This paper addresses the challenges of deploying reinforcement learning (RL) models for traffic signal control (TSC) in real-world environments. Real-world training can prevent mismatches between simulation environments and the actual traffic conditions, thereby achieving better performance of agent upon deployment. However, free explorations by agents during real-world training can disrupt traffic operations. To mitigate this, this paper proposes a reference mechanism to guide the decision-making process within the RL framework. A reference timing policy, typically a model-based signal strategy, is integrated into the learning process to provide agents with reference actions. Specifically, an additional Q-value function is introduced to evaluate both the agent's actions and those of the reference policy, allowing for adjustments before the actions are executed in real traffic system. Numerical results indicate that the reference mechanism effectively enhances system performance early in the training process, thus accelerating learning. We also combine the reference RL method with a pretraining procedure and a jump-start algorithm, respectively. Experimental results demonstrate their effectiveness in further enhancing system performance and facilitating real-world training.

Urged by a close future perspective of a traffic flow made of a mix of human-driven vehicles and automated vehicles (AVs), research has recently focused on studying the traffic flow characteristics of Adaptive Cruise Controls (ACCs), the most typical AV. However, in most works, the ACC system is studied under a simplifying and unrealistic assumption, or the ACC system modeled is inaccurate. This paper proposes a general hierarchical control system to model ACC systems with several assumptions based on the deficiencies above. Moreover, a field experiment was conducted, and the corresponding experimental data was used to verify the proposed hierarchical control system and assumptions. In addition, string stability is explored along with sensitivity analyses of control parameters based on an example under the constant time gap policy. The results show that different upper-level controller parameters have different delays, where the delay of the speed is negligible; the introduction of actuator delay and lag in the lower-level controller can significantly improve the model goodness of fit. Furthermore, optimizing the delay and lag in the lower-level controller can significantly enhance the string stability of ACCs than optimizing the control parameters.