This paper addresses the challenges of deploying reinforcement learning (RL) models for traffic signal control (TSC) in real-world environments. Real-world training can prevent mismatches between simulation environments and the actual traffic conditions, thereby achieving better performance of agent upon deployment. However, free explorations by agents during real-world training can disrupt traffic operations. To mitigate this, this paper proposes a reference mechanism to guide the decision-making process within the RL framework. A reference timing policy, typically a model-based signal strategy, is integrated into the learning process to provide agents with reference actions. Specifically, an additional Q-value function is introduced to evaluate both the agent's actions and those of the reference policy, allowing for adjustments before the actions are executed in real traffic system. Numerical results indicate that the reference mechanism effectively enhances system performance early in the training process, thus accelerating learning. We also combine the reference RL method with a pretraining procedure and a jump-start algorithm, respectively. Experimental results demonstrate their effectiveness in further enhancing system performance and facilitating real-world training.

@en

Drivers initiate a discretionary lane change when they perceive an anticipated improvement in their own driving condition from moving to another lane. However, such a lane change can slow down other vehicles on the target lane, and even worse initiate a disturbance. In this work, we argue that the blocking effect triggered by individual lane changes results from the heterogeneity in the desired speeds of vehicles, and thus using desired speed information of vehicles when regulating lane-changing decisions can improve traffic efficiency. In doing so, our work also exemplifies the usefulness of incorporating user preferences into control decisions. The proposed lane guidance system uses an optimization-based approach to update the target range of desired speeds on each lane in real time, and accordingly recommends individual lane changes. The control system coordinates the lane-changing decisions at the link level, for which the road stretch is subdivided into multiple sections that are controlled independently. We evaluate the performance of the lane guidance system in micro-simulation, for different network demands and desired speed distributions. The results highlight that the proposed approach utilizing the desired speed preferences of drivers results in positive efficiency gains for most traffic compositions in free flow. Moreover, the highest gains are expected in medium to high demand, and when the traffic composition includes a higher proportion of vehicles desiring higher speeds. The gains also increase when the desired speeds of vehicles that want to drive fast and those that want to drive slower are more separated.

@en

Multimodal arterial signal coordination for buses and passenger vehicles can improve arterial travel smoothness and efficiency. However, uncertainty in bus operations requires signal priority at intersections, which impacts coordination and increases stop times for other traffic types. Therefore, this study proposes a stochastic priority-integrated signal coordination (SPIC) method. It includes an offline stochastic programme to determine the arterial signal coordination, i.e. cycle length and offsets, considering the stochastic signal priority, and an online mixed-integer nonlinear programme to determine the signal priority together with the bus arrival and departure times at and from stops and intersections in a connected vehicle environment. A scenario-based heuristic algorithm is proposed to solve the SPIC efficiently. Numerical studies have validated that SPIC can improve the efficiency of buses and passenger vehicles. Sensitivity analyses show that the SPIC effectively reduces delays with fluctuations in the bus travel time, dwell time, and passenger vehicle demands.

@en

Model-based predictive signal control is a popular method to pro-actively control traffic and to reduce the effects of congestion in urban networks. In combination with structure-free controllers, which adapt signal settings in arbitrary order and combination (no imposed cycles), these predictive control methods have a high potential to increase system performance by adapting to individual vehicle patterns, which are increasingly available due to new technology. However, most of these control methods assume perfect predictions, while in practice there are prediction errors due to various reasons. In this paper, the sensitivity of the system performance to these prediction errors is analyzed, for an urban corridor with spillback. In a microscopic simulator, first the ideal world is created for the structure-free model-based predictive signal controller, in which perfect predictions are made and the controller can reach its optimal performance. Then prediction errors are introduced in this perfect world, distinguished in aggregation errors that arise using a macroscopic prediction model and biases that represent structural errors in the prediction model or in its demand and state input. The effects of these prediction errors on the system performance are analyzed, as a function of the prediction horizon and update frequency of the control system. The results show that, even under errors, longer prediction horizons lead to better performance, up to a certain optimal prediction horizon length. A high update frequency dampens the influence of prediction errors, enabling the structure-free controller to correct mistakes faster. However, there remains a significant performance loss due to aggregation errors and biases in the prediction model, indicating a promising performance gain of more reliable predictions and the incorporation of information on individual vehicles in future control applications. Moreover, for all model quantities one direction of the bias has more impact on the system performance than the other direction, indicating guidelines towards a more robust control system that suffers less from erroneous predictions.

@en

Conventional reinforcement learning (RL) models of variable speed limit (VSL) control systems (and traffic control systems in general) cannot be trained in real traffic process because new control actions are usually explored randomly, which may result in high costs (delays) due to exploration and learning. For this reason, existing RL-based VSL control approaches need a traffic simulator for training. However, the performance of those approaches are dependent on the accuracy of the simulators. This paper proposes a new RL-based VSL control approach to overcome the aforementioned problems. The proposed VSL control approach is designed to improve traffic efficiency by using VSLs against freeway jam waves. It applies an iterative training framework, where the optimal control policy is updated by exploring new control actions both online and offline in each iteration. The explored control actions are evaluated in real traffic process, thus it avoids that the RL model learns only from a traffic simulator. The proposed VSL control approach is tested using a macroscopic traffic simulation model to represent real world traffic flow dynamics. By comparing with existing VSL control approaches, the proposed approach is demonstrated to have advantages in the following two aspects: (i) it alleviates the impact of model mismatch, which occurs in both model-based VSL control approaches and existing RL-based VSL control approaches, via replacing knowledge from the models by knowledge from the real process, and (ii) it significantly reduces the exploration and learning costs compared to existing RL-based VSL control approaches.

@en

The literature on green mobility and eco-driving in urban areas has burgeoned in recent years, with special attention to using infrastructure to vehicle (I2V) communications to obtain optimal speed trajectory which minimize the economic and environmental costs. This article shares the concept with these studies but turns the spotlight on cyclists. It examines the problem of finding optimal speed trajectory for a cyclist in signalised urban areas. Unlike the available studies on motorised vehicles which predominantly designed for pre-defined, fixed traffic lights timing, this article uses an algorithm based on stochastic dynamic programming to explicitly address uncertainty in traffic light timing. Moreover, through a comprehensive set of simulation experiments, the article examines the impact of the speed advice's starting point as well as the cyclist's willingness for changing his/her speed on enhancing the performance. The proposed approach targets various performance metrics such as minimising the total travel time, energy consumption, or the probability of stopping at a red light. Hence, the resulting speed advice can be tailored according to the personal preferences of each cyclist. In a simulation case study, the results of the proposed approach is also compared with an existing approach in the literature.

@en

Signalized traffic control is important in traffic management to reduce congestion in urban areas. With recent technological developments, more data have become available to the controllers and advanced state estimation and prediction methods have been developed that use these data. To fully benefit from these techniques in the design of signalized traffic controllers, it is important to look at the quality of the estimated and predicted input quantities in relation to the performance of the controllers. Therefore, in this paper, a general framework for sensitivity analysis is proposed, to analyze the effect of erroneous input quantities on the performance of different types of signalized traffic control. The framework is illustrated for predictive control with different adaptivity levels. Experimental relations between the performance of the control system and the prediction horizon are obtained for perfect and erroneous predictions. The results show that prediction improves the performance of a signalized traffic controller, even in most of the cases with erroneous input data. Moreover, controllers with high adaptivity seem to outperform controllers with low adaptivity, under both perfect and erroneous predictions. The outcome of the sensitivity analysis contributes to understanding the relations between information quality and performance of signalized traffic control. In the design phase of a controller, this insight can be used to make choices on the length of the prediction horizon, the level of adaptivity of the controller, the representativeness of the objective of the control system, and the input quantities that need to be estimated and predicted the most accurately.

@en

In this paper a novel computationally efficient model predictive control (MPC) method for optimizing flows at urban intersections is proposed. Several linear and quadratic MPC approaches have been developed in the literature to reduce the computational complexity of the problem, but without considering the back-propagating waves associated with spillback. As the principal contribution of this work, a linear optimization problem for an MPC approach is formulated, which considers downstream propagating waves linked to free-flow traffic, queuing dynamics, and upstream propagating waves related to spillback (i.e. forward and backward moving waves, respectively). The linear optimization problem is obtained by describing link dynamics using the link transmission model, and aggregating the traffic dynamics to (several) tens of seconds. The performance of the proposed controller is compared with two other existing strategies; a store-and-forward model-based, and a cell transmission model-based approach. The total time spent (TTS) by all the vehicles in the network and the computation time is applied as performance indexes for the evaluation of the control strategies. Simulation results show that including upstream propagating waves results in better controller performance, due to the explicit modeling of the impact of link outflow on the maximum link inflow.

@en

This paper develops a model-based hierarchical control method for coordinated ramp metering on freeway networks with multiple bottlenecks and on- and off-ramps. The controller consists of two levels where at the upper level, a Model Predictive Control (MPC) approach is developed to optimize total network travel time by manipulating total inflow from on-ramps to the freeway network. The lower level controller distributes the optimal total inflows to each on-ramp of the freeway based on local traffic state feedback. The control method is based on a parsimonious aggregated traffic model that relates the freeway total outflow to the number of vehicles on the freeway sections. Studies on aggregated traffic modeling of networks have shown the existence of a well-defined and low-scatter Macroscopic Fundamental Diagram (MFD) for urban networks. The MFD links network aggregated flow and density (accumulation). However, the MFD of freeway networks typically exhibits high scatter and hysteresis loops that challenge the control performance of MFD-based controllers for freeways. This paper addresses these challenges by modelling the effect of density heterogeneity along the freeway and capacity drop on characteristics of freeway MFD using field traffic data. In addition, we introduce a model to predict the evolution of density heterogeneity that is essential to reproduce the dynamics of freeway MFD accurately. The proposed model is integrated as the prediction model of the MPC in the hierarchical control method. The proposed coordinated ramp metering method shows desirable performance to reduce the vehicles total time spent and eliminate congestion. The control approach is compared with other coordinated ramp metering controllers based on the MPC framework with different traffic prediction models (e.g. CTM and METANET). The outcomes of numerical experiments highlight that the MFD-based hierarchical controller (i) is better able to overcome the modeling mismatch between the prediction model and the plant (process model) in the MPC framework and (ii) requires less computation effort than other nonlinear controllers.

@en

Freeway congestion can reduce the freeway throughput due to the capacity drop or due to blocking caused by spillback to upstream ramps. Research has shown that congestion can be reduced by the application of ramp metering and variable speed limits. Model predictive control is a promising strategy for the optimization of the ramp metering rates and variable speed limits to improve the freeway throughput. However, several challenges have to be addressed before it can be applied for the control of freeway traffic. This paper focuses on the challenge of reducing the computation time of MPC strategies for the integration of variable speed limits and ramp metering. This is realized via a parameterized control strategy that optimizes the upstream and downstream boundaries of a speed-limited area and the parameters of the ALINEA ramp metering strategy. Due to the parameterization, the solution space reduces substantially, leading to an improved computation time. More specifically, the number of optimization variables for the variable speed limit strategy becomes independent of the number of variable message signs, and the number of optimization variables for the ramp metering strategy becomes independent of the prediction horizon. The control strategy is evaluated with a macroscopic model of a two-lane freeway with two ON-ramps and OFF-ramps. It is shown that parameterization realizes improved throughput when compared with a non-parameterized strategy when using the same amount of computation time.

@en

Intelligent vehicle technologies are opening new possibilities for decentralized vehicle routing systems, suitable for regulating large traffic networks, and at the same time, capable of providing customized advice to individual vehicles. In this study, we perform a rigorous simulation-based analysis of an in-vehicle routing strategy that aims to achieve a user-equilibrium distribution in traffic. Novel features of the approach include: a mechanism based on forward propagation of individual vehicle decisions to anticipate future traffic dynamics; time-dependent prediction of route travel times with neural network-based link predictors; and a stochastic routing policy for in-vehicle decision-making based on predicted travel times. However, for an effective application of the approach, design choices need to be made regarding the accuracy of the link predictors, and some control settings. These choices may depend on the network size and structure. We investigate the impact of two important design aspects: sequentially using link-level predictors for route travel time estimation, and the control parameter values, on the equilibrium performance at the network-level. The results suggest functional scalability of the approach, in terms of the prediction model accuracy and routing performance. Overall, the work contributes to a qualitative and quantitative understanding of emergent performance from the given routing approach.

@en

Although there exists algorithms that give speed advice for cyclists when approaching traffic lights with uncertainty in the timing, they all need to know, and thus assume, the cyclist's response to the advice in order to be able to optimize the advice. To relax this assumption, in this paper an algorithm is proposed that combines reinforcement learning and planning to learn the reaction of cyclist to the advice and deploys this information for planning the best next advice on-the-fly. Rather than a single search procedure, which is conventional in the existing architectures, two sample-based search procedures are suggested to be used in the algorithm. This makes it possible to obtain an accurate local approximation of the action-value function, in spite of the short computation time that is available in each decision epoch. The algorithm is tested in a simulation case study where the impact of a proper initialisation of action-value function as well as the importance of using two search procedures are affirmed.

@en

Signalized intersections are one of the most common sources of inconvenience for cyclists. The aim of this paper is to develop an approach that helps cyclists to meet their cycling preferences (regarding, e.g., the energy they use and their preference to avoid unnecessary stopping) while crossing intersections. Uncertainty in traffic light timing is considered explicitly in the approach, which makes it applicable for intersections with traffic-responsive signals. The suggested approach provides cyclists with optimal and personalized speed advice. The advice is communicated to the cyclists through a roadside sign located upstream of the intersection. It is assumed that the roadside sign can measure the speed of the approaching bike and can also communicate with the traffic light to ascertain the traffic light’s state and give advice accordingly. To consider the behavior of cyclists when they disregard the advice as they get close to the intersection, the problem is separated in two parts and formulated as a Markov reward process combined with a Markov decision process and stochastic dynamic programming is used to solve the corresponding optimization problem. The approach is generic in relation to the underlying process model and the objective function. The results of an illustrative case study show how much improvement, in relation to the cyclist’s average number of stops, and average energy consumption, could be achieved by use of the suggested approach in a simulated intersection. We also investigate how the location of the sign may affect the performance of the approach.@en

In this paper, we develop a hierarchical approach to optimize the signal timings in an urban traffic network taking into account the different dynamics in all traffic regimes. The proposed hierarchical control framework consists of two layers. The first layer--the network coordination layer--uses a model predictive control strategy based on a simplified traffic flow model to provide reference outflow trajectories. These reference outflow trajectories represent average desired link outflows over time. These are then mapped to green-red switching signals which can be applied to traffic lights. To this end, the second layer--the individual intersection control layer--then selects at every intersection the signal timing stage that realizes an outflow which has the smallest error with respect to the reference outflow trajectory. The proposed framework is tested using both macroscopic and microscopic simulations. It is shown that the control framework can outperform a greedy control policy that maximizes the individual intersection outflows, and the control framework can distribute the queues over the network in a way that the network outflow is improved. Simulations using a macroscopic model allow the direct application of the reference outflows computed by the network coordination layer, and the results indicate that the mapping of the reference outflows to the detailed signal timings by the individual intersection control layer only introduces a small performance loss.

@en

When travelling in urban areas with signalised intersections, cyclists are currently unable to optimize their speed profile according to their preferences. By integrating information from traffic signal phase and timing and cyclists' preferences, in this paper an algorithm is developed that generates personalised optimal speed advice for cyclists. Results from the simulation experiment indicate that in spite of stochastic nature of traffic light phase and timing, with the developed algorithm based on stochastic dynamic programming, an speed profile can be advised that meets the cyclist's preferences in terms of travel time, energy consumption, or the number of stops.@en

This paper extends an existing linear quadratic model predictive control (LQMPC) approach to multi-destination traffic networks, where the correct origin-destination (OD) relations are preserved. In the literature, the LQMPC approach has been presented for efficient routing and intersection signal control. The optimization problem in the LQMPC has a linear quadratic formulation that can be solved quickly, which is beneficial for a real-time application. However, the existing LQMPC approach does not preserve OD relations and thus may send traffic to wrong destinations. This problem is tackled by a heuristic method presented is this paper. We present two macroscopic models: 1) a non-linear route-specific model which keeps track of traffic dynamics for each OD pair and 2) a linear model that aggregates all route traffic states, which can be embedded into the LQMPC framework. The route-specific model predicts traffic dynamics and provides information to the LQMPC before the optimization and evaluates the optimal solutions after the optimization. The information obtained from the route-specific model is formulated as constraints in the LQMPC to narrow the solution space and exclude unrealistic solutions that would lead to flows that are inconsistent with the OD relations. The extended LQMPC approach is tested in a synthetic network with multiple bottlenecks. The simulation of the LQMPC approach achieves a total time spent close to the system optimum, and the computation time remains tractable.

@en

In the area of active traffic management, new technologies provide opportunities to improve the use of current infrastructure. Vehicles equipped with in-car communication systems are capable of exchanging messages with the infrastructure and other vehicles. This new capability offers many opportunities for traffic management. This paper presents a novel merging assistant strategy that exploits the communication capabilities of intelligent vehicles. The proposed control requires the cooperation of equipped vehicles on the main carriageway in order to create merging gaps for on-ramp vehicles released by a traffic light. The aim is to reduce disruptions to the traffic flow created by the merging vehicles. This paper focuses on the analytical formulation of the control algorithm, and the traffic flow theories used to define the strategy. The dynamics of the gap formation derived from theoretical considerations are validated using a microscopic simulation. The validation indicates that the control strategy mostly developed from macroscopic theory well approximates microscopic traffic behaviour. The results present encouraging capabilities of the system. The size and frequency of the gaps created on the main carriageway, and the space and time required for their creation are compatible with a real deployment of the system. Finally, we summarise the results of a previous study showing that the proposed merging strategy reduces the occurrence of congestion and the number of late-merging vehicles. This innovative control strategy shows the potential of using intelligent vehicles for facilitating the merging manoeuvre through use of emerging communications technologies.

@en

In this paper we develop a fast model predictive control (MPC) approach for variable speed limit coordination to resolve freeway jam waves. Existing MPC approaches that are based on the second-order traffic flow models suffer from high computation load due to the non-linear and non-convex optimization formulation. In recent years, simplified MPC approaches which are based on discrete first-order traffic flow models have attracted more and more attention because they are beneficial for real-time applications. In literature, the type of traffic jam resolved by these approaches is limited to the standing queue in which the jam head is fixed at the bottleneck. Another type of traffic jam known as the jam wave, has been neglected by the discrete first-order model-based MPC approaches. To fill this gap, we develop a fast MPC approach based on a more accurate discrete first-order model. The model keeps the linear property of the classical discrete first-order model, meanwhile takes traffic flow features of jam waves propagation into consideration. A classical non-linear MPC and a recently proposed linear MPC are compared with the proposed MPC in terms of computation speed and jam wave resolution by a benchmark problem. Simulation results show that the proposed MPC resolves the jam wave with a real-time feasible computation speed.

@en

This paper validates the prediction model embedded in a model predictive controller (MPC) of variable speed limits (VSLs). The MPC controller was designed based on an extended discrete first-order model with a triangular fundamental diagram. In our previous work, the extended discrete first-order model was designed to reproduce the capacity drop and the propagation of jam waves, and it was validated with reasonable accuracy without the presence of VSLs. As VSLs influence traffic dynamics, the dynamics including VSLs needs to be validated, before it can be applied as a prediction model in MPC. For conceptual illustrations, we use two synthetic examples to show how the model reproduces the key mechanisms of VSLs that are applied by existing VSL control approaches. Furthermore, the model is calibrated by use of real traffic data from Dutch freeway A12, where the field test of a speed limit control algorithm (SPECIALIST) was conducted. In the calibration, the original model is extended by using a quadrangular fundamental diagram which keeps the linear feature of the model and represents traffic states at the under-critical branch more accurately. The resulting model is validated using various traffic data sets. The accuracy of the model is compared with a second-order traffic flow model. The performance of two models is comparable: both models reproduce accurate results matching with real data. Flow errors of the calibration and validation are around 10%. The extended discrete first-order model-based MPC controller has been demonstrated to resolve freeway jam waves efficiently by synthetic cases. It has a higher computation speed comparing to the second-order model-based MPC.

@en