Human-like autonomous car-following model with deep reinforcement learning

This study proposes a framework for human-like autonomous car-following planning based on deep reinforcement learning (deep RL). Historical driving data are fed into a simulation environment where an RL agent learns from trial and error interactions based on a reward function that signals how much the agent deviates from the empirical data. Through these interactions, an optimal policy, or car-following model that maps in a human-like way from speed, relative speed between a lead and following vehicle, and inter-vehicle spacing to acceleration of a following vehicle is finally obtained. The model can be continuously updated when more data are fed in. Two thousand car-following periods extracted from the 2015 Shanghai Naturalistic Driving Study were used to train the model and compare its performance with that of traditional and recent data-driven car-following models. As shown by this study’s results, a deep deterministic policy gradient car-following model that uses disparity between simulated and observed speed as the reward function and considers a reaction delay of 1 s, denoted as DDPGvRT, can reproduce human-like car-following behavior with higher accuracy than traditional and recent data-driven car-following models. Specifically, the DDPGvRT model has a spacing validation error of 18% and speed validation error of 5%, which are less than those of other models, including the intelligent driver model, models based on locally weighted regression, and conventional neural network-based models. Moreover, the DDPGvRT demonstrates good capability of generalization to various driving situations and can adapt to different drivers by continuously learning. This study demonstrates that reinforcement learning methodology can offer insight into driver behavior and can contribute to the development of human-like autonomous driving algorithms and traffic-flow models.     

A simple nonparametric car-following model driven by field data

Car-following models are always of great interest of traffic engineers and researchers. In the age of mass data, this paper proposes a nonparametric car-following model driven by field data. Different from most of the existing car-following models, neither driver’s behaviour parameters nor fundamental diagrams are assumed in the data-driven model. The model is proposed based on the simple k-nearest neighbour, which outputs the average of the most similar cases, i.e., the most likely driving behaviour under the current circumstance. The inputs and outputs are selected, and the determination of the only parameter k is introduced. Three simulation scenarios are conducted to test the model. The first scenario is to simulate platoons following real leaders, where traffic waves with constant speed and the detailed trajectories are observed to be consistent with the empirical data. Driver’s rubbernecking behaviour and driving errors are simulated in the second and third scenarios, respectively. The time–space diagrams of the simulated trajectories are presented and explicitly analysed. It is demonstrated that the model is able to well replicate periodic traffic oscillations from the precursor stage to the decay stage. Without making any assumption, the fundamental diagrams for the simulated scenario coincide with the empirical fundamental diagrams. These all validate that the model can well reproduce the traffic characteristics contained by the field data. The nonparametric car-following model exhibits traffic dynamics in a simple and parsimonious manner.     

Vehicle headway modeling and its inferences in macroscopic/microscopic traffic flow theory: A survey

Vehicle headway distribution models are widely used in traffic engineering fields, since they reflect the fundamental uncertainty in drivers' car-following maneuvers and meanwhile provide a concise way to describe the stochastic feature of traffic flows. This paper presents a systematic review of vehicle headway distribution studies in the last few decades. Since it is impossible to enumerate the merits and drawbacks of all of existing distribution models, we emphasize four advances of headway distribution modeling in this paper. First, we highlight the chronicle of key assumptions on the existing distribution models and explain why this evolution occurs. Second, we show that departure headways measured for interrupted flows on urban streets and headways measured for uninterrupted flows on freeways have common features and can be simulated by a unified microscopic car-following model. The interesting finding helps gather two kinds of headway distribution models under one umbrella. Third, we review different approaches that aim to link microscopic car-following models and mesoscopic vehicle headway distribution models. Fourth, we show that both the point scattering on the density-flow plot and the shape of traffic flow breakdown curve implicitly depend on the vehicular headway distribution. These findings reveal pervasive connections between macroscopic traffic flow models and mesoscopic headway distribution. All these new insights bring new vigor into vehicle headway studies and open research frontiers in this field.     

A multi-commodity Lighthill–Whitham–Richards model of lane-changing traffic flow

Systematic lane changes can seriously deteriorate traffic safety and efficiency inside lane-drop, merge, and other bottleneck areas. In our previous studies (Jin, 2010a, Jin, 2010b), a phenomenological model of lane-changing traffic flow was proposed, calibrated, and analyzed based on a new concept of lane-changing intensity. In this study, we further consider weaving and non-weaving vehicles as two commodities and develop a multi-commodity, behavioral Lighthill–Whitham–Richards (LWR) model of lane-changing traffic flow. Based on a macroscopic model of lane-changing behaviors, we derive a fundamental diagram with parameters determined by car-following and lane-changing characteristics as well as road geometry and traffic composition. We further calibrate and validate fundamental diagrams corresponding to a triangular car-following fundamental diagram with NGSIM data. We introduce an entropy condition for the multi-commodity LWR model and solve the Riemann problem inside a homogeneous lane-changing area. From the Riemann solutions, we derive a flux function in terms of traffic demand and supply. Then we apply the model to study lane-changing traffic dynamics inside a lane-drop area and show that the smoothing effect of HOV lanes is consistent with observations in existing studies. The new theory of lane-changing traffic flow can be readily incorporated into Cell Transmission Model, and this study could lead to better strategies for mitigating bottleneck effects of lane-changing traffic flow.     

A probabilistic stationary speed–density relation based on Newell’s simplified car-following model

Probabilistic models describing macroscopic traffic flow have proven useful both in practice and in theory. In theoretical investigations of wide-scatter in flow–density data, the statistical features of flow density relations have played a central role. In real-time estimation and traffic forecasting applications, probabilistic extensions of macroscopic relations are widely used. However, how to obtain such relations, in a manner that results in physically reasonable behavior has not been addressed. This paper presents the derivation of probabilistic macroscopic traffic flow relations from Newell’s simplified car-following model. The probabilistic nature of the model allows for investigating the impact of driver heterogeneity on macroscopic relations of traffic flow. The physical features of the model are verified analytically and shown to produce behavior which is consistent with well-established traffic flow principles. An empirical investigation is carried out using trajectory data from the New Generation SIMulation (NGSIM) program and the model’s ability to reproduce real-world traffic data is validated.     

Online calibration for microscopic traffic simulation and dynamic multi-step prediction of traffic speed

Simulating driving behavior in high accuracy allows short-term prediction of traffic parameters, such as speeds and travel times, which are basic components of Advanced Traveler Information Systems (ATIS). Models with static parameters are often unable to respond to varying traffic conditions and simulate effectively the corresponding driving behavior. It has therefore been widely accepted that the model parameters vary in multiple dimensions, including across individual drivers, but also spatially across the network and temporally. While typically on-line, predictive models are macroscopic or mesoscopic, due to computational and data considerations, nowadays microscopic models are becoming increasingly practical for dynamic applications. In this research, we develop a methodology for online calibration of microscopic traffic simulation models for dynamic multi-step prediction of traffic measures, and apply it to car-following models, one of the key models in microscopic traffic simulation models. The methodology is illustrated using real trajectory data available from an experiment conducted in Naples, using a well-established car-following model. The performance of the application with the dynamic model parameters consistently outperforms the corresponding static calibrated model in all cases, and leads to less than 10% error in speed prediction even for ten steps into the future, in all considered data-sets.     

A car-following model considering asymmetric driving behavior based on long short-term memory neural networks

Asymmetric driving behavior is a critical characteristic of human driving behaviors and has a significant impact on traffic flow. In consideration of the asymmetric driving behavior, this paper proposes a long short-term memory (LSTM) neural networks (NN) based car-following (CF) model to capture realistic traffic flow characteristics by incorporating the driving memory. The NGSIM data are used to calibrate and validate the proposed CF model. Meanwhile, three characteristics closely related to the asymmetric driving behavior are investigated: hysteresis, discrete driving, and intensity difference. The simulation results show the good performance of the proposed CF model on reproducing realistic traffic flow features. Moreover, to further demonstrate the superiority of the proposed CF model, two other CF models including recurrent neural network based CF model and asymmetric full velocity difference model, are compared with LSTM-NN model. The results reveal that LSTM-NN model can capture the asymmetric driving behavior well and outperforms other models.     

Incorporating human-factors in car-following models: A review of recent developments and research needs

Over the past decades there has been a considerable development in the modeling of car-following (CF) behavior as a result of research undertaken by both traffic engineers and traffic psychologists. While traffic engineers seek to understand the behavior of a traffic stream, traffic psychologists seek to describe the human abilities and errors involved in the driving process. This paper provides a comprehensive review of these two research streams.It is necessary to consider human-factors in CF modeling for a more realistic representation of CF behavior in complex driving situations (for example, in traffic breakdowns, crash-prone situations, and adverse weather conditions) to improve traffic safety and to better understand widely-reported puzzling traffic flow phenomena, such as capacity drop, stop-and-go oscillations, and traffic hysteresis. While there are some excellent reviews of CF models available in the literature, none of these specifically focuses on the human factors in these models.This paper addresses this gap by reviewing the available literature with a specific focus on the latest advances in car-following models from both the engineering and human behavior points of view. In so doing, it analyses the benefits and limitations of various models and highlights future research needs in the area.     

Calibration of nonlinear car-following laws for traffic oscillation prediction

Frequency-domain analysis has been successfully used to (i) predict the amplification of traffic oscillations along a platoon of vehicles with nonlinear car-following laws and (ii) measure traffic oscillation properties (e.g., periodicity, magnitude) from field data. This paper proposes a new method to calibrate nonlinear car-following laws based on real-world vehicle trajectories, such that oscillation prediction (based on the calibrated car-following laws) and measurement from the same data can be compared and validated. This calibration method, for the first time, takes into account not only the driver’s car-following behavior but also the vehicle trajectory’s time-domain (e.g., location, speed) and frequency-domain properties (e.g., peak oscillation amplitude). We use Newell’s car-following model (1961) as an example and calibrate its parameters based on a penalty-based maximum likelihood estimation procedure. A series of experiments using Next Generation Simulation (NGSIM) data are conducted to illustrate the applicability and performance of the proposed approach. Results show that the calibrated car-following models are able to simultaneously reproduce observed driver behavior, time-domain trajectories, and oscillation propagation along the platoon with reasonable accuracy.     

Modeling car-following behavior on urban expressways in Shanghai: A naturalistic driving study

Although car-following behavior is the core component of microscopic traffic simulation, intelligent transportation systems, and advanced driver assistance systems, the adequacy of the existing car-following models for Chinese drivers has not been investigated with real-world data yet. To address this gap, five representative car-following models were calibrated and evaluated for Shanghai drivers, using 2100 urban-expressway car-following periods extracted from the 161,055 km of driving data collected in the Shanghai Naturalistic Driving Study (SH-NDS). The models were calibrated for each of the 42 subject drivers, and their capabilities of predicting the drivers’ car-following behavior were evaluated.The results show that the intelligent driver model (IDM) has good transferability to model traffic situations not presented in calibration, and it performs best among the evaluated models. Compared to the Wiedemann 99 model used by VISSIM®, the IDM is easier to calibrate and demonstrates a better and more stable performance. These advantages justify its suitability for microscopic traffic simulation tools in Shanghai and likely in other regions of China. Additionally, considerable behavioral differences among different drivers were found, which demonstrates a need for archetypes of a variety of drivers to build a traffic mix in simulation. By comparing calibrated and observed values of the IDM parameters, this study found that (1) interpretable calibrated model parameters are linked with corresponding observable parameters in real world, but they are not necessarily numerically equivalent; and (2) parameters that can be measured in reality also need to be calibrated if better trajectory reproducing capability are to be achieved.     

Modelling supported driving as an optimal control cycle: Framework and model characteristics

Driver assistance systems support drivers in operating vehicles in a safe, comfortable and efficient way, and thus may induce changes in traffic flow characteristics. This paper puts forward a receding horizon control framework to model driver assistance and cooperative systems. The accelerations of automated vehicles are controlled to optimise a cost function, assuming other vehicles driving at stationary conditions over a prediction horizon. The flexibility of the framework is demonstrated with controller design of Adaptive Cruise Control (ACC) and Cooperative ACC (C-ACC) systems. The proposed ACC and C-ACC model characteristics are investigated analytically, with focus on equilibrium solutions and stability properties. The proposed ACC model produces plausible human car-following behaviour and is unconditionally locally stable. By careful tuning of parameters, the ACC model generates similar stability characteristics as human driver models. The proposed C-ACC model results in convective downstream and absolute string instability, but not convective upstream string instability observed in human-driven traffic and in the ACC model. The control framework and analytical results provide insights into the influences of ACC and C-ACC systems on traffic flow operations.     

Evaluation of the General Motors based car-following models and a proposed fuzzy inference model

This paper evaluates the properties of the General Motors (GM) based car-following models, identifies their characteristics, and proposes a fuzzy inference logic based model that can overcome some of the shortcomings of the GM based models. This process involves developing a framework for evaluating a car-following model and comparing the behavior predicted by the GM models with the behavior observed under the real world situation. For this purpose, an instrumented vehicle was used to collect data on the headway and speeds of two consecutive vehicles under actual traffic conditions. Shortcomings of the existing GM based models are identified, in particular, the stability conditions were analyzed in detail. A fuzzy-inference based model of car-following is developed to represent the approximate nature of stimulus–response process during driving. This model is evaluated using the same evaluation framework used for the GM models and the data obtained by the instrumented test vehicle. Comparison between the performance of the two models show that the proposed fuzzy inference model can overcome many shortcomings of the GM based car-following models, and can be useful for developing the algorithm for the adaptive cruise control for automated highway system (AHS).     

Speed or spacing? Cumulative variables,and convolution of model errors and time in traffic flow models validation and calibration

This paper proves that in traffic flow model calibration and validation the cumulative sum of a variable has to be preferred to the variable itself as a measure of performance. As shown through analytical relationships, model residuals dynamics are preserved if discrepancy measures of a model against reality are calculated on a cumulative variable, rather than on the variable itself. Keeping memory of model residuals occurrence times is essential in traffic flow modelling where the ability of reproducing the dynamics of a phenomenon – as a bottleneck evolution or a vehicle deceleration profile – may count as much as the ability of reproducing its order of magnitude. According to the aforesaid finding, in a car-following models context, calibration on travelled space is more robust than calibration on speed or acceleration. Similarly in case of macroscopic traffic flow models validation and calibration, cumulative flows are to be preferred to flows. Actually, the findings above hold for any dynamic model.     

A high fidelity traffic simulation model based on cellular automata and car-following concepts

A high fidelity cell based traffic simulation model (CELLSIM) has been developed for simulation of high volume of traffic at the regional level. Straightforward algorithms and efficient use of computational resources make the model suitable for real time traffic simulation. The model formulation uses concepts of cellular automata (CA) and car-following (CF) models, but is more detailed than CA models and has realistic acceleration and deceleration models for vehicles. A simple dual-regime constant acceleration model has been used that requires minimal calculation compared to detailed acceleration models used in CF models. CELLSIM is simpler than most CF models; a simplified car-following logic has been developed using preferred time headway. Like CA models, integer values are used to make the model run faster. Space is discretized in small intervals and a new concept of percent space occupancy (SOC) is used to measure traffic congestion. CELLSIM performs well in congested and non-congested traffic conditions. It has been validated comprehensively at the macroscopic and microscopic levels using two sets of field data. Comparison of field data and CELLSIM for trajectories, average speed, density and volume show very close agreement. Statistical comparison of macroscopic parameters with other CF models indicates that CELLSIM performs as good as detailed CF models. Stability analyses conducted using mild and severe disturbances indicate that CELLSIM performs well under both conditions.     

Short-term traffic flow rate forecasting based on identifying similar traffic patterns

The ability to timely and accurately forecast the evolution of traffic is very important in traffic management and control applications. This paper proposes a non-parametric and data-driven methodology for short-term traffic forecasting based on identifying similar traffic patterns using an enhanced K-nearest neighbor (K-NN) algorithm. Weighted Euclidean distance, which gives more weight to recent measurements, is used as a similarity measure for K-NN. Moreover, winsorization of the neighbors is implemented to dampen the effects of dominant candidates, and rank exponent is used to aggregate the candidate values. Robustness of the proposed method is demonstrated by implementing it on large datasets collected from different regions and by comparing it with advanced time series models, such as SARIMA and adaptive Kalman Filter models proposed by others. It is demonstrated that the proposed method reduces the mean absolute percent error by more than 25%. In addition, the effectiveness of the proposed enhanced K-NN algorithm is evaluated for multiple forecast steps and also its performance is tested under data with missing values. This research provides strong evidence suggesting that the proposed non-parametric and data-driven approach for short-term traffic forecasting provides promising results. Given the simplicity, accuracy, and robustness of the proposed approach, it can be easily incorporated with real-time traffic control for proactive freeway traffic management.     

Comparison of parametric and nonparametric models for traffic flow forecasting

Single point short-term traffic flow forecasting will play a key role in supporting demand forecasts needed by operational network models. Seasonal autoregressive integrated moving average (ARIMA), a classic parametric modeling approach to time series, and nonparametric regression models have been proposed as well suited for application to single point short-term traffic flow forecasting. Past research has shown seasonal ARIMA models to deliver results that are statistically superior to basic implementations of nonparametric regression. However, the advantages associated with a data-driven nonparametric forecasting approach motivate further investigation of refined nonparametric forecasting methods. Following this motivation, this research effort seeks to examine the theoretical foundation of nonparametric regression and to answer the question of whether nonparametric regression based on heuristically improved forecast generation methods approach the single interval traffic flow prediction performance of seasonal ARIMA models.     

Characterization of traffic oscillation propagation under nonlinear car-following laws

Unlike linear car-following models, nonlinear models generally can generate more realistic traffic oscillation phenomenon, but nonlinearity makes analytical quantification of oscillation characteristics (e.g, periodicity and amplitude) significantly more difficult. This paper proposes a novel mathematical framework that accurately quantifies oscillation characteristics for a general class of nonlinear car-following laws. This framework builds on the describing function technique from nonlinear control theory and is comprised of three modules: expression of car-following models in terms of oscillation components, analyses of local and asymptotic stabilities, and quantification of oscillation propagation characteristics. Numerical experiments with a range of well-known nonlinear car-following laws show that the proposed approach is capable of accurately predicting oscillation characteristics under realistic physical constraints and complex driving behaviors. This framework not only helps further understand the root causes of the traffic oscillation phenomenon but also paves a solid foundation for the design and calibration of realistic nonlinear car-following models that can reproduce empirical oscillation characteristics.     

An anisotropic continuum model considering bi-directional information impact

In traffic flow with naturalistic driving only, stimulus information pre-dominantly comes from the preceding vehicles with drivers occasionally responding to the following vehicles through the inspection of rear-view mirrors. Such one-sided information propagation may potentially be altered in future connected vehicle environment. This brings new motivations of modeling vehicle dynamics under bi-directional information propagation. In this study, stemming from microscopic bi-directional car-following models, a continuum traffic flow model is put forward that incorporates the bi-directional information impact macroscopically but can still preserve the anisotropic characteristics of traffic flow and avoid non-physical phenomenon such as wrong-way travels. We then analyze the properties of the continuum model and respectively illustrate the condition that guarantees the anisotropy, eradicates the negative travel speed, preserves the traveling waves and keeps the linear stability. Through a series of numerical experiments, it is concluded that (1) under the bi-directional looking context only when the backward weight ratio belongs to an appropriate range then the anisotropic property can be maintained; (2) forward-propagating traffic density waves and standing waves emerge with the increasing consideration ratio for backward information; (3) the more aggressive driving behaviors for the forward direction can delay the backward-propagating and speed up the forward-propagating of traffic density waves; (4) positive holding effect and negative pushing effect of backward looking can also be observed under different backward weight ratios; and (5) traffic flow stability varies with different proportion of backward traffic information contribution and such stability impact is sensitive to the initial traffic density condition. This proposed continuum model may contribute to future development of traffic control and coordination in future connected vehicle environment.