PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Reinforcement Learning-Based Eco-Driving Strategy for Connected and Automated Vehicles at Signalized Intersections

Zhengwei Bai, Peng Hao, Wei Shangguan, Baigen Cai, Matthew J. Barth

arXiv:2201.07833v2eess.SYcs.LGcs.SY

TL;DR

Mixed traffic at signalized intersections makes it challenging to improve throughput and energy efficiency amid traffic complexity and uncertainty. The paper proposes an HRL eco-driving framework that combines rule-based strategies with deep RL and integrates vision perception with V2I information. In experiments, it reports lower energy consumption and travel time than Graph and IDM baselines.

  • Problem

    Conventional approaches make assumptions such as traffic-free conditions or full connected-vehicle penetration, while few RL methods address long-term intersection eco-driving across energy, signal, and traffic tasks.

    Source: §I · p. 2

  • Method

    The HRL framework combines rule-based strategies with deep RL, using multi-sensor inputs and a long-short-term reward model for mixed-traffic intersection eco-driving.

    Source: §I · p. 2, §VI · p. 13

  • Results

    12.70% lower energy consumption and 11.75% less travel time versus the Graph method; versus IDM, the reductions were 56.01% and 5.66%, respectively.

    Source: §VI · p. 13

  • Takeaways & Limitations

    The framework can be implemented for other vehicle types by replacing the energy-reward function and vehicle dynamic model.

    Source: §VI · p. 13

Abstract

from arXiv · show

Taking advantage of both vehicle-to-everything (V2X) communication and automated driving technology, connected and automated vehicles are quickly becoming one of the transformative solutions to many transportation problems. However, in a mixed traffic environment at signalized intersections, it is still a challenging task to improve overall throughput and energy efficiency considering the complexity and uncertainty in the traffic system. In this study, we proposed a hybrid reinforcement learning (HRL) framework which combines the rule-based strategy and the deep reinforcement learning (deep RL) to support connected eco-driving at signalized intersections in mixed traffic. Vision-perceptive methods are integrated with vehicle-to-infrastructure (V2I) communications to achieve higher mobility and energy efficiency in mixed connected traffic. The HRL framework has three components: a rule-based driving manager that operates the collaboration between the rule-based policies and the RL policy; a multi-stream neural network that extracts the hidden features of vision and V2I information; and a deep RL-based policy network that generate both longitudinal and lateral eco-driving actions. In order to evaluate our approach, we developed a Unity-based simulator and designed a mixed-traffic intersection scenario. Moreover, several baselines were implemented to compare with our new design, and numerical experiments were conducted to test the performance of the HRL model. The experiments show that our HRL method can reduce energy consumption by 12.70% and save 11.75% travel time when compared with a state-of-the-art model-based Eco-Driving approach.

I. INTRODUCTION

The paper targets time-efficient eco-driving at signalized intersections in mixed traffic, where conventional approaches rely on restrictive assumptions and RL faces a multi-task challenge. It proposes an HRL framework and frames the task around energy, travel time, and safe interaction with traffic and signals.

I. INTRODUCTION

  • Earlier methods often assume traffic-free conditions or 100% connected-vehicle penetration, while some eco-driving methods omit lateral maneuvers.The paper identifies these simplifications as limiting the ability to incorporate complex dynamics in complex or unknown environments.Source: §I · p. 2
  • Few RL algorithms had addressed long-term intersection eco-driving, which combines energy-efficient driving with signal and traffic interaction.The paper describes this as a multi-logical task involving energy use, signal response, and safe distance from other vehicles.Source: §I · p. 2
  • The proposed HRL framework combines rule-based policies and reinforcement learning to learn long-term intersection eco-driving strategies in mixed traffic.Its stated innovations include a long-short-term reward model and a multi-sensor RL network.Source: §I · p. 2

II. PROBLEM FORMULATION

  • The policy is intended to save energy, reduce travel time, and interact safely with traffic and signals.The problem is framed as an RL task in which the ego-vehicle interacts with mixed traffic and a signalized intersection through states, actions, and rewards.Source: §II · pp. 2–3
  • The paper situates the approach in an intersection environment with an ego-vehicle, other vehicles, and traffic signals, using front-camera and SPaT perception.The RL framing identifies the ego-vehicle as agent, the proposed eco-driving policy as policy, and short- and long-term benefits as rewards.Source: §II · p. 3

A. Traffic Environment

The traffic environment combines conventional vehicles with varied driving dynamics and a signalized intersection whose initial phase and time vary across simulation starts.

A. Traffic Environment

  • The simulated traffic environment includes the ego-vehicle, conventional vehicles, and a five-lane signalized intersection as an example.Source: A. Traffic Environment · p. 3
  • Conventional vehicles use five dynamic models and behavior strategies, with IDM generating longitudinal acceleration when a leading vehicle is present or absent.Their lateral control uses varying lane-changing rates to represent human-driver behavior.Source: A. Traffic Environment · p. 3
  • The intersection uses a fixed signal plan with 20s green, 3s yellow, 40s red, and 1s all-red phases, while initial phase and time are randomly sampled each run.Source: A. Traffic Environment · p. 3

B. Agent

The agent is modeled as an electric CAV, and its observations combine infrastructure signal information with onboard sensing and vehicle-state data.

B. Agent

  • The ego-vehicle is modeled as an electric CAV with maximum acceleration and deceleration set to 3m/s2.Source: B. Agent · p. 3

1) Observation:

  • Observations combine V2I SPaT messages, three radar distances, a 320x160-pixel front-camera image at 50fps, and OBD speed and acceleration.Source: 1) Observation · p. 4
  • Radar warnings reduce the observation representation to a 12-dimensional vector containing signal, warning, distance, speed, and acceleration information.The warning variables include forward, left, and right warnings; the observation elements are listed in the paper's observation-space definition.Source: 1) Observation · p. 4

2) Energy Model:

The energy model excludes braking-energy regeneration, and the action space covers longitudinal acceleration and target-lane choices while reducing the combined action dimensions.

2) Energy Model:

  • The energy model uses a calibrated model derived from 2013 NISSAN LEAF on-road data, sets road grade to zero, and truncates braking-energy regeneration.The authors report that removing regeneration was needed for the learning system to run successfully.Source: 2) Energy Model · p. 4

3) Action:

  • The agent’s actions include longitudinal acceleration and target lane, with lane changes restricted to zero longitudinal acceleration, reducing the action space from 33 dimensions to 13.The lateral choices are left, current lane, or right; the passage does not provide the complete longitudinal action definition.Source: 3) Action · p. 4

III. HRL-BASED ECO-DRIVING FRAMEWORK

The proposed system uses an HRL brain to coordinate rule-based and learned driving policies for intersection driving. Its on-board sensing and communication inputs support acceleration and lane-target outputs across multiple driving tasks.

III. HRL-BASED ECO-DRIVING FRAMEWORK

  • The system architecture is proposed for electric passenger vehicles and centers on an on-board computer housing the hierarchical-RL decision-making brain.The brain is presented as the system’s decision-making center.Source: A. System Architecture · p. 4
  • The on-board computer processes vehicle and traffic observations and returns longitudinal acceleration and target-lane information to vehicle control.The described inputs include radar, camera, OBD speed and acceleration, and traffic-light information relayed by a roadside unit.Source: A. System Architecture · p. 5
  • The HRL brain combines rules and reinforcement learning to address cruise control, lane changing, stopping at red or yellow, and starting at green.The authors describe these as multi-phase tasks that are difficult for one RL algorithm to handle.Source: A. System Architecture · p. 5

B. HRL Framework

The HRL framework combines learned and rule-based driving policies through a situation-dependent manager. Select-stack preprocessing prepares spatiotemporal observations, and a final safety rule constrains the chosen actions.

B. HRL Framework

  • The HRL agent contains Deep RL, IDM, and emergency-braking policies, coordinated by the Decision Manager and followed by a final safety-based rule policy.The policies generate longitudinal actions, while the Deep RL policy generates both longitudinal and lateral actions.Source: B. HRL Framework · p. 5

C. Data Preprocessing

  • Select-stack preprocessing resizes image frames and selects and stacks frames to form spatiotemporal inputs; it is also applied to logical data.The method uses 80×80×3 image frames with Nselect and Nstack both set to 4.Source: C. Data Preprocessing · p. 5

D. Decision Manager

  • The Decision Manager divides driving into situations and combines Deep RL, IDM, and emergency-braking policies; its Stop-in-Red warning distinguishes signal and vehicle conditions.IDM is activated when the signal turns green and the vehicle needs to start moving; emergency braking activates when front distance is below its threshold.Source: D. Decision Manager · pp. 5–6

E. Safety-Based Rule Policy

  • The Safety-Based Rule Policy checks the Decision Manager’s action and overrides it when warning factors indicate danger, including substituting emergency braking for a forward warning.The policy also blocks a lane change when the corresponding side warning is active.Source: E. Safety-Based Rule Policy · p. 6

F. Deep RL for Long-term Eco-Driving

Long-term intersection eco-driving requires balancing energy use and speed across multiple driving phases. The paper pairs Dueling DQN with a long-short-term reward that combines immediate indicators and rule-based long-term benefits.

F. Deep RL for Long-term Eco-Driving

  • The Long-Short Term Reward model combines short-term indicators for velocity, energy, lane changes, and danger with long-term indicators for time and reaching the stop line on green.The reward also includes a rule-based long-term benefit for reaching the stop line when the light is green.Source: F. Deep RL for Long-term Eco-Driving · pp. 6, 8
  • Dueling DQN estimates state value and action advantages in separate streams, while prioritized experience replay is used to address correlation and non-static data distribution.The policy selects actions from the action space using an ϵ-greedy algorithm.Source: F. Deep RL for Long-term Eco-Driving · pp. 6–7
  • The central learning challenge is balancing energy saving against speed while planning across multiple driving phases and long-term benefits.The described task spans the approach to and passage through an intersection, with operating logic changing across stages.Source: F. Deep RL for Long-term Eco-Driving · p. 7
  • The reward principles distinguish whether the vehicle can pass the intersection at its current speed under red or yellow versus green phases when deciding whether to accelerate.The four listed cases combine signal phase and passability to specify when to accelerate or not.Source: F. Deep RL for Long-term Eco-Driving · p. 7
  • The reward includes penalties of −0.1 for lane changes and −0.5 for dangerous actions.These are terms in the LSTR reward definition.Source: F. Deep RL for Long-term Eco-Driving · p. 8

G. Neural Network Structure

The network maps observations to action values through separate vision and logical-data feature streams, then selects an action using a Dueling DQN policy.

G. Neural Network Structure

  • The main network maps each observation to action values through hidden-feature extraction and a Dueling DQN policy network.The policy selects the action with the maximum Q-value.Source: G. Neural Network Structure · p. 8
  • The Green-Pass Reward function takes speed, acceleration, distance, time, and signal phase as inputs and returns a reward.Its algorithm assigns reward values according to the inputs and driving conditions.Source: G. Neural Network Structure · p. 8
  • A CNN processes image flow, while radar, OBD, and V2I data form a 13-dimensional vector transformed into temporal data for an LSTM stream.The two streams are concatenated before the dense policy network.Source: G. Neural Network Structure · p. 8
  • The Dueling DQN combines state value and action advantage to generate Q-values for possible actions.The action with the maximum Q-value is selected.Source: G. Neural Network Structure · p. 8

H. Network Update and Hyperparameters

The network update uses evaluation and target networks with epsilon-greedy action selection, and Adam updates the evaluation-network weights.

H. Network Update and Hyperparameters

  • Each update predicts action values, selects an epsilon-greedy action, generates target-network values, then calculates loss and updates the evaluation network.Adam is used to update the network weights.Source: H. Network Update and Hyperparameters · p. 9
  • The learning rate was 0.00025, discount factor 0.99, batch size 64, and replay memory size 50,000.Training used 2,000,000 steps and testing used 10,000 steps.Source: H. Network Update and Hyperparameters · p. 9

IV. GAME ENGINE BASED SIMULATION EXPERIMENTS

The authors built a Unity-based intersection simulator with Unity ML-Agents and a TensorFlow reinforcement-learning script to train and evaluate the ego vehicle.

IV. GAME ENGINE BASED SIMULATION EXPERIMENTS

B. Model Training and Numerical Testing Design

The experiments trained and tested HRL in a Unity-based mixed-traffic intersection simulator against IDM and Graph-based trajectory planning. Across tested conditions, HRL improved energy and travel-time outcomes, while its performance varied with entry conditions and traffic interactions.

B. Model Training and Numerical Testing Design

  • Training used epsilon-greedy exploration over 2 million steps, and testing varied signal entry time from 0 to 50 seconds and initial speed from 10 to 50 kph.IDM and Graph-based trajectory planning were comparison methods; the rule-based lane-change decision used front distance, target speed, and accumulated time conditions.Source: B. Model Training and Numerical… · pp. 9–10, Table V · p. 10

A. Training Results

  • During training, HRL speed increased, energy consumption decreased, and Green-Pass Reward rose after about halfway through training as signal interaction was learned.HRL also learned fewer lane-changing maneuvers, yielding a smoother driving strategy.Source: A. Training Results · pp. 10–11, Figure 10 · p. 10

B. Numerical Testing Results

  • HRL adjusted acceleration to traffic conditions, using lower acceleration to avoid waiting at an intersection or higher acceleration when it could pass before green ended.The trajectories contrasted scenarios C30 S20 and C40 S20.Source: B. Numerical Testing Results · p. 11, Figure 11 · p. 11
  • Unlike IDM’s speed-limit target and Graph’s SPaT-adjusted target, HRL dynamically adjusted target speed; in a long-red queue, Graph’s trajectory was disrupted by vehicles ahead.The cited comparison reports HRL performing better than Graph in energy consumption and travel time in that setting.Source: B. Numerical Testing Results · pp. 11–12, Figure 11 · p. 11
  • In 30 scenarios, HRL achieved the best energy-consumption improvement in 21 and the best travel-time improvement in 27.Table VI presents the comparisons with Graph and IDM across the traffic scenarios.Source: Table VI · p. 12, B. Numerical Testing Results · p. 12
  • 12.70% (up to 35.69%) less energy consumption than Graph was reported for HRL under different initial signal-phase conditions.Table VII reports average performance, with improvement defined as HRL compared with Graph.Source: B. Numerical Testing Results · p. 12, Table VII · p. 13
  • With regenerative braking, HRL reduced energy consumption by 23.60% versus Graph and 41.59% versus IDM.Energy consumption decreased for all evaluated methods when regenerative-braking charging was included.Source: B. Numerical Testing Results · p. 13
  • In a reported C50, 30-kph entry case, HRL saved 42.94% energy consumption and 55.83% travel time compared with Graph.The analysis reports that entry time and speed jointly influence HRL’s performance, with a sweet spot in the tested conditions.Source: B. Numerical Testing Results · p. 13, Figure 13 · p. 13, Figure 14 · p. 13

VI. CONCLUSIONS & FUTURE WORK

The HRL framework combines rule-based driving with deep reinforcement learning for eco-driving in mixed intersection traffic. Experiments report energy and travel-time savings, while vehicle diversity, cooperation, and more realistic evaluation remain future work.

VI. CONCLUSIONS & FUTURE WORK

  • HRL combines condition-based safety and efficiency rules with deep RL using OBD, camera, radar, and V2I data to output acceleration and a target lane, optimizing short- and long-term rewards.Source: §VI · p. 13
  • 12.70% versus Graph and 56.01% versus IDM energy-consumption reductions, and 11.75% versus Graph and 5.66% versus IDM travel-time savings, are reported for HRL.The comparison is against the respective Graph/IDM methods.Source: §VI · p. 13
  • Future work will examine other vehicle types, multi-agent cooperative eco-driving in complex traffic networks, and more realistic microsimulation and field experiments.Source: §VI · p. 13
Loading 2201.07833v2…