PaperlayerView exact revision on arXiv

Source-linked AI summary

Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge

Pin Wang, Ching-Yao Chan

arXiv:1709.02066v3cs.LGcs.AI

TL;DR

On-ramp merging combines unpredictable interactions with other vehicles and a need to account for historical driving information, which prior studies had not extensively incorporated. The paper proposes an LSTM-based Deep Reinforcement Learning architecture that passes a history-informed internal state to a Deep Q-network for action selection. The work is preliminary, with verification, training, refinement, and performance evaluation left for future study.

  • Problem

    Prior studies had not extensively incorporated historical driving information, although merging involves other vehicles whose dynamic states may be unpredictable to the merging vehicle.

    Source: §II · p. 2

  • Method

    The architecture uses an LSTM to represent historical interactions in an internal state, which is passed to a Deep Q-network for action selection.

    Source: §III · p. 2, A. LSTM · p. 2

  • Results

    The authors present a Deep Reinforcement Learning architecture for learning an on-ramp merge driving policy.

    Source: §V · p. 4

  • Takeaways & Limitations

    The internal state is intended to incorporate historical interactive driving behavior into action selection, while experience replay and a target network are used to relieve local-optima and instability problems.

    Source: §V · p. 4

    The work is preliminary; verification and validation remain future work, and model training, refinement, and performance evaluation are still pending.

    Source: §II · p. 2, §IV · p. 4, §V · p. 4

Abstract

from arXiv · show

Multiple automakers have in development or in production automated driving systems (ADS) that offer freeway-pilot functions. This type of ADS is typically limited to restricted-access freeways only, that is, the transition from manual to automated modes takes place only after the ramp merging process is completed manually. One major challenge to extend the automation to ramp merging is that the automated vehicle needs to incorporate and optimize long-term objectives (e.g. successful and smooth merge) when near-term actions must be safely executed. Moreover, the merging process involves interactions with other vehicles whose behaviors are sometimes hard to predict but may influence the merging vehicle optimal actions. To tackle such a complicated control problem, we propose to apply Deep Reinforcement Learning (DRL) techniques for finding an optimal driving policy by maximizing the long-term reward in an interactive environment. Specifically, we apply a Long Short-Term Memory (LSTM) architecture to model the interactive environment, from which an internal state containing historical driving information is conveyed to a Deep Q-Network (DQN). The DQN is used to approximate the Q-function, which takes the internal state as input and generates Q-values as output for action selection. With this DRL architecture, the historical impact of interactive environment on the long-term reward can be captured and taken into account for deciding the optimal control policy. The proposed architecture has the potential to be extended and applied to other autonomous driving scenarios such as driving through a complex intersection or changing lanes under varying traffic flow conditions.

II. LITERATURE REVIEW

Prior on-ramp merging approaches use fixed rules or basic reinforcement-learning setups, but unpredictable interactions and continuous states and actions make the problem difficult to represent. The review identifies a need for a history-aware environment model and a Q-function approximator suited to these conditions.

II. LITERATURE REVIEW

  • Rule-based merging models depend on predefined behavior and may fail in unforeseen situations, potentially causing crashes or severe traffic disturbance.Examples include cooperative gap creation, slot-based merging, and gap-acceptance rules.Source: §II · p. 1
  • Reinforcement learning learns actions through trial and error by balancing exploration and exploitation and maximizing cumulative reward.Unlike supervised learning, it does not require ground-truth input/output pairs.Source: §II · p. 2
  • Basic Q-learning applications to ramp metering use discrete states and actions and model the problem as a Markov Decision Process.The cited applications regulate vehicles entering the mainline or seek to increase capacity at weaving sections.Source: §II · p. 2
  • Ramp merging is non-Markovian from the merging agent’s perspective and has continuous state and action spaces, making tabular Q-learning impractical.Other agents’ dynamic states may be unpredictable; Q-function approximation is presented as suitable for non-MDP or POMDP settings.Source: §II · p. 2
  • Prior deep-learning driving work leaves a gap in incorporating long-term historical information into a representative interactive-environment model and Q-function approximator.The review also says CNN-based DQNs are limited to image inputs and are not ready for real-world driving scenarios.Source: §II · p. 2

III. METHODOLOGY

The proposed on-ramp merging framework combines an LSTM model of the interactive driving environment with a DQN that selects actions from the LSTM’s internal state. Its design targets long-term effects, unpredictable or adversarial agents, and continuous states and actions.

III. METHODOLOGY

  • The framework uses LSTM to model the interactive environment and DQN to select actions from its internal state representation.The stated aims include incorporating long-term effects, mitigating unexpected and adversarial impacts, and handling continuous states and actions.Source: §III · p. 2
  • The LSTM internal state represents historical driving information and is passed to the Q-network for action selection.The Q-network is updated using experience replay and a second target network to address local optima and divergence concerns.Source: §III · p. 2
  • Figure 1 presents an overview of the proposed deep reinforcement-learning architecture.The passage identifies the figure as an architecture overview without specifying its visual components.Source: §III · p. 2

A. LSTM

The LSTM encodes interactions and historical driving information into a compact internal state for the Q-network. Its observation and state modules support prediction of the next observation and construction of a more informative state.

A. LSTM

  • The LSTM internal state compactly represents the history of interactions between the ego vehicle and surrounding vehicles for input to the Q-network.The passage describes LSTM as retaining information over both short and long durations.Source: A. LSTM · p. 2
  • The observation module estimates the next observation from the previous internal state and action, while the state module maps the current observation to a more informative internal state.The LSTM is trained by supervised learning.Source: A. LSTM · p. 2
  • The representative internal state is passed both to the next LSTM unit and to the Q-network for action decisions.The retained historical environment information informs this state representation.Source: A. LSTM · pp. 2–3
  • Figure 2 is identified as the LSTM architecture.The supplied caption gives no further figure details.Source: A. LSTM · p. 3

B. Deep Q-learning

Deep Q-learning approximates the Q-function to select actions from the LSTM state, with updates based on reward and future Q-values. The design also uses continuous control actions, exploration noise, experience replay, and a target network.

B. Deep Q-learning

  • The Q-network takes the LSTM internal state as input and produces Q-values used to select an action.The learning process also updates network parameters using the loss between predicted and target Q-values.Source: B. Deep Q-learning · p. 3
  • The Q-function approximator is designed as a quadratic function of action, with neural networks defining A(s), B(s), and C(s).The stated design goal is to ensure an optimal action exists in a given state and to support fast convergence.Source: B. Deep Q-learning · p. 3
  • Actions comprise acceleration and steering, each constrained to a range while remaining continuous within that range.The selected highest-Q action receives random noise for exploration, described as similar to ε-greedy exploration.Source: B. Deep Q-learning · p. 3
  • The reward function evaluates merging safety, smoothness, and timeliness using acceleration, steering, speed, and distance to surrounding vehicles.Large acceleration or deceleration, small surrounding-vehicle distance, and low speed under free-flow conditions receive large negative rewards.Source: B. Deep Q-learning · p. 3
  • The loss is the mean squared error between predicted and target Q-values, with the target based on immediate reward and the next internal state’s maximum Q-value.The discount factor γ lies in [0,1].Source: B. Deep Q-learning · p. 3
  • Experience replay samples transition batches uniformly at random, while a separate target network’s parameters remain fixed for a number of iterations and are periodically updated.These choices address sequential-data correlation and stability concerns in Q-learning.Source: B. Deep Q-learning · p. 3

IV. IMPLEMENTATION ON ON-RAMP MERGE CASE

The implementation models an on-ramp merge involving the merging vehicle and two mainline vehicles, and applies LSTM-based state representation with deep Q-learning. Real-world driving data inform the modeled interactions, while verification and validation remain future work.

  • The simulated merge centers on a ramp vehicle seeking a gap between a lag vehicle and a front vehicle on the mainline.The training scenarios are based on simulated sections of the real-world I-80 merging area.Source: §IV · pp. 3–4
  • Nine variables represent the merging vehicle and two mainline vehicles; the merge vehicle acts through acceleration and steering angle.The merging vehicle is modeled with speed, position, heading, and distances to lane markings; the other vehicles contribute speed and position.Source: §IV · p. 4
  • Real-world, bird’s-eye camera recordings are analyzed to obtain vehicle dynamics, and LSTM units learn interactive behavior as an internal state.The resulting internal state is passed to neural networks that calculate Q-values for action selection.Source: §IV · p. 4
  • The deep Q-learning procedure stores transitions in replay memory, samples mini-batches, and updates network parameters using a target Q-value.The algorithm adds noise to the selected action and uses a second target network, with parameters reset after a specified number of iterations.Source: §IV · p. 4
  • Verification and validation of the implemented methods are identified as next steps.The paper characterizes the work as preliminary.Source: §IV · p. 4

V. CONCLUSION AND DISCUSSION

The paper presents an LSTM–DQN architecture for learning an on-ramp merging policy from historical and interactive driving behavior. It also proposes supervisory control as a further research direction, while training, refinement, and performance evaluation remain future work.

  • The proposed architecture feeds LSTM internal states into a Q-function approximator for action selection, with experience replay and a second target network.The LSTM incorporates historical and interactive driving behavior into the state used for selecting actions.Source: §V · p. 4
  • Training, fine-tuning, methodology refinement, and performance evaluation remain topics for further study.Source: §V · p. 4
  • The authors envision supervisory control in which manual and automated modes coexist and drivers may selectively provide inputs during a merge.Cooperative inverse reinforcement learning is suggested as a possible way to accommodate a balanced, common reward.Source: §V · p. 4
Loading 1709.02066v3…