PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Braking System via Deep Reinforcement Learning

Hyunmin Chae, Chang Mook Kang, ByeoungDo Kim, Jaekyum Kim, Chung Choo Chung, Jun Won Choi

arXiv:1702.02302v2cs.AI

TL;DR

The paper targets the difficulty of designing autonomous braking that handles uncertain pedestrian threats without braking too early or too late. It trains a DQN braking policy with a reward balancing accident damage and getting out of risk quickly; simulations report desirable control and collision avoidance in uncertain scenarios.

  • Problem

    Rule-based autonomous braking is limited in handling all real-road scenarios, while uncertain pedestrian behavior makes choosing safe braking actions challenging.

    Source: §I · p. 1, A. Scenarios · p. 2

  • Method

    A DQN learns braking actions from vehicle speed and relative pedestrian position, using a reward that balances accident penalties with getting out of risk quickly.

    Source: §I · pp. 1–2, C. Reward Function · p. 4

  • Results

    The agent avoided collisions for TTC values above 1.5 s, while collisions below 1.5 s were cases deemed unavoidable despite full braking.

    Source: C. Test Results · p. 5

  • Takeaways & Limitations

    Simulations show desirable and consistent braking behavior across scenarios with uncertain pedestrian behavior.

    Source: §V · p. 6

Abstract

from arXiv · show

In this paper, we propose a new autonomous braking system based on deep reinforcement learning. The proposed autonomous braking system automatically decides whether to apply the brake at each time step when confronting the risk of collision using the information on the obstacle obtained by the sensors. The problem of designing brake control is formulated as searching for the optimal policy in Markov decision process (MDP) model where the state is given by the relative position of the obstacle and the vehicle's speed, and the action space is defined as whether brake is stepped or not. The policy used for brake control is learned through computer simulations using the deep reinforcement learning method called deep Q-network (DQN). In order to derive desirable braking policy, we propose the reward function which balances the damage imposed to the obstacle in case of accident and the reward achieved when the vehicle runs out of risk as soon as possible. DQN is trained for the scenario where a vehicle is encountered with a pedestrian crossing the urban road. Experiments show that the control agent exhibits desirable control behavior and avoids collision without any mistake in various uncertain environments.

I. INTRODUCTION

Conventional rule-based braking protocols are limited across the situations that can arise on real roads. The paper proposes a DRL braking system that adapts to changing obstacle positions, with simulations reporting desirable control across uncertain scenarios, including Euro NCAP AEB tests.

I. INTRODUCTION

  • Rule-based braking protocols are limited in handling all scenarios that can occur on real roads.The paper frames intelligent, goal-oriented braking as a response to this limitation.Source: §I · p. 1
  • Prior work applied DNNs to autonomous-driving perception and end-to-end sensing-to-control, while DRL had been used for vehicle control.The introduction situates the proposal within these machine-learning applications.Source: §I · p. 1
  • The proposed DRL braking system adapts brake control to changing obstacle positions to minimize accident risk.The agent receives obstacle-position information from sensors and responds to state changes.Source: §I · p. 1, Figure 1 · p. 1
  • The reward design balances accident penalties against rewards for getting the vehicle out of risk quickly, and trains DQN to choose braking timing for a crossing pedestrian.The paper also proposes a DQN design intended to learn rapidly to avoid rare accidents.Source: §I · p. 2
  • Simulations tested uncertainty in initial vehicle velocity, pedestrian position, and whether the pedestrian crosses; the system showed desirable control, including in Euro NCAP AEB tests.The paper evaluates the proposed braking system through computer simulations.Source: §I · p. 2

II. SYSTEM DESCRIPTION

The system addresses uncertain pedestrian threats by selecting braking actions according to the pedestrian's state. It models pedestrian behavior as a discrete-state Markov process and learns a policy through interaction with the environment.

A. Scenarios

  • The same pedestrian detection can call for different actions: stop when the pedestrian crosses, but remain on standby when appropriate.The paper frames the challenge as choosing brake actions for different pedestrian states despite uncertainty about when a pedestrian becomes a threat.Source: Figure 2 · p. 2, A. Scenarios · p. 2
  • The scenario includes uncertainty in vehicle speed, pedestrian crossing timing and direction, sensor error, and road condition.The system is designed to decide actions based on pedestrian state, including position and velocity.Source: A. Scenarios · p. 2
  • Pedestrian behavior follows a discrete-state Markov process with no-obstacle, sidewalk-stay, and road-crossing states.The pedestrian can begin on either the farside or nearside, and walking speed and vehicle initial speed can vary.Source: A. Scenarios · p. 2, Figure 3 · p. 2

B. Autonomous Braking System

The braking system uses vehicle speed and relative pedestrian position to select among discrete braking actions. Its RL agent learns a policy through episodes, with the stated goal of maximizing expected accumulated reward.

B. Autonomous Braking System

  • The system takes vehicle speed and the pedestrian's relative position as inputs, then chooses among no braking and three braking intensities.The paper notes that finer discrete actions or continuous braking are not considered.Source: B. Autonomous Braking System · pp. 2–3

A. Structure of DRL

  • The agent receives sensor-derived state feedback, selects an action under its policy, and seeks to maximize expected accumulated reward.Episodes begin when a pedestrian is detected and use random initial pedestrian positions and vehicle velocities.Source: A. Structure of DRL · p. 3, §III · p. 3
  • An episode can end when the vehicle stops, bumps past the safety line during a crossing, passes without an accident, or the pedestrian crosses the road.The agent learns from interaction with the environment episode by episode.Source: A. Structure of DRL · p. 3

B. Deep Q-Network

The section presents DQN as a neural-network approximation to Q-learning’s state-action values, with target-network updates and replay batches used to stabilize and train the estimate.

B. Deep Q-Network

  • Q-learning iteratively searches for an optimal policy using state-action values that represent expected future rewards discounted over time.The discount factor reduces the contribution of rewards farther in the future.Source: B. Deep Q-Network · p. 3
  • The Bellman equation characterizes the Q-value, while Q-learning uses a one-step backup update when the exact value is difficult to obtain.Source: B. Deep Q-Network · p. 3
  • DQN approximates state-action values with a deep neural network because continuous state spaces make exact optimal values impractical to obtain.The network parameters are optimized to minimize squared temporal-difference error.Source: B. Deep Q-Network · p. 3
  • A periodically cloned target network estimates next-state values separately from the Q-network to improve convergence.Source: B. Deep Q-Network · p. 3
  • Random batches of one-step backups from replay memory form the loss minimized through stochastic gradient descent.Source: B. Deep Q-Network · p. 4

C. Reward Function

The reward function balances avoiding collisions with leaving risky situations quickly, while penalizing premature deceleration and high-speed impacts.

C. Reward Function

  • The reward function balances collision avoidance against getting the vehicle out of risk quickly, since imbalance can make braking overly conservative or reckless.Source: C. Reward Function · p. 4
  • A deceleration penalty proportional to squared pedestrian distance discourages braking when the pedestrian is far away.An accident penalty depends on vehicle velocity, reflecting the greater pedestrian damage associated with higher-speed collisions.Source: C. Reward Function · p. 4
  • The accident penalty combines vehicle speed with an indicator for collision, and weight parameters control the trade-off between the objectives.Source: C. Reward Function · p. 4

D. Trauma Memory

Because collisions are rare in ordinary replay, the method adds a trauma memory that retains rare-event backups for training.

D. Trauma Memory

  • Rare collisions leave few associated backups in replay memory, making them unlikely to be sampled enough for the DQN to learn accident avoidance reliably.Source: D. Trauma Memory · p. 4
  • Trauma memory stores backups from rare events such as collisions and samples them alongside replay-memory backups during training.Source: D. Trauma Memory · p. 4
  • Persistent accident reminders let the agent learn to maintain speed and avoid collisions reliably, regardless of its current policy.Source: D. Trauma Memory · p. 4

IV. EXPERIMENTS

The experiments evaluate the braking system in computer simulations using a pedestrian-crossing environment with randomized timing and behavior.

IV. EXPERIMENTS

  • The proposed braking system is evaluated through computer simulations.Source: §IV · p. 4

A. Simulation Setup

  • PreScan models vehicle dynamics, while simulations train the DQN with random pedestrian behavior and noisy relative-position measurements.Source: A. Simulation Setup · p. 4
  • Initial time-to-collision is sampled uniformly from 1.5 to 4 seconds, and vehicle speed from 2.78 to 16.67 m/s.Source: A. Simulation Setup · p. 4
  • Training selects equally between a pedestrian crossing the road and one staying put; crossing begins at a trigger point, and the agent chooses among braking levels every 0.1 seconds.The pedestrian safety distance is 3 m.Source: A. Simulation Setup · p. 4

B. Training of DQN

The DQN uses a specified network and replay-memory configuration to learn brake control. With trauma memory, its value function converges during training, unlike training without trauma memory.

B. Training of DQN

  • The DQN uses a fully connected network with five hidden layers, leaky ReLU, and RMSProp at a learning rate of 0.0005.The listed layer sizes are 15, 100, 70, 50, 70, 100, and 4 nodes.Source: B. Training of DQN · p. 5
  • Training uses replay memory of 10,000 samples with batches of 32, alongside trauma memory of 1,000 samples and batches of 10.The reward-function parameters are α = 0.001, β = 0.1, η = 0.01, and λ = 100.Source: B. Training of DQN · p. 5
  • With trauma memory, the value function converges after 2,000 episodes and maintains high total reward; without it, the policy keeps fluctuating without convergence.The comparison is reported for training with and without trauma memory.Source: B. Training of DQN · p. 5

C. Test Results

Tests show collision avoidance for TTC values above 1.5 seconds; collisions below that threshold were attributed to high initial vehicle speed despite full braking. The trajectories also show graduated braking and generally stopping about five metres ahead of the pedestrian.

C. Test Results

  • For TTC values above 1.5 s, the agent avoids collisions; below 1.5 s, some collisions occur when high initial speed makes collision unavoidable despite full braking.Collision rate was measured over 10,000 trials for each TTC value.Source: C. Test Results · p. 5, Table I · p. 6
  • In the TTC = 1.5 s example, the vehicle starts decelerating about 20 m away and stops about 5 m ahead of the pedestrian.The trajectory figure shows position, velocity, and brake actions; braking begins weakly and becomes stronger as the vehicle approaches.Source: C. Test Results · p. 5, Figure 6 · p. 5
  • In Scenario 2, the agent passed the pedestrian without an unnecessary stop in all cases; separately, safety testing measured collision rates over 10,000 trials for each TTC value in Scenario 1.Source: C. Test Results · p. 5
  • Across 1,000 Scenario 1 trials, the vehicle stops around 5 m in front of the pedestrian in most cases, compared with a 3 m safety distance.The paper says this spacing can be adjusted by changing reward parameters.Source: C. Test Results · p. 5, Figure 7 · p. 6
  • The authors report consistent brake-control performance across all cases considered.The pedestrian-position and final relative-distance distributions are shown in Fig. 7.Source: C. Test Results · p. 6, Figure 7 · p. 6

D. Test Results for Euro NCAP AEB Pedestrian Test

The system passed the specified Euro NCAP AEB pedestrian tests without collision. Tests covered farside and nearside crossings over vehicle speeds from 20 to 60 km/h.

D. Test Results for Euro NCAP AEB Pedestrian Test

V. CONCLUSIONS

The paper presents a DQN-based autonomous braking system with a carefully designed reward function and a modified DQN structure. Simulations show desirable, consistent braking across scenarios with uncertain pedestrian behavior.

V. CONCLUSIONS

  • The proposed system learns brake control from simulated experience using DQN and a carefully designed reward function.The DQN structure was modified to enhance learning stability.Source: §V · p. 6
  • Computer simulations show desirable and consistent braking behavior across scenarios with uncertain pedestrian behavior.The conclusion describes the system as an autonomous braking system based on deep reinforcement learning.Source: §V · p. 6
Loading 1702.02302v2…