Source-linked AI summary
Navigating Occluded Intersections with Autonomous Vehicles using Deep Reinforcement Learning
David Isele, Reza Rahimi, Akansel Cosgun, Kaushik Subramanian, Kikuo Fujimura
TL;DR
Intersection handling must account for driver intent, and occlusions challenge TTC-based rules. This paper uses Deep Q-Networks to learn intersection policies and exploratory creeping; the learned systems improve efficiency and success rates over the rule-based method but still occasionally collide.
Problem
TTC-based intersection handling assumes full knowledge of surroundings, while unpredictable human driving complicates rule-based methods.
Source: §I · p. 1Method
The paper uses Deep Q-Networks for intersection handling under partial knowledge and investigates learned creeping as an exploratory action for occlusions.
Source: §I · pp. 1–2Results
The learned networks have better task efficiency and success rates than the rule-based method, learn departure times and creeping behaviors, and have some ability to generalize.
Source: §V · p. 6Takeaways & Limitations
The analysis identifies scenarios where TTC can be improved and suggests methods that could improve it beyond the specific learner choice.
Source: §I · p. 2The networks only have some ability to generalize and occasionally collide, so further research is needed to improve robustness.
Source: §V · p. 6
Abstract
from arXiv · showhide
Providing an efficient strategy to navigate safely through unsignaled intersections is a difficult task that requires determining the intent of other drivers. We explore the effectiveness of Deep Reinforcement Learning to handle intersection problems. Using recent advances in Deep RL, we are able to learn policies that surpass the performance of a commonly-used heuristic approach in several metrics including task completion time and goal success rate and have limited ability to generalize. We then explore a system's ability to learn active sensing behaviors to enable navigating safely in the case of occlusions. Our analysis, provides insight into the intersection handling problem, the solutions learned by the network point out several shortcomings of current rule-based methods, and the failures of our current deep reinforcement learning system point to future research directions.
I. INTRODUCTION
Unsignaled intersections require autonomous vehicles to interpret driver intent, manage occlusions, and balance safety with efficiency. The paper investigates Deep Q-Networks and exploratory creeping behaviors for this setting.
I. INTRODUCTION
- Prior intersection-handling strategies include cooperative and heuristic approaches; cooperative methods require vehicle-to-vehicle communication, limiting their scalability to general intersection handling.TTC is described as a widely used heuristic and as the method employed on the authors’ autonomous vehicle.Source: §I · p. 1
- Occlusions expose a limitation of TTC methods, which assume full knowledge of surroundings, while constant-velocity assumptions overlook driver intent and can lead to excessive caution.The paper motivates machine-learning approaches because coding special behaviors for the many possible scenarios requires engineers to identify and manage them.Source: §I · p. 1
- The paper explores Deep Q-Networks for intersection handling under partial knowledge, with exploratory actions as a complement to history-based approaches to partial observability.At an occluded intersection, an exploratory action can improve the agent’s understanding of the environment before it navigates.Source: §I · p. 1
- The paper examines deep learning for intersection handling, including network designs whose top-performing system learns when to start accelerating.The authors also identify scenarios where TTC could be improved and suggest methods for doing so.Source: §I · pp. 1–2
- The paper analyzes creeping as an exploratory action for occlusions and reports that deep reinforcement learning agents can discover this behavior.The authors present exploratory actions, specifically creeping behaviors, as a means of improving vehicle safety.Source: §I · p. 2
II. APPROACH
The approach formulates intersection handling as reinforcement learning and uses a Deep Q-Network to learn a state-action value function. It frames the task around maximizing expected return over a sequence of actions.
II. APPROACH
- The paper treats intersection handling as a reinforcement-learning problem and uses a Deep Q-Network to learn the state-action value function.An agent takes actions in states, transitions to new states, and receives rewards along a trajectory.Source: §II · p. 2, A. Reinforcement Learning · p. 2
A. Reinforcement Learning
- The reinforcement-learning task is formulated as a Markov Decision Process with states, actions, transition probabilities, rewards, and a discount factor.The Markov assumption makes the next-state transition depend on the current state and action, not the full prior history.Source: A. Reinforcement Learning · p. 2
- The learning objective is to choose actions that maximize expected return, and the paper uses Q-learning for this optimization.The return discounts future rewards.Source: A. Reinforcement Learning · p. 2
B. Q-learning
Q-learning represents expected return with an action-value function and derives an optimal policy by selecting the highest-valued action. Deep Q-learning approximates that function with a neural network and trains it against an estimated return.
B. Q-learning
- In Q-learning, the action-value function gives expected return for a state-action pair under a policy, and an optimal policy selects the action with the highest optimal value.The optimal action is chosen by maximizing Q*(s, a) at each time step.Source: B. Q-learning · p. 2
- Deep Q-learning approximates the optimal value function with a neural network and learns it by minimizing error between expected return and the network’s prediction.Because the true return is unknown in practice, it is often approximated with the one-step return.Source: B. Q-learning · p. 2
C. Action Representations
The paper studies three action representations that vary in control flexibility and support different behaviors. They range from choosing a departure time to sequential control and a restricted option for creeping before departure.
C. Action Representations
- The paper focuses on three action representations after finding that representation choice significantly affects an agent’s ability to learn.Source: C. Action Representations · p. 2
- Time-to-Go supplies a desired path and lets the agent choose departure timing through repeated wait decisions followed by a go action.After selecting go, the agent cannot wait again.Source: C. Action Representations · p. 2
- Sequential actions let the agent accelerate, decelerate, or maintain velocity along the desired path, allowing more complex behavior such as slowing and waiting mid-intersection.This representation enables analysis of whether complex actions add benefit beyond changing departure time.Source: C. Action Representations · p. 2
- Creep-and-Go combines wait, slow forward movement, and go, letting the agent creep and stop before committing to cross.Its restricted action space offers the interpretability and learnability of Time-to-Go while retaining exploratory actions.Source: C. Action Representations · p. 2
D. State Representation
The paper compares action and state representations for intersection navigation, then tests how well learned policies perform across intersection scenarios and generalize to new ones. Real-valued representations performed better in preliminary studies, while the sequential-action network learned more slowly and required more training than Time-to-Go.
D. State Representation
- Real-valued representations performed better in preliminary studies, and the sequential-action representation used heading, velocity, and time-to-collision features in a spatial grid.The paper hypothesizes that real values help generalization; Time-to-Go instead uses an 18 × 26 grid without calculated time-to-collision.Source: D. State Representation · pp. 2–3, A. Algorithm Comparison Study · p. 3
- For occluded states, the grid adds an occlusion indicator and the car’s offset to the pixel boundary.The sequential-action representation discretizes space into a 5 × 11 grid spanning 0 to 20 meters ahead and ±90 meters to either side.Source: D. State Representation · p. 3, A. Algorithm Comparison Study · p. 3
III. EXPERIMENTS
- The first experiments compared Sequential Actions and Time-to-Go with TTC across five scenarios, including a dense six-lane Challenge intersection.Challenge traffic density was higher than in the other scenarios.Source: §III · p. 3, A. Algorithm Comparison Study · p. 3
A. Algorithm Comparison Study
- The sequential-action network learned more slowly and performed below Time-to-Go, even with one million simulations per scenario versus 250,000 for Time-to-Go.The sequential network also received an added time-to-collision feature and longer training to bring its performance on par.Source: A. Algorithm Comparison Study · p. 4
- Generalization was tested by applying a policy trained in one scenario to another without additional training, using 25,000 training simulations to prevent over-fitting.Experiments used the Sumo traffic simulator; the paper also reports collecting real-vehicle data at an unsigned T-junction.Source: A. Algorithm Comparison Study · pp. 3–4
B. Experiments with Occlusion
The occlusion experiments test a left turn across two lanes when objects obstruct the closest lane and hide some vehicles. The setup varies occluder placement and traffic conditions, and compares approaches with different opportunities to explore.
B. Experiments with Occlusion
- The occlusion figure distinguishes gray occluding objects, yellow occluded cars, and green visible cars.Braking time was not measured because traffic cars do not brake for the ego-car.Source: Figure 3 · p. 4, B. Experiments with Occlusion · p. 4
- The occlusion task is a left turn across two lanes, with closest-lane objects blocking the vehicle’s view and ray tracing marking occluded grid cells.Cars in occluded cells are not added to the representation.Source: B. Experiments with Occlusion · p. 4
- Occluders vary in side, length, width, and position; the experiments use a dueling network with prioritized replay and doubled traffic density.The time limit was extended to 60 seconds, and other cars were not allowed to decelerate.Source: B. Experiments with Occlusion · p. 4
IV. RESULTS
Across the reported intersection results, DQN policies were faster and often more successful than TTC, while TTC’s tuned threshold achieved zero collisions. The speed-safety comparison favors DQN across the tested TTC thresholds, but the authors note that DQN methods seldom matched TTC’s collision minimization.
IV. RESULTS
- Time-to-Go had the highest success rate in all but one experiment, and both DQN methods reached the goal at least 99.5% of the time outside Challenge.In Challenge, Time-to-Go reached the goal 98.46% of the time, while TTC reached it 39.2% of the time.Source: §IV · pp. 4–5
- DQN performance dominated TTC across every tested speed–collision trade-off; Time-to-Go averaged 28% faster than TTC, while Sequential averaged 19% faster.Figure 4 excludes the Challenge scenario for scale, though its results are described as similar.Source: §IV · pp. 4–5, Figure 4 · p. 5
- TTC had zero collisions after its safety threshold was tuned per scenario, while DQN methods seldom minimized collisions as successfully.The paper reports comparable braking time across methods and no conclusive evidence that DQNs were more aggressive than TTC.Source: §IV · pp. 4–5
A. Safety, Generalization, and Transfer
Transfer performance varies by training scenario, and no method transfers well to every task. The authors also emphasize that collision-free behavior remains an important safety goal and that overly aggressive learned behavior can disrupt other drivers.
A. Safety, Generalization, and Transfer
- Zero percent collisions remains important, and overly aggressive close-following can make the vehicle's intent difficult to interpret and disrupt other drivers.The authors describe close-following behind a passing car as a learned efficiency strategy that may be useful in dense traffic.Source: A. Safety, Generalization, and Transfer · p. 5
- The transfer test copied a network trained in one scenario directly to every other scenario as an initial investigation of out-of-sample generalization.The results are shown in Table III.Source: A. Safety, Generalization, and Transfer · p. 5, Table III · p. 5
- Networks trained on Left transferred better to Right and Forward, while Left2-trained networks transferred well to Challenge; no method transferred well to all tasks.The authors attribute the pattern largely to state representation.Source: A. Safety, Generalization, and Transfer · p. 5, Table III · p. 5
B. Occlusion
Occlusion experiments show a trade-off between reaching full visibility quickly and learning a successful creeping behavior. Without creeping, TTC can collide by assuming the road is clear, while the DQN learns the specialized behavior more efficiently than TTC with creeping.
B. Occlusion
- Sequential actions reach full visibility fastest, while the restricted action space makes creeping easier to learn and yields a higher success rate.Sequential actions therefore tend to be faster than the creeping approach.Source: B. Occlusion · p. 5
- Creeping moves the vehicle closer to the goal while the intersection is blocked, so TTC and Time-to-Go without creeping tend to take longer on average.Source: B. Occlusion · p. 5
- TTC without creeping collides by incorrectly assuming the road is clear; TTC with creeping avoids collisions but has many timeouts, while the DQN learns the behavior more efficiently.Source: B. Occlusion · p. 5
C. Qualitative Analysis
The learned DQN strategies account for other vehicles' predicted behavior, unlike TTC's requirement that all cars clear its path. Departure-time selection can improve over TTC without dynamic acceleration and deceleration, but occlusions require exploratory movement to a better vantage point.
C. Qualitative Analysis
- DQNs predict that distant-lane traffic will pass before arrival and anticipate whether oncoming traffic has time to brake.The few collisions appear related to discretization effects, with the vehicle nearly missing oncoming traffic.Source: C. Qualitative Analysis · pp. 5–6
- TTC waits for all cars to clear and applies the same safety margin to distant and closer lanes, often missing crossing opportunities.Source: C. Qualitative Analysis · p. 6
- Choosing departure time can improve over TTC without dynamic acceleration or deceleration, but occlusions cause TTC and Time-to-Go to collide without movement to a better vantage point.The study identifies occlusions as a case requiring exploratory behaviors and shows DQNs can learn them.Source: C. Qualitative Analysis · p. 6
V. CONCLUSIONS
The authors conclude that DQNs can learn intersection-departure timing and exploratory behaviors, outperforming a rule-based method in efficiency and success rates. Their system has some ability to generalize but still occasionally collides, leaving robustness as further work.
V. CONCLUSIONS
- The DQN system learns exploratory behaviors to better understand the scene in the intersection-handling problem.Source: §V · p. 6
- The learned networks have better task efficiency and success rates than the rule-based method, and can learn both departure times and creeping behaviors.Source: §V · p. 6
- The networks have some ability to generalize to novel domains and out-of-sample data but occasionally collide, so more research is needed for robustness.Source: §V · p. 6