Source-linked AI summary
Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models
Ashesh Jain, Hema S. Koppula, Bharad Raghavan, Shane Soh, Ashutosh Saxena
TL;DR
The paper tackles how to anticipate driving maneuvers before they occur, using an AIO-HMM to model temporal interactions between driver intention and inside- and outside-car context. On natural-driving data, it reports 77% precision for jointly predicting all maneuvers, 34% above Morris et al., with predictions averaging 3.5 seconds ahead. The authors report that inference takes milliseconds and identify vision failures and unmodeled events as limitations.
Problem
Maneuver anticipation requires temporal modeling of multimodal driving context and driver intention, whose interactions are complex and whose intention is not directly observable.
Source: §1 · pp. 1–2Method
The AIO-HMM jointly models inside- and outside-vehicle context and driver intention, including temporal dependencies in inside-vehicle events.
Source: §1 · p. 2, §4.2 · p. 4Results
77% precision for jointly predicting all maneuvers is 34% higher than Morris et al., and AIO-HMM predicts maneuvers 3.5 seconds before they occur on average.
Source: §6.2 · p. 7Takeaways & Limitations
The authors report that millisecond inference makes the approach suited for real-time use and could give drivers more time to react to dangerous maneuvers.
Source: §7 · p. 8Wrong anticipations can arise from vision-pipeline failures and unmodeled events, including passenger interactions and overtakes.
Source: §6.3 · p. 8
Abstract
from arXiv · showhide
Advanced Driver Assistance Systems (ADAS) have made driving safer over the last decade. They prepare vehicles for unsafe road conditions and alert drivers if they perform a dangerous maneuver. However, many accidents are unavoidable because by the time drivers are alerted, it is already too late. Anticipating maneuvers beforehand can alert drivers before they perform the maneuver and also give ADAS more time to avoid or prepare for the danger. In this work we anticipate driving maneuvers a few seconds before they occur. For this purpose we equip a car with cameras and a computing device to capture the driving context from both inside and outside of the car. We propose an Autoregressive Input-Output HMM to model the contextual information alongwith the maneuvers. We evaluate our approach on a diverse data set with 1180 miles of natural freeway and city driving and show that we can anticipate maneuvers 3.5 seconds before they occur with over 80\% F1-score in real-time.
1. Introduction
The paper addresses maneuver anticipation so drivers can be alerted before dangerous maneuvers, using temporal reasoning over driving context and driver intention. It proposes an AIO-HMM and evaluates it on natural driving data.
Motivation
- Maneuver anticipation matters because existing alerts may arrive only after a dangerous maneuver, leaving drivers less time to react.The introduction notes that more than 33,000 people die in US road accidents each year, the majority due to inappropriate maneuvers.Source: §1 · p. 1
Approach
- The model reasons over time-series context from driver-facing and road-facing video, vehicle dynamics, GPS, and street maps.The paper identifies temporal modeling and detecting useful contextual cues as central challenges.Source: §1 · p. 1
- AIO-HMM jointly models inside- and outside-vehicle context with driver intention, capturing temporal and generative interactions.The model represents how outside events affect intention, which then generates events inside the vehicle.Source: §1 · p. 2
Evaluation
- The approach anticipates maneuvers 3.5 seconds before they occur on average, with 80% precision and recall.Evaluation uses 1,180 miles of natural freeway and city driving from 10 drivers across two states.Source: §1 · p. 2
Contributions
- The work releases a natural-driving dataset with inside- and outside-car video, GPS, and speed information.The authors describe it as the first such dataset.Source: §1 · p. 2
2. Related Work
Prior driver-intent approaches often use trajectory or limited camera context, while this work centers on rich context and temporal interactions between driving conditions and driver intention. The overview illustrates processing multimodal input into maneuver probabilities.
Assistive Features for Vehicles
- The paper positions its work as complementing existing ADAS and driver-monitoring techniques by anticipating maneuvers several seconds ahead.Existing vehicle assistive features warn drivers when they perform potentially dangerous maneuvers.Source: §2 · p. 2
Driver Intent
- Prior driver-intent work commonly uses vehicle trajectories, while most such studies omit rich camera, GPS, and map context.Trivedi et al. and Morris et al. use cameras inside and outside the vehicle but assume informative cues occur at fixed times.Source: §2 · p. 2
Anticipation and Modeling Humans
- AIO-HMM models interactions between driving context and driver intention, which the cited human-motion approaches do not address in this driving setting.The paper also distinguishes its generative approach from prior temporal models and compares it experimentally with HMM variants.Source: §2 · p. 2
System Overview
- The system overview shows multimodal inputs processed into inside- and outside-driving-context features, then maneuver probabilities.The vision pipeline extracts cues such as driver head movements.Source: Figure 2 · p. 3
3. Problem Overview
The problem is to anticipate lane changes and turns before they occur by combining local and global driving context. Useful cues arise at variable times, making their temporal order relevant.
Problem Overview
- The task includes anticipating lane changes before wheels touch lane markings and predicting whether a driver will turn or continue straight near intersections.Driver head movements can provide useful cues, such as checking side traffic before lane changes and scanning cross traffic at intersections.Source: §3 · p. 3
- Anticipation requires modeling cues that appear at variable times before a maneuver, with their timing affected by speed, traffic, and GPS location.Figure 3 depicts facial-point trajectories over time and notes that the order of cues matters.Source: §3 · p. 3, Figure 3 · p. 3
- The model must combine local context, including nearby vision, GPS, and speed, with global route factors such as the driver's destination.The system uses driver- and road-facing cameras, a speed logger, and GPS with maps; these sources help identify head movements, road conditions, and upcoming intersections or exits.Source: §3 · p. 3
4. Our Approach
The approach models maneuvers by jointly reasoning over outside driving context, the driver’s latent intention, and inside-vehicle observations. It learns separate maneuver models and uses them to estimate maneuver probabilities from recent driving context.
4. Our Approach
- The approach treats driver intention as a latent state because it is not directly observable and interacts with driving context.The context comprises features from inside and outside the vehicle.Source: §4 · p. 3
4.1. Modeling driving maneuvers
- AIO-HMM extends an Input-Output HMM with autoregressive connections to model temporal dependencies in inside-vehicle observations.Outside events affect latent driver states, which generate inside-vehicle observations.Source: §4.1 · p. 4, Figure 4 · p. 4
4.2. Modeling Maneuvers with AIO-HMM
- The model represents outside features as inputs, driver intention as hidden states, and inside features as outputs, with autoregressive links for temporally correlated observations.The output emissions use normal distributions, with their mean influenced by outside features and the preceding inside features.Source: §4.2 · p. 4
4.3. Learning AIO-HMM parameters
- Training maximizes maneuver-data log-likelihood using EM, estimating hidden states in the E-step and updating model parameters in the M-step.The E-step uses forward-backward inference; closed-form updates are used for parameters other than transition weights, which are optimized by gradient descent.Source: §4.3 · pp. 4–5
4.4. Inference of Maneuvers
- At inference, separate maneuver models score recent inside- and outside-feature sequences, and the resulting likelihoods and maneuver priors determine maneuver probabilities.Inference uses a forward pass and assumes a uniform prior over maneuvers.Source: §4.4 · p. 5
5. Features
The system extracts inside-vehicle head-motion features and outside-vehicle road and vehicle features to represent driving context. Together, these feature sets provide inputs from both within the car and its surroundings.
5. Features
- The feature representation distinguishes inside features Z from outside features X.The inside features are derived from the driver-facing camera, while outside features encode the environment and vehicle dynamics.Source: §5 · p. 5, §5.1 · p. 5, §5.2 · p. 6
5.1. Inside-vehicle features.
- Inside-vehicle features represent head movement through per-frame face features aggregated over every 20 frames.The vision pipeline detects and tracks the face; motion histograms represent horizontal movement and angular motion, alongside mean face-center movement.Source: §5.1 · pp. 5–6
5.2. Outside-vehicle features.
- Outside-vehicle features combine lane presence, proximity to mapped road artifacts, and vehicle speeds over the previous five seconds in a six-dimensional vector.Road artifacts include intersections, turns, and highway exits; speeds include average, maximum, and minimum values.Source: §5.2 · p. 6
6. Experiment
The experiment evaluates maneuver anticipation on natural-driving data against baseline algorithms, then examines performance, prediction timing, computation, and failure modes.
6.1. Experimental Setup
- Natural-driving data spans 1,180 miles of freeway and city driving, collected from 10 drivers across two states.The dataset includes 2 million video frames and 700 annotated events.Source: §6.1 · p. 6
6.2. Results and Discussion
- 77% precision for jointly predicting all maneuvers—34% higher than Morris et al. and 26% higher than Random-Forest; average anticipation is 3.5 seconds.Precision exceeds 80% for lane changes and turns, and turn anticipation reaches up to 4 seconds.Source: Table 1 · p. 7, §6.2 · p. 7
6.1. Experimental Setup
- Five-fold cross-validation evaluates unseen maneuvers in lane-change, turn, and joint settings, all including instances of driving straight.Baselines include Chance, SVM, Random-Forest, HMM variants, and IOHMM; AIO-HMM features match those used for HMM E + F.Source: §6.1 · p. 6, §6.2 · p. 7
6.2. Results and Discussion
- AIO-HMM precision is 3% higher than IOHMM on average, and its false-positive predictions are 10% lower than IOHMM and HMM E + F.HMM F has 11% false-positive predictions but 36% recall.Source: §6.2 · p. 7
- F1-scores increase as predictions approach maneuver onset, while IOHMM and AIO-HMM scores remain stable as the prediction threshold varies.Figure 8 compares fixed time-to-maneuver settings; Figure 9 varies the prediction threshold.Source: Figure 8 · p. 8, Figure 9 · p. 8, §6.2 · p. 8
- Jointly modeling inside and outside context outperforms context-only HMMs; AIO-HMM has the highest precision for each maneuver.For joint prediction, HMM F precision is 13% below AIO-HMM and HMM E precision is 14% below it.Source: §6.2 · pp. 7–8
- Dynamic programming lets each 0.8-second anticipation process only incoming context, reducing inference complexity from O(K|S||I|^3) to O(|S||I|^3).The reported average prediction time is under 3.6 milliseconds.Source: §6.2 · p. 8
6.3. Qualitative discussion
- Illumination changes caused tracker failure in 6% of maneuvers, and prediction timing remains ambiguous when drivers scan traffic but do not maneuver.Other cited failure sources include vision-pipeline failures and unmodeled events such as passenger interactions and overtakes.Source: §6.3 · p. 8
7. Conclusion
The conclusion presents AIO-HMM as a way to anticipate driving maneuvers from inside- and outside-car context. It reports improvement over baselines on 1,180 miles of driving data and millisecond-scale inference.
7. Conclusion
- AIO-HMM jointly models driver intention and driving context from inside and outside the car, addressing temporal and generative aspects of maneuver anticipation.Source: §7 · p. 8
- Evaluation on 1,180 miles of driving data showed improvement over many baselines.Source: §7 · p. 8
- Inference takes only a few milliseconds, and the authors state that the approach is suited for real-time use.The conclusion also says the natural-driving dataset will be publicly released.Source: §7 · p. 8