Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture

Ashesh Jain; Avi Singh; Hema S Koppula; Shane Soh; Ashutosh Saxena

Source-linked AI summary

Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture

Ashesh Jain, Avi Singh, Hema S Koppula, Shane Soh, Ashutosh Saxena

arXiv:1509.05016v1Sep 16, 2015cs.CVcs.AIcs.RO

TL;DR

The paper addresses early maneuver anticipation from partial observations across multiple sensory streams, a robotics problem relevant to safer driving. It proposes an LSTM-based sensory-fusion RNN trained sequence-to-sequence and reports improved maneuver-anticipation performance on 1180 miles of natural driving data, reaching 90.5% precision and 87.4% recall with 3D head-pose features.

Problem
The paper addresses anticipating driving maneuvers several seconds before they occur using partial temporal context and multiple sensory streams, because late driver alerts may leave too little time to avoid dangerous maneuvers.
Method
The paper proposes a sensory-fusion RNN architecture with LSTM units, sequence-to-sequence training for partial-context prediction, and a loss layer designed to prevent over-fitting.
Results
On 1180 miles of natural driving data, adding the driver’s 3D head pose increased precision to 90.5% and recall to 87.4%.
Takeaways & Limitations
The architecture outperformed the previous state-of-the-art on the evaluated maneuver-anticipation task and is described as applicable to other activity-anticipation problems.

Abstract

from arXiv · show

Anticipating the future actions of a human is a widely studied problem in robotics that requires spatio-temporal reasoning. In this work we propose a deep learning approach for anticipation in sensory-rich robotics applications. We introduce a sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal dependencies. We train our architecture in a sequence-to-sequence prediction manner, and it explicitly learns to predict the future given only a partial temporal context. We further introduce a novel loss layer for anticipation which prevents over-fitting and encourages early anticipation. We use our architecture to anticipate driving maneuvers several seconds before they happen on a natural driving data set of 1180 miles. The context for maneuver anticipation comes from multiple sensors installed on the vehicle. Our approach shows significant improvement over the state-of-the-art in maneuver anticipation by increasing the precision from 77.4% to 90.5% and recall from 71.2% to 87.4%.

I. INTRODUCTION

The paper addresses anticipation from limited temporal context and multiple sensors by proposing an LSTM-based sensory-fusion RNN trained for early prediction. On 1180 miles of natural driving data, richer camera features and the architecture improve maneuver-anticipation performance.

Motivation: Activity anticipation predicts future events from limited temporal context and must fuse information from multiple sensors in sensory-rich robotics.Previous anticipation work commonly used single modalities and shallow architectures that cannot handle long temporal dependencies.
Approach: Sequence-to-sequence training maps every partial context (x1, ..., xt) to the future event, enabling anticipation from incomplete observations.At test time, the goal is to predict the future event as soon as possible using only a partial temporal context.
Approach: The proposed architecture uses RNNs with LSTM units to learn rich representations and optimally fuse multiple sensory streams.Each sensory stream is modeled with an RNN, whose high-level representations are nonlinearly combined for final prediction.
Approach: A novel loss layer prevents over-fitting and encourages early anticipation.The loss layer is introduced specifically to support anticipation during sequence-to-sequence training.
Results: 90.5% precision and 87.4% recall were achieved after adding richer camera features, including the driver’s 3D head pose, on 1180 miles of driving data.Without the richer camera features, precision increased from 77.4% to 84.5% and recall from 71.2% to 77.1%.
Contributions: The paper contributes a sensory-fusion RNN-LSTM architecture, a rich-feature vision pipeline, and state-of-the-art maneuver anticipation on 1180 miles of driving data.The data include natural driving from 10 drivers with variation in routes, traffic conditions, and driving styles.

II. RELATED WORK

Prior work studied human activity anticipation, driver behavior, and recurrent sequence prediction, but often relied on shallow or single-modality approaches. This paper situates sensory-fusion RNN-LSTM modeling within that literature and defines anticipation as early prediction from partial observations.

Human Activity Anticipation: Human activity anticipation has supported human-robot collaboration and sociable robot navigation, while video methods have also addressed activity forecasting.These applications include collaborative tasks and planning trajectories around humans.
Human Activity Anticipation: Earlier anticipation approaches commonly used shallow architectures and single data modalities, limiting their ability to model temporal aspects and sensory-rich settings.The paper identifies these as shortcomings of prior human-activity anticipation work.
Driver Maneuver Anticipation: Driver maneuver anticipation uses contextual information from multiple cameras, GPS, and vehicle dynamics to predict maneuvers several seconds before they happen.Prior methods included sensory fusion through concatenated feature vectors.
Driver Maneuver Anticipation: Jain et al. reported that feature concatenation does not capture rich maneuver context and used an AIO-HMM with linear sensory-stream fusion.The present work instead learns an expressive architecture to combine information from multiple sensors.
Recurrent Sequence Prediction: RNNs and LSTM units provide the paper’s sequence-modeling foundation, extending recurrent modeling to multimodal sensory-fusion activity anticipation.The contribution is framed as applying RNN-LSTM models to sensory-rich robotics applications.
Problem Definition: Anticipation predicts an event several seconds early from observations available up to the present, unlike recognition, which uses the complete test-time sequence.Training supplies observation sequences paired with their terminal events; testing uses partial sequences.

A. Recurrent Neural Networks

Recurrent neural networks transform temporal observations into high-level representations and event probabilities, while LSTM units preserve long-term context through gated memory updates. The architecture uses LSTM-based recurrence to address long temporal dependencies relevant to anticipation.

Recurrent Neural Networks: A standard RNN maps an input sequence (x1, x2, ..., xT) to high-level representations (h1, h2, ..., hT) through nonlinear transformations.At each time step, it can produce softmax probabilities for events given observations up to the present.
Recurrent Neural Networks: Standard RNNs with tanh or sigmoid activations suffer from vanishing gradients, limiting their ability to capture long temporal dependencies.
Long-Short Term Memory Cells: LSTM units implement a memory cell that maintains state over time, enabling recurrent networks to remember long-term context dependencies.
Long-Short Term Memory Cells: LSTM updates memory using input and forget gates, computes an output gate, and produces a hidden representation from the memory cell.The memory update combines retained prior memory with new candidate values derived from the current observation and previous hidden representation.
Long-Short Term Memory Cells: The anticipation architecture uses RNNs with LSTM units as its core recurrent model.

IV. NETWORK ARCHITECTURE FOR ANTICIPATION

The anticipation architecture predicts future events from partial temporal context rather than complete sequences, addressing the mismatch between anticipation and recognition. It uses sequence-to-sequence training and combines information from multiple sensor modalities.

NETWORK ARCHITECTURE FOR ANTICIPATION: Anticipation predicts a future event from observations available before the event, whereas activity recognition uses the complete temporal context.At test time, the model must predict from a partial sequence and do so as early as possible.
NETWORK ARCHITECTURE FOR ANTICIPATION: In sensory-rich robotics, different sensor modalities provide contextual information whose fusion is critical to final application performance.
NETWORK ARCHITECTURE FOR ANTICIPATION: Sequence-to-sequence training maps a complete observation sequence to event labels at every time step, teaching the RNN to predict the future from partial observations.
NETWORK ARCHITECTURE FOR ANTICIPATION: The architecture uses LSTM-based RNNs to capture temporal dependencies while explicitly learning anticipation from partial context.

B. Fusion-RNN: Sensory fusion RNN for anticipation

The Fusion-RNN processes sensory streams independently, fuses their high-level recurrent representations, and predicts maneuvers from the combined representation. Its exponentially time-weighted loss encourages earlier correction while reducing pressure to fit very early, context-poor predictions.

Fusion-RNN: Two sensory streams are processed independently by separate RNNs before their high-level representations are concatenated and passed through a fusion layer.
Fusion-RNN: The fusion-layer output is passed to a softmax layer to produce anticipation predictions, and the framework can extend to more sensory streams.
Exponential loss-layer for anticipation: The proposed loss multiplies the prediction loss by an exponential term, so its weight grows with time.The loss uses the probability assigned to the ground-truth event at each time step.
Exponential loss-layer for anticipation: The time-growing loss penalizes later mistakes more strongly, encouraging the model to correct errors as early as possible.
Exponential loss-layer for anticipation: The loss penalizes early mistakes less when insufficient context is available, acting as a regularizer against very-early over-fitting.

D. Model training and data augmentation

Training uses temporal subsequences to augment data and reduce over-fitting in a network with more than 25,000 learnable parameters. Optimization uses RMSprop with a fixed step size of 10^-4.

Model training and data augmentation: More than 25,000 parameters make over-fitting a major challenge for the maneuver-anticipation architecture.
Model training and data augmentation: The training data are augmented by extracting temporal observation subsequences, introducing redundancy that acts as a regularizer.
Model training and data augmentation: The network is trained with RMSprop gradients using a fixed step size of 10^-4.
Model training and data augmentation: Experiments found that the proposed exponentially growing loss layer worked best among the tested softmax-loss variants for anticipation.

V. CONTEXT FOR MANEUVER ANTICIPATION

Maneuver anticipation uses temporal context from multiple vehicle sensors, including driver and road cameras, GPS, and vehicle dynamics. The proposed pipeline improves facial features and fuses sensory streams with an RNN architecture.

The anticipation context combines driver-facing and road-facing cameras, GPS, and vehicle dynamics.
Fusion-RNN combines high-level representations from inside-vehicle and outside-vehicle sensory streams.
CLNF provides more consistent 2D trajectories and estimates the driver’s 3D head pose.
The architecture processes temporal context and outputs probabilities for five maneuvers: turns, lane changes, and straight driving.
The improved vision pipeline tracks 68 fixed facial landmarks with CLNF instead of discriminative points tracked by KLT.

VI. EXPERIMENTS

The experiments evaluate sensory-fusion RNN variants and conventional baselines on natural driving data. The models use separate recurrent processing and compare uniform versus exponentially growing losses.

The vehicle data combines inside and outside video views, speed, and GPS coordinates.
The dataset contains 700 annotated events: 274 lane changes, 131 turns, and 295 randomly sampled straight-driving instances.
The study compares the deep RNN architecture with chance, random forest, SVM, IOHMM, and AIO-HMM baselines.
Fusion-RNN variants process sensor streams separately before fusing their high-level representations, using uniform or exponentially growing losses.

A. Evaluation setup

Evaluation computes maneuver-specific precision, recall, and time-to-maneuver from thresholded probability predictions. Results are averaged across five cross-validation folds and exclude straight driving from precision and recall.

At each time step, the algorithm uses the preceding 0.8 seconds of features to estimate probabilities for the maneuver classes.The feature window contains 20 frames.
A prediction is made using a confidence threshold p_th, with driving straight as the default when other maneuver probabilities are insufficient.
Precision and recall are computed from total instances, true positives, and predicted instances for each maneuver.
Driving straight is excluded from precision and recall because algorithms predict it by default when they lack confidence in other maneuvers.
The reported metrics use uniformly random five-fold cross-validation, training on four folds and testing on the fifth.

B. Results

The results support separate recurrent processing, exponentially growing loss, and richer facial features. The final system improves maneuver-anticipation performance over prior approaches across evaluated comparisons.

The deep-learning variants outperform the previous state-of-the-art a majority of the time on Jain et al.’s features.
F-RNN-EL achieves 6% higher precision and recall than S-RNN when predicting all maneuvers.
F-RNN-EL improves precision and recall by 2% over F-RNN-UL for all-maneuver prediction.The authors conjecture that exponentially growing loss acts like a regularizer by reducing early over-fitting risk.
The CLNF-based vision pipeline increases F-RNN-EL precision by 6% and recall by 10%.It tracks 68 facial landmarks and estimates 3D head pose.
F-RNN-EL has higher precision than AIO-HMM on every maneuver when both use the same features.
The maximum F1-score is 4% higher than AIO-HMM with the same features and 13% higher with the new vision pipeline.

VII. CONCLUSION

The paper presents an RNN-LSTM architecture that fuses multiple sensory streams and learns anticipation from partial temporal context. On natural driving data, it outperforms prior state of the art, with further gains from head tracking and 3D head pose.

The proposed architecture uses RNNs with LSTM units to model long temporal dependencies while fusing multiple sensory streams.It is trained in a sequence-to-sequence manner to anticipate from partial temporal context.
The architecture explicitly learns to anticipate using only a partial temporal context and includes a loss layer that prevents over-fitting.
The deep learning architecture outperformed the previous state of the art on 1180 miles of natural driving data.
The confusion matrix represents algorithm predictions by rows and actual maneuvers by columns, with diagonal values indicating precision.
90.5% precision and 87.4% recall were achieved when head tracking was improved and the driver’s 3D head pose was included as a feature.The reported baseline architecture improved precision from 78% to 84.5% and recall from 71.1% to 77.1%.