Source-linked AI summary
End-to-end Learning of Driving Models from Large-scale Video Datasets
Huazhe Xu, Yang Gao, Fisher Yu, Trevor Darrell
TL;DR
Existing driving-policy methods are limited by narrow, vehicle-specific or simulated training settings, motivating models learned from diverse crowdsourced video. The paper proposes an FCN-LSTM that predicts future egomotion distributions from monocular observations and prior vehicle state, using semantic segmentation as privileged side-task supervision. It evaluates future-egomotion prediction on held-out sequences and reports that side-task training improves performance, while real-car execution remains future work.
Problem
Learning robust perception-action policies remains difficult because existing deep visuomotor approaches are generally limited to in-situ models learned from a single vehicle or simulation environment.
Method
The paper learns a generic future-egomotion distribution from monocular camera observations and previous vehicle state using an end-to-end FCN-LSTM trained on large-scale crowdsourced driving video.
Results
Semantic-segmentation side-task supervision improves future-egomotion prediction performance under the privileged learning paradigm on held-out sequences.
Takeaways & Limitations
Large-scale crowdsourced driving data and privileged side-task training support learning a generic motion model across heterogeneous vehicle actuators.
Takeaways & Limitations
The model is evaluated on held-out sequences and is not yet executed on a real car, with policy coverage across undemonstrated regions left for future work.
Abstract
from arXiv · showhide
Robust perception-action models should be learned from training data with diverse visual appearances and realistic behaviors, yet current approaches to deep visuomotor policy learning have been generally limited to in-situ models learned from a single vehicle or a simulation environment. We advocate learning a generic vehicle motion model from large scale crowd-sourced video data, and develop an end-to-end trainable architecture for learning to predict a distribution over future vehicle egomotion from instantaneous monocular camera observations and previous vehicle state. Our model incorporates a novel FCN-LSTM architecture, which can be learned from large-scale crowd-sourced vehicle action data, and leverages available scene segmentation side tasks to improve performance under a privileged learning paradigm.
1. Introduction
The paper addresses limits of end-to-end driving policies trained on narrowly calibrated data by learning generic future-motion predictions from large-scale crowdsourced video. It introduces an FCN-LSTM model, curated data, and semantic-segmentation side-task supervision, while leaving real-vehicle execution for future work.
- Learning-based driving policies are motivated by the need to handle complex, rare, and multi-agent scenarios.
- Prior pixel-to-actuation methods depend on data collected with a specifically calibrated actuation setup or corresponding simulations.
- The paper advocates learning generic driving models from large-scale uncalibrated crowdsourced dashcam video and releases a curated dataset.
- The proposed model predicts future motion from current visual observations and previous vehicle state, enabling actuator-independent motion plans.
- The FCN-LSTM jointly learns demonstration and semantic-segmentation losses while producing multimodal predictions.
- The evaluation predicts future egomotion on held-out video examples, while real-car driving and coverage of undemonstrated policy regions remain future work.
2. Related Work
Related work spans early and modern pixel-to-action driving systems, interpretable affordance-based policies, visual prediction, privileged-information learning, and recurrent convolutional sequence models. The paper combines these directions through an FCN-LSTM driving model trained with side information.
- Early neural driving systems demonstrated navigation from pixels in simple scenarios, including lane-following, obstacle avoidance, and highway driving.
- Affordance-based methods provide interpretable intermediate outputs but require defining comprehensive measures and manually associating them with actions.
- Visual-prediction approaches model future driving frames using methods such as VAE-GAN combinations, treating driving as a video-prediction problem.
- Privileged-information learning uses additional training-time labels or metadata to improve models that rely only on test-time observations.
- The paper applies recurrent convolutional sequence modeling through a novel combination of a fully convolutional network and an LSTM.
3. Deep Generic Driving Networks
The paper defines generic driving models that predict feasible future egomotion from visual observations and vehicle-state history, rather than vehicle-specific actuation. It combines visual and temporal encoders, supports multimodal action distributions, evaluates sequence predictions with perplexity, and uses semantic segmentation as privileged side supervision.
- 3.1. Generic Driving Models: The driving model F(s, a) scores the feasibility of a potential motion action a under world state s.The state incorporates prior observations and vehicle state.
- 3.1. Generic Driving Models: Generic models predict future egomotion likelihoods from raw pixels and current and prior vehicle-state signals, instead of directly predicting vehicle-specific actuation.The action space can represent coarse actions or finer future egomotion headings.
- 3.2. Architecture: The proposed FCN-LSTM fuses a fully convolutional visual encoder with an LSTM temporal encoder and jointly supports motion prediction and pixel-level supervised tasks.The visual encoder preserves spatial information, while the LSTM integrates visual and sensor history.
- 3.2. Architecture: Temporal convolution is evaluated as an alternative to LSTM for fusing multiple visual representations across time.The temporal convolution uses an n × 1 kernel over the time dimension.
- 3.3. Driving Perplexity: Perplexity evaluates the probability assigned to held-out driving sequences, while accuracy compares the most probable predicted action with the driver’s carried-out action.The model outputs a probability distribution over possible actions, including multiple modes for ambiguous scenes.
- 3.4. Discrete and Continuous Action Prediction: Continuous action distributions are represented non-parametrically by discretizing action spaces into many small bins because a single parametric family may not fit all scenarios.Discrete actions are trained by minimizing perplexity, effectively using cross entropy against the demonstrated action.
- 3.5. Driving with Privileged Information: Privileged training adds semantic segmentation as a side task to raw-pixel motion prediction, encouraging meaningful features and improving performance on small relevant scene phenomena.This contrasts with motion-reflex learning from raw pixels and mediated perception driven only by semantic labels.
4. Dataset
BDDV combines real driving video with sensor data across diverse locations and conditions, providing a large-scale basis for learning generic driving models.
- Dataset composition: BDDV contains real driving videos with GPS/IMU data across cities, highways, towns, and rural areas.The dataset also includes sensor modalities such as GPS, IMU, gyroscope, and magnetometer.
- Scale: Over 10,000 hours of driving dash-cam video make BDDV at least two orders of magnitude larger than prior public benchmark datasets.Robotcar provides 214 hours, while KITTI has fewer than an hour and Cityscapes no more than 100 hours upon request.
- Sensor information: The dataset’s sensor readings and fused measurements can support recovery of vehicle trajectories and dynamics.Available measurements include course and speed in addition to raw sensor readings.
- Diversity: BDDV targets generic driving models across scene types, car makes and models, driving behaviors, lighting conditions, and locations.Examples include nighttime, daytime, city, highway, and rural driving.
5. Experiments
Experiments evaluate discrete and continuous future-motion prediction, compare architectural variants, and test privileged segmentation supervision. The FCN-LSTM predicts driving actions from video and state, while privileged training achieves the strongest reported discrete-action performance.
- Experimental setup: 21,808 videos were used for training, 1,470 for validation, and 3,561 for testing after preprocessing.Videos were truncated to 36 seconds, resized to 640 × 360, and sampled at 3Hz.
- Discrete action prediction: The discrete task predicts straight, stop, left-turn, or right-turn actions over the next 1/3 second.Evaluation uses log perplexity and maximum-likelihood accuracy on held-out videos.
- Discrete action prediction: Temporal convolution improves with longer history, while CNN-LSTM achieves comparable performance to TCNN9 and FCN remains comparable to CNN without privileged supervision.Inverse-frequency loss weighting encourages rare-action predictions but does not improve perplexity.
- Discrete action prediction: The model changes from predicting straight to stop as a traffic light changes from yellow to red.It also shifts toward stopping as the vehicle approaches a stopped car.
- Continuous action prediction: Data-driven binning performs best and linear binning worst for continuous lane-following prediction.The task predicts angular speed over the future 1/3 second using 180 bins.
- Learning with privileged information: Privileged Training achieves the best perplexity and accuracy among the evaluated approaches.It uses segmentation supervision during training and predicts stopping more reliably in examples involving a distant red light or a stopped vehicle.
6. Conclusion
The paper presents an end-to-end approach for learning generic driving models from large-scale crowd-sourced video. Its FCN-LSTM predicts future egomotion from monocular observations and prior state, while semantic segmentation side tasks improve performance under privileged learning.
- Conclusion: The approach learns generic driving models from large-scale crowd-sourced video using an end-to-end trainable architecture.The accompanying dataset is designed for learning driving models from crowd-sourced driving behaviors.
- Conclusion: The model predicts a distribution over future egomotion from monocular camera observations and previous egomotion states.Its FCN-LSTM architecture learns from driving behaviors and can use semantic segmentation as a side task.
- Conclusion: The study evaluates future-egomotion prediction and privileged learning on held-out sequences across diverse conditions.The conclusion frames the dataset and architecture as resources for generic driving-model learning.