Source-linked AI summary
Learning a Driving Simulator
Eder Santana, George Hotz
TL;DR
The paper addresses how to learn realistic road-video simulation when direct interaction with the driving world is limited and pixel-space prediction is difficult. It embeds frames with variational autoencoders and learned adversarial costs, then trains action-conditioned RNN transitions in latent space. The resulting model produces realistic-looking predictions for multiple frames, while simulating all relevant driving events remains unresolved.
Problem
The paper studies learned video simulation for real-world driving, where prior approaches lacked compact representations or action-conditioned transitions and direct prediction is high-dimensional and correlated.
Method
The approach trains an autoencoder with variational Gaussian regularization and adversarially learned cost functions, followed by an action-conditioned RNN transition model in the embedded space.
Results
The transition model preserves road structure for 100 frames and generates realistic-looking driving events, although it fails to simulate curves.
Takeaways & Limitations
Learned latent-space transitions can generate realistic-looking road video without explicitly optimizing a cost function in pixel space.
Abstract
from arXiv · showhide
Comma.ai's approach to Artificial Intelligence for self-driving cars is based on an agent that learns to clone driver behaviors and plans maneuvers by simulating future events in the road. This paper illustrates one of our research approaches for driving simulation. One where we learn to simulate. Here we investigate variational autoencoders with classical and learned cost functions using generative adversarial networks for embedding road frames. Afterwards, we learn a transition model in the embedded space using action conditioned Recurrent Neural Networks. We show that our approach can keep predicting realistic looking video for several frames despite the transition model being optimized without a cost function in the pixel space.
1 Introduction
The paper frames learned simulation as an alternative to hand-coded simulators for driving, focusing on generating front-facing road video from human-agent examples. It targets compact, action-conditioned video prediction without graphics-engine or world-model assumptions.
- Driving offers abundant, contextually rich data for vision-based self-driving research and may extend to action recognition and planning.
- When unbounded interaction with the world is unaffordable, researchers can either hand-code a simulator or learn to simulate.Hand-coded simulation requires domain expertise to model physics and world randomness.
- This paper learns to simulate aspects of the world from human-agent examples by generating front-facing windshield-camera video.
- The approach learns realistic-looking video predictions with a low-dimensional compact representation and action-conditioned transitions, while leaving simulated-world controls for future work.
2 Dataset
The released dataset contains windshield-camera driving video paired with vehicle and navigation sensors, providing data for machine-learning and controls research. This paper focuses on downsampled camera frames, steering angle, and speed.
- The publicly released driving dataset uses the same video and sensors as comma.ai’s self-driving car test platform.
- A windshield-mounted Point Grey camera on a 2016 Acura ILX captured road images at 20 Hz across 7.25 hours and 11 videos.
- The dataset includes car speed, steering angle, GPS, gyroscope, and IMU measurements, with sensors interpolated to 100Hz.
- Raw sensor and camera-frame data are released in HDF5 files for machine-learning and controls software.
- The paper focuses on camera frames, steering angle, and speed after downsampling frames to 80 × 160 and renormalizing pixels between -1 and 1.
3 Problem definition
The paper defines road simulation as predicting the next frame from a history of frames and associated speeds and steering angles. It decomposes this high-dimensional problem by embedding frames in a Gaussian latent space before learning transitions.
- Road simulation estimates F to predict x_t+1 from n preceding frames, ego speeds, and steering angles.The mapping takes frame sequences and two corresponding control-signal sequences as input.
- The high-dimensional, highly correlated formulation can converge slowly or underfit, while prior complex-texture video models lacked action-conditioned transitions and compact representations.
- The paper learns F piecewise, first embedding frames into a 2048-dimensional Gaussian latent space and then learning transitions there.The latent-space formulation replaces direct pixel-space transition learning and uses a Gaussianity assumption to support realistic decoding within a high-density region.
4 Driving simulator
The driving simulator separates frame embedding from action-conditioned latent-space prediction. It combines a variational autoencoder with learned discriminator-based costs and an RNN transition model.
- 4 Driving simulator: The architecture uses an autoencoder for dimensionality reduction and an action-conditioned RNN for learning transitions.The complete model couples these two separable components rather than training an end-to-end system.
- 4.1 Autoencoder: The autoencoder combines a Gaussian latent prior with discriminator-based feature and adversarial costs to learn frame embeddings.The encoder, generator, and discriminator are jointly optimized using the VAE regularizer, discriminator-layer reconstruction error, and GAN cost.
- 4.2 Transition model: After the autoencoder is trained and fixed, the transition model predicts the next latent code from the current code, hidden state, and concatenated speed and steering signals.The transition model is trained with mean squared error, and predicted codes are decoded into future frames.
- 4.2 Transition model: Training uses teacher forcing for the first five frames of 15-frame sequences, then feeds predicted outputs back as inputs for the remaining ten frames.This setup exposes the RNN to its own predictions during multi-step simulation.
5 Results
The learned cost function produced more visually appealing road reconstructions than MSE, while the transition model preserved road structure over extended predictions but failed on curves.
- Generative adversarial networks cost functions produced more visually appealing decoded road images than mean square error, despite similar MSE values.Both models had MSE in the order of 10−2 and PSNR in the order of 10.
- The transition model generated realistic-looking samples even though its generator was not optimized to make those samples realistic.The samples supported the assumption that the transition model remained within the learned code space.
- Mean square error reconstructions blurred lane markings into a single long lane and failed to preserve leading car edges.Preserving leading car edges matters because they provide clues for visual odometry and leading car distance estimation.
- The transition model preserved road structure for 100 frames and simulated passing lanes, approaching leading cars, and leading cars moving away.It was trained on 5Hz videos and generated these events from different seed frames.
- The model failed to simulate curves, quickly straightening lane geometry when initialized with curved-road frames.
6 Conclusions
The paper combines autoencoders and RNNs to learn a driving simulator from video, producing realistic-looking images and transitions while leaving comprehensive event simulation for future research.
- The approach first trains an autoencoder with generative adversarial network cost functions, then trains an RNN transition model in the embedded space.
- Both the autoencoder and transition model produced realistic-looking results, but more research is needed to simulate all relevant driving events.
- The paper releases a driving dataset with video and sensors, along with code for training the investigated networks.