PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveGAN: Towards a Controllable High-Quality Neural Simulation

Seung Wook Kim, Jonah Philion, Antonio Torralba, Sanja Fidler

arXiv:2104.15060v1cs.CVcs.RO

TL;DR

Realistic robotics simulators are important, but hand-crafted environments are costly and data-driven methods have limited ability to model controllable dynamics in pixels. DriveGAN learns a high-quality neural simulator from video and actions, disentangles scene components for unsupervised control, and supports differentiable re-simulation. Trained on multiple datasets including 160 hours of real driving data, it surpasses existing neural simulators and enables scene controls such as weather and non-player-object locations.

  • Problem

    Hand-crafted simulators require extensive design effort, while the paper seeks scalable pixel-space simulation from video frames and associated actions without human annotations.

  • Method

    DriveGAN combines a disentangled image latent space with a learned dynamics engine and image generator to simulate future frames, edit scene factors, and re-simulate recorded trajectories.

  • Results

    DriveGAN surpasses existing neural simulators by a significant margin while supporting controls over weather, non-player-object locations, and re-simulation of recorded scenes.

  • Takeaways & Limitations

    Differentiable, disentangled simulation enables interactive scenario editing and uses real-world videos to recover variations and train robots in recreated environments.

Abstract

from arXiv · show

Realistic simulators are critical for training and verifying robotics systems. While most of the contemporary simulators are hand-crafted, a scaleable way to build simulators is to use machine learning to learn how the environment behaves in response to an action, directly from data. In this work, we aim to learn to simulate a dynamic environment directly in pixel-space, by watching unannotated sequences of frames and their associated action pairs. We introduce a novel high-quality neural simulator referred to as DriveGAN that achieves controllability by disentangling different components without supervision. In addition to steering controls, it also includes controls for sampling features of a scene, such as the weather as well as the location of non-player objects. Since DriveGAN is a fully differentiable simulator, it further allows for re-simulation of a given video sequence, offering an agent to drive through a recorded scene again, possibly taking different actions. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data. We showcase that our approach greatly surpasses the performance of previous data-driven simulators, and allows for new features not explored before.

1. Introduction

DriveGAN frames data-driven neural simulation as a scalable alternative to hand-designed environments, learning controllable high-fidelity scene dynamics from video and actions. It additionally supports user control over scene elements and re-simulation of recorded scenarios.

  • Motivation: Robotic simulators support safe, repeatable testing of autonomous systems before real-world deployment.They also enable fair comparisons because scenarios can be repeated under controlled conditions.
  • DriveGAN: DriveGAN learns high-fidelity real-world scenes from user controls while enabling control over weather and object placement.Its stated simulator capabilities include action-conditioned rendering and manipulation of scene aspects.
  • Motivation: Hand-designed simulators require substantial content creation and complex behavior modeling, motivating scalable data-driven alternatives.The paper notes that building realistic virtual environments can take years and hundreds of artists and engineers.
  • Motivation: Prior data-driven scene-generation methods synthesize layouts or objects but do not model environmental dynamics and object behaviors.These methods target diversity or unsupervised scene construction without learning how the environment evolves over time.
  • Motivation: Neural simulators learn pixel-space environment responses to agent actions from large video datasets, avoiding human annotations beyond cheaply obtained actions.The paper follows this route despite the challenge of learning complex worlds and dynamic agents from high-resolution views.
  • DriveGAN: DriveGAN’s differentiable simulator can re-create recorded scenarios so an agent can drive through them again with different actions.The model learns from video footage and associated ego-agent actions.

2. Related Work

Prior work spans video generation, data-driven scene construction, neural simulation, and model-based reinforcement learning. DriveGAN focuses on high-quality future-frame simulation from controls in real-world driving data while adding interactive scene editing.

  • Video generation and prediction: Video-generation and prediction models use architectures including VAEs, autoregressive models, flow-based models, and GANs to produce realistic frames and transitions.Conditional video generation extends these approaches by incorporating additional information such as controls.
  • Data-driven simulation: Data-driven simulation methods generate scene parameters, LiDAR point clouds, or future frames, with World Model learning transitions through a VAE and LSTM.These approaches differ in whether they construct scenes, synthesize sensor outputs, or predict controlled future observations.
  • Neural simulation and model-based RL: GameGAN combines a GAN and memory module to mimic game engines, while model-based reinforcement learning learns dynamics models for planning.Earlier neural simulators often targeted simple environments with known ground-truth simulators.
  • DriveGAN: DriveGAN applies neural simulation to real-world driving data and emphasizes higher-quality simulations with interactive scene editing.Its related-work positioning contrasts real-world driving with prior work on simpler environments.

3. Methodology

DriveGAN learns a controllable neural simulator by separating image themes and spatial content, then modeling action-conditioned latent dynamics. Its differentiable design supports scene editing, stochastic simulation, and re-simulation from real videos.

  • Overview: DriveGAN targets controllable simulation by learning future frames from observations and actions while separately controlling the ego agent and scene factors.The method aims to modify objects or background conditions in addition to applying agent actions.
  • Pre-trained Latent Space: The encoder produces spatial zcontent and global ztheme codes, which a StyleGAN-based decoder uses to represent content and appearance separately.ztheme controls AdaIN scaling and bias, while zcontent supplies spatial information to the generator.
  • Pre-trained Latent Space: KL-loss balancing is essential because stronger reconstruction can produce an overfit latent space whose transitions are harder for the dynamics model to learn.The β parameter trades off reconstruction quality against conformity to the prior used for dynamics learning.
  • Dynamics Engine: A pre-trained encoder-decoder is fixed while the Dynamics Engine learns latent transitions conditioned on actions, reducing training complexity compared with image-space modeling.The engine uses recurrent spatial processing to model viewpoint shifts and produces the next latent codes for decoding.
  • Dynamics Engine: The Dynamics Engine disentangles content into action-dependent and action-independent features without supervision, using separate recurrent pathways and reparameterized variables.The architecture assigns action-dependent features to scene changes driven by controls and action-independent features to other content variation.
  • Differentiable Simulation: Because DriveGAN is differentiable, it can infer latent scene factors from real videos, replay scenarios with different actions, and modify weather or objects during re-simulation.The model also uses stochastic reparameterization to generate diverse future scenarios.

4. Experiments

Experiments evaluate DriveGAN across simulated, indoor, and real-world driving datasets using video-quality, action-consistency, human-preference, controllability, differentiable-simulation, and downstream-prediction tests. The results show high-quality, action-faithful generation together with editable scene factors and successful recovery of actions and intermediate frames.

  • Datasets and evaluation: Experiments use Carla, Gibson, and Real World Driving datasets containing extracted frames paired with ego-agent actions.The RWD dataset represents approximately 160 hours of driving data.
  • Datasets and evaluation: DriveGAN is evaluated against Action-RNN, SAVP, GameGAN, and World Model baselines using shared initial screens and action sequences.World Model* additionally uses DriveGAN’s proposed latent space for its MDN-RNN component.
  • Quantitative and human evaluation: DriveGAN achieves lower FVD than every baseline except GameGAN on Gibson, while human subjects strongly prefer its visual quality, including on Gibson.The authors attribute Gibson’s exception partly to brightness changes in DriveGAN outputs.
  • Quantitative and human evaluation: Action consistency is measured by predicting the actions responsible for transitions between generated frames and comparing predictions with input actions.The predictor is trained on real-video image pairs using mean-squared error.
  • Controllability: DriveGAN disentangles scene factors without supervision, enabling sampling or swapping of theme, independent, and content latents to alter weather, interiors, layouts, and objects.Spatial content latents allow cell-level edits such as erasing or adding trees and buildings.
  • Applications and additional experiments: DriveGAN supports editable simulation scenarios, including replay with modified weather or objects and sampling the first screen unlike some previous work.Simulated sequences were also realistic enough for LiftSplat to produce accurate bird’s-eye-view lane predictions.
  • Differentiable simulation: Differentiable simulation recovers driving actions and stochastic variables from videos, with optimized-action losses of 1.91 for Carla and 0.57 for RWD.These losses are lower than baseline values of 3.64 and 1.01 and support recovery of unobserved actions and intermediate frames.
  • Applications and additional experiments: Compared with a projected StyleGAN latent space, DriveGAN obtains 411.9 FVD and 0.050 action-prediction loss versus 636.8 FVD and 0.225 loss.The comparison uses 25% of Gibson data because the projection process requires approximately 19,200 GPU hours for Gibson.

5. Conclusion

The conclusion presents DriveGAN as a controllable high-quality simulator whose latent dynamics support unsupervised scene editing and differentiable re-creation of real-world video environments.

  • 5. Conclusion: DriveGAN combines a novel encoder and image GAN with a dynamics engine that learns transitions in the resulting latent space.The conclusion describes this design as supporting controllable high-quality simulation.
  • 5. Conclusion: Unsupervised disentanglement lets users sample and modify different scene components interactively to create unique simulation scenarios.The conclusion frames scene editing as a direct consequence of the disentangled latent representation.
  • 5. Conclusion: Differentiable simulation supports discovering variation factors from real-world videos and training robots in recreated environments.

A. Model Architecture and Training

This architecture section specifies tensor-dimension notation for the model architecture and training descriptions.

  • A. Model Architecture and Training: Unless otherwise noted, tensor dimensions are written as H × W × D for feature-map height, width, and channel count.

A.1. Pre-trained Latent Space

The pre-trained latent-space stage learns an image representation through an encoder-decoder setup, producing disentangled content and theme latents for the dynamics engine.

  • A.1. Pre-trained Latent Space: The latent space is pretrained with an encoder, generator, and discriminator.
  • A.1. Pre-trained Latent Space: The encoder produces zcontent and ztheme, which together form the disentangled latent space used by the dynamics engine.The figure identifies Gaussian blocks as reparameterization steps.

A.1.1 Encoder

The encoder maps each RGB image into disentangled theme and spatial-content latent codes, while the decoder and multi-scale discriminators support high-quality image synthesis.

  • Encoder: The encoder ξ produces ztheme ∈R128 and zcontent ∈R4×4×64 from a 256×256×3 RGB image.Separate feature-extraction and encoding heads produce the two latent components.
  • Decoder: The generator transforms zcontent through a convolutional input and uses ztheme through eight 1024-dimensional linear layers for adaptive instance normalization.Its design follows StyleGAN while concatenating zcontent with the initial constant tensor.
  • Discriminators: The image discriminator uses three scales: a full-image scalar discriminator, a 16×16 patch discriminator, and an 8×8 patch discriminator on downsampled images.The adversarial inputs are real and generated 256×256×3 images, with the third discriminator receiving 2× downsampled inputs.

A.1.4 Training

Training first learns an image latent space with combined VAE and GAN objectives, then trains a recurrent Dynamics Engine to predict disentangled latent transitions conditioned on actions.

  • Image pretraining: The pretraining objective combines the VAE and GAN losses as Lpretrain = LVAE + LGAN.The reconstruction term uses perceptual rather than pixel-wise distance, with separate KL weights for theme and content.
  • Dynamics Engine: The Dynamics Engine learns latent-code transitions from pre-extracted image codes, given the agent action at each time step.This separates image representation learning from temporal transition learning.
  • Action-dependent dynamics: The convolutional LSTM models action-dependent spatial transitions, combining the action, theme code, and content code before producing the next latent variables.Its hidden state is reparameterized into action-dependent content and theme codes.
  • Action-independent dynamics: A second plain LSTM receives only the current latent code and therefore models information independent of the action.Its input is the flattened 1152-dimensional latent representation.
  • Latent adversarial training: Training uses separate single-latent and temporal action-conditioned discriminators to regularize generated latent codes and their transitions.The temporal discriminator combines representations of consecutive latent codes with an action embedding over 32-step sequences.

A.2.2 Training

Dynamics training uses scheduled ground-truth conditioning, latent reconstruction, and KL regularization to align generated latent trajectories with training sequences.

  • Optimization: The model is optimized for 400,000 steps with Adam at learning rate 0.0001 using batches of 128 sequences of 32 time steps.A warm-up phase initially supplies ground-truth latent codes for 18 steps and linearly reduces that number to 1.
  • Training objectives: A mean squared latent reconstruction loss weighted by 10.0 encourages generated zt to match the input latent codes.The training procedure also reduces the KL penalty for the action-dependent latent variables.
  • Dataset-specific regularization: KL weights differ by dataset: Carla uses βadep = βaindep = 0.1, while Gibson and RWD use βadep = 0.5 and βaindep = 0.25.The theme KL weight remains βtheme = 1.0 in all listed datasets.

B. Additional Analysis on Experiments

Additional experiments evaluate discriminator design, differentiable sequence optimization, and interactive scene control, showing improved image quality and lane alignment for optimized simulations.

  • Discriminator analysis: Multi-patch multi-scale discrimination improves Carla image FID from 72.3 with the StyleGAN discriminator to 67.1.The authors attribute the architectural choice to datasets containing diverse objects across multiple locations.
  • Differentiable simulation: DriveGAN-optimized sequences produce lane predictions that follow ground-truth lanes across different look-ahead distances.The comparison uses Random, DriveGAN, and Ground-Truth distances between BEV lane predictions and fitted quadratics.
  • Interactive control: The simulator UI exposes keyboard controls for steering and speed and allows users to randomize theme, content, or action-independent latent variables.Users can also select predefined themes and objects for targeted scene changes.
Loading 2104.15060v1…