Source-linked AI summary
Learning to drive from a world on rails
Dian Chen, Vladlen Koltun, Philipp Krähenbühl
TL;DR
The paper addresses how to learn safe, interactive vision-based driving policies from passive logs despite sparse safety-critical data. It factorizes dynamics under a world-on-rails assumption, computes tabular Bellman action-values by dynamic programming, and distills them into a reactive visuomotor policy. The method achieves stronger driving performance with substantially less data and is more sample-efficient on ProcGen navigation tasks.
Problem
Sparse safety-critical observations and biased expert trajectories make it difficult to learn driving policies that handle costly mistakes from passive experience.
Method
The method factorizes a world-on-rails model into a replayed environment and learned ego-vehicle dynamics, then uses dynamic-programming action-values to supervise a reactive visuomotor policy.
Results
25% higher driving score was achieved on the CARLA leaderboard with 40× less training data, while ProcGen navigation required an order of magnitude fewer observations than baseline algorithms.
Takeaways & Limitations
World-on-rails supervision can support effective driving policies in reactive environments and improve sample efficiency across driving and navigation benchmarks.
Takeaways & Limitations
The world-on-rails assumption does not hold in simulators or the real world because other agents react to the ego vehicle and its actions.
Abstract
from arXiv · showhide
We learn an interactive vision-based driving policy from pre-recorded driving logs via a model-based approach. A forward model of the world supervises a driving policy that predicts the outcome of any potential driving trajectory. To support learning from pre-recorded logs, we assume that the world is on rails, meaning neither the agent nor its actions influence the environment. This assumption greatly simplifies the learning problem, factorizing the dynamics into a nonreactive world model and a low-dimensional and compact forward model of the ego-vehicle. Our approach computes action-values for each training trajectory using a tabular dynamic-programming evaluation of the Bellman equations; these action-values in turn supervise the final vision-based driving policy. Despite the world-on-rails assumption, the final driving policy acts well in a dynamic and reactive world. At the time of writing, our method ranks first on the CARLA leaderboard, attaining a 25% higher driving score while using 40 times less data. Our method is also an order of magnitude more sample-efficient than state-of-the-art model-free reinforcement learning techniques on navigational tasks in the ProcGen benchmark.
1. Introduction
The paper addresses sparse safety-critical data and biased expert trajectories by learning a policy that simulates action effects offline. A world-on-rails factorization enables action-value supervision from recorded driving logs, yielding strong benchmark results.
- Motivation: Safety-critical observations are rare in expert trajectories, making behavior-cloning agents difficult to train for costly mistakes.Reinforcement learning also requires prohibitively many samples for safe driving, even in simulation.
- Approach: The method learns a navigation policy that recovers from mistakes without executing them during training.A static world model simulates actions, computes action-values, and supervises a reactive visuomotor policy using raw RGB images and speed readings.
- Approach: The world-on-rails assumption makes the world model factorize into a reactive ego-vehicle component and a passively replayed environment.The agent’s actions affect only its own state, while the environment follows pre-recorded trajectories.
- Approach: Dynamic programming and backward induction compute action-values over potential agent states, providing dense supervision across viewpoints, speeds, and command augmentations.The action-values are computed from a predefined reward and the learned forward model.
- Results: 25% higher driving score was achieved on the CARLA leaderboard while using 40× less training data than the prior top-ranking entry.The method also outperformed prior methods on NoCrash and used an order of magnitude fewer observations than baselines on ProcGen navigation tasks.
2. Related Work
The paper sits between imitation learning, model-based reinforcement learning, policy distillation, and cost-volume planning. Its distinctive combination is learning from passive logs and using tabular action-values to supervise the final visuomotor policy.
- Imitation learning: Imitation-learning driving methods rely on human or privileged-expert trajectories, limiting learning to the expert’s observations and actions.This paper instead learns from passive driving logs.
- Model-based reinforcement learning: Model-based reinforcement learning uses forward models to generate imagined trajectories or improve value estimates, whereas this work factorizes the forward world model.The supplied passage introduces the comparison but does not detail the factorization further.
- Policy distillation: Prior policy-distillation methods supervise visuomotor agents with privileged policies, while this work distills a tabular action-value function.A reactive driving policy is formed only after distillation.
- Cost-volume planning: Cost-volume planners score future ego-vehicle trajectories for inference, whereas this method uses action-values for offline policy supervision and ground-truth states.The comparison distinguishes both training-time use and state information.
3. Method
The method factorizes world dynamics into a learned ego-vehicle model and replayed world trajectories, then evaluates Bellman action-values by backward induction. These values supervise a reactive visuomotor policy over many ego states and actions.
- Method overview: The system learns a reactive policy π(I) from trajectory data containing sensor readings, driving logs, and executed actions.The logs include ego-vehicle, traffic-participant, lane, and traffic-light states.
- Method overview: Algorithm 1 fits an ego-vehicle forward model, estimates action-values, and distills a visuomotor policy.The three stages are represented by FitForward, EstimateQ, and DistillPolicy.
- Factorized forward model: The forward model separates controlled-vehicle dynamics from a world component that cannot react to ego actions under the world-on-rails assumption.The world transition depends only on its prior world state, allowing recorded trajectories to represent the environment directly.
- Factorized forward model: The world-on-rails assumption is unrealistic in simulators and the real world, where other agents react to the ego vehicle.The paper reports that the simplification nevertheless outperforms full-world forward models in its experiments.
- Action-value estimation: Bellman action-values are evaluated over recorded world states and all possible ego-vehicle states using dynamic programming and backward induction.A compact ego state enables a tabular value approximation with discretization, interpolation, and a discretized action maximization.
- Policy distillation: The resulting action-values represent optimal-policy returns and supervise the visuomotor policy for arbitrary ego states and augmented observations.An entropy regularizer encourages more diverse policy outputs.
4. Implementation
Implementation uses a compact ego-vehicle dynamics model, discretized Bellman values, and a vision policy conditioned on RGB images and speed. Rewards encode lane following, stopping, braking, and command-dependent driving behavior.
- Forward model: The ego-vehicle model is trained on trajectories spanning steering, throttle, and braking actions, predicting the next position, orientation, and speed.Inputs include 2D location, orientation, and speed; the model is trained autoregressively with L1 loss.
- Bellman evaluation: The value function is discretized over position, velocity, and orientation bins, with linear interpolation for states between bins.The implementation uses 96 × 96 spatial bins, 4 velocity bins, and 5 orientation bins.
- Policy network: The policy uses a ResNet34 RGB backbone, concatenates ego speed, and predicts a categorical distribution over discretized actions.The network first applies global average pooling before the fully connected layers.
- Policy network: High-level commands are supervised simultaneously, while semantic segmentation provides an auxiliary loss that improves generalization to new environments.Image augmentations are also used.
- Reward design: The reward combines ego state, world state, action, and high-level command, using lane information to determine the target lane.Zero-speed zones and brake rewards remove the need for an explicit collision penalty.
5. Experiments
Experiments evaluate the method on CARLA, NoCrash, and ProcGen using camera-based driving and navigational tasks. The method outperforms prior approaches while using less data, and ablations support dense action-value supervision and auxiliary semantic segmentation.
- Evaluation setup: The CARLA evaluation uses RGB images, speed, and high-level navigation commands to predict steering, throttle, and braking across challenging routes and held-out towns.The CARLA leaderboard measures driving score, route completion, and infraction score; NoCrash evaluates routes under varying traffic densities in training and unseen towns.
- CARLA results: 25% higher driving score than the prior leading entry was achieved on the CARLA leaderboard while using 40× less data.The driving score is the leaderboard’s primary ranking measure and combines route completion and traffic-related performance through the evaluation metrics.
- NoCrash results: The method outperforms LBC and IA on all 12 NoCrash tasks and conditions, including traffic scenarios in both towns despite training only on Town1.The comparison uses CARLA 0.9.10; IA was trained with all towns and weathers, while the presented method does not require expert actions.
- Model-based comparisons: The approach outperforms model-based baselines on almost all CoRL17 tasks, while Dreamer with a factorized world model still fails to generalize beyond straight driving.The easier CoRL17 benchmark was added because NoCrash was too difficult for Dreamer; the observed weakness may reflect few turning scenarios in its training set.
- Ablations: Camera and speed augmentation consistently improve NoCrash performance, supporting dense action-values computed through factorized Bellman updates.The augmented supervision also performs well without trajectory-noise injection.
- Ablations: Semantic-segmentation auxiliary supervision consistently outperforms the action-only baseline, especially under generalization settings.The same pattern was observed for the LBC baseline.
6. Conclusion
The paper argues that treating the world as independent of the agent significantly simplifies reinforcement learning, while preserving strong driving and navigation performance despite modeling constraints.
- The world-on-rails assumption factorizes dynamics into an agent-independent environment and an ego-vehicle model, simplifying reinforcement learning.
- Even though true independence rarely holds, the reported training-efficiency gains outweigh the resulting modeling constraints.
- The method is an order of magnitude more sample-efficient than state-of-the-art reinforcement learning on challenging ProcGen navigation tasks.
B. ProcGen Training Levels Returns
Figure 5 compares the method with PPO, PPG, and privileged-information PPO on ProcGen navigational tasks using average training-level episode returns.
- Figure 5 measures average episode returns on the ProcGen training levels.
- The comparison includes the presented method and PPO.
- The comparison also includes PPG and PPO with access to privileged information.
C. Additional NoCrash Experiments
The additional NoCrash experiments compare route completion and noisy-trajectory training against prior methods under specified experimental settings.
- Table 7 compares route completion rates for Rails and prior state-of-the-art methods on the CARLA NoCrash benchmark.
- Table 8 evaluates a variant trained with noisy trajectories generated using Ornstein-Uhlenbeck noise.
- The noisy-trajectory experiment uses an experimental setup equivalent to Table 4.
D. Action-value Computation
Action-value computation uses horizon-based reward discretization and backward induction, while related figures and tables report benchmark comparisons for returns and route completion.
- In CARLA, the method uses a planning horizon of H = 5 and computes rewards from t to t + H − 1 around the ego state at time t.
- In ProcGen, the planning horizon is H = 30.
- Values and action-values are computed for each time t using backward induction.
- Figure 5 reports average episode returns on ProcGen training levels, while Table 7 reports mean route completion on NoCrash.
E. CARLA Controls
CARLA control uses separate steering and throttle probabilities to smooth outputs from discretized actions, with additional throttle control and model ensembling for stability.
- Control smoothing: Steering and throttle are modeled independently, with softmax probabilities used to compute smooth control values.The training policy factorizes steering and throttle logits when the brake indicator is inactive.
- Throttle control: A bang-bang throttle controller sets throttle to 0 when vehicle speed exceeds a predefined threshold.The experiments use tb = 0.5.
- Ensembling: A 6 model ensemble provides more stable control for the top leaderboard submission.
G. Training Hyperparameters
The CARLA experiments use specified image augmentations and additional hyperparameter settings, while NoCrash evaluation includes trajectories collected with injected Ornstein–Uhlenbeck noise.
- Image augmentation: CARLA training applies seven image augmentations, including blur, Gaussian noise, pixel dropout, contrast changes, grayscale, and elastic transformations.The listed augmentations also include multiplicative scaling.
- NoCrash evaluation: Table 8 compares NoCrash success rates after training on trajectories collected with injected Ornstein–Uhlenbeck noise.Its metric and evaluation protocol are comparable to Table 4, while the data-collection protocol follows LBC.
- Hyperparameters: Table 9 provides additional training hyperparameters for reference.