ReSim: Reliable World Simulation for Autonomous Driving

1 Introduction

• Real-world driving data is dominated by safe expert demonstrations, leaving hazardous events and non-expert actions underrepresented and challenging for reliable simulation.The introduction connects this restricted state-action coverage to severe hallucinations on unseen non-expert actions.Source: §1 · pp. 1–2
• ReSim enriches human driving logs with simulator-collected non-expert behaviors and uses multi-stage training to integrate visual and action conditions.Its training recipe includes unbalanced noise sampling and a dynamics consistency loss to emphasize motion coherence, including for non-expert actions.Source: §1 · pp. 2–3
• Video2Reward estimates scalar rewards from ReSim’s simulated futures, while ReSim’s imagination can also serve as a closed-loop synthetic environment for validating learned driving policies.These capabilities support action evaluation as well as visual simulation.Source: §1 · pp. 2–3
• On NAVSIM, ReSim’s video prediction-based policy improves +2.0 over state-of-the-art world model-based planners and +2.6 over an end-to-end supervised baseline; policy selection improves 55.3% over weak candidate policies.The policy-selection process uses estimated rewards to choose among trajectories generated by candidate policies.Source: §1 · p. 2

2.1 Heterogeneous Data Compilation

• Real-world NAVSIM and OpenDV data are supplemented with CARLA expert and non-expert driving, addressing the scarcity of unsafe behaviors in human demonstrations.The datasets contain 4M OpenDV, 85K NAVSIM, and 88K CARLA video samples.Source: §2.1 · p. 3

2.2 Controllable World Model

• ReSim adapts a text-conditioned diffusion transformer for history, command, and waypoint conditioning, and adds a dynamics consistency loss across multiple time intervals.The dynamics loss supervises latent motion as well as frame-level video prediction.Source: §2.2 · pp. 3–4
• A three-stage curriculum combines OpenDV, NAVSIM, and CARLA training, while noise sampling favors higher-noise timesteps to improve learning of complex driving dynamics.The frequency of sampling timesteps in [500, 1000] increases from 1/2 to 2/3.Source: §2.2 · p. 4

2.3 Reward Estimation from Video

• Video2Reward learns video-based reward estimates from CARLA infraction scores, then scores ReSim’s predicted future for a proposed trajectory.Its supervision includes safe and hazardous simulated data and penalizes factors such as collisions and excessively low speeds.Source: §2.3 · p. 5, Figure 2 · p. 5

2.4 Applications

• For a video prediction-based policy, ReSim imagines future frames from history and a command, and an inverse dynamics model converts them into an ego trajectory.The policy does not condition on actions when generating its visual plan.Source: §2.4 · p. 5, Figure 3 · p. 5
• On zero-shot Waymo action-conditioned prediction, ReSim surpasses baselines by a large margin under both action conditions.Source: Table 1 · p. 6
• Reward-guided selection renders candidate trajectories into predictive videos, scores them with Video2Reward, and selects the highest-reward trajectory.The authors report that selection significantly improves over individual policy candidates by leveraging their advantages in different situations.Source: §2.4 · pp. 5–6
• ReSim can simulate closed-loop visual states by generating a new state from each driving agent action and returning it to the agent for its next decision.Source: §2.4 · p. 6

3.1 Results of Simulation Reliability

• On unseen Waymo data, ReSim improves Trajectory Difference results by 80% and 54% over Vista for action-free and expert-action conditioning, respectively.For non-expert actions, human preferences favor ReSim for visual realism and trajectory following; qualitative comparisons show reliable hazardous outcomes, and Figure 6 reports zero-shot control across expert and non-expert actions.Source: §3.1 · p. 6, Figure 5 · p. 7, Figure 6 · p. 7
• Without nuScenes training samples or explicit action conditioning, ReSim outperforms prior in-distribution driving world models on prediction fidelity.For long-term prediction, Vista becomes oversaturated and loses scene semantics, while ReSim remains visually rich at 30 seconds.Source: §3.1 · pp. 6–7, Table 2 · p. 7

3.2 Results of Applications

• 74.1 PDMS is achieved by ReSim plus Video2Reward on 300 challenging NAVSIM scenarios, exceeding candidate policies, an averaged ensemble, and Vista reward guidance.Table 3 reports a +26% PDMS improvement over candidate policies; the result is closest to oracle selection.Source: Table 3 · p. 8, §3.2 · p. 8
• ReSim with a lightweight inverse dynamics model outperforms world-model and end-to-end NAVSIM planners using only history observations and a high-level command.In closed-loop visual simulation, it also generates successive 4-second futures for a policy using front-view video, reaching states absent from pre-recorded data.Source: Table 4 · p. 8, §3.2 · p. 8, Figure 8 · p. 9

3.3 Ablation Study

• Removing CARLA data worsens expert and non-expert action controllability, reward estimates, and reward-guided policy selection.The qualitative comparison also shows scenario-structure inconsistency for non-expert actions without simulated data.Source: Figure 5 · p. 7, §3.3 · p. 9
• Unbalanced noise sampling produces more consistent agent motion and scenario layout than uniform noise sampling.Figure 9 also reports improved motion and scenario consistency.Source: §3.3 · p. 9, Figure 9 · p. 9
• Dynamics consistency loss and a longer maximum interval K for latent motion extraction yield more coherent future predictions; Figure 10 identifies K = 4 as best.Source: §3.3 · p. 9, Figure 10 · p. 9

4 Conclusion and Outlook

• ReSim combines hazardous non-expert data from a driving simulator with mostly safe human driving data, and uses dynamics consistency loss, unbalanced noise sampling, and multi-stage learning.Video2Reward estimates reward from ReSim's simulated future for driving applications beyond visual simulation.Source: §4 · p. 9
• Iterative denoising bottlenecks inference efficiency, and how to train agents within ReSim's synthesized world remains unresolved.The authors also identify fair closed-loop planning benchmarks as future work.Source: §4 · p. 9

A.1 World Model

• World models represent environmental dynamics by mapping past states or observations and policy actions to a next state and reward estimate, with long-term continuous rollouts.Prior work has used them for representation learning, planning, and model-based reinforcement learning.Source: §A.1 · p. 17

A.2 Predictive Model for Driving Scenes

• Driving world models trained mainly on human logs struggle with bad-action consequences, while open-world driving rewards are difficult to specify from high-dimensional observations and agent-environment relationships.The authors also note limitations of rule-based and uncertainty-based reward approaches.Source: §A.2 · p. 17

A.3 Video Generation

• Video diffusion transformers can produce spatiotemporally consistent video, but existing web-trained generators lack reliable driving prediction and action controllability.ReSim addresses this gap with model-structure modifications and training protocols.Source: §A.3 · p. 18

Universal Driving Videos

• 1700 hours of worldwide, uncalibrated OpenDV videos provide broad scenario and camera coverage; turning clips are sampled at 5×, yielding 4M video clips.High-level commands are pseudo-labeled from estimated flow.Source: §B.1 · p. 18

Expert Driving Data

• 85K NAVSIM training samples supply detailed ego-action labels unavailable in online videos, while nuScenes and Waymo are reserved for held-out evaluation.Source: §B.1 · p. 18

Explorable simulated data

• 88K successful and hazardous Bench2Drive executions, collected across 220 routes with varied weather and time of day, provide simulated expert and non-expert data.PDM-Lite collects successful executions; rule-based exploration covers a larger action space, with each type accounting for roughly half the cases.Source: §B.1 · p. 18
• The non-expert agent first follows PDM-Lite for 1s, then explores for 4s using strategies including forced lane changes and tailgating.Exploration randomizes parameters such as turning angle and speed.Source: §B.1 · pp. 18–19

B.2 Model and Training

• ReSim uses a 2B diffusion transformer with T5 language encoding, a 3D causal VAE, and a lightweight trajectory encoder for action conditioning.The trajectory encoder has two attention blocks and a linear head.Source: §B.2 · p. 19
• Historical-video latent corruption uses separately sampled timesteps during training and t-aug = 0 at inference to improve robustness in multi-round prediction.Source: §B.2 · p. 19
• Text commands are dropped with probability 0.5 and NAVSIM trajectories with probability 0.5, while CARLA trajectories are retained to guide abnormal and hazardous behaviors.The authors state that such behaviors cannot be accurately inferred from history alone and that unconditioned hazardous behaviors could interfere with learning expert patterns.Source: §B.2 · p. 19
• Video2Reward extracts frame features with DINOv2, aggregates them through two spatial-temporal attention blocks, and predicts a scalar reward supervised by CARLA Infraction Scores.Source: §B.2 · pp. 19–20
• Two separately trained inverse dynamics models estimate 8-waypoint trajectories at 2Hz for expert-controllability evaluation and video-prediction-based planning.The models share an XVO visual-odometry backbone and lightweight attention head; their training datasets differ.Source: §B.2 · p. 20
• The VO-based planner uses historical frames only, unlike the NAVSIM inverse dynamics model, which receives history and future frames.Source: §B.2 · p. 20

B.3 Sampling

• ReSim samples 4s futures at 10Hz and 512×896 resolution using 50 DDIM steps, then can iteratively reuse the latest 9 frames to extend the rollout.Inputs include 9 historical frames and optional high-level commands and 8 future ego-trajectory waypoints at 2Hz.Source: §B.3 · p. 20

• 15 participants answered 40 questions each, choosing among Vista, ReSim without simulated data, and ReSim for visual realism and trajectory following.The model-generation order was shuffled for each testing scenario.Source: §B.4 · p. 20, Figure S.12 · p. 21

C.1 Action Controllability

• On unseen nuScenes and Waymo samples, visualizations show ReSim controlled by both ground-truth expert trajectories and randomly associated non-expert trajectories.Source: §C.1 · p. 21

C.2 Ablation Study for Simulated Data

• Joint training with simulated data improves ReSim’s controllability for non-expert actions in open-world scenarios.The examples use the OpenDV validation set with randomly associated trajectories from other labeled datasets.Source: §C.2 · p. 21

C.3 Action-free Prediction

• Without action inputs, ReSim synthesizes a possible future from historical frames, which may differ from ground truth because driving scenarios are multi-modal.Source: §C.3 · p. 21

C.4 Long-horizon Prediction

• In multi-round rollouts without action conditions, ReSim produces a more visually rich future over a longer horizon than Vista from the same scenario.Both models iteratively condition on previously generated sequences to extend the prediction horizon.Source: §C.4 · p. 21

C.5 Failure Mode

• Despite improved fidelity and controllability over previous methods, ReSim still faces challenges.The passage refers readers to the Societal Impact discussion for limitations.Source: §C.5 · p. 21

Inference Efficiency

• Diffusion sampling’s multiple denoising rounds may bottleneck real-world application, with fewer sampling steps or distillation offered as possible latency improvements.Source: D Limitations and Broader Impact · p. 21

World Model for Policy Training

• Using a world model as a dynamic environment for policy training is presented as a promising direction that could allow direct deployment of trained policies to autonomy.Source: D Limitations and Broader Impact · p. 21
• The authors identify scaling ReSim’s visual simulation for vision-based driving policy learning as future work.Source: D Limitations and Broader Impact · p. 22

Closed-loop Benchmark

• ReSim can expose policies to new states in closed-loop simulation, but its front-view-only training limits ready use of multi-view methods, and fair quantitative benchmarking remains open.Source: D Limitations and Broader Impact · p. 22

Societal Impact

• Diffusion stochasticity may cause visual artifacts, complex multi-agent scenes may be hallucinated, and uncurated geographic data distributions may bias learned behavior.Source: D Limitations and Broader Impact · p. 22

E License of Assets

• Public datasets and benchmarks support training and evaluation, while CARLA data collected to improve action diversity is under CC-BY and follows Bench2Drive scenarios under CC BY-NC-SA 4.0.Source: E License of Assets · p. 22
• ReSim is developed upon CogVideoX, with both code and model under Apache License 2.0.Source: E License of Assets · p. 22
• The paper uses DINOv2 under Apache License 2.0, XVO under CC BY-NC-SA 4.0, and Vista under Apache License 2.0; the authors state their code and models will be released under Apache License 2.0.Source: E License of Assets · p. 22