PaperlayerView exact revision on arXiv

Source-linked AI summary

ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstruction

Chaojun Ni, Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Wenkang Qin, Xinze Chen, Guanghong Jia, Guan Huang, Wenjun Mei

arXiv:2508.08170v2cs.CV

TL;DR

Reconstruction-based simulators are constrained by recorded camera trajectories, while realistic closed-loop training also requires physically feasible vehicle motion. ReconDreamer-RL combines diffusion-based appearance modeling and kinematic modeling with agents for corner cases and action diversity; it achieves a 5× reduction in Collision Ratio versus imitation-learning methods.

Abstract

from arXiv · show

Reinforcement learning for training end-to-end autonomous driving models in closed-loop simulations is gaining growing attention. However, most simulation environments differ significantly from real-world conditions, creating a substantial simulation-to-reality (sim2real) gap. To bridge this gap, some approaches utilize scene reconstruction techniques to create photorealistic environments as a simulator. While this improves realistic sensor simulation, these methods are inherently constrained by the distribution of the training data, making it difficult to render high-quality sensor data for novel trajectories or corner case scenarios. Therefore, we propose ReconDreamer-RL, a framework designed to integrate video diffusion priors into scene reconstruction to aid reinforcement learning, thereby enhancing end-to-end autonomous driving training. Specifically, in ReconDreamer-RL, we introduce ReconSimulator, which combines the video diffusion prior for appearance modeling and incorporates a kinematic model for physical modeling, thereby reconstructing driving scenarios from real-world data. This narrows the sim2real gap for closed-loop evaluation and reinforcement learning. To cover more corner-case scenarios, we introduce the Dynamic Adversary Agent (DAA), which adjusts the trajectories of surrounding vehicles relative to the ego vehicle, autonomously generating corner-case traffic scenarios (e.g., cut-in). Finally, the Cousin Trajectory Generator (CTG) is proposed to address the issue of training data distribution, which is often biased toward simple straight-line movements. Experiments show that ReconDreamer-RL improves end-to-end autonomous driving training, outperforming imitation learning methods with a 5x reduction in the Collision Ratio.

Abstract

This section presents ReconDreamer-RL, which combines diffusion-enhanced scene reconstruction and physical vehicle modeling with adversarial scenario generation and trajectory augmentation for end-to-end autonomous-driving reinforcement learning. It reports improved training outcomes and examines the contributions of ReconSimulator, DAA, and CTG.

Introduction

Overview of the ReconDreamer-RL framework

  • ReconDreamer-RL trains policies in two stages: behavior cloning initializes the plan, then closed-loop reinforcement learning optimizes the policy while DAA creates corner cases.CTG diversifies ego-vehicle actions during imitation learning.Source: Introduction · p. 1, Overview of the ReconDreamer-RL… · p. 3

ReconSimulator

  • ReconSimulator combines 3D Gaussian Splatting with a video diffusion prior for appearance modeling and a kinematic model to keep vehicle trajectories physically feasible.DriveRestorer improves rendered novel-trajectory videos, which are used to fine-tune the reconstruction model.Source: Introduction · p. 1, ReconSimulator · p. 3

Dynamic Adversary Agent

  • DAA selects surrounding vehicles and modifies their trajectories relative to the ego vehicle to generate realistic interactions such as cut-ins.It creates scenarios for imitation learning and reinforcement learning, with trajectories checked for feasibility.Source: Dynamic Adversary Agent · p. 4

Cousin Trajectory Generator

  • 4× more non-straight-line driving maneuvers appear in the Cousin-nuScenes dataset than in nuScenes (Caesar et al. 2020).CTG extends and interpolates expert trajectories, checking generated paths for physical plausibility and collision avoidance.Source: Cousin Trajectory Generator · p. 5

Main Results

  • 3× reduction in Collision Ratio compared with RAD (Gao et al. 2025) is reported for ReconDreamer-RL.The comparison is reported across the evaluation metrics in Table 1.Source: Main Results · pp. 6–7, Table 1 · p. 7
  • 404.5% improvement in Collision Ratio over imitation methods is reported for cut-in corner cases.In qualitative cut-in tests, ReconDreamer-RL avoids collisions, including when a cut-in is followed by sudden braking.Source: Table 4 · p. 7, Main Results · p. 7
  • ReconSimulator’s rendering speed is reported as efficient enough to satisfy reinforcement-learning task requirements.The comparison is presented in Table 5.Source: Table 5 · p. 7, Main Results · p. 7

Ablation Studies

  • Without ReconSimulator, NTA-IoU and NTL-IoU dropped by 54.46% and 9.71%, while FID increased by 104.43% at a 6m lane shift.Source: Ablation Studies · p. 8
  • Adding DAA decreased Collision Ratio from 0.172 to 0.117 by generating corner-case scenarios for training.Source: Ablation Studies · p. 8
  • CTG reduced CR from 0.172 to 0.143 and DR from 0.073 to 0.052, indicating improved trajectory adherence.The passage attributes these benefits to addressing data bias and enhancing the action distribution learned during imitation learning.Source: Ablation Studies · p. 8

Conclusion

  • ReconDreamer-RL reduced Collision Ratio by 5× compared with imitation-based learning methods for end-to-end autonomous driving.Source: Conclusion · p. 8
  • ReconSimulator combines video diffusion priors for appearance modeling with a kinematic model for physical dynamics, while DAA generates complex traffic scenarios such as sudden lane changes.Source: Conclusion · p. 8
  • CTG extends and interpolates trajectories to increase diversity beyond training data dominated by simple straight-line movements.Source: Conclusion · p. 8

Learning via Diffusion-based Scene Reconstruction

The supplementary material collects implementation details and additional evidence about ReconDreamer-RL's generalization and capabilities.

Implementation Details

The implementation reconstructs nuScenes and Waymo as 3DGS environments, evaluates policy safety, deviation, and rendering realism, and trains RAD through imitation and reinforcement learning. The section also documents corner-case scenarios and implementation settings.

Datasets

  • nuScenes and Waymo are reconstructed into 3DGS environments, and Waymo is additionally used to assess generalization across varied urban conditions.The reconstructed environments are intended to be made publicly available.Source: Datasets · p. 11

Metric

CR = DCR + SCR. (14)

Training Details

  • RAD uses perception supervision and expert demonstrations for imitation learning, then optimizes its policy with PPO through interaction with clip-specific 3DGS environments.The section points to separate hyperparameter tables for imitation and reinforcement learning.Source: Training Details · p. 12, Table 6 · p. 12, Table 7 · p. 12

Details of Edited Scenes

  • The designed corner cases include BlockedIntersection, DynamicCutIn, OppositeLaneIntrusion, ParkingCutIn, HardBrake, and HazardAtSideLane.Source: Details of Edited Scenes · p. 12
  • Additional edited scenes include wrong-way approaches, conflicting lane changes, and blocked lanes that require maneuvering into opposing traffic.Source: Details of Edited Scenes · pp. 12–13
  • The listed training settings include learning rates, optimizers, and PPO-related parameters such as GAE values and clipping thresholds.Source: Details of Edited Scenes · p. 12
  • DAA can generate different trajectories for a target vehicle at the same timestep.Source: Figure 8 · p. 13

Additional Experimental Results.

Additional experiments compare ReconDreamer-RL with RAD on Waymo and assess diffusion-prior rendering gains, qualitative corner-case behavior, and DAA trajectory edits.

ReconSimulator

  • On Waymo novel-view rendering quality for lane change, integrating the video diffusion prior improves NTA-IoU by 13.58% and NTL-IoU by 1.41% while reducing the FID score by 18.89%.Source: Additional Experimental Results · p. 13

Quantitative Results on Waymo

Qualitative Results on Waymo

  • In HardBrake and DynamicCutIn comparisons, ReconDreamer-RL avoids collisions by adjusting speed, with the authors attributing this to the diffusion-based simulator and data augmentation.Figure 9 highlights collisions with red boxes.Source: Additional Experimental Results · p. 14, Figure 9 · p. 14

Visualization of DAA

  • DAA can fine-tune edited-scene vehicle trajectories, including speed, at a given timestep without changing the pipeline's overall timing.Figure 8 shows different DAA-generated trajectories at the same timestep.Source: Additional Experimental Results · p. 14

Qualitative Results on Waymo

Loading 2508.08170v2…