PaperlayerView exact revision on arXiv

Source-linked AI summary

RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework

Hao Gao, Shaoyu Chen, Yifan Zhu, Yuehao Song, Wenyu Liu, Qian Zhang, Xinggang Wang

arXiv:2604.15308v1cs.CV

TL;DR

Autonomous-driving planners need multimodal trajectory modeling that remains stable under closed-loop interaction, but diffusion-based imitation learning lacks corrective feedback and struggles with optimization. RAD-2 combines a diffusion generator, an RL-trained discriminator, temporally consistent optimization, on-policy generator refinement, and BEV-Warp simulation. The framework reports substantial collision-rate reductions and improvements in safety, efficiency, and real-world perceived driving quality.

  • Problem

    Diffusion-based planners can produce unstable trajectories because imitation learning lacks corrective negative feedback, while sparse rewards complicate optimization in high-dimensional trajectory spaces.

  • Method

    RAD-2 generates diverse diffusion-based trajectory candidates, reranks them with an RL-optimized discriminator, refines the generator through structured longitudinal feedback, and trains in BEV-Warp.

  • Results

    56% collision-rate reduction versus strong diffusion-based planners is reported, alongside improved closed-loop safety and efficiency metrics and lower open-loop collision rates.

  • Takeaways & Limitations

    The generator-discriminator framework supports stable closed-loop planning with multimodal trajectories while improving safety and efficiency across benchmarks.

  • Takeaways & Limitations

    BEV-Warp is constrained for architectures using raw camera pixels or unified latent embeddings without explicit spatially equivariant grid structures.

Abstract

from arXiv · show

High-level autonomous driving requires motion planners capable of modeling multimodal future uncertainties while remaining robust in closed-loop interactions. Although diffusion-based planners are effective at modeling complex trajectory distributions, they often suffer from stochastic instabilities and the lack of corrective negative feedback when trained purely with imitation learning. To address these issues, we propose RAD-2, a unified generator-discriminator framework for closed-loop planning. Specifically, a diffusion-based generator is used to produce diverse trajectory candidates, while an RL-optimized discriminator reranks these candidates according to their long-term driving quality. This decoupled design avoids directly applying sparse scalar rewards to the full high-dimensional trajectory space, thereby improving optimization stability. To further enhance reinforcement learning, we introduce Temporally Consistent Group Relative Policy Optimization, which exploits temporal coherence to alleviate the credit assignment problem. In addition, we propose On-policy Generator Optimization, which converts closed-loop feedback into structured longitudinal optimization signals and progressively shifts the generator toward high-reward trajectory manifolds. To support efficient large-scale training, we introduce BEV-Warp, a high-throughput simulation environment that performs closed-loop evaluation directly in Bird's-Eye View feature space via spatial warping. RAD-2 reduces the collision rate by 56% compared with strong diffusion-based planners. Real-world deployment further demonstrates improved perceived safety and driving smoothness in complex urban traffic.

1 Introduction

RAD-2 addresses instability and credit-assignment challenges in closed-loop autonomous-driving planning by separating trajectory generation from RL-based evaluation and using scalable simulation. Its generator-discriminator design combines multimodal diffusion generation with low-dimensional, structured optimization and temporal consistency.

  • Robust, safe, and human-like motion planning is a central goal for high-level autonomous driving systems.
  • Diffusion-based imitation-learning planners model multimodal continuous trajectories but can produce unstable, low-quality behaviors because driving data are noisy and unevenly distributed.High-dimensional trajectories and imitation-only training further limit optimization and provide no negative feedback against unrealistic behaviors.
  • Directly applying sparse scalar rewards to high-dimensional, temporally structured trajectories creates severe credit-assignment problems and unstable optimization.
  • RAD-2 combines BEV-Warp with the generator-discriminator architecture to enable stable policy updates and efficient closed-loop interaction.BEV-Warp supports feature-level simulation, while the framework is reported to reduce collision rates by over 56% on large-scale benchmarks and improve perceived safety during real-vehicle testing.
  • On-policy Generator Optimization restricts reinforcement learning to the longitudinal component, while Temporally Consistent sampling and optimization address trajectory credit assignment.This design uses structured feedback and temporally coherent sampling rather than optimizing the full high-dimensional trajectory directly.
  • RAD-2 decouples diffusion-based trajectory exploration from RL-based reranking, using a generator for diverse candidates and a discriminator for long-term quality evaluation.The discriminator evaluates candidate sets sampled from the generator, while the generator is refined through on-policy optimization.

2 Related Work

Prior work spans trajectory scoring, reinforcement-learning integration, and closed-loop simulation, but each line retains limitations in flexibility, credit assignment, fidelity, or computational cost. RAD-2 positions BEV-Warp as a feature-level alternative that directly warps BEV features for high-throughput simulation.

  • Trajectory Scoring and Selection: Trajectory-scoring methods evolved from fixed vocabularies and rule-based teachers toward coarse-to-fine filtering, preference optimization, and dynamic candidate evaluation.
  • Reinforcement Learning for Driving: Reinforcement learning can address imitation-learning issues, but sparse rewards make optimizing high-dimensional driving outputs difficult because of severe credit-assignment challenges.
  • Closed-loop Simulation: Traditional interactive simulators often suffer sim-to-real gaps, while reconstruction-based and generative approaches improve fidelity but can remain computationally intensive or drift temporally.
  • Closed-loop Simulation: BEV-Warp enables high-throughput simulation by directly warping BEV features around the ego vehicle, bypassing expensive image-level rendering.

3.1 Generator-Discriminator Framework

The framework separates trajectory generation from trajectory evaluation: a diffusion generator produces multimodal candidates, while a discriminator uses trajectory–scene interactions to score and rerank them.

  • Generator: The diffusion-based generator produces diverse candidate trajectories conditioned on the current observation and scene representation.It encodes static map elements, dynamic agents, and navigation inputs into a unified scene embedding for trajectory generation.
  • Generator: For multiple independent modes, noisy trajectory samples are iteratively denoised to form the final candidate set.The denoising network conditions each update on the current noisy trajectory, scene embedding, and denoising step.
  • Generator: Each candidate is a continuous (x, y) trajectory over the planning horizon and is passed to the discriminator for evaluation and selection.The candidates define the policy’s trajectory set.
  • Discriminator: The discriminator embeds trajectory points, aggregates them with a Transformer [CLS] query, and conditions that query on multi-source scene context.Its scene encoders share the generator’s architecture but use independent parameters, while cross-attention fuses trajectory and scene information.
  • Discriminator: The discriminator applies a sigmoid to the fused trajectory–scene representation to produce scalar scores for candidate prioritization and reranking.Scores may be normalized across candidates, although the implementation directly uses sigmoid outputs.

3.2 Closed-Loop Simulation Environment and Controller

BEV-Warp performs closed-loop simulation by warping recorded bird’s-eye-view features according to the simulated pose, while an iLQR controller tracks the selected trajectory.

  • BEV-Warp Environment: BEV-Warp constructs a high-throughput environment by manipulating BEV features over time instead of repeatedly rendering images.The simulation starts from recorded real-world data and extracts the reference BEV feature and pose at each step.
  • BEV-Warp Environment: At each step, the planner generates candidate trajectories, selects an optimal trajectory, and tracks it to update the simulated pose.The updated pose drives the next closed-loop simulation step.
  • BEV-Warp Environment: The warp matrix is computed from the simulated and logged reference poses to realign the recorded environment with the simulated ego-state.The formulation assumes constant altitude and neglects pitch and roll rotations.
  • Trajectory-Based Controller: Both environments use an iLQR-based controller that locally linearizes vehicle dynamics and iteratively refines controls to track the planned trajectory.The first optimal command updates the vehicle state and produces the next ego pose.
  • BEV-Warp Environment: The next-step BEV feature is synthesized through spatial warping, enabling continuous interaction without expensive image-level rendering.The synthesized feature is coupled with the updated pose for subsequent evaluation.

3.3 Joint Policy Optimization

RAD-2 jointly optimizes a diffusion generator and an RL-trained discriminator through temporally consistent closed-loop rollouts. The discriminator learns long-horizon trajectory quality, while structured generator optimization shifts outputs toward safer and more efficient behaviors.

  • Joint Framework: The joint policy combines a diffusion generator for multimodal trajectory proposals with an RL-trained discriminator for candidate selection.The framework optimizes their hybrid policy through a cyclic structured pipeline in the BEV-Warp environment.
  • Temporally Consistent Rollout: High-frequency switching among trajectory modes weakens the correlation between individual decisions and long-term driving outcomes, causing inefficient credit assignment and unstable policy improvement.Trajectory reuse addresses this by committing selected controls over a fixed execution horizon while retaining asynchronous safety termination.
  • Reward Modeling: The closed-loop reward combines collision-risk and navigational-efficiency objectives to balance safety with driving efficiency.Collision reward uses the worst-case temporal safety margin, while efficiency is anchored to a target progress interval relative to expert pacing.
  • Discriminator Optimization via RL: TC-GRPO persists sampled trajectory hypotheses over short horizons and assigns sparse rewards to the sustained hypothesis, denoising policy gradients and stabilizing optimization.The discriminator is optimized at sparse trajectory-sampling decision points using group-relative advantages and a clipped objective.
  • Generator Optimization: On-policy Generator Optimization converts closed-loop feedback into structured longitudinal signals and shifts the generator toward high-reward trajectory regions.This avoids applying scalar reinforcement-learning rewards directly to the generator’s full high-dimensional trajectory output.

4 Experiment

Experiments evaluate RAD-2 across closed-loop simulation, photorealistic simulation, open-loop benchmarks, scaling studies, ablations, and qualitative interactions. The framework improves safety and efficiency while joint optimization and design choices contribute to stable performance.

  • Experimental Setup: Approximately 50,000 hours of real-world driving data pretrain the diffusion generator, while 50k real-world clips support closed-loop training and evaluation.The open-loop benchmark covers car-following, lane changing, intersections, curves, and heavy braking scenarios.
  • Experimental Results: Closed-loop BEV-Warp evaluation reduces CR from 0.533 to 0.234 and AF-CR from 0.264 to 0.092 versus ResAD.Safety@1/2 increases from 0.418/0.281 to 0.730/0.596, while EP-Mean rises from 0.970 to 0.988 and EP@1.0/0.9 from 0.516/0.894 to 0.736/0.984.
  • Experimental Results: Open-loop evaluation lowers overall CR to 0.142% versus 0.288% for Senna-2, with FDE and ADE reduced to 0.553 m and 0.208 m.The reductions also include both dynamic and static collision components.
  • Scaling Behavior Analysis: Joint generator-discriminator optimization scales more efficiently than discriminator-only or sequential two-stage training, producing steeper performance gains and better final performance.Shared rollouts improve data efficiency and faster convergence through tightly coupled updates.
  • Ablation Studies: On-policy generator optimization reduces collision rate from 0.533 to 0.287, while IL fine-tuning restores cruise performance without removing the safety gains.Further discriminator RL enhances the resulting safety-efficiency balance.
  • Ablation Studies: An execution horizon of 8 balances temporal consistency and reactivity, while rollout filtering, pretrained discriminator initialization, group size 4, entropy regularization, and mixed scenarios improve training behavior.Larger groups can improve efficiency but reduce safety, and mixed scenario training balances collision avoidance with navigation progress.
  • Additional Analyses: Increasing inference-time candidate count scales EP@1.0 from 0.667 to 0.814, although safety metrics exhibit minor fluctuations.Qualitative results show proactive deceleration avoiding collisions and agile lane changes improving efficiency in dynamic traffic.

5 Limitations and Future Work

The framework’s BEV-Warp efficiency depends on explicit spatially equivariant BEV feature maps, limiting direct applicability to architectures without such structures. Future work targets generalized transformations, generative world models, broader scenarios, and the simulation-to-reality fidelity gap.

  • Representation Specificity: BEV-Warp is constrained for raw-pixel or unified-latent architectures lacking explicit spatially equivariant grid structures.Such systems would require a more generalized transformation module or a direct latent-space world model.
  • Transition to Generative World Models: Generative world models offer greater flexibility and photorealism but currently impose computational overhead and long-horizon temporal drift that limit large-scale RL training.The paper identifies integrating the optimization pipeline with such models as a future direction.
  • Future Work: Future work must scale training-scenario diversity and bridge the remaining fidelity gap between simulation and complex open-world driving dynamics.This boundary concerns the transition from simulated evaluation to real-world driving conditions.

6 Conclusion

RAD-2 combines a generator-discriminator framework with temporally consistent optimization and BEV-Warp simulation for reinforcement learning in diffusion-based motion planning. Experiments show improved safety and efficiency across benchmarks, with ablations supporting the proposed formulation and optimization strategy.

  • Contributions: RAD-2 presents a unified generator-discriminator framework for stable reinforcement learning in diffusion-based motion planning.The framework decouples trajectory generation and evaluation within closed-loop learning.
  • Optimization Strategy: Temporally Consistent Group Relative Policy Optimization improves credit assignment through temporally coherent sampling, while On-policy Generator Optimization refines trajectory distributions using structured feedback.These methods provide the proposed temporally consistent optimization strategy.
  • Simulation: BEV-Warp enables scalable closed-loop learning through a feature-level simulation pipeline.The conclusion presents it as the infrastructure for efficient large-scale training.
  • Findings: Experiments demonstrate improved safety and efficiency across diverse benchmarks, including substantial collision-rate reductions and reliable closed-loop navigation.Ablation studies further verify the generator-discriminator formulation and temporally consistent optimization strategy.
Loading 2604.15308v1…