PaperlayerView exact revision on arXiv

Source-linked AI summary

ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving

Zihao Sheng, Xin Ye, Jingru Luo, Sikai Chen, Liu Ren

arXiv:2604.02714v2cs.CV

TL;DR

VLA behavior cloning reproduces expert demonstrations without exploring alternatives, and sparse text or waypoint supervision omits scene appearance and geometry. ExploreVLA pairs trajectory prediction with future RGB and depth generation, then uses world-model uncertainty in a safety-gated RL reward. Experiments report state-of-the-art NAVSIM performance and validation on nuScenes.

  • Problem

    VLA behavior cloning limits policies to observed expert behavior, while sparse text and waypoint targets omit scene appearance and geometry, and RL needs a learned model of driving consequences.

    Source: §1 · pp. 2–3

  • Method

    ExploreVLA jointly predicts trajectories and future RGB and depth images, then uses image-prediction uncertainty in a safety-gated reward optimized with GRPO.

    Source: §1 · p. 3

  • Results

    Experiments validate effectiveness on NAVSIM and nuScenes, with state-of-the-art NAVSIM performance and PDMS 93.7.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    The framework targets diverse yet safe driving strategies beyond expert imitation by combining dense scene supervision with uncertainty-guided exploration.

    Source: §5 · p. 15

    The framework uses a single front-view camera, and offline/open-loop RL post-training may bias the policy toward training data and limit learning recovery behaviors from self-induced states.

    Source: §5 · p. 15

Abstract

from arXiv · show

End-to-end autonomous driving models based on Vision-Language-Action (VLA) architectures have shown promising results by learning driving policies through behavior cloning on expert demonstrations. However, imitation learning inherently limits the model to replicating observed behaviors without exploring diverse driving strategies, leaving it brittle in novel or out-of-distribution scenarios. Reinforcement learning (RL) offers a natural remedy by enabling policy exploration beyond the expert distribution. Yet VLA models, typically trained on offline datasets, lack directly observable state transitions, necessitating a learned world model to anticipate action consequences. In this work, we propose a unified understanding-and-generation framework that leverages world modeling to simultaneously enable meaningful exploration and provide dense supervision. Specifically, we augment trajectory prediction with future RGB and depth image generation as dense world modeling objectives, requiring the model to learn fine-grained visual and geometric representations that substantially enrich the planning backbone. Beyond serving as a supervisory signal, the world model further acts as a source of intrinsic reward for policy exploration: its image prediction uncertainty naturally measures a trajectory's novelty relative to the training distribution, where high uncertainty indicates out-of-distribution scenarios that, if safe, represent valuable learning opportunities. We incorporate this exploration signal into a safety-gated reward and optimize the policy via Group Relative Policy Optimization (GRPO). Experiments on the NAVSIM and nuScenes benchmarks demonstrate the effectiveness of our approach, achieving a state-of-the-art PDMS score of 93.7 and an EPDMS of 88.8 on NAVSIM. The code is available at https://zihaosheng.github.io/ExploreVLA/.

1 Introduction

VLA driving policies trained by behavior cloning reproduce expert demonstrations without exploring alternatives, while sparse text and waypoint supervision leaves scene appearance and geometry underrepresented. ExploreVLA combines future-image world modeling with uncertainty-guided, safety-gated RL exploration and reports state-of-the-art NAVSIM performance.

1 Introduction

  • Behavior cloning limits VLA policies to observed expert behaviors, leaving them without exploratory experience for scenarios outside the expert distribution.The introduction also identifies sparse text and waypoint targets as insufficient to capture scene geometry and fine-grained appearance needed for safe planning.Source: §1 · pp. 2–3
  • Future RGB and depth generation adds dense visual and geometric supervision alongside trajectory prediction, enriching the planning backbone's representations.The image-generation objectives require prediction of fine-grained appearance and metric geometry in future scenes.Source: §1 · p. 3
  • Image-prediction uncertainty measures trajectory novelty relative to the training distribution, rather than distance from a single ground-truth trajectory.The world model is trained on expert demonstrations, so high uncertainty indicates an out-of-distribution trajectory.Source: §1 · p. 3
  • A safety-gated reward gives exploration bonuses to high-PDMS trajectories with high prediction uncertainty, and GRPO optimizes the composite reward.The reward is designed to encourage novel, successful strategies while maintaining driving safety.Source: §1 · pp. 2–3
  • 93.7 PDMS and 88.8 EPDMS on NAVSIM accompany state-of-the-art performance, with nuScenes used to validate generalizability.The approach augments trajectory prediction with future RGB and depth generation and uses the world model's image-prediction uncertainty as an exploration signal.Source: §1 · p. 3

2 Related Work

Related work spans learned driving-world prediction and increasingly unified VLA systems, while noting that sparse supervision remains common. ExploreVLA combines RGB and depth generation with trajectory prediction and uses world-model uncertainty as an intrinsic exploration signal.

2.1 World Models for Autonomous Driving

  • Driving world models span video prediction, geometric occupancy forecasting, and planning integration; ExploreVLA also uses its world model to provide intrinsic rewards for exploratory behavior.The cited examples include generated driving frames, occupancy and ego-motion prediction, and world-model-based trajectory evaluation or imagined-future rewards.Source: §2.1 · pp. 3–4

2.2 VLA Models for Autonomous Driving

  • Existing VLA models primarily use text and action trajectories as supervision, whereas ExploreVLA adds RGB and depth targets to capture richer visual and geometric cues for planning.The related-work passage describes VLA development from scene narration and modular planning toward unified end-to-end architectures.Source: §2.2 · pp. 4–5

2.3 Unified Understanding and Generation Models

  • Unified future-scene generation and action prediction is used to support informed, anticipatory planning; ExploreVLA additionally derives intrinsic exploration rewards from world-model uncertainty.The stated goal is to encourage exploratory driving behaviors and improve generalization beyond the training distribution.Source: §2.3 · p. 5

3 Methodology

ExploreVLA combines trajectory prediction with future RGB and depth generation, then uses world-model uncertainty in reward-guided policy exploration. Its method specifies a unified model, dense visual and geometric supervision, a safety gate, and a two-stage training strategy.

3.1 Overview

  • The framework jointly predicts trajectories and future images, then uses world-model uncertainty as an intrinsic reward for GRPO-guided exploration beyond imitation.Its three components are a unified VLM backbone, future RGB and depth generation, and uncertainty-guided policy optimization.Source: §3.1 · pp. 5–6

3.2 Problem Formulation

  • The model conditions on driving context to predict future waypoints, RGB frames, and depth maps as a unified understanding-and-generation task.Inputs include current and past front-view images, a language command, and ego-vehicle status.Source: §3.2 · p. 6, Input · p. 6, Output · p. 6

3.3 Unified Understanding and Generation Architecture

3.4 Dense Supervisory Signals via World Modeling

3.5 Intrinsic Reward from World Model for Exploration

3.6 Training Strategy

  • Training first pre-trains on image generation with ground-truth future actions, then fine-tunes jointly on actions and images before reward-based policy refinement.The second stage uses an intrinsic reward derived from the world model to encourage exploration.Source: §3.6 · p. 10

4 Experiments

Experiments evaluate ExploreVLA on NAVSIM and nuScenes, with the reported NAVSIM results leading the benchmark comparisons. Further analyses examine exploration novelty, dense visual supervision, reward design, safety thresholds, and trajectory corrections after RL post-training.

4.1 Experimental Setup

  • Evaluation uses NAVSIM navtrain/navtest and nuScenes train/val splits; the model is trained in two stages, then refined with GRPO.NAVSIM plans over four seconds; nuScenes evaluation uses L2 displacement error and collision rate.Source: Datasets · p. 10, Implementation Details · p. 10

4.2 Main Results

4.3 Analysis of Intrinsic Reward Modeling

  • On 1,000 straight-driving scenes, uncertainty-based bonuses better capture behavioral novelty than L2 error across speed and direction perturbations.Speed changes produce larger L2 error despite little novelty, while direction changes produce smaller L2 error and higher exploration bonuses.Source: §4.3 · p. 12, Figure 3 · p. 12, Table 3 · p. 13

4.4 Ablation Study

  • Joint RGB and depth generation raises PDMS to 88.5, with the modalities providing complementary appearance and geometric supervision.Either modality alone improves on trajectory-only training.Source: Effect of Dense Visual Supervision · p. 13, Table 4 · p. 13
  • Combining PDMS and image-based exploration rewards achieves PDMS 90.36, versus 90.19 for PDMS-only and 88.53 for exploration-only.Without the PDMS safety gate, exploration alone provides only a marginal gain over the Stage 1 baseline of 88.50.Source: Effect of Reward Design · p. 13, Table 5 · p. 14
  • Exploration reward improves over the PDMS-only baseline across tested safety thresholds, with δ = 0.9 performing best.About 99% of sampled Stage 1 trajectories already exceed PDMS 0.6, limiting the lower thresholds’ ability to separate riskier trajectories.Source: Effect of the Safety Threshold · p. 14, Table 6 · p. 14
  • Across three scenarios, Stage 2 RL corrects Stage 1 failures involving a vehicle collision, dangerous proximity to pedestrians, or running a stop sign.The figure compares Stage 1 and Stage 2 trajectories against ground truth.Source: Qualitative Analysis · p. 14, Figure 4 · p. 15

5 Conclusion

ExploreVLA combines trajectory, future RGB, and depth prediction with uncertainty-guided, safety-gated GRPO to address sparse supervision and limited policy exploration. The authors report benchmark effectiveness and identify single-view inputs and offline/open-loop post-training as limitations.

5 Conclusion

  • ExploreVLA jointly predicts future trajectories, RGB images, and depth maps, using dense world-model supervision to enrich scene representations.The framework also uses prediction uncertainty as an intrinsic novelty measure with safety-gated reward and GRPO to guide exploration beyond expert imitation.Source: §5 · p. 15
  • Experiments on NAVSIM and nuScenes validate the approach, with state-of-the-art performance reported on NAVSIM.Source: §5 · p. 15
  • The framework uses a single front-view camera, while offline/open-loop RL may bias policies toward training data and limit learning recovery from self-induced states.The authors propose multi-view inputs and large-scale closed-loop RL and evaluation as future work.Source: §5 · p. 15

A More Implementation Details

The implementation builds on Show-o with Phi-1.5 and MAGVIT-v2, using current and historical front-view images to predict future RGB and depth. Training uses AdamW, with LoRA and specified GRPO settings in later stages.

A More Implementation Details

  • The model uses Show-o with Phi-1.5 as its language-model backbone and pre-trained MAGVIT-v2 as its image tokenizer.Input images are resized to 256 × 448, and Metric3D-ViT-Giant2 generates ground-truth depth supervision.Source: A More Implementation Details · p. 20
  • The model takes current and 0.5-second-old front-view images as input and predicts RGB and depth images 0.5 seconds ahead.Source: A More Implementation Details · p. 20
  • AdamW is used throughout; Stage 1 uses a 3×10−5 learning rate, while Stage 2 applies rank-32 LoRA at 3 × 10−6.Source: A More Implementation Details · p. 20
  • GRPO uses group size 8, KL penalty 0.01, clipping range 0.1, safety threshold 0.9, and exploration bonus weight 0.5.Experiments use 4×H200 GPUs, with per-GPU batch sizes of 8 in Stage 1 and 1 in Stage 2.Source: A More Implementation Details · p. 20

B.1 Evaluation on NAVSIM

On NAVSIM navtest, qualitative examples show planned trajectories across straight-driving, turning, and intersection scenarios, alongside future RGB and depth predictions. The reported visualizations indicate road-aligned planning and preservation of global scene layout, with some fine-detail degradation in generated images.

B.1 Evaluation on NAVSIM

  • Across straight, turning, and intersection scenarios, predicted trajectories closely follow road topology and traffic context, including reasonable plans in complex intersections.Straight cases show stable lane keeping, while turns are described as smooth and well timed.Source: More Qualitative Results · p. 20, Figure 5 · p. 21
  • Future RGB images and depth maps align with ground truth in global scene geometry and layout, though texture sharpness and thin structures degrade.Examples compare the current frame with ground-truth and predicted future RGB images and depth maps.Source: Image Generation from Dense World… · p. 20, Figure 6 · p. 22
  • The trajectory examples pair front-view camera images with BEV representations overlaying ground-truth trajectories in green and predictions in orange.Source: Figure 5 · p. 21

B.2 Evaluation on nuScenes

On nuScenes, the Stage 1 model is evaluated under ST-P3 and UniAD open-loop protocols, without Stage 2 reinforcement-learning post-training. It reports a 0.10% average collision rate under ST-P3 and competitive average L2 errors under both protocols.

B.2 Evaluation on nuScenes

  • Both open-loop protocols evaluate 1s, 2s, and 3s trajectories using waypoint L2 error and collision rate.The reported nuScenes results use Stage 1 only because the dataset lacks established closed-loop metrics for RL reward signals.Source: Evaluation Metrics · p. 23, Quantitative Analysis · p. 23
  • 0.44m average L2 error and 0.10% average collision rate under ST-P3; the collision rate is lowest on average and matches OpenDriveVLA.ExploreVLA has the best collision rate at 1s and 2s horizons.Source: Quantitative Analysis · p. 23
  • 0.77m average L2 error under UniAD is competitive with UniAD at 1.03m and AutoVLA at 0.70m.Source: Quantitative Analysis · p. 23

B.3 Closed-Loop Evaluation on HUGSIM

ExploreVLA is evaluated in reactive closed-loop HUGSIM without closed-loop fine-tuning or domain adaptation. It performs competitively on three domains, while Waymo reveals a notable gap attributed primarily to open-loop RL fine-tuning.

B.3 Closed-Loop Evaluation on HUGSIM

  • HUGSIM evaluation tests the model in a fully reactive setting across four source domains, using HD-Score to aggregate five driving criteria.The evaluation is zero-shot: the model is deployed without closed-loop fine-tuning or domain adaptation.Source: §B.3 · p. 24
  • ExploreVLA achieves the best HD-Score on KITTI-360 and ranks second on nuScenes and PandaSet in zero-shot closed-loop evaluation.It is deployed without closed-loop fine-tuning; the table reports HD-Score across four source domains.Source: §B.3 · p. 24, Table 8 · p. 24
  • A notable Waymo gap is attributed primarily to open-loop RL fine-tuning, which provides limited experience recovering from self-induced states during closed-loop rollout.The authors also note that the exploration reward may anchor the policy toward the training distribution and identify closed-loop RL fine-tuning as future work.Source: §B.3 · p. 24

Related papers

Loading 2604.02714v2…