PaperlayerView exact revision on arXiv

Source-linked AI summary

Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models

Canyu Chen, Yuguang Yang, Zhewen Tan, Yizhi Wang, Ruiyi Zhan, Haiyan Liu, Xuanyao Mao, Jason Bao, Xinyue Tang, Linlin Yang, Bingchuan Sun, Yan Wang, Baochang Zhang

arXiv:2603.06049v1cs.CVcs.RO

TL;DR

Driving VLA training can develop a Narrow Policy in which imitation learning limits behavioral diversity and weakens later reinforcement learning. Curious-VLA addresses this with feasible trajectory expansion, normalized trajectory representations, diversity-aware sampling, and spanning rewards, achieving state-of-the-art Navsim results and a Best-of-N PDMS of 94.8.

  • Problem

    Driving VLAs suffer from a Narrow Policy limitation: the IL-RL pipeline can produce insufficient behavioral diversity, restricting exploration and feedback diversity for subsequent RL.

  • Method

    Curious-VLA combines feasible trajectory expansion and step-wise normalization during IL with adaptive diversity-aware sampling and spanning driving reward during RL.

  • Results

    Curious-VLA achieves state-of-the-art Navsim performance, including PDMS 90.3, EPDMS 85.3, and Best-of-N PDMS 94.8.

  • Takeaways & Limitations

    The framework balances exploitation and exploration across data, sampling, and reward, supporting more diverse and reliable autonomous-driving planning.

Abstract

from arXiv · show

We identify a fundamental Narrow Policy limitation undermining the performance of autonomous VLA models, where driving Imitation Learning (IL) tends to collapse exploration and limit the potential of subsequent Reinforcement Learning (RL) stages, which often saturate prematurely due to insufficient feedback diversity. Thereby, we propose Curious-VLA, a framework that alleviates the exploit-explore dilemma through a two-stage design. During IL, we introduce a Feasible Trajectory Expansion (FTE) strategy to generate multiple physically valid trajectories and a step-wise normalized trajectory representation to adapt this diverse data. In the RL stage, we present Adaptive Diversity-Aware Sampling (ADAS) that prioritizes high-diversity samples and introduce Spanning Driving Reward (SDR) with a focal style weighting to amplify reward's value span for improving sensitivity to driving quality. On the Navsim benchmark, Curious-VLA achieves SoTA results (PDMS 90.3, EPDMS 85.4) and a Best-of-N PDMS of 94.8, demonstrating its effectiveness in unlocking the exploratory potential of VLA models. Code: https://github.com/Mashiroln/curious_vla.git.

1. Introduction

Driving VLAs commonly use IL followed by RL, but this pipeline can suffer from Narrow Policy: imitation learning limits behavioral diversity and weakens subsequent exploration. Curious-VLA addresses this through diverse feasible trajectories and exploration-oriented RL components.

  • Existing VLA Training: Current driving VLAs use either an additional trajectory planner or direct trajectory tokens, followed by IL and then RL.The two paradigms differ architecturally but share the same two-stage training pattern.
  • Narrow Policy Evidence: Behavioral diagnostics show baselines with low trajectory diversity and quality, including mean-pFDE values of 0.20m for Qwen2.5-VL and 0.33m for ReCogDrive.Qwen2.5-VL also has a min-FDE of 1.05m, while Curious-VLA achieves the best diversity, quality, and performance in the comparison.
  • IL-stage Design: Curious-VLA expands imitation data with multiple physically valid driving paths and normalizes each prediction step to separate diverse behaviors.The approach treats the ground-truth trajectory as one potential human driving behavior rather than the only valid behavior.
  • RL-stage Design: During RL, ADAS prioritizes exploratory samples while SDR expands reward values to improve sensitivity to driving quality.Together, these components target insufficient exploration and premature concentration on a single behavioral pattern.

2. Related Work

Related work extends VLMs from scene understanding toward planning and control, while driving VLA methods remain organized around planner-based and token-based paradigms. Curious-VLA targets the shared Narrow Policy limitation through coordinated changes to data, sampling, and reward.

  • VLA Evolution: VLM-based driving research has progressed from scene understanding and reasoning toward direct action planning in unified VLA systems.Early work focused on captioning, question answering, and intention recognition, while later systems incorporate richer context for reasoning and control.
  • VLA Paradigms: Driving VLAs comprise VLA-Planner methods with an additional trajectory planner and VLA-Token methods that directly generate trajectory tokens.AutoVLA and SMART are examples of the VLA-Token paradigm.
  • Shared Limitation: Both paradigms commonly use an IL-then-RL pipeline, but existing methods suffer from Narrow Policy associated with the explore–exploit dilemma and insufficient diversity.Curious-VLA follows VLA-Token while addressing data, sampling, and reward across both training stages.
  • Verifiable Reward RL: RL research has expanded from RLHF to verifiable rewards, with GRPO removing value networks through group-based normalization and later methods refining exploration or advantage estimation.Autonomous driving has also adopted RLVR-style fine-tuning methods.

3. Preliminary and Narrow Policy

The paper formulates driving VLA training as supervised trajectory modeling followed by GRPO-style RL, then analyzes Narrow Policy through objective mismatch, physical-scale imbalance, advantage collapse, and behavioral diagnostics.

  • VLA Training Pipeline: A driving VLA maps multimodal observations to an action sequence of spatial and speed actions, with inputs including images, instructions, and ego states.In the VLA-Token setting, SFT maximizes the likelihood of generated trajectory tokens to establish basic planning capabilities.
  • VLA Training Pipeline: GRPO samples groups of outputs, combines a clipped policy objective with a KL constraint, and standardizes rewards within each group to compute advantages.The group mean and standard deviation determine the normalized reward-based advantage.
  • Narrow Policy Analysis: Cross-entropy SFT treats all non-ground-truth tokens as equally incorrect, providing no smoother preference for near-correct trajectory predictions.This discrete supervision encourages overconfidence in the ground-truth trajectory and can collapse the policy around one expert mode.
  • Narrow Policy Analysis: Far-horizon waypoint variance can dominate the SFT loss, while near-horizon steering actions contribute less and behavioral diversity is reduced.The paper contrasts waypoint variance at 4s with that at 0.5s, reporting orders-of-magnitude larger variance at the farther horizon.
  • Narrow Policy Analysis: When a policy collapses to one trajectory mode, sampled rewards become nearly identical, causing normalized advantages to approach zero and gradients to vanish.This is identified as advantage collapse in the RL stage.
  • Behavioral Diagnostics: Behavioral Diagnostics measure diversity through mean pairwise ADE/FDE, quality through minimum ADE/FDE, and performance through mean Navsimv1 PDMS.The diagnostics sample k = 8 trajectories over a 4-second, 8-step horizon.
  • Behavioral Diagnostics: A balanced policy should have high Diversity@k, low Quality@k, and high Performance@k; diversity collapse with stagnant quality signals the Narrow Policy bottleneck.The metrics jointly assess both the breadth and effectiveness of policy exploration.

4. Curious-VLA

Curious-VLA combines Feasible Trajectory Expansion for imitation learning with diversity-aware reinforcement learning to broaden trajectory exploration and reward feedback. Its pipeline uses step-wise normalization, Adaptive Diversity-Aware Sampling, and a focal-style Spanning Driving Reward.

  • Curious-VLA: Curious-VLA combines Feasible Trajectory Expansion in imitation learning with diversity-aware reinforcement learning to address narrow-policy exploration limitations.The framework spans trajectory data expansion, sampling, and reward design across both training stages.
  • Imitation Learning: Feasible Trajectory Expansion identifies 12k challenging segments from NavTrain and synthesizes multiple feasible trajectories for standard supervised fine-tuning.The selected segments include multi-lane, intersection, and occlusion scenarios.
  • Imitation Learning: Step-wise normalization equalizes gradient magnitudes across horizons, improving trajectory-pattern separability and supporting exploration.During SFT, normalized trajectories are used for training and later de-normalized for testing.
  • Reinforcement Learning: ADAS selects scenarios with diverse stochastic-policy rollouts to maintain reward variance for stable GRPO optimization.It uses offline rollouts to estimate reward statistics before active-set filtering and online training.
  • Reinforcement Learning: SDR reformulates Navsim metrics with focal-style weighting, amplifying differences between suboptimal and optimal behaviors and increasing reward sensitivity to driving quality.The formulation is based on PDMS and EPDMS components, including safety constraints and weighted driving objectives.

5. Experiment

Curious-VLA is evaluated across Navsim v1/v2 and nuScenes, achieving strong benchmark performance and improved exploration through its IL and RL designs. Ablations show that step-wise normalization, diversity-aware sampling, and spanning rewards are central to converting trajectory diversity into quality and performance.

  • Navsim Results: 90.3 PDMS establishes a new Navsim v1 SOTA under single-front-camera input, improving 1.2 PDMS over AutoVLA.Curious-VLA also approaches methods using additional sensors or larger bases while using a 3B Qwen2.5-VL model.
  • Navsim Results: 94.8 Best-of-N PDMS with N=6 surpasses AdaThinkDrive† by 1.8 PDMS and matches the Human GT level.The result is attributed to trajectories that combine high diversity with high quality.
  • Navsim Results: 85.3 EPDMS on Navsim v2 sets a new SOTA, improving 0.8 over DiffusionDrive’s 84.5.Performance is supported particularly by Drivable Area Compliance, Ego Progress, and Time to Collision.
  • nuScenes Results: Curious-VLA outperforms existing VLAs and end-to-end models on nuScenes 3-second L2 error and Collision rate.The pipeline is applied to a 28k-sample training subset with an ADE-based reward for RL.
  • Exploration Analysis: Step-wise Normalization raises diversity while maintaining 91.31 mean-PDMS, and the RL stage reaches 1.415 mean-pFDE and 0.547 minFDE.Expanded CoT data without normalization degrades trajectory quality, whereas normalization enables effective learning from diverse trajectories.
  • Ablation Study: 90.3 PDMS is achieved by combining ADAS with SDR, exceeding ADAS alone at 89.6 and avoiding collapse observed with several alternative sampling strategies.Rejecting unimodal reward distributions reaches 88.8 PDMS, while Human Difficulty, Random Sample, and Full Trainset reach 35.2 PDMS.

6. Conclusion

Curious-VLA identifies Narrow Policy as an exploit-explore imbalance in driving VLA training and addresses it through a framework spanning data, sampling, and reward. Its design aims to increase behavioral and policy diversity for more comprehensive planning.

  • Narrow Policy in driving VLA is attributed to an exploit-explore imbalance that previous works largely neglected.
  • Imitation Learning provides a robust foundation from high-value ground-truth data but suffers limited behavioral diversity due to data scarcity.
  • Reinforcement Learning alone is insufficient to resolve the exploit-explore dilemma, particularly for critic-free RL algorithms such as GRPO in VLMs.
  • Curious-VLA systematically addresses the imbalance through coordinated changes to data, sampling, and reward across IL and RL.

Supplementary Material

The supplementary material specifies the SFT and GRPO configurations, including a two-stage SFT procedure and the reward redesign used for RL training.

  • The supplementary material provides detailed configurations for both the Imitation Learning and Reinforcement Learning stages.
  • SFT: SFT first aligns the LLM with structured chain-of-thought and external reasoning tokens while freezing the vision encoder and projector.
  • SFT: SFT then unfreezes all parameters for end-to-end fine-tuning that jointly optimizes the visual and language components.
  • Configurations: The supplementary material lists separate hyper-parameter tables for SFT and GRPO.
  • RL: The RL reward is redesigned by adapting EPDMS into an additive focal objective with strict safety constraints, while removing EC for training efficiency.

B.1. Details on Exploratory Data Expansion

Curious-VLA expands exploratory training data by filtering challenging scenes, varying driving intents, and retaining candidates that satisfy safety and diversity criteria. The resulting samples broaden behavioral coverage while preserving efficient inference.

  • Scenario filtering: The pipeline identifies 12k challenging driving segments from the 103k navtrain split, including multi-lane roads, intersections, and occlusions.
  • Scenario filtering: A prompt template filters scenarios that semantically contain multiple feasible driving trajectories.
  • Intent expansion: The expansion process explores execution variations within an intent and systematically switches among Go Straight, Turn Left, Turn Right, and Unknown intents.
  • Candidate filtering: Candidates are retained only when they pass safety and diversity filtering, with safety requiring a PDMS score above 95.0.
  • Output: 39k non-ground-truth yet physically feasible exploratory samples are produced, broadening behavioral coverage beyond narrow human ground truth.
  • Trajectory prediction: Curious-VLA uses structured reasoning from visual, kinematic, and intent inputs to predict normalized four-second trajectory waypoints.
  • Efficiency: Curious-VLA achieves 1.57s per sample, 7.74s faster than AutoVLA’s text waypoint mode and close to its 1.31s Action+RFT mode.

D. RL Training Stability

The RL training analysis reports stable improvement across the full training process and across independent runs, whereas random sampling leads to RL collapse.

  • Across 130 training steps, Curious-VLA reports validation reward and test PDMS while using ADAS with three-times sampling and three outer loops.
  • The Random Sample baseline without ADAS shows RL collapse, supporting the need for diversity-aware sampling.
  • Across k = 4 independent training runs with ADAS 1x, critic and validation-reward curves improve consistently with low variance.

E. External Analytical Experiments

External analyses show that Curious-VLA maintains diverse feasible behaviors where baseline selection can collapse exploration, while its RL training remains stable and improves consistently across runs.

  • Extended exploration analysis: Curious-VLA achieves a superior balance across Quality, Diversity, and Performance against DiffusionDrive’s confidence-selected trajectory.DiffusionDrive’s 20-candidate pool collapses at Top-1 selection to the lowest diversity, with mean-pADE/FDE of 0.037 / 0.076.
  • RL training curves: The Random Sample baseline shows RL collapse across validation reward and test PDMS over 130 training steps.
  • Stability analysis: Critic and validation-reward curves improve consistently with low variance across k = 4 ADAS 1x training runs.
  • Behavior visualization: Visualization indicates that Curious-VLA alleviates the narrow policy bottleneck and preserves diverse feasible behaviors.
Loading 2603.06049v1…