PaperlayerView exact revision on arXiv

Source-linked AI summary

Sometimes You Gotta Run Before You Can Walk: Run-then-Walk Scheduling Strategy for VLM Autonomous Driving

Yuqi Ye, Shangkun Sun, Junhong Lin, Jiayi Zhao, Changhao Peng, Wei Zheng, Guoqing Liu, Tiesong Zhao, Wei Gao

arXiv:2609.25831v1cs.ROcs.CV

TL;DR

VLM-driving GRPO recipes face a progress–safety trade-off and lengthy training. Run-then-Walk separates progress discovery from safety repair, and experiments report higher driving performance with 40–50% fewer training steps across planners and benchmarks.

  • Problem

    GRPO recipes for VLM driving struggle to balance progress and safety, while existing approaches require lengthy training.

    Source: §1 · p. 2

  • Method

    Run-then-Walk first explores progress under relaxed constraints, then applies endpoint and safety objectives to repair unsafe behaviors.

    Source: §1 · p. 2, §4.1 · pp. 4–5

  • Results

    Across VLM-based autoregressive and diffusion planners on multiple benchmarks, driving performance improves with 40–50% fewer RL training steps.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings support a progress-first, safety-refinement curriculum as an efficient and effective strategy for VLM-based driving RL.

    Source: §7 · p. 9

    The theoretical analysis is idealized and explains reward direction rather than guaranteeing convergence.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

Recent VLM-based autonomous driving planners adopt GRPO-style reinforcement learning to optimize driving performance. However, existing GRPO recipes either optimize driving efficiency, risking progress-seeking but unsafe behavior, or enforce early safety constraints, leading to overly conservative behavior; both require lengthy training. To solve these problems, we first reveal two distinct RL regimes: a progress regime (Run-GRPO) that aggressively explores high progress, and a safety regime (Walk-GRPO) that restores safety under stable progress. Based on this finding, we propose $\textit{Run-then-Walk}$, a simple yet effective two-stage reward scheduling strategy for GRPO, achieving both better performance and faster convergence. Unlike one-stage RL, which may focus on progress, safety, or a mixture of both within a single training phase, this schedule explicitly separates progress discovery from safety repair. In the $\textit{Run}$ phase, we focus on progress, allowing the policy to escape the conservative bias and discover high-progress modes. In the subsequent $\textit{Walk}$ phase, we introduce endpoint and safety strategy to repair unsafe behaviors from the Run phase. This reversed schedule overcomes the conservatism of Walk-first methods and the unsafe progress-seeking of joint optimization. We validate it with various VLM-based planners on multiple benchmarks: NAVSIMv1, NAVSIMv2, Navhard, and nuScenes. Extensive experiments demonstrate improved driving performance while requiring 40--50\% fewer RL training epochs than the baselines.

1 INTRODUCTION

The introduction frames the progress–safety trade-off and long training as challenges for GRPO-based VLM driving. It proposes Run-then-Walk, which explores progress before repairing safety, and reports improved performance with fewer training steps.

1 INTRODUCTION

  • Balancing progress and safety remains a central challenge because composite closed-loop driving metrics entangle both objectives.Recent VLM planners use GRPO-style RL to improve closed-loop driving quality.Source: §1 · p. 2
  • Prior GRPO strategies either remain progress-biased with high collision, become overly conservative, or require 10 RL epochs.These shortcomings motivate a faster, more effective training paradigm for VLM-based driving.Source: §1 · p. 2
  • Run-GRPO rapidly explores high progress but struggles to repair safety, whereas Walk-GRPO preserves safety while progress improves slowly.The controlled comparison starts both strategies from the same SFT checkpoint on NAVSIM using AutoDrive-P3.Source: §1 · p. 2
  • Run-then-Walk first explores progress under relaxed constraints, then uses endpoint and safety objectives to repair unsafe behaviors.The introduction distinguishes this ordered schedule from one-stage mixed-objective RL and generic easy-to-hard curricula.Source: §1 · p. 2
  • Across autoregressive and diffusion planners on NAVSIMv1, NAVSIMv2, Navhard, and nuScenes, driving performance improves with 40–50% fewer RL training steps.The introduction also reports 5 epochs for AutoDrive-P3 and 6 for ReCogDrive, versus 10 epochs for the original RL policies.Source: §1 · p. 2

2 RELATED WORK

Related work describes the shift from modular driving systems to VLM planners and the adoption of GRPO for their reinforcement-learning fine-tuning. Existing schedules expose a progress–safety trade-off, while how RL regimes shape that trade-off remains poorly understood.

2 RELATED WORK

  • VLM planners either use model features with a downstream trajectory generator or directly emit trajectories as language tokens.The paper evaluates the autoregressive AutoDrive-P3 and diffusion-based ReCogDrive as representative planners.Source: §2 · p. 3
  • Driving GRPO approaches include mixed Walk-and-Run schedules that may retain unsafe progress-seeking and safety-first schedules that may suppress progress and require long training.The passage associates these approaches with ReCogDrive (Li et al., 2025c) and AutoDrive-P3, respectively.Source: §2 · p. 3
  • How RL training regimes shape the progress–safety trade-off in VLA planners remains poorly understood.The related-work discussion presents systematic characterization of these regimes as important for designing strategies with better performance and faster convergence.Source: §2 · p. 3

3 TOY EXPERIMENT: RUN-GRPO VS. WALK-GRPO FROM SFT

The toy experiment compares direct progress-oriented Run-GRPO with safety-constrained Walk-GRPO from the same SFT initialization. They fail in opposite ways, motivating a schedule that discovers progress first and repairs safety afterward.

3 TOY EXPERIMENT: RUN-GRPO VS. WALK-GRPO FROM SFT

  • Direct Run-GRPO raises ego progress but increases collisions and drivable-area violations, leaving overall PDMS below the SFT baseline.The experiment uses PDMS, a composite reward in which ego progress is the easiest component to optimize from the SFT initialization.Source: §3 · p. 3, Figure 3 · p. 4
  • After Run-GRPO enters an aggressive regime, safety repair searches for safer trajectories within that regime, making a balanced policy difficult to recover.The figure compares direct Run and constrained Walk for three epochs from the same SFT model.Source: §3 · p. 4, Figure 3 · p. 4
  • Direct Walk-GRPO preserves and improves safety but constrains exploration around the SFT solution, so PDMS improves only slowly.Its reward combines endpoint consistency with the PDMS safety terms DAC and NC.Source: §3 · pp. 3–4
  • The opposing failure modes motivate Run-then-Walk: expand the progress distribution first, then refine the policy for safe, stable closed-loop driving.The proposed ordering aims to break conservative bias before repairing the safety deficits introduced by exploration.Source: §3 · p. 4

4 METHOD

Run-then-Walk first uses relaxed constraints to discover high-progress actions, then adds endpoint and safety objectives to repair unsafe behavior. Its GRPO implementation preserves planner architecture and native auxiliary objectives, while the theoretical analysis is explicitly idealized rather than a convergence guarantee.

4.1 RUN-THEN-WALK PLANNING STRATEGY DESIGN

  • The reward is applied to decoded trajectories rather than a particular output representation, making the construction planner-agnostic across autoregressive and diffusion planners.The strategy preserves each planner’s native auxiliary supervision.Source: §4.1 · pp. 4–5
  • The Walk endpoint reward uses L1 distance from the predicted final point to the expert endpoint and decays in steps, with smaller ∆ enforcing a tighter constraint.A larger ∆ makes Walk looser and more aggressive; a larger η accelerates reward decay.Source: §4.1 · pp. 4–5
  • Run-then-Walk discovers high-progress actions under relaxed constraints, then adds endpoint and safety rewards to produce a safe, high-progress policy.Run intentionally relaxes safety repair to move beyond conservative SFT behavior; Walk introduces endpoint and safety objectives.Source: §4.1 · pp. 4–5, Figure 4 · p. 5

4.2 RUN-THEN-WALK STRATEGY

  • For each query, the strategy samples planner outputs, computes stage-wise rewards, normalizes them into group-relative advantages, and optimizes with GRPO.Architecture-native rewards can remain fixed across stages.Source: §4.2 · p. 5
  • Run expands the progress distribution from the SFT checkpoint; Walk continues from a selected Run checkpoint to repair safety without collapsing to the conservative SFT solution.The Run and Walk stages use their respective planning rewards and reference policies.Source: §4.2 · p. 6

4.3 IDEALIZED THEORETICAL JUSTIFICATION

  • In the idealized objective, quality is bounded by safety and weighted quality, so progress alone cannot compensate for poor safety.The analysis frames Run-then-Walk as reward-direction reasoning rather than a convergence guarantee.Source: §4.3 · p. 6
  • Under the idealized population update, a positive total-reward gap can increase the probability of samples passing Walk’s safety gate.The stated sufficient gap depends on the implementation’s planning-reward weight and the range of unchanged auxiliary rewards.Source: §4.3 · p. 6
  • The theoretical statements require useful safe-progress samples from Run, adequate group coverage, a fixed evaluation distribution, and a measured KL bound.Endpoint proximity is neither a path-safety nor a progress certificate.Source: §4.3 · p. 7

5 EXPERIMENTS

Experiments report improvements across planner families and benchmarks with fewer RL epochs than the 10-epoch baseline. The AutoDrive-P3 schedule uses three Run epochs followed by two Walk epochs, while ReCogDrive uses five Run epochs and one Walk epoch.

Benchmarks and baselines

  • Evaluation covers NAVSIMv1, NAVSIMv2, Navhard, and nuScenes with autoregressive AutoDrive-P3 and diffusion-based ReCogDrive, except nuScenes uses AutoDrive-P3 only.The experiments use PDMS on NAVSIMv1 and EPDMS on NAVSIMv2 and Navhard.Source: §5 · p. 7

Run/Walk strategy evolution

  • After three Run epochs increase progress but hurt safety, two Walk epochs restore safety and raise PDMS above the SFT baseline in five total epochs.The plotted curves describe AutoDrive-P3’s Run-then-Walk strategy.Source: Figure 5 · p. 8

Benchmarks and baselines

  • The experiments set the aggressive reward to PDMS and use ∆=10 and η=0.2 for endpoint reward across experiments.The supplied table captions also identify Run/Walk recipe comparisons and an analysis of safety metrics in Walk.Source: §5 · p. 8, Table 5 · p. 8, Table 6 · p. 8

Main results across planners and benchmarks

  • 40–50% fewer RL training epochs than the 10-epoch baseline accompany improvements across planner families and benchmarks; NAVSIMv1 PDMS rises 90.6 to 91.0 for ReCogDrive and 90.2 to 91.8 for AutoDrive-P3.NAVSIMv2 paired EPDMS rises from 82.7 to 83.7 for ReCogDrive and from 88.7 to 89.6 for AutoDrive-P3.Source: §5 · p. 8

6 ANALYSIS AND VISUALIZATION

Run-then-Walk reaches the strongest reported performance while reducing training epochs, and the analysis shows why the order of progress exploration and safety repair matters. Reward ablations and scenario visualizations further illustrate the safety–progress trade-off.

Analysis of Run-then-Walk Training Strategy

  • 91.8 PDMS for AutoDrive-P3 and 91.0 PDMS for ReCogDrive accompany restored safety and 40–50% fewer epochs than 10-epoch baselines.AutoDrive-P3 uses 5 RL epochs and ReCogDrive uses 6.Source: §6 · p. 8

Analysis of different RL recipes

  • Run-then-Walk achieves the highest PDMS; Only Run reaches EP 88.8 and PDMS 85.7, Only Walk reaches EP 81.5 and PDMS 89.7, and Walk-then-Run is suboptimal.Only Run's lower PDMS is attributed to poor safety, while Only Walk has the best safety but limits progress.Source: §6 · pp. 8–9

Effect of safety signal composition

  • DAC alone yields 91.5 PDMS; adding NC yields 91.8 PDMS and 88.9 EP, while adding TTC lowers EP to 86.9 and PDMS to 91.2.The analysis identifies DAC+NC as the best safety–progress trade-off and associates TTC as a reward signal with over-conservatism.Source: §6 · p. 9

Visualization

  • Across two scenarios, Walk-and-Run collides (NC = 0), Walk-then-Run advances less (EP = 0.65), while Run-then-Walk maintains high EP and safety.These are qualitative trajectory comparisons on two representative driving scenarios.Source: §6 · p. 9

7 CONCLUSION

Run-then-Walk separates progress exploration from safety repair while preserving each planner's architecture and native auxiliary objectives. It uses a progress-oriented planning reward in Run and endpoint and safety rewards in Walk.

7 CONCLUSION

  • Across autoregressive and diffusion planners on different benchmarks, Run-then-Walk improves driving performance with 40–50% fewer RL training steps.The strategy first explores progress under relaxed constraints, then repairs safety with endpoint rewards.Source: §7 · p. 9

A.1 PLANNER-AGNOSTIC REWARD INTERFACE

  • For each query, group-sampled planner outputs are decoded into trajectories, while native auxiliary objectives remain unchanged and only the planning reward and its schedule differ.Autoregressive planners assign reward to sampled token sequences; diffusion planners assign it to sampled continuous trajectories.Source: §A.1 · p. 13

A.2 PLANNING REWARD: RUN VS. WALK

  • Run uses PDMS without endpoint/L2 constraints or explicit safety rewards; Walk switches to an endpoint reward and safety gating based on NC and DAC.The endpoint reward is based on the L1 distance between the final predicted point and the expert endpoint.Source: §A.2 · p. 13

B IDEALIZED THEORETICAL JUSTIFICATION

The theoretical justification models driving quality as a safety-weighted objective, then gives conditional arguments for safety repair, progress retention, and score improvement. Its guarantees depend on reward separation, sampling support, and distribution-level KL conditions.

Safety direction of Walk

  • Under a fixed query and positive reference support, a sufficient reward gap makes passing samples more likely under the idealized Walk update.The stated sufficient gap is (3/2)λplan − ω > 0; ω must be established from the implementation’s auxiliary-reward scale.Source: B IDEALIZED THEORETICAL JUSTIFICATION · pp. 13–15

Objective and progress–safety bottleneck

  • The objective multiplies safety/compliance by a quality term that combines normalized progress with remaining quality, so high progress cannot offset vanishing safety.For PDMS, S = NC DAC and α = 5/12; for EPDMS, S = NC DAC DDC TLC and α = 5/16.Source: B IDEALIZED THEORETICAL JUSTIFICATION · p. 13

Local GRPO direction

  • At the Run checkpoint, an idealized group-normalized GRPO direction favors gate-passing samples when a group contains both passing and failing outcomes.Shared parameters, finite groups, advantage recomputation, and clipping mean the actual update need not follow that direction or reach the Gibbs optimizer.Source: B IDEALIZED THEORETICAL JUSTIFICATION · p. 15

Progress retention under KL

  • A measured full-output KL budget bounds changes in bounded statistics, but progress retention is nonvacuous only when Run safe-progress support exceeds the KL and Walk-failure terms.A nominal GRPO KL coefficient alone does not establish the required full-distribution bound; zero KL prevents repair, while very large KL weakens retention.Source: B IDEALIZED THEORETICAL JUSTIFICATION · p. 15

Final-score improvement

  • The condition (sW − sR)cR > sW ℓ is sufficient for strict score improvement, even if some scenes worsen.It can be estimated from paired trajectory scores; separate averages of NC, DAC, and EP do not establish it.Source: B IDEALIZED THEORETICAL JUSTIFICATION · p. 15

Sampling and scope conditions

  • Useful safe-progress samples must have Run support and adequate group coverage; the switch should precede loss of that support, with no universal schedule implied.The training gate is not the full evaluation safety factor, and endpoint proximity certifies neither path safety nor progress; protocol-specific analyses are needed for additional evaluation effects.Source: B IDEALIZED THEORETICAL JUSTIFICATION · pp. 15–16

C TRAINING HYPERPARAMETERS

The hyperparameter choices reflect each planner’s baseline and the progress-discovery stopping rule.

Run-stage duration

  • ReCogDrive uses five Run epochs, two more than AutoDrive-P3, because its EP takes longer to plateau from its SFT checkpoint.AutoDrive-P3 starts near EP 81.7 and ReCogDrive near 80; the Run stage stops once EP has essentially plateaued.Source: C TRAINING HYPERPARAMETERS · p. 16

Stage-specific rewards

  • Both planners use PDMS as the Run planning reward and Eq. 14 as the Walk reward, while only AutoDrive-P3 retains its baseline auxiliary reward.ReCogDrive does not require the additional auxiliary term to match its original baseline.Source: C TRAINING HYPERPARAMETERS · p. 16

D STAGE-BY-STAGE QUALITATIVE WALKTHROUGH: SFT →RUN →WALK

The walkthrough contrasts a safe but short SFT policy with Run’s progress discovery and Walk’s subsequent safety repair.

Stage 1: SFT (conservative imitation)

  • SFT imitates demonstrations collision-free but brakes early and stops short, producing low endpoint progress and capped PDMS.The described scenes typically show PDMS of 0.83–0.93.Source: D STAGE-BY-STAGE QUALITATIVE… · p. 16

Stage 2: Run-GRPO (progress discovery)

  • Run-GRPO’s unconstrained PDMS-only reward discovers farther-reaching trajectories, but frequent collisions or departures from the drivable area reduce NC and PDMS to zero.The passage characterizes this safety regression as a temporary exploration trade-off.Source: D STAGE-BY-STAGE QUALITATIVE… · p. 16

Stage 3: Walk-GRPO (safety repair)

  • Walk-GRPO reintroduces endpoint and safety rewards, retaining nearly Run-level progress while restoring collision-free trajectories and improving EP and PDMS over SFT.Across six scenes, Walk consistently exceeds SFT progress while matching its safety; PDMS is often 0.95–1.00.Source: D STAGE-BY-STAGE QUALITATIVE… · p. 17

E QUALITATIVE COMPARISON

The qualitative comparisons contrast Run-then-Walk with safety-first training that stops short and progress-dominant training that collides. The stage examples also show Walk retaining Run’s progress while removing its unsafe behavior.

Comparison setup

  • The comparison uses matched NAVSIM scenes with human reference, baseline, and Run-then-Walk trajectories; Figures 9–10 isolate conservative training and Figures 11–12 isolate progress-seeking risk.The passage specifies the same scene for each triplet.Source: E QUALITATIVE COMPARISON · p. 17

Walk-then-Run (AutoDrive-P3)

Walk-and-Run (ReCogDrive)

Stage-by-stage comparison

  • Across the stage examples, Run reaches farther but is unsafe; Walk inherits that progress and removes collisions or off-road events.The described Walk trajectories end close to or above the human progress level while remaining collision-free.Source: Figure 7 · p. 18, Figure 8 · p. 19

Related papers

Loading 2609.25831v1…