Reflect-VLA: Internalizing Reflective Guidance via Dual-Path On-Policy Self-Distillation for Autonomous Driving

1 Introduction

• Across model scales, Reflect-VLA improves planning over SFT while requiring substantially less training time than RL and enabling guidance-free single-pass planning.This comparison is summarized in Figure 1.Source: Figure 1 · p. 1
• Scalar planning scores collapse distinct scene-understanding, behavior-selection, and trajectory-execution failures without indicating how to correct them.The paper identifies this as a particular challenge in complex and long-tail scenarios.Source: §1 · pp. 1–2
• Reflective Guidance can improve an initial trajectory, but requires that trajectory and its evaluation, making it unavailable for direct single-pass inference.The challenge is to transfer its planning preferences into direct trajectory generation rather than perform iterative inference-time refinement.Source: §1 · p. 2
• Dual-OPSD routes student-generated trajectories by planning score, distilling teacher corrections for risky plans and using self-imitation to preserve reliable decisions.Reflective Guidance is privileged information available during training; the score routes trajectories while teacher distributions provide fine-grained supervision.Source: §1 · p. 2
• The teacher is progressively built through driving-knowledge pretraining, planning-and-refinement supervision, and self-refinement rejection sampling to support guidance internalization.The final stage uses student-generated failures and verified refinements to reduce the distribution gap; inference then needs no guidance, online evaluator, or iterative correction.Source: §1 · p. 2

2.1 VLA models for autonomous driving

• Autonomous-driving VLA pipelines commonly use driving-data supervised fine-tuning followed by reinforcement learning, but scalar rewards offer limited supervision for diverse failures.ELF-VLA (Luo et al. 2026a) uses teacher feedback to diagnose failures and construct improved training samples; this work instead internalizes structured guidance for inference.Source: §2.1 · p. 2

2.2 On-Policy Distillation and On-Policy Self-Distillation

• On-policy distillation queries a teacher using student-generated prefixes, providing token-level supervision on trajectories sampled from the student policy.On-policy self-distillation can condition the same model on privileged information to produce the teacher signal.Source: §2.2 · p. 2
• Reflect-VLA applies privileged reflective guidance to student rollouts, distilling corrections when needed and preserving decisions on reliable trajectories.This quality-aware approach targets autonomous-driving planning, where the application of on-policy distillation remains limited.Source: §2.2 · p. 2

3.1 VLA Inputs Formulation

• The base input combines front-view imagery, a navigation command, ego state, and historical trajectory; guidance adds an initial trajectory and structured diagnosis and correction.The guidance corpus includes a trajectory verdict, planning diagnosis, expert alignment, and refinement directive.Source: §3.1 · p. 3

3.2 Data Construction

• Refinement data pairs low-quality evaluated trajectories with Reflective Guidance and a target that is ground truth or an evaluator-verified high-quality refinement.Initial candidates include classical-planner trajectories and rule-based perturbations assessed with the NAVSIM evaluator, map constraints, and trajectory-level analysis.Source: §3.2 · p. 4

3.3 Three-stage SFT for Planning and Refinement

• The teacher is built through driving-knowledge pretraining, planning-and-refinement SFT, and self-refinement rejection sampling to correct failures from its unguided rollouts.The three stages establish domain knowledge, train direct generation and guided refinement, then improve reliability for subsequent on-policy distillation.Source: §3.3 · p. 4

3.4 Dual-Path On-Policy Self-Distillation

• Dual-OPSD routes student rollouts by planning score: reliable trajectories are self-preserved, while risky ones receive token-level corrective preferences from a frozen guidance-conditioned teacher.The student remains restricted to standard inputs during this distillation.Source: §3.4 · p. 4

4.1 Implementation details

• The evaluation covers NAVSIM v1 and v2 trajectory planning and Bench2Drive closed-loop performance, using PDMS, EPDMS, and Bench2Drive driving metrics.NAVSIM v2 extends evaluation to traffic-rule compliance, lane-level behavior, and temporal consistency.Source: §4.1 · p. 5

4.2 Performance Comparison

• Reflect-VLA improves NAVSIM v1 PDMS over corresponding RL baselines at all three model scales, with scores of 89.3, 90.0, and 91.5 for 0.8B, 2B, and 4B.The 4B model achieves the best overall PDMS score, with improved TTC while maintaining strong performance on other metrics.Source: §4.2 · pp. 5–6
• Reflect-VLA-4B achieves a Bench2Drive Driving Score of 80.15, Efficiency of 212.20, and Success Rate of 59.09%, outperforming existing baselines including AutoVLA.The evaluation reports closed-loop driving performance on Bench2Drive.Source: §4.2 · p. 6, Table 3 · p. 6
• OPSD is 4.1× faster than GRPO for 0.8B, 13.1× faster for 2B, and 13.9× faster for 4B in training time per 10,000 samples.For 2B and 4B, OPSD requires 1.7 and 2.4 hours, respectively.Source: §4.2 · p. 6, Table 5 · p. 6

4.3 Qualitative Analysis

• In qualitative examples, Dual-OPSD transfers safer longitudinal and lateral corrections to the student, while token distributions show the teacher preferring a 2.08 m rather than 2.15 m forward displacement.The examples include a collision-risk case and a drivable-area violation; the student reproduces feasible trajectories without guidance at inference.Source: §4.3 · p. 6, Figure 3 · p. 7

4.4 Ablation Studies

• Pretraining raises PDMS from 88.1 to 88.5, while adding OPSD to self-refinement raises it from 90.2 to 91.5; self-refinement alone reaches 88.4.The authors attribute the combined gain to refinement samples from model rollouts aligning guidance-conditioned behavior with on-policy failures.Source: §4.4 · p. 7, Table 6 · p. 7
• The complete reflective-guidance design achieves PDMS 93.2, and removing any component lowers performance; Expert Alignment and Planning Diagnosis yield the largest gains.These components identify planning failures and relate them to deviations from expert behavior.Source: §4.4 · p. 7, Table 7 · p. 7

5 Conclusion

• Reflect-VLA transfers token-level corrective distributions from guidance-conditioned generation into direct trajectory prediction, with competitive NAVSIM and Bench2Drive performance and up to a 10× training speedup over reinforcement-learning baselines.The conclusion emphasizes guidance-free single-pass planning across model scales.Source: §5 · p. 7

A Appendix

• The appendix adds nuScenes planning results and ablations, construction and implementation details, and qualitative token-level analyses of corrective preference transfer.It also reports data construction, model architecture, training parameters, hardware, and benchmark metrics.Source: A Appendix · p. 10

B Question

• Reflective Guidance is unavailable for direct inference because it requires an initial trajectory and evaluation, so Reflect-VLA internalizes its preferences to avoid an online evaluator.The appendix states that direct use would require an additional prediction–evaluation–refinement loop.Source: B Question · p. 10
• Dual-OPSD distills teacher distributions for risky rollouts while preserving reliable predictions through self-preservation.Teacher correction may unnecessarily modify predictions that are already reliable.Source: B Question · p. 10
• Self-Refinement Rejection Sampling exposes the teacher to model-generated failures and retains evaluator-verified improvements to better align corrections with student rollouts.The motivation is that offline refinement data may not capture the student’s failure patterns.Source: B Question · p. 10
• Reflective Guidance uses deterministic templates built from trajectory-evaluation results, geometric differences from a reference trajectory, and the navigation command, without another VLM or manual annotation.Source: B Question · p. 10

• Reflect-VLA-4B achieves state-of-the-art performance among VLA-based methods, with average L2 error of 0.41 m and collision rate of 0.14%.Source: §C.1 · p. 10
• Reflect-VLA-2B reports average L2 error of 0.52 m and collision rate of 0.20%, outperforming OpenDriveVLA and approaching FSDrive.Source: §C.1 · p. 10
• The 0.8B variant remains comparable to AutoVLA on the nuScenes results.Source: §C.1 · p. 10

C.2 Ablation Studies

• Increasing top-k support from 16 to 64 raises PDMS from 90.2 to 91.5, while larger supports yield only marginal gains; the paper uses k = 64.Source: §C.2 · p. 10, Table 9 · p. 11
• Reverse KL raises PDMS from 91.0 to 91.5; the authors adopt it because it concentrates student probability on the teacher’s high-probability corrective tokens.Forward KL favors broader distribution coverage, which may dilute dominant correction preferences.Source: §C.2 · p. 10, Table 10 · p. 11
• Raising routing threshold s from 0.65 to 0.85 gradually improves PDMS, but s = 0.95 also routes reliable trajectories and may disrupt effective behavior; the chosen threshold is 0.85.Source: §C.2 · p. 10, Table 11 · p. 11

D.1 Trajectory Tokenization

• Trajectories are autoregressively represented as relative longitudinal, lateral, and heading displacements, uniformly quantized at 0.01 resolution and decoded back into future trajectories.Each quantized index maps to a special token, with the three components concatenated in longitudinal–lateral–heading order.Source: §D.1 · pp. 10–11

D.2 Reflective Guidance Construction

• Reflective Guidance uses evaluator results, navigation commands, and a reference trajectory to produce a verdict, diagnosis, expert alignment, and ordered refinement directive.Diagnoses prioritize safety-critical failures; directives specify adjustments while preserving smooth waypoints and navigation intent.Source: §D.2 · pp. 11–12

D.3 Details of Pretraining Data

• Driving-knowledge pretraining draws on open-source datasets including LingoQA, DriveGPT4, Senna, DriveLM, NuScenes-QA, NuInstruct, and DriveLMM.Source: §D.3 · p. 12

D.4 Details of SFT Data

• Base planning samples pair front-view images and ego state with navigation commands, historical trajectories, and ground-truth future trajectories.Trajectories are sampled at 0.5-second intervals, and future trajectories are converted to discrete tokens for autoregressive prediction.Source: §D.4 · p. 12
• Guidance-conditioned refinement samples pair base driving inputs with a low-quality initial trajectory, Reflective Guidance, and a reference trajectory.Initial trajectories come from classical-planner outputs or rule-based perturbations of valid trajectories.Source: §D.4 · p. 12
• Planner-generated failures are retained when correctable in safety, drivable-area compliance, or progress, while rule perturbations vary longitudinal, lateral, or heading dimensions.Source: §D.4 · p. 12
• Candidates are evaluator- and map-checked; low-quality but valid trajectories become initial inputs, with expert or evaluator-verified improved trajectories as targets.Source: §D.4 · p. 12
• Self-refinement examples retain only model-generated refinements that satisfy trajectory and map constraints and score higher than their unguided initial trajectories.When several valid refinements are available, the highest-scoring one is retained.Source: §D.4 · p. 12

E.1 Training Parameters and Hardware Configuration

• Driving-knowledge pretraining runs for 2 epochs with batch size 1 and learning rate 1 × 10−5 on 8 NVIDIA H20 GPUs.Source: §E.1 · p. 12
• Planning-and-Refinement SFT and Self-Refinement Rejection Sampling each use 2 epochs, batch size 4, and learning rate 4 × 10−5.Source: §E.1 · p. 12
• Dual-OPSD runs for 2 epochs with learning rates 8 × 10−6 and 2 × 10−6; reliable trajectories use self-preservation, while risky ones receive teacher token-level supervision.Rollouts use greedy decoding at temperature 0; distillation retains teacher top-k support k = 64 and minimizes reverse KL, with routing threshold s = 0.85.Source: §E.1 · pp. 12–13

E.2 Metrics

• NAVSIMv1 uses PDMS, combining No At-Fault Collision and Drivable Area Compliance penalties with weighted Ego Progress, Time-to-Collision, and Comfort.Source: §E.2 · p. 13
• Bench2Drive reports Driving Score, Success Rate, Efficiency, and Comfort; nuScenes open-loop trajectory quality uses L2 error and collision rate.Driving Score combines route completion with multiplicative infraction penalties, while Success Rate counts routes completed within the time limit without traffic violations.Source: §E.2 · p. 13
• NAVSIMv2 uses EPDMS, extending the metric set to include driving-direction and traffic-light compliance, lane keeping, and history and extended comfort.Source: §E.2 · p. 13

F Visualization Analysis

• Across three representative failures, the distilled student makes safer trajectory adjustments resembling the guidance-conditioned teacher without inference-time guidance.The cases are drivable-area violation, insufficient TTC margin, and at-fault collision risk.Source: F Visualization Analysis · p. 13, Figure 4 · p. 14
• For drivable-area violation, the teacher shifts away from the road boundary into the valid lane corridor, and the student shows a similar lateral adjustment.Source: F Visualization Analysis · p. 13
• For insufficient TTC margin and at-fault collision risk, the teacher reduces or postpones progress and increases clearance; token distributions shift the student toward safer motions.Source: F Visualization Analysis · p. 13, Figure 4 · p. 14