PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving

Shaoyuan Xie, Zihan Zhang, Jingxuan Wang, Jiashu Qu, Xiaoqing Liang, Lingdong Kong, Junchi Lu, Henrik I. Christensen, Qi Alfred Chen

arXiv:2610.20609v1cs.RO

TL;DR

VLAs must judge whether open-ended navigation instructions are safe in the current scene, a decision that modular planners previously handled explicitly. SafeDriveVLA combines explicit compliance modes with navigation-conditioned world-model dreaming and is evaluated alongside benchmarks for following, conflict awareness, and adversarial instructions. It reports improved closed-loop language following and fewer collisions under unsafe instructions.

  • Problem

    VLAs must learn to resolve navigation signals against local context, while their open-ended language interface makes this judgment more challenging.

    Source: §1 · p. 2

  • Method

    SafeDriveVLA predicts strict, cautious, or fallback compliance modes and uses dreamed outcomes of strict navigation-signal execution to inform conflict reasoning before action generation.

    Source: §1 · p. 2

  • Results

    SafeDriveVLA improves closed-loop navigation following and reduces unsafe behavior under conflicting instructions; it reports a 49.3% lift in closed-loop language following over the best SimLingo variant.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    The benchmarks jointly assess whether a driving VLA follows safe signals, recognizes scene-dependent conflicts, and handles instructions that are unsafe in any circumstance.

    Source: §1 · p. 2

    The mode labels are inferred from expert trajectories rather than driver intent, benchmark language uses fixed paraphrase pools, and world-model dreaming does not provide a formal safety guarantee.

    Source: §7 · p. 12

Abstract

from arXiv · show

Vision-Language-Action (VLA) models are gaining attention in autonomous driving, inheriting commonsense priors from large language models (LLMs) and widening the navigation interface from a fixed command set to free-form natural-language instructions. However, this open-ended interface exposes a new vulnerability surface: an inattentive driver may issue instructions that are unsafe given the current scene, and a malicious user may issue dangerous ones on purpose. We present the first benchmark suite that evaluates navigation following and conflict awareness together, along with adversarial robustness. Existing VLAs fall short on both: SimLingo completes no language-instructed lane change in closed loop, and all models degrade sharply under conflicting signals. We propose SafeDriveVLA, the first driving VLA that (1) explicitly predicts a three-level navigation-compliance mode to decouple conflict reasoning from action generation, and (2) uses navigation-conditioned world model dreaming to inform this prediction. Unlike prior world models that roll out expert or planned actions, ours dreams of strictly following the navigation signal, showing when following is safe. SafeDriveVLA reaches 88.16 Driving Score (DS) and 68.64% Success Rate (SR) on Bench2Drive, lifts closed-loop language following by 49.3% and reduces collisions under unsafe instructions from 60 to 38 compared to the best SimLingo variant. Video demonstrations are available on our project page.

SafeDriveVLA

Figure 1 contrasts explicit conflict resolution in modular driving stacks with the implicit judgment VLAs must learn.

SafeDriveVLA

  • Modular stacks resolve navigation signals against perception in a behavior planner, while VLAs must learn this judgment implicitly.A navigation command, waypoint, or language instruction can be unsafe given the observation.Source: Figure 1 · p. 1

1 Introduction

The introduction frames navigation-scene conflict as a safety challenge for VLAs and presents a benchmark suite and SafeDriveVLA to address it. SafeDriveVLA reports improved closed-loop following and fewer collisions under unsafe instructions.

1 Introduction

  • Modular behavior planners resolve navigation signals against local context, but end-to-end models internalize this judgment, and VLAs expand the interface to open-ended language.This shift removes the explicit safety mechanism that resolved navigation-scene conflicts.Source: §1 · p. 2
  • The first benchmark suite jointly evaluates navigation following and conflict awareness, addressing the risk that ignoring every instruction looks safe while following every instruction looks good on conflict tests.It also covers adversarial instructions that are unsafe under any circumstance.Source: §1 · p. 2
  • SimLingo completes none of its language-instructed lane changes in closed loop despite an 83% open-loop rate, while all models degrade under conflicting signals.The models execute conflicting signals without grounding them in the scene.Source: §1 · p. 2
  • SafeDriveVLA predicts strict, cautious, or fallback compliance modes and dreams of strict navigation-signal execution to inform conflict reasoning before action generation.The authors state that this dreaming makes the model less conservative than using the mode token alone.Source: §1 · p. 2
  • 88.16 Driving Score (DS) and 68.64% Success Rate (SR) on Bench2Drive, 49.3% higher closed-loop language following, and B2D-C collisions reduced from 60 to 38.The language-following comparison is against the best SimLingo variant.Source: §1 · p. 2

2 Related Work

Related work spans language-action alignment, driving world models, perception and safety, and navigation evaluation. SafeDriveVLA focuses on whether a navigation request is safe in its current scene and on closed-loop driving behavior.

2.1 Language–Action Alignment for Autonomous Driving

  • Prior driving language-action work explores reasoning, planning, shared representations, and instruction following; SafeDriveVLA’s mode token marks whether a requested maneuver should be executed, delayed, or overridden.The cited approaches include DriveGPT4 [29], Dolphins [30], DriveVLM [31], DriveMLM [32], RDA-Driver [33], and LinkVLA [34].Source: §2.1 · pp. 2–3

2.2 World Models and Prospective Action Reasoning

  • Driving world models predict pixels or latent states, often under expert actions or candidate intentions; SafeDriveVLA conditions rollout on the navigation-implied action to expose conflicts.The paper describes its dreamed latents as a prospective decision aid, not event-specific causal counterfactual prediction.Source: §2.2 · p. 3

2.3 Robust Perception and Safety-Aware Driving

  • Related safety work addresses perception robustness, scene representations, traffic rules, hazards, and unsafe requests, but these approaches cover complementary aspects of driving safety.The cited examples include perception and geometry methods, SafeAuto’s traffic-rule verification, and risk-detection work.Source: §2.3 · pp. 3–4

2.4 Navigation and Grounded Driving Evaluation

  • Prior navigation and driving benchmarks assess robustness, grounding, or risk, but a correct explanation or discrete action answer does not establish safe execution.SafeDriveVLA evaluates following, navigation-scene conflict, and adversarial instructions through closed-loop CARLA driving and real-world NavSim data.Source: §2.4 · p. 4

3 Benchmark

The benchmark suite separates safe navigation following, scene-dependent conflicts, and instructions that are unsafe in any circumstance. Its components test these capabilities through closed-loop CARLA and NavSim settings.

3 Benchmark

  • The benchmark suite evaluates navigation following, navigation-scene conflict awareness, and adversarial instructions that are unsafe under any circumstance.The three settings are CARLA-F, B2D-C and NavSim-C, and B2D-Adv, respectively.Source: §3 · p. 4

3.1 Navigation Following Benchmark

  • CARLA-F tests closed-loop following across three navigation modalities, using chained natural-language instructions as its primary signal.Its routes begin with a speed instruction and continue with maneuver instructions; background traffic is removed and lights are forced green so every signal should be followed.Source: §3.1 · p. 5
  • CARLA-F reports Navigation Compliance Rate (NCR) per meta-command, weighted by signal count and excluding speed, plus Speed Error (SE) against the instructed speed.Source: §3.1 · p. 5

3.2 Navigation-Scene Conflict Benchmark

  • B2D-C pairs Bench2Drive scenarios with unsafe language instructions triggered when safety-critical events occur, testing whether danger depends on the current scene.It reports Driving Score, Success Rate, and per-infraction collision, traffic-violation, and out-of-route breakdowns.Source: §3.2 · p. 5
  • NavSim-C contains 300 scenarios where a replaced left- or right-turn command would cause a collision or leave the drivable area.It covers intersection turns and lane changes and follows NavSim’s evaluation protocol.Source: §3.2 · p. 5
  • At inference, the navigation signal selects a meta-command action anchor; the frozen world model rolls the current latent forward five steps, and the VLA receives the dreamed latents with the camera frame and navigation signal.For language instructions, the meta-command is parsed by vocabulary matching.Source: §3.2 · p. 6

3.3 Adversarial Instruction Benchmark

  • The adversarial benchmark tests whether a VLA refuses directly harmful instructions that are unsafe under any circumstance.Source: §3 · p. 4, §3.3 · p. 6

4 SafeDriveVLA

SafeDriveVLA separates navigation-conflict reasoning from action generation by predicting a driving mode and conditioning imagined futures on strict execution of the navigation signal. Its pipeline combines mode-labeled expert data, a latent world model, and a VLA that consumes observations, navigation, and dreamed latents.

4 SafeDriveVLA

  • SafeDriveVLA predicts a strict, cautious, or fallback mode before generating actions, decoupling conflict reasoning from action generation.The model also uses navigation-conditioned dreamed latents as input to its VLA backbone.Source: §4 · p. 6

4.3 Vision-Language-Action Model

  • The VLA emits a mode token followed by discrete action tokens, while an auxiliary path head supplies lateral control and the action tokens set longitudinal target speed.The action sequence uses 10 tokens to encode a 2.5 s trajectory at 4 Hz.Source: §4 · p. 6, §4.3 · p. 8

4.1 Driving Mode Attribution

  • Expert frames are labeled strict for immediate execution, cautious for following while yielding to hazards, or fallback when no safe action remains.The corresponding mode token is supervised directly and predicted at inference from observations, navigation, and dreamed latents.Source: §4.1 · pp. 6–7

4.2 Latent World Model

  • The latent world model is pretrained on CARLA to predict future representations from observations and ego actions, using two-frame latents that encode 0.25 s of motion.It uses a frozen V-JEPA 2 ViT-L target encoder and a Transformer predictor trained over a 2.5 s horizon at 2 Hz.Source: §4.2 · p. 7
  • Predicted latents recover 49–78% of the encoder-oracle hazard-detection gap, and ground-truth-action rollouts remain closer to the oracle than zeroed- or mirrored-action rollouts at every horizon.The probing uses 5,095 clips that start hazard-free.Source: §4.2 · p. 7, Table 2 · p. 7
  • Unlike prior driving world models that roll out expert actions, SafeDriveVLA dreams futures under hypothetical actions that strictly execute the navigation signal to expose maneuver feasibility.The resulting latent trajectory informs whether the navigation-implied maneuver is feasible in the current scene.Source: §4.2 · p. 7

4.3 Vision-Language-Action Model

  • One action anchor is precomputed per meta-command, selected from the planner command, a waypoint's accompanying command, or vocabulary-matched language instruction.The world model rolls forward five steps under the selected anchor; lane-change anchors are calibrated.Source: §4.3 · p. 8

5 Experiment

The experiments evaluate navigation following, scene-conflict handling, and adversarial robustness, comparing SafeDriveVLA with SimLingo and other driving VLAs. SafeDriveVLA improves Bench2Drive results, while ablations show complementary effects from mode prediction and navigation-conditioned world-model dreaming.

5.2 Main Results

  • 88.16 DS and 68.64% SR on Bench2Drive; SafeDriveVLA improves over same-data SimLingo by +2.15 DS and +1.89% SR on average across two navigation modalities.Experiments also use CARLA-F, B2D-C, B2D-Adv, and NavSim-C; training uses PDM-Lite expert trajectories and samples waypoint, command, and language navigation signals.Source: §5.1 · pp. 8–9, §5.2 · p. 9

5.3 Navigation Following Benchmark

  • Table 4 compares CARLA-F command-based VLAs under official protocols and evaluates SimLingo and SafeDriveVLA with both commands and free-form language instructions.SimLingo commentary, instruction-following, and safety variants use their respective default mode or prepended tags.Source: §5.3 · p. 9

Comparison with SimLingo. SafeDriveVLA

  • SafeDriveVLA raises average language-instruction NCR to 83.0% and lane-change NCR from 0% to 63.6% left and 72.4% right, while command NCR reaches 78.8%.Command-only ORION has higher average NCR than SafeDriveVLA, but SafeDriveVLA matches or exceeds command-only models on lane changes.Source: Comparison with SimLingo. SafeDriveVLA · p. 9

5.4 Navigation-Scene Conflict Benchmark

  • 85.9 DS and 67.3% SR on B2D-C, with 38 collisions, 9 traffic violations, and 4 out-of-route events, versus 60, 61, and 11 for SimLingo-Safe.On NavSim-C, existing VLAs degrade under conflicting commands; CuriousVLA retains the strongest conflict awareness at 84.4 PDMS, a drop of 6.2.Source: §5.4 · pp. 9–10

5.5 Adversarial Instruction Benchmark

  • On paired B2D-Adv routes, SafeDriveVLA’s DS drops 10.8 points under adversarial instructions, versus 19.7 for SimLingo and 23.3 for the same backbone without mode tokens or world-model inputs.The routes are driven both with and without adversarial instructions, isolating the instruction effect from route difficulty.Source: §5.5 · p. 10

5.6 Ablation Study

  • Scaling data from ×0.2 to ×1.0 improves B2D-C safety but degrades CARLA-F NCR; adding a mode token raises B2D-C DS from 79.2 to 87.3 but CARLA-F NCR only from 55.1% to 59.8%.The mode prediction agrees with rule-derived labels on 90.2% of 600 held-out frames, with lowest accuracy on turn_left (0.770).Source: §5.6 · p. 11, Table 7 · p. 12
  • Navigation-conditioned world-model dreaming improves CARLA-F NCR by 23.2 points, from 59.8% to 83.0%, while retaining safety; ego-history extrapolation lowers performance on all three benchmarks.The authors attribute the following gain to anticipating conflicts by dreaming strict execution of the navigation signal; SafeDriveVLA also reaches 83.9 versus 78.6 DS on B2D-Adv.Source: §5.6 · pp. 11–12
  • Speed error rises with both components, and the authors identify speed-unaware action anchors as a limitation; together, the components put SafeDriveVLA on the Pareto frontier at both data scales.The anchors are defined per meta-command, so speed instructions are dreamed under follow_road without conveying the instructed speed to the world model.Source: §5.6 · p. 12

6 Conclusion

SafeDriveVLA addresses navigation-scene conflict, which existing VLAs often fail to ground in the scene. Its benchmark suite and model improve conflict-aware behavior and closed-loop navigation following.

6 Conclusion

  • Existing VLAs systematically follow navigation signals without grounding them in the scene, exposing navigation-scene conflict as a vulnerability.Source: §6 · p. 12
  • SafeDriveVLA decouples conflict reasoning from action generation through explicit driving-mode attribution and navigation-conditioned world-model dreaming.Source: §6 · p. 12
  • SafeDriveVLA outperforms SimLingo on Bench2Drive while improving closed-loop navigation following and reducing unsafe behavior under conflicting instructions.Source: §6 · p. 12

7 Limitations

The paper identifies limitations in its driving-mode labels, benchmark instructions, world-model conditioning, and the safety implications of dreamed outcomes. Its appendix describes how the CARLA-F instruction-following benchmark is constructed.

7 Limitations

  • Kinematic rules infer driving modes from expert trajectories rather than annotations of driver intent, so the labels may not fully capture mode semantics.Source: §7 · p. 12
  • Benchmark instructions come from fixed paraphrase pools, so they do not cover fully free-form user inputs.Source: §7 · p. 12
  • The world model uses a per-meta-command action anchor that omits speed, increasing speed error; speed-aware conditioning is proposed as future work.Source: §7 · p. 12
  • Dreamed outcomes can inherit world-model errors, so mode predictions do not provide a formal safety guarantee.Source: §7 · p. 12

A.1.1 Language Instruction Following Generation

  • CARLA-F routes reuse Bench2Drive starts but rebuild planner-valid routes, disable surrounding traffic, and set traffic lights green.Source: §A.1.1 · p. 15
  • The benchmark samples feasible maneuvers at high-actionability waypoints, chains them into routes, and hides planner commands and route geometry from the model.Source: §A.1.1 · p. 15

A.1.2 Evaluation Metrics

CARLA-F evaluates language instruction compliance and target-speed error using matched route windows, then aggregates results across instructions. Its statistics describe 210 evaluation routes.

A.1.2 Evaluation Metrics

  • Navigation Compliance Rate (NCR) measures the fraction of instructions that pass compliance checks, while endpoint Speed Error compares estimated speed with the target.Source: §A.1.2 · p. 15
  • Lane-follow and lane-change compliance is judged by the majority road-and-lane pair in each window’s final 20%, penalizing missed and wrong-lane changes.Source: §A.1.2 · pp. 15–16
  • Turn instructions pass when the executed and ground-truth route endpoints differ by less than 15 m; speed error is measured near the target distance.Source: §A.1.2 · pp. 16–17
  • Instructions are scored sequentially; after a failure, remaining lateral instructions on that route are counted as failed because geometry references no longer match.Source: §A.1.2 · p. 17
  • Per-meta-command NCR pools passed and total instructions across routes, and average NCR weights commands by instruction count.Source: §A.1.2 · p. 17

A.1.3 Benchmark Statistics

  • The instruction-following benchmark contains 210 evaluation routes, each beginning with a target-speed instruction, and covers a balanced range of vehicle maneuvers.Source: §A.1.3 · p. 17

A.1.4 Per-Meta-Command Ablation

The NavSim-C results show degraded performance under command-scene conflict, while the CARLA-F ablation reports where the world model helps. The appendix also details conflict benchmarks and their evaluation setup.

A.1.4 Per-Meta-Command Ablation

  • The CARLA-F ablation finds that the world model mainly improves lane-change and go-straight categories.Source: §A.1.4 · p. 17

A.2 B2D-C

  • B2D-C manually labels unsafe language instructions triggered during safety-critical events, including instructions whose danger depends on the scene.Source: §A.2 · pp. 17–18

A.3.1 Benchmark Setup

  • NavSim-C selects 300 scenarios and injects conflicting commands that risk leaving the drivable area, entering oncoming lanes, or colliding.Source: §A.3.1 · p. 18

A.3.2 Evaluation Metrics

  • PDMS combines safety-violation penalty gates with weighted progress and comfort scores; driving-direction compliance is reported diagnostically but excluded.Source: §A.3.2 · pp. 18–19

A.3.3 Benchmark Statistics

A.3.4 Detailed Results

  • NavSim-C results show a significant performance drop under command-scene conflict, suggesting models blindly follow instructions without consulting the scene.Source: §A.3.4 · p. 19, Table 10 · p. 19

A.4.1 Benchmark Details

  • B2D-Adv extends B2D-C with adversarial prompts on 15 safety-critical routes spanning 13 scenario types.Source: §A.4.1 · p. 19

B.1 Training

The appendix describes world-model pre-training and probing, including tests of forecasting quality and action dependence. It also outlines the world-token projector used during VLA training.

B.1 Training

  • The predictor uses two real consecutive frames per tubelet to encode a 0.25 s motion sample, while future ego states are masked during training.Teacher-forced and autoregressive losses train predictions over a 2.5 s horizon; autoregressive depth ramps from 2 to 5 steps.Source: §B.1 · p. 20

B.2 World Token Projector

  • A learnable projector maps the 16×16 latent to one language-model token and is the only world-model component trained during VLA training.The encoder, predictor, and target normalization remain frozen.Source: §B.2 · p. 20

B.3 World Model Probing

  • The predictor improves hazard AUC over the frozen-frame baseline by +0.04 at 0.5 s and +0.09 at 2 s, recovering 49–78% of the oracle gain.Across all horizons, predictor AUC lies strictly between the frozen-frame baseline and encoder oracle.Source: §B.3 · p. 21
  • At every horizon, rollout error is lowest with ground-truth actions, higher with mirrored actions, and highest with zeroed actions, supporting action-conditioned prediction.At 2 s, the error gaps are +0.07 for zeroed actions and +0.02 for mirrored actions relative to ground-truth actions.Source: §B.3 · p. 21

C.1 Training Details

This section details VLA training, rule-derived navigation-compliance labels, and meta-command action anchors for world-model rollouts. It reports mode-prediction accuracy and qualitatively distinct actions across modes.

C.1 Training Details

  • InternVL3-1B is fine-tuned with LoRA, using action and mode cross-entropy, path-point MSE, and text cross-entropy objectives.Training runs for 14 epochs with batch size 12.Source: §C.1 · p. 21

C.2 Mode Attribution Details

  • Mode labels are derived from each command and the expert’s next 2.5 s of waypoints, current speed, and road speed limit.For follow_road, strict requires cruising above 6 m/s with little lateral deviation; stopped or hard-braking cases can be fallback, while other cases are cautious.Source: §C.2 · pp. 21–22

SafeDriveVLA

  • SafeDriveVLA attains 90.2% overall mode accuracy, with turn_left at 0.770 versus turn_right at 0.910.The authors attribute the gap to unprotected left turns, where strict and cautious labels depend on subtle oncoming-traffic and right-of-way cues.Source: SafeDriveVLA · pp. 22–23

C.4 Meta-Command Action Anchor

  • For each meta-command, a five-action ego-frame sequence serves as the world model’s maneuver-conditioned rollout input.Lane-change anchors are calibrated to include 3.5 m of lateral movement at the first 0.5 s step.Source: §C.4 · p. 23, Figure C.1 · p. 23

C.5 Mode-Conditioned Action Generation

  • Different mode tokens produce distinct action strategies: strict makes more progress, cautious proceeds more carefully, and fallback trajectories mostly stay in place.The qualitative study samples 12 trajectories per mode.Source: §C.5 · pp. 23–24, Figure C.2 · p. 24

D Qualitative Results

The qualitative results illustrate instruction following in safe scenes and refusal when an instructed maneuver would cause a collision. The appendix also lists scenario-based instruction sets for evaluation.

D Qualitative Results

  • The NavSim-C visualization notes worse navigation-scene conflict awareness at intersections than on cruising roads and suggests training-data diversity as a possible reason.Orange dots indicate predictions; green dots indicate ground truth.Source: Figure D.1 · p. 26
  • The CARLA-F demonstration shows strict language-instruction following for a left turn and two lane changes when the environment is safe.The captions beneath the subfigures are simplified commands, not the actual language inputs.Source: Figure D.2 · p. 27
  • In the B2D-C demonstration, the model refuses a left turn at 8 m/s after the instructed speed would cause a collision, sacrificing driving score to avoid it.The model initially accelerates and prepares to turn, then falls back to going straight at the fourth frame.Source: Figure D.3 · p. 27

E Public Resources Used

  • The appendix describes route-based instruction sets by scenario and a set of 15 adversarial natural-language instructions grouped by safety-critical scenario.Each scenario in the instruction set has five evaluation routes, with one instruction assigned to each route.Source: Table 12 · p. 28, Table 13 · p. 31

Related papers

Loading 2610.20609v1…