PaperlayerView exact revision on arXiv

Source-linked AI summary

Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation

Jinghui Lu, Jiayi Guan, Zhijian Huang, Jinlong Li, Guang Li, Lingdong Kong, Yingyan Li, Han Wang, Shaoqing Xu, Yuechen Luo, Fang Li, Chenxu Dang, Junli Wang, Tao Xu, Jing Wu, Jianhua Wu, Xiaoshuai Hao, Wen Zhang, Tianyi Jiang, Lingfeng Zhang, Lei Zhou, Yingbo Tang, Jie Wang, Yinfeng Gao, Xizhou Bu, Haochen Tian, Yihang Qiu, Feiyang Jia, Lin Liu, Yigu Ge, Hanbing Li, Yuannan Shen, Jianwei Cui, Hongwei Xie, Bing Wang, Haiyang Sun, Jingwei Zhao, Jiahui Huang, Pei Liu, Zeyu Zhu, Yuncheng Jiang, Zibin Guo, Chuhong Gong, Hanchao Leng, Kun Ma, Naiyan Wang, Guang Chen, Kuiyuan Yang, Hangjun Ye, Long Chen

arXiv:2604.18486v3cs.CVcs.CLcs.RO

TL;DR

Autoregressive chain-of-thought is too slow for real-time driving, while prior latent methods remain sequential and rely on abstract language representations. OneVL combines compact latent reasoning with language and visual world-model supervision, and across four benchmarks it is the only latent CoT method reported to outperform explicit CoT while matching answer-only inference speed.

  • Problem

    Autoregressive CoT is too slow for real-time driving, while prior latent CoT remains sequential and compresses abstract language rather than scene dynamics.

    Source: §1 · p. 3

  • Method

    OneVL routes reasoning through compact latent tokens supervised by auxiliary language reconstruction and visual future-frame prediction decoders.

    Source: §1 · p. 3

  • Results

    Across NAVSIM, ROADWork, Impromptu, and Alpamayo-R1, OneVL is the only latent CoT method to outperform explicit autoregressive CoT while matching answer-only inference speed.

    Source: §1 · p. 5, §6 · p. 22

  • Takeaways & Limitations

    World-model-grounded latent CoT provides more effective reasoning than verbose token-by-token generation within the reported benchmarks.

    Source: §6 · p. 22

    Training requires roughly 3× memory for three full 4B model instances, imposing nontrivial infrastructure requirements.

    Source: §6 · p. 22

Abstract

from arXiv · show

Chain-of-Thought (CoT) reasoning has become a powerful driver of trajectory prediction in VLA-based autonomous driving, yet its autoregressive nature imposes a latency cost that is prohibitive for real-time deployment. Latent CoT methods attempt to close this gap by compressing reasoning into continuous hidden states, but consistently fall short of their explicit counterparts. We suggest that this is due to purely linguistic latent representations compressing a symbolic abstraction of the world, rather than the causal dynamics that actually govern driving. Thus, we present OneVL (One-step latent reasoning and planning with Vision-Language explanations), a unified VLA and World Model framework that routes reasoning through compact latent tokens supervised by dual auxiliary decoders. Alongside a language decoder that reconstructs text CoT, we introduce a visual world model decoder that predicts future-frame tokens, forcing the latent space to internalize the causal dynamics of road geometry, agent motion, and environmental change. A three-stage training pipeline progressively aligns these latents with trajectory, language, and visual objectives, ensuring stable joint optimization. In inference, the auxiliary decoders are discarded, and all latent tokens are prefilled in a single parallel pass, matching the speed of answer-only prediction. Across four benchmarks, OneVL becomes the first latent CoT method to surpass explicit CoT, delivering superior accuracy at answer-only latency. These results show that with world model supervision, latent CoT produces more generalizable representations than verbose token-by-token reasoning. Code has been open-sourced to the community. Project Page: https://xiaomi-embodied-intelligence.github.io/OneVL

1 Introduction

OneVL addresses the latency and representation limits of explicit and prior latent CoT for driving with compact latents supervised by language and future-frame prediction. It pre-fills latents for single-pass inference and is reported to outperform explicit autoregressive CoT.

Introduction

  • OneVL is the only latent CoT method reported to outperform explicit autoregressive CoT, with superior performance across four benchmarks.The introduction says prior latent CoT methods underperform explicit CoT across all benchmarks.Source: §1 · p. 3
  • Prior latent CoT methods use linguistic latents that compress symbolic scene descriptions rather than spatiotemporal causal dynamics, and still generate hidden states sequentially.Explicit autoregressive CoT also incurs latency proportional to chain length, a problem for safety-critical real-time driving.Source: §1 · p. 3
  • Dual auxiliary decoders reconstruct human-readable CoT and predict future frames, supervising latents to encode semantic intent and causal scene dynamics.Future-frame prediction targets dynamics such as agent trajectories, road geometry evolution, and emerging hazards.Source: §1 · pp. 3–4
  • A three-stage training pipeline progressively aligns the latent bottleneck with trajectory prediction so representations capture causal structure rather than memorized patterns.The reported ablations find gains from both decoders and say the staged training recipe is essential.Source: §1 · p. 5
  • NAVSIM prefill latency matches AR answer-only and is 1.5× faster than explicit CoT; ROADWork prefill is 2.3× faster than its explicit counterpart.For deployment, an MLP trajectory head reduces latency to 0.24s (4.16 Hz), 5.4% of the AR model’s latency.Source: §1 · p. 5
  • The language decoder recovers high-quality CoT text, while the visual decoder produces spatially coherent future-frame previews for linguistic and visual interpretability.At inference, the auxiliary decoders are discarded and latent tokens are prefilled into the prompt.Source: Figure 2 · p. 4, §1 · p. 5

2 Related Work

Related work spans latent chain-of-thought methods, language-grounded driving models, and autonomous-driving world models. OneVL combines compressed latent reasoning with future visual-token prediction during training, then removes the auxiliary decoder for answer-only inference.

2.1 Implicit and Latent Chain-of-Thought

  • Latent-CoT work spans progressive internalization [22], continuous thought tokens [41], compressed summaries [16], sequence distillation [87], mixed tokens [91], and per-step alignment [106].These methods were developed for language-only tasks and do not transfer effectively to VLA-based autonomous driving.Source: §2.1 · p. 5

2.2 VLM and VLA for Autonomous Driving

  • Driving VLM/VLA research develops map and traffic understanding, BEV scene fusion, safety-focused evaluation, and language-reasoned waypoint/action prediction.Examples include MapLM, BEV feature injection [23], corner-case and risk-localization tests [12] [82], and action-linked VLA systems [69] [70] [117], including DriveVLA-W0 [62], AdaThinkDrive [78], LaST-VLA [80], and Alpamayo-R1 [105].Source: §2.2 · pp. 5–6

2.3 World Modeling for Autonomous Driving

  • Unlike prior driving world models, OneVL trains future visual-token prediction alongside compressed latent CoT, then discards that auxiliary for prefilled answer-only inference.Prior driving world models serve data generation, closed-loop evaluation, and representation learning; examples include Cosmos [3], DICC [33], AD-R1 [114], DriveVLA-W0 [62], and DynVLA [86].Source: §2.3 · p. 6

3 Model Architecture

OneVL extends a pretrained vision-language model with visual and language latent tokens, supervised by auxiliary decoders for future-scene prediction and human-readable reasoning. At inference, it discards those decoders and prefills the latent tokens so the model generates only trajectory tokens autoregressively.

3.1 Main Vision-Language Model

  • The backbone is Qwen3-VL-4B-Instruct [5], comprising a ViT vision encoder, MLP visual projector, and LLM initialized from the checkpoint and fully trainable in Stages 0 and 2.It uses cross-entropy next-token prediction for trajectory answers and latent reasoning tokens.Source: §3.1 · p. 6

3.2 Latent Token Design

  • OneVL routes implicit reasoning through visual and language latent tokens whose hidden states train auxiliary decoders to predict future-frame tokens and reconstruct chain-of-thought text.The visual decoder targets frames at 0.5s and 1.0s; the language decoder reconstructs CoT reasoning.Source: §3.2 · p. 6, Figure 3 · p. 7, §3.3 · p. 7
  • OneVL places 4 visual latent tokens before 2 language latent tokens in the response, where they carry future-scene spatial-temporal reasoning and language-grounded implicit reasoning, respectively.The language tokens occupy the position where explicit CoT would appear.Source: §3.2 · p. 7

3.3 Language Auxiliary Decoder

  • The language auxiliary decoder uses language-latent hidden states and current-frame ViT embeddings, mapped into its embedding space, to predict ground-truth CoT text.Its cross-entropy objective encourages scene information in the latent states to be decodable as natural-language reasoning.Source: §3.3 · pp. 7–8

3.4 Visual Auxiliary Decoder

  • The visual auxiliary decoder conditions on current-frame ViT embeddings and visual-latent hidden states to predict anticipated future-frame visual tokens, providing world-model supervision of physical dynamics alongside semantic intent.Training visual token sequences are constructed offline from ground-truth future frames using the IBQ Emu3.5 tokenizer, with 131,072 visual codes and maximum image resolution 512x512 pixels.Source: §3.4 · p. 8

3.5 Combined Training Objective

  • The combined training loss weights the language explanation loss at 1.0 and visual explanation loss at 0.1 alongside the main-model cross-entropy loss.The smaller visual-loss weight reflects the difficulty of visual-token reconstruction and prevents it from dominating training.Source: §3.5 · p. 8

3.6 Prefill Inference

  • At inference, OneVL discards the auxiliary decoders and prefills all latent tokens in parallel, generating trajectory tokens autoregressively at latency nearly identical to answer-only prediction.Language and visual explanations remain available through the auxiliary decoders for post-hoc generation.Source: §3.6 · p. 9

4 Three-Stage Training Pipeline

OneVL first pretrains the visual decoder, then uses three stages to establish meaningful latent representations, align the auxiliary decoders, and jointly fine-tune all components.

4.1 Preliminary: Visual Auxiliary Decoder Self-Supervised Pretraining

  • The visual decoder is first pretrained on current-frame embeddings to predict the next two timestamps, then conditioned on latent tokens for action-conditioned rollouts.Pretraining establishes a visual-dynamics prior before the latent conditioning signal is introduced.Source: §4.1 · pp. 9–10

4.2 Stage 0: Main Model Warmup

  • Stage 0 trains the main VLM end-to-end on trajectory prediction with latent tokens, establishing prediction capability and latent positions that trajectory tokens can attend to.This creates information-routing pathways for later auxiliary-decoder supervision.Source: §4.2 · p. 10

4.3 Stage 1: Auxiliary Decoder Warmup

  • Stage 1 freezes the warmed-up main model while training the language decoder on annotated CoT text and the visual decoder to predict two future frames.The fixed main-model representations give both decoders a consistent semantic distribution to learn from.Source: §4.3 · p. 10

4.4 Stage 2: Joint End-to-End Fine-tuning

  • Skipping the three-stage pipeline degrades performance, while Stage 2 jointly fine-tunes the main model and both auxiliary decoders.Language and visual losses then shape latent representations for trajectory prediction, explanation, and visual prediction.Source: §4.4 · p. 10

5 Experiments

The experiments evaluate OneVL’s benchmark performance and latency, including answer-only-like prefill inference and a faster MLP-head variant. They also examine how auxiliary decoders and staged training support trajectory prediction, explanations, and latent-space alignment.

5.1 Datasets / 5.2 Experimental Setups

  • Experiments use NAVSIM, ROADWork, Impromptu, and Alpamayo-R1, with CoT annotations sourced or constructed for each benchmark.Evaluation reports PDM-score on NAVSIM, ADE/FDE on ROADWork and Alpamayo-R1, and ADE/FDE plus four-second trajectory L2 error on Impromptu; inference latency is also reported. Baselines include AR and latent-CoT methods built on Qwen3-VL-4B-Instruct and prior benchmark-specific methods.Source: §5.1 · p. 11, §5.2 · pp. 11–12

5.3 Main Results

  • OneVL records 88.84 PDM-score on NAVSIM and 12.49 ADE / 28.80 FDE on ROADWork, surpassing the cited prior state-of-the-art results.The paper also reports the same overall pattern on Impromptu and Alpamayo-R1, while noting that OneVL’s Alpamayo-R1 FDE is slightly worse than Cosmos-Reason’s (7.53 vs. 7.42).Source: Table 1 · p. 12, Table 2 · p. 13, Table 3 · p. 13
  • OneVL’s NAVSIM latency is 4.46s, essentially matching AR Answer’s 4.49s, and it is much faster than AR CoT+Answer.On ROADWork, Impromptu, and Alpamayo-R1, OneVL records 4.71s, 4.02s, and 3.23s latency, respectively, comparable to or below answer-only prediction.Source: §5.3 · pp. 13–14
  • Adapted latent CoT baselines generally underperform AR Answer, whereas explicit CoT improves NAVSIM PDM-score by 0.80 over answer-only prediction.The authors attribute OneVL’s advantage to auxiliary decoders that verify latent content and to staged training that progressively aligns the bottleneck.Source: §5.3 · pp. 14–15

5.5 Ablation Study

  • Removing the visual decoder lowers NAVSIM PDM-score by 0.87, removing the language decoder lowers it by 0.31, and omitting staged training drops it by 21.71.Direct joint training also produces an initial gradient norm of 378.22 versus 0.28 with staged training, and a final trajectory prediction loss of 0.186 versus 0.136; the visual decoder overfits and adds noise without the curriculum.Source: §5.5 · pp. 17, 18, 20

5.4 Explanation Quality

  • On the NAVSIM test set, OneVL achieves 71.00 Meta Action Accuracy, 78.26 STS, and 79.13 LLM Judge, outperforming SIM-CoT across all evaluation dimensions while AR CoT+Answer leads every metric.Source: §5.4 · pp. 15–16, Table 6 · p. 20

5.6 Towards Real-World Deployment

  • The MLP-head variant reaches 86.83 PDM-score at 0.24s latency, compared with full OneVL’s 88.84 at 4.46s.The authors report the MLP variant’s latency as 5.4% of the AR model’s and approximately 4.16 Hz end-to-end.Source: §5.6 · p. 20, Table 8 · p. 21

5.7 In-Depth Analysis: Where Does the Benefit Come From?

  • 88.84 vs. 88.29: OneVL’s implicit latent reasoning outperforms explicit AR CoT+Answer despite using fewer reasoning tokens.The analysis proposes compression and world-model grounding as explanations for the difference.Source: §5.7 · p. 21
  • Compact latent tokens filter redundant reasoning, while future-frame supervision makes them encode causal scene dynamics that language-level abstractions can miss.The analysis attributes prior latent CoT failures to lacking visual world-model supervision and, typically, staged training; without visual grounding, representations lose geometric precision for trajectory prediction.Source: §5.7 · pp. 21–22
  • +0.87 PDM-score from the visual auxiliary decoder exceeds +0.31 from the language auxiliary decoder.Future-frame reconstruction at 0.5–1.0 seconds is aligned with trajectory geometry and forces latents to represent scene dynamics, unlike symbolic language supervision alone.Source: §5.7 · p. 21
  • 4.46s for prefill-mode OneVL is essentially indistinguishable from 4.49s for AR Answer on NAVSIM.Prefill processes latent tokens in one parallel pass, with negligible overhead beside the image patches; an MLP head reaches 0.24s at the cost of performance.Source: §5.7 · p. 21
  • 87.97 (OneVL w/o visual decoder, staged) versus 67.13 (no staged training), while full OneVL reaches 88.84, showing staged training and visual supervision are complementary.Removing visual supervision alone accounts for a −0.87 drop; the curriculum stabilizes the latent space for world-model grounding.Source: §5.7 · p. 22

6 Conclusion

OneVL grounds latent reasoning in future-scene dynamics through visual world-model supervision, aiming to compress causal information that language alone may not capture. It combines this training approach with single-pass latent prefill for fast trajectory prediction.

Conclusion

  • OneVL uses a visual decoder to predict future-frame tokens, grounding latent compression in scene dynamics rather than abstract linguistic descriptions.The paper argues that predicting visual evolution requires internalizing agent motion and scene interaction, which language alone may not reliably encode.Source: §6 · p. 22
  • At inference, OneVL discards its decoders and prefills latent tokens for single-pass trajectory generation at answer-only latency.Its three-stage curriculum establishes latent representations, aligns decoders to a stable latent space, then jointly optimizes both sides; skipping it can cause memorization and catastrophic performance loss.Source: §6 · p. 22
  • OneVL achieves state-of-the-art results on NAVSIM, ROADWork, Impromptu, and Alpamayo-R1; it is the only latent CoT method reported to outperform explicit autoregressive CoT.It matches answer-only inference speed; adding a lightweight MLP head reduces latency to 5% of the original while retaining competitive performance.Source: §6 · p. 22
  • Training requires roughly 3× memory for three full 4B model instances, while the latent-token count was chosen empirically and its capacity trade-off remains unstudied.DeepSpeed ZeRO-2 mitigates, but does not remove, the infrastructure burden.Source: §6 · p. 22
  • Future work targets non-autoregressive trajectory decoding, multi-camera future-scene prediction, and driver-facing visual and verbal explanations that could support trust and reinforcement-learning feedback.Trajectory generation remains autoregressive and constrains latency; the paper identifies parallel decoding as a step toward real-time deployment.Source: §6 · p. 23

Appendix · A Appendix · A.1 Data Format Example

A OneVL NAVSIM sample pairs driving-scene inputs with an assistant response containing four visual latent tokens, two language latent tokens, and a trajectory answer.

A.1 Data Format Example

  • Each NAVSIM sample’s assistant response contains four visual latent tokens, two language latent tokens, and a trajectory answer.The example formats the trajectory as a sequence of coordinate triples inside <answer> tags.Source: §A.1 · p. 32
  • The user input includes a front-view scene image, the command MOVE FORWARD, velocity [1.5, 0.0], and acceleration [0.3, 0.0].It also includes an instruction to output reasoning and a predicted trajectory.Source: §A.1 · p. 32
  • In the front-view driving scene, the MOVE FORWARD command is paired with velocity [1.5, 0.0] and acceleration [0.3, 0.0], while the historical trajectory shifts along the first coordinate.Source: §A.1 · p. 32
  • Visual latent tokens are bracketed by <|start-latent-vis|> and <|end-latent-vis|>, followed by language latent tokens bracketed by <|start-latent|> and <|end-latent|>.Source: §A.1 · p. 32
  • The think_steps label for the language decoder describes a clear road, no vehicles or pedestrians requiring braking, and continuing straight at current speed.Source: §A.1 · p. 32
  • The future_image_tokens label for the visual decoder uses image-start and image-end markers around visual-token placeholders.Source: §A.1 · p. 32

A.2 Training Configuration

OneVL uses staged training, beginning with self-supervised visual-decoder pretraining and progressing through latent-token warm-up, auxiliary-decoder training, and joint fine-tuning.

A.2 Training Configuration

  • Visual-decoder pretraining predicts the next frame from ViT features alone, without the autoregressive backbone, for 13 040 optimizer steps at global batch size 256.The pretraining duration is specified in optimizer steps rather than a fixed epoch count.Source: §A.2 · p. 32
  • Stage 0 warms up the main VLM while introducing latent tokens.Source: §A.2 · p. 32
  • In Stage 1, auxiliary decoders train with the main model frozen, and visual latent tokens make the visual decoder action-conditioned.Source: §A.2 · p. 32
  • Stage 2 jointly fine-tunes the full stack with all parts updated together.Source: §A.2 · p. 32

A.3 CoT Annotation Construction

The appendix describes dataset-specific CoT annotation pipelines for work-zone and unstructured driving, plus reproduced Alpamayo-R1 labels and a constructed test set. The annotations cover scene understanding, driving decisions, and planning rationales.

A.3 CoT Annotation Construction

  • ROADWork CoT annotations use an in-house pipeline for work-zone reasoning, unlike NAVSIM annotations sourced from AdaThinkDrive [78].They cover hazards such as cones, barriers, signage, and workers; temporary lane layouts; and work-zone-specific speed and clearance decisions.Source: §A.3 · p. 32
  • Impromptu VLA annotations use a VLM-centric, CoT-prompted pipeline tailored to unstructured driving scenarios.Labels classify four challenging scenario types and describe ambiguous boundaries, temporary facilities or rules, unusual obstacles, adverse conditions, avoidance, speed and clearance decisions, meta-action plans, trajectories, and textual rationales.Source: §A.3 · pp. 32–33
  • Because Alpamayo-R1 CoT labels and its official test set were unavailable, the authors reproduced training labels and sampled 700 available video clips for testing.They used the public checkpoint and subsampled 64 waypoints to 8 while retaining the last points, since 64 were too dense for autoregressive modeling.Source: §A.3 · p. 33

A.4 LLM-as-Judge Evaluation Prompt

The evaluation uses a vision-language judge to score predicted driving CoT against ground truth, prioritizing safety-critical motion and planning decisions over minor perception differences.

A.4 LLM-as-Judge Evaluation Prompt

  • The evaluator compares predicted CoT with ground-truth CoT using the front-camera image as driving context.The paper identifies the judge as gemini-3.1-flash-lite-preview2.Source: §A.4 · p. 34
  • Motion-state mismatches and deviations from the ground-truth ego action receive severe penalties, while small distance or location differences incur only slight penalties.Language fluency is also assessed, with minor grammatical errors penalized.Source: §A.4 · p. 34
  • The judge assigns an integer score from 0 to 100, returned with brief penalty reasoning in a strict JSON format.Raw scores are divided by 100 to normalize them to [0, 1], and temperature is set to 0.1 for scoring stability.Source: §A.4 · p. 34

A.5 Reproducing Impromptu … A.9 Alpamayo-R1 qualitative examples

The appendices explain how the Impromptu comparison result was selected and show baseline-versus-OneVL examples across several datasets. Impromptu examples illustrate CoT-guided trajectory changes around merging vehicles and intersections.

A.5 Reproducing Impromptu

  • The authors select the better-performing of two reproduced Impromptu results as the representative baseline because neither matches the original paper’s claimed performance.The two results come from the author-provided checkpoint and a model trained from scratch using the official script.Source: §A.5 · p. 35

A.6 NAVSIM qualitative examples

  • NAVSIM examples compare baseline answer-only predictions with OneVL, showing decoded future frames and text CoT alongside ground-truth and predicted trajectories.The plots overlay trajectories on front-camera views.Source: §A.6 · p. 35

A.7 Roadwork qualitative examples

  • ROADWork examples compare baseline answer-only predictions with OneVL and overlay ground-truth and predicted trajectories on front-camera views.The examples also show two future frames decoded by the visual auxiliary decoder and text CoT from the language decoder.Source: §A.7 · p. 35

A.8 Impromptu qualitative examples

  • CoT prompts guide OneVL away from merging large vehicles and toward ground-truth trajectories, while a right-turn prompt produces a more centered lane trajectory at a Y-intersection.The examples illustrate how CoT prompts can help in scenarios requiring changes in driving behavior, including interactions with vehicles and crossing intersections.Source: §A.8 · p. 35, Figure 20 · p. 47, Figure 21 · p. 48
  • At a Y-shaped intersection, a right-turn CoT prompt yields a reasonable trajectory that stays more centered in the lane.The ground-truth and model-predicted trajectories are shown in green and red, respectively.Source: §A.8 · p. 35, Figure 21 · p. 48

A.9 Alpamayo-R1 qualitative examples

  • The Alpamayo-R1 appendix presents a qualitative comparison of answer-only baseline and OneVL cases.Figure 22 shows the example on the Alpamayo-R1 dataset.Source: §A.9 · p. 35, Figure 22 · p. 49

Related papers

Loading 2604.18486v3…