Source-linked AI summary
Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
Jinghui Lu, Jiayi Guan, Zhijian Huang, Jinlong Li, Guang Li, Lingdong Kong, Yingyan Li, Han Wang, Shaoqing Xu, Yuechen Luo, Fang Li, Chenxu Dang, Junli Wang, Tao Xu, Jing Wu, Jianhua Wu, Xiaoshuai Hao, Wen Zhang, Tianyi Jiang, Lingfeng Zhang, Lei Zhou, Yingbo Tang, Jie Wang, Yinfeng Gao, Xizhou Bu, Haochen Tian, Yihang Qiu, Feiyang Jia, Lin Liu, Yigu Ge, Hanbing Li, Yuannan Shen, Jianwei Cui, Hongwei Xie, Bing Wang, Haiyang Sun, Jingwei Zhao, Jiahui Huang, Pei Liu, Zeyu Zhu, Yuncheng Jiang, Zibin Guo, Chuhong Gong, Hanchao Leng, Kun Ma, Naiyan Wang, Guang Chen, Kuiyuan Yang, Hangjun Ye, Long Chen
TL;DR
Explicit CoT improves driving planning but is too slow for real-time use, while prior latent CoT compresses language without directly representing scene dynamics. OneVL adds visual world-model supervision to compact latent reasoning and prefilled inference, and reports superior performance to explicit CoT across four benchmarks at answer-only latency.
Problem
Autoregressive CoT has prohibitive real-time latency, while language-only latent representations do not directly encode the causal dynamics governing driving.
Method
OneVL jointly supervises compact visual and language latent tokens with future-frame prediction and CoT reconstruction, then pre-fills them for single-pass inference.
Results
OneVL is the only latent CoT method reported to outperform explicit autoregressive CoT across NAVSIM, ROADWork, Impromptu, and Alpamayo-R1 while matching answer-only inference speed.
Takeaways & Limitations
World-model grounding makes compressed latent reasoning more effective than verbose token-by-token generation within the reported benchmarks and latency setting.
Takeaways & Limitations
Training requires roughly 3× memory for three full 4B model instances, and the latent token count was chosen empirically.
Abstract
from arXiv · showhide
Chain-of-Thought (CoT) reasoning has become a powerful driver of trajectory prediction in VLA-based autonomous driving, yet its autoregressive nature imposes a latency cost that is prohibitive for real-time deployment. Latent CoT methods attempt to close this gap by compressing reasoning into continuous hidden states, but consistently fall short of their explicit counterparts. We suggest that this is due to purely linguistic latent representations compressing a symbolic abstraction of the world, rather than the causal dynamics that actually govern driving. Thus, we present OneVL (One-step latent reasoning and planning with Vision-Language explanations), a unified VLA and World Model framework that routes reasoning through compact latent tokens supervised by dual auxiliary decoders. Alongside a language decoder that reconstructs text CoT, we introduce a visual world model decoder that predicts future-frame tokens, forcing the latent space to internalize the causal dynamics of road geometry, agent motion, and environmental change. A three-stage training pipeline progressively aligns these latents with trajectory, language, and visual objectives, ensuring stable joint optimization. In inference, the auxiliary decoders are discarded, and all latent tokens are prefilled in a single parallel pass, matching the speed of answer-only prediction. Across four benchmarks, OneVL becomes the first latent CoT method to surpass explicit CoT, delivering superior accuracy at answer-only latency. These results show that with world model supervision, latent CoT produces more generalizable representations than verbose token-by-token reasoning. Code has been open-sourced to the community. Project Page: https://xiaomi-embodied-intelligence.github.io/OneVL
1 Introduction
OneVL addresses the tension between CoT’s planning benefits and autoregressive latency by grounding compact latent reasoning in both language and future visual dynamics. Its prefilled latent tokens, dual decoders, and staged training target fast inference with interpretable multimodal explanations.
- Motivation: Autoregressive CoT improves trajectory prediction but produces latency proportional to reasoning-chain length, making the gap from answer-only prediction prohibitive for real-time driving.Explicit chains are also described as redundant, motivating compact representations that retain essential causal structure.
- Motivation: Prior latent CoT methods compress language, but driving-language abstractions omit the spatiotemporal causal dynamics that determine future outcomes.Their latent hidden states also remain autoregressively generated, leaving inference sequential.
- OneVL Framework: OneVL uses language and visual auxiliary decoders to reconstruct CoT text and predict future-frame tokens, supervising compact latents with linguistic reasoning and causal scene dynamics.The visual decoder targets agent trajectories, road-geometry evolution, and emerging hazards rather than abstract symbolic summaries.
- Inference and Interpretability: Prefill inference discards the auxiliary decoders and inserts all latent tokens into the prompt for single-pass generation, achieving answer-only-like speed while retaining vision-language explanations.On NAVSIM, latency is 1.5× faster than explicit autoregressive CoT; on ROADWork, it is 2.3× faster than its explicit counterpart.
- Training: A three-stage training pipeline progressively aligns the latent bottleneck with trajectory prediction, while visual and language reconstruction encourage generalizable structure rather than superficial correlations.The two auxiliary objectives provide complementary validation through coherent reasoning text and plausible future frames.
- Results: Across four benchmarks, OneVL is reported as the only latent CoT method outperforming explicit autoregressive CoT, with ablations supporting contributions from both decoders and staged training.The paper also reports state-of-the-art performance across the evaluated benchmarks.
2 Related Work
Related work spans latent reasoning, richer language-grounded driving supervision, trajectory-producing VLAs, and world models. OneVL differs by combining compressed latent CoT with a training-only short-horizon visual world-model auxiliary inside one VLA.
- Latent Reasoning: Latent reasoning methods replace explicit reasoning tokens with continuous representations, but the cited work was developed for language-only tasks and does not transfer effectively to VLA driving.Examples include curriculum-based latent thought, self-distillation, and text-decoding auxiliary supervision.
- Driving Supervision: Language-grounded driving research adds benchmarks for map and traffic-scene understanding, bird’s-eye-view context, corner-case evaluation, and risk localization.These efforts broaden evaluation and supervision signals for multimodal driving reasoning.
- Trajectory Prediction: Trajectory-focused VLA work pairs language reasoning with waypoint or action outputs, while related systems use adaptive CoT, large-scale driving data, long-tail action prediction, or world modeling.OneVL builds on these foundations to address explicit-CoT latency through dual-modal latent supervision.
- World Models: World-model research includes model-based reinforcement learning, spatial reasoning, data generation, closed-loop evaluation, and physically faithful driving simulations.The cited approaches often emphasize data generation, simulators, or separate representation stacks.
- OneVL’s Position: OneVL uses short-horizon future visual-token prediction as a training-only world-model auxiliary paired with compressed latent CoT in a single VLA.The auxiliary guides the bottleneck toward causal scene dynamics and is discarded at inference, enabling prefilled latents with answer-only latency.
3 Model Architecture
OneVL augments a pretrained VLM with visual and language latent tokens whose hidden states support trajectory prediction and multimodal explanations. Auxiliary decoders supervise future-frame prediction and CoT reconstruction during training, while inference retains only parallel-prefilled latent tokens and trajectory generation.
- Backbone and latent interface: OneVL uses a pretrained VLM with a compact latent-token interface for trajectory prediction and multimodal explanation.The backbone processes interleaved image and text inputs, while latent tokens carry implicit reasoning.
- Visual auxiliary decoder: The visual auxiliary decoder predicts future-frame visual tokens at +0.5s and +1.0s from current visual context and visual latent states.The visual tokenizer represents images as discrete token sequences for future-frame prediction.
- Latent token design: Visual latent tokens encode future-scene spatial and temporal reasoning, while language latent tokens occupy the position of explicit CoT before the trajectory answer.The design uses fixed-length visual and language latent sequences, with the visual sequence preceding the language sequence.
- Language auxiliary decoder: The language auxiliary decoder reconstructs human-readable CoT reasoning from language latent hidden states conditioned on current-frame visual embeddings.An MLP maps the latent and visual branches into the decoder space before CoT prediction.
- Training objective: The total training loss combines main trajectory cross-entropy with language and visual auxiliary losses, weighted by λl = 1.0 and λv = 0.1.The smaller visual-loss weight reflects the greater difficulty of visual token reconstruction and limits its dominance during optimization.
- Inference: At inference, auxiliary decoders are discarded, latent tokens are prefilled in parallel, and the model autoregressively generates only trajectory tokens.This yields latency nearly identical to answer-only autoregressive prediction while retaining implicit reasoning activation.
4 Three-Stage Training Pipeline
OneVL uses staged optimization to align the main VLM and its auxiliary decoders before joint training. The pipeline first establishes future-frame and trajectory capabilities, then aligns fixed latent representations with language and visual supervision, and finally fine-tunes all components together.
- Pipeline motivation: The pipeline addresses joint optimization across the main VLM, language decoder, and visual decoder, which begin with different objectives and alignment states.A preliminary self-supervised step precedes three main training stages.
- Preliminary visual pretraining: Preliminary visual-decoder pretraining learns future-frame generation from current-frame ViT embeddings before latent conditioning is introduced.This supplies an unconditional visual prior before the decoder receives latent information.
- Action-conditioned world model: After pretraining, adding visual latent tokens converts the decoder from unconditioned next-frame generation into action-conditioned future-scene generation.The latent tokens provide an additional conditioning signal alongside ViT embeddings and encode the planned driving action.
- Stage 0: main-model warmup: Stage 0 trains the main VLM end-to-end on trajectory prediction so latent positions acquire useful representations and routing pathways.Trajectory tokens can attend to latent positions, enabling later auxiliary-decoder supervision.
- Stage 1: decoder alignment: Stage 1 freezes the main model while language and visual auxiliary decoders learn to decode CoT text and predict two future frames.Freezing maintains a consistent latent-feature distribution during decoder alignment.
- Stage 2: joint optimization: Stage 2 jointly fine-tunes all components with the combined loss, allowing auxiliary gradients to shape latents for trajectory, language, and visual objectives.Ablations report degraded performance when three-stage training is skipped.
5 Experiments
OneVL delivers stronger trajectory prediction than prior baselines while retaining answer-only-like latency. Its gains come from dual visual-language supervision and staged optimization, which stabilize and enrich the latent bottleneck.
- Main Results: 88.84 PDM-score makes OneVL best on NAVSIM, surpassing AdaThinkDrive by +2.64 and LaST-VLA by +1.54.
- Inference Efficiency: 4.46s NAVSIM latency matches answer-only prediction at 4.49s, while OneVL remains faster than autoregressive CoT across the evaluated benchmarks.Reported latencies on ROADWork, Impromptu, and Alpamayo-R1 are 4.71s, 4.02s, and 3.23s, respectively.
- Main Results: OneVL is the only latent CoT method reported to outperform explicit CoT, whereas adapted linguistic latent CoT methods generally underperform answer-only AR prediction.
- Ablation Study: Dual auxiliary decoders improve the full model: removing visual supervision lowers NAVSIM from 88.84 to 87.97, while removing language supervision lowers it to 88.53.The visual decoder contributes +0.87 score and the language decoder contributes +0.31.
- Ablation Study: Direct joint fine-tuning drops PDM-Score by 21.71 points, from 88.84 to 67.13, while staged training keeps gradient norms stable during optimization.The reported gradient norm is 378.22 for direct training versus 0.28 with the staged strategy.
6 Conclusion
OneVL argues that world-model-supervised latent compression improves generalization while prefilled latent tokens reduce reasoning overhead. The paper also identifies remaining training-memory and trajectory-decoding constraints.
- Core contribution: OneVL compresses scene dynamics through a visual decoder that predicts future-frame tokens, complementing language-based reasoning supervision.The target is intended to capture agent motion and scene interaction rather than only abstract linguistic descriptions.
- Training: A three-stage training pipeline progressively aligns the latent bottleneck before joint optimization, while skipping the curriculum causes compression to collapse into memorization.The stages establish meaningful latents, align auxiliary decoders, and tighten the bottleneck through bidirectional optimization.
- Inference: Prefilled latent tokens enable single-pass trajectory generation with auxiliary decoders discarded at inference.This removes iterative latent-token generation overhead, although trajectory tokens remain autoregressive.
- Results: OneVL achieves state-of-the-art results on NAVSIM, ROADWork, Impromptu, and Alpamayo-R1 while matching answer-only inference speed.It is reported as the only latent CoT method to outperform explicit autoregressive CoT.
- Limitations: Training requires roughly 3× memory, and the empirically chosen latent-token count leaves its trade-off with representation capacity for future study.DeepSpeed ZeRO-2 mitigates but does not remove the infrastructure requirements.
- Deployment boundary: A lightweight MLP regression head retains competitive performance at only 5% of the original latency, but overall latency remains constrained by autoregressive decoding.Parallel or non-autoregressive trajectory decoding is identified as a future step toward real-time deployment.
7 Contributions and Acknowledgments
The supplied contribution and acknowledgment passages list the paper’s authors. They contain no substantive contribution or acknowledgment details beyond the author lists.
- Authors: The section lists Jinghui Lu, Jiayi Guan, Zhijian Huang, Jinlong Li, Guang Li, Lingdong Kong, and additional coauthors.
- Authors: The section also lists Lingfeng Zhang, Lei Zhou, Yingbo Tang, Jie Wang, Yinfeng Gao, and additional coauthors.
A.1 Data Format Example
The NAVSIM data-format example shows an image-based driving input paired with latent visual and language tokens, a trajectory answer, and auxiliary decoder labels.
- Sample structure: Each NAVSIM sample contains four visual latent tokens, two language latent tokens, and a trajectory answer.
- Input: The user input includes a front-view image, a MOVE FORWARD command, velocity and acceleration, historical trajectory, and an instruction for reasoning and trajectory prediction.
- Output: The assistant output places visual latent tokens before language latent tokens and the predicted trajectory answer.
- Language supervision: The language-decoder label is a driving explanation describing a clear road and continuing straight without braking.
- Visual supervision: The visual-decoder label is represented as future image tokens enclosed by image boundary markers.
A.2 Training Configuration
Training begins by pretraining the visual auxiliary decoder, then progressively warms up the VLM, aligns auxiliary decoders, and jointly fine-tunes the system.
- Pretraining: The visual auxiliary decoder is pretrained to predict the next frame from ViT features alone for 13 040 optimizer steps with global batch size 256.
- Stage 0: Stage 0 warms up the main VLM while introducing latent tokens.
- Stage 1: Stage 1 trains auxiliary decoders with the main model frozen and makes visual prediction action-conditioned through visual latent tokens.
- Stage 2: Stage 2 performs joint end-to-end fine-tuning of the complete system.
A.3 CoT Annotation Construction
ROADWork and Impromptu annotations extend CoT supervision with domain-specific reasoning about hazards, scene structure, decisions, and challenging driving conditions. Alpamayo-R1 labels and waypoints are reproduced or subsampled to support evaluation and autoregressive modeling.
- ROADWork: Work-zone labels identify hazards, interpret non-standard lanes, and justify speed and clearance decisions.
- Impromptu: Impromptu annotations classify unstructured scenarios, describe complex scene elements, and reason about non-standard driving behavior.
- ROADWork: ROADWork annotations cover planning-oriented meta-actions, trajectories, and textual rationales for driving maneuvers.
- Alpamayo-R1: Alpamayo-R1 CoT labels are reproduced with a released checkpoint, while waypoint sequences are heuristically reduced from 64 to 8 points.
- Examples: The supplied examples include lane-keeping, work-zone stopping, and intersection-turn decisions grounded in scene-specific factors.
A.4 LLM-as-Judge Evaluation Prompt
The evaluation appendix defines an LLM-based CoT judge and presents qualitative comparisons across four driving benchmarks. It emphasizes safety-relevant motion and planning decisions, trajectory overlays, and differences between OneVL and answer-only baselines.
- A.4 LLM-as-Judge Evaluation Prompt: The judge scores predicted CoT from 0 to 100 using image context, ground-truth CoT, and predicted CoT.
- A.4 LLM-as-Judge Evaluation Prompt: Motion-state mismatches receive severe penalties because they are treated as safety hazards, while minor distance differences incur slight penalties.
- A.4 LLM-as-Judge Evaluation Prompt: Judge scores are normalized to [0, 1] by dividing by 100, with temperature set to 0.1 for scoring stability.
- A.5 Reproducing Impromptu: Baseline reproduction on Impromptu follows official training and checkpoint settings, with results reported in Table 10 alongside trajectory L2 error.
- A.6 NAVSIM qualitative examples: NAVSIM and ROADWork qualitative figures compare answer-only baselines with OneVL, decoded future frames, text CoT, and ground-truth versus predicted trajectories.
- A.8 Impromptu qualitative examples: On Impromptu, the qualitative comparison states that CoT prompting helps OneVL avoid large vehicles merging into traffic and stay closer to ground truth.
- A.9 Alpamayo-R1 qualitative examples: Additional qualitative examples are provided for NAVSIM, ROADWork, Impromptu, and Alpamayo-R1, with green ground-truth and red predicted trajectories where specified.