PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Reasoning VLA: Latent Thinking and Prediction for Vision-Language-Action Models

Shuanghao Bai, Jing Lyu, Wanqi Zhou, Zhe Li, Dakai Wang, Lei Xing, Xiaoguang Zhao, Pengwei Wang, Zhongyuan Wang, Cheng Chi, Badong Chen, Shanghang Zhang

arXiv:2602.01166v2cs.RO

TL;DR

VLA CoT methods face inference overhead and a mismatch between discrete reasoning representations and continuous robotic perception and control. LaRA-VLA progressively transfers multimodal textual and visual CoT into continuous latent reasoning for action generation, achieving strong performance across simulation and real-robot evaluations while reducing inference latency by up to 90% versus explicit CoT-based approaches.

  • Problem

    Existing VLA CoT methods incur high inference overhead and rely on discrete reasoning representations that mismatch continuous perception and control.

  • Method

    LaRA-VLA uses curriculum-based training to transition from explicit multimodal CoT supervision to continuous latent reasoning coupled directly to action generation.

  • Results

    LaRA-VLA consistently surpasses textual and visual CoT methods across simulation benchmarks and long-horizon real-robot manipulation tasks.

  • Takeaways & Limitations

    Latent multimodal reasoning provides an efficient, action-oriented paradigm for real-time embodied control without explicit CoT generation at inference time.

  • Takeaways & Limitations

    Latent representations may collapse without explicit supervision, so the implementation restricts reasoning to one latent token per step, potentially limiting expressiveness.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models benefit from chain-of-thought (CoT) reasoning, but existing approaches incur high inference overhead and rely on discrete reasoning representations that mismatch continuous perception and control. We propose Latent Reasoning VLA (LaRA-VLA), a unified VLA framework that internalizes multi-modal CoT reasoning into continuous latent representations for embodied action. LaRA-VLA performs unified reasoning and prediction in latent space, eliminating explicit CoT generation at inference time and enabling efficient, action-oriented control. To realize latent embodied reasoning, we introduce a curriculum-based training paradigm that progressively transitions from explicit textual and visual CoT supervision to latent reasoning, and finally adapts latent reasoning dynamics to condition action generation. We construct two structured CoT datasets and evaluate LaRA-VLA on both simulation benchmarks and long-horizon real-robot manipulation tasks. Experimental results show that LaRA-VLA consistently outperforms state-of-the-art VLA methods while reducing inference latency by up to 90\% compared to explicit CoT-based approaches, demonstrating latent reasoning as an effective and efficient paradigm for real-time embodied control. Project Page: https://loveju1y.github.io/Latent-Reasoning-VLA/

1. Introduction

LaRA-VLA addresses the latency and representation limitations of explicit CoT in VLA models by internalizing multimodal reasoning into continuous latent states. Its curriculum-based framework and structured datasets support efficient reasoning and action generation for embodied manipulation.

  • Motivation: Existing VLA CoT methods use explicit textual or visual intermediate representations, including language tokens and discrete visual tokens.Textual CoT covers decomposition and planning, while visual CoT models future observations or intermediate visual states.
  • Motivation: Text-based CoT can reduce control frequency below 5 Hz or around 1 Hz because long reasoning traces increase token length, memory use, and latency.These overheads are problematic for real-time robotic control.
  • Approach: LaRA-VLA performs multimodal reasoning and prediction in continuous latent space, aligning reasoning with continuous perception and control.The framework internalizes both textual and future-oriented visual CoT rather than retaining discrete reasoning traces.
  • Approach: Its curriculum progressively transitions from explicit textual and visual CoT supervision to latent embodied reasoning and action-conditioned latent dynamics.Visual prediction objectives and stabilized visual encoders support the transition.
  • Implications: The framework allocates computation to compact latent thought steps instead of verbose textual reasoning, reducing token expansion and inference latency.This design targets efficient, action-oriented reasoning for real-time robotic control.
  • Evaluation: LaRA-VLA is supported by the LIBERO-LaRA and Bridge-LaRA structured multimodal CoT datasets and evaluations in simulation and long-horizon real-robot tasks.The datasets provide embodied manipulation reasoning annotations.

2. Related Work

Prior VLA research includes textual and visual CoT, while broader latent-reasoning work replaces explicit traces with continuous internal states. LaRA-VLA adapts this direction to embodied control by progressively internalizing both textual and future-oriented visual reasoning.

  • VLA Reasoning: VLA research has pursued architectural advances and training- and inference-time optimizations since RT-2, with textual and visual CoT as major reasoning categories.Textual CoT and visual CoT differ in their intermediate representation forms.
  • Comparison: Unlike Fast-ThinkAct, LaRA-VLA internalizes both textual CoT and future-oriented visual CoT into continuous latent representations.Fast-ThinkAct mainly latentizes textual CoT while retaining visually grounded reasoning as discrete traces.
  • Latent Reasoning: Explicit CoT improves reasoning but can impose verbose outputs, high inference latency, and dependence on discrete tokens.These limitations motivate implicit or continuous reasoning in latent space.
  • Latent Reasoning: Prior latent-reasoning methods use continuous internal states for multi-step computation, with supervised stabilization addressing optimization instability as latent-token counts grow.Related work includes language and vision-language settings.
  • LaRA-VLA: LaRA-VLA extends latent reasoning to VLA by initializing structured reasoning with explicit CoT and progressively replacing it with latent representations.Its visual CoT is aligned with continuous visual representations from the perception backbone.

3. Method

LaRA-VLA builds embodied reasoning through structured CoT data, multimodal latent prediction, curriculum-based latent internalization, and latent-conditioned action generation. Its three-stage training and constrained attention connect visual, textual, and action information while removing explicit CoT at inference.

  • Dataset Construction: LaRA-VLA constructs structured CoT datasets for simulated LIBERO and SimplerEnv environments and applies the framework to long-horizon real-world manipulation.
  • Stage I: Explicit CoT Fine-Tuning: Stage I fine-tunes the VLM with explicit CoT annotations, future visual-latent prediction, and action supervision for embodied manipulation.The model uses visual observations and language instructions to generate CoT tokens, predicts future visual representations, and infers actions through inverse dynamics.
  • Stage I: Explicit CoT Fine-Tuning: Future visual latents are stabilized with an exponential moving average of online visual-encoder parameters to prevent representation collapse.
  • Stage II: Curriculum-based Latent Reasoning: Stage II progressively replaces discrete CoT tokens with continuous latent states, reducing explicit textual supervision until reasoning is internalized in latent space.The curriculum retains textual CoT likelihood and visual-latent prediction objectives while decreasing the proportion of discrete CoT tokens.
  • Stage III: Action Generation: Stage III adapts the multimodal latent context to an action expert through flow matching, generating continuous actions directly without an additional action latent.The context combines current visual and language inputs, intermediate text-based reasoning, and predicted future visual latents.
  • Attention and Objectives: LaRA-VLA constrains attention across text, current-image, future-image, and action tokens differently across training stages to regulate multimodal information flow.Stages I and II use autoregressive action-token attention, while Stage III trains over text and vision tokens without action-token attention.

4. Experiments

LaRA-VLA is evaluated across simulation benchmarks, long-horizon real-world manipulation, ablations, robustness tests, and inference efficiency. It consistently outperforms comparison methods while maintaining stable latent representations and substantially lower inference latency.

  • Simulation Experiments: LaRA-VLA achieves the best overall LIBERO performance, with 97.9% average success, including 99.8% on Object and 96.6% on Long.It also attains the highest SimplerEnv-WidowX average success rate at 68.8%.
  • Simulation Experiments: 68.8% average success on SimplerEnv-WidowX surpasses No-CoT, textual CoT, and visual CoT baselines under real-to-sim evaluation.
  • Real-World Experiments: LaRA-VLA achieves the highest average success rate in real-world evaluation and performs best on three of four long-horizon manipulation tasks.The gains are especially notable for multi-stage reasoning and sustained temporal coordination.
  • Ablation Study: Latent textual CoT yields larger success-rate gains than explicit textual CoT, while adding latent visual CoT produces the best ablation performance.Latent visual CoT supplies predictive future-state information and multimodal alignment.
  • Robustness Analysis: LaRA-VLA maintains higher success rates than Qwen-GR00T across Gaussian blur and noise perturbations at both severity levels.Its latent reasoning tokens remain semantically consistent with only moderate distributional shifts.
  • Inference Efficiency: 135 ms per rollout is the lowest reported inference latency, with up to 90% lower inference time than explicit CoT methods.The efficiency is attributed to fewer latent reasoning tokens, compact visual goal latents, and a lightweight action expert.

5. Limitations

LaRA-VLA retains two limitations: latent representations can collapse without explicit supervision, and the curriculum increases training cost as CoT-related tokens grow.

  • Latent reasoning limitations: Latent CoT representations may collapse toward homogeneous states without explicit supervision, especially as the number of latent tokens increases.The current implementation restricts latent reasoning to one token per step to mitigate this risk, potentially limiting expressiveness.
  • Training limitations: Curriculum learning increases training cost because progressively replacing explicit CoT tokens with latent representations expands the number of CoT-related tokens during training.Improving training efficiency while preserving stable latent reasoning remains an open direction.

6. Conclusion

LaRA-VLA internalizes textual and visual chain-of-thought reasoning into continuous latent representations and uses curriculum training to transfer explicit reasoning into latent space. Results across simulation and long-horizon real-robot tasks indicate strong performance with improved inference efficiency.

  • Conclusion: LaRA-VLA internalizes chain-of-thought reasoning across textual and visual modalities using continuous latent representations.It replaces long explicit inference-time CoT sequences with compact textual CoT latents and uses visual latents as implicit supervisory signals.
  • Conclusion: Experiments on simulated benchmarks and long-horizon real-robot manipulation tasks show strong performance with significantly improved inference efficiency.The conclusion supports realizing structured embodied reasoning in latent space without explicit chain-of-thought generation.

Impact Statement

The paper aims to improve the efficiency and scalability of reasoning in Vision-Language-Action models for robotic manipulation. Its approach is intended to support effective reasoning without explicit chain-of-thought generation and potentially facilitate real-time deployment.

  • Impact Statement: The paper aims to improve the efficiency and scalability of reasoning in Vision-Language-Action models for robotic manipulation.The stated scope includes broader robotic applications such as automation and assistive technologies.
  • Impact Statement: The proposed approach enables effective reasoning without explicit chain-of-thought generation and may facilitate real-time robotic deployment.

A.1. Implementation Details of LaRA-VLA

LaRA-VLA uses a progressive three-stage training setup that moves from explicit supervision to implicit latent reasoning and then removes explicit CoT supervision. Experiments use scenario-specific action horizons and standardized multi-GPU training.

  • Training Paradigm: Stage I combines explicit CoT annotations, next-frame feature-prediction tokens, and tokenized action tokens in the input sequence.The training example includes textual task instructions, a subtask, a bounding box, reasoning, and repeated next-frame prediction tokens.
  • Training Paradigm: Stage II substitutes concrete CoT steps with <thinking> tokens to support implicit reasoning learning.
  • Training Paradigm: Stage III discards explicit CoT supervision after the preceding stages.
  • Training Data: Figure 9 presents examples of the training data formats.
  • Training Configuration: The action horizon is 16 for Bridge, 8 for LIBERO, and 25 for real-world experiments.All models are trained using 8 NVIDIA H100 GPUs.

A.2. Implementation Details Baselines in Real-world Experiments

The real-world baselines include ACT, GR00T N1.5, and an ECoT-style ablation, with implementation settings specified for each comparison.

  • ACT: ACT uses the LeRobot implementation with 50-step action chunks and a ResNet-18 perception backbone.Its transformer has a 4-layer encoder, 1-layer decoder, dimension 512, 8 attention heads, and a 3200-dimensional feedforward network.
  • GR00T N1.5: GR00T N1.5 uses its default architecture with a continuous-action flow-matching head.For real-robot experiments, it uses 25-step action chunks, batch size 128, and trains only the projector and diffusion policy head.
  • ECoT: The ECoT baseline retains explicit textual CoT supervision while removing the action expert, visual CoT, and latent textual CoT components.Other model configurations and training hyperparameters remain consistent with LaRA-VLA for a fair comparison.

B. Details of Data Pipeline

The data pipeline jointly captures subtask structure, target-object location, and manipulator motion to support embodied reasoning supervision.

  • Pipeline goals: The pipeline targets three coupled components: subtask analysis, spatial grounding, and directional motion reasoning.These components respectively support long-horizon reasoning, object localization, and translation of intent into executable control signals.
  • Subtask Annotation: Subtask annotations identify semantic object anchors and segment robot trajectories into atomic manipulation stages.Stages include pre-grasp, grasp, move, and release, using gripper-state changes as temporal cues.
  • Target Object Bounding Boxes: Target-object annotations use GroundingDINO and SAM3 to produce temporally consistent 2D bounding-box trajectories.The Bridge dataset additionally uses GroundingDINO detections from five uniformly sampled frames for multi-frame SAM prompting.
  • Motion Reasoning: Motion annotations compute global motion toward the segment goal and local instantaneous motion from end-effector trajectories.The resulting motion vectors are mapped to directional descriptors and added to the CoT annotations.

C. Analysis of Real-World Experiments

Real-world analysis distinguishes tasks by subtask dependency, showing that sequentially coupled tasks require coordinated reasoning and action execution.

  • Temporal dependency: Two-subtask basket and bowl tasks are largely decoupled, so failures in one subtask need not propagate to the other.Overall success improvements primarily reflect independent gains in individual subtasks.
  • Temporal dependency: Find the Block and Place It in the Basket has strong sequential dependency because placing the block depends on first finding it.Errors in the initial subtask directly limit the achievable success rate of the subsequent subtask.
  • Implications: Sequentially dependent tasks require coherent reasoning and action execution across subtasks rather than isolated subtask-level optimization.The analysis reports consistent LaRA-VLA gains over GR00T N1.5 on this tightly coupled task.

D.1. Additional Analysis

Additional analyses examine action and CoT supervision, inference-time latent reasoning, and EMA-based stabilization of latent representations.

  • CoT supervision: CoT supervision raises average SimplerEnv success from 55.2% to 61.4% even when CoT tokens are removed during inference.This supports a training-time representation effect beyond using intermediate tokens at inference.
  • CoT supervision: Enabling latent reasoning at inference further increases average SimplerEnv success to 68.7%.The result indicates an additional benefit from explicit latent reasoning beyond implicit gains from CoT-supervised training.
  • EMA stability: EMA improves latent stability by providing a slowly evolving target that reduces representation drift during curriculum transition.It is introduced for visual-latent supervision rather than as an independent contribution.
  • EMA stability: Without EMA, bounding-box and motion latent tokens overlap more strongly, whereas EMA produces better-separated latent clusters.The comparison associates EMA with improved semantic separation between reasoning components.
Loading 2602.01166v2…