Source-linked AI summary
LaST$_{0}$: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Model
Zhuoyang Liu, Jiaming Liu, Hao Chen, Jiale Yu, Ziyu Guo, Chengkai Hou, Chenyang Gu, Xiangju Mi, Renrui Zhang, Kun Wu, Zhengping Che, Jian Tang, Pheng-Ann Heng, Shanghang Zhang
TL;DR
Prior explicit CoT approaches for VLA models face inference-latency and representational challenges in robotic manipulation. LaST0 addresses these challenges with compact spatio-temporal latent reasoning and a dual-system architecture, while ablations show gains from multimodal latent states, token allocation, and temporal coverage.
Problem
Explicit CoT VLA methods face non-negligible inference latency, while linguistic reasoning is limited for capturing physical and robotic dynamics.
Method
LaST0 uses a token-efficient latent CoT that predicts future image, point-cloud, and proprioceptive representations, coordinated with high-frequency action generation through a dual-system architecture.
Results
82% success is achieved with one latent token per modality, while multimodal latent representations and four-step temporal coverage improve latent decision-state performance.
Takeaways & Limitations
LaST0 shifts reasoning into compact continuous latent representations that encode future physical dynamics and support temporally coherent robotic manipulation.
Takeaways & Limitations
Pretraining coverage of complex mobile and dexterous manipulation is limited by the scarcity of public robotic datasets, and complex object interactions remain challenging.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have recently shown strong generalization, with some approaches seeking to explicitly generate linguistic reasoning traces or predict future observations prior to execution. However, explicit reasoning typically incurs non-negligible inference latency, which constrains the temporal resolution required for robotic manipulation. Moreover, such reasoning is confined to the linguistic space, imposing a representational bottleneck that struggles to faithfully capture ineffable physical attributes. To mitigate these limitations, we propose LaST$_0$, a framework that enables efficient reasoning before acting through a Latent Spatio-Temporal Chain-of-Thought (CoT), capturing fine-grained physical and robotic dynamics that are often difficult to verbalize. Specifically, we introduce a token-efficient latent CoT space that models future visual dynamics, 3D structural information, and robot proprioceptive states, and further extends these representations across time to enable temporally consistent implicit reasoning trajectories. Furthermore, LaST$_0$ adopts a dual-system architecture implemented via a Mixture-of-Transformers design, where a reasoning expert conducts low-frequency latent inference and an acting expert generates high-frequency actions conditioned on robotics-oriented latent representations. To facilitate coordination, LaST$_0$ is trained with heterogeneous operation frequencies, enabling adaptive switching during deployment. Across 10 real-world tasks spanning tabletop, mobile, and dexterous hand manipulation, LaST$_0$ improves mean success rates by 13%, 14% and 14% over prior SOTA VLA methods, respectively.
1. Introduction
LaST0 addresses the latency and representational limits of explicit linguistic reasoning in VLA models with compact latent spatio-temporal reasoning and a dual-system architecture. It coordinates low-frequency reasoning with high-frequency action generation for robotic manipulation.
- Explicit CoT VLA methods face non-negligible inference latency and difficulty representing ineffable physical attributes in language.
- LaST0 predicts future visual, geometric, and proprioceptive latent tokens to model physical dynamics and robot-environment interactions.
- The latent CoT extends across future keyframes to support temporally consistent reasoning and coherent closed-loop actions.
- LaST0 is pretrained on diverse robotic manipulation datasets and trained with heterogeneous operating ratios for adaptive execution frequencies.
- A Mixture-of-Transformers dual system separates low-frequency latent reasoning from high-frequency action generation.
2. Related Work
Related work has explored continuous action generation in VLA models and latent reasoning in general VLMs. These directions motivate compact reasoning representations integrated with downstream prediction.
- Vision-Language-Action Model: Recent VLA research uses diffusion- and flow-based generative policy heads to model complex continuous action distributions.
- Latent CoT: Latent CoT methods perform multi-step inference in continuous spaces, making intermediate reasoning compact, implicit, and integrated with downstream prediction.
3. Method
LaST0 combines multimodal future-state latents with a dual-frequency Mixture-of-Transformers policy. Its latent CoT compresses visual, geometric, and proprioceptive dynamics, while asynchronous experts coordinate reasoning and control.
- 3.1. Preliminaries: The VLA task maps language and visual observations to action sequences represented in SE(3)-based robot control spaces.
- 3.2. LaST0 Architecture: LaST0 repurposes a decoder-only transformer into a unified Mixture-of-Transformers architecture that separates slow latent reasoning from fast action execution.
- 3.2. LaST0 Architecture: Current RGB observations provide real-time context, while future RGB frames supply ground-truth visual targets for latent CoT supervision.
- 3.3. Latent Spatio-Temporal Chain-of-Thought: Future RGB frames, point clouds, and robot states form multimodal latent representations organized chronologically across a temporal horizon.
- 3.3. Latent Spatio-Temporal Chain-of-Thought: Average pooling compresses each modality into one token per step, producing a latent sequence of length 3 × H and avoiding pixel- or text-level decoding costs.
- 3.3. Latent Spatio-Temporal Chain-of-Thought: The reasoning expert is trained by continuous latent regression with cosine similarity rather than discrete token likelihoods.
- 3.4. Asynchronous Frequency Coordination: The slow expert reasons at sparse keyframes, while the fast expert acts at every native control timestep using periodically updated latent knowledge.
4. Experiment
Experiments evaluate LaST0 across simulation, ablations, and real-world manipulation, showing strong benchmark performance and benefits from its latent modalities, temporal reasoning, and dual-system coordination.
- Simulation Results: LaST0 achieves an 82% mean success rate across 10 RLBench tasks, surpassing HybridVLA-7B by 8%, π0.5-3B by 17%, and CogACT-7B by 21%.It reaches the highest success rate on 7 of 10 tasks.
- Simulation Results: LaST0 operates at 15.4 Hz without action chunking, exceeding CoT-VLA at 1.1 Hz and remaining competitive with π0.5 at 13.8 Hz.The reported inference speed is measured for the RLBench evaluation.
- Simulation Results: LaST0 reaches a 98.1% mean success rate on LIBERO, including 99.2% on Spatial, 99.6% on Object, and 95.6% on LIBERO-Long.On LIBERO-Long, it outperforms OpenVLA-OFT at 94.5% and π0.5 at 92.4%.
- Ablation Study: Using image, point-cloud, or robot-state latents alone yields 74%, 76%, and 75% success rates, while combining modalities provides further improvement.The ablation supports modeling complementary physical dynamics across modalities.
- Ablation Study: Increasing temporal coverage from 0 to 4 future steps raises success from 68% to 82%, while additional coverage provides no significant improvement.The final configuration uses 4 latent future steps.
- Ablation Study: Mixed collaboration-ratio training achieves 82% success with a 1:4 testing ratio, whereas overly infrequent 1:8 collaboration drops to 74%.Ratios of 1:1, 1:2, and 1:4 achieve comparable performance between 75% and 79%.
- Ablation Study: A single-backbone variant reaches 74% success, while the MoT dual-system design achieves an 8% higher result.The ablation evaluates jointly optimized latent and action streams in the single-backbone configuration.
- Real-World Results: On real-world tasks, LaST0 obtains a 72% ±3 mean success rate on the Franka platform, compared with SpatialVLA at 41% ±2, π0.5 at 59% ±4, and CoT-VLA at 50% ±2.The evaluation excludes the long-horizon task from this Franka mean.
5. Conclusion
LaST0 combines compact spatio-temporal latent reasoning with a fast-slow dual-system architecture for physically grounded, closed-loop robotic manipulation.
- LaST0 shifts reasoning from explicit traces to compact latent representations that capture future semantic, geometric, and proprioceptive dynamics.
- The spatio-temporal latent representation autoregressively models future dynamics across time for temporally coherent reasoning.
- A Mixture-of-Transformers design decouples low-frequency deliberative reasoning from high-frequency action generation.
- LaST0 targets more physically grounded reasoning in robotic foundation models while preserving fine-grained physical information for closed-loop control.
6. Limitations and Future Work
The authors identify limited pretraining coverage, challenging object interactions, and remaining robustness and scalability needs as directions for future work.
- Public robotic-data scarcity limits pretraining coverage of complex mobile and dexterous manipulation.
- Handling complex object interactions remains challenging for LaST0.
- Future work will enforce physical constraints and 3D relational graphs within the Latent CoT.
- The authors plan to explore reinforcement learning for post-training and joint optimization for longer-horizon, complex tasks.
A. Model and Training Details
The paper summarizes LaST0’s model and training configurations in Table 4.
- Table 4 summarizes the model and training configurations for LaST0.
B. Large Scale Pre-training Datasets
LaST0 is pretrained on diverse robotic data and evaluated across varied single-arm, bimanual, mobile, and dexterous manipulation settings.
- Large-scale pretraining: The pretraining corpus contains 400K trajectories and 28M frames from Open-X-Embodiment, DROID, and RoboMIND repositories.
- Robot platforms: The physical deployment uses multimodal robotic platforms, including Franka arms, Agilex mobile manipulation, and TienKung dexterous-hand setups.
- Robot platforms: Agilex tasks use three RGB views and a unified 20-DoF action space combining bimanual arm controls with mobile-chassis velocities.
- Robot platforms: TienKung tasks use one RGB image and a unified 26-DoF action space with separate 13-DoF blocks for the two arms.
- Evaluation: The evaluation covers 10 tasks designed to test deliberative reasoning alongside high-frequency execution across tabletop, mobile, and dexterous manipulation.
- Evaluation: The tasks include spatially precise tabletop manipulation, long-horizon and bimanual coordination, mobile-arm control, and dexterous finger-object interaction.
E. Additional Ablation Study
Additional ablations examine whether latent-modality ordering and temporal sparsity affect LaST0’s manipulation performance. The results assess robustness across modality permutations and latent strides in dense-action settings.
- Modality Ordering: LaST0 remains robust across different permutations of visual, geometric, and state modalities, while the original ordering consistently achieves the best results.
- Latent Stride: Latent-stride experiments evaluate temporal sparsity in latent reasoning while maintaining dense action outputs on LIBERO.
F. Additional Latent Space Analysis
Additional analyses characterize LaST0’s latent representations, action-feature separability, modality contributions, attention patterns, and generated geometric inputs. Together, these analyses examine how latent reasoning represents temporal and physical structure for action generation.
- Latent Dynamics: PCA analysis finds well-separated Image, Point Cloud, and State clusters across RLBench tasks, reflecting inter-token discrimination that captures temporal dynamics.
- Action Features: LaST0 yields an action-feature inter-class distance of 1.34 versus 1.03 without Latent CoT, indicating greater motion-category separation.
- Modality Contributions: Removing any Image, Point Cloud, or Proprioceptive State modality reduces action clustering quality, showing that each contributes distinct information.
- Attention: Compared with No-CoT and Explicit-CoT variants, LaST0’s attention focuses on robot-object interactions rather than irrelevant background textures.
- Geometric Inputs: VGGT-generated point clouds provide spatial geometric information for LaST0 pretraining despite limited point-cloud data and camera extrinsics in pretraining datasets.
- Task Execution: Representative executions show smooth, continuous motions in single-arm, dual-arm, mobile, and dexterous-hand tasks.
H. Failure Case Analysis
Failure cases reveal errors in manipulation height, object placement, and grasp position across tabletop, dual-arm, and dexterous tasks. These examples identify concrete boundaries of reliable execution.
- Tabletop Manipulation: An incorrect wiping height left part of the whiteboard pattern unerased, with limited visual feedback increasing the probability of this failure.
- Dual-Arm Manipulation: During dual-arm stacking, an incorrectly estimated height and position caused plate collision and shifted the first plate.
- Dexterous Manipulation: In dexterous manipulation, insufficient forward movement prevented the hand from grasping the drawer handle and caused a failed pull.
I. Additional Method Details
LaST0 formulates robotic reasoning as a latent intermediate process between observations and actions. Its continuous latent CoT predicts future multimodal dynamics, while cached slow-expert states support efficient fast control.
- Reasoning Formulation: The policy decomposes into reasoning over an intermediate variable Z followed by action prediction conditioned on Z.
- Latent CoT: Unlike explicit language- or image-token CoT, LaST0 represents Z as continuous embeddings that autoregressively predict future 2D images, 3D point clouds, and proprioceptive states.
- Efficient Inference: During inference, cached Key-Value states from the slow expert let the acting expert retrieve latent CoT information without repeatedly invoking slow reasoning.