Source-linked AI summary
DualCoT-VLA: Visual-Linguistic Chain of Thought via Parallel Reasoning for Vision-Language-Action Models
Zhide Zhong, Junfeng Li, Junjie He, Haodong Yan, Xin Gong, Guanyi Zhao, Yingjie Cai, Jiantao Gao, Xu Yan, Bingbing Liu, Yingcong Chen, Liuqing Yang, Haoang Li
TL;DR
VLA models and existing CoT approaches struggle to combine fine-grained spatial perception with long-horizon planning while avoiding autoregressive latency and cumulative errors. DualCoT-VLA addresses these gaps with visual and linguistic CoT streams implemented through parallel learnable query tokens, achieving state-of-the-art results on LIBERO and RoboCasa GR1 and in real-world deployments.
Problem
Existing CoT-based VLA models use isolated modalities and step-by-step autoregressive decoding, limiting combined spatial-planning reasoning and causing latency and cumulative errors.
Method
DualCoT-VLA combines visual CoT for low-level spatial understanding and linguistic CoT for high-level planning through parallel learnable query tokens and single-step latent reasoning.
Results
DualCoT-VLA achieves state-of-the-art performance on LIBERO and RoboCasa GR1, with evaluations also conducted on real-world platforms.
Takeaways & Limitations
The dual-stream design provides both low-level 3D spatial perception and high-level logical task planning while bypassing autoregressive reasoning.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models map visual observations and language instructions directly to robotic actions. While effective for simple tasks, standard VLA models often struggle with complex, multi-step tasks requiring logical planning, as well as precise manipulations demanding fine-grained spatial perception. Recent efforts have incorporated Chain-of-Thought (CoT) reasoning to endow VLA models with a ``thinking before acting'' capability. However, current CoT-based VLA models face two critical limitations: 1) an inability to simultaneously capture low-level visual details and high-level logical planning due to their reliance on isolated, single-modal CoT; 2) high inference latency with compounding errors caused by step-by-step autoregressive decoding. To address these limitations, we propose DualCoT-VLA, a visual-linguistic CoT method for VLA models with a parallel reasoning mechanism. To achieve comprehensive multi-modal reasoning, our method integrates a visual CoT for low-level spatial understanding and a linguistic CoT for high-level task planning. Furthermore, to overcome the latency bottleneck, we introduce a parallel CoT mechanism that incorporates two sets of learnable query tokens, shifting autoregressive reasoning to single-step forward reasoning. Extensive experiments demonstrate that our DualCoT-VLA achieves state-of-the-art performance on the LIBERO and RoboCasa GR1 benchmarks, as well as in real-world platforms.
1. Introduction
DualCoT-VLA addresses the limits of single-modal and autoregressive reasoning in VLA models by combining visual and linguistic CoT with parallel reasoning. It is evaluated on simulation benchmarks and real-world platforms, achieving state-of-the-art performance on LIBERO and RoboCasa GR1.
- VLA models struggle with long-horizon logical planning and precise spatial perception, while existing CoT methods typically address only one modality.
- DualCoT-VLA combines visual CoT for low-level spatial cues with linguistic CoT for high-level logical plans.The two streams are designed to provide comprehensive multimodal reasoning.
- Two sets of learnable CoT query tokens enable implicit parallel reasoning in a single forward pass instead of autoregressive decoding.The mechanism avoids the latency and compounding errors associated with step-by-step inference.
- The method is evaluated across simulation benchmarks and real-world platforms.The evaluation covers both LIBERO and RoboCasa GR1, alongside real-world robot experiments.
- DualCoT-VLA achieves state-of-the-art performance on LIBERO and RoboCasa GR1 and transfers task planning and 3D spatial perception to real-world environments.
2. Related Work
Prior VLA reasoning methods separate linguistic and visual CoT, limiting their ability to combine semantic planning with spatial perception, and often decode reasoning autoregressively. DualCoT-VLA uses parallelized CoT query tokens to internalize both forms of reasoning in one forward pass.
- Single-Modal Chain-of-Thought in VLA models: Linguistic CoT supports high-level logic but often lacks physical grounding, whereas Visual CoT captures visual details but struggles with abstract planning.
- Single-Modal Chain-of-Thought in VLA models: Existing single-modal approaches fail to synergize high-level semantic logic with low-level spatial perception.
- Autoregressive Reasoning in VLA models: Autoregressive reasoning generates steps token-by-token, causing severe inference latency and preventing full parallelization.
- Autoregressive Reasoning in VLA models: DualCoT-VLA simultaneously distills 3D spatial priors and internalizes high-level linguistic plans through parallelized CoT query tokens in a single forward pass.The design eliminates token cascading errors and reduces inference latency.
3. Methodology
DualCoT-VLA combines visual and linguistic implicit CoT streams with a parallel query-token mechanism, then conditions a diffusion action head on their unified reasoning context. Visual geometric distillation captures spatial priors, while linguistic supervision internalizes task plans.
- Architecture: DualCoT-VLA comprises a VLM backbone, dual-stream implicit CoT mechanism, and downstream Diffusion Transformer action head.The VLM processes multimodal inputs while the two reasoning paths provide visual and linguistic features for action prediction.
- Parallel Implicit CoT: Two learnable query-token sets let the VLM produce visual and linguistic CoT hidden states simultaneously in one forward pass.Visual queries contain M = 16 tokens, linguistic queries contain N = 4 tokens, and self-attention integrates both with visual observations and language instructions.
- Visual CoT: Visual CoT uses cross-attention to reconstruct frozen Depth Anything 3 dense features, training compact hidden states to encode low-level spatial priors.A geometric teacher supplies dense spatial features, while a mean squared error objective supervises reconstruction from the visual CoT tokens.
- Linguistic CoT: Linguistic CoT projects its hidden states into a frozen auxiliary language model, whose task-planning text reconstruction provides step-level supervision.The projected states act as prefix tokens for the decoder, and cross-entropy training encourages logical planning in N = 4 continuous tokens.
- Action Prediction: The action head uses a Diffusion Transformer with Flow Matching, conditioning vector-field prediction on noisy actions, time, robot state, and the unified VLM context.The VLM hidden states combine spatial grounding and logical planning before being injected into the action head.
4. Experiments
DualCoT-VLA is evaluated on simulation benchmarks, real-world manipulation, qualitative reasoning behavior, inference latency, and component ablations. Across these evaluations, it combines complementary visual and linguistic reasoning with parallel latent-space inference.
- Experimental Setup: Experiments cover LIBERO, RoboCasa GR1, and AgileX Cobot deployments, testing spatial reasoning, long-horizon execution, and high-dimensional action coordination.LIBERO uses a 7-DoF arm, RoboCasa GR1 uses a 29-DoF dexterous hand, and the real-world platform uses onboard RGB cameras.
- Benchmark Results: 98.2% Long, 99.4% Spatial, and 99.8% Object success rates show complementary benefits from linguistic planning and visual geometric reasoning.Linguistic CoT supports long-horizon planning, while Visual CoT extracts geometric information for visually intensive suites.
- Benchmark Results: 55.1% average success rate across 24 RoboCasa GR1 tasks includes 80.0% on CuttingboardToPan and 74.0% on PlacematToPlate.The strongest improvements are reported for spatially constrained dexterous-manipulation tasks.
- Reasoning Analysis: Qualitative probes recover dense 3D geometry from visual query tokens, while linguistic tokens decode scene state, object locations, and next-step plans.The visual probe maps compressed hidden states into depth maps, and an auxiliary decoder translates linguistic hidden states into structured text.
- Ablation Study: Ablations show Visual-only CoT reaches 99.4% on Spatial, Linguistic-only CoT raises Long from 92.0% to 96.0%, and the full model averages 98.8%.These results support complementary roles: linguistic CoT handles long-horizon planning, while visual CoT supplies geometric perception.
5. Conclusion
DualCoT-VLA provides a dual-stream reasoning paradigm for robotic manipulation, combining low-level 3D spatial perception with high-level logical task planning in latent space. Its parallel implicit CoT mechanism uses learnable query tokens to support single-step reasoning and efficient control.
- DualCoT-VLA combines low-level 3D spatial perception with high-level logical task planning for robotic manipulation.
- Parallel implicit CoT reasoning with learnable query tokens shifts reasoning from explicit autoregressive inference to single-step latent-space processing.
- The approach addresses single-modal reasoning isolation while bypassing the latency and compounding errors associated with autoregressive inference.
- Extensive experiments across simulation benchmarks and real-world platforms validate the effectiveness of the approach for generalist robotic control.