Source-linked AI summary
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, Ankur Handa, Ming-Yu Liu, Donglai Xiang, Gordon Wetzstein, Tsung-Yi Lin
TL;DR
VLAs generally lack explicit intermediate reasoning for complex manipulation because they map observations directly to actions. CoT-VLA generates future subgoal images as visual chain-of-thought steps before producing short action sequences, and experiments report improved performance across simulation and real-world tasks. The approach also enables action-less videos to train visual reasoning, but intermediate image generation creates substantial inference overhead.
Problem
Existing VLAs directly map observations to actions without explicit intermediate reasoning steps for complex manipulation tasks.
Method
CoT-VLA autoregressively generates a future subgoal image as an intermediate visual reasoning step, then conditions a short action sequence on the observation and generated goal.
Results
CoT-VLA improves policy performance over prior VLA approaches in simulation and real-world experiments, with state-of-the-art performance across multiple robot platforms and tasks.
Takeaways & Limitations
Using subgoal images lets CoT-VLA leverage information already present in robot manipulation data and potentially use abundant action-less video for visual reasoning.
Takeaways & Limitations
Generating 256 intermediate image tokens before action tokens causes a 7× average slowdown with an action chunk size of 10.
Abstract
from arXiv · showhide
Vision-language-action models (VLAs) have shown potential in leveraging pretrained vision-language models and diverse robot demonstrations for learning generalizable sensorimotor control. While this paradigm effectively utilizes large-scale data from both robotic and non-robotic sources, current VLAs primarily focus on direct input--output mappings, lacking the intermediate reasoning steps crucial for complex manipulation tasks. As a result, existing VLAs lack temporal planning or reasoning capabilities. In this paper, we introduce a method that incorporates explicit visual chain-of-thought (CoT) reasoning into vision-language-action models (VLAs) by predicting future image frames autoregressively as visual goals before generating a short action sequence to achieve these goals. We introduce CoT-VLA, a state-of-the-art 7B VLA that can understand and generate visual and action tokens. Our experimental results demonstrate that CoT-VLA achieves strong performance, outperforming the state-of-the-art VLA model by 17% in real-world manipulation tasks and 6% in simulation benchmarks. Project website: https://cot-vla.github.io/
1. Introduction
Existing VLAs map observations and language directly to actions, without explicit intermediate reasoning. CoT-VLA instead generates visual subgoals before actions and combines this reasoning with action chunking and hybrid attention.
- Existing VLAs typically map language instructions and visual observations directly to robot actions without explicit intermediate reasoning steps.
- CoT-VLA generates a subgoal image representing a planned future state before conditioning action generation on the observation and visual goal.The subgoal image serves as an intermediate visual reasoning step in pixel space.
- Action-less video data can train subgoal image generation, expanding the data sources available for visual reasoning beyond action-annotated robot demonstrations.
- CoT-VLA combines causal attention for text and image generation with full attention for action prediction, and incorporates action chunking.The model is built on a unified multimodal foundation model and trained on robot demonstrations plus action-less videos.
- Experiments in simulation and the real world report improved policy performance over prior VLA approaches and state-of-the-art results across multiple platforms and tasks.
2. Related Work
Related work spans multimodal chain-of-thought reasoning and vision-language models used in robot perception, planning, and control. CoT-VLA connects these directions by using visual intermediate goals within an end-to-end VLA.
- Multimodal chain-of-thought methods extend sequential reasoning beyond language by iteratively processing visual information.
- Vision-language models have been used for task decomposition, object detection, dense rewards, goal generation, and pretrained representations in robot systems.
- Recent VLA approaches fine-tune pretrained vision-language models on robot demonstrations for direct action prediction.
3. CoT-VLA
CoT-VLA predicts a future subgoal image and then an action sequence, training visual reasoning on both robot demonstrations and action-less videos. Its architecture uses VILA-U with hybrid attention and joint visual-action objectives.
- 3.1. Visual Chain-of-Thought Reasoning: CoT-VLA uses robot demonstrations and action-less videos for pretraining, with language instructions and image sequences as inputs or targets.Robot demonstrations include actions, whereas action-less videos contain language descriptions and images without action annotations.
- 3.1. Visual Chain-of-Thought Reasoning: The model first predicts a subgoal image n frames ahead, then predicts m actions conditioned on the current observation, instruction, and subgoal.
- 3.2. The Base Vision-Language Model: Causal attention handles text and image generation, while full attention predicts action dimensions in parallel.The training objective combines visual-token and action losses.
- 3.2. The Base Vision-Language Model: VILA-U provides autoregressive understanding and generation of image, video, and language tokens through a unified multimodal foundation model.The model uses discrete visual tokens and is pretrained on interleaved multimodal pairs.
- 3.3. Training Procedures: Pretraining uses Open X-Embodiment robot data plus EPIC-KITCHENS and Something-Something V2 videos, with 256 × 256 images and action chunks of size 10.
- 3.3. Training Procedures: LIBERO results report average success rates and standard errors across Spatial, Object, Goal, and Long suites using 3 seeds and 500 episodes per suite.
4. Experiments
Experiments evaluate CoT-VLA across simulation and real-world manipulation settings, comparing it with established baselines and isolating the effects of action chunking, hybrid attention, visual chain-of-thought reasoning, and pretraining. CoT-VLA achieves strong performance across benchmarks, with its highest average performance on Franka-Tabletop and a 46.7% relative pretraining improvement.
- Experimental Setup: Evaluations span LIBERO simulation, Bridge-V2 real-robot manipulation, and Franka-Tabletop adaptation with 10–150 demonstrations per testing scenario.LIBERO contains four suites of 10 tasks with 50 demonstrations per task; Bridge-V2 uses 45k language-annotated trajectories.
- Benchmark Results: CoT-VLA achieves best or competitive performance on LIBERO and comparable or better performance across Bridge-V2’s visual, motion, semantic, and language generalization categories.LIBERO evaluations use 500 trials per task suite with 3 random seeds, while Bridge-V2 categories use 10 trials each.
- Benchmark Results: CoT-VLA achieves the highest average performance on Franka-Tabletop, improving across both single-instruction and multi-instruction scenarios.OpenX-pretrained models adapt better to multi-instruction tasks, while Diffusion Policy performs best on some single-instruction tasks.
- Ablation Study: Action sequence prediction consistently outperforms single-action prediction, hybrid attention further improves performance, and CoT-VLA achieves the best LIBERO ablation results.The ablations evaluate action chunking, hybrid attention, and visual chain-of-thought reasoning on LIBERO-Spatial and LIBERO-Goal.
- Ablation Study: 46.7% relative improvement, from 53.7% to 78.8%, is achieved by CoT-VLA with OpenX and action-less video pretraining over direct fine-tuning of the base VILA-U model.The comparison is conducted on Franka-Tabletop demonstrations and is reported as improved downstream task adaptation.
- Visual Reasoning: CoT-VLA still struggles with out-of-distribution subgoal generation, while scaling large video and image models is identified as a promising direction.This limitation concerns visual reasoning generalization rather than the reported in-distribution benchmark comparisons.
5. Conclusion, Limitations and Future Work
CoT-VLA introduces explicit intermediate visual goals for VLA reasoning and demonstrates strong robotic manipulation performance. Its main limitations are inference overhead, lower visual quality, and discontinuities from action chunking.
- Conclusion: CoT-VLA introduces intermediate visual goals as explicit reasoning steps, using subgoal images rather than abstract representations such as bounding boxes or keypoints.The system is built upon VILA-U and targets diverse robotic manipulation tasks.
- Limitations: Generating 256 image tokens before action tokens causes a 7× average slowdown with an action chunk size of 10.Action chunking and parallel decoding improve speed, but image generation remains the primary bottleneck.
- Limitations: Autoregressive image generation produces lower visual quality than state-of-the-art diffusion-based models.The authors identify faster unified multimodal models as a possible improvement direction.
- Limitations: Action chunking can introduce discontinuous actions between chunks and lacks high-frequency feedback during execution.The authors suggest temporal smoothing and per-step prediction as potential remedies.
6. Implementation Details
The implementation details specify datasets, manually selected subgoal-horizon bounds, hyperparameter tables, fine-tuning settings, and substantial computational requirements.
- Datasets: Open X-Embodiment provides robot demonstration pre-training data, while Something2Something and EPIC-KITCHEN-100 provide action-less video data.The predicted subgoal horizon uses manually set upper and lower bounds for each dataset.
- Hyperparameters: Tables 4 and 5 present dataset weights and hyperparameters for pre-training.These tables summarize implementation settings used in the training pipeline.
- Hyperparameters: The paper separately lists important hyperparameters for the model pre-training and pose-training stages.The implementation section frames these settings as key reproducibility details.
- Fine-tuning: Fine-tuning on LIBERO and Franka-Tabletop uses a constant learning rate of 1e-5 for 150 epochs.The LLM backbone, projector, and depth transformer are fine-tuned.
- Compute: Pre-training uses 12 A100 GPU nodes with 8 GPUs each and consumes 11K A100 GPU hours in total.LIBERO and Franka-Tabletop fine-tuning takes 10–24 hours on a single A100 GPU node, depending on dataset size.