Source-linked AI summary
ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, Feifei Feng
TL;DR
Existing VLA approaches struggle to combine embodied control with multimodal understanding because robot training can cause spurious forgetting and joint training can create task interference. ChatVLA addresses these issues with Phased Alignment Training and a Mixture-of-Experts architecture, achieving strong multimodal understanding and robot control across benchmark and real-world tasks. The framework provides a unified model for conversational ability, multimodal understanding, and physical interaction.
Problem
Existing VLA methods often compromise multimodal understanding or embodied control, while unified models must learn both distributions effectively.
Method
ChatVLA first trains embodied control before incrementally integrating multimodal data and uses Mixture-of-Experts layers to isolate task-specific processing while sharing attention.
Results
ChatVLA achieves competitive VQA and multimodal understanding performance and outperforms OpenVLA on 25 real-world robot manipulation tasks, including 6x higher MMMU performance and 47.2 on MMStar versus ECoT.
Takeaways & Limitations
A single parameter-efficient framework can support conversational ability, general multimodal understanding, and physical robot control within the reported evaluation scope.
Takeaways & Limitations
The unified objective assumes control and multimodal understanding can share beneficial representations while still requiring separable task-specific learning.
Abstract
from arXiv · showhide
Humans possess a unified cognitive ability to perceive, comprehend, and interact with the physical world. Why can't large language models replicate this holistic understanding? Through a systematic analysis of existing training paradigms in vision-language-action models (VLA), we identify two key challenges: spurious forgetting, where robot training overwrites crucial visual-text alignments, and task interference, where competing control and understanding tasks degrade performance when trained jointly. To overcome these limitations, we propose ChatVLA, a novel framework featuring Phased Alignment Training, which incrementally integrates multimodal data after initial control mastery, and a Mixture-of-Experts architecture to minimize task interference. ChatVLA demonstrates competitive performance on visual question-answering datasets and significantly surpasses state-of-the-art vision-language-action (VLA) methods on multimodal understanding benchmarks. Notably, it achieves a six times higher performance on MMMU and scores 47.2% on MMStar with a more parameter-efficient design than ECoT. Furthermore, ChatVLA demonstrates superior performance on 25 real-world robot manipulation tasks compared to existing VLA methods like OpenVLA. Our findings highlight the potential of our unified framework for achieving both robust multimodal understanding and effective robot control.
1. Introduction
VLA models face a central trade-off: robot-focused training supports physical interaction but weakens multimodal understanding, while vision-language models understand images and text but cannot act physically. ChatVLA addresses this challenge with phased training and task-separating architecture, achieving strong understanding and control results.
- Robot-focused VLA training excels at low-level manipulation but often struggles with image-text interpretation and multimodal reasoning.
- Vision-language models provide multimodal scene understanding but lack physical interaction, motivating a unified embodied system.
- The work analyzes existing VLA training paradigms, including robot-only training, reasoning-augmented robot data, and co-training settings.
- ChatVLA combines Phased Alignment Training with Mixture-of-Experts layers to unify understanding and manipulation while reducing task interference.The model first develops embodied control, then incrementally integrates multimodal data; shared attention supports transfer while task-specific MLPs limit interference.
- ChatVLA achieves competitive TextVQA and DocVQA performance, strong general multimodal understanding, and superior control across 25 real-world robot tasks.
- 6x higher MMMU performance and MMStar improvement from 0 to 47.2 are achieved versus ECoT, using 3.5x fewer VLM-backbone parameters.
2. Related Work
Related work spans multimodal large language models for cross-modal understanding and vision-language-action models that adapt pretrained vision-language backbones for robot action prediction.
- Multimodal understanding: Multimodal large language models integrate visual and linguistic information for scene comprehension, VQA, image captioning, and spatial reasoning.
- Vision-language-action models in robot learning: Vision-language-action models typically fine-tune pretrained vision-language models to predict robot actions from visual and linguistic inputs.
3. Methodology
This section analyzes why existing VLA training struggles to unify multimodal understanding with robot control and presents ChatVLA’s phased-training and expert-routing strategy. The framework separates task-specific processing while preserving shared representations, yielding simultaneous gains across understanding, conversation, and control.
- Training paradigms: The study compares robot-only training, reasoning-augmented robot training, and co-training on visual-text and robot data across understanding and real-world control evaluations.
- Empirical analysis: Robot-only training reduces performance to 0 across the reported understanding benchmarks, while visual-text-plus-robot training produces the smallest drop relative to the Qwen2-VL base model.
- Empirical analysis: Robot training can cause spurious forgetting by misaligning previously acquired conversational and understanding capabilities; fixed reasoning templates can reactivate visual-text alignment.
- Empirical analysis: Adding visual-text pairs during co-training significantly reduces real-world task success, indicating partial task interference between action-generation and understanding representations.
- ChatVLA: ChatVLA first trains embodied control with reasoning data, then incrementally integrates multimodal data, while its Mixture-of-Experts layers route robot and visual-text processing separately.
- ChatVLA: Shared self-attention layers enable knowledge transfer because robot control and understanding use overlapping semantic representations for scene and object reasoning.
4. Experiment
ChatVLA is evaluated across multimodal understanding, visual question answering, and real-world robot manipulation. It achieves competitive understanding performance and consistently strong control across diverse long-horizon and multi-task settings.
- Evaluation setup: ChatVLA is evaluated on seven VQA benchmarks, six multimodal understanding benchmarks, and 25 real-world manipulation tasks.The robot evaluation comprises 528 trials across the 25 tasks.
- Multimodal understanding and VQA: 71.2 on TextVQA surpasses ECoT and DiVLA by relative improvements of 9.2x and 9.5x, respectively.On MMStar, ChatVLA scores 37.4, corresponding to 2.2x and 6.9x improvements over DiVLA and ECoT.
- Real robot tasks: ChatVLA outperforms OpenVLA and Octo across all high-level-policy task configurations.These tasks vary object types, manipulation skills, and shelf heights, testing compositional reasoning and interference adaptation.
- Real robot tasks: ChatVLA achieves superior performance to Octo and OpenVLA across bathroom, kitchen, and tabletop multi-task categories.The evaluation combines picking, placing, pushing, and hanging skills across diverse real-world environments.
- Ablation study: Co-training with LLaVA-1.5 visual-text data produces comparable VQA and MLLM results to Qwen2-VL.ChatVLA performs slightly worse than Qwen2-VL in art, medicine, and social science, with gaps linked to missing expert knowledge in the visual-text subdatasets.
- Ablation study: A 1:3 visual-text-to-robot-data ratio is evaluated alongside 1:1 and 3:1 under equal training steps.The study reports that the smaller amount of visual-text data achieved better performance and may suffice to reactivate visual-text alignment.
5. Conclusion
ChatVLA addresses the difficulty of jointly achieving multimodal understanding and embodied control in VLA models. Its phased training and MoE design achieve competitive understanding while excelling at real-world control with fewer parameters than OpenVLA and ECoT.
- Conclusion: ChatVLA combines Phased Alignment Training with a Mixture-of-Experts architecture to unify conversation, multimodal understanding, and robot control.Phased training prioritizes embodied control before multimodal linking, while MoE supports the two tasks.
- Conclusion: ChatVLA outperforms OpenVLA and ECoT on 25 real-world robot tasks with 3.5x fewer parameters.The framework also achieves competitive VQA and general multimodal understanding performance.
6. Appendix
The appendix describes ChatVLA’s training setup, datasets, evaluation metrics, and diverse real-world manipulation scenarios, including long-horizon and cross-skill tasks.
- Implementation Details: The visual-text co-training data comes from LLaVA, with a 1:3 visual-text-to-robot data ratio.The visual-text data was randomly sampled from the LLaVA fine-tuning dataset.
- Implementation Details: ChatVLA uses phased alignment training: robot data trains control first, followed by co-training visual-text and robot data.The control expert and action head are activated in stage one; both experts are trained in stage two with learning rate 2e-5.
- Evaluation Metrics: Long-horizon performance awards one point per successfully completed step and reports average successful-sequence length as “Avg. Len.”For cross-skill multi-tasking, “Avg.” is the average success rate across all tasks.
- Robot Tasks: Long-horizon evaluations include direct language prompting and high-level policy planning in tabletop toy scenarios.Example tasks include sorting toys, stacking cubes, moving blocks to baskets, opening drawers, and placing toys into drawers.
- Robot Tasks: Cross-skill multi-tasking evaluates bathroom, kitchen, and tabletop manipulation tasks requiring varied object interactions and skills.The tasks include placing soap, hanging cups, moving bread and bananas, stacking cubes, and transferring objects between containers.