Source-linked AI summary
ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledge
Zhongyi Zhou, Yichen Zhu, Junjie Wen, Chaomin Shen, Yi Xu
TL;DR
Existing VLA fine-tuning can lose key pretrained VLM capabilities, motivating models that preserve and use those capabilities for generalizable robotic control. ChatVLA-2 combines dynamic mixture-of-experts, reasoning-following enhancement, and two-stage training, and demonstrates open-world reasoning on novel math and spatial tasks. The approach supports end-to-end robotic generalization, although pretrained knowledge is not fully retained and experiments are mainly tabletop-based.
Problem
Existing VLA fine-tuning can lose key pretrained VLM capabilities, motivating models that preserve and use those capabilities for generalizable robotic control.
Method
ChatVLA-2 combines dynamic mixture-of-experts, reasoning-following enhancement, and two-stage training to connect pretrained multimodal knowledge with robotic actions.
Results
ChatVLA-2 demonstrates open-world reasoning on out-of-distribution math matching and spatial toy-placement tasks involving novel equations, instructions, or objects.
Takeaways & Limitations
The approach supports end-to-end robotic generalization across diverse tasks that traditionally require explicit planning by an external agent.
Takeaways & Limitations
The method does not fully retain pretrained VLM knowledge and is currently focused mainly on tabletop tasks.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models have emerged as the next generation of models in robotics. However, despite leveraging powerful pre-trained Vision-Language Models (VLMs), existing end-to-end VLA systems often lose key capabilities during fine-tuning as the model adapts to specific robotic tasks. We argue that a generalizable VLA model should retain and expand upon the VLM's core competencies: 1) Open-world embodied reasoning - the VLA should inherit the knowledge from VLM, i.e., recognize anything that the VLM can recognize, be capable of solving math problems, and possess visual-spatial intelligence, 2) Reasoning following - effectively translating the open-world reasoning into actionable steps for the robot. In this work, we introduce ChatVLA-2, a novel mixture-of-expert VLA model coupled with a specialized two-stage training pipeline designed to preserve the VLM's original strengths while enabling actionable reasoning. To validate our approach, we design a math-matching task wherein a robot interprets math problems written on a whiteboard and picks corresponding number cards from a table to solve equations. Remarkably, our method exhibits exceptional mathematical reasoning and OCR capabilities, despite these abilities not being explicitly trained within the VLA. Furthermore, we demonstrate that the VLA possesses strong spatial reasoning skills, enabling it to interpret novel directional instructions involving previously unseen objects. Overall, our method showcases reasoning and comprehension abilities that significantly surpass state-of-the-art imitation learning methods such as OpenVLA, DexVLA, and pi-zero. This work represents a substantial advancement toward developing truly generalizable robotic foundation models endowed with robust reasoning capacities.
1 Midea Group 2 East China Normal University
ChatVLA-2 is presented as the proposed model, with the paper listing Midea Group and East China Normal University affiliations.
- ChatVLA-2 is the proposed model introduced in the paper.
- The paper identifies Midea Group as an affiliated institution.
1 Introduction
The introduction argues that VLA models can lose pretrained VLM capabilities during robotic fine-tuning, motivating ChatVLA-2’s focus on preserving open-world reasoning and connecting it to action.
- Fine-tuning VLA models for robotic tasks can erode pretrained multimodal comprehension and reasoning abilities.
- ChatVLA-2 targets VLA models that retain VLM knowledge while generalizing across diverse robotic tasks.
- Its dynamic MoE architecture separates multimodal understanding from robotic action while preserving shared representations.
- A reasoning-enhancement module aligns action outputs with the model’s internal reasoning process.
- The evaluation uses out-of-distribution math matching and spatial toy-placement tasks involving novel equations, instructions, or objects.
2 Related Work
Related work situates ChatVLA-2 within VLA research using pretrained VLM backbones and embodied reasoning methods, while its architecture addresses reasoning-action alignment and feature competition.
- VLA research commonly uses pretrained VLMs as backbones for language comprehension and observational understanding in robot control.
- ChatVLA-2’s architecture combines a reasoning-following module with dynamic MoE feature disentanglement.
- Embodied reasoning work incorporates chain-of-thought methods to support more complex, context-aware robotic actions.
3 Methodology
ChatVLA-2 combines a VLM-based architecture, dynamic experts, reasoning-conditioned action generation, and two-stage training to preserve knowledge while improving open-world robotic behavior.
- Architecture: ChatVLA-2 adopts DexVLA with Qwen2-VL as its core VLM and leverages textual reasoning outputs.
- Dynamic mixture-of-expert: Dynamic MoE addresses competition between multimodal understanding and robotic manipulation within dense VLA parameter spaces.
- Dynamic mixture-of-expert: Dynamic experts preserve the original VLM architecture while selectively enabling task-specific adaptation.
- Reasoning following enhancement module: The reasoning-following module projects reasoning tokens through an MLP and uses them to condition action generation.
- Training Strategy: Stage 1 co-trains image-text and robot data to connect pretrained multimodal knowledge with robotic actions.
- Training Strategy: Stage 2 freezes the pretrained VLM and trains only the action expert so actions follow generated reasoning.
- Experimental setup: The experiments use math matching and toy placement setups with robots performing specified pick-and-place actions.
4 Experiments
ChatVLA-2 is evaluated on real-robot open-world reasoning, understanding, and manipulation using math matching and toy placement tasks. It outperforms compared methods in open-world settings, while ablations show that both mixture-of-experts and two-stage training are important.
- Experimental setup: The experiments evaluate mathematical reasoning, OCR, object recognition, spatial reasoning, and manipulation in both in-domain and open-world settings.Open-world targets or equations are absent from training data, requiring recognition, spatial understanding, and task execution.
- Robot setup: ChatVLA-2 was tested on a bimanual ARX-R5 robot with two 6-DoF arms, a top RealSense L515 camera, and 50 Hz teleoperation.The combined state and action space is 14-dimensional.
- Math matching game: 82.7% manipulation success, 3.58 OCR accuracy, and 1.73 mathematical reasoning accuracy were achieved by ChatVLA-2 in the open-world math matching game.None of the compared methods successfully completed manipulation tasks in the open-world condition.
- Toy placement: 81.4% average success rate in open-world toy placement represented a 3.52-times improvement over DexVLA.The result is attributed to strong spatial reasoning and reasoning-following capabilities.
- Ablation studies: Removing mixture-of-experts caused a significant drop in math-matching success, and increasing the dense model to 7B parameters did not resolve the failure.Dense models failed completely on unseen equations and often recognized incorrect answers.
- Ablation studies: Removing Stage 2 reduced open-world robotic control performance to 23%, while removing Stage 1 produced near-zero OCR and mathematical reasoning scores.Stage 1 generates open-world reasoning, whereas Stage 2 injects that reasoning into action execution.
5 Conclusion
ChatVLA-2 combines a dynamic mixture-of-experts architecture with two-stage training to preserve pretrained multimodal knowledge while connecting reasoning to robotic actions. Its experts separate shared multimodal features from task-specific representations, and its training stages establish knowledge-action links before teaching reasoning-following.
- Architecture: ChatVLA-2 integrates a dynamic mixture-of-experts module atop a pretrained vision-language backbone to manage different task requirements.Some experts share multimodal features, while others specialize in task-specific representations.
- Training strategy: Its two-stage training first connects pretrained multimodal knowledge to robotic actions, then trains the model to translate reasoning outputs into corresponding actions.The second stage specifically targets reasoning-following behavior.
A Limitation
The method does not yet fully retain pretrained VLM knowledge after fine-tuning and is currently focused mainly on tabletop tasks.
- A Limitation: Fine-tuning with robot data still causes some pretrained VLM capabilities to disappear, a challenge left for future work.The authors state that current approaches cannot fully resolve this retention problem.
- A Limitation: The current method is mainly evaluated on tabletop tasks rather than mobile-manipulator or longer-horizon real-world settings.The authors aim to expand the embodiment in future work.
B.1 Training details.
Training uses two stages combining image-text and robot data before freezing the VLM backbone, with curated multimodal datasets and flexible reasoning annotations.
- B.1 Training details.: Stage 1 co-trains on image-text and robot data for 15k steps, while stage 2 freezes the VLM backbone and trains for 50k steps.Both stages start at 2e-5, use cosine scheduling down to 2e-6, and stage 2 warms up for 3k steps.
- B.1 Training details.: The image-text mixture includes approximately 32k COCO, 20k TextVQA, and 54k GQA samples plus task-aligned data.These datasets support the reasoning template used in the toy placement task.
- B.1 Training details.: Image-text examples are limited to five dialogue turns, and TextVQA samples exclude numeric OCR tokens and mathematical operators.Images are processed at 320 × 240 resolution.
- B.1 Training details.: Robot data receive sub-reasoning annotations initialized with fixed templates and augmented using GPT-4o.The augmentation keeps reasoning phrasing flexible so the action expert does not dominate through a single template.
C.1 Ablation study on number of experts.
The expert-count ablation examines how to retain pretrained VLM knowledge while controlling resources, finding that more experts improve robotic generalization.
- C.1 Ablation study on number of experts.: Increasing both the total experts and the experts selected during inference enhances generalization in robotic scenarios.The study evaluates expert counts as a trade-off between pretrained knowledge and resource consumption.
- C.1 Ablation study on number of experts.: A limited number of experts may develop visual-similarity selection biases, encouraging robot-data overfitting and neglect of pretrained VLM knowledge.This is offered as a possible explanation for the observed performance pattern.
C.2 Ablation Study on Layers for Injecting Reasoning-Following Enhancement Module.
The reasoning-following module injects reasoning tokens into later action-expert layers, while injecting them into earlier layers destabilizes action generation and lowers success.
- C.2 Ablation Study on Layers for Injecting Reasoning-Following Enhancement Module.: Reasoning tokens replace the observation embedding and condition scale and shift parameters in the latter half of the action expert.This design injects reasoning context into the action-generation process.
- C.2 Ablation Study on Layers for Injecting Reasoning-Following Enhancement Module.: Injecting reasoning information into the former half layers increases action instability and significantly reduces task success rates.The authors hypothesize that removing observation information harms action generation because observations may carry critical action cues.