ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledge

1 Introduction

• VLAs integrate pretrained VLMs and have demonstrated dexterous manipulation, long-horizon tasks [1] [2], and open-world generation [3] [4].The paper attributes their advantages over traditional imitation learning to pretrained multimodal knowledge and architectures.Source: §1 · p. 2
• Fine-tuning VLAs on robotics data can erode pretrained VLM knowledge, causing failures on tasks absent from training; ChatVLA [7] illustrates this degradation.The paper notes that VLMs can recognize objects, reason about spatial relationships, and solve math problems, abilities that may be lost during adaptation.Source: §1 · p. 2
• ChatVLA-2 targets generalization through VLM priors, separating multimodal and control features while preserving shared representations and ensuring actions follow internal reasoning.The paper frames this as a feasible demonstration of leveraging pretrained knowledge, rather than building a robot able to perform every conceivable task.Source: §1 · pp. 2–3
• The proposed dynamic mixture-of-experts disentangles multimodal and action features while identifying shared representations, and a reasoning-enhancement module aligns action outputs with internal reasoning.Training first preserves pretrained multimodal knowledge while connecting it to robotic actions; then it freezes the VLM backbone and trains only the action expert.Source: §1 · p. 3
• On out-of-distribution math matching and toy-placement tasks, ChatVLA-2 showed superior reasoning and generalization to OpenVLA, DexVLA [2], and π0 [1].The math equations and many objects and directional instructions were unseen during training.Source: §1 · p. 3

Vision-language-action models in robot learning

• VLAs typically fine-tune large pretrained VLMs to predict robot actions [25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 39, 54, 55], perform strongly across simulated and real-world tasks and varied embodiments, environments, and objects, yet lack generalization beyond training data.ChatVLA-2 is designed to retain and leverage pretrained VLM knowledge, enabling robots to perform open-world tasks through pretrained reasoning and extensive general knowledge.Source: §2 · p. 3

Embodied Reasoning in VLA models

• Embodied-CoT and CoA-VLA use structured spatial text, CoT-VLA and VPP use generated visual imagery, and DiffusionVLA, DexVLA, and π0.5 use plain-language instructions [57, 58, 59, 60, 61, 2, 3].These methods explicitly train reasoning representations, limiting broader generalization to knowledge in their training datasets.Source: §2 · p. 4
• ChatVLA-2 advances embodied reasoning by leveraging pretrained VLM knowledge to support open-world reasoning and generalization.This contrasts with approaches whose explicitly trained reasoning is limited to their training datasets.Source: §2 · pp. 3–4

3.1 Preliminary: Vision-Language-Action Model

• DexVLA [2] supplies the backbone: Qwen2-VL [62] [6] produces reasoning and action tokens, while a pretrained 1B ScaleDP [63] serves as the action expert.The authors chose DexVLA because it supports unstructured textual reasoning, letting the approach use pretrained VLM knowledge and generalize across diverse scenes.Source: §3.1 · pp. 4–5

3.2 Model Architecture

• Dynamic MoE routes inputs to selected experts, preserving the VLM’s intact architecture while enabling task-specific adaptation; the model uses eight experts and activates two at inference.Expert MLP layers are initialized with pretrained MLP weights; the authors motivate dynamic routing as a way to avoid disrupting pretrained knowledge with static or shared experts.Source: §3.2 · p. 5
• For novel reasoning, ChatVLA-2 replaces observation embeddings with MLP-projected reasoning tokens and uses them with timestep embeddings to condition scale and shift parameters in the latter half of action-expert layers.The authors report that selective integration handles open-world reasoning without sacrificing in-domain accuracy.Source: §3.2 · p. 5

3.3 Training Strategy

• Stage one co-trains on image-text and robot data, using COCO [67], TextVQA [68], GQA [69], and robotics image-text pairs without targeting OCR, math, or spatial reasoning specifically.Text augmentation increases query diversity. Robot data include 600 math-matching and 300 toy-placement trajectories, with an image-text-to-robot-data ratio of 1:3; training runs for 50k steps.Source: §3.3 · p. 6, Figure 3 · p. 6
• Stage two freezes the pretrained VLM and trains only the action expert, preserving pretrained knowledge while strengthening how actions follow model-generated reasoning in unseen scenarios.The training strategy is presented as a way to improve instruction following while retaining the open-world reasoning developed in stage one.Source: §3.3 · p. 6, Figure 3 · p. 6

4 Experiments

• The real-robot experiments used math matching and toy placement to assess open-world reasoning and understanding transferred to physical interaction; simulation benchmarks were not used.The study evaluated mathematical reasoning, spatial reasoning, OCR, and object recognition and localization, with many test scenarios absent from training.Source: §4 · p. 7

4.1 Mathematical Reasoning: Math Matching Game

• In open-world math matching, ChatVLA-2 scored 3.58 in OCR accuracy, 1.73 in mathematical reasoning accuracy, and 82.7% manipulation success, while all comparators failed manipulation.In-domain manipulation performance was comparable to π0 and DexVLA, which already achieved near-perfect performance; open-world tests used equations absent from training and unfamiliar typography.Source: §4.1 · pp. 7–8, Table 1 · p. 8

4.2 Spatial Reasoning: Toy Placement

• On open-world toy placement, ChatVLA-2 achieved 81.4% average success, 3.52 times DexVLA, and object-recognition performance of 0.94.The target and reference objects were entirely unseen during training; other models except π0 had near-random action success, while ChatVLA’s open-world performance was much lower than 0.94.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 Ablation Study

• Without MoE, dense models’ open-world math-matching success dropped substantially, and scaling to a 7B VLM did not prevent complete failure on unseen equations.The authors hypothesize that conflicts between robotic actions and reasoning/understanding in parameter space explain the decline, and report that dense models often recognize the wrong answer.Source: Table 3 · p. 9, §4.3 · p. 9
• Removing Stage 2 reduced open-world robot control success to 23%, while removing Stage 1 left OCR and mathematical reasoning scores near zero.Stage 1 generates open-world reasoning through co-training with image-text data; Stage 2 is intended to inject that reasoning into action execution.Source: Table 4 · p. 9, §4.3 · p. 9

• ChatVLA-2 uses pretrained vision-language reasoning and understanding to enable VLA models to perform diverse tasks end to end.Source: §5 · p. 9
• Its dynamic Mixture-of-Experts module combines experts that share multimodal features with others dedicated to task-specific representations.Source: §5 · pp. 9–10
• Training first connects pretrained multimodal knowledge to robotic actions, then teaches the model to translate reasoning outputs into corresponding actions.Source: §5 · p. 10

A Limitation

• Fine-tuning on robot data inevitably removes some pretrained VLM capacity, a knowledge-retention challenge the current approach cannot fully resolve.The authors leave this problem to future work.Source: A Limitation · p. 15
• The method is mainly limited to tabletop tasks; future work aims to extend it to mobile manipulators for longer-horizon, more complex real-world tasks.Source: A Limitation · p. 15

B.1 Training details.

• Training uses mixed precision (FP16) and the AdamW optimizer.Source: §B.1 · p. 15
• Stage 1 co-trains on image-text and robot data for 15k steps, starting at a learning rate of 2e-5.Source: §B.1 · p. 15
• In stage 2, the VLM backbone is frozen.Source: §B.1 · p. 15
• Stage 2 trains for 50k steps, starting at a learning rate of 2e-5 with a 3k-step warm-up.Source: §B.1 · p. 15
• Both stages use a cosine learning-rate scheduler that scales the rate down to 2e-6.Source: §B.1 · p. 15
• Total training costs 340 GPU hours.Source: §B.1 · p. 15

B.2 Data details.

• The image-text dataset includes approximately 32k COCO samples, 20k TextVQA samples, and 54k GQA samples, alongside additional data specifically constructed to align with the task formulation.Robotics-related image-text pairs use the reasoning template used in the toy placement task; all collected data are combined during training.Source: §B.2 · p. 15
• Approximately 2k simulated RoboPoint samples enhanced spatial understanding despite lower visual quality, and 5k real-world samples cover tabletop setups and broader scenes.The real-world samples use a question-answer format similar to LLaVA.Source: §B.2 · p. 15
• Image-text examples are limited to five dialogue turns; longer examples retain the first turn and randomly sample four others.Source: §B.2 · p. 15
• TextVQA samples containing numeric OCR tokens or mathematical operators are excluded, and images are processed at 320 × 240 resolution.Source: §B.2 · p. 15

C More Ablation Studies

• The additional ablation section addresses the mixture-of-experts choice and two-stage training strategy, which the paper identifies as key components.The supplied passage introduces these ablation questions but does not report their results.Source: C More Ablation Studies · p. 16

C.1 Ablation study on number of experts.

• Increasing both the total number of experts and the number selected during inference enhances the model’s generalization in robotic scenarios.The experiments examine how expert count affects the use of pretrained VLM knowledge while maintaining resource consumption.Source: §C.1 · p. 16
• With too few experts, selection may favor visually similar task images, leading to robot-data overfitting and neglect of pretrained VLM knowledge.The authors offer this as a possible explanation for the performance degradation associated with limited experts.Source: §C.1 · p. 16

C.2 Ablation Study on Layers for Injecting Reasoning-Following Enhancement Module.

• The study compares injection locations after replacing the observation embedding with reasoning tokens that condition scale and shift parameters in the latter half of the action expert.The authors report the placement results in Table 6; no numerical values are provided in the supplied passages.Source: §C.2 · p. 16
• Injecting reasoning information into the former half of the action expert increases action-generation instability and significantly reduces task success rates.The authors hypothesize that replacing the original observation embedding may remove information critical to action generation.Source: §C.2 · p. 16