Source-linked AI summary
TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformers
Bin Yu, Shijie Lian, Xiaopeng Lin, Yuliang Wei, Zhaolong Shen, Changti Wu, Yuzhuo Miao, Xinming Wang, Bailing Wang, Cong Huang, Kai Chen
TL;DR
VLA fine-tuning can cause catastrophic forgetting, undermining the general VLM capabilities needed for embodied intelligence. TwinBrainVLA preserves those capabilities with frozen and trainable VLM pathways linked by AsyMoT, and evaluations report strong performance across simulated and real-robot settings. The approach nevertheless adds deployment computation, motivating knowledge distillation toward a single model.
Problem
Standard robotic fine-tuning disrupts pre-trained VLM features through catastrophic forgetting, compromising the general visual understanding VLA models aim to leverage.
Method
TwinBrainVLA coordinates a frozen Left Brain and trainable Right Brain, using AsyMoT to fuse preserved semantic features with embodied-control representations.
Results
TwinBrainVLA significantly outperforms mainstream VLAs across SimplerEnv, RoboCasa, and real-robot experiments.
Takeaways & Limitations
Retaining general capabilities is reported to boost embodied performance, with TwinBrainVLA matching commercial models such as π0.5 without massive pre-training.
Takeaways & Limitations
The dual-stream architecture inevitably increases deployment computational overhead, motivating Twin-to-One knowledge distillation and further efficiency work.
Abstract
from arXiv · showhide
The fundamental premise of Vision-Language-Action (VLA) models is to harness the extensive general capabilities of pre-trained Vision-Language Models (VLMs) for generalized embodied intelligence. However, standard robotic fine-tuning inevitably disrupts the pre-trained feature space, leading to "catastrophic forgetting" that compromises the general visual understanding we aim to leverage. To effectively utilize the uncorrupted general capabilities of VLMs for robotic tasks, we propose TwinBrainVLA, which coordinates two isomorphic VLM pathways: a frozen generalist (also called "Left Brain") and a trainable specialist (also called "Right Brain"). Our architecture utilizes a Asymmetric Mixture-of-Transformers (AsyMoT) mechanism, enabling the Right Brain to dynamically query and fuse intact semantic knowledge from the Left Brain with proprioceptive states. This fused representation conditions a flow-matching action expert for precise continuous control. Empirical results on SimplerEnv and RoboCasa benchmarks demonstrate that by explicitly retaining general capabilities, TwinBrainVLA achieves substantial performance gains over baseline models in complex manipulation tasks.
1 Introduction
VLA models seek to combine VLM semantic reasoning with robotic control, but fine-tuning can destroy the general capabilities they are meant to exploit. TwinBrainVLA addresses this tension with two coordinated VLM pathways and an asymmetric information-sharing mechanism.
- VLA models aim to transfer pre-trained VLM semantic reasoning and open-world generalization to robotic control.
- Standard robotic fine-tuning updates the VLM backbone for narrow action prediction, causing catastrophic forgetting of general capabilities.
- TwinBrainVLA structurally separates semantic understanding from embodied control through frozen and trainable VLM pathways.
- AsyMoT lets the trainable Right Brain dynamically query and fuse semantic features from the frozen Left Brain.
- Experiments on SimplerEnv, RoboCasa, and real-robot settings evaluate TwinBrainVLA, AsyMoT, and the proposed training strategy.
2 Related Work
Related work situates TwinBrainVLA within multimodal understanding and VLA research, while Figure 2 documents catastrophic forgetting under prevailing VLA training paradigms.
- Multimodal understanding: VLMs combine visual encoders with language models to support semantic understanding and visual question answering.
- Vision-Language-Action Models: VLA models fine-tune pre-trained VLMs on robotic manipulation data to map multimodal observations to actions.
- Catastrophic forgetting evidence: Figure 2 compares robot-only VLA training and co-training, evaluating general visual understanding across four benchmarks during fine-tuning.
3 Motivation: The Dilemma of Utilizing General Capabilities in VLA
VLA fine-tuning creates a dilemma: adapting VLMs to precise sensorimotor control can overwrite their pre-trained semantic features. Experiments show that neither robot-only training nor co-training fully preserves general visual understanding.
- VLA models seek to use VLM semantic reasoning and open-world understanding to guide robots through complex, unseen environments.
- Robot-dataset fine-tuning differs from internet-scale pre-training and can trigger catastrophic forgetting of the VLM feature space.
- The study evaluates Qwen2.5-VL-3B-Instruct and Qwen3-VL-4B-Instruct before and after VLA Training and Co-Training.
- 88.87% to 0.04%: Qwen3-VL's POPE score nearly collapses after standard VLA Training.
- Co-Training also fails to prevent severe degradation, merely mitigating symptoms rather than solving catastrophic forgetting fundamentally.
- Catastrophic forgetting undermines the premise of building generalized robotic intelligence from pre-trained VLMs.
4 Method: TwinBrainVLA
TwinBrainVLA uses two parallel VLM pathways to separate preserved semantic understanding from trainable robotic control. Its AsyMoT mechanism lets the specialist query frozen general knowledge, while a flow-matching action expert generates continuous actions.
- Asymmetric Dual-VLM Backbone: TwinBrainVLA comprises identical parallel VLM pathways: a frozen Left Brain preserves general capabilities, while a trainable Right Brain specializes in robotic control.The Left Brain provides semantic representations, and the Right Brain processes control-relevant inputs.
- Asymmetric Dual-VLM Backbone: The Right Brain combines visual, textual, and proprioceptive inputs before conditioning the action expert.A State Encoder projects proprioceptive state into the VLM embedding space.
- AsyMoT Mechanism: AsyMoT creates a unidirectional information bridge in which the Right Brain queries semantic features from the frozen Left Brain while preserving independent pathway computation.The Left Brain remains frozen and attends only to itself, whereas the Right Brain attends to both pathways.
- AsyMoT Mechanism: The Right Brain’s asymmetric joint attention constructs shared keys and values from both brains while using its own queries.This design differs from cross-attention because the Right Brain can attend to itself as well as the Left Brain.
- Flow-Matching Action Expert: A flow-matching Action Expert uses Right Brain representations to synthesize continuous action trajectories from noise.The model is implemented as a conditional Diffusion Transformer for continuous control.
- Training Strategy: Training minimizes only the robotic action objective while updating the Right Brain, Action Expert, and State Encoder, leaving the Left Brain frozen.The dual-stream design allows control specialization while the frozen pathway safeguards semantic and linguistic priors.
5 Experiment
TwinBrainVLA is evaluated across simulation, benchmark, real-robot, and ablation settings to assess generalization, manipulation performance, and architectural choices.
- SimplerEnv: 64.5% success with Qwen3-VL-4B-Instruct on SimplerEnv surpasses Isaac-GR00T-N1.6 at 57.1% by +7.4%.Qwen2.5-VL-3B-Instruct achieves 58.4%; results use Avg@480 over OOD environments.
- RoboCasa: TwinBrainVLA exceeds Isaac-GR00T-N1.6, QwenGR00T, and QwenPI on RoboCasa by +7.0%, +6.8%, and +10.7%, respectively.RoboCasa evaluates 24 tabletop manipulation tasks with the GR1 robot.
- LIBERO: 97.6% average success is achieved on LIBERO across all four task suites without massive pre-training.The model is trained on combined data from all four suites and evaluated as a single model.
- Real-robot experiments: Real-robot experiments on Franka Research 3 show performance comparable to π0.5 despite significantly less pre-training data, with robust OOD and Pick-All results.The evaluation uses teleoperated demonstrations for a pick-and-place task.
- Ablation experiments: Making the Left Brain trainable causes a 7% success-rate drop with Qwen3-VL-4B on SimplerEnv.The ablation evaluates the asymmetric freezing strategy under identical benchmark conditions.
- Ablation experiments: Removing the Left Brain and inter-VLM attention reduces the architecture to a Vanilla VLA and causes a performance drop of nearly 7 percentage points.The interaction-frequency ablation further examines sparsified AsyMoT connections.
6 Discussion
TwinBrainVLA preserves general semantic capabilities through a frozen Left Brain while using the Right Brain for embodied control. The discussion also identifies deployment overhead as a limitation and explores distillation toward simpler architectures.
- General semantic preservation: A strictly frozen Left Brain preserves open-world semantic knowledge while the Right Brain specializes in embodied control.The architecture separates semantic understanding from motor specialization during training.
- AsyMoT interaction: Runtime attention visualizations show the Right Brain attending to semantic information from the Left Brain through AsyMoT.The figure provides a runtime visualization of cross-pathway information use.
- Discussion: TwinBrainVLA matches commercial models such as π0.5 without massive pre-training and supports robust instruction following and semantic generalization in real-world scenarios.The discussion links preserved backbone semantics with complex task handling in real-world settings.
- Limitations and future direction: The dual-stream architecture introduces additional deployment computation, motivating Twin-to-One knowledge distillation into a simpler Vanilla VLA.Preliminary distillation improves performance, while the detailed experiments are deferred to Appendix G.
7 Conclusion
TwinBrainVLA addresses catastrophic forgetting in VLA models by preserving pretrained VLM capabilities through an asymmetric dual-stream design. The paper reports broad empirical gains while acknowledging remaining efficiency and semantic-fusion challenges.
- TwinBrainVLA is introduced to address catastrophic forgetting of VLM general capabilities during robotic fine-tuning.
- The framework uses a frozen Left Brain for open-world understanding and a trainable Right Brain for embodied action through AsyMoT.This structurally decouples multimodal understanding from sensorimotor control while allowing semantic features to be queried without corruption.
- The authors identify higher efficiency and deeper semantic fusion as remaining challenges.These limitations and future directions are discussed in Appendix H.
A.4 Real Robot Evaluation
The real-robot evaluation uses three settings to test in-domain performance, color-based out-of-domain generalization, and long-horizon pick-all behavior.
- The evaluation is divided into in-domain, out-of-domain, and pick-all settings.
- In-Domain: In-domain tasks retain the training objects and instructions while randomly altering block positions.
- Out-of-Domain: Out-of-domain evaluation changes the grasped block color, using the instruction to pick up the yellow block and place it into the gray box.
- Pick-All: Pick-all requires placing four colored blocks into the gray box, testing semantic comprehension and long-horizon composition of learned atomic actions.
B Implementation Details
The implementation specifies VLM initialization, fine-tuning, proprioceptive-state handling, baseline architectures, benchmark structure, and prompt templates for the two brains.
- Training: TwinBrainVLA uses Qwen2.5-VL-3B-Instruct and Qwen3-VL-4B-Instruct as VLM backbones, fine-tuned for 40K steps with AdamW and a 1e-5 learning rate.
- State Handling: The state encoder processes joint angles and gripper states, but is omitted in vision-only simulation experiments and retained for real-robot control.
- Baselines: QwenGR00T uses late fusion, QwenPI uses layer-wise deep fusion, and QwenOFT predicts discretized action bins autoregressively.These baselines differ in how the VLM interacts with action generation.
- Evaluation: RoboCasa Tabletop contains 24 distinct subtasks, with detailed success rates reported in Table 8.
- Prompt Templates: The Left Brain receives images and instructions, whereas the Right Brain additionally receives robot states and predicts the next action.
F Implementation Details on the LIBERO Benchmark
The LIBERO experiments train one generalist model on all four task suites and evaluate it over 500 trials, while the distillation study transfers TwinBrainVLA representations to a vanilla VLA student.
- Training Dataset: Training combines LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long into one dataset for a single generalist model.
- Evaluation Protocol: Evaluation follows the official LIBERO protocol across 500 trials covering 10 tasks with 50 episodes per task.The reported result is the average success rate over all trials.
- Knowledge Distillation: The Twin-to-One framework distills the trained TwinBrainVLA teacher into a standard vanilla VLA student to reduce deployment overhead.
- Distillation Objective: Distillation adds a feature-alignment loss between the teacher’s and student’s final hidden states alongside the standard action-prediction loss.The teacher’s final Right Brain hidden states provide the supervision signal.
- Distillation Experiment: The distillation experiment uses identical SimplerEnv training data and hyperparameters to the main experiments, with a student architecture matching TwinBrainVLA’s Right Brain.
G.3 Results and Analysis
TwinBrainVLA’s distillation results show that a single-stream student can outperform Vanilla VLA, while the two-stream design incurs efficiency costs and leaves semantic fusion unresolved.
- Performance Gains: 58.4% average success rate for the Distilled Student exceeds Vanilla VLA’s 55.2% by +3.2%.Table 9 reports the comparison on SimplerEnv using the Qwen3-VL-4B-Instruct backbone.
- Performance Gains: The results support attributing TwinBrainVLA’s gains to semantics-rich features fused from the Left Brain.
- Efficiency: TwinBrainVLA adds training and deployment costs because it uses a second VLM component.The paper prioritizes generalizability over efficiency and acknowledges this overhead as a limitation.
- Efficiency: The two 4B-model architecture outperforms a larger 8B-parameter model, despite its added component cost.
- Towards Deeper Semantic Fusion: Seamlessly fusing retained semantic information into downstream VLA tasks for true generalization remains unresolved.The paper identifies the architecture as a viable pathway but calls for further exploration of semantic-policy integration.