PaperlayerView exact revision on arXiv

Source-linked AI summary

MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalization

Zaibin Zhang, Junlan Xiao, Zhongbo Zhang, Yifan Wang, Li Kang, Yiran Qin, Changxing Xia, Heng Zhou, Talas Fu, Enshen Zhou, Ruimao Zhang, Zhenfei Yin, Huchuan Lu, Lijun Wang

arXiv:2608.25864v1cs.RO

TL;DR

Existing VLAs often treat language as one global command, leaving per-arm task allocation implicit and limiting transfer to collaboration patterns absent from training. MA-VLA assigns atomic prompts to individual arms and uses Arm Shuffle to encourage role-agnostic recomposition; across simulation and real-world evaluations, it reports consistent gains over strong baselines. The benchmark explicitly tests new compositions formed from training-known atomic actions.

  • Problem

    Most existing VLAs lack explicit task decomposition and per-arm subtask allocation, while test-time collaboration may require unseen combinations of layouts, arm states, and inter-arm dependencies.

  • Method

    MA-VLA uses a VLM-based Planner to produce temporally ordered per-arm atomic sub-goals and a VLA Executor to ground them into actions, with Arm Shuffle permuting arm input bundles during training.

  • Results

    MA-VLA consistently outperforms competitive imitation-learning and VLA baselines in in-domain performance and multi-arm compositional generalization across RoboFactory, RoboTwin 2.0, and real-world SO101.

  • Takeaways & Limitations

    Structured per-arm atomic action assignment offers a practical route to scalable multi-arm instruction following and generalization to unseen collaboration patterns.

  • Takeaways & Limitations

    The evaluation requires test compositions that are absent from training while reusing the training action set: P_test is not a subset of P_train.

Abstract

from arXiv · show

Multi-arm collaboration is becoming a core capability in embodied manipulation. Recent vision-language-action (VLA) models integrate perception, language, and control, but most represent language as a single global instruction and do not provide an explicit mechanism for assigning and composing arm-specific behaviors. This design limits transfer to collaboration patterns that differ from those observed during training. We present MA-VLA, a unified framework for multi-arm collaboration via atomic action assignment. MA-VLA decomposes cooperative behavior into mid-level atomic prompts and allocates them to individual arms, enabling explicit subgoal specification and compositional reuse across tasks. To reduce reliance on fixed execution roles, we introduce Arm Shuffle, a training-time permutation of the observation, state, and assigned atomic prompts for each arm. This permutation enforces role-agnostic instruction following and supports recomposition into unseen coordination patterns, which we term multi-arm compositional generalization. We also construct a benchmark in which test-time collaboration patterns are absent in training set. Across simulation and real-world evaluations, prior state-of-the-art VLAs largely fail under these unseen collaborations, while MA-VLA consistently succeeds. These results indicate that structured, per-arm atomic action assignment offers a practical route to scalable generalization in multi-arm embodied systems. Code, models, and data are available at https://github.com/zhangzaibin/future-robots

1 Introduction

Multi-arm collaboration matters for embodied manipulation, but existing VLAs usually lack explicit per-arm task decomposition and allocation, limiting transfer to unseen coordination patterns. MA-VLA addresses this with atomic action assignment and Arm Shuffle, and reports consistent gains across simulation and real-world evaluations.

  • Multi-arm systems can solve tasks infeasible for a single arm through parallel execution and coordinated control.
  • Most existing VLAs map a single static command to control without explicitly decomposing tasks or allocating subtasks across arms.
  • Unseen collaboration patterns require recombining familiar atomic actions with assignment patterns across new layouts, arm states, and inter-arm dependencies.
  • MA-VLA decomposes high-level instructions into interpretable mid-level atomic actions and assigns them to individual arms within one end-to-end model.
  • Arm Shuffle permutes each arm’s state, local observation, and assigned atomic prompts during training to promote role-agnostic behavior and recomposition.
  • MA-VLA consistently outperforms competitive imitation and VLA baselines across RoboFactory, RoboTwin 2.0, and real-world SO101 evaluations.

2 Related Work

Prior work spans robot policy learning and VLA models, with generalization studies emphasizing environmental, visual, or task-level changes. Multi-arm collaborative generalization remains comparatively underaddressed, motivating Arm Shuffle.

  • Behavior cloning and offline reinforcement learning remain dominant policy-learning paradigms but depend heavily on expert demonstrations.
  • VLA models integrate vision, language, and action generation to map visual inputs and natural-language instructions into robot actions.
  • Prior generalization studies mainly address environmental, visual, compositional, or parameterized task changes, largely in single- or dual-arm settings.
  • Arm Shuffle targets multi-arm collaborative generalization by randomizing arm roles during training to break fixed dependencies.

3 Preliminaries

This section formalizes multi-arm VLA control and compositional generalization by representing collaboration as arm-assigned atomic actions whose compositions can differ between training and testing.

  • Vision-Language-Action Models: A single-arm VLA maps visual input, proprioception, and an instruction to an action through a policy trained with behavioral cloning.The observation combines visual input and robot state, while the action-level loss can use MSE or cross-entropy.
  • Vision-Language-Action Models: For multiple arms, a shared policy predicts each arm’s action from the joint observations, states, and instruction, optimizing summed per-arm behavioral-cloning losses.
  • Multi-arm Compositional Generalization: Multi-arm compositional generalization recombines familiar atomic actions into unseen collaborative executions without retraining.The shared atomic-action library includes behaviors such as grasp, lift, align, and place.
  • Multi-arm Compositional Generalization: A multi-arm execution is represented as a sequence of arm-tagged atomic actions, with actions potentially occurring in parallel across arms.
  • Multi-arm Compositional Generalization: The test distribution retains the training atomic-action library but introduces collaboration compositions absent from training, including new assignments, ordering, synchronization, intermediate states, and interactions.
  • Multi-arm Compositional Generalization: The MA-VLA pipeline decomposes a high-level instruction into atomic prompts and represents each arm with its prompt, state, wrist-view observation, and ground-truth action; Arm Shuffle permutes these tuples during training.

4 Method

MA-VLA combines a VLM-based planner that assigns temporally ordered atomic sub-goals to arms with a unified VLA executor that generates coordinated actions. Training uses Arm Shuffle and View Dropout to promote role-invariant coordination and robust visual grounding.

  • System overview: The architecture couples a VLM-based Planner for per-arm atomic sub-goal decomposition with a VLA Executor for grounding those sub-goals into actions.The planner handles collaboration structure, while the executor performs fine-grained physical control.
  • VLM-based planner: Given language and visual observations, the planner produces a temporally ordered sequence of stage-wise prompts specifying which arm performs each atomic action.Atomic prompts come from a shared curated vocabulary and each stage runs until a task-dependent termination condition.
  • VLA executor: At each timestep, per-arm prompts are concatenated into one collaborative instruction, and the unified executor jointly predicts all arms’ actions from visual observations and proprioceptive states.This design captures inter-arm dependencies while allowing distinct, complementary behaviors and uses arm-specific action heads after shared processing.
  • Training strategies: Arm Shuffle randomly permutes each arm’s state, view, prompt, and action correspondence during training, preventing overfitting to fixed positional or identity-based correlations.The permutation forces arms to interpret atomic prompts semantically rather than structurally.
  • Training strategies: View Dropout randomly masks visual inputs during training, encouraging the model to use redundant spatial cues and learn consistent multi-view reasoning.Both Arm Shuffle and View Dropout are stochastic augmentations integrated under the same behavioral-cloning action loss.

5 Experiments

The experiments evaluate MA-VLA across two simulation benchmarks and a real-world dual-arm platform, using in-domain and compositional out-of-domain settings. Results show improvements over baselines in simulation and real-world collaboration, including unseen coordination patterns.

  • Experimental Setup: The evaluation spans RoboFactory, RoboTwin2.0 (Hard), and real-world dual-arm SO101 under in-domain and out-of-domain settings.RoboFactory uses 2–4 arms, RoboTwin2.0 includes strong visual disturbances, and SO101 has 6 degrees of freedom per arm.
  • Experimental Setup: 150 expert demonstrations are collected per simulation task, while each SO101 task uses 50 teleoperated demonstrations and 20 evaluation episodes.SO101 training runs for 15,000 gradient steps with batch size 32.
  • In-domain Results: Atomic action conditioning consistently improves in-domain RoboFactory collaboration, with larger gains for 3–4-arm tasks.The results suggest atomic prompts help disambiguate per-arm responsibilities as coordination complexity increases.
  • In-domain Results: MA-VLA remains robust on RoboTwin2.0 (Hard), improving over the Pi0 backbone and diffusion-based baselines under strong visual disturbances.The benchmark includes distractors, background variation, and lighting changes.
  • Out-of-domain Results: MA-VLA improves out-of-domain success by up to 13.0 in unseen collaboration tasks where end-to-end imitation baselines collapse.The evaluated scenarios contain novel role orders and coordination structures unseen during training.
  • Real-world Results: On real-world SO101, MA-VLA improves Pi0 in-domain and achieves non-zero success in role-reversed out-of-domain settings where Pi0 fails.The real-world tasks include Stack Two Bowls, Place Two Cubes, Pass Two Toys, and Stack Two Cubes.

6 Ablation Study

Ablations show that atomic actions improve in-domain performance, Arm Shuffle drives out-of-domain generalization, and View Dropout further improves transfer with minimal in-domain degradation. Increasing shuffle probability raises out-of-domain performance while leaving in-domain accuracy relatively stable.

  • 6.1 Ablation of Components: Atomic actions markedly improve in-domain performance on the Three Robots Stack Cube unseen-order task.This task evaluates the in-domain blue–green–red order and three unseen orderings.
  • 6.1 Ablation of Components: Arm Shuffle produces non-zero out-of-domain success where prior models fail, demonstrating clearer compositional generalization.The ablation targets unseen stacking orders and coordination structures.
  • 6.1 Ablation of Components: View Dropout further boosts out-of-domain compositional generalization with only minimal in-domain degradation.The added regularization addresses visual discrepancies in environments, target appearance, and arm configurations.
  • 6.2 Ablation of Shuffle Probability: Increasing shuffle probability from 0% to 100% raises out-of-domain performance from near-zero to significantly higher levels.The shuffle probability controls random permutation of arms’ states, viewpoints, and instructions.
  • 6.2 Ablation of Shuffle Probability: In-domain accuracy experiences a mild drop as shuffle rate increases but remains stable.The controlled study indicates a trade-off between in-domain accuracy and compositional generalization.
  • Separate Model Comparison: Separate per-arm VLA models show poor compositional generalization despite following their assigned atomic actions.Each model receives only its own viewpoint, state, and action history, then runs in parallel at test time.

7 Visualization

Figure 6 presents qualitative MA-VLA rollouts in RoboFactory and on the real-world SO101 platform. The visualizations compare in-domain behavior with out-of-domain recombination of seen atomic actions into unseen coordination patterns.

  • Qualitative Rollouts: Figure 6 shows MA-VLA rollouts in simulation and on real-world SO101 under in-domain and out-of-domain splits.The out-of-domain split requires recombining seen atomic actions into unseen multi-arm coordination patterns.

8 Conclusion

MA-VLA makes multi-arm collaboration explicit by assigning interpretable atomic actions to individual arms within one VLA model. With Arm Shuffle, it improves transfer to unseen collaboration patterns across simulation and real-world evaluations.

  • Conclusion: MA-VLA decomposes high-level instructions into interpretable atomic actions and assigns them to individual arms within a single model.Arm Shuffle encourages role-agnostic behavior and supports compositional transfer.
  • Conclusion: Experiments on RoboFactory, RoboTwin 2.0, and real-world SO101 show consistent gains over imitation-learning and VLA baselines.The gains occur in both in-domain performance and multi-arm compositional generalization.
Loading 2608.25864v1…