PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Bimanual Vision-Language-Action Models via Embarrassingly Simple Modality Masking

Dongzhou Cheng, Ziang Li, Yixiao Zhou, Haojuan Li, Jinghao Zhang, Lei Lei, Minjing Dong, Jie Gui, Jiaqi Wang

arXiv:2608.22419v1cs.ROcs.CV

TL;DR

Query-based VLAs enable low-latency bimanual control but can fail when multi-view fusion follows distracting or spurious cues. M3 introduces training-only modality masking to expose policies to controlled partial observations. Across simulated and real-world evaluations, M3 improves robustness over the Adapter baseline without architectural changes or large-scale pretraining.

  • Problem

    Query-based VLAs can learn spurious multi-view correlations and attend to distracting cues, causing discontinuous actions and failures under distribution shift.

  • Method

    M3 stochastically masks modality channels and action-query subsets during training while preserving structured partial observability, including the egocentric view as a stable reference.

  • Results

    21.7% average success improvement in Clean and 11.4% in Clean2Rand are reported over the Adapter baseline, alongside improved real-world full-task success across three long-horizon tasks.

  • Takeaways & Limitations

    Structured training-time masking is a practical strategy for strengthening query-based VLA policies for bimanual manipulation without inference-time architectural changes.

  • Takeaways & Limitations

    Broader evaluation across robot platforms, task categories, VLA architectures, richer modalities, and masking effects on multimodal fusion remains future work.

Abstract

from arXiv · show

Query-based Vision-Language-Action (VLA) models offer low-latency inference that is attractive for bimanual robotic manipulation, but we observe that they can still exhibit discontinuous actions and execution failures in complex dual-arm tasks. We hypothesize that unstable multi-view and language fusion is one contributing factor in these failures, often coinciding with attention spreading to distracting regions. To improve robustness, we introduce the Modality Masking Mechanism (M3), an embarrassingly simple, training-only strategy that requires no architectural changes or large-scale robot pretraining. M3 stochastically masks subsets of modality channels during training, exposing the policy to controlled partial observations and encouraging it to rely less on distracting cues and more on evidence that remains reliable. We evaluate M3 on ten bimanual tasks from RoboTwin 2.0 and on three long-horizon real-world tasks. Compared with the Adapter baseline, M3 improves average success by 21.7% in the Clean setting and 11.4% in Clean2Rand, where policies are trained on clean demonstrations and evaluated on randomized scenes, while also improving averaged real-world full-task success by over 30%. These results suggest that structured training-time masking is a practical way to improve the robustness of query-based VLA policies for bimanual manipulation.

1 Introduction

Query-based VLAs provide efficient bimanual control but can fail when multi-view attention follows distracting or spurious cross-view cues. M3 addresses this through structured training-time masking and improves simulated and real-world robustness.

  • Motivation: Query-based VLAs can produce discontinuous actions and poor execution when attention spreads to irrelevant image regions or contact areas.Multi-view inputs can introduce salient but task-irrelevant activations associated with unstable rollouts and reduced success.
  • Motivation: Always-visible camera views can encourage spurious cross-view correlations, such as idle-arm distractors influencing actions for the active arm.These correlations can break under distractors, occlusions, lighting changes, or other distribution shifts.
  • Results: M3 produces more stable end-effector paths and contact alignment, with consistent improvements over the plain baseline across clean simulation and real-world evaluations.The figure contrasts discontinuities near contact for the plain VLA with more stable M3 execution.
  • Method: M3 trains under structured view dropout by preserving the egocentric view while jointly masking wrist views and selectively masking language and action queries.The design aims to prioritize reliable evidence and reduce dependence on wrist-to-egocentric saliency matching.
  • Evaluation: M3 requires no architectural changes or large-scale pretraining and is evaluated on RoboTwin 2.0 and three long-horizon real-world tasks.It is presented as a low-overhead, training-only intervention for query-based bimanual VLA policies.
  • Results: 21.7% average success improvement in Clean and 11.4% in Clean2Rand are reported for M3 over the Adapter baseline.Clean2Rand trains on clean demonstrations and evaluates on randomized scenes.

2 Preliminaries

Query-based VLA policies convert multi-view images and language into action chunks using visual tokens, language tokens, and action queries. Related implementations differ in how queries enter the vision-language-action pipeline and how actions are decoded.

  • Query-based VLA: A query-based VLA maps multi-view images and language to an H-step action chunk in one forward pass.The predicted chunk contains actions at successive offsets h = 0, ..., H−1.
  • Query-based VLA: The model represents camera observations as visual tokens Vt, instructions as language tokens T, and decoder inputs as action queries Q.These token groups form the inputs to the generic policy gθ.
  • Query-based VLA: Actions are trained with an ℓ1 regression objective against the ground-truth action sequence.The objective supervises the predicted action chunk rather than requiring autoregressive single-step decoding.
  • Related architectures: OpenVLA-OFT uses Q = 0 as a special query, while VLA-Adapter uses learnable queries injected through per-layer bridge attention.The Adapter action latent is modulated using self-attention and cross-attention before action decoding.

3 Modality Masking Mechanism (M3)

M3 trains query-based VLA policies under structured partial observability by masking modality channels and subsets of action queries while preserving useful spatial and task information. Its design uses coordinated view masking, query dropout, and additive attention masks to encourage robust, complementary representations.

  • M3 Strategy: M3 stochastically hides modality channels and action-query subsets during training while retaining the standard L1 regression objective.The policy must approximate ground-truth actions despite missing signals.
  • Design Guidelines: The egocentric view remains visible as a stable spatial reference, while wrist views are masked jointly rather than independently.This preserves global localization and reduces spurious cross-view matching between wrist and egocentric views.
  • Design Guidelines: Random query-subset masking never removes all queries, discouraging co-adaptation and encouraging different queries to specialize in complementary task information.At least one action query remains active at every training step.
  • Mask Construction: M3 integrates modality visibility through a unified additive attention mask that permits interactions only between visible tokens.The modality mask is combined element-wise with the causal mask, without changing the embeddings.
  • Query Rescaling: Visible query embeddings are rescaled to preserve expected feature energy, encouraging task-relevant information to spread across dynamically available query slots.This query-dimension dropout mechanism promotes diverse, complementary contextual representations.

4 Experiments

Experiments evaluate M3 on RoboTwin 2.0 simulation and real-world bimanual tasks under clean and distribution-shifted conditions. M3 consistently outperforms the Adapter baseline, with especially large gains on long-horizon and OOD evaluations.

  • 4.1 Setup: RoboTwin 2.0 evaluates ten selected tasks using 50 clean demonstrations and 100 held-out scenarios in Clean and Clean2Rand settings.Clean uses held-out clean scenes, whereas Clean2Rand evaluates the same clean-trained policies on randomized scenes.
  • 4.2 Main Results: 41.0% to 62.7%: M3 raises overall Clean success on RoboTwin 2.0 and outperforms the Adapter baseline and compared pretrained policies.Long-horizon average success increases from 33.7% to 61.0%.
  • 4.2 Main Results: 15.1% overall success: M3 achieves the highest Clean2Rand performance and gains +11.4% over the Adapter baseline.The largest reported per-task gains are +38% on Put Bottles Dustbin and +26% on Grab Roller.
  • 4.2 Main Results: 44.4% to 69.4% in clean and 12.5% to 61.1% under OOD clutter: M3 improves averaged full-task success across three real-world tasks.Qualitative rollouts show smoother bottle disposal and more stable handover alignment for M3.

5 Analysis

Analysis examines whether M3 transfers across backbones, focuses spatial attention, improves training efficiency, and depends on structured rather than generic masking. The evidence supports gains from preserving the ego view, jointly masking wrist views, and combining vision with query masking.

  • Cross-backbone transfer: 32.2% to 53.5%: M3-OFT improves average success for OpenVLA-OFT under the domain-clean RoboTwin protocol.Gains occur across short-, medium-, and long-horizon tasks.
  • Visualization of M3: M3 concentrates attention more compactly on objects and contact regions, whereas the Adapter baseline attends more to distractors, grippers, and backgrounds.Attention remains partly diffuse on the harder Block Rank Size task.
  • Training efficiency: Under identical settings, M3 reaches strong performance earlier and sustains a higher success-rate margin than Adapter on short- and long-horizon tasks.The paper defines training efficiency as reaching a target success rate with fewer optimization steps without sacrificing the final plateau.
  • Ablation of M3: Preserving the ego view while jointly masking both wrist views outperforms masking the ego view or only one wrist view.A light query-mask ratio performs best, while overly aggressive query masking reduces success.
  • Ablation of M3: 64.0% Avg. SR: full vision-language-query masking exceeds single-component masking, vision-plus-language masking, and generic dropout or augmentation baselines.Vision plus query masking reaches 60.3% Avg. SR, while token dropout and modality dropout reach 31.8% and 24.1%.

6 Related Works

Related work situates M3 among vision-language-action models, bimanual manipulation methods, and training-time regularization. The paper contrasts existing VLA architectures and sensing approaches with its structured masking strategy.

  • Vision-Language-Action Models: VLAs connect pretrained vision-language models to robotic action, with existing categories including autoregressive models that discretize continuous actions into language tokens.The related-work discussion names RT-2 and OpenVLA as autoregressive examples.
  • Learning-based Bimanual Manipulation: Recent bimanual manipulation methods use structured reinforcement learning, coordination priors, visuo-tactile fusion, wrist-force sensing, and human-video action signals.These approaches target robust transfer, contact-state estimation, robustness, data efficiency, or embodiment gaps.
  • Dropout and training-time regularization: Dropout and modality dropout regularize multimodal models by deactivating neurons or modalities during training.The paper compares these generic methods with its structured masking design.

7 Limitations

The paper reports consistent gains across RoboTwin 2.0 and three real-world tasks, but identifies broader validation and deeper mechanism analysis as future work.

  • Limitations: M3 still requires evaluation on more robot platforms, task categories, VLA architectures, and richer modalities such as 3D geometric information.The paper also calls for deeper analysis of structured masking in multimodal fusion and diffusion-based action decoding.

8 Conclusion

The paper introduces M3 as a training-only modality masking strategy for robust query-based bimanual VLA control, while retaining parameter-efficient, low-latency model choices. Its evaluation spans simulation and real-world settings, with additional transfer evidence for OpenVLA-OFT.

  • Conclusion: M3 consistently improves over the Adapter baseline across RoboTwin 2.0 and three long-horizon real-world tasks without inference-time architectural changes.The reported gains include improved average success in Clean and Clean2Rand and higher averaged full-task success under clean and OOD clutter conditions.
  • Conclusion: VLA-Adapter remains a practically reasonable backbone because it is resource-efficient and reported as broadly comparable to OpenVLA-OFT in simulation.The authors also observe that the Adapter baseline achieves a higher average success rate than OpenVLA-OFT in their domain-clean evaluation, while noting benchmark and implementation differences.
  • Conclusion: M3 stochastically masks partial modality channels during training to address unstable multi-view and language fusion without changing inference architecture.The strategy is motivated by attention spreading toward distracting regions and reliance on unstable correlations across views.
  • Conclusion: VLA-Adapter uses a frozen 0.5B vision-language backbone with lightweight adapter and policy modules, enabling parameter-efficient fine-tuning without large-scale robotic pretraining.Its Bridge Attention connects control-relevant multimodal features to the action space, and one-shot decoding supports low-latency inference.
  • Conclusion: The study evaluates bimanual manipulation methods on RoboTwin 2.0 task statistics and reports additional architectural comparisons involving OpenVLA-OFT.OpenVLA-OFT uses parallel decoding with bidirectional attention and zero-query embeddings for action axes and chunk size.

A.4 Tasks.

The additional task setup covers ten RoboTwin 2.0 bimanual tasks grouped by temporal horizon, with evaluation also varying language familiarity and scene domain. This organization enables analysis across task complexity, linguistic perturbation, and transfer beyond the primary backbone.

  • A.4 Tasks: The RoboTwin 2.0 evaluation selects 10 bimanual manipulation tasks spanning coordination, contact-rich interaction, and spatiotemporal reasoning.The tasks are organized by planning horizon and average step count.
  • A.4 Tasks: Short-horizon tasks include Click Bell, Grab Roller, and Place Phone Stand, averaging approximately 95 steps.These tasks typically involve fundamental reaching or grasping primitives.
  • A.4 Tasks: Medium-horizon tasks average 197 steps and require sequential pick-and-place operations or dual-arm coordination.Examples include Place Bread Basket, Place A2B Right, Place Shoe, and Stack Blocks Two.
  • A.4 Tasks: Long-horizon tasks demand extended reasoning and multi-stage execution, with Block Rank Size extending beyond 500 steps and averaging 461 steps.The category also includes Handover Block and Put Bottles Dustbin.
  • A.4 Tasks: The multi-horizon organization exposes baseline drift or instability on longer sequences and supports domain-clean comparisons across Adapter and OpenVLA-OFT variants.Table 8 marks M3 integrations with +M3 and defines Δoft as relative improvement over OpenVLA-OFT.
  • A.4 Tasks: The benchmark evaluates both seen instructions from training and unseen instructions with varied phrasing during evaluation.Unseen instructions preserve semantic intent while testing robustness to linguistic diversity.

B Additional Experiments

An additional OpenVLA-OFT experiment tests whether M3 transfers beyond the primary backbone. The integration improves domain-clean average success across short-, medium-, and long-horizon tasks.

  • B Additional Experiments: M3 improves OpenVLA-OFT’s domain-clean average success rate by 21.3% over the standard OpenVLA-OFT baseline.The gains remain consistent across short-, medium-, and long-horizon tasks.

B.1 Quantitative Attention-Misalignment Analysis

The attention analysis links discontinuous rollouts with greater attention outside prompt-relevant visual patches, especially during later contact-rich execution. M3 rollouts tend to maintain lower attention on such regions.

  • B.1 Quantitative Attention-Misalignment Analysis: The analysis compares attention misalignment across Place Phone Stand, Place Shoe, and Handover Block trajectories.It forms an important patch set from prompt-to-visual relevance and measures action-to-visual attention outside that set.
  • B.1 Quantitative Attention-Misalignment Analysis: Discontinuous trajectories tend to show higher unimportant-attention values than smooth trajectories across normalized rollout time.The comparison uses rollouts normalized to the interval [0, 1].
  • B.1 Quantitative Attention-Misalignment Analysis: The gap between discontinuous and smooth trajectories is more visible during middle-to-late execution, when contact formation, object transfer, and final placement require stable localization.These stages involve accurate localization and temporally stable evidence selection.
  • B.1 Quantitative Attention-Misalignment Analysis: M3 trajectories tend to maintain lower attention outside prompt-relevant patches throughout rollout, suggesting reduced drift toward distractors during contact-rich stages.The result quantitatively complements the paper’s heatmaps and is consistent with reduced reliance on task-irrelevant cues during cross-view fusion.
  • B.1 Quantitative Attention-Misalignment Analysis: Bimanual manipulation makes idle-arm wrist views especially relevant as distractors because they can remain salient while being action-irrelevant for the current sub-step.This creates competition among cross-view evidence that is less characteristic of aligned single-arm settings.

C Visualization Results

The visualizations compare M3 with the baseline through real-world phase-level outcomes, attention patterns, execution timelines, and long-horizon rollouts. Across these views, M3 is associated with more focused attention and more stable task execution, including under OOD clutter.

  • Qualitative Real-World Cases: M3 completes more later-stage real-world sequences in examples where the baseline stalls, fails transfer or disposal, or misses an early grasp.These differences appear in Bottle Cleanup, Stack & Shelf, and Veggie Centering across clean and OOD cases.
  • Real-World Evaluation: Table 9 reports per-phase and full-task success rates for three real-world bimanual tasks across clean and OOD evaluations.Full-task rows include pooled rates and sample standard deviations across three evaluation rounds.
  • Real-World Evaluation: Under OOD clutter, baseline rates decline across all phases, while M3 degrades comparatively less.The qualitative cases align with these phase-level trends across Bottle Cleanup, Stack & Shelf, and Veggie Centering.
  • Qualitative Attention Patterns: Baseline attention is often scattered across table surfaces and irrelevant regions, whereas M3 exhibits more focused attention in representative bimanual tasks.The attention patterns are presented as qualitative evidence concerning distraction from background textures, grippers, and other irrelevant regions.
  • Qualitative Long-Horizon Rollouts: In long-horizon rollouts, M3 completes sorting, bottle disposal, and handover sequences that the baseline disrupts through ranking errors or unstable contact.The shown baseline failures include a ranking error at T = 120s and a knocked-over bottle at T = 169s.
  • Discussion: The discussion hypothesizes that M3’s dynamic visibility constraints may regularize learning by reducing reliance on salient but task-irrelevant features.The authors connect this hypothesis to focused attention, lower execution instability, and improved long-horizon performance.
Loading 2608.22419v1…