PaperlayerView exact revision on arXiv

Source-linked AI summary

Targeted Modality Dropout for Real-Robot Manipulation Robust to Intermittent Vision Loss

Genki Shikada, Kazuki Osamura, Masaru Ide, Tetsuya Ogata, Kanata Suzuki

arXiv:2610.09566v1cs.RO

TL;DR

Multimodal imitation policies may over-rely on a dominant modality such as vision, making inference-time loss a robustness concern. TMD estimates modality dependence using attention, selectively drops the most dominant modality, and regularizes the dependence distribution with entropy. In real-robot evaluation, baseline success counts fell under vision loss while TMD’s total count stayed unchanged; RMD failed on many tasks even without vision loss.

  • Problem

    Multimodal imitation policies can over-rely on a dominant modality, and loss of vision at inference may severely impair execution.

    Source: §I · p. 1

  • Method

    TMD estimates modality dependence from attention, targets and drops the most dependent modality, and applies entropy regularization to the dependence distribution.

    Source: §I · p. 1

  • Results

    Under 10% per-inference-chunk loss of all camera images, Baseline success count fell from 14/30 to 2/30 (total Δ = −12), while TMD remained at 15/30 (total Δ = 0).

    Source: 2) Vision-Loss Setting · p. 5

  • Takeaways & Limitations

    On the evaluated bimanual robot, TMD’s value was robustness under vision loss rather than better success rates in normal conditions.

    Source: D. Discussion · p. 6

    The real-robot trial count was limited, TMD and RMD differed in both selection rule and regularization, and evaluated vision loss was short-term.

    Source: §V · p. 7

Abstract

from arXiv · show

Imitation learning policies that integrate multiple sensory modalities are prone to overreliance on a dominant modality, such as vision, during training, which can disrupt policy execution when that modality is lost at inference time. In this paper, we introduce Targeted Modality Dropout (TMD), in which the dependence on each modality is estimated using attention and the most dominant modality is selectively dropped. This is combined with entropy regularization over the dependence distribution. Through real-robot evaluation using a bimanual manipulator, we show that under vision loss the success rate of the baseline policy drops substantially, whereas TMD sustains task execution. In contrast, a conventional dropout that selects the dropped modality at random, without the entropy regularization, fails on many tasks even without vision loss.

I. INTRODUCTION

Multimodal imitation policies can over-rely on vision, making inference-time vision loss a practical robustness concern. The paper proposes TMD, which uses attention-estimated dependence to target the dominant modality and adds entropy regularization; real-robot evaluation reports sustained execution under vision loss, while RMD fails on many tasks even without it.

I. INTRODUCTION

  • Imitation learning trains a policy from teleoperated demonstrations by supervised action prediction, then executes it in a sensor-driven closed loop.Recorded inputs include camera images, robot state, and commanded actions.Source: §I · p. 1
  • Vision can become dominant in multimodal training, so its loss at inference may severely impair execution; camera failures and occlusions occur in real deployments.The paper identifies overreliance on a particular modality as a practical robustness concern.Source: §I · p. 1
  • Figure 1 contrasts ordinary training, where estimated dependence concentrates on vision, with TMD, which drops the dominant modality and balances dependence.The unfilled vision bar marks what inference-time vision loss removes.Source: Figure 1 · p. 1
  • TMD estimates modality dependence from attention, drops the most dependent modality as a target, and regularizes the dependence distribution with entropy.The stated aim of the entropy regularization is to alleviate concentration on any single modality.Source: §I · p. 1
  • The study evaluates inference-time stochastic vision loss during closed-loop task execution on a real bimanual manipulator and compares TMD with random modality selection.The comparison method is Random Modality Dropout (RMD).Source: §I · p. 1
  • TMD sustains task execution under vision loss, whereas RMD fails on many tasks even without vision loss.RMD selects the dropped modality randomly and omits entropy regularization.Source: §I · pp. 1–2

II. RELATED WORK

Manipulation policies combine partially substitutable inputs, while their relative reliance is learned implicitly under an assumption that all inputs are available. Prior approaches address modality reliance or missing inputs in different ways, but real-robot closed-loop evaluation under stochastic inference-time vision loss has rarely been examined.

A. Multimodal Imitation Learning for Manipulation

  • Images provide object position and orientation, language specifies the task, and force or tactile signals report contact that images cannot capture.Transformer policies combine inputs as a token sequence, and vision-language-action models use pretrained visual and language encoders [1],,.Source: A. Multimodal Imitation Learning for… · p. 2

B. Controlling Modality Reliance

  • Fusion design, phase-dependent weighting, visual degradation, gradient rebalancing, and modality dropout address modality reliance, but each has stated constraints,,,,,,,.The passage notes architecture dependence, assumes modality availability, does not identify dominance, or does not train for modality absence; prior audiovisual dropout tuned probabilities in advance.Source: B. Controlling Modality Reliance · p. 2

C. Evaluation under Missing Modalities

  • Real-robot closed-loop task execution under stochastic inference-time vision loss has rarely been examined; prior missing-modality work mainly used classification benchmarks or simulated manipulation, [16].Visuotactile work addressed only partially restricted vision [14].Source: C. Evaluation under Missing Modalities · p. 2

A. Problem Formulation and Notation

The paper formulates a three-modality Transformer vision-language-action policy that integrates tokens through cross-attention and generates action chunks. It estimates each modality’s relative dependence from cross-attention weights for use by the proposed method.

A. Problem Formulation and Notation

  • The policy integrates multiple input modalities via cross-attention and generates an action chunk.It is a Transformer-based vision-language-action model.Source: A. Problem Formulation and Notation · p. 2
  • The three modalities are language instructions, images, and state, with M containing language, vision, and state and |M| = 3.Here, state denotes motor currents, used as a signal reflecting contact states without an external force sensor.Source: A. Problem Formulation and Notation · p. 2
  • The proposed method uses dependence estimated from cross-attention to suppress the dominant modality as a target.The passage introduces subsequent dependence estimation, targeted dropout, and entropy regularization descriptions.Source: A. Problem Formulation and Notation · p. 2
  • Modality dependence is estimated from the Transformer’s cross-attention weights.The passage defines layer and head indices, query set Q, and key/value tokens T_m for modality m.Source: B. Estimation of Modality Dependence · p. 2
  • The layer- and head-averaged attention weight and modality dependence are calculated from attention weights.The passage presents these quantities as the basis for the dependence estimate.Source: B. Estimation of Modality Dependence · p. 2
  • The dependence p_t,m represents the relative magnitude of dependence on modality m, and the modality values sum to one.The normalization is stated as the sum over m in M of p_t,m = 1.Source: B. Estimation of Modality Dependence · p. 2

C. Targeted Modality Dropout (TMD)

TMD selects the modality with the greatest estimated dependence and stochastically masks it at the modality-token level during training. The mask probability is scheduled to rise and then return to zero, while dependence is estimated and used to choose the subsequent step’s target.

C. Targeted Modality Dropout (TMD)

  • TMD targets the modality with the largest estimated dependence and stochastically drops it during training.The selected target is the argmax over modality dependence values.Source: C. Targeted Modality Dropout (TMD) · p. 2
  • For each sample, a uniform random draw determines whether to zero all tokens of the previously selected target modality.Dropout is applied independently across samples, with probability ρ_t.Source: C. Targeted Modality Dropout (TMD) · p. 2
  • The dependence estimate drives both targeted token masking and entropy inclusion in the training loss.Figure 2 depicts the target modality’s tokens being zeroed in some samples and entropy of p_t entering the loss; its illustrated target is vision.Source: Figure 2 · p. 3, C. Targeted Modality Dropout (TMD) · p. 3
  • The training step schedules ρ_t, samples whether to mask the target, runs the masked minibatch, estimates dependence, updates the loss, and selects the next target.The algorithm uses the dependence at step t to determine m†_t for use at step t + 1.Source: C. Targeted Modality Dropout (TMD) · p. 3
  • The mask coefficient applies to a modality’s tokens, blocking information at the modality level when the target tokens are zeroed.The coefficient is shared across token positions within a sample, so all target-modality tokens are zeroed simultaneously.Source: C. Targeted Modality Dropout (TMD) · p. 3
  • The dropout probability starts at zero, ramps up to an upper bound, and decays back to zero late in training.Training therefore ends without dropout.Source: C. Targeted Modality Dropout (TMD) · p. 3

D. Entropy Regularization

Entropy regularization discourages the modality-dependence distribution from concentrating on one modality. It adds a weighted negative-entropy term to the task loss.

D. Entropy Regularization

  • The regularizer applies the confidence-penalty form to the dependence distribution over modalities rather than the output distribution over classes.Source: D. Entropy Regularization · p. 3
  • Maximizing dependence-distribution entropy moves modality dependence toward a uniform distribution, approximately 1/M ≃ 0.33.The regularizer is intended to suppress concentration on any single modality.Source: D. Entropy Regularization · p. 3
  • The total loss subtracts λH(pt) from task loss, with λ set to 0.001 in this study.Source: D. Entropy Regularization · p. 3

IV. EXPERIMENTS

The experiments evaluate a real-robot manipulation policy using multiple sensory inputs across six tasks. The section covers experimental conditions, evaluation conditions, results, and discussion.

IV. EXPERIMENTS

  • The experiments section presents conditions, evaluation conditions, results, and discussion.Source: §IV · p. 3
  • The real-robot setup receives images from head and hand cameras as well as motor currents.Source: Figure 3 · p. 4
  • The evaluation includes bottle, zipper closing and opening, towel folding, trash disposal, and unplugging tasks.Source: Figure 3 · p. 4

A. Experimental Setup

The study evaluates a 471-million-parameter vision-language-action policy on a bimanual robot, with training and targeted-dropout schedules specified for the experiments.

A. Experimental Setup

  • CobotMagic uses 14-dimensional actions comprising each arm’s six joint angles and one gripper command, with camera images, motor currents, and language instructions as inputs.Source: 1) Implementation · p. 4
  • The policy uses a TVF-DiT-based architecture with frozen pretrained DINOv3 and SigLIP2 visual encoders and a Diffusion Transformer action decoder trained with Conditional Flow Matching.Source: 1) Implementation · p. 4
  • Training uses 100 demonstrations per task and 300k steps; targeted-dropout probability peaks at 0.6 at 60k steps, then decays to zero from 240k to 300k.Source: 1) Implementation · p. 4

2) Comparison Conditions:

The experiments compare ordinary training, targeted modality dropout with entropy regularization, and a random-dropout control. Real-robot evaluation spans six tasks under normal and stochastic vision-loss settings.

2) Comparison Conditions:

  • RMD selects the dropped modality uniformly at random and omits entropy regularization, unlike TMD’s dependence-targeted selection and regularization.Source: 2) Comparison Conditions · p. 4

3) Evaluation:

  • Six real-robot tasks are evaluated with five trials per condition under normal input availability and 10% per-chunk vision loss.Source: 3) Evaluation · pp. 4–5

B. Real-Robot Evaluation

1) Normal Setting:

With all modalities available, Baseline and TMD had comparable total success counts; under vision loss, Baseline’s count dropped while TMD’s total remained unchanged. During training, TMD balanced modality dependence, whereas random modality dropout did not.

1) Normal Setting:

  • With all modalities available, Baseline and TMD had comparable totals (14/30 and 15/30), while RMD achieved 4/30 and was excluded from subsequent vision-loss evaluation.Source: 1) Normal Setting · p. 5

2) Vision-Loss Setting:

  • Under 10% per-chunk vision loss, Baseline fell from 14/30 to 2/30 successes (Δ = −12), while TMD stayed at 15/30 (Δ = 0), with task-wise gains and losses canceling.The unchanged TMD total masks decreases in towel folding and trash disposal and an increase in unplug.Source: Table II · p. 5, 2) Vision-Loss Setting · p. 5

1) Correction of Modality Dependence during Training:

  • During training, TMD reduced vision dependence and balanced the modalities, whereas random-target RMD left vision dependence dominant.Source: 1) Correction of Modality Dependence… · p. 5
  • At the evaluated end-of-training checkpoint, TMD dependence was 0.29 for language, 0.37 for vision, and 0.34 for state; RMD vision dependence was 0.95.Before dropout ramp-up, entropy regularization alone yielded dependence of 0.29, 0.37, and 0.34 for language, vision, and state; during dropout, TMD targeted vision in 0.51, state in 0.45, and language in 0.05 of steps.Source: 1) Correction of Modality Dependence… · p. 5

2) Continuity of Actions under Vision Loss:

Vision-loss chunks produce larger action changes in both conditions, but TMD shows a smaller within-trial increase than Baseline. The comparison uses adjacent-step action deltas averaged separately for loss and normal chunks.

2) Continuity of Actions under Vision Loss:

  • 3.12 for Baseline versus 2.29 for TMD was the mean within-trial ratio of action delta under vision loss to normal-setting action delta.The ratio comparison was significant (Mann–Whitney U test, p = 1.6 × 10^-8, n = 30 per condition).Source: Figure 5 · p. 6, 2) Continuity of Actions under Vision… · p. 6
  • Action delta sums both arms’ L2 changes in commanded action vectors between adjacent steps, averaged per trial separately for loss-generated and normal chunks.Source: 2) Continuity of Actions under Vision… · pp. 5–6, Figure 5 · p. 6
  • Under vision loss, mean action delta was 0.039 for Baseline and 0.022 for TMD; increases over each condition’s normal level were 0.027 and 0.012, respectively.Normal-setting action delta was 0.010 for TMD and 0.012 for Baseline, so absolute values partly reflect TMD’s smaller adjacent-step changes overall.Source: 2) Continuity of Actions under Vision… · p. 6

D. Discussion

The discussion finds TMD's value in greater robustness under vision loss rather than higher normal-setting success. The conclusion reports that TMD sustains execution under vision loss, while noting limits in trial count, comparison design, and loss duration.

D. Discussion

  • Under vision loss, Baseline success counts drop on many tasks, while the drop is smaller for TMD; TMD’s value is robustness, not normal-setting success gains.TMD does not outperform Baseline on every normal-setting task, including close zipper.Source: D. Discussion · p. 6
  • The real-robot evaluation used N = 5 trials per condition, and vision loss was short-term at each inference chunk, leaving sustained loss unevaluated.Small success-count differences may be subject to error.Source: D. Discussion · pp. 6–7, §V · p. 7

V. CONCLUSION

  • TMD targets the dominant modality using attention-estimated dependence with entropy regularization; its action commands are less perturbed under vision loss, while RMD fails many tasks normally.The TMD–RMD difference cannot be attributed solely to the selection rule because the methods also differ in entropy regularization.Source: D. Discussion · p. 6, §V · p. 7
Loading 2610.09566v1…