PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Act under Visual Interruptions with Vision-Language-Action Models

Mingle Jiang, Rui Xu, Yunke Wang, Chang Xu

arXiv:2609.35003v1cs.ROcs.AI

TL;DR

VLA benchmarks typically assume camera streams remain available, leaving closed-loop performance under visual interruption insufficiently understood. MAIL-Bench measures that setting, and MINT combines training with missing inputs and selectively gated visual predictions. Across two VLA backbones, MINT improves interruption robustness and is also deployed on a real robot.

  • Problem

    Standard evaluations typically assume all camera streams remain available, while the effect of missing views during closed-loop manipulation remains insufficiently understood.

    Source: §1 · p. 1

  • Method

    MAIL-Bench pairs successful healthy rollouts with role-specific camera interruptions, and MINT trains policies for missing inputs before selectively supplying reliable predicted views.

    Source: §1 · pp. 1–2, §4 · p. 5

  • Results

    Across π0.5 and GR00T N1.5, MINT improves interruption robustness, and the same interface transfers to real-robot deployment.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Camera-loss severity depends on the visual role lost, and MINT improves robustness without modifying the policy architecture.

    Source: §1 · p. 2, §6 · p. 9

    MAIL-Bench covers one embodiment and 18 RoboCasa365 Atomic-Seen tasks; broader validation across benchmarks and robot platforms remains future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Vision-language-action (VLA) models have demonstrated strong capabilities in robotic manipulation, but they are typically developed and evaluated with all camera streams available throughout task execution. When a camera stops delivering frames during task execution, the policy must continue acting without access to subsequent observations from the missing view. Despite its practical importance, how such interruptions affect closed-loop manipulation remains insufficiently understood. To investigate this problem, we introduce MAIL-Bench, a benchmark that evaluates visual interruptions with VLA models. By interrupting different cameras at multiple stages of each policy's successful reference trajectory, MAIL-Bench measures how well policies retain their capabilities when visual inputs become unavailable. Building on this benchmark, we propose MINT, which first trains VLA policies to remain functional under missing visual inputs. At inference time, MINT selectively supplements missing observations using optical-flow extrapolation or an action-conditioned world model, and withdraws predicted views when they become unreliable. Experiments on $\pi_{0.5}$ and GR00T N1.5 show that MINT significantly improves task success under camera loss over the original models. Experiments on AgiBot G2 further demonstrate the real-robot deployment under camera loss. The benchmark is available at https://minglejiang.github.io/Mail-Bench/

1 INTRODUCTION

VLA evaluation typically assumes uninterrupted camera streams, leaving their behavior under complete visual-stream loss insufficiently understood. MAIL-Bench measures this loss, and MINT combines missing-input training with selective prediction; experiments report improved interruption performance and real-robot deployment.

1 INTRODUCTION

  • Existing benchmarks typically keep camera streams available, leaving limited evidence about VLA performance when one or more views disappear during execution.Unlike changes to lighting, pose, or image quality, interruption removes a source of information altogether.Source: §1 · p. 1
  • MAIL-Bench pairs rollouts in the same scenes and interrupts distinct visual roles at 30%–60% of each policy’s own successful completion time.It compares success conditional on intact-camera success.Source: §1 · pp. 1–2
  • Camera loss removes a large share of both base policies’ successes, while MINT recovers part of that loss on every visual role.Figure 1 compares scenes solved out of 900 under healthy and interrupted conditions.Source: Figure 1 · p. 2
  • MINT trains policies to act with missing inputs, then selectively supplies optical-flow or action-conditioned predictions and withdraws them when reliability checks fail.The method uses optical flow for missing wrist views or all-vision loss, and a world model for missing third-person views.Source: §1 · p. 2
  • Across π0.5 and GR00T N1.5, MINT improves task success under camera loss; an AgiBot G2 deployment completes closed-loop tasks with a camera interrupted mid-task.On GR00T N1.5, it also outperforms a matched clean fine-tuning control with the same training budget.Source: §1 · p. 2

2 RELATED WORK

Prior robustness evaluations largely perturb available images or camera conditions, whereas MAIL-Bench tests persistent loss of a visual stream during closed-loop execution. MINT’s missing-input fine-tuning follows prior masked-training work without adding a policy component.

2 RELATED WORK

  • Generalist and perturbation benchmarks assess intact observations or alter appearance, geometry, language, pose, or sim-to-real conditions while frames remain available.Prior methods also study task-aware visual selection and run-time intervention (Xu et al., 2026b; Feng et al., 2026).Source: §2 · p. 2
  • MAIL-Bench distinguishes itself by evaluating complete visual-stream loss during an ongoing rollout rather than changing the content of an available frame.Its interruption timing follows each policy’s healthy trajectory, and paired outcomes are scored conditional on healthy success.Source: §2 · pp. 2–3, Table 1 · p. 3
  • Prior absent-sensor approaches include masked multimodal training, camera dropout, missing-state inference, retrieval, and availability-aware representations; MINT follows the masked-training line without changing the policy.Examples include Skand et al. (2025) and Vanjani et al. (2025).Source: §2 · p. 3

3 MAIL-BENCH: MANIPULATION UNDER VISUAL INTERRUPTION

MAIL-Bench evaluates camera-loss robustness on paired scenes and policies’ own successful healthy trajectories. Its protocol groups cameras by visual role, varies interruption timing, and scores retained success only on scenes the policy can already solve.

3 MAIL-BENCH: MANIPULATION UNDER VISUAL INTERRUPTION

  • MAIL-Bench evaluates how visual-stream interruptions affect manipulation policies on tasks they can otherwise complete.The benchmark is built on 18 RoboCasa365 Atomic-Seen tasks with 50 fixed scenes each, totaling 900 scenes.Source: §3 · p. 3, §3.1 · p. 3

3.1 BENCHMARK CONSTRUCTION

  • Camera groups represent the wrist view, onboard third-person views, or all vision; once interrupted, a group supplies no further frames and has availability flag 0.The healthy condition retains all cameras, while all-vision loss leaves proprioception and the instruction.Source: §3.1 · p. 3
  • Interruptions begin at 30%, 45%, or 60% of each policy’s successful healthy completion time, making onset relative to execution speed.Figure 2 depicts the same scene replayed with one role interrupted while unaffected cameras continue streaming.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 BENCHMARK EVALUATION

  • Each interrupted rollout restarts from the healthy rollout’s saved initial state with the same policy random seed and receives normal observations until interruption.After interruption, future frames from the missing group are withheld from both the policy and any recovery module; replacement views must use runtime-available information.Source: §3.2 · p. 4
  • The protocol discards remaining actions in the current chunk and immediately queries the policy when an interrupted camera becomes unavailable.From that query onward, the missing visual role is represented by no fresh frame and an availability value of 0.Source: §3.2 · p. 4
  • Robustness is measured only on scenes solved with all cameras; the interruption score measures how much of that healthy capability remains.MAIL-Bench weights healthy performance and nine interruption conditions equally per task, then averages equally over tasks.Source: §3.2 · pp. 4–5

4 MINT: ACTING UNDER VISUAL INTERRUPTION

MINT pairs training for missing visual inputs with selective prediction of unavailable views. It supplies predictions only while they are considered reliable, then falls back to the policy’s learned missing-input behavior.

4 MINT: ACTING UNDER VISUAL INTERRUPTION

  • MINT adapts policies to changing observation availability, then selectively supplies predicted views when reliable so they can act without depending on reconstruction.The policy predicts action chunks, executing only the first few actions before the next query.Source: §4 · p. 5

4.1 LEARNING WITH MISSING VISUAL INPUTS

  • Training samples use demonstration observations, state, instruction, and actions under temporal availability patterns spanning healthy, missing-view, persistent, and intermittent conditions.Missing-view tokens are excluded from attention, while the policy architecture, optimization schedule, and action objective remain unchanged.Source: §4.1 · p. 5

4.2 PREDICTING MISSING VISUAL INPUTS

  • MINT uses short-horizon optical-flow extrapolation for wrist or all-vision loss, and an action-conditioned world model for third-person loss.The world model predicts missing third-person views and an auxiliary wrist view using rollout memory, the surviving wrist observation, committed actions, and task instruction.Source: §4.2 · p. 6

4.3 RELIABILITY-GATED FALLBACK

  • Route-specific checks gate predictions; once rejected, a route stays withdrawn for the episode and the policy uses its learned missing-input behavior.Route reliability requires all checks to fall within their admissible ranges.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

Experiments evaluate MINT on two VLA backbones using MAIL-Bench and matched controls, then test closed-loop deployment on an AgiBot G2. MINT improves interruption success in benchmark and real-robot settings.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers two RoboCasa365 Human300-trained backbones and their MINT variants across 18 tasks and 900 frozen scenes.Tasks are grouped into pick-and-place, small-appliance, and fixture-and-navigation categories.Source: §5.1 · p. 7

5.2 BENCHMARK RESULTS

  • MINT outperforms the base models across almost all benchmark interruption conditions on both VLA backbones, with gains across camera roles and early and late onsets.The benchmark table reports healthy scores, interruption scores, and improvement over each base checkpoint.Source: §5.2 · p. 8, Table 2 · p. 8
  • Figure 4 summarizes the GR00T N1.5 ablation across task categories at each interruption onset, with H marking each variant’s healthy-input score.The supplied results do not state the ablation outcome.Source: Figure 4 · p. 8

5.3 REAL ROBOT EXPERIMENT

  • On the AgiBot G2, MINT outperforms GR00T N1.5 in every missing-input setting without degrading healthy performance.Figure 6 reports success rates over 10 trials per task and condition.Source: §5.3 · p. 9, Figure 6 · p. 9
  • In a Cube to Drawer episode, the policy completes grasping, transport, release, and drawer closure after the third-person camera is interrupted and world-model prediction begins.The wrist-camera stream remains available.Source: §5.3 · p. 9, Figure 5 · p. 9

6 CONCLUSION

The conclusion presents MAIL-Bench as a systematic evaluation of camera interruption and MINT as an approach combining missing-input training with selective visual prediction. It reports robustness gains across two backbones and real-robot deployment, while identifying limits in benchmark scope and inference overhead.

6 CONCLUSION

  • MAIL-Bench shows that interruption severity depends strongly on which visual role is lost.The paper presents it as a systematic evaluation of VLA models under camera interruption.Source: §6 · p. 9
  • MINT combines learning under missing visual inputs with selective visual prediction to improve robustness.Source: §6 · p. 9
  • Across two VLA backbones, MINT improves interruption robustness without changing policy architecture and transfers to real-robot deployment.Source: §6 · p. 9
  • The benchmark covers one embodiment and 18 RoboCasa365 Atomic-Seen tasks, while the world-model route adds inference and memory overhead.The paper identifies broader validation and more efficient prediction methods as future work.Source: §6 · p. 9

A MAIL-BENCH DETAILS

MAIL-Bench fixes a task suite and horizons and evaluates π0.5 and GR00T N1.5 on 900 scenes. It also distinguishes official benchmark scoring from auxiliary fault diagnostics and describes controls for paired rollouts.

A MAIL-BENCH DETAILS

  • The suite fixes 18 RoboCasa365 Atomic-Seen tasks, 50 scenes per task, and their control-step horizons.The suite is versioned under RoboCasa365 1.0.1.Source: A MAIL-BENCH DETAILS · p. 13, Table 3 · p. 13
  • Under healthy observations, π0.5 solves 362 of 900 scenes and GR00T N1.5 solves 405; missing-view training does not reduce GR00T N1.5 healthy success.GR00T N1.5 reaches H = 0.532 with Clean FT + prediction, 0.540 with Missing-view FT, and 0.544 with MINT.Source: A MAIL-BENCH DETAILS · p. 13
  • Blackout, freeze, stale delivery, burst dropout, finite-duration loss, and recovery operators are diagnostic options, not part of the official MAIL-Bench score.Source: A MAIL-BENCH DETAILS · p. 13
  • Paired rollouts reset to the same frozen scene state and use seeded policy sampling, with small GPU-renderer differences recorded.Source: A MAIL-BENCH DETAILS · p. 13

B MINT IMPLEMENTATION DETAILS

MINT trains with camera-availability variation and masks unavailable visual tokens from policy computation. Its serving routes predict missing views subject to checks, and hyperparameters are fixed using disjoint development scenes.

B.1 MISSING-VIEW TRAINING

  • Training represents unavailable cameras with an availability flag and inert zero image placeholder, while excluding their visual tokens from policy computation.The training curriculum includes healthy, wrist-missing, third-person-missing, and all-vision-missing observations.Source: §B.1 · p. 13

B.2 SERVING AND PREDICTION ROUTES

  • The optical-flow route uses a flow-confidence check after a minimum tenancy, while the world-model route checks each generated block and withdraws at rejection or its generation cap.The two third-person cameras are admitted and withdrawn together; a failed route returns that role to missing input for the rest of the episode.Source: §B.2 · pp. 13–14
  • PersistWorld is adapted to the 18 tasks with a low-rank fine-tune; on held-out demonstrations, PSNR rises from 16.4 to 18.9 dB and out-of-domain conditioning states fall from 0.18 to 0.01.The rollout memory starts from the last available observations, and auxiliary wrist prediction is used only for the consistency check.Source: §B.2 · p. 14

B.3 CHOICE OF HYPERPARAMETERS

  • Hyperparameters are fixed on development scenes disjoint from the 900 benchmark scenes before benchmark evaluation.Source: §B.3 · p. 14

C MAIL-BENCH PER-TASK AND PER-CONDITION RESULTS

The per-task tables report healthy-success counts and identify tasks without healthy success; counts based on few scenes should be interpreted cautiously.

C MAIL-BENCH PER-TASK AND PER-CONDITION RESULTS

  • Task categories are pick-and-place, small appliances, and fixtures and navigation, with the listed tasks assigned to each category.The categories cover the five PickPlace tasks; six small-appliance tasks; and seven fixtures-and-navigation tasks.Source: C MAIL-BENCH PER-TASK AND… · p. 14
  • N_H_j counts healthy successes out of 50 scenes, while “n/a” marks a task without healthy success and contributes zero to S_j.Source: C MAIL-BENCH PER-TASK AND… · p. 14
  • Per-task values based on small N_H_j rest on few scenes.Source: C MAIL-BENCH PER-TASK AND… · p. 14

C.1 COMPUTE COST

The compute-cost section reports policy, evaluation, and fine-tuning costs, as well as additional world-model costs under interruption.

C.1 COMPUTE COST

  • MINT does not increase per-step wall-clock time during healthy execution; under interruption, optical flow runs on the CPU.MAIL-Bench evaluation takes 60–70 single-worker hours per arm; fine-tuning takes 12.9 GPU-hours for GR00T N1.5 and 15.1 GPU-hours for π0.5.Source: §C.1 · p. 14
  • Table 5 reports healthy success, conditional interruption scores, and S_MAIL averaged over 18 tasks.Source: Table 5 · p. 15
  • Table 6 describes a GR00T N1.5 ablation, and Table 7 gives paired bootstrap intervals for S_MAIL and per-condition differences.Table 7 resamples scenes within each task and pairs arm differences on the same resampled scenes.Source: Table 6 · p. 15, Table 7 · p. 15
  • The world model adds 9.2 GiB of GPU memory and about 100 TFLOPs per generation.It averages 7.35 generations per third-person-missing cell for GR00T N1.5 and 7.72 for π0.5.Source: §C.1 · p. 15
  • Tables 8–11 provide per-task, per-condition results for π0.5 and GR00T N1.5, with and without MINT.Source: Table 8 · p. 16, Table 9 · p. 16, Table 10 · p. 17

D REAL-ROBOT SETUP

The real-robot setup uses an AgiBot G2 with head and wrist cameras and evaluates four tabletop tasks after fine-tuning on successful teleoperated episodes.

D REAL-ROBOT SETUP

  • The AgiBot G2 has a head-mounted camera and one camera on each wrist, with the right arm used for manipulation.Source: D REAL-ROBOT SETUP · p. 18
  • The study collected 224 teleoperated episodes across four tabletop tasks and fine-tuned GR00T N1.5 on 203 successful episodes.Source: D REAL-ROBOT SETUP · p. 18
  • Figure 7 shows healthy episodes for drawer, cube-into-bowl, cup-onto-plate, and cup-stacking tasks at five time points.Each panel shows the head camera above and right-wrist camera below.Source: Figure 7 · p. 18

Related papers

Loading 2609.35003v1…