PaperlayerView exact revision on arXiv

Source-linked AI summary

PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training

Mingyu Liu, Chonghao Sima, Tianjian Feng, Hanqing Wang, Cong Chen, Hao Chen, Chunhua Shen

arXiv:2610.04616v1cs.ROcs.CV

TL;DR

VLA policies can complete familiar tasks while relying on cues from demonstrations rather than the evidence needed to decide what action to take. Perturbot changes the training data, and GroundFscore evaluates shortcut reliance offline. Together, they frame scaling as expanding the decisions that depend on the scene, instruction, and observed outcome.

  • Problem

    Task success cannot distinguish policies using task evidence from policies replaying actions when familiar cues appear, and repeated demonstration correlations may preserve shortcuts.

    Source: §1 · pp. 1–2

  • Method

    Perturbot adds task-preserving wrist-view perturbations, decision-relevant captions, and relabeled random and failed trajectory segments; GroundFscore scores shortcut reliance offline.

    Source: §1 · p. 2

  • Results

    GroundFscore requires actions to stay unchanged under irrelevant edits and change under task-relevant edits, so ignoring an input does not help.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper argues that scaling should expand decisions depending on the scene, instruction, and observed outcome, not only repeat familiar behavior.

    Source: §7 · p. 9

    Perturbot depends on detailed behavior-matched captions and recorded random or failed segments; relabeling cannot create unrecorded behaviors such as recovery.

    Source: §6 · p. 9

Abstract

from arXiv · show

A vision--language--action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundingFscore, an offline score that diagnoses how severely a policy relies on modality shortcuts. Task success rate shows whether a policy improves, while GroundingFscore reveals whether the policy scales healthily, relying on task evidence rather than shortcuts. Together, Perturbot and GroundingFscore provide a training-and-evaluation framework for disentangling VLA decisions from shortcut priors while preserving responsiveness to task-relevant evidence.

1 INTRODUCTION

VLA policies can succeed while relying on visual, lexical, or motor shortcuts rather than the evidence needed to choose actions. Perturbot changes training data to make task evidence more usable, while GroundFscore diagnoses shortcut reliance offline.

1 INTRODUCTION

  • Modality shortcuts let visual, lexical, or motor cues predict expert actions without the evidence the underlying decision requires.Successful demonstrations can pair the target with the wrist view, an object name with one operation, or gripper closure with a lift.Source: §1 · p. 1
  • More demonstrations that preserve the same correlations need not remove shortcuts, so familiar-task success can rise without more reliable decisions when cues and evidence disagree.This can leave target selection, instruction following, and outcome assessment brittle.Source: §1 · p. 2
  • GroundFscore is an offline score for diagnosing shortcut reliance; unlike task success, it requires both unchanged actions under irrelevant edits and known changes under relevant edits.The paired edits mean that ignoring an input does not raise the score.Source: §1 · pp. 2–3
  • The paper evaluates all eight component combinations on 50 real-robot General PnP cases and assesses broader skills in RoboTwin 2.0.Ablations test whether success and evidence use rise together as data grow, and whether GroundFscore predicts online failures.Source: §1 · p. 2
  • The paper identifies salience capture, noun lock-in, and motor inertia as modality shortcuts in VLA learning.Scaling demonstrations that preserve these correlations need not remove them.Source: §1 · p. 3
  • Perturbot uses three independently ablatable data interventions that keep action supervision valid, change what is scaled, and leave inference unchanged.The interventions are task-preserving wrist-view perturbations, behavior-supported decision-relevant captions, and relabeled random-motion and failed-execution segments.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans VLA scaling and evaluation, shortcut learning across modalities, and data-centric robustness interventions. The paper distinguishes its approach by tying shortcuts to the decisions they affect while keeping action labels valid and inference unchanged.

2 RELATED WORK

  • VLA progress tracks growth in demonstration data and studies of data diversity, while evaluations have expanded from nominal benchmarks to perturbed scenes and instructions.GroundFscore also requires actions to change when task-relevant evidence changes, unlike sensitivity to task-preserving perturbations alone, which has been used to select demonstrations (D’urso et al., 2026).Source: §2 · p. 3
  • Prior shortcut-learning work spans dataset bias and language-prior effects in visual question answering, while imitation-learning work shows that action matching need not identify the evidence controlling behavior.Prior VLA studies examine modality use; this paper ties each shortcut to the affected decision and traces all three to regularities in successful demonstrations.Source: §2 · p. 3
  • Prior data-centric robustness methods vary appearance, edit demonstrations, enrich instructions, or widen behavioral coverage, typically targeting one axis of variation.Perturbot targets the cue that substitutes for evidence in a decision, keeps every action label valid, and leaves inference unchanged.Source: §2 · p. 3

3 METHOD

The method frames VLA errors as modality shortcuts and counters them with data interventions plus an offline paired-edit score for evidence use.

3.1 MODELING MODALITY SHORTCUTS

  • Modality shortcuts substitute a predictive cue for the evidence a decision requires; collinearity or an invariant decision can preserve them as demonstrations scale.The examples span wrist-view salience, noun–verb pairings, and gripper closure followed by a lift.Source: §3.1 · pp. 3–4

3.2 PERTURBOT

  • Perturbot combines task-preserving wrist-view distractors, decision-relevant captions, and relabeled random or failed trajectory segments to make shortcuts insufficient and evidence more usable.Segments are labeled for their local behavior, broadening state–action coverage without itself teaching recovery.Source: §3.2 · p. 4

3.3 GROUNDFSCORE

  • GroundFscore pairs null edits that should preserve actions with causal edits targeting a known change, then combines unwanted movement and causal sensitivity so ignoring inputs earns no score.It is computed from paired forward passes on 100 held-out General PnP observations, without rollouts.Source: §3.3 · p. 5, Figure 2 · p. 5
  • Evaluation uses 100 held-out General PnP observations, while each policy also faces the same 50 stored real-robot cases; GroundFscore remains diagnostic, not a competence certificate.The real-robot cases use six objects and three containers.Source: §3.3 · p. 5, Figure 3 · p. 6

4 EXPERIMENTS

The experiments test Perturbot on real-robot pick-and-place and broader bimanual skills, including whether its components improve success and evidence use. The full method improves General PnP and RoboTwin 2.0 results.

4.1 PROTOCOL

  • General PnP uses unseen layouts with six objects and three containers drawn from 100 items, and 50 rule-generated layouts are restored so policies run the same cases.All conditions share initialization, optimizer, steps, and batch size; data recordings and scene groups are split before segmentation, captioning, or augmentation.Source: §4.1 · p. 6

4.2 GENERAL PICK AND PLACE

  • The evaluation covers all eight V/C/R combinations and probes distractor capture and empty-grasp lifting; verb-change noun-lock-in probes run in RoboTwin 2.0.Controls test regularization, data quantity, discarded visual evidence, and inference-time correction.Source: §4.2 · p. 7
  • The full method doubles baseline success to 84% on General PnP, with the best readout in all three GroundFscore families; removing a component costs 16–20 points.Other controls stay within 6 points of baseline; camera removal lowers distractor capture but also loses 16 points of success and lowers GFV to 0.16.Source: §4.2 · p. 7

4.3 GENERALITY ACROSS SKILLS

  • On RoboTwin 2.0, Perturbot improves every Table 2 column; its Clean2Random gain is 12.2 points on randomized scenes versus 2.7 on clean scenes.Removing V costs 8.4 points on randomized scenes in Full and 2.6 in Clean2Random.Source: §4.3 · p. 7, Table 2 · p. 7

5 ABLATION STUDY

The ablations examine data scaling, component specificity, controllability, and whether GroundFscore predicts online failures. Perturbot’s gains in success accompany stronger evidence use, unlike scaling the baseline with more demonstrations.

5.1 DATA SCALING

  • Perturbot gains 16.2 success points per doubling versus 6.0 for fine-tuning; from 1/4× to 2×, its success and GroundFscore rise by 48 points and 0.30.At 1/4×, Perturbot matches fine-tuning at 2× with eight times fewer demonstrations; fine-tuning instead adds 18 success points while GF moves only 0.04.Source: §5.1 · p. 8, Table 3 · p. 8

5.2 COMPONENT SPECIFICITY

  • Budget-matched controls show targeted mechanisms matter: V reaches GFV 0.51 versus at most 0.38, C raises GFL by 0.23 versus at most 0.06, and random plus failed segments lower MI to 44%.Extra successful demonstrations leave MI near baseline, while random placement of distractors drops success below baseline because on-path decoys invalidate the recorded action.Source: §5.2 · pp. 8–9, Table 4 · p. 8

5.4 VALIDITY OF GROUNDFSCORE

  • The validity ablation reports GroundFscore’s correlation with negative online failure rate over 36 checkpoints; stability alone ranks the no-wrist policy above baseline.The ablation tests whether the offline score predicts online failures and whether its terms suffice alone.Source: §5 · p. 7, Table 6 · p. 9

5.3 CONTROLLABILITY

  • Perturbot picks a distractor in 20% of probes versus 68% for fine-tuning, yet selects the instructed object in 90%; it lifts after empty grasps in 24% versus 82%.It lifts secured objects nearly as often as fine-tuning: 92% versus 94%.Source: §5.3 · p. 9, Table 5 · p. 9

6 LIMITATIONS

Perturbot depends on detailed captions and relabeled random and failed trajectory segments.

6 LIMITATIONS

  • Relabeling cannot add behaviors absent from the recorded trajectories, such as recovery.The method also requires random and failed trajectories, in addition to demonstrations, with relabeling matched to recorded behavior.Source: §6 · p. 9

7 CONCLUSION

VLA policies can rely on modality shortcuts even when they succeed, so Perturbot changes the training data while GroundFscore assesses shortcut reliance offline. The conclusion argues that scaling should expand the range of decisions grounded in scene, instruction, and outcome evidence.

7 CONCLUSION

  • Perturbot adds task-preserving wrist-view perturbations, decision-relevant captions, and relabeled random and failed segments without changing inference.Source: §7 · p. 9
  • GroundFscore complements task success by diagnosing offline whether policy improvement relies on task evidence or modality shortcuts.Source: §7 · p. 9
  • The authors argue that scaling robot data should broaden the decisions dependent on scene, instruction, and observed outcome, not only repeat familiar behavior.Source: §7 · p. 9

A EXPERIMENTAL SETUP

The real-robot platform uses two side-by-side AgileX PiPER X arms facing a tabletop workspace, with head and wrist cameras providing views of the scene.

A.1 REAL-ROBOT PLATFORM

  • The platform records head- and wrist-camera streams; the head camera supplies the third-person view and the wrist cameras supply wrist views.Source: §A.1 · p. 18
  • Two AgileX PiPER X arms are mounted side by side and face the tabletop workspace.Source: Figure 5 · p. 18, §A.1 · p. 18
  • A RealSense D435 head camera on the central post provides the third-person view, while a wrist camera beside each gripper provides wrist views.Source: Figure 5 · p. 18, §A.1 · p. 18

A.2 GENERAL PNP EVALUATION

The General PnP real-robot evaluation uses 50 fixed cases and reports success and progress alongside separate probe trials. RoboTwin 2.0 evaluation uses two training settings, while its auxiliary simulated trajectories include random motions and failed executions.

A.2 GENERAL PNP EVALUATION

  • The 50 General PnP cases use fixed object identities, targets, destinations, and instructions, with randomized tabletop positions; layouts are restored for every rollout.Source: §A.2 · p. 18
  • Evaluation reports overall and five-block success plus mean 0–5 progress; separate 50-trial probe lists add no physical-robot task.Intervals are 95% percentile bootstrap half-widths from 10,000 resamples of the evaluation units.Source: §A.2 · p. 18

A.3 ROBOTWIN 2.0

  • RoboTwin 2.0 tests Full and Clean2Random on held-out clean and randomized trials, with Clean2Random testing unseen randomized scenes.Source: §A.3 · p. 18
  • Table 7 lists published RoboTwin 2.0 task-averaged success results from different architectures and recipes; the matched π0.5 comparison is in Table 2.Source: Table 7 · p. 19
  • Simulated auxiliary data use workspace-sweeping random motions and failed expert rollouts with injected grasp or placement perturbations, retaining only failed outcomes.The trajectories are segmented, captioned, and relabeled; Clean2Random uses clean-scene rollouts only.Source: §A.3 · p. 19

B TRAINING DATA

Training combines task demonstrations with auxiliary behavior segments and applies wrist-view perturbations and detailed captions to both sources. The annotation pipeline segments behaviors, relabels them, and describes actions using visible, decision-relevant details.

B TRAINING DATA

  • Task demonstrations supervise the task, while auxiliary recordings supply R and are neither completed tasks nor evaluation cases.Source: B TRAINING DATA · p. 19

B.1 GENERAL PNP DEMONSTRATIONS

  • The demonstration pool contains 1,000 teleoperated examples, with main conditions using a fixed 500-example subset and larger budgets using all 1,000.Source: §B.1 · p. 19

B.2 RANDOM-MOTION AND FAILED-EXECUTION RECORDINGS

  • Auxiliary recordings capture varied contacts through workspace-sweeping random motions and deliberately imperfect interactions, including misses, empty grasps, and failed pushes.Source: §B.2 · p. 20

B.3 SEGMENTATION, RELABELING, AND MIXTURE

  • Auxiliary recordings are split into atomic-skill segments, labeled for local behavior, and mixed into batches at a fixed probability rather than adding optimizer updates.Failed-attempt goals are not retained as labels, and recorded states and actions are not overwritten.Source: §B.3 · p. 20

D WRIST-VIEW PERTURBATION

  • Wrist-view edits add plausible distractors only when the target, spatial relations, grasp path, and contact evidence remain readable; other inputs and the action label stay unchanged.Source: D WRIST-VIEW PERTURBATION · pp. 20–21

E DETAILED-CAPTION ANNOTATION

  • Detailed captions describe the acting hand, operation, position, object state and attributes, object identity, and grasp point using visible, checked evidence.Source: E DETAILED-CAPTION ANNOTATION · p. 21

Prompt Used to Annotate Captions

Prompt Used to Annotate Captions (Continued)

  • The continued prompt specifies unambiguous descriptions using hand state, action, position, object details, and grasp point, and requests timestamped descriptions in a plain JSON array.Source: Prompt Used to Annotate Captions… · p. 22

F ADDITIONAL ABLATION RESULTS

The sampling-rate sweep varies each component around validation-selected settings, while verb-change probes test whether instructions override noun-linked operations. Removing detailed captions restores most noun lock-in.

F.2 VERB-CHANGE PROBES FOR NOUN LOCK-IN

  • 52% of fine-tuning probes perform the familiar operation, versus 36% completing the instructed one; removing C restores 43% lock-in, while removing V or R raises it by at most 7 points.Perturbot reverses the fine-tuning pattern; the authors attribute the gain to making requested operations explicit with verbs already present in the demonstrations.Source: Table 11 · p. 23, §F.2 · p. 23

F.1 SENSITIVITY TO THE SAMPLING RATES

  • Table 12 compares success on 50 real-robot General PnP cases with brief and detailed instructions, defining Δ as detailed minus brief.Brief-instruction values repeat Table 1.Source: Table 12 · p. 23
  • Each sampling rate is varied one at a time around its validation-selected value; Figure 4c plots success change against rate to test shortcut resistance versus useful conditioning.The other two rates remain at their selected values; level 0 disables the component and 1× is the full method.Source: §F.1 · pp. 22–23, Table 10 · p. 23

F.2 VERB-CHANGE PROBES FOR NOUN LOCK-IN

  • The probe tests five object pairs in both directions, with 10 trials per direction and 100 trials per policy on clean scenes.Each pair uses an object that permits both operations from the same initial state; only the verb differs between the restored task and the brief instruction.Source: §F.2 · p. 23

F.3 DETAILED-INSTRUCTION EVALUATION

With detailed captions, detailed instructions add success while brief-instruction success remains higher than every other configuration under either test style. GroundFscore's paired advantage excludes zero across checkpoints, unlike task success's interval.

F.3 DETAILED-INSTRUCTION EVALUATION

  • The primary evaluation uses brief instructions, then replays the same 50 cases with detailed instructions while keeping the rest of the protocol unchanged.The detailed prompts are written from stored case specifications and checked against the initial scene.Source: §F.3 · p. 23
  • Without C, detailed instructions cost 6–10 points; with C, they add 2–6 points.The comparison uses detailed-instruction success relative to brief-instruction success.Source: §F.3 · p. 23
  • Perturbot's brief-instruction success exceeds every other configuration under both brief and detailed test instructions.The result indicates that Perturbot does not depend on detailed instructions at test time.Source: §F.3 · p. 23

F.4 CONFIDENCE INTERVALS FOR THE VALIDITY OF GROUNDFSCORE

  • GroundFscore's interval lower end exceeds every other predictor's point estimate and its paired advantage excludes zero, while task success's interval includes zero.Table 13 reports Spearman ρ with negative online failure rate and 95% cluster-bootstrap intervals over the 36 checkpoints.Source: Table 13 · p. 24, §F.4 · p. 24
  • The 36 checkpoints are not independent because they share evaluation cases and training recipes.They comprise twelve conditions and three seeds.Source: §F.4 · p. 24
  • The reported 95% intervals use 10,000 two-level cluster-bootstrap resamples, with paired predictor differences from shared draws.Resampling draws conditions and seeds, then evaluation units within each checkpoint.Source: §F.4 · p. 24

Related papers

Loading 2610.04616v1…