Source-linked AI summary
When Listening Becomes Easier: Scrubbing Visual Cues for Shortcut-Free VLAs
Jasper Gerigk, Kenzo Aspuru-Takata, Chin-Hsuan Wu, Mohammad Mohammadi, Shuhong Zheng, Igor Gilitschenski
TL;DR
Robot policies can exploit task-correlated visual cues instead of language, and it remains unclear why VLA architectures differ in shortcut susceptibility or how to suppress shortcuts without discarding useful vision. The paper analyzes shortcut representations and introduces task scrubbing, which improves out-of-distribution robustness across simulation and real-robot experiments.
Problem
VLA policies may substitute task-correlated visual cues for language, and the paper identifies a gap in explaining architecture-dependent susceptibility and suppressing shortcuts without losing useful visual information.
Source: §I · p. 1Method
The paper uses action margin and attention knockout to analyze shortcut representations, then applies adversarial task scrubbing to make visual task cues less reliable for action prediction.
Source: §I · p. 2Results
Across LIBERO, RoboTwin, and real-robot experiments, task scrubbing reduces shortcut behavior and improves OOD success.
Source: §VII · p. 7Takeaways & Limitations
Manipulating information routing can prevent visual shortcut learning without requiring visual task information to disappear from the representations.
Source: §VI · p. 7, §VII · p. 8Evidence is scoped to visual shortcut learning in VLAs, with evaluations reported across LIBERO, RoboTwin, and real-robot experiments.
Source: §VII · p. 7
Abstract
from arXiv · showhide
Shortcut learning is a prevalent issue in robot learning. The limited diversity of robot demonstration datasets can mislead policies into exploiting spurious correlations between tasks and irrelevant features, such as viewpoint or background. Collecting sufficiently diverse robot demonstrations is costly and inefficient, motivating algorithmic alternatives. We focus on vision-language-action (VLA) models and discover that different vision-language model backbones exhibit substantially different levels of susceptibility to visual shortcut learning. We find that model behavior correlates with our proposed representation-level metric, action margin, which requires no policy rollouts. Visual shortcuts consistently enter action representations in early layers, with models differing in the extent to which later layers correct them by incorporating language information. To boost models' attention to language, we introduce task scrubbing, a new domain-adversarial training method that decreases models' likelihood of using visual shortcuts and improves VLAs' generalization. Experiments in both simulation and the real world across multiple VLAs and visual cues show that task scrubbing improves out-of-distribution robustness and often eliminates visual shortcut learning.
I. INTRODUCTION
VLAs can exploit task-correlated visual cues instead of language, and existing explanations do not account for why architectures differ in this behavior. The paper analyzes shortcut susceptibility and introduces task scrubbing to strengthen language’s influence on action prediction.
I. INTRODUCTION
- Visual shortcuts let policies substitute task-correlated cues for language, producing cue-following behavior when the cue–instruction association breaks out of distribution.Such cues can arise from camera position, backgrounds, or task setup.Source: §I · p. 1
- The paper examines why VLA architectures trained on the same data differ in shortcut susceptibility and how to suppress shortcuts without discarding useful visual information.Source: §I · p. 1
- Visual cues shift action representations toward cue-consistent behavior in early layers, while VLM backbones differ in whether later computation restores alignment with the instruction.Source: §I · p. 2
- Shortcut susceptibility depends strongly on architecture: variants trained on the same data differ substantially, while backbone size and general performance do not predict robustness.Source: §I · p. 2
- Action margin and attention knockout provide a rollout-free framework for tracking shortcut reliance and analyzing how visual and language information shape action representations.Source: §I · p. 2
- Task scrubbing uses gradient reversal to make task prediction from visual representations harder during training, increasing reliance on language and reducing shortcut learning.Source: §I · p. 2
II. BACKGROUND AND RELATED WORK
VLA robustness depends on how pretrained vision-language representations are adapted for control, while shortcut learning can arise from task-correlated features in limited or fragmented data. Existing mitigations involve trade-offs between data collection, removing inputs, and preserving pretrained knowledge.
A. Vision-Language-Action Models
- VLA policies connect pretrained vision-language representations to actions through either the backbone itself or a separate action expert, but adaptation can trade off action accuracy against generalization.Broad fine-tuning may degrade pretrained structure, while freezing the VLM does not generally improve language generalization and can reduce task performance.Source: A. Vision-Language-Action Models · p. 2
B. Shortcut Learning in Robot Policy Training
- Shortcut learning can emerge when limited or fragmented demonstrations entangle task identity with scenes, cameras, embodiments, or trajectories; more data helps only when it adds relevant variation.Source: B. Shortcut Learning in Robot Policy… · p. 2
- Prior mitigations include broader data collection, removing observation channels, and recovering pretrained knowledge, each with practical or information-retention trade-offs.The paper instead investigates a training modification intended to avoid visual shortcuts without requiring more data or permanently removing information.Source: B. Shortcut Learning in Robot Policy… · p. 2
C. Adversarial Representation Learning
The paper defines visual shortcut learning as relying on a task-correlated but irrelevant visual cue instead of the language instruction. It applies domain-adversarial representation learning to shift task-predictive visual representations so policies rely on instructions, and evaluates this behavior on in-distribution and crossed task–cue combinations.
C. Adversarial Representation Learning
- Domain-adversarial training makes representations from different data sources harder to distinguish by training an auxiliary classifier to recover an attribute while reversing its gradient into the encoder.Here, continually shifting task-predictive visual representations can be sufficient to make policies rely on instructions rather than visual shortcuts.Source: C. Adversarial Representation Learning · pp. 2–3
A. Formal Definition of Shortcut Learning
- Visual shortcut learning occurs when a policy relies on an irrelevant visual cue Z instead of the language-specified task T, despite the cue being statistically associated with the task in training.The intended policy uses task-relevant visual information U and T, whereas a shortcut policy can let Z substitute for the instruction.Source: A. Formal Definition of Shortcut… · p. 3
- In-distribution evaluation retains task–cue pairs seen in training, while out-of-distribution evaluation crosses those pairs and measures success and shortcut rate.Shortcut rate is the percentage of OOD episodes in which the policy attempts the cue-consistent task rather than the instructed one; an episode cannot both succeed and shortcut.Source: A. Formal Definition of Shortcut… · p. 3
B. LIBERO Spatial Viewpoint-Disparity Setup
On LIBERO Spatial, shortcut susceptibility varied substantially across VLA backbones trained on the same data, and action margin tracked shortcut rate across model–disparity combinations. The viewpoint-disparity setup compares task-specific views, while action margin estimates whether crossed OOD representations align with the instructed task or visual cue.
B. LIBERO Spatial Viewpoint-Disparity Setup
- The LIBERO Spatial setup uses two tasks with task-specific viewpoints and progressively larger viewpoint differences in the Low, Mid, and High conditions.A rollout counts as shortcut use when the gripper is closer to the incorrect bowl location at its first close.Source: B. LIBERO Spatial Viewpoint-Disparity… · p. 3
- SmolVLA and SmolVLA-VLM showed no shortcut behavior, while replacing SmolVLA’s backbone generally worsened performance; PaliGemma was most susceptible.SmolVLA achieved nonzero OOD success at all three disparity levels, and Qwen2.5-VL’s high average benchmark performance did not predict the resulting VLA’s shortcut propensity.Source: B. LIBERO Spatial Viewpoint-Disparity… · p. 3, Figure 2 · p. 3
C. Representation-Level Shortcut Magnitude
- Action margin compares crossed-OOD action representations with task- and cue-matched reference representations to estimate whether actions follow the instruction or visual cue without rollouts.Positive values indicate closer alignment to the cue-matched centroid; negative values indicate closer alignment to the instructed task.Source: C. Representation-Level Shortcut… · p. 4
- Across 21 model–viewpoint-disparity combinations, action margin strongly correlated with shortcut rate, indicating it tracks shortcut reliance across architectures.The model-level value averages crossed configurations’ first observations with flow-matching noise fixed to avoid confounding from divergent rollout states.Source: Figure 3 · p. 3, C. Representation-Level Shortcut… · p. 4
D. Mechanistic Analysis
Attention knockouts show that visual information shifts action representations toward visual cues in early layers, while language and action-token interactions can correct that alignment later. The correction varies by model, motivating training methods that increase language’s relative influence.
D. Mechanistic Analysis
- Visual information injects shortcut alignment in early layers, while later language and action-token interactions can steer representations back toward the instructed task, with correction varying by model.Among the tested models, SmolVLA most effectively reverses early visual alignment through language features.Source: D. Mechanistic Analysis · p. 4
- Language share measures the relative steering of action representations along the task axis by language versus visual cues, using counterfactual instruction and cue contrasts.It is their normalized balance, rising toward 1 as language assumes control and near 0 for a pure-shortcut policy; the metric is computed on first trajectory frames under fixed flow-matching noise.Source: D. Mechanistic Analysis · pp. 4–5
- Task scrubbing uses an adversarial task classifier to make visual features less informative about task identity, making language a more reliable task-identifying signal during policy training.The classifier learns to recover task identity while the visual pathway learns to frustrate it.Source: Figure 5 · p. 5
IV. TASK SCRUBBING
Task scrubbing makes task identity less stable in visual representations so language becomes the more reliable signal for action selection. An adversarial task classifier encourages this shift without aiming to erase all task-decodable visual information.
IV. TASK SCRUBBING
- Task scrubbing destabilizes visually encoded task identity so task-irrelevant cues are less convenient bases for action selection.The goal is not to remove all task-decodable visual information, which may be infeasible when task views differ strongly.Source: §IV · p. 5
- A classifier predicts task identity from mean-pooled projected visual tokens, while gradient reversal makes the visual pathway frustrate that prediction.The classifier operates before the language-model or action backbone and is trained jointly with the policy.Source: §IV · p. 5
- When the visual encoder is frozen, SmolVLA uses a trainable residual adapter on the visual projection so reversed gradients can update the pathway.The adapter adds negligible computational overhead to training.Source: §IV · p. 5
V. RESULTS
On high-disparity LIBERO, task scrubbing improves out-of-distribution (OOD) success and reduces shortcut behavior across most evaluated VLA families. The π0.5 exception shows that the response can vary across seeds and may trade off in-distribution performance.
A. Task Scrubbing Improves LIBERO Robustness
- For π0, OOD success rises from 0% to 48% while shortcut rate (SC) falls from 99% to 0%, with no meaningful reduction in ID success.The high-viewpoint-disparity LIBERO comparison reports ID success, OOD success, and SC for the baseline and task-scrubbed policies.Source: A. Task Scrubbing Improves LIBERO… · p. 5, Table I · p. 6
- Across π0, the SmolVLA family, and MiniVLA, task scrubbing reduces SC and increases OOD success.The SmolVLA variants show the strongest overall response: observed shortcut behavior is eliminated and ID performance increases.Source: A. Task Scrubbing Improves LIBERO… · pp. 5–6
- For π0.5, mean SC falls and mean OOD success rises, but only two of three seeds succeed; the remaining seed retains approximately 90% SC.Mean ID success also falls from 66% to 45%.Source: A. Task Scrubbing Improves LIBERO… · p. 6
B. Task Scrubbing Alters the Internal Information Pathway
Task scrubbing changes which information pathway is easiest for action selection rather than permanently removing task information from images. The intervention also lowers shortcut rates and boosts OOD success in RoboTwin, though some models lose ID performance.
B. Task Scrubbing Alters the Internal Information Pathway
- Task scrubbing makes visual task signals less stable for action learning, while language remains a direct and stable source of task identity.Task identity can still be decoded by linear probes after training, so the method changes pathway reliance rather than deleting image information.Source: B. Task Scrubbing Alters the Internal… · p. 6
- With scrubbing, language share rises sharply during training, whereas without it language share does not increase sufficiently or may decrease.Action pretraining raises initial language share in both SmolVLA and π0 comparisons, while pretrained models respond more slowly to scrubbing.Source: B. Task Scrubbing Alters the Internal… · p. 6
C. RoboTwin
- In RoboTwin Place-Block, task scrubbing decreases SC and boosts OOD success across tested models, although models more often lose ID performance.MiniVLA was not evaluated because its ID performance was too low for a meaningful comparison; SmolVLA-VLM had one seed significantly worse than the other two.Source: C. RoboTwin · p. 6, Table II · p. 6
D. Real-Robot Experiments
On the real ALOHA robot, task scrubbing improves both tested models under cup-color and combined color-background cues. Directly penalizing action-to-image attention can raise language share but does not match task scrubbing on high-disparity LIBERO, especially for the π family.
D. Real-Robot Experiments
- On a physical ALOHA platform, the evaluation tests language-instructed cup placement to a left or right target under two visual shortcut conditions: cup color alone, or cup color plus background appearance.In the second condition, purple-left demonstrations include a blue towel, while green-right demonstrations use the normal table background.Source: D. Real-Robot Experiments · pp. 6–7
- With combined cup-color and background cues, task scrubbing eliminates observed shortcut episodes and yields OOD success above 90% for both real-robot models.With cup color alone, little shortcut learning occurs, but OOD success still benefits from task scrubbing.Source: D. Real-Robot Experiments · p. 7, Table III · p. 7
E. Ablation: Direct Attention Regularization
- Attention regularization drives language share above 90% in all evaluated models, but trails task scrubbing in success rates and leaves π0 at 68% SC and π0.5 at 79% SC.It works for SmolVLA variants but fails for the π family on high-disparity LIBERO.Source: E. Ablation: Direct Attention… · p. 7, Table IV · p. 7
- Forcing high language attention does not guarantee language controls action because attention amounts incompletely measure information flow.The regularizer penalizes action-to-image attention across cross-modal heads and layers.Source: E. Ablation: Direct Attention… · p. 7
VI. DISCUSSION
VLA backbones differ in shortcut susceptibility, while task scrubbing shifts action computation toward language without removing decodable visual task information. Across simulation and real-robot experiments, it reduces shortcut behavior and improves OOD success, with limitations in scope and consistency.
VI. DISCUSSION
- Models trained on the same demonstrations can differ substantially in shortcut behavior, and replacing SmolVLA’s backbone can change its robustness.The authors argue shortcut learning should not be treated solely as a dataset property; architecture, pretraining, and later language use also matter.Source: §VI · p. 7
- Task scrubbing leaves visual task identity decodable but makes it unstable during training, increasing language’s influence on action computation.Robustness depends on whether visual information dominates action computation and whether later layers use language to restore task alignment.Source: §VI · p. 7
- Action margin can help identify cue-following backbones, while language share may monitor task scrubbing and inform auxiliary-signal adjustment.These diagnostics are proposed for model selection and training monitoring under controlled crossings.Source: §VI · p. 7
VII. CONCLUSION AND LIMITATIONS
- Across LIBERO, RoboTwin, and real-robot experiments, task scrubbing reduces shortcut behavior and improves OOD success.The authors conclude that manipulating information routing is effective against visual shortcut learning in VLAs.Source: §VII · pp. 7–8
- Task scrubbing requires task labels and trainable visual-pathway parameters, and its evaluations cover paired tasks rather than multitask settings.Frozen architectures require an adapter; some seeds retain shortcut behavior, and ID performance can decrease.Source: §VII · p. 8
APPENDIX
The appendix describes how task-scrub scales were selected for each architecture and task and identifies the table listing the selected values.
APPENDIX
- Task-scrub scale was selected separately for each architecture and task.Selection used a single seed, aiming to reduce SC and maximize OOD success without reducing ID success when possible.Source: APPENDIX · p. 8
- A single seed was used to choose a scale that reduces SC and maximizes OOD success without reducing ID success when possible.Source: APPENDIX · p. 8
- Table V lists the selected auxiliary-loss scales for task scrubbing.Source: Table V · p. 8