Source-linked AI summary
What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies
Vivek Chavan, Pengtao Xie, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger
TL;DR
Visuomotor policies can select the wrong object or destination when visually similar distractors appear, even while retaining manipulation skill. The paper diagnoses these conditional grounding failures across ACT and a pretrained VLA policy, then evaluates interventions that improve target selection while preserving control-relevant geometry. The interventions recover robustness in simulation, on a physical UR3e, and in a state-conditioned VLA routing task.
Problem
Visuomotor imitation policies can fail when visually similar objects or receptacles are introduced, and aggregate success does not distinguish incorrect selection from lost manipulation ability.
Method
The paper uses controlled ACT distractor experiments, diagnosis-driven augmentation, phase-dependent attention regularization, appearance-based prompting, and a pretrained-VLA state-conditioned routing case study.
Results
The interventions substantially improve robustness across simulation, a physical UR3e, and the VLA routing task, while analyses indicate that preserving task-relevant spatial structure is important.
Takeaways & Limitations
Visual distractors can cause incorrect context-dependent selection while motor behavior remains intact, so explicitly improving target selection can recover performance across visuomotor policy regimes.
Takeaways & Limitations
The conclusions are bounded by asset-specific color-shape effects, correlational attention and representation analyses, small hardware and VLA evaluations, and nonequivalent VLA guidance conditions.
Abstract
from arXiv · showhide
Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state. Using Action Chunking with Transformers (ACT), we systematically introduce distractor objects and receptacles with controlled color and shape similarity and localize failures to picking and placement. We find that distractor sensitivity is specific to both the type of visual similarity and the manipulation stage. Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting as complementary interventions for improving target selection while preserving spatial information required for control. These interventions substantially improve robustness in simulation and on a physical UR3e. We further examine the same failure pattern in a pretrained vision-language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination. Together, the results show that visual distractors can cause incorrect object or destination selection even when the underlying manipulation skill remains intact, and that explicitly improving target selection can substantially recover performance across distinct visuomotor policy-learning regimes.
I. INTRODUCTION
The paper frames distractor-induced failures as conditional visual grounding errors: the correct referent changes with manipulation phase and, in VLA tasks, observed state. It localizes failures to target selection and evaluates diagnosis-driven interventions.
- Conditional visual grounding requires selecting a context-dependent referent from candidate objects or receptacles.
- In ACT, the object matters during target acquisition and grasping, while the receptacle matters during placement.
- In the VLA case study, the medical instrument’s observed state determines the correct destination.
- The study asks whether failures can be localized by cue, referent, and execution stage, and whether that diagnosis supports effective interventions.
II. RELATED WORK
Prior work documents visual-distractor sensitivity in imitation and vision-language-action policies, while proposing semantic, region-aware, masking, prompting, and runtime observation interventions. This paper complements those approaches by jointly diagnosing the cue, referent, and stage of grounding failure.
- Visuomotor policies can exploit irrelevant distractor correlations and fail under changes in appearance, scene configuration, and visual clutter.
- Existing interventions improve robustness through semantic guidance, task-relevant region-aware representations, grounding masks, and visual prompts.
- Pretrained VLA policies remain sensitive to task-irrelevant visual content, motivating runtime observation interventions and supervision of task-relevant action factors.
- The paper focuses on identifying which visual cues induce incorrect grounding, which referent is affected, and when the error occurs.
III. METHOD
The method combines controlled distractor experiments in ACT with diagnosis-driven augmentation, phase-dependent attention regularization, and appearance-based prompts, then tests the perspective in a pretrained VLA policy.
- Controlled diagnosis with ACT: Two simulated pick-and-place tasks separate target and destination grounding, using clean demonstrations and competitors matched by color, shape, or neither.
- Diagnosis-driven intervention: Image-space copy-paste augmentation inserts synthetic competitors to reduce reliance on incidental appearance correlations.
- Diagnosis-driven intervention: Phase-dependent attention regularization directs a decoder attention head toward the phase-relevant referent and away from synthetic distractors.
- Diagnosis-driven intervention: Appearance-based prompts provide positionless crops of the target object and receptacle, with a jointly trained phase predictor selecting the prompt.
- Pretrained-VLA case study: The VLA case study fine-tunes π0.5 on 231 teleoperated episodes and evaluates state-dependent routing among five subgoals.
IV. EXPERIMENTS AND RESULTS
ACT failures are stage- and cue-specific: distractors mainly disrupt object or receptacle selection while downstream manipulation remains effective after correct selection. Augmentation, attention regularization, and prompting improve robustness while preserving task geometry, with preliminary analogous evidence in π0.5.
- ACT failures are cue-specific and stage-localized: Under three color-matched object competitors, P(pick) falls to 39.2%, while P(lift | pick) and P(place | pick, lift) remain 93.8% and 95.5%.This contrasts with shape-matched receptacle competitors, which leave P(pick) at 97.5% but reduce conditional placement to 33.9%.
- ACT failures are cue-specific and stage-localized: Under full mixed distractors, end-to-end success falls to 39.5% on Task 1 and 14.0% on Task 2, while preserved downstream conditional success points to incorrect grounding.
- Object-centric interventions recover robustness: Augmentation alone reaches 100.0% on Task 1 and 64.0% on Task 2 without explicit target prompts, while ACT-Modified reaches 94.5% and 88.5%, respectively.
- Object-centric interventions recover robustness: On hardware under mixed distractors, standard ACT fails all tested trials, whereas ACT-Modified succeeds in 13/20 and 12/20 trials on Tasks 1 and 2.
- Robust representations require invariance without loss of task geometry: Standard ACT retains only 13.1% of clean pick/place separation under distractors, whereas robust variants suppress nuisance variation while preserving relevant spatial structure.
- Direct prompt intervention: Replacing the target prompt with a distractor prompt increases distractor selection from 0-1% to 35-91%, showing that prompt identity can redirect closed-loop selection.
- The VLA case study provides preliminary cross-regime evidence: The prompt-trained π0.5 checkpoint improves from 20/25 to 25/25 overall and from 5/10 to 10/10 on ambiguous routing when its RGB cue is restored.The regularized clean-input checkpoint also reaches 25/25 overall and 10/10 on routing.
V. DISCUSSION AND OUTLOOK
Conditional visual grounding differs across ACT and VLA settings: manipulation phase selects the relevant referent in ACT, while instrument state selects the destination in the VLA task. Robust placement requires suppressing distractor variation while preserving task-relevant geometry, but the conclusions are bounded by limited trials, correlational analyses, and task-specific conditions.
- In ACT, manipulation phase determines whether the object or receptacle is relevant, whereas the VLA task uses instrument state to determine the destination.
- Robust placement requires both clean/full distractor invariance and retained structure encoding the target container’s position.Augmentation-only ACT shows the strongest clean/full invariance, while ACT-Modified preserves clearer container-position structure; t-SNE projections are illustrative.
- Across both settings, competent motor behavior can coexist with incorrect selection of a context-dependent referent, motivating a common framework for localizing visual failures.
- The conclusions are limited by asset-specific color-shape effects, incomplete visual-factor counterbalancing, correlational analyses, and small hardware and VLA evaluations.The hardware evaluation has 20 trials per cell, while the exploratory VLA study has five trials per subgoal and nonequivalent guidance conditions.