Source-linked AI summary
ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation
Jinxuan Zhu, Jiaheng Wang, Chao Tang, Mengfan Wang, Hao Wei, Shengbao Li, Hong Yin, Yiwen Gao, Chenrui Tie, Tingguang Li
TL;DR
High-precision manipulation can be difficult for VLA policies whose attention is dispersed across task-irrelevant regions. ActGaze derives gaze supervision from counterfactual visual interventions on action prediction, and improves task success and visual attention across four real-world tasks.
Problem
VLA policies may attend to task-irrelevant regions, while standard action training does not directly constrain where they attend during high-precision manipulation.
Source: §I · p. 1, A. Spatial Guidance in VLA Policies · p. 2Method
ActGaze uses counterfactual visual interventions to derive action-grounded spatial supervision and transfers the learned gaze back to refine VLA attention without external annotations or grounding models.
Source: §I · pp. 1–2Results
Across four real-world sub-millimeter-precision tasks, ActGaze improves average success rate from 77.0% to 91.5% and produces more task-relevant visual attention.
Source: §I · p. 2Takeaways & Limitations
Experiments support the practical effectiveness of action-grounded spatial guidance for high-precision manipulation without external annotation or grounding-model supervision.
Source: §VI · p. 7Gaze learning is constrained by the initial policy’s knowledge and biases, which may persist across refinement rounds and limit further performance gains.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Current Vision-Language-Action (VLA) models often struggle with high-precision robotic manipulation. We attribute this limitation primarily to their visual attention being dispersed across task-irrelevant regions. To address this issue, we propose ActGaze, a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements. Unlike prior methods that rely on external labels for gaze supervision, ActGaze derives spatial supervision directly from the VLA's own action objective by using counterfactual visual interventions to identify regions that are critical for action prediction. Extensive real-robot experiments on four high-precision robotic manipulation tasks demonstrate that ActGaze induces more focused visual attention on task-relevant regions and consistently outperforms the base VLA policy and other visual-grounding approaches.
I. INTRODUCTION
ActGaze addresses dispersed visual attention in high-precision manipulation by deriving spatial gaze supervision from changes in action prediction under counterfactual visual interventions. It avoids external spatial annotations and grounding models while targeting visual regions relevant to action.
I. INTRODUCTION
- Standard end-to-end VLA training supervises actions but does not directly constrain attention, while high-precision tasks may depend on a small subset of visual regions.The paper identifies attention to background clutter and visual distractors as a major failure mode.Source: §I · p. 1
A. Spatial Guidance in VLA Policies
- External gaze, mask, keypoint, or model-generated supervision can guide attention, but requires additional effort to obtain spatial labels.Other approaches use policy internals or execution history as relevance proxies, which may not directly reflect a region’s influence on action prediction.Source: §I · p. 1, A. Spatial Guidance in VLA Policies · p. 2
I. INTRODUCTION
- Across four real-world sub-millimeter-precision tasks, average success rate rises from 77.0% to 91.5%, exceeding CoT Grounding and Implicit Grounding by 32.5% and 25.0%, respectively.ActGaze also produces more task-relevant visual attention without the external spatial supervision used by those baselines.Source: §I · p. 2
- ActGaze converts intervention-induced action-prediction changes into spatial supervision without external annotations or grounding models.The approach probes camera views and image regions, then transfers the learned gaze back to the VLA to guide its action attention.Source: §I · pp. 1–2
A. Spatial Guidance in VLA Policies
- ActGaze estimates a region’s action relevance from intervention-induced changes in action prediction, providing explicit spatial guidance without external labels.This contrasts with approaches that use internal representations, attention patterns, or historical context as proxies for relevance.Source: A. Spatial Guidance in VLA Policies · p. 2
B. Interventional Analysis of Action Policies
ActGaze turns intervention-based estimates of visual-region relevance into spatial gaze supervision for VLA policies. Its three-stage pipeline initializes an action-capable policy, distills relevance into a gaze head, and transfers that guidance back to refine policy attention.
B. Interventional Analysis of Action Policies
- Interventional analysis has been used to interpret vision models and learned policies, including perturbation-based policy saliency.ActGaze uses the intervention signal for training rather than post hoc analysis alone.Source: B. Interventional Analysis of Action… · p. 2
- Unlike prior intervention studies that analyze or improve policies, ActGaze converts intervention-derived action relevance into gaze supervision for on-the-fly learning and VLA performance improvement.Prior work includes interventional visual-action attribution [8] and analysis of sensitivity to task-irrelevant regions [22].Source: B. Interventional Analysis of Action… · p. 2
III. PROBLEM FORMULATION
- The gaze head estimates a distribution over visual patches to capture action-relevant evidence, with joint normalization allowing relevance to be allocated across camera views.The goal is to use this spatial guidance to improve VLA grounding for high-precision manipulation.Source: §III · p. 2
IV. METHODOLOGY
- ActGaze initializes the VLA on demonstrations, distills intervention-derived spatial targets into a multi-view gaze head, then transfers gaze back to refine attention while preserving the action objective.The three stages are policy initialization, action-to-gaze distillation, and gaze-to-action refinement.Source: §IV · pp. 2–3, Figure 2 · p. 3
A. Policy Initialization
ActGaze first trains and freezes an action-capable VLA, then derives gaze supervision by intervening on visual evidence and observing changes in action prediction. The intervention signal is evaluated hierarchically across camera views and image regions.
A. Policy Initialization
- The VLA is trained on action demonstrations until validation action loss falls below an empirical threshold, then frozen for gaze learning.Behavioral competence does not guarantee correct visual grounding; a policy may rely on correlated but non-essential cues and fail under small pose, viewpoint, or distraction changes.Source: A. Policy Initialization · p. 3
B. Action-to-Gaze Distillation
- With the VLA frozen, ActGaze measures how controlled visual perturbations affect action prediction and distills the resulting relevance signals into a gaze head.The gaze head predicts a task-relevant visual distribution from the original observations.Source: B. Action-to-Gaze Distillation · p. 3
1) Counterfactual Visual Intervention:
- A region is treated as more action-relevant when intervening on it causes greater degradation in action prediction, with non-visual inputs and prediction conditions held constant.The hierarchical interventions mask whole camera views and blur candidate patch-aligned regions; camera-level relevance weights patch-level gaze relevance.Source: 1) Counterfactual Visual Intervention · p. 4
2) Gaze Learning:
ActGaze distills sparse intervention responses into a gaze head that predicts relevance from clean observations, then aligns VLA attention with the frozen gaze prediction while retaining action optimization. The experiments are framed around gaze transfer, manipulation performance, and design choices.
2) Gaze Learning:
- The gaze head learns a relative distribution over candidate regions by matching its clean-observation predictions to intervention-derived relevance, with camera-level relevance weighting supervision across views.Sampling a small candidate set avoids exhaustive patch-wise interventions and supports on-the-fly supervision.Source: 2) Gaze Learning · p. 4
C. Gaze-to-Action Refinement
- Gaze-to-action refinement jointly optimizes the action objective and an attention-alignment term, encouraging VLA attention to follow the learned gaze while preserving action prediction.The gaze head is frozen during refinement, and λ_align controls the strength of alignment.Source: C. Gaze-to-Action Refinement · p. 4
V. EXPERIMENT
- The experiments ask whether intervention-derived gaze transfers to the VLA, improves real-world high-precision manipulation against base and grounding policies, and whether design choices contribute to performance.These are posed as three research questions covering gaze learning and transfer, task performance, and ablations.Source: §V · p. 4
A. Experimental Setup
ActGaze is evaluated on four precision-demanding real-world tasks, with spatial alignment measured and transfer of learned gaze to policy attention assessed. The results show improved alignment over the base policy and qualitative transfer of task-relevant focus.
A. Experimental Setup
- The evaluation covers Straw Insertion, Hose Insertion, LEGO Assembly, and Connector Plug-in, spanning rigid insertion, deformable manipulation, assembly, and multi-pin alignment.The tasks require millimeter- to sub-millimeter-scale geometric precision.Source: A. Experimental Setup · p. 4, Figure 3 · p. 5
- All methods use the same 200 expert demonstrations, batch size, and training steps, and are evaluated in the same session to reduce environmental and robot-condition variation.ActGaze uses π0.5 as its base policy and runs on a Rainbow RB-Y1 robot with two wrist cameras and one head-mounted camera.Source: A. Experimental Setup · p. 5
B. Action-Grounded Gaze Learning and Transfer (Q1)
- Ground-truth alignment is evaluated using annotated task-relevant regions from 200 sparsely sampled frames per camera view; annotations are withheld during ActGaze training.The Ground-Truth Alignment Score measures probability mass assigned to annotated relevant regions.Source: B. Action-Grounded Gaze Learning and… · p. 5
- 40.3% for learned gaze and 35.7% after refinement, versus 16.3% for base π0.5 attention on average Ground-Truth Alignment Score.The learned gaze improves alignment across all four tasks, and refined policy attention exceeds twice the base policy score.Source: Table I · p. 5, B. Action-Grounded Gaze Learning and… · pp. 5–6
- The three evaluated maps are base-policy action-to-vision attention, the gaze-head prediction, and action attention after gaze-to-action refinement.Qualitative maps show learned gaze concentrating on task-relevant regions and refined policy attention shifting toward similar regions.Source: B. Action-Grounded Gaze Learning and… · pp. 5–6, Figure 4 · p. 6
- Table II compares real-world success rates across the high-precision tasks against base policy π0.5.The supplied caption identifies the comparison scope but provides no success-rate values.Source: Table II · p. 6
C. Real-World High-Precision Manipulation (Q2)
In real-world comparisons, ActGaze improves success over the base policy and two visual-grounding paradigms. Its action-conditioned guidance differs from semantic object-level grounding.
C. Real-World High-Precision Manipulation (Q2)
- On Hose Insertion and Connector Plug-in, ActGaze averages 87.5%, versus 55.0% for CoT Grounding and 62.5% for Implicit Grounding.Table III presents real-world success rates for two representative tasks.Source: Table III · p. 6, C. Real-World High-Precision… · p. 6
- ActGaze raises average success from 77.0% to 91.5% across all four high-precision tasks.The largest reported task-level increase is on Hose insertion, from 68.0% to 92.0%.Source: C. Real-World High-Precision… · p. 6
- For a controlled comparison, CoT Grounding predicts task-region boxes and Implicit Grounding reconstructs task-region crops; both use the same backbone, 200 demonstrations, and Grounding-DINO labels.ActGaze derives spatial guidance from action demonstrations rather than external spatial supervision.Source: C. Real-World High-Precision… · p. 6
- External grounding supplies semantic object-level relevance, while ActGaze identifies action-conditioned, phase-dependent cues such as openings or mating interfaces.The relevant region can shift from the manipulated object before grasping to the target interface during alignment and insertion.Source: C. Real-World High-Precision… · p. 7
D. Ablation Study of Key Design Choices (Q3)
Ablations indicate that counterfactual supervision and staged training support gaze learning, while later refinement rounds add only marginal gains. The conclusion notes that guidance remains bounded by initial-policy biases.
D. Ablation Study of Key Design Choices (Q3)
- Without counterfactual supervision, gaze becomes nearly uniform and average success falls to 37.5%, versus 92.5% for ActGaze on Straw Insertion and LEGO Assembly.The ablation attributes weak gaze gradients to limited action-loss variation and reports that aligning to the uniform gaze may impair action prediction.Source: D. Ablation Study of Key Design… · p. 7
- In LEGO Assembly, success rises from 78% at Round 0 to 90% at Rounds 1–2 and 92% at Rounds 3–4, with most SGT gains in round one.The authors use one refinement round to balance the gains against computational cost.Source: Figure 5 · p. 7, D. Ablation Study of Key Design… · p. 7
- Joint training from the beginning yields 62.5% average success, versus 92.5% with staged training on Straw Insertion and LEGO Assembly.The authors report that early unreliable action predictions make counterfactual supervision unstable; Stage I supplies a stable reference, though convergence does not guarantee strong real-world performance.Source: D. Ablation Study of Key Design… · p. 7
VI. CONCLUSION AND LIMITATIONS
- ActGaze converts intervention-induced changes in action prediction into spatial supervision and improves attention and task performance without external annotations or grounding models.The framework is described as potentially applicable to a broad class of visuomotor policies.Source: §VI · p. 7
- Visual interventions estimate relevance rather than physical causality; perturbation-induced distribution shifts and initial-policy biases can constrain guidance and further gains.The authors state that biases may persist across refinement rounds.Source: §VI · p. 8
APPENDIX
The appendix adds theoretical motivation for counterfactual visual intervention and supplementary annotation, task-specification, and robustness material. It interprets intervention responses as policy-level estimates of relative predictive relevance, not physical causal effects.
A. Theoretical Analysis of Counterfactual Visual Intervention
- Under squared-error loss, V(S) is nonnegative and increases when full-view and region-removed optimal predictions differ.If removing S leaves the optimal prediction unchanged, V(S) is zero.Source: A. Theoretical Analysis of… · p. 8
- A region’s relevance depends on the remaining visual context, since other regions or camera views may compensate for its removal.Thus, relevance is not an intrinsic property of the region alone.Source: A. Theoretical Analysis of… · p. 8
- ActGaze approximates ideal information removal by suppressing a region and comparing action losses under a fixed initialized policy.A larger intervention response suggests stronger reliance on that visual evidence for predicting the demonstrated action.Source: A. Theoretical Analysis of… · p. 8
- The practical score is not an unbiased estimate of ideal predictive value or a physical causal effect; masking or blurring may shift distributions, and estimates inherit initialized-policy biases.The paper treats the score as relative policy-level relevance and reports that empirical improvements in visual alignment and manipulation performance support its practical utility.Source: A. Theoretical Analysis of… · p. 9
B. Ground-Truth Region Annotation
Four annotators followed a shared protocol to mark localized visual evidence for each manipulation step. The annotations serve as evaluation references and are not used to train ActGaze.
B. Ground-Truth Region Annotation
- Ground-truth task-relevant regions are used only to evaluate spatial alignment and are never exposed to ActGaze during training.They provide the reference regions for the Ground-Truth Alignment Score SGT.Source: B. Ground-Truth Region Annotation · p. 9
- Annotators marked localized evidence for each frame and camera view rather than broadly covering an object or workspace.Four annotators used a shared protocol.Source: B. Ground-Truth Region Annotation · p. 9
- Annotations shift by phase, from the object or grasping area during acquisition to the target opening or mating interface during alignment and insertion.Views without useful evidence for the current action are left unannotated.Source: B. Ground-Truth Region Annotation · p. 9
C. Task Specifications
The evaluation spans four precision-demanding tasks with varied object properties, geometric tolerances, and spatial randomization.
C. Task Specifications
- Together, the four tasks cover rigid insertion, deformable-object manipulation, assembly, and multi-pin alignment.They test varying geometric precision, object properties, and spatial randomization.Source: C. Task Specifications · p. 9
- Straw insertion uses an 11.8 mm straw and a 13.8 mm cup opening, with the cup randomized within a 10 cm × 10 cm workspace.Source: C. Task Specifications · p. 9
- Hose insertion uses a 4.6 mm compliant hose and a 5.0 mm opening, with up to 10 cm translation and 15° rotation of the opening base.Hose deformation adds geometric variation during manipulation.Source: C. Task Specifications · p. 9
- LEGO assembly fits an eight-hole brick onto a mating brick, with randomized object positions and rotations.The yellow brick varies within a 5 cm × 5 cm workspace and up to ±30° rotation; the orange brick varies within 15 cm × 15 cm.Source: C. Task Specifications · p. 9
- Connector plug-in aligns a 24-pin ATX connector with a socket at clearance within 0.4 mm, while randomizing motherboard and connector positions.The motherboard shifts approximately 3 cm, and the connector varies over a 10 cm horizontal range.Source: C. Task Specifications · p. 9
D. Robustness to Visual Distractors
On LEGO assembly, ActGaze improved success under visual clutter but showed no clear improvement under strong lighting variation. The reported robustness benefit was more pronounced for spatial distractors than global appearance changes.
D. Robustness to Visual Distractors
- Under visual clutter, ActGaze raised LEGO Assembly success from 13/20 to 15/20; both methods performed similarly in the clean setting, suggesting task-relevant visual guidance reduces sensitivity to nearby task-irrelevant content.Source: D. Robustness to Visual Distractors · p. 10
- 11/20 for both methods: ActGaze showed no clear improvement under strong lighting variation, which affected the entire observation, including task-relevant regions.The reported robustness benefit was more pronounced for spatial distractors than for global appearance changes.Source: D. Robustness to Visual Distractors · p. 10