GIFT: Guided Intermediate Feature Training via Action-Oriented Structural Supervision for Robotic Manipulation

1 INTRODUCTION

• Action-only supervision and pixel-focused visual prediction may leave intermediate features without metric relations, object-centric interaction structure, or instruction-conditioned targets.Action-only objectives can reward background or camera-artifact shortcuts, while visually plausible forecasts can retain task-irrelevant detail and omit 3D or object-semantic structure.Source: §1 · pp. 1–2
• GIFT improves zero-shot robustness across seven LIBERO-Plus shifts, with GIFT-VLA and GIFT-WAM-Fast outperforming their matched baselines across shifts and on average.The introduction also reports gains for all three configurations under LIBERO-Plus transfer, with pronounced advantages for WAM variants on RoboCasa articulated-object tasks and robustness to unseen visual and spatial perturbations.Source: Figure 1 · p. 1, §1 · p. 2
• GIFT trains intermediate features to preserve geometry for motion feasibility, affordances for instruction-relevant entities and interactions, and goals for task-relevant regions.It operationalizes these structures through geometry alignment, affordance prediction, and goal-region reconstruction.Source: §1 · p. 2
• The same guidance is instantiated in a direct-action VLA, a direct-action diffusion WAM, and a future-conditioned inverse-dynamics WAM while retaining model-specific action formulations.The study compares injection and no-injection designs to distinguish feature learning from supplying auxiliary predictions as action conditions.Source: §1 · p. 2
• Across LIBERO and RoboCasa, the configurations retain competitive in-distribution performance; ablations report complementary effects from the three guidance signals and better performance without injection.Attention visualizations show more consistent focus on task-relevant objects and interaction regions.Source: §1 · p. 2

2.1 Vision-Language-Action Policies and Structured Guidance

• VLA approaches add geometric representations, affordance and goal cues, reasoning traces, or specialized heads, while action-only objectives do not specify which spatial, interaction, and task evidence features should preserve.The section also describes geometry-guided feature alignment and reconstruction of instruction-relevant regions.Source: §2.1 · p. 2

2.2 World-Action Models and Predictive Representation Learning

• WAMs use future observations, latent states, or video prediction for dynamic supervision, but pixel-oriented RGB latents can overrepresent appearance and omit structure needed for action.Related approaches add geometric, semantic, object-centric, or action-conditioning targets; coarse latent actions may still lack detail for precise control.Source: §2.2 · p. 3

2.3 Distinction between GIFT and Prior Work

• GIFT supervises shared intermediate visual features with geometry, affordance, and goal objectives rather than specializing decoder heads or requiring auxiliary predictions as action inputs.The paper distinguishes this approach from prior guidance interfaces and reports evaluation across a VLA, direct-action WAM, and inverse-dynamics WAM with matched controls.Source: §2.3 · p. 3

• A manipulation sample combines multi-view observations, a language instruction, proprioceptive state, and a future action chunk; native action conditions also include future-video context for inverse dynamics.Source: §3.1 · p. 3
• GIFT guides intermediate features to encode geometry for motion feasibility, affordances for instruction-relevant entities and object–end-effector relations, and goals for task-relevant regions.Source: §3.1 · p. 3, Figure 2 · p. 4
• Geometry uses shallow backbone features at layer 6, while affordance and goal prediction use final-layer features and lightweight heads trained on current-frame visual tokens.Source: §3.1 · p. 3
• The auxiliary predictions are excluded from action inputs and shape shared features through backpropagated losses, leaving action generation model-specific.Source: §3.1 · p. 3

3.2.1 Geometry Guidance

• Geometry guidance aligns student visual tokens with frozen VGGT features by separately matching feature direction and magnitude.Teacher and student grids are resampled to corresponding patch locations; the teacher is used only to construct training targets, not at deployment.Source: §3.2.1 · pp. 4–5

3.2.2 Affordance Guidance

• Affordance guidance predicts instruction-relevant entity roles and object-centric end-effector configurations, encoding what to interact with and how.The first object interacted with in the demonstration anchors the representation, which reduces dependence on the global camera frame.Source: §3.2.2 · p. 5

3.2.3 Goal Guidance

• Goal guidance predicts an instruction-conditioned binary mask identifying the image-space region where the intended interaction should occur.The mask is the sole goal target; together, goal and affordance guidance distinguish where interaction occurs from how relevant entities should be interacted with.Source: §3.2.3 · p. 5

3.3 Instantiation Across Policy Paradigms

• All three policy paradigms apply geometry, affordance, and goal objectives to intermediate visual tokens without assuming a particular action parameterization.Each retains its native action-generation mechanism.Source: §3.3 · p. 5

3.3.1 GIFT-VLA: Semantics-Centered Direct Action Policy

• GIFT-VLA appends goal, affordance, and action-query tokens to the multimodal sequence and predicts continuous action chunks through a BridgeAttention head.The VLA uses a Qwen3-VL backbone, with one action query per predicted action step; direct action chunks are trained by ℓ1 regression.Source: §3.3.1 · p. 6

3.3.2 GIFT-WAM-Fast: Direct Action from World-Model Features

• GIFT-WAM-Fast attaches guidance to current-frame video tokens while retaining direct action diffusion and cached current-frame features at inference.During training, action tokens attend to current-frame video tokens but not noisy future-video tokens; inference does not generate a future video.Source: §3.3.2 · p. 6

3.3.3 GIFT-WAM-IDM: Future-Conditioned Inverse Dynamics

• GIFT-WAM-IDM conditions action generation on a future-video branch, first denoising a visual trajectory and then using its features for action denoising at inference.The action expert learns an inverse mapping from a predicted transition to demonstrated action; comparing it with GIFT-WAM-Fast tests transfer across direct diffusion and future-conditioned inverse dynamics.Source: §3.3.3 · p. 6

3.4 Optional Auxiliary-Feature Injection

• No-injection is the main design because it tests whether structured intermediate supervision alone produces control-oriented features.In this formulation, auxiliary predictions are not routed into the action-generation interface.Source: §3.4 · p. 6
• Injection appends affordance and goal features to the model-specific action condition, using layer-aligned tokens for the VLA and projected learned-query features for both WAMs.The injected route is active during training and inference, with features predicted internally from the current observation and instruction.Source: §3.4 · pp. 6–7
• Geometry remains alignment-only because its scene-level structure is absorbed by visual tokens already consumed through each policy’s native pathway.Affordance and goal features instead provide compact optional action conditions for object-level interaction and task relevance.Source: §3.4 · p. 7

3.5 Overall Training Objective

• The complete training objective combines the selected native policy loss with weighted geometry, affordance, and goal objectives.The nonnegative weights control the contributions of the three auxiliary objectives.Source: §3.5 · p. 7
• Injection and no-injection use the same structured targets and overall objective, differing only in whether predicted affordance and goal features reach the action generator.In the default no-injection model, auxiliary prediction heads and the geometry teacher need not run at deployment.Source: §3.5 · p. 7
• With injection, predicted affordance and goal features condition the action head during training and inference; without injection, they provide auxiliary supervision only.Geometry remains alignment-only because its scene-level structure is absorbed by visual tokens already entering the native action pathway.Source: Figure 3 · p. 7

4.3 Comparison with State-of-the-Art Methods

• 97.9%, 97.7%, and 98.5% are the LIBERO overall success rates for GIFT-VLA, GIFT-WAM-Fast, and GIFT-WAM-IDM, respectively, with small gains over matched baselines.The baselines are near-saturated on standard LIBERO, leaving limited headroom.Source: Table 1 · p. 8, §4.3 · p. 9
• 4.6, 12.6, and 5.2 points are the respective LIBERO-Plus overall success-rate gains for GIFT-VLA, GIFT-WAM-Fast, and GIFT-WAM-IDM over their matched baselines.The models reach 79.6%, 72.6%, and 87.8%, respectively, after training on original LIBERO trajectories without adaptation.Source: Table 2 · p. 8, §4.3 · p. 9
• 12.6, 9.0, and 8.4 points are the respective RoboCasa average-success-rate gains for GIFT-VLA, GIFT-WAM-Fast, and GIFT-WAM-IDM over their matched baselines.GIFT-WAM-IDM achieves 84.3% on articulated interaction, while GIFT-WAM-Fast achieves 83.7% on pick-and-place.Source: Table 3 · p. 9, §4.3 · p. 9

4.4 Ablation Study and In-depth Analysis

• Each individual geometry, affordance, and goal signal improves overall LIBERO-Plus success over its matched baseline, while combining all three yields the best result in every policy configuration.The combined results are 79.6% for GIFT-VLA, 72.6% for GIFT-WAM-Fast, and 87.8% for GIFT-WAM-IDM.Source: §4.4 · pp. 9–10
• Removing auxiliary-feature injection improves GIFT-VLA by 2.9 points on LIBERO-Plus and 2.8 points on RoboCasa; both WAM variants show smaller improvements.The results show that intermediate-token supervision does not require auxiliary predictions to be fed into the action head.Source: §4.4 · p. 10, Table 5 · p. 10
• Across VLA and WAM examples, geometry preserves scene layout, affordance recovers target-anchored interaction configurations, and goal masks isolate instruction-relevant regions; GIFT attention also focuses more consistently on task-relevant regions.The attention pattern is reported on both LIBERO-Plus and RoboCasa.Source: §4.4 · pp. 10–11, Figure 5 · p. 12, Figure 6 · p. 12

Q5: When can structured intermediate guidance fail?

• Guidance can fail when noise or subtle depth changes impair geometry, viewpoint shifts confuse affordance roles, or out-of-distribution scenes cause goal masks to include irrelevant regions.The authors conclude that effectiveness depends on preserving accurate scene structure, interaction roles, and task relevance under distribution shifts.Source: Q5: When can structured intermediate… · pp. 11–12, Figure 7 · p. 13

4.5 Design Choices

• 2 denoising steps attain the best GIFT-WAM-IDM result and tie the best GIFT-WAM-Fast result; structured supervision adds 3.9 points beyond the no-guidance action-head variant.Across 2, 5, 10, and 15 steps, GIFT-WAM-IDM varies from 87.6% to 87.8%, and GIFT-WAM-Fast from 72.3% to 72.6%.Source: §4.5 · pp. 12–13, Table 6 · p. 13, Table 7 · p. 13

5.1.2 Data Collection

• The four tasks receive 100 human teleoperation demonstrations each, yielding 400 trajectories across placement, cabinet manipulation, and bimanual test-tube insertion.Half of the collected data is randomly inpainted to remove the demonstrator’s green gloves, then mixed with the original demonstrations for training.Source: §5.1.2 · p. 13

5.1.3 Data Annotation

• Human-verified masks and per-frame 6-DoF object poses, together with end-effector poses and closure states, provide temporally consistent targets for goal and affordance supervision.The first object interacted with in each demonstration is selected as the anchor; bimanual objects are paired with their designated arm.Source: §5.1.3 · p. 13

5.1.4 Implementation Details

• Models train on collected demonstrations for 50,000 iterations, predict 32-step relative-joint action chunks, and replan every 32 control steps.GIFT and each baseline use the same training configuration within their policy family.Source: §5.1.4 · pp. 13–14

5.1.1 Hardware Setup

• The xArm7 uses third-person and wrist cameras, while the dual-arm ARX X5 uses a head-mounted camera and one wrist camera per arm; both use parallel grippers.Models for the two platforms are trained separately.Source: §5.1.1 · p. 13, §5.1.4 · p. 13, Figure 8 · p. 14

5.3 Results

• 57.5% versus 35.0% are the real-world overall success rates for GIFT-VLA and StarVLA-OFT, while GIFT-WAM-IDM reaches 87.5% versus 52.5% for Fast-WAM-IDM.The comparison covers four tasks spanning color- and layer-conditioned placement, articulated-object manipulation, and bimanual test-tube insertion.Source: Table 8 · p. 14, §5.3 · p. 14
• 27.5% and 67.5% are the aggregate success rates for GIFT-VLA and GIFT-WAM-IDM under four unseen perturbation conditions, versus 5.0% and 15.0% for their matched baselines.On Task 4, GIFT-WAM-IDM succeeds in 8/10 Level 1 trials and 5/10 Level 2 trials.Source: Table 9 · p. 14, §5.3 · pp. 14, 16

5.2 Task Setting

• Each task is evaluated in 10 trials; robustness tests perturb layer-conditioned placement and bimanual insertion with unseen lighting, background, background-object, and task-object-pose changes.Level 2 adds a different-color tablecloth on Task 2 and rotates the test tubes and rack by approximately 15°–20° on Task 4.Source: §5.2 · p. 14, Figure 9 · p. 15, Figure 10 · p. 15

6 CONCLUSION

• Geometry, affordance, and goal objectives teach features to preserve scene structure, object-centric interaction configurations, and instruction-relevant regions without changing action interfaces or feeding auxiliary predictions as action inputs.Source: §6 · p. 16
• GIFT spans a semantics-centered VLA, a fast-mode WAM, and an inverse-dynamics WAM; ablations support complementary guidance signals and find feature shaping alone sufficient, sometimes better than injecting predictions into the action head.Source: §6 · p. 16
• GIFT improved robustness across LIBERO-Plus distribution shifts and real-robot perturbations while maintaining competitive in-distribution control on LIBERO and RoboCasa.Real-robot tests covered lighting, background, layout, and object-pose changes.Source: §6 · p. 16
• Future work will explore additional guidance, scale up training, and improve guidance-signal reliability under severe perceptual distribution shifts.Source: §6 · p. 16