PaperlayerView exact revision on arXiv

Source-linked AI summary

GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization

Xiaosong Jia, Bowen Yang, Zuhao Ge, Xian Nie, Yuchen Zhou, Cunxin Fan, Yufeng Li, Yilin Chai, Chao Jing, Zijian Liang, Qingwen Bu, Haidong Cao, Chao Wu, Qifeng Li, Zhenjie Yang, Chenhe Zhang, Hongyang Li, Zuxuan Wu, Junchi Yan, Yu-Gang Jiang

arXiv:2605.12369v2cs.RO

TL;DR

VLA action decoders are expected to learn task-relevant factors implicitly from demonstrations, while unguided attention may rely on spurious correlations. GuidedVLA assigns manually specified factors to specialized attention heads; across simulation and real-robot evaluations, it improves performance, and factor quality correlates positively with task success.

  • Problem

    VLA training typically relies on end-to-end supervision for action decoders to learn task-relevant factors implicitly.

    Source: §I · p. 2

  • Method

    GuidedVLA manually specifies task-relevant factors and supervises specialized attention heads to capture them.

    Source: §I · p. 2

  • Results

    Across simulation and real-world experiments, GuidedVLA boosts π0 and surpasses recent feature-training methods [96] [75].

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported factor-performance correlation and decoupled features support explicitly guiding action-decoder learning as a direction for more robust VLA models.

    Source: §I · p. 2, §VI · p. 9

    The method requires manual selection of task-relevant factors, which can be domain-dependent; automating factor discovery remains an open challenge, especially for continuous tasks.

    Source: §VI · p. 9, Q. Limitations and Future Work · p. 30

Abstract

from arXiv · show

Vision-Language-Action (VLA) models aim for general robot learning by aligning action as a modality within powerful Vision-Language Models (VLMs). Existing VLAs rely on end-to-end supervision to implicitly enable the action decoding process to learn task-relevant features. However, without explicit guidance, these models often overfit to spurious correlations, such as visual shortcuts or environmental noise, limiting their generalization. In this paper, we introduce GuidedVLA, a framework designed to manually guide the action generation to focus on task-relevant factors. Our core insight is to treat the action decoder not as a monolithic learner, but as an assembly of functional components. Individual attention heads are supervised by manually defined auxiliary signals to capture distinct factors. As an initial study, we instantiate this paradigm with three specialized heads: object grounding, spatial geometry, and temporal skill logic. Across simulation and real-robot experiments, GuidedVLA improves success rates in both in-domain and out-of-domain settings compared to strong VLA baselines. Finally, we show that the quality of these specialized factors correlates positively with task performance and that our mechanism yields decoupled, high-quality features. Our results suggest that explicitly guiding action-decoder learning is a promising direction for building more robust and general VLA models.

I. INTRODUCTION

GuidedVLA addresses unstable, task-irrelevant attention in VLA action decoders by assigning dedicated heads to task-relevant factors. The paper reports gains across simulation and real-world evaluations, alongside evidence about factor quality and feature decoupling.

I. INTRODUCTION

  • Unguided VLA decoders can rely on spurious visual correlations and shifting attention, leaving their task-relevant representations unstable.End-to-end supervision alone is expected to make the action decoder learn task-relevant factors implicitly.Source: §I · p. 2
  • GuidedVLA manually assigns task-relevant factors to specialized attention heads, instantiated for object grounding, skill recognition, and geometry perception.The authors also report that factor understanding correlates positively with task success and that specialized heads yield better-decoupled features than joint supervision.Source: §I · p. 2
  • GuidedVLA significantly boosts π0 across simulation and real-world experiments and surpasses recent feature-training methods [96] [75].The evaluations report improvements in both in-domain and out-of-distribution settings.Source: §I · p. 2

II. RELATED WORK

  • GuidedVLA complements prior object-centric, skill-based, and geometry-aware approaches by supervising these structures as separable pathways inside the VLA action decoder.Related work also covers multi-source embodied data, benchmarks, training and inference recipes, and alternative action representations.Source: §II · p. 2

A. Problem Setup and Motivation

VLA action tokens may attend to irrelevant background regions rather than extracting task-critical information from visual and language features. GuidedVLA specializes attention heads around three complementary factors: grounding localizes a target, skill recognition identifies behavior, and geometry provides spatial constraints.

A. Problem Setup and Motivation

  • Action-token attention can diffuse over irrelevant background regions, motivating guidance that directs action decoding toward task-relevant information.The action generation process regresses robot trajectories by denoising action tokens conditioned on vision and language tokens.Source: A. Problem Setup and Motivation · p. 2
  • GuidedVLA uses distinct object, skill, and depth heads for factor-specific attention and supervision, then merges specified-head outputs with the main branch through ControlNet-style residual fusion.The depth head attends to depth-encoder features; object and skill heads receive their respective auxiliary objectives.Source: Figure 2 · p. 3, A. Problem Setup and Motivation · p. 3

B. What to Guide: Three Task-Relevant Factors

  • The three complementary factors are object grounding for target localization, skill recognition for the current sub-skill or phase, and geometry perception for 3D spatial constraints.The paper frames them together as a semantic interface between high-level VLM representations and low-level control.Source: B. What to Guide: Three Task-Relevant… · p. 3

C. How to Guide: Attention Head Specialization

GuidedVLA assigns distinct task-relevant factors to selected attention heads, while a residual adapter preserves the pretrained attention path and gradually adds factor-specific signals. Its object head guides attention toward labeled task-relevant regions.

C. How to Guide: Attention Head Specialization

  • GuidedVLA assigns specific attention heads to distinct factors and applies different supervision signals to those heads.The paper presents multi-head attention as a way to capture decoupled task-relevant information with minimal structural modification.Source: C. How to Guide: Attention Head… · p. 3
  • The residual adapter adds factor-specific attention through a zero-initialized projection, preserving the pretrained main branch initially and gradually injecting task-relevant biases during training.The adapter copies weights from the base policy.Source: C. How to Guide: Attention Head… · pp. 3–4, Figure 3 · p. 4

1) Object Head (Visual Grounding):

  • The object head concentrates selected heads’ attention mass on ground-truth object-region masks, with non-object image patches and non-image tokens assigned zero target weight.Mean-head aggregation combines the selected object heads; the object-mass loss is applied when the target object is visible.Source: 1) Object Head (Visual Grounding) · p. 4
  • Object masks mark the object to grasp or destination, while the model remains free to distribute attention within the marked region.Foundation models help construct the masks; stage-aware mask construction is described separately.Source: 1) Object Head (Visual Grounding) · p. 4

2) Skill Head (Temporal Logic Intent):

GuidedVLA uses selected skill heads to represent temporally extended intent and constrains depth heads to use depth-derived features. These signals join the flow-matching objective while other heads remain free to learn data-driven patterns.

2) Skill Head (Temporal Logic Intent):

  • Skill heads pool features across guided layers, heads, and action queries, then predict a skill distribution trained against a future-horizon soft label with KL divergence.Skill labels combine foundation-model annotation and manual correction; LIBERO uses three task-level classes plus a null/background class.Source: 2) Skill Head (Temporal Logic Intent) · p. 4

3) Depth Head (3D Structure):

  • Depth heads attend only to keys and values projected from a frozen depth encoder, imposing a structural constraint for 3D geometry without a depth loss.The design addresses standard vision encoders’ lack of explicit 3D awareness, as described in the passage.Source: 3) Depth Head (3D Structure) · p. 4
  • Training combines flow matching with object and skill auxiliary losses; depth guidance instead uses depth keys and values, leaving unsupervised heads free to learn data-driven patterns.The remaining heads preserve model flexibility and expressivity.Source: 3) Depth Head (3D Structure) · p. 4

D. Guidance Dataset Construction

The paper describes an automatic factor-annotation pipeline and evaluates GuidedVLA in simulation and on two real-robot platforms. Its analysis considers benchmark performance, factor-task alignment, and attention-head specialization.

D. Guidance Dataset Construction

  • The annotation pipeline uses Qwen3-VL prompts and SAM2 mask propagation for object grounding, Qwen3-VL skill labels, and frozen depth features without depth labels.Human verification remains part of the process; 92% of episodes require no correction, and annotating 50 episodes takes about 4 minutes versus around 43.5 minutes manually.Source: D. Guidance Dataset Construction · pp. 4–5, Figure 4 · p. 5
  • The LIBERO-Plus results identify task-specific single-head strengths: object on Object and Long, skill on Goal, and depth on Spatial; the proposed model has the highest average success rate.Source: Table I · p. 5

A. Simulation Experiments

  • LIBERO-Plus tests distribution-shift robustness across seven dimensions, while RoboTwin 2.0 provides multi-task evaluation under randomized, unseen conditions.The RoboTwin evaluation covers eight representative tasks.Source: A. Simulation Experiments · p. 5

B. Real-World Experiments

  • Real-world experiments use two dual-arm platforms to assess in-domain action generation and cross-platform generalization across household and chemistry-lab manipulation tasks.The chemistry-lab tasks focus on transparent rigid objects and tight geometric constraints, not complete safety-critical procedures; each task and model receives 20 trials.Source: B. Real-World Experiments · pp. 5–6
  • 90.63% average success on RoboTwin 2.0: the full model achieves the best average across eight manipulation tasks, integrating capabilities of specialized heads.The benchmark evaluates randomized, unseen instructions, environments, and object placements on the Agilex Piper dual-arm setup.Source: A. Simulation Experiments · p. 5, Figure 5 · p. 6

V. ANALYSIS

  • The analysis asks whether factor guidance closes a vision-language representation gap, improves in- and out-of-distribution performance, and whether head specialization yields decoupled features.It also examines which factors matter for task types and how architectural choices affect performance.Source: §V · p. 6

A. Task-suite Analysis and Cross-benchmark Generalization

GuidedVLA combines object, skill, and depth heads for broad task generalization, evaluated in simulation and on ALOHA household and PSI-Bot lab tasks. Controlled sensitivity analyses find task success rises with the quality of each factor.

A. Task-suite Analysis and Cross-benchmark Generalization

  • Object, skill, and depth heads excel in different task niches, while their combination supports generalization across diverse domains.Examples include 82.5% on LIBERO-Plus Object, 96% on RoboTwin Lift Pot, and depth-head gains from 35% to 63% on Click Bell and 78% to 96% on Beat Hammer Block.Source: A. Task-suite Analysis and… · pp. 6–7
  • 90.63% average RoboTwin 2.0 success versus 77.38%, while real-robot success exceeds baseline across settings by up to 52.7%.The full model is the only variant reported to reliably generalize across scene, lighting, and position variation.Source: A. Task-suite Analysis and… · p. 7, Table II · p. 7

B. Sensitivity Analysis: Does Factor Quality Matter?

  • LIBERO-Plus success rises from 61.3% to 74.6% as object-mask-aligned attention increases from 0.25 to 1.0.The π0 baseline has 26.5% intrinsic object focus; GuidedVLA raises object-region attention from 8.1% to 84.0% and argmax-hit accuracy from 2.2% to 84.7%.Source: B. Sensitivity Analysis: Does Factor… · p. 8
  • LIBERO-Plus success rises from 66.2% to 72.9% as skill accuracy increases from 0.25 to 1.0; π0 skill accuracy is 48.4%.Source: B. Sensitivity Analysis: Does Factor… · p. 8
  • LIBERO-Plus success rises from 15.6% to 76.7% as the true depth-feature ratio increases from 0 to 1.0.Across all three factors, success increases monotonically with factor quality, not merely factor presence.Source: B. Sensitivity Analysis: Does Factor… · p. 8

C. Specialization Enables Decoupled Feature Learning

GuidedVLA specializes attention heads rather than jointly supervising all heads with every factor objective. The paper reports that specialization yields better-decoupled features, and evaluates head and adapter design choices.

C. Specialization Enables Decoupled Feature Learning

  • Jointly supervising every attention head with object, geometry, and skill objectives entangles features and underperforms specialization and other non-factorized controls.The t-SNE visualization contrasts separated clusters for specialized heads with overlapping clusters for the mixture alternative.Source: C. Specialization Enables Decoupled… · pp. 8–9, Figure 10 · p. 9, Figure 9 · p. 9
  • GuidedVLA reaches 75.4% overall average success, outperforming DreamVLA (69.9%), VLA-Adapter (59.1%), AdaMoE (50.1%), and Spatial Forcing (29.1%).The authors attribute gains to explicit specialization of object grounding, skill recognition, and geometric cues.Source: C. Specialization Enables Decoupled… · p. 9

E. Ablation of Design Choices

  • The design ablations favor object-region suppression, soft skill targets, downsampled depth features, and a zero-initialized control branch.The branch introduces auxiliary signals gradually without disrupting the base model at training start.Source: E. Ablation of Design Choices · p. 9

VI. CONCLUSION

  • The conclusion reports consistent in-domain and distribution-shift gains, positive factor-quality/task-success correlation, and better-decoupled features from dedicated heads.The method relies on predefined factors; automatic factor discovery remains open, especially for continuous tasks with difficult skill labeling.Source: §VI · p. 9

APPENDIX

The appendix points to full real-robot, LIBERO-Plus, and RoboTwin 2.0 result tables and details the head, adapter, and factor-quality ablation implementations.

B. Complete Results for All Datasets

C. Implementation Details of Each Head and Adapter

D. Implementation Details of Factor Quality Ablation

  • Factor-quality ablations clamp object grounding strength m to α ∈ {0.25, 0.5, 0.75, 1.0}, vary depth-feature ratio δ from 0 (pure noise) to 1.0 (clean signal), and target skill soft accuracy S at γ ∈ {0.25, 0.5, 0.75, 1.0}.Source: D. Implementation Details of Factor… · pp. 15, 21

E. Ablation on Head and Adapter Design Choices

The passage points to detailed explanations for material corresponding to Sections III-C and V-B.

E. Ablation on Head and Adapter Design Choices

1) Object Head:

The ablations compare object, depth, and skill supervision choices and feature-fusion strategies. Region-based object supervision, a small depth encoder with token downsampling, soft skill labels, and zero-initialized convolution each perform well in their respective evaluations.

1) Object Head:

  • 83.33% average success rate with object region supervision, compared with 72.00% for Gaussian-prior supervision on RoboTwin 2.0.The region-based strategy uses binary masks to constrain attention to object-relevant regions without specifying the distribution within them.Source: 1) Object Head · p. 22, Table VI · p. 23

2) Depth Head:

  • 83.00% average success rate with the Depth Anything 3–small encoder on RoboTwin 2.0; removing token downsampling drops performance to 68.00%.The large variant reaches 82.00% and the base variant 69.67%; adding the depth head raises stacking success from 35% to 50% over Spatial Forcing and from 25% to 45% over π0.Source: 2) Depth Head · p. 22

3) Skill Head:

  • 75.00% average success rate across three RoboTwin 2.0 tasks with soft skill labels, outperforming hard labels on Click Bell (39% vs. 36%) and Dump Bin BigBin (94% vs. 82%).Soft targets encode the distribution of skill classes within an action segment, accommodating ambiguous transitions and mixed-intent windows.Source: 3) Skill Head · p. 22

4) Feature Fusion via ControlNet-style Adapter:

  • 83.33% average success rate with zero-initialized convolution, versus 64.00% with direct addition and 73.33% with gated addition on RoboTwin 2.0.Zero initialization makes the fused path initially behave as an identity function; the largest task-specific gain is on Click Bell.Source: 4) Feature Fusion via ControlNet-style… · pp. 23–24

F. Ablation on Overall Architecture

The ablations identify a mid-to-upper layer range as the most effective place for guidance and describe the model, data annotation, and training procedures used in the experiments. They also compare feature-fusion strategies and document implementation choices affecting training stability and validation.

G. Complete Model Architecture of GuidedVLA

  • GuidedVLA combines a multi-view SigLIP vision tower, PaliGemma language backbone, and Gemma-based action expert, with optional Depth Head, Skill Head, and ControlAttention modules.The optional branches can be toggled according to task setup.Source: G. Complete Model Architecture of… · p. 24, Table XI · p. 26

H. Code Details

  • The implementation converts LeRobot 2.0 datasets to LeRobot 3.0, uses torch.compile for training speed, and defaults to mixed precision after full bfloat16 training degraded performance.Full float32 and mixed precision match the reported performance of 90, while full bfloat16 reaches 10; gradient accumulation is disabled by default.Source: H. Code Details · p. 24, Table XII · p. 27

J. Experiment Details

  • The paper reports that monitoring auxiliary losses during training is important for effective learning, so it adds a 93:7 training-validation split and open-loop validation-loss tracking.The official codebase described in the passage lacks a validation set and training-time evaluation pipeline.Source: H. Code Details · p. 24

1) Object Masks:

  • Object masks are generated stage by stage using Qwen3-VL proposals, SAM2 tracking, and human verification, then converted to patch-level targets on a 16 × 16 image-token grid.Frames outside annotated stages and frames with empty propagated masks are unlabeled for object supervision.Source: 1) Object Masks · pp. 24, 26

F. Ablation on Overall Architecture

  • On RoboTwin 2.0, among direct addition, elementwise tanh-gated addition, and zero-initialized convolution for incorporating auxiliary features into the action decoder, zero-initialized convolution achieved the highest average success rate (83.33%), particularly on Click Bell.Source: Table IX · p. 25
  • 75.4 LIBERO-Plus total score with guidance on the third layer quartile, above all-layer guidance (74.1) and the first, second, and fourth quartiles (74.4, 74.3, and 73.8).The authors suggest mid-to-upper layers may capture higher-level semantic and task-relevant representations.Source: F. Ablation on Overall Architecture · p. 24, Table X · p. 25

2) Skill Labels:

  • Skill targets are normalized histograms of stage-wise skill labels over each action segment, becoming one-hot when only one class occurs.For LIBERO, the four classes include three task-level skills and a null/background class; the distribution supervises trajectory-level intent.Source: 2) Skill Labels · p. 26

J. Experiment Details

  • LIBERO-Plus evaluation is zero-shot after unguided π0 fine-tuning on LIBERO followed by object- and skill-guided training; RoboTwin uses 1,000 demonstrations per task and 100 evaluation rollouts per task.The real-world experiments cover six tasks with approximately 50 demonstrations each; the compared models use identical settings for LIBERO-Plus.Source: J. Experiment Details · pp. 26–27

K. Training Hyperparameter Ablation.

This material specifies the real-robot deployment and evaluation conditions, generalization shifts, and rollout visualizations. It also reports stage-aware object attention and depth attention focused on spatially meaningful regions in LIBERO-Plus examples.

K. Training Hyperparameter Ablation.

1) Real-world Deployment Setup:

  • Real-robot inference outputs 50-step action chunks at an effective 20 Hz, interpolated to 50 Hz control, with a 4.0 s joint-convergence timeout.Deployment uses a frozen Depth Anything V3 small encoder; camera extrinsics match training and objects stay within a 50 cm × 60 cm × 30 cm workspace.Source: 1) Real-world Deployment Setup · p. 27

2) Detailed Evaluation Setup:

  • The evaluation covers three ALOHA household tasks and three PSI-Bot laboratory tasks, with trials lasting up to 120 s and usually requiring a 1 s stable dwell.Success criteria are task-specific, including completing the required sequence for tabletop cleaning and stable placement of the beaker on mesh for heating.Source: 2) Detailed Evaluation Setup · pp. 27–28

M. Real-World Generalization Settings

  • Real-world generalization tests isolate one shift per trial: positional perturbations, scene clutter, or colored lighting.Positional shifts use nine anchors spaced 1–2 cm apart; scene shifts add 3–5 non-occluding distractors; lighting uses three settings that preserve task observability.Source: M. Real-World Generalization Settings · p. 28, Figure 11 · p. 28

N. Visualization of LIBERO-Plus Results

  • LIBERO-Plus visualizations show seven keyframes per successful episode across its spatial, object, goal, and long task suites, with RGB, object attention, depth, and depth-head attention rows.The spatial-task figure identifies the same four visual rows across seven episode stages.Source: N. Visualization of LIBERO-Plus Results · pp. 28–29, Figure 12 · p. 29

1) Object attention is stage-aware:

  • The object head shifts attention from the object of interest during early stages to the target container or placement area as the task progresses.The described examples include attention to a bowl or book before shifting toward the goal state.Source: 1) Object attention is stage-aware · p. 29

2) Depth attention captures geometric awareness:

  • The depth head highlights the robot arm and task-relevant objects, including contours and relative positions in stacking and container-reaching examples.The passage connects these visual patterns with geometry-aware reasoning for occlusion, stacking, and motion planning.Source: 2) Depth attention captures geometric… · p. 29

1) Visualization of Real-World Tasks Rollouts:

  • Real-robot rollout grids show seven manually selected stages for each task under positional, lighting, and scene shifts.Rows correspond to the three shifts; stages include approach, grasp, transport, placement, and completion.Source: 1) Visualization of Real-World Tasks… · p. 29

2) Head-wise Mechanism Visualization on Real-World Robots:

Real-robot diagnostics align each specialized head with tasks where its factor is most critical, while baseline failures cluster around grounding, geometry, and temporal progression. The paper notes that selecting task-relevant factors remains manual and may depend on the domain.

2) Head-wise Mechanism Visualization on Real-World Robots:

  • The object, depth/geometry, and skill/temporal heads are aligned with T1/T4, T2/T5, and T3/T6, respectively, and diagnostics pair single-head success rates with qualitative examples.Single-head variants are evaluated only on their aligned tasks; rollout visualizations show attention overlays, depth cues, and skill progression.Source: 2) Head-wise Mechanism Visualization… · p. 29, Figure 27 · p. 38, Figure 28 · p. 39

P. Failure Case Analysis (Tasks 1–6)

1) Object grounding failures:

2) Metric geometry and clearance failures:

3) Temporal skill collapse in multi-stage execution (T3/T6):

Q. Limitations and Future Work

  • Task-relevant factors must be selected manually and may be domain-dependent; automating factor discovery and adding factors such as force/torque reasoning remain future directions.Source: Q. Limitations and Future Work · p. 30
Loading 2605.12369v2…