PaperlayerView exact revision on arXiv

Source-linked AI summary

Act with Intent: Distilling Behavior Intent for Vision-Language-Action Models

Sangoh Lee, Sangwoo Mo, Wook-Shin Han

arXiv:2608.23478v1cs.ROcs.AIcs.CV

TL;DR

VLA action decoders largely imitate demonstrated commands without explicitly representing the local objective served by the behavior. INDI distills that behavior-level intent from executed demonstrations into an intermediate decoder state, and improves performance and generalization across simulation and real-world manipulation. The method’s semantic targets remain bounded by the fixed teacher’s understanding, with additional runtime cost from decoder representations.

  • Problem

    Behavior cloning supervises demonstrated motor commands, while future-based supervision captures particular future realizations rather than the local objective the behavior serves under an instruction.

  • Method

    INDI uses a frozen teacher VLM to interpret executed behavior and trains the VLA decoder to recover its multimodal intent from standard inputs for downstream action and grounding prediction.

  • Results

    INDI improves performance and OOD generalization across SimplerEnv-Bridge, RoboCasa Kitchen, and real-world manipulation, including 84.7% versus 64.3% average success for GR00T-N1.7 on SimplerEnv-Bridge.

  • Takeaways & Limitations

    Recovered intent captures behavior objective and execution progress and organizes downstream predictions in an objective-dependent manner without requiring the teacher at deployment.

  • Takeaways & Limitations

    INDI’s semantics are bounded by the fixed teacher’s physical-behavior understanding, and deployment adds decoder parameters and inference time.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models can turn multimodal context into robot actions, but their action decoders are still trained largely by behavior cloning. This supervises which motor command was demonstrated while leaving implicit the local objective served by the behavior under the instruction. Future-based supervision enriches action learning with frames, latent observations, trajectories, or motion representations, but these signals capture particular realizations of what may happen rather than the shared semantic objective of the forthcoming behavior. We propose Intention Distillation (INDI), which distills behavior-level intent into the action decoder. During training, a frozen teacher VLM interprets a demonstrated segment from the current observation, instruction, coarse action summary, and corresponding execution video. From its standard inputs, the deployed VLA recovers the resulting multimodal intent representation at an intermediate decoder layer and uses it to organize action prediction together with representations of how the behavior unfolds and what it achieves. On SimplerEnv-Bridge, INDI improves GR00T-N1.7 from 64.3% to 84.7%, and on RoboCasa Kitchen it improves the controlled GR00T-N1.7 baseline from 64.1% to 70.3%, with consistent gains on $π_{0.5}$ across both benchmarks. In real-world tasks, INDI improves average success from 62.0% to 68.7%, with gains of up to 12.0 pp on longer-horizon tasks. Further analyses show that the recovered latent is used by the decoder, captures behavior objective and execution progress, and organizes downstream predictions in an objective-dependent manner. These results show that action decoders benefit from explicitly modeling the semantic objective of the behavior they generate.

1 INTRODUCTION

VLA action decoders largely imitate demonstrated commands or predict future realizations without explicitly representing the local objective those behaviors serve. INDI distills behavior-level intent into the decoder to organize action and outcome-related predictions, improving performance across benchmarks and real-world manipulation.

  • Motivation: Behavior cloning supervises which motor command to reproduce, while future-based signals describe possible future states or motion rather than the shared objective of behavior.Similar manipulation stages can serve different objectives, and one objective can have diverse executions.
  • Approach: INDI defines intent as the local objective forthcoming behavior should achieve under the instruction and distills it into the action decoder.A frozen teacher VLM interprets executed behavior, while the student recovers the resulting multimodal representation from standard VLA inputs.
  • Approach: The decoder uses recovered intent to organize action prediction together with representations of how behavior unfolds and what it achieves.The intent is recovered at an intermediate decoder layer and supports subsequent predictions.
  • Results: 84.7% versus 64.3%: INDI improves GR00T-N1.7 average success on SimplerEnv-Bridge.The supplied result reports the comparison as 64.3% to 84.7%.
  • Results: INDI improves performance and OOD generalization across SimplerEnv-Bridge, RoboCasa, and real-world manipulation, while analyses show the latent captures objective and progress.The recovered representation also organizes downstream predictions in an objective-dependent manner.

2 RELATED WORK

Related robot-policy work augments action learning with future, structured, spatial, motion, semantic, and latent-plan representations. INDI differs by distilling a fixed teacher’s semantic interpretation of executed behavior rather than learning reconstruction-oriented plans jointly with the policy.

  • Future-based and structured supervision: Future-based methods add generated frames, subgoal images, predictive features, future latent states, world-action representations, or structured world knowledge.Other approaches encode spatial, motion, or semantic structure as intermediate policy signals.
  • Intent and latent plans: Related latent-plan methods derive continuous or language-regularized discrete plans from trajectories, while other work uses trajectory abstractions, action priors, and virtual targets.These approaches provide alternative intermediate representations for robot policies.
  • Intent and latent plans: INDI distills a fixed teacher’s semantic interpretation of behavior, unlike related latents that function as reconstruction codes learned jointly with the policy.This distinction concerns how the latent representation is obtained and supervised.

3 METHODOLOGY

INDI trains a VLA decoder to recover behavior-level intent from standard inputs and use it as an intermediate state for action and grounding prediction. Training derives multimodal, textual-purpose, and endpoint-visual targets from executed behavior, while deployment removes teacher-side modules.

  • Problem formulation: A frozen VLM maps the current observation and instruction to context tokens, while the decoder predicts actions and forms an intermediate intent state.The intent state represents the local objective served by the action under the instruction.
  • Intent supervision from executed behavior: Training derives intent from executed segments using the observation, instruction, coarse action summary, and execution video.The teacher also generates a functional-purpose statement and cached hidden states.
  • Intent supervision from executed behavior: Teacher targets combine a multimodal intent representation, a textual-purpose representation, and an endpoint visual feature.The endpoint visual target grounds the resulting scene change, while the textual target grounds functional meaning.
  • Intent-aware action decoding: Intent queries recover the teacher target at an intermediate decoder layer, and the recovered decoder-space state is used by remaining layers.Intermediate supervision allows the representation to shape subsequent computation.
  • Intent-aware action decoding: Action and grounding rows share one decoder and are jointly processed so recovered intent organizes their subsequent predictions.Asymmetric attention routes information from intent to grounding and action streams.
  • Training objective: The training objective combines action, intent, visual-grounding, textual-grounding, and intent-mismatch losses.The mismatch term forces matched intent to yield lower downstream loss than mismatched intent by margin m.
  • Training objective and deployment: At deployment, the teacher, cached targets, alignment projections, and mismatch branch are removed, leaving the original VLA inputs and internal intent states.Only additional decoder representations remain at runtime.

4 EXPERIMENTS

INDI is evaluated across simulation, real-world, and controlled studies to test performance, generalization, intent structure, and computational cost. It consistently improves VLA success, with larger benefits on longer multi-stage tasks and evidence that teacher-derived intent contributes the main gain.

  • Simulation benchmarks: INDI improves GR00T-N1.7 average success from 64.3% to 84.7% on SimplerEnv-Bridge and from 64.1% to 70.3% on RoboCasa Kitchen.The RoboCasa comparison covers 24 tasks; gains also transfer to π0.5 across both benchmarks.
  • Real-world benchmarks: INDI improves real-world success under clean, held-out-object, and distractor conditions, with the largest task-level gains on longer-horizon manipulation.Average task improvements are 7.3 pp for Cross-Bin Stacking and 10.7 pp for Drawer Storage, versus 4.7 and 4.0 pp for the shorter tasks.
  • Real-world benchmarks: Stage profiles show distinct improvement mechanisms: Cross-Bin Stacking benefits at a subgoal transition, whereas Drawer Storage improves from grasping through closure.On Drawer Storage, completion rises from 50.0% to 70.0% at grasping and from 24.0% to 36.0% at closure.
  • Controlled analyses: Intent supervision alone reaches 76.0%, exceeding the action-only baseline by 14.5 pp and future supervision by 8.0 pp.Adding visual and textual groundings raises average success further to 85.5%.
  • Controlled analyses: Interventions preserve 84.5% success for same-objective edits but reduce success to 45.2% for cross-objective edits and below 8% for phase forcing.These interventions test whether recovered representations organize execution according to task objective and progress.
  • Computational cost: INDI adds modest deployment cost because teacher inference is offline, increasing GR00T-N1.7 inference time from 56.1 to 61.5 ms per policy query.The parameter count rises from 3.46B to 3.50B; π0.5 inference time rises from 24.4 to 29.7 ms.

5 CONCLUSION

INDI distills behavior-level intent into pretrained VLA action decoders, using a teacher-derived representation recovered from standard deployed inputs. Across benchmarks, it improves strong VLA backbones and makes the latent functionally relevant to action and grounding prediction.

  • INDI distills behavior-level intent into pretrained VLA action decoders.
  • INDI improves strong VLA backbones across SimplerEnv-Bridge, RoboCasa Kitchen, and real-world manipulation without requiring the teacher at deployment.
  • The recovered state represents objective and progress, is used by the policy, and influences closed-loop execution.
  • The vision-language modules remain frozen while the complete action decoder and added intent and grounding parameters are optimized.
  • For GR00T-N1.7, INDI recovers intent at block 16 of 32, while π0.5 recovers it at layer 9 of 18.

A.2 DATASETS

The evaluation spans simulation and real-world tabletop manipulation, with diverse tasks, embodiments, baselines, supervision controls, and teacher-target construction choices. Controlled results indicate that teacher-derived intent contributes substantially and complements grounding supervision.

  • Datasets: SimplerEnv-Bridge contains 53,192 episodes and 1,893,026 frames for four WidowX manipulation tasks.
  • Datasets: RoboCasa Kitchen contains 2,400 episodes across 24 household-manipulation tasks, evaluated with three camera views and three independent runs.
  • Datasets: The real-world suite uses four tabletop tasks, each with 100 demonstrations and a single multitask policy trained on identical data per method.
  • Baselines: The evaluated baselines include generalist VLA architectures such as GR00T-N1.7, π0.5, OpenVLA, RoboVLMs, SpatialVLA, and π0.
  • Intent targets: The teacher receives the current observation, instruction, coarse action summary, and execution video to interpret each demonstrated behavior segment.
  • Controlled results: Teacher-derived intent reaches 76.0%, while adding visual and textual groundings increases average success to 85.5%.The controlled comparisons use matched demonstrations, optimization budgets, and evaluation protocols.
  • Controlled results: The joint grounding effect is +9.5 percentage points, exceeding the +8.5 points from the individual visual and textual gains.On EP-Basket, either stream alone reaches roughly 73%, whereas both together reach 100.0%.

B.2 INTENT-DEPENDENCE MECHANISM ABLATIONS

The ablations test whether downstream decoding depends on recovered intent through context bottleneck routing and intent-mismatch training. Removing either mechanism lowers success, and removing both lowers it further.

  • Ablation design: The full model combines a context bottleneck with intent-mismatch training to make downstream action and grounding prediction depend on recovered intent.
  • Results: 85.5% falls to 70.0% when the context bottleneck is removed and to 74.0% when intent-mismatch training is removed.
  • Results: Removing both mechanisms further lowers average success to 65.5%.
  • Interpretation: Bottleneck routing encourages contextual information to pass through intent, while Lmis makes downstream decoding sensitive to intent content rather than latent-pathway presence alone.

B.3 FUNCTIONAL ROLE AND STRUCTURE OF RECOVERED INTENT

Recovered intent is used by the action decoder and encodes both behavior objective and execution progress beyond action-sequence realization. Objective-specific interventions alter downstream behavior, with effects shaped by task relatedness and execution stage.

  • Decoder dependence: Zeroing recovered intent reduces average success from 85.5% to 45.5%, below the 61.5% action-only baseline.The intervention preserves other policy inputs and decoder computation, indicating that recovered intent is used in downstream action decoding.
  • Progress structure: Three-way phase prediction reaches 77.8% five-fold cross-validation accuracy, showing progress structure shared across the four Bridge manipulation tasks.The recovered representation organizes segments by early, middle, and late execution stages across tasks with different objects and target relations.
  • Objective structure: Recovered intent separates behavior objectives more strongly than matched local action snippets, including when the same objective uses dissimilar action sequences.For progress-matched pairs, recovered-intent similarity is 0.150 for the same objective with dissimilar sequences versus −0.017 for different objectives with similar sequences.
  • Information propagation: Intent-row task purity peaks at 0.93 before alignment, while visual and textual grounding purity reaches 0.95 and 0.97 after alignment.Action and state rows remain near chance, motivating edits to intent and grounding rows through the decoder tail.
  • Objective intervention: Same-objective interventions retain 84.5% average success, versus 45.2% for cross-objective interventions and 1.0% under Gaussian corruption.The contrast indicates that the intent-mediated pathway carries objective-dependent content rather than functioning as an undifferentiated gate.
  • Task relatedness and stage control: Spoon and Carrot have the closest recovered-intent centroids, while EP-Basket is most separated, but scene and objective differences prevent causal interpretation of transfer compatibility.Stage-forced injections also produce behavior consistent with the injected stage, such as approach without grasp under early forcing and placement under late forcing.

B.4 VISUAL AND TEXTUAL READOUT OF INTENT

Readout experiments show that recovered intent contains compact semantic information that can be decoded into visual outcomes and textual purpose. Interventions also redirect downstream visual and textual predictions toward the injected behavior.

  • Visual readout: Visual grounding retrieves the paired endpoint at rank one in 72.6% of held-out segments and within the top ten in 95.3%.Purpose semantics are most compactly represented in recovered intent, while textual grounding provides a downstream semantic realization.
  • Intent intervention: Visual readout quality decreases from recovered to zero to cross-task intent, showing dependence on the content supplied through the intent pathway.The intervention preserves the remaining policy inputs while changing recovered intent.
  • Visual-outcome retrieval: Recovered intent preserves the current object, scene, and resulting relation, whereas cross-task intent redirects retrieved endpoints toward the injected behavior.Across evaluated pairs, recovered intent retrieves the current endpoint in 70.5% of cases and cross-task replacement retrieves the injected endpoint in 78.6%.
  • Textual readout: Recovered intent is more readable than final textual grounding, improving object F1 from 0.314 to 0.693 and verb F1 from 0.195 to 0.432.This localizes the semantic bottleneck primarily in recovered intent while retaining textual grounding as a downstream representation used during joint decoding.
  • Joint semantic hierarchy: Recovered intent and downstream grounding predictions change consistently under intent intervention, supporting a hierarchy in which intent stores a compact behavior-level representation.Visual and textual readouts are evaluated with frozen policy representations and separately trained readout decoders.

B.5 TEACHER QUALITY AND INTENT-TARGET GEOMETRY

Teacher quality substantially affects INDI performance: useful targets require non-degenerate, state-resolved variation rather than high effective rank alone. Controlled teacher comparisons support this geometry-performance relationship as a diagnostic, with limited causal scope.

  • Target geometry: Qwen3.5-9B provides an almost degenerate alignment signal with an angular budget of 0.0027, despite its high effective rank.A nearly constant prediction can satisfy much of its cosine-alignment objective, so effective rank alone does not indicate a useful teacher.
  • Readout structure: Recovered intent supports visual and textual behavior readouts alongside teacher-target analysis, with injected intent redirecting both modalities toward the injected behavior.The readout evidence shows that the learned representation has observable semantic realizations beyond the teacher-cache geometry.
  • Target geometry: Cosmos preserves high within-episode variation while increasing angular budget by 3.3× over Qwen3-VL, accompanied by a further 12.0-point downstream gain.Qwen3-VL recovers a 12.0-percentage-point gain over the action-only baseline in the controlled comparison.
  • Interpretation: Across three teachers, angular budget and within-episode share increase monotonically with downstream success, but the small sample supports a diagnostic condition rather than a predictive law.The stated causal interpretation is limited to the Qwen3-VL–Cosmos pair, where physical-AI post-training is the controlled difference.

B.6 INTENT-TARGET SOURCE ABLATION

The source of the teacher intent target matters: both evidence-side and generated-response representations improve over action-only supervision, but the evidence-side multimodal target performs better.

  • Target-source comparison: 85.5% average success from the evidence-side multimodal target exceeds 79.5% from the generated-response target.Both variants use the same teacher, target-slot count, student architecture, grounding targets, optimization budget, and evaluation protocol.
  • Target-source comparison: The evidence-side target provides a further 6.0 percentage-point gain over the generated-response target.The generated-response target still substantially outperforms the 61.5% action-only baseline.
  • Interpretation: Directly distilling the representation formed during multimodal evidence processing is more effective than using the representation formed after reasoning and response generation.The comparison supports useful behavior-level supervision from both target sources while favoring the evidence-side representation.

C.1 FULL QUANTITATIVE RESULTS

The full RoboCasa Kitchen results compare INDI with controlled and larger-data baselines across all 24 tasks, while representative rollouts illustrate execution on four real-world tasks. Per-task variation shows that future supervision can redistribute performance rather than improve uniformly.

  • Per-task variation: The future-supervision variant improves markedly on some tasks but degrades on others where competing variants are near ceiling.Examples include gains on Open Double Door and Turn Off Stove, alongside degradation on Turn On Sink Faucet and Coffee Press Button.
  • Per-task variation: Category and overall averages for future supervision reflect a redistribution of per-task performance rather than uniform improvement.
  • Full RoboCasa Kitchen results: Table 20 reports success rates across all 24 RoboCasa Kitchen tasks for controlled GR00T-N1.7, π0.5, a GR00T-N1.7 future-supervision variant, and larger-data GR00T checkpoints.Controlled variants and π0.5 results are reported as mean ± sample standard deviation over three evaluation runs; GR00T G3000 provides single-value data-scale references.
  • Real-world rollouts: Representative rollouts cover Threading, Basket Nesting, Cross-Bin Stacking, and Drawer Storage across third-person, head, and wrist-camera views.The two latter tasks require coordinated execution across multiple stages.

D LIMITATIONS AND FUTURE WORK

The paper identifies three boundaries for INDI: dependence on the teacher’s embodied understanding, recovery of a single deterministic intent, and reliance on successful single-window demonstrations. It proposes execution-validated, interactive, hierarchical, and failure-informed intent as future directions.

  • Teacher grounding: INDI’s available intent semantics are bounded by the teacher VLM’s understanding of physical behavior.A teacher without embodied grounding produces nearly directionless targets, causing performance below the action-only baseline; the framework cannot revise targets when execution contradicts them.
  • Teacher grounding: The paper proposes validating intent against execution outcomes or acquiring it through policy interaction rather than reading it solely from a fixed interpreter.
  • Intent interface: The deployed policy reconstructs one deterministic intent per query and immediately consumes it, preventing distributions over ambiguous objectives, persistent rollout intent, or stated corrections.Although interventions show the intent state is addressable, the policy does not use it as an interface.
  • Intent interface: The paper identifies planner-, human-, or execution-feedback interfaces for intent as its most consequential extension.
  • Demonstration scope: INDI reads intent from single windows of successful demonstrations, leaving nested objectives and failed or corrected episodes for future hierarchical and failure-informed intent learning.The authors note that failed or corrected episodes may state objectives more explicitly while representing substantial robot experience.
Loading 2608.23478v1…