PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Foresight Actionable: Repurposing Representation Alignment in World Action Models

Lu Qiu, Yizhuo Li, Yi Chen, Yuying Ge, Yixiao Ge, Xihui Liu

arXiv:2606.12217v1cs.CVcs.AIcs.RO

TL;DR

World Action Models can generate plausible visual futures without producing accurate actions, as their decoders may miss relevant interaction regions or respond to background perturbations. AGRA aligns video-model features with spatially structured semantic features. On real-world manipulation tasks, it improves reported ID success and generalization over baseline WAM, though the experiments cover a limited task set and the mismatch remains only partially mitigated.

  • Problem

    Plausible visual futures do not necessarily yield accurate actions, and baseline WAM decoders can miss interaction regions or respond to task-irrelevant background perturbations.

    Source: §1 · p. 2, §3.2 · p. 4

  • Method

    AGRA regularizes the world-action interface by aligning selected video-model hidden states with spatially structured semantic features from a frozen foundation visual encoder.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    AGRA reaches 80% ID success versus 34% for baseline WAM and improves Semantic, Instance-Level, and Attribute Generalization by 27%, 32%, and 32%, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AGRA is reported to improve action grounding, including interaction-region attention, object localization, affordance understanding, and robustness to task-irrelevant perturbations.

    Source: §1 · p. 2, §5 · p. 8

    Real-world experiments cover a limited task set, and AGRA only partially mitigates the video-action mismatch, which remains challenging to quantify.

    Source: §5 · p. 8

Abstract

from arXiv · show

World Action Models (WAMs) offer a promising route for robot manipulation by using video generation models to model future scene evolution before producing control actions. However, our empirical observations reveal a phenomenon: generating plausible visual futures does not always guarantee the extraction of accurate actions. To diagnose this failure, we conduct action-head attention analysis and causal interventions. We find that the action decoder fails to focus on task-relevant interaction regions and remains sensitive to perturbations in task-irrelevant areas. This reveals a representation mismatch: hidden states optimized for visual reconstruction are not inherently organized in a form useful for low-level action control. In this paper, we propose AGRA, an Action-Grounded Representation Alignment objective that regularizes the world-action interface by aligning intermediate video diffusion features with spatially coherent semantic representations from a foundation visual encoder. We evaluate AGRA on real-world manipulation tasks. Experiments show that AGRA makes world model representations more action-grounded: by focusing the action decoder on the correct interaction regions, it improves object localization accuracy and affordance understanding, and makes the policy more robust to perturbations in task-irrelevant regions. As a result, AGRA consistently improves both in-distribution performance and out-of-distribution generalization over the baseline world action model.

1 Introduction

The introduction identifies an action-grounding gap: plausible video futures do not ensure accurate actions, and baseline decoders can rely on irrelevant regions. AGRA aligns video-model features with spatially structured semantic features, and the paper reports improved real-world execution and generalization.

1 Introduction

  • Baseline WAM action decoders can miss hand-object interaction regions and remain sensitive to perturbations in task-irrelevant background areas.The introduction notes that plausible, task-consistent futures can still produce incorrect motions.Source: §1 · p. 2
  • Video diffusion features emphasize dense appearance details, whereas action prediction depends on sparse localized factors such as target objects, contact regions, and affordance geometry.This mismatch can leave the decoder attending to spurious or irrelevant regions.Source: §1 · p. 2
  • AGRA aligns selected video-diffusion hidden states with spatially structured semantic features from a frozen foundation visual encoder to regularize the world-action interface.The semantic targets encourage coherent structures for regions with similar semantic or functional roles while reducing the dominance of appearance variations.Source: §1 · p. 2
  • AGRA reaches 80% ID success versus 34% for baseline WAM and improves Semantic, Instance-Level, and Attribute Generalization by 27%, 32%, and 32%, respectively.The paper also reports improved cross-embodiment transfer ability.Source: §1 · p. 2

2 Related Work

Related work presents video generation as a way to model scene evolution and describes both explicit generation-and-control and implicit future-representation approaches for robot policies. It distinguishes AGRA from prior representation alignment, which primarily targets generative modeling.

Video Generation as World Modeling

  • Video generators are viewed as implicit world models of scene evolution and physical interaction, with embodied versions used for robot-domain training, synthetic data, evaluation, and planning.The cited work spans autoregressive and diffusion-based generation [21] [22] [23] [24] [15], interactive world models [26] [28] [29], and embodied models [16] [30] [31] [32].Source: §2 · pp. 2–3

Video Prediction and Video Representations for Robot Policies

  • Robot policies use either explicit future-frame generation followed by MPC or inverse dynamics, or implicit future representations as inputs to policies or inverse dynamics models.Explicit methods require full video denoising at inference, slowing execution; LingbotVA [11] and DreamZero [10] exemplify more recent paradigms.Source: §2 · p. 3

Representation Alignment for Generative Models

  • Prior representation-alignment work uses foundation visual features to improve diffusion generation, while AGRA repurposes alignment for action-grounded robot policy learning.The related work also distinguishes compact latent world models, which learn dynamics from scratch, from WAMs using pretrained video representations [57] [58].Source: §2 · p. 3

3 AGRA: Action-Grounded Representation Alignment

The section describes a WAM whose plausible video predictions can still yield poorly grounded actions, then introduces AGRA to align world-model features with spatially structured visual representations for control.

3.1 Baseline World Action Model Architecture

  • The baseline WAM uses a Video DiT to predict future frames and an Action DiT to transform intermediate video representations into continuous actions.The video model is evaluated once at a fixed high-noise level to extract features, while the action model iteratively denoises in action space.Source: §3.1 · p. 3

3.2 Diagnosing the Action-Grounding Gap in World Action Models

  • Attention often misses the hand-object interaction site, while causal interventions show that task-irrelevant background regions can strongly influence predicted actions.The Figure 2 example shows attention distracted by the stationary left hand and table background; interventions also identify background regions as influential.Source: §3.2 · p. 4, Figure 2 · p. 4
  • PCA visualizations show DINOv2 features organizing similar regions coherently across scenes, while Cosmos features remain more sensitive to local appearance.Cosmos features vary with background clutter, textured tablecloths, and appearance changes in functionally similar surfaces.Source: Figure 3 · p. 4, §3.2 · p. 5

3.3 Repurposing Representation Alignment for Action Grounding

  • AGRA regularizes the world-action interface by aligning selected video-model hidden states with spatially structured semantic features from a frozen visual encoder.The method projects selected world-model layers into the DINOv2 feature space and minimizes negative cosine similarity to aligned targets.Source: §3.3 · p. 5
  • The alignment loss supplements rather than replaces the baseline WAM training objectives, constraining intermediate world-model features exposed to the action decoder.Source: §3.3 · p. 5
  • Figure 4 organizes real-world evaluation around in-distribution scenarios, three out-of-distribution scenarios, and action-head attention analysis.Source: Figure 4 · p. 6

4 Experiments

Experiments on the IRON-R01-1.11 humanoid robot cover Pick-and-Place and Open-Steamer-Transfer-Bun in in-distribution and three out-of-distribution settings. AGRA improves real-world execution and action grounding, while further analyses identify design choices that affect its performance.

4.2.1 AGRA Improves Action-Grounded Control

  • AGRA achieves 80% ID success versus 34% for baseline WAM and boosts Semantic, Instance-Level, and Attribute Generalization performance by 27%, 32%, and 32%.These comparisons assess real-world execution and robustness across the reported settings.Source: §4.2.1 · p. 6
  • AGRA improves Attention in Mask Ratio and Centroid Error, indicating more precise action-head attention to control-relevant interaction regions.The metrics use manually annotated hand-object interaction masks; higher Attention in Mask Ratio and lower Centroid Error are better.Source: §4.2.1 · pp. 6–7
  • Across Mean, Zero, Shuffle, and Swap interventions, AGRA achieves a higher Matched Sensitivity Ratio than baseline WAM.The ratio compares action deviation from perturbing interaction tokens with deviation from perturbing background tokens; a higher ratio indicates greater sensitivity to interaction regions.Source: §4.2.1 · p. 7, Table 1 · p. 7
  • Real-world execution cases show AGRA producing more stable grasps, including targeting a banana’s center and closing its fingers more tightly.The baseline WAM often misses target objects because of spatial deviations.Source: §4.2.1 · p. 7, Figure 6 · p. 7

4.2.2 Further Analysis

  • Aligning Cosmos layer 8 outperforms layer 15, while simultaneous multi-layer and hierarchical alignment do not improve performance or degrade it, respectively.The authors suggest shallow semantic alignment leaves deeper layers to model motion and fine-grained spatiotemporal dynamics.Source: §4.2.2 · pp. 7–8
  • DINOv2’s object-centric, spatially coherent features yield clearer object-background boundaries than SigLIP’s diffuse features, while AGRA-SiglipL8 does not improve semantic generalization.The reported SigLIP bottleneck is execution-level failure to achieve stable grasps, not failure to comprehend instructions.Source: §4.2.2 · p. 8
  • AGRA-BridgeL8, which reuses the aligned layer-8 feature across action cross-attention layers, reaches 0% success on Open-Steamer-Transfer-Bun.The aligned layer provides object identity and scene layout but lacks spatial, geometric, and dynamic detail for precise manipulation; AGRA is an interface regularizer, not a replacement for predictive world features.Source: §4.2.2 · p. 8
  • Adding EgoDex human data yields little improvement for baseline WAM, whereas AGRA benefits substantially, especially in OOD scenarios.The authors attribute this to AGRA reducing dependence on embodiment-specific cues and exposing more invariant object-centric and interaction-relevant structure.Source: §4.2.2 · p. 8

5 Conclusion and Limitation

The paper concludes that AGRA improves action grounding and generalization in real-world manipulation. Its evaluation remains limited in task coverage, and the method does not fully resolve the video-action mismatch.

5 Conclusion and Limitation

  • Real-world manipulation experiments show AGRA improves attention to hand-object regions, object localization, causal sensitivity to task-relevant areas, and generalization.The conclusion frames AGRA as a method for regularizing the world-action interface through alignment of video diffusion features with spatially coherent semantic representations.Source: §5 · p. 8
  • Real-world experiments cover a limited set of tasks, and the method only partially mitigates the action-grounding problem without fully solving video-action mismatch.The authors note that broader tasks could better assess visual representations, including SigLIP on tasks requiring richer semantic reasoning.Source: §5 · p. 8

A.1 Dual-DiT Architecture

The section describes a dual-DiT world-action model, its training and inference procedures, and RoboCasa GR1 evaluation, where AGRA outperforms strong policy baselines.

A.1 Dual-DiT Architecture

  • The video DiT predicts future visual latents from the current observation and instruction, while the action DiT generates action chunks conditioned on predictive visual features and robot state.A multi-layer bridge supplies different video-model layer representations to the action decoder’s cross-attention layers.Source: §A.1 · pp. 13–14

A.2 Action Sampling

  • At inference, one high-noise video pass provides guidance for multi-step action denoising, reducing latency while retaining predictive visual guidance.Preliminary experiments found higher task success with single-step than four-step video denoising features.Source: §A.2 · p. 14

A.3 Optimization Strategy

  • Video prediction uses randomly sampled noise levels, while action training uses a separate fixed-high-noise video pass to match the action head’s inference inputs.The full objective combines video-prediction and action-prediction flow-matching losses, balanced by λact.Source: §A.3 · p. 15

B.1 RoboCasa GR1 Tabletop Simulation

  • RoboCasa GR1 evaluation covers 24 tabletop tasks over 50 episodes each, with full-data comparisons and few-shot ablations plus three out-of-distribution test scenarios.The full-data regime uses 24,000 trajectories; the ablation regime uses 2,400.Source: §B.1 · pp. 15–16

B.2.1 Comparison Against State-Of-The-Art Policies

  • 66.4% overall success on RoboCasa GR1 exceeds GR00T by an 18.8-point absolute margin and improves over FLARE, DiT4DiT, and LDA-1B by more than 10%.Figure 9 presents the full-training-data evaluation.Source: Figure 9 · p. 16, §B.2.1 · p. 16

B.2.2 Ablation study

The ablations examine AGRA’s effect, alignment placement, layer choice, and training design; the framework description specifies the model configuration used.

B.2.2 Ablation study

  • Jointly optimizing the world model and action head lets action gradients reshape reconstruction-focused hidden states into an action-conditioned representation space.Freezing the backbone after video-only fine-tuning yields a low performance ceiling, with the action head tending to overfit to proprioception and ignore vision.Source: §B.2.2 · p. 17
  • 66.28 on Unseen Combinations falls to 59.57 when AGRA is removed, with the decline especially pronounced in out-of-distribution generalization.Simulation gains are marginal; baseline failures often involve post-grasp slippage, which visual improvements alone cannot solve.Source: §B.2.2 · p. 17
  • AGRA performance peaks at Layer 8; applying the constraint at shallow layers, roughly one third of model depth, yields the best performance.Figure 10 shows the downstream manipulation performance across layer choices.Source: §B.2.2 · p. 17
  • The separate pure-noise action-conditioning pass outperforms shared video denoising, keeping training representations consistent with those consumed at inference.This focuses the action head on high-noise video representations used at inference.Source: §B.2.2 · pp. 17–18
  • Applying AGRA to the action-conditioning pass outperforms aligning the random-timestep video-generation pass and provides more explicit semantic grounding.The authors report superior empirical robustness for alignment on the exact hidden states consumed by the action decoder.Source: §B.2.2 · p. 18
  • Multi-layer bridges outperform every single-layer configuration, combining high-level object-centric topology with fine-grained spatial dynamics for continuous control.Figure 11 compares single-layer and multi-layer bridge performance.Source: §B.2.2 · p. 18

C.1 Model Framework

  • The framework uses eight selected Cosmos layers, applies default AGRA at layer 8 with frozen DINOv2 targets, and generates actions with four denoising steps.Cosmos features are extracted after one inference denoising step.Source: §C.1 · p. 18

C.2.1 Dataset Collection

The real-world data collection combines factory and human demonstrations for pretraining with smaller laboratory datasets for task-specific fine-tuning.

C.2.1 Dataset Collection

  • The real-world tasks are Pick-and-Place and Open-Steamer-Transfer-Bun, a more challenging bimanual task requiring coordinated lid removal and bun transfer.The latter uses the left hand to set aside the lid and the right hand to move the bun to a plate.Source: §C.2.1 · p. 18
  • Pretraining combines 40,000 factory trajectories with 37,000 EgoDex human demonstrations, then trains for 60,000 steps.The factory data spans 92 hours; the EgoDex subset spans 42 hours.Source: §C.2.1 · pp. 18–19
  • Laboratory fine-tuning uses 560 Pick-and-Place trajectories and 180 Open-Steamer-Transfer-Bun trajectories for 2,000 steps.The Pick-and-Place data covers varied objects; the bun task includes three bun types and plates of different colors.Source: §C.2.1 · p. 19

C.2.2 Evaluation

Real-robot evaluation measures task completion across seen-object performance and three forms of out-of-distribution generalization.

C.2.2 Evaluation

  • Each real-robot trial counts as successful only when the instructed manipulation goal is completed.The evaluation includes one in-distribution regime and three out-of-distribution regimes.Source: §C.2.2 · p. 19
  • In-distribution Pick-and-Place evaluation tests 10 seen objects at five tabletop positions, for 50 trials.Source: §C.2.2 · p. 19
  • Semantic generalization tests instruction-following across distractors and target-container choices after swapping their positions, for 40 trials.The scenarios comprise eight object-selection cases and two placement cases, each evaluated under four trials.Source: §C.2.2 · p. 19
  • Instance-level generalization tests 10 unseen objects at five positions for 50 trials; attribute generalization tests changed plates, tablecloths, and bun attributes for 40 trials.Source: §C.2.2 · p. 19

C.3 Training Details

AGRA is trained through staged video and video-action optimization, then fine-tuned for real-world tasks. Real-world rollouts include in-distribution and several forms of out-of-distribution generalization.

C.3 Training Details

  • Cosmos first receives 4,000–5,000 steps of video-only training before video-action joint training, using loss weights 1.0, 1.0, and 0.01 for video, action, and AGRA losses.Source: §C.3 · p. 19
  • Real-world training uses 60,000 joint steps and 2,000 task-specific fine-tuning steps, with Cosmos frozen during fine-tuning; sampling every three frames matches 16 RGB frames to a 48-step action chunk.Source: §C.3 · pp. 19, 21
  • Simulation training runs for 80,000 steps in the full-data regime or 40,000 in the few-shot regime, with a 16-step action chunk.Source: §C.3 · p. 21

D.1 Real-World Execution Cases

  • AGRA identifies correct target regions and executes more precise reaches and grasps than baseline WAM in the real-world examples.Source: §D.1 · p. 21

D.2 Analysis for Representation Alignment Layer

The analysis examines where Cosmos develops instruction-conditioned object correspondence and how the action head reads video representations. It also compares the spatial structure of DINOv2 and SigLIP features.

D.2 Analysis for Representation Alignment Layer

  • At Cosmos layer 8, instruction tokens attend most spatially consistently to their corresponding objects, making it a suitable alignment stage while deeper layers remain available for motion and geometry.Source: §D.2 · p. 21, Figure 14 · p. 22
  • Action-head attention progresses from broad scene context in early layers to robot hands and hand-object interaction regions in middle and later layers.Source: §D.2 · p. 22, Figure 15 · p. 23

D.3 Visualization for Different Semantic Feature Types

  • DINOv2 features are more spatially structured and object-centric than SigLIP features, with clearer boundaries, geometry, and functional-region separation.Source: §D.3 · p. 22, Figure 16 · p. 23
Loading 2606.12217v1…