PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Representations for World-Action Modeling

Haoyi Jiang, Liu Liu, Xinjiang Wang, Zhihao Sun, Zequn Chen, Sen Wang, Xinjie Wang, Xia Chen, Jingfeng Yao, Weiheng Zhao, Shanglin Yuan, Zhizhong Su, Wei Sui, Wenyu Liu, Xinggang Wang

arXiv:2609.38163v1cs.CVcs.RO

TL;DR

World-action models jointly predict future observations and learn robot policies, but the representation connecting those tasks is not well characterized by reconstruction fidelity alone. ReWAM calibrates and temporally organizes DINO features, then routes action-loss gradients to shape its compact state representation. It reports strong results on RoboTwin 2.0 and RoboDojo without generative video pre-training.

  • Problem

    World-action models need a representation interface for prediction and control, but reconstruction-oriented latents may prioritize appearance rather than control-relevant information.

    Source: §1 · p. 1

  • Method

    ReWAM calibrates frozen DINO features, organizes them into compact states with a Temporal Representation Bottleneck, and routes only action-loss gradients to that bottleneck.

    Source: §1 · p. 2

  • Results

    93.6% success in both RoboTwin 2.0 settings; with embodied pre-training, ReWAM reaches a 12.29 average score and 8.28% success rate on RoboDojo.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Effective world-action modeling depends on representation construction and adaptation, since reconstruction fidelity alone does not determine control performance.

    Source: §5 · p. 10

Abstract

from arXiv · show

World-action models jointly learn robot policies and predict future observations, making the representation space an interface between control and prediction. We study the design of this space through controlled comparisons, finding that neither reconstruction fidelity nor pre-trained perceptual features alone ensure effective policy learning. These findings motivate ReWAM, a representation-centric world-action model built on pre-trained DINO features. Feature Calibration and a Temporal Representation Bottleneck organize these features into compact world states suited to dynamics modeling. Action-Grounded Representation Shaping routes only action-loss gradients to the bottleneck, thereby letting the policy shape what the representation encodes while the world model learns how it evolves. Without generative video pre-training, ReWAM achieves 93.6% success on RoboTwin 2.0. On RoboDojo, it achieves an average score of 12.29 and a success rate of 8.28% using approximately 600 hours of embodied pre-training data.

1 Introduction

World-action models need representations that support both prediction and control, yet reconstruction fidelity and perceptual pretraining alone do not ensure control utility. ReWAM calibrates and temporally organizes DINO features, then uses action-grounded shaping to achieve strong results without generative video pre-training.

1 Introduction

  • Raw DINO features outperform Video-VAE latents on RoboTwin 2.0 despite lower pixel-reconstruction SSIM, showing reconstruction fidelity alone does not establish control utility.Source: §1 · p. 1
  • Feature Calibration improves average RoboTwin 2.0 success by 12.20 percentage points, while addressing frame-wise features' lack of explicit temporal variation and possible mismatch with diffusion training.Source: §1 · p. 1
  • ReWAM calibrates frozen DINO features and organizes them into compact states with a Temporal Representation Bottleneck; AGRS routes action-loss gradients to the bottleneck while the world model learns state transitions.The design assigns the policy the role of shaping what the representation encodes and the world model the role of learning how it evolves.Source: §1 · p. 2
  • 93.6% success in both RoboTwin 2.0 settings; with embodied pre-training, ReWAM reaches a 12.29 average score and 8.28% success rate on RoboDojo.On RoboDojo, it also leads compared methods without embodied pre-training.Source: §1 · p. 2
  • The study demonstrates ReWAM without generative video pre-training across two simulation benchmarks and real-robot tasks.Source: §1 · p. 2

2 Related Work

Prior world-action models connect visual prediction with policy learning through video-generation systems, representation features, and visual-action tokenizers. Related representation work studies predictive feature spaces and generative latents; ReWAM focuses on constructing and adapting a shared space for prediction and policy learning.

2.1 World-Action Models

  • World-action models combine video generation and action learning, while representation-space approaches predict visual features or subgoals and adapt interfaces with tokenizers; ReWAM adds calibration, temporal organization, and asymmetric gradient routing.Examples include joint denoising [3], world/action streams [1] [2] [14], Fast-WAM [4], LDA-1B [5], DexWorldModel [6], and LaWAM [29].Source: §2.1 · p. 3

2.2 Representations for Prediction and Generation

  • Predictive feature-space work includes V-JEPA [34] [35], DINO-world [36], and Spa3R [37], while generative representation methods align or use pre-trained latents [38] [39] [41] [42] [43] [44] [45].Probing studies examine representation content and support for spatial understanding, dynamics prediction, and control [8] [9] [47]; ReWAM instead addresses construction and adaptation for joint prediction and policy learning.Source: §2.2 · p. 3

3 ReWAM

ReWAM pairs a policy with a predictor of future visual representations, using calibrated DINO features and compact temporal states. Its asymmetric gradient routing lets action learning shape the representation while the world model learns its dynamics.

3.1 Overview

  • ReWAM predicts future visual representations and action chunks from encoded observations, proprioception, and language through separate world and action branches.At inference, the world branch processes the current state once to condition action generation without generating future states.Source: §3.1 · pp. 3–4

3.2 Feature Calibration

  • Feature Calibration averages selected DINO layers and adds the final layer’s spatial mean to each patch, combining encoder depths with image-level context.Frozen-feature baseline targets use fixed pre-training statistics; ReWAM normalizes changing TRB outputs with running statistics.Source: §3.2 · p. 4
  • The representation-aware schedule samples separate flow times for future states, shifting toward noisier inputs for larger prediction targets while keeping the current state clean.The shift accounts for target token count and channel width.Source: §3.2 · p. 4

3.3 Temporal Representation Bottleneck

  • The TRB converts frame-wise DINO features into compact world states by grouping 2 × 2 × 2 spatiotemporal blocks and projecting attended tokens to 128 dimensions.Its outputs use per-channel running statistics updated by exponential moving average and held fixed at inference.Source: §3.3 · pp. 4–5

3.4 Action-Grounded Representation Shaping

  • Both branches use flow matching, while AGRS sends only action-loss gradients through current state z0 to the TRB and stops world-loss gradients at representations.The action objective shapes the TRB; the future-prediction objective trains the world model in the resulting representation space.Source: §3.4 · p. 5

4 Experiments

Experiments compare ReWAM across simulation, real-robot evaluation protocols, and representation ablations. ReWAM leads the reported RoboTwin 2.0 and RoboDojo results, while the ablations distinguish reconstruction fidelity, feature calibration, temporal organization, and action-grounded shaping.

4.1 Setup

  • RoboTwin 2.0 evaluation averages success across 50 tasks, while RoboDojo reports partial-progress score and binary success averaged across five capability categories.The evaluation protocols use 100 episodes per RoboTwin task per setting and compare RoboDojo methods within each embodied-pretraining setting.Source: §4.1 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6

Real-Robot Evaluation.

  • The real-robot protocol evaluates three bimanual Piper tasks after 10k steps of task-specific post-training for each method, reporting success over 20 trials per task.ReWAM and Fast-WAM first receive 75k steps of embodied pre-training on the same 600 hours of demonstrations.Source: Real-Robot Evaluation · p. 7, §4.1 · p. 7

4.2 Main Results

  • ReWAM reaches 93.6% success in both RoboTwin 2.0 settings, beating the strongest method by 0.2 points on clean and 1.4 on random, and leads RoboDojo without embodied pre-training at 9.42 average score and 5.82% success.With embodied pre-training, its RoboDojo average score and success rate rise to 12.29 and 8.28%.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 7

4.3 Representation Design and Ablations

  • Raw DINO features outperform Video-VAE latents by 4.90/9.22 percentage points on clean/random despite lower RGB reconstruction SSIM (0.70 versus 0.91).Removing generative video pre-training reduces Fast-WAM success by 21.68/24.11 percentage points on clean/random.Source: §4.3 · p. 7

Feature Calibration.

  • Feature calibration improves average success by 12.20 points over raw DINO features through successive gains from normalization, a representation-aware noise schedule, and multi-layer aggregation.The successive gains are 3.92, 1.41, and 6.87 percentage points, respectively.Source: Feature Calibration · p. 7, Table 3 · p. 8

Temporal Representation Bottleneck.

  • Fixed random projection improves average success by 2.89 points over calibrated frame-wise features, while RGB-reconstruction codec training substantially reduces success and DINO-reconstruction codec training falls slightly below fixed projection.Fixed random projection concatenates spatial features across a temporal window and projects them to 512 dimensions.Source: Temporal Representation Bottleneck · pp. 7–8, Table 4 · p. 8

Action-Grounded Representation Shaping.

  • AGRS reaches 90.70%/89.12% clean/random success, exceeding fixed random projection by 3.94/3.18 points; routing world-loss gradients to the TRB lowers average success by 6.30 points.AGRS performs best among tested widths at 128 dimensions and remains effective at 64, while fixed random projection loses 3.29 average-success points when reduced from 512 to 128 dimensions.Source: Action-Grounded Representation Shaping · p. 8, Table 5 · p. 8

4.4 Action-Related Structure in Representations

Across two tasks, AGRS representations show clearer action-direction grouping and higher nearest-neighbor direction agreement than Video-VAE representations. In physical-robot trials, ReWAM exceeds Fast-WAM overall, with gains on the more challenging tasks.

4.4 Action-Related Structure in Representations

  • AGRS has higher mean motion-direction agreement among ten nearest neighbors than Video-VAE on both tasks, and its representation variations show clearer local grouping by commanded direction.The comparison uses the same 242 transitions per task across the three projections.Source: §4.4 · p. 8, Figure 2 · p. 9

4.5 Real-Robot Evaluation

  • ReWAM succeeds in 55 of 60 real-robot trials (91.7%), versus 48 of 60 (80.0%) for Fast-WAM, with gains of 15 points on Fold Clothes and 20 on Unpack Lunchbox.Both methods complete all 20 Collect Objects trials.Source: Figure 3 · p. 9, §4.5 · p. 10

5 Conclusion

The conclusion argues that world-action modeling depends on constructing and adapting its representation space, not reconstruction fidelity alone. ReWAM combines calibrated, temporally organized perceptual features with action-grounded shaping and demonstrates effectiveness without generative video pre-training.

5 Conclusion

  • ReWAM combines feature calibration and temporal organization with action-grounded shaping, allowing the policy to shape what the representation encodes while the world model learns its dynamics.Results on RoboTwin 2.0, RoboDojo, and real-robot manipulation demonstrate the design without generative video pre-training.Source: §5 · p. 10

A Implementation and Evaluation Details

The implementation samples multi-view observation clips over a 64-action horizon and builds compact spatiotemporal states from camera-specific DINO patch grids. At inference, the world branch caches current-state features while the policy generates and executes action chunks.

Temporal sampling and horizons

  • Training clips contain ten multi-view observations sampled every eight control steps, forming a 64-action horizon; inference uses historical and current observations, substituting the current frame when history is missing.Padded targets are excluded from their corresponding losses.Source: A Implementation and Evaluation Details · p. 16

TRB architecture

  • The TRB converts head- and wrist-camera DINO grids into 120 tokens per state, initially 8192-dimensional, then mixes views within each temporal pair using one eight-head attention layer.The head grid is 16 × 20 patches and each wrist grid is 8 × 10.Source: A Implementation and Evaluation Details · p. 16

Inference and action execution

B Embodied Pre-training Data Scale

ReWAM uses approximately 600 hours of embodied pre-training data on RoboDojo, substantially less than the reported amounts for listed baselines. With this data, it achieves an average score of 12.29 and a success rate of 8.28%.

B Embodied Pre-training Data Scale

  • With approximately 600 hours of embodied pre-training data, ReWAM achieves an average score of 12.29 and a success rate of 8.28% on RoboDojo.The reported duration is substantially less than the amounts for the listed baselines; the matched Fast-WAM baseline uses the same 600-hour dataset.Source: B Embodied Pre-training Data Scale · p. 16, Table 8 · p. 16
  • Differences in data composition should be considered when interpreting the comparison between ReWAM and the listed baselines.Source: B Embodied Pre-training Data Scale · p. 16
  • Table 8 compares the reported durations of embodied pre-training datasets for selected methods evaluated on RoboDojo.Hours measure embodied training-data duration; “+” marks a reported lower bound and “≈” an approximate duration.Source: Table 8 · p. 16

C Contribution of World Co-training

Disabling the world loss in the raw DINO WAM configuration isolates the contribution of world co-training. World co-training improves clean/random success by 2.60/2.99 percentage points beyond gains from pre-trained DINO features alone.

C Contribution of World Co-training

D Detached Pixel-Reconstruction Probe

The SSIM probe measures how much pixel information can be recovered from the representation. Its auxiliary decoder reconstructs future RGB observation pairs from detached latents without affecting the encoder or WAM.

D Detached Pixel-Reconstruction Probe

E Representation Analysis Protocol

The representation analysis selects RoboTwin transitions using action and state records, then balances highlighted transitions by direction and displacement magnitude. t-SNE projections include all selected transitions, but are fitted independently and cannot be compared by orientation or absolute position.

E Representation Analysis Protocol

  • The analysis examines all 500 random-setting RoboTwin demonstrations per displayed task and selects 242 transitions each for Move Stapler Pad and Place Mouse Pad.Selection uses action and state records independently of representation features or projections.Source: E Representation Analysis Protocol · p. 17
  • Transitions qualify for highlighting with displacement of at least 0.04 m within 45° of their assigned signed axis.Directions with fewer than 20 eligible candidates are excluded, and equal counts are sampled per direction within displacement-magnitude bins.Source: E Representation Analysis Protocol · p. 17
  • Direction-balanced highlighting yields 144 Move Stapler Pad transitions and 168 Place Mouse Pad transitions across +X, −X, +Y, and −Y.The subsets contain 36 and 42 transitions per direction, respectively; bins with fewer than three candidates in a retained direction are omitted.Source: E Representation Analysis Protocol · p. 17
  • Each t-SNE fit includes all 242 transitions, with transitions outside the highlighted subset shown in gray.Fits are independent, so orientations and absolute positions cannot be compared across panels.Source: E Representation Analysis Protocol · p. 17
Loading 2609.38163v1…