PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Distilling Future Video Representations for World Action Models

Huiwon Jang, Suhyeok Jang, Hyeonmo Kang, Seungku Kim, Jinwoo Shin, Sihyun Yu

arXiv:2610.30320v1cs.ROcs.CV

TL;DR

Single-pass WAMs reduce inference cost by using noise-generated future video representations to condition action prediction, but the quality of those representations matters. FRED uses self-distillation toward representations from ground-truth future observations and adds historical context. It outperforms competing WAMs across real-world generalization and human-to-robot transfer, including tasks absent from robot training data.

  • Problem

    Single-pass WAMs avoid iterative video denoising, but their action decoders depend directly on how well noise-generated hidden states capture future robot-object dynamics.

    Source: §1 · p. 3

  • Method

    FRED distills noise-generated future representations toward targets from ground-truth future observations and conditions the VDM on historical observations.

    Source: §1 · p. 3, §3.2 · p. 6

  • Results

    FRED consistently outperforms competing WAMs across real-world settings, raising human-to-robot transfer success from 19.4% to 50.0% on tasks absent from robot training data.

    Source: §1 · p. 3

  • Takeaways & Limitations

    FRED improves robot control in zero-shot environment generalization and human-to-robot skill transfer settings.

    Source: §5 · p. 11

    FRED relies on the language-understanding capabilities of its video diffusion backbone and adds no language-grounding supervision; without human-video co-training, it achieves 0% instruction-following on human skill-transfer tasks.

    Source: F. Limitations and Future Work · p. 24

Abstract

from arXiv · show

World Action Models (WAMs) have emerged as a promising approach to learning generalist robot policies using rich spatiotemporal priors from pre-trained video diffusion models. A recent line of WAMs improves inference efficiency by conditioning a separate action decoder on intermediate future video representations extracted from a single denoising step, avoiding costly iterative denoising required for video generation. This design makes action prediction depend directly on the quality of these future representations. In this paper, we introduce Future REpresentation learning via self-Distillation with temporal context (FRED), a framework that trains WAMs to improve the quality of future video representations from noise in a single denoising step. Specifically, FRED uses self-distillation to predict future representations from noise, using representations extracted from ground-truth future observations as targets. We further condition the video model on historical observations, providing temporal context for predicting future representations better. Across diverse real-world manipulation scenarios, FRED consistently outperforms competing WAMs. It achieves particularly large gains on long-horizon and out-of-domain tasks under zero-shot environment generalization, and substantially improves human-to-robot skill transfer, reaching 50.0% task success on tasks absent from robot training data and demonstrated only in human videos, compared with 19.4% for the best competing WAM.

1. Introduction

Single-pass WAMs make action prediction efficient by conditioning a separate decoder on future video representations, but control depends on how informative those representations are. FRED improves them through self-distillation and historical context, and outperforms competing WAMs across real-world generalization and human-to-robot transfer settings.

1. Introduction

  • WAMs build on pre-trained video diffusion models and jointly predict future frames and actions to bring video-learned spatiotemporal priors to robot control.The introduction describes these models as a promising approach to generalist robot policies.Source: §1 · p. 3
  • Single-pass WAMs use a separate action decoder conditioned on VDM hidden states, reducing inference latency by avoiding iterative denoising.Because the decoder directly uses these states, action prediction depends on how well they capture future robot-object dynamics.Source: §1 · p. 3
  • Noise-generated VDM hidden states become more predictive of future motion during WAM training, but their probing accuracy remains below that of clean-future representations.Their structural similarity to clean-future representations also increases during training (Figure 1).Source: §1 · p. 3, Figure 1 · p. 4
  • FRED aligns noise-generated representations with targets from ground-truth future observations through self-distillation and conditions the VDM on short observation histories.The paper reports that FRED reduces the future-motion probing gap and improves structural similarity to clean-future representations.Source: §1 · pp. 3–4
  • FRED improves WAM results across zero-shot generalization and human-to-robot transfer, raising long-horizon success from 61.7% to 81.7%, out-of-domain success from 10.4% to 52.1%, and transfer success from 19.4% to 50.0%.At the 14B scale, the long-horizon and out-of-domain comparisons are reported; transfer tasks were absent from robot training data.Source: §1 · p. 3

2. Related Work

Related work uses video diffusion models for robot control, with recent WAMs reducing inference cost by separating action decoding from video generation. FRED focuses on improving the future representations used by one such single-step design, using observed future frames as distillation targets rather than a fixed external representation.

World Action Models

  • Recent WAMs reduce video denoising to one step; some mask future features or decode from video-backbone states, while another line conditions actions on future representations from noise.FRED follows the latter design and seeks more informative representations for action generation.Source: §2 · p. 4

Diffusion Models and Representation Learning

  • Representation-learning work aligns diffusion features with clean image or video features, uses the diffusion model's own lower-noise features, or integrates self-supervision into flow matching.FRED instead distills noise-generated WAM representations toward features from observed future frames, without aligning them to a fixed external representation.Source: §2 · p. 4
  • FRED's overview treats the VDM as a future-representation generator and applies self-distillation to improve noise-generated representations.The method also uses historical observations as conditioning input for temporal context.Source: Figure 2 · p. 5, §2 · p. 5

3. Method

FRED treats the VDM as a future-representation generator whose single-pass hidden state conditions a separate action decoder. It trains noise-generated representations toward teacher targets from noisy future observations, adds historical context, and combines distillation with video and action training objectives.

3.1. Overview

  • At inference, a separate diffusion-based action decoder uses an intermediate VDM hidden state generated from noise in a single forward pass.The VDM is interpreted as a future-representation generator, with h = fθ(ϵ|o_t, c).Source: §3.1 · p. 5

3.2. FRED

  • FRED trains the student VDM to predict future representations from noise, using a momentum-teacher encoder on noisy ground-truth future observations to produce targets.The targets are extracted under the same conditioning as the student representations.Source: §3.2 · p. 5
  • FRED trains representations by minimizing the patch-averaged KL divergence between student and teacher output distributions.The student and teacher projection heads produce the distributions; teacher logits are centered and sharpened to prevent representation collapse.Source: §3.2 · p. 6
  • Both student and teacher encoders condition on an H-frame history of observations, adding temporal cues such as recent robot motion to future prediction.In practice, the policy predicts multiple future actions and observations at once.Source: §3.2 · pp. 5–6
  • The WAM objective combines video flow-matching and self-distillation losses, while action flow matching trains the action decoder rather than the representation generator.At inference, the decoder generates actions conditioned on the current proprioceptive state and the future representation.Source: §3.2 · p. 6

4. Experiments

The experiments test FRED in zero-shot transfer to a new environment, human-video co-training, and analyses of its components and future representations. Across these settings, they assess policy performance, representation quality, and inference cost.

4.2. Zero-shot Environment Generalization

  • At 14B, FRED outperforms DreamZero on long-horizon tasks (81.7% vs. 61.7%) and out-of-domain tasks (52.1% vs. 10.4%), with gains across all task types.In the 5B setup, it also improves long-horizon performance from 31.7% to 46.7% and out-of-domain success from 10.4% to 37.5% over the strongest competing WAM.Source: Table 1 · p. 7, §4.2 · p. 8

4.3. Human-robot Co-training

  • On in-domain composite tasks, robot-only FRED reaches 70.8% task progress versus 50.0% for the strongest baseline, and human-video co-training raises it to 75.0%.Co-training also raises FRED's atomic-task success from 91.7% to 97.2%.Source: §4.3 · p. 9
  • With human-video co-training, FRED achieves 50.0% task success on human-only demonstrated skills, while DreamZero reaches 0% and AGRA 0%.FRED's instruction-following rate is 97.2%; without human-video co-training, FRED and the baselines achieve near-zero task success.Source: Table 2 · p. 8, §4.3 · p. 9

4.4. Analysis and Ablation Studies

  • Removing self-distillation lowers success from 68.8% to 65.1%, removing historical conditioning to 64.8%, and removing both to 61.4%.The two components provide complementary gains in the RoboCasa Kitchen ablation.Source: §4.4 · p. 9, Table 3 · p. 9
  • Across RoboCasa Kitchen, GR-1 Tabletop, and LIBERO, FRED improves more consistently than VideoREPA, SRA, and Self-Flow.The comparison fine-tunes models on each benchmark's training data for 100K iterations.Source: §4.4 · p. 9, Table 4 · p. 9
  • Task success correlates more strongly with noisy future-representation quality (r=0.89) than with one-step video quality (r=0.24) or observed-future representation quality (r=0.49).The analysis uses DiT4DiT variants with different video-model training objectives on RoboCasa Kitchen.Source: §4.4 · p. 10, Figure 4 · p. 10
  • On RoboCasa Kitchen, teacher noise τ=0.25, layer index 18, and λSD=0.01 perform best among evaluated choices; stopping action-loss gradients raises success from 65.6% to 68.8%.A larger λSD=0.1 is comparable, while shallower or deeper feature layers degrade performance.Source: §4.4 · pp. 10–11, Table 5 · p. 10
  • FRED improves success with both Wan2.2-5B and Cosmos-Predict2.5-2B backbones, while at 14B its inference is approximately 8× faster than DreamZero.At 5B, latency is comparable to VLA baselines and only 1.2× that of DiT4DiT with one-frame input.Source: §4.4 · p. 11, Table 6 · p. 11, Table 7 · p. 11

5. Conclusion

FRED explicitly learns future representations for action generation from a single video denoising step, using self-distillation and observation history. Experiments show improved robot control in complex real-world scenarios, including zero-shot generalization and human-to-robot skill transfer.

5. Conclusion

  • FRED learns the future representations used for action generation from a single video denoising step via self-distillation.It aligns representations with self-distillation targets extracted from noisy future observations and uses observation history as additional temporal context.Source: §5 · p. 11
  • FRED consistently improves robot control in complex real-world scenarios, including zero-shot environment generalization and human-to-robot skill transfer.Source: §5 · p. 11
  • The authors hope this work encourages research toward efficient, generalizable robot foundation models that better leverage large-scale pre-trained video models.Source: §5 · p. 11

A. Implementation Details

Implementation uses video diffusion backbones with historical-frame conditioning and predicts future frames for action generation. Training, inference, simulation configurations, and baseline reproduction procedures vary by experiment setting.

A. Implementation Details

  • Real-world experiments condition the video model on four historical frames plus the current frame and predict eight future frames with a temporal stride of three.Franka uses Wan2.2-TI2V-5B or Wan2.1-I2V-14B; OpenArm uses Wan2.2-TI2V-5B.Source: A. Implementation Details · p. 15
  • FRED uses the same intermediate layer for self-distillation and action conditioning, with γ1 = 0.1, γ2 = 0.04, teacher noise level τ = 0.25, and λSD = 0.01.The teacher is updated with an exponential-moving-average warmup decay schedule.Source: A. Implementation Details · p. 15
  • Real-world Franka models train for 150K iterations at 5B and 100K at 14B, while OpenArm models train for 50K iterations.OpenArm robot-only batches contain 32 samples; co-training batches contain 32 robot and 32 human samples, and inference uses four denoising steps.Source: A. Implementation Details · p. 15
  • Baseline checkpoints are used when available; otherwise methods are trained or reproduced following their official implementations or reported procedures.AGRA is reproduced in-house, while VideoREPA, SRA, and Self-Flow follow their described alignment procedures.Source: A. Implementation Details · pp. 15–16
  • Simulation experiments use a Wan2.2-TI2V-5B backbone, a 16-layer DiT action decoder, 100K training iterations, and a 16-step action prediction horizon.The model conditions on four historical frames plus the current frame and predicts eight future frames at temporal stride two.Source: A. Implementation Details · p. 16
  • Image augmentations used for FRED are summarized in Table 9.Source: A. Implementation Details · p. 16, Table 9 · p. 16

B.1. Zero-shot Environment Generalization Tasks on DROID

The DROID evaluation tests policies in a new environment without environment-specific training, using four task sets that cover in-domain and out-of-domain behavior. Trials report partial-credit success or progress under task-specific action-step limits.

B.1. Zero-shot Environment Generalization Tasks on DROID

  • Policies trained on DROID are evaluated in a new environment without additional training there, using a Franka Research 3 with a Robotiq 2F-85 gripper.The evaluation setting differs from training in location, background, objects, and some instructions.Source: §B.1 · p. 17, Figure 5 · p. 17
  • Evaluation covers in-domain easy, in-domain long-horizon, out-of-domain, and object-grounding task sets.A task is out-of-domain when its instruction is absent from DROID; object grounding tests discrimination between similarly colored or shaped objects.Source: §B.1 · p. 18, Figure 6 · p. 18
  • Each task is evaluated over 12 trials with partial credit, with limits of 1,500 action steps for long-horizon tasks and 500 for other tasks.Tasks include clearing a table, revealing an object under a mug, and selecting specified cups or cutlery.Source: §B.1 · p. 18

B.2. Human-robot Co-training on OpenArm

The OpenArm experiments combine robot and egocentric human-video data and evaluate seven manipulation tasks, including human skill transfer. The baselines span vision-language-action models and video-diffusion-based models.

B.2. Human-robot Co-training on OpenArm

  • OpenArm uses two 7-DoF arms with 6-DoF hands, a 2-DoF neck, and stereo egocentric camera views.The arms are teleoperated with an exoskeleton, while gloves capture finger motion for conversion to joint angles.Source: §B.2 · p. 19, Figure 7 · p. 19
  • The robot dataset contains 553 demonstrations and 192K frames across 53 instructions, four skills, and six object categories.Human co-training adds 1,187 egocentric videos and 321K frames across seven skills and seven object categories.Source: §B.2 · p. 20, Figure 8 · p. 20, Figure 9 · p. 20
  • Seven OpenArm tasks are evaluated over 12 trials each, with mean task scores reported as percentages and partial credit for partial completion.Tasks include atomic and composite in-domain manipulation as well as transfer tasks such as novel-object placement and pouring coffee.Source: §B.2 · p. 21, Figure 10 · p. 21

C. Baselines

  • The considered baselines include diffusion-based VLAs and WAMs, with WAMs using either shared video-action modeling or separate video and action transformers.Separate-transformer variants differ in how video features condition action prediction or whether future video prediction is omitted at inference.Source: C. Baselines · p. 22

D. Analysis Setup Details

This appendix specifies the analysis and generation setups, then describes the flow-matching process underlying FRED’s one-step setting.

D.1. Attentive Probe

  • The attentive probe classifies 15 motion primitives from frozen video tokens, using labels based on 16-frame clips where gripper changes take priority.It uses a learnable query with cross-attention; representations come from layer 18, and the probe is trained on 300 demonstrations per task and evaluated on 50.Source: §D.1 · p. 23

D.2. CKNNA

  • CKNNA compares noisy representations across checkpoints with clean future-observation representations from the Base WAM at 100K iterations.The analysis uses 4,096 validation clips and mean-pools 192 future-frame tokens from layer 18.Source: §D.2 · p. 23

D.3. Generated Video Quality and One-Step Distillation

  • One-step video quality is measured with FVD against RoboCasa clips; DMD2 distills a fine-tuned Wan2.2-TI2V-5B EMA teacher into a one-step student.The model is fine-tuned for 100K iterations, followed by 5K iterations of distillation.Source: §D.3 · p. 23

E. Flow Matching Process in FRED

  • Flow matching trains a conditional velocity network to reconstruct ε−x; inference integrates from noise, while this setting uses one Euler step and extracts hidden states for action generation.The one-step pass uses x_τ=1=ε and Δτ=1.Source: E. Flow Matching Process in FRED · p. 24

F. Limitations and Future Work

The authors identify limits in language grounding, cross-embodiment transfer, and the robot-domain priors of current video backbones.

F. Limitations and Future Work

  • FRED adds no language-grounding supervision, so instruction following relies on the underlying video diffusion backbone’s language understanding.Without human-video co-training, FRED has 0% instruction-following performance on human skill-transfer tasks; co-training substantially improves it.Source: F. Limitations and Future Work · p. 24
  • Human-to-robot transfer remains implicit because joint video training does not explicitly learn representation correspondences across embodiments.The authors suggest explicit cross-embodiment representation learning as future work.Source: F. Limitations and Future Work · p. 24
  • Current video diffusion models are predominantly trained on human videos and may require substantial adaptation for downstream robotic tasks.The authors propose exploring larger video backbones with stronger robot-domain priors.Source: F. Limitations and Future Work · p. 24

G. Full Benchmark Results

The appendix reports full per-task results for DROID and OpenArm, using different performance metrics for each task category.

G.1. DROID

  • Table 11 reports DROID-trained policies’ per-task zero-shot results in a new real-world environment, using progress rates for in-domain long-horizon tasks and success rates for other task types.The policies are evaluated without environment-specific fine-tuning.Source: Table 11 · p. 25, §G.1 · p. 25

G.2. OpenArm

  • Table 12 reports OpenArm per-task results for robot-only and human-robot co-training, with metrics spanning atomic tasks, composite tasks, and human skill transfer.The table includes task success and instruction-following performance for human skill-transfer tasks.Source: Table 12 · p. 26, §G.2 · p. 26

H. Additional Simulation Benchmark Results

On the RoboCasa Kitchen simulation benchmark, FRED has the highest average success rate among the compared state-of-the-art robot foundation models.

H. Additional Simulation Benchmark Results

  • FRED achieves the highest average success rate on the RoboCasa Kitchen benchmark among the compared state-of-the-art robot foundation models.For this comparison, FRED is trained with global batch size 256 for 100K iterations; the authors note the results differ from Table 4 because training configurations were adjusted for comparability.Source: Table 13 · p. 27, H. Additional Simulation Benchmark… · p. 27

Related papers

Loading 2610.30320v1…