PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes World Action Models Generalize? An Empirical Study of Test-Time Future Modeling

Renping Zhou, Zanlin Ni, Zihao Fan, Guohao Fu, Zeyu Liu, Hao Shi, Jie Zhang, Chi Bene Chen, Yang Yue, Xueyang Fu, Gao Huang

arXiv:2609.34981v1cs.CVcs.RO

TL;DR

The paper asks whether WAMs must generate future video at inference, given the cost of denoising and disagreement over whether future prediction is needed beyond training. It compares matched paradigms across three generalization axes and finds that a single pass over a fully noised future retains most of the benefit, enabling Simple-WAM to pair strong generalization with low inference cost.

  • Problem

    Whether inference-time future modeling is necessary remains disputed: explicit WAMs denoise future video at high cost, while latent WAMs skip it and report little in-distribution difference.

    Source: §1 · p. 2

  • Method

    The paper compares matched WAM configurations across environmental perturbation, data efficiency, and task generalization, then proposes Simple-WAM with one pass over fully noised future tokens and a training schedule adapted to that inference behavior.

    Source: §1 · p. 2, §5.2 · p. 7

  • Results

    79.5 for Simple-WAM versus 67.7 for the explicit paradigm and 53.8 for the latent one under environmental perturbation; matched comparisons also find latent WAMs trail explicit WAMs across all three generalization axes.

    Source: §1 · p. 2, §6.3 · p. 8

  • Takeaways & Limitations

    A single pass over fully noised future video tokens retains most of the generalization benefit without generating a clean future.

    Source: §1 · p. 2

    The conclusions rest on a 5B backbone without embodied pretraining; whether they hold at larger scale or with embodied pretraining remains open.

    Source: §7 · p. 10

Abstract

from arXiv · show

World action models (WAMs) predict the future alongside actions during \emph{training}. Due to the heavy computation cost of video denoising, whether the future must still be generated during \emph{inference} is disputed: Explicit WAMs denoise it into clean frames along with every action chunk, whereas Latent WAMs discard it entirely for acceleration. We find that latent WAMs, despite matching explicit ones on in-distribution tasks, fail to retain the generalization benefits that originally motivated WAMs. To demonstrate this, we evaluate generalization along three axes: \emph{environmental perturbation}, \emph{data efficiency}, and \emph{task generalization}. Controlled comparisons with a matched backbone, training data, and budget reveal consistent degradation across all three axes when the action expert no longer conditions on future representations. Further analysis shows that the gap arises almost entirely from the first denoising step: the benefit comes from \emph{preparing} the future, not \emph{generating} it. We therefore propose \textbf{Simple-WAM}, which simplifies future modeling into a single forward pass of fully noised video tokens and adapts the training-time noise schedule to this inference behavior. Across simulation and real-world tasks, Simple-WAM achieves the best of both worlds, leading explicit WAMs in generalization performance with efficiency comparable to Latent WAMs. Project Page: \href{https://zrporz.github.io/Simple-WAM-Web/}{\textcolor{panton}{\texttt{https://zrporz.github.io/Simple-WAM-Web}}}

1. Introduction

The paper asks whether inference-time future modeling is needed for WAM generalization, since explicit video denoising is costly while latent WAMs omit future generation. Matched comparisons show that latent models’ in-distribution parity with explicit models does not extend to generalization, and Simple-WAM retains future conditioning with a single pass over noisy future tokens.

1. Introduction

  • WAMs jointly generate future dynamics and actions, using observation-space supervision beyond sparse action labels and video priors for generalization.The paper contrasts this with VLA backbones pretrained predominantly on static image-text pairs.Source: §1 · p. 1
  • Matched comparisons find that latent WAMs match explicit WAMs in distribution but fall behind across environmental perturbation, data efficiency, and task generalization.The protocol examines robustness to environmental perturbations, fewer demonstrations, and tasks unseen in action training.Source: §1 · p. 2
  • A single forward pass over fully noised future video tokens recovers most of the generalization gap without denoising them into clean frames.The result indicates that inference-time future conditioning can retain most of the benefit without generating a clean future.Source: §1 · p. 2
  • Simple-WAM restores inference-time future conditioning to latent WAMs at negligible additional cost, keeping future video tokens in the policy context as noise.The video expert runs once rather than over the full denoising schedule, while training shifts its flow-time sampling toward that inference point.Source: §1 · p. 2
  • 79.5 for Simple-WAM versus 67.7 for the explicit paradigm and 53.8 for the latent one under environmental perturbation; it leads the explicit paradigm on seven of eight measurements and the latent one on all eight.Simple-WAM runs at 3.8× the speed of the explicit paradigm and within 1.2× the latency of the latent one.Source: §1 · p. 2

2. Related Works

Prior WAM work reports benefits in sample efficiency, robustness, and transfer, while efficient latent variants skip inference-time future generation. The paper organizes embodied-policy generalization into three axes and compares paradigms under a matched setting.

2.1. World Action Models and Inference Efficiency

  • Earlier WAMs jointly predict future frames and actions, attributing gains to observation-space training signals and video backbones with priors on scene evolution.Prior work reports sample-efficiency, scene-robustness, and transfer gains.Source: §2.1 · pp. 2–3
  • Latent WAMs retain video training but omit future generation at deployment, reporting near-explicit success rates at lower latency; that evidence is in-distribution.The paper situates its study among work on architecture, inference formulation, and training recipes for efficient WAMs.Source: §2.1 · p. 3

2.2. Generalization of Embodied Policies

  • Embodied-policy generalization spans environmental variation, fewer demonstrations per task, and transfer to tasks withheld from training or learned from action-free video.LIBERO-Plus covers seven environmental factors involving camera, scene, language, and robot initial state.Source: §2.2 · p. 3
  • The paper consolidates these dimensions and compares paradigms in one matched setting to isolate inference-time future modeling.Prior properties were generally demonstrated by individual systems with different backbones, data, and training budgets.Source: §2.2 · p. 3

3. Preliminaries

A WAM combines a video expert and an action expert, training them on future-video and action prediction. At inference, explicit and latent paradigms differ in whether future tokens are present and how their noise level changes, with video denoising driving much of the cost.

3.1. World Action Models

  • A WAM couples a pretrained video expert over current and future frame latents with an action expert that predicts an action chunk from observations, language, and proprioception.The future-frame latents carry a flow time from clean data to Gaussian noise, while the current observation is clean.Source: §3.1 · p. 3
  • Training minimizes action and video losses, and independently sampled action and video flow times expose the action expert to future latents across noise levels.The training schedule also supplies the noise levels used for inference steps.Source: §3.1 · p. 4

3.2. Inference-Time Future Conditioning

  • At inference, the action expert updates its action latent using a video-expert representation as its conditioning feature.The map from the video expert to the representation read by the action expert is denoted Φθ.Source: §3.2 · p. 4
  • Explicit WAMs retain future tokens and denoise them toward clean data, so the action expert reads progressively sharper future features; latent WAMs omit those tokens.Both paradigms are distinguished by whether future tokens are present and by the schedule applied to their flow time.Source: §3.2 · p. 4
  • Explicit inference costs K video-expert passes plus K action steps, whereas latent inference uses one current-frame video pass plus K action steps.The video term dominates because the video expert is larger and processes far more future-video tokens than action tokens; cost depends on denoising depth, not how often the action expert reads the future.Source: §3.2 · p. 4
  • The study varies future-token access and its noise schedule separately while holding the other components fixed.Earlier comparisons changed both factors at once.Source: §3.2 · p. 4

4. A Controlled Study of Inference-Time Future Conditioning

A matched evaluation finds that explicit WAMs generalize better than latent WAMs beyond in-distribution tasks, while nearly all of the benefit comes from one pass over a fully noised future. The evaluation spans environmental perturbations, reduced demonstrations, and unseen tasks.

4.1. Evaluation Protocol for Generalizability

  • The protocol tests environmental perturbations, reduced demonstrations, and unseen tasks using configurations matched in architecture, training hyperparameters, and budget.Perturbations cover seven factors; data efficiency uses 10 demonstrations per task, and task generalization tests held-out suites with and without action-free video.Source: §4.1 · p. 5

4.2. Finding 1: Generalization Deteriorates without Inference-time Future Modeling

  • Explicit WAMs match latent WAMs in distribution at 97.75 against 96.85, but lead by 13.97 points under environmental perturbation and 8.45 under data efficiency.For held-out tasks with action-free video, the video is worth 63.65 points to explicit WAMs and 3.80 to latent WAMs; the reported success rates are 69.90 and 5.90, respectively.Source: §4.2 · p. 5, Table 1 · p. 5

4.3. Finding 2: Fully Noised Future is Sufficient for Generalization

  • A sweep that varies video denoising only at inference, without retraining, finds that almost all of the generalization gain comes from the first pass.At tdenoise = 1, the future tokens are pure Gaussian noise; this single pass outperforms the latent paradigm by 26.44, 8.2, 13.4, and 89.8 points across four settings.Source: §4.3 · p. 6, Figure 2 · p. 6
  • The gain comes from the intermediate representation read by the action expert, not from generating a denoised future.The findings point to a mismatch between the visual fidelity the video model is trained for and the feature the action expert requires.Source: §4.3 · p. 6

5. Method

Simple-WAM retains future-video conditioning while reducing inference to one video-expert pass over fully noised future tokens. Its training schedule increases emphasis on the noise level used at inference while retaining supervision across the rest of the schedule.

5. Method

  • Simple-WAM keeps future-video conditioning but uses a single video-expert forward pass, targeting explicit-WAM generalization with latent-WAM-like inference cost.The changes add neither a module nor a loss term.Source: §5 · p. 6, Figure 3 · p. 7

5.1. Inference: One Pass over a Fully Noised Future

  • At inference, one video-expert pass over future tokens left at Gaussian noise supplies the feature used to condition all K action steps.Action denoising is unchanged; the video-related cost is one pass rather than iterative future-video denoising.Source: §5.1 · p. 6

5.2. Training: A Mixed Schedule

  • The mixed training schedule increases the share of future-video tokens sampled at τ = 1, while retaining sampling across the explicit paradigm’s schedule.The paper sets p = 0.5 and applies the video loss at the sampled flow time.Source: §5.2 · p. 7

6. Experiments

Across simulation and real-world tests, Simple-WAM preserves generalization while avoiding iterative future-frame denoising; ablations support conditioning on pretraining-aligned noised video tokens.

6.1. Implementation Details

  • Simple-WAM uses pretrained Wan2.2-5B, 32-action chunks, nine video frames per chunk, and video loss weight λ = 3.0.Inference uses 10 denoising steps with CFG scale 1.0; latency is measured on a single RTX 5090 GPU.Source: §6.1 · p. 7

6.2. Experimental Setup

  • Evaluation spans LIBERO, RoboTwin 2.0, LIBERO-Plus, and real-world dual-arm tasks, testing environmental perturbation, data efficiency, and task generalization.LIBERO uses 5- and 10-demonstration settings and four-fold suite cross-validation; RoboTwin uses 10 demonstrations and withholds tasks; LIBERO-Plus covers seven variation factors.Source: §6.2 · pp. 7–8

6.3. Main Results

  • 79.5% under environmental perturbation leads the explicit paradigm (67.7%) and latent paradigm (53.8%) on LIBERO-Plus.With fewer demonstrations, Simple-WAM scores 92.4% and 97.2% on LIBERO at 5 and 10 shots and 37.2% on RoboTwin; on withheld tasks, it reaches 10.1% without data and 73.6% with action-free video on LIBERO.Source: Table 2 · p. 8, §6.3 · p. 8
  • 74.7 ms per action chunk is 3.8× faster than the explicit paradigm’s 286.9 ms and within 12.7 ms of the latent paradigm’s 62.0 ms.Simple-WAM averages the best across the four generalization settings.Source: Figure 4 · p. 8, §6.3 · p. 9
  • On real-world environmental perturbations, Simple-WAM scores 69.2% versus 25.2% for the latent paradigm, and scores 69.6% versus 37.2% with 50 demonstrations per task.On held-out Store in Order with action-free video, Simple-WAM reaches 87.8%, compared with 68.9% for explicit and 10.0% for latent WAMs.Source: Figure 5 · p. 9, §6.3 · p. 9

6.4. Ablation Study

  • Replacing noised future video tokens with queries or zeros lowers performance on every generalization axis.The authors attribute this to those inputs departing from video-backbone pretraining; mixed-schedule probabilities 0.25, 0.5, and 0.75 differ by only 1.3 points on the three-axis average, while both endpoints score lower.Source: §6.4 · p. 9, Table 3 · p. 10

7. Conclusion

The conclusion argues that inference-time future conditioning matters for generalization, while fully noised future tokens recover most of the gap without clean-frame denoising.

7. Conclusion

  • Matched explicit and latent WAMs perform similarly in distribution but diverge across all three generalization axes.Future video tokens left at pure noise recover almost all of that gap.Source: §7 · p. 10
  • Simple-WAM conditions on a fully noised future in one pass and leads the explicit paradigm on nearly every measurement at 3.8× its speed.Source: §7 · p. 10
  • The conclusions are based on a 5B backbone without embodied pretraining; transfer to larger models or models with such pretraining remains open.Source: §7 · p. 10

A.1. Training Details

The appendix points to a training-configuration table and notes that other baselines use their official settings.

A.1. Training Details

A.2. Per-Suite Simulation Results

Table A2 provides detailed results for data efficiency and task generalization, with benchmark-specific evaluation conditions.

A.2. Per-Suite Simulation Results

  • Table A2 breaks down data-efficiency and task-generalization results by LIBERO suite and RoboTwin scene condition.LIBERO suites are held-out folds in four-fold cross-validation; RoboTwin reports clean and randomized scenes, with data efficiency measured at 10 demonstrations per task.Source: §A.2 · p. 13, Table A2 · p. 13

A.3. Per-Task Real-World Results

Table A3 reports per-task real-world results, including held-out-task evaluation without and with action-free video.

A.3. Per-Task Real-World Results

  • Table A3 reports mean and standard deviation for real-world results by task.The held-out task is evaluated both without and with action-free video.Source: §A.3 · p. 14, Table A3 · p. 14
  • T1–T4 denote the four training tasks shown in Fig. 6.Source: §A.3 · p. 14
  • Each run is scored by the fraction of listed sub-goals completed, and each cell averages thirty runs.The sub-goals are listed in Sec. 6.2.Source: §A.3 · p. 14
  • “Avg.” averages the four task means and is the quantity quoted in Sec. 6.3.Source: §A.3 · p. 14

Related papers

Loading 2609.34981v1…