PaperlayerView exact revision on arXiv

Source-linked AI summary

GlanceWAM: Sparse Test-Time Imagination for World-Action Models

Linhan Wang, Zijian An, Mingyuan Zhang, Chen Dai, Yi Xu, Can Cui, Zichong Yang, Yinlin Chen, Lifeng Zhou, Chang-Tien Lu

arXiv:2608.23927v1cs.CV

TL;DR

Synchronous visual imagination makes world-action control latency-prohibitive, while removing imagination discards explicit foresight. GlanceWAM decouples asynchronous lookahead from latent-space action decoding, reaching high success with 48 ms chunks on benchmark tasks.

  • Problem

    Existing WAMs face a trade-off between latency-heavy synchronous visual imagination and imagination-free control that discards explicit visual foresight.

  • Method

    GlanceWAM asynchronously generates a single seconds-ahead latent lookahead frame within a unified video DiT and reuses it for fast action decoding.

  • Results

    72.2% RoboCasa success, 99.0% LIBERO success, and 48 ms action decoding outperform or match the reported benchmark comparisons.

  • Takeaways & Limitations

    Sparse asynchronous visual foresight can provide both explicit lookahead conditioning and real-time latent-space control.

  • Takeaways & Limitations

    Training steps across systems are not compute-matched; reported values are empirical plateau values.

Abstract

from arXiv · show

Video generative models provide rich physical priors for robot learning, yet existing world-action models (WAMs) face a fundamental trade-off: synchronous video generation at control rate is latency-prohibitive, while abandoning test-time visual imagination sacrifices task success. We show that visual imagination achieves both real-time inference and superior success rates when generated asynchronously off the critical path and consumed directly in latent space. We introduce GlanceWAM, which decouples imagination from control within a single video DiT: an asynchronous proposer glances ahead on a slow clock to imagine a single lookahead frame seconds into the future in the background, while an action head decodes action chunks at control rate (48 ms) purely in latent space without blocking. Enabled by a non-interfering attention mask that isolates video representations and staleness-robust horizon training that accommodates asynchronous lookahead aging, GlanceWAM breaks the speed-success dilemma. Trained purely on demonstrations, it attains 72.2% on the 24-task RoboCasa kitchen benchmark (surpassing synchronous Cosmos Policy at 67.1% and imagination-free co-training at 64.4%) and 99.0% on LIBERO, executing at 48 ms per chunk on an NVIDIA A100 GPU (24x faster than synchronous baselines). Code is available at https://github.com/linhanwang/GlanceWAM.

1 INTRODUCTION

GlanceWAM addresses the speed–success dilemma by moving sparse visual foresight off the control path and consuming it directly in latent space. This design improves manipulation success while preserving 48 ms action-chunk decoding.

  • Existing WAMs synchronously couple future video generation to high-frequency action decoding, creating heavy latency and near-static prediction horizons.
  • 72.2% RoboCasa success surpasses synchronous Cosmos Policy at 67.1% and imagination-free co-training at 64.4%.
  • GlanceWAM asynchronously generates one latent lookahead frame about 3 s ahead while decoding action chunks independently at 48 ms.
  • The approach uses a non-interfering attention mask and staleness-robust horizon training to preserve isolated video representations during asynchronous execution.
  • 99.0% LIBERO success and 48 ms action decoding demonstrate strong benchmark performance alongside real-time inference.

2 RELATED WORK

Related work spans dense predict-then-act rollouts, jointly trained world-action models, offline representation shaping, and visual-subgoal planning. GlanceWAM combines test-time visual foresight with asynchronous, unified action execution.

  • Early predict-then-act methods synthesize dense video rollouts and decode actions through inverse dynamics, incurring severe latency on the control path.
  • Recent WAMs jointly model video and actions but typically pay diffusion overhead at every short action-chunk horizon.
  • GlanceWAM unifies visual foresight and action policy learning while decoupling their timescales.
  • Fast-WAM and AHA-WAM remove test-time visual imagination through causal masking and offline representation shaping.
  • Prior visual-foresight methods create image-based subgoals, filtered candidates, or high-level video plans refreshed every few seconds.

3 METHOD

GlanceWAM trains a unified latent video-and-action model with isolated lookahead representations and randomized temporal offsets. At inference, asynchronous latent lookahead is reused across several fast action chunks.

  • Dual-timescale formulation: GlanceWAM formulates foresight at Hf ≈3.0 s and control at Ha = 0.8 s, requiring policies to handle a decaying lookahead offset.
  • Unified training: Training jointly supplies observation history, a noised long-horizon target, and a clean randomized intermediate lookahead condition.
  • Causal isolation: A two-pass VAE encodes generative and policy streams independently so future targets cannot contaminate lookahead latents.
  • Attention isolation: The 3-class prefix-LM mask blocks lookahead access from observation and future-prediction representations while retaining lookahead conditioning for actions.
  • Joint objective: The shared video DiT and action head optimize video forward-dynamics and inverse-dynamics flow-matching objectives jointly.
  • Staleness robustness: Randomized offsets with explicit temporal conditioning train the action head to tolerate aging lookahead latents during asynchronous deployment.
  • Multi-layer conditioning: +1.3% RoboCasa performance comes from conditioning on multi-layer visual features combining shallow spatial details with deep semantic destinations.
  • Inference: Generated lookahead latents remain undecoded and are reused across approximately four 0.8 s chunks, each decoded in 48 ms.

4 EXPERIMENTS

Experiments evaluate GlanceWAM across benchmark comparisons, component ablations, lookahead usage, prediction horizons, and inference efficiency. The results show that asynchronous lookahead conditioning improves success while preserving fast latent-space control and robustness to reduced generative compute.

  • Experimental setup: GlanceWAM is evaluated on state-of-the-art imitation policies and world-action models across RoboCasa kitchen and LIBERO.The experimental questions also cover component contributions, causal lookahead dependence, real-time execution, and generative compute allocation.
  • Benchmark comparison: 72.2% average success on RoboCasa exceeds Cosmos Policy by +5.1% with 50 demonstrations per task and surpasses baselines trained with 300 episodes per task.On LIBERO, GlanceWAM reaches 99.0% average success across four suites.
  • Component isolation: +7.1% from single-layer lookahead conditioning raises success to 71.5%, while four-layer extraction adds +0.7% to reach 72.2%.Lookahead-conditioned models reach peak validation performance within 10k steps, whereas lookahead-free co-training plateaus near 64% across 60k steps.
  • Lookahead contribution: Ablating lookahead tokens reduces success from 71.5% to 61.6%, below the 64.4% lookahead-free baseline, indicating functional dependence on lookahead information.Attention and localized token perturbation analyses likewise show that the action head incorporates lookahead representations into control decisions.
  • Prediction horizon: 71.6% success at a 3.0 s lookahead horizon outperforms 66.3% at 1.4 s and slightly exceeds 71.1% at 3.8 s.The horizon sweep supports distal subgoals rather than short-horizon transitions as the most effective visual foresight target.
  • Inference efficiency: 48 ms per action chunk on an NVIDIA A100 is 24–80× lower latency than comparable synchronous architectures, while asynchronous lookahead generation runs in the background.Pipelining causes only a −0.5% difference from synchronous execution when lookaheads update at the scheduled cadence; success remains 71.2–71.5% across 1–30 Euler steps, enabling a 10× proposer-compute reduction.

5 CONCLUSION

GlanceWAM resolves the speed–success tension by generating sparse visual foresight asynchronously and conditioning real-time action decoding in latent space.

  • 48 ms per chunk enables real-time control while GlanceWAM achieves state-of-the-art success on RoboCasa kitchen and LIBERO.Lookahead generation runs off the critical control path, so action decoding remains nonblocking.
  • Sparse asynchronous lookahead generates distal subgoals without forcing visual prediction to match each high-frequency action chunk.The unified video diffusion architecture separates foresight from execution.
  • Latent-space action conditioning provides spatial guidance that remains robust across sampling budgets and asynchronous execution delays.Empirical analyses support the robustness of lookahead conditioning under these variations.

A CONCURRENT WORK

GlanceWAM compares favorably with concurrent world-action models while using RGB demonstrations alone and a single network.

  • 72.2% RoboCasa success exceeds DeVA’s 72.0%, while 99.0% LIBERO success matches DeVA’s 99.0% under respective protocols.The comparison uses benchmark numbers reported under each system’s evaluation protocol.
  • GlanceWAM reaches these results from RGB demonstrations alone, without affordance or depth supervision, pretrained extractors, or label pipelines.Its design uses a single network rather than separate video and action experts.
  • DeVA and Flex-π retain synchronous joint denoising or multi-stream future prediction, whereas GlanceWAM distinguishes itself through sparse asynchronous execution.The concurrent systems use different supervision and deployment configurations.

B LATENCY MEASUREMENT

Latency measurements separate the nonblocking action path from asynchronous lookahead refreshes and compare GlanceWAM with same-hardware baselines.

  • Latency comparisons use NVIDIA A100 40 GB hardware, while the benchmark caption notes that concurrent-model numbers follow their respective protocols.The protocol uses batch size 1 and synchronized timing; the table comparison is not a single unified evaluation protocol.
  • 108.3 ms hold-path latency measures control using an already available lookahead latent, while refresh generation occurs asynchronously.The reported control latency therefore excludes blocking on proposer refresh.
  • 145 ms is the approximate one-step refresh-call latency, and 478.6 ms is the trained 10-step refresh-call latency off the critical path.The asynchronous proposer runs concurrently, so refresh cost does not block the control loop.
  • A fully synchronous design would pay 478.6 ms per chunk at the 10-step budget instead of keeping refresh generation off the control path.This is approximately 10× the asynchronous path at that sampler budget.
  • 48 ms per chunk is GlanceWAM’s latency versus 91.5 ms for Fast-WAM on the same A100 hardware class.Both systems eliminate video generation from the per-step action forward pass; GlanceWAM also uses fewer parameters and reports higher LIBERO success.

C MECHANISM ANALYSIS

Mechanism analyses indicate that GlanceWAM primarily uses coarse, on-manifold spatial layout, with performance robust to sampling budget and lookahead aging but sensitive to manifold violations.

  • Mechanism Analysis: 71.5% single-layer performance is used to isolate one lookahead pathway for instrumentation.This checkpoint setting defines the scope of the analyses below.
  • Mechanism Analysis: A 2-step lookahead perturbs the latent by 13.4% relative to 10-step lookahead without reducing performance, whereas an equal off-manifold edit costs −28.2 points.The comparison suggests manifold membership matters more than visual fidelity for the policy’s coarse layout read.
  • Mechanism Analysis: Lookahead contributes 0.399 value share versus 0.340 raw attention mass in the action head’s cross-attention readout.These readouts quantify how the action head uses lookahead tokens alongside observation tokens.
  • Mechanism Analysis: The strongest localized lookahead block causes 0.093 of a full lookahead-drop displacement, indicating that conditioning is distributed across many tokens.Perturbations replace localized lookahead blocks with corresponding observation tokens on the token manifold.
  • Mechanism Analysis: SkyReels-V2-DF reaches 71.5%, ahead of Wan2.1 at 70.3% and Self-Forcing-DMD at 68.9% under the identical recipe.The reported margins are within one evaluation sigma pairwise, so the ordering is interpreted cautiously.
  • Mechanism Analysis: The evidence does not separate amortized test-time compute from a training-time scaffold as explanations for the lookahead benefit.The flat sampler-dose curve is consistent with both candidate mechanisms.

D.1 STALENESS TOLERANCE OF THE LOOKAHEAD

GlanceWAM tolerates hold aging within its trained horizon, but source-stale lookaheads fall outside training and can mislead the policy rather than being ignored.

  • Training covers hold aging by supervising lookahead offsets uniformly over u ∼U(0, Hf], matching consumption until refresh.This trains the policy to consume lookaheads as the execution process catches up to them.
  • Lookahead fidelity improves with sampler budget, but success remains flat across K=1–30, indicating control uses content rather than pixel fidelity.The one-step lookahead visibly smears the arm and manipulated object, while higher fidelity does not improve success.
  • Source-stale lookaheads are out-of-distribution because training never generates them from a world state that the actual trajectory has left.Robustness to source staleness therefore requires measurement rather than assumption.
  • 70.8% success under pipelined one-chunk-late adoption is only 0.5 points below the synchronous baseline.The deployed configuration refreshes every chunk, keeps lookaheads exactly 0.8 s old, and fits generation within one 800 ms chunk.
  • 68.8% success at 0.8 s naive lag falls to 60.5% at 1.6 s and 49.9% at 3.2 s, revealing a narrow deployment envelope.The reported crossover is treated as a deployment boundary, not as a mechanism statement.

D.2 LOOKAHEAD FIDELITY ACROSS SAMPLER BUDGETS

Increasing sampler compute visibly improves generated-lookahead fidelity, but it does not improve policy success across the tested budget range.

  • Fidelity visibly improves with sampler budget, while success does not across the budgets evaluated in Table 4.The comparison uses the generated lookaheads visualized in Figure 8.
  • The policy therefore reads task-relevant content that a single sampling step already fixes rather than requiring pixel-perfect lookaheads.

E REPRODUCIBILITY AND TRAINING DETAILS

GlanceWAM uses a causal video-VAE latent representation and is trained with a specified multi-GPU mixed-precision optimization setup.

  • The spatiotemporal causal video VAE compresses raw video into continuous latents with temporal ratio pt = 4, spatial ratio ps = 8, and dimension C = 16.Latents are patchified with ph = pw = 2, producing N = (H/16) × (W/16) spatial tokens per frame.
  • Training uses PyTorch on 4× NVIDIA H200 GPUs with bfloat16 mixed precision and AdamW optimization.The optimizer uses β1 = 0.9, β2 = 0.95, ϵ = 10^-8, weight decay 10^-8, and gradient clipping at 1.0.
  • The video DiT and action head use learning rates of 1.0×10^-5 and 1.0×10^-4, respectively, with cosine decay and 5000 warmup steps.
Loading 2608.23927v1…