PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Use Imagination: Progress-Conditioned Future Utilization for World Action Models

Yijie Zhu, Zitong Yu, Wei Li, Hui Ma, Wen Li, Rui Shao, Liqiang Nie

arXiv:2609.06578v1cs.CV

TL;DR

Existing WAMs give limited attention to adapting imagined-future use as execution progresses, although future utility varies across stages and among latents. ProWAM represents execution progress and uses it to modulate future use hierarchically. It reports consistent gains over strong VLA and WAM baselines across simulation and real-world platforms.

  • Problem

    Existing WAMs give limited attention to adapting imagined-future use as execution evolves, despite stage-level variation in future utility and heterogeneous latent relevance within a progress state.

    Source: §1 · pp. 1–2

  • Method

    ProWAM uses SS-DTPE to represent progress from short-term action–observation evidence and long-term history, then HPIM modulates future use globally and by latent relevance.

    Source: §1 · pp. 2–3, §4.1 · p. 5, §4.3 · p. 8

  • Results

    ProWAM reports consistent gains over strong VLA and WAM baselines across five simulation benchmarks and two real-world platforms, particularly on progress-aware and long-horizon manipulation.

    Source: §1 · p. 3, §6 · p. 16

  • Takeaways & Limitations

    The results support progress-conditioned regulation of imagined futures as an approach evaluated across simulation and real-world manipulation settings.

    Source: §6 · p. 16

    The real-world evaluation primarily covers tabletop-scale manipulation with relatively structured task objectives.

    Source: §3 · p. 23

Abstract

from arXiv · show

World Action Models (WAMs) extend Vision-Language-Action (VLA) models by incorporating future visual dynamics into action generation. However, existing WAMs often utilize imagined futures with limited adaptation to evolving execution progress, potentially introducing distracting or unreliable predictive cues. This limitation arises from two empirically identified forms of non-uniformity in future utility: (i) at the inter-progress level, the utility of imagined futures varies across execution stages as control demands change; and (ii) at the intra-progress level, individual future latents exhibit heterogeneous relevance within the same progress state. To address these limitations, we propose ProWAM, a Progress-Conditioned World Action Model that introduces execution progress as an explicit intermediate representation for adaptive imagination utilization. ProWAM comprises two tightly coupled components: (1) To obtain a reliable representation of execution progress, we propose the Self-Supervised Dual-Temporal Progress Encoder (SS-DTPE). SS-DTPE couples short-term action-observation interaction modeling with long-term recurrent progress aggregation to capture recent execution feedback and accumulated task history. (2) Conditioned on the progress representation from SS-DTPE, we propose the Hierarchical Progress-Conditioned Imagination Modulation (HPIM) to adapt imagination utilization to execution progress. HPIM operates at two complementary levels: an inter-progress global modulation mechanism adapts future utilization across execution stages, while an intra-progress relevance mechanism differentiates individual future latents within each progress state. Extensive experiments demonstrate consistent gains over strong VLA and WAM baselines.

1 INTRODUCTION

ProWAM addresses limited progress adaptation in world action models by modeling execution progress and using it to regulate imagined futures. Its encoder represents progress from short- and long-term evidence, while its modulation mechanism adjusts future use globally and by latent.

1 INTRODUCTION

  • Existing WAMs largely use fixed or progress-agnostic imagined-future interactions, despite evolving relevance that can make predictive cues distracting or unreliable.The paper distinguishes this utilization gap from prior efforts to improve future representations, grounding, memory, or inference.Source: §1 · p. 1
  • Future utility varies across execution stages and among individual future latents within the same progress state.These two forms of non-uniformity motivate adapting imagination use as control demands evolve.Source: §1 · pp. 2–3
  • ProWAM uses explicit execution progress to regulate imagined futures through global inter-progress gating and intra-progress latent relevance.HPIM applies these complementary mechanisms to the action-to-future attention pathway.Source: §1 · pp. 2–3
  • SS-DTPE learns progress representations by combining short-term action–observation evidence with long-term recurrent history, without manual progress annotations.Its self-supervised objectives use demonstration semantics and temporal ordering to guide progress learning.Source: §1 · pp. 2–3
  • ProWAM consistently gains over strong VLA and WAM baselines across five simulation benchmarks and two real-world platforms, particularly in progress-aware and long-horizon manipulation.The evaluation spans general manipulation, robustness and generalization, progress-aware execution, and temporal memory.Source: §1 · p. 3

2 RELATED WORK

Prior work develops VLAs, predictive policies, WAMs, and progress-aware manipulation methods. ProWAM differs by explicitly learning execution progress and conditioning hierarchical future-latent modulation on it.

2.1 Vision-Language-Action Models

  • VLAs ground visual observations and language instructions in executable actions, while predictive VLAs and future-state models add future-oriented guidance for action generation.The cited approaches include explicit future images and latent representations of anticipated state transitions.Source: §2.1 · p. 3

2.2 World Action Models

  • WAM research has advanced future representations, physical grounding, efficient deployment, and adaptive mechanisms, but often regulates denoising, execution horizons, or memory rather than structured progress-conditioned future latents.ProWAM explicitly translates execution progress into hierarchical regulation of imagined-future attention.Source: §2.2 · p. 3

2.3 Progress Modeling for Robotic Manipulation

  • Prior progress-aware methods use task advancement for monitoring, policy learning, reward modeling, skill acquisition, or WAM execution and replanning.ProWAM instead learns progress from short-term action–observation evidence and long-term recurrent history, then conditions HPIM on that state.Source: §2.3 · p. 3

3 PROBLEM FORMULATION

The paper frames WAM action generation as joint prediction of actions and future visual latents, then examines whether those latents are equally useful throughout execution. Its diagnostics find stage-dependent and latent-specific differences in future utility.

3.1 Preliminaries: Video–Action Generation in WAMs

  • Action-only policies predict action chunks from the current observation and instruction, whereas WAMs jointly model actions with future visual latents in a shared generative process.The action horizon is denoted Ha; the future visual sequence spans horizon Hv.Source: §3.1 · pp. 3–4
  • A Mixture-of-Transformers backbone preserves modality-specific video and action computation while shared attention lets action tokens access imagined visual cues.Training uses rectified flow-matching losses for future-video prediction and action generation.Source: §3.1 · p. 4

3.2 Rethinking Imagination Usage

  • Progress stages impose different control demands: pre-contact search and approach can benefit from foresight, while contact-rich interaction depends more on current evidence and precise local control.The motivating question is whether future latents should influence action generation uniformly throughout execution.Source: §3.2 · p. 4
  • The diagnostic study evaluates five real-world long-horizon tasks over 500 trials and compares a joint WAM with an action-only variant while perturbing near-, mid-, and long-horizon latent groups.The platform is Galaxea R1 Lite; the action-only comparison holds the action backbone, language conditioning, observation encoder, and action objective unchanged.Source: §3.2 · pp. 4–5

Finding 1: Future Latents Show Progress-Dependent Overall Utility

  • The action-only variant has more failures in search/transit and approach, while joint WAM reduces early-stage failures and remaining failures more often occur in contact-rich stages.This pattern suggests future latents are particularly helpful before physical interaction, when foresight can support target-directed motion and approach.Source: §3.2 · p. 5

Finding 2: Future Latents Show Progress-Dependent Local Relevance

  • Pre-contact stages are more sensitive to mid- and long-horizon futures, while grasping, placement, and handover are more sensitive to near-future latents.Distant imagined states may be less reliable or relevant for immediate control in contact-rich stages.Source: Finding 2: Future Latents Show… · p. 5
  • The diagnostics identify inter-progress variation across execution stages and intra-progress heterogeneity among future latents within a progress state.The analysis motivates an online progress representation; without it, future latents can only be exposed as fixed spatio-temporal context.Source: Finding 2: Future Latents Show… · p. 5

4 METHOD

ProWAM estimates execution progress from action–observation feedback and task history, then uses progress-conditioned modulation to adapt how imagined future latents influence action generation. Its method includes progress encoding and mechanisms for modulating future use across execution stages and individual latents, with progress updated online during receding-horizon execution.

4.1 Overall Framework

  • ProWAM directly modulates the MoT attention path from action tokens to future latents, using learned progress tokens to strengthen or suppress imagined information.The progress tokens are inferred online from the current observation, previous action, instruction, and prior progress state.Source: §4.1 · pp. 5–6

4.2 Self-Supervised Dual-Temporal Progress Encoder

  • SS-DTPE couples short-term action–observation feedback with recurrent progress memory and learns progress from demonstration structure rather than manual annotations.Its self-supervision aligns progress with instruction prefixes and constrains semantic completion to follow temporal execution order.Source: §4.2 · pp. 6–7

4.3 Hierarchical Progress-Conditioned Imagination Modulation

  • HPIM combines a progress-conditioned global gate for overall future reliance with token-wise relevance weights that distinguish useful future latents.It applies these signals to action-to-future attention, with modulation weaker in shallow and deep layers and stronger in intermediate layers.Source: §4.3 · pp. 8–9
  • Weak keyframe labels train the global gate to favor imagined futures during transit, search, and approach, reduce reliance during contact-rich interaction, and exclude unreliable cues.Interaction anchors include gripper-state transitions, contact changes, end-effector motion, and task-specific cues associated with grasping, releasing, insertion, or placement.Source: §4.3 · p. 9
  • Intra-progress relevance receives no token-level labels and is learned end-to-end through the joint WAM objective to support action prediction under the current progress condition.The overall HPIM objective combines standard action and future-latent generation losses with weak supervision for the inter-progress gate.Source: §4.3 · p. 9

4.4 Training and Inference Procedure

  • Training has two stages: SS-DTPE first learns progress representations from demonstration trajectories, then is frozen while HPIM and the WAM train jointly using its progress tokens.Source: §4.4 · p. 9
  • At each replanning step, ProWAM updates progress from the current observation, prior action, instruction, and prior state, then executes an action chunk without requiring offline progress annotations or changing the WAM rollout interface.The executed action feeds the next progress update, enabling continuous adaptation as execution evolves.Source: §4.4 · pp. 9–10

5 EXPERIMENTS

Experiments evaluate ProWAM in simulation and real-world manipulation, ablate its progress representation and imagination modulation, and analyze execution across successive stages.

5.1 Implementation Details

  • The experiments specify a two-stage training setup: SS-DTPE is trained for five epochs, then frozen while the WAM with HPIM is trained for ten epochs.Both stages use AdamW with a 1×10^-4 learning rate, 1×10^-2 weight decay, cosine scheduling, and gradient clipping at 1.0.Source: §5.1 · p. 10

5.2 Simulation and Real-World Setup

  • Evaluation spans five simulation benchmarks and two mobile bimanual robot platforms, with real-world tasks covering progress-aware long-horizon manipulation and generalization and robustness.The real-world evaluation uses 150–200 expert demonstrations and 50 rollouts per task, randomizing object positions and varying lighting and scene configurations for robustness tests.Source: §5.2 · pp. 10–11

5.3 Evaluation on Simulation Benchmarks

  • ProWAM achieves 99.1% average success on LIBERO and 93.4% average on RoboTwin2.0, outperforming compared methods across both benchmarks.On LIBERO, it exceeds the best competing WAM by 0.6 percentage points; on RoboTwin2.0, it reports 93.9% Clean and 92.8% Randomized success, without additional embodied pre-training.Source: Table 1 · p. 11, §5.3 · p. 11, Table 2 · p. 11
  • On VLABench, ProWAM achieves 68.4% SR, 85.4% IS, and 81.2% PS, exceeding Fast-WAM by 10.0, 4.2, and 6.7 percentage points; on RoboEval, it scores 0.375 SR and 0.494 TP, outperforming all compared methods.Source: §5.3 · pp. 11–12

5.4 Evaluation on Real-World Tasks

  • On the Galaxea R1 Lite and AgileX Cobot Magic, ProWAM reaches 72% and 71% full-task success on long-horizon tasks, and 69% and 68% on generalization and robustness tasks.On Galaxea long-horizon tasks, its average completion rate falls only 5 percentage points from first stage to full completion; on out-of-distribution Fold Cloth, its final-stage success exceeds π0.5 and Fast-WAM by 12 and 22 points, respectively.Source: §5.4 · p. 12, Table 6 · p. 13, Table 7 · p. 13

5.5 Ablation Study

  • SS-DTPE and HPIM each improve over the baseline, while full ProWAM performs best in every reported column; Mamba aggregation also outperforms alternatives across benchmarks.The component ablations show full-model gains of 3.0/0.4/1.4/3.0 points across LIBERO suites and 16/11 on real-world Long./Gen.; Mamba-based aggregation is strongest among the tested long-term aggregators.Source: Table 8 · p. 13, §5.5 · p. 13, Table 9 · p. 13
  • Semantic Progress Alignment and Temporal Structure Learning provide complementary supervision; removing both progress signals from HPIM’s global gate lowers real-world Long. and Gen. by 15 and 11 percentage points, respectively.Removing progress conditioning from local relevance lowers LIBERO-Long from 97.8% to 95.8% and real-world Gen. from 69% to 57%. Action-to-All-Visual modulation reduces LIBERO-Long, real-world Long., and Gen. by 1.4, 7, and 6 percentage points, respectively.Source: §5.5 · p. 14, Table 12 · p. 14

5.6 More Qualitative and Quantitative Analysis

  • Qualitative analyses show that progress representations evolve approximately monotonically, global gates vary by execution stage, and future-latent relevance is non-uniform and progress-dependent.On LIBERO-Spatial, the top-20% latent relevance mass changes from 93.0% during Object Approach to 44.7% during Lift and 89.5% during Transport.Source: Figure 7 · p. 15, Figure 8 · p. 15, §5.6 · p. 15
  • The analysis reports more coherent execution across successive stages.Source: §5.6 · p. 16

6 CONCLUSION

ProWAM conditions imagined-future use on execution progress, combining short-term feedback with task history and modulating future information across stages and individual latents. It performs strongly across five simulation benchmarks and two real-world platforms.

6 CONCLUSION

  • ProWAM learns when and how to use imagined future information by conditioning its use on execution progress.It combines short-term action–observation feedback with long-term interaction history and modulates future use across execution stages and individual latents.Source: §6 · p. 16
  • SS-DTPE integrates short-term action–observation feedback with long-term interaction history to learn structured progress representations.The representations condition HPIM’s hierarchical modulation of the action-to-future attention pathway.Source: §6 · p. 16
  • ProWAM demonstrates strong performance across five simulation benchmarks and two real-world platforms.Source: §6 · p. 16

1 DETAILED EXPERIMENTAL SETUP

The experimental setup spans five simulation benchmarks, two real-world platforms, progress-sensitive evaluation metrics, and automatically constructed weak stage supervision. Real-world tasks cover long-horizon manipulation and variations in object and scene conditions.

1.1 Simulation Benchmarks

  • Five simulation benchmarks cover general manipulation, bimanual control, semantic and physical reasoning, progress-aware execution, and temporal memory.LIBERO uses four suites; RoboTwin2.0 has over 50 tasks; VLABench has 100 categories and over 2,000 objects; RoboEval covers eight bimanual task families; Mikasa-Robo includes five memory-dependent tasks.Source: §1.1 · p. 19

1.2 Detailed Real-World Setup

  • Real-world evaluation uses two complementary platforms and eight long-horizon tasks, with additional tests varying appearance, geometry, position, lighting, and scene configuration.The tasks include sequential manipulation, bimanual handover, deformable-object interaction, fine-grained insertion, and generalization under out-of-distribution conditions.Source: §1.2 · pp. 19–20, Figure 11 · p. 19

1.3 Evaluation Metrics

  • Evaluation combines terminal Success Rate with benchmark-specific measures of intention, intermediate progress, task-stage completion, and cumulative real-world stage completion.The final cumulative stage-completion rate corresponds to full-task success and captures error accumulation as execution progresses.Source: §1.3 · p. 21

1.4 Automatic Weak-Supervision Construction for Global Modulation

  • Weak stage labels come from interaction events detected using gripper transitions and, when needed, end-effector height or velocity cues.Detected anchors are expanded over neighboring frames using 20-frame pre- and post-windows; label refinement removes short isolated segments and merges small gaps.Source: §1.4 · pp. 21–22

2 QUALITATIVE ROLLOUT VISUALIZATION

Qualitative rollouts show ProWAM progressing through varied simulated and real-world tasks, including stage changes, contact-rich interactions, and generalization conditions.

Qualitative Simulation Results

  • Across five simulation benchmarks, representative rollouts show coherent progression through manipulation, bimanual coordination, memory-dependent control, and semantic-grounding tasks.The examples include adaptation between goal-directed motion and precise interaction, with no obvious loss of execution coherence at stage transitions.Source: §2 · p. 22

Qualitative Real-World Results

  • Real-world rollouts show successive-stage execution across long-horizon tasks including packing, marker insertion, stacking, tray transfer, drawer manipulation, and handover.Cloth folding and handover also require adaptation to interaction feedback and changes in scene configuration.Source: §2 · pp. 22–23, Figure 13 · p. 23
  • Generalization rollouts show task progression under variations in object appearance, geometry, and scene conditions, including out-of-distribution cloth folding.Examples include color-conditioned sorting, multi-object clearing, and bowl-on-plate stacking.Source: §2 · p. 23

3 LIMITATIONS AND FUTURE WORK

The real-world evaluation is bounded by tabletop-scale tasks with relatively structured objectives, and large-scale cross-embodiment transfer is outside the study’s scope.

Scope of real-world evaluation

  • Real-world evaluation focuses on tabletop-scale manipulation with relatively structured task objectives.Broader settings such as articulated environments, tool-mediated interaction, and longer spatial and temporal dependencies remain future evaluation directions.Source: §3 · p. 23

Cross-embodiment evaluation scale

  • Large-scale cross-embodiment transfer is beyond scope because training data, observation spaces, and action representations remain specific to each benchmark or platform.The study evaluates heterogeneous simulated embodiments and two real-world platforms, but not shared progress representations across robots with different configurations.Source: §3 · p. 23
Loading 2609.06578v1…