PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation

Xiangcheng Zhan, Zirui Chen, Yicheng Zhao, Ziteng Gao, Shuo Yang

arXiv:2609.37398v1cs.AIcs.RO

TL;DR

Post-deployment learning for WAMs often targets behavior without requiring better predictions of deployment interactions, a gap that matters in dexterous manipulation. DEWO learns from matched successful and failed visual futures alongside action supervision, then uses progress-gated guidance. It improves average success across three WAM formulations and strengthens real-world spatial generalization in cells with initial successes.

  • Problem

    Post-deployment learning for WAMs typically targets behavior without requiring better predictions of deployment interactions, which can challenge both action generation and predictive representations in dexterous manipulation.

    Source: §1 · p. 1

  • Method

    DEWO organizes matched successful and failed futures around interaction turning points, trains outcome-conditioned visual prediction alongside successful-action supervision, and uses progress estimates to gate classifier-free guidance.

    Source: §1 · p. 2

  • Results

    DEWO improves average success across three WAM paradigms; on the fixed cells with at least one initial success, pooled success rises from 51.0% to 71.7%.

    Source: §1 · p. 2, §3.3 · p. 7

  • Takeaways & Limitations

    The findings support continued predictive learning from deployment experience as part of WAM adaptation, with successful and failed visual experience improving prediction and control beyond action supervision alone.

    Source: §6 · p. 9

Abstract

from arXiv · show

World-Action Models (WAMs) couple action generation with predictions of how physical interactions unfold. However, current post-deployment learning paradigms typically improve behavior without requiring better world predictions. Especially in dexterous manipulation, small execution errors can compound in high-dimensional action spaces, hindering policy improvement and pushing interactions beyond the world model's training distribution. Motivated by this, we propose Direct Experience World-Model Optimization (DEWO), a post-deployment learning paradigm for WAMs that, alongside action imitation, refines world representations through visual experience to better condition action generation. Specifically, it identifies interaction turning points and learns from successful and failed futures to support classifier-free guidance. An additional value head estimates task progress from video representations and activates guidance when progress stalls during inference. Across five DexJoCo tasks, DEWO improves average success across all three WAM formulations. Ablations show that visual supervision from successful and failed continuations improves both prediction and control beyond action supervision alone. On four real-world tasks across Wuji and Sharpa, 3 x 3 grid evaluations show that two rounds of deployment learning increase success from 51.0% to 71.7% in cells with at least one initial success, a gain of 20.7 percentage points. These findings support continued predictive learning for improving control through deployment experience, making world modeling an active part of WAM adaptation.

1 INTRODUCTION

Post-deployment learning often improves behavior without requiring better predictions of deployment interactions, a gap that matters when dexterous-manipulation errors compound. DEWO pairs action supervision with visual learning from successful and failed futures, then uses progress-gated guidance to improve action generation.

1 INTRODUCTION

  • In dexterous manipulation, small action deviations can change contact and physical futures, causing errors to compound as executions depart from the training distribution.This challenges both action generation and the predictive representations that support it.Source: §1 · p. 1
  • DEWO identifies interaction turning points and learns outcome-conditioned world representations from matched successful and failed visual futures, using action targets only from successful continuations.The visual experience complements action imitation to better condition action generation.Source: §1 · p. 2
  • A value head estimates task progress from video representations and activates classifier-free guidance when progress stalls.Guidance steers actions toward the learned success condition.Source: §1 · p. 2, Figure 2 · p. 3
  • DEWO improves average success across three WAM paradigms, while visual-future supervision improves prediction and control beyond action supervision alone.On five DexJoCo tasks, it improves average success across joint modeling, video-then-action inverse dynamics, and FastWAM.Source: §1 · pp. 2–3

2 DIRECT EXPERIENCE WORLD-MODEL OPTIMIZATION

DEWO continues world-model learning on deployment experience, organizing matched continuations around interaction turning points. Outcome-conditioned visual prediction uses both successful and failed futures, while action supervision is reserved for successful behavior and progress estimates gate action guidance.

2.1 CONTINUED WORLD MODELING FROM DEPLOYMENT EXPERIENCE

  • DEWO treats observed physical futures from deployment as continued world-model learning targets, selected around interaction turning points.The paradigm organizes direct experience for predictive learning rather than introducing a new visual loss.Source: §2.1 · p. 3

2.2 FROM INTERACTION TURNING POINTS TO DIVERGENT FUTURES

  • Replay exploration probes failed-rollout contexts and retains matched successful and failed continuations whose futures diverge from comparable starting configurations.On physical robots, matching uses comparable task-relevant configurations rather than identical images or observation histories; sampled recoverability does not establish physical irrecoverability.Source: §2.2 · pp. 3–4

2.3 OUTCOME-CONDITIONED WORLD-MODEL LEARNING

  • Successful continuations supervise visual and action prediction under the success condition, while failed continuations supervise visual prediction under the failure condition.Failed futures extend predictive learning beyond the successful behavior available for action supervision.Source: §2.3 · p. 4

2.4 PROGRESS-GATED ACTION GUIDANCE

  • At inference, a value head predicts discounted terminal success from video representations and activates classifier-free action guidance when progress estimates fail to rise sufficiently.The gate adds the learned success direction to action prediction, while visual prediction remains base-conditioned.Source: §2.4 · p. 5

3 EXPERIMENTS

The experiments evaluate DEWO across simulation benchmarks and repeated real-world deployment. DEWO improves dexterous-control results across WAM formulations, while real-world gains are concentrated in positions with initial successes.

3.1 EXPERIMENTAL SETUP

  • The simulation evaluates five DexJoCo tasks across three 50-scene sets per task, while real-world grasp-and-place tests use four objects, two hands, and 3 × 3 position grids.Each real-world grid cell receives ten trials; R0 is the initial DEWO-S model trained near the center, followed by two DEWO collection–optimization rounds.Source: §3.1 · pp. 5–6

3.2 SIMULATION RESULTS

  • DEWO raises FastWAM success from 80.7% to 82.3% and FACT from 64.5% to 81.2%, exceeding their SFT comparisons.The advantage on both models extends beyond retaining visual supervision, since WAM SFT also uses visual futures.Source: Table 1 · p. 6, §3.2 · p. 6
  • Joint, IDM, and FastWAM improve by 4.8, 1.9, and 1.6 percentage points, respectively; DEWO-S reaches 89.9% average success versus 86.7% for adapted Joint.DEWO-S trains a Joint model from pretrained video and action components and provides the starting point for real-world evaluation.Source: Table 1 · p. 6, §3.2 · p. 6

3.3 REAL-WORLD RESULTS: ITERATION AND SPATIAL GENERALIZATION

  • Full-grid success rises from 20.6% (148/720) to 28.9% (208/720), with gains on both hands and all eight object–hand pairs improving from R0 to R2.Wuji rises from 18.9% to 26.7%, and Sharpa from 22.2% to 31.1%.Source: §3.3 · p. 7, Figure 3 · p. 7
  • On the fixed 29 cells with at least one R0 success, pooled success rises from 51.0% to 72.1% in R1 and 71.7% in R2.Off-center success rises from 40.0% to 65.7% and 65.2%, while all 43 initially zero cells remain at zero.Source: §3.3 · p. 7, Figure 3 · p. 7

4 ANALYSIS

The analysis tests visual-future supervision, turning-point experience selection, and guidance timing. It reports gains in task success and video prediction, with selective guidance outperforming continuous guidance in the tested tasks.

4.1 LEARNING CONSEQUENCES BEYOND ACTION IMITATION

  • Adding successful and then failed visual futures raises success from 86.7% to 88.7% to 90.7% on Water Plant and from 72.0% to 74.0% to 77.3% on Fold Glasses.Failed continuations supply no action-imitation targets, yet their observed consequences add a further control improvement.Source: §4.1 · p. 8, Figure 4 · p. 8
  • Full visual supervision reduces held-out video loss by 32.7% on Water Plant and 17.8% on Fold Glasses relative to action-only training.The loss falls on both tasks as supervision expands from successful to failed continuations.Source: §4.1 · p. 8, Figure 4 · p. 8

4.2 EXPERIENCE SELECTION AND SELECTIVE GUIDANCE

  • At matched data sizes and training budgets, critical experience reaches 90.7% versus 89.3% on Water Plant and 77.3% versus 74.0% on Fold Glasses for full trajectories.The comparison organizes supervision around consequential interactions and their divergent outcomes.Source: §4.2 · p. 8, Figure 4 · p. 8
  • Value-gated guidance achieves 90.7% and 77.3% success, versus 86.7% and 73.3% without guidance and 78.0% and 52.0% with continuous guidance.The reported gains support activating guidance selectively when estimated progress stalls.Source: Figure 4 · p. 8, §4.2 · p. 9

5 RELATED WORK

Prior work post-trains robot policies through action targets, rewards, or predictive objectives, and explores different data sources and settings for WAM adaptation. DEWO positions continued world modeling around interaction turning points and matched successful and failed continuations.

World-action models for robot control

  • WAMs use visual dynamics prediction to support action generation, either through predicted intermediate goals and inverse dynamics or joint future-and-action generation.FastWAM retains video prediction during training without explicitly generating futures at inference.Source: §5 · p. 9

Learning from deployment experience

  • Post-deployment learning includes action supervision and value-based reinforcement learning, while dexterous-control methods also use structured exploration and WAM predictive priors.The passage frames deployment futures as additional experience for refining representations that support action generation.Source: §5 · p. 9

Post-training world-action models

  • Prior WAM post-training uses imagined interactions, successful online rollouts, student-visited histories, or failed rollouts with masked action imitation.DEWO organizes matched successful and failed continuations around interaction turning points.Source: §5 · p. 9

6 CONCLUSION

DEWO makes continued world modeling part of post-deployment WAM learning, using matched successful and failed continuations around interaction turning points. The paper reports gains across WAM formulations and real-world embodiments, while framing sustained self-improvement as an open question.

6 CONCLUSION

  • DEWO makes continued world modeling an explicit part of post-deployment WAM learning, using diverging visual futures from matched successful and failed continuations to support control.Ablations show that this visual experience improves prediction and control beyond successful-action supervision alone.Source: §6 · p. 9
  • Real-world evaluations indicate that DEWO makes existing spatial generalization more reliable, while sustained self-improvement through interaction turning points remains an open question.The conclusion envisions agents learning how interaction changes can lead to success and applying that knowledge in later attempts.Source: §6 · p. 10

A.1 SIMULATION TASKS AND EVALUATION SETS

  • Simulation evaluations pool 150 trials per task across three 50-scene sets, with five-task averages weighting tasks equally across 750 trials.Seeds index evaluation scenes, not training runs.Source: §A.1 · p. 13

A.2 INITIALIZATION AND COMPARISON SCOPE

  • Table 1B compares Joint, IDM, and FastWAM before and after DEWO in a shared MoT, while Table 1A compares SFT, RECAP, and DSRL on π0.5 and FastWAM and SFT with DEWO on FACT.Within each base-model comparison, methods share the corresponding initial checkpoint; FACT uses FastWAM settings but its own backbone and initial checkpoint.Source: §A.2 · p. 13

A.3 COLLECTION BUDGETS AND TRAINING DATA

The appendices describe simulation collection budgets, data construction, and the optimization and checkpoint-selection settings for DEWO and DEWO-S.

A.3 COLLECTION BUDGETS AND TRAINING DATA

  • DEWO collection queries up to 20 anchors with ten continuations each, allowing at most 200 extra attempts per failed trajectory, with early stopping able to reduce the cost.Simulation collection uses one round and 200 initial attempts per task; SFT and RECAP use the same collected data as DEWO in the FastWAM comparison.Source: §A.3 · p. 13
  • Training pools include complete successful trajectories, value-supervision scan entries, retained successful continuations or stitches, and failed cliff windows; source counts are recorded before window expansion.DEWO uses deployment experience, while DEWO-S additionally includes 500 expert demonstrations; FastWAM windows span 33 action-clock frames, and validation is split by episode before window expansion.Source: §A.3 · pp. 13–14, Table 2 · p. 14

A.4 OPTIMIZATION AND CHECKPOINT SELECTION

  • DEWO adapts the five-task FastWAM checkpoint with a frozen backbone and low-rank adapters, while DEWO-S trains the full Joint WAM; checkpoints are selected by open-loop validation loss.The DEWO checkpoint is selected after 10,000 adaptation updates; DEWO-S is selected after 35,000 training updates. Closed-loop evaluation sets report success but do not select checkpoints.Source: §A.4 · pp. 14–15, Table 3 · p. 14, Table 4 · p. 15

B DEWO IMPLEMENTATION DETAILS

DEWO implementation selects turning-point experience from rollout continuations, applies outcome-conditioned world-model losses, and uses a value head to gate action guidance when estimated progress stalls. DEWO-S applies the objective during construction of a Joint WAM from pretrained components.

B.1 CRITICAL-EXPERIENCE QUERIES

  • Simulation queries estimate recoverability from ten continuations at each anchor; neighboring-anchor drops select failed cliff windows, while terminal zero-success queries stop exploration and retain a successful counterpart only at the preceding nonzero anchor.For real robots, anchors are proposed by diagnosing failed rollouts and restoring a corresponding pre-task configuration; restoration does not reproduce identical camera frames or observation prefixes.Source: §B.1 · pp. 15–16

B.2 OUTCOME CONDITIONS, POOL MIXING, AND LOSSES

  • Successful trajectories and continuations provide video, action, and value supervision, while failed windows provide video and value supervision with action targets masked; the objective also includes adapter residual regularizers.Failed windows receive a failure condition, and successful continuations use the success condition with a 0.1 probability of replacement by the base instruction; DEWO-S omits the adapter regularizers.Source: §B.2 · p. 16

B.3 VALUE LEARNING AND VIDEO FEATURES

  • The value head predicts discounted terminal success from detached video-representation features, and its regression targets use continuation averages at queried contexts, taking precedence over individual retained-branch returns.The value head uses current-observation tokens and requires no future-video generation at inference.Source: §B.3 · pp. 16–17

B.4 DISCOUNT-INFORMED PROGRESS GATING

  • Progress gating detects a stall when no value estimate in the monitoring window reaches the required increase over its starting value, then applies guidance only within the eligible replanning interval.The discount-based reference motivates α = 1.27 over H = 5 replans; for five FastWAM simulation tasks, guidance is enabled from replans 10 through 24, and DEWO-S uses w = 0.Source: §B.4 · p. 17

B.5 DEWO-S: CONSTRUCTION FROM PRETRAINED COMPONENTS

  • DEWO-S constructs a Joint WAM from pretrained VideoDiT and ActionDiT components, then trains the full model under the DEWO objective before deployment adaptation.The value head’s final layer is initialized to zero.Source: §B.5 · pp. 17–18

C.1 PLATFORMS, TASKS, AND SUCCESS CRITERIA

The real-world evaluation measures complete grasp-and-place performance across two dexterous-hand platforms, four objects, fixed spatial grids, and successive deployment-learning checkpoints.

C.1 PLATFORMS, TASKS, AND SUCCESS CRITERIA

  • The evaluation pairs Wuji and Sharpa hands with four objects, and counts success only when the robot grasps the object and places it in the target basket.Grasp acquisition and stability are identified as the principal difficulties by inspection.Source: §C.1 · p. 18
  • Each object–embodiment pair uses a fixed 3 × 3 grid, with ten trials per cell, for 90 trials per pair and 720 trials per checkpoint.All nine cells enter the full-grid metric, including locations with no observed successes.Source: §C.1 · p. 18

C.2 VISUAL OBSERVATIONS AND PREPROCESSING

  • Real-world input concatenates resized head-view and wrist-view RGB images into a 224 × 448 image; videos are stored at 30 frames/s.Simulation uses front and wrist RGB views, each resized to 224 × 224 before concatenation.Source: §C.2 · p. 18

C.3 ROUND DEFINITIONS AND CONTINUATIONS

  • R0, R1, and R2 denote the initial DEWO-S checkpoint and checkpoints after one and two additional collection–optimization cycles.The separate π0.5 reference is not a DEWO-S deployment round.Source: §C.3 · p. 18

C.4 FIXED SPATIAL GROUPS AND AGGREGATION

  • Spatial groups are fixed by Round 0 outcomes: the initially nonzero group has 29 cells and a pooled denominator of 290 trials per round.The full-grid success rate is reported separately over all 72 cells.Source: §C.4 · p. 18, Table 6 · p. 19
  • 23 of 29 cells with a nonzero initial count improve from R0 to R2; none of the 43 initially zero cells records a later success.The authors describe the group-wise trends as descriptive because group membership is defined using R0 outcomes.Source: §C.4 · p. 18

D ABLATION PROTOCOLS AND VIDEO METRIC

The ablations compare action and visual-future supervision, experience selection, and guidance timing. The accompanying guidance analysis frames the method as a reference-guided preference, while limiting what that interpretation establishes.

D.1 VISUAL AND ACTION SUPERVISION

  • The supervision ablation compares successful-action supervision with variants adding successful visual futures and then failed visual futures, with failed action targets masked.The comparison reports control and prediction outcomes for the variants.Source: §D.1 · p. 19

D.2 OFFLINE VALIDATION VIDEO LOSS

  • The offline prediction metric is the model’s native video loss on a validation set, while checkpoint selection uses the total validation objective.The video metric uses the model’s loss normalization, masks, and generative-time/noise handling.Source: §D.2 · p. 19
  • Full supervision reduces validation video loss from 0.196 to 0.132 on Water Plant and from 0.214 to 0.176 on Fold Glasses, with higher task success on both tasks.These are relative loss reductions of 32.7% and 17.8%, respectively.Source: §D.2 · p. 19

D.3 EXPERIENCE SELECTION AND GUIDANCE TIMING

  • The selection ablation compares full-trajectory training with critical experience at matched training-data quantities and budgets; guidance timing varies inference settings on one adapted checkpoint.The guidance settings are no guidance, continuous guidance, and value-gated guidance.Source: §D.3 · p. 19

E REFERENCE-GUIDED ACTION GENERATION

  • The guidance analysis treats the action branches as idealized critical-success and base-reference distributions, not as a ratio identified with true environment success probability.Its interpretation is time-local and does not establish that the complete guided sampler exactly produces the clean-action density or monotonically improves true task success.Source: E REFERENCE-GUIDED ACTION GENERATION · pp. 19–20

F SIMULATION RESULTS BY EVALUATION SET

The simulation-results materials describe evaluation-set and table aggregation formats, but the supplied passages do not report simulation outcome values or comparisons.

F SIMULATION RESULTS BY EVALUATION SET

  • Each seed identifies one fixed 50-scene evaluation set, and reported table means are arithmetic means rather than variation across independently trained models.Table 1 reports per-task means and equally weighted task averages.Source: F SIMULATION RESULTS BY EVALUATION SET · p. 20
  • Tables 7–11 are captioned as per-evaluation-set success-rate tables for π0.5, FastWAM, Joint WAM, IDM WAM, and DEWO-S, respectively.The supplied captions do not include the table entries.Source: Table 7 · p. 21, Table 8 · p. 21, Table 9 · p. 22
  • Table 12 reports FACT success rates across three 50-scene evaluation sets per task, with all-task rows aggregating 250 trials per set and 750 overall.Its caption specifies one fixed checkpoint per method.Source: Table 12 · p. 23

G REAL-WORLD SPATIAL EVALUATION GRIDS

The real-world spatial evaluation displays complete grasp-and-place trial counts on a 3 × 3 grid for the reference and three DEWO-S checkpoints.

G REAL-WORLD SPATIAL EVALUATION GRIDS

  • Each grid entry is successes out of ten trials, arranged top-left to bottom-right; C marks the center, and R0/R1/R2 denote successive DEWO-S checkpoints.To align with Figure 3a, water-bottle matrices are vertically flipped and other object matrices transposed; this preserves counts and center-cell membership.Source: G REAL-WORLD SPATIAL EVALUATION GRIDS · p. 23
  • Table 13 presents π0.5 and DEWO-S Rounds 0–2 in four panels, with each spatial entry reporting complete grasp-and-place successes out of 10 trials.Overall reports totals out of 90 trials and the corresponding success rate.Source: Table 13 · p. 24

Related papers

Loading 2609.37398v1…