PaperlayerView exact revision on arXiv

Source-linked AI summary

Can World and Action Modeling Benefit Each Other under Unified Causal Training?

Yue Su, Bokai Lin, Xinkai Wang, Chubin Zhang, Xiangnan Wu, Kuan Zhang, Weiyang Jin, Jiwen Yu, Yiming Li, Cewu Lu, Lixin Yang, Xihui Liu

arXiv:2610.21095v1cs.CV

TL;DR

The reciprocal benefits of world modeling and action generation under unified causal training remain unclear. OneWM combines clean-action-conditioned forward prediction and action generation in a causal video-action model, and experiments report benefits in both directions. The results support reciprocal benefits within the tested settings.

  • Problem

    How action-conditioned forward dynamics and action generation affect each other under temporal causality remains an open question.

    Source: §1 · pp. 1–2

  • Method

    OneWM mixes all-clean, partial-clean, and all-noisy action conditioning to train forward prediction and action generation in a shared causal model.

    Source: §1 · p. 2

  • Results

    OneWM improves policy performance and generalization with clean-action dynamics learning, while action learning improves action-conditioned world prediction under the same architecture.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results support reciprocal benefits between forward-dynamics learning and action generation under unified causal training.

    Source: §5 · p. 10

Abstract

from arXiv · show

World and action modeling are closely related, yet their reciprocal benefits under unified causal training remain unclear. We introduce OneWM, a simple framework for unified causal training that treats actions as both conditions for world prediction and targets for policy learning within a shared video-action architecture. Using this framework, we investigate whether learning forward dynamics conditioned on specified actions improves action learning, and whether learning to generate actions in turn improves action-conditioned world modeling. We use LIBERO, LIBERO-Plus, and RoboCasa365 as embodied testbeds to evaluate both policy learning and world modeling, complemented by real-world policy experiments. Learning the visual consequences of specified actions improves policy performance and generalization across changing scenes. In turn, action learning improves multi-view world-prediction fidelity under the same architecture. Together, these results answer both questions positively, supporting reciprocal benefits between world modeling and action generation under unified causal training.

1 Introduction

World and action modeling are complementary, but their reciprocal benefits under temporal causality remain unclear. OneWM unifies action-conditioned world prediction and action generation to test whether each improves the other.

1 Introduction

  • World models predict future observations from specified actions, whereas WAMs use video generation to guide action generation, motivating a unified study of both objectives.The paper frames the objectives as closely related and complementary.Source: §1 · p. 1
  • Existing video prediction and WAM approaches support action learning, but joint denoising with noisy actions differs from forward prediction conditioned on specified clean actions.This distinction motivates testing whether clean-action forward dynamics improves action learning and whether action generation improves action-conditioned world modeling.Source: §1 · pp. 1–2
  • OneWM mixes all-clean, partial-clean, and all-noisy action conditioning to train forward-dynamics prediction and action generation within one causal model.The framework compares clean-action conditioning with WAMs that use noisy actions or mask action keys.Source: §1 · p. 2, Figure 1 · p. 2
  • Across simulated and real-world environments, OneWM improves action-prediction performance over evaluated WAMs and action generation also improves action-conditioned world modeling.The results provide positive answers to both questions under unified causal training.Source: §1 · p. 2

2 Related Work

Prior work uses predictive video representations, learned dynamics, and shared video-action models to support policies and world prediction. Reported reciprocal benefits and trade-offs across modeling settings leave open how temporal causality shapes those outcomes.

2.1 World Action Models

  • World-action models support policies through visual plans, predictive video features, and joint future-observation/action learning, with temporal supervision and video pretraining reported to improve generalization and data efficiency.Some approaches recover executable actions from generated videos using inverse dynamics or goal-conditioned controllers.Source: §2.1 · p. 3
  • Efficient WAMs reduce or bypass future-video synthesis at deployment, while latent, condition-space, and distillation approaches transfer predictive representations to policies.This paper makes visual consequences of specified actions an explicit policy-training target.Source: §2.1 · p. 3

2.2 World Models

  • World models learn transition dynamics in latent or visual spaces, and robotic models connect visual priors to commands through action injection, kinematic conditioning, or latent-action pretraining.Predictions that follow supplied actions are distinguished from visual plausibility; action-generation supervision is also used here to strengthen prediction representations.Source: §2.2 · p. 3

2.3 Unified World and Action Modeling

  • Unified video-action models span bidirectional diffusion and causal designs, with prior studies reporting both mutual benefits and trade-offs.WorldVLA reports improved action generation and long-horizon visual prediction, whereas Cosmos 3 reports a small decline in video-generation fidelity from action co-training.Source: §2.3 · p. 3
  • Different prior modeling settings leave open how temporal causality shapes physical-dynamics learning and video-action outcomes, motivating OneWM’s shared causal framework.OneWM explicitly couples action-conditioned forward-dynamics learning with action generation.Source: §2.3 · p. 3

3 Method

OneWM jointly models future video and action chunks from visual context and language instructions. Its causal attention and mixed action-conditioning regimes train the model for both specified-action prediction and action generation.

3.1 Problem Formulation

  • OneWM formulates future-video prediction and action generation from an observed frame prefix and language instruction, using a video VAE for latents and action embeddings for action chunks.The observed video prefix stays clean while future latents are denoising targets.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
  • Separate video and action experts exchange information through temporally causal joint attention, with each receiving language conditioning through its own text cross-attention.Across samples, actions act as clean conditions, noisy targets, or a clean prefix followed by a noisy suffix.Source: §3.1 · p. 4, §3.2 · p. 4, Figure 2 · p. 4

3.2 Temporally Causal Video--Action Modeling

  • Future-video queries access video and action prefixes through their aligned block, while action queries access observed visual context and the corresponding action prefix.The causal visibility relations connect the tasks while preserving temporal ordering between blocks.Source: §3.2 · pp. 4–5

3.3 Unified Action Conditioning

  • All-clean actions condition future-video prediction; partial-clean actions condition denoising of the remaining suffix; all-noisy actions are generated from visual context and instruction.The regime and clean-prefix length are sampled per example, mixing specified-action dynamics and action generation within a batch.Source: §3.3 · p. 5

3.4 Joint Training Objective

  • Joint flow-matching training combines video and action velocity losses, with action loss applied only to examples that contain a noisy action suffix.Thus, all-clean examples supervise forward dynamics, while partial-clean and all-noisy examples also supervise action generation; the coupled experts are optimized through causal joint attention.Source: §3.4 · pp. 5–6

3.5 Inference

  • At inference, OneWM either predicts future video from fixed clean actions or generates actions from noise using visual context and instruction.For closed-loop control, it generates a new action chunk from the latest observations at each replanning step.Source: §3.5 · p. 6

4 Experiments

Experiments test both transfer directions across simulation benchmarks and three real-world manipulation tasks, using success rates for policies and image/video metrics for prediction.

4.2 World Modeling for Action Learning

  • OneWM outperforms both WAM variants on LIBERO and LIBERO-Plus, and WAMmask and listed reference policies across RoboCasa365 task groups; it also beats Fast-WAM across LIBERO-Plus perturbations.The comparison spans in-distribution tasks, perturbations, and complex stationary and mobile manipulation.Source: Table 1 · p. 7, §4.2 · pp. 7–8

4.1 Experimental Setup

  • Evaluation covers LIBERO, LIBERO-Plus, and RoboCasa365 in simulation, plus three real-world tasks; policy success rates and PSNR, SSIM, LPIPS, FID, and FVD measure the two tasks.Comparisons include action-masked and noisy-action-conditioned WAMs, and a clean-action-conditioned world-model baseline without action prediction.Source: §4.1 · p. 7

4.2 World Modeling for Action Learning

  • OneWM improves over WAMmask on block stacking, block insertion, and breakfast arrangement, especially insertion and breakfast arrangement.The real-world comparison tests relative placement, precise alignment, and successive scene-changing manipulations.Source: Figure 4 · p. 8, §4.2 · p. 8

4.3 Action Learning for World Modeling

  • OneWM achieves the best PSNR, SSIM, and LPIPS on all three benchmarks and improves paired prediction quality and FVD across them, as well as FID on LIBERO-Plus and RoboCasa365.WM is consistently second on PSNR, SSIM, and LPIPS; WMAdaLN has higher FID/FVD in some settings.Source: Table 2 · p. 9, §4.3 · p. 9

4.4 Effect of the Action-Conditioning Mixture

  • The 60/25/15 mixture best balances policy generalization and paired world-prediction quality among tested mixtures, while a larger clean-conditioning share weakens policy performance, particularly under distribution shift.Performance remains broadly stable between 60/25/15 and 50/35/15; the latter brings a small in-distribution policy gain.Source: §4.4 · pp. 9–10, Table 3 · p. 10

4.5 Endless Dream: Closing the Loop in Imagination

  • Without additional training, OneWM alternates action generation and action-conditioned prediction for 16 LIBERO rounds, producing 512 actions over a modeled horizon of 25.6 seconds.Each round generates 32 actions and eight future frames, feeding the final predicted camera frames into the next round without further environment observations.Source: §4.5 · p. 10

5 Conclusion

OneWM combines action-conditioned forward prediction and action generation in a causal model, with experiments indicating reciprocal benefits for policy learning and world prediction.

5 Conclusion

  • Explicit forward-dynamics learning improves policy performance and generalization, while action learning improves action-conditioned world prediction within the same architecture.Real-world manipulation experiments further validate the policy benefit.Source: §5 · p. 10
  • OneWM mixes all-clean, partial-clean, and all-noisy examples to combine clean-action-conditioned forward prediction with action generation.The mixture study finds that moderate rebalancing preserves the benefit, while larger shifts toward clean conditioning mainly weaken policy generalization.Source: §5 · p. 10
  • The results suggest the objectives supervise complementary aspects of interaction: the visual consequences of controls and the scene structure needed to select them.The conclusion motivates studying reciprocal learning with more diverse data and longer interaction horizons, and using learned dynamics for planning and policy improvement.Source: §5 · p. 10

A.1 Training and Deployment

Training and deployment use dataset-specific budgets across simulation and real-world tasks, with action generation and world prediction sharing the model.

A.1 Training and Deployment

  • LIBERO and LIBERO-Plus training uses 20 epochs on demonstrations from four LIBERO suites, while RoboCasa365 training uses 90k optimizer steps.The global batch sizes are 128 for LIBERO and LIBERO-Plus, and 1024 for RoboCasa365.Source: §A.1 · p. 16
  • The three real-world tasks use 150 demonstrations total, with 50 demonstrations per task and 20 training epochs.Training uses a global batch size of 96.Source: §A.1 · p. 16
  • Unless stated otherwise, OneWM samples all-noisy, partial-clean, and all-clean action conditioning with probabilities 60%, 25%, and 15%.Internal WAM and WM ablations use the same dataset-specific data, batch size, and training budget.Source: §A.1 · p. 16
  • At deployment, policies denoise action chunks from Gaussian noise using current visual observations and language, while world prediction holds specified actions clean during future-video denoising.Observed video representations are reused during action denoising without generating future video.Source: §A.1 · p. 16

A.2 Evaluation Protocols

Policy success and future-video prediction are evaluated across benchmark-specific tasks and views, using complementary reconstruction and distributional metrics.

A.2 Evaluation Protocols

  • Policy evaluation reports closed-loop success across benchmark-specific task groups and aggregates, with each real-world task measured over 15 trials per model.RoboCasa365 results cover atomic-seen, composite-seen, and composite-unseen tasks.Source: §A.2 · p. 16
  • Video evaluation uses expert-rollout sets of 100 complete episodes total across LIBERO and LIBERO-Plus, plus 100 for RoboCasa365, with multi-view future predictions from external and wrist cameras.Prediction chunks span 32 action steps at stride 4 for LIBERO and LIBERO-Plus, and 16 steps at stride 2 for RoboCasa365; each chunk evaluates eight future frames, excluding conditioning frames.Source: §A.2 · p. 16
  • PSNR measures pixel reconstruction accuracy and SSIM compares local luminance, contrast, and structure, with higher values better for both metrics.LPIPS measures perceptual distance using a VGG16 backbone, with lower values better.Source: §A.2 · p. 16
  • FID compares generated and reference image-feature distributions, while FVD compares video features; lower values are better for both.FVD uses eight future frames in temporal order.Source: §A.2 · p. 17
  • PSNR, SSIM, and LPIPS average over evaluated future frames, whereas FID and FVD compute Fréchet distance from feature statistics pooled over each benchmark.The Fréchet metrics do not average per-batch scores.Source: §A.2 · p. 17

B Qualitative Analysis of Action-Conditioned World Prediction

Matched qualitative predictions compare OneWM with world-model baselines across LIBERO, LIBERO-Plus, and RoboCasa365 views, illustrating differences in conditional spatial fidelity.

B Qualitative Analysis of Action-Conditioned World Prediction

  • In main-camera LIBERO and LIBERO-Plus cases, OneWM more closely matches object, gripper, and scene relationships than WMAdaLN and WM.The six cases compare ground truth, OneWM, and the indicated baseline; orange boxes mark details enlarged at t + 32.Source: B Qualitative Analysis of… · p. 18, Figure 5 · p. 18
  • Wrist-camera cases show OneWM better preserving object position, shape, and pose near the gripper than WMAdaLN and WM.The matched comparisons enlarge the same marked region at t + 32 across rows.Source: B Qualitative Analysis of… · p. 19, Figure 6 · p. 19
  • PSNR, SSIM, and LPIPS compare predictions with corresponding references, but pooled FID and FVD statistics are unchanged if generated clips are reassigned to different action sequences.A lower distributional distance therefore need not indicate more accurate action-conditioned geometry.Source: B Qualitative Analysis of… · p. 19
  • On RoboCasa365, OneWM more closely tracks forearm withdrawal, gripper–door motion, gripper descent, and orientation across left, wrist, and right views.The baselines overpredict viewpoint change in wrist views, shifting fridge geometry and bringing a bowl into view too early; Figure 7 enlarges details at t + 16.Source: B Qualitative Analysis of… · p. 20, Figure 7 · p. 20

C Endless Dream on LIBERO

The LIBERO endless-dream experiment alternates action prediction and action-conditioned world prediction, generating later visual context from one initial observation.

C Endless Dream on LIBERO

  • Each rollout alternates action prediction with action-conditioned world prediction for 16 rounds, with 32 actions and eight predicted frames per round.Only the initial observation comes from the environment.Source: C Endless Dream on LIBERO · p. 21
  • The generated action chunk passes unchanged to world prediction, whose final multi-view prediction becomes the next visual context.Each rollout starts from main- and wrist-camera images plus a task instruction.Source: C Endless Dream on LIBERO · p. 21
  • Both inference stages use 10 denoising steps and a fixed seed of 42.The jointly trained LIBERO model uses the default conditioning mixture.Source: C Endless Dream on LIBERO · p. 21
  • Figure 8 shows six rollouts with main- and wrist-camera views; blue outlines mark initial observations, and columns indicate modeled time through round 16.Every later frame is generated.Source: Figure 8 · p. 21

Related papers

Loading 2610.21095v1…