PaperlayerView exact revision on arXiv

Source-linked AI summary

OneWorld: Learning Consistent Physics Across Actions in World Models

Ke He, Yichen Ding, Bin Yang

arXiv:2609.30946v1cs.CVcs.AI

TL;DR

Action-conditioned futures from the same initial scene can appear plausible while implying incompatible physical properties. OneWorld aggregates branch-level mechanism evidence to couple such futures during training and sampling. Relative to ACWM-DiT, it reduces mean shared-world gap by 84.3% while improving mean M-MSE by approximately 7.7 × 10^-4.

  • Problem

    Current action-conditioned world models do not explicitly enforce consistency across futures from the same initial world, which can each appear plausible while implying incompatible physical properties.

    Source: §1 · pp. 1–2

  • Method

    OneWorld infers a latent mechanism distribution for each action–outcome branch and aggregates branch evidence to guide shared-world flow training and sampling.

    Source: §1 · pp. 2–3

  • Results

    OneWorld reduces mean shared-world gap by 84.3% relative to ACWM-DiT while improving mean M-MSE by approximately 7.7 × 10^-4.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across controlled environments, cross-intervention physical consistency improves while single-rollout prediction quality remains strong.

    Source: §5 · p. 9

    The study evaluates consistency in simulation and on public video-prediction benchmarks, not in a live robotic system.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Action-conditioned video world models aim to predict scene evolution under different actions, a capability that is essential for reliable planning, decision-making, and interaction in dynamic environments. However, futures generated independently from the same initial scene may each appear plausible while implying incompatible physical properties, such as friction or mass. This inconsistency can lead to contradictory predictions across interventions, making it difficult for the model to maintain a coherent understanding of the underlying world and limiting its reliability for planning and decision-making. To address these issues, we propose OneWorld, a shared-mechanism counterfactual generation framework that jointly models multiple action-conditioned futures under a common latent physical mechanism. A physical mechanism interpreter first infers a distribution over latent mechanisms from each action-outcome branch. These distributions are then aggregated into shared-world evidence, which captures whether the branches admit a common physical explanation while accounting for uncertainty in less informative branches. This evidence constrains flow training and guides sampling, encouraging consistency in the underlying physical mechanism while preserving the distinct outcomes induced by different actions. We further introduce a multi-intervention evaluation protocol in controlled environments, following the interaction settings of ACWM-Phys, to assess whether generated futures can be jointly explained by the same physical parameters, alongside standard measures of single-rollout prediction quality. Experiments in these environments show that OneWorld improves cross-intervention physical consistency while maintaining competitive single-rollout prediction quality.

1 INTRODUCTION

Action-conditioned futures from the same initial world can be individually plausible yet imply incompatible physical properties. OneWorld couples their physical interpretations, and its controlled-environment evaluation reports improved cross-intervention consistency alongside single-rollout quality.

1 INTRODUCTION

  • ACWM-DiT futures can each appear plausible yet require incompatible physical properties; the shared-world gap measures the extra fitting error imposed by requiring one physical configuration.Nonzero gaps are reported for ACWM-DiT.Source: §1 · pp. 1–2
  • Interventions provide unequal information about physical properties, so compatible branches may support the same mechanism with different uncertainty.A barely disturbed object can remain consistent with many physical configurations, while a strong interaction can constrain plausible mechanisms more sharply.Source: §1 · p. 2
  • OneWorld infers a mechanism distribution for each action–outcome branch and aggregates the distributions into shared-world evidence that tolerates uncertainty in less informative branches.The evidence guides shared-world flow training and sampling while allowing different videos and mechanism posteriors across branches.Source: §1 · pp. 2–3
  • The evaluation uses Push Cube, Push Rope, and Pour Water, spanning rigid-body, deformable-object, and particle-based dynamics.Multiple interventions can be executed from the same underlying physical configuration in these controlled environments.Source: §1 · pp. 2–3
  • OneWorld reduces mean shared-world gap by 84.3% relative to ACWM-DiT while improving mean M-MSE by approximately 7.7 × 10^-4.The evaluation measures both single-rollout prediction quality and whether multiple futures can be explained by one physical configuration.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans action-conditioned prediction, physical reasoning and evaluation, and counterfactual consistency. OneWorld differs by assessing whether multiple action-conditioned futures share a compatible physical explanation.

2 RELATED WORK

  • Action-conditioned world-model objectives primarily supervise each rollout individually, while planning and interaction systems condition future observations on actions.Diffusion and flow matching provide generative foundations for visual prediction.Source: §2 · p. 3
  • Physical-prior methods and motion benchmarks address plausible dynamics, but existing evaluation is largely single-rollout rather than testing a shared configuration across futures (Xue et al., 2026).ACWM-Phys provides the interaction settings and action-conditioned backbone used in the controlled experiments.Source: §2 · p. 3
  • Prior counterfactual approaches include latent-factor reasoning, causal consistency, action-responsiveness constraints, and coupling through shared prefixes or future noise (Shi et al., 2026; Ma et al., 2026).OneWorld instead couples futures through compatibility of inferred physical explanations without requiring identical branch posteriors or shared sampling noise.Source: §2 · p. 3

3 METHOD

OneWorld jointly predicts action-conditioned futures from a shared initial scene while encouraging them to support a common latent physical mechanism. It uses inferred mechanism distributions to constrain both flow training and sampling without forcing branches to produce identical futures.

3 METHOD

  • OneWorld uses branch-wise physical mechanism interpretations to couple action-conditioned futures through a shared compatibility signal for training and sampling.The framework keeps branch-specific action predictions while constraining their underlying physical explanations.Source: §3 · p. 3, Figure 2 · p. 4

3.1 TASK FORMULATION AND FLOW PRELIMINARIES

  • Different interventions may produce distinct outcomes, but futures from the same initial world should be compatible with one shared latent mechanism.The model treats that mechanism as action-independent world properties such as friction, mass, or stiffness.Source: §3.1 · p. 4

3.2 MECHANISM INTERPRETATION AND SHARED-WORLD EVIDENCE

  • The mechanism interpreter maps each action–outcome branch to a mechanism distribution, and shared-world evidence scores their compatibility relative to an initial-scene prior.The score accommodates unequal information across branches, so weakly informative interventions can remain uncertain.Source: §3.2 · p. 5

3.3 LEARNING THE MECHANISM INTERPRETER

  • Relational supervision trains the interpreter to distinguish same-world groups from groups combining branches generated under different physical configurations.The interpreter and prior are trained before the video model, then fixed as the compatibility criterion.Source: §3.3 · p. 5

3.4 SHARED-WORLD FLOW TRAINING

  • Shared-world flow training raises group compatibility while retaining branch-wise flow matching, coupling predictions through physical interpretations rather than directly matching videos or latents.The interpreter is frozen while gradients from the shared-world loss update the video model.Source: §3.4 · pp. 5–6

3.5 SHARED-WORLD FLOW GUIDANCE

  • At inference, a group-level guidance correction uses the other branches’ physical explanations to steer each branch toward greater compatibility.Guidance grows stronger as generation becomes more informative; branch actions and Gaussian noise remain independent, so futures need not be identical.Source: §3.5 · p. 6

4 EXPERIMENTS

OneWorld is evaluated across controlled physical environments for both single-rollout prediction quality and whether action-conditioned futures share a physical explanation. It reduces the shared-world gap while retaining or improving prediction quality, and the analyses examine alternative coupling strategies, RoboDesk behavior, and component sensitivity.

4.1 EXPERIMENTAL SETUP

  • The benchmark covers Push Cube, Push Rope, and Pour Water, spanning rigid-body, deformable-object, and fluid dynamics; baselines include independent prediction and alternative cross-branch coupling strategies.Same-world groups share an initial state and hidden physical configuration while taking different actions; mismatched-world groups vary the hidden configuration while matching the visible scene.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • Mean Δworld falls from 0.02382 to 0.00374 versus ACWM-DiT (84.3%); OneWorld also has the lowest M-MSE in all three environments, improving the mean by approximately 7.7 × 10−4.E_ind remains low and E_shared decreases substantially, so the smaller gap does not arise from uniformly poor physical fits.Source: Table 1 · p. 7, §4.2 · pp. 7–8
  • Vid2World, CoCo, and Twin Rollouts reduce the shared-world gap versus ACWM-DiT but remain above OneWorld; Shared Noise slightly improves M-MSE while increasing the gap.Posterior Alignment is the strongest alternative on consistency metrics, but is weaker than OneWorld’s prior-corrected shared-support criterion.Source: §4.2 · p. 8

4.3 QUALITATIVE ANALYSIS ON ROBODESK

  • On RoboDesk, OneWorld preserves distinct outcomes for slow versus fast drawer pulls and short versus long door pushes while maintaining coherent post-contact dynamics.The comparison accounts for contact and release conditions rather than requiring identical displacements.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 ABLATION STUDIES

  • Removing mismatched-world supervision worsens the gap, while shared-world training and flow guidance together yield the lowest gap and E_shared among tested variants.On Push Cube, increasing K from 2 to 5 raises Δworld with small M-MSE changes; increasing λ_shared and λ_max trades consistency against prediction quality, while importance-sampling gains are limited beyond M = 64.Source: §4.4 · p. 9, Table 2 · p. 9, Figure 4 · p. 9

5 CONCLUSION

OneWorld couples action-conditioned futures through a shared physical explanation, and the conclusion reports improved cross-intervention consistency with strong single-rollout quality. The appendices explain the evidence score, its prior-matching neutrality, and the scope of external prediction evaluation.

5 CONCLUSION

  • Across three controlled environments, OneWorld improves cross-intervention physical consistency while maintaining strong single-rollout prediction quality.Recovery and interpolation analyses also find shared explanations aligned with simulator configurations beyond the discrete training settings; the conclusion argues joint evaluation can reveal inconsistencies missed by single-rollout evaluation.Source: §5 · p. 9

A.1 AN IDEALIZED EVIDENCE-RATIO INTERPRETATION

  • Under idealized Bayesian assumptions, shared-world evidence compares a common-mechanism model with independent branch mechanisms, but its learned score is not a calibrated Bayes factor.The correction removes repeated prior factors, leaving one prior contribution in the shared-mechanism model.Source: §A.1 · p. 12

A.2 NEUTRALITY OF A PRIOR-MATCHING BRANCH

  • Adding a branch whose posterior matches the initial-scene prior leaves shared-world evidence unchanged; this algebraic property does not guarantee the learned interpreter will match that prior for uninformative interventions.The score is 1 for a single branch and when all branch posteriors equal the prior.Source: §A.2 · p. 13

B EXTERNAL ACTION-CONDITIONED VIDEO PREDICTION

  • The external BAIR and RoboNet evaluation measures conventional single-rollout quality because those datasets lack the grouped physical supervision required for shared-world evaluation.Both use two observed context frames, with 12 future frames on BAIR and 10 on RoboNet; shared-world metrics are reserved for controlled environments.Source: B EXTERNAL ACTION-CONDITIONED VIDEO… · p. 13, §B.1 · p. 13

B.2 PREPROCESSING AND TEMPORAL ALIGNMENT

  • On BAIR and RoboNet, evaluation reuses held-out clips across locally evaluated models and reports FVD, PSNR, SSIM, and LPIPS on matched-resolution benchmarks.Baseline values come from published results or matched-protocol comparisons, and results from different spatial resolutions are not combined in one table.Source: §B.2 · p. 13, §B.5 · p. 14, §B.4 · p. 14

B.3 MODEL CONFIGURATION ON BAIR AND ROBONET

  • The appendix’s external backbone training uses standard branch-wise action-conditioned flow matching, without the mechanism interpreter, shared-world loss, or shared-world guidance.The datasets do not provide OneWorld’s grouped physical supervision.Source: §B.3 · p. 14

B.6 EXTERNAL PREDICTION RESULTS

The action-conditioned DiT backbone achieves competitive conventional prediction metrics on BAIR and RoboNet. The accompanying RoboDesk setup evaluates qualitative transfer using hidden physical properties and same-world versus mismatched-world intervention groups.

B.6 EXTERNAL PREDICTION RESULTS

  • 41.2 FVD versus 45.0 for DyDiff on BAIR; the DiT backbone also records PSNR 27.9, SSIM 95.8, and LPIPS 2.8.These results support the paper’s characterization of strong conventional single-rollout prediction quality.Source: §B.6 · p. 15, Table 5 · p. 15
  • 52.6 FVD, 30.6 PSNR, 95.3 SSIM, and 1.7 LPIPS are reported for the DiT backbone on RoboNet.Together with BAIR, the authors characterize the backbone as competitive under standard video-prediction metrics.Source: §B.6 · p. 15

C ROBODESK QUALITATIVE EVALUATION SETUP

  • RoboDesk’s Open Drawer and Sliding Door tasks hide friction and damping while fixing the visible scene, enabling qualitative comparison of action-conditioned futures across latent interaction properties.Each task uses nine physical configurations; the displayed slow/fast pulls or short/long pushes differ in action duration or speed.Source: C ROBODESK QUALITATIVE EVALUATION SETUP · p. 15, §C.1 · p. 15

C.2 SAME-WORLD AND MISMATCHED-WORLD GROUPS

  • RoboDesk same-world groups use three actions under one initial state and physical configuration, while mismatched-world groups retain the visible scene but vary hidden physical properties.Figure 3 displays two branches from held-out three-branch groups; the third still contributes to mechanism learning and shared-world training.Source: §C.2 · p. 16

C.3 TRAINING PROTOCOL

The appendix details the RoboDesk training and qualitative comparison protocol, then specifies the mechanism interpreter, relational supervision, evidence estimation, and gradient flow used to couple branches.

C.3 TRAINING PROTOCOL

  • For RoboDesk, OneWorld starts from the ACWM-DiT checkpoint, learns the interpreter and prior on grouped data, freezes them, then optimizes the video model with flow and shared-world losses.Vid2World and CoCo use the same task-specific backbone, training trajectories, and optimization budget; no method receives hidden friction or damping values.Source: §C.3 · p. 16

C.4 QUALITATIVE COMPARISON PROTOCOL

  • The qualitative comparison reuses held-out states and identical actions across methods, with matched Gaussian noise seeds across methods but independent noise between branches.For Sliding Door, the key comparison is post-contact motion conditioned on release velocity, since contact duration makes different displacements expected.Source: §C.4 · pp. 16–17

D.1 PHYSICAL MECHANISM INTERPRETER

  • The interpreter estimates a 32-dimensional diagonal-Gaussian mechanism distribution from scene, action, future, and flow-time features; branch evidence updates an initial-scene prior.Weakly informative interventions approach the prior, while more informative ones yield sharper estimates.Source: §D.1 · p. 17

D.2 MECHANISM-LEARNING GROUPS

  • Relational supervision distinguishes same-world groups with shared hidden configurations from mismatched-world groups with matched visible scenes but different physical configurations.The interpreter receives no semantic labels for simulator parameters, so its latent coordinates are not assumed to map one-to-one to named physical properties.Source: §D.2 · p. 18

D.3 NUMERICAL EVALUATION OF SHARED-WORLD EVIDENCE

  • Shared-world evidence is estimated by importance sampling from a proposal covering the prior and branch posteriors, using 64 samples per group and log-space aggregation.Log-sum-exp stabilizes sample aggregation.Source: §D.3 · pp. 18–19

D.4 GRADIENT COMPUTATION

  • The interpreter is frozen during shared-world flow training, but gradients pass through its inputs to adjust video predictions; sampling guidance increases as clean-future estimates become more informative.The schedule applies little guidance early, when those estimates are uncertain.Source: §D.4 · p. 19

D.5 TRAINING SCHEDULE

OneWorld uses a two-stage training schedule and adds a trade-off between shared-world consistency and prediction error as branch count or regularization rises. Guided sampling costs more than the no-guidance path.

D.5 TRAINING SCHEDULE

  • Training first optimizes the interpreter and prior for 20,000 updates with the video backbone frozen, then freezes the interpreter and prior and trains the video model for 12,000 updates.Groups contain three branches by default; inference uses independent Gaussian noise and shared-world guidance over 50 flow steps.Source: §D.5 · pp. 19–20

D.6 SENSITIVITY TO BRANCH COUNT AND HYPERPARAMETERS

  • Increasing branch count from K = 2 to 5 gradually increases the shared-world gap while causing only small M-MSE changes; stronger regularization or guidance eventually increases prediction error.Moderate shared-loss and guidance strengths substantially reduce the gap, while larger importance-sampling budgets improve estimator stability with modest gains beyond M = 64.Source: §D.6 · p. 20

D.7 COMPUTATIONAL COST

  • The no-guidance path has essentially ACWM-DiT’s inference cost, whereas full OneWorld guidance increases latency and memory while retaining practical group-level throughput.The guided path evaluates the interpreter and 64-sample evidence estimator at each guided flow step.Source: §D.7 · p. 20

E IMPLEMENTATION OF SHARED-WORLD EVALUATION

  • The shared-world evaluator converts generated videos into observable state trajectories, then measures compatibility through environment replay and physical-parameter fitting.It is independent of the learned mechanism interpreter.Source: E IMPLEMENTATION OF SHARED-WORLD… · p. 21

E.1 CONTROLLED ENVIRONMENT CONSTRUCTION

The appendix defines controlled multi-intervention groups and a simulator-based evaluator for measuring whether generated branches share a physical explanation. It also checks evaluator-grid sensitivity and interprets the shared-world gap alongside individual and shared physical fit quality.

E.1 CONTROLLED ENVIRONMENT CONSTRUCTION

  • The controlled suite uses reconstructed Pymunk environments for Push Cube and Push Rope and a separate particle-based environment for Pour Water.The cube and rope configurations each use two physical factors with three levels; fluid uses viscosity and cohesion with four levels each.Source: §E.1 · p. 21, Table 9 · p. 21

E.2 SAME-WORLD AND MISMATCHED-WORLD CONSTRUCTION

  • Same-world groups reset every branch to the same initial state and physical configuration, while mismatched-world groups hold the visible initial state fixed but vary physical configuration.The group labels depend on compatibility of action-conditioned outcomes, not on generated videos from the evaluated model.Source: §E.2 · pp. 21–22

E.3 FROZEN STATE EXTRACTORS

  • A separately trained state extractor is frozen before evaluation, and the same extractor is applied to all compared methods.The extractors use simulator-rendered frames paired with ground-truth states and are never fine-tuned on model outputs.Source: §E.3 · p. 22

E.4 ENVIRONMENT-SPECIFIC STATE DISTANCES

  • The evaluator uses environment-specific state distances, with normalization constants fixed before testing and shared across prediction methods.The cube distance accounts for translation and periodic orientation disagreement, while the fluid distance combines occupancy mismatch and centroid displacement.Source: §E.4 · pp. 22–23

E.5 SIMULATOR REPLAY AND PHYSICAL-PARAMETER SEARCH

  • For each branch, the evaluator replays its action sequence from the exact initial simulator state under candidate configurations and compares the resulting trajectory with the extracted prediction.Each branch may select its own best-fitting configuration for individual fit, whereas shared fit requires one configuration for all branches.Source: §E.5 · p. 23, §E.7 · p. 24

E.6 EVALUATOR GRID-DENSITY SENSITIVITY

  • The ACWM-DiT versus OneWorld shared-world-gap ordering persists across 3 × 3, 5 × 5, and 9 × 9 evaluator grids, with only small absolute-gap reductions from densification.The model outputs are fixed across columns, so this sensitivity check changes only evaluator-grid density.Source: §E.6 · p. 23, Table 11 · p. 23

E.7 INDIVIDUAL AND SHARED PHYSICAL FITS

  • The shared-world gap measures the added fitting cost of requiring one physical explanation across branches, and the reported metrics interpret consistency alongside overall physical fit quality.The measure is limited to compatibility within the specified simulator family and candidate parameter space; it does not establish a unique underlying mechanism.Source: §E.7 · p. 24

E.8 GROUND-TRUTH PHYSICAL CONFIGURATION RECOVERY

The section distinguishes joint compatibility from recovery of the physical configuration that generated a world. OneWorld has the strongest reported recovery across the three controlled environments.

E.8 GROUND-TRUTH PHYSICAL CONFIGURATION RECOVERY

  • Ground-truth recovery complements the shared-world gap because joint compatibility alone does not show whether the common fit matches the generating configuration.The evaluation uses simulator ground truth only for scoring and adds no learned evaluator.Source: §E.8 · p. 24
  • OneWorld reaches 87.5% recovery on Push Cube, 87.5% on Push Rope, and 80.0% on Pour Water, for an 85.0% macro-average and gains of 28.1, 25.0, and 30.0 percentage points over ACWM-DiT, respectively.Its macro-average exceeds Posterior Alignment, the strongest alternative, at 69.0%.Source: §E.8 · p. 25, Table 12 · p. 25
  • OneWorld’s recovery advantage remains substantial on Pour Water, where its 80.0% accuracy exceeds Posterior Alignment by 20.0 percentage points.The passage notes that nearby fluid configurations can yield compatible occupancy and centroid trajectories.Source: §E.8 · p. 25
  • Together, the two evaluations indicate that OneWorld’s branches are more mutually compatible and more faithful to the controlled world’s latent physical configuration.The reported recovery average is the macro-average of the three environment-level accuracies.Source: §E.8 · p. 25

E.9 GENERALIZATION TO UNSEEN INTERPOLATED PHYSICS

This protocol tests generalization to intermediate physical values excluded from both mechanism learning and world-model training, while also holding out test initial states. It evaluates joint consistency and closeness to the interpolated physical configuration.

E.9 GENERALIZATION TO UNSEEN INTERPOLATED PHYSICS

  • The interpolation tests use held-out initial states and physical values inside the training range that never appear during mechanism learning or world-model training.Both physical factors are held out jointly: four unseen configurations for Push Cube and Push Rope, and nine for Pour Water.Source: §E.9 · pp. 25–27
  • The protocol uses range-normalized coordinates for physical values, while preserving the simulator-native configuration.Interpolation coordinates are excluded from training and reserved for held-out evaluation.Source: §E.9 · pp. 25–26, Table 13 · p. 26
  • The evaluator includes the held-out interpolation values and measures both cross-branch consistency and normalized physical-parameter error.The candidate set is densified over the same ranges rather than forcing unseen worlds to match discrete training configurations.Source: §E.9 · p. 26
  • The shared-world gap and normalized physical-parameter error should be interpreted together: low values on both indicate compatibility near the actual interpolated physics.A small gap alone can also reflect mutually compatible branches centered on an incorrect configuration.Source: §E.9 · p. 27

E.10 INDEPENDENCE OF TRAINING AND EVALUATION

The learned latent mechanism and the explicit simulator parameters used for evaluation are separate objects. The evaluator does not use OneWorld’s learned compatibility function or intermediate features.

E.10 INDEPENDENCE OF TRAINING AND EVALUATION

  • OneWorld’s latent mechanism W and the evaluator’s explicit simulator parameter vector w are separate objects.The learned interpreter models compatibility among action–outcome branches; its dimensions are not used by the simulator-based evaluator.Source: §E.10 · p. 27
  • The evaluator relies on frozen state extractors, environment replay, and explicit candidate configurations, without accessing OneWorld’s learned compatibility function or intermediate features.Source: §E.10 · p. 27
  • The reported shared-world-gap improvement cannot arise from directly evaluating predictions with the learned compatibility function that supplied the training objective.Source: §E.10 · p. 27

Related papers

This paper cites

Loading 2609.30946v1…