Source-linked AI summary
MOSH-WM: Mask-Grounded Soft-Hamiltonian Dynamics for Object-Centric World Models
Zhekai Wang, Haoxiang Huang, Xiang Liu, Zhikang Chen, Yueqing Sun, Qi Gu, Shiji Zhou, Miao Liu, Sen Cui
TL;DR
Object-centric world models often evolve visual slots whose dynamics variables do not explicitly represent object geometry while also needing to preserve decoder-relevant appearance and identity. MOSH-WM separates these roles with a mask-grounded phase state and causal visual context, achieving lower long-horizon errors across OBJ3D and CLEVRER than the strongest object-centric baseline.
Problem
Object-centric world models often use unconstrained visual features for dynamics, which can entangle geometry, appearance, identity, and decoder-specific semantics during long closed-loop rollouts.
Method
MOSH-WM derives Q from slot-owned mask support and P from temporal differences, applies soft-Hamiltonian guidance to the phase state, and separately preserves appearance and identity in causal context for decoder-compatible slot reconstruction.
Results
25.0% lower LPIPS and 33.7% lower spatial MSE on OBJ3D, with corresponding reductions of 14.5% and 18.7% on CLEVRER versus the strongest object-centric baseline.
Takeaways & Limitations
The complete model accumulates error more slowly throughout the 30-frame closed-loop rollout, with horizon curves and ablations supporting the combined phase-state and contextual pathway.
Takeaways & Limitations
Results are single validation-selected runs; the state inherits observer errors, energy provides a soft bias rather than identified conserved Hamiltonian dynamics, and evaluation focuses on passive rollout rather than control or planning.
Abstract
from arXiv · showhide
Object-centric world models forecast future videos by evolving a set of entity slots, but the variables receiving dynamics supervision are often unconstrained visual features. We introduce \method{}, a mask-grounded soft-Hamiltonian world model that makes its position-like state explicitly depend on slot-owned image support. A frozen video-slot encoder produces slots and masks; spatial moments of mask-owned support form a canonical state $Q$, temporal differences form $P$, and a learned energy supplies a soft directional bias to a bounded learned increment. Decoder-relevant appearance and identity are stored separately in a causal visual context. A gated composer and bounded residual then combine this context with the propagated phase state to reconstruct decoder-compatible slots. On OBJ3D, given six observed frames and evaluated over the following 30 frames, \method{} reduces LPIPS by 25.0\% and spatial MSE by 33.7\% relative to the strongest object-centric baseline. On CLEVRER, given six observed frames and evaluated over the following ten frames, the corresponding reductions are 14.5\% and 18.7\%. Horizon-resolved visual and object-state measurements show that the complete model accumulates error more slowly throughout the 30-frame closed-loop rollout. Project page:https://github.com/moshwm-anon/-moshwm-anon.github.io.
1 Introduction
MOSH-WM addresses weakly constrained long-horizon slot dynamics by grounding phase variables in object support while retaining separate decoder-relevant visual information. Its soft-Hamiltonian dynamics and contextual pathway target more stable closed-loop predictions.
- Object-centric predictors can entangle geometry, appearance, identity, and decoder semantics, leaving closed-loop rollouts weakly constrained.Strong predictors may match the first future frame yet drift in object layout and appearance over 30 steps.
- A useful physical state should depend on each object’s supported image region while preserving separate visual information for decoding.
- Hamiltonian structure provides coordinates, momenta, and energy, but arbitrary visual slots may not represent object geometry and compact phase states may omit decoder information.MOSH-WM therefore applies Hamiltonian guidance to geometric phase variables while routing appearance and identity through context.
- MOSH-WM constructs position-like Q from owned masks and fixed spatial bases, derives momentum-like P from temporal differences, and excludes appearance-bearing slot features.
- A soft-Hamiltonian module propagates (Q, P) with energy-gradient directions blended with a bounded learned field, while causal context, gated composition, and bounded residuals restore decoder-compatible slots.
2 Related Work
Prior work spans object-centric video representations, Hamiltonian and physics-informed latent dynamics, and broader latent or pixel-space world models. MOSH-WM differs by grounding its phase state in owned image support and using Hamiltonian structure as a soft closed-loop bias.
- Object-centric video world models: Object-centric methods represent scenes as exchangeable entities, with video models extending this interface through temporal correspondence and persistent object slots.
- Hamiltonian and physics-informed dynamics: Hamiltonian and physics-informed models impose geometric or interaction structure through energy, Lagrangian, symplectic, continuous-time, or graph-based dynamics.
- Hamiltonian and physics-informed dynamics: Object-centric physics models estimate latent coordinates and momenta or factorize planner latents before applying physical modules and spatiotemporal reasoning.
- Hamiltonian and physics-informed dynamics: MOSH-WM grounds Q in owned image support, obtains P by finite differences, and uses a soft Hamiltonian field while reserving visual context for decoder compatibility.
- Latent and pixel-space world models: Latent and pixel-space world models learn compact rollouts or future-frame predictions for planning, behavior learning, or visual forecasting.
3 Method
MOSH-WM separates directly propagated mask-grounded phase state from autoregressive visual context. A soft-Hamiltonian transition evolves (Q, P), while gated reconciliation and bounded correction produce slots compatible with the frozen decoder.
- Mask-Grounded Phase-State Factorization: A frozen SAVi interface returns visual slots and masks; MOSH-WM maintains canonical phase state (Q, P) separately from causal visual context C.
- Mask-Grounded Phase-State Factorization: Winner-take-all ownership assigns each pixel to a slot, enabling mask support to separate object geometry from entangled slot appearance.
- Mask-Grounded Phase-State Factorization: Mask-owned pooling over fixed spatial bases constructs position-like Q, while raw temporal differences provide momentum-like P without using appearance-bearing slot vectors.
- Physics–Appearance Reconciliation: A causal spatiotemporal encoder produces C from slot history, and context remains outside the phase-state extractor for the decoder pathway.
- Soft-Hamiltonian Closed-Loop Dynamics: The Soft-Hamiltonian transition blends energy-gradient directions with learned increments, using a soft physical bias rather than strict Hamiltonian dynamics.The update directly propagates (Q, P) during rollout.
- Physics–Appearance Reconciliation: A gated composer combines normalized phase displacement with C, then a causal module supplies bounded decoder-space correction before frozen decoding and context updates.
- Learning Objective: Training freezes the SAVi encoder and decoder, encodes six frames, and unrolls future steps under a composite objective supervising slots, images, Hamiltonian alignment, energy, trajectories, and residuals.
4 Experiments
Experiments test interface fidelity, benchmark performance, long-horizon stability, ablations, and transfer under matched protocols. MOSH-WM improves visual and object-state prediction while accumulating error more slowly, with both phase dynamics and causal context contributing to the gains.
- Experimental design: The experiments evaluate interface fidelity, OBJ3D and CLEVRER prediction, long-horizon feedback, component ablations, and transfer to a different frozen observer.The protocols use shared observation windows, evaluators, and metrics within each dataset setting.
- Interface fidelity: LPIPS changes by only 0.0002 and PSNR by 0.24 dB when phase representations are mapped back to decoder-compatible slots before rollout.This reconstruction test compares the phase-to-slot interface with direct frozen-SAVi reconstruction.
- Benchmark results: 25.0% lower LPIPS, 33.7% lower spatial MSE, 2.43 dB higher PSNR, and 0.8916 SSIM are reported on OBJ3D versus the strongest object-centric baseline.MOSH-WM also outperforms the reported PredRNN baseline on all four visual metrics.
- Benchmark results: 15.4% lower centroid ADE, 9.5% lower FDE, 6.6% lower velocity error, 14.9% lower pairwise-distance error, and 32.2% lower slot MSE are reported on OBJ3D.Image-space and observer-space measurements agree on improved object-state evolution.
- Benchmark results: CLEVRER improves across all five visual metrics over SlotFormer and SAVi-dyn under the shared six-observation, ten-prediction protocol.The matched setup evaluates transfer to collisions, occlusions, and multi-object interactions.
- Long-horizon stability: MOSH-WM shows slower closed-loop error growth, with lower visual, centroid, and pairwise-geometry errors throughout the OBJ3D rollout.Its absolute gap to SlotFormer grows from 0.0159 at Mean@10 to 0.0286 at Mean@30, while final-frame LPIPS is 0.1744 versus 0.2063.
- Ablation study: Removing physics nearly doubles LPIPS and more than triples Slot@30, while removing causal context also causes large visual degradation.Physics-only prediction is stronger than residual-only prediction but remains behind their learned combination.
- Ablation study: The two-block configuration achieves LPIPS/MSE/Slot@30 of 0.0873/4.789/0.02755, outperforming one- and three-block alternatives under the shared budget.The MLP sweep likewise favors the default expansion ratio of two, indicating that the gain is not explained by simply increasing capacity.
5 Discussion and Limitations
The paper reports broad gains for MOSH-WM but bounds their interpretation by run selection, observer dependence, soft rather than identified physics, transfer protocols, and the passive-rollout setting.
- Discussion: MOSH-WM leads in image and observer space on OBJ3D, with a growing margin under feedback and higher error after component removal.The evidence is based on the complete model and component-removal comparisons.
- Limitations: Results are based on single validation-selected runs rather than multi-seed uncertainty estimates.This limits confidence in the stability of reported comparisons across repeated runs.
- Limitations: The state inherits errors from the frozen observer, and the energy provides a soft bias rather than an identified conserved Hamiltonian.The paper distinguishes structural guidance from exact physical identification.
- Limitations: PredRNN lacks aligned slots, while CLEVRER and Physion are fixed-compute transfers with protocol-specific metrics.These evaluation choices constrain direct cross-model and cross-dataset interpretation.
- Limitations: The study evaluates passive rollout rather than control or planning.Control and planning are outside the reported scope.
6 Conclusion
MOSH-WM separates a mask-grounded phase state from decoder-oriented context, using soft Hamiltonian guidance for rollout and causal context to restore appearance. Across OBJ3D and transfer evaluations, the reported evidence supports improved long-horizon prediction and the combined pathway.
- Conclusion: MOSH-WM separates mask-grounded phase state from decoder-oriented context for forecasting.Slot support defines Q, temporal differences define P, and causal context restores decoder-relevant appearance.
- Evidence: 25.0% lower LPIPS and 33.7% lower spatial MSE are reported on 6-to-30 OBJ3D versus the strongest object-centric baseline.Object-state metrics, horizon curves, and ablations agree with these benchmark gains.
- Evidence: CLEVRER transfers favorably, while Physion confirms the combined pathway under a distinct STEVE interface.The conclusion summarizes transfer evidence beyond the primary OBJ3D benchmark.
- Future work: Future work targets phase extraction under occlusion and adds interaction, depth, and action variables.These are the paper’s stated directions for extending the representation.
A Additional Method Details
The additional method details factor dynamics and visual decoding: mask-owned support defines a canonical phase state, while causal context preserves appearance and identity for decoder-compatible reconstruction.
- Mask-Grounded Phase-State Factorization: Mask-owned support and fixed spatial bases form the position-like Q, while temporal finite differences define momentum-like P.Appearance-bearing slot vectors are excluded from the dynamics input.
- Causal Visual Context: A causal spatiotemporal encoder produces context from slot history, reserving it for the decoder-compatible pathway.Context does not enter the phase-state extractor.
- Soft-Hamiltonian Dynamics: The Soft-Hamiltonian transition blends energy-gradient directions with learned increments to provide a soft physical bias.The learned path captures residual dynamics while the gradient path organizes phase evolution.
- Decoder Reconciliation: A gated composer combines normalized phase displacement with causal context, followed by a bounded decoder-space correction.The frozen decoder’s output is appended to the history used for subsequent context updates.
- Training: Training freezes the SAVi encoder and decoder, encodes six frames, and unrolls future steps under a composite objective.The objective combines slot, image, Hamiltonian, energy, centroid, and residual terms, with physical coefficients warmed up early.
B Training Diagnostic
The training diagnostic documents validation-based checkpoint selection and short-budget component sensitivity, with the complete model preferred in capacity sweeps.
- Training Diagnostic: The main OBJ3D training run uses a validation curve for checkpoint selection.Figure 7 reports Validation Slot@30 during the main run.
- Training Diagnostic: Component-sensitivity experiments vary one architectural or functional component at a time under a shared short-budget protocol.Shape-compatible tensors inherit full-model initialization, while changed shapes are reinitialized; results are directional rather than multi-seed confidence intervals.
- Capacity Sensitivity: The complete model is preferred in both capacity sweeps.The two-block configuration balances expressivity and rollout stability, while the default MLP ratio avoids degradation seen at narrower and wider settings.
C Additional OBJ3D Results
An additional OBJ3D visualization evaluates all methods under the same six-observation, 30-frame closed-loop rollout, emphasizing identity retention, relative layout, and error growth under repeated feedback.
- Additional OBJ3D Results: The visualization emphasizes identity retention, relative layout, and the growth of errors under repeated feedback.All methods receive the same six observations and generate the next 30 frames autoregressively.
- Additional OBJ3D Results: The additional strip compares ground truth with three predictors across future horizons under the same closed-loop protocol.Columns represent future horizons, while rows compare the ground-truth sequence with three predictors.
Error and Transfer Summaries
The paper supplements its main results with ablations, transfer visualizations, and an explicit closed-loop evaluation procedure that separates propagated phase state from decoder-preserving context.
- Error and Transfer Summaries: Removing the physical, context, or residual pathway increases relative error in the compact OBJ3D ablation summary.The plot reports relative degradation when each functional pathway is removed.
- Error and Transfer Summaries: Nondecreasing temporal weights make late rollout errors visible during optimization without changing evaluation or reported metrics.The weighting complements energy and centroid constraints, which address phase consistency and mask-summarized spatial support.
- Error and Transfer Summaries: The soft-Hamiltonian update blends a bounded learned proposal with a clipped energy-gradient direction, preserving a learned path when the energy signal is weak.The clipping operator limits occasional high-magnitude gradients without changing ordinary update directions.
- Error and Transfer Summaries: The composer uses phase information to locate expected motion while context preserves decoder-relevant details omitted from the canonical phase state.This post-update separation permits evaluation through different frozen observers with observer-specific decoder interfaces.
- Error and Transfer Summaries: The rollout encodes observations, extracts phase variables, updates them with learned and energy-guided directions, composes decoder-compatible slots, and feeds predictions back.During closed-loop evaluation, predicted frames and slots are fed back without target feedback, exposing accumulated rollout error.
- Error and Transfer Summaries: The Physion transfer experiment changes the upstream observer and treats slot error as the primary measurement because the frozen STEVE decoder limits pixel-quality interpretation.The full model remains below two reduced variants across the listed horizons.
Interpretation of the Transfer Result
The transfer result supports the phase/context factorization under a different frozen observer, while the authors restrict interpretation of absolute reconstruction error because the STEVE decoder is not pixel-optimized.
- Interpretation of the Transfer Result: The full model remains below the two reduced variants throughout the listed Physion horizons.The authors relate this pattern to phase carrying evolving spatial state and context preserving downstream decoder information.
- Interpretation of the Transfer Result: Absolute reconstruction error is not treated as a standalone image-quality claim because the frozen STEVE decoder is not optimized for pixel-perfect reconstruction.The scenario table uses the same locked test protocol rather than constituting a new training set.