PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Object Permanence in World Models

Haotian Zhang, Fengyuan Yu, Dezhi Luo, Haoran Sun, Zehong Zhao, Qingying Gao, Yihan Li, Siyuan An, Huayi Qin, Yilan Zhang, Zhengze Jiang, Pinyuan Feng, Renrui Zhang, Ziyu Guo, Letian Wang, Mengyue Yang, Kangfu Mei, Maijunxian Wang, Ran Ji, Vikash Kumar, Freda Shi, Chandra Sripada, Vincent C. Muller, Philip Torr, Alan Yuille, Nikolaus Kriegeskorte, Felix Juefei-Xu, Lvmin Zhang, Jieneng Chen, Yilun Du, Hokin Deng

arXiv:2609.28654v1cs.AIcs.CV

TL;DR

Video-generation models can make occluded objects reappear impossibly or pass through barriers, raising whether they represent object permanence and solidity, foundational aspects of physical intelligence. WROP provides 150 Blender generators across six task families, a 1.5-million-sample corpus, and a 300-question exam; the authors fine-tune PWM-WROP on the corpus. PWM-WROP ranks first among true-continuation models in a blind pairwise study with 20 raters, competitive with frontier commercial systems despite lower native resolution.

  • Problem

    Video-generation models can make occluded objects reappear impossibly or pass through barriers, raising whether they represent object permanence and solidity, foundational aspects of physical intelligence.

    Source: §1 · p. 2

  • Method

    WROP provides 150 Blender generators across six task families, a 1.5-million-sample corpus, and a 300-question exam; the authors fine-tune PWM-WROP on the corpus.

    Source: §1 · p. 2, §3.2 · p. 6, §4.1 · p. 7

  • Results

    PWM-WROP ranks first among true-continuation models in a blind pairwise study with 20 raters, competitive with frontier commercial systems despite lower native resolution.

    Source: §6 · p. 16

  • Takeaways & Limitations

    The authors present WROP as a foundation for evaluating and training object permanence and solidity in video generation models.

    Source: §1 · pp. 2–3

    Full-reference metrics measure reference-video similarity rather than object permanence or solidity reasoning, and interface differences make them less suitable for ranking models.

    Source: §5.3 · p. 16

Abstract

from arXiv · show

Object permanence and solidity are hallmarks of human cognitive priors. Recent studies show that video generation models, a paradigmatic class of current world models, have begun to show emerged reasoning abilities, making them ideal candidates for building human-like physical intelligence. Do video models have emerged object permanence in them? If not, could we train them with a core-cognition inspired dataset? We introduce WROP (World Reasoning with Object Permanence), a data infrastructure of 150 hand-designed cognitive science inspired tasks, divided into six cognitive categories. We build Blender generators that randomize speed, lighting, camera angle, and other nuisance parameters while preserving each task's cognitive structure, yielding 10,000+ samples per task. We release a 1.5M-sample training corpus and a 300-question exam. On this exam we evaluate 14 video models: 3 reference-to-video, 7 edit, and 4 continuation, among which PWM-WROP, our 16B world model. In a blind pairwise Elo study, PWM-WROP ranks first among continuation models and third overall, behind only a statistical tie between two reference-to-video models. We release the data, exam, model answers, scores, weights, and PWM, our native-PyTorch training stack on AWS Trainium2.

1. Introduction

The paper asks whether video-generation models represent object permanence (OP) and object solidity (OS), foundational physical capabilities, despite failures such as objects reappearing impossibly or passing through barriers. It introduces WROP as a benchmark and training corpus, then reports that fine-tuned PWM-WROP ranks first among continuation models in a blind human study.

1. Introduction

  • The paper frames OP and OS as foundational physical capabilities, while noting video models can make occluded objects reappear impossibly or pass through solid barriers.It argues that failures in these capabilities could be structurally upstream of valid collision, support-removal, and higher-level scene reasoning.Source: §1 · p. 2
  • WROP is a Blender-built 3D benchmark with 150 generators across six task families, a 1.5-million-sample training corpus, and a fixed 300-question exam.Models receive the input half of each clip and generate the target half; human evaluators compare continuations against hand-authored, physically consistent animations.Source: §1 · p. 2
  • PWM-WROP ranks first among continuation models at Elo 1679.5, behind a statistical tie between two commercial reference-to-video systems at 1723.6.The blind pairwise study covered 14 models and 20 raters; PWM-WROP ran at 320×192, below competitors’ 720p and 1080p outputs.Source: §1 · p. 2
  • The paper also provides the training corpus, exam, model answers, scores, model weights, and a native-PyTorch training stack for AWS Trainium2.Its evaluation combines human preference judgments with automatic target-fit metrics and qualitative analyses.Source: §1 · pp. 2–3

2. Related Works

Prior work makes video-generation models plausible candidates for world models, but existing investigations leave gaps in physical-inference targets, V2V evaluation, scale, training splits, and scoring. The paper situates object permanence and solidity in cognitive and philosophical accounts of core knowledge.

Video Models as World Models

  • Prior video-model studies probe tasks such as maze-solving, temporal induction, and abstract rule following, but are mainly 2D, omit video-to-video evaluation, and lack dedicated structured physical-inference benchmarks.The review also identifies small per-task scale, minimal or absent training splits, and reliance on VLM scoring, which the authors consider unreliable for intuitive-physics evaluation.Source: §2 · p. 3

Object Permanence and Solidity: Cognitive and Philosophical Foundations

  • Developmental studies report object permanence from 3.5 months and infant sensitivity to solidity violations within the first half-year; philosophical accounts also treat persistent objects as central to experience.The cited work further connects solidity expectations with support removal and describes them as persisting in adult visual cognition.Source: §2 · pp. 3–4
  • The review presents OP and OS as primitive core-knowledge capacities that provide scaffolding for physical inference and are relevant to human-like world models.It describes WROP’s core-cognition approach as operationalizing cognitive-scientific task paradigms for 3D evaluation and training.Source: §2 · pp. 3–4

3. Dataset

WROP organizes 150 Blender generators into six cognitively grounded families probing object permanence and solidity. Its split clips, varied task structure, and randomized visual properties support training and evaluation across controlled physical-reasoning challenges.

3. Dataset

  • WROP’s six families probe object permanence through dynamic occlusion, static-scene occlusion, and container-based concealment, and solidity through obstruction, support removal, and collision.Figure 2 depicts representative early-input, split-boundary, and target frames for each family.Source: Figure 2 · p. 4, §3.1 · p. 4

3.1. Cognitive Taxonomy

  • Each generator places its key physical event at or after the midpoint; structural parameters vary task difficulty while independent surface variations diversify appearance without changing the physical problem.The task families are adapted from established experimental paradigms where applicable, and otherwise designed for video-generation evaluation.Source: §3.1 · pp. 4–5
  • Object-permanence tasks test re-emergence with identity and motion intact, reinstatement of hidden objects’ number and arrangement, and tracking concealed objects with moving containers.Container tasks require updating location relative to the container rather than retaining the object’s world-origin position.Source: §3.1 · p. 5
  • Object-solidity tasks test whether objects respect aperture size, fall when support is removed, and retain identity and count while following physically consistent collision trajectories.The tasks probe geometric solidity, support-contingent gravity, and contact-mediated solidity, respectively.Source: §3.1 · p. 5

3.2. Data Statistics

  • The corpus contains 1,500,000 samples across 150 generators, with 10,000 per generator, and the exam contains 300 questions, with two per generator.Each sample is a 120-frame hand-authored animation split into 60-frame input and target clips, accompanied by a prompt, object-pose trajectories, and scene-state metadata; motion uses Blender keyframes rather than a physics engine.Source: §3.2 · p. 6

3.3. Data Generation Pipeline

  • Generators are self-contained parameterized Blender scenes with authored objects, geometry, motion, contact events, camera, prompt, and expected outcome; no rigid-body solver produces their trajectories.Scene geometry and trajectories are revised when inspection finds interpenetration or other physical violations.Source: §3.3 · p. 6
  • A shared Blender backend renders 120-frame samples, splits them at frame 60, and packages video, prompt, trajectory, and metadata; recorded seeds vary surface appearance while preserving authored geometry and physical mechanisms.The task is to generate a physically plausible continuation directly, rather than infer an infant’s expectations from a violation-of-expectation presentation.Source: §3.3 · p. 6
  • Automated checks verify sample files, schemas, frame counts, timing, and required trajectory and metadata fields, with failed samples rejected and regenerated.Representative samples from every generator are manually inspected before release for task fidelity and correct split placement.Source: §3.3 · p. 7

4. Evaluation

The evaluation fine-tunes PWM-WROP on WROP’s video-to-video task and compares it with 13 external systems using a shared inference protocol. Blind pairwise human preferences are primary; full-reference metrics provide a secondary measure of target fit.

4.1. PWM-WROP

  • PWM-WROP is the only evaluated model trained on WROP data, fine-tuned for one epoch on 1.5 million samples from Cosmos3-Nano while retaining its architecture and tokenizer.It receives the input clip and prompt without task labels, and predicts the target portion under the same video-to-video contract used in evaluation.Source: §4.1 · p. 7

4.2. Models Evaluated

  • The evaluation compares PWM-WROP with 13 open-weight and proprietary systems, withholding each benchmark item’s target video from all models.The 14 systems span true continuation, reference-to-video, and edit or transfer interfaces, which differ in how they use the conditioning clip and whether they generate later events.Source: §4.2 · pp. 7–8
  • The paper treats interface class as an explicit analysis factor because true continuation, reference-to-video, and edit models engage with the conditioning clip under different assumptions.Edit and transfer outputs occupy the input’s temporal span and cannot depict events that unfold after the occlusion boundary.Source: §4.2 · p. 8

4.3. Inference Protocol

  • A unified harness passes prompts verbatim with conditioning videos, applies no model-specific prompt engineering, and admits outputs after checking that videos decode and have expected geometry.PWM-WROP conditions on the last 57 input frames and predicts 60 frames at 320×192 using a fixed seed and specified sampler settings.Source: §4.3 · p. 8

4.4. Human Judgment

  • Human preference is measured through blind pairwise judgments in which raters assess prompt alignment, natural and plausible motion, and object permanence.Twenty raters completed 476 of 480 scheduled judgments; Bradley–Terry aggregation treats ties as half-wins, and bootstrap resampling over raters yields intervals.Source: §4.4 · p. 9

4.5. Automatic Metrics

  • Full-reference metrics compare generated clips with target videos across pixel, signal-fidelity, perceptual, temporal, and final-frame measures, with clips resampled to a common 320×192 scale.Because reference proximity does not directly measure physical reasoning and can reward reproducing pre-event scenes, these metrics are secondary.Source: §4.5 · p. 9

5. Results

On the WROP exam, PWM-WROP ranked first among continuation models and third overall, behind two reference-to-video leaders. Results varied by task family, and qualitative comparisons exposed failures in object tracking and physical-event continuity; automatic metrics offered only a limited view of reasoning quality.

5.1.1. Overall Results

  • PWM-WROP ranked third overall at 1680 [1604, 1782] and first among continuation models, 224 Elo points ahead of Grok Imagine (video extend) at 1457 [1368, 1555].Wan 3.0 Prime and MiniMax H3 tied for first at 1724 each; the Figure 5 visual summarizes human-preference Elo across the 14 models.Source: §5.1.1 · p. 10, Figure 5 · p. 11
  • The top four systems had nonzero bootstrap top-1 probabilities, while the next five systems spanned only 109 Elo points with overlapping intervals.The analysis suggests interface type explains the leaderboard better than model scale.Source: §5.1.1 · p. 10

5.1.2. Performance by Task Family

  • PWM-WROP ranked first in object-static occlusion, third in two other object-permanence families, and second, fifth, and eighth across the three solidity families.The profile suggests fine-tuning was more effective for occlusion tracking than contact-based dynamics.Source: §5.1.2 · p. 11

5.2. Qualitative Analysis

  • Across six qualitative cases, failures clustered into representation dropout and causal decoupling, including lost or added objects and events disconnected from physical interactions.Examples include objects reappearing inconsistently after occlusion and balls failing to fall when support is removed.Source: §5.2 · p. 15

5.3. Automatic Metrics

  • At 320×192, PWM-WROP had the best LPIPS (0.081) and MS-SSIM (0.921), but full-reference metrics measure target similarity rather than physical reasoning.These metrics can reward scene reproduction without correct event generation, so they are most useful as diagnostics of visual and geometric consistency.Source: §5.3 · pp. 15–16

6. Conclusion

WROP combines a cognitively grounded benchmark and training corpus with a fixed exam, and PWM-WROP ranks highest among true-continuation models in a blind pairwise study. The authors describe the result as preliminary evidence that cognitively principled synthetic data is a viable path toward physical reasoning in video generation models.

6. Conclusion

  • PWM-WROP ranked highest among true-continuation models in a blind pairwise study with 20 raters, despite operating at lower native resolution than frontier commercial systems.The result offers preliminary evidence that training on cognitively principled synthetic data is a viable path toward physical reasoning in video generation models.Source: §6 · p. 16
  • WROP provides a 1.5-million-sample corpus and a fixed 300-question exam, built from 150 hand-designed Blender generators across six cognitively grounded task families.The corpus includes per-sample trajectories and scene-state metadata.Source: §6 · p. 16
  • The release includes the corpus, exam, model answers, scores, weights, and PWM, a native-PyTorch training stack on AWS Trainium2.The authors hope WROP will serve as a foundation for measuring, understanding, and improving physical reasoning in video generation models.Source: §6 · p. 16

A. Task Inventory

The inventory covers 150 generators across six task families, each named with its core physical scenario. The listed scenarios include occlusion, obstruction, object drop, and collision tasks.

A. Task Inventory

  • Table 4 inventories all 150 WROP generators across six task families by generator name and core physical scenario.The listed scenarios show the range of physical events represented in the task collection.Source: Table 4 · p. 20
  • The inventory includes occlusion scenarios such as cup swaps, shell games, hidden-ball transfers, and boxes that open to reveal objects.Other listed scenarios include doors, lids, and covers moving to conceal or reveal objects.Source: A. Task Inventory · p. 21
  • OS-1 obstruction tasks include objects stopped or redirected by walls, gates, beams, and apertures of different sizes.Examples include a vehicle blocked by a low beam and size-dependent passage through a gate.Source: A. Task Inventory · p. 21
  • OS-2 tasks test object drops after support removal or when objects encounter holes, trapdoors, slopes, and moving platforms.The inventory also includes varied mechanisms such as retractable pins, hinged shelves, and released hooks.Source: A. Task Inventory · p. 21
  • OS-3 tasks cover collisions and resulting motion, including bouncing balls, velocity exchange, deflections, and objects toppling or sliding.Scenarios include collisions between balls, balls and blocks, and balls and pendulums.Source: A. Task Inventory · p. 22

B. PWM on Trainium2

The PWM package trains and samples Cosmos3-Nano on Trainium2 using native PyTorch, and engineering changes reduce measured training-step time while preserving loss trajectories. The appendix also reports validation results and constraints on attention geometry and vocabulary-embedding optimization.

B. PWM on Trainium2

  • The released package shards public diffusers-layout weights over a tensor-parallel × FSDP2 device mesh and compiles each Mixture-of-Transformers block once with static shapes.The production 36-layer layout uses tensor parallelism degree 4 and FSDP degree 16 across 64 logical NeuronCores on a single trn2.48xlarge.Source: B. PWM on Trainium2 · p. 23
  • 15.1 s to 5.71 s per training step after successive optimizations on one trn2.48xlarge, with loss trajectories unchanged step for step.The final configuration reaches 12,454 tokens/s at batch size 16 for the specified 288 × 512 × 30 latent-frame geometry.Source: B. PWM on Trainium2 · p. 23
  • At 6.74 s per step, FSDP2 all-gathers and reduce-scatters, the root-unit all-gather, the optimizer, and gradient clipping contribute to time beyond block forward and backward.The collectives are latency-bound, so the authors report that gains come from eliminating collectives rather than reducing their payload.Source: B. PWM on Trainium2 · p. 23
  • A row-concatenation reduce-scatter copy-in variant cuts step time by 2.25×, while multi-tensor AdamW reduces optimizer time from 1,324 ms to 126 ms per step.The AdamW parameters and moments remained bitwise-equal over five steps.Source: B. PWM on Trainium2 · pp. 23–24
  • The vocabulary embedding remains an open optimization, while the implemented flash-attention backward path is limited to sequences of at most 8,192 tokens and excludes 720p geometries.The released default is the compiled scaled-dot-product path, which has not been A/B-tested end to end against the flash-attention path.Source: B. PWM on Trainium2 · p. 24
  • Validation includes bitwise CPU forward parity, 7 × 10^-7 two-layer real-weight parity, bit-exact 64-rank resumption, and overfitting checks on example and physics-rendered clips.For 64 physics-rendered clips, continuation-half cosine was 0.993–0.999 against 0.85–0.995 for the base model.Source: B. PWM on Trainium2 · p. 24

C. Detailed Per-family Human Preference Data and Leaderboards

The per-family leaderboard reports Bradley–Terry Elo and within-family ranks for all 14 models across six WROP task families. The associated figure visualizes these rankings, while the reported intervals caution against treating adjacent ranks as reliably distinct.

C. Detailed Per-family Human Preference Data and Leaderboards

  • The authors treat patterns as most reliable when they recur across multiple families or are corroborated by qualitative analysis.This qualification accompanies the wide bootstrap intervals in the per-family comparisons.Source: C. Detailed Per-family Human… · p. 25
  • Table 6 reports Bradley–Terry Elo and within-family rank for all 14 models across the six WROP task families.Its caption states that bootstrap confidence intervals are wide, especially for OS families with 36–44 total games, so ranks indicate tendencies rather than significant differences.Source: Table 6 · p. 25, C. Detailed Per-family Human… · p. 25
  • Figure 9 plots per-family Elo strengths on a mean-1500 scale, with 95% rater-clustered bootstrap intervals and colors marking interface class.Panels are sorted independently by within-family rank; adjacent ranks are rarely distinguishable, especially in OS families.Source: Figure 9 · p. 26
Loading 2609.28654v1…