PaperlayerView exact revision on arXiv

Source-linked AI summary

World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models

Shinseong Kim, Jaeseok Jeong, Jianlong Fu, Youngjung Uh

arXiv:2610.16365v1cs.AI

TL;DR

Video world models must produce physically plausible futures, but existing approaches leave open where the generator should represent scene evolution. The Physical Registry introduces forecast-initialized, future-supervised slots that co-evolve with video tokens. It achieves the largest gain among compared predictive-state interfaces and improves Physics-IQ Verified scores across evaluated Cosmos and WAN backbones.

  • Problem

    Video models used as world models should generate physically plausible phenomena, while existing control cues may depend on available supervision and predictive features can remain entangled with appearance.

    Source: §1 · p. 1

  • Method

    The Physical Registry uses observation-conditioned JEPA forecasts to initialize dedicated registry slots, which co-evolve with video tokens and receive supervision toward future-frame representations.

    Source: §3.2 · p. 3, §3.3 · p. 4

  • Results

    The Physical Registry with predictive supervision yields the largest gain among compared interfaces, and improves Physics-IQ Verified scores across evaluated backbones; Cosmos 3 Nano 16B reaches 43.56/49.54 on I2V/V2V.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    The results suggest that predictive dynamics can be represented as an explicit, readable state within the video generator.

    Source: §5 · p. 9

Abstract

from arXiv · show

Video world models must generate physically plausible futures, not just realistic frames. Existing approaches typically use semantic, physical, or predictive information to guide the visual generation stream, leaving a fundamental question: where inside the generator should predictions of scene evolution live? Standard video tokens jointly encode appearance and dynamics, but provide no distinct state for learning and reading predictive dynamics. We answer this question with the Physical Registry: temporally grounded registry slots inside a video DiT that are initialized from an observation-conditioned forecast, co-evolve with video tokens, and receive future-representation supervision, making predictive dynamics an explicit, readable state of the generator. In a controlled comparison of predictive-state interfaces on Physics-IQ Verified, Physical Registry improves vanilla fine-tuning by 9.9/8.7 points on I2V/V2V, the largest gain among the interfaces we compare. Across all Cosmos and WAN backbones, Physical Registry improves Physics-IQ Verified scores. With only 0.45% additional parameters, it enables Cosmos 3 Nano 16B to surpass the published I2V score of Cosmos 3 Super 64B (43.56 vs. 42.7). On WorldModelBench, it also improves physical adherence and the total score for all four evaluated backbones. Because the registry becomes candidate-specific while remaining readable, the Physical Registry Selector (PRS) ranks emerging futures by how far their intermediate registry readouts depart from the shared forecast before full denoising is complete. Across four backbones, this selection improves I2V scores by 5.6-15.9% without completed-video external scoring. Together, these results support a broader principle: dynamics prediction should live as an explicit model state that co-evolves with the video representation. Project page: https://grainsack.github.io/physical-registry/

1 INTRODUCTION

Video world models need physically plausible futures, but it remains unclear how to internalize scene evolution when control signals may be unavailable and appearance and dynamics can remain entangled. The Physical Registry uses forecast-initialized, future-supervised slots to represent those dynamics, and is reported to outperform the other compared predictive-representation interfaces.

1 INTRODUCTION

  • Video world models should generate physically plausible phenomena, yet recent video models often fail to do so.The passage names solid mechanics, fluid dynamics, optics, thermodynamics, and magnetism as examples.Source: §1 · p. 1
  • Physical-prior approaches use cues such as motion, force, or physical controls, but depend on the availability of controls and matching supervision.Source: §1 · p. 1
  • The Physical Registry places future-dynamics representations in dedicated slots, initialized from a source-frame-conditioned forecast and supervised toward future-frame representations.The slots and video tokens co-evolve through transformer blocks.Source: §1 · p. 2
  • The registry also provides an internal signal for selecting among samples from different initial noises before videos are fully generated.Source: §1 · p. 2
  • The Physical Registry outperforms the other compared interfaces for predictive representations.The authors report the largest gain among the compared interfaces for a dedicated registry combined with predictive supervision.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses physical plausibility with external controls, representation alignment, joint generation of visual and representation states, and inference-time candidate scoring. The Physical Registry combines predictive representations and register tokens by holding forecast-initialized representations in dedicated, temporally grounded slots and supervising them toward future representations.

Physical plausibility in video generation

  • Physical Registry represents predicted future dynamics inside the generator, unlike prior approaches that express intended evolution through external control signals.Source: §2 · p. 2

Predictive representations inside video generators

  • Prior generative approaches either align pretrained predictive representations with intermediate features or jointly generate visual and representation states; Physical Registry instead holds forecast-initialized representations in dedicated, temporally grounded slots.It supervises these slots toward future representations.Source: §2 · p. 2

Inference-time guidance and candidate selection

  • Unlike external reward models or verifiers, PRS ranks candidates using the generator’s own intermediate registry states.Source: §2 · p. 2

Predictive representations inside video generators

  • During training, forecast-initialized registry tokens co-evolve with video tokens under predictive supervision; at inference, their readouts support PRS.Source: Figure 1 · p. 3

3 METHOD

The method adds forecast-initialized registry slots to a video DiT, co-processes them with video tokens, and trains them toward representations of the realized future. At inference, PRS uses candidate-specific registry readouts to select a trajectory before denoising is complete.

3.1 CONDITIONAL FLOW MATCHING

  • Conditional rectified flow uses a linear interpolation path, velocity target, and objective for video generation.The method augments this formulation with the Physical Registry.Source: §3.1 · p. 3

3.2 PHYSICAL REGISTRY

  • Forecast-initialized registry slots co-evolve with video tokens in shared attention blocks, making their hidden states candidate-specific.A frozen JEPA encoder and predictor derive the forecast from the observed frame; the registry occupies dedicated slots alongside, rather than within, the video latent.Source: §3.2 · pp. 3–4

3.3 PREDICTIVE SUPERVISION OF THE REGISTRY

  • At an intermediate DiT block, predictive supervision aligns registry readouts with representations of the realized future while leaving subsequent blocks for further interaction.The registry begins each denoising step from an observation-only prediction, and the target features come from the corresponding full training video.Source: §3.3 · p. 4

3.4 TRAINING OBJECTIVE

  • The shared DiT is optimized with flow-matching and alignment terms, applying generation training to video tokens and predictive supervision to registry slots.The objective is L = LFM + λalignLalign.Source: §3.4 · p. 4

3.5 THE PHYSICAL REGISTRY SELECTOR

  • PRS ranks candidates by the angular departure of their registry readouts from a shared forecast, then continues denoising only the selected trajectory.Candidates share the forecast initialization; ranking uses a K-step prefix, and the selected trajectory continues for the remaining D −K steps.Source: §3.5 · p. 5, Figure 2 · p. 5

4 EXPERIMENTS

Experiments compare predictive-state interfaces under controlled held-view conditions, then evaluate Physical Registry across video backbones, benchmarks, candidate selection, and registry ablations. Physical Registry improves Physics-IQ Verified and WorldModelBench results, while registry readouts also support early candidate selection.

4.1 EXPERIMENTAL DETAILS

  • The evaluation uses Physics-IQ Verified and WorldModelBench, with Cosmos evaluated for I2V and V2V and WAN evaluated for I2V only.Fine-tuning uses Enlarge videos while excluding Physics-IQ videos; candidate selection compares a ten-candidate pool, with Plain reporting its average and WMReward or PRS selecting one candidate.Source: §4.1 · pp. 5–6

4.2 CONTROLLED COMPARISON OF PREDICTIVE-STATE INTERFACES

  • 51.2/55.1: predictive supervision with a dedicated registry scores best among the compared interfaces, versus +3.7/+3.9 for registry only and +0.5/+0.5 for REPA.The comparison uses center-view training and held-out left/right evaluation.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 FINE-TUNING ON THE ENLARGE DATASET ACROSS VIDEO BACKBONES

  • 43.56/49.54 versus 42.7/48.02: Physical Registry improves Cosmos 3 Nano 16B’s I2V/V2V scores over the published Cosmos 3 Super 64B reference with 0.45% additional parameters.It improves Physics-IQ Verified over every evaluated backbone.Source: §4.3 · p. 6, Table 2 · p. 7
  • Physical Registry improves WorldModelBench physical adherence and total score across all four backbones, while instruction following decreases on Cosmos v2.5 and WAN TI2V-5B.On EWMBench, it also improves Cosmos 3 Nano’s scene consistency and all three motion metrics, with the largest gain in trajectory agreement.Source: §4.3 · p. 6, Table 3 · p. 8

4.4 SELECTION FROM REGISTRY READOUTS

  • 5.6%–15.9%: PRS improves I2V scores over Plain across four default backbone configurations using intermediate registry readouts without completed-video external scoring.On Cosmos v2.5, selection after 10 of 35 steps reaches 38.36/46.68 on I2V/V2V, slightly exceeding WMReward’s 37.74/46.59; continuing the selected trajectory reduces serial denoising by 64%.Source: Figure 4 · p. 8, §4.4 · p. 8

4.5 ABLATION OF THE PHYSICAL REGISTRY

  • 36.01 to 16.78: removing the projected forecast sharply lowers Nano’s I2V score, while moderate Gaussian perturbation has only a small effect.WAN 14B shows the same trend, and replacing the forecast with source-video JEPA features shifts generated motion toward that source while holding initial frame, prompt, and noise fixed.Source: §4.5 · p. 8, Table 4 · p. 9
  • Input-independent learnable registers underperform vanilla fine-tuning; forecast initialization improves over them, and predictive supervision improves further.The ablation separates registry capacity, forecast initialization, and predictive supervision.Source: Table 4 · p. 9, §4.5 · p. 9

4.6 REGISTRY ATTENTION AND ACTIVATION

  • Figure 5 labels video-token queries attending to time-aligned registry slots, while r1 alone denotes that slot’s activation during balloon inflation.Foreground queries are video tokens inside the generated video’s Physics-IQ motion mask.Source: Figure 5 · p. 9

5 DISCUSSION AND CONCLUSION

The Physical Registry gives predictive representations an explicit, temporally grounded state alongside the video representation. The authors conclude that explicit predictive states provide a useful interface for representing and reading physically plausible futures in world models.

5 DISCUSSION AND CONCLUSION

  • The Physical Registry gives predictive representations an explicit, temporally grounded state within the video backbone rather than leaving them entangled with video representations.The authors report that this dedicated state can be explicitly supervised toward future dynamics while remaining readable during inference.Source: §5 · p. 9
  • The authors conclude that explicit predictive states provide a useful interface for representing and reading physically plausible futures in world models.Source: §5 · p. 9
  • The dedicated state can be supervised toward future dynamics while remaining readable during inference.Source: §5 · p. 9

AI USE STATEMENT

The appendices document AI use, registry architecture and supervision, training data, and inference settings. They also describe sampling costs and the expanded-prompt setting.

AI USE STATEMENT

  • Generative AI assisted with wording, literature discovery, routine experiment execution, and preliminary manuscript drafting; authors checked cited papers and reviewed AI-assisted material.Source: AI USE STATEMENT · p. 10

APPENDIX

  • The appendices cover experimental details, additional benchmark results, selector and source-transfer probes, mechanistic diagnostics, and qualitative comparisons.Source: APPENDIX · p. 15

A.1 REGISTRY ARCHITECTURE AND SUPERVISION

  • The registry alignment loss supervises an early-to-middle generation block, while selection reuses a frozen readout and trains no additional selection module.The adapter maps the forecast into the video-latent geometry using spatio-temporal convolutions, per-position temporal attention, resizing, and channel projection.Source: §A.1 · pp. 15–16, Figure 6 · p. 16
  • For Cosmos v2.5 2B, the adapter maps eight forecast slots into eight registry slots at the backbone's 88 × 160 latent resolution.The normalized registry is concatenated with 24 video latent frames, and its slots receive temporally aligned RoPE coordinates.Source: §A.1 · p. 15

A.2 TRAINING AND DATA

  • Enlarge combines physics-oriented, web, simulation, and robot videos; fine-tuning uses RGB videos and captions without physical annotations or simulator states.Physics-IQ videos are excluded from the dataset.Source: §A.2 · p. 16, Table 5 · p. 16, Figure 7 · p. 16

A.3 INFERENCE AND SAMPLING

  • The registry raises per-step sampling cost to 1.67× on Cosmos v2.5 2B and 1.55× on Cosmos 3 Nano 16B.Sampling settings otherwise follow each model's official inference defaults, with matched settings for each backbone pair.Source: §A.3 · p. 17, Table 8 · p. 17, Table 7 · p. 17
  • For the expanded-prompt Nano setting, Claude Opus 5 expands the original description and conditioning image into Cosmos's structured prompt format.The expanded prompts are generated once and reused in every run.Source: §A.3 · pp. 17–18

A.4 EVALUATION AND VISUALIZATION DETAILS

The evaluation spans Physics-IQ, WorldModelBench, and EWMBench, with matched protocols for each backbone pair. It also specifies candidate-pool comparisons and visualization procedures.

A.4 EVALUATION AND VISUALIZATION DETAILS

  • Physics-IQ Verified scores physical plausibility against recorded futures, while WorldModelBench evaluates instruction following, physical adherence, and common sense across 350 prompts.EWMBench evaluates embodied robot episodes using visual consistency, motion correctness, and task-instruction alignment.Source: §A.4 · p. 18
  • Matched pools contain ten candidates per input; Plain averages pool scores, while WMReward and PRS select one candidate.Candidates are compared within the same scene, view, and conditioning input.Source: §A.4 · p. 18
  • In the controlled interface comparison, training uses center views and evaluation uses held-out left and right views with shared protocols.JEPA features from views of the same scenario are closer than features from different scenarios in L2 distance and cosine similarity.Source: §A.4 · p. 19, Figure 8 · p. 19
  • Registry activation measures hidden-state variance over denoising steps 0–10 at each aligned slot and spatial position, averaged across feature channels.Figure 5 attention maps average native QK-softmax responses across selected denoising steps.Source: §A.4 · p. 19
  • Appendix text-attention maps aggregate lexical token groups, while text/video self-attention and registry self-attention are measured over different denoising-step ranges.Text and video self-attention use steps 0–10; registry self-attention uses steps 11–22.Source: §A.4 · p. 19

B EVALUATION CURVES: VANILLA FINE-TUNING VERSUS PHYSICAL REGISTRY

Under matched data and update budgets, Physical Registry consistently exceeds vanilla fine-tuning on WAN TI2V-5B. The appendix also reports Physics-IQ component comparisons and WorldModelBench reference scores.

B EVALUATION CURVES: VANILLA FINE-TUNING VERSUS PHYSICAL REGISTRY

  • The control uses WAN TI2V-5B with the same Enlarge data, schedule, and optimizer, differing from Physical Registry by the absence of registry tokens and predictive alignment.Source: B EVALUATION CURVES: VANILLA… · p. 20
  • +1.98 points, the smallest Physical Registry gain, exceeds vanilla fine-tuning's largest gain of +1.25 across eight evaluation points.Vanilla fine-tuning ends at +0.15 after 8,000 iterations, while Physical Registry exceeds it at every evaluation point.Source: Figure 9 · p. 20, B EVALUATION CURVES: VANILLA… · p. 20

C PHYSICS-IQ COMPONENTS AND ORIGINAL PHYSICS-IQ SCORE

D WORLDMODELBENCH REFERENCE LEADERBOARD

E EMBODIED WORLD MODEL BENCHMARK

On EWMBench, Physical Registry improves scene consistency and motion metrics, with the largest gains in trajectory agreement. The accompanying protocol covers a limited public robot-episode split and compares candidate-selection policies.

E EMBODIED WORLD MODEL BENCHMARK

  • The public ground-truth split covers 21 episodes across seven tasks, with each episode generated three times for 126 videos across both arms.Generated clips are 121 frames at 24 fps, versus 30 fps for recorded episodes.Source: E EMBODIED WORLD MODEL BENCHMARK · p. 22
  • Published EWMBench results cover ten tasks with ten episodes each, whereas this study uses the public 21-episode, seven-task release; EnerVerse-FT and LTX-FT, unlike these models, are robot-domain adapted.Source: E EMBODIED WORLD MODEL BENCHMARK · p. 22
  • Physical Registry improves scene consistency and all three motion metrics on EWMBench; trajectory agreement gains are largest, with NDTW rising 0.073 and DYN 0.076.The table reports higher-is-better metrics on a 0–1 scale.Source: E EMBODIED WORLD MODEL BENCHMARK · p. 22, Table 11 · p. 22

F.1 AGGREGATION AND PROTOCOL

  • Selection comparisons use ten candidates per input; WMReward scores decoded candidates, PRS uses intermediate registry readouts, and Oracle is an upper bound rather than a policy.The protocol also reports Random and Inverse controls against PRS.Source: §F.1 · p. 22, Table 12 · p. 23, Figure 10 · p. 23

F.2 DIRECTION AND CONSISTENCY

The terminal registry readout tracks candidate-specific generated futures, and PRS uses those readouts to select candidates before denoising finishes. Prefix selection and continuation reduce selection cost while preserving useful selection quality.

F.2 DIRECTION AND CONSISTENCY

  • The terminal registry readout has higher mean cosine similarity to the generated-video reference than the shared forecast, 0.974 versus 0.764, and varies across candidates sharing that forecast.Source: §F.2 · p. 23
  • PRS outperforms Inverse and Random in every configuration, remains comparable to completed-video WMReward, and retains positive gains over both controls in pools of four to six candidates.For Cosmos v2.5 I2V at pool size four, departure correlates with Verified quality at Spearman 0.297 [0.209, 0.383].Source: §F.2 · p. 24

F.3 EARLY SELECTION

  • Cosmos v2.5 prefix selection first exceeds WMReward after 8/10 steps; at ten steps it slightly exceeds WMReward on both tasks, with full-schedule decision agreement of 72.7%/56.1% on I2V/V2V.Agreement increases with more denoising steps used for selection.Source: §F.3 · p. 24, Figure 11 · p. 24

F.4 SELECTION COST

  • Measured wall-clock speedups range from 2.84× to 4.06×; Cosmos v2.5 reaches 2.84× against a 2.80× (64%) candidate-step reduction.Table 13 times complete best-of-N and PRS on the same Physical Registry model, with K = 10 as an illustrative prefix length.Source: §F.4 · p. 25, Table 13 · p. 25

G.1 SETUP AND SCOPE

  • The motion-transfer probe holds the receiver image, general prompt, initial noise, and sampler fixed while replacing its native forecast with source-video JEPA features.The transfer input has eight features and shares the trained projection and generation interface with the native-forecast configuration.Source: §G.1 · p. 26, Figure 12 · p. 26

G.2 SOURCE EXAMPLES

Matched-input source-feature replacements shift generated motion toward the source video. The feature visualization also finds a localized response at the alignment-supervised layer.

G.2 SOURCE EXAMPLES

  • Bowling-ball features suppress rebound: maximum upward return falls from 80.1 to 3.6 pixels, compared with three diminishing bounces under the native forecast.The matched outputs use the same conditioning image, prompt, initial noise, and sampler.Source: §G.2 · pp. 26, 28, Figure 13 · p. 27, Table 14 · p. 28
  • Balloon features slow descent: traversing the same height range takes 478 versus 371 ms (1.29×), first contact shifts from 0.375 to 0.500 s, and rebound drops from 40.9 to 12.1 pixels.First-frame silhouette areas differ by only 1.0%.Source: §G.2 · p. 28, Figure 14 · p. 28
  • Changing only registry content changes generated motion, indicating that the generator uses the registry’s predicted dynamics when producing motion.Source: §G.2 · pp. 26, 28

H.1 REGISTRY FEATURE VISUALIZATION

  • At denoising step 16, alignment-supervised layer 14 responds around the balloon and expanding contour, with 1.13× foreground and 1.05× change-region enrichment.Layers 7, 21, and 27 retain broad scene-level activation; layer 14 has the largest measured enrichment among the four layers.Source: §H.1 · p. 29, Figure 15 · p. 29

I.1 ADDITIONAL QUALITATIVE COMPARISONS

Qualitative comparisons show Physical Registry completing intended physical events and avoiding some spurious or duplicated objects. Additional visualizations show registry attention and activation tending to concentrate near changing regions.

I.1 ADDITIONAL QUALITATIVE COMPARISONS

  • On Cosmos v2.5, Physical Registry propagates a domino topple after contact and extinguishes a lit match reaching water; other examples show fewer duplicated objects and cup deformation under a kettlebell.Source: §I.1 · p. 30, Figure 16 · p. 30

I.2 WORLDMODELBENCH QUALITATIVE EXAMPLES

  • WorldModelBench examples show Cosmos 3 Nano 16B generations across robotics, driving, industrial, and natural-scene prompts, with eight frames per prompt.The examples are arranged as storyboards read left to right and top to bottom.Source: §I.2 · p. 30, Figure 19 · p. 32, Figure 20 · p. 33

I.3 ADDITIONAL ATTENTION AND REGISTRY VISUALIZATIONS

  • Across additional Physics-IQ examples, registry attention and activation tend to concentrate more locally around interacting or changing regions, while text cross-attention is often broader.The examples span solid mechanics, fluid dynamics, optics, and magnetism.Source: §I.3 · pp. 30–31, Figure 17 · p. 31, Figure 18 · p. 31

Related papers

Loading 2610.16365v1…