One Model, Two Physical Stories: Auditing Misalignment in Multi-Modal World Modeling

1 Introduction

• A shared backbone can produce individually convincing outputs that tell incompatible physical stories, which matters when world models support planning, counterfactual evaluation, or synthetic data generation.A correct verbal answer alone does not establish that the sampled future obeys the stated physics.Source: §1 · p. 1
• The motivating robot check found 0.448 frame-heights of rise for the Isaac hold, versus 0.004 for the Cosmos hold and 0.006 for the Cosmos slip.The vial remained detected in every frame.Source: §1 · p. 1
• Internal misalignment is video–text disagreement, whereas external misalignment is video–environment disagreement, even when the references predict the same outcome.The paired robot schematic shows text predicting a secure grasp and Isaac lifting the vial while generated video fails to realize the hold/lift.Source: §1 · p. 2, Figure 1 · p. 2
• The A ladder adds the model’s own text-derived magnitude and trajectory to test internal alignment, while the B ladder adds corrected physical information to test external alignment.The ladders progressively supply event, magnitude, and keyframes.Source: §1 · p. 2, Figure 2 · p. 3
• The paper formalizes both alignment tests, builds a four-mechanism intervention testbed, and evaluates claims using canonical scoring of 448 clips, sensor-validity checks, and multi-seed evaluation.Its measurement approach extracts verifiable predictions from text and video and uses simulation measurements as ground truth.Source: §1 · pp. 2–3

Physical evaluation of video generators

• Physical video benchmarks compare generated clips with external references, while this study adds within-checkpoint comparisons and one-parameter tests of directional response.The reported sweep pattern concerns the sampled conditional distribution, not universal model behavior or any single clip’s realism.Source: §2 · p. 3

Understanding–generation gaps

• GapEval measures bidirectional understanding–generation consistency; this study is narrower in model breadth but intervenes on known parameters and compares trajectories with internal and external targets.The A/B internal-target experiment is described as diagnostic rather than an equivalence result.Source: §2 · p. 3

Multi-modal world models

• Prior work frames video models as predictors for understanding and planning, while the paper audits which parameters the two pathways share to support its reasoning-versus-generation framing.The cited examples include Cosmos 3, V-JEPA 2, and related work.Source: §2 · p. 3

Steering and aligning generated physics

• Existing steering approaches can improve or control generated-video physics; this paper’s crossed directives instead test what the text channel controls about event occurrence and magnitude.The passage says event occurrence is measured in both directions, while magnitude is tested on an unmodified checkpoint.Source: §2 · p. 3

3 Physics Fidelity Framework

• A shared contract C = (z, m, u, K, I) aligns event class, measurable quantity, units, keyframes, and scene invariants across video, text, and physical references.The calibrated pixel extractor reports trajectories in object-relative units, making measurements camera-calibration-free.Source: §3 · p. 4
• Both alignment distances use the same measured video contract and change only the reference: Ttext for Dint and Tphys for Dext.The physical trajectory is checked against specified initial conditions and governing dynamics before evaluation.Source: §3 · p. 4
• Mechanism-specific scores test event or magnitude agreement, with clips tolerance-correct within max(0.5 units, 25%) or in the correct incline regime.The paired distances distinguish faithful-and-correct, faithful-but-wrong, unfaithful-but-correct, and unfaithful-and-incorrect outcomes; when Ttext = Tphys, the distances coincide numerically.Source: §3 · p. 4
• Tier 0 supplies scene and parameter, Tier 1 adds the event, and Tiers 2–3 add either the model’s own magnitude and keyframes or corrected physical ones.The A ladder is evaluated by Dint and the B ladder by Dext.Source: §3 · p. 4
• Degenerate clips include vanish, spawn, morph, teleport, or corrupted frame geometry; only measurable clips receive numerical scores, and degeneration is also counted as failure in an unconditional score.The tracker does not gate frame-geometry corruption, which is reported by inspection.Source: §3 · p. 4

4 Experiments

• Cosmos3-Super is a jointly trained Mixture-of-Transformers checkpoint with separate 31.2B understanding and generation streams, 1.6B shared parameters, and two-way joint attention.Text, image–text, and video modes load the same shards but execute largely distinct stream parameters.Source: §4 · pp. 4–5
• The Figure 3 schematic maps a rollout to event, magnitude, units, keyframes, and invariants, then compares the same measured contract with Ttext or Tphys.Those comparisons yield Dint and Dext, respectively.Source: Figure 3 · p. 5
• Vanish, duplicate, morph, teleport, and non-physical-drift failures are rejected rather than assigned numerical scores.Source: Figure 3 · p. 5

• Bounce measures rebound-apex height after a drop, while incline sliding and frictional push measure down-slope travel and stopping distance, respectively.The incline cube slides iff µ < tan θ; the push cube coasts to rest under kinetic friction.Source: Tested Scenarios (see Appendix A.3 for… · p. 5
• The pendulum setup measures swing amplitude and reads period from zero-crossings; the gripper setup measures vial rise when friction permits lifting.The pendulum is a bob on a rigid rod released from θ0, and the grasp condition is µ·grip ≥ mg.Source: Tested Scenarios (see Appendix A.3 for… · p. 5
• Isaac Sim 6.0.1 (PhysX) supplies controllable bounce, incline, projectile, and pendulum scenes, each with one intervened parameter and one pixel-measurable observable.The passage also names an additional setting without specifying it here.Source: Tested Scenarios (see Appendix A.3 for… · p. 5
• A conditioning-frame-locked tracker maps clips to event, object-relative magnitude, displacement, and corruption status, recovering event class on 6/6 lossless simulator feeds.RGB and sensor traces agree framewise (R2=0.989/0.997).Source: Tested Scenarios (see Appendix A.3 for… · p. 5
• Each mechanism and parameter setting is posed identically to text and image+text pathways at temperature 0, while video is generated from the same conditioning frame under a neutral prompt.Video sizes and parameter are stated, but the outcome is not.Source: Tested Scenarios (see Appendix A.3 for… · p. 5
• The tiered protocol shares T0 scene/parameter and T1 event, then separates model-derived A and corrected-physics B references at T2 magnitude and T3 keyframes.The paired final-script A/B arm contains 32 clips at four seeds; because most text and physics targets coincide, it is diagnostic rather than a powered comparison of ladder slopes.Source: Tested Scenarios (see Appendix A.3 for… · p. 6
• Wrong-directive arms hold the physical parameter, conditioning frame, and duration fixed while language falsely denies the event or specifies a false magnitude.These arms test the direction of language control rather than whether the model recalls physics from the parameter.Source: Tested Scenarios (see Appendix A.3 for… · p. 6

5.1 Prerequisite: Is the text reference well defined and correct?

• The text pathway is correct on 22/22 probes, while image+text is strict-correct on 19/22 parsed final answers.Three image+text responses exhaust a 4096-token budget.Source: §5.1 · p. 6

5.2 External and internal alignment under intervention

• At fixed noise, event-wording edits flip outcomes in the same seeds, whereas parameter-number edits are not likely to explain unresponsiveness.Incline flips from 0/8 to 6/8 and pendulum from 7/8 to 3/8 under event-wording changes.Source: §5.2 · p. 6
• Changing guidance leaves one over-amplified pendulum seed swinging at all three tested scales, so the gain persists when a swing is rendered.Its measured amplitude exceeds the physical maximum by about 90%, while bob pixel diameter varies only 16–26%.Source: §5.2 · p. 6

5.3 Can corrected information close misalignment?

• The intervention crosses physical parameters with correct or deliberately wrong language across 192 clips, holding the frame and duration fixed across eight seeds.Correct directives increase from neutral parameter-only through qualitative, magnitude, and keyframed prompts; wrong directives specify a false outcome.Source: §5.3 · p. 6

5.2 External and internal alignment under intervention

• Across 20 neutral-prompt settings, three mechanisms show flat, seed-striped responses; the pendulum swings only at L ≥0.5 and gains a non-physical 1.2–2×.The sweep uses eight seeds per setting, with physical references shown for comparison.Source: §5.2 · p. 6, Figure 4 · p. 7

5.3 Can corrected information close misalignment?

• Language controls binary event occurrence in both directions: keyframing unlocks incline motion in 6/8 seeds, while a wrong pendulum directive suppresses five of eight swings.Neither effect is available to the physical parameter itself; the table distinguishes valid events from magnitudes within tolerance under external and internal scoring.Source: §5.3 · p. 7, Table 2 · p. 8
• Language does not reliably control magnitude: bounce and pendulum directives miss targets, while push travel stays near 10 cube-widths regardless of stated speed.Push travel is about 2.6× the true stopping distance, with only 0–1/8 clips within tolerance; clips may also be degenerate or unscorable.Source: §5.3 · pp. 7–8
• No B-ladder tier is generally monotone in physical fidelity: richer specification more reliably elicits motion than its magnitude, while sometimes increasing degeneration.Incline alone shows a dose–response, reaching the evaluator’s visible ceiling; bounce is non-monotone, pendulum remains over-amplified, and push remains parameter-blind.Source: §5.3 · p. 8

6 Limitations

• The physical reference is Isaac/PhysX simulation, not hardware truth.The evaluation covers one model series, four mechanisms, and eight sweep seeds.Source: §6 · p. 8
• The model scope is limited to one series.The passage identifies four tested mechanisms and eight sweep seeds.Source: §6 · p. 8
• The reported scope does not establish behavior beyond the four mechanisms and eight sweep seeds tested.These are stated as qualifications of the study’s conclusions.Source: §6 · p. 8

7 Conclusion

• Shared contracts over event, magnitude, and timing enable comparison of generated video with both Ttext and Tphys.These comparisons separate internal disagreement from disagreement with the physical environment.Source: §7 · p. 8
• Across four mechanisms and 20 settings, all 22 text probes were correct, but neutral video frequently failed to realize the corresponding dynamics.The intervention ladders altered behavior but did not consistently close the gaps across mechanisms.Source: §7 · p. 8
• The results suggest current unified multimodal backbones may not express one coherent physical world model across modalities.Evaluation should test whether outputs agree with one another and with their intended environment, not only whether each is plausible in isolation.Source: §7 · p. 8

A.1 Architecture and inference audit

• Cosmos3-Super has 64 layers with hidden size 5120.Each layer contains separate understanding and generation streams, each with 31.2B parameters, plus approximately 1.6B shared parameters.Source: §A.1 · p. 10
• The model also includes a 0.6B vision encoder and a 0.7B video VAE.Text and image–text reasoning use temperature 0; video uses UniPC at 30 steps with guidance 6.0 and flow-shift 10.Source: §A.1 · p. 10
• Code inspection verifies causal self-attention over prompt tokens in the understanding stream and generation attention over concatenated understanding/generation keys and values.This establishes a route for information, not where quantitative content is lost.Source: §A.1 · p. 10

A.2 Scene and solver specifications

• The four controlled Isaac Sim twins isolate restitution, Coulomb friction, ballistic motion, and oscillatory timing, with observables expressed in object-diameter units.The scenes use a 60 Hz physics step and the default TGS solver; the originating Franka vial-grasp case is not counted among the four mechanisms.Source: §A.3 · p. 10, §A.2 · p. 10, Table 3 · p. 10

A.3 Test Setting Visualizations

• The test-setting visualizations show Isaac-twin conditioning frames and make each mechanism’s observable pixel-measurable in object-relative units.The four observables are rebound apex, down-slope displacement, stopping distance, and swing amplitude.Source: §A.3 · p. 10, §A.4 · p. 11, Figure 5 · p. 11

A.4 Explicit Validation Example

• For the 25° incline with µ=0.2, Tphys predicts a monotonic downslope trajectory reaching about 15 cube-widths at 0.87 s.The analytical prediction is checked against the reference simulator’s state trajectory.Source: §A.4 · pp. 10–11

A.5 Original pilot and canonical replacement

• The original pilot used four mechanisms, six prompt cells, and two seeds for 48 clips; its ladder suggested language could switch a coarse event without establishing magnitude fidelity.The canonical study broadened the ladder to eight seeds; push rendered a slide at a fixed, parameter-blind magnitude.Source: §A.5 · p. 11

A.6 Evaluator qualification details

• The evaluator gates clips as DYNAMIC, STATIC, or DEGENERATE, treating degeneration as failure in its unconditional score.Its segmentation uses a box declared from the conditioning frame; incline visibility is limited and the push pixel scale is calibrated from release-to-rest displacement.Source: §A.6 · p. 12

A.7 Stochastic realization

• Per-seed success@1 versus oracle best-of-8 is bounce 0.04→0.33, pendulum 0.00→0.00, incline 0.50→0.50, and push 0.05→0.20.The oracle is a diagnostic capacity bound selected using simulator truth, not a deployable procedure.Source: §A.7 · p. 12

A.8 Second checkpoint: Cosmos3-Nano

• Nano changes motion frequency but not parameter sensitivity: push displacement stays about 1.5 cube-widths across speeds, while long-pendulum gain remains 1.2–1.8×.The eight settings were repeated with four seeds; the authors interpret this as a shared failure family, not a scale comparison.Source: §A.8 · p. 12