PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models

Jonas Kneifl, Jakub Skalski, Bartłomiej Twardowski, Kamil Deja

arXiv:2610.03154v1cs.CVcs.LG

TL;DR

The paper asks whether video DiTs represent physical state or merely reproduce familiar motion, against a background of physical-reasoning deficits in generated videos. It probes three open-weight DiTs against simulator ground truth and finds that activations add information beyond inputs, store it locally around objects, and yield directions that transfer across tested scene changes and steer generated motion.

  • Problem

    Video generations can look plausible while violating physical constraints, leaving open whether DiTs represent physical state or only reproduce familiar motion.

    Source: §1 · p. 2, §4 · p. 9

  • Method

    The authors probe three open-weight video DiTs against simulator ground truth and test readout transfer and steering of generated motion.

    Source: §5 · p. 9, §4 · p. 9

  • Results

    Activations outperform the noised-latent baseline, and probe directions survive tested scene changes and steer motion in every model, though Open-Sora transfers poorly.

    Source: §4 · p. 9

  • Takeaways & Limitations

    The readout direction can be causally active in generation, but in-distribution decodability alone does not distinguish world-model candidates.

    Source: §4 · p. 9

    Decodability does not establish that models represent the laws relating physical states, and the models were not trained specifically for physically correct video.

    Source: §4 · p. 9

Abstract

from arXiv · show

Video generation models produce strikingly realistic sequences and are increasingly proposed as world models, yet recent benchmarks reveal pronounced deficits in their physical reasoning. This raises the question of whether these models internalize physical principles or merely reproduce familiar motion patterns. We address this by probing internal representations of video Diffusion Transformers (DiTs) for simulator-derived ground-truth physical quantities spanning kinematic motion and rigid-body dynamics under gravity and contact. We find that these quantities are linearly decodable with high accuracy early in the denoising process, substantially outperforming a baseline decoded directly from the model's own noised latents, indicating that the relevant physical information is actively constructed during denoising rather than already present in the input. Additionally, we show that activations at on-object tokens carry the relevant physical information and that quantities defined over multiple frames are readable from single latent frames. Hence, information is sharply localized within the token sequence and is computed globally but stored locally. The probes further show partial extrapolation, transferring to scene variations and object configurations outside their training regime, so what they read is not simply a correlate of the scenes they were fit on. When fitted directly in the full-resolution activation space, the probing directions can serve as steering vectors to change the model's output.

1 RELATED WORK

Prior work has evaluated physical plausibility mainly through generated outputs, while a parallel interpretability literature probes representations. This paper’s stated distinction is to test generative video DiTs directly for ground-truth continuous physical quantities.

Physics benchmarks

  • Output-focused benchmarks use perceptual metrics, judgments, tracked ground truth, or symbolic tests, and report plausible-looking generations that can fail closer inspection.Reported model rankings are mixed across evaluations.Source: §1 · p. 3

Physics probing in video models

  • Prior representation studies probe video encoders or plausibility, while some examine DiTs, causal influence, failure mechanisms, or candidate ranking.Examples include Joseph et al. (2026) on encoders, Esmati et al. (2026) on plausibility and causal influence, and Tang et al. (2026) on candidate ranking.Source: §1 · p. 3
  • This work probes generative video DiTs for ground-truth continuous physical quantities, unlike prior work focused on encoders or plausibility readouts.The closest cited work includes Joseph et al. (2026) on video encoders and Esmati et al. (2026), whose quantitative-target experiment used smaller models and simple 2D animations.Source: §1 · p. 3

Steering in video models

  • Video-DiT activation steering has been adapted for safety, motion control, generation quality, and multi-event generation.Examples include supervised safety directions (Kundu et al., 2026), contrastive control, and test-time latent optimization for motion (Xiao et al., 2024).Source: §1 · p. 3

2 SETUP

The study uses simulator-rendered videos with exact physical labels and captures activations from three open-weight video DiTs during denoising. This setup supports probing quantities across controlled motion scenarios and noise levels.

2.1 SYNTHETIC DATA GENERATION

  • Simulator-rendered videos provide exact analytic or simulator-derived annotations for each physical observable.The authors use the Genesis physics simulator because exact physical labels are difficult to annotate in real-world video.Source: §2.1 · p. 3
  • Three scenarios cover rolling kinematics, multi-ball elastic collisions and total momentum, and projectile gravity and launch angle.Each scenario has 200 videos with varied appearances and initial conditions; gravity and launch angle cannot be detected from any single frame.Source: §2.1 · p. 3

2.2 MODELS AND ACTIVATION CAPTURE

  • The study captures activations from Wan 2.1, CogVideoX-1.5, and Open-Sora 2.0, encoding labeled clips and running a denoising pass at selected noise levels.The first ground-truth frame conditions the pass and the prompt is empty, preserving exact labels without mismatched text conditioning.Source: §2.2 · p. 4
  • Video latents are compressed, split into spatio-temporal tokens, and processed by DiT blocks whose attention spans the full sequence.Recorded layer activations and simulator labels form the activation–label pairs used to train probes.Source: §2.2 · p. 4

3 PROBING PHYSICAL OBSERVABLES

The probes test whether physical quantities can be read from DiT activations, where that information is localized, and whether the readout transfers or can steer generated motion. Across the supplied results, activations add information beyond the input, while readouts and steering vary by quantity, model, and setting.

3.1 METHOD

  • Ridge probes use video-level five-fold cross-validation, and ROI and global readouts differ only in which mean-pooled tokens they include.The input baseline concatenates the noised latent and clean conditioning latent; probes are evaluated across layers and noise levels.Source: §3.1 · p. 4

3.2 PHYSICAL QUANTITIES ARE LINEARLY DECODABLE

  • At −40 dB, the matched input baseline is at chance across evaluated quantities while residual-stream activations precisely resolve base kinematics and improve other scenarios.Figure 2 compares ROI activation probes with masked latent references across noise levels.Source: Figure 2 · p. 5, §3.2 · p. 5
  • Gravity and launch-angle probes remain +0.40 to +0.69 above the baseline in every model at the cleanest measured input; approach angle remains +0.47 and +0.30 for Wan and CogVideoX, while Open-Sora scores −0.05.Wan and CogVideoX reach peak performance while inputs remain heavily noised, whereas Open-Sora decodability rises near the diffusion midpoint.Source: §3.2 · pp. 5–6

3.3 SPATIO-TEMPORAL LOCALIZATION

  • An object-centered ROI using roughly 1–2% of tokens outperforms the global readout and the center token; decoy results tie the signal to object location.The ROI exceeds the center token by 0.04–0.09 mean R2, and its benefit saturates around a radius of 1.5 times the object size.Source: §3.3 · pp. 6–7
  • Collision token-frames read their own phase most strongly, but retain cross-phase information: anticipation retains 80%, 57%, and 51%, versus memory’s 67%, 51%, and 10% across Wan, CogVideoX, and Open-Sora.The two phases are nearly independent (r = 0.22), so anticipation is not merely reflecting phase correlation.Source: §3.3 · pp. 6–7
  • Single-frame activations recover more temporal information than a single pixel frame or the clean latent, including gravity at R2 = 0.41–0.81 versus 0.07–0.12 from the clean latent.The authors attribute the additional readability to DiT aggregation across frames, beyond temporal information already introduced by VAE compression.Source: §3.3 · p. 7

3.4 GENERALIZATION ACROSS OBJECTS AND SCENES

  • After recalibration, ROI probes retain high median R2 in Wan and CogVideoX (0.876 and 0.905) but transfer poorly in Open-Sora (0.295); global probes transfer through shape change but collapse under other tested shifts.The global-probe medians are 0.773 for shape, 0.032 for background, 0.095 for perspective, and −0.006 for object count.Source: Figure 4 · p. 7, §3.4 · p. 8

3.5 STEERING

  • The full-resolution rightward probe direction steers all three backbones above the prompt and negative control; the reversed-direction control flips none of the seeds.The steering test fixes left-rolling seeds and intervenes during the high-noise phase; the direction-flip ratio is the fraction that switches rightward.Source: §3.5 · p. 8, Figure 5 · p. 8

4 DISCUSSION AND LIMITATIONS

The DiTs add physical information beyond their inputs, with dynamics readable at high noise and representations computed globally but stored locally. Scene-transfer and steering results support causal activity, while the findings remain bounded by the probes’ scope and setup.

4 DISCUSSION AND LIMITATIONS

  • Across depth, DiT activations outperform the noised-latent baseline, and physical quantities remain readable at high noise.The discussion interprets high-noise readability as indicating that dynamics are fixed before detailed appearance.Source: §4 · p. 9
  • Small object-centered ROIs score higher and transfer better than global readouts, while distant tokens carry little relevant information.Known object position suffices to select a small token region here; real video would require tracking, and multi-object per-object quantities require per-object readouts.Source: §4 · p. 9
  • Probe directions survive scene changes but their scale does not; moving activations along a direction probe changes generated motion in every model.The direction’s gain and offset shift across scenes, so probes do not transfer one-to-one.Source: §4 · p. 9
  • Open-Sora matches Wan and CogVideoX in-distribution at low noise on five of six quantities, but barely transfers and steers least effectively.In-distribution decodability alone does not distinguish world-model candidates; transfer against a target-scene ceiling and steering with the probe’s own direction do.Source: §4 · p. 9
  • Decodability alone does not establish physical understanding, and the study’s scope is rigid-body mechanics in uncluttered, few-object scenes.The study omits dissipation, rotation, deformable or articulated bodies, multi-body contact, and phenomena beyond mechanics.Source: §4 · p. 9
  • Activations come from forward-noised real videos rather than the models’ own sampling trajectories, so the captured states are off-trajectory.Capture requires no inversion.Source: §4 · p. 9
  • Momentum magnitude and kinetic energy correlate at r = 0.85, and both track total mass at r = 0.71 and 0.73, complicating target attribution.The other main targets are uncorrelated within each scenario; targets decodable from clean latents also cannot distinguish denoising from physics becoming decodable.Source: §4 · p. 9
  • High-scoring probes for properties beyond direction are available only from pooled activations, while full-resolution probes fail with limited samples.Uniformly injecting a single vector did not yield functional steering; ROI steering remains a possible approach whose application during generation is open.Source: §4 · p. 9

5 CONCLUSION

The conclusion reports that physical readout directions survive controlled scene changes and can steer generated motion, but Open-Sora transfers poorly despite matching in-distribution. The accompanying statements describe AI use, ethics, and plans for reproducibility.

5 CONCLUSION

  • Physical readout directions survive controlled scene changes and can steer generated motion, but Open-Sora transfers poorly despite matching in-distribution.The authors say Open-Sora’s decodable state alone does not ensure physically consistent generation.Source: §5 · pp. 9–10

AI USE STATEMENT

  • The authors used generative AI across hypothesis and method development, analysis, coding, literature work, and drafting, and report reviewing AI-assisted work.They say code was manually reviewed and tested, and AI-suggested hypotheses, experiments, and interpretations were assessed against results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study uses procedurally rendered simulator scenes and public frozen checkpoints, with no human subjects or personal data.The authors identify steering as dual-use, while limiting their experiments to lateral rolling of one object in synthetic scenes.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors plan to release the rendered videos, simulator labels, and code for reproducing the experiments upon publication.The stated code covers simulation, activation capture, probing, steering, and figure and table export.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 SIMULATION SCENARIOS

The appendix describes three simulator scenarios for probing physical quantities, with scene variants that change one factor at a time for transfer tests. It also specifies scenario-specific visibility and collision constraints.

A.1 SIMULATION SCENARIOS

  • The scenarios use rigid objects with sampled physical and visual properties, while mass is tied to rendered radius so it can be decoded from video.Without that relation, absolute momentum and kinetic energy are unidentifiable from the video; collision and projectile trajectories use analytical rendering because Genesis contacts are effectively inelastic.Source: §A.1 · pp. 13–14
  • Rolling without slipping probes a ball’s constant speed and direction as it moves across a horizontal floor.Initial conditions keep the ball in frame throughout the clip.Source: §A.1 · p. 14
  • The elastic-collision scenario probes the two-ball system’s total momentum and the angle between their velocities before contact.Setups with negligible impulse or balls leaving the frame are rejected.Source: §A.1 · p. 14
  • The projectile scenario probes gravity magnitude and launch angle from a ball’s parabolic flight, with gravity low enough that it does not reach the ground during the clip.The launch has three velocity components, and the trajectory lies in a vertical plane.Source: §A.1 · p. 14
  • Transfer variants change one factor while holding each scenario’s sampler and physics fixed, including background, shape, viewpoint, or object count.Figure A.1 pairs a probe-fit scene on the left with its scored target scene on the right for each shift.Source: §A.1 · p. 14, Figure A.1 · p. 15

A.2 MODELS AND IMPLEMENTATION DETAILS

The appendix details the three models’ capture geometry and the probing protocol, including noise settings, pooling choices, input baselines, and a permutation null. It also specifies how temporal targets and off-object and multi-object readouts are handled.

A.2 MODELS AND IMPLEMENTATION DETAILS

  • The three largest open-weight I2V variants are run at their native resolutions and frame counts, and Table A.2 covers their backbones, checkpoints, and capture geometry.The table caption notes that Open-Sora’s MM-DiT has 19 double-stream plus 38 single-stream blocks, indexed consecutively.Source: §A.2 · p. 15, Table A.2 · p. 15
  • All three models use a causal VAE with 8× spatial and 4× temporal compression, but CogVideoX-1.5 groups latent frames in pairs after padding to 22.Wan 2.1 and Open-Sora 2.0 instead produce 21 token-frames from an 81-frame clip.Source: §A.2 · p. 15
  • Table A.1 identifies sampled scenario parameters, defines U as a uniform draw and T as clip duration, and marks radius and colour as appearance nuisances.It also defines mref and rref as the reference mass and radius of the density law.Source: Table A.1 · p. 16
  • Captures use six shared noise levels from σ = 0.990 to 0.137, reported as −40 to +16 dB; localization is usually read at 0 dB and a fixed mid-depth block.The shared σ grid makes the noise axis comparable across model schedules.Source: §A.2 · p. 16

B.1 PROBING PROTOCOL DETAILS

  • For time-varying targets, spatial pooling retains the latent-frame sequence and each frame is a sample for one shared probe; targets are averaged over the video frames compressed into each latent frame.The probe remains d-dimensional.Source: §B.1 · p. 16
  • Off-object controls mirror the object’s position and either stay pinned or track its mirrored trajectory, while collision readouts concatenate ROIs for approach angle and average them for total momentum.The off-object controls are evaluated only in single-object rolling and projectile scenarios.Source: §B.1 · p. 16
  • The input baseline concatenates the noised video latent with the repeated conditioning latent and pools them using the same variants as the activations.This allows probing the physical signal already present in the model input.Source: §B.1 · p. 17
  • 50 shuffled-label permutations leave R2 at chance across quantities, models, layers, and noise levels, with a mean of −0.056.The probes are refit under the same folds, penalty search, and pooling.Source: §B.1 · p. 17

B.2 ADDITIONAL PROBING RESULTS

Table B.1 reports probe R2 for ROI and global readouts at σ = 0.500, averaged across ten transformer blocks, with each arm’s gap over its own input baseline.

B.2 ADDITIONAL PROBING RESULTS

  • Table B.1 compares ROI and global probe R2 at σ = 0.500, averaging across ten transformer blocks and giving each readout’s gap over its own ztc baseline.Block 0 is excluded; time-varying rows use one ridge across video and token-frame pairs, and per-ball rows average the two balls.Source: Table B.1 · p. 17
  • Time-varying rows report a single ridge over video–token-frame pairs rather than an average of per-frame scores.Per-ball rows are averaged across the two balls, which agree to within 0.03, and have no comparable global cell.Source: Table B.1 · p. 17

B.2.1 DECODABILITY ACROSS DEPTH, NOISE AND ROI SIZE

The depth–noise grid summarizes probe performance across network depth and noise levels, while a disk-radius sweep tests the ROI size used for localized readout.

B.2.1 DECODABILITY ACROSS DEPTH, NOISE AND ROI SIZE

  • Each grid cell is a probe’s R2 at one noise level and transformer block; the plotted line and band give the column mean and min–max.Depth is normalized by L −1 across models with 40, 42, and 57 blocks; each panel samples eleven blocks and includes a matched ztc baseline.Source: Figure B.1 · p. 18
  • The evaluated backbones are Wan 2.1, CogVideoX-1.5, and Open-Sora 2.0.Source: §B.2.1 · p. 18
  • Speed R2 stays flat from 1× to 1.5× the object-token radius, then falls, motivating the 1.5× default ROI.The sweep uses a tracked disk at one token-frame in the rolling closeup-room scenario.Source: Figure B.2 · p. 18

B.2.2 EXTRAPOLATION EXTENDED RESULTS

The extended transfer results distinguish the transfer of a probe’s ordering from its calibration and compare ROI and global readouts across scene shifts. ROI readouts retain stronger cross-scene ordering than global readouts, while unchanged-probe R2 can be poor despite high correlation.

B.2.2 EXTRAPOLATION EXTENDED RESULTS

  • 26 of 57 unchanged ROI-probe evaluations have negative R2, with median R2 0.16 despite median Pearson r = 0.896.The passage attributes the apparent mismatch mainly to scene-shifted gain and offset, which can overwhelm an informative ordering.Source: §B.2.2 · pp. 18–19
  • The table separates unchanged-probe as-is scores, Pearson r for ordering independent of scale, and recalibrated scores after fitting a target-scene gain and offset.Recalibration tests transfer after a scene-specific change of units rather than refitting the probe.Source: §B.2.2 · p. 19, Table B.2 · p. 20
  • Median target-scene Pearson r is 0.896 for ROI probes versus 0.327 for global probes; unchanged global probes are negative in 51/57 cells, compared with 26/57 for ROI probes.The worst global as-is score reaches −2483.Source: §B.2.2 · p. 19
  • The object-swap summary rows cover four quantities because the can’s fixed radius gives the target no variance, making radius unevaluable.Source: Table B.2 · p. 20

C STEERING PROTOCOL DETAILS

The steering protocol uses probe-derived directions to push activations during denoising and evaluates whether left-rolling seeds change course. Full-resolution directions are used because pooled directions do not steer without video degradation.

C STEERING PROTOCOL DETAILS

  • A full-resolution direction reshapes a probe’s weight vector into one vector per token; pooled directions lack token layout and do not steer.Stronger pooled pushes deteriorate or collapse the video before the ball flips, while weak pushes leave the flip ratio at noise level.Source: C STEERING PROTOCOL DETAILS · p. 21
  • All reported steering results use full-resolution directions because stronger pooled pushes add background drift that smears or pans the video before object motion changes.Source: C STEERING PROTOCOL DETAILS · p. 21
  • Each push rotates token activations a fraction α toward their target direction on the unit sphere, at one block during the first quarter of denoising.A full-resolution push uses each token’s own direction row; a pooled push repeats one vector across positions.Source: C STEERING PROTOCOL DETAILS · p. 21
  • Steered generations use a neutral prompt, while the prompt reference differs by specifying that the ball rolls right.Source: C STEERING PROTOCOL DETAILS · p. 21
  • Direction flip ratio is measured on seeds whose unsteered clips roll left, using tracked object masks to classify motion as left, right, or unclassified.Each steered clip is compared with an unsteered clip sharing its seed; the prompt and reversed-push references provide context.Source: C STEERING PROTOCOL DETAILS · pp. 21–22
  • The probe direction exceeds the directed-prompt reference on every backbone, while the reversed push flips none of the 52 seeds to the right.The comparison uses the same known left-rolling seeds for each backbone.Source: C STEERING PROTOCOL DETAILS · p. 22, Table C.1 · p. 22
Loading 2610.03154v1…