Source-linked AI summary
Interpreting Physics in Video Encoders using Multi-Object Interaction Videos
Ayushi Mehrotra, Sai Vidyaranya Nuthalapati
TL;DR
Because latent rollouts constrain physical-AI planning, it matters which quantities video encoders represent; behavioral scores alone cannot distinguish a physical quantity from a visible correlate. This paper probes relational collision and friction properties in frozen encoders and finds mixed evidence: some collision distinctions emerge, but other readouts show limited transfer.
Problem
Prior work primarily probes individual-object quantities, while relational physical quantities have not been directly measured in video representations; behavioral scores cannot distinguish physical encoding from visible correlates.
Source: §1 · p. 1, B Extended Related Work · p. 12Method
Results
Takeaways & Limitations
The tests provide evidence of selective relational information, but successful probes do not establish that encoders explicitly represent the underlying physical quantities.
Source: §3.3 · p. 5, §3.4 · p. 6The exploration uses simple synthetic stimuli and only two encoders, and decodability may not reflect downstream use.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Video encoder models are increasingly used as world models for physical AI, where a policy plans by rolling out the latent, so what physics the latent holds bounds what the agent can plan over. Moreover, existing interpretability work probes quantities belonging to a single object; the quantities that make physics relational, such as momentum transfer, restitution, and friction, are defined between bodies and have not been examined. We probe relational physics attributes in two frozen self-supervised video encoders (V-JEPA2 and VideoMAE) under two-body settings and test if they transfer across environments. Our early findings are mixed. Both encoders separate elastic from inelastic collisions reliably. V-JEPA2's own predictor is less likely to predict a collision that creates kinetic energy compared to one that loses it. Momentum-violation detection appears in only one of the two encoders, a kinetic-energy readout survives a change of environment for V-JEPA2 but not for VideoMAE, and neither model holds a friction coefficient that survives a change of geometry.
1 Introduction
The introduction argues that internal evidence is needed to determine whether video encoders capture physical quantities rather than visible correlates, especially for relational physics. Initial tests find reliable elasticity distinctions but mixed evidence for other interaction properties.
1 Introduction
- Behavioral benchmarks cannot identify whether a model encodes a physical quantity or only a visible correlate that works in the evaluated environment.Those explanations come apart when a change of scene, geometry, or embodiment breaks the correlate's link to the quantity.Source: §1 · p. 1
- Prior interpretability work primarily probes individual-object properties, leaving quantities defined between bodies or between a body and a surface unmeasured in video representations.Examples include momentum transfer, restitution, and friction.Source: §1 · pp. 1–2
- The study probes relational collision and friction properties in frozen VideoMAE and V-JEPA2 encoders and tests whether their readouts transfer across scene families.The probes target elasticity, momentum conservation, kinetic-energy change, and friction, with readouts compared against an observer using visible kinematics.Source: §1 · p. 2
- Both encoders distinguish elastic from inelastic collisions, but evidence is mixed across energy prediction, momentum-violation detection, and friction transfer.V-JEPA2's predictor is more surprised by energy-creating than energy-losing collisions, and its kinetic-energy readout transfers to spheres; VideoMAE's readout performs at chance, and neither encoder's friction readout transfers across floor geometries.Source: §1 · p. 2
2 Experimental Setup
The experimental setup probes pooled encoder activations against ground-truth physical quantities from rendered interactions, with controls designed to distinguish representation-based readouts from visible kinematic cues.
2 Experimental Setup
- Videos are rendered in Kubric with exact simulator states recorded per frame, and activations are pooled over nine object-or-frame and temporal-region combinations.The temporal windows are before contact, after contact, and the whole clip; because both encoders attend bidirectionally over the full clip, these windows are pooling choices, not causal isolation.Source: §2 · p. 2
- Ridge probes map a single pooled activation vector to a physical target, using out-of-fold R2 for continuous targets and AUC for binary targets.Each probe is refit with 12 fold seeds, with mean and standard deviation reported.Source: §2 · p. 2
- Every result is paired with a visible-scalar observer and a permutation null that corrects p-values for searching across layers and pools.A probe is informative only insofar as it beats the matched observer.Source: §2 · p. 2
- The experiments cover two-body collisions and friction, testing whether relational quantities are recoverable and whether readouts generalize across scene or floor geometries.Collision experiments address momentum conservation, elasticity, and kinetic energy; friction experiments use a separate cube–floor dataset.Source: §2 · p. 2
3 Two-Body Collisions
Two-body collision tests show that relational physics is unevenly represented: both encoders classify elasticity, but momentum detection and cross-environment transfer vary by model and quantity. The experiments use controlled collisions to distinguish what probes decode from what native predictions reveal.
3 Two-Body Collisions
- Individual-object attributes are highly decodable, but frame-wide pooling preserves sums of object attributes rather than their differences.Whether these attributes appear to emerge with network depth depends on pooling, not the encoder.Source: §3 · p. 3
3.1 Dataset Construction
- Matched collision clips vary momentum conservation and restitution independently, with identical pre-contact trajectories; a separate edit creates energy while conserving momentum.The collision dataset uses two spheres with independently sampled radius and density, and simulator-recorded physical states.Source: §3.1 · p. 3
3 Two-Body Collisions
- VideoMAE detects momentum violations, both encoders distinguish elastic from inelastic collisions, and only V-JEPA2’s kinetic-energy readout transfers across environments; friction does not transfer across geometries.V-JEPA2’s native predictor is systematically more surprised by energy-creating than energy-losing outcomes, while VideoMAE’s reconstruction evidence is inconclusive.Source: §3.2 · p. 4, §3.3 · p. 4, §3.4 · p. 6
3.2 Momentum
- The two-stage norm route detects momentum violations in VideoMAE in both regimes, but V-JEPA2 only marginally in elastic collisions; frame-wide pooling recovers net momentum change better than object-local pooling.Direct binary probes remain near chance, and frame-wide pooling predicts Δp with R2 = 0.426 versus R2 = −0.404 for concatenated object-local pools.Source: §3.2 · p. 4
3.3 Elasticity
- Both encoders classify elasticity reliably, but an observer using velocities projected onto the contact normal reaches AUC 1.000, so classification alone does not establish explicit restitution representation.Frame-wide representations outperform object-local representations throughout.Source: §3.3 · pp. 4–5
3.4 Kinetic Energy
- V-JEPA2’s native predictor finds energy-creating outcomes more surprising than energy-losing ones, while the kinetic-energy readout transfers across environments only for V-JEPA2.VideoMAE’s transfer is at chance; transferred size and speed factors outperform the direct kinetic-energy readout on both models.Source: §3.4 · p. 6
- A friction-coefficient readout does not survive a change of floor geometry, although cube size transfers across the same shift.The two geometries make friction a different function of visible kinematics.Source: Table 2 · p. 7
4 Friction
The friction experiments compare probe readouts from incline and flat-floor videos, finding that friction is less directly decodable than incline angle or acceleration and that friction readouts do not transfer between geometries.
4 Friction
- The study decoded incline angle, down-slope acceleration, and friction from videos of cubes sliding on inclined planes, with a companion dataset of cubes sliding on flat, banded floors.The incline angle and friction coefficient were sampled independently across 150 scenes; acceleration was measured from the cube’s trajectory.Source: §4 · p. 7
- Best θ R² was 0.991 for VideoMAE-base and 1.000 for V-JEPA2-ViT-L; acceleration reached 0.791 ± 0.013 and 0.905 ± 0.006, while direct µ reached 0.512 ± 0.037 and 0.765 ± 0.013.The θ readouts were already 0.957 and 1.000 at patch embedding, suggesting angle was largely accessible from static scene geometry.Source: §4 · pp. 7–8
- Direct and composed friction readouts were similar, but the comparison cannot distinguish direct friction information from recovery through decoded acceleration and incline angle.For VideoMAE-base, direct and composed R² were 0.512 ± 0.037 and 0.498 ± 0.034; for V-JEPA2-ViT-L, they were 0.765 ± 0.013 and 0.759 ± 0.018.Source: §4 · p. 8
- Transfer R² was −0.663 for VideoMAE-base and −0.925 for V-JEPA2-ViT-L incline-to-flat, and −0.167 and −0.553, respectively, in reverse.Probes were trained on one geometry and applied unchanged to the other, with evaluation restricted to the overlapping friction range.Source: §4 · p. 8
- Cube size transferred substantially better than friction, with incline-to-flat correlations of 0.939 and 0.948 and flat-to-incline correlations of 0.440 and 0.826.The control indicates that the geometry shift did not prevent all properties from generalizing.Source: §4 · p. 8
- Across geometries, friction probes predicted acceleration better than friction, consistent with reliance on geometry-dependent kinematic cues rather than a transferable friction readout.This does not establish that the encoders contain no friction information.Source: §4 · p. 8
5 Conclusion
The encoders show mixed evidence for relational physics: some collision properties are decodable, but momentum and friction findings are limited. The study is an early exploration with synthetic, simple stimuli and only two encoders.
5 Conclusion
- Both encoders distinguish elastic from inelastic collisions, but the mass-free label reflects visible speeds rather than establishing kinetic-energy representation.V-JEPA2’s predictor is more surprised by energy-creating than energy-losing collisions; momentum-violation detection appears in only one encoder, and neither retains friction across floor geometry.Source: §5 · p. 8
- The exploration uses synthetic, simple stimuli and two encoders, so it cannot distinguish whether the momentum split reflects training objective or model scale.The energy finding rests on one restitution ratio and one model’s predictor; decodability also does not establish use, since downstream computation may be nonlinear.Source: §5 · p. 8
A Physical Attributes in Multi-Object Interaction Videos
In two-object videos, apparent attribute emergence depends on how representations are pooled. Frame-wide pooling carries a sum of object velocities, while object-local pooling exposes individual attributes early.
A Physical Attributes in Multi-Object Interaction Videos
- Frame-wide pooling makes every attribute curve start at zero and rise through the network’s first third, whereas object-local attributes are already available at the patch embedding.The comparison uses out-of-fold R2 across depth on 150 momentum-conserving clips; the mask-only floors are negligible.Source: Figure 5 · p. 10, A Physical Attributes in Multi-Object… · p. 11
- Per-object attributes are highly decodable from object-specific tokens, with reported R2 values from 0.922 to 0.982 across the listed attributes and encoders.The mask-only floors are negligible, supporting the conclusion that object-local pooling is not carrying these scores.Source: A Physical Attributes in Multi-Object… · p. 10
- The frame-wide pool recovers the symmetric velocity sum strongly (0.955 and 0.938 R2) but the velocity difference only weakly (maxima 0.011 and 0.023).The difference remains unrecoverable at every layer; Table 3 also reports that frame-wide pools read the sum better than any object-local pool reads anything.Source: Figure 6 · p. 11, Table 3 · p. 11
- Momentum transfer decodes from frame-wide pooling (R2 = +0.426) but not from object-local pools (−0.404 concatenated), consistent with its mass-weighted sum over both objects.The authors attribute linear-readout accessibility to pooling geometry rather than to the physics itself.Source: A Physical Attributes in Multi-Object… · p. 12
B Extended Related Work
Prior work evaluates physical behavior and probes individual-object or scene-level properties, leaving relational quantities unmeasured at the representational level. The paper situates its probing protocol among methods that compare representations against matched controls.
B Extended Related Work
- Behavioral benchmarks score physical predictions or preferences, but cannot distinguish a representation of a physical quantity from one tracking a visible correlate [Yi et al., 2020, Bear et al., 2021, Riochet et al., 2020, Upadhyay et al., 2026].The cited work includes CLEVRER, Physion, IntPhys, and WorldBench; Garrido et al. [2025] and V-JEPA 2 [Assran et al., 2025] also report competent behavior.Source: B Extended Related Work · p. 12
- Mechanistic studies probe layerwise physical variables, plausibility, and interventions, including the Physics Emergence Zone reported by Joseph et al. [2026].Other cited work probes frozen features, inverts video-diffusion trajectories, or injects concept activation vectors [Punzo et al., 2026, Esmati et al., 2026, Alam, 2026, Kim et al., 2018].Source: B Extended Related Work · p. 12
- Prior probes target individual-object properties or clip-level plausibility, not quantities defined between objects such as momentum transfer, restitution, and friction.The paper states that these relations cannot be read from either body in isolation and, to its knowledge, had not been measured representationally.Source: B Extended Related Work · p. 12
- The protocol follows matched-control probing: probes are compared with an observer using visible scalars under the same pipeline and folds [Hewitt and Liang, 2019, Belinkov, 2022].The paper uses a visible-scalar observer rather than a randomized control target because the physical relations are often algebraic functions of those scalars.Source: B Extended Related Work · pp. 12–13
C Definitions of the Probed Quantities
The paper defines per-object and between-object targets for probing, with simulator-state measurements under controlled collision conditions. It also treats friction quantities separately on cube-and-floor scenes.
C Definitions of the Probed Quantities
- Collision targets come from recorded simulator state; friction and gravity are absent, contact occurs near frame 20, and pre- and post-contact velocities are sampled at frames 18 and 22.These sampling choices avoid the solver’s interpenetration window while remaining close enough that no other event intervenes.Source: C Definitions of the Probed Quantities · p. 13
- Per-object targets include pre-contact velocity, speed, direction, radius, and log mass, with density independently sampled and signaled by rendered material.On these clips, direction is 0.87 linear in velocity because the speed distribution is narrow.Source: C Definitions of the Probed Quantities · p. 13
- Relational targets include momentum transfer, restitution, kinetic-energy change, and binary conservation, elasticity, and energy-creation labels.Momentum transfer is a mass-weighted sum of velocity changes; restitution is a mass-free ratio of approach and separation speeds along the contact normal, while kinetic energy requires mass.Source: C Definitions of the Probed Quantities · p. 13
- Friction is probed on cube-and-floor scenes using the Coulomb coefficient and incline angle, with down-slope acceleration defined differently for incline and flat-floor motion.On an incline, a = g(sin θ − µ cos θ) with g = 0.45; on a flat floor, a = µg with g = 1.0.Source: C Definitions of the Probed Quantities · pp. 13–14
D Samples from Datasets
Figure 7 presents five two-body regimes, with contact at frame 20 in every clip. Paired clips match through contact, while separate stimuli isolate energy creation from momentum edits.
D Samples from Datasets
- Only the final regime creates kinetic energy while conserving momentum exactly.The figure reports KEpost/KEpre for each row; the elastic momentum-violating twin reaches 2.15.Source: Figure 7 · p. 14
- Paired clips share a seed and crop and remain identical up to contact, diverging only afterward.Contact occurs at frame 20 in every scene, so pre-contact frames do not distinguish pair members.Source: Figure 7 · p. 14
- Momentum edits also shift kinetic energy, so the energy experiment uses separate stimuli.The elastic momentum-violating twin has a KEpost/KEpre ratio of 2.15; green, red, and purple borders mark conserving, violating, and physically impossible cases.Source: Figure 7 · p. 14