PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?

Anton Hristov, Aleksandar Slavov, Vasil Vasilev

arXiv:2611.01815v1cs.CVcs.LG

TL;DR

Prediction errors are often treated as physical surprise, although they can also reflect appearance, static content, motion, or ordinary prediction difficulty. The paper evaluates representations and native errors separately, finding that physical information can be represented while prediction error remains an unreliable measure of physical surprise.

  • Problem

    Prediction-error scores are used to assess plausibility and unexpected events, but they may reflect nonphysical video properties as well as physical inconsistency.

    Source: §1 · p. 1

  • Method

    The framework separately tests frozen representations, latent geometry, and native prediction errors under controlled changes to actions, appearance, motion, and physical validity.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    V-JEPA2 exposes modest possible-versus-impossible information, but error responses to appearance and static content and reversed Kubric violation ordering show that native error does not reliably measure physical surprise.

    Source: §1 · p. 2, §4.1 · p. 4

  • Takeaways & Limitations

    Recoverable physical information and sensitivity to incorrect actions alone are insufficient to interpret a model’s native prediction error as physical surprise.

    Source: §1 · p. 2

    The comparisons vary architecture, pretraining, capacity, and native score, and do not estimate general limits on planning or anomaly-detection performance.

    Source: §6 · p. 8

Abstract

from arXiv · show

Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty. We present an evaluation framework that separates what frozen representations contain, how they organize appearance and physical differences, and what changes a model's own prediction error. On IntPhys2, a benchmark of physically possible and impossible events, we compare representations from latent prediction, masked pixel reconstruction, and simple visual statistics. Appearance remains easy to decode, while V-JEPA2 exposes modest information about whether events are possible or impossible. Looking at the latent geometry, appearance changes become smaller compared with differences between videos, their directions line up more consistently across scenes, and their overlap with physical changes decreases in deeper layers. Across action-conditioned models, incorrect actions raise prediction error, while appearance changes also produce substantial responses. In Physion++ video clips, repeating a single frame can produce higher error than the original clip, while changing color throughout the clip can lower it. Using Kubric, a simulation and rendering framework, we find that impossible events can produce lower error than possible events. A fixed error threshold also incorrectly flags many possible events as impossible. A model can therefore represent physical information and respond to incorrect actions without its native prediction error reliably measuring physical surprise.

1 Introduction

Prediction error is often used to assess physical plausibility, but its value can reflect physical inconsistency as well as appearance, static content, motion, or ordinary prediction difficulty. The paper separates information in frozen representations from the behavior of models’ native prediction errors.

1 Introduction

  • Prediction errors and related scores are used to assess plausibility, identify unexpected events, and select plausible generated videos [Smith et al., 2019, Piloto et al., 2022, Riochet et al., 2018, Dai et al., 2023, Garrido et al., 2025, Bordes et al., 2025, Maes et al., 2026; Baldassarre et al., 2025, Yuan et al., 2026].Their interpretation matters when scores locate events, compare videos, or reward trajectories.Source: §1 · p. 1
  • A prediction-error score measures difficulty in a chosen representation, so a large value may reflect physical inconsistency, appearance, static content, motion, or ordinary model error.Physical information may be accessible to a classifier even when prediction error responds mainly to other video properties.Source: §1 · p. 1
  • On IntPhys2, appearance labels are readily decoded, while V-JEPA2-G offers a modest advantage for classifying possible versus impossible events.Across controlled changes, incorrect actions raise error, but appearance and static-content changes also alter it; Kubric results show error ordering can track trajectory rather than physical validity.Source: §1 · p. 2

2 Related work

Prior work studies predictive video representations, physical information in latent geometry, and prediction-error responses to physical violations. This paper connects those lines by testing what native scores respond to and whether their interpretation transfers across controlled changes.

2 Related work

  • Predictive video representations arise from latent prediction, masked-pixel reconstruction, and action-conditioned prediction objectives, which yield different representations and native error scales.The paper retains released pipelines and tests score meaning through controlled interventions.Source: §2 · p. 2
  • Prior studies probe physical information and latent geometry; this work asks separately whether measured geometry predicts native-error differences in the same model and setting.Related work includes inverse-dynamics probes [Yeom et al., 2026], layerwise analyses [Joseph et al., 2026, Punzo et al., 2026], and physical-invariance studies [Zeng et al., 2026].Source: §2 · p. 2
  • Physical prediction and violation-of-expectation evaluations compare possible and impossible events, while this paper tests whether a fixed native score responds specifically to violations across motion, appearance, and context changes.The cited evaluation traditions include IntPhys and IntPhys2 [Riochet et al., 2018, Bordes et al., 2025] and Physion and Physion++ [Bear et al., 2021, Tung et al., 2023].Source: §2 · p. 2

3 Evaluation framework

The framework distinguishes evidence about encoded representations from evidence about prediction scores and physical decisions. It evaluates native pipelines while keeping score magnitudes comparable only within each model.

3 Evaluation framework

  • The framework retains each pipeline’s encoders, token selection, normalization, and native distance, and compares score magnitudes only within a model.Context-only prediction and full-clip encoding before context-token selection are treated as distinct computations.Source: §3 · p. 3
  • Readout and geometry assess encoded information, while other checks assess predictive sensitivity and physical decisions.For threshold detection, a lawful clip above threshold is a false alarm and an impossible clip below threshold is a missed violation.Source: §3 · p. 3, Table 1 · p. 3
  • Evaluations span V-JEPA2 variants, released action-conditioned models, IntPhys2, Kubric, Physion++, TwoRoom, and PushT, with thresholds and regression coefficients fitted on separate data.DINO-WM and JEPA-WM share a frozen DINOv2 encoder, so their comparison varies only the predictor; uncertainty is estimated over independent units appropriate to each setting.Source: §3 · p. 3

4 Results

Across the evaluated settings, prediction error does not reliably order events by physical validity or transfer as a fixed detector. The results also show appearance information in representations and layerwise changes in its geometry.

4.1 Physical-violation ordering and threshold transfer

  • Impossible-minus-possible error reverses sign between solid support and a gap, under both clean and persistent-hue appearances; a single threshold reaches 53.9% balanced accuracy but varies sharply by setting.The four adjusted intervals exclude zero; the threshold falsely flags about 69% of lawful clips, including every lawful solid-support continuation.Source: Figure 1 · p. 4, §4.1 · p. 4

4.2 Static-content and persistent-appearance controls

  • On Physion++ Mass, repeating a frame raises mean error relative to the original video, while persistent hue shifts and pixel darkening lower error; target-only effects remain uncertain.Repeating frames changes both content and history, so this comparison does not isolate motion’s causal effect; persistent-versus-target-only hue ordering reverses in Collision.Source: §4.2 · p. 4, Table 2 · p. 5

4.3 Physical-label and appearance readouts

  • On IntPhys2, linear probes readily decode appearance, and V-JEPA2-G has modest physical-label accuracy; its all-appearance physics advantage over each comparator has intervals above zero.The clean-only comparison with PixelSummary remains uncertain, and architecture and training differences prevent isolating the JEPA objective.Source: Table 3 · p. 5, §4.3 · p. 5

4.4 Layerwise representation geometry

  • Appearance shifts shrink relative to between-video distances in deeper layers, align more consistently across scenes, and overlap less with physical changes, though the trends are not uniformly monotonic.V-JEPA2-G’s hue ratio falls from approximately 57% to 8%; final overlap is approximately 6%, above the roughly 0.28% dimension-matched random reference.Source: §4.4 · p. 5

Appendix B.3 gives the complete geometry and source-video retrieval results.

The section tests whether representation geometry helps predict native-error differences and examines how action and appearance changes affect those errors. Its findings distinguish physical information in representations from reliable physical-surprise scoring, while bounding conclusions to the tested models and settings.

Appendix B.3 gives the complete geometry and source-video retrieval results.

  • On 57 IntPhys2 scenes, Figure 2 reports displacement relative to median unrelated-video distance, directional coherence above shuffled correspondences, and the fraction of physical-pair displacement captured by a four-dimensional appearance subspace fitted on other scenes.Source: Figure 2 · p. 6

4.5 Predicting native-error differences from geometry

  • Adding final-encoder geometry changes physical-margin R2 from approximately 0.771 to 0.774, but its adjusted improvement interval includes gains and losses; no tested improvement interval is wholly positive.For appearance-score differences, both models perform worse than the fitting-set-mean reference.Source: §4.5 · p. 6

4.6 Action and appearance sensitivity

  • Across all three action-conditioned models, incorrect actions raise prediction error when the observed outcome is held fixed.Table 4 reports opposite-minus-lawful action contrasts over 256 episodes per task or scorer; native-MSE magnitudes are not comparable across models.Source: §4.6 · p. 6, Table 4 · p. 7
  • Color changes also raise LeWM error at fixed simulator state; at the 60°/0.75 PushT match, hue-induced increases exceed action-mismatch increases for DINO-WM and JEPA-WM.For DINO-WM, the respective increases are 0.04977 versus 0.00307; for JEPA-WM, they are 0.01738 versus 0.00284, relative to each lawful baseline.Source: §4.6 · pp. 6–7, Figure 3 · p. 7

4.7 Task-associated color controls

  • In matched PushT color controls, LeWM scores differ by color despite fixed simulator state; the target-only blue/orange gap is positive in 233 of 256 episodes.Persistent changes reduce the mean gaps but retain both color orderings; target-only score gaps algebraically track target-encoder shifts, while outcome-prediction utility remains open.Source: §4.7 · p. 7

5 Discussion

  • V-JEPA2-G representations expose physical labels at about 58% accuracy against 50% chance, yet native prediction error does not consistently track physical validity; selected geometry adds little predictive value.The evaluation framework distinguishes representation accessibility, geometry, action sensitivity, and error specificity; physical-surprise use calls for violation-ordering and transferred-threshold error checks.Source: §5 · pp. 7–8

6 Limitations

  • The controlled support experiment covers one procedural family with restricted cameras, colors, and lighting; real-world and material generalization are untested, and reward or safety use needs task-specific validation.The IntPhys2 readout and geometry-to-error transfer are post-hoc; their intervals condition on fitted models, and tested linear geometry models establish neither a mechanism nor the effect of removing appearance directions.Source: §6 · p. 8

A Evidence map and interpretation

The appendix maps which inferences each experiment supports and describes the IntPhys2 readout protocol and uncertainty estimates. It also clarifies that the displayed physics-readout figure visualizes existing estimates rather than adding an independent experiment.

A Evidence map and interpretation

  • The evidence map separates representation accessibility, representation geometry, action-conditioned sensitivity, and error specificity, with status labels bounding the inference each evidence block supports.The completed crossed-support and Physion++ follow-ups are included in the map; the geometry-to-error regression is distinct from the structural-geometry criterion.Source: A Evidence map and interpretation · pp. 12–13, Table 6 · p. 12

B.1 IntPhys2 readout protocol and uncertainty

  • The corrected IntPhys2 sealed test contains 114 matched possible/impossible pairs from 57 held-out scenes, with appearance variants and frozen probes evaluated using five-fold scene cross-fitting.The neural pipelines use the same decoded frames and matched relative depths, while Table 7’s checkpoint-family comparison does not isolate training objective or architecture.Source: §B.1 · p. 13, Table 7 · p. 13

B.2 Linear-probe results

  • Physics-readout bootstrap intervals resample the 57 sealed scenes with trained probes held fixed, omitting uncertainty from probe refitting and fold assignment.Figure 4 visualizes the physics-readout estimates reported with the classification tasks in Table 3; it is not an independent confirmation.Source: §B.1 · p. 13, §B.2 · p. 14, Figure 4 · p. 14

B.3 Geometry estimands and complete results

Across representation geometry and controlled prediction-error tests, appearance shifts show structured layerwise behavior, while color, actions, and target difficulty all affect native scores. The analyses probe these effects through exploratory geometry and matched PushT controls.

B.3 Geometry estimands and complete results

  • At block 30, V-JEPA2-G hue ΔMRL was 0.784 [0.781, 0.797], brightness was 0.954 [0.953, 0.961], and recoloring was 0.712 [0.708, 0.720]; final projection remained 0.060 [0.057, 0.060].Projection bases were fit with five-fold scene cross-fitting; k = 4 was retained from development before sealed effects were observed.Source: §B.3 · pp. 14–15

B.4 Exploratory transformation trajectories (run after the sealed test)

  • At final depth, brightness was most ray-like (straightness 0.907), while selective recoloring had the highest effective rank (13.72).For hue, a harmonic curved model had lower held-out relative error than a shared-ray model (0.747 versus 0.798), but three nonzero doses spanning at most 90° cannot test periodicity or a latent color wheel.Source: §B.4 · pp. 15–16, Figure 5 · p. 16
  • At final depth and k = 4, the local appearance-tangent basis captured 0.0668 of held-out physical-displacement energy, versus 0.0416 for the strongest-endpoint basis.The paired tangent-minus-endpoint difference was 0.0252 [0.0072, 0.0450], but the advantage was not universal across depth.Source: §B.4 · p. 16, Figure 5 · p. 16

C.1.1 LeWorldModel, DINO-WM, and JEPA-WM controls

  • Action tests preserve each model’s native pipeline and score; action contrasts use the same target, while hue-versus-opposite-action comparisons use different target baselines.Native score magnitudes are interpreted within each model, not compared across models.Source: §C.1.1 · p. 17, Table 14 · p. 18

C.1.2 Separating target difficulty from intervention effects

  • At the matched PushT condition, mean hue increments exceeded action increments for all three models, but higher moving lawful baselines made DINO-based absolute hue-versus-opposite-action contrasts negative.That cross-target contrast therefore cannot identify which intervention changes the score more.Source: §C.1.2 · pp. 17, 19, Table 15 · p. 18

C.2.1 Question and matched design

  • PushT recolorings compare role colors with magnitude-matched novel colors at zero simulator-state difference; target-only edits leave context, actions, and predictor outputs unchanged.The conditions recolor the block to agent blue or the agent to block gray, each paired with a matched novel color.Source: §C.2.1 · pp. 19–20

C.2.2 Matched contrasts

  • All four preregistered role-color-minus-matched-novel score contrasts were positive in mean, including with substantially matched target RGB change and affected-pixel fraction.Contrasts were evaluated over 256 fresh episodes.Source: §C.2.2 · p. 20, Table 17 · p. 20

C.2.3 Exact latent decomposition

The latent-MSE decomposition locates target-only role-color score gaps primarily in target-encoder color-direction sensitivity, while persistent predictor and target shifts show partial agreement.

C.2.3 Exact latent decomposition

  • Native latent-MSE changes decompose into residual interaction and shift magnitude; target-only role-color score gaps nearly match target-encoder shifts, locating the abrupt effect primarily there.The identity reconstructs intervention scores within 2.38 × 10−7; persistent predictor and target shifts have cosines from 0.71 to 0.93.Source: §C.2.3 · p. 20, Table 18 · p. 20

D.1 Full-population controls

The Physion++ controls show that static-frame and appearance manipulations can change prediction error, with important limits on interpreting those changes. The Kubric experiment crosses support, trajectory, and appearance while keeping appearance changes from altering simulated trajectories.

D.1 Full-population controls

  • In the released full-clip scoring implementation, target-only image changes can affect the predictor even when future pixels are identical.All 16 frames are encoded before context tokens are selected; the authors retained this implementation after seeing the results.Source: §D.1 · p. 21
  • 84.46% of windows scored no higher than their average static baseline, but this is not a physical-violation false-positive rate.The mean original-minus-static residual was −0.028264, with pair 95% interval [−0.028999,−0.027525].Source: §D.1 · p. 21
  • Persistent hue and darkening lowered mean error, while final-four-frame-only changes had uncertain original-relative effects.The persistent-versus-target-only hue ordering reversed in Collision, so the overall mean does not describe every scenario.Source: §D.1 · p. 21

D.3 Motion association and static candidates

  • Raw score correlated with motion (r = 0.596), yet 6 of 10 videos also had high-scoring near-static candidates with nonpositive frozen-frame residuals.The 6/10 finding is diagnostic, not a population false-positive rate or evidence about V-JEPA2-G benchmark accuracy.Source: §D.3 · p. 22

E.1 Population, rendering, and native score

  • The Kubric evaluation crossed 256 configurations with support, lawful or inconsistent stopping or falling, and clean, persistent-hue, or abrupt-hue appearance.Appearance changes did not alter the simulated trajectory; the scorer used 12 context frames and 36 future frames.Source: §E.1 · pp. 22–23

E.2 Estimands, threshold, and audit disposition

The final Kubric evaluation tests whether one unchanged threshold and native prediction error distinguish lawful from inconsistent events across support settings. The ordering reverses by support, and the threshold performs unevenly across those settings; a small exact-rerendering pilot provides exploratory hue-versus-displacement contrasts.

E.2 Estimands, threshold, and audit disposition

  • Clean solid-support inconsistent-minus-lawful error was −0.006734, while clean gap error was +0.006908; persistent-hue margins likewise had opposite signs.All four 98.75% intervals excluded zero in their respective directions, so averaging across support settings conceals the reversal.Source: §E.2 · p. 23, Figure 9 · p. 24
  • With the unchanged development threshold, every lawful solid-support stop was flagged as inconsistent, while performance was substantially better over gaps; the balanced rows average the two worlds.The threshold was fitted on 24 clean development configurations by maximum balanced accuracy, with largest-threshold tie breaking.Source: §E.2 · p. 23, Figure 10 · p. 25

E.3 Kubric exact rerendering

  • The eight-scene exact-rerendering pilot found that both predeclared physical-minus-appearance contrasts had intervals including zero.The contrasts compared 120° hue change with 0.25-diameter displacement, and 180° hue change with 0.50-diameter displacement; they were exploratory.Source: §E.3 · p. 25, Figure 11 · p. 26

F Geometry-to-error regression

The geometry-to-error analysis tests whether feature geometry adds predictive value beyond physical and appearance baselines. Its post-hoc comparisons do not support incremental prediction from geometry, and the fixed-coefficient intervals omit development-fit uncertainty.

F.1 Development-only geometry fit, evaluated post-hoc

  • The physical and appearance models add relative feature distance and an interaction to their respective baselines, with coefficients fit by development-only ordinary least squares.The combined models were evaluated after final outcomes had been inspected, so this is not an untouched confirmation.Source: §F.1 · p. 26
  • Bootstrap intervals keep coefficients fixed and therefore omit development-fit uncertainty; zero-crossing intervals do not establish equivalence.The ten incremental-R2 contrasts use 99.5% intervals.Source: §F.1 · p. 26
  • Across five fixed geometry locations and both tasks, no positive lower bound supports incremental prediction over the baseline.The analysis was post-hoc, and the reported intervals keep development coefficients fixed.Source: Figure 12 · p. 27, Table 23 · p. 27, Table 24 · p. 27
  • The shuffled-geometry R2 distributions are descriptive, not causal or confirmatory randomization tests.No result demonstrates that removing appearance improves native error or that the model lacks physical information.Source: Table 25 · p. 27, §F.1 · p. 28

G Reproducibility resources

The paper provides code, numeric experiment records, and documentation for reproducing its experiments. Large inputs and model assets must be obtained separately.

G Reproducibility resources

  • Code and numeric experiment records are available in the project repository.The listed repository path is the local-launchers-2026-10-31 release.Source: G Reproducibility resources · p. 28
  • The release identifies result directories for representation and action studies, crossed-support experiments, Physion++ controls, and geometry-transfer regression.It also points to documentation for hardware, runtimes, checkpoint identities, execution instructions, and file verification.Source: G Reproducibility resources · p. 28
  • Datasets, model weights, rendered videos, and large feature caches must be obtained separately.Execution instructions specify required input files, software versions, and hardware.Source: G Reproducibility resources · p. 28
Loading 2611.01815v1…