Source-linked AI summary
Foresight at the Event Boundary: Evaluating Physical Prediction in Video World Models
Estela Monserrat Arriaga Santana, Julian Rosas Scull, Ehécatl Sacamch'en Núñez Rico, Hugo Jair Escalante
TL;DR
Existing evaluations of video world models mainly measure reference similarity, physical-law consistency, or plausibility judgments, leaving anticipation of an observed event's consequence indirect. This work evaluates consequence production, timing, and physical realization using event-anchored free-fall recordings and human predictions. Models show distinct failure stages, while human predictions favor the recorded future in aggregate but retain ambiguity among plausible continuations.
Problem
Video evaluations largely measure reference similarity, physical-law consistency, or judgment plausibility, so whether a model initiates an observed event's expected consequence is assessed only indirectly.
Source: §1 · p. 1Method
The paper introduces an event-anchored protocol that separates consequence production, temporal placement, and physical realization, using controlled free-fall recordings and a human prediction study.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Physical foresight is a sequence of distinct challenges: initiating a consequence, anchoring it in time, and realizing its motion.
Source: §7 · p. 8Runway and Veo receive only the final context frame, so their delayed onset may partly reflect missing temporal context rather than model quality alone.
Source: §6.1 · p. 6
Abstract
from arXiv · showhide
Video world models are largely regarded as predictive models of the physical world and are therefore expected to anticipate the consequences of observed events. However, evaluation has mainly focused on reference similarity, physical-law consistency, or judgment plausibility, estimating anticipation only indirectly. We address this directly: when a release or impact has just occurred but its consequence is withheld, can a world model anticipate what should happen next? We introduce an event-anchored evaluation based on 62 controlled real-world free-fall recordings and 124 clips spanning three object types, with fine-grained release and impact annotations and ground-truth trajectories. The protocol separates consequence production, temporal placement, and physical realization. Across six contemporary video generation and world models, Runway and Veo produce release and subsequent impact events at rates above 93% but often initiate them substantially late, whereas Cosmos-Predict-2.5 and MAGI-1 frequently preserve the pre-event state and produce little or no measurable consequence. Among measurable falls, plausible timing does not necessarily imply physically consistent motion. We further conduct a 15-participant, 20-condition human study in which participants describe the expected consequence from a single event-anchored frame and draw its trajectory. Human predictions favor the recorded future in aggregate while revealing genuine ambiguity among plausible continuations. Overall, physical foresight emerges as a sequence of distinct challenges: initiating a consequence, anchoring it in time, and realizing its motion.
1. Introduction
Video world models are expected to continue the consequences of observed physical events, but common evaluations measure this only indirectly. This work tests event consequence, timing, and realization directly, finding distinct failure patterns across models and ambiguity alongside human anticipation.
Motivation and approach
- The evaluation asks whether a model carries an observed event through to its consequence, when the consequence begins, and whether its motion is physically plausible.Existing reference-similarity and physical-constraint evaluations can miss whether the expected consequence is initiated at all.Source: §1 · pp. 1–2
- The study uses controlled real-world free-fall recordings of three objects, annotated at release and impact with measured trajectories, and evaluates six models alongside a human study.The recorded continuation is one possible future; human predictions provide a reference for consequences and trajectories reasonably anticipated from the available evidence.Source: §1 · p. 1
Main findings
- Runway and Veo usually produce release consequences but often start them substantially late; Cosmos-Predict-2.5 and MAGI-1 frequently fail to initiate them, while Cosmos-3 and PhyWorld fall between these patterns.Once measurable free-fall motion is produced, its timing can be close to the real fall even when event onset is misplaced.Source: §1 · pp. 1–2
- Human predictions give the recorded future the strongest aggregate support from minimal evidence while also revealing genuine ambiguity among plausible continuations.The work contributes an annotated resource of 62 recordings and 124 event-anchored clips, plus a matched human study of consequence and trajectory prediction.Source: §1 · p. 2
2. Related Work
Prior evaluations examine generated-video similarity, physical-law consistency, or judgments of physical failures, but do not isolate the response to an observed event. This work adds event-level consequence measurement and open-ended human predictions from the same anchors.
Existing evaluation approaches
- Reference-based evaluation can penalize physically valid alternatives when the initial state does not determine a single future, while law-based benchmarks test physical admissibility without isolating response to an observed event.Existing approaches do not directly measure whether a consequence begins after release or impact, how delayed its onset is, or whether the response is suppressed.Source: §2 · p. 2
Event-level evaluation
- The proposed protocol ends observation at an annotated release or impact and evaluates whether the expected response occurs, when it begins, and whether its motion is physically plausible.It is presented as complementary to established physical evaluations, which do not isolate event-level consequence production.Source: §2 · p. 2
Human reference
- The study compares model continuations with open-ended human predictions from the same event anchors rather than relying on predefined contact judgments.Participants freely describe and draw expected continuations, following prior use of trajectory drawings to probe intuitive physics [2] [16].Source: §2 · p. 2
3. Dataset
The dataset comprises controlled real-world free-fall recordings and paired release- and impact-anchored clips across three objects. It includes event annotations and metric trajectories, alongside generated clips for six evaluated models.
Source recordings and event clips
- The dataset contains 62 real free-fall recordings of a ball, red cube, and spinning top, split into 124 release- and impact-anchored clips.Recordings use two viewpoints under fixed indoor conditions; each clip supplies 33 context frames ending at the anchor, with prompts that withhold the future trajectory and outcome.Source: §3 · p. 2
Annotations
- Real recordings are annotated for object locations and release, impact, and rest events, with calibrated metric trajectories used for physics measurements.Generated videos receive object tracks, event markers, and failure-mode labels including suppressed motion, disappearance, duplication, and outcome substitution.Source: §3 · p. 2
Generated and evaluated clips
- The six models produce 743 videos in total; manual evaluation uses 96 balanced event clips per model, for 576 annotated generated videos.The balanced subset covers 12 object, viewpoint, and anchor-event strata, with eight clips per stratum.Source: §3 · p. 2
4. Proposed Evaluation Protocol
The protocol separates event consequence production, physical realization, and alignment with human expectations. It anchors observation at release or impact, then evaluates generated and human continuations using event, motion, and trajectory criteria.
4. Proposed Evaluation Protocol
- The protocol evaluates whether an observed event leads to its expected consequence, whether the resulting motion is physically plausible, and whether the future aligns with human expectations under minimal visual evidence.Event-level consequence measurement and open-ended human references target aspects not isolated by established physics-based evaluation practice.Source: §4 · pp. 2–3
4.1. Measuring consequence production
- Consequence production credits an expected event whenever it appears in the generated continuation, without requiring accurate timing, trajectory, or physical plausibility.For a release anchor, release, subsequent impact, and rest are recorded independently; release and impact are the substantive tests, while rest has weaker evidential weight because a clip may end before settling. For an impact anchor, the object must react rather than remain frozen; motion is determined from trajectory displacement exceeding half the object's size.Source: §4.1 · p. 3
4.2. Measuring physical realization
- Physical realization is assessed by release-to-impact duration against theoretical free fall, horizontal displacement from release, and estimated acceleration from the tracked vertical trajectory.Measurements apply only when release and impact can be identified; duration allows a one-frame tolerance, horizontal displacement allows 10 px, and acceleration is fit to a constant-acceleration model.Source: §4.2 · p. 3
4.3. Human foresight reference
- Fifteen participants predict outcomes for 20 free-fall conditions from an anchor frame and outcome-blind description, providing free-text predictions, trajectory drawings, and confidence scores.The recorded trajectory is not shown to participants. Human and generated consequences share a classification based on the first displacement exceeding 0.5d0; trajectory comparisons normalize by object size and use discrete Fréchet distance without synchronized timestamps.Source: §4.3 · p. 4, Figure 2 · p. 4
- Human comparisons are visually matched for Runway and Veo, but comparisons with video-conditioned models measure alignment with a minimal-evidence human reference, not a controlled test of context length.Models use their native conditioning: some receive the full 0.55 s event-ending context, Cosmos-Predict-2.5 uses five frames, and Runway and Veo receive only the anchor frame.Source: §4.3 · p. 4
5. Experimental Settings
The evaluation preserves each model’s native conditioning modality while ending all input at the annotated event. Prompts describe the setup and anchor event without revealing the subsequent outcome.
Models and conditioning
- The six evaluated systems include three open-weight models, PhyWorld, Runway Gen-4.5, and Google Veo 3.1.Open-weight models were self-hosted on an NVIDIA A100; Runway and Veo were accessed through Replicate.Source: §5 · p. 5
- Conditioning ends at release or impact, with complete event-ending context for some models, about 0.31 seconds for Cosmos-Predict-2.5, and only the anchor frame for Runway and Veo.The models retain their native conditioning interfaces; no input contains post-anchor information.Source: §5 · p. 5
Prompting
- Prompts describe the setup, camera, and anchor event while withholding subsequent motion, trajectory, and outcome.Auxiliary constraints discourage camera motion and generation artifacts where supported.Source: §5 · p. 5
6. Results
The results separate whether models produce a consequence, when they place it, and whether its motion is physically plausible. Human comparisons show that the recorded future has the greatest aggregate support, while measurable model trajectories vary in production and geometry.
6.1. Does the model produce the consequence?
- After impact, Cosmos-Predict-2.5 and MAGI-1 leave objects frozen in 72.9% and 63.8% of clips, while Veo and Runway replay releases in 17 and 4 clips.These outcomes distinguish response suppression from replay of an earlier causal stage.Source: §6.1 · p. 5
- Runway and Veo produce release and subsequent impact in over 93% of clips, versus roughly one third for Cosmos-3 and PhyWorld; Cosmos-Predict-2.5 never releases and MAGI-1 does so once.For Cosmos-3 and PhyWorld, impact rates closely track release rates, suggesting most failures occur at initiation. Stationary outputs alone do not establish that an object came to rest after moving.Source: §6.1 · p. 5, Table 2 · p. 6
- Median release onset is 1.73 s for Runway and 0.94 s for Veo, versus 0.23 s for Cosmos-3 and 0.22 s for PhyWorld; Runway and Veo’s delays occur mainly before release.Once falling begins, their fall durations are 0.25 s and 0.21 s, close to the real 0.20 s; their anchor-frame-only conditioning may partly explain delayed onset.Source: §6.1 · pp. 5–6, Table 3 · p. 6
6.2. Is the resulting motion physically plausible?
- Across measurable falls, PhyWorld passes the duration criterion in 50% but estimates g=3.36±3.12 m/s2; Cosmos-3’s 8.09±11.94 m/s2 mean coexists with 46% temporal and 8% horizontal pass rates.These measurements apply only when both release and impact can be identified; duration, horizontal deviation, and acceleration capture different properties.Source: §6.2 · p. 6, Table 4 · p. 7
6.3. Do model predictions align with human foresight?
- The recorded future receives mean human support of 0.61 for release and 0.47 for impact, exceeding model maxima of 0.52 and 0.39, respectively.Support varies by stimulus, and some different first-event descriptions may refer to points in a shared physical sequence.Source: Table 5 · p. 7, §6.3 · p. 7
- Cosmos-Predict-2.5 produces no measurable trajectory in 19 evaluable windows, while MAGI-1 does so in 2 of 20; trajectory production is evaluated before geometry.Discarding static outputs would introduce survivorship bias.Source: Table 6 · p. 7, §6.3 · p. 7
- Among measurable release trajectories, Veo is closer to ground truth than Cosmos-3 (D(M, GT)=0.40d0 versus 1.61d0), but these geometry comparisons are conditional on motion.The reported model distances require at least five measurable trajectories.Source: §6.3 · p. 7
7. Conclusion
The evaluation separates consequence production, temporal anchoring, and physical realization, revealing distinct failure stages across models and ambiguity in human predictions.
7. Conclusion
- Some models fail to initiate an observed event’s implied consequence, while others produce it only after substantial delay.Physically plausible timing does not necessarily imply physically consistent motion.Source: §7 · p. 8
- Human predictions from a single event-anchored frame often anticipate the expected consequence, while allowing ambiguity among plausible futures.Source: §7 · p. 8
- Physical prediction should be evaluated as separable capabilities rather than reduced to one similarity or physics score.Source: §7 · p. 8