Source-linked AI summary
CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models
León Begiristain, Olaf Dünkel, Adam Kortylewski
TL;DR
It remains unclear whether video models learn causal physical structure or exploit superficial visual correlations. CRONOS benchmarks this through controlled visual interventions and finds substantial inconsistencies, especially for viewpoint and object-type changes.
Problem
It remains unclear whether video prediction models learn causal representations that support stable predictions under controlled visual changes rather than relying on superficial correlations.
Method
CRONOS generates photorealistic videos of fixed physical events while systematically intervening on viewpoint, scene, object category, and object appearance.
Results
Models often fail to generate physically consistent videos and are especially sensitive to viewpoint and object-type changes; the strongest model achieves only 0.22 overall success rate.
Takeaways & Limitations
CRONOS provides a controlled testbed for diagnosing counterfactual physical consistency and guiding more robust, structured video predictions.
Takeaways & Limitations
Because CRONOS uses synthetic Unreal Engine renderings, its results diagnose controlled physical prediction rather than directly estimating real-video performance.
Abstract
from arXiv · showhide
Video prediction is increasingly viewed as a path toward generalizable world models, yet it remains unclear whether these systems learn underlying causal structure or merely exploit superficial visual correlations for future prediction. We introduce CRONOS, an intervention-based benchmark designed to evaluate counterfactual physical consistency: whether a model's predictions of physical events respond appropriately to controlled changes in the visual input, such as variations of scene context, viewpoint, object appearance, and object category. Built in a photorealistic Unreal Engine environment, CRONOS enables controlled, high-fidelity generation of videos across diverse scenes and dynamics. In contrast to previous benchmarks, CRONOS systematically intervenes on four key factors - viewpoint, scene, object category, and object appearance - while keeping the underlying physical event type, such as a collision, occlusion, or fall, fixed. Our evaluation of recent open-source video generators reveals substantial failures in counterfactual physical consistency: prediction quality for the same physical event type is affected by appearance, environment, and, particularly by viewpoint changes. CRONOS provides a controlled and reproducible testbed for diagnosing how the quality of generated videos changes for different interventions, establishing a concrete target for developing models that perform consistently across changes of multiple conditions. The dataset and code are available at our project page.
1 Introduction
CRONOS addresses whether video models learn physical structure or exploit superficial visual correlations by testing predictions under controlled changes to visual input. It provides a photorealistic, intervention-based benchmark for evaluating counterfactual physical consistency across physical events and visual conditions.
- Motivation and contribution: CRONOS evaluates whether predicted physical events remain coherent when the visual input is changed in controlled ways.The benchmark targets counterfactual physical consistency rather than perceptual quality alone.
- Motivation and contribution: The benchmark uses photorealistic Unreal Engine sequences while keeping the underlying physical event type fixed across interventions.This controlled construction enables fine-grained analysis of how predictions respond to changes in the visual world.
- Benchmark design: 675 videos span 3 events, 5 scenes, 5 object categories, up to 4 viewpoints, and 3 appearances.Viewpoint variation is omitted for occlusion to preserve the visibility structure.
- Evaluation: Object-centric metrics disentangle 3D motion from appearance, while intervention sensitivity diagnoses counterfactual consistency in I2V and V2V settings.The metrics are applied to several state-of-the-art open-source video generation models.
- Findings: Models often fail to generate physically consistent videos, with especially high sensitivity to viewpoint and object type changes.Video conditioning improves over image conditioning, whereas scaling model size does not necessarily improve consistency.
2 Related Work
Related work spans video generation architectures, evaluation methods for visual quality and physical realism, and synthetic environments for probing physical understanding. Existing simulators and benchmarks motivate controlled tests of physical-event prediction and generalization, while CRONOS varies counterfactual visual factors across fixed events.
- Video generation models: Video generation models evolved from temporally augmented latent diffusion architectures toward transformer-based diffusion architectures for conditioning on text, images, video frames, or combinations.
- Evaluating video generation: Video-generation evaluation began with image-based quality metrics such as FVD and expanded to benchmarks assessing physical commonsense, laws, and scientific concepts.
- Simulators for probing visual understanding: Synthetic video-reasoning benchmarks use controlled environments to evaluate understanding of simple scenes, while IntPhys focuses on detecting violations of physics.
- Simulators for probing visual understanding: CRONOS renders collision, fall, and occlusion events under counterfactual changes to scene context, camera viewpoint, object category, and object appearance.
- Simulators for probing visual understanding: Physion, PhysWorld, PISA, and WorldBench study physical-event prediction, visual-property generalization, physics modeling, or physical understanding using diverse or synthetic environments.
3 CRONOS Benchmark
CRONOS evaluates video generators through controlled counterfactual experiments in Unreal Engine, rendering fixed physical events under systematic interventions to visual and physical conditions. It uses three event types, a full-factorial intervention design with a viewpoint exception for occlusion, and metrics assessing per-video quality and intervention sensitivity.
- Controlled setup: CRONOS defines each experiment around a physical event specified by initial states, impulses, and simulator parameters, then varies one factor while preserving the underlying event.The controlled factors are camera viewpoint, scene, object appearance, and object category.
- Physical events: CRONOS includes fall, collision, and occlusion events that respectively test changing contact and free-fall motion, interaction dynamics, and long-range inference through hidden motion.All three scenarios begin from standardized initial conditions in which an impulse initiates object motion.
- Interventions: Interventions change scene layout, viewpoint, appearance, or object category to test contextual reliability, perspective consistency, appearance–dynamics disentanglement, and coherent adaptation to altered physical properties.Object-category changes can affect shape, material, mass, and friction, whereas appearance changes leave physical parameters unchanged.
- Evaluation: The dataset uses a full-factorial design for each event, except that viewpoint is fixed for occlusion to preserve visibility, while evaluation combines per-video quality metrics with intervention sensitivity.Sensitivity is the average deviation between best and worst performance across groups and metrics, with lower values indicating greater counterfactual consistency.
4 Results
CRONOS evaluates open-source video generators with validated metrics and finds broad failures in physical event generation and counterfactual consistency. Video conditioning improves several stability measures, while increasing Cosmos model size does not improve physical consistency.
- Experimental setup: The evaluation covers Cosmos2.5, CogVideoX1.5, MAGI-1, and Wan2.2 in I2V, with Cosmos and MAGI-1 also evaluated in V2V using five conditioning frames.Cosmos2.5 is tested at 2B and 14B parameters.
- Metric validation: Positive Pearson correlations between metric performance and human ratings support using the proposed metrics for subsequent benchmark analysis.The study collected median-aggregated ratings from three annotators for 540 videos on a 1–5 quality scale.
- Physical event generation: 22% is the strongest model success rate, while most models remain below 15%, showing that open-source models fail to reliably generate short clips of basic rigid-body physics.A video succeeds only when every per-video metric passes its calibrated threshold.
- Counterfactual consistency: Models show substantial sensitivity to interventions, with viewpoint and object-category changes producing especially large quality variations.Appearance changes are tolerated best, yet even the most robust models vary by around 20% under this perturbation.
- Effect of video conditioning: Video conditioning generally improves motion fidelity, background stability, and object stability for Cosmos and MAGI-1 compared with I2V.The additional frames provide temporal information about motion direction and magnitude.
- Effect of model size: Scaling Cosmos2.5 from 2B to 14B parameters yields no improvement in physical event generation and can reduce performance across nearly every metric.In V2V, the success rate drops from 22% to 14%.
5 Limitations
CRONOS’s limitations include a synthetic-to-real domain gap, reliance on single-reference rollouts, and evaluation restricted to open-source models. Even the strongest evaluated model achieves only 22% success, indicating the benchmark remains far from saturated.
- Synthetic-to-real domain gap: Unreal Engine renderings introduce a synthetic-to-real domain gap, so results diagnose controlled physical prediction rather than directly estimating real-video performance.The controlled rendering setup enables matched counterfactual interventions and is common in synthetic physics benchmarks.
- Single-reference rollouts: Most metrics compare generations with one rendered reference despite multiple plausible futures, motivating multi-seed evaluation and future distribution-based assessment.CRONOS also uses detailed text prompts and reference-independent stability metrics to address this limitation.
- Scope of evaluated models: The evaluation covers fixed-weight open-source models rather than closed systems such as Veo, Sora, or Kling, limiting coverage of the model landscape.The benchmark remains far from saturated: even the strongest evaluated model reaches only 22% success rate.
6 Conclusions
CRONOS is an intervention-based benchmark for evaluating counterfactual physical consistency in video generation models using controlled physical events. Results reveal that current models struggle with simple physical events and vary substantially under targeted visual interventions, indicating low counterfactual consistency.
- Conclusions: CRONOS evaluates counterfactual physical consistency using high-quality synthetic videos of controlled collisions, falling dynamics, and occlusions.The benchmark is designed to diagnose how video models respond to controlled visual interventions while evaluating physical events.
- Conclusions: Current video models struggle to generate simple physical events and show substantial variation under targeted visual interventions.These limitations persist even among the strongest current video models.
- Conclusions: The findings reveal low counterfactual physical consistency and suggest that many models rely on superficial visual correlations rather than stable scene-dynamics predictions.CRONOS is intended to expose these weaknesses and support more robust, structured video-model predictions.
- Conclusions: CRONOS provides a testbed for diagnosing model limitations and guiding development of more robust and structured predictions of the visual world.Its purpose is to help evaluate and improve prediction stability under controlled interventions.
A Dataset details · B Text Prompts
CRONOS uses handcrafted object–scene combinations with explicitly controlled physical parameters and renders intervention variations across three events, with occlusion exempt from viewpoint changes. Video models also receive manually designed, structurally consistent prompts that describe event dynamics and intervention values coarsely.
- A Dataset details: Objects are selected for geometries and materials that support sliding or rolling on smooth surfaces, producing visually plausible and dynamically rich motion patterns.The dataset is handcrafted from combinations of object and scene assets.
- A Dataset details: Mass, friction, and restitution coefficients are explicitly defined per object and kept consistent across events.
- A Dataset details: All variations are rendered for the three physical events, except occlusion, whose view remains unchanged to preserve the intended visibility effect.The number of rendered videos per event is reported in Table 3.
- A Dataset details: Table 2 reports the number of variations generated for each intervention.
- B Text Prompts: Text prompts are provided to all video models alongside the visual conditioning signal.Prompts are manually designed for each physical event and use the same structure across interventions.
- B Text Prompts: Prompts coarsely describe expected event dynamics without many details, while adding intervention-specific variation values and event-required information.
C Evaluation details
The evaluation combines robust aggregation, object tracking, visibility filtering, and metrics for object, background, motion, shape, and physical-event consistency. A calibrated success rate then prevents catastrophic failures in any single metric from being hidden by other scores.
- Robust aggregation: Scores aggregate the worst k frames rather than averaging all frames or using only the single worst frame, balancing sensitivity and stability.Brief but meaningful errors can be diluted by averages, whereas a single worst frame is sensitive to metric limitations.
- Tracking and filtering: Object masks and point tracks are propagated from ground-truth first-frame renders using SAM3 and CoTracker3.The first-frame prompts always match the ground-truth renders before per-frame predictions are obtained.
- Tracking and filtering: Disappearance and occlusion filters exclude object-centric evaluations when objects vanish spuriously or have too few visible pixels.The disappearance detector distinguishes abrupt disappearance from legitimate occlusion or leaving the scene, while the occlusion filter retains frames with low occlusion.
- Metric suite: The metric suite measures object stability, background stability, motion similarity, shape stability, and physical plausibility.These metrics target semantic object distortions, background alterations, appearance-independent motion, object morphing, and high-level physical events.
- Metric suite: Physical plausibility uses event-specific and general questions evaluated by a VLM, with inverse scaling that penalizes videos containing several incorrect answers.The VLM returns True/False answers, justifications, and confidence, and the metric compares answers with expected outcomes.
- Success rate: Success rate thresholds are calibrated against human annotations so catastrophic failure in one metric cannot be smoothed by other results.A video is successful when it exceeds the calibrated thresholds, based on annotated metrics with median human scores of 3 or higher.
D User Study · E Additional examples
The user study used detailed instructions, qualification screening, and compensated Prolific annotators to assess video quality, while additional sections provide complete generated-video examples.
- D User Study: The study provided annotators with detailed participation instructions.The instructions are shown in Figure 7.
- D User Study: The annotation interface evaluated videos along the considered quality axis.The GUI is shown in Figure 8.
- D User Study: All annotators had to pass a qualification exam before participating.
- D User Study: The study employed 8 hired Prolific annotators.
- D User Study: Annotators received compensation of 14 £/hour.
- E Additional examples: The additional-examples section shows the complete set of generated videos for the examples in Section 3.These videos are presented in Figures 9, 10, and 11.
F Broader impacts · G Asset Licenses and Release Documentation
CRONOS is intended to diagnose failures in counterfactual physical consistency while recognizing risks of misuse, over-interpretation, computational cost, and limited event coverage. The release provides generated benchmark artifacts and evaluation code under documented asset, tool, and dataset licensing constraints.
- F Broader impacts: CRONOS exposes object drift, broken object permanence, implausible motion, and sensitivity to viewpoint, scene, category, and appearance changes.This diagnostic purpose can help researchers avoid overestimating the physical reliability of visually plausible video predictions.
- F Broader impacts: Improving performance using CRONOS could make misleading synthetic media more physically plausible.Strong CRONOS performance also would not necessarily demonstrate general physical reasoning or suitability for safety-critical use.
- F Broader impacts: Synthetic Unreal Engine rendering reduces privacy risks compared with real-world video datasets.Releases should document intended use, known limitations, and applicable licenses for rendered assets and third-party tools.
- F Broader impacts: CRONOS evaluates a narrow set of controlled events and should not support claims of broad physical competence.The benchmark’s intended interpretation is limited to its controlled synthetic rigid-body event coverage.
- F Broader impacts: Rendering high-resolution videos and running segmentation, 3D reconstruction, motion encoding, and VLM judging add computational cost beyond standard evaluation.These model-based evaluation components contribute to the benchmark’s environmental considerations.
- G Asset Licenses and Release Documentation: Third-party Unreal Engine assets were purchased under ECLA and selected only when GenAI-related research and benchmark generation were allowed.CRONOS uses these assets in controlled synthetic scenes and does not rely on scraped videos or unlicensed real-world footage.
- G Asset Licenses and Release Documentation: The release includes rendered RGB videos, text prompts, physical-event and intervention metadata, and evaluation code.These artifacts support reproduction of the benchmark and computation of the reported evaluation metrics.
- G Asset Licenses and Release Documentation: Third-party Unreal Engine source assets with redistribution restrictions are not redistributed; the package instead provides shareable data, code, metadata, and license information.Release documentation includes attribution and license notes for third-party tools and assets.
H Detailed results
This section provides additional results, including further generated examples. The supplementary material is presented without further detail in the supplied passages.
- H Detailed results: The section introduces additional results beyond the preceding analysis.The passage does not specify the experiments or findings included.
- H Detailed results: The supplied passages identify supplementary examples but do not report quantitative findings.No metrics, comparisons, or numerical results are included in the provided text.
- H Detailed results: Figure 9 presents additional generated examples.The supplied figure caption provides no further description of the examples.
I Computational resources
The section provides an overview of CRONOS’s computational resources, including the hardware used for benchmark generation and evaluation.
- I Computational resources: Table 10 summarizes compute used for benchmark generation and evaluation.Each evaluated model configuration generated 2025 videos; generation used NVIDIA H100 GPUs, while evaluation used NVIDIA A100 GPUs.