PaperlayerView exact revision on arXiv

Source-linked AI summary

CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?

Jonathan Sadeghi, Jenny Seidenschwarz, Jesse Allardice, Sirish Srinivasan, Benjamin Graham, Jeffrey Hawke

arXiv:2608.16829v1cs.LGcs.AI

TL;DR

Existing video benchmarks do not test fine-grained statistical calibration of specific stochastic physical phenomena. CaliBench evaluates discrete outcomes against analytically known distributions while separating scorability from calibration, revealing systematic probability-mass concentration and mode collapse across frontier models.

  • Problem

    Existing video evaluations emphasize perceptual plausibility or learned embedding distances, leaving statistical calibration of stochastic physical outcomes unmeasured.

  • Method

    CaliBench evaluates discrete physical outcomes against closed-form reference distributions and separates performance into scorability and total variation calibration.

  • Results

    Across six models and nine stochastic scenes, generations systematically over-concentrate probability mass, often collapsing to a single outcome despite realistic individual trajectories.

  • Takeaways & Limitations

    CaliBench exposes substantial stochastic-dynamics vulnerabilities in frontier video world models and establishes a mathematical baseline for evaluating physical calibration.

  • Takeaways & Limitations

    The benchmark evaluates image-to-video pipelines and marginal outcome distributions rather than text-to-video stochasticity or full continuous trajectories.

Abstract

from arXiv · show

Video world models approximate the stochastic distribution of physical outcomes through generative sampling, but existing benchmarks score individual generations or compare distributions coarsely over a whole dataset, leaving the fine-grained aleatoric uncertainty of specific phenomena untested. We introduce CaliBench, which scores outcomes in a physically interpretable discrete space - a bin index, a die face, a suit, a colour - rather than a learned feature space such as in FID, so the distance from a known reference distribution is measured directly. We curate outcome spaces whose reference is known in closed form (binomial Galton boards, Bernoulli forks, uniform dice/cards/lottery, a skewed European-roulette colour), enabling an exact calibration test. We decompose performance into two orthogonal axes that a single accuracy metric conflates: scorability, the fraction of generations yielding a scoreable outcome, and calibration, the total variation distance from the reference on that sample. A chi-squared test assesses significance; as calibration is its null hypothesis it can evidence only miscalibration, and at N=32 per cell detects only large deviations. We apply it to nine scenes and six image-to-video models (WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, Cosmos3-Super), 32 generations each. Models consistently concentrate probability mass on a few outcomes rather than reproducing the reference. Most scene-model combinations are significantly miscalibrated, in the extreme collapsing to one outcome, as Veo 3.1 does on dice. On roulette, generations often leave the ball ambiguously placed, giving several models low scorability. Performance varies by scene: no model dominates all nine. We release the protocol and a metric (mean normalised total variation, mnTV) for comparing new models against our results.

1 Introduction

CaliBench evaluates whether video world models reproduce physically grounded stochastic outcome distributions, addressing the limits of perceptual and per-sample video metrics. Its protocol separates valid-outcome generation from distributional calibration across analytically tractable discrete spaces.

  • Motivation: Existing video metrics such as FVD assess abstract embedding distances or per-sample plausibility, but do not measure whether generated physical outcomes are statistically calibrated.Their latent coordinates are not physically interpretable and do not map to conservation laws.
  • Benchmark: CaliBench benchmarks nine physical environments whose discrete outcomes have analytically known reference distributions, enabling exact calibration tests without learned feature spaces or empirical dataset proxies.Outcome spaces include bin indices, die faces, and suit or colour identifiers, with binomial, Bernoulli, uniform, or skewed references.
  • Evaluation protocol: The protocol separates scorability, the fraction of structurally valid generations, from calibration, the Total Variation Distance between scoreable outcomes and the reference distribution.This distinguishes structural instability from mode collapse, including high scorability with severe distributional error.
  • Results and limitations: Across six image-to-video models and nine scenes, χ2 testing shows that most models are significantly miscalibrated, often concentrating probability mass despite plausible individual frames.The test treats calibration as its null hypothesis, so non-significant results are inconclusive; with N=32 per cell, it reliably detects only large miscalibrations.
  • Reproducibility: The authors release the protocol, conditioning imagery, VLM prompts, and analysis code, and introduce Mean Normalised Total Variation (mnTV) to track progress toward physically grounded stochastic simulation.The release supports reproducible and standardized comparisons between models.

2 Related Work

Existing video-generation benchmarks emphasize per-sample quality, plausibility, or learned feature-space distributional similarity, but do not directly test fine-grained physical calibration against known reference distributions. CaliBench addresses this gap by auditing calibration against established physical distributions.

  • Per-Sample Quality and Physical Plausibility: Existing suites such as VBench evaluate individual video rollouts across 16-plus dimensions, while diversity remains distinct from calibration.A model may produce diverse generations while still distributing probability mass incorrectly.
  • Feature-Space Distributional Metrics: FVD and JEDi measure appearance-level divergence in learned perceptual feature spaces after aggregating statistics across entire datasets.This global aggregation limits physical interpretability and cannot isolate a specific phenomenon’s aleatoric uncertainty.
  • Distributional Calibration without a Known Reference: Calibration methods for image generation typically compare samples against empirical datasets without closed-form references using learned feature-space tests.Examples include Voronoi-cell partitioning, precision and recall from density estimates, and effective-support bounds.
  • Uncertainty Estimation in Video Generation: Uncertainty-estimation frameworks such as S-QUBED and C3 quantify predictive or pixel-level uncertainty but do not evaluate calibration against established physical distributions.CaliBench explicitly targets this evaluation gap.
  • Benchmarking Video World Models: WorldModelBench, STEVO-Bench, TiViBench, PAI-Bench, PhysicsIQ, and PhyGenBench assess per-sample physics adherence, object permanence, reasoning, plausibility, trajectory matching, or commonsense laws.These benchmarks evaluate individual continuations or prompts rather than calibration of stochastic outcomes against a physical reference distribution.

3 CaliBench

CaliBench evaluates video world models on nine stochastic physical scenes with analytically known discrete outcome distributions, using 32 trajectories per model–scene pair. It separates outcome scorability from calibration error and aggregates floor-corrected total variation into mnTV while retaining per-scene results.

  • Evaluation environments: CaliBench samples N = 32 trajectories for each model–scene pair across nine scenes grouped by their reference-distribution properties.The scenes cover binomial, Bernoulli, uniform, and known non-uniform outcome spaces.
  • Outcome spaces: The benchmark includes Galton boards with a Binomial(10, 1/2) target, symmetric Bernoulli scenes, uniform dice/cards/lottery, and skewed European roulette.The Galton-board distribution has 11 bins with a central bin-6 mode of approximately 24.6%; roulette assigns red and black probability 18/37 each and green probability 1/37.
  • Metrics: Performance is decomposed into scorability ρ and Total Variation Distance, because perfect scorability can coexist with severe calibration failure and does not ensure physical plausibility.TVD provides a unified metric for categorical spaces where inter-outcome distances are undefined, despite ignoring ordinal proximity in Galton boards.
  • Statistical testing: Calibration significance is evaluated with Monte Carlo p-values from 50,000 reference-distribution samples using the observed χ2 statistic.The procedure compares the observed statistic against replicates of size nvalid drawn from the reference distribution.
  • Aggregate benchmark score: The aggregate metric mnTV averages floor-corrected per-scene excess TV across all nine scenes, approaches zero for fully scoreable calibrated models, and is reported with bootstrap confidence intervals.mnTV is bounded above by 1 and is intended to complement, not replace, the per-scene grid because averaging can mask reversals across heterogeneous scenes.

4 Results

CaliBench results show substantial, heterogeneous miscalibration across models and scenes, with no model dominating overall or across all environments. SeeDance-2.0 has the lowest aggregate calibration penalty, but most tested cells remain significantly miscalibrated after correction.

  • Aggregate results: SeeDance-2.0 achieves the lowest overall calibration penalty (mnTV = 0.39), but overlapping bootstrap intervals support broad performance tiers rather than a strict ranking.All empirical aggregates remain far above the perfectly calibrated baseline (mnTV ≈0).
  • Per-scene heterogeneity: Across eight informative scenes, no model dominates: the per-scene leader changes among Runway Gen-4.5, SeeDance-2.0, HappyHorse, Cosmos3-Super, and Veo 3.1.Cards is a three-way tie in TV, and most scenes still show significant miscalibration for every model.
  • Significance testing: 28 of the 44 testable scene–model cells remain significant after multiple-comparison correction, including severe mode collapses that survive Bonferroni correction.Severe collapses have p < 0.001, while only two cells change verdict relative to an uncorrected threshold.
  • Significance testing: Runway Gen-4.5 on the physical board and SeeDance-2.0 on lottery become unrejected after correction, whereas Cosmos3-Super on dice remains significant (p = 0.011→q = 0.019).The two newly unrejected cells have p = 0.038→q = 0.057 and p = 0.049→q = 0.070, respectively.
  • Distributional patterns: Galton-board generations generally over-concentrate near the central bin, while animated-board SeeDance-2.0 instead places excessive mass on bins 1 and 2.The reference distribution is Binomial(10, 1/2), whose expected peak probability is 24.6%.

5 Discussion, Limitations, and Future Work

CaliBench reveals systemic probability-mass over-concentration in frontier world models, often despite realistic individual samples, while no model consistently achieves both high scorability and low total variation across scenes. The evaluation is limited by image-to-video scope and extraction noise, but its computationally intensive protocol supports future distributional verification and broader scene testing.

  • Discussion: Frontier world models’ dominant failure mode is systemic probability-mass over-concentration, sometimes collapsing entirely despite high individual-sample realism.The discussion attributes this to shared a priori biases and localized conditioning cues in the initial image.
  • Limitations: 93.8% annotation agreement supports human validation of automated VLM outcome extraction, though extraction still introduces a minor noise floor.The limitation concerns image-to-video pipelines and does not evaluate text-to-video stochasticity.
  • Future Work: 1,728 video rollouts and 5,184 VLM queries are required across the current evaluation matrix, making multi-sample evaluation computationally intensive.The authors nevertheless describe the protocol as viable for low-step or autoregressive architectures and as a pathway toward calibrated physical simulators.
  • Discussion: No evaluated model consistently approaches the calibration target of high scorability and low total variation across all nine scenes.The target is the bottom-right region of each scorability-versus-TVD panel.
  • Future Work: Future extensions include testing more reference scenes and whether closed-form-scene calibration predicts distributional fidelity on open-world footage.Open-world reference distributions would need to be estimated empirically from large observational samples.

6 Conclusion

CaliBench benchmarks calibration in discrete, physically meaningful outcome spaces and shows that current video world models are unreliable stochastic simulators. Despite realistic individual trajectories, the models exhibit severe probability mass concentration and mode collapse, which the framework exposes through separate scorability and distributional-calibration measures.

  • CaliBench evaluates stochastic calibration across six state-of-the-art models and nine classic stochastic environments.
  • Current architectures routinely fail as reliable stochastic simulators despite producing highly realistic, physically plausible individual trajectories.
  • Severe probability mass over-concentration and mode collapse systematically undermine the stochastic dynamics of frontier world models.
  • By separating scorability from distributional calibration, CaliBench exposes critical vulnerabilities and establishes a rigorous mathematical baseline.

A mnTV Definition and Per-Cell Decomposition · B Empirical Distributions for All Scenes

mnTV is a noise-corrected aggregate of per-scene calibration that penalizes unscorable generations, while its interpretation depends on fixed sample size and outcome extraction. Empirical distributions are compared with known references across scenes, with the ball-fork scene showing generally close calibration.

  • A mnTV Definition and Per-Cell Decomposition: mnTV subtracts expected finite-sample variation, yielding excess error esm = 0 at perfect calibration and 1 for total concentration on an impossible outcome.Signed excess error can be slightly negative when calibration is better than the sampling-noise floor.
  • A mnTV Definition and Per-Cell Decomposition: Unscorable generations contribute through (1−ρ), which can dominate mnTV on cards, roulette, lottery, and pendulum scenes.Improving scorability could therefore change conclusions about calibration on null-heavy scenes.
  • A mnTV Definition and Per-Cell Decomposition: The double-pendulum component contributes approximately 1 −ρ for every model because no model produces more than three valid samples.The valid-sample count is too small for informative esm estimation.
  • A mnTV Definition and Per-Cell Decomposition: mnTV must be interpreted at the specific sample size N and outcome-extraction protocol used, and Table 2 reports lower-is-better scores bounded above by 1.The benchmark’s baseline comparability requires keeping these experimental hyperparameters invariant.
  • A mnTV Definition and Per-Cell Decomposition: Because mnTV combines scorability and calibration across nine heterogeneous scenes, it is an ordering tool rather than a substitute for per-scene evaluation.Per-scene reversals are expected, including strong calibration in one environment and total mode collapse in another.
  • A mnTV Definition and Per-Cell Decomposition: The null baseline from 50,000 perfectly calibrated, fully scorable simulations at N = 32 has mean ≈0, median ≈0, and central 95% interval ≈[−0.04, 0.04].Each simulation sets ρ = 1 and samples independently from the true reference distribution.
  • B Empirical Distributions for All Scenes: The empirical distributions compare each model’s outcomes with the reference across Figures 6–12, while the ball-fork scene is generally close to calibrated.The fork reference is Bernoulli(1/2), uniform over left and right, with only mild imbalance.

C Statistical Power of the χ2 Test · D Multiple-Comparison Correction

The χ2 test’s power depends on scene-specific sensitivity thresholds, so non-rejection is informative only relative to the detectable TVD at the available sample size. Multiple-comparison corrections preserve the conclusion that severe mode collapse is statistically significant while narrowing the set of rejections overall.

  • C Statistical Power of the χ2 Test: At N = 32 and α = 0.05, non-rejection is not definitive evidence of calibration without sufficient power to detect moderate deviations.The analysis establishes each scene configuration’s minimum reliably detectable discrepancy.
  • C Statistical Power of the χ2 Test: The power analysis finds the minimum contamination magnitude yielding at least 80% χ2-test rejection probability for each scene.Contamination mixes the analytical reference distribution with a point mass on an empirically dominant failure outcome.
  • C Statistical Power of the χ2 Test: Sensitivity is reported as minimum detectable TVD, with thresholds referenced at N = 32 and N = 16 to approximate scorability degradation.Continuous sensitivity across sample allocations is shown through power curves.
  • C Statistical Power of the χ2 Test: Low-threshold roulette and binary-fork scenes make non-rejection highly informative, whereas Galton boards, dice, cards, and lotteries can conceal moderate miscalibration.Every non-rejection should therefore be interpreted against its corresponding scene-specific threshold.
  • D Multiple-Comparison Correction: Table 3 evaluates 54 scene–model configurations, but only 44 provide enough valid generations for a finite χ2 goodness-of-fit p-value.Ten configurations are excluded because of near-empty or single-valid-generation cells.
  • D Multiple-Comparison Correction: BH rejects 28 of 44 null hypotheses at α = 0.05, using a critical raw threshold of p ≤0.0297.The Benjamini–Hochberg procedure is less conservative than Bonferroni in this analysis.
  • D Multiple-Comparison Correction: Bonferroni rejects only configurations with raw p ≤0.00114, bracketing the conclusions more conservatively than BH.Both corrections reject every environment exhibiting severe mode collapse with raw p < 0.001.

E MCAR Null-Sensitivity Analysis

In null-heavy scenes, unscoreable generations make the null-extended term dominate total variation, so improved scorability could substantially change reported calibration. Extreme MCAR imputations bound this sensitivity, while lottery and roulette reveal concentrated or distorted outcome probabilities.

  • Null-heavy scenes: Unscoreable generations dominate reported TV in cards, roulette, lottery, and pendulum because the null-extended term (1 −ρ) is large.The analysis examines how calibration might change if scorability improved.
  • Outcome-specific sensitivity: Lottery’s uniform 20-number reference exposes severe collapse, with probability mass concentrated on a small number of ball numbers across all models.The reference distribution is Uniform{1, . . . , 20}.
  • Outcome-specific sensitivity: Roulette over-represents green among most models with non-trivial scorability, while HappyHorse produces no green outcomes.The reference probability for green is 1/37, or 2.7%; HappyHorse records 0/31 green outcomes, whereas Cosmos3-Super records 50% on 6/12 valid outcomes.
  • MCAR bounds: MCAR bounds assign missing outcomes to minimise or maximise full-N total variation, bracketing best- and worst-case calibration.Table 7 compares these bounds with the reported charge TVrep.

F VLM Extraction Validation · G Conditioning Text Prompts · H VLM Extraction Prompts

The paper validates automated VLM outcome extraction against human annotations, finding high overall agreement with limited disagreement concentrated in a few scenes. It also specifies fixed scene prompts and strict null-handling rules for extracting discrete outcomes from frames or videos.

  • F VLM Extraction Validation: F VLM Extraction Validation: Human annotators labeled 162 videos, comprising 18 generations per each of nine environments, with either a physical outcome or null.The stratified corpus spans six architectures and provides ground truth for automated extraction.
  • F VLM Extraction Validation: F VLM Extraction Validation: 152/162 (≈93.8%) automated labels exactly agreed with humans; Galton board and dice reached 100%, while cards reached 83%.Exact agreement treats null as a distinct discrete state.
  • F VLM Extraction Validation: F VLM Extraction Validation: The false-null rate was 4/107 (≈3.7%), while disagreements included two wrong labels and rendering artefacts affecting cards, roulette, and Galton-board extraction.False-null errors conservatively reject human-valid outcomes and deflate observed scorability; missed-null errors introduce structural noise.
  • F VLM Extraction Validation: F VLM Extraction Validation: With 6.2% overall disagreement, correcting labels could change reject/non-reject verdicts only for cells whose p-values are near 0.05, not severe mode-collapse cells at p < 0.001.The fragile cells were Runway Gen-4.5 on the physical board and SeeDance-2.0 on lottery, which the paper declines to interpret as miscalibration evidence.
  • G Conditioning Text Prompts: G Conditioning Text Prompts: Each scene uses a fixed text prompt alongside its conditioning frame, describing physically accurate terminal-state or direction-of-motion scenarios.Terminal-state scenes use final frames, while ball fork, double pendulum, and walking use full videos for extraction.
  • H VLM Extraction Prompts: H VLM Extraction Prompts: Gemini 3.1 Pro receives scene-specific instructions to output a discrete label or null when the outcome is ambiguous, with each query repeated three times and majority-voted.The protocol covers bin indices, left/right directions, pendulum position, die faces, card suits, lottery numbers, and roulette colours.
  • H VLM Extraction Prompts: H VLM Extraction Prompts: Roulette extraction requires a visible ball inside a numbered pocket and returns null for ambiguous, invalid, or split-pocket placements.The prompt rejects balls outside the pocket ring, on the hub, on the inner slope, or equally split between pockets; otherwise it reports red, black, or green.

I Generation-Setting Ablations · I.1 Resolution ablation (SeeDance-2.0, 480p vs. 720p) · I.2 Duration ablation

Generation-setting ablations test whether resolution or duration drives calibration results. SeeDance-2.0’s strongly miscalibrated scenes remain unchanged at 720p, while duration is standardized to 5 seconds where feasible and compared using aggregate mnTV.

  • I Generation-Setting Ablations: The ablations vary output resolution and video duration separately to test whether either generation setting drives calibration results.Each ablation changes one setting while holding the other conditions fixed.
  • I.1 Resolution ablation (SeeDance-2.0, 480p vs. 720p): SeeDance-2.0 is regenerated at 720p with identical frames, prompts, and 32 seeds, changing only resolution from its benchmarked 480p.The outputs are re-extracted and re-scored against the 480p results.
  • I.1 Resolution ablation (SeeDance-2.0, 480p vs. 720p): Strongly miscalibrated animated-board, dice, and card scenes remain unchanged across resolutions, with p < 0.001 at both 480p and 720p.These scenes are identified as strongly miscalibrated under both settings.
  • I.1 Resolution ablation (SeeDance-2.0, 480p vs. 720p): Borderline SeeDance-2.0 cells mostly worsen at 720p, including lottery and roulette.The supplied passage truncates the list after mentioning these scenes.
  • I.2 Duration ablation: WAN-2.7, SeeDance-2.0, and HappyHorse-1.0 are regenerated at uniform 5-second duration using identical frames, prompts, and 32 seeds.Only duration changes for these models.
  • I.2 Duration ablation: Runway Gen-4.5 and Cosmos3-Super are reused near 5 seconds, while Veo 3.1 remains at benchmarked 4 seconds because its API accepts only {4, 6, 8} seconds.The duration comparison reports aggregate mnTV, with lower values indicating better calibration.

J Classifier-Free Guidance Sweep on Cosmos3-Super Dice

Cosmos3-Super dice calibration depends strongly on CFG: lower guidance approaches the uniform reference but sharply reduces scorability, while higher guidance preserves valid outputs yet remains miscalibrated. The default CFG = 6.0 therefore contributes to, but does not fully explain, the observed miscalibration.

  • Interpretation: The baseline CFG = 6.0 result of TV = 0.28 and p = 0.011 is therefore partially an artefact of the default guidance setting.The sweep shows that attenuating guidance can improve calibration, but only by reducing scorability.
  • Experimental setup: Seven CFG settings were evaluated with 32 generations each under a fixed 61-frame horizon, varying only the guidance scale.The sweep used CFG ∈{1.0, 1.5, 3.0, 4.5, 6.0, 7.5, 9.0}.
  • Calibration–scorability trade-off: 0.26 TV at CFG = 4.5 was the minimum for CFG ≥3.0, but TV rose to 0.51 at CFG = 9.0 while every setting remained significantly miscalibrated.In this regime, scorability saturated at ρ ≥0.97 and p ≤0.042.
  • Calibration–scorability trade-off: 0.17 TV at CFG ∈[1.0, 1.5] made the distribution statistically indistinguishable from uniform, with p = 0.70 and p = 0.39.These low-guidance settings had the lowest scorability, so TV was computed on fewer valid generations.
  • Calibration–scorability trade-off: ρ fell to 0.78 at CFG = 1.5 and 0.66 at CFG = 1.0, reflecting substantially more unscorable sequences at the settings with improved calibration.At higher guidance, probability mass concentrates on faces 1 and 5, while face 6 is essentially absent for CFG ≥3.0.

K Explicit Target-Face Ablation on Dice

The explicit target-face ablation shows that prompt steering is strongly shaped by geometric visibility: visible die faces are easier to request, while occluded faces often remain unreachable. Steering ability also does not guarantee baseline physical calibration, because some models structurally fail to generate particular faces.

  • Evaluation protocol: Compliance measures the proportion of valid generations whose rolled face matches the explicitly requested face, against a 1/6 ≈16.7% random baseline.Each face is requested by appending a conditional sentence to the baseline prompt, with five random seeds per model and requested face.
  • Geometric bias: Visible 1- and 5-pip faces achieve 60–100% compliance across most models, whereas the occluded 6-pip face has 0% compliance for five of six models.The conditioning image anchors which faces can be requested, and the compliance pattern follows this geometric bias.
  • Steering versus calibration: SeeDance-2.0 reaches the highest aggregate steering accuracy at 63.3%, with 60–100% compliance across five of six requested faces.This indicates that the model can generate hidden facets when explicitly conditioned, despite its biased unsteered distribution.
  • Steering versus calibration: High prompt-steering compliance does not imply baseline physical calibration, so severe marginal deviations may reflect a genuine capability gap rather than only an uncalibrated default distribution.SeeDance-2.0 combines the highest steering accuracy with an outcome-agnostic distribution concentrating 56% of its unsteered mass on face 1.
  • Structural generation failures: HappyHorse-1.0 never generates faces 2, 3, 4, or 6, Veo 3.1 never generates face 2, and WAN-2.7 never generates face 6 across the 30 evaluation sequences.These failures can make biased calibration defaults observationally indistinguishable from structural generation failures.
Loading 2608.16829v1…