PAWBench: How Far Are We from Probabilistically Aligned World Modeling?

1 INTRODUCTION

• Probabilistic alignment requires a model’s distribution under a fixed observation and action to reflect both which futures are physically possible and how likely each is.This matters for interaction and planning because decisions depend on both possible consequences and their relative likelihoods.Source: §1 · p. 1
• Existing benchmarks assess generated videos individually, so plausible rollouts can conceal a narrow range of outcomes or incorrectly weighted futures.Single-video scores for visual quality, temporal coherence, text or action alignment, and physical plausibility do not reveal the conditional distribution over futures.Source: §1 · p. 2
• PAWBench evaluates 50 scenarios across eight mechanism groups, using repeated rollouts and PAWEval to measure probability alignment or valid-outcome recovery.Its two 25-scenario suites are PAW-Calibration, with specified reference probabilities, and PAW-Coverage, with enumerable valid outcomes whose relative probabilities cannot be reliably specified.Source: §1 · pp. 2–3
• No tested model consistently achieves accurate outcome probabilities, broad valid-future coverage, and reliable performance across scenes.More rollouts support the main evaluation’s sampling budget, and PAWEval agrees with human judgments on videos with clear terminal outcomes; distributions respond too little to changed physical transitions but can shift with non-causal cues.Source: §1 · pp. 2–3
• The paper probes language guidance, initial-noise sampling, and fine-tuning as ways to steer, broaden, or reshape predicted future distributions.Language makes possible futures explicit, sampling varies noise while leaving the future unspecified, and fine-tuning tests learning how future probabilities vary with physical state.Source: §1 · pp. 2–3

• A video world model is represented by a conditional distribution over future trajectories, which may yield different valid outcomes from the same observation and action.Probabilistic alignment concerns both which futures the distribution supports and the probability assigned to each.Source: §2 · p. 3
• Mapping trajectories to terminal outcomes induces an outcome distribution; with a reference distribution, probability-mass alignment requires equality, while support alignment concerns which outcomes are possible.This outcome-level distribution abstracts away differences among trajectories that reach the same result.Source: §2 · p. 3

3.1 BENCHMARK DESIGN

• Each item fixes an initial image and action, defines finite valid outcomes and sometimes a reference distribution, then maps K repeated rollouts to an empirical outcome distribution.Holding inputs constant makes variation across rollouts reflect the model-induced future distribution.Source: §3.1 · pp. 3–4
• Across 50 scenarios in eight mechanism groups, 25 Calibration cases test reference-probability matching and 25 Coverage cases test recovery of valid futures; scores remain separate.Calibration scenarios have analytic or symmetry-derived reference probabilities, while Coverage scenarios have enumerable outcomes whose probabilities cannot be reliably specified.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 SCENARIO CURATION

• Scenarios require visible physical stochasticity, one atomic action that can be judged in the video, and a finite, visually distinguishable outcome set; all 50 are manually reviewed.Source: §3.2 · p. 4

3.3 EVALUATING DISTRIBUTIONS OVER POSSIBLE FUTURES

• PAWEval uses Gemini 3.5 Flash and scenario rubrics to label repeated rollouts, reports readout failures separately, and scores Calibration by TVD and Coverage by observed valid-outcome fraction.Only readable, in-schema outcomes contribute to the distribution and metrics.Source: §3.3 · p. 5

4.1 EVALUATION SETUP

• The evaluation tests eleven video generators with K = 50 rollouts per scenario and system, using the same source image and fixed action prompt across rollouts.A scene passes when at least 20 of 50 outcomes are readable and in-schema; Scene Pass Rate (SPR) is the percentage of 25 track scenes that pass.Source: §4.1 · p. 5

4.2 BENCHMARK RESULTS

• No model combines well-aligned probability mass, broad support recovery, and reliable scene-level performance across PAWBench.Cosmos 3 Super I2V has the lowest Calibration TVD, but only 80.0% of its Calibration scenes pass; LTX-2.3 has the highest Coverage average over the 72.0% of Coverage scenes that pass. Seedance 2 passes every Calibration scene and LingBot-Video-MoE every Coverage scene, but neither leads its track’s conditional metric.Source: Table 1 · p. 6, §4.2 · p. 6
• Recovering broad valid-outcome support does not ensure that a model assigns the correct probabilities to those outcomes.PAW-Coverage and PAW-Calibration capture complementary failures: models can expose plausible alternatives but misallocate probability mass, or match observed frequencies while leaving valid alternatives unseen.Source: §4.2 · p. 6
• Models shift outcome distributions incompletely or in the wrong direction under causal interventions, while distractor text redirects probability under non-causal interventions.The causal pencil tilt changes the physical transition; Galton-board text is non-causal and should leave the future distribution unchanged. The same pattern appears across paired controls, video generators, and direct VLM future samplers.Source: §4.2 · pp. 6–7, Figure 4 · p. 6

4.3 ROBUSTNESS OF THE EVALUATION PROTOCOL

• Doubling rollout budget leaves calibration largely unchanged, and 99% of matched reference-sampling simulations have average TVD below 9.22, versus 31.2 observed across generators.Coverage rises for three of four models as samples increase; the authors use K = 50 as a shared budget, not a convergence point.Source: §4.3 · p. 7, Figure 5 · p. 7
• PAWEval agrees with decisive human judgments on 722 of 888 videos with clear labels from both, or 81.3%.Seven independent human judgments were collected per video using the same scene-specific outcome space; disagreement over clear outcomes alone cannot account for the benchmark gap.Source: §4.3 · p. 7

5.1 PROMPT ENGINEERING

• GPT-5.5 prompt engineering selections score 44.3 Calibration TVD and 35.0% Coverage before synthesis, while Oracle PE lowers Calibration TVD and raises Coverage for every generator, yet only 37.6–58.1% of requested outcomes are realized.Prompt-based control is limited both by selecting the wrong distribution of futures and by generators missing supplied targets.Source: §5.1 · pp. 8–9, Table 3 · p. 8, Figure 7 · p. 8

5.2 INITIAL NOISE SAMPLING

• Across passing scenes, C2C lowers mean Calibration TVD and raises mean Coverage for all three generators, but does not consistently raise SPR or change the model’s learned distribution.C2C couples K = 50 initial-noise samples with negative dependence while preserving each sample’s standard Gaussian marginal, broadening finite-gallery exploration of existing possibilities.Source: §5.2 · p. 9, Table 4 · p. 9

5.3 UPDATING MODEL PARAMETERS

• Training more left-fall examples shifts generated left-fall frequency upward nonlinearly in both scenes, but no adapted model matches both the 50/50 upright and 100/0 left-leaning references.At 20% left-fall training, the model comes closest to the upright reference while the left-leaning scene remains 50/50; at 80% or 100%, the left-leaning scene reaches its reference but the upright pencil falls left almost every time.Source: §5.3 · pp. 9–10, Figure 8 · p. 9, Table 5 · p. 9

Video Generators as World Models

• PAWBench shifts world-model evaluation from coherent, controllable, or realistic rollouts to whether repeated samples assign probability mass to the right futures.World models have supported planning through action-consequence prediction, and recent video generators and interactive systems pursue controllable visual rollouts (Sutton, 1990; Ha & Schmidhuber, 2018; Hafner et al., 2020; Schrittwieser et al., 2020; OpenAI, 2024; Bruce et al., 2024; Google DeepMind, 2025a; NVIDIA, 2025; Yang et al., 2024).Source: §6 · p. 10

Video-generation and world-model benchmarks

• Prior video, physics, and world-model benchmarks assess visual quality, physical plausibility, action following, or task utility, whereas PAWBench compares repeated fixed-input rollouts’ outcome distributions or support.Relevant benchmark work includes video quality and alignment (Liu et al., 2024; Huang et al., 2024a;b), physics and action following (Bansal et al., 2024; Meng et al., 2024; Guo et al., 2025; Sanli et al., 2025; Pu et al., 2026; Li et al., 2026), and world-model prediction and utility (Li et al., 2025; Qin et al., 2025; Kang et al., 2024; Tian et al., 2023).Source: §6 · p. 10

Calibration and coverage under fixed actions

• PAWBench separates probability calibration from support coverage, complementing CaliBench’s nine stochastic scenes with controlled probes of causal state, language, sampling noise, and training distribution.PAW-Calibration applies when defensible reference probabilities are available; PAW-Coverage applies when only valid support can be specified. CaliBench reports scorability separately from conditional TVD and tests miscalibration (Sadeghi et al., 2026).Source: §6 · p. 10

7 LIMITATIONS

• Terminal-outcome comparisons make distributional evaluation tractable but do not capture trajectory-level dynamics or intermediate physical processes.Future work should extend probabilistic alignment to richer state trajectories.Source: §7 · p. 10
• Finite rollout estimates become more reliable with larger sampling budgets, but increase evaluation cost and do not correct biased model distributions.Future work should develop more efficient and reliable rollout-based estimators.Source: §7 · pp. 10–11
• Controlled, visually parseable scenarios isolate stochastic future modeling but leave longer-horizon, interactive, and embodied environments for future study.The paper also calls for scaling the benchmark to interactive settings and training models to learn calibrated distributions over futures across physical states.Source: §7 · pp. 10–11

8 CONCLUSION

• Across eleven current video generators, no model consistently matches reference probabilities while recovering valid futures, and finite sampling or human disagreement on clear outcomes does not explain the gap.PAWBench evaluates probabilistic alignment through repeated video rollouts.Source: §8 · p. 11
• Model distributions do not reliably track causal changes in the physical process, according to controlled interventions.Source: §8 · p. 11
• Language can request individual futures, coupled noise can broaden finite-budget exploration, and fine-tuning can shift outcome frequencies, but none reliably recovers scene-conditioned future distributions.Source: §8 · p. 11
• Plausible, diverse, or controllable rollouts alone do not establish probabilistically aligned world modeling across different physical states.Source: §8 · p. 11

A.1 SCENARIO DEFINITION AND TASK TAXONOMY

• Each PAWBench scenario fixes a source image, an atomic-intervention action prompt, a finite set of valid terminal outcomes, outcome-readout criteria, and any independently derivable reference distribution before model evaluation.Source: §A.1 · p. 16

A.1.1 PAW-CALIBRATION TASKS

• PAW-Calibration fixes reference distribution q before evaluation, deriving it from sector proportions, combinatorial counts, physical symmetry, or visible causal state rather than model outputs or assumed uniformity.Source: §A.1.1 · p. 16
• Scenarios without a defensible reference distribution from the initial observation and action are assigned to PAW-Coverage or discarded; PAW-Calibration tests whether rollouts allocate mass according to q.Source: §A.1.1 · p. 16

A.1.2 PAW-COVERAGE TASKS

• PAW-Coverage tests recovery of enumerated valid outcomes across repeated rollouts without assuming equal probabilities or requiring justified reference probabilities.PAWBench contains 50 scenarios, evenly split between PAW-Calibration and PAW-Coverage, across eight stochastic mechanism groups.Source: §A.1.2 · p. 16, Figure 9 · p. 16

A.2 SCENARIO CONSTRUCTION AND QUALITY REVIEW

• Source images are manually selected to show the physical mechanism and action target, avoid hidden initial conditions, and support distinguishable terminal outcomes.Candidates with obscured mechanisms, ambiguous initial states, or competing action targets are rejected.Source: §A.2 · p. 16
• Scenarios fix a source image, atomic action, terminal outcomes, readout criteria, and—only when independently justified—a PAW-Calibration reference distribution.Quality review checks mechanism visibility, action specificity, distinguishable outcomes, and ambiguity before model evaluation; unsuitable scenarios are revised or discarded.Source: §A.2 · pp. 16–17

B.1 PAWEVAL OUTCOME READOUT

• PAWEval applies a frozen scene-specific rubric to assign each rollout an in-schema terminal label or the shared readout-failure label ⊥.More specific visible endpoints are mapped to the corresponding outcome category before aggregation.Source: §B.1 · p. 17
• Readout failures determine whether a scene passes the outcome-readout gate and affect SPR, but are excluded from the conditional distribution over readable, in-schema outcomes.A readable endpoint outside the schema counts as a readout failure.Source: §B.1 · p. 18
• The separate trustworthiness audit flags action execution, physical process, and object continuity without changing terminal labels, TVD, Coverage, or SPR.For example, an implausible rotation ending clearly on blue retains the blue outcome label while the trajectory is flagged by the audit.Source: §B.2 · p. 18, §B.1 · p. 18, Table 7 · p. 18

B.2 AUXILIARY TRUSTWORTHINESS DIAGNOSTICS

• Physical-process errors are the largest share of trustworthiness failures overall, while action-execution and object-continuity errors remain substantial across mechanism groups.These flags describe how readable endpoints are reached and do not change PAWBench scores.Source: §B.2 · p. 18

B.3 METRIC COMPUTATION AND AGGREGATION

• A scene passes the readout gate when at least 20 of 50 rollouts have readable, in-schema outcomes; otherwise it receives no conditional alignment score.This is equivalent to requiring no more than 30 readout failures.Source: §B.3 · p. 18
• PAW-Calibration measures valid-only conditional total variation distance from each scenario’s reference distribution, reporting 100 × TVD; lower values indicate closer probability-mass alignment.Readout failures are excluded from the conditional outcome distribution.Source: §B.3 · p. 20
• PAW-Coverage measures recovery of the enumerable valid outcome set, reports coverage as a percentage, and fixes K = 50 for primary comparisons; Fig. 5 tests K = 100.Higher coverage means repeated rollouts recover a broader set of valid terminal outcomes.Source: §B.3 · p. 20
• Mechanism-group scores average scene-level conditional metrics over passing scenes, while track Avg. averages all passing scenes; Scene Pass Rate uses the full 25-scene roster.Conditional scores and pass rate should be read together, and PAW-Calibration and PAW-Coverage remain separate rather than forming one ranking.Source: §B.3 · p. 20

B.4 FINITE-SAMPLE EFFECTS ON CALIBRATION

• 31.2 observed mean TVD exceeded the 8.33 matched-reference mean; 99% of simulated means were below 9.22, and every generator exceeded its own baseline’s 99th percentile.Alternative aggregation and a stricter minimum-readout threshold preserve the conclusion; the analysis covers conditional Calibration TVD, not Coverage or readout failures.Source: §B.4 · p. 20, Figure 12 · p. 21, Table 8 · p. 21

B.5 FULL CAUSAL AND NON-CAUSAL CONTROLS

• Eight controlled pairs compare response direction across eleven video generators and five direct VLM future samplers, not cross-modality performance.Each system receives K = 50 samples per scene; causal variants change the transition and reference distribution, while non-causal variants preserve them.Source: §B.5 · p. 21, Figure 13 · p. 22

D INTERVENTION EXPERIMENTS

• Section 5 probes whether future distributions can be shaped through language, initial noise, or the model’s training distribution.The appendix records systems, prompt interfaces, sampling budgets, and readout protocols for these interventions.Source: D INTERVENTION EXPERIMENTS · p. 24, Table 10 · p. 25

D.1 LANGUAGE-SIDE DIAGNOSTICS AND INTERVENTIONS

• Language-side tests separate predicting an outcome distribution from producing a requested outcome, using repeated VLM predictions, Prompt Engineering, and Oracle PE.VLM predictions are projected into scene-specific outcomes; invalid or unmapped responses count toward a gate that fails above 30 of 50 queries. Oracle PE schedules 50 requests per scene from reference probabilities for Calibration or evenly across valid outcomes for Coverage.Source: §D.1 · pp. 25–26
• Oracle PE lowered mean Calibration TVD and raised mean Coverage for all four generators, but only 37.6–58.1% of rollouts reached the requested outcome.PE raised Calibration TVD for all four generators and improved Coverage for only two; repeated VLM predictions remained misaligned with reference distributions.Source: §D.1 · p. 26

D.2 COUPLED NOISE SAMPLING

• C2C couples groups of five initial-noise samples across ten groups, yielding 50 rollouts per model–scene pair while preserving each rollout’s standard-Gaussian marginal.The matched IID condition uses 50 independent noises; source image, prompt, generator, generation settings, readout gate, and metric denominator are held fixed.Source: §D.2 · pp. 26–27
• Because C2C preserves each rollout’s marginal noise, it leaves the learned one-rollout conditional law unchanged; improved coverage would indicate finite-budget exploration, not better calibration.The probe asks whether a finite IID gallery misses futures the generator can already realize.Source: §D.2 · p. 27

D.3 TRAINING-DISTRIBUTION INTERVENTION

• Five 2,000-example datasets vary only the left/right outcome ratio across 0/100, 20/80, 50/50, 80/20, and 100/0, with a shared neutral motion prompt.Each direction’s source videos are repeated uniformly; the interior mixtures retain both futures, while the endpoints serve as boundary conditions.Source: §D.3 · p. 27
• Each mixture adapts the same Wan2.2 I2V-A14B base using paired rank-32 LoRA adapters for its high- and low-noise experts.Adapters use 832×480, 49-frame clips, a 10−4 learning rate, and 2,000 updates; the original model is the unadapted reference.Source: §D.3 · p. 27
• Evaluation uses 50 matched seeds per model–scene pair on upright and left-leaning pencils, whose reference distributions are 50/50 and 100/0, respectively.Dose–response comparisons use the five LoRA models; Base is shown only as unadapted behavior because its output geometry and diffusion-profile metadata differ.Source: §D.3 · p. 27
• Across three interior training mixtures, conditional left-fall frequency rises in both scenes, but the models recover neither exact ratios nor scene-specific probabilities.The scenes have different reference distributions (50/50 and 100/0), yet respond similarly.Source: Table 11 · p. 27, §D.3 · p. 27

E QUALITATIVE EXAMPLES AND FAILURE CASES

• Thirteen case cards pair rollout strips with scene and generation details to show how PAWEval reads terminal outcomes; these qualitative examples do not enter aggregate metrics.They complement the benchmark’s aggregate results and illustrate why plausible-looking videos may not complete the intended physical process.Source: E QUALITATIVE EXAMPLES AND FAILURE CASES · p. 28
• The cards include action-execution failures, object-continuity breaks, inconsistent trajectories, apparatus instability, and questionable but readable clips, alongside successful trials.Examples include a spinner with inconsistent motion and a ball that settles on a divider rather than entering a bin.Source: E QUALITATIVE EXAMPLES AND FAILURE CASES · p. 28, Figure 20 · p. 29, Figure 21 · p. 29
• Other rollouts end in readable physical outcomes, including a coin heads-up, a die showing five, and a ball landing inside a cup.These examples show terminal outcomes that can be assessed alongside the failure cases.Source: E QUALITATIVE EXAMPLES AND FAILURE CASES · p. 28, Figure 18 · p. 28, Figure 19 · p. 28