PaperlayerView exact revision on arXiv

Source-linked AI summary

PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements

Kaiwei Chen, Moran Li, Zhenye Gan, Yabiao Wang, Chengjie Wang, Kexin Zhang, Yong Liu, Jiangning Zhang

arXiv:2610.26711v1cs.CV

TL;DR

Video generation can look convincing while remaining physically implausible, and existing evaluations provide limited object-level, quantitative measurement. PVBench combines specialized physical measurements, physical-law checks, and MLLM scoring where direct measurement is difficult. Its numerical scores are more discriminative and more consistent with human judgments than MLLM semantic scores.

  • Problem

    Existing evaluations rely on semantic judgments or camera-centric geometry and offer limited computable, testable physical quantities for generated videos.

    Source: §1 · p. 1, §5 · p. 9

  • Method

    PVBench evaluates 46 dimensions using specialized vision models to measure physical quantities, checks measurements against physical laws, and uses MLLM scoring for hard-to-quantify dimensions.

    Source: §1 · p. 2, §3.3.2 · p. 5, §5 · p. 9

  • Results

    Numerical scores correlate with human ratings at Spearman 0.84 versus 0.14 for MLLM scores and are more discriminative.

    Source: §4.4 · p. 9, §5 · p. 10

  • Takeaways & Limitations

    Numerical physical-quantity scores provide a more discriminative and human-consistent basis for quantifiable video evaluation.

    Source: §5 · p. 10

    The evaluation largely checks consistency with physical laws and lacks ground-truth references for most dimensions.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent video generation models produce visually compelling outputs, yet their physical plausibility remains poorly measured. Physical plausibility asks whether generated objects obey geometry, kinematics, dynamics, and conservation laws. Existing benchmarks rely on coarse visual-language model judgments or target embodied world models rather than video generation itself. We introduce PVBench, a physical video benchmark that quantitatively measures the plausibility of generated videos across 46 dimensions organized into seven domains, spanning: i) instruction and semantic fidelity, ii) behavioral and causal consistency, iii) perceptual and temporal quality, iv) geometric and spatial consistency, v) material properties and field states, vi) kinematic and spatiotemporal persistence, vii) and mechanical plausibility with conservation laws. PVBench comprises 600 carefully crafted prompts, of which 30% involve physical embodied agents such as robot arms, humanoids, quadrupeds, and multi-agent systems. For challenging physically tractable dimensions, it pairs each with specialized vision models that derive verifiable, physically grounded measurements, such as object-level 6D pose, metric scale, surface normals, kinematics, and numerical conservation-law residuals. It thus avoids relying solely on subjective scoring, while supporting both text-to-video and image-to-video generation. Finally, we validate the agreement between automatic metrics and human judgments through human evaluation. Meanwhile, we evaluate mainstream open-source and closed-source models and present several insights. By releasing the benchmark, prompts, and calibration protocol, we aim to foster physically grounded video generation.

1 Introduction

Visually convincing generated videos can still violate physical laws, undermining their use in downstream applications. PVBench addresses limitations in existing evaluations with a systematic benchmark and physical-quantity-based measurements, and its physical scores generally agree with human judgments.

Motivation

  • Generated videos can look real while objects clip through one another, vanish, or appear at incorrect scales, undermining downstream usability.The paper identifies embodied intelligence, content creation, and simulation as affected applications.Source: §1 · p. 1

Limitations of existing evaluation

  • Existing benchmarks miss object-level physical quantities: semantic yes/no judgments do not localize or quantify violations, while coverage often remains narrow.Metric scale and 6D pose are identified as important for manipulation, reconstruction, motion planning, and simulation initialization.Source: §1 · pp. 1–2

PVBench

  • PVBench organizes 46 dimensions into seven domains and uses paired same-scene T2V/I2V prompts, with 30% involving embodied-agent scenarios.Specialized vision models measure physical quantities and test them against physical laws; MLLM scoring supplements dimensions that cannot be directly computed.Source: §1 · p. 2, Figure 1 · p. 2

Findings

  • Across six T2V and six I2V model configurations, physical-layer scores are substantially lower than semantic and perceptual scores, while physical dimensions generally agree with human judgments.The authors plan to release the code and prompt set.Source: §1 · p. 2

2 Related Work

Video foundation models are diversifying in architecture, input modalities, and interactive capabilities. Video-generation benchmarks assess visual quality, while physical-plausibility benchmarks use judgments or limited quantitative proxies; PVBench instead measures numerical physical quantities against physical laws.

2.1 Video Foundation Models

  • Video foundation models span diffusion and flow-matching approaches, with newer systems exploring unified multimodal inputs, efficient long-video generation, and autoregressive generation.Some video generators are also framed as world models, including systems for physical AI and action-controlled environments.Source: §2.1 · pp. 2–3

2.2 Benchmarks for Video Generation

  • General-purpose video suites emphasize semantic and perceptual metrics, while physical-plausibility benchmarks often rely on semantic judgments or camera-centric geometry.VBench-2.0 adds physics checks, but these remain coarse; Physics-IQ uses proxy motion measures, and WorldScore reconstructs camera geometry.Source: §2.2 · p. 3
  • PVBench addresses these evaluation limits by extracting numerical physical quantities with specialized models and checking them against physical laws.The cited related work describes prior quantitative approaches as measuring only camera-centric geometry.Source: §2.2 · p. 3

3 Methodology

PVBench organizes video evaluation around semantic, perceptual, and physical measurements, using paired prompts and specialized models to make physical plausibility quantitatively testable. Its design combines numerical metrics with human validation and compares observable object states against prompt-conditioned constraints and physical laws.

3.2 Prompt Design for Physical Plausibility Evaluation

  • PVBench contains 600 prompt groups pairing T2V and I2V variants, with 30% involving embodied-agent scenarios and additional hybrid-dimension prompts and visual confounders.Prompts are designed around individual dimensions, dimension pairs, and difficult scenes involving agents, occlusion, or camera motion.Source: §3.2 · pp. 3–4

3.3.1 Hierarchical Metrics of PVBench

  • The semantic layer covers 14 dimensions across instruction fidelity and behavioral, causal, and commonsense consistency, while the perceptual layer assesses frame quality and temporal dynamics across seven dimensions.Semantic checks include instruction following and whether actions and timing unfold as instructed; perceptual checks include subject consistency, motion smoothness, flicker, and imaging quality.Source: §3.3.1 · p. 5
  • PVBench organizes 46 dimensions into semantic, perceptual, and physical layers, with 25 physical dimensions spanning four domains of object properties, relations, and dynamics.The physical domains cover geometry and spatial consistency, material properties and field states, kinematics and persistence, and mechanics and conservation laws.Source: §3.1 · p. 3, §3.3.1 · pp. 4–5
  • Table 1 maps specialized evaluation models to the dimension codes they cover, showing which models supply measurements for different parts of the framework.Dimension names are abbreviated in the table, with their full correspondence provided in the appendix.Source: §3.3.1 · p. 5, Table 1 · p. 5

3.3.2 From Specialized Models to Physical Quantities

  • For 17 of the 25 physical dimensions, specialized models produce numerical measurements; four use pure MLLM scoring and four combine specialized measurement with MLLM judgment.The framework uses MLLM-assisted judgment when quantities cannot be directly quantified.Source: §3.3.2 · p. 5

3.3.3 Score Mapping

  • Physical quantities are mapped to scores from 1 to 5 using manually set reasonable-range thresholds, then averaged equally across dimensions before conversion to a percentile scale.Human validation is used to check the fit of the thresholds to human judgments.Source: §3.3.3 · p. 5

3.4 Human Alignment For Benchmark Validation

  • Human alignment was assessed by annotating five prompts per physical dimension for three T2V models, with each sample independently scored by 10 annotators.Annotators received metric definitions, guidelines, and examples before scoring; the scores were averaged.Source: §3.4 · pp. 5–6

3.5 Novelty: From Semantic Judgment to Physical-State Evaluation

  • PVBench evaluates observable physical states against prompt-conditioned constraints and laws, rather than relying only on holistic judgments of whether a video looks plausible.Its evaluation unit is object states and physical laws, in contrast with prior approaches based on semantic judgments, binary outcomes, proxy metrics, or camera-centric geometry.Source: §3.5 · p. 6

4 Experiments

The experiments compare six T2V and six I2V configurations across PVBench’s seven domains, finding a persistent gap between physical and semantic/perceptual scores. Supplementary analyses test difficult scenes, human alignment, first-frame priors, and evaluator choices.

4.1 Experimental Setup

  • Six mainstream models were evaluated in T2V and I2V configurations on all 600 prompt groups under a unified 720p, 10-second, fixed-seed protocol.Evaluator versions and weights were fixed, and final scores were reported on a percentile scale.Source: §4.1 · p. 6

4.2 Main Results and Cross-Layer Model Behavior

  • SANA-Video 2.0 T2V scores 58.2 in perception and temporal quality (PTQ) but 39.0 in behavioral and causal commonsense (BC), showing that perceptual quality and behavioral plausibility can diverge.Across models, semantic and perceptual layers distinguish capabilities but do not yield a completely consistent ranking.Source: §4.2 · pp. 6–7
  • Physical-layer scores are markedly lower than semantic- and perceptual-layer scores; Seedance 2.5 performs best overall, followed by MiniMax H3.Table 2 reports percentile scores across seven domains, and Figure 4 visualizes the cross-layer results.Source: §4.2 · p. 6, Table 2 · p. 6, Figure 4 · p. 7
  • Geometric space and kinematic persistence score higher than material field states and mechanical conservation, and continuity or completed actions do not ensure mechanical-law compliance.Mechanical conservation remains weak across models, including collisions, forces, and energy changes.Source: §4.2 · p. 7

4.3 Supplementary Analysis

  • On the high-difficulty subset, 5 of 6 T2V models drop by an average relative 7.1%, and 5 of 6 I2V models drop by 10.9%.The subset combines embodied-agent, occlusion, visual-clutter, and camera-motion prompts; automatic scores correlate with human ratings at Spearman 0.772 and Kendall 0.643 overall.Source: §4.3 · p. 7, Table 3 · p. 8, Table 4 · p. 8
  • For MiniMax H3, I2V improves material field states by 3.4 percentage points but drops 3.7 points in geometric space and 5.2 in mechanical conservation versus T2V.Kinematic persistence is roughly flat, so first-frame conditioning does not improve physical dimensions uniformly.Source: §4.3 · p. 8, Table 5 · p. 8

4.4 Ablation Studies for PVBench

  • Numerical physical scores outperform direct MLLM scores in human alignment (Spearman 0.84 vs. 0.14) and discriminability; VLM-backend pairwise Spearman correlations range from 0.71 to 0.84.The calibrated thresholds 0.10 for SAM3 detection confidence and 0.1% for trajectory object area perform best in coverage and human-score correlation.Source: §4.4 · p. 9, Table 7 · p. 9, Table 8 · p. 9
  • Removing mechanical conservation raises the six-T2V average physical score from 38.5 to 43.4, while removing kinematic persistence lowers it to 33.6.Removing material field states raises the average to 41.9; removing geometric space lowers it to 35.0, supporting separate reporting of the four domains.Source: §4.4 · p. 9

5 Conclusion and Future Work

PVBench evaluates video-generation plausibility through numerical physical measurements checked against physical laws, using MLLM scoring for hard-to-quantify dimensions. Its numerical scores are reported as more discriminative than MLLM semantic scores and highly consistent with human judgments, while the evaluation still has limited ground-truth coverage and relies on specialized vision models.

5 Conclusion and Future Work

  • PVBench extracts numerical measurements with specialized vision models, checks them against physical laws, pairs same-scene T2V/I2V prompts, and uses MLLM scoring for hard-to-quantify dimensions.Source: §5 · p. 9
  • Numerical physical-quantity scores are more discriminative than MLLM semantic scores and highly consistent with human judgments.The authors present these findings as enabling more reliable and quantifiable evaluation.Source: §5 · p. 10
  • Material and phase-change dimensions still use MLLM scoring, and a full ablation of specialized evaluators is left for future work.The study ablates only the VLM backend; evaluation depends on specialized vision models and skips a small number of unobservable samples.Source: §5 · p. 10
  • The evaluation largely checks consistency with physical laws and lacks ground-truth references for all but a few dimensions.The authors plan to build references for conditional image-to-video models using physics engines and other simulation tools.Source: §5 · p. 10

AI Use Statement

The authors used generative AI to polish the writing and assist in constructing the prompt set; they reviewed and verified all AI-assisted content and take responsibility for the final work.

AI Use Statement

  • The authors used generative AI for writing polish and prompt-set construction, then reviewed and verified the assisted content.Source: AI Use Statement · p. 11

Ethics Statement

The authors state that they follow the ICLR Code of Ethics and describe the human study's annotation protocol.

Ethics Statement

  • The human study validated automatic evaluators, with trained annotators independently scoring each sample and scores averaged across 10 annotators.The annotation followed the protocol described in the paper.Source: Ethics Statement · p. 11

Reproducibility Statement

The paper describes its evaluation and reproducibility materials, fixes model versions, weights, and random seeds, and says it will release code, prompts, and calibration protocols. The appendix outlines benchmark specifications, results, scoring, generation protocols, costs, and reproducibility details.

Reproducibility Statement

  • Evaluation fixes model versions, weights, and random seeds, and the authors will release code, the prompt set, and calibration protocol.The experiments cover 12 model configurations and 600 prompt groups—about 7,200 generated videos—and were evaluated on a single H20 GPU.Source: Reproducibility Statement · p. 11

Appendix Overview

  • The appendices document the 46 dimensions and their prior-benchmark coverage, per-dimension human alignment, and automatic scores for all 12 configurations.Source: Appendix Overview · p. 15
  • Appendix materials also specify scoring models, paradigms, quantities, formulas, thresholds, semantic and perceptual criteria, prompt-group statistics, generation configurations, costs, and reproducibility details.Source: Appendix Overview · p. 15

A Evaluation Dimensions: Description and Master Table

PVBench reports scores over the dimensions applicable to each modality and organizes its full evaluation set by domain and assessment layer. Its physical-quantity coverage extends across three classes, unlike prior benchmarks, which cover at most a small subset, mainly through semantic judgment.

A.1 Dimension Description

  • T2V evaluates 45 dimensions and I2V evaluates all 46 because IF-10 applies only to I2V with a reference image; aggregation equally weights evaluated dimensions.The master table lists each dimension's ID, name, domain (D1–D7), and assessment layer (L1–L3).Source: §A.1 · p. 15, §A.2 · p. 15

A.2 Master Table of Evaluation Dimensions

  • The master table organizes all 46 evaluation dimensions by ID, name, domain, and assessment layer.Per-dimension physical-layer human-alignment results and scores across 12 configurations are reported in separate appendix tables.Source: §A.2 · p. 15

A.3 Physical Dimension Coverage vs. Prior Work

  • PVBench provides numerical evaluation across all three classes of physical dimensions, whereas prior benchmarks cover at most a small subset, mainly through semantic judgment.The comparison covers the 25 physical dimensions.Source: §A.3 · p. 15

B Per-Dimension Human Alignment Results

Automatic physical-dimension scores correlate with human scores overall, though dimensions involving detection, tracking, or long-term consistency have lower correlations in part because samples are skipped or unobservable.

Per-Dimension Human Alignment Results

  • ρ = 0.772 and τ = 0.643 are the overall Spearman and Kendall correlations with human scores across the 25 physical dimensions.The table reports per-dimension correlations using three merged T2V models and 15 paired samples per dimension.Source: B Per-Dimension Human Alignment Results · p. 15, Table A4 · p. 18
  • Dimensions relying on detection, tracking, or long-term consistency have lower human-score correlations, mainly because some samples are skipped or unobservable.The per-dimension table covers 25 physical dimensions, with three T2V models and 15 samples per dimension.Source: B Per-Dimension Human Alignment Results · p. 15, Table A4 · p. 18

C Per-Dimension Automatic Score Tables

This appendix reports automatic scores across PVBench’s dimensions, describes how semantic, perceptual, and physical dimensions are evaluated and mapped to scores, and summarizes paired T2V/I2V differences and prompt-set composition. It also presents statistics on the 600 prompt groups and specifies the shared generation and evaluation protocol.

C Per-Dimension Automatic Score Tables

  • Tables A6–A12 report mean scores for 12 configurations across all 46 dimensions, with physical-layer dimensions generally scoring lower than semantic and perceptual layers.Skipped or errored samples are excluded from dimension means and recorded as not applicable.Source: C Per-Dimension Automatic Score Tables · p. 19

C.1 Paired T2V/I2V Differences Across Models

  • Table A13 reports I2V-minus-T2V paired differences across seven domains for six model configurations.The supplied passage gives no difference values.Source: §C.1 · p. 19, Table A13 · p. 21

D Per-Dimension Scoring Principles

  • Physical dimensions use specialized-model measurements, measurements followed by MLLM judgment, or pure MLLM semantic scoring; D1–D3 use semantic anchors.The physical dimensions’ quantities, formulas, criteria, and score thresholds are detailed in later subsections.Source: D Per-Dimension Scoring Principles · p. 21

D.1 Specialized Models and Dimension/Paradigm Mapping

  • Geometric and kinematic dimensions mainly use specialized models for numerical measurements, while Qwen3.8-27B semantically scores material, field, instruction-following, and commonsense dimensions.Light/shadow and object permanence are measured by specialized models and then judged by an MLLM.Source: §D.1 · p. 21

D.2 Physical Quantities, Formulas, and Criteria

  • The 25 physical dimensions combine measurements such as depth ordering, surface-normal consistency, motion, and conservation residuals with semantic judgments of physical plausibility.For example, depth ordering compares occluded objects with their occluders, while force-effects checks assess prompt-specific mechanics and use metric scale to compare acceleration with gravity.Source: §D.2 · pp. 21–24

D.3 Mapping Physical Quantities to 1 to 5 Scores

  • Physical quantities are mapped to scores from 1 to 5 using direction-specific thresholds, with reconstruction-noise baselines relaxed where needed.Without simulation ground truth, geometric dimensions use measured quantities, while several semantic, scale, and pose dimensions receive direct Qwen3.8-27B scores.Source: §D.3 · p. 24

D.4 Scoring Principles for Semantic and Perceptual Dimensions

  • D1–D3 receive semantic or numerical-anchor scores, including counterfactual trajectory checks for action causality and perception-model measures of consistency, smoothness, flicker, and image quality.Action causality combines tracked 3D displacement and direction effects with semantic verification.Source: §D.4 · pp. 24–26

E Additional Prompt-Set Statistics

  • The dataset contains 600 prompt groups, each with T2V and I2V versions; Table A16 summarizes their composition and embodied-scenario share.The supplied passage does not state the composition breakdown or share value.Source: E Additional Prompt-Set Statistics · p. 26
  • Table A16 summarizes category counts, embodied samples, and modality pairing across the 600 prompt groups.Source: Table A16 · p. 26

F Generation Protocol Details

  • Models generate all 600 prompt groups for text-to-video and image-to-video under a shared initial scene and standardized output specification.Image-to-video uses a GPT-image-2-synthesized first frame; outputs are 720p, about 10 seconds at 24 fps, with a fixed seed, then resampled for evaluators.Source: F Generation Protocol Details · pp. 26–27

G Evaluation Cost and Reproducibility

The evaluation fixes specialized-model versions and weights and the random seed, and covers 12 model configurations and 600 prompt groups.

Evaluation Cost and Reproducibility

  • Evaluation fixes specialized-model versions and weights and the random seed for reproducibility.The experiments cover 12 model configurations and 600 prompt groups, about 7,200 videos, and run the evaluation models on a single H20 GPU with 95 GB of memory.Source: G Evaluation Cost and Reproducibility · p. 27
  • Evaluation covers 12 model configurations and 600 prompt groups, or about 7,200 videos.Per-video runtime and total GPU hours vary with hardware and evaluated dimensions.Source: G Evaluation Cost and Reproducibility · p. 27
  • Evaluation models run on a single H20 GPU with 95 GB of memory, while runtime and total GPU hours vary with hardware and evaluated dimensions.Further statistics, runtime environment, and model-weight versions are to be provided with the code release.Source: G Evaluation Cost and Reproducibility · p. 27

Related papers

Loading 2610.26711v1…