PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantitative Video World Model Evaluation for Geometric-Consistency

Jiaxin Wu, Yihao Pi, Yinling Zhang, Yuheng Li, Xueyan Zou

arXiv:2605.15185v1cs.CVcs.AI

TL;DR

Evaluating whether generative videos obey 3D geometry remains difficult because existing metrics are subjective, geometry-agnostic, or insufficiently diagnostic. PDI-Bench audits generated videos with explicit projective-geometry residuals and identifies subtle spatial inconsistencies across stress-test scenarios.

  • Problem

    Existing video evaluations provide insufficiently transparent and geometry-specific evidence for verifying 3D physical consistency in generated videos.

  • Method

    PDI-Bench lifts 2D pixel dynamics into 3D geometric reasoning through a Target-Uplift-Anchor pipeline and Perspective Distortion Index.

  • Results

    PDI-Bench identifies subtle spatial inconsistencies across five geometric stress-test scenarios in six state-of-the-art video models.

  • Takeaways & Limitations

    PDI-Bench provides a geometric yardstick that complements semantic-based metrics for evaluating generative video world models.

  • Takeaways & Limitations

    PDI-Bench depends on off-the-shelf perception tools, assumes rigid bodies, and cannot fully disentangle 3D rotation from axial translation using monocular cues.

Abstract

from arXiv · show

Generative video models are increasingly studied as implicit world models, yet evaluating whether they produce physically plausible 3D structure and motion remains challenging. Most existing video evaluation pipelines rely heavily on human judgment or learned graders, which can be subjective and weakly diagnostic for geometric failures. We introduce PDI-Bench (Perspective Distortion Index), a quantitative framework for auditing geometric coherence in generated videos. Given a generated clip, we obtain object-centric observations via segmentation and point tracking (e.g., SAM 2, MegaSaM, and CoTracker3), lift them to 3D world-space coordinates via monocular reconstruction, and compute a set of projective-geometry residuals capturing three failure dimensions: scale-depth alignment, 3D motion consistency, and 3D structural rigidity. To support systematic evaluation, we build PDI-Dataset, covering diverse scenarios designed to stress these geometric constraints. Across state-of-the-art video generators, PDI reveals consistent geometry-specific failure modes that are not captured by common perceptual metrics, and provides a diagnostic signal for progress toward physically grounded video generation and physical world model. Our code and dataset can be found at https://pdi-bench.github.io/.

1. Introduction

PDI-Bench addresses the gap between visual plausibility and geometric rigor in generative video by evaluating 2D motion under 3D projective-geometry constraints. It extracts 3D geometric evidence from video pixels to provide diagnostic measurements of geometric hallucinations and physical inconsistency.

  • Motivation: Generative video models can appear visually realistic while violating spatial scale, perspective, and Euclidean consistency through artifacts such as volume breathing.These failures expose a gap between visual plausibility and geometric rigor.
  • Motivation: Existing evaluators rely on subjective human or LMM judgments, or emphasize high-level physical categories without rigorous geometric verification.Automated suites such as PhysBench and WorldBench are described as largely geometry-agnostic.
  • Framework: PDI-Bench evaluates scale-depth alignment, 3D motion consistency, and 3D structural rigidity as three orthogonal geometric metrics.The metrics target scale hallucinations, non-physical world-space motion, and non-Euclidean deformations.
  • Framework: Its Target-Uplift-Anchor workflow combines SAM 2 segmentation with MegaSaM monocular reconstruction to lift 2D observations into a unified 3D environment.The workflow uses precise 2D boundaries and scale priors, reconstructed world-coordinate pointmaps, and camera poses for camera-motion-invariant evaluation.
  • Contributions: The paper introduces PDI-Bench, the Perspective Distortion Index, and PDI-Dataset for quantitative geometric-consistency evaluation across six state-of-the-art open- and closed-source video models.The contributions frame 3D lifting as a way to decouple kinematic errors from geometric alignment and detect physical hallucinations.

2. Related Work

Related work spans video generation, perceptual and prompt-following evaluation, physical-consistency benchmarks, and visual perception tools for 3D auditing. PDI-Bench targets the unresolved distinction between genuine 3D simulation and replicated 2D statistical patterns with a geometric yardstick.

  • Video Generation and World Model: Diffusion-based video generators have progressed from temporal extensions of 2D diffusion to high-fidelity, long-duration sequences with complex semantic alignment.Examples include Sora and Wan2.2.
  • Video Quality Assessment and Prompt Following: FVD and IS prioritize frame-level textural and aesthetic quality, while CLIP-based scores assess text-video semantic alignment and prompt following.VBench and T2V-CompBench extend evaluation to dynamic quality, temporal consistency, and compositional prompt adherence.
  • Physical Consistency and Scene Understanding: Existing physical-consistency benchmarks probe high-level semantic plausibility through macroscopic laws, interaction dynamics, dynamic constraints, physical constants, and world-generation capabilities.The cited benchmarks include VIDEOPHY, WorldModelBench, PhysBench, PhyGenBench, WorldBench, and WorldScore.
  • Physical Consistency and Scene Understanding: MEt3R and TRAJAN assess implicit consistency through learned representations, using latent 3D pointmaps and trajectory autoencoders to analyze multi-view consistency and point-track anomalies.These approaches are described as closer to PDI-Bench than high-level semantic benchmarks.
  • Visual Perception and 3D Reconstruction: Foundation models enable 3D physical auditing through SAM 2 for video segmentation, CoTracker3 for dense temporal correspondence, and MegaSaM for semantic-aware monocular 3D reconstruction.Together, these tools support object isolation, pixel-level tracking, and reconstruction from dynamic monocular sequences.

3. Method

PDI-Bench bridges 2D pixel dynamics and 3D physical regularities through a Target-Uplift-Anchor pipeline that extracts semantic masks, world-space geometry, and structural trajectories. It synthesizes these observations into the Perspective Distortion Index (PDI), which measures scale-depth, trajectory, and rigidity failures with distinct residuals.

  • Targeting: The Target-Uplift-Anchor pipeline uses Florence-2 and SAM 2 to identify an auditing subject and propagate temporal binary masks.The masks provide per-frame pixel heights and spatial boundaries for subsequent geometric analysis.
  • Uplifting: MegaSaM reconstructs depth, focal length, and camera poses, then projects pixels into unified 3D world-space pointmaps that decouple object motion from camera ego-motion.This lifting step yields P_world ∈ R^T×H×W×3 for downstream kinematic and structural audits.
  • Anchoring: CoTracker3 seeds and filters 2D anchor trajectories within the initial subject mask, then lifts tracked anchors through MegaSaM pointmaps into 3D trajectories.Anchors are initialized using a SIFT → Shi-Tomasi → uniform-grid cascade and filtered by visibility and displacement-jump thresholds.
  • Index construction: PDI combines three orthogonal weighted residuals—scale, trajectory, and rigidity—to quantify distinct geometric failure modes.Scale and trajectory residuals use RMSE, whereas rigidity uses a temporal mean of MAD to avoid an additional second-order penalty.
  • Scale-depth alignment: The scale residual tests the pinhole-camera invariant h_tZ_t for non-physical scaling, using the median of the first five frames as a stable baseline and final RMSE as severity.The formulation targets artifacts such as volume breathing while using per-frame pixel height and median object depth.
  • Motion and rigidity: Trajectory auditing decomposes 3D kinematic anomalies into acceleration-magnitude and directional-continuity penalties, while rigidity auditing measures temporal consistency of internal 3D pairwise distances.Directional penalties activate only above a speed threshold, and rigidity uses filtered anchor pairs plus MAD-based distance ratios to resist monocular scale drift.

4. Experiments

Experiments use PDI-Bench to evaluate geometric consistency across diverse generated-video scenarios and validate the perception-to-3D pipeline. Results expose a physics gap between visual realism and geometric consistency, with Seedance 2.0 and CogVideoX-3 strongest overall and scenario-specific failures across other models.

  • Benchmark and pipeline: PDI-Dataset contains 183 video sequences from 28 textual prompts, spanning five geometric challenges: longitudinal convergence, dynamic tracking, biological motion, curved motion, and partial occlusion.The benchmark includes 15 high-quality real-world Ground Truth videos and evaluates open- and closed-source generators.
  • Benchmark and pipeline: The multi-stage Fidelity Guard independently audits SAM 2 segmentation, CoTracker3 trajectories, and MegaSaM 3D pointmaps before PDI synthesis.MegaSaM reconstruction is checked through cross-frame reprojection using thresholds for Coverage, MAE, and L2 distance.
  • Overall results: GT videos achieve a PDI Score of 0.1206 and scale residual 𝜖_s= 0.0660, establishing the physical-world lower bound for generative auditing.Lower PDI values indicate higher physical realism.
  • Overall results: Seedance 2.0 records 0.0% outliers and an 89.3% MathPass rate, while CogVideoX-3 reaches 𝜖_t= 0.2033 and 𝜖_r= 0.2065.These results identify Seedance 2.0 and CogVideoX-3 as the generative models with highest fidelity to physical laws.
  • Overall results: Visually renowned Sora and HunyuanVideo exhibit scale residuals exceeding 1.67, standard deviations above 1.7, and 14.3% outlier ratios, revealing severe scale hallucinations.These failures violate the perspective invariant h⋅Z= const.
  • Scenario and human-alignment analysis: Scenario analyses show distinct failures: Sora reaches PDI= 2.13 in curved motion, HunyuanVideo reaches PDI= 2.41 under partial occlusion, and CogVideoX-3 reaches PDI= 0.16 in dynamic tracking.Seedance 2.0 leads biological motion at PDI= 0.25, while expert rankings perfectly align with automated PDI scores (𝜌= 1.0).

5. Case Study: Diagnosing Autoregressive Extrapolation

The case study evaluates Self-Forcing autoregressive extrapolation from 81 to 129 frames across 28 prompts, revealing stable 3D kinematics alongside severe geometric scale-depth collapse. PDI further localizes failures to partial occlusion and nonlinear motion, demonstrating fine-grained diagnostic value.

  • Evaluation setup: Self-Forcing extrapolates 81-frame trained sequences to 129 frames across 28 prompts to assess 3D consistency beyond the training context window.The evaluation uses the Wan2.1-T2V-1.3B architecture.
  • Kinematic Success vs. Geometric Collapse: The overall 3D Kinematic Trajectory error remains stable, with a mean of 0.3170 across scenarios.Rolling KV caching and Self-Forcing mitigate high-frequency spatial jitter and unnatural reversals during long autoregressive generation.
  • Kinematic Success vs. Geometric Collapse: The overall Scale-Depth Alignment error surges to 2.8583, indicating severe scale hallucination beyond the 81-frame training horizon.Objects expand or contract independently of depth as the model loses spatial memory of their original 3D volume.
  • Vulnerability to Spatial Disconnects: Scale error is 0.1515 during Dynamic Tracking but PDI reaches 2.7570 under Partial Occlusion, where object permanence fails after re-emergence.The contrast shows sensitivity to continuous visual context and loss of structural anchors when subjects are temporarily obscured.
  • Vulnerability to Spatial Disconnects: Curved Motion maintains trajectory but raises the rigidity residual to 0.4177, indicating a mild jello effect during nonlinear displacement.The result links rotational complexity to residual geometric deformation despite preserved kinematics.

6. Conclusion · A. Additional Experimental Details

PDI-Bench provides a quantitative framework for auditing geometric consistency in generative video world models by translating 2D pixel dynamics into verifiable 3D reasoning. The framework also has limitations tied to perception failures, rigid-body assumptions, and difficult 3D rotation disentanglement.

  • 6. Conclusion: PDI-Bench audits perspective and scale consistency in generative video world models with a quantitative framework.It is presented as a geometric yardstick for evaluating generated videos.
  • 6. Conclusion: The Target-Uplift-Anchor workflow translates 2D pixel dynamics into verifiable 3D geometric reasoning.
  • 6. Conclusion: PDI identifies subtle video hallucinations including “volume breathing” and “skating” effects through a multi-modal metric.
  • 6. Conclusion: PDI-Bench accuracy depends on off-the-shelf perception tools, and failed 3D uplifting forces less precise 2D proxies.This limitation is especially relevant in degraded videos.
  • 6. Conclusion: PDI-Bench’s geometric invariants assume rigid bodies, limiting theoretical suitability for highly non-rigid or amorphous subjects.
  • A. Additional Experimental Details: Disentangling complex 3D rotation remains a further limitation of the framework.The supplied passage truncates before specifying the full limitation.

A.1. PDI-Dataset Construction … B. Computing Resources

The benchmark combines 183 real-world and synthetic videos spanning five geometric stress-test scenarios, standardizes evaluation through reconstruction-aware PDI scoring and robust reporting, and uses a multi-GPU pipeline for processing. Its protocol includes explicit prompt design, quality-conditioned residual weighting, repeated evaluation, and GT-anchored normalization.

  • A.1. PDI-Dataset Construction: PDI-Dataset contains 183 video sequences partitioned into real-world and synthetic subsets.The dataset includes 15 real-world clips and 168 synthetic videos.
  • A.1. PDI-Dataset Construction: The synthetic subset contains 168 videos generated from 28 text prompts by six representative video models.The models are Wan 2.2, HunyuanVideo, CogVideoX-3, Seedance 2.0-Fast, Sora, and Veo 3.1-Fast.
  • A.1. PDI-Dataset Construction: The 28 prompts target five projective-geometric scenarios: Longitudinal Convergence, Dynamic Tracking, Biological Motion, Curved Motion, and Partial Occlusion.Examples include a car on a straight road, a bus entering a roundabout, and a tractor turning at a field boundary.
  • A.2. Complete PDI-Bench Prompt Gallery: PDI synthesizes three orthogonal physical residuals using weights that sum to 1.When MegaSAM reconstruction passes ground-plane SVD and reprojection checks, the weights are (w1, w2, w3) = (0.4, 0.4, 0.2).
  • A.3. Evaluation Protocol: For each model and scenario, PDI reports median scores and 95% Bootstrap Confidence Intervals across valid sequences.Videos are evaluated multiple times with different random seeds for anchor sampling, and the resulting scores are averaged.
  • A.3. Evaluation Protocol: GT-Anchored Normalization defines a physics-perfect baseline from real-world Ground Truth residual medians and MADs, then maps robust Z-scores to [0, 100].The protocol reports both raw PDI-Error residuals and normalized PDI-Score values.
  • B. Computing Resources: Experiments run on a Linux workstation equipped with NVIDIA GeForce RTX 3090 GPUs, each providing 24,576 MiB of GDDR6X video memory.The GPUs use compute capability 8.6 and the Ampere architecture.
  • B. Computing Resources: Multi-GPU parallelization assigns each video sequence to one GPU worker for sequential SAM2 segmentation, Co-Tracker tracking, and Mega-SAM reconstruction.This worker-based pipeline processes the benchmark across GPUs.

C. Proofs of Main Theorems … D.2. Perspective Coupling via Angular Alignment

The paper formally derives the geometric identities behind PDI and implements diagnostics for scale-depth alignment, motion consistency, rigidity, longitudinal convergence, and perspective coupling. These diagnostics use projective invariants, 3D observations, fallback strategies, vanishing-point geometry, and angular residuals to identify physically inconsistent motion and structure.

  • C. Proofs of Main Theorems / C.1. Geometric Invariants of Perspective Projection: PDI’s formal derivations establish a perspective invariant: for rigid objects undergoing linear motion, projected height divided by radial displacement remains constant.The derivation uses pinhole projection and the assumptions that physical height H and horizontal coordinate X remain constant.
  • C.2. Square-Inverse Scaling Law: Physically correct perspective scaling follows a square-inverse depth law, whereas linear-in-1/Z behavior produces systematic velocity distortion.The result follows by differentiating h = fH/Z and applying a first-order approximation for nearby depths.
  • C.3. Spatio-temporal Coupling in the Image Plane: For fixed 3D velocities, pinhole projection makes pixel displacement quadratically suppressed with depth.The derivation considers a 3D displacement (ΔX, ΔY, ΔZ) and recovers the image-plane motion expression used in the main paper.
  • C.4. Metric Implementation Details: The Scale metric measures violations of the perspective relation h_t z_t ≈ const using SAM2 pixel heights and Mega-SAM aligned depths.Residuals are computed framewise in log space and aggregated into the Scale component of PDI.
  • C.4. Metric Implementation Details: The Traj metric audits world-coordinate centroid motion using robust 3D pooling, temporal median denoising with k = 3, and velocity-based residuals.Invalid foreground estimates inherit the last valid centroid, and motion is parameterized with Δt = 1/fps.
  • Structural Rigidity (Rigidity).: Rigidity is measured through a prioritized hierarchy: 3D pairwise rigidity, 3D height stability, and degraded 2D pairwise consistency.The active strategy directly supplies PDI’s third component; unavailable evidence can yield a rigidity value of 0.
  • D. Geometric Invariants and Perspective Coupling in Longitudinal Motion / D.1. Diagnostic Perspective Analysis for Longitudinal Convergence: For longitudinal motion, Generalized H-VP Homogeneity couples centroid convergence toward a vanishing point with scale reduction through inverse depth.The distance-based probe is used when a stable vanishing point can be estimated, while near-transverse motion uses the angular formulation.
  • D.2. Perspective Coupling via Angular Alignment: The angular coupling residual Δθ compares motion-derived and geometry-derived vanishing-point directions, ranging from 0 for aligned motion to 1 for contradictory perspective.Cosine similarity is preferred over Euclidean distance because it remains robust when transverse-motion vanishing points approach infinity.

D.3. Mathematical Derivation of H-VP Homogeneity · E. Qualitative Visualizations

The derivation establishes that a rigid object moving linearly in world space projects toward a vanishing point, with scale changes and radial convergence linearly coupled in 3D-consistent videos.

  • D.3. Mathematical Derivation of H-VP Homogeneity: The proof assumes a 3D point on a rigid object undergoing linear motion in world space.
  • D.3. Mathematical Derivation of H-VP Homogeneity: Under pinhole projection, the object’s image coordinates converge to the vanishing point as its depth parameter s_t approaches infinity.
  • D.3. Mathematical Derivation of H-VP Homogeneity: The derivation expresses the horizontal centroid-to-vanishing-point distance using the projected point coordinates.
  • D.3. Mathematical Derivation of H-VP Homogeneity: Substituting the linearly evolving X_t and Z_t coordinates simplifies the numerator of the distance expression.
  • D.3. Mathematical Derivation of H-VP Homogeneity: The 2D Euclidean distance to the vanishing point follows Dist(p_t, VP) = f⋅C/Z_t, where C is determined by initial geometry and motion direction.The relationship is stated across the distance formula and its definition of C.
  • D.3. Mathematical Derivation of H-VP Homogeneity: Dividing the initial state by the state at time t yields the homogeneity relation.
  • D.3. Mathematical Derivation of H-VP Homogeneity: The derivation concludes that scale changes and radial convergence must be linearly coupled in 3D-consistent videos.

E.1. Visualizing the PDI Computation Pipeline

The PDI pipeline transparently converts a generated video into semantic masks, coherent 3D world-space observations, and tracked 3D anchors before auditing scale, trajectory, and rigidity. It aggregates these errors into a weighted PDI score and produces diagnostics for specific physical failure modes.

  • Pipeline execution: Florence-2 prompts SAM 2 to segment and propagate temporal masks, isolating the target and providing frame-wise pixel heights and 2D boundaries.The masks are denoted {M_t}^T_{t=1}.
  • Pipeline execution: MegaSaM estimates depth, focal length, and camera poses, then projects pixels into unified 3D world coordinates to decouple object motion from camera ego-motion.It produces world-space pointmaps P_world ∈ R^T×H×W×3.
  • Pipeline execution: CoTracker3 seeds anchors inside the initial mask and lifts their 2D trajectories into structurally meaningful 3D trajectories using MegaSaM pointmaps.This transforms tracked pixel-space motion into 3D anchor coordinates.
  • Three-Dimensional Geometric Auditing: The geometric audit measures scale-depth invariance, Newtonian trajectory consistency, and rigidity through temporal variation in 3D anchor-pair distances.The three residuals penalize violations of h·Z = const, abrupt acceleration or reversals, and “volume breathing” or “Jello effect” artifacts.
  • Score Synthesis & Final Report: The final PDI combines dimension errors with weights w1 = 0.4, w2 = 0.4, and w3 = 0.2; lower scores indicate superior physical rationale.The pipeline also generates a terminal audit report with fine-grained diagnostics for specific failure modes.

F. Qualitative Analysis of Failure Modes

Qualitative analysis identifies distinct geometric failure modes that persist in state-of-the-art generative models despite PDI-Bench’s quantitative scores. Veo 3.1-Fast can abruptly invert or morph object features, creating temporal and structural incoherence that challenges SfM-based reconstruction.

  • Overview: Qualitative examination reveals distinct geometric failure modes persisting in state-of-the-art generative models.These failures remain visible alongside PDI-Bench’s quantitative evaluation.
  • Temporal Orientation and Structural Incoherence: Veo 3.1-Fast exhibits abrupt inversion or morphing of object features, swapping or deforming frontal and rear characteristics mid-sequence.The failure is described as temporal orientation and structural incoherence.
  • Temporal Orientation and Structural Incoherence: Instantaneous topology changes can occur within a duration shorter than the SfM reconstruction window, causing pointmaps to struggle with tracking them.The rapid transition undermines the ability of the underlying pointmaps to follow the change.
  • Qualitative Examples: Figure 10 depicts motion-induced blur and feature morphing in a flying bird, plus a bus whose appearance and 3D structure become decoupled across frames.The bus example illustrates temporal inconsistencies in both appearance and reconstructed structure.

G. Limitations and Future Directions · H. Broader Impacts

PDI-Bench remains limited by perception-module failures and rigid-body assumptions, motivating learned, richer-physics, and multimodal extensions. Its broader impacts include improved evaluation and interpretability alongside risks from increasingly convincing synthetic media, underscoring the need for safeguards and responsible deployment.

  • G. Limitations and Future Directions: PDI-Bench depends on SAM 2, CoTracker3, and MegaSAM, whose failures in low-texture or low-parallax regimes reduce the depth sensitivity of final scores.The framework falls back to 2D proxies and reconstruction-aware weighting when these modules fail.
  • G. Limitations and Future Directions: The core geometric invariants assume rigid bodies, although the rigidity residual partially compensates for this limitation.
  • G. Limitations and Future Directions: Future work could distill PDI-Bench into a lightweight neural assessor that approximates geometric residuals without explicit SfM for larger-scale or near real-time auditing.This direction integrates learned priors into the evaluation pipeline.
  • G. Limitations and Future Directions: Other extensions would cover fluid dynamics, articulated bodies, and multi-object interactions, while multi-view or multi-sensor inputs could reduce inherent limitations.Proposed inputs include stereo, depth sensors, and IMU traces; hybrid Lagrangian–Eulerian formulations are also suggested.
  • H. Broader Impacts: PDI-Bench may improve the reliability and interpretability of generative video systems by systematically evaluating geometric and physical plausibility.The benchmark could benefit applications including robotics, embodied AI, simulation, and digital content creation.
  • H. Broader Impacts: More physically realistic generated videos could enable misinformation, deceptive content, malicious manipulation, and harder synthetic-media detection in sensitive contexts.
  • H. Broader Impacts: Quantifying physical consistency and identifying model weaknesses may support trustworthy modeling, robustness analysis, and synthetic-media detection, alongside safeguards and responsible deployment.
Loading 2605.15185v1…