PaperlayerView exact revision on arXiv

Source-linked AI summary

PAI-Bench: A Comprehensive Benchmark For Physical AI

Fengzhe Zhou, Jiannan Huang, Jialuo Li, Deva Ramanan, Humphrey Shi

arXiv:2512.01989v1cs.CV

TL;DR

Evidence remains limited on whether MLLMs and video generative models can handle the perception and prediction demands of Physical AI. PAI-Bench evaluates those capabilities across three video tracks using real-world cases, and finds that visual fidelity can coexist with weak physical consistency while MLLMs lag human performance. The authors conclude that Physical AI remains nascent and largely unsolved.

  • Problem

    Evidence is limited on MLLMs' effectiveness in specialized Physical AI applications and on whether VGMs can predict complex interactions and obey physical laws.

    Source: §1 · p. 2

  • Method

    PAI-Bench evaluates Physical AI perception and prediction through video generation, conditional generation, and video understanding tracks grounded in real-world data.

    Source: §1 · p. 2

  • Results

    VGMs often achieve high visual fidelity but fail to maintain physical consistency, while MLLMs show a significant deficit against human baselines in physical common-sense reasoning and embodied forecasting.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The findings indicate that Physical AI remains nascent and largely unsolved, and the authors hope PAI-Bench will catalyze research toward models that internalize physics for robust action.

    Source: §6 · p. 11

    The discussion notes that artifacts have limited impact on quantitative rankings, while methods that encourage risk-taking in motion generation without compromising visual quality remain a future direction.

    Source: E. Discussion and Future Work · p. 27

Abstract

from arXiv · show

Physical AI aims to develop models that can perceive and predict real-world dynamics; yet, the extent to which current multi-modal large language models and video generative models support these abilities is insufficiently understood. We introduce Physical AI Bench (PAI-Bench), a unified and comprehensive benchmark that evaluates perception and prediction capabilities across video generation, conditional video generation, and video understanding, comprising 2,808 real-world cases with task-aligned metrics designed to capture physical plausibility and domain-specific reasoning. Our study provides a systematic assessment of recent models and shows that video generative models, despite strong visual fidelity, often struggle to maintain physically coherent dynamics, while multi-modal large language models exhibit limited performance in forecasting and causal interpretation. These observations suggest that current systems are still at an early stage in handling the perceptual and predictive demands of Physical AI. In summary, PAI-Bench establishes a realistic foundation for evaluating Physical AI and highlights key gaps that future systems must address.

1. Introduction

PAI-Bench addresses the limited evaluation of perception and prediction for Physical AI with a unified benchmark grounded in physically meaningful tasks and real-world video. Its evaluation finds that strong visual fidelity does not ensure physical coherence, while MLLMs still lag human performance in Physical AI tasks.

1. Introduction

  • The benchmark addresses limited evidence about MLLMs in specialized Physical AI applications and about whether video generative models can model complex interactions and physical laws.Existing assessments of MLLMs emphasize abstract reasoning and simple daily-life perception, while VGM benchmarks have focused more on aesthetics and temporal consistency.Source: §1 · p. 2
  • PAI-Bench evaluates video generation, conditional generation, and video understanding through three tracks targeting plausibility, control-signal conformance, and Physical AI video understanding.The tracks are PAI-Bench-G, PAI-Bench-C, and PAI-Bench-U, respectively.Source: §1 · p. 2
  • All three tracks use real-world captures and cover subdomains including autonomous vehicles, robotics, and ego-view scenes across 2,808 cases.The benchmark grounds evaluation in physically meaningful tasks and real-world data.Source: §1 · p. 2
  • Leading video generative models can achieve high visual fidelity yet fail to follow basic physical laws or model complex real-world dynamics, while powerful MLLMs lag human performance.The authors conclude that the prediction and perception capabilities required for Physical AI remain nascent.Source: §1 · p. 2
  • The paper presents PAI-Bench as a unified benchmark for video generation, conditional video generation, and video understanding, with task-aligned metrics and a large-scale evaluation of recent models.The evaluation covers 15 VGMs, four conditional VGMs across five control settings, and 16 MLLMs.Source: §1 · pp. 2–3

2. Related Work

Existing benchmarks often assess prediction and perception separately, with gaps in both physical plausibility and control-guided generation for VGMs and physically grounded video reasoning. PAI-Bench addresses these gaps with a unified framework focused on Physical AI.

2. Related Work

  • VGM benchmarks often assess visual quality and temporal consistency [38] [39] [101] or physical plausibility alone [11] [14] [57], while PAI-Bench-G combines domain and quality scores.The benchmark emphasizes Physical AI applications.Source: §2 · p. 3
  • No existing benchmark systematically tests controllability in multimodal-control-guided video synthesis, motivating PAI-Bench-C.The passage identifies depth maps as an example of a control signal.Source: §2 · p. 3
  • Physically grounded video reasoning remains limited in perception benchmarks [20] [27] [48] [59] [73], and PAI-Bench-U combines general video comprehension with dedicated physical challenges.Existing video-understanding benchmarks include general tasks such as action recognition [37] [49] [70] [72] [92] [102].Source: §2 · pp. 3–4
  • PAI-Bench focuses on Physical AI and evaluates prediction and perception in one framework for assessing model preparedness for real-world physical interaction.Its practical-domain coverage includes autonomous vehicles, industry, embodied AI, and ego-centric views.Source: §2 · p. 4, Table 1 · p. 4

3. PAI-Bench: A Comprehensive Benchmark For Physical AI

PAI-Bench organizes real-world Physical AI evaluation into generation, conditional generation, and video understanding tracks. Each track uses task-specific metrics or questions to assess physical plausibility, control, or reasoning.

3.1. PAI-Bench-G: Video Generation

  • PAI-Bench-G assesses generated-video quality and physical plausibility using Quality Score and a Domain Score based on answers to curated physical and semantic questions.Its curated collection contains 1,044 video-prompt pairs and 5,636 QA pairs across six domains.Source: §3.1 · pp. 4–5

3.2. PAI-Bench-C: Conditional Video Generation

  • PAI-Bench-C evaluates whether generated videos follow blur, edge, depth, and segmentation controls, alongside visual quality and diversity.The track uses 600 videos sampled from robotics, autonomous-driving, and ego-centric datasets.Source: §3.2 · p. 5

3.3. PAI-Bench-U: Physical Video Understanding

  • PAI-Bench-U covers physical common-sense reasoning about space, time, and the physical world, plus embodied reasoning about action effects and physical constraints.Its curated questions cover task completion, next-plausible actions, and action affordance.Source: §3.3 · pp. 6–7

4. Experiments

Experiments assess generation, conditional control, and video understanding. They find a gap between visual quality and physical plausibility, benefits from multi-signal conditioning, and substantial remaining limits in model understanding.

4.1. PAI-Bench-G

  • VGMs score competitively with source videos on Quality Score, but every evaluated model falls short of the real-world source-video baseline on Domain Score.The authors identify adherence to fundamental physical laws as a continuing challenge.Source: Table 3 · p. 8, §4.1 · p. 8

4.2. PAI-Bench-C

  • For both Cosmos-Transfer models, the All control-signal condition achieves the highest Quality Score, while segmentation control yields the lowest Mask mIoU.The authors hypothesize that temporally inconsistent segmentation masks provide unreliable supervision.Source: §4.2 · p. 9, Table 4 · p. 9

4.3. PAI-Bench-U

  • With zero video frames, performance falls to random guessing, while the substantial gap between one and 32 frames indicates that tasks require temporal context.These tests support the authors’ claim that PAI-Bench-U neutralizes language priors and static-frame shortcuts.Source: §4.3 · p. 10, Figure 7 · p. 10
  • All evaluated MLLMs trail the human accuracy of 93.2, and proprietary models do not uniformly outperform open-source models.Qwen3-VL-235B achieves higher accuracy than GPT-5.Source: §4.3 · p. 10, Table 5 · p. 10
  • Text-only thinking slightly degrades Qwen3-VL performance, especially on embodied reasoning, whereas GPT-5 with medium reasoning effort shows gains using textual and visual thinking.The authors argue that visual thinking needs to capture fine-grained details for subsequent reasoning to be grounded.Source: Table 6 · p. 10, §4.3 · p. 11

5. Further Analysis and Discussion

Across generation and understanding, the authors find that Physical AI remains largely unsolved, with low absolute performance and little differentiation between leading proprietary and open-source models.

  • Leading open-source VGMs approach Veo3 on the reported comparison, while Qwen3-VL-235B exceeds GPT-5; the authors identify either a data gap or a failure to learn Physical AI capabilities.They conclude that the field remains nascent.Source: §5 · p. 11

6. Conclusion

PAI-Bench evaluates perception and prediction across three video tracks and establishes a baseline for Physical AI. Its results show continuing physical-consistency and human-performance gaps.

6. Conclusion

  • PAI-Bench evaluates video generation, conditional control, and physical understanding, finding that VGMs often lack physical consistency and MLLMs trail human baselines in physical reasoning and embodied forecasting.The authors conclude that Physical AI remains nascent and largely unsolved.Source: §6 · p. 11

A.1. Data Source

PAI-Bench-G draws on established open-source benchmarks and web-sourced videos. Its quality metrics assess visual and temporal properties, while the Domain Score evaluates domain-specific physical and semantic constraints.

A.1. Data Source

  • Data curation combines established open-source benchmarks with videos collected from YouTube and stock-footage platforms.The open-source sources include EgoExo4D, HowTo100M, Physics-IQ, WISA-80K, Agibot, BridgeDatav2, and Open X-Embodiment.Source: §A.1 · p. 20

A.2. Details of Quality Score Calculation

  • Motion Smoothness assesses physical plausibility and temporal coherence by comparing omitted odd-indexed frames with frame-interpolated reconstructions.The frame difference is measured as MAE and the final score is normalized to [0, 1], with higher values indicating smoother motion dynamics.Source: §A.2 · p. 20
  • Imaging Quality evaluates frame-level distortions such as overexposure, noise, and blur using MUSIQ scores averaged after normalization.Aesthetic Quality is described as measuring composition, color harmony, and artistic quality with the LAION aesthetic predictor.Source: §A.2 · pp. 20–21
  • Quality measures include semantic and stylistic video–prompt alignment, reference-image subject and background consistency, and subject and background stability across frames.The measures use ViCLIP, DINO, DreamSim, and CLIP features, respectively.Source: §A.2 · pp. 20–21

A.3. Details of Domain Score Calculation

  • The Domain Score measures compliance with domain-specific physical and semantic constraints using Qwen3-VL-235B-A22B-Instruct responses checked against expected answers.Evaluation uniformly samples frames at 2 fps and uses greedy decoding for deterministic inference.Source: §A.3 · p. 21

A.4. Details about Experiments

The experiments specify generation settings for evaluated video generative models and use human pairwise judgments to assess video quality and physical plausibility. Rankings are computed separately for these dimensions and jointly for an overall score.

  • Generation configurations report each evaluated VGM’s checkpoint, spatial resolution, frame count, and FPS.The configurations are listed in Table 7.Source: §A.4 · p. 22
  • Human annotators compare two generated videos given a text prompt and reference image, judging video quality and physical plausibility.For each dimension, choices are A Better, B Better, Both Good, or Both Bad.Source: §A.4 · p. 22
  • ELO ratings use an initial rating of 1000 and K of 32, with separate trackers for video quality and physical plausibility.The overall rating applies ELO updates to the average of the two preference scores; detailed results are reported in Table 8.Source: §A.4 · p. 22

A.5. Qualitative Visualization

The appendix presents qualitative examples from PAI-Bench-G and PAI-Bench-C and details PAI-Bench-C metrics for control fidelity, visual quality, and generation diversity. Its fidelity measures compare generated videos with reference controls across multiple representation spaces.

A.5. Qualitative Visualization

  • PAI-Bench-G visualizations cover autonomous vehicles, robotics, industrial settings, common-sense reasoning, human activity, and physical dynamics.The examples are provided in assets 11–16, respectively.Source: §A.5 · p. 22

B. PAI-Bench-C Score Calculation Details

  • PAI-Bench-C assesses control fidelity, visual quality, and generative diversity under identical conditions.The appendix describes a unified extraction and fidelity-operator formulation and scoring pipeline.Source: B. PAI-Bench-C Score Calculation Details · p. 23

B.1. Control Fidelity

  • Control fidelity compares generated videos with reference signals in low-frequency structure, edges, depth geometry, and semantic segmentation.The measures are blurred-image SSIM, Canny-edge F1, depth si-RMSE, and segmentation mIoU; segmentation matches below IoU 0.1 are filtered.Source: §B.1 · p. 23

B.2. Visual Quality

  • Visual quality is measured with the reference-free DOVER-technical video-quality metric, whose higher values indicate better clarity and stability.The score is averaged across the dataset.Source: §B.2 · p. 23

B.3. Generation Diversity

  • Generation diversity is measured by averaging pairwise LPIPS distances among outputs under identical conditioning.Higher Diversity-LPIPS values indicate greater diversity and reduced mode collapse.Source: §B.3 · p. 24

B.4. Qualitative Visualization

  • PAI-Bench-C qualitative examples show test cases and generated results from autonomous-vehicle, robotics, and human domains.The examples are illustrated in Figures 17–19, respectively.Source: §B.4 · p. 24

C.1. Detailed Data Curation Process

The embodied-reasoning curation standardizes question templates and action granularity to ground evaluation in visual input and make next-action predictions less ambiguous. It draws on clips from robotics, assistance, and autonomous-vehicle datasets, with task-specific questions.

  • Question templates are standardized to ground reasoning in visual input rather than textual biases, and action labels follow a hierarchy of atomic actions, coarse subtasks, and dataset-specific goals.The action taxonomy is attributed to [13].Source: §C.1 · p. 24
  • Figure 9 plots PAI-Bench-U accuracy against the number of uniformly sampled input frames and marks random-guess baselines.The supplied passage describes the figure’s comparison but does not state its outcome.Source: Figure 9 · p. 25
  • RoboVQA and proprietary autonomous-vehicle questions test task completion and affordance, with the vehicle set also probing likely next actions.RoboVQA uses 101 validation clips; the proprietary set contains 100 videos with diverse lateral and longitudinal behaviors and agent interactions.Source: §C.1 · p. 25
  • RoboFail examples assess affordance and task-completion reasoning under complex temporal dynamics, physical constraints, and nuanced conditions beyond simple perception mismatches.The benchmark uses 100 manually annotated examples.Source: §C.1 · p. 25
  • BridgeData V2 questions use task instructions and visual history to elicit the most plausible immediate next action from 100 validation clips.Source: §C.1 · p. 25
  • AgiBot and HoloAssist questions ask models to predict the next subtask from task or coarse-action context, with distractors drawn from other subtasks or fine-grained annotations.Each source contributes 100 samples or QA pairs.Source: §C.1 · p. 25

C.2. Detailed Experiment Setup

The appendix details the MLLM evaluation setup and examines how frame sampling, human baselines, and normalized radar plots inform interpretation of PAI-Bench results.

C.2. Detailed Experiment Setup

  • PAI-Bench-U uses a unified prompt that asks MLLMs to provide a rationale before an answer, after receiving temporally sampled frames and a question.Figure 10 shows the standardized inference prompt template.Source: §C.2 · pp. 25–26, Figure 10 · p. 26
  • Proprietary models use default inference settings, GPT-5 is evaluated at medium and minimal reasoning effort, and open-source models use official defaults via vLLM.Table 9 lists the evaluated proprietary model versions.Source: §C.2 · p. 26, Table 9 · p. 26

C.3. Discussion on Input Frame Count

  • The frame-count experiment tests GPT-5 and three Qwen3-VL models using 0, 1, 2, 4, 8, 16, and 32 uniformly sampled frames.Source: §C.3 · p. 26
  • Performance gains from increasing sampled frames below 8 saturate beyond 8, with dense sampling potentially obscuring motion cues and reasoning limits constraining further gains.The authors attribute the plateau to a reasoning bottleneck and possible temporal ambiguity from highly similar adjacent frames.Source: §C.3 · p. 26

C.4. User Study Details

  • A user study collects human answers to benchmark questions conditioned on video clips, with aggregated accuracy serving as the model-comparison baseline.Figure 21 depicts the annotation interface.Source: §C.4 · p. 27

D. PAI-Bench Leaderboard Visualization

  • Radar plots present model performance across PAI-Bench dimensions using normalized scores to support comparisons across heterogeneous metrics.The visualizations are shown in Figure 22.Source: D. PAI-Bench Leaderboard Visualization · p. 27

E. Discussion and Future Work

The discussion identifies limitations in consistency metrics, video generation, and automated judging. It points to more capable encoders and safer ways to encourage dynamic motion as directions for further work.

Enhancing Metric Robustness with Advanced Encoders

  • Consistency metrics depend on video–text encoder capabilities; VCLIP’s limits on lengthy, detailed prompts constrain sensitivity, motivating next-generation encoders for finer-grained evaluation.Source: E. Discussion and Future Work · p. 27

Addressing Conservative Generation Strategies

  • Some video models favor static fidelity over complex, high-risk dynamics to avoid artifacts, motivating methods that encourage motion without sacrificing visual quality.The authors say this conservative strategy has limited impact on quantitative rankings.Source: E. Discussion and Future Work · p. 27

Necessity and Imperfection of MLLM Judges

  • MLLM judges have inherent limits in interpreting video temporal dynamics, but remain current best practice for scalable semantic assessment because effective alternatives are lacking.Source: E. Discussion and Future Work · p. 27
Loading 2512.01989v1…