PaperlayerView exact revision on arXiv

Source-linked AI summary

Apple-$π$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

Runmao Yao, Kairui Hu, Yukang Cao, Ruisi Wang, Shulin Tian, Ziang Cao, Weichen Fan, Ziqi Huang, Yuhao Dong, Hao Li, Zhaoxi Chen, Zhongang Cai, Lei Yang, Ziwei Liu

arXiv:2607.16401v1cs.CV

TL;DR

Existing video benchmarks assess physical plausibility from outputs without testing whether models reason through governing laws. Apple-π introduces an auditable Perception–Formulation–Deduction benchmark, finding that current video models remain far from dependable law-grounded world models, with the best video model averaging 0.473.

  • Problem

    Existing benchmarks evaluate what video models output, not whether they identify physical quantities, invoke governing laws, and deduce dynamics law-groundedly.

  • Method

    Apple-π combines the 400-video Orchard dataset, a Perception–Formulation–Deduction protocol, and physics-law-grounded objective evaluation.

  • Results

    0.473 is the best video-model average score, while unified understanding-generation models substantially outperform video models across most tracks, especially Perception and Formulation.

  • Takeaways & Limitations

    Apple-π positions reliable physical intelligence as requiring explicit understanding alongside temporal generation, particularly for formulating laws, executing dynamics, and composing across law transitions.

  • Takeaways & Limitations

    Models can copy annotations while assigning their physical meaning incorrectly, leading to erroneous target states and deduction trajectories.

Abstract

from arXiv · show

Modern video generation models are increasingly hailed as emerging world models with an internalized grasp of physical law. Yet existing benchmarks largely evaluate physical plausibility only at the output level, without verifying whether the model arrives there through a faithful, law-grounded reasoning process. We introduce Apple-PI, the first benchmark that anchors video-model evaluation explicitly in physical laws. Apple-PI comprises three components. 1) Orchard: a dataset of 400 videos covering ten canonical tasks in classical mechanics. It separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization. 2) Benchmark Protocol: a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction. It uses chain-of-frames prompting on infographic-annotated first frames, treating the generated video as the model's visible reasoning trace. 3) Evaluation Suite: a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. This enables stage-resolved diagnosis of not only whether a model fails, but where it fails. Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators, with the best video model scoring only 0.473. Our stage-, pillar-, and source-resolved analyses further expose a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap. These findings position Apple-PI as a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence.

1 Introduction

Apple-π addresses the gap between visually plausible video outputs and verified physical reasoning by making video generation an auditable, law-grounded scientific reasoning task. Its benchmark diagnoses whether models perceive quantities, formulate governing laws, and deduce consistent future dynamics.

  • Motivation: Video models are increasingly viewed as emerging world models because they implicitly absorb geometry, motion, and causality and may produce chain-of-frames reasoning traces.This motivates evaluating not only what models generate, but how they arrive at those outputs.
  • Motivation: Existing benchmarks evaluate only model outputs, so they cannot determine whether success reflects physical-law reasoning or whether failure arose from perception, law identification, or deduction.This leaves unresolved whether video models genuinely reason about physical laws.
  • Benchmark Design: Apple-π converts Newton-style scientific reasoning into a three-stage protocol requiring models to perceive physical quantities, formulate governing laws, and deduce future dynamics.The protocol is designed to make law-grounded reasoning auditable through video generation.
  • Benchmark Design: Orchard contains 400 physical videos spanning ten canonical tasks, drawing on real-world recordings, internet footage, and high-fidelity simulators.The dataset is the benchmark’s video foundation.
  • Findings: Current video models possess useful physical priors but remain far from dependable law-grounded world models, since visually plausible motion may not reflect correctly grounded quantities or laws.Models can often generate plausible motion from an annotated first frame, but plausibility alone is insufficient evidence of grounded reasoning.
  • Benchmark Design: Apple-π explicitly anchors evaluation in physical laws and decomposes reasoning into diagnosable stages, unlike representative physical-intelligence benchmarks.Table 1 distinguishes real and simulated sources and marks support as full, partial, or unsupported.

2 Related Work

Related work frames video as a potential substrate for physical simulation, develops benchmarks for physical intelligence, and extends chain-of-thought reasoning from text and images to video. Existing efforts span video world models, understanding and generation benchmarks, and chain-of-frames reasoning.

  • Video World Models: Video world models have long pursued learning world models from video, with recent generation, interactive, and unified systems making video increasingly plausible for physical simulation.Sora renewed this vision by framing large-scale video generation as a path toward world simulation.
  • Benchmarking Physical Intelligence: Physical-intelligence benchmarks cover synthetic intuitive-physics probes, realistic physics question-answering suites, and generation evaluations based on human ratings or pixel-level metrics.The passage distinguishes understanding-side benchmarks from generation-side assessments such as VideoPhy and Physics-IQ.
  • Thinking with Video: Chain-of-thought prompting has progressed from text to images and most recently video, with chain-of-frames formalizing frame-by-frame generation as step-by-step reasoning.Subsequent work extends this direction across benchmarking, generation, and test-time scaling.

3 Apple-π

Apple-π is a law-grounded benchmark that evaluates video models through Orchard scenarios, a three-stage scientific-reasoning protocol, and hybrid subjective/objective metrics. Its design exposes where models fail across Perception, Formulation, and Deduction rather than judging only final physical plausibility.

  • Benchmark overview: Apple-π couples the Orchard dataset, a Perception–Formulation–Deduction protocol, and an evaluation suite combining MLLM-based subjective and physics-law-grounded objective metrics.The three components are designed to diagnose failures by reasoning stage.
  • Orchard: Orchard organizes 400 law-first cases around explicit classical-mechanics laws, physically specified conditions, and law-predicted motion for analyzable, reproducible evaluation.Its cases retain controlled real-world visual diversity across simulated, self-recorded, and Internet-sourced videos.
  • Orchard: Orchard separates single-law cases for controlled diagnosis from multi-law compositions that test whether models can chain laws handled individually.Examples include inclined-plane motion followed by projectile motion and circular motion followed by collision.
  • Benchmark protocol: The protocol uses five subtracks: Perception-Text, Perception-Graphic, Formulation-Text, Formulation-Graphic, and Deduction.All subtracks pair an infographic-annotated first frame with a chain-of-frames prompt, while generated video serves as the visible reasoning trace.
  • Benchmark protocol: Deduction generates complete law-consistent dynamics from the annotated first frame and evaluates the full sequence frame by frame against the law-predicted ground-truth trajectory.This stage directly tests whether the model produces physically correct evolution over time.
  • Evaluation suite: Apple-π scores every subtrack on a [0, 1] scale using MLLM-based subjective scoring for validity and format compliance alongside physics-law-grounded objective measures for law consistency.Objective measures include segmentation-IoU for predicted object regions and trajectory-based evaluation for Deduction.

4 Experiments

Experiments evaluate 11 models across 400 cases, five subtracks, and three rollouts, revealing that current video models remain weak law-grounded physical reasoners. Results expose progressive stage bottlenecks, limited multi-law and Sim-to-Real generalization, and failures in annotation semantics and visual-output control.

  • Overall Results: 0.473 is the best video-model average score, while GPT Image 2 and Nano Banana 2 reach 0.704 and 0.699 overall.The results indicate that plausible video synthesis alone does not ensure explicit quantity binding, law identification, or law-consistent dynamics.
  • Stage-wise Bottlenecks: Video-model scores decline progressively from Perception to Formulation to Deduction, indicating that visual cues and partial laws do not reliably transfer into temporal dynamics.Perception recovers entities, Formulation abstracts or instantiates laws, and Deduction rolls those laws forward through time.
  • Law Generalization: Multi-law cases are generally harder than Grav., Mom., and N1 because states produced by one law must initialize the next.This exposes weak compositional transfer of position, velocity, direction, or contact states across law transitions.
  • Source Generalization: Both video and unified models perform worse on real-world than simulated cases, although unified models retain higher absolute scores.The Sim-to-Real drop is attributed mainly to grounding and tracking failures under realistic visual conditions.
  • Failure Analysis: Annotation-semantic errors, OCR, segmentation, rendering, and instruction-following failures cause models to misbind physical meanings or produce invalid visual outputs.A recognized initial-velocity arrow can receive the wrong physical direction, corrupting both target-state formulation and deduction trajectories.

5 Conclusion · Appendix Contents

Apple-π is introduced as a law-grounded benchmark for testing whether video generation models achieve auditable physical intelligence beyond visual plausibility. Its Orchard-based evaluation anchors cases in classical-mechanics laws and separates reasoning into Perception, Formulation, and Deduction, while experiments find current models remain far from reliable law-grounded simulators.

  • 5 Conclusion: Apple-π evaluates whether video generation models move beyond visual physical plausibility toward auditable physical intelligence.
  • 5 Conclusion: The benchmark is built on Orchard and anchors each case in explicit classical-mechanics laws.
  • 5 Conclusion: Apple-π decomposes evaluation into Perception, Formulation, and Deduction.
  • 5 Conclusion: Experiments show that current video models acquire useful physical priors from large-scale video training.
  • 5 Conclusion: Despite these priors, current video models remain far from reliable law-grounded simulators.
  • 5 Conclusion: The conclusion frames Apple-π as a benchmark for diagnosing law-grounded physical intelligence rather than visual plausibility alone.

A Orchard Design Principles and Data Card · A.1 Design Principles

Orchard is designed as a law-first, diagnostic benchmark that separates single-law diagnosis from multi-law generalization while standardizing visual objects and spatially grounded initial conditions. Its infographic annotations specify starting conditions without supplying answers, requiring models to identify governing laws and deduce future trajectories.

  • A.1 Design Principles: Orchard begins each case with an explicit classical-mechanics principle, initial condition, and law-predicted trajectory rather than retrospectively labeling arbitrary videos.This law-first construction localizes model failures to Perception, Formulation, or Deduction.
  • A.1 Design Principles: Single-law cases isolate one dominant physical principle, enabling diagnosis with minimal confounding.The single-law branch is intended for clean attribution of failures to individual physical principles.
  • A.1 Design Principles: Multi-law cases compose two or more principles to test transfer of state variables from one law to the next.Examples include an inclined-plane stage followed by projectile motion and a curved-track stage followed by collision.
  • A.1 Design Principles: The benchmark standardizes dynamic objects across simulated, self-recorded, and Internet-sourced cases to four primitive solids: sphere, cube, cylinder, and cone.This controls object-specific semantic priors while supporting consistent centers, masks, contact surfaces, and size parameters.
  • A.1 Design Principles: Physical quantities are attached to visual referents, including mass to objects, friction to contact surfaces, angles to ramps, and velocity vectors to moving bodies.This spatial grounding supports reproducible law-grounded evaluation.
  • A.1 Design Principles: An infographic-style annotated first frame specifies initial conditions but not the answer, requiring models to identify the governing law and deduce future states or trajectories.The interface presents the physical setup while preserving the reasoning challenge.

A.2 Dataset Composition and Native Frame Rates … C Benchmark Protocol and Prompt Templates

Apple-π’s Orchard dataset combines simulator and real-world mechanics cases with source-specific ground truth, metadata, and quality controls. Its benchmark protocol uses annotated first frames and model-family-specific prompts to evaluate perception, formulation, and deduction outputs.

  • A.2 Dataset Composition and Native Frame Rates: 400 cases span ten canonical mechanics tasks, combining simulator, self-recorded, and Internet-sourced videos with source-specific ground-truth procedures and native frame rates.Simulator cases use programmatic physics-engine ground truth, whereas real-world cases use measurement-backed and human-reviewed annotations; native GT fps is stored per case.
  • A.3 Task Taxonomy: The taxonomy separates the first nine tasks as single-law diagnostics from multi-law compositions, with instances varying parameters such as initial velocity, friction, restitution, and gravity.Representative formulas define or verify cases, while task-specific parameters can vary across individual instances.
  • A.4 Released Metadata: Each case directory includes evaluation-critical native GT fps, physical duration, timestamps, available object masks, formula metadata, and target-time state.Source-specific fields are optional, but these evaluation-critical fields are always present.
  • B.1 Simulator Ground Truth: Simulator cases use deterministic Isaac Sim configurations and engine-derived RGB, masks, depth, poses, velocities, and contact events, with mass computed as m = ρV and shared visual-physics material keys.Configurations specify object, material, geometry, motion, environment, camera, and rendering parameters; closed-form solutions are used when available.
  • B.2 Real-World Ground Truth: Real-world cases are measurement-first: self-recorded clips use 30 fps, Internet clips retain native fps, and parameters rely on measurements, calibration cues, specifications, and human review.The stated goal is not to make real videos identical to simulated videos.
  • B.3 Mask and Tracking Annotation: Masks use semi-automatic propagation with human correction and identity checks, with engine-rendered masks canonical for simulators and human-refined masks canonical for real-world cases.These masks support graphic-subtrack IoU and Deduction trajectory metrics, including contact and collision-frame checks.
  • B.4 Formula Answer-Key Construction: Formula answer keys provide one governing formula plus confusing, unrelated, and fabricated alternatives, testing law selection rather than superficial symbol matching.The confusing distractor shares annotation symbols but does not govern the case, while the fabricated option is syntactically plausible but physically invalid.
  • B.5 Three-Pass Review Protocol: Every case receives one primary annotation pass and two independent reviews, with disagreements resolved through analytic backstops, senior arbitration, or domain-expert escalation.The review checks visual, temporal, mask, velocity, formula, and source metadata; simulator review corrections demonstrate active label-quality improvement, while agreement is measured with Cohen’s κ and ICC(2,1).

C.3 Protocol Ablations

Protocol ablations indicate that infographic annotations do not artificially inject solutions, while removing detailed output-format instructions mainly harms graphic-output tracks.

  • Protocol Ablations: The Text-Parameter ablation produces only a small average change when physical quantities are supplied as structured text instead of infographic overlays.This suggests infographic annotations are not a shortcut for solving the tasks.
  • Protocol Ablations: Infographic annotations provide a spatially grounded way to bind physical quantities to visual referents.
  • Protocol Ablations: The Concise-Prompt ablation removes detailed output-format instructions, with the main degradation appearing on graphic-output tracks.

D Evaluation Standardization Across Model Outputs … D.6 Invalid Outputs and Rollout Aggregation

The evaluation standardizes heterogeneous model outputs into a canonical packet, aligns comparisons in prompt-defined physical time, and scores all outputs under shared spatial, temporal, graphic, and rollout rules. This ensures model differences are evaluated consistently while invalid artifacts remain penalized rather than excluded.

  • D.1 Output Canonicalization: All model responses are converted into a canonical evaluation packet before scoring, despite differing formats, resolutions, frame rates, and durations.The canonicalization procedure is summarized in Table 12.
  • D.2 Time and FPS Alignment: Physical-time alignment uses case-specific requested duration and source-dependent GT frame rates, while generated frames are treated as trajectories over the requested interval.Raw provider duration is ignored; effective generated fps is decoded-frame count divided by requested physical duration.
  • D.3 Timestamp Grid: Deduction metrics compare generated and GT frames at canonical physical timestamps, including event-critical moments alongside uniform samples.Video outputs are densely decoded and sampled at the nearest generated frame for each timestamp.
  • D.4 Resolution Normalization: Pixel- and mask-level metrics resize every generated output directly to GT resolution using bilinear interpolation for RGB and nearest-neighbor interpolation for masks.The protocol neither crops nor pads, applying identical canvas handling across models.
  • D.5 Graphic-Subtrack Mask IoU: Perception-Graphic and Formulation-Graphic use objective mask IoU metrics after resolution normalization against object-only and target-time scene ground truth, respectively.For multi-object cases, masks use identity matching when available or Hungarian matching otherwise; missing objects receive zero IoU.
  • D.6 Invalid Outputs and Rollout Aggregation: Each model receives three independent rollouts per case and subtrack, whose scores are aggregated into a rollout mean.The score is defined over model, case, subtrack, and rollout indices.
  • D.6 Invalid Outputs and Rollout Aggregation: Undecodable rollouts and persistent judge or parser failures receive zero scores rather than being removed from the denominator.Transient judge or parser failures are retried before persistent failures are logged and penalized.

E Metric Definitions and Score Aggregation … F.1 Primary Judge

Apple-PI combines deterministic MLLM judging with objective comparisons against ground-truth masks, pixels, and velocities, then aggregates scores hierarchically across rollouts, cases, slices, and subtracks. Its Deduction score emphasizes physics accuracy while adapting judge prompts and score fusion to each model output format.

  • E.1 MLLM-Judge Rubrics: MLLM criteria use temperature 0, scores in [0, 1], applicability filtering, and renormalized group weights during aggregation.Undecodable responses receive zero for the corresponding subtrack after retries.
  • E.1.1 MLLM-Judge Prompt Templates: Deduction judging differs by output format: video models are assessed as full videos, whereas unified models are evaluated frame-by-frame on timestamped keyframes.Unified-model keyframe scores are averaged across requested timestamps, with object-position match serving as the motion-smoothness proxy.
  • E.1.1 MLLM-Judge Prompt Templates: Subtrack rubrics evaluate annotation fidelity, object preservation, symbolic laws, substitutions, target states, velocity attributes, visual quality, consistency, smoothness, and physics accuracy.Perception-Text, Perception-Graphic, Formulation-Text, and Formulation-Graphic use grouped criteria tailored to their artifacts.
  • E.3 Deduction Score Fusion: Deduction fuses integrity, fidelity, and physics groups with weights 0.20, 0.20, and 0.60, respectively.The larger physics weight prevents visual plausibility from compensating for physically incorrect dynamics.
  • E.2 Deduction Objective Metrics: Deduction objective metrics compare time-aligned generated and ground-truth frames using normalized PSNR, masked PSNR, motion-mask IoUs, and law-predicted velocity accuracy.Spatial IoU measures where motion occurs, Spatiotemporal IoU measures where and when, and Weighted Spatial IoU additionally captures motion frequency.
  • E.4 Hierarchical Aggregation: Scores are averaged from three rollouts to cases, cases to slices, and slices to models, while overall scores average across cases and subtracks.Pillar-wise and source-wise results are case-level means over restricted subsets rather than means of previously reported table cells.
  • F.1 Primary Judge: Gemini 3 Flash at temperature 0 is the canonical MLLM judge, while programmatic metrics are used whenever masks, pixels, or velocities can be directly compared with ground truth.The judge covers validity, format compliance, rendering, visual quality, object consistency, velocity-arrow interpretation, and high-level physics plausibility.

F.2 Open-Weights Cross-Check

An open-weights cross-check finds that leaderboard conclusions are robust across judges: Qwen3-VL and Gemini 3 Flash strongly agree on per-model averages and preserve quality-tier ordering. Remaining differences reflect judge leniency and error-propagation behavior rather than changed rankings.

  • Cross-check design: The cross-check evaluates seven representative generation models across all five subtracks with Qwen3-VL-30B-A3B-Instruct-FP8 as an open-weights judge.It targets ranking robustness rather than exact score equality because judges may calibrate the [0, 1] scale differently.
  • Agreement: The two judges strongly agree on per-model mean scores and preserve the same quality-tier ordering.Table 14 reports agreement between Gemini 3 Flash and Qwen3-VL.
  • Judge differences: Residual disagreement arises because Qwen3-VL can be more lenient on visually plausible law-violating motion, while Gemini 3 Flash more consistently propagates earlier errors downstream.Qwen3-VL may score rubric items more independently, whereas the primary judge can carry a wrong formula choice into later formula and substitution criteria.
  • Interpretation: Because tier-level rankings remain unchanged, Qwen3-VL serves as a reproducible robustness check while Gemini 3 Flash remains the primary judge.The preserved ranking supports treating the open-weights evaluation as a cross-check rather than a replacement.

G Experimental Details

The benchmark evaluates 11 models under a standardized setup spanning 400 cases, five subtracks, and three independent rollouts per model. Output formats vary by model family and subtrack, with videos or timestamped keyframes providing the evaluated artifacts.

  • Experimental Setup: 11 models—five video-generation and six unified understanding-generation models—are evaluated on 400 cases across five subtracks with three independent rollouts each.This yields 6000 responses per model.
  • Experimental Setup: Models use the same annotated first frame and subtrack prompt template within each output family, without model-specific prompt tuning.The setup standardizes inputs while preserving output-family differences.
  • Output Protocol: For non-Deduction subtracks, unified models generate answer artifacts while video models produce chain-of-frames clips evaluated by their last frame.For Deduction, video models generate full clips and unified models generate timestamped keyframes at evaluation timestamps.

H Limitations

Apple-π’s conclusions are bounded by its narrow physics, object, scene, and interface scopes, as well as imperfect evaluation and real-world ground-truth precision. Its temporal normalization further assumes that each generated clip represents the requested physical duration.

  • Physics scope: Apple-π evaluates only classical rigid-body mechanics, excluding fluids, thermodynamics, electromagnetism, deformable bodies, fracture, granular media, biological motion, and quantum phenomena.Its covered laws include gravity-driven motion, momentum-conservation collisions, Newton’s first law, and simple compositions.
  • Object and scene scope: Dynamic objects are limited to spheres, cubes, cylinders, and cones, while single-camera cases omit articulated, organic, deformable, irregular, multi-view, and novel-view challenges.The restricted object set improves annotation consistency but reduces scene and object diversity.
  • Interface scope: The annotated first frame grounds physical quantities visually, so the benchmark tests perception, law formulation, and dynamics deduction rather than constructing a complete physical scene from text alone.This interface removes unnecessary reference ambiguity while narrowing the task’s scope.
  • Evaluation scope: MLLM judges are imperfect physical oracles, and objective metrics introduce limitations through segmentation quality, harmless appearance penalties, and noisy velocity estimates.The evaluation therefore combines subjective judgments with objective comparisons when masks, pixels, or velocities can be matched to ground truth.
  • Real-world ground-truth noise: Real-world labels remain less exact than simulator labels because they rely on measurement, mask refinement, and frame-based annotation despite recording controls, instrument precision, review, and agreement analysis.Simulator cases instead provide exact engine-state ground truth.
  • Temporal normalization assumption: Generated-video evaluation assumes that the fully decoded output represents the prompt’s requested physical duration, despite providers returning fixed-length containers.The rule is applied consistently across models but may conflate physical time with nominal file duration.
Loading 2607.16401v1…