PaperlayerView exact revision on arXiv

Source-linked AI summary

PIVOT: Physics-Grounded Verification for AI-Generated Audio-Video Detection

Bo Zheng, Kangran Zhao, Xiaoyu Zhang, Weinan Guan, Zhiheng Li, Yize Chen, Haizhou Li, Qingshan Liu, Siwei Lyu, Baoyuan Wu

arXiv:2609.15562v1cs.CVcs.AIcs.MM

TL;DR

Artifact cues may become less stable as generative models advance, motivating evidence based on whether depicted events satisfy physical constraints. PIVOT estimates audio-video quantities, selects relevant measurable laws, and verifies their constraints. On PhysForensics-Bench, it outperforms direct Gemini 3.1 Pro inspection on the reported accuracy and F1 comparisons.

  • Problem

    Artifact cues used by existing detectors may weaken or become less stable as generators evolve, while realistic-looking generated events can still violate physical behavior.

    Source: §I · pp. 1–2

  • Method

    PIVOT estimates structured audio-video quantities, selects event-relevant laws that can be verified from them, and checks those laws’ constraints.

    Source: §I · p. 2

  • Results

    70.30% accuracy and 64.29% F1 on Real+Seedance, and 72.16% accuracy and 65.82% F1 on Real+Veo, versus Gemini 3.1 Pro’s 53.96% accuracy and 60.09% F1 and 57.22% accuracy and 63.44% F1, respectively.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results provide empirical support for physical-consistency verification as a practical, structured, inspectable direction that complements artifact-based AIGC detection.

    Source: §VII · p. 11

    The current instantiation focuses on event-centric clips whose observable physical evidence primarily comes from object motion, contact dynamics, scene geometry, and associated acoustic events.

    Source: §I · p. 2

Abstract

from arXiv · show

As generative models continue to advance, AI-generated content (AIGC) is becoming increasingly realistic, weakening the artifact cues commonly exploited by existing detectors. Nevertheless, faithfully reproducing the physical behavior of real-world events remains challenging for current generators. We therefore explore detecting AIGC by assessing whether the depicted event satisfies measurable constraints derived from physical laws. We introduce PIVOT, a physics-grounded AIGC detector, instantiated here for audio-video clips, that estimates physical quantities from video and audio, selects physical laws relevant to each clip, and verifies their measurable constraints. Beyond a real/fake decision, PIVOT returns supporting evidence that records the verification outcome, relevant time window, and supporting quantities for each applicable law. Although instantiated and evaluated here on audio-video data, the framework can, in principle, extend to other AIGC modalities whenever the physical quantities required for verification can be estimated reliably. We also introduce PhysForensics-Bench, comprising paired real and generated audio-video clips from nine event-centric scene families and two recent audio-video generators. On PhysForensics-Bench, PIVOT achieves 70.30% accuracy and 64.29% F1 score on Real+Seedance, and 72.16% accuracy and 65.82% F1 on Real+VEO. In comparison, direct inspection with Gemini 3.1 Pro obtains 53.96% accuracy and 60.09% F1 on Real+Seedance, and 57.22% accuracy and 63.44% F1 on Real+Veo. These results demonstrate the practical promise of physical-consistency verification as a structured and inspectable source of evidence that complements artifact-based AIGC detection.

I. INTRODUCTION

PIVOT frames audio-video AIGC detection as checking whether depicted events satisfy measurable physical constraints, producing evidence organized by law and time window. The paper motivates this approach as complementary to artifact cues and evaluates it on a benchmark of paired real and generated clips.

I. INTRODUCTION

  • Artifact cues may become weaker or less stable as generators evolve, while perceptually realistic audio-video can still violate physical behavior.The paper presents physical-law evidence as complementary to artifact cues.Source: §I · pp. 1–2
  • In the bouncing-ball example, rebound heights rise from 0.64 m to 0.75 m and audio inter-onset intervals from 533 ms to 631 ms, contrary to passive-bounce expectations.PIVOT links these measurements to energy-decay and bounce-interval constraints and returns the corresponding verification evidence.Source: §I · p. 2
  • 70.30% accuracy and 64.29% F1 on Real+Seedance, and 72.16% accuracy and 65.82% F1 on Real+Veo, compared with Gemini 3.1 Pro’s 53.96% accuracy and 60.09% F1 and 57.22% accuracy and 63.44% F1, respectively.PhysForensics-Bench pairs real and generated clips across nine event-centric scene families and two recent generators.Source: §I · p. 2
  • PIVOT selects event-relevant, measurable physical laws, checks their constraints over estimated quantities, and returns supporting evidence by law and time window.The formulation focuses on event-centric clips with observable evidence from object motion, contact dynamics, scene geometry, and associated acoustic events.Source: §I · p. 2

II. RELATED WORK

  • Prior physics-inspired video detectors apply predefined formulations, while PIVOT selects laws according to the depicted event and available measurable quantities.General audio-video authenticity detection using per-law measurable evidence remains less studied than face-centric, lip-speech, or local cross-modal detection.Source: §II · pp. 3–4

III. METHODOLOGY

PIVOT uses a caption–estimate–select–verify workflow to test audio-video events against physical constraints. Its quantity interface is architecture-agnostic: it exposes time-aligned records for downstream law selection and verification.

III. METHODOLOGY

  • PIVOT captions a clip, estimates time-indexed audio and video quantities, selects relevant and measurable laws, then verifies their constraints and returns a structured decision trace.The decomposition keeps observations, expected physical behavior, and verification outcomes explicit.Source: §III · p. 4

A. Captioning

  • Captions identify objects, surfaces, interactions, and temporal phases for law selection without predicting whether a clip is real or generated.The reported implementation retains independent descriptions from Gemini 3.1 Pro and GPT-5.5.Source: A. Captioning · p. 4

B. Physical Quantity Estimation

  • The replaceable estimator interface exposes named, time-aligned quantities as event-level audio records and frame-level video records, rather than hidden estimator features.Timestamps, units, and coordinate conventions are retained where applicable, allowing estimation tools to be substituted without changing selection or verification.Source: B. Physical Quantity Estimation · p. 5, Figure 3 · p. 5

1) Video Physical Estimator:

  • Video estimators can provide object attributes and kinematics, including position, velocity, and acceleration, while other estimators may add object-pair relations or further quantities.The interface requires consistent object identities, timestamps, field definitions, and coordinate conventions, not a fixed segmentation, geometry, or tracking model.Source: 1) Video Physical Estimator · p. 5

2) Audio Physical Estimator:

PIVOT turns localized audio events and structured physical quantities into event-specific, measurable law checks, then records constraint judgments and combines repeated verification labels.

2) Audio Physical Estimator:

  • Audio estimation fuses low-band semantic event detection with high-band transient detection on a shared time axis aligned with video, then compresses informative quantity records for downstream reasoning.Records can include onset, duration, peak dB, RMS level, fundamental frequency, and spectral descriptors; compression preserves metadata needed to interpret retained rows.Source: 2) Audio Physical Estimator · pp. 5–6

C. Physical Law Selection

  • The law selector chooses event-relevant laws that can be checked with available quantity fields and specifies constraints, required fields, and applicability or failure conditions.The decision basis can encode an equality, inequality, ordering, trend, or temporal relation.Source: C. Physical Law Selection · p. 6

D. Physical Consistency Verification

  • The verifier assesses selected physical constraints against captions and compressed audio-video quantities, recording each applicable law’s status as supported, violated, or uncertain.Verification is formula-guided consistency assessment, not exact physical simulation or full system identification.Source: D. Physical Consistency Verification · p. 6
  • Each of three runs returns a binary label and structured report; majority vote sets the final label, while law weights and weighted rates remain diagnostics.Reports retain law-window states and supporting quantities; the verifier can link decisions to inconsistencies such as discontinuous trajectories or mismatched contact and sound.Source: D. Physical Consistency Verification · pp. 6–7

IV. PHYSFORENSICS-BENCH

PhysForensics-Bench targets event-centric audio-video detection where physical-law consistency provides a forensic signal. Its generated clips are paired with real counterparts, though the set is not perfectly balanced.

IV. PHYSFORENSICS-BENCH

  • The benchmark foregrounds event-centric physical behavior, including trajectory continuity, rebound energy decay, and synchronization between visible contact and acoustic onset.Its focus is distinct from face identity or artifacts tied to a single generator.Source: §IV · p. 7

A. Data Sources and Generation

  • Generated clips from Veo 3.1 Fast and Seedance 2.0 use a real clip’s first frame and an event prompt as context for synthesizing the continuation and its sounds.This makes each generated clip a counterpart sharing the real clip’s initial visual context and high-level event description.Source: A. Data Sources and Generation · p. 7
  • The pair-oriented benchmark is not perfectly balanced because platform safety or community-policy filters block some generations, leaving some real clips without all generated variants.Examples include a visible face in the reference frame or a rejected prompt.Source: A. Data Sources and Generation · p. 7

B. Physical Scene Taxonomy

PhysForensics-Bench spans nine deliberately varied event-centric scenes, and PIVOT performs best on both generator-specific test subtasks without benchmark-label training or calibration.

B. Physical Scene Taxonomy

  • Nine scenes—ball bouncing, balloon deflation, ball collisions, basketball dribbling, catapult launches, coin drops, slingshot releases, slope descent, and cart descent on a ramp—cover event-centric physics.The set varies timing, contact localization, deformation, continuous motion, and energy trends across objects, surfaces, viewpoints, and motion patterns.Source: B. Physical Scene Taxonomy · p. 7

C. Benchmark Split

  • Splitting groups each real seed clip with all available Seedance and Veo counterparts, preventing shared initial frames or event descriptions from crossing splits.The split contains 878 training, 293 validation, and 293 test samples.Source: C. Benchmark Split · pp. 7–8

A. Experimental Setup

  • Main comparisons use Real+Seedance and Real+Veo binary subtasks and exclude benchmark-specific detector training; PIVOT also uses no benchmark labels for adaptation.Released checkpoints may have been trained on their original source domains.Source: A. Experimental Setup · p. 8

B. Main Results

  • Generator-specific subtasks expose prediction bias that a single aggregate score would conceal.Gemini 3.1 Pro has relatively high F1 but lower Accuracy, while released video detectors and out-of-domain FGI transfer poorly.Source: B. Main Results · pp. 9–10
  • 70.30% Accuracy and 64.29% F1 on Seedance, and 72.16% Accuracy and 65.82% F1 on Veo; PIVOT performs best in all four generator-specific metrics.PIVOT uses no benchmark labels for detector training or operating-point calibration.Source: Table III · p. 10, B. Main Results · p. 10

C. Per-Scene Analysis

Per-scene results vary with observability: sustained trajectories and repeated events are stronger than subtle deformation, small objects, and brief contacts. The paper proposes broader law coverage, uncertainty-aware verification, and expanded testing, and concludes that PIVOT leads the evaluated methods in the primary benchmark comparison.

C. Per-Scene Analysis

  • 78.57%/72.73% Accuracy/F1 on Seedance and 80.18%/75.00% on Veo make slope descent the strongest, most consistent scene.Sustained trajectories expose position, velocity, acceleration, and object-surface relations across many frames.Source: C. Per-Scene Analysis · p. 10
  • PIVOT detects 3 of 26 Seedance generated coin drops as fake versus 13 of 22 Veo clips, yielding 17.65% F1 versus 65.00%.Slingshot shows a similar generator gap: 10 of 45 Seedance versus 28 of 45 Veo generated clips detected, while 42 of 45 real clips are correctly classified for both.Source: C. Per-Scene Analysis · p. 10

VI. FUTURE DIRECTIONS

  • The current estimator covers object extent, 3D kinematics, and event-level acoustics; added quantities could extend verification to broader physical laws.Examples of quantities not yet explicitly recovered include mass, volume, density, material properties, elasticity, friction, and contact force.Source: §VI · p. 11
  • Future verification could propagate measurement uncertainty and use executable operators to evaluate numerical constraints reproducibly.The proposed estimators could provide calibrated confidence intervals, observation quality, and alternative measurement hypotheses.Source: §VI · p. 11
  • Future benchmark versions could broaden domains, objects, capture conditions, acoustic environments, and generator families, with matched and longitudinal tests.These evaluations would test robustness to evolving generators on previously unseen generators and events.Source: §VI · p. 11

VII. CONCLUSION

  • PIVOT achieves the highest accuracy and F1 among evaluated methods in the primary comparison on PhysForensics-Bench.The benchmark pairs real and generated clips across nine scene families, with Seedance 2.0 and Veo 3.1 Fast generated samples.Source: §VII · p. 11
  • The conclusion frames event-aware, measurability-constrained physical-consistency verification as a complementary direction for AIGC detection.The framework connects physical perception with multimedia forensics and structures evidence from physical behavior of depicted events.Source: §VII · p. 11

APPENDIX

The appendix describes PIVOT’s staged agent workflow, notation and implementation mapping, and representative prompts, quantity records, and verification output. The examples show how the selector limits checks to available quantities and the verifier reports law-specific evidence and a verdict.

A. LLM Agent Workflow

  • PIVOT’s agent workflow captions the clip, estimates physical quantities, selects measurable event-relevant laws, then verifies those constraints and returns a verdict.The verifier reports support, violations, and uncertainty alongside the real/fake decision.Source: A. LLM Agent Workflow · p. 12

B. Notation and Implementation Mapping

  • The notation maps θ to a stage’s model and prompting configuration, distinguishes vector and scalar quantities, and marks deterministic procedures with upright function names.The appendix directs readers to Table V for concrete backends and Table VI for remaining symbols.Source: B. Notation and Implementation Mapping · p. 12

C. Prompt and Response Examples

  • The example records aligned audio timing and acoustic descriptors alongside sparse video trajectories and object-relation distances for impact and trajectory checks.Small minimum distances around audio onsets support impact-synchronization checks; velocity and acceleration support trajectory-continuity checks.Source: C. Prompt and Response Examples · p. 13
  • The law selector uses captions and a compact quantity-field summary, choosing only constraints evaluable from available fields before seeing full numerical tables.The verifier then receives selected laws, their constraints, and compressed quantity tables, and evaluates those constraints without selecting new laws.Source: C. Prompt and Response Examples · p. 13
  • In the ball-bounce example, PIVOT flags an unsourced early transient and stronger audio at a lower-energy rebound, then returns a physically implausible, Fake verdict.The response ties each finding to a named law and time; it also notes that most bounce onsets align and the trajectory is broadly gravity-consistent.Source: C. Prompt and Response Examples · p. 14

D. Video Physical Quantity Estimator

The video estimator combines object masks with calibrated camera geometry to derive physical quantities. It smooths finite-difference motion estimates, while the ablation removes the enhanced mask and geometry components but retains downstream PIVOT stages.

D. Video Physical Quantity Estimator

  • The video estimator combines a frozen object-understanding path with a simulation-calibrated geometry path.GroundingDINO and SAM 2 provide object masks; frozen ViPE estimates camera parameters, motion, and depth, with adapters calibrating intrinsics and depth.Source: D. Video Physical Quantity Estimator · p. 14
  • Masked pixels are back-projected through depth and camera geometry into a fixed world frame to recover object coordinates in meters.The object center is estimated from the 2D mask centroid and median object depth.Source: D. Video Physical Quantity Estimator · p. 14
  • Backward finite differences produce raw velocities and accelerations, then separate exponential moving averages smooth frame-level depth and mask jitter.The smoothing is intended to suppress derivative noise while retaining motion trends; α_v and α_a control velocity and acceleration smoothing.Source: D. Video Physical Quantity Estimator · pp. 14–15
  • The video ablation restores the default ViPE mask path and bypasses intrinsics and depth adapters, while leaving downstream quantity estimation and verification stages unchanged.Source: D. Video Physical Quantity Estimator · p. 15

E. Unity Simulation Calibration Data

The Unity simulation archive supplies camera and object ground truth for calibrating video-side quantity estimates, without using PhysForensics-Bench labels. The reported held-out audit shows improved mask and depth measurements from the calibrated components.

E. Unity Simulation Calibration Data

  • The calibration set contains 260 Unity ball-dropping episodes rendered from four fixed cameras, yielding 1,040 monocular object runs.Episodes include known camera parameters and poses, object identities and masks, and depth; the set is used only to improve video-side quantity estimates.Source: E. Unity Simulation Calibration Data · p. 15
  • Calibration selects bidirectional GroundingDINO+SAM 2 masks and trains intrinsics and residual-depth adapters on simulation data, without fitting or selecting on PhysForensics-Bench labels.Splits are assigned by simulation episode to prevent the same physical trajectory appearing across train, validation, and test through different camera views.Source: E. Unity Simulation Calibration Data · p. 15
  • On held-out Unity runs, bidirectional masks achieve about 0.94 foreground IoU versus 0.54–0.55, while depth adapters reduce full-frame absolute relative error from 0.393 to 0.057.Object-region absolute relative error falls from 0.450 to 0.188; these audit results characterize quantity estimation, not benchmark detection.Source: E. Unity Simulation Calibration Data · p. 15

F. Audio Physical Quantity Estimator

The audio estimator detects events through semantic and native-rate spectral proposal paths, then merges them into event records with acoustic descriptors and detector-source metadata.

F. Audio Physical Quantity Estimator

  • The semantic path detects frame-level sound events in 10 s chunks, thresholds probabilities at 0.2, and refines coarse regions using waveform onsets and envelope boundaries.Coarse regions use a 9-frame median filter and a 50 ms minimum duration.Source: F. Audio Physical Quantity Estimator · p. 15
  • The native-rate spectral path detects brief, weakly semantic, or high-frequency physical sounds using STFT band novelty with a 25 ms window and 10 ms hop.It compares band power in dB against a causal 120 ms median baseline, skipping bands above Nyquist when necessary.Source: F. Audio Physical Quantity Estimator · p. 15
  • Spectral proposals require median novelty plus 6 MAD, at least 6 dB energy rise, and a −75 dB peak floor before duration filtering and merging.Regions are padded by 20 ms, kept if at least 30 ms long, and merged within 80 ms; their label remains the neutral spectral_transient.Source: F. Audio Physical Quantity Estimator · p. 15
  • The main audio branch merges semantic and spectral proposals when they overlap or lie within 80 ms, retaining semantic labels when available and recording proposal sources.Event records include energy, HNR, spectral descriptors, decay, dominant band, and sample rate; spectral-only events remain physical transient candidates without fabricated semantic labels.Source: F. Audio Physical Quantity Estimator · pp. 15–16

G. Physics-Aware Quantity Compression

Physics-aware compression selects deterministic visual sampling policies from clip captions and converts sampled visual and event-level audio quantities into compact tables with sampling metadata.

G. Physics-Aware Quantity Compression

  • The caption-based selector chooses among six deterministic policies, including relation, kinematic, area-change, uniform, and hybrid-event sampling.Policies retain relevant extrema or event candidates, with uniform coverage used when no reliable event cue exists.Source: G. Physics-Aware Quantity Compression · p. 16
  • Selected visual core frames retain two neighbors on each side, while audio event quantities and sampled visual quantities are converted to CSV tables with metadata marking sparse visual rows as event-centered.Together, the tables and metadata form the compact representation used for verification.Source: G. Physics-Aware Quantity Compression · p. 16
  • The cited result reports 8.9k tokens per clip, or about 94%.Source: G. Physics-Aware Quantity Compression · p. 16

H. Physical Quantity Estimator Ablations

Estimator ablations quantify the contributions of video and audio enhancements; the paper also describes qualitative physical checks, representative constraint families, and inference and compression costs.

H. Physical Quantity Estimator Ablations

  • Without video enhancement, Accuracy/F1 fall from 70.30%/64.29% to 60.40%/54.55% on Seedance and 72.16%/65.82% to 62.89%/57.14% on VEO.This ablation restores the default mask path and removes Unity-calibrated intrinsics and depth adapters.Source: H. Physical Quantity Estimator Ablations · p. 16
  • Without high-band transient detection, Accuracy/F1 are 64.85%/60.34% on Seedance and 65.46%/59.88% on VEO.The full architecture's video enhancement and semantic event detector remain in place.Source: H. Physical Quantity Estimator Ablations · p. 16

I. Qualitative Verification of Physical Constraints

  • Most constraints are qualitative or ordinal: the verifier checks trends such as decreasing rebound heights rather than requiring universally calibrated absolute thresholds.Monocular geometry, mask propagation, and acoustic-event localization are not yet accurate enough for universal absolute thresholds; checks cover alignment, ordering, monotonic trends, and qualitative consistency.Source: §I · p. 16

J. Representative Physical Constraints

  • Onset-to-event checks anchor audio events to nearby contact, release, rebound, relation-distance, velocity, or area-change cues; strong onsets without plausible visual sources are also checked.Selected laws must be relevant to the clip, supported by available quantities, and specified with a physical basis, operational check, applicability, and failure conditions.Source: J. Representative Physical Constraints · p. 17
  • Kinematic and energy checks test plausible gravity, inertia, dissipation, and projectile continuity, while material checks compare acoustic duration, decay, novelty, and band energy with video-implied interactions.Examples include bounce-energy dissipation, post-release trajectory continuity, material timbre, and rolling-friction envelope coverage.Source: J. Representative Physical Constraints · p. 17
  • Scene-geometry checks cover stable reference structures, balloon topology and orientation, and projectile free-flight continuity.These checks can identify geometry-level violations that local audio-visual synchronization alone may miss.Source: J. Representative Physical Constraints · p. 17

K. Compute Cost

  • Across 1,464 samples, complete PIVOT inference uses 10,248 successful API calls and averages 144.8k input, 16.5k output, and 161.8k total tokens per sample.The input includes 18.1% served from cache; failed retries and extra successful reruns are excluded.Source: K. Compute Cost · p. 17
  • On 135 stratified clips, compressed quantity prompts average 8.9k input tokens versus 154.3k for raw quantities, a reduction of about 94%.Quantity-block size falls from 294.7k to 8.4k characters on average (97.1%); the visual sampler retains 31.2 of 90.3 frames on average (34.5%).Source: K. Compute Cost · p. 17

Related papers

Loading 2609.15562v1…