PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Rift: Cross-Scale Coupling Mismatch for AI-Generated Video Detection

Siyu Li, Jin Yang, Weiheng Liang

arXiv:2609.00742v1cs.CVcs.MM

TL;DR

AI video generators can achieve realism while violating the natural coupling between macro temporal dynamics and micro residual patterns, creating a detection gap. RIFT independently extracts both scales with orthogonal constraints and measures their conditional dependency, achieving high benchmark F1 and strong unseen-generator detection. The method is also relatively insensitive to encoder scale and family, though its current evaluation remains bounded by cropping fragility, compute concentration, and limited next-generation coverage.

  • Problem

    AI generators systematically violate the cross-scale coupling between macro temporal dynamics and micro residuals, but detecting this mismatch requires independent extraction and relationship measurement.

  • Method

    RIFT combines orthogonal macro-micro decomposition, a geometric-topological macro baseline, a residual-based micro probe, and conditional coupling-divergence analysis.

  • Results

    RIFT achieves 99.33% F1 on VidProM and 99.72% on GenVidBench, with 97.87% unseen-generator detection in leave-one-out evaluation.

  • Takeaways & Limitations

    Detection remains nearly unchanged across 14× DINOv2 parameter scaling and drops only 0.73 pp when switching to DINOv1.

  • Takeaways & Limitations

    RIFT is fragile under aggressive spatial cropping, the macro Transformer accounts for 86% of post-backbone compute, and broader evaluation beyond the Seedance 2.0 case study remains needed.

Abstract

from arXiv · show

As AI video generators achieve cinematic realism, reliable detection becomes essential for safeguarding digital trust. We identify cross-scale coupling mismatch as a new forensic signal, where scale refers to the level of abstraction (semantic dynamics vs. pixel-level residuals): in natural videos, macro-level temporal dynamics and micro-level residual patterns are intrinsically coupled by the unified imaging physics pipeline, whereas AI generators, whose training objectives do not explicitly preserve this joint distribution, systematically violate this coupling. Detecting such mismatch is challenging because it requires independently extracting information at both scales while simultaneously quantifying their cross-scale relationship. We propose RIFT (Representation Inconsistency Forensics on Trajectories), an orthogonal forensic framework that addresses this through three interlocking components: a macro stream that builds a dynamic baseline of expected temporal evolution via differential geometry and persistent homology on learned manifold trajectories, a micro stream that acts as a sensitive forensic probe via steganalytic filtering and temporal modeling, and a coupling divergence module that measures the conditional dependency between the two streams. Gram-Schmidt orthogonality guarantees the information-theoretic validity of this measurement. Experiments on two benchmarks (VidProM, 120K videos, 7 generators; GenVidBench, 68K videos, 4 generators) demonstrate that RIFT achieves 99.33% and 99.72% F1-score respectively, with 97.87% unseen-generator detection rate in leave-one-out evaluation, while exhibiting encoder agnosticism: scaling from ViT-S/14 (22M) to ViT-L/14 (300M) changes F1 by less than 0.1%, and switching to a different encoder family (DINOv1) reduces F1 by only 0.73 pp. Code is available at https://github.com/Litsay/RIFT

1 Introduction

RIFT identifies cross-scale coupling mismatch as a forensic signal: natural videos couple macro temporal dynamics with micro residuals, while AI-generated videos systematically violate this relationship. It addresses this gap with orthogonal macro and micro streams plus conditional coupling-divergence analysis.

  • Research Gap: Existing detectors separately model spatial artifacts, temporal patterns, or single-scale physics statistics, while hierarchical dual-path methods do not directly establish cross-scale dependency.These paradigms therefore leave the relationship between semantic dynamics and pixel-level residuals insufficiently addressed.
  • Motivation: Natural videos intrinsically couple macro temporal dynamics and micro residual patterns through the unified imaging physics pipeline.AI generators optimize perceptual quality and temporal coherence at individual scales without explicitly preserving their joint distribution.
  • Approach: RIFT orthogonally decouples representations into macro and micro subspaces, then builds a macro dynamic baseline using manifold geometry and persistent homology.The micro stream independently probes noise fingerprints and temporal forensic patterns.
  • Approach: The coupling-divergence module estimates conditional dependency between macro and micro signals, with both streams active in parallel rather than arranged as a bypassable cascade.The paper identifies this divergence as the distinguishing mechanism beyond single-scale trajectory methods.
  • Results: 99.33% and 99.72% F1 are achieved on VidProM and GenVidBench, respectively, with 97.87% unseen-generator detection in leave-one-out evaluation.The contribution passage reports 7 generators for VidProM and 4 for GenVidBench.

2 Method

RIFT extracts frozen frame and motion representations, orthogonally separates macro and micro subspaces, and models their temporal behavior independently. It combines a geometric-topological macro baseline, a residual-based micro probe, and conditional coupling analysis to detect cross-scale mismatch.

  • Pipeline: RIFT’s five-stage pipeline extracts frozen multimodal frame features, orthogonally decouples macro and micro subspaces, models each stream, and measures P(micro | macro).The coupling module quantifies cross-scale mismatch for binary authentic-versus-generated classification.
  • Orthogonal Decoupling: Gram-Schmidt constraints and reconstruction make the macro and micro subspaces non-redundant, preventing shared information from inflating coupling estimates through self-correlation.The coupling measurement requires independently extracted information from both streams.
  • Macro Stream: The macro stream embeds 128-dimensional representations into a 24-dimensional manifold space before extracting local geometry and global topology.The embedding restores angular variance from approximately 0.008 to 0.042, while persistent homology summarizes connected components and loops.
  • Macro Stream: Differential geometry summarizes trajectory bending, twisting, direction changes, speed, acceleration, and jerk into a 36-dimensional local descriptor.These statistics capture the trajectory’s local kinematic profile.
  • Micro Stream: The micro stream suppresses scene content with SRM and Bayar high-pass filters, then models residual-feature sequences using a convolutional encoder and bidirectional GRU.Motion-compensated residual statistics and frame-to-frame forensic changes supplement the temporal embedding.
  • Coupling Divergence: The coupling module predicts micro-statistics distributions from macro states, using conditional likelihood to encode strong coupling as low NLL and mismatch as high NLL.This conditional dependency is the framework’s distinguishing forensic mechanism.

3 Experiments

RIFT is evaluated across two benchmarks, unseen generators, component ablations, encoder variants, interpretability analyses, and common degradations. It achieves strong benchmark performance and generalization, while coupling divergence and macro geometry emerge as central signals.

  • Main Results: 99.33% F1 on VidProM and 99.72% F1 on GenVidBench establish RIFT as state of the art across both benchmarks.These results exceed ReStraV by 1.50 percentage points on VidProM and MPF-Net by 2.69 percentage points on GenVidBench.
  • Cross-Generator Generalization: 97.87% average unseen-generator detection rate is maintained in leave-one-generator-out evaluation across all seven VidProM generators.The hardest cases—OpenSora, CogVideo, and Pika—still exceed 94%.
  • Ablation Study: 97.99% F1 from Macro only versus 83.54% from Micro only shows that macro geometry and topology carry most discriminative information.Removing coupling divergence reduces F1 by 1.32 percentage points, nearly matching the Full-to-Macro-only gap of 1.34 points.
  • Encoder Ablation: 0.02% F1 change from ViT-S/14 to ViT-L/14 and a 0.73-point drop from DINOv2 to DINOv1 demonstrate encoder agnosticism.The DINOv1 switch changes AUC by only 0.12 percentage points.
  • Interpretability: Real videos show higher mutual information and lower conditional NLL than AI-generated videos, with Cohen’s d values of 1.37 and 4.46 respectively.The orthogonality check gives cos(z_macro, z_micro) ≈ 0.001 for both classes, while separation remains consistent across all seven generators.
  • Interpretability: 52% higher mean Menger curvature for real trajectories and 27% micro-gate suppression for fake videos reveal distinct geometric and gating behaviors.The curvature distributions overlap substantially, supporting coupling divergence as the decisive signal rather than geometry alone.

4 Related Work

Prior AI-video detectors largely isolate spatial, temporal, or physics-informed cues, while newer multi-scale methods still leave cross-branch coupling insufficiently addressed. RIFT instead treats conditional macro–micro coupling divergence as core forensic evidence and reports encoder agnosticism.

  • Existing paradigms: Existing detectors separately target frame artifacts, temporal patterns, or single-scale physical priors.Spatial methods discard temporal structure; temporal methods operate at one semantic scale; physics-informed methods use handcrafted statistics from one scale.
  • Multi-scale methods: Recent multi-scale approaches combine localization and verification or use hierarchical dual paths, but MPF-Net lacks cross-branch interaction.RIFT runs both streams in parallel with orthogonal constraints and measures their coupling divergence.
  • RIFT distinction: RIFT measures P(micro | macro) as the central forensic evidence rather than treating the two branches as independent detectors.This distinguishes RIFT’s coupling-centered design from prior multi-scale approaches.
  • Encoder robustness: RIFT exhibits encoder agnosticism, with less than 0.1% within-DINOv2 variation and a 0.73 pp reduction across encoder families.The comparison contrasts RIFT with MPF-Net’s backbone-sensitive performance.
  • Topological modeling: Persistent homology supplies global structural invariants that complement local differential geometry on learned manifold trajectories.RIFT applies Vietoris–Rips filtrations to temporal trajectories for AI-generated video detection.

5 Conclusion

RIFT frames AI-video detection as identifying cross-scale coupling that natural imaging physics preserves but current generators do not explicitly optimize. It combines orthogonal macro–micro analysis with coupling divergence and reports strong benchmark, generalization, efficiency, and robustness results, while noting remaining deployment and coverage limits.

  • Conclusion: RIFT identifies cross-scale coupling mismatch as a structural limitation of generators that optimize individual scales without explicit joint supervision.The hypothesis concerns the joint distribution between macro dynamics and micro residuals.
  • Conclusion: 99.33% F1 on VidProM and 99.72% F1 on GenVidBench establish high reported detection performance across 7 and 4 generators.Leave-one-out evaluation reaches a 97.87% unseen-generator detection rate.
  • Conclusion: RIFT uses 4.7M trainable parameters with frozen pretrained encoders, trains on one RTX 3080 with 10 GB, and reaches approximately 182 ms end-to-end latency.These are the reported resource and inference characteristics of the full pipeline.
  • Limitations and Future Work: Aggressive spatial cropping makes RIFT fragile because it alters the context used for patch statistics and optical flow.The authors identify spatial augmentation during feature extraction as a direct remedy.
  • Limitations and Future Work: Broader evaluation beyond the Seedance 2.0 case study, including Sora and Kling, is needed to assess the hypothesis’s generality.The authors also motivate distillation because the macro Transformer accounts for 86% of post-backbone compute.

A.1 Two-Stage Training Procedure

RIFT uses two-stage training with frozen backbone encoders: first shaping orthogonal macro–micro representations and manifold trajectories, then jointly optimizing the trainable modules for classification and dependency modeling. Its objective combines focal classification, orthogonality, reconstruction, manifold, and conditional losses.

  • A.1 Two-Stage Training Procedure: RIFT’s two-stage procedure keeps DINOv2 ViT-S/14 and RAFT-Large encoders frozen throughout training.The trainable modules differ between orthogonal-decoupling pretraining and joint training.
  • Stage 1: Orthogonal Decoupling Pretraining: Stage 1 trains orthogonal projectors, a reconstruction decoder, and the macro analyzer for 30 epochs with elevated manifold regularization γ_manifold = 0.2.Fusion, conditional dependency, and micro-stream modules remain frozen during this stage.
  • Stage 2: Joint Training: Stage 2 unfreezes all trainable modules for 50 epochs and reduces γ_manifold to 0.02 so classification loss dominates.Training uses warmup, cosine annealing with restarts, and validation-based early stopping.
  • Hyperparameters: Table 6 provides the complete final-model hyperparameter settings.The supplied training description identifies Table 6 as the hyperparameter summary.
  • A.3 Loss Function: The total loss is L = L_focal + αL_ortho + βL_recon + γL_manifold + δL_cond.The terms respectively cover class-balanced focal classification, stream orthogonality, CLS reconstruction, manifold regularization, and conditional prediction.

B.1 VidProM

VidProM evaluation uses a 50/50 stratified test split of approximately 60K samples, with per-generator rates computed under single-segment evaluation. CogVideo and OpenSora are the most challenging generators in the reported results.

  • B.1 VidProM: The VidProM test set uses a 50/50 stratified split with approximately 60K samples.Per-generator detection rates are reported on this test set.
  • B.1 VidProM: Per-generator rates use single-segment evaluation, whereas aggregate Table 1 metrics use the standard multi-segment protocol.This protocol distinction matters when comparing per-generator and aggregate results.
  • B.1 VidProM: 97.24% for CogVideo and 98.36% for OpenSora are the lowest reported generator detection rates, making them the most challenging cases.The passage attributes this pattern to their more temporally coherent outputs possibly preserving coupling patterns better.

B.2 GenVidBench

RIFT achieves near-perfect detection across GenVidBench generators, with the Main subset’s hardest case at 99.61% and perfect detection on Plants.

  • 99.61% detection was achieved for CogVideo on the Main subset, the most challenging GenVidBench generator result.
  • 100% detection was achieved for every generator on the Plants subset, where visual randomness is high.

C Full Robustness Results

RIFT remains effective under most tested degradations, but spatial cropping severely damages performance by disrupting feature extraction inputs.

  • F1 = 88.19% was the peak matched-bitrate result for H.265 at 2 Mbps.
  • F1 exceeded 88% under Gaussian noise at σ∈{5, 15, 20}.
  • F1 fell to 35.34% at 70% spatial cropping, the most damaging tested degradation.Cropping alters spatial context used to compute patch statistics and optical flow, disrupting the feature extraction pipeline rather than the classifier.
  • F1 showed nonmonotonic degradation behavior, including 79.89% at σ=10 and 67.82% for H.264 at 1 Mbps.The authors attribute these patterns to the limited evaluation subset (n=996) and interactions between degradation parameters and the precomputed feature pipeline.

D Per-Generator Coupling Mismatch Statistics

All seven generators exhibit cross-scale coupling breakdown relative to real videos, while orthogonality is maintained and curvature varies across generators.

  • All 7 generators show substantially lower MI and positive NLL than real videos, indicating universal coupling breakdown.
  • |cos(zmacro, zmicro)| < 0.003 for every generator, confirming orthogonality between the streams.
  • Curvature varies widely across generators, from 1.50 for T2VZero to 28.87 for Pika.

E Seedance 2.0 Full Results

RIFT generalizes to Seedance 2.0 in zero-shot evaluation: all 27 videos are correctly classified, although confidence is lowest for slow-moving underwater content.

  • 73.4% was the lowest confidence, for an underwater-ocean video with slow-moving marine life.Minimal motion structure weakens the macro baseline’s geometric features.
  • Seedance 2.0 showed weaker cross-scale coupling than the seven training generators, including MI = −17.74 and g_micro = 0.49.The extreme MI value may partly reflect MINE estimator instability on out-of-distribution data and is treated as a relative indicator rather than calibrated measurement.
Loading 2609.00742v1…