PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-free Detection of Generated Videos via Spatial-Temporal Likelihoods

Omer Ben Hayun, Roy Betser, Meir Yossef Levi, Levi Kassel, Guy Gilboa

arXiv:2603.15026v2cs.CVcs.LG

TL;DR

Reliable synthetic-video detection is difficult because image-based methods ignore temporal dynamics and supervised video detectors generalize poorly to unseen generators. STALL addresses this gap with a training-free probabilistic detector that fuses spatial and temporal likelihoods using real-video statistics, and it consistently outperforms prior detectors across benchmarks including recent frontier generators.

  • Problem

    Image-based detectors ignore temporal dynamics, while supervised video detectors generalize poorly to unseen generators, creating a need for reliable zero-shot video detection.

  • Method

    STALL jointly models per-frame spatial likelihoods and inter-frame temporal likelihoods using a real-video calibration set without training or generated samples.

  • Results

    STALL consistently outperforms prior supervised and zero-shot image/video detectors across multiple benchmarks, including benchmarks featuring Sora and Veo3.

  • Takeaways & Limitations

    The method provides an efficient, training-free detector that remains robust across spatial and temporal perturbations, calibration choices, and aggregation settings.

  • Takeaways & Limitations

    STALL requires a calibration set of real videos to define its spatial and temporal reference distributions.

Abstract

from arXiv · show

Following major advances in text and image generation, the video domain has surged, producing highly realistic and controllable sequences. Along with this progress, these models also raise serious concerns about misinformation, making reliable detection of synthetic videos increasingly crucial. Image-based detectors are fundamentally limited because they operate per frame and ignore temporal dynamics, while supervised video detectors generalize poorly to unseen generators, a critical drawback given the rapid emergence of new models. These challenges motivate zero-shot approaches, which avoid synthetic data and instead score content against real-data statistics, enabling training-free, model-agnostic detection. We introduce STALL, a simple, training-free, theoretically justified detector that provides likelihood-based scoring for videos, jointly modeling spatial and temporal evidence within a probabilistic framework. We evaluate STALL on two public benchmarks and introduce ComGenVid, a new benchmark with state-of-the-art generative models. STALL consistently outperforms prior image- and video-based baselines. Code and data are available at https://omerbenhayun.github.io/stall-video.

1. Introduction

Rapid progress in realistic video generation increases the need for reliable synthetic-video detection, but existing image- and supervised video-based detectors struggle with temporal evidence and unseen generators. STALL addresses this gap with a training-free, theoretically grounded detector that jointly models spatial and temporal likelihoods and achieves state-of-the-art results across benchmarks.

  • Synthetic videos’ realistic, controllable generation raises concerns about misinformation, fraud, impersonation, and intellectual-property violations.
  • Image detectors operate per frame, ignoring temporal dependencies and artifacts that emerge across video frames.
  • Supervised video detectors require large labeled datasets and generalize poorly to unseen generators.
  • D3 provides zero-shot video detection through temporal transitions but ignores per-frame spatial information and lacks principled theoretical foundations.
  • STALL jointly models spatial content and temporal transitions using a probabilistic, training-free formulation calibrated only on real videos.
  • STALL achieves state-of-the-art performance on two established benchmarks and a new benchmark containing videos from recent high-performing generators.

2. Background and Related work

Prior detection work progressed from supervised image classifiers toward zero-shot image methods, but image-only approaches remain limited for videos. STALL’s qualitative comparison motivates combining spatial and temporal evidence rather than relying on either modality alone.

  • Supervised image detectors often generalize poorly to unseen generators, motivating zero-shot methods that avoid synthetic-content exposure.
  • Image-only detectors assess authenticity frame by frame and therefore overlook temporal dependencies in videos.
  • Figure 2 contrasts spatial-only ZED, temporal-only D3, and STALL across clips with natural or unnatural spatial and temporal behavior.
  • STALL fuses spatial and temporal likelihoods because either modality alone can fail when the other dimension is misleading.
  • Whitened visual embeddings can be approximated by Gaussian distributions, enabling closed-form image-likelihood approximation without additional training.

3. Preliminaries

The preliminaries establish whitening and Gaussian likelihood approximation as the mathematical basis for scoring embeddings. The method overview then extends this foundation from frame embeddings to normalized temporal differences and fused video-level scoring.

  • Whitening transform: Whitening transforms centered embeddings so that the resulting representation has zero mean and identity covariance.
  • Gaussian likelihood: Under an isotropic Gaussian model, the squared whitened norm provides a closed-form proxy for embedding log-likelihood.
  • Asymptotic Gaussian projections: High-dimensional vectors with nearly uniform directions and concentrated norms have low-dimensional linear projections that are approximately Gaussian.
  • Method overview: STALL calibration computes spatial statistics from frame embeddings and temporal statistics from normalized inter-frame differences.
  • Method overview: At inference, frame embeddings are transformed and scored, temporal embeddings are scored, and the resulting components are aggregated into a video score.

4. Method: STALL

STALL detects generated videos by jointly scoring per-frame spatial embeddings and normalized inter-frame transitions against real-video statistics. It aggregates complementary likelihood evidence into a scale-free unified score, with robustness supported by ablations and real-only calibration.

  • Method: STALL jointly scores per-frame spatial embeddings and inter-frame transitions through spatial and temporal likelihoods.
  • Likelihoods: Spatial likelihoods use whitened frame embeddings, while temporal likelihoods model normalized frame-to-frame differences whose directions exhibit Gaussian-like behavior.
  • Unified score: The method selects maximal spatial likelihood and minimal temporal likelihood because this combination yields the lowest cross-domain correlation and complementary information.
  • Unified score: Percentile normalization places spatial and temporal scores on comparable, scale-free ranges relative to real calibration data.
  • Robustness: The unified score performs best among tested component and fusion choices, while remaining robust across aggregation choices and calibration settings.
  • Calibration: Calibration uses real videos only, with the calibration set disjoint from evaluation benchmarks and other paper data.

5. Evaluations

STALL is evaluated on three benchmarks against image- and video-based detectors, with experiments covering efficiency, calibration, encoder choice, perturbations, and temporal settings. It achieves strong detection performance while remaining efficient and robust across tested conditions.

  • Experimental settings: VideoFeedback, GenVideo, and ComGenVid evaluate STALL on diverse generated videos, including recent Veo3 and Sora outputs.The benchmarks pair generated videos with real videos and ComGenVid is designed to stress cross-model generalization.
  • Benchmark results: STALL achieves the highest average performance on each benchmark and outperforms supervised video detectors included in Figure 6b.Other methods sometimes produce AUC values below 0.5 for individual generators, whereas STALL maintains consistent separation.
  • Efficiency: 0.49s inference latency per 16-frame video makes STALL one of the fastest evaluated methods.D3 and RIGID require 0.5s and 0.6s, while ZED and T2VE require 0.92s and 0.97s.
  • Calibration: Calibration data from the tested distribution is only slightly better, while alternative datasets remain competitive.Calibration-set sizes below 5k cause significant performance drops; larger sizes are more stable.
  • Robustness: STALL maintains strong separation under JPEG compression, Gaussian blur, resized crops, additive noise, and varied temporal settings.Temporal robustness holds across frame rate, video duration, and temporal step size, while aggregation choices also have limited impact.

6. Conclusion

STALL detects fully generated videos by combining spatial and temporal likelihoods within a training-free probabilistic framework. Across benchmarks including frontier generators, it consistently outperforms prior detectors while remaining efficient and robust.

  • Method: STALL fuses per-frame spatial and inter-frame temporal likelihoods using only real videos to define reference distributions.The detector requires no training and no generated samples.
  • Results: STALL consistently outperforms prior supervised and zero-shot image and video detectors across benchmarks including Sora and Veo3.The conclusion identifies modeling the statistical structure of real videos as a promising path for detection.
  • Implications: The method is efficient and robust to spatial and temporal perturbations, calibration choices, calibration-set size, and aggregation choices.The paper notes that room for improvement remains as video generation continues to develop rapidly.

— Supplementary Material —

The supplementary material identifies the paper’s authors and institutional affiliation. It also provides the publication’s organizational context.

  • Authors: The paper is authored by Omer Ben Hayun, Roy Betser, Meir Yossef Levi, Levi Kassel, and Guy Gilboa.
  • Affiliation: The authors are affiliated with the Viterbi Faculty of Electrical and Computer Engineering.
  • Affiliation: The stated institution is the Technion – Israel Institute of Technology in Haifa, Israel.

Abstract

The supplementary material documents STALL’s reproducible implementation and evaluation procedures, including whitening, spatial and temporal scoring, calibration, inference, preprocessing, and balanced comparisons. It also records evaluation caveats involving class imbalance and frame duplication.

  • Scoring: Spatial scores use whitened frame embeddings, while temporal scores use whitened normalized differences between consecutive embeddings.
  • Calibration: STALL calibration uses one frame per calibration video for spatial statistics and all normalized inter-frame differences for temporal statistics.
  • Inference: During inference, STALL computes spatial and temporal scores, converts them to calibration percentile ranks, and averages the two ranks.The resulting video score is svideo = 1/2(percspat + perctemp).
  • Preprocessing: The default evaluation samples videos at 8 FPS and truncates them to 2 seconds, yielding 16 frames per video.Videos shorter than 2 seconds or below 8 FPS are filtered out before sampling.
  • Evaluation protocol: Balanced sampling uses equal numbers of real and generated videos for pairwise comparisons, addressing class imbalance in evaluation.
  • Evaluation caveats: Unbalanced comparisons can inflate Average Precision, while duplicating 3-FPS real frames to 8 FPS can inflate scores for inter-frame detectors.

A.4.3. D3 Ablation Results

The section reports protocol-sensitive D3 evaluation and describes the Anderson–Darling and D’Agostino–Pearson tests used to assess Gaussianity.

  • D3 evaluation: Each protocol difference individually inflates D3 mean AP on GenVideo, while their combination produces a large gap from the reported results.The differences concern class imbalance and FPS duplication.
  • Normality tests: The Anderson–Darling test emphasizes tail discrepancies when comparing samples with a reference normal distribution.Larger A2 values indicate stronger deviations from normality.
  • Normality tests: The analysis accepts normality under the Anderson–Darling criterion A2 < 0.752.The statistic is compared with tabulated critical values.
  • Normality tests: The D’Agostino–Pearson test combines sample skewness and kurtosis into a statistic that approximately follows a chi-square distribution with two degrees of freedom.The p-value is treated as compatible with Gaussianity when p > 0.05.

B.1.3. Results

The results show that normalization makes temporal differences approximately Gaussian, while frame embeddings and video embeddings are also compatible with Gaussian likelihood modeling.

  • Normality-test results: Raw temporal differences are strongly non-Gaussian, whereas ℓ2-normalized differences exhibit high normality-test pass rates.The analysis uses 40 groups of 250 embeddings and evaluates each coordinate with AD and DP tests.
  • Temporal differences: ℓ2 normalization transforms temporal-difference histograms from clear Gaussian deviations into close matches across the first four DINOv3 dimensions.The comparison uses adjacent frame pairs from the VATEX calibration set and moment-matched Gaussian curves.
  • Frame embeddings: Raw and generated video frame embeddings both follow approximately Gaussian shapes, despite slight differences in their means and variances.The distributions are shown for randomly sampled GenVideo frames.
  • Theoretical basis: The Maxwell–Poincaré result motivates Gaussian behavior for coordinates of high-dimensional vectors uniformly distributed on the sphere.The first-coordinate distribution approaches a standard normal as dimension increases.
  • Theoretical basis: Cosine-similarity distributions compare normalized VATEX temporal differences with points sampled uniformly from the unit sphere.The comparison uses 3k randomly sampled normalized temporal differences and 3k sphere points.

C.1. ComGenVid benchmark

ComGenVid combines real and generated videos from multiple sources, with standardized sampling and dedicated calibration and evaluation subsets.

  • ComGenVid sources: ComGenVid includes sampled real videos from MSVD and generated videos from Sora and Veo3, with 1.7k videos selected for each source.The Sora videos were manually collected from distinct users, while Veo3 videos came from ShareVeo3.
  • ComGenVid sources: MSR-VTT contributes 1.4k real videos because it contains substantially more real videos than any generative model.The selection follows the paper’s pairwise comparison protocol.
  • Evaluation protocol: GenVideo evaluation standardizes videos to 8 FPS and 2 seconds, or uses 1-second real comparisons for models producing shorter clips.The standard setting contains 16 frames, while the shorter-clip setting contains 8 frames.
  • VideoFeedback: VideoFeedback evaluation retains videos with dynamic-degree scores 3–4, indicating motion that can be distinguished from a static image.Videos are evaluated only when they meet the duration requirements, with a one-second exception for HotShot-XL.
  • Calibration data: VATEX, Kinetics400, and PE provide calibration or calibration-ablation data, while the paper points to tables for complete benchmark statistics.The default ablation configuration uses VATEX, DINOv3, and 8-FPS, 2-second generative videos.

D.1. Derivative order ablations

The ablations examine temporal derivative order, branch composition, aggregation, calibration, and robustness to perturbations, frame rates, and video lengths. Results indicate similar performance across derivative orders, robust calibration and sampling choices, and sensitivity to abrupt flash frames.

  • D.1. Derivative order ablations: All derivative orders D ∈{1, 2, 3, 4} produce strongly correlated scores and very similar performance.The evaluation reports AUC, Pearson correlation, and Spearman correlation across orders.
  • D.2. Components ablation: STALL compares spatial-only, temporal-only, and combined detectors using raw, percentile-ranked, mean-fused, and product-fused scores.The configurations report average AP and AUC across all three benchmarks.
  • D.3. Frame-level aggregation ablation: The frame-level aggregation ablation sweeps minimum, mean, and maximum operators independently for spatial and temporal branches, reporting average AUC and AP as heatmaps.Rows encode spatial aggregation and columns encode temporal aggregation.
  • D.6. Calibration and robustness: 0.8110 average AUC and 0.8046 average AP result from using all frame differences for temporal whitening, versus 0.8105 and 0.8044 from one sampled difference per video.The two calibration variants yield Pearson correlation 0.9994 and Spearman correlation 0.9992.
  • D.5. Temporal perturbations: Temporal likelihood is largely unaffected by reversal and shuffling but is strongly affected by inserted black or white flash frames.Reversal and shuffling preserve adjacent-frame difference statistics, whereas flash frames introduce abrupt temporal inconsistency.
  • D.6.2–D.6.3. FPS and video length ablations: Performance remains essentially unchanged across the tested frame rates and 1–4-second video durations, despite whitening calibration at 8 FPS and 2 seconds.The FPS and duration ablations therefore report robustness to these sampling variations.
  • D.6.6. Calibration set sources: Calibration-source choice shows stable average AUC across all three benchmarks, while performance may degrade when the test domain is absent from calibration.The calibration set defines the detector’s reference feature statistics and can also support adaptation to a target domain.

E.2. memory analysis

The memory analysis profiles model-loading and inference memory in isolated processes. It finds substantial resource variation across detection approaches, including different memory trade-offs among image and supervised video methods.

  • Memory profiling: The profiling study separately measures one-time model-loading memory and inference memory for each method in isolated processes.Isolation is used to eliminate memory pollution or interference between measurements.
  • Zero-shot image methods: AEROBLADE has a substantially higher memory footprint than ZED among zero-shot image methods.The comparison covers both loading and inference phases.
  • Supervised video methods: T2VE requires more GPU memory for model loading than AIGVdet, whereas AIGVdet consumes more memory during inference.The result illustrates a phase-dependent memory trade-off within supervised video detection.
Loading 2603.15026v2…