Benchmarking Spatial, Spectral, and Self-Supervised Cues for Face Forgery Detection under Realistic Degradation

Lucas Cunha; Lucas Sotomaior; Lucas Gasperin; Beatriz Caldas; Eduardo Pianovski; Rayson Laroca

Source-linked AI summary

Benchmarking Spatial, Spectral, and Self-Supervised Cues for Face Forgery Detection under Realistic Degradation

Lucas Cunha, Lucas Sotomaior, Lucas Gasperin, Beatriz Caldas, Eduardo Pianovski, Rayson Laroca

arXiv:2609.01511v1Sep 1, 2026cs.CV

TL;DR

Face forgery detectors can perform strongly on controlled benchmarks yet remain unreliable under realistic degradation. This paper benchmarks six model families and spatial, spectral, and hybrid representations on clean and degraded MFFI partitions, finding that Xception leads clean performance while frozen DINOv3 leads degraded performance. The results support degraded evaluation and self-supervised representations as promising directions for robust detection.

Problem
Face forgery detectors show limited practical reliability when compression, resizing, blurring, and re-encoding alter the test distribution.
Method
The paper establishes a standardized MFFI benchmark comparing six model families and spatial, spectral, and hybrid representations on clean and degraded partitions.
Results
Clean-set performance is an unreliable robustness proxy: Xception reaches ROC-AUC = 0.884 clean, whereas frozen DINOv3 reaches 0.726 degraded.
Takeaways & Limitations
Fourier cues are most useful as auxiliary RGB information, while self-supervised visual representations are a promising basis for robust face forgery detection.
Takeaways & Limitations
Using a single dataset measures robustness to MFFI degradation rather than cross-dataset generalization, and frozen DINOv3 does not establish fine-tuning behavior.

Abstract

from arXiv · show

Face forgery detectors often achieve strong results on controlled benchmarks, but their reliability under realistic image degradations remains limited. This paper presents a standardized benchmark for face forgery detection using the Multi-Dimensional Face Forgery Image (MFFI) dataset and evaluates performance on both clean and degraded test partitions. We compare six model families, including convolutional networks, transformer-based models, and a frozen self-supervised DINOv3 backbone, across spatial, spectral, and hybrid input representations. The results show that clean-set performance is not a reliable indicator of robustness under compression, resizing, and blurring. Xception with RGB obtains the best clean performance, reaching 0.884 mean ROC-AUC, but degrades substantially on the harder partition. In contrast, frozen DINOv3 achieves the strongest degraded-set result, with 0.726 mean ROC-AUC, while training only a linear classification head. The representation analysis indicates that Fourier-domain cues are most useful when combined with RGB information, whereas purely spectral inputs consistently underperform spatial representations. Qualitative attribution maps further suggest that convolutional detectors focus on localized artifacts, while DINOv3 relies on broader facial structure. These findings reinforce the need for degraded evaluation protocols and highlight self-supervised visual representations as a promising direction for robust face forgery detection. Our source code is publicly available at https://github.com/lucasdocunha/FaceForgery-Benchmark/.

I. INTRODUCTION

The paper addresses the limited reliability of face forgery detectors under realistic image degradation by establishing a standardized MFFI benchmark. It evaluates architectures and spatial, spectral, and hybrid representations under common clean and degraded protocols.

Realistic compression, resizing, blurring, and re-encoding can suppress forensic traces and sharply reduce detector performance after distribution shifts.
MFFI provides more than 50 forgery techniques, diverse authentic sources, and clean and degraded test partitions for realistic robustness evaluation.
The benchmark evaluates six model families spanning convolutional, attention-based, and self-supervised architectures under fixed protocols.
Seven spatial, spectral, and hybrid input representations are compared to identify when frequency cues help or fail.
Attribution methods analyze the evidence used by accurate and degraded detectors, alongside comparisons with state-of-the-art MFFI results.

II. RELATED WORK

Prior work established strong in-distribution face forgery detection and motivated frequency-domain analysis. These studies also exposed sensitivity to distribution changes and motivated evaluating multiple spectral and hybrid representations.

FaceForensics++ and Xception demonstrated effective detection of several facial manipulations on curated benchmarks.
A CNN can generalize to unseen generators with suitable augmentation, but performance decreases when the test distribution changes.
Prior studies reported structured Fourier artifacts and measurable spectrum discrepancies in generated images, motivating frequency-aware detection.
This paper extends frequency-domain work by comparing seven spatial, spectral, and hybrid representations rather than a single spectral input.

C. Foundation Models and Self-Supervised Learning

The paper examines whether pretrained visual representations can provide forensic robustness without learning all evidence solely from MFFI. It includes frozen DINOv3 features within a benchmark spanning diverse data and degradation conditions.

CLIP, DINOv2, and DINOv3 provide pretrained visual representations learned through image-text supervision or self-supervision.
Recent robust deepfake detection systems use adapted DINOv3 backbones, motivating evaluation of frozen DINOv3 features without backbone fine-tuning.
The benchmark uses MFFI binary classification with official train, validation, and test splits, evaluating robustness on degraded Test-Hard images.

B. Input Representations

The benchmark constructs spatial, spectral, and hybrid inputs from RGB images and evaluates them across heterogeneous architectures. Spectral modes encode Fourier-derived information, while hybrid modes combine spatial and spectral channels.

B. Input Representations: Spectral inputs convert RGB images to grayscale luminance before applying a centered two-dimensional FFT.
B. Input Representations: The seven representations include RGB, log-magnitude, phase, real and imaginary components, high-pass magnitude, and hybrid spatial-spectral inputs.
B. Input Representations: Log-magnitude captures energy across spatial frequencies, phase preserves structural information, and high-pass magnitude emphasizes high-frequency artifacts.
B. Input Representations: Hybrid modes concatenate spatial and spectral channels, and channel-mismatched first layers are reinitialized for the resulting inputs.
C. Architectures: The benchmark spans convolutional, attention-based, and self-supervised model families, with DINOv3 using a frozen backbone and trained linear head.

D. Training and Evaluation

The benchmark uses a fixed optimization and evaluation protocol across models, with augmentation, imbalance handling, repeated runs, and threshold-independent primary scoring.

All models use AdamW, cross-entropy loss, weight decay 10^-4, gradient clipping, mixed precision, early stopping, and plateau-based learning-rate scheduling.
Class imbalance is addressed with a weighted random sampler, while augmentation applies resized crops, flips, and color jitter before FFT computation.
Results are reported as means and standard deviations across three independent runs using different random seeds.
The validation-selected decision threshold is fixed for Test and Test-Hard, with ROC-AUC as the primary metric and accuracy reported at that threshold.
Grad-CAM analyzes convolutional models and DINOv3, whereas Attention Rollout analyzes the ViT and CLIP-style encoder.

IV. RESULTS AND DISCUSSION

Clean performance favors convolutional detectors, but degradation changes the ranking: frozen DINOv3 performs best on Test-Hard, while spectral cues help mainly when combined with RGB.

Clean-set performance favors task-specific convolutional models, whereas degradation changes the model ranking.
0.884 ROC-AUC is Xception’s best clean RGB result, exceeding ResNet-18 at 0.846 and MobileNetV3 at 0.839.
0.726 ROC-AUC is frozen DINOv3’s Test-Hard result, surpassing Xception with RGB+Mag at 0.650 and ResNet-18 with RGB+Freq at 0.648.
Xception with RGB falls from 0.884 on Test to 0.609 on Test-Hard, while DINOv3 falls from 0.809 to 0.726.
The ViT’s small absolute degradation drop reflects low initial performance, creating a floor effect rather than evidence of strong robustness.

B. Spatial, Spectral, and Hybrid Inputs

Spectral information provides limited clean-set benefit but becomes more useful under degradation when combined with RGB. Purely spectral inputs remain weaker, while frozen DINOv3 is competitive with published degraded-set results under comparable protocols.

Hybrid representations: Hybrid RGB+Mag or RGB+Freq representations match or outperform RGB for every convolutional architecture on Test-Hard.The gains indicate that spectral cues can complement spatial evidence when degradation weakens RGB-based cues.
Spectral representations: Purely spectral representations remain substantially weaker than spatial representations on both clean and degraded partitions.Spectral cues do not replace RGB information in the evaluated models.
Published comparison: 0.726 mean ROC-AUC is achieved by frozen DINOv3 on Test-Hard, marginally exceeding the strongest published degraded-set result of 0.720.DINOv3 uses only a linear head on a frozen backbone.
Published comparison: 0.884 ROC-AUC is achieved by Xception with RGB on the clean partition, exceeding SRM at 0.877 and the official Xception baseline at 0.852.The comparison uses the comparable MFFI-training protocol.
Published comparison: Clean-set leaders are not necessarily the strongest models after degradation.Published comparisons distinguish clean performance from robustness on Test-Hard.

D. Qualitative Evidence from Attribution Maps

Attribution maps reveal distinct evidence patterns across model families: convolutional detectors localize facial artifacts, from-scratch transformers are diffuse, and DINOv3 spreads evidence across facial landmarks. These qualitative patterns align with differing false-positive behavior and degraded robustness.

Analysis setup: Attribution maps are generated for six RGB models using Grad-CAM for convolutional detectors and DINOv3, and Attention Rollout for the transformers.The analysis uses four common samples: two forged and two authentic examples.
Observed predictions: All six models correctly classify both forged samples, but ViT, CLIP-style encoder, and DINOv3 classify both authentic samples as forged.The illustrative samples are not a quantitative evaluation.
Convolutional detectors: Convolutional detectors activate on localized regions such as the nose, mouth, eyes, or blending boundaries in forged images.Their authentic-image maps are more diffuse, suggesting reliance on the absence of localized anomalies.
From-scratch transformers: The ViT and CLIP-style encoder show scattered or weakly localized activations and broad activation on authentic faces.The CLIP-style encoder’s pattern is consistent with its false-positive errors.
DINOv3: DINOv3 distributes evidence across multiple semantically meaningful facial landmarks rather than concentrating on one region.This broader allocation is consistent with Test-Hard robustness but may reduce clean-authentic discrimination.

E. Limitations

The study’s conclusions are constrained by its single-dataset evaluation, mostly from-scratch training, frozen-only DINOv3 assessment, and qualitative four-sample attribution analysis.

Scope: All experiments use one dataset, so robustness is measured only against the MFFI degradation protocol rather than cross-dataset generalization.The reported ranking should be validated on additional forgery sources.
Training regime: Five of six model families are trained from scratch, isolating architecture learning on MFFI but not large-scale pretraining performance.This design does not estimate performance attainable with pretrained models.
DINOv3 evaluation: DINOv3 is evaluated only as a frozen feature extractor with a linear head.The experiments do not establish whether its robustness advantage persists after fine-tuning.
Attribution analysis: Attribution analysis is qualitative, correlational, and based on only four samples.Grad-CAM maps for the frozen transformer require token reshaping and are approximate visualizations.

V. CONCLUSIONS

The benchmark shows that clean-set performance is an unreliable proxy for robustness under realistic degradation: Xception leads on clean images, while frozen DINOv3 leads on degraded images. Hybrid spatial-spectral inputs can help convolutional models, whereas purely spectral inputs underperform RGB, and attribution patterns differ across model families.

V. CONCLUSIONS: 0.884 ROC-AUC: Xception achieved the strongest clean result, but performance decreased substantially after compression, resizing, and blurring.Frozen DINOv3 obtained the highest degraded-set result, 0.726, while training only a linear classification head.
V. CONCLUSIONS: 0.726 degraded-set result: frozen DINOv3 achieved the strongest degraded performance while requiring only a trained linear classification head.This contrasts with Xception’s strongest clean result and substantial degradation after common image transformations.
V. CONCLUSIONS: Hybrid spatial-spectral inputs can improve convolutional models under degradation, whereas purely spectral representations consistently underperform RGB.The findings identify Fourier-domain information as most useful when used as an auxiliary cue.
V. CONCLUSIONS: Attribution analysis suggests convolutional detectors emphasize localized facial evidence, while DINOv3 distributes evidence across broader facial structure.These attribution patterns provide a qualitative distinction between the evidence used by convolutional and self-supervised representations.
V. CONCLUSIONS: Future evaluation should separate robustness to image degradation from generalization across forgery sources by testing additional datasets and degradation types.The paper also identifies comparisons of frozen versus fine-tuned backbones and degradation-aware training as directions for future work.