Source-linked AI summary
Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues
Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, Jing Shao
TL;DR
Face forgery detection remains challenging when refined manipulations and compression hide artifacts from human and RGB-domain inspection. F3-Net uses complementary frequency-aware decomposition and local frequency statistics in a collaborative two-stream network, outperforming reference methods across FaceForensics++ quality settings, especially in low-quality media.
Problem
Refined face manipulations can evade human inspection, and compression may obscure forgery artifacts in RGB images, motivating frequency-aware detection.
Method
F3-Net combines frequency-aware image decomposition, local frequency statistics, and cross-attention-based collaborative learning, using DCT for frequency transformation.
Results
F3-Net outperforms reference methods on RAW, HQ, and LQ FaceForensics++ settings, achieving 90.43% Acc and 0.933 AUC in LQ evaluation.
Takeaways & Limitations
Frequency-aware FAD and LFS clues provide an effective basis for face forgery detection, particularly for challenging low-quality media.
Takeaways & Limitations
FAD partitions the DCT spectrum into bands with roughly equal energy, reflecting an assumption about natural-image spectral distributions.
Abstract
from arXiv · showhide
As realistic facial manipulation technologies have achieved remarkable progress, social concerns about potential malicious abuse of these technologies bring out an emerging research topic of face forgery detection. However, it is extremely challenging since recent advances are able to forge faces beyond the perception ability of human eyes, especially in compressed images and videos. We find that mining forgery patterns with the awareness of frequency could be a cure, as frequency provides a complementary viewpoint where either subtle forgery artifacts or compression errors could be well described. To introduce frequency into the face forgery detection, we propose a novel Frequency in Face Forgery Network (F3-Net), taking advantages of two different but complementary frequency-aware clues, 1) frequency-aware decomposed image components, and 2) local frequency statistics, to deeply mine the forgery patterns via our two-stream collaborative learning framework. We apply DCT as the applied frequency-domain transformation. Through comprehensive studies, we show that the proposed F3-Net significantly outperforms competing state-of-the-art methods on all compression qualities in the challenging FaceForensics++ dataset, especially wins a big lead upon low-quality media.
1 Introduction
Face forgery detection is difficult because refined manipulations can evade human and RGB-domain inspection, particularly after heavy compression. F3-Net addresses this challenge by collaboratively mining complementary frequency-aware clues through FAD and LFS.
- The proposed framework targets security concerns arising from malicious distribution of forged face media that can be imperceptible to human observers.
- Heavy compression can contaminate or obscure forgery artifacts in RGB images, while frequency-domain analysis can reveal unusual distributions relative to real faces.
- FAD decomposes images into learnable frequency-band components to expose subtle manipulation patterns, especially in higher-frequency components.
- LFS extracts local frequency statistics that describe statistical discrepancies between real and forged faces while retaining spatial structure suitable for CNNs.
- F3-Net progressively fuses FAD and LFS features through the cross-attention MixBlock in a two-stream collaborative framework.
2 Related Work
Prior face forgery detection primarily used spatial-domain cues, whereas frequency-domain methods transform images to expose artifacts and distributions that may be difficult to detect in color space. F3-Net combines frequency-aware decomposition, local statistics, and collaborative feature interaction.
- Spatial-Based Forgery Detection: Most spatial-domain methods use RGB or HSV information, but subtle manipulation clues may remain difficult to detect in color space.
- F3-Net introduces FAD for frequency-aware image decomposition, LFS for local frequency statistics, and MixBlock for collaborative feature interaction.
- Frequency-Based Forgery Detection: Frequency-domain forgery detection applies transformations such as Wavelet or Discrete Fourier Transform to mine underlying artifacts.
3 Our Approach
F3-Net mines complementary frequency-aware forgery clues through Frequency-aware Decomposition and Local Frequency Statistics, then fuses their representations with cross-attention.
- FAD: Frequency-Aware Decomposition: FAD adaptively partitions input images into learnable frequency bands and reconstructs frequency-aware image components for CNN-based forgery-pattern learning.DCT is used for frequency transformation, and learnable filters adapt the fixed base-band partitioning.
- FAD: Frequency-Aware Decomposition: FAD uses DCT because its frequency layout separates low and high responses, while JPEG and H.264 also use DCT.This makes DCT-based decomposition compatible with describing compression artifacts alongside forgery patterns.
- LFS: Local Frequency Statistics: LFS applies sliding-window DCT, gathers statistics within learnable frequency bands, and reassembles them into a spatially aligned multi-channel map.Band-wise statistics reduce the representation and smooth distributions by limiting outlier interference.
- Two-stream Collaborative Learning Framework: The two-stream framework processes FAD components and LFS maps with Xception branches, exchanging information through cross-attention MixBlocks.MixBlock computes cross-attention between branch feature maps rather than simply concatenating them.
- Experimental Setup: Table 1 evaluates the method on FaceForensics++ under LQ, HQ, and RAW quality settings against Xception-based and artifact-feature baselines.LQ denotes heavy compression, HQ light compression, and RAW uncompressed videos.
4 Experiment
Experiments on FaceForensics++ evaluate F3-Net across video qualities, manipulation types, video backbones, and component ablations. F3-Net consistently improves detection, with especially strong gains in low-quality settings and when frequency-aware components are combined.
- Evaluations on Different Quality Settings: F3-Net outperforms all reference methods on FaceForensics++ RAW, HQ, and LQ settings, reaching 90.43% Acc and 0.933 AUC on LQ.On LQ, it gains about 3.5% Acc over Xception-PAFilters, which achieves 87.16%.
- Towards Different Manipulation Types: F3-Net improves Acc by about 4.2% over Xception when detecting NeuralTextures forgeries in low-quality videos.NeuralTextures is described as especially challenging because its synthesized faces have few noticeable artifacts.
- Comparing with previous methods: In t-SNE embeddings, Xception cannot separate real videos from NeuralTextures forgeries, whereas F3-Net places them farther apart.This visualization provides a feature-space view of F3-Net’s improved discrimination on the LQ task.
- Video-based Extensions: Slowfast-F3-Net achieves 93.02% Acc and 0.958 AUC on LQ videos, compared with 90.53% and 0.936 for Slowfast alone.The frequency-aware extension also wins over 3% on NeuralTextures manipulation.
- Effectiveness of LFS, FAD and MixBlock: Adding FAD, then LFS, then MixBlock progressively improves Acc and AUC, with the complete model reaching 90.43% and 0.933.The ablation results describe FAD and LFS as complementary and attribute additional gains to MixBlock cooperation.
- Ablation study on FAD: Among FAD-Low, FAD-Mid, and FAD-High, the high-frequency variant achieves the best scores, indicating that high-frequency clues help forgery detection.The component analysis also compares FAD-All, which uses all frequency bands.
- Ablation study on LFS: SWDCT improves substantially over traditional full-image DCT, while frequency statistics are more robust to unstable or noisy spectra.The proposed adaptive frequency statistics further optimize the use of local spectral information.
5 Conclusions
The paper concludes that F3-Net detects face forgeries by combining complementary frequency-aware clues with collaborative learning. Extensive FaceForensics++ experiments demonstrate effectiveness, particularly for low-quality media.
- 5 Conclusions: F3-Net combines frequency-component partitioning, local frequency-statistics discrepancy, and a cross-attention module for two-stream collaborative learning.The two branches target subtle forgery patterns and small-scale statistical differences between real and forged images.
- 5 Conclusions: Extensive FaceForensics++ experiments demonstrate F3-Net’s effectiveness, especially on the challenging low-quality task.
6 Appendix
The appendix examines evaluation thresholds and SWDCT hyperparameters for F3-Net, including accuracy reporting across FaceForensics++ quality settings and manipulation methods. It selects a window size of 10 and stride of 2 as practical settings for LFS.
- Threshold selection: Accuracy depends strongly on the classification threshold θ, so the appendix compares greedy-searched thresholds with the standard θ = 0.5.The greedy search selects the highest validation accuracy on FaceForensics++.
- Accuracy evaluation: The appendix reports accuracy comparisons for F3-Net and baselines across low-, high-, and raw-quality FaceForensics++ data.The tables distinguish LQ heavy compression, HQ light compression, and RAW videos without compression.
- Accuracy evaluation: Accuracy is also compared across DeepFakes, Face2Face, FaceSwap, and NeuralTextures in the low-quality FaceForensics++ setting.The comparisons include both greedy-searched and standard-threshold accuracy results.
- SWDCT window size: A window size of 10 is selected for SWDCT because size 2 misses high-frequency structure near edges, whereas sizes 20 and 30 reduce sensitivity to local abnormal statistics.The appendix evaluates window sizes 2, 5, 10, 20, and 30 in low-quality tasks with stride 2.
- SWDCT stride size: A stride of 2 is selected because it provides the best trade-off between detection effectiveness and computational efficiency.Stride values 10, 6, 4, 3, 2, and 1 are evaluated in low-quality tasks with window size 10.