Source-linked AI summary
Exploiting Fine-grained Face Forgery Clues via Progressive Enhancement Learning
Qiqi Gu, Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, Ran Yi
TL;DR
Realistic face forgeries create security concerns, while existing frequency-based methods use coarse information and vanilla learning struggles with fine-grained traces. The paper proposes PEL, combining fine-grained frequency decomposition with a two-branch RGB-frequency network and self- and mutual-enhancement modules. Experiments report state-of-the-art performance on seen and unseen datasets, alongside evidence of generalization and robustness.
Problem
Existing frequency-information exploitation is coarse-grained, and vanilla learning struggles to extract fine-grained forgery traces from high-quality forged faces.
Method
PEL decomposes RGB images into fine-grained frequency components and progressively enhances RGB and frequency features through self-enhancement and mutual-enhancement modules in a two-branch network.
Results
PEL achieves state-of-the-art performance on both seen and unseen datasets, with cross-dataset evaluations indicating generalization and perturbed evaluations indicating robustness.
Takeaways & Limitations
Fine-grained frequency clues and progressive feature enhancement provide a framework for exploiting subtle forgery traces in high-quality faces.
Abstract
from arXiv · showhide
With the rapid development of facial forgery techniques, forgery detection has attracted more and more attention due to security concerns. Existing approaches attempt to use frequency information to mine subtle artifacts under high-quality forged faces. However, the exploitation of frequency information is coarse-grained, and more importantly, their vanilla learning process struggles to extract fine-grained forgery traces. To address this issue, we propose a progressive enhancement learning framework to exploit both the RGB and fine-grained frequency clues. Specifically, we perform a fine-grained decomposition of RGB images to completely decouple the real and fake traces in the frequency space. Subsequently, we propose a progressive enhancement learning framework based on a two-branch network, combined with self-enhancement and mutual-enhancement modules. The self-enhancement module captures the traces in different input spaces based on spatial noise enhancement and channel attention. The Mutual-enhancement module concurrently enhances RGB and frequency features by communicating in the shared spatial dimension. The progressive enhancement process facilitates the learning of discriminative features with fine-grained face forgery clues. Extensive experiments on several datasets show that our method outperforms the state-of-the-art face forgery detection methods.
1 Introduction
Face forgery detection addresses increasingly realistic manipulations that are difficult for humans to discern and harmful to the community. PEL combines fine-grained frequency decomposition with progressive enhancement of RGB and frequency features to extract subtle forgery clues, achieving strong results across datasets.
- Realistic face forgeries are difficult to distinguish by eye and have been used in pornography and political rumors, motivating face forgery detection.
- Prior detectors use handcrafted or semantic CNN features, while newer methods mine artifacts through auxiliary supervision such as blending boundaries or forged masks.
- PEL decomposes RGB inputs into fine-grained frequency components with patch-wise DFT and sliding windows, providing complementary information and more feature combinations for artifact discovery.
- Its two-stream network uses self-enhancement for spatial-noise and channel-attention processing, and mutual-enhancement for communication between RGB and frequency branches.
- Enhancements are inserted after convolutional blocks so earlier improvements facilitate later feature extraction and magnify hidden forgery cues.
- PEL achieves state-of-the-art performance on FaceForensics++ and WildDeepfake, with cross-dataset evaluations indicating generalization and perturbed evaluations indicating robustness.
2 Related work
Face forgery detection has progressed from handcrafted artifact descriptors to auxiliary supervision, local-region similarity, and frequency-domain cues. These approaches improve artifact mining, but mask-dependent methods are limited on some wild datasets, motivating broader applicability.
- Early methods distinguish authentic and forged faces using handcrafted steganalysis, convolutionalized features, facial-region descriptors, or facial inconsistency estimates.
- Auxiliary-supervision approaches jointly use classification and segmentation or emphasize manipulated regions and blending boundaries to mine forgery cues.
- Local-region similarity methods achieve superior performance on benchmark datasets, while frequency-domain methods target artifacts that are difficult to observe directly.
- Manipulated masks required by some methods are not applicable to certain wild datasets, including WildDeepfake, limiting practical application.
3 Method
PEL decomposes RGB inputs into fine-grained frequency components and progressively enhances complementary RGB and frequency features through self- and mutual-enhancement modules.
- 3.1 Fine-grained Clues: PEL transforms RGB inputs into fine-grained frequency components and feeds them with RGB features into a two-stream EfficientNet network.The decomposition preserves spatial relationships and produces a 64-channel frequency input after adaptive 1×1 convolution.
- 3.4 Progressive Enhancement: Enhancement modules are inserted throughout the backbone so earlier strengthened features support later extraction while preserving stream-specific clues.Mutual enhancement follows self-enhancement after the second through penultimate convolutional blocks; final stream outputs are concatenated for classification.
- 3.2 Self-Enhancement Module: The self-enhancement module separately strengthens each stream using noise enhancement and channel attention to expose subtle forgery traces.Noise enhancement extracts feature noise with median filtering, Sigmoid amplification, and depth-wise convolution before adding it back to the input feature.
- 3.3 Mutual-Enhancement Module: The mutual-enhancement module concatenates dual-stream features to generate spatial attention maps that enhance both branches through shared spatial communication.Each stream is multiplied by its attention map, intensity-adjusted with depth-wise 1×1 convolution, and added to the input without changing feature size.
- 3.4 Loss Function: The network is trained for binary classification with Binary Cross-Entropy loss and back-propagation.The concatenated outputs of the two streams are used for classification before optimization.
4 Experiment
Experiments evaluate PEL across standard, cross-dataset, robustness, ablation, denoising, and visualization settings. The method is tested on five public datasets and shows strong performance, generalization, robustness, and complementary RGB-frequency behavior.
- Experimental Setup: Experiments use five public datasets, with FaceForensics++ and WildDeepfake supporting training and evaluation, and Celeb-DF, DFD, and DFDC used for cross-dataset evaluation.FaceForensics++ includes HQ and LQ settings; WildDeepfake contains diverse internet-collected face sequences.
- Intra-testing: PEL outperforms all compared methods in Acc and AUC across FaceForensics++ quality settings and WildDeepfake.The authors attribute these gains to excavating and magnifying fine-grained forgery cues.
- Cross-testing: PEL significantly outperforms competitors on unseen WildDeepfake, Celeb-DF, DFD, and DFDC datasets under cross-dataset evaluation.Models trained on FF++ (LQ) are evaluated using AUC and EER; training on WildDeepfake yields similar results.
- Robustness: PEL degrades the least in most perturbation cases, including Gaussian noise, salt-and-pepper noise, and Gaussian blur.Robustness is assessed by the decay of Acc and AUC, denoted ΔAcc and ΔAUC.
- Components: Ablations show peak performance when the RGB-frequency two-stream design uses both self-enhancement and mutual-enhancement modules.The two inputs are distinct and complementary, while self-enhancement extracts within-stream information and mutual-enhancement integrates the streams.
- Visualizations: Visualizations show complementary attention and enhancement patterns: RGB focuses on manipulated regions, whereas frequency features cover broader facial areas and reveal subtler artifacts.Self-enhancement is stronger in local regions such as eyes or mouth, while mutual-enhancement intensifies manipulated regions across both streams.
5 Conclusion
PEL enhances discriminative representations using fine-grained frequency information alongside RGB input, achieving state-of-the-art performance on seen and unseen datasets.
- PEL decomposes images into fine-grained frequency components and processes them with RGB input in a two-stream network.The frequency representation supports the subsequent enhancement modules.
- The self-enhancement module captures traces separately across input spaces using spatial noise enhancement and channel attention.
- The mutual-enhancement module concurrently enhances RGB and frequency branches through feature communication in their shared spatial dimension.
- PEL fully exploits fine-grained clues in high-quality forgery faces and achieves state-of-the-art performance on seen and unseen datasets.
- Feature-map and class-activation visualizations reveal the framework’s inner mechanism and explain its effectiveness.