Source-linked AI summary
Representative Forgery Mining for Fake Face Detection
Chengrui Wang, Weihong Deng
TL;DR
Vanilla CNN fake-face detectors can perform well but tend to focus on limited facial regions, motivating a method that broadens their forgery attention. RFM uses attention-guided erasing during training, and the paper reports representative-forgery visualization and state-of-the-art performance on DFFD and Celeb-DF.
Problem
Vanilla CNN detectors tend to seek forgeries in a limited facial region, indicating limited understanding of representative forgery.
Method
RFM generates gradient-based Forgery Attention Maps and uses Suspicious Forgeries Erasing to occlude Top-N sensitive regions during training.
Results
RFM enables a vanilla CNN-based detector to achieve state-of-the-art performance on DFFD and Celeb-DF while separately visualizing representative forgeries from different manipulation techniques.
Takeaways & Limitations
The method guides detectors toward previously ignored facial regions and can be integrated with various CNN models without structure modification.
Takeaways & Limitations
The work formulates fake-face detection as a binary classification problem.
Abstract
from arXiv · showhide
Although vanilla Convolutional Neural Network (CNN) based detectors can achieve satisfactory performance on fake face detection, we observe that the detectors tend to seek forgeries on a limited region of face, which reveals that the detectors is short of understanding of forgery. Therefore, we propose an attention-based data augmentation framework to guide detector refine and enlarge its attention. Specifically, our method tracks and occludes the Top-N sensitive facial regions, encouraging the detector to mine deeper into the regions ignored before for more representative forgery. Especially, our method is simple-to-use and can be easily integrated with various CNN models. Extensive experiments show that the detector trained with our method is capable to separately point out the representative forgery of fake faces generated by different manipulation techniques, and our method enables a vanilla CNN-based detector to achieve state-of-the-art performance without structure modification.
1. Introduction
The paper identifies limited facial attention as a weakness of vanilla CNN detectors and proposes RFM to guide them toward broader, manipulation-representative forgery regions.
- Problem: Vanilla CNN detectors may inspect forgeries within a limited facial region, unlike humans who identify representative forgery across the face.The paper motivates broader attention as necessary for detecting forgeries that vary by manipulation technique.
- Problem: Deepfakes and Face2Face forgeries mainly occur at facial boundaries, whereas StyleGAN and PGGAN forgeries can appear anywhere on the face.
- Method: RFM generates a gradient-based Forgery Attention Map and erases Top-N sensitive facial regions to expose previously ignored regions during training.The erased inputs encourage the detector to mine more representative forgery without requiring additional supervision.
- Method: RFM refines training data dynamically and can be integrated with various CNN models without structure modification or a sophisticated training set.
- Results: RFM enables vanilla CNN detectors to achieve state-of-the-art performance on DFFD and Celeb-DF while visualizing representative forgery from different manipulation techniques.
2. Related Work
The related work covers face-manipulation procedures, existing fake-face detectors, and conventional data-augmentation methods, positioning RFM among guided erasing approaches.
- Face Manipulation Techniques: Face manipulation techniques are broadly divided into one-stage and two-stage methods according to their technical procedure.
- Face Manipulation Techniques: Two-stage techniques generate or extract a target face and blend its identity into a source face, supporting identity-swap and expression-manipulation applications.
- Face Manipulation Techniques: One-stage techniques commonly use GANs to synthesize entire faces or manipulate facial expressions and attributes without complex physical models.
- Fake Face Detection: Prior fake-face detectors use attention layers, semantic segmentation, spatial-frequency inputs, or generator digital fingerprints to locate or detect forgeries.
- Data Augmentation: Data augmentation methods include cropping, flipping, dropout, Mixup, Cutout, Random Erasing, and CAM-guided Adversarial Erasing.
3. Proposed Method
RFM combines an image-level Forgery Attention Map with Suspicious Forgeries Erasing to guide attention away from over-sensitive facial regions. It dynamically occludes top-ranked sensitive regions while preserving facial information and avoiding erasing-mask overfitting.
- Forgery Attention Map: RFM uses Forgery Attention Map (FAM) to reveal the detector’s sensitivity across facial image regions.FAM is computed from the maximum absolute difference between gradients of the real and fake logits.
- Forgery Attention Map: FAM measures pixel-level sensitivity by comparing how perturbations affect the detector’s real and fake logits.The maximum value is taken along the channel axis, and absolute values represent perturbation impact.
- Comparison with well-known erasing methods: SFE differs from Adversarial Erasing because it uses image-level FAM rather than last-convolutional-layer Class Activation Mapping to guide occlusion.The paper contrasts FAM’s sensitivity localization with Class Activation Mapping’s decision-region highlighting.
- Suspicious Forgeries Erasing: Suspicious Forgeries Erasing (SFE) sorts facial coordinates by FAM values and occludes the Top-N most sensitive regions during training.The method generates FAM for each mini-batch image before selecting erasing anchors.
- Comparison with well-known erasing methods: The erasing algorithm fills selected image blocks with random integers to produce an erased image for detector training.Its inputs include the facial image, FAM, erasing-block count, erasing probability, and maximum erase sizes.
- Suspicious Forgeries Erasing: SFE uses multiple flexible erasing blocks to target separated sensitive regions while preserving as much effective facial information as possible.Unlike random erasing, the blocks are guided by FAM and can cover forgeries at different locations.
4. Experiments
Experiments on DFFD and Celeb-DF show that RFM improves detection, remains effective on less-forgery faces, and visualizes representative regions across manipulation techniques.
- Ablation and method components: RFM combines Forgery Attention Map guidance with Suspicious Forgeries Erasing to refine detector attention during training.FAM locates sensitive facial regions, while erasing-based augmentation redirects attention toward previously ignored regions.
- Ablation and method components: 8.33% TDR0.01% and 3.88% TDR0.1% improvements over the baseline detector were achieved by combining MEB and FAM on DFFD.The combined setting outperformed the other ablation settings by a large margin.
- Hyper-parameter analysis: Hyper-parameter effects varied across Celeb-DF subsets, with Hmax&Wmax more decisive than p and overly large settings degrading performance on subsetA.The authors recommend increasing parameters from small values until reaching optimal performance.
- Comparison with baselines: RFM outperformed baseline and other erasing methods on DFFD and achieved competitive state-of-the-art performance on DFFD and Celeb-DF.RFM-generated images also improved Patch, indicating that the method integrates with different detector backbones.
- Comparison with baselines: 6.16% TDR0.1% and 6.61% TDR0.01% improvements over [11] were obtained with RFM on incomplete DFFD under matched training iterations.The comparison used the same iteration budget as the reference setting.
- Robustness and visualization: RFM maintained detection performance on less-forgery faces, whereas detectors without RFM experienced severe degradation.Average FAMs also highlighted more comprehensive regions, including facial boundaries for two-stage techniques and entire skin regions for one-stage techniques.
- Robustness and visualization: Average FAMs revealed representative forgery patterns by manipulation technique, with similar techniques producing similar maps and four frames sufficient for a discriminative contour.The contour and inner structure became more complete as the number of frames increased from 4 to 256.
5. Conclusion
Refining training data with Representative Forgery Mining guides detectors toward representative forgeries across previously ignored facial regions. The method enables vanilla CNN detectors to identify manipulation-specific forgeries and achieve state-of-the-art performance on DFFD and Celeb-DF.
- Representative Forgery Mining improves fake face detection by refining training data to guide exploration of previously ignored facial regions.
- The method discovers representative forgeries corresponding to different manipulation techniques without requiring well-designed supervision.
- A vanilla CNN-based detector using the method achieves state-of-the-art performance on the DFFD and Celeb-DF datasets.