Source-linked AI summary
Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection
Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, Jue Wang
TL;DR
Deepfake detectors struggle to generalize from known forgery datasets and methods to unseen ones. The paper synthesizes diverse, challenging forgeries and trains detectors to predict their configurations, reporting improved generalization and favorable comparisons with state-of-the-art detectors.
Problem
Deepfake detectors show a performance drop when recognizing forgeries from unseen datasets and methods.
Method
The method combines adversarial forgery augmentation with self-supervised prediction of forgery configurations.
Results
The proposed strategies are effective and perform favorably against other state-of-the-art detectors.
Takeaways & Limitations
Training on dynamically synthesized forgeries and configuration-prediction tasks produces more robust feature representations and a more generalizable detector.
Takeaways & Limitations
The synthesizer's augmentation types remain limited.
Abstract
from arXiv · showhide
Recent studies in deepfake detection have yielded promising results when the training and testing face forgeries are from the same dataset. However, the problem remains challenging when one tries to generalize the detector to forgeries created by unseen methods in the training dataset. This work addresses the generalizable deepfake detection from a simple principle: a generalizable representation should be sensitive to diverse types of forgeries. Following this principle, we propose to enrich the "diversity" of forgeries by synthesizing augmented forgeries with a pool of forgery configurations and strengthen the "sensitivity" to the forgeries by enforcing the model to predict the forgery configurations. To effectively explore the large forgery augmentation space, we further propose to use the adversarial training strategy to dynamically synthesize the most challenging forgeries to the current model. Through extensive experiments, we show that the proposed strategies are surprisingly effective (see Figure 1), and they could achieve superior performance than the current state-of-the-art methods. Code is available at \url{https://github.com/liangchen527/SLADD}.
1 The University of Adelaide 2 Tencent AI Lab
The passage lists contact email addresses for the authors.
- The listed contact addresses use the gmail.com domain.
- The address list contains four comma-separated entries.
- The addresses are presented as author contact information.
1. Introduction
Deepfake detectors often lose performance on forgeries from unseen datasets and methods. The paper proposes diverse synthesized forgeries, configuration-prediction tasks, and adversarial training to improve generalization.
- Detectors perform well when training and testing use the same dataset and forgery methods, but performance drops on forgeries outside the training data.
- Limited augmentation variety and dataset-sensitive handcrafted artifacts can hinder generalization across deepfake datasets.
- The method synthesizes forgeries from configurations specifying the forgery region, blending type, and blending ratio.
- The detector predicts forgery configurations in addition to classifying inputs as real or fake.
- Experiments report effectiveness for adversarial augmentation and self-supervised tasks, with favorable performance against other state-of-the-art detectors.
2. Related Works
Prior deepfake-detection research uses handcrafted forensic cues, auxiliary tasks, and adversarial learning, while this paper positions its approach as tailored to forgery generation.
- Deepfake detection: Deepfake detectors have used blending artifacts, high-frequency details, phase spectra, lip reading, facial decomposition, and landmark geometry as classification cues.
- The paper defines a configuration as a specific way of synthesizing a forgery image.
- The cited figure provides examples of pristine images, reference images, and corresponding synthesized adversarial forgeries.
- Deepfake detection: Auxiliary tasks can improve detection, but their generalization is questionable and obtaining annotations is expensive.
- Adversarial learning: Adversarial training commonly combines a generator and discriminator to augment samples or choose augmentation policies.
- Adversarial learning: Unlike general adversarial-training strategies, this method models its adversarial synthesis process on deepfake generation.
3. Proposed Method
The method combines adversarial forgery synthesis with self-supervised prediction of forgery configurations. A synthesizer generates challenging forgeries, while a detector predicts authenticity, regions, blending types, and blending ratios.
- 3. Proposed Method: Adversarial data augmentation enriches forgery types, while self-supervised tasks enforce sensitivity to forgery configurations.The synthesizer acts as a generator and the detector as a discriminator in adversarial training.
- 3.1. Selecting Space and Synthesizing Forgery: The synthesizer takes pristine and reference images and outputs a region index, blending type, and mix-up blending ratio.The region and blending type are sampled from predicted distributions, while the ratio is generated as a continuous scalar.
- 3.1. Selecting Space and Synthesizing Forgery: The forgery configuration space covers 10 facial regions, four blending choices including do-nothing, and a mix-up ratio in [0, 1].The three active blending techniques are alpha, Poisson, and mixup; the ratio is effective only for mixup blending.
- 3.1. Selecting Space and Synthesizing Forgery: New forgeries are formed by deforming and blurring the selected region, cropping facial parts from the reference, and blending them into the pristine image.Alpha and Poisson blending use OpenCV implementations, while mixup blending uses the generated ratio and deformed mask.
- 3.2. Self-supervised Tasks: The detector jointly predicts real versus fake labels, forgery regions, blending types, and blending ratios when mixup applies.The region task uses an L1 loss against generated or dataset-provided masks, and the ratio loss is gated by mixup selection.
- 3.3. Adversarial Training: Training solves a weighted min-max objective in which the detector minimizes combined task losses and the synthesizer maximizes the detector’s training loss.The generator and discriminator are updated iteratively using gradient-based optimization.
4. Experiments
Experiments evaluate generalization across datasets, compression levels, multitask baselines, state-of-the-art detectors, and controlled ablations. The proposed method generally performs best or favorably, with adversarial augmentation and self-supervised tasks contributing to performance.
- Experimental setup: Models are trained on four FF++ forgery methods and evaluated on CelebDF, DFDC, and DF1.0, where both pristine and forged test data are unseen during training.This setting directly tests cross-dataset generalization under unseen data sources and forgeries.
- Generalizability comparisons: The proposed method outperforms other models in most cross-dataset cases and achieves the overall best performance by AUC.Comparisons include Xception, Face X-ray, F3Net, RFM, and SRM under matched training settings.
- Compression robustness: The proposed method is substantially less affected by compression levels and outperforms other methods when testing uses unseen compression conditions.Methods relying on imperceptible image patterns or shared facial regions experience larger drops on unseen low-quality data.
- Multitask comparisons: The model outperforms other multitask detectors on samples containing both seen and unseen deepfake techniques, according to reported ACC results.Its auxiliary task recognizes forgery configurations rather than requiring forgery-region annotations.
- State-of-the-art comparisons: When trained on FF++ and tested on CelebDF, the method achieves the best performance among the compared state-of-the-art detectors, while remaining competitive on FF++.The comparison uses AUC and includes results from multiple existing detectors.
- Ablation studies: Adversarial augmentation alone improves the Xception baseline by around 4% AUC, while replacing it with random augmentation causes a significant drop.These ablations indicate that adversarial training is essential for the system’s augmentation strategy.
- Ablation studies: Each of the three auxiliary self-supervised tasks contributes to performance, and the complete combination performs best.Removing any task decreases overall performance, while using only the classification loss performs significantly worse.
5. Conclusions and Discussions
The proposed detector targets unseen scenarios by dynamically generating forgeries and learning robust representations. Its main limitation is the synthesizer’s restricted augmentation variety.
- Conclusions: The method uses a synthesizer and adversarial training framework to dynamically generate forgeries for unseen-scenario detection.Training on generated forgeries is intended to produce more robust feature representations and a more generalizable detector.
- Conclusions: Extensive experiments demonstrate the effectiveness of the proposed deepfake detection method.
- Discussions: The synthesizer’s augmentation types remain limited, motivating future use of generative models and self-supervised auxiliary tasks.The proposed direction includes controlling generated forgeries with latent variables and predicting those variables.