Source-linked AI summary
Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection
Zhiyuan Yan, Yuhao Luo, Siwei Lyu, Qingshan Liu, Baoyuan Wu
TL;DR
Deepfake detectors often lose generalization when training and testing distributions differ, partly because they overfit to forgery-specific features. LSDA expands forgery representations through within- and cross-domain latent-space augmentation, then refines a binary classifier using enhanced forgery and real-face features. Experiments report improved generalization and robustness over baselines and state-of-the-art detectors across deepfake benchmarks.
Problem
Deepfake detectors often struggle with cross-dataset generalization when training and testing distributions differ, while overfitting to particular forgery features can impede transfer to other forgery types.
Method
LSDA enlarges the forgery space through within-domain interpolation of challenging examples, cross-domain latent Mixup, and real-face features from ArcFace for binary classification.
Results
LSDA shows substantial improvement over baseline methods and enhanced cross-dataset generalization and robustness, favorably outperforming recent state-of-the-art detectors.
Takeaways & Limitations
A wider range of latent forgery representations is reported to support a more adaptable decision boundary and reduce forgery-specific overfitting.
Abstract
from arXiv · showhide
Deepfake detection faces a critical generalization hurdle, with performance deteriorating when there is a mismatch between the distributions of training and testing data. A broadly received explanation is the tendency of these detectors to be overfitted to forgery-specific artifacts, rather than learning features that are widely applicable across various forgeries. To address this issue, we propose a simple yet effective detector called LSDA (\underline{L}atent \underline{S}pace \underline{D}ata \underline{A}ugmentation), which is based on a heuristic idea: representations with a wider variety of forgeries should be able to learn a more generalizable decision boundary, thereby mitigating the overfitting of method-specific features (see Fig.~\ref{fig:toy}). Following this idea, we propose to enlarge the forgery space by constructing and simulating variations within and across forgery features in the latent space. This approach encompasses the acquisition of enriched, domain-specific features and the facilitation of smoother transitions between different forgery types, effectively bridging domain gaps. Our approach culminates in refining a binary classifier that leverages the distilled knowledge from the enhanced features, striving for a generalizable deepfake detector. Comprehensive experiments show that our proposed method is surprisingly effective and transcends state-of-the-art detectors across several widely used benchmarks.
1. Introduction
Deepfake detectors often fail to generalize across datasets because they overfit to forgery-specific features. LSDA enlarges the forgery space through latent-space augmentation and combines enriched forgery and real-face representations to improve generalization.
- Motivation: Cross-dataset distribution shifts expose a generalization weakness in many deepfake detectors.These detectors can work effectively within a dataset but struggle when training and testing distributions differ.
- Motivation: Overfitting to one forgery type can impede generalization to other forgery types.
- Method: LSDA enlarges the forgery space by interpolating latent representations within and across forgery domains.Within-domain augmentation diversifies each forgery type with challenging examples, while cross-domain Mixup creates transitions between distinct forgery features.
- Method: The detector also uses ArcFace-derived real-face features to refine a binary classifier from comprehensive real and forgery representations.
- Advantages: Latent-space augmentation avoids directly relying on pixel-level artifacts that can be altered by compression or blurring and may not cover entire face synthesis.
- Results: Experiments report substantial improvement over baselines and favorable performance against recent state-of-the-art detectors in cross-dataset generalization.
2. Related Work
Related work addresses deepfake detection through multiple directions, including augmentation, frequency clues, identity information, and specialized learning strategies. Data-augmentation methods synthesize training examples using mechanisms such as self-blending, blending-boundary encouragement, and adversarially selected choices.
- Deepfake Detectors toward Generalization: Deepfake detection research explores data augmentation, frequency clues, identity information, disentanglement, network design, reconstruction, and 3D decomposition.
- Deepfake Detectors Based on Data Augmentation: FWA generates synthetic faces through self-blending and transformations designed to expose wrapping artifacts.
- Deepfake Detectors Based on Data Augmentation: Face X-ray encourages detectors to learn blending boundaries in fake images.
- Deepfake Detectors Based on Data Augmentation: I2G generates synthetic data for pair-wise self-consistency learning, while SLADD adversarially generates challenging blending choices.
3. Method
LSDA uses dedicated teacher encoders and latent-space augmentation to enrich forgery representations, then distills them into a student encoder for deepfake detection. Within-domain transformations diversify individual forgery domains, while cross-domain interpolation connects different forgery types; ArcFace supplies real-face features.
- Framework overview: The framework assigns dedicated teacher encoders to forgery types and real images, then distills their learned features into one student encoder with a binary classifier.Training uses domain-specific teachers followed by distillation; inference applies only the student encoder.
- Within-domain augmentation: Within-domain augmentation uses centrifugal, affine, and additive transformations to enlarge each forgery domain and improve representation robustness.Centrifugal transformation creates hard examples away from a domain centroid, while affine and additive transformations create neighboring or perturbed samples.
- Within-domain augmentation: Centrifugal augmentation moves latent samples away from their domain centroid directly or toward existing hard examples, using a randomly sampled scaling factor β.The hard example is the sample with the largest Euclidean distance from the centroid.
- Objective and real-feature learning: The real representation comes from a pre-trained, frozen ArcFace model, while domain, distillation, and binary-classification losses are combined into the final weighted objective.The student aligns its fake features with augmented forgery representations and its real feature with the ArcFace representation.
- Cross-domain augmentation: Cross-domain augmentation applies latent-space Mixup to linearly combine representations from distinct forgery domains, encouraging shared forgery features and smoother decision boundaries.The mixing weight α is randomly sampled between 0 and 1.
- Fusion and distillation: A learnable convolutional fusion layer concatenates original and augmented forgery representations to produce the final latent representation used for student distillation.The fusion layer aligns the augmented features with the student encoder’s output shape.
4. Experiments
Experiments evaluate LSDA under cross-dataset generalization, comparisons with state-of-the-art and augmentation-based detectors, robustness to perturbations, and ablations of its components. The method consistently improves generalization, robustness, and performance across the reported evaluations.
- 4.2. Generalization Evaluation: The evaluation also examines robustness to unseen perturbations and compares LSDA with three RGB-based augmentation methods using video-level AUC.Robustness is measured across five degradation levels for five perturbation types.
- 4.1. Settings: Experiments train models on FF++ c23 and evaluate them on previously unseen datasets, following a cross-dataset generalization protocol.The evaluation includes datasets such as CDF and DFDC, with experiments conducted using DeepfakeBench settings.
- 4.2. Generalization Evaluation: LSDA consistently outperforms competing detectors across tested scenarios, achieving an average 5% performance improvement on the unified benchmark.The comparison uses frame-level AUC in Table 1.
- 4.2. Generalization Evaluation: LSDA achieves the best video-level AUC performance on both CDF-v2 and DFDC against recent state-of-the-art methods.Table 2 compares methods trained on FF++ c23, with SBI additionally reproduced under consistent conditions.
- 4.3. Ablation Study: Ablations show incremental generalization gains from combining within-domain and cross-domain augmentation, while ArcFace pretraining improves the real encoder over alternative backbones.The WD and CD variants are evaluated across datasets, and ArcFace is compared with EFNB4 and iResNet101 without face-recognition pretraining.
5. Visualizations
The visualizations compare the baseline with LSDA in learned latent space and activated image regions. LSDA produces a more robust decision boundary and focuses detection on manipulated face areas, encouraging more general forgery features.
- The baseline responds to similar, limited regions across different forgeries, indicating reliance on forgery-specific artifacts.
- LSDA localizes forgery detection predominantly to manipulated face areas and captures more general forgery features.
- LSDA learns a more robust decision boundary than the un-augmented baseline in the FF++ c23 testing feature space.The comparison uses t-SNE visualizations of 5,000 randomly selected samples.
6. Conclusion
The paper concludes that LSDA enlarges the forgery space by simulating within- and across-forgery variations in latent space. Extensive experiments report superior generalization and robustness to state-of-the-art methods.
- LSDA is designed to generalize to unseen deepfake datasets by learning from a wider range of forgery representations.
- The method constructs and simulates variations within and across forgery features in latent space to mitigate forgery-specific overfitting.
- LSDA shows superior generalization and robustness to state-of-the-art methods in extensive experiments.