Source-linked AI summary
UCF: Uncovering Common Features for Generalizable Deepfake Detection
Zhiyuan Yan, Yong Zhang, Yanbo Fan, Baoyuan Wu
TL;DR
Deepfake detectors often overfit forgery-irrelevant content and method-specific patterns, making generalization to unknown forgeries difficult. The paper disentangles these factors, isolates common forgery features, and uses them for detection; experiments report superior generalization to current state-of-the-art methods on unseen datasets.
Problem
Deepfake detection has difficulty generalizing to new forgery types because existing methods overfit forgery-irrelevant features and method-specific patterns.
Method
A multi-task disentanglement framework separates forgery-irrelevant, method-specific, and common forgery features, aided by a conditional decoder and contrastive regularization.
Results
The framework achieves superior generalization compared with current state-of-the-art methods on unseen testing datasets.
Takeaways & Limitations
Using only common forgery features provides the paper’s approach to avoiding overfitting to content and method-specific forgery textures.
Takeaways & Limitations
Existing disentanglement methods may still fail to completely separate forgery features, leaving them overfit to method-specific textures and limiting generalization to unseen forgeries.
Abstract
from arXiv · showhide
Deepfake detection remains a challenging task due to the difficulty of generalizing to new types of forgeries. This problem primarily stems from the overfitting of existing detection methods to forgery-irrelevant features and method-specific patterns. The latter has been rarely studied and not well addressed by previous works. This paper presents a novel approach to address the two types of overfitting issues by uncovering common forgery features. Specifically, we first propose a disentanglement framework that decomposes image information into three distinct components: forgery-irrelevant, method-specific forgery, and common forgery features. To ensure the decoupling of method-specific and common forgery features, a multi-task learning strategy is employed, including a multi-class classification that predicts the category of the forgery method and a binary classification that distinguishes the real from the fake. Additionally, a conditional decoder is designed to utilize forgery features as a condition along with forgery-irrelevant features to generate reconstructed images. Furthermore, a contrastive regularization technique is proposed to encourage the disentanglement of the common and specific forgery features. Ultimately, we only utilize the common forgery features for the purpose of generalizable deepfake detection. Extensive evaluations demonstrate that our framework can perform superior generalization than current state-of-the-art methods.
1. Introduction
Deepfake detection struggles to generalize when test forgeries use unknown procedures, partly because detectors overfit content and method-specific patterns. The paper proposes disentangling these factors and detecting with common forgery features, reporting stronger performance on unseen datasets.
- Unknown forgery procedures create training–testing differences that cause poor detection performance and limit practical use.
- Existing generalization methods often rely on predefined artifacts or remove content while retaining method-specific patterns, limiting performance on unseen forgeries.
- The proposed framework separates content from forgery fingerprint, then divides fingerprint features into method-specific and common components.
- A multi-task disentanglement framework addresses overfitting to forgery-irrelevant features and method-specific textures by uncovering common features.
- A conditional decoder and contrastive regularization support disentanglement of irrelevant, specific, and common forgery features.
- Extensive experiments show the framework outperforms current state-of-the-art methods on unseen testing datasets, demonstrating effectiveness in generalization.
2. Related Work
Deepfake detection research has increasingly targeted generalization across changing forgery techniques, but existing methods remain constrained by predefined patterns and incomplete disentanglement. These limitations motivate separating forgery-related information into more generalizable components.
- Image and video forgery detection are the two broad deepfake detection task categories, while this paper focuses on image forgery.
- Detection Methods toward Generalization: Generalization fails when training and testing use different data distributions, because detectors trained on known forgery techniques perform poorly on unknown procedures.
- Detection Methods toward Generalization: Generalization-oriented detectors use blending, frequency, spatial-temporal, noise, similarity, and augmentation-based cues, but often rely on predefined forgery patterns or the full feature space.
- The t-SNE visualization shows baseline Xception features separating by forgery method, while the proposed common and content modules capture shared forgery and forgery-irrelevant features, respectively.
- Disentanglement Learning for Deepfake Detection: Disentanglement methods separate forgery-irrelevant and forgery-related features, yet many still overfit to method-specific patterns after removing content influence.
3. Methods
The method disentangles each image into content, method-specific fingerprint, and common fingerprint features, then trains separate objectives for reconstruction, classification, and contrastive separation. Detection uses the common forgery representation to target features shared across forgery methods.
- 3.1. Motivation: The framework addresses content and method-specific overfitting by disentangling inputs into content, specific forgery, and common forgery features.
- 3.3. Architecture: The encoder extracts content, specific fingerprint, and common fingerprint components, while the decoder recombines fingerprint and content features to reconstruct images.
- 3.3. Architecture: The decoder uses AdaIN to align content-code mean and variance with fingerprint statistics before convolutional processing and upsampling.AdaIN is applied to fuse the fingerprint as a condition with content during reconstruction.
- 3.4. Objective Function: The training objective combines common and specific classification losses, contrastive regularization, and pixel-level reconstruction losses in a weighted sum.Reconstruction includes self-reconstruction and cross-reconstruction using different fingerprint-content combinations.
- 3.4. Objective Function: Two classification heads learn method-specific textures and common forgery features by predicting forgery method identity and real-versus-fake labels, respectively.The heads share architecture but not parameters.
- 3.4. Objective Function: Contrastive regularization minimizes distances between similar embeddings and enlarges the gap between dissimilar embeddings using margin α.For common features, real-image attributes form similar pairs and manipulated-image attributes form dissimilar pairs; same-method images supervise specific features.
4. Experiments
Experiments evaluate UCF across multiple datasets and configurations, showing stronger generalization than competing detectors and disentanglement-based baselines. Ablations support the contributions of disentanglement, multi-task learning, and contrastive regularization.
- Experimental setup: Experiments use FF++, DFD, DFDC, and CelebDF, with FF++ providing four manipulation types and multiple compression settings.Cross-dataset evaluation trains on FF++ and tests on DFD, DFDC, and CelebDF.
- Experimental setup: AUC is the default comparison metric, supplemented by accuracy, average precision, and equal error rate.
- Comparison with prior methods: Frequency-based and blended-artifact detectors generalize less effectively when post-processing or forgery patterns differ across datasets.
- Generalization results: UCF outperforms Liang et al. [27] on all tested datasets, supporting the value of uncovering common forgery features beyond content removal.
- Generalization results: UCF achieves the best AUC on both CelebDF and DFDC among the compared state-of-the-art methods.The comparison covers 27 detectors, including 10 implemented in the study and 17 referenced.
- Generalization results: UCF reaches 82.4% AUC on CelebDF, outperforming Zhuang et al. [60] at 72.8%.
- Ablation study: The full model with disentanglement, multi-task learning, and contrastive regularization performs best across within-dataset and cross-dataset evaluations.Basic disentanglement improves cross-dataset results, while multi-task learning further improves them.
- Feature analysis: Common forgery features generalize better than specific features and outperform using whole forgery features for binary classification.
5. Conclusion
The paper concludes that UCF improves deepfake detection on unseen datasets by learning common forgery features while avoiding forgery-irrelevant and method-specific textures. Conditional decoding and contrastive regularization strengthen the disentanglement process.
- UCF disentangles content, specific forgery, and common forgery features, using only common features for detection.
- The framework combines multi-task disentanglement, a conditional decoder, and contrastive regularization to improve feature separation.
- Extensive benchmark experiments show that UCF generalizes well to unseen deepfake datasets compared with existing state-of-the-art methods.