Source-linked AI summary
Dual Contrastive Learning for General Face Forgery Detection
Ke Sun, Taiping Yao, Shen Chen, Shouhong Ding, Jilin L, Rongrong Ji
TL;DR
Face forgery detectors often generalize poorly because category-level classification does not capture essential real-fake discrepancies, especially across unseen domains. DCL constructs positive and negative pairs and applies Inter-ICL and Intra-ICL with hard sample selection; experiments report improved generalization and state-of-the-art performance across several settings.
Problem
Existing classification-based face forgery detectors emphasize category-level differences, while unseen domain gaps produce severe performance drops.
Method
DCL constructs transformed positive pairs and informative negative pairs, then applies Inter-ICL across instances and Intra-ICL between local regions within instances.
Results
DCL achieves state-of-the-art performance across multiple evaluation settings, including unseen domains, low-quality images, cross-manipulation, and multisource cross-manipulation scenarios.
Takeaways & Limitations
The results support dual-granularity contrastive learning as an effective approach for learning generalized face-forgery representations.
Abstract
from arXiv · showhide
With various facial manipulation techniques arising, face forgery detection has drawn growing attention due to security concerns. Previous works always formulate face forgery detection as a classification problem based on cross-entropy loss, which emphasizes category-level differences rather than the essential discrepancies between real and fake faces, limiting model generalization in unseen domains. To address this issue, we propose a novel face forgery detection framework, named Dual Contrastive Learning (DCL), which specially constructs positive and negative paired data and performs designed contrastive learning at different granularities to learn generalized feature representation. Concretely, combined with the hard sample selection strategy, Inter-Instance Contrastive Learning (Inter-ICL) is first proposed to promote task-related discriminative features learning by especially constructing instance pairs. Moreover, to further explore the essential discrepancies, Intra-Instance Contrastive Learning (Intra-ICL) is introduced to focus on the local content inconsistencies prevalent in the forged faces by constructing local-region pairs inside instances. Extensive experiments and visualizations on several datasets demonstrate the generalization of our method against the state-of-the-art competitors.
1 Introduction
Face forgery detection is necessary because high-quality manipulated videos can create serious societal and political threats, yet methods trained for similar data distributions lose performance across unseen domains. DCL addresses this generalization problem with contrastive learning at instance and local-region granularities.
- High-quality forged videos can be difficult for humans to distinguish and may be abused to cause severe societal or political problems.
- CNN-based binary classifiers perform well when training and test distributions are similar, but unseen domain gaps cause severe performance drops.
- Existing general forgery detectors still emphasize category-level differences rather than essential discrepancies between real and fake faces.
- DCL constructs transformed views as positive pairs to remove task-independent information and preserve instance variation through alignment and uniformity.
- Inter-ICL uses hard negative samples to learn task-related discriminative features, while Intra-ICL contrasts fake and real local regions to mine essential forgery clues.
- Experiments and visualizations show that DCL improves generalization and performs effectively against state-of-the-art competitors.
2 Related Work
Face forgery detection has progressed from handcrafted cues to CNN- and frequency-based methods, but strong in-domain performance does not reliably transfer to unseen domains. The paper motivates contrastive learning as a way to address category-level bias and feature-distribution limitations.
- Earlier detectors used handcrafted cues such as eye blinking, inconsistent head poses, and visual artifacts before CNNs became widely used.
- CNN-based methods highlight manipulated regions with attention mechanisms, while other approaches supplement RGB information with frequency-domain clues.
- Methods that perform promisingly in matched training and test domains experience significant performance drops in unseen-domain scenarios.
- General forgery detectors such as Face X-ray, LTW, and SRM-based methods address specific biases but retain a focus on category-level differences rather than essential real-fake discrepancies.
- Supervised contrastive learning can change feature distributions and reduce intra-class variance, providing motivation for contrastive approaches to representation learning.
3 Proposed Method
DCL combines supervised classification with contrastive learning across views, instances, and local regions to learn generalized face-forgery features. Its modules generate transformed views, select hard negatives, and contrast real and forged regions within images.
- Architecture of Contrastive Learning: DCL transforms each training image into two views, extracts query and key feature maps, and trains a supervised contrastive architecture.The query encoder and classifier are used for final predictions, while the key encoder is updated from the query encoder by exponential moving average.
- Data Views Generation: Data Views Generation uses transformations including high-frequency enhancement, frame shifts, and corresponding mixup to reduce task-irrelevant or obvious forgery information.The operations are randomly combined to form the two views; corresponding mixup is applied only when the input is fake.
- Inter-Instance Contrastive Learning: Inter-Instance Contrastive Learning pulls together different views of the same image and pushes away true negatives stored in separate real and fake feature queues.It uses normalized cosine similarity and an InfoNCE-based loss, while avoiding aggregation of all same-category instances to preserve intra-class variance.
- Inter-Instance Contrastive Learning: Hard sample selection compares fake and real features with opposite-class prototypes, enqueuing samples that exceed a threshold as difficult negatives.The selected samples retain their queue labels while being difficult for the model to distinguish, making them more suitable for negative pair construction.
- Intra-Instance Contrastive Learning: Intra-Instance Contrastive Learning uses masks to divide forged features into real and forgery parts, then contrasts their local similarities within each image.The loss pulls real pairs closer, pulls real–fake pairs apart, and does not aggregate fake parts so counterfeit-trace diversity is preserved.
- Overall Loss Function: The overall objective combines binary cross-entropy with two InfoNCE-based contrastive losses, balanced by a hyper-parameter φ.The classification branch uses a fully connected classifier after query feature extraction.
4 Experiments
Experiments evaluate DCL across cross-dataset, cross-manipulation, multi-source, visualization, and ablation settings, showing stronger generalization and the contribution of its components.
- Experimental Setting: DCL is evaluated using five challenging datasets, including FaceForensics++ with four manipulation approaches and separate video-quality settings.The experiments use FaceForensics++ and evaluate generalization across manipulated data distributions.
- Cross-database Evaluation: Cross-database evaluation trains on FF++(HQ) and tests on DFD, DFDC, Wild Deepfake, and Celeb-DF using AUC and EER.FF++ is treated as intra-domain, while the other datasets represent unseen domains.
- Cross-manipulation Evaluation: DCL outperforms competitors in most intra-manipulation and cross-manipulation cases, gaining over 15% average AUC when trained on Deepfakes and tested on Faceswap.Diagonal results represent intra-domain performance in the cross-manipulation table.
- Multi-source Manipulation Evaluation: In multi-source manipulation evaluation, DCL achieves state-of-the-art AUC and ACC across all protocols and outperforms LTW by around 5% on low-quality FF++.The evaluation changes the backbone to EfficientNet-b0 for fair comparison.
- Ablation Study: Ablations show approximately 5% improvement from Inter-ICL on Celeb-DF, around 1% from hard-sample selection on both datasets, and further gains from Intra-ICL.Combining all proposed components produces the best performance on Celeb-DF and DFD.
- Visualization: Self-similarity distributions are significantly separated by DCL in both intra-domain and unseen-domain settings, unlike the baseline model.The visualization attributes this separation to the Intra-ICL module.
- Hard Sample Strategy: The hard-sample strategy selects higher-quality, difficult examples as negative pairs to promote mining essential forgery clues.The strategy compares samples with negative prototypes and uses threshold-based selection.
5 Conclusion
DCL is a holistic framework for general face forgery detection that combines paired data views, dual-granularity contrastive learning, and hard negative selection. Experiments across three settings report significant superiority over state-of-the-art methods.
- DCL generates data views as positive pairs and uses Inter-ICL and Intra-ICL to learn associations among instances and inconsistencies within samples.Inter-ICL operates across instances, while Intra-ICL focuses within each sample.
- The framework adds hard sample selection to identify informative hard negative samples for contrastive learning.
- Experiments on three settings demonstrate significant superiority over state-of-the-art methods.