Source-linked AI summary
CORE: Consistent Representation Learning for Face Forgery Detection
Yunsheng Ni, Depu Meng, Changqian Yu, Chengbin Quan, Dongchun Ren, Youjian Zhao
TL;DR
Face forgery detectors based on vanilla CNNs can overfit, and erasing-based augmentation does not explicitly ensure consistent representations across views. CORE addresses this by aligning representations from paired augmentations with a consistency loss, and it performs favorably across in-dataset and cross-dataset evaluations, including state-of-the-art benchmark performance.
Problem
Vanilla CNNs suffer from overfitting, while erasing-based methods lack explicit regularization of representations across augmented views.
Method
CORE uses different augmentations, a shared encoder, and a consistency loss to explicitly align representations before supervised classification.
Results
CORE performs favorably against recent state-of-the-art face forgery detection methods and enables a vanilla CNN to achieve state-of-the-art performance across several benchmarks.
Takeaways & Limitations
Explicitly regularizing augmented-view representations provides a flexible framework that can integrate with almost any other method.
Takeaways & Limitations
Cosine loss is used because erased views may not contain identical information, making exact representation matching potentially harmful.
Abstract
from arXiv · showhide
Face manipulation techniques develop rapidly and arouse widespread public concerns. Despite that vanilla convolutional neural networks achieve acceptable performance, they suffer from the overfitting issue. To relieve this issue, there is a trend to introduce some erasing-based augmentations. We find that these methods indeed attempt to implicitly induce more consistent representations for different augmentations via assigning the same label for different augmented images. However, due to the lack of explicit regularization, the consistency between different representations is less satisfactory. Therefore, we constrain the consistency of different representations explicitly and propose a simple yet effective framework, COnsistent REpresentation Learning (CORE). Specifically, we first capture the different representations with different augmentations, then regularize the cosine distance of the representations to enhance the consistency. Extensive experiments (in-dataset and cross-dataset) demonstrate that CORE performs favorably against state-of-the-art face forgery detection methods.
1 Tsinghua University 2 University of Science and Technology of China 3 Meituan
The passage lists contact email addresses for the paper’s authors.
- Contact email addresses are provided for the listed authors.
1. Introduction
Face forgery detection is motivated by increasingly realistic manipulations and vanilla CNN overfitting, while CORE explicitly aligns representations from augmented views. The framework preserves model structure and achieves state-of-the-art performance across several evaluation settings.
- Face manipulations are increasingly difficult for humans to distinguish, creating risks for societal trust and security.
- Vanilla CNNs face overfitting issues, motivating erasing-based augmentations for more general forgery representations.
- CORE explicitly regularizes representations to attend to more intrinsic forgery evidence without modifying model structure.
- CORE captures representations from differently augmented views and explicitly constrains their consistency with a consistency loss.
- CORE enables a vanilla CNN to obtain state-of-the-art performance on FF++, Celeb-DF, DFFD, and DFD benchmarks.
2. Related Work
Prior face forgery detection work includes binary classification, localization, specialized architectures, domain-combined representations, and consistency-based learning. The paper positions consistent representation learning within fully supervised face forgery detection.
- Face forgery detection is commonly formulated as a binary classification problem with multiple benchmarked architectures and representation strategies.
- Prior methods include patch-based, blending-boundary localization, multi-attention, color-frequency, and other specialized approaches.
- The paper relates consistent representation learning to semi-supervised and contrastive learning while studying it in fully supervised face forgery detection.
3. Proposed Method
CORE generates two augmented views of each image, encodes them with shared parameters, classifies both representations, and explicitly regularizes their consistency. Its cosine-based consistency loss encourages augmentation-invariant features without forcing representations to be identical.
- Data augmentation and representation learning: Two random augmentations produce paired views whose representations are compared for augmentation invariance while both remain available for classification.The framework applies augmentations twice, encodes the resulting 2N images, and applies consistency loss to each pair.
- Encoder and classifier: A shared Xception encoder maps the two views into representations f1 and f2, which a classifier converts into fake-face probabilities.The encoder and classifier are separate components, with the classifier using a linear layer and softmax normalization.
- Consistency loss: Cosine similarity loss penalizes distance between paired representations after L2 normalization, aligning their directions rather than their magnitudes.This choice avoids forcing augmented views to have exactly identical representations when an augmentation removes facial information.
- Classification loss: The classification objective uses standard cross-entropy with the ground-truth label for the predictions from the paired views.The two representations are fed into the classifier, and their classification losses are combined over N image pairs.
- Overall objective: CORE combines cross-entropy and consistency losses using balance weight α to jointly optimize prediction and representation consistency.The overall objective is the sum of the classification loss and α-weighted consistency loss.
4. Experiments
Experiments show that CORE improves face forgery detection through explicit representation consistency, with gains across in-dataset and cross-dataset evaluations. Cosine consistency and suitable augmentation choices are especially effective, while performance varies by dataset quality and domain.
- In-dataset Ablation Study: 0.013 AUC, 2.336 TDR0.1%, and 2.070 TDR0.01% gains over Xception+ show the benefit of consistent representation learning in-dataset.Xception+ already improves over Xception, but CORE performs best in the in-dataset comparison.
- In-dataset Ablation Study: Cosine consistency performs best across AUC, TDR0.1%, and TDR0.01%, whereas L1 and L2 penalties can underperform on TDR0.01%.The cross-dataset ablation likewise reports cosine penalty as best under average AUC.
- Data Augmentation: RaAug is the strongest in-dataset augmentation, reaching 99.971 AUC and 93.980 TDR0.01%, while DFDC selim produces the largest cross-domain gains.Across in-dataset augmentations, CORE improves AUC by +0.02 to +0.07 and TDR0.1% by +2.4 to +5.0 over baselines.
- Visualization Analysis: CORE’s CAM focuses inside forgery regions, and its t-SNE features distinguish forgery less by manipulation method than the baseline.This indicates that CORE extracts features less tied to method-specific artifacts, according to the reported visualization analysis.
- Cross-dataset Ablation Study: 2.118% average AUC gain over Xception+ demonstrates that CORE improves cross-dataset performance under the same augmentation setting.On Celeb-DF, the reported AUC gain reaches 5.708.
- Comparison with Other Methods: CORE outperforms prior methods on FF++ RAW and HQ, Celeb-DF, and DFFD, while achieving similar or second-best performance on some cross-dataset benchmarks.The method is reported as best on FF++ RAW/HQ, Celeb-DF, and DFFD, and as second or similar to state of the art on DFDC-P and Celeb-DF cross-dataset evaluation.
5. Conclusion
CORE explicitly constrains consistency across augmented representations and performs favorably against recent state-of-the-art face forgery detection methods.
- CORE explicitly constrains the consistency of representations from different augmentations and performs favorably against recent state-of-the-art face forgery detection approaches.