Source-linked AI summary
Exploring Disentangled Content Information for Face Forgery Detection
Jiahao Liang, Huafeng Shi, Weihong Deng
TL;DR
Face forgery detectors often perform well in-domain but overfit content information instead of artifact traces, limiting generalization. The paper introduces an embeddable disentanglement framework with C2C and GRCC, plus unbalanced datasets for studying content bias. Experiments show competitive detection while reducing content interference and emphasizing suspicious artifacts.
Problem
Detectors can overfit recognizable content information rather than artifact traces, causing poor cross-domain generalization.
Method
The paper disentangles content and artifact features, uses only artifact features for detection, and adds C2C and GRCC to improve feature independence.
Results
Extensive visualizations and experiments show competitive detection while ignoring content interference and guiding the detector toward suspicious artifact traces.
Takeaways & Limitations
Content-bias analysis and artifact-focused disentanglement provide a framework for improving the generalization of face forgery detection.
Takeaways & Limitations
The framework assumes that an image’s latent representation can be decomposed into content and artifact features.
Abstract
from arXiv · showhide
Convolutional neural network based face forgery detection methods have achieved remarkable results during training, but struggled to maintain comparable performance during testing. We observe that the detector is prone to focus more on content information than artifact traces, suggesting that the detector is sensitive to the intrinsic bias of the dataset, which leads to severe overfitting. Motivated by this key observation, we design an easily embeddable disentanglement framework for content information removal, and further propose a Content Consistency Constraint (C2C) and a Global Representation Contrastive Constraint (GRCC) to enhance the independence of disentangled features. Furthermore, we cleverly construct two unbalanced datasets to investigate the impact of the content bias. Extensive visualizations and experiments demonstrate that our framework can not only ignore the interference of content information, but also guide the detector to mine suspicious artifact traces and achieve competitive performance.
1 Introduction
The paper identifies content bias as a source of poor cross-domain face forgery detection and proposes disentangling content from artifact features. Its framework, constraints, and unbalanced datasets aim to improve artifact-focused detection and generalization.
- Motivation: Cross-domain evaluations expose detectors’ reliance on recognizable content information, local regions, or background cues rather than forgery artifacts.This content overfitting is proposed as a cause of generalization failure.
- Method: The proposed framework disentangles content and artifact features, then uses only artifact features for face forgery detection.The design is intended to ignore interference from content information.
- Method: Content Consistency Constraint (C2C) preserves corresponding information, while Global Representation Contrastive Constraint (GRCC) strengthens independence and feature purity.The constraints address the limited independence explored by prior disentanglement methods.
- Experiments: Two unbalanced FaceForensics++ datasets investigate how intrinsic content bias affects detection performance.The paper describes identity-unbalanced and background-unbalanced datasets as a novel investigation of content bias.
- Results: Visualizations and experiments show that the framework ignores content interference, mines suspicious artifact traces, and achieves competitive face forgery detection performance.The framework is embedded with multiple backbones and evaluated through experiments and ablations.
2 Related Works
Related work spans handcrafted artifact and physiological-signal methods, learning-based detectors, and disentangled representation learning. These approaches improve detection but remain sensitive to compression, noise, and dataset quality or distribution.
- Face Forgery Detection: Early detectors used handcrafted facial cues such as blinking, head pose, and lip movement to capture forgery artifacts.Other early approaches explored PPG signals representing heart-rate information.
- Face Forgery Detection: Learning-based methods achieved significant progress, while shallow local textures and correlations between facial regions were proposed as useful forgery indicators.These methods primarily exploit spatial-domain information.
- Face Forgery Detection: Existing detection performance remains sensitive to image compression, noise, and dataset quality or distribution.This sensitivity motivates methods that address dataset-related robustness issues.
- Disentangled Representation Learning: Disentangled representation learning decomposes coupled information into distinguishable features, including identity–pose or pose–appearance representations for face-related tasks.Prior work applies disentanglement to synthesis, recognition, and physiological measurements.
3 Methods
The method disentangles content and artifact features, detects forgery using artifact features, and adds reconstruction, content-consistency, and contrastive constraints to improve feature completeness and independence.
- Basic Disentanglement Framework: Artifact and content encoders separate the two feature types, while the classifier uses only artifact features for forgery detection.The framework uses independent encoders, a decoder for reconstruction, and a classifier connected to the artifact encoder.
- Basic Disentanglement Framework: Pairwise inputs combine content and artifact features from the same image for self-reconstruction and from different images for cross-reconstruction.The decoder reconstructs original images from same-image combinations and produces cross-reconstructed images from swapped content or artifact features.
- Basic Disentanglement Framework: Reconstruction losses preserve pixel-level image consistency and feature consistency, encouraging completeness of the disentangled representation.The total reconstruction objective includes image and feature reconstruction terms.
- Enhanced Independence of Disentangled Features: GRCC uses InfoNCE over Gram-matrix representations and cosine similarity to encourage artifact and content features to remain independent.The feature Gram matrices are flattened into vectors, and contrastive comparisons are made between artifact and content representations.
- Enhanced Independence of Disentangled Features: C2C preserves identity and background information in cross-reconstructed images through identity and content-perception losses.Identity preservation uses ArcFace features, while background consistency uses VGG features.
- Overall Loss: The final training objective is a weighted sum of classification, reconstruction, identity, background, and contrastive losses.The weights λ1 through λ5 balance the auxiliary loss terms against cross-entropy classification.
4 Experiments
Experiments evaluate the framework across in-dataset, cross-method, cross-dataset, ablation, augmentation, and content-bias settings. Results show competitive performance, with C2C and GRCC central to gains and artifact-focused representations reducing content-bias interference.
- In-Dataset Evaluation: The framework improves detectors across FF++ in-dataset evaluation and outperforms competing methods in the reported average comparisons.Table 1 evaluates ACC (%) on four FF++ LQ real–fake sub-datasets and reports gains after embedding detectors into the framework.
- Cross-Dataset and Cross-Method Evaluation: On Celeb-DF cross-dataset evaluation, the method outperforms same-backbone Xception methods and achieves the best result with ResNest-50.Cross-method performance still drops substantially, and the framework mitigates rather than eliminates overfitting in that setting.
- Ablation Study: Combining C2C and GRCC raises performance by 11.80%, whereas adding them separately improves AUC by 4.97% and 7.46%.The basic disentanglement framework alone decreases performance by 0.81%, identifying the constraints as the dominant contributors.
- Augmentation Study: Feature-level augmentation improves cross-dataset performance more than in-dataset performance and is significantly better than image-level augmentation.The study evaluates Random Erasing, Horizontal Flip, and Mixup at image and artifact-feature levels.
- Content-Bias Analysis: Performance suffers sharply on identity- and background-unbalanced datasets, while the framework maintains high performance by stripping content features.Background imbalance causes more severe degradation than identity imbalance, and t-SNE shows content-similar samples clustering more closely than forgery-method-similar samples.
- Visualization: Grad-CAM shows the baseline attending to small local regions or content noise, whereas the proposed method activates comprehensively and nearly matches forgery masks.The visualization supports the framework’s goal of guiding detection toward suspicious artifact regions rather than misleading content information.
5 Conclusion
The paper concludes that face forgery detectors can overfit content information instead of mining artifact traces, harming generalization. Its disentanglement framework, C2C, and GRCC are presented as an embeddable approach that reduces content-bias interference and supports competitive detection.
- Conclusion: The framework disentangles content and artifact information, then uses artifact features to reduce content-bias interference in forgery detection.The paper also constructs unbalanced datasets to investigate content bias and evaluates the framework through visualizations and experiments.
- Conclusion: C2C and GRCC enhance the independence of disentangled features, helping the detector mine suspicious artifact traces and achieve competitive performance.The framework is described as easily embeddable across detector backbones.