Source-linked AI summary
Protecting Celebrities from DeepFake with Identity Consistency Transformer
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Ting Zhang, Weiming Zhang, Nenghai Yu, Dong Chen, Fang Wen, Baining Guo
TL;DR
Face forgery detection must remain effective when low-level artifacts are degraded or absent, especially for widely available celebrity imagery. The paper proposes Identity Consistency Transformer, which learns inner- and outer-face identities with a consistency loss and can incorporate reference identities. ICT and ICT-Ref show improved generalization across datasets and image degradations, while the method is specifically limited to identity-inconsistent face swaps.
Problem
Existing face forgery detectors rely heavily on low-level artifacts, while real-world degradations and identity-preserving reenactments limit the scope of identity-inconsistency detection.
Method
Identity Consistency Transformer uses a Transformer to learn inner and outer identity information jointly, adds a consistency loss, and can leverage celebrity reference identities.
Results
ICT and ICT-Ref improve generalization across datasets and image degradations, including real-world video applications, and ICT is effective where low-level methods fail.
Takeaways & Limitations
High-level identity consistency provides a basis for face forgery detection and can be augmented with available celebrity identity information.
Takeaways & Limitations
The method targets identity-inconsistent face swaps and may fail on face reenactment where the identity remains the same.
Abstract
from arXiv · showhide
In this work we propose Identity Consistency Transformer, a novel face forgery detection method that focuses on high-level semantics, specifically identity information, and detecting a suspect face by finding identity inconsistency in inner and outer face regions. The Identity Consistency Transformer incorporates a consistency loss for identity consistency determination. We show that Identity Consistency Transformer exhibits superior generalization ability not only across different datasets but also across various types of image degradation forms found in real-world applications including deepfake videos. The Identity Consistency Transformer can be easily enhanced with additional identity information when such information is available, and for this reason it is especially well-suited for detecting face forgeries involving celebrities. Code will be released at \url{https://github.com/LightDXY/ICT_DeepFake}
1. Introduction
The paper proposes detecting face forgeries through identity inconsistency between inner and outer face regions, addressing limitations of artifact-based detection and leveraging reference identities for celebrity protection.
- Deepfake realism and widespread manipulation of public figures create an urgent need for face forgery detection.
- Existing detectors often rely on low-level artifacts, which can be weakened by image degradation and convincing deepfakes.
- Identity Consistency Transformer detects whether the inner and outer face regions belong to the same person using high-level semantic identity information.
- ICT learns separate inner- and outer-face identity vectors because conventional face verification tends to emphasize the inner face and misses outer-face identity information.
- ICT can train on swapped faces without generated manipulations, and its identity-consistency scores distinguish examples missed by state-of-the-art detectors.
- Reference-assisted ICT uses celebrity identity information and improves generalization across datasets and real-world image degradations, including video applications.
2. Related Work
Prior work includes manipulation-generation methods, artifact-based detectors, and semantic cues such as physical inconsistencies; ICT instead focuses on identity consistency with Transformer-based global attention.
- Face manipulation methods evolved from landmark-based swapping toward higher-quality methods, while current manipulations still focus primarily on the inner face.
- Existing detection methods commonly search for low-level generation artifacts, while some video methods exploit head-pose or eye-blink abnormalities.
- ICT investigates identity information as a high-level semantic signal for robust face forgery detection.
- Compared with related identity-consistency methods, ICT can predict from a suspect image and automatically retrieve references in its reference-assisted variant.
- Transformer: Transformer-based global attention can learn semantically meaningful features for fine-grained face classification.
- Transformer: ICT introduces a consistency loss that adapts Transformer features to identity-consistency detection.
3. Identity Consistency Transformer
Identity Consistency Transformer extracts inner- and outer-face identity information with a Transformer and detects forgeries through identity consistency. It uses swapped-face training, consistency-aware distances, and optional reference identities, while offering broad manipulation and degradation generalization but remaining targeted to identity-inconsistent face swaps.
- 3.1. Identity Extraction Model: The model applies a vision Transformer to learn inner and outer identity information simultaneously from face images.Two learnable tokens represent the inner and outer identities, whose final outputs feed a shared classification head.
- 3.1. Identity Extraction Model: Training uses inner-face swaps between different identities, allowing the identity extraction model to learn without generated manipulation-specific fake images.The training dataset pairs an inner identity from one real face with an outer identity from another, using cosine-based softmax classification.
- 3.1. Identity Extraction Model: A shared classification head makes same-person inner and outer identity vectors similar, while a consistency loss further encourages this relationship when predicted labels are correct.The consistency loss is applied only to correctly predicted identities so it operates on meaningful identity vectors.
- 3.2. Identity Consistency Detection: Detection compares extracted inner and outer identity vectors, with smaller distance indicating greater consistency and a higher likelihood that the image is real.The method uses Euclidean distance by default and can modulate distances with identity-similarity weights.
- 3.2. Identity Consistency Detection: Reference-assisted detection retrieves nearest neighbors from real-image identity pairs and combines corresponding distances into an enhanced consistency measure.The reference set contains identity-vector pairs from available real images, supporting additional identity information when references exist.
- 3.3. Benefits and Limitations and Impacts: The approach is designed to generalize across manipulation methods and image degradations, but specifically targets identity-inconsistent face swaps and may fail on identity-preserving reenactment.Its high-level identity evidence is described as more resistant to degradation, while the stated limitation restricts scope to face swaps.
4. Experiment
Experiments evaluate ICT against state-of-the-art detectors for unseen datasets, image degradations, celebrity deepfake videos, and component choices. ICT generalizes strongly, while ICT-Ref further improves benchmark and celebrity-video performance; ablations support the roles of consistency loss and training-data processing.
- Comparison with State-of-the-art Methods: Experiments compare ICT with manipulation-specific and general deepfake detectors using frame-level AUC on unseen datasets and degradations.The evaluation includes conventional benchmarks, carefully crafted Internet videos, and five severity levels for each degradation form.
- Comparison with State-of-the-art Methods: ICT improves detection on newer low-artifact datasets, while ICT-Ref achieves state-of-the-art performance across all benchmark datasets.The paper attributes this improvement to identity information being more reliable than low-level textures in these cases.
- Comparison with State-of-the-art Methods: Nearly 95% average accuracy on carefully crafted celebrity deepfake videos rises to 100% with the reference-assisted ICT-Ref.Most existing methods achieve relatively low accuracy on this real-world scenario.
- Comparison with State-of-the-art Methods: ICT maintains high detection performance as image degradation severity increases, except that Gaussian noise causes degradation at extreme severity.All compared methods drop drastically as degradation becomes more severe.
- Analysis of ICT: Removing consistency loss reduces performance by about 23%–40%, indicating that the loss is critical to the identity consistency Transformer.The analysis also reports AUCs of 85.71% for ICT, 92.52% using only inner identity, 87.45% using only outer identity, and 94.43% for ICT-Ref on Celeb-DeepFake v2.
- Analysis of ICT: A shared Res50 backbone fails to converge, whereas separate inner- and outer-identity models perform slightly worse than ICT with four times as many parameters.The analysis links ICT’s advantage to global attention that can separate inner and outer identities.
- Analysis of ICT: Saliency maps visualize the inner- and outer-identity attention regions across different poses.The figure presents the inner identity in the second column and the outer identity in the fourth column.
- Analysis of ICT: Mask deformation and color correction both help the final model, while small or incomplete reference sets can make ICT-Ref underperform ICT.A larger identity drop rate has a stronger negative influence on reference-assisted performance.
5. Conclusion
The paper proposes Identity Consistency Transformer for detecting forged faces through identity consistency, using high-level semantics rather than only low-level artifacts. It also extends the approach with additional celebrity identity information and evaluates its effectiveness extensively.
- Identity Consistency Transformer detects forged faces by learning inner and outer identity information with a Transformer and consistency loss.
- The high-level semantic approach is especially effective when low-level-based detection methods fail.
- Additional identity information from celebrities further enhances the approach for celebrity-related face forgery detection.