Source-linked AI summary
Learning Self-Consistency for Deepfake Detection
Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, Wei Xia
TL;DR
Deepfake detectors often target stitching artifacts that forgery methods can reduce. This paper detects source-feature inconsistency with PCL and I2G, reporting superior accuracy across seven datasets, including strong cross-dataset AUC scores.
Problem
Existing deepfake detectors target artifacts from stitching processes, while forgery techniques evolve to reduce those artifacts, motivating alternative detection cues.
Method
PCL extracts local source features and measures their self-consistency, while I2G dynamically generates forged images with manipulated-region annotations for training.
Results
PCL achieves AUC scores of 99.11%, 99.07%, 99.41%, 98.30%, and 90.03% on five cross-dataset evaluations, with superior detection accuracy across seven datasets.
Takeaways & Limitations
PCL detects stitching-based face forgeries and localizes manipulated regions, providing a competitive baseline against state-of-the-art methods.
Takeaways & Limitations
PCL may not handle entire-face synthesis that produces self-consistent images, and its performance can be further improved on low-quality data.
Abstract
from arXiv · showhide
We propose a new method to detect deepfake images using the cue of the source feature inconsistency within the forged images. It is based on the hypothesis that images' distinct source features can be preserved and extracted after going through state-of-the-art deepfake generation processes. We introduce a novel representation learning approach, called pair-wise self-consistency learning (PCL), for training ConvNets to extract these source features and detect deepfake images. It is accompanied by a new image synthesis approach, called inconsistency image generator (I2G), to provide richly annotated training data for PCL. Experimental results on seven popular datasets show that our models improve averaged AUC over the state of the art from 96.45% to 98.05% in the in-dataset evaluation and from 86.03% to 92.18% in the cross-dataset evaluation.
1. Introduction
The paper detects stitching-based deepfakes through spatial inconsistency in source features rather than artifacts, using PCL and I2G to learn and supervise this cue. Evaluations across seven datasets report strong in-dataset and cross-dataset detection results, while the approach remains bounded by future forgery techniques that remove source inconsistency.
- Deepfake detectors commonly target stitching artifacts, creating an arms race as forgery methods evolve to reduce those artifacts.Examples include eye blinking, face warping, blending boundaries, and fake prototypes.
- The method instead detects inconsistency among spatially local source features, hypothesized to remain identifiable after current deepfake generation processes.Pristine images are expected to have consistent source features across positions, unlike forged images assembled from different sources.
- PCL trains a ConvNet with pair-wise cosine-similarity supervision and an image-level binary classifier to detect whether locations originate from the same source.Pairs from the same source are encouraged to have high similarity, while pairs from different sources are encouraged to have low similarity.
- I2G generates stitched forged images online from pristine source and target images, supplying the pixel-level annotations required by PCL without expensive deep-network synthesis.The simplified generator supports training with annotations of manipulated regions.
- 99.79%, 99.98%, and 94.38% AUC are achieved on FF++, CD2, and DFDC-P in-dataset evaluations, while cross-dataset AUC reaches 99.11%, 99.07%, 99.41%, 98.30%, and 90.03%.The consistency maps also can localize modified regions.
- The method is intended for existing stitching-based deepfakes because entire-face synthesis or future stitching methods could remove or obscure source-feature inconsistency.The stated evaluation scope is the deepfake imagery represented in the datasets used.
2. Related Work
The paper situates its approach among deepfake detection and image-forensics methods, emphasizing source-feature inconsistency as a less-attended cue. It combines an end-to-end consistency architecture with I2G-generated, finely stitched training images.
- Deepfake generation includes entire-image synthesis, facial-attribute or expression modification, and face identity swapping, often blending generated segments into original images.
- Detection research targets whole-image synthesis cues as well as released datasets and countermeasures for other facial manipulations.
- The paper differs by using source-feature inconsistency and by designing a lightweight consistency predictor that can function as a plugin on common backbones.
- Earlier consistency-learning work compares image patches or uses specialized two-stream or Siamese approaches for forensic inconsistencies and metadata.
- The proposed architecture estimates image self-consistency in one forward pass while capturing internal relations among patches.
- PCL uses a consistency branch for patch-source relations and a classification branch after the source feature map for binary deepfake prediction.
3. Our Approach
The approach learns source-feature self-consistency with a pairwise consistency branch and uses a classification branch for real-versus-fake detection. I2G supplies dynamically generated, masked training examples and annotations for learning these representations.
- Pair-Wise Self-Consistency Learning: PCL compares local source-feature vectors pairwise to learn whether image locations originate from the same source.The consistency branch computes similarities across feature-map patches and supervises them using source-location agreement.
- Pair-Wise Self-Consistency Learning: The consistency branch produces a 4D volume of pairwise consistency scores and a 2D heatmap for visualizing modified regions.The volume compares every local patch with all others, while the heatmap is fused and up-sampled to input resolution.
- Pair-Wise Self-Consistency Learning: Ground-truth consistency is derived from a downsampled manipulation mask, with pristine images represented by an all-one consistency volume.Values near one indicate consistent patch pairs, while values near zero indicate inconsistency.
- Pair-Wise Self-Consistency Learning: A classification branch applies convolution, global average pooling, and a fully connected layer to predict real-versus-fake labels.The branch is supervised with a two-class cross-entropy loss.
- Inconsistency Image Generator (I2G): I2G generates self-inconsistent images and masks from pristine frames using aligned cross-identity faces, morphed masks, blending, and augmentation.Its mask pipeline uses facial landmarks, convex-hull extraction, elastic deformation, Gaussian blurring, and blending; augmentations include compression, noise or blur, brightness and contrast changes, erasing, and color jittering.
- Inconsistency Image Generator (I2G): I2G reduces synthesis cost by stitching real source and target images, enabling dynamic CPU generation during training and data augmentation.This design provides the patch-level manipulated-region annotations required by PCL.
4. Experiments
The experiments evaluate PCL with I2G across in-dataset and cross-dataset settings, using video-level detection metrics and multiple deepfake benchmarks. Results show strong average gains, while performance remains constrained by low-quality data and future forgery methods that remove source inconsistency.
- Evaluation settings: Training uses real and deepfake videos for in-dataset evaluation, but only real FF++ videos augmented by I2G for cross-dataset evaluation.The cross-dataset setting is intended to assess generalization without prior knowledge of attacks.
- In-dataset results: AUC improves from 96.45% to 98.05% over the state of the art across FF++, CD2, and DFDC-P in in-dataset evaluation.The method reaches 99.98% AUC on CD2 and improves the state-of-the-art DFDC-P result by 3.28%.
- Cross-dataset results: On DFD, the model achieves 99.07% AUC versus 93.47% for Face X-ray, while remaining on par with Face X-ray on FF++.These experiments train only on real FF++ videos and test on unseen datasets.
- Cross-dataset results: Cross-dataset average AUC improves from 86.03% to 92.18% over the state of the art across five evaluated datasets.The method also reports 99.51% AUC on DFR and 67.52% on DFDC as cross-dataset baselines.
- Ablation studies: PCL improves DFDC AUC by 15.8% in ablation experiments, and large λ values indicate that the consistency branch dominates performance.The reported ablation links the performance improvement to the consistency-learning component.
- Qualitative results: PCL produces consistency visualizations that are usually blank for real images and align with modified regions in deepfakes.Failure cases include mistaken inconsistencies from lighting or unusual texture and false negatives on highly compressed or poorly exposed samples.
- Limitations: The method is evaluated on existing deepfake generation methods because entire-face synthesis or future stitching could remove or obscure source-feature inconsistency.The authors also identify low-quality data as a remaining weakness.
5. Conclusion
PCL detects stitching-based face forgeries and localizes manipulated regions using inconsistency in source features, while I2G efficiently generates annotated training images.
- PCL detects stitching-based face forgeries and localizes manipulated regions using inconsistency in source features within modified images.It is designed as a plugin module with few parameters for common backbone networks.
- I2G dynamically generates forged images and annotations to support PCL training efficiently.The method is lightweight and synthesizes training data with manipulated-region annotations.
- PCL and I2G achieve competitive performance against state-of-the-art methods across seven popular datasets.The authors present the results as a strong baseline for future research.
6. Appendix
The appendix reports additional evaluation, visualization, computational, and backbone analyses, including frame-level cross-dataset results and PCL's modest computational overhead.
- Qualitative Results: Consistency visualizations are produced by upsampling a fused heatmap derived from the predicted consistency volume.Figure 5 shows predicted consistency maps intended to localize modified regions, with DSSIM-generated ground-truth regions.
- Frame-Level Results on Celeb-DF-v2: Our model outperforms another state-of-the-art method by over 8% in frame-level AUC on CD2 under cross-dataset evaluation.The model is trained on FF++ and evaluated on CD2; baseline results are cited from Masi et al.
- Computational Complexity: PCL adds 48.12M FLOPs, 65.5K parameters, and 0.0009 seconds to a single-image forward pass.With ResNet-34, the total is 9.62G FLOPs, 21.3M parameters, and 0.0234 seconds on one NVIDIA Tesla V100.
- Backbone Ablation: Performance saturates as backbone depth increases in the ablation study using λ = 10.The study compares different backbone depths.