Source-linked AI summary
Detecting Deepfakes with Self-Blended Images
Kaede Shiohara, Toshihiko Yamasaki
TL;DR
Deepfake detectors can lose performance on unknown manipulations, so this paper introduces self-blended images (SBIs) as synthetic training data. SBIs blend transformed pseudo source and target images from one pristine image, and the method improves generalization across unseen manipulations and scenes, including reported gains on DFDC and DFDCP.
Problem
Deepfake detectors often perform well on known manipulations but significantly lose performance on fakes created by unknown manipulations.
Method
SBIs are generated by blending pseudo source and target images made from a single pristine image through transformations that reproduce common forgery artifacts.
Results
4.90% and 11.78% points are the reported improvements over the unsupervised baseline on DFDC and DFDCP in cross-dataset evaluation, respectively.
Takeaways & Limitations
Experiments report improved generalization to unseen manipulations and scenes, with good results across datasets and network architectures.
Takeaways & Limitations
The frame-level model cannot capture temporal inconsistencies, and whole-image synthesis detection achieves only 69.11% AUC on the evaluated set.
Abstract
from arXiv · showhide
In this paper, we present novel synthetic training data called self-blended images (SBIs) to detect deepfakes. SBIs are generated by blending pseudo source and target images from single pristine images, reproducing common forgery artifacts (e.g., blending boundaries and statistical inconsistencies between source and target images). The key idea behind SBIs is that more general and hardly recognizable fake samples encourage classifiers to learn generic and robust representations without overfitting to manipulation-specific artifacts. We compare our approach with state-of-the-art methods on FF++, CDF, DFD, DFDC, DFDCP, and FFIW datasets by following the standard cross-dataset and cross-manipulation protocols. Extensive experiments show that our method improves the model generalization to unknown manipulations and scenes. In particular, on DFDC and DFDCP where existing methods suffer from the domain gap between the training and test sets, our approach outperforms the baseline by 4.90% and 11.78% points in the cross-dataset evaluation, respectively.
1. Introduction
Deepfake detectors often generalize poorly to unknown manipulations, motivating synthetic data that captures common forgery artifacts without relying on specific source-target pairs. The paper proposes SBIs and reports stronger cross-dataset and cross-manipulation performance.
- Deepfake detection performance significantly drops when test fakes use unknown manipulations.
- Synthetic training data is used to encourage models to learn generic representations for deepfake detection.Prior approaches reproduce quality degradation or blending artifacts, but some generated samples can be difficult to recognize under compression or exposure conditions.
- Self-blended images blend transformed pseudo source and target images derived from one pristine image to reproduce common forgery artifacts.The method actively generates artifacts rather than relying only on gaps between distinct selected images.
- 4.90% and 11.78% points are the reported cross-dataset improvements over the unsupervised baseline on DFDC and DFDCP, respectively.The evaluation trains on FF++ and tests on CDF, DFD, DFDC, DFDCP, and FFIW.
- 99.64% versus 99.11% is the reported whole-FF++ AUC comparison, while cross-manipulation AUC reaches 99.99%, 99.88%, 99.91%, and 98.79% for DF, F2F, FS, and NT.
2. Related Work
Prior work develops specialized representations and synthetic samples for deepfake detection, but synthetic blended images can contain irregular or insufficient artifacts. SBIs instead generate diverse artifacts from a single base image.
- Deepfake detectors use neural architectures, frequency-domain traces, specialized facial representations, and patch-wise source-target consistency.
- Training-data synthesis targets generic forgery features because detectors often overfit to artifacts from known manipulations.FWA reproduces a quality gap, while other methods generate blended images from pristine image pairs.
- Existing blended-image samples may contain irregular swaps and may not provide enough artifact types for robust detection.The concern follows from their dependence on landmark-matched source-target pairs and oriented representations.
- SBIs generate pseudo source and target images from one base image and combine them with a diverse, deformed blending mask.
3. Self-Blended Images (SBIs)
SBIs are generated by transforming one pristine image into pseudo source and target images, creating a varied mask, and blending the results. The resulting samples train binary classifiers to detect forged images from actively generated artifacts.
- Source-Target Generator (STG): The source-target generator creates pseudo source and target images from a single input by applying transformations that introduce statistical inconsistencies.Transformations include color changes, sharpening or downsampling, and other image processing operations.
- Source-Target Generator (STG): Resizing and translating the source image reproduce blending boundaries and landmark mismatches.The resized dimensions and translation vector are sampled from continuous uniform distributions.
- Mask Generator (MG): The mask generator derives a facial-region mask from predicted landmarks, deforms it, and varies its shape and blending ratio.
- SBI synthesis: I_SB = I_s ⊙ M + I_t ⊙ (1-M) blends the pseudo source and target images using the generated mask.
- SBI synthesis: SBIs contain artifacts seen in forged faces even though they are not intended for counterfeiting.
- Training with SBIs: A binary classifier is optimized using binary cross-entropy on input images and binary labels.The classifier output F(x) is the probability that x is fake; target images are labeled Real rather than base images.
4. Experiments
Experiments evaluate SBI-trained detectors across datasets, manipulations, architectures, and inference settings. The results show strong cross-dataset and cross-manipulation generalization, with attention directed toward manipulation-independent artifacts.
- Experimental setting: Cross-dataset evaluation compares models trained on FF++ with tests on CDF, DFD, DFDC, DFDCP, and FFIW.The evaluation uses official splits except FFIW, whose original validation set serves as the test set.
- Cross-dataset evaluation: 6.08%, 5.17%, and 5.23% points are the gains over frame-level state-of-the-art methods on CDF, DFDC, and DFDCP.The average improvement is 4.58 percentage points, from 82.75% to 87.33%.
- Cross-dataset evaluation: 6.28%, 3.16%, 1.42%, 12.15%, and 10.36% points are the gains on CDF, DFD, DFDC, DFDCP, and FFIW against video-level methods.The average improvement is 6.68 percentage points, from 80.15% to 86.83%.
- Cross-manipulation evaluation: 99.99%, 99.88%, 99.91%, and 98.79% AUCs are obtained on DF, F2F, FS, and NT, respectively, with 99.64% on whole FF++.The whole-FF++ result exceeds 99.11%, and the method outperforms or nearly equals existing methods across the four manipulations.
- Ablations: All tested architectures achieve good results without critical degradation, while larger networks tend to provide greater generality.Even vanilla ResNet-50 outperforms previous methods on CDF, DFDCP, and FFIW.
- Qualitative analysis: Grad-CAM++ visualizations show sparser attention for the SBI model, which detects minor artifacts independent of manipulation type.The model also distinguishes forged faces from real ones, and SBIs are distributed across the four manipulation classes in feature space.
5. Limitations
The method has two stated limitations: it cannot capture temporal inconsistencies and performs poorly on whole-image synthesis.
- Frame-level detection cannot capture temporal inconsistencies across video frames.Sophisticated deepfake techniques with fewer spatial artifacts may therefore pass the detector.
- The method performs poorly on whole-image synthesis, where the face region or background is not the defined manipulation target.On a 20k-image FFHQ and Style-GAN synthesis set, its AUC is only 69.11%.
6. Conclusion
The paper concludes that self-blended images provide synthetic training data for learning generic, robust deepfake-detection representations without forged face images. Experiments show generalization to unseen manipulations and scenes across architectures and training datasets.
- SBIs blend slightly transformed pseudo source and target images from a single real image to reproduce forgery artifacts.The method trains detectors without forged face images.
- The paper reports superior performance to state-of-the-art methods for unseen manipulations and scenes.
- The method generalizes across different network architectures and training datasets.
A. Additional Experiments
Additional experiments examine landmark similarity, joint SBI–BI training, and source-target augmentation, supporting the importance of SBI design choices for generalization.
- Landmark Similarity: Training on the top 20 landmark-similar BI source images outperforms original BI on CDF and DFDCP.It is on par on DFD, DFDC, and FFIW, indicating that easy low-similarity samples do not contribute to generality.
- Joint Training of SBIs and BI: Joint training on SBIs and BI produces lower results than training with the proposed SBIs alone.
- Applying Source-Target Augmentation to BI: Source-target augmentation improves original BI but remains below SBIs on four of five test sets.The result indicates that blending identical images and augmenting source and target images are both important for general detection.
B. Comprehensive Results
The paper provides comprehensive video-level evaluation results and statistical details, including AUC, AP, and counts of real and fake videos, with additional evaluation on FaceShifter and DeeperForensics1.0.
- The comprehensive evaluation reports video-level area under the receiver operating characteristic curve and average precision.
- The evaluation includes the number of real and fake videos.
- Additional evaluation covers FaceShifter and DeeperForensics1.0 datasets.For DeeperForensics1.0, c23 lightly compressed real videos from FF++ are used.