Source-linked AI summary
DeepFake Detection by Analyzing Convolutional Traces
Luca Guarnera, Oliver Giudice, Sebastiano Battiato
TL;DR
Deepfake face images can be highly realistic, motivating detection methods that expose hidden generation traces. This paper uses Expectation Maximization to model convolutional features and evaluates naive classifiers across five GAN architectures. The method discriminates generated images and distinguishes architectures, while its broader deployment remains subject to stated scope boundaries.
Problem
Realistic Deepfakes are difficult to authenticate, while existing image-analysis methods tend to fail on them.
Method
Expectation Maximization extracts local features modeling convolutional traces, which naive classifiers use to analyze face images generated by five GAN architectures.
Results
The extracted information discriminated Deepfakes and distinguished generation architectures, reaching 99.31% maximum accuracy between STYLEGAN and STYLEGAN2.
Takeaways & Limitations
The modeled fingerprint provides an explainable basis for forensic detection and attribution of realistic face Deepfakes.
Takeaways & Limitations
Future work must assess robustness to image editing, compression, antiforensics, video analysis, and operation without prior knowledge of the generation process.
Abstract
from arXiv · showhide
The Deepfake phenomenon has become very popular nowadays thanks to the possibility to create incredibly realistic images using deep learning tools, based mainly on ad-hoc Generative Adversarial Networks (GAN). In this work we focus on the analysis of Deepfakes of human faces with the objective of creating a new detection method able to detect a forensics trace hidden in images: a sort of fingerprint left in the image generation process. The proposed technique, by means of an Expectation Maximization (EM) algorithm, extracts a set of local features specifically addressed to model the underlying convolutional generative process. Ad-hoc validation has been employed through experimental tests with naive classifiers on five different architectures (GDWCT, STARGAN, ATTGAN, STYLEGAN, STYLEGAN2) against the CELEBA dataset as ground-truth for non-fakes. Results demonstrated the effectiveness of the technique in distinguishing the different architectures and the corresponding generation process.
1. Introduction
Deepfakes are machine-learning-generated or altered face media that can appear realistic enough to evade human recognition, creating risks for news authenticity, politics, companies, and privacy. The paper introduces an explainable detection method that models convolutional traces across five GAN architectures and evaluates it on varied image sizes.
- Motivation: Deepfakes synthetically generate, alter, or swap faces in images and videos using deep-learning generative models.The term covers multimedia content created or altered through machine-learning generative models.
- Motivation: Realistic Deepfakes can be difficult for human observers to distinguish from authentic media.The paper describes generated multimedia as not easily recognizable as real or fake by the human eye.
- Motivation: Deepfakes threaten the authenticity of mass-media news and pose risks to politics, companies, and individual privacy.The introduction motivates tools that can unmask or detect such content.
- Contribution: The proposed method uses an Expectation Maximization algorithm to extract local features modeling convolutional traces, then trains naive classifiers across GDWCT, STARGAN, ATTGAN, STYLEGAN, and STYLEGAN2.The method targets images of human faces and treats the modeled information as related to the generating architecture.
- Contribution: Experiments use an almost-in-the-wild dataset spanning five generation techniques and different image sizes, unlike existing techniques limited to specific sizes and usually one GAN family.The paper presents this broader evaluation setting as a distinction from the state of the art.
2. Related Works
Related work covers GAN generation, prominent face-generation architectures, and detection methods based on visual, frequency, and learned image characteristics. The paper positions its approach as reverse engineering of generator traces to support both classification and forensic attribution across architectures and image sizes.
- GAN foundations: GANs train a generator and discriminator adversarially so the generator learns to approximate the training-data distribution.The discriminator estimates whether samples come from training data or the generator, while the generator learns by encouraging discriminator errors.
- Deepfake generation techniques: STARGAN performs image-to-image translation across multiple domains using one model and facial-attribute or expression labels.It was trained on CELEBA and RaFD datasets and reported superior visual quality relative to compared methods.
- Deepfake generation techniques: STYLEGAN controls generated face style through an intermediate latent space and injected noise, while STYLEGAN2 modifies the generator to reduce imperfections.STYLEGAN2 includes redesigned normalization, multiresolution, and regularization methods.
- Deepfake generation techniques: ATTGAN applies an attribute-classification constraint to generated images to preserve correct modifications of desired facial attributes.The reported experiments found that ATTGAN exceeded state of the art for realistic facial-attribute modification.
- Deepfake generation techniques: GDWCT uses group-wise deep whitening and coloring to improve styling capacity, with reported gains in computational efficiency and generated-image quality.Its evaluation covered face, artwork, cat-to-dog, and other image-translation datasets.
- Deepfake detection methods: Deepfake authentication remains difficult even for human eyes, and conventional image-analysis methods tend to fail on these manipulations.Prior work includes image-history reconstruction, CFA interpolation analysis, and compression-parameter analysis.
- Deepfake detection methods: Prior detectors use contrastive characteristics, frequency spectra, or GAN simulation to classify authentic and fake images.Examples include automated video-frame benchmarks and frequency-domain methods such as AutoGAN-based detection.
- Positioning of the proposed method: The proposed distinction is reverse engineering of a GAN’s final computational layer to capture periodic traces and identify both fakery and the likely generation technique.The paper relates this goal to camera-model identification and evaluates five techniques with different image sizes.
3. Extracting Convolutional Traces
The method models local pixel correlations to extract convolutional traces associated with transpose-convolution layers in Deepfake generation. An EM algorithm estimates these relationships and produces feature vectors for classification.
- Method rationale: Transpose-convolution operations in GANs are treated as the source of local pixel correlations that can reveal generation traces.The approach focuses on the transpose-convolution layers used by the described Deepfake-generation techniques.
- Feature extraction: The method applies convolution with an N × N kernel to numerically represent relationships among neighboring pixels.The kernel vector is intended to capture the hidden relationship exploited as a forensic trace.
- Expectation-Maximization: The algorithm models pixels using two distributions, including a zero-mean Gaussian model with unknown variance and a uniform model.The Gaussian variance is estimated during maximization, after which kernel parameters are estimated by least squares.
- Expectation-Maximization: EM alternates between estimating model-membership probabilities and estimating weighted parameters for the models.The expectation step computes membership probabilities, while maximization estimates parameters from those probabilities.
- Feature representation: The EM steps are iterated, and the resulting kernel parameters form a feature vector extracted separately from RGB channels.For a 3 × 3 kernel, excluding k_0,0, concatenation across three channels yields 24 features.
- Computational cost: The algorithm’s computational complexity is linear in the number of input characteristics, objects, and iterations.These quantities are denoted d, n, and t, respectively.
4. Classification Phase and Results
The method evaluates EM-derived convolutional-trace features on six image datasets using KNN, SVM, and LDA classifiers. It distinguishes authentic images from several GAN architectures, separates STYLEGAN from STYLEGAN2, and achieves lower performance when grouping all Deepfakes against CELEBA.
- Experimental setup: Six datasets pair authentic CELEBA faces with Deepfakes generated by STARGAN, STYLEGAN, STYLEGAN2, GDWCT, and ATTGAN.STYLEGAN and STYLEGAN2 images were downloaded, while the other three GANs generated images in inference mode.
- Feature extraction and classification: EM extracts convolutional-trace feature vectors with 3×3, 4×4, 5×5, and 7×7 kernels, which KNN, SVM, and LDA classify.The pipeline tests authentic-versus-one-GAN and authentic-versus-all-Deepfake tasks.
- Authentic-versus-GAN results: 93.17% maximum accuracy distinguishes CELEBA from STARGAN using linear SVM with a 7×7 kernel.The corresponding maxima are 92.67% for ATTGAN and 88.40% for GDWCT, both using KNN with a 3×3 kernel.
- Authentic-versus-GAN results: 99.81% maximum accuracy distinguishes CELEBA from STYLEGAN2 using linear SVM with a 4×4 kernel.CELEBA versus STYLEGAN reaches 99.65% with KNN and a 4×4 kernel.
- Cross-architecture discrimination: 99.31% maximum accuracy separates STYLEGAN from STYLEGAN2, supporting sensitivity to generator-structure differences.The authors link the differing traces to modifications of STYLEGAN2’s generator and convolution layers.
- All-Deepfake binary classification: 90.22% maximum accuracy is obtained when classifying CELEBA against all Deepfakes, requiring nonlinear classifiers because the samples are not linearly separable.The best result uses KNN with K=5 and a 3×3 EM kernel; VGG-16 reaches 53% on the same binary task.
5. Conclusions and future works
The study concludes that EM-derived features provide an effective fingerprint for discriminating images generated by recent GAN architectures. Future work will examine broader operating conditions, including deployment without prior knowledge of the generation process.
- EM-derived features formed an effective fingerprint for discriminating images generated by recent GAN architectures.The conclusion specifically targets realistic human-face generation architectures.
- Figure 4 represents CELEBA and DeepNetwork features in two dimensions across five classification tasks and all kernel sizes.
- Figure 5 represents CELEBA-versus-DeepNetwork binary classifications using different kernel sizes.
- Future works: Adapting the method to in-the-wild situations without a priori knowledge of the generation process remains a future-work objective.