Source-linked AI summary
Deepfake Detection using Spatiotemporal Convolutional Networks
Oscar de Lima, Sean Franklin, Shreshtha Basu, Blake Karwoski, Annet George
TL;DR
Most Deepfake detectors analyze individual frames and leave temporal information across video frames unused. This paper evaluates spatiotemporal video-classification methods on Celeb-DF, and reports that several outperform state-of-the-art frame-based methods, with R3D outperforming the other tested networks.
Problem
Most Deepfake detection methods rely on single-frame analysis and do not leverage temporal information across video frames.
Source: §1 · p. 1Method
The paper applies 3D-input video-classification techniques to Deepfake detection and evaluates spatiotemporal convolutional methods on Celeb-DF.
Source: §1 · p. 1Results
Several tested networks incorporating temporal information beat state-of-the-art frame-based methods, and R3D outperformed the other networks, including I3D.
Source: §5 · p. 5Takeaways & Limitations
The paper concludes that incorporating temporal information helped several tested networks outperform frame-based baselines on the evaluated Deepfake detection task.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Better generative models and larger datasets have led to more realistic fake videos that can fool the human eye but produce temporal and spatial artifacts that deep learning approaches can detect. Most current Deepfake detection methods only use individual video frames and therefore fail to learn from temporal information. We created a benchmark of the performance of spatiotemporal convolutional methods using the Celeb-DF dataset. Our methods outperformed state-of-the-art frame-based detection methods. Code for our paper is publicly available at https://github.com/oidelima/Deepfake-Detection.
1. Introduction
Deepfake detection is a binary real-versus-fake video task, but most existing methods analyze frames individually rather than using temporal information. This work applies video-classification techniques to the task and evaluates them on the challenging Celeb-DF v2 dataset.
1. Introduction
- Deepfakes are manipulated videos that the paper identifies as a growing threat to social security, privacy, and democracy.Source: §1 · p. 1
- Most Deepfake detectors analyze individual frames and do not leverage temporal information across video frames.The paper frames detection as binary classification and notes that action recognition investigates information across frames.Source: §1 · p. 1
- The paper applies video-classification techniques using 3D input to Deepfake classification.The tested networks, apart from RCN, were pretrained on Kinetics, and methods were analyzed on fixed-length video clips.Source: §1 · p. 1
- Celeb-DF v2 contains 590 real videos and 5,639 synthesized videos and was selected as a challenging dataset because its fakes have fewer noticeable visual artifacts.Source: §1 · p. 1
2. Related Work
Related work covers the evolution of face synthesis and detection, the datasets used to evaluate detectors, and video-classification architectures that model spatiotemporal information.
2. Related Work
- The Deepfake maker method detects, aligns, and crops faces before an encoder-decoder generates a target person's face with the source expression [25].The paper situates it among earlier synthesis methods and later GAN and style-transfer techniques.Source: §2 · p. 1
- Celeb-DF was presented as more realistic because its synthesis produces fewer visual artifacts; it contains 590 real videos and 5,639 fakes from interviews with 59 celebrities [25].The videos average 13 seconds at 30 fps and have 256 x 256 resolution.Source: §2 · p. 1
- Prior detection approaches include XceptionNet trained on FaceForensics++ [31] and methods such as Two-Stream [39], MesoNet [7], Headpose [38], and FWA [24].The paper also lists VA, Multi-task [27], capsule [28], and DSP-FWA.Source: §2 · p. 1
- Li et al. [25] tested the methods on Celeb-DF without training on it, whereas Kumar and Bhavsar [23] trained and tested on Celeb-DF using the XceptionNet architecture with metric learning.Source: §2 · pp. 1–2
- Video-classification work includes C3D [37], Recurrent CNN, ResNet-3D [17], mixed 3D-2D ResNet [37], ResNet (2+1)D [37], and I3D [10].The paper notes that these spatiotemporal methods had not achieved the same level of success as deep-learning image classification.Source: §2 · p. 2
3. Method
The method evaluates multiple video-network architectures for DeepFake classification, with random cropping and temporal jittering applied across methods. Its designs range from a Fourier-feature frame classifier and an LSTM-based recurrent model to several 3D convolutional architectures.
3. Method
- The method section applies random cropping and temporal jittering to all evaluated methods.Source: §3 · p. 2
3.1. Pre-Processing
- Preprocessing crops faces in every frame, restacks them as video, and avoids stretching source frames to prevent distortion.RetinaFace [12] was chosen after Haar Cascades produced false positives and BlazeFace bounding boxes jittered between frames.Source: §3.1 · p. 2
3.2. DFT
- The non-temporal DFT baseline compresses an image's 2D amplitude spectrum into a 300 × 1 feature vector using azimuthal averaging for binary classification.The paper also describes using these features with k-means clustering to classify unlabeled datasets.Source: §3.2 · p. 2
3.3. RCN
- The RCN combines a VGG-11 feature-extraction encoder with a three-layer LSTM decoder to analyze sequences of 10 randomly selected consecutive frames.Per-frame features are stacked over time and passed to the LSTM, whose descriptors feed fully connected classification layers.Source: §3.3 · p. 2
3.4. R3D
- R3D uses residual connections with 3D convolutions and pooling to capture motion features from adjacent frames.The implementation is an 18-layer model with 33.17 million weights, pretrained on Kinetics, and takes 16 RGB frames at 112 x 112 as input.Source: §3.4 · p. 2
3.5. ResNet Mixed 3D-2D Convolution
- MC3 begins with 3D convolutions and switches to 2D convolutions from layer 3 onward.The implementation has 18 layers and 11.49 million weights, uses 16 RGB frames at 112 × 112, and is pretrained on Kinetics [4].Source: §3.5 · p. 3
3.6. ResNet (2+1)D
- R(2+1)D replaces each 3D convolution with a spatial 2D convolution followed by a temporal 1D convolution.The paper describes the factorization as introducing more nonlinearities and making optimization easier; its 18-layer model has 31.30 million parameters and Kinetics pretraining.Source: §3.6 · p. 3
3.7. I3D
- I3D inflates 2D convolution filters and pooling kernels to 3D, using an Inception-V1 base network to learn spatiotemporal features.The implementation uses RGB input and cropped faces resized to 256 x 256 before a random 224 x 224 crop; its 12.29 million weights were pretrained on Charades.Source: §3.7 · p. 3
4. Experiments
The experiments evaluate video-based methods on Celeb-DF against frame-level baselines, using test accuracy and ROC-AUC. The reported classical frame-based method achieves 66.8% accuracy, which the authors relate to weak statistical differences between real and fake images in this dataset.
4. Experiments
- The experiments measure top test accuracy and top ROC-AUC, rounding scores to four decimal places.Source: §4 · p. 4
4.1. Baselines
- The video-based methods are compared with frame-level baselines, including previously benchmarked methods and methods trained and tested on Celeb-DF.The paper also reimplemented the discrete-Fourier-transform method and trained it on Celeb-DF.Source: §4.1 · p. 4
4.2. Result and analysis
- The spatiotemporal networks were trained on Celeb-DF from published pretrained weights without frozen layers for over 25 epochs, with best ROC-AUC and test accuracy recorded.Training used stochastic gradient descent with momentum, cross-entropy loss, and inverse-frequency class weighting to address sample imbalance.Source: §4.2 · p. 4
- 66.8% accuracy was achieved by the classical frame-based method on Celeb-DF.The authors suggest its lower performance may reflect the reduced statistical discrepancy between real and fake images in Celeb-DF compared with FaceForensics++.Source: §4.2 · p. 4
5. Conclusions
The paper evaluates action-recognition methods for Deepfake detection using temporal information. Several networks beat frame-based baselines, and R3D outperformed the other tested networks, including I3D.
5. Conclusions
- The evaluated action-recognition networks make decisions using temporal information, distinguishing them from previously explored methods.The conclusion attributes several networks’ advantage over frame-based methods to incorporating this extra information.Source: §5 · pp. 4–5
- Figure 5 compares power-spectrum means and standard deviations for real and fake images in FaceForensics and Celeb-DF.Source: Figure 5 · p. 5
- R3D outperformed the other tested networks, including I3D, while several temporal networks beat state-of-the-art frame-based methods.Figure 4 presents top test accuracies for spatiotemporal convolutional methods.Source: §5 · p. 5, Figure 4 · p. 5