Source-linked AI summary
Deepfake Video Detection Using Convolutional Vision Transformer
Deressa Wodajo, Solomon Atnafu
TL;DR
Deepfakes can be used maliciously, while existing detection methods have limited generalizability. This paper proposes a Convolutional Vision Transformer combining convolutional feature extraction with Vision Transformer attention. Tested on 400 unseen DFDC videos, the model achieved 91.5 percent accuracy, an AUC value of 0.91, and a loss value of 0.32.
Problem
Existing Deepfake detection methods lack generalizability, including reliable detection of multiple spoofing techniques and unseen techniques, while Deepfakes can be used maliciously.
Method
The proposed Convolutional Vision Transformer combines a CNN feature-learning component with a Vision Transformer component that processes learned features using attention.
Results
91.5 percent accuracy, an AUC value of 0.91, and a loss value of 0.32 were achieved on 400 unseen DFDC videos.
Takeaways & Limitations
Combining CNN and Transformer learning capacities supports the paper's generalized Deepfake detection model across diverse facial images and features.
Takeaways & Limitations
The model's accuracy is affected by inherent problems in the face detection deep-learning libraries used during processing.
Abstract
from arXiv · showhide
The rapid advancement of deep learning models that can generate and synthesis hyper-realistic videos known as Deepfakes and their ease of access to the general public have raised concern from all concerned bodies to their possible malicious intent use. Deep learning techniques can now generate faces, swap faces between two subjects in a video, alter facial expressions, change gender, and alter facial features, to list a few. These powerful video manipulation methods have potential use in many fields. However, they also pose a looming threat to everyone if used for harmful purposes such as identity theft, phishing, and scam. In this work, we propose a Convolutional Vision Transformer for the detection of Deepfakes. The Convolutional Vision Transformer has two components: Convolutional Neural Network (CNN) and Vision Transformer (ViT). The CNN extracts learnable features while the ViT takes in the learned features as input and categorizes them using an attention mechanism. We trained our model on the DeepFake Detection Challenge Dataset (DFDC) and have achieved 91.5 percent accuracy, an AUC value of 0.91, and a loss value of 0.32. Our contribution is that we have added a CNN module to the ViT architecture and have achieved a competitive result on the DFDC dataset.
1. Introduction
Deepfakes are increasingly accessible and can be used both creatively and maliciously. Existing detection methods face generalizability and preprocessing weaknesses, motivating a generalized CViT architecture.
- Deepfakes replace a targeted person’s face with someone else’s face in video and can also support creative media applications.
- Malicious uses of deceptive Deepfakes pose a threat to the general public.
- Current detection methods lack generalizability because they often target artifacts or behaviors associated with particular creation tools.
- The paper identifies data preprocessing and generality as two weaknesses in existing Deepfake detection systems.
- The proposed generalized CViT combines CNN and Transformer architectures, emphasizes preprocessing, and trains on diverse DFDC face images.
2. Related Work
Related work describes Deepfake generation through generative models and identity or expression manipulation, alongside detection methods based on behavior, biological signals, visual artifacts, and deep networks.
- Deepfakes can be generated with GANs, autoencoders, and techniques including face swap, puppet-master, lip-sync, face reenactment, and synthetic media generation.
- FakeApp used two autoencoder networks with a shared encoder and separate decoders to swap source and target faces.
- Face swapping inserts a source face into a target image or video and can use GANs, traditional computer-vision tools, or autoencoders.
- Facial-expression reenactment transfers or alters expressions between people, turning one identity into a puppet.
- Detection methods target physical behavior, biological signals or GAN fingerprints, and visual artifacts, with artifact-based methods requiring substantial training data.
- Prior detectors include CNNs, CNN-RNN systems, and ensembles addressing forged images, videos, replay attacks, face swaps, and reenactments.
3. Convolutional Vision Transformer
The proposed CViT pipeline preprocesses video faces and combines convolutional feature learning with a Vision Transformer for Deepfake classification. Its CNN extracts features, while the ViT processes embedded feature patches through attention-based encoding.
- The detection model has preprocessing and detection components, with face extraction, augmentation, training, validation, and testing stages.
- Face extraction prepares video data by extracting 224 x 224 RGB face images for model processing.
- The ViT receives Feature Learning outputs, splits feature maps into seven patches, and embeds them into a 1 x 1024 sequence with positional information.
- The Feature Learning component is a CNN without a fully connected classification layer, designed to extract face-image features for the ViT.
- The Feature Learning component uses stacked convolutional operations followed by batch normalization and ReLU nonlinearities.
- The ViT Transformer uses an encoder with multi-head self-attention and multilayer perceptron blocks to process the embedded features.
4. Experiments
The experiments combine extensive face preprocessing with CViT training and evaluate detection on Deepfake datasets. Results show strong DFDC performance but sensitivity to dataset and preprocessing conditions.
- Dataset preparation: 162,174 balanced facial images were stored as 224 x 224 JPEGs and split 70:15:15 into training, validation, and testing sets.The sets contained 112,378 training, 24,898 validation, and 24,898 testing images, with equal real and fake classes.
- Dataset preparation: 308,130 facial images resulted after Albumentations augmentation was applied to 90 percent of the face images.Albumentations provides a large class of image transformations for data augmentation.
- Model training: The CViT was trained for 50 epochs using binary crossentropy, normalized mini-batches of 32 images, Adam optimization, and augmentation during training.Normalization used mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225].
- Evaluation: 91.5 percent accuracy, an AUC value of 0.91, and a loss value of 0.32 were achieved on 400 unseen DFDC videos.The classification process used 30 facial images from each video.
- Evaluation: The model performed well on DFDC, UADFV, and FaceForensics++, but poorly on the FaceForensics++ FaceShifter dataset.The authors suggest that the model likely failed to learn the difficult visual artifacts in FaceShifter videos.
- Effects of data processing: 637 nonface regions were found during preliminary testing, and retaining them reduced accuracy to 69.5 percent while increasing loss to 0.4.The test covered varied DFDC settings, including lighting, pose, movement, gender, skin color, and subject count.
- Effects of data processing: Face recognition was used as a filter for BlazeFace detections because it rejected more false positives than the other tested libraries.The study combined BlazeFace, MTCNN, and face recognition to improve face detection accuracy.
5. Conclusion
The study presents CViT as a generalized Deepfake detector that combines CNN and Transformer capabilities with extensive preprocessing and diverse training data. Tested on 400 DFDC videos, the model achieved 91.5 percent accuracy, while the authors identify broader dataset integration as future work.
- CViT combines CNNs and Transformers to learn local and global image features and relate nonlocal features through attention.The CNN learns local features, while the Transformer contributes global modeling and attention-based relationships.
- The model emphasizes data preprocessing during both training and classification and uses a diverse Deepfake-detection dataset.
- 91.5 percent accuracy was achieved when CViT was tested on 400 DFDC videos.
- The authors plan to add other Deepfake research datasets to improve diversity, accuracy, and robustness.