Source-linked AI summary
Combining EfficientNet and Vision Transformers for Video Deepfake Detection
Davide Coccomini, Nicola Messina, Claudio Gennaro, Fabrizio Falchi
TL;DR
Realistic and increasingly accessible deepfakes make manipulated videos difficult to distinguish from real ones. The paper combines EfficientNet B0 feature extraction with Vision Transformers and proposes voting across multiple faces, achieving competitive detection results without distillation or ensembles.
Problem
Increasingly realistic and accessible deepfakes create a need for effective methods to distinguish manipulated videos from real ones.
Method
The paper combines EfficientNet B0 convolutional feature extraction with Vision Transformers and uses a voting scheme to handle multiple faces during video inference.
Results
The models obtain competitive results, with the best model achieving an AUC of 0.951 and an F1 score of 88.0% on the DFDC.
Takeaways & Limitations
Mixed convolutional-transformer networks can achieve state-of-the-art results without distillation or ensemble techniques, while EfficientNet B0-based patch extraction and voting provide effective design choices.
Takeaways & Limitations
Results for Convolutional ViT on the DFDC test set were obtained by executing the authors’ available test code and pretrained model rather than reported in the original paper.
Abstract
from arXiv · showhide
Deepfakes are the result of digital manipulation to forge realistic yet fake imagery. With the astonishing advances in deep generative models, fake images or videos are nowadays obtained using variational autoencoders (VAEs) or Generative Adversarial Networks (GANs). These technologies are becoming more accessible and accurate, resulting in fake videos that are very difficult to be detected. Traditionally, Convolutional Neural Networks (CNNs) have been used to perform video deepfake detection, with the best results obtained using methods based on EfficientNet B7. In this study, we focus on video deep fake detection on faces, given that most methods are becoming extremely accurate in the generation of realistic human faces. Specifically, we combine various types of Vision Transformers with a convolutional EfficientNet B0 used as a feature extractor, obtaining comparable results with some very recent methods that use Vision Transformers. Differently from the state-of-the-art approaches, we use neither distillation nor ensemble methods. Furthermore, we present a straightforward inference procedure based on a simple voting scheme for handling multiple faces in the same video shot. The best model achieved an AUC of 0.951 and an F1 score of 88.0%, very close to the state-of-the-art on the DeepFake Detection Challenge (DFDC).
1 Introduction
Deepfakes make highly realistic misleading videos possible, creating a severe societal problem and motivating effective detection methods. This paper combines EfficientNet B0 with Vision Transformers and proposes voting-based video inference.
- Highly realistic face-swapped videos can make it appear that replaced people are present and saying things they never said.
- Deepfake accessibility and realism force researchers to distinguish manipulated videos from real ones.
- The study analyzes combinations of EfficientNet B0 convolutional features with different Vision Transformers and compares them with state-of-the-art methods.
- CNNs retain spatial locality, an architectural prior important for detecting image-patch abnormalities and maintaining data efficiency.
- A simple voting mechanism is proposed for video inference, yielding better and more stable results.
2 Related Works
Deepfake detection research spans generative-model backgrounds, expanding datasets, classical and spatiotemporal detectors, and newer convolutional-Transformer methods. Recent approaches increasingly use Vision Transformers, sometimes with distillation from EfficientNet B7.
- GANs and VAEs are the two main generative approaches used to obtain realistic synthetic faces.
- GANs use a generator to modify videos and a discriminator to identify whether they are fake, producing increasingly credible results.
- Deepfake datasets are organized into three generations, with later generations containing larger datasets and more frames.
- DFDC supports methods ranging from EfficientNet B7 ensembles and EfficientNet-GRU models to 3DCNN, optical-flow, K-nearest-neighbor, SVM, and frequency-based detectors.
- Vision Transformer methods combine Transformers with convolutional patch extraction, while later work distills knowledge from a DFDC-pretrained EfficientNet B7.
3 Method
The proposed systems classify pre-extracted faces with mixed convolutional-Transformer architectures, then aggregate face-level outputs across time and faces to classify a video shot. They include Efficient ViT and a two-branch Convolutional Cross ViT.
- Faces are pre-extracted with MTCNN, and the models output the probability that each face has been manipulated as a supervised binary classification task.
- Models are trained per face, then video-shot decisions aggregate outputs over time and across multiple faces.
- Efficient ViT: Efficient ViT uses EfficientNet B0 to extract features from 7 × 7 image chunks before Vision Transformer processing and CLS-token classification.
- Convolutional Cross ViT: Convolutional Cross ViT uses small- and large-patch branches whose Transformer outputs interact through cross attention.
- Convolutional Cross ViT: The two branch CLS outputs become separate logits, which are summed before a final sigmoid produces probabilities.
4 Experiments
The experiments evaluate mixed convolutional-transformer models on DFDC and FaceForensics++, including architecture choices, training, and multi-face inference. EfficientNet-based Cross ViT models perform competitively, while voting provides a small inference improvement.
- Evaluation setup: The models were evaluated against Convolutional ViT, distilled ViT, and EfficientNet B7 methods on DFDC and FaceForensics++.The study used two widely used datasets and included comparisons with the DFDC challenge winner.
- Inference: Using more than 30 faces at inference appears statistically useless because the F1-score reaches a plateau with no more than 30 faces.This test was conducted on one of the networks.
- DFDC results: EfficientNet models achieve considerably higher AUC and F1-scores than the Convolutional ViT on DFDC.Cross Vision Transformer models obtain the best results among the proposed architectures.
- DFDC results: Cross Vision Transformer with EfficientNet B0 reaches competitive performance using less than 1/3 of the parameters of the top methods.Its results remain slightly below the first two state-of-the-art methods in AUC and F1-score, without distillation or ensembles.
- Inference: The proposed voting procedure slightly improves results over averaging scores from all faces indistinctly within a video.The procedure groups faces by actor, averages scores over time, and then applies hard voting.
- FaceForensics++ results: On FaceForensics++, the models outperform the original Convolutional ViT on all sub-datasets except DeepFakes, while Convolutional Cross ViT with EfficientNet B0 reaches 80% mean accuracy.NeuralTextures produces lower accuracy values than some other sub-datasets, and the proposed models have a higher average accuracy overall.
5 Conclusions
The paper demonstrates mixed convolutional-transformer networks for face-based deepfake detection, combining EfficientNet feature extraction with Vision Transformers. It also introduces multi-scale processing and a voting-based inference scheme, achieving competitive results without distillation or ensembles.
- 5 Conclusions: Mixed convolutional-transformer networks combine EfficientNet B0 visual feature extraction with Vision Transformers for deepfake detection.The Transformer provides a global description for the downstream task.
- 5 Conclusions: The Convolutional Cross ViT captures local and global image details through multi-scale analysis.The architecture uses two scales to analyze manipulation artifacts.
- 5 Conclusions: The models obtain state-of-the-art results without requiring distillation or ensemble techniques.The conclusion specifically highlights EfficientNet B0 as effective despite being the smallest EfficientNet network.
- 5 Conclusions: The voting scheme averages multiple actor-face scores over time before hard voting, producing slightly better and more stable results than global score averaging.This explicitly addresses videos containing multiple faces.