Source-linked AI summary
Deepfakes Detection with Automatic Face Weighting
Daniel Mas Montserrat, Hanxiang Hao, S. K. Yarlagadda, Sriram Baireddy, Ruiting Shao, János Horváth, Emily Bartusiak, Justin Yang, David Güera, Fengqing Zhu, Edward J. Delp
TL;DR
Realistic deepfakes are increasingly widespread, while detectors may not generalize beyond research datasets. This paper uses face-focused CNN and RNN processing with automatic weighting and evaluates it on DFDC, achieving competitive leaderboard performance but remaining limited to visual face manipulation detection.
Problem
Realistic deepfake media is increasingly widespread, while existing detectors can fail when evaluated beyond the datasets used in research environments.
Method
The method combines face detection, EfficientNet feature extraction with ArcFace, automatic face weighting, and GRU-based temporal prediction.
Results
The method achieves a 0.321 log-likelihood error and ranks 117 of 2275 teams, placing it in the top 6% of the public leaderboard.
Takeaways & Limitations
Combining convolutional and recurrent networks with automatic face weighting and boosting yields high DFDC detection accuracy and predictions in under eight seconds on a single GPU.
Takeaways & Limitations
The work focuses on face manipulation detection and excludes audio analysis, which the authors identify as a possible source of future accuracy improvement.
Abstract
from arXiv · showhide
Altered and manipulated multimedia is increasingly present and widely distributed via social media platforms. Advanced video manipulation tools enable the generation of highly realistic-looking altered multimedia. While many methods have been presented to detect manipulations, most of them fail when evaluated with data outside of the datasets used in research environments. In order to address this problem, the Deepfake Detection Challenge (DFDC) provides a large dataset of videos containing realistic manipulations and an evaluation system that ensures that methods work quickly and accurately, even when faced with challenging data. In this paper, we introduce a method based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs) that extracts visual and temporal features from faces present in videos to accurately detect manipulations. The method is evaluated with the DFDC dataset, providing competitive results compared to other techniques.
1. Introduction
Deepfakes are increasingly realistic and widely distributed, making manipulated media harder to distinguish from authentic content. The paper proposes a CNN–RNN detector with automatic face weighting and evaluates it on DFDC.
- Deepfakes increasingly blur the distinction between real and fake multimedia, while social media enables large-scale misinformation spread.
- Facial manipulations such as face swaps and altered speech can fabricate people’s locations, statements, or actions.
- Existing manipulation detectors often fail on newer, realistic deep-learning alterations, motivating robust and rapid detection methods.
- The proposed architecture combines CNN and RNN processing, automatically weights reliable frames, and predicts whether a video is real or fake.
- The method achieves a 0.321 log-likelihood error on DFDC, ranking 117 of 2275 teams in the public leaderboard.
2. Related Work
Related work covers established face-generation techniques and detection methods based on visual artifacts or temporal CNN–RNN pipelines. The proposed system is illustrated as a face-focused detection pipeline.
- Face manipulation methods include FaceSwap, Face2Face, DeepFakes, and NeuralTextures, spanning computer-graphics and learning-based approaches.
- DeepFakes uses shared-encoder autoencoders to reconstruct source and target faces before blending the generated face into target video.
- Detection research includes CNN classifiers for generated images, forensic fingerprints, and temporal CNN–RNN pipelines for manipulated videos.
- The proposed system detects faces with MTCNN, extracts face-region features with EfficientNet, then uses AFW and GRU components to classify videos.
3. Deepfake Detection Challenge Dataset
DFDC provides a large, varied video dataset containing face and audio manipulations under challenging recording conditions. Its private evaluation uses log-likelihood loss that heavily penalizes confident errors.
- DFDC contains 123,546 ten-second videos with 300 frames each, including face and audio manipulations involving one or more people.
- The videos span varied backgrounds, illumination, orientations, and quality, with training resolutions of 1920 × 1080 or 1080 × 1920 pixels.
- The private test set reports log-likelihood loss, which sharply penalizes predictions that are both confident and wrong.
- Because accuracy can coexist with high log-likelihood error, the evaluation system creates an additional challenge for detection methods.
4. Proposed Method
The proposed method detects facial manipulations by combining face-focused CNN features with automatic face weighting and GRU-based temporal modeling. It emphasizes reliable face regions, integrates information across frames, and uses ArcFace and video-level losses during end-to-end training.
- Pipeline: The pipeline detects faces with MTCNN, extracts features using EfficientNet-b5, and predicts manipulation with AFW and GRU layers.It processes face regions rather than entire frames because manipulations are located within faces and faces occupy small frame regions.
- Face Feature Extraction: ArcFace replaces softmax cross-entropy to produce more discriminative real-versus-fake face features for high-quality photo-realistic fakes.Its penalization terms reduce intra-class variation and enlarge inter-class separation in angular space.
- Motivation: Naively averaging frame predictions can fail with false face detections, multiple faces, or blurry frames, motivating reliability-aware video aggregation.A correct frame-level CNN prediction can still yield an incorrect video-level decision after uniform averaging.
- Automatic Face Weighting: Automatic face weighting emphasizes reliable face regions and aggregates their logits into a video-level manipulation probability.The mechanism assigns nonnegative weights to face-region logits, suppressing less reliable detections before weighted aggregation.
- Temporal Modeling: A GRU combines face features, logits, weights, and weighted probabilities across sampled frames to produce a final video estimate.Each GRU input has 2051 dimensions: 2048 EfficientNet features plus the estimated logit, weight, and manipulated probability.
- Training Process: The method trains EfficientNet, GRU, and AFW components end-to-end with ArcFace for frame-level features and BCE losses for video-level outputs.An optional 2000-batch ArcFace initialization did not improve accuracy but enabled faster convergence and more stable training.
5. Experimental Results
Experiments show that face-focused models outperform full-video processing and that automatic face weighting with a GRU further improves detection. Boosting and test-time augmentation reduce the DFDC log-likelihood error to 0.321, ranking the method 117th of 2275 teams.
- Benchmark comparison: Automatic face weighting with a GRU further improves balanced accuracy over face-region baselines and EfficientNet-b5 and Xception.The comparison is reported in Table 1, whose caption identifies balanced accuracy for the presented method and previous works.
- Benchmark comparison: Conv-LSTM improves considerably when restricted to detected face regions because full-video feature extraction misses manipulations within faces.The experiments also report good accuracy for Xception and EfficientNet-b5.
- Ablation by stage: The GRU prediction achieves the highest balanced accuracy among the EfficientNet logits, automatic face weighting, and GRU stages.Table 2 reports the balanced accuracy at each stage of the method.
- Error analysis: The network typically fails on highly realistic manipulations in blurry or low-quality images but detects challenging manipulations in high-quality videos.Figure 4 presents correctly and incorrectly detected examples.
- Test-time enhancements: 0.321 log-likelihood error places the method 117th of 2275 teams when boosting and test augmentation are both used.The reported leaderboard position is 5.1% of the competition’s teams.
6. Conclusions
The paper concludes that combining convolutional and recurrent networks with automatic face weighting detects face manipulations accurately on DFDC. The method processes videos in under eight seconds, while future work should address emerging manipulations and incorporate audio.
- Conclusion: Combining convolutional and recurrent neural networks achieves high detection accuracies on the DFDC dataset.The conclusion presents the combined CNN-RNN approach as the paper’s main result.
- Conclusion: Automatic weighting of face regions and boosting techniques provide more robust predictions while processing videos in less than eight seconds with one GPU.The conclusion links the weighting and boosting components to robustness and reports the processing-time scope.
- Future work: The approach remains open to different face detectors, backbone architectures, and multi-frame prediction techniques because its design is modular.These are identified as possible improvements rather than evaluated extensions.
- Limitations: The work focuses on face-manipulation detection and omits audio analysis, which the authors identify as a possible source of future accuracy improvement.The paper also notes that new deepfake-generation techniques continue to emerge.