Source-linked AI summary
Two-branch Recurrent Network for Isolating Deepfakes in Videos
Iacopo Masi, Aditya Killekar, Royston Marian Mascarenhas, Shenoy Pratik Gurudatt, Wael AbdAlmageed
TL;DR
Deepfakes create a need for video-based detection of hyper-realistic face manipulations. The paper uses a two-branch recurrent architecture and a separation-oriented loss, reporting performance that outperforms or matches state of the art while leaving room for improvement at low false alarm rates.
Problem
The rapid spread of realistic deepfake videos threatens trust in visual media and motivates detection methods tailored to video streams.
Method
A recurrent model processes aligned face sequences using RGB and LoG branches that combine color and artifact-amplified frequency information, with a loss compacting natural-face representations and separating manipulated ones.
Results
The method outperforms or matches state-of-the-art results across the evaluated deepfake-detection benchmarks.
Takeaways & Limitations
The study shows good video-level performance for cross-testing, supporting the method’s potential for video-based deepfake detection within the reported scope.
Takeaways & Limitations
The method processes sequences, or windows, of aligned video faces with cardinality F.
Abstract
from arXiv · showhide
The current spike of hyper-realistic faces artificially generated using deepfakes calls for media forensics solutions that are tailored to video streams and work reliably with a low false alarm rate at the video level. We present a method for deepfake detection based on a two-branch network structure that isolates digitally manipulated faces by learning to amplify artifacts while suppressing the high-level face content. Unlike current methods that extract spatial frequencies as a preprocessing step, we propose a two-branch structure: one branch propagates the original information, while the other branch suppresses the face content yet amplifies multi-band frequencies using a Laplacian of Gaussian (LoG) as a bottleneck layer. To better isolate manipulated faces, we derive a novel cost function that, unlike regular classification, compresses the variability of natural faces and pushes away the unrealistic facial samples in the feature space. Our two novel components show promising results on the FaceForensics++, Celeb-DF, and Facebook's DFDC preview benchmarks, when compared to prior work. We then offer a full, detailed ablation study of our network architecture and cost function. Finally, although the bar is still high to get very remarkable figures at a very low false alarm rate, our study shows that we can achieve good video-level performance when cross-testing in terms of video-level AUC.
1 Introduction
Deepfakes have made realistic video manipulation easier and more socially harmful, motivating video-level detectors with low false alarm rates. The paper proposes two complementary components: frequency-enhancing representation learning and a loss that separates natural from manipulated faces.
- Deepfakes replaced labor-intensive face swaps with realistic videos that can falsely portray victims and spread political chaos, revenge porn, and defamation.
- 14,678 realistic-looking fake videos were estimated to exist, up from 7,964 in December 2018, amid a reported 100% annual growth rate.
- The paper proposes a densely connected two-branch extractor that combines color-domain information with multi-scale LoG frequency information.
- The LoG branch suppresses low-level image content and acts as a band-pass filter to amplify manipulation artifacts.
- A novel loss compacts natural-face representations and pushes manipulated faces away, unlike binary cross-entropy classification objectives.
- The method models aligned face sequences with a backbone and bi-directional LSTM, while reporting TAR at low FAR alongside AUC, pAUC, and tAUC.
2 Prior Work
Prior work spans specialized semantic inconsistency detectors, data-driven classifiers, and GAN-synthesis detectors, while newer benchmarks emphasize video-level evaluation. This paper differs by fusing color and frequency information in a recurrent two-branch architecture.
- Face-forensics research lacked large-scale datasets for training and evaluation, and earlier manipulation detectors were mainly tested on still images.
- FaceForensics++ contains 1.8 million manipulated frames generated with FaceSwap, DeepFakes, Face2Face, and NeuralTextures.
- The DFDC preview dataset introduced web-oriented deepfake detection evaluation, with performance considered at the video level rather than only the frame level.
- Prior detectors used semantic cues such as blinking and head-pose inconsistencies, hand-crafted artifact features, or learned classifiers including XceptionNet.
- GAN-synthesis detectors model fingerprints in entirely generated faces, whereas this work targets deepfakes and fuses RGB information with the frequency domain.
- The architecture processes video face sequences using two DenseBlocks, fused features, and a bi-directional LSTM supervised by a novel loss.
3 Method
The method combines RGB and multi-scale LoG representations in a two-branch recurrent network, then learns a feature space that compacts natural faces and separates manipulated faces with two hyperspherical margins.
- 3.1 Network Architecture and Optimization: The network maps aligned face-frame sequences to embeddings using a two-branch backbone followed by bidirectional LSTM modeling.One branch processes RGB information, while the other applies LoG filtering to suppress face content and emphasize artifacts.
- 3.2 Deep Laplacian of Gaussian: The LoG branch uses a multi-scale Laplacian of Gaussian bottleneck to suppress low-level visual content and amplify multi-band frequency artifacts.The resulting LoG feature responses emphasize edges, whereas the RGB branch focuses more on the global face structure.
- 3.3 Loss Function to Isolate Manipulated Faces: The proposed loss compresses natural-face embeddings around a reference center while pushing manipulated-face embeddings beyond an outer radius.It uses inner and outer hyperspheres with 0 < r− < r+ and margin m = r+−r−.
- 3.3 Loss Function to Isolate Manipulated Faces: The two hyperspherical margins prevent the loss from merely sending manipulated samples arbitrarily far away without compressing natural-face representations.The outer radius is essential because indefinite separation alone could reduce the cost without enforcing compactness of natural faces.
- 3.3 Loss Function to Isolate Manipulated Faces: Compared with binary cross-entropy, the proposed loss produces more compact natural-face distributions and clearer separation from manipulated faces in the feature space.The binary-cross-entropy logits show overlapping spread, whereas center distances under the proposed loss show compression and separation in confusing regions.
4 Experimental Evaluation
Experiments evaluate the method on FF++, Celeb-DF, and DFDC Preview using video- and frame-level measures, with emphasis on low false alarm rates and cross-dataset transfer. Ablations examine the loss, two-branch backbone, recurrent modeling, and feature fusion.
- Benchmarks and metrics: The evaluation covers FaceForensics++, Celeb-DF, and the DFDC Preview Dataset at both video and frame levels.Comparisons account for sequence-level sampling so methods observe comparable data quantities.
- Benchmarks and metrics: Low-false-alarm evaluation reports TAR, pAUC, and tAUC alongside standard AUC.tAUC considers only TAR up to a specified FAR cutoff, excluding higher false alarm rates.
- FaceForensics++ ablations: Ablations compare optimization and loss settings, single versus two-branch designs, recurrent modeling, and branch-fusion choices.The best recurrent configuration uses 128 hidden nodes, concatenated bidirectional outputs, and grouped convolutional filters for branch fusion.
- Celeb-DF: On Celeb-DF, the method achieves a clear improvement over prior methods in cross-dataset frame- and video-level evaluation.The authors report higher AUC than other methods on Celeb-DF while retaining high AUC on FF++ Deepfakes.
- Celeb-DF: Qualitative Celeb-DF errors are associated with poor illumination for natural faces and strong facial hair for manipulated faces.The method also detects a challenging manipulated sequence that could be perceived as real.
5 Conclusions and Future Work
The paper concludes that recurrent modeling over aligned-face sequences, combined with a two-branch backbone and isolating loss, provides competitive video-based deepfake detection. It identifies low-false-alarm performance as the main practical gap and proposes augmentation, additional natural faces, and explainability as future directions.
- Conclusions: The proposed method processes aligned-face sequences recurrently with a two-branch backbone and a loss designed to isolate manipulated sequences.The authors report performance that outperforms or matches the state of the art.
- Limitations: Low-false-alarm performance remains a significant area for improvement in practical web-scale applications.The authors specifically state that the bar remains high for very strong results at very low false alarm rates.
- Future work: Future work includes measuring data augmentation and additional external natural faces, followed by unsupervised explainability for face manipulations.The explainability mechanism is planned without pixel-wise supervision.