Source-linked AI summary
Video Face Manipulation Detection Through Ensemble of CNNs
Nicolò Bonettini, Edoardo Daniele Cannas, Sara Mandelli, Luca Bondi, Paolo Bestagini, Stefano Tubaro
TL;DR
The paper addresses detection of realistic facial manipulations in videos, which pose serious social risks and challenge traditional forensic methods. It ensembles EfficientNetB4-based CNNs using attention and siamese training, achieving promising results across two public datasets and reaching the DFDC public-test top 3%.
Problem
Realistic facial manipulations are difficult to detect because techniques vary, affect small regions, and leave subtle forensic traces after common video processing.
Method
The method ensembles EfficientNetB4-based CNNs that incorporate an attention mechanism and triplet siamese training.
Results
The four-model ensemble reached the top 3% on the DFDC public-test leaderboard and improved detection over the baseline on both evaluated datasets.
Takeaways & Limitations
Evaluation on two public datasets containing almost 120 000 videos identifies the proposed ensemble strategy as a valid solution for facial manipulation detection.
Abstract
from arXiv · showhide
In the last few years, several techniques for facial manipulation in videos have been successfully developed and made available to the masses (i.e., FaceSwap, deepfake, etc.). These methods enable anyone to easily edit faces in video sequences with incredibly realistic results and a very little effort. Despite the usefulness of these tools in many fields, if used maliciously, they can have a significantly bad impact on society (e.g., fake news spreading, cyber bullying through fake revenge porn). The ability of objectively detecting whether a face has been manipulated in a video sequence is then a task of utmost importance. In this paper, we tackle the problem of face manipulation detection in video sequences targeting modern facial manipulation techniques. In particular, we study the ensembling of different trained Convolutional Neural Network (CNN) models. In the proposed solution, different models are obtained starting from a base network (i.e., EfficientNetB4) making use of two different concepts: (i) attention layers; (ii) siamese training. We show that combining these networks leads to promising face manipulation detection results on two publicly available datasets with more than 119000 videos.
I. INTRODUCTION
Modern facial manipulation tools make realistic video forgeries accessible, creating serious social risks and challenging traditional forensic detection. The paper addresses this problem with an ensemble of CNN models designed for modern manipulation techniques and practical deployment constraints.
- Accessible facial manipulation tools enable realistic video forgeries with malicious applications including fake news and revenge porn.
- Traditional forensic detectors rely on editing footprints that can be subtle after compression, multiple edits, downsampling, or realistic face-specific manipulation.
- Modern facial forgeries are difficult to detect because multiple techniques exist, affect small regions, and are often resized and recompressed by social platforms.
- The paper studies ensembles based on EfficientNetB4, an attention-augmented variant, and two training strategies including siamese training.
- The solution targets analysis of 4 000 videos in under 9 hours on one NVIDIA P100 GPU while keeping trained models below 1GB.
- Experiments use disjoint FF++ and DFDC datasets, and results report that attention and siamese strategies help the ensemble outperform the FF++ baseline on both datasets.
II. RELATED WORK
Prior work detects facial manipulation through frame-based CNNs, temporal modeling, processing traces, semantic inconsistencies, and localization. This paper instead emphasizes a lightweight multi-dataset ensemble under DFDC competition constraints.
- CNN-based methods perform frame-by-frame fake-face analysis, with XceptionNet reported to outperform the shallower MesoNet.
- LSTM-based methods model temporal evolution by combining frame-level features with recurrent mechanisms.
- Other detectors exploit warping traces produced when donor faces are adapted to host videos.
- Semantic approaches analyze head pose, lighting inconsistencies, or eye blinking, while newer methods provide localization through segmentation or attention.
- The proposed work focuses on network ensembles that operate across multiple datasets while remaining sufficiently lightweight for DFDC rules.
III. PROPOSED METHOD
The proposed method ensembles CNN classifiers to capture complementary high-level facial information. It uses EfficientNet as the base architecture and varies both attention modeling and training strategy.
- The method investigates whether differently trained CNN classifiers can capture complementary high-level semantic information for ensemble prediction.
- EfficientNet provides the starting architecture because its accuracy and efficiency help satisfy the DFDC hardware and time constraints.
- Two design paths modify the EfficientNet-based models for ensembling, with further details covering attention and network training strategies.
A. EfficientNet and attention mechanism
EfficientNetB4 processes cropped face images, while EfficientNetB4Att adds an attention block that weights informative feature-map regions. The attention map also provides a human-interpretable view of the network’s focus.
- EfficientNet baseline: EfficientNetB4 is selected as the baseline for its trade-off among model size, runtime, and classification performance.It has 19 million parameters and 4.2 billion FLOPS, with 83.8% ImageNet top-1 accuracy.
- EfficientNet baseline: The network takes a squared color face image and produces a 1792-element feature vector before final face classification.
- Attention mechanism: Embedding the attention block into EfficientNetB4 defines the proposed EfficientNetB4Att architecture.
- Attention mechanism: The attention variant selects 28×28×56 feature maps, applies a 1×1 convolution and Sigmoid, then multiplies the resulting attention map into those maps.
- Attention mechanism: The attention mechanism focuses processing on relevant feature-map regions and maps its learned emphasis back onto the input face.
B. Network training
The paper trains CNN detectors either end-to-end or with siamese learning, using classification or triplet-margin objectives to separate real and fake faces.
- Siamese training: Siamese training learns an encoding that separates real and fake faces by encouraging within-class similarity and between-class separation.The resulting feature extractor is finalized by fine-tuning a simple classification layer using the end-to-end approach.
- End-to-end training: The end-to-end strategy updates network weights using LogLoss on predicted face scores and binary real/fake labels.The Sigmoid function converts scores within the loss formulation; pristine faces use label 0 and fake faces label 1.
- Siamese training: The siamese objective is a triplet-margin loss defined from distances between anchor-positive and anchor-negative face encodings.The loss uses a strictly positive margin and compares the anchor with same-class and different-class samples.
- Siamese training: Triplets use a real anchor, another real positive sample, and a fake negative sample.These roles make the positive pair same-class and the negative pair different-class for the stated training configuration.
IV. EXPERIMENTS
Experiments evaluate the proposed facial-manipulation detector on FF++ and DFDC, covering established manipulation methods and a large challenge dataset.
- Datasets: The evaluation uses two datasets: FF++ and DFDC.The paper reports results on the test sets after dataset-specific train, validation, and test splits.
- FF++: FF++ contains more than 1.8 million images from 4000 manipulated videos generated with Face2Face, FaceSwap, DeepFakes, and NeuralTextures.The source material comprises 1000 pristine YouTube videos, each processed by the four manipulation methods.
- DFDC: DFDC contains more than 119 000 real and fake video sequences created for a challenge, with diversity in gender, skintone, age, and backgrounds.Its fake videos derive from real videos using different DeepFake techniques, while the precise generation algorithms are unknown.
B. Networks
The experimental setup compares XceptionNet with EfficientNetB4 variants, trains them under two paradigms, and limits video processing to 32 frames with face-focused preprocessing.
- Networks: The network comparison includes XceptionNet, EfficientNetB4, and EfficientNetB4Att.XceptionNet is the yardstick, EfficientNetB4 is selected for accuracy and efficiency, and EfficientNetB4Att is designed to emphasize relevant face regions.
- Networks: Models are trained and tested separately on both datasets using end-to-end and siamese strategies for the EfficientNet variants.The FF++ evaluation uses videos generated with constant rate quantization equal to 23.
- Setup: The experiments limit each video to 32 analyzed frames because very small frame counts encourage overfitting while more frames do not improve performance justifiably.The paper notes that 32 frames help prevent overfitting, although 15 frames do not improve the minimum validation loss relative to 32.
- Setup: Face crops are extracted from each frame with BlazeFace to reduce the data processed by the networks.The preprocessing focuses analysis on the subjects’ faces rather than the full frames.
- Setup: Training and validation use augmentation including downscaling, flipping, color changes, noise, and JPEG compression.The implementation uses Albumentation for augmentation and PyTorch as the deep-learning framework.
- Setup: End-to-end training processes batches of 32 faces and runs for at most 20k iterations or until validation loss plateaus.Validation occurs every 500 iterations on 6000 validation samples.
- Setup: Siamese training uses batches of 12 triplets while retaining the end-to-end strategy’s iteration count, validation routine, and learning-rate scheduling.The triplets comprise six real-fake-fake and six fake-fake-real combinations.
V. RESULTS
The results section introduces the collection of findings from the experimental campaign but provides no substantive result in the supplied passage.
- This section collects all results obtained during the experimental campaign.
A. EfficientNetB4Att explainability
The attention mechanism highlights detailed facial regions, while siamese-trained features separate real and fake samples in a reduced feature space.
- A. EfficientNetB4Att explainability: Attention maps highlight detailed facial regions such as the eyes, mouth, nose, and ears, while largely ignoring flat regions.The map is produced from the attention block and up-scaled onto the input face.
- A. EfficientNetB4Att explainability: The attention network tends to select the eyes, mouth, and nose in analyzed FF++ faces.
- A. EfficientNetB4Att explainability: Siamese-trained EfficientNetB4Att features cluster frames from the same videos into small sub-regions and separate real from fake samples.The t-SNE projection places real samples in the top region and fake samples in the bottom region.
C. Architecture independence
The study motivates ensembling by showing that the networks produce partly independent scores, and reports improved detection metrics over the baseline on both datasets.
- C. Architecture independence: Different networks produce slightly different frame-level scores, supporting their combination in an ensemble.The score clouds do not perfectly align, unlike perfectly correlated models.
- C. Architecture independence: The evaluated systems include XceptionNet as baseline and four proposed EfficientNet models, with ensemble face scores computed by averaging model outputs.The proposed models are EfficientNetB4, EfficientNetB4Att, EfficientNetB4ST, and EfficientNetB4AttST.
- C. Architecture independence: Ensembling generally improves performance, with the best top-3 results consistently reached by combinations of two or more networks.The reported measures are AUC for detection accuracy and LogLoss for detection quality.
- C. Architecture independence: On both FF++ and DFDC, ensemble LogLoss and AUC are better than the baseline.
- C. Architecture independence: The four-model ensemble placed the team in the top 3% of the public DFDC leaderboard.The private-test leaderboard had not yet been disclosed.
VI. CONCLUSIONS
The paper targets facial manipulation detection with an EfficientNet-based ensemble combining attention and triplet siamese training. Across two large public datasets, the ensemble is reported as a valid solution, while temporal information remains future work.
- VI. CONCLUSIONS: The paper addresses facial manipulation detection in video sequences involving both classical computer graphics and deep-learning-generated fake videos.
- VI. CONCLUSIONS: The method combines EfficientNet-based models trained with attention mechanisms and triplet siamese training.The attention mechanism also provides a human-comprehensible inference of informative regions.
- VI. CONCLUSIONS: Results over two publicly available datasets containing almost 120 000 videos identify the ensemble strategy as a valid facial manipulation detection solution.
- VI. CONCLUSIONS: Future work will embed temporal information, potentially using intelligent voting schemes over multiple frames to increase accuracy.