Source-linked AI summary
MesoNet: a Compact Facial Video Forgery Detection Network
Darius Afchar, Vincent Nozick, Junichi Yamagishi, Isao Echizen
TL;DR
Compressed video makes face-forgery detection difficult, especially for Deepfake and Face2Face manipulations. The paper proposes compact mesoscopic deep networks and evaluates them on a new online-video Deepfake dataset and FaceForensics, reporting average detection rates of 98% and 95%, respectively.
Problem
Video-forgery detection remains difficult because compression strongly degrades frames, limiting the direct use of image-forensics methods.
Method
The paper proposes two compact deep-learning networks with few layers that analyze mesoscopic image properties for detecting Deepfake and Face2Face forgeries.
Results
Average detection rates were 98% for Deepfake videos and 95% for Face2Face videos under real internet-diffusion conditions.
Takeaways & Limitations
The method provides efficient, low-computational-cost detection architectures and a dataset devoted to Deepfake videos.
Takeaways & Limitations
Deepfake auto-encoders can poorly reconstruct fine details, producing blurry faces, and the dataset construction relies on publicly available internet videos rather than newly generated training data.
Abstract
from arXiv · showhide
This paper presents a method to automatically and efficiently detect face tampering in videos, and particularly focuses on two recent techniques used to generate hyper-realistic forged videos: Deepfake and Face2Face. Traditional image forensics techniques are usually not well suited to videos due to the compression that strongly degrades the data. Thus, this paper follows a deep learning approach and presents two networks, both with a low number of layers to focus on the mesoscopic properties of images. We evaluate those fast networks on both an existing dataset and a dataset we have constituted from online videos. The tests demonstrate a very successful detection rate with more than 98% for Deepfake and 95% for Face2Face.
1. Introduction
The paper targets detection of Deepfake and Face2Face video forgeries, motivated by the difficulty of extending image-forensics methods to compressed video. It introduces these forgery processes and describes their principal mechanisms and limitations.
- Video edition remains difficult to detect because compression strongly degrades frames, limiting direct extension of image-forensics methods.
- 1.1. Deepfake: Deepfake replaces one person’s face with another’s by training two auto-encoders with a shared encoder and separate decoders.An image of person A is encoded through the shared encoder and decoded with person B’s decoder.
- 1.1. Deepfake: Deepfake processing extracts and aligns faces frame by frame, generates replacement faces preserving illumination and expression, and merges them back into video.
- 1.1. Deepfake: Auto-encoders often reconstruct fine details poorly, producing blurry faces, while larger encoding spaces can reduce realism by passing morphological data to the decoder.
- 1.2. Face2Face: Face2Face performs real-time, photorealistic, markerless facial reenactment by fitting a morphed target-face blendshape to the source actor’s expression.It requires a short prerecorded target-person training sequence and tracks source and target expressions at runtime.
2. Proposed method
The paper uses compact mesoscopic deep networks to detect compressed face forgeries, balancing low representation and computational cost with classification performance. It proposes Meso-4 and MesoInception-4, which share a common compact design strategy but differ in their convolutional modules.
- 2. Proposed method: The method analyzes mesoscopic image properties because compression degrades microscopic noise and human observers struggle with higher-level facial forgery cues.
- 2. Proposed method: The proposed architectures use few layers and few parameters while achieving the best classification scores among the tested networks.
- 2.1. Meso-4: Meso-4 combines four successive convolution-and-pooling layers with a one-hidden-layer dense network.ReLU, Batch Normalization, and Dropout are used for nonlinearity and regularization.
- 2.1. Meso-4: Meso-4 contains 27,977 trainable parameters.
- 2.2. MesoInception-4: MesoInception-4 replaces the first two Meso-4 convolutional layers with inception modules using multiple kernel shapes and 3 × 3 dilated convolutions.The design adds 1×1 convolutions for dimension reduction and a parallel 1×1 skip connection.
- 2.2. MesoInception-4: Replacing more than two layers with inception modules did not improve classification, and the selected MesoInception-4 configuration has 28,615 trainable parameters.
3. Experiments
The experiments evaluate the two architectures for detecting the studied digital forgeries and examine robustness to video compression in online-video conditions.
- The evaluation implements both proposed architectures for detecting Deepfake and Face2Face forgeries and examines their robustness to video compression.
3.1. Datasets
The paper constructs a Deepfake dataset from online videos because no suitable dataset was available, and supplements it with real faces for balanced classification. It also evaluates Face2Face detection using the existing FaceForensics dataset under multiple compression levels.
- 3.1.1 Deepfake dataset: The authors created a Deepfake video dataset because, to their knowledge, no dataset gathered videos generated by Deepfake.
- 3.1.1 Deepfake dataset: The Deepfake dataset contains 175 forged-video rushes downloaded from different platforms, spanning two seconds to three minutes at minimum 854 × 480 resolution.
- 3.1.1 Deepfake dataset: Faces were extracted with Viola-Jones, aligned using a trained landmark detector, and sampled in proportion to camera-angle and illumination changes.
- 3.1.1 Deepfake dataset: The dataset was doubled with real face images and manually reviewed, while resolution distributions were matched across classes to reduce classification bias.
- 3.1.2 FaceForensics dataset: FaceForensics supplies over a thousand Face2Face-forged videos with originals and predefined training, validation, and testing splits.
- 3.1.2 FaceForensics dataset: The Face2Face evaluation used lossless, light-compression level 23, and strong-compression level 40 H.264 videos to assess compression robustness.
3.2. Classification Setup
The classifiers minimize expected prediction error and are trained on fixed-size image batches with ADAM, scheduled learning-rate decay, and random input transformations. Dataset tables define class cardinalities and reserve 10% of training data for validation.
- Ten percent of each dataset’s training set is used for model validation, while the remaining data support training and evaluation as specified in Table 2.
- The classification task minimizes expected loss E(f) = E[l(f(X), Y)] over prediction functions mapping inputs to actions.The loss is defined as l(a, y) = 1/2(a − y)^2.
- Both networks are optimized with ADAM using batches of 75 images sized 256 × 256 × 3.The implementation uses default ADAM parameters.
- The learning rate starts at 10^-3, is divided by 10 every 1000 iterations, and decreases to 10^-6.
- Random zoom, rotation, and horizontal transformations are applied to input batches to improve generalization and robustness.
3.3. Image classification results
Frame-level classification is evaluated on Deepfake and Face2Face datasets, with ROC curves and dataset-specific score tables. Deepfake frame scores are around 90%, while Face2Face performance deteriorates under strong compression and differs from reported Xception results.
- Around 90% frame-level classification scores are achieved by both networks on the Deepfake dataset.The authors attribute the ceiling partly to very low-resolution facial images in the dataset.
- Face2Face detection scores deteriorate notably at the strong video compression level.Table 4 reports frame-level classification scores for the FaceForensics dataset.
- 96.1% at compression level 0 and 93.5% at level 23 are obtained when fine-tuning Xception under the FaceForensics configuration.The paper states that these results make interpretation unclear relative to the dataset paper’s reported results.
- Figure 6 plots ROC curves for evaluated classifiers on Deepfake and Face2Face videos compressed at rate 23.
3.4. Image aggregation
Because compression degrades video frames, the method averages frame predictions across a video. This aggregation substantially improves detection, exceeding 98% with MesoInception-4 on Deepfake.
- Video-level prediction is obtained by averaging network predictions over sampled frames from the same video.The paper notes that correlated frames do not theoretically justify a confidence interval or guaranteed score gain.
- A majority of clear, stable frames can outweigh punctual movement blur, face occlusion, and random misprediction during aggregation.
- More than 98% detection is achieved by MesoInception-4 on the Deepfake dataset after image aggregation.On Face2Face, both networks reach the same score, although they misclassify different videos.
3.5. Aggregation on intra-frames
The paper tests aggregation using only intra-frames to reduce interpolation-related compression artifacts, but short videos may provide too few such frames. This variant slightly worsens classification while remaining above single-image scores.
- Aggregation is repeated using only intra-frames, which are not interpolated over time, to test whether fewer compression artifacts improve classification.
- Videos lasting a few seconds may contain as few as three I-frames, limiting the smoothing effect expected from aggregation.
- The I-frame variant slightly worsens classification, although the difference is small.The resulting scores remain higher than single-image classification and may support quick aggregation.
3.6. Intuition behind the network
The networks can be interpreted by visualizing inputs that maximize filter activation or by comparing mean layer outputs for real and forged images. These visualizations indicate that Deepfake detection relies on differences in facial detail, especially around the eyes, nose, and mouth, versus the background.
- Activation visualization: Maximizing a filter’s activation with input regularization reveals the image signal to which that filter responds.The objective is E(x) = f_ij(x) − λ∥x∥_p, where regularization reduces noise in the synthesized input.
- Activation visualization: Positive-weighted Meso4 neurons emphasize detailed eyes, nose, and mouth, whereas negative-weighted neurons emphasize detailed backgrounds and smooth faces.The weight sign indicates whether activation pushes the final classification toward the real or forged class.
- Activation visualization: This activation pattern is consistent with Deepfake faces being blurrier or less detailed than surrounding image regions left untouched.
- Mean-output visualization: Mean layer outputs provide another way to identify image regions that contribute strongly to classification.The approach compares activation differences across batches of real and forged images.
- Mean-output visualization: For MesoInception-4 on Deepfake data, eyes activate strongly in real images, while backgrounds show the highest peaks in deepfake images.The authors again relate this difference to relative image detail after face dimension reduction.
4. Conclusion
The paper concludes that efficient low-cost networks can detect Deepfake and Face2Face video forgeries, while visualization helps explain the networks’ decisions. It identifies the eyes and mouth as especially important for detecting Deepfake-forged faces.
- Conclusion: 98% average detection for Deepfake videos and 95% for Face2Face videos was achieved under real internet diffusion conditions.
- Conclusion: The paper provides two network architectures designed to detect face forgeries efficiently with low computational cost.
- Conclusion: Visualizing network layers and filters helps evaluate the qualities and limitations of the learned solution.
- Conclusion: The eyes and mouth play a paramount role in detecting faces forged with Deepfake.