Source-linked AI summary
Recurrent Convolutional Strategies for Face Manipulation Detection in Videos
Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, Prem Natarajan
TL;DR
The paper addresses the limited attention given to detecting manipulated faces in videos despite the growing spread of realistic synthetic content. It combines recurrent convolutional modeling with face preprocessing, evaluates the approach across video manipulation benchmarks, and reports improved state-of-the-art performance.
Problem
The paper addresses the limited study of detecting tampered faces in videos while exploiting temporal information, amid increasing realistic synthetic misinformation.
Method
The paper combines recurrent convolutional models with face cropping and alignment to detect temporal discrepancies in manipulated video faces.
Results
The experiments evaluate Deepfake, Face2Face, and FaceSwap detection on FaceForensics++, with the best configuration using landmark alignment, bidirectional recurrence, and DenseNet.
Takeaways & Limitations
The paper concludes that combining recurrent-convolutional modeling with face alignment improves video face-manipulation detection over the state of the art.
Abstract
from arXiv · showhide
The spread of misinformation through synthetically generated yet realistic images and videos has become a significant problem, calling for robust manipulation detection methods. Despite the predominant effort of detecting face manipulation in still images, less attention has been paid to the identification of tampered faces in videos by taking advantage of the temporal information present in the stream. Recurrent convolutional models are a class of deep learning models which have proven effective at exploiting the temporal information from image streams across domains. We thereby distill the best strategy for combining variations in these models along with domain specific face preprocessing techniques through extensive experimentation to obtain state-of-the-art performance on publicly available video-based facial manipulation benchmarks. Specifically, we attempt to detect Deepfake, Face2Face and FaceSwap tampered faces in video streams. Evaluation is performed on the recently introduced FaceForensics++ dataset, improving the previous state-of-the-art by up to 4.55% in accuracy.
1. Introduction
The paper frames video face manipulation as a growing misinformation concern and motivates detectors that exploit temporal information absent from still-image analysis. It proposes recurrent convolutional modeling with face preprocessing to detect abnormal faces in manipulated video streams.
- Facial manipulation is presented as a way to disseminate false information and libel prominent individuals.
- Video manipulation detection can exploit temporal coherence, which provides features beyond those available in still images.
- Face-swapping and face-reenactment tools have become readily available through machine-learning-assisted systems.
- Videos may appear more believable than still images because they show activities in progress and require coherent manipulation across frames.
- The proposed approach leverages temporal artifacts with recurrent convolutional models and explores face alignment to improve manipulation detection accuracy.
2. Related Work
Related work covers temporal video models, emerging video-based face-manipulation benchmarks, and existing detection methods. The paper positions recurrent convolutional modeling, face alignment, bidirectional recurrence, and multi-manipulation evaluation as its distinguishing choices.
- Video processing with deep models: Video-processing approaches include two-stream networks, recurrent convolutional layers, and 3D convolutions for learning temporal or spatio-temporal information.
- Video processing with deep models: The paper selects a CNN encoder with a recurrent model to capture temporal anomalies in face appearance while avoiding the filter growth of 3D convolutions.
- Face manipulation benchmarks: FaceForensics++ extends FaceForensics with Deepfake and FaceSwap manipulations and provides 1,000 YouTube videos split into training, validation, and test sets.
- Face Manipulation Detection: Existing detection work includes XceptionNet and MesoNet baselines, while the literature on video face-manipulation detection remains relatively sparse.
- Novelty: Compared with prior work, this paper studies face alignment, bidirectional rather than unidirectional recurrence, and multiple manipulation types instead of only deepfakes.
3. Method
The method preprocesses aligned face sequences and applies recurrent-convolutional detection to exploit temporal discrepancies in manipulated video frames.
- Face preprocessing: The pipeline first crops and aligns faces from video frames before detecting manipulation in the preprocessed facial regions.Landmark-based alignment compensates for scale, in-plane rotation, and translation using a similarity transformation.
- Video-based Face Manipulation Detection: The recurrent-convolutional model processes a sequence of tightly aligned face crops, or face “tubelet,” from successive video frames.The input is a sequence of spatio-temporal face crops across video frames.
- Video-based Face Manipulation Detection: Temporal discrepancies are exploited because frame-by-frame manipulations can produce inconsistent low-level facial features across frames.The model uses temporal evidence rather than relying only on spatial artefacts in individual images.
- Model architecture: The backbone is trained for binary real-versus-synthetic classification, then extended with recurrent layers and trained end-to-end.ResNet and DenseNet are explored as CNN backbones, with GRU-based recurrent processing.
- Model architecture: The experiments compare recurrence on final CNN features with multi-recurrent processing at multiple CNN hierarchy levels.The multi-recurrent design separately processes micro-, meso-, and macroscopic features.
4. Experiments
Experiments test backbone choice, alignment, temporal input length, recurrence direction, and architectural variations on Deepfakes, Face2Face, and FaceSwap detection.
- Ablations: The evaluation compares DenseNet and ResNet, face alignment, five-frame versus single-frame input, and unidirectional versus bidirectional recurrence.These experiments use a single recurrent network on top of final CNN features.
- Ablations: DenseNet consistently outperforms ResNet, while face alignment and image sequences improve performance over their respective alternatives.The reported comparisons cover all three manipulation types.
- Ablations: Bidirectional recurrence is superior to unidirectional recurrence for the five-frame input experiments.The comparison is part of the temporal ablation study using a single recurrent network.
- Model selection: The best Table 1 configuration is a five-frame bidirectional DenseNet with alignment, which serves as the base model for Table 2 variations.Table 1 reports accuracy across manipulation types against the FF++ baseline.
- Model selection: Spatial-transformer alignment and multi-recurrent architectures both fail to improve performance and exhibit a decline relative to the base model.The authors associate the spatial-transformer drop with unstable training and changing affine parameters affecting the following CNN.
5. Conclusion
The paper concludes that recurrent convolution combined with face alignment improves video face-manipulation detection, with landmark alignment and bidirectional DenseNet recurrence performing best.
- Conclusion: Landmark-based face alignment with a bidirectional recurrent DenseNet performs best for face manipulation detection in videos.The conclusion frames this combination as improving upon the state of the art.
- Conclusion: The study also explores alternative alignment strategies and ways to combine CNN features through recurrence.These alternatives include the broader design choices evaluated in the paper.