Source-linked AI summary
Sharp Multiple Instance Learning for DeepFake Video Detection
Xiaodan Li, Yining Lang, Yuefeng Chen, Xiaofeng Mao, Yuan He, Shuhui Wang, Hui Xue, Quan Lu
TL;DR
The paper addresses DeepFake videos in which only some faces are manipulated while supervision is available only at the video level. It proposes S-MIL with spatial-temporal instances and evaluates it using FFPMS and existing benchmarks, reporting superior performance across partially attacked video and single-frame settings.
Problem
Partial face attacks leave some faces in fake videos unmanipulated, while practical DeepFake detection often provides only video-level labels.
Method
S-MIL directly maps weighted instance embeddings to bag predictions and uses spatial-temporal encoded instances to model intra-frame and inter-frame inconsistency.
Results
S-MIL is reported superior to other methods on partially attacked FFPMS videos and DFDC, and achieves state-of-the-art performance when adapted to single-frame FF++ detection.
Takeaways & Limitations
The approach supports DeepFake detection when manipulation is partial and supervision is available at the video level, while also extending to traditional single-frame detection.
Takeaways & Limitations
Traditional MIL can suffer gradient vanishing when one instance is predicted positive, limiting its use for judging whether individual frames are fake.
Abstract
from arXiv · showhide
With the rapid development of facial manipulation techniques, face forgery has received considerable attention in multimedia and computer vision community due to security concerns. Existing methods are mostly designed for single-frame detection trained with precise image-level labels or for video-level prediction by only modeling the inter-frame inconsistency, leaving potential high risks for DeepFake attackers. In this paper, we introduce a new problem of partial face attack in DeepFake video, where only video-level labels are provided but not all the faces in the fake videos are manipulated. We address this problem by multiple instance learning framework, treating faces and input video as instances and bag respectively. A sharp MIL (S-MIL) is proposed which builds direct mapping from instance embeddings to bag prediction, rather than from instance embeddings to instance prediction and then to bag prediction in traditional MIL. Theoretical analysis proves that the gradient vanishing in traditional MIL is relieved in S-MIL. To generate instances that can accurately incorporate the partially manipulated faces, spatial-temporal encoded instance is designed to fully model the intra-frame and inter-frame inconsistency, which further helps to promote the detection performance. We also construct a new dataset FFPMS for partially attacked DeepFake video detection, which can benefit the evaluation of different methods at both frame and video levels. Experiments on FFPMS and the widely used DFDC dataset verify that S-MIL is superior to other counterparts for partially attacked DeepFake video detection. In addition, S-MIL can also be adapted to traditional DeepFake image detection tasks and achieve state-of-the-art performance on single-frame datasets.
1 INTRODUCTION
The paper targets partially attacked DeepFake videos, where video-level labels are available but only some faces may be manipulated. It proposes S-MIL, spatial-temporal instances, and FFPMS to address and evaluate this setting.
- Partial faces attack occurs when real and fake faces coexist in one frame or one person is only partly manipulated within a video.
- Existing frame-based methods may perform poorly because they treat video labels as frame labels and require many labeled frames.
- S-MIL directly maps weighted sums of instance embeddings to bag predictions, allowing bag-level loss to propagate directly to instance embeddings.
- The method theoretically alleviates gradient vanishing, while spatial-temporal instances model intra-frame and inter-frame inconsistency from independent frame attacks.
- FFPMS provides frame-level and video-level annotations, and experiments report S-MIL superiority on partially attacked videos, DFDC, and adaptable single-frame detection.
2 RELATED WORK
Related work covers image- and video-level DeepFake detection and multiple-instance learning. It highlights limitations of frame-score aggregation and describes instance-space, embedded-space, and attention-based MIL approaches.
- DeepFake detection methods are categorized into image-level and video-level detection, with early approaches using handcrafted features.
- Deep networks were introduced as DeepFake faces became more realistic, learning discriminative features or manipulation traces.
- Frame-based video detection commonly samples frames and aggregates maximum or average scores, but these operations can produce false alarms or false negatives.
- MIL learns bag-label predictors from groups of bags containing many instances without requiring individual instance labels.
- MIL algorithms include instance-space and embedded-space paradigms, while attention mechanisms focus learning on key instances.
3 OUR APPROACH
The approach detects and encodes face sequences as spatial-temporal instances, organizes them into multiple bags, and applies S-MIL for video prediction. S-MIL directly maps instance embeddings to bag predictions, while spatial-temporal encoding captures temporal face inconsistency.
- Algorithm Overview: The pipeline detects faces, extracts CNN feature maps, encodes face sequences into spatial-temporal instances, and applies S-MIL to produce video-level fake scores.Multiple spatial-temporal bags represent each input video before final prediction.
- Problem Formulation: Traditional frame-level supervision is inaccurate for partial attacks because assigning video labels to every frame introduces label noise.Maximum and average frame-score aggregation can respectively produce false alarms and false negatives when manipulated faces are only partially present.
- Problem Formulation: MIL treats faces as instances and videos as bags when only video-level annotations are available.A positive bag contains at least one positive instance, whereas a negative bag contains only negative instances.
- Sharp Multi-Instance Learning: S-MIL applies a sigmoid to a weighted sum of instance embeddings, directly connecting bag prediction with embedded-space representations.The weighting mechanism emphasizes informative instances and can make positive instances more influential in positive bags.
- Sharp Multi-Instance Learning: S-MIL is designed to alleviate gradient vanishing because its loss has a sharper gradient surface than traditional MIL.The sharper surface yields a smaller region with vanished gradients during backpropagation from bag predictions to instance embeddings.
4 EXPERIMENTS
Experiments evaluate S-MIL on partially and fully attacked DeepFake datasets, including FFPMS, DFDC, Celeb, and FF++, using video- and frame-level comparisons, ablations, and visualization. S-MIL generally outperforms frame- and video-based counterparts, while spatial-temporal encoding and additional sampled frames further improve performance.
- DeepFake Video Detection Datasets: FFPMS combines 14,000 frames from FF++ testing videos with several fake frames replaced by corresponding real frames, providing frame- and video-level labels for partial-attack evaluation.The number of replaced fake frames ranges from 1 to 19, and the dataset includes DF, F2F, FS, NT, and original videos.
- Evaluation Setup: Experiments compare S-MIL with XceptionNet, D-FWA, LSTM, and I3D on DFDC, Celeb, and FFPMS using the average accuracy of fake and real testing videos.Because FF++ contains few unmodified faces in fake training videos, 25% of fake training frames are replaced with corresponding real frames for S-MIL training.
- Evaluation on Partially Attacked Datasets: S-MIL outperforms frame-based and video-based methods in most cases, and S-MIL-T improves further when spatial-temporal encoding is added.On FFPMS, S-MIL-T remains effective when only 10% of input faces are fake and significantly outperforms XceptionNet and LSTM.
- Evaluation on Fully Attacked Datasets: On FF++, S-MIL achieves state-of-the-art performance with only 10% training data and outperforms traditional MIL under both frame-level and video-level settings.The component analysis attributes gains to alleviating gradient vanishing, weighting important instances, and using spatial-temporal encoding.
- Ablation Study: On DFDC, S-MIL-T accuracy increases as more testing frames are sampled, whereas XceptionNet and LSTM accuracies change little.This comparison evaluates how the number of sampled frames affects video-based inference under noisy conditions.
5 CONCLUSION
The paper addresses partially attacked DeepFake videos, where only some faces are manipulated, using MIL and the proposed S-MIL framework. It introduces spatial-temporal instances, the FFPMS dataset, and reports superiority over counterparts on DeepFake detection benchmarks.
- S-MIL directly connects bag-label prediction to instance embeddings, alleviating gradient vanishing in traditional MIL.The paper also provides theoretical analysis of S-MIL's sharp loss property.
- Spatial-temporal instances model inconsistency between faces in nearby frames to further improve detection performance.
- FFPMS provides a dataset for evaluating video-level DeepFake detection under partial face manipulation.
- Experiments on FFPMS and DFDC verify that S-MIL is superior to other counterparts for DeepFake video detection.
- S-MIL also adapts to traditional single-frame detection and achieves state-of-the-art performance on single-frame datasets.
6 APPENDIX
The appendix analyzes MIL formulations and their gradients, then visualizes how S-MIL reduces gradient-vanishing regions. It also reports implementation details for face detection and comparative experiments.
- Appendix: MIL represents multiple instance predictions as a bag prediction trained with cross-entropy loss.The formulation merges instance outputs into a final bag probability.
- Appendix: Traditional MIL can produce vanishing gradients when an instance probability approaches one, limiting optimization of other positive instances.
- Appendix: S-MIL retains a nonzero gradient in the analyzed case, whereas traditional MIL's gradient vanishes.
- 6.2 Visualization: For two instances, the gradient surface is evaluated over p_1 and p_2 to compare traditional MIL with sharp MIL.
- 6.2 Visualization: The visualization shows a smaller gradient-vanishing area for sharp MIL than for traditional MIL.
- Implementation: Face extraction uses dlib for FF++ and MTCNN for DFDC and Celeb, without face recognition or tracking.