Source-linked AI summary
Face Forensics in the Wild
Tianfei Zhou, Wenguan Wang, Zhiyuan Liang, Jianbing Shen
TL;DR
Face-forgery detection remains inadequate for multi-person videos because existing benchmarks mostly represent single-person or simple scenes. The paper introduces FFIW10K and a multiple-instance-learning attention model trained with video-level labels, reporting improved classification and localization performance on multi-person forgery data. The study also identifies limitations tied to face-swapping algorithms and difficult slow-illumination, stable-motion scenes.
Problem
Existing datasets provide insufficient representation of real-world multi-person videos, where only some faces may be manipulated.
Method
The paper constructs FFIW10K with automated Q-Net quality control and proposes a discriminative attention model using multiple instance learning, temporal aggregation, and sparse face selection.
Results
The proposed model outperforms representative approaches on FFIW10K for forgery classification and localization and generalizes to existing benchmarks.
Takeaways & Limitations
FFIW10K provides a multi-person benchmark for studying face forgery detection under video-level supervision.
Takeaways & Limitations
The dataset’s difficulty is limited to the adopted face-swapping algorithms, while the model struggles with slow illumination changes and stable motions.
Abstract
from arXiv · showhide
On existing public benchmarks, face forgery detection techniques have achieved great success. However, when used in multi-person videos, which often contain many people active in the scene with only a small subset having been manipulated, their performance remains far from being satisfactory. To take face forgery detection to a new level, we construct a novel large-scale dataset, called FFIW-10K, which comprises 10,000 high-quality forgery videos, with an average of three human faces in each frame. The manipulation procedure is fully automatic, controlled by a domain-adversarial quality assessment network, making our dataset highly scalable with low human cost. In addition, we propose a novel algorithm to tackle the task of multi-person face forgery detection. Supervised by only video-level label, the algorithm explores multiple instance learning and learns to automatically attend to tampered faces. Our algorithm outperforms representative approaches for both forgery classification and localization on FFIW-10K, and also shows high generalization ability on existing benchmarks. We hope that our dataset and study will help the community to explore this new field in more depth.
1. Introduction
Existing face-forgery benchmarks poorly represent multi-person videos, where only some faces are manipulated. FFIW10K addresses this gap with a large-scale dataset and a video-level-supervised attention model for classification and localization.
- Accurate face-forgery detection could help mitigate malicious deepfake uses such as face-recognition attacks and fake news.
- Existing datasets favor trimmed, single-person videos, making them insufficient for evaluating forgery detection in real-world multi-person scenes.
- FFIW10K contains 10,000 high-fidelity manipulated videos averaging three faces per frame, with one to fifteen identities represented.The dataset contains 33 hours of video in total, with videos averaging 12 seconds.
- FFIW10K provides face- and video-level annotations for forgery classification and localization, including a video-level-only training setting.
- The proposed discriminative attention model uses multiple instance learning and temporal feature aggregation to attend to manipulated faces using only video-level labels.Its components aggregate multi-scale temporal tracklet features, attend across face tracklets, and impose sparse face selection.
2. Related Work
Earlier face-forensics datasets improved realism and benchmark scale but largely remained limited to simple, one- or two-person scenarios. FFIW10K is designed to provide unconstrained multi-person complexity, scalable quality control, and a benchmark for methods operating in realistic settings.
- Recent datasets use advanced synthesis methods, yet most prior videos still contain only one or two identities and inadequately represent real-world scenes.
- FFIW10K targets unconstrained multi-person detection with real-world complexity, high-fidelity manipulation, and automatic scalable construction.Its quality is managed by the model-agnostic Q-Net quality assessment network.
- Manual filtering makes large-scale face-forensics dataset construction costly and time-consuming, motivating automated quality assessment.
- Current deep learning methods commonly classify trimmed videos and remain prone to failure in real-world multi-person scenarios.
3. FFIW10K Dataset
FFIW10K is constructed from unconstrained multi-person videos and automatic face swapping, then filtered for quality. Its statistics emphasize many faces per frame and the presence of both real and manipulated faces.
- FFIW10K is introduced as a benchmark intended to expose conditions under which current face-forgery algorithms fail.
- The source collection contains 4,000 high-resolution YouTube videos gathered through diverse multilingual keyword searches by 10 people.Videos were required to be at least 480p.
- Face swapping selects source and target videos, extracts face tracklets, and chooses long-duration, high-quality tracks for manipulation.
- 10,000 fake videos with high quality scores are selected after low-scoring synthetic faces are discarded.The quality filtering is performed automatically by the quality assessment network.
- FFIW10K contains 1–15 faces per frame, averaging 3.15, and includes videos containing both real and fake faces.
- The dataset provides 3.2M real-face and 1.1M fake-face annotations covering 3.6K persons, alongside video-level labels.
4. Domain-Adversarial Quality Control
Q-Net is trained to assess swapped-face quality while learning representations that remain invariant across generative-model domains. After training, it automates quality control by scoring manipulated face tracklets and retaining those above a quality threshold.
- Q-Net training: Q-Net learns to predict synthesized-face quality from pseudo scores derived from generative-model training progress.The pseudo score is si = 0.9×n/N, where n is the iteration number and N is the maximum iteration.
- Q-Net training: Direct training could overfit artifacts specific to individual generative models and generalize poorly under domain shift.The training domains are StyleGAN, StyleGAN2, and PGGAN, while testing targets FFIW10K.
- Q-Net training: Domain-adversarial regularization trains Q-Net to predict both quality scores and domain labels while encouraging domain-invariant feature representations.The objective combines an l1 quality loss with a negatively weighted cross-entropy domain loss.
- Q-Net-based quality control: After training, Q-Net scores each swapped face, and manipulated face tracklets are preserved only when their average quality score exceeds a threshold.This quality-control procedure automates selection during FFIW10K construction.
5. Face Forgery Detection Framework
The framework treats each video as a bag of face-tracklet instances and uses multiple instance learning to classify videos while attending to potentially manipulated faces. It builds discriminative tracklet representations from short-, long-, and global-temporal information, then aggregates them with sparse attention for classification and localization.
- 5.1. Discriminative Attention Model: Each video is a bag of face-tracklet instances with an observed video-level label and unobserved instance labels.The model formulates multi-person forgery detection in a multiple-instance-learning setting.
- 5.2. Multi-Temporal-Scale Instance Feature Aggregation: Tracklet representations combine short-term, long-term, and global feature aggregation to capture temporal patterns and produce compact descriptors.Short-term aggregation uses densely connected dilated temporal convolutions, while long-term aggregation models pairwise temporal-feature relations.
- 5.3. Attention-Based Bag Feature Aggregation: The attention-based bag module adaptively weights all tracklet representations to form a video descriptor and identify potentially manipulated faces.Attention weights support video-level classification and face-forgery localization; a tracklet is regarded as fake when its attention exceeds 0.75.
- 5.4. Loss Function: Sparse regularization on the attention vector selects a few likely tampered faces because most videos manipulate only a sparse subset of faces.The training objective combines binary cross-entropy with an l1-norm sparsity term weighted by β.
- Preprocessing and Training Details: Faces are detected and associated across frames into tracks, resized to 224×224, and represented using ResNet-50 features with D = 2048.The whole network is trained end-to-end using these tracklet-level face representations.
6. Experiment
The experiments evaluate face forgery classification, localization, cross-dataset generalization, and model components on FFIW10K. The proposed model performs strongly across these evaluations, including under video-level supervision alone.
- 6. Experiment: Experiments evaluate classification, localization, cross-dataset generalization, and ablations on FFIW10K.Classification uses AUC and ACC, while localization uses mAP.
- 6.1. Face Forgery Classification: The proposed model outperforms all compared methods on FFIW10K classification despite using only video-level labels.It also significantly outperforms S-MIL, which is trained with video-level labels.
- 6.2. Face Forgery Localization: The proposed approach achieves the best localization performance without precise face-level supervision.Video-based methods consistently outperform image-based methods in mAP, and the model isolates tampered faces in complex multi-person scenes.
- 6.3. Cross-Dataset Evaluation and Generalization: Training on FFIW10K produces stronger cross-dataset performance than training on FF++, including on DFDC Preview and Celeb-DF.The authors interpret these results as evidence that FFIW10K has low data bias.
- 6.4. Model Ablations: Removing short-term aggregation reduces AUC from 70.9% to 69.7% and mAP from 30.8% to 29.6%.Discarding the long-term aggregation module shows a similar trend.
- 6.4. Model Ablations: The attention-based bag aggregation improves classification over max- and average-pooling baselines by highlighting likely discriminative instances.The sparsity constraint also improves localization mAP from 28.5% to 30.8%.
7. Limitation and Discussion
The dataset is limited to the adopted face-swapping algorithms, while the model struggles with slowly changing illumination and stable motion. The discussion therefore identifies broader forgery types as an open scope.
- 7. Limitation and Discussion: FFIW10K covers only the adopted face-swapping algorithms, and the model faces difficulties under slow illumination changes and stable motion.The authors suggest extending forgery research to body movements, facial expressions, talking heads, and controllable camera conditions.