Source-linked AI summary
Exploring Temporal Coherence for More General Video Face Forgery Detection
Yinglin Zheng, Jianmin Bao, Dong Chen, Ming Zeng, Fang Wen
TL;DR
Face forgery detectors often fail to generalize to unseen manipulations and can depend on spatial artifacts that are vulnerable to perturbations. This paper introduces an end-to-end FTCN plus Temporal Transformer framework that models short- and long-range temporal incoherence, achieving superior robustness and generalization without pre-training or external datasets.
Problem
Existing detectors have limited generalization to unseen forgeries, remain vulnerable to common perturbations, and may rely on spatial artifacts instead of temporal incoherence.
Method
The framework uses FTCN with spatial kernel dimensions set to 1 for short-term temporal cues, followed by a Temporal Transformer for long-term incoherence, trained end-to-end.
Results
Extensive experiments show superior robustness and generalization to unseen forgeries compared with previous methods.
Takeaways & Limitations
Temporal information has significant effects for video face forgery detection, and the framework can locate and visualize temporal incoherence without manual annotations.
Takeaways & Limitations
Prior methods such as LipForensics and Face X-ray have specific dependencies or vulnerabilities: pre-training for the former and hand-crafted data plus perturbation susceptibility for the latter.
Abstract
from arXiv · showhide
Although current face manipulation techniques achieve impressive performance regarding quality and controllability, they are struggling to generate temporal coherent face videos. In this work, we explore to take full advantage of the temporal coherence for video face forgery detection. To achieve this, we propose a novel end-to-end framework, which consists of two major stages. The first stage is a fully temporal convolution network (FTCN). The key insight of FTCN is to reduce the spatial convolution kernel size to 1, while maintaining the temporal convolution kernel size unchanged. We surprisingly find this special design can benefit the model for extracting the temporal features as well as improve the generalization capability. The second stage is a Temporal Transformer network, which aims to explore the long-term temporal coherence. The proposed framework is general and flexible, which can be directly trained from scratch without any pre-training models or external datasets. Extensive experiments show that our framework outperforms existing methods and remains effective when applied to detect new sorts of face forgery videos.
1. Introduction
Face forgery detection must generalize beyond known manipulations and remain robust to video perturbations, while exploiting temporal incoherence that frame-wise generation introduces. The paper proposes FTCN and a Temporal Transformer to detect short- and long-range temporal artifacts without pre-training or manual annotations.
- Known-technique detectors suffer dramatic performance drops on unseen manipulations, while newer methods remain vulnerable to compression, noise, and other perturbations.
- Frame-by-frame face forgery generation introduces flickering and discontinuities because altered faces are generated independently.
- Forged videos contain spatial artifacts and temporal incoherence, but unconstrained spatio-temporal models may rely mainly on the more significant spatial artifacts.
- FTCN sets spatial convolution dimensions to 1 while preserving temporal kernels, restricting spatial artifact processing and encouraging temporal-feature classification.
- A lightweight Temporal Transformer follows FTCN to capture long-range discontinuities, such as wrinkles or moles gradually appearing or disappearing.
- The end-to-end framework can locate and visualize temporal incoherence without manual annotations and operates without pre-training knowledge or hand-crafted datasets.
- Experiments report significantly better generalization to unseen forgeries and robustness to varied video perturbations than previous methods.
2. Related Work
Prior work largely emphasizes spatial artifacts or specialized cues, while visualizations compare how different classifiers represent real and manipulated clips. The related-work evidence frames temporal modeling as a complementary direction for generalization.
- Early image face-forgery detectors primarily use deep CNN binary classifiers to distinguish real and fake images through spatial artifacts.
- The t-SNE visualizations represent each video clip as a dot and compare features from different models on the FF++ test set.
- Face X-ray detects blending boundaries for generalization but remains susceptible to common perturbations, while LipForensics targets unnatural mouth motion with spatio-temporal networks.
3. Methods
The framework combines FTCN, which emphasizes local temporal artifacts by restricting spatial kernels, with a Temporal Transformer that models long-range temporal incoherence. Together, these components produce temporal features for final real-or-fake classification.
- Motivation: Current frame-wise manipulation can create flickering and discontinuities, motivating detection based on temporal incoherence.Subtle appearance changes across independently processed frames can yield temporally incoherent fake videos.
- Motivation: Standard spatio-temporal classifiers may rely on stronger spatial artifacts rather than learning temporal incoherence.The framework addresses this imbalance by restricting spatial processing.
- Fully Temporal Convolution Network: FTCN keeps temporal convolution kernels unchanged while setting spatial kernel sizes to 1, encouraging classification from temporal-related artifacts.The proposed 3D R50-FTCN uses a 3D R50 backbone with this kernel restriction.
- Fully Temporal Convolution Network: Compared with 2D R50 and 3D R50, 3D R50-FTCN produces more mixed fake-data features, indicating more general temporal representations.All classifiers use the same FF++ training set and training and inference settings.
- Temporal Transformer: The Temporal Transformer receives FTCN features as temporal sequences and captures long-range temporal incoherence using a class token and Transformer encoder.An MLP head uses the final class-token representation for prediction.
- Overall Framework: The complete pipeline applies FTCN to extract temporal features, then models long-term incoherence with the Temporal Transformer before final MLP prediction.The two stages are trained end-to-end for video face forgery detection.
4. Experiments
Experiments evaluate generalization across unseen manipulations and datasets, robustness to perturbations, and the contributions of temporal design choices. The method achieves strong cross-manipulation and cross-dataset performance while ablations support its fully temporal representation and Temporal Transformer.
- Generalization to unseen manipulations: The experiments train on three FF++ manipulation methods and test on the remaining method using video-level AUC(%).
- Generalization to unseen manipulations: 99.7% average AUC demonstrates generalization to novel forgeries, exceeding Face X-ray and LipForensics by 4.8% and 2.6%, respectively.The comparison uses four unseen manipulation methods and no pre-training or external training data.
- Generalization to unseen datasets: Training on FF++ and testing on Celeb-DF-v2, DFDC, FaceShifter, and DeeperForensics yields the best performance on every unseen dataset.Results are especially strong on FaceShifter and DeeperForensics, while CDF and DFDC produce relatively low scores across methods.
- Robustness to unseen perturbations: Across five intensity levels of block-wise distortion, saturation change, Gaussian blur, and resize, the method achieves better average robustness to unseen perturbations.The evaluation reports video-level AUC(%).
- Ablation study: Reducing spatial convolution improves generalization: 3D R50-FTCN outperforms comparable spatial variants, and pixel shuffling still leaves reasonable performance.The ablation attributes the gain mainly to the fully temporal design and suggests reliance on temporal-related information.
- Ablation study: Increasing clip size improves performance, while the authors identify clip size 32 as a trade-off between performance and computation cost.The gain from clip sizes 32 to 64 is tiny, with possible limits from temporal depth and face alignment.
- Ablation study: Adding the Temporal Transformer improves generalization capability, while the ablation also compares two- and three-layer encoder variants.
- Ablation study: The framework can robustly distinguish real and fake clips and accurately localize regions containing subtle temporal defects.The paper presents localization and visualization of temporal incoherence without manual annotations.
5. Conclusion
The paper finds that temporal cues significantly affect video face forgery detection, with the proposed solution achieving robustness and generalization against previous methods.
- Temporal information has significant effects on video face forgery detection.
- The proposed solution shows superior capabilities in robustness and generalization compared with previous methods.