Source-linked AI summary
Spatiotemporal Inconsistency Learning for DeepFake Video Detection
Zhihao Gu, Yang Chen, Taiping Yao, Shouhong Ding, Jilin Li, Feiyue Huang, Lizhuang Ma
TL;DR
DeepFake detectors have paid limited attention to spatial-temporal inconsistency, despite temporal artifacts remaining visible when individual forged faces appear genuine. The paper introduces a STIL block that combines spatial modeling, orthogonal adjacent-frame temporal differences, and information fusion in a plug-in 2D CNN design. STIL outperforms state-of-the-art methods on four widely used benchmarks.
Problem
Existing DeepFake detection methods mainly use frame-based or video-based binary classification and give limited attention to spatial-temporal inconsistency in forged videos.
Method
STIL combines spatial inconsistency modeling, horizontal and vertical adjacent-frame temporal differences, and information supplementation within a flexible block for existing 2D CNNs.
Results
STIL outperforms state-of-the-art methods on four widely used public benchmarks.
Takeaways & Limitations
Spatial-temporal inconsistency learning provides a unified representation for DeepFake detection by combining spatial and temporal facial traces.
Abstract
from arXiv · showhide
The rapid development of facial manipulation techniques has aroused public concerns in recent years. Following the success of deep learning, existing methods always formulate DeepFake video detection as a binary classification problem and develop frame-based and video-based solutions. However, little attention has been paid to capturing the spatial-temporal inconsistency in forged videos. To address this issue, we term this task as a Spatial-Temporal Inconsistency Learning (STIL) process and instantiate it into a novel STIL block, which consists of a Spatial Inconsistency Module (SIM), a Temporal Inconsistency Module (TIM), and an Information Supplement Module (ISM). Specifically, we present a novel temporal modeling paradigm in TIM by exploiting the temporal difference over adjacent frames along with both horizontal and vertical directions. And the ISM simultaneously utilizes the spatial information from SIM and temporal information from TIM to establish a more comprehensive spatial-temporal representation. Moreover, our STIL block is flexible and could be plugged into existing 2D CNNs. Extensive experiments and visualizations are presented to demonstrate the effectiveness of our method against the state-of-the-art competitors.
1 INTRODUCTION
DeepFake detection is motivated by security threats and the limitations of frame-based analysis, which can miss temporal inconsistencies between manipulated frames. The paper therefore proposes STIL, integrating spatial and temporal inconsistency modeling in a flexible 2D CNN block.
- DeepFake face manipulation poses cyber, social, and political security threats, motivating effective detection technologies.
- Frame-based detectors mine image-level forgery patterns but can miss unnatural transitions and facial jitter across video frames.
- SIM models spatial inconsistency, TIM models temporal inconsistency, and ISM fuses both streams within the STIL block.The block can be plugged into existing 2D CNNs such as ResNets.
- STIL formulates DeepFake video detection as learning spatial and temporal inconsistencies for a more comprehensive representation.
- TIM computes temporal differences between adjacent frames along horizontal and vertical directions to model temporal inconsistency.
- The method outperforms state-of-the-art methods on four widely used public benchmarks.
2 RELATED WORK
Prior DeepFake detection and video-temporal modeling methods provide foundations but leave room for task-specific inconsistency modeling. STIL addresses this gap with orthogonal temporal differences designed for fine-grained temporal inconsistency in forged videos.
- Early DeepFake generation and detection methods used hand-crafted features, while later approaches adopted deep neural networks as manipulations became more realistic.
- The STIL block integrates spatial and temporal features within a unified framework and is designed to plug into existing 2D CNNs.
- Existing approaches combine 2D backbones with recurrent operations or frame averaging, making performance dependent on the representation capacity of those backbones.
- 3D CNNs and two-stream inflated 3D CNNs model spatiotemporal features, while neighboring-frame temporal differences offer a more efficient alternative.
- STIL differs by using adjacent-frame temporal differences along two orthogonal directions to capture fine-grained temporal inconsistency for DeepFake detection.
3 OVERVIEW
The STIL block reframes DeepFake detection as learning spatial and temporal inconsistency within a unified 2D CNN framework. It combines SIM, TIM, and ISM to capture forged patterns, adjacent-frame differences, and complementary information.
- STIL Overview: STIL splits input features into two streams, using SIM for spatial forged patterns and TIM for temporal inconsistency, then connects them through ISM.The block is designed as a flexible component for existing 2D CNNs.
- Spatial Inconsistency Module: Spatial inconsistency includes artifacts such as checker-board patterns, blending borders, and image-quality mismatches between pristine and forged regions.SIM processes image-level features without temporal information and highlights spatial inconsistency areas.
- Temporal Inconsistency Module: TIM exploits feature differences between adjacent frames along horizontal and vertical directions to produce importance weights for temporally inconsistent regions.Its multi-fields-of-views structure uses vertical and horizontal temporal inconsistency enhancement, while temporal differencing adds no extra parameters.
- Information Supplement Module: ISM supplements the temporal pathway with selected spatial channels by combining global average pooling, channel-wise 1D convolution, and two-stream feature fusion.The studied spatial-to-temporal information flow, S →T, achieves the best performance in practice.
4 EXPERIMENTS
Experiments evaluate STIL on four public datasets through intra-dataset, cross-dataset, and ablation settings. The method generally outperforms comparison methods and shows stronger cross-dataset generalization.
- Evaluation setup: STIL is evaluated against state-of-the-art methods on four public face-forgery benchmarks under video-level settings.The datasets are FF++, Celeb-DF, DFDC, and WildDeepfake.
- Evaluation setup: Experiments use cropped facial frames, eight training frames, sixteen testing frames, and a ResNet50 backbone with STIL blocks replacing 3 × 3 convolutions.Dlib detects faces for FF++, while MTCNN is used for the other datasets.
- Intra-dataset comparison: 91.78% accuracy on NT c40 exceeds S-MIL-T by 4.99%, while STIL outperforms nearly all competitors across FF++ settings except F2F c23.The c40 setting is described as more challenging because manipulations create subtle facial artifacts.
- Intra-dataset comparison: The two-stream STIL model achieves the best performance, while frame-based XN-avg surpasses many video-based methods on several datasets.These results support retaining spatial information alongside temporal inconsistency modeling.
- Cross-dataset comparison: Cross-dataset testing trains on FF++ c40 and evaluates on Celeb-DF using AUC, where STIL achieves significantly higher AUC than all listed opponents.Frame-based methods show a severe performance drop after transfer to the unseen Celeb-DF dataset.
- Cross-dataset comparison: Ignoring temporal cues leads to poor generalization, whereas STIL balances spatial and temporal information to facilitate transfer across manipulation methods.The passage attributes frame-based degradation to overfitting on seen forged patterns.
5 ABLATION STUDY
Ablations examine the contributions and information flow of SIM, TIM, and ISM. Results favor combining spatial and temporal streams with the proposed spatial-to-temporal connection and attention-map fusion.
- Module effectiveness: Removing either SIM or TIM substantially decreases accuracy, while the complete STIL block combines SIM, TIM, and ISM.The variants include SIM alone, TIM alone, both without ISM, and the complete block.
- Information flow: Only the SIM-to-TIM connection produces consistent performance gains among the tested unidirectional and bidirectional information flows.The alternatives are T →S and bidirectional S + T.
- Difference fusion: Averaging horizontal and vertical attention maps consistently outperforms summing directional temporal information before the sigmoid function.The authors conjecture that summation can affect single-direction inconsistency representations.
6 VISUALIZATION AND ANALYSIS
Visualizations show how STIL responds to different manipulation types and how SIM and TIM contribute complementary spatial-temporal evidence. TIM separately localizes traces along vertical and horizontal directions.
- STIL outputs: STIL feature maps focus on the whole face for DeepFake and FaceSwap, but localize most forged areas for Face2Face and NeuralTextures.The model captures characteristics of each manipulation method despite training with only video-level labels.
- SIM and TIM attention: Combining SIM and TIM produces broader activation over difficult FaceSwap traces than either module alone.Both modules locate forged areas, but their individual maps are not fully comprehensive.
- SIM and TIM attention: For a NeuralTextures example, TIM corrects SIM’s false activations on non-forged areas, and their combination produces the correct final activation map.The example illustrates complementary spatial and temporal information.
- TIM directional maps: TIM’s h−t maps trace forgery along the vertical direction, whereas w−t maps trace it along the horizontal direction.The two directional maps are projected back onto the original video frames.
7 CONCLUSION
The paper frames DeepFake detection as spatial-temporal inconsistency learning through STIL, combining spatial and temporal information for a comprehensive representation. STIL uses directional adjacent-frame differences and outperforms state-of-the-art methods on four public benchmarks.
- STIL formulates DeepFake detection as spatial-temporal inconsistency learning for a more comprehensive representation.
- TIM captures temporal inconsistency by comparing adjacent frames along horizontal and vertical directions.
- STIL outperforms state-of-the-art methods on four widely used public benchmarks.
- The paper also conducts in-depth ablation studies and extensive visualizations to investigate and demonstrate the method's effectiveness.