Source-linked AI summary
TALL: Thumbnail Layout for Deepfake Video Detection
Yuting Xu, Jian Liang, Gengyun Jia, Ziming Yang, Yanhao Zhang, Ran He
TL;DR
Deepfake video detectors need to model temporal inconsistencies without the high computational cost of existing video methods. TALL converts masked consecutive frames into a spatial-temporal thumbnail and integrates it with Swin Transformer as TALL-Swin. Experiments report validity and superiority across intra-dataset and cross-dataset settings, with robustness to unseen deepfake types and common corruptions.
Problem
Existing video deepfake detection methods can perform well but are computationally intensive, motivating efficient modeling of temporal information alongside spatial cues.
Method
TALL masks consecutive frames and rearranges resized sub-images into a predefined thumbnail, while TALL-Swin incorporates this representation into Swin Transformer.
Results
TALL-Swin outperforms previous methods in intra-dataset and cross-dataset scenarios and achieves promising results across unseen deepfake types and common corruptions.
Takeaways & Limitations
TALL provides a simple, model-agnostic strategy for joint spatial-temporal modeling without additional computation or parameters.
Abstract
from arXiv · showhide
The growing threats of deepfakes to society and cybersecurity have raised enormous public concerns, and increasing efforts have been devoted to this critical topic of deepfake video detection. Existing video methods achieve good performance but are computationally intensive. This paper introduces a simple yet effective strategy named Thumbnail Layout (TALL), which transforms a video clip into a pre-defined layout to realize the preservation of spatial and temporal dependencies. Specifically, consecutive frames are masked in a fixed position in each frame to improve generalization, then resized to sub-images and rearranged into a pre-defined layout as the thumbnail. TALL is model-agnostic and extremely simple by only modifying a few lines of code. Inspired by the success of vision transformers, we incorporate TALL into Swin Transformer, forming an efficient and effective method TALL-Swin. Extensive experiments on intra-dataset and cross-dataset validate the validity and superiority of TALL and SOTA TALL-Swin. TALL-Swin achieves 90.79$\%$ AUC on the challenging cross-dataset task, FaceForensics++ $\to$ Celeb-DF. The code is available at https://github.com/rainy-xu/TALL4Deepfake.
1. Introduction
Deepfake detectors must capture spatio-temporal inconsistencies without the computational burden of video models. TALL addresses this gap by arranging masked consecutive frames into thumbnails, while TALL-Swin achieves strong performance across evaluation settings.
- Motivation: Existing video-based methods model spatio-temporal dependencies but are computationally intensive, motivating a cheaper image-level strategy.
- Contributions: TALL transforms masked consecutive frames into a thumbnail that preserves spatial and temporal information for deepfake detection.Frames are resized into sub-images and rearranged in a predefined layout.
- Contributions: TALL is model-agnostic and incorporates spatial-temporal dependencies without additional computation or parameters.
- Contributions: TALL-Swin combines TALL with Swin Transformer and is evaluated on four benchmark datasets for deepfake detection.
- Results: TALL-Swin outperforms previous methods in both intra-dataset and cross-dataset scenarios.
2. Related Work
Prior deepfake detectors use spatial artifacts, temporal inconsistencies, or transformer-based representations, but these approaches involve generalization or efficiency trade-offs. The related work therefore spans image-level, video-level, and transformer methods.
- Image-level methods: Image-level methods exploit spatial artifacts such as local discrepancies, frequency patterns, and global texture statistics.
- Video-level methods: Video-level methods improve detection generalization by modeling temporal incoherence and spatio-temporal inconsistencies.
- Transformer methods: Vision-transformer methods can outperform CNN-based models but sacrifice computational efficiency.
3. Method
TALL converts sampled video frames into a masked thumbnail so image-level models can learn spatial and temporal deepfake cues efficiently. TALL-Swin adapts Swin windows to strengthen cross-frame interactions while reducing computational complexity.
- 3.1. Motivation: TALL preserves intra-frame artifacts and inter-frame inconsistencies by combining spatial and temporal information within the thumbnail.
- 3.2. Thumbnail Layout (TALL): Fixed-position masking across frames encourages the model to detect inconsistencies between adjacent tampered frames.
- 3.3. TALL-Swin: TALL-Swin enlarges early-stage windows so self-attention more frequently captures detailed spatio-temporal dependencies across thumbnail frames.
- 3.3. TALL-Swin: TALL-Swin has the lowest computational complexity among the compared image- and video-level transformer methods.
4. Experiments
Experiments evaluate TALL across backbones, datasets, corruptions, and design choices, showing improved accuracy, generalization, and robustness with low computational overhead.
- Scaling over backbones: TALL improves image-based backbones without computation increases, including ViT+TALL at 86.58% AUC on Celeb-DF with 55.4G FLOPs versus ViViT’s higher computation.Swin+TALL gains 7.6% on CDF and 3.6% on DFDC, compared with 4.5% and 1.5% for ResNet+TALL.
- Intra-dataset evaluations: TALL-Swin has comparable performance and lower consumption than prior video transformers on FF++ HQ, but performs unsatisfactorily on severely compressed LQ videos.The authors attribute the LQ degradation possibly to spatial information lost during frame scaling.
- Robustness to unseen perturbations: TALL-Swin remains more robust than previous methods across seven unseen corruption types and five severity levels, retaining high performance as perturbations increase.The authors associate this robustness with TALL’s mask augmentation and consecutive multi-frame input.
- Ablation Study: A 2×2 resized thumbnail layout raises TALL-Swin’s AUC by 3.2% over the baseline, whereas 4×4 layouts sharply reduce performance because sub-images become too small.The compact layout also shortens distances between images, supporting temporal dependence learning.
- Ablation Study: Masking four sub-images outperforms standard Cutout by 1.46% and Mixup plus Cutmix by 1.02%, supporting improved learning of subtle spatio-temporal variations.The study also finds a 1.74% AUC gain from expanding windows in the first three phases, while overly large windows can weaken local-information learning.
5. Conclusion
TALL offers a simple, cost-free representation for joint spatio-temporal deepfake detection, and TALL-Swin captures frame inconsistencies efficiently while showing robustness across unseen deepfake types and common corruptions.
- TALL-Swin efficiently captures inconsistencies between deepfake video frames and shows robustness to unseen deepfake types and common corruptions.