Source-linked AI summary
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection
Junke Wang, Zuxuan Wu, Wenhao Ouyang, Xintong Han, Jingjing Chen, Ser-Nam Lim, Yu-Gang Jiang
TL;DR
Deepfake detection must identify increasingly photorealistic manipulations whose traces vary in scale and may be obscured in RGB after compression. The paper proposes M2TR, combining multi-scale RGB transformers with frequency-domain features and cross-modality fusion, and introduces SR-DF for development and evaluation. Extensive experiments report effective detection across datasets, including state-of-the-art performance and gains from the frequency and multi-scale components.
Problem
Photorealistic Deepfakes are difficult to distinguish, existing methods commonly analyze cropped faces at one scale, and compression can make forgery artifacts imperceptible in RGB.
Method
M2TR combines multi-scale RGB transformers, learnable frequency filters, and interactive cross-modality fusion for Deepfake detection.
Results
M2TR achieves state-of-the-art detection performance on different datasets, while ablations report a 1.7% AUC gain from frequency filtering under low-quality conditions.
Takeaways & Limitations
SR-DF provides a large-scale, high-quality dataset and evaluation criteria for developing and assessing Deepfake detection methods.
Abstract
from arXiv · showhide
The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images. In this paper, we aim to capture the subtle manipulation artifacts at different scales using transformer models. In particular, we introduce a Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels. M2TR further learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block. In addition, to stimulate Deepfake detection research, we introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods. We conduct extensive experiments to verify the effectiveness of the proposed method, which outperforms state-of-the-art Deepfake detection methods by clear margins.
1 INTRODUCTION
Deepfake detection is challenging because realistic manipulations contain artifacts at varying spatial scales, while compression can obscure them in RGB images. M2TR addresses these challenges with multi-scale RGB and frequency-domain modeling, alongside the SR-DF dataset and evaluation criteria.
- Motivation: Photorealistic forged images are difficult to distinguish from authentic images, motivating deep neural-network-based Deepfake forensics.
- Motivation: Forgery traces vary from small color mismatches to image-spanning visible boundaries, making multi-scale region analysis critical.The examples include color mismatch near mouth corners and visible boundaries spanning almost the entire image.
- M2TR: M2TR uses a two-stream architecture: multi-scale RGB transformers model regional inconsistency, while learnable frequency filters reveal complementary forgery features.A cross-modality fusion block combines the two streams interactively before classification.
- Dataset gap: Existing datasets contain clear artifacts and limited diversity, leaving a gap between benchmark samples and forged images circulating online.Celeb-DF uses one face-swapping method, and existing work lacks unbiased, comprehensive dataset-quality metrics.
- SR-DF: SR-DF is a large-scale, high-quality dataset generated with state-of-the-art face-swapping and facial-reenactment methods, accompanied by evaluation criteria for dataset quality.The dataset and evaluation system are intended for Deepfake detection development and evaluation.
- Results: M2TR achieves state-of-the-art Deepfake detection performance across different datasets in extensive experiments.
2 RELATED WORK
Deepfake detection research includes two-stream, localization-oriented, boundary-based, attention-based, and transformer-based approaches. The cited transformer methods generally model token interactions at a single scale.
- Deepfake detection: Two-stream methods separately capture facial manipulation clues and patch inconsistency, while multi-task learning can jointly detect forged faces and manipulated regions.
- Deepfake detection: Face X-ray detects blending boundaries, whereas MaDD uses multiple attention maps to capture Deepfake artifacts.
- Transformer methods: DCViT extracts face-image features with a CNN before feeding them to a traditional single-scale transformer for forgery detection.
- Transformer methods: Visual transformers typically model interactions between tokens at the same scale using self-attention mechanisms.The related-work discussion frames this as the typical visual-transformer design.
3 APPROACH
M2TR combines multi-scale RGB analysis with frequency-domain filtering and cross-modality fusion to detect subtle forgery artifacts. Its training also uses classification, segmentation, and contrastive objectives to improve supervision and feature separation.
- Architecture: M2TR uses stacked multi-scale transformers, frequency filters, and cross-modality fusion blocks to detect forgery clues.The integrated features are iteratively updated through N = 4 stacked blocks before global semantic features are extracted.
- Multi-scale Transformer: Multi-scale self-attention operates on patches of different sizes to capture forgery patterns at multiple spatial scales.Each head extracts patches with its own size, computes patch-wise self-attention, reshapes the result spatially, and combines heads through a residual block.
- Frequency Filter: The frequency stream applies a 2D FFT and learnable frequency filters, then uses an inverse FFT to produce frequency-aware features that complement RGB features.The filter models dependencies among different frequency-band components, addressing artifacts that may become imperceptible after compression.
- Cross Modality Fusion: A query-key-value Cross Modality Fusion block combines RGB features T_i and frequency features W_i into a unified representation M_i.The block embeds both modalities with 1 × 1 convolutions, applies spatially flattened attention, adds a residual connection, and uses a 3 × 3 convolution.
- Training Objectives: M2TR predicts image-level real-or-fake labels with fully connected layers and produces a binary face mask through an auxiliary decoder.The ground-truth mask marks manipulated pixels with 1 and other pixels with 0.
- Training Objectives: A contrastive loss pulls pristine samples toward their feature center relative to manipulated samples, while λ1 = 1 and λ2 = 0.001 balance the objectives.The full training objective combines classification, segmentation, and contrastive losses.
4 SR-DF DATASET
SR-DF is constructed from diverse, state-of-the-art face swapping and facial reenactment methods, with harmonization post-processing and evaluation across image quality, temporal consistency, and feature diversity.
- 4.1 Dataset Construction: SR-DF is built from pristine FF++ videos spanning diverse genders, ages, and ethnic groups, with forged videos generated using state-of-the-art techniques.Faces are cropped from video frames before synthesis, followed by image harmonization post-processing.
- 4.1 Dataset Construction: Four identity-agnostic manipulation methods provide diversity: FSGAN and FaceShifter for face swapping, plus First-order-motion and IcFace for facial reenactment.These methods can be applied to arbitrary face images without paired training identities.
- 4.1 Dataset Construction: DoveNet harmonization reduces color mismatch and stitched boundaries between synthesized faces and their backgrounds.Foreground and background masks are generated with a face parsing model.
- 4.2 Comparisons to current Deepfake Datasets: Dataset quality is benchmarked through identity retention, authenticity, temporal smoothness, and diversity using a comprehensive quantitative evaluation system.The paper presents these perspectives as criteria for comparing existing datasets and SR-DF.
- 4.2 Comparisons to current Deepfake Datasets: SR-DF achieves the highest average Mask-SSIM among compared datasets, while perceptual loss and E_warp assess visual similarity and short-term temporal consistency.Mask-SSIM compares masked face regions; lower perceptual loss indicates better image quality, and lower E_warp indicates smoother temporal results.
- 4.2 Comparisons to current Deepfake Datasets: Feature-space visualization shows Celeb-DF concentrated, FF++ easily separable between real and forged images, and SR-DF comparatively more diverse.The visualization uses ImageNet-pretrained ResNet-18 features, t-SNE, and one frame per video.
5 EXPERIMENTS
Experiments evaluate M2TR across multiple datasets, quality settings, classification granularities, video detection, cross-dataset generalization, and ablations. M2TR achieves strong detection results, with frequency, multi-scale, and contrastive components contributing to performance.
- Evaluation on FaceForensics++: M2TR achieves state-of-the-art frame-level performance across RAW, HQ, and LQ versions of FaceForensics++.The evaluation reports effectiveness across different visual qualities.
- Evaluation on FaceForensics++: 92.35% accuracy on FF++ (LQ) exceeds F3-Net by 1.92%, showing a benefit from frequency information under compression.The comparison is reported for the low-quality setting, where forged images are compressed.
- Evaluation on Celeb-DF and SR-DF: M2TR reaches 99.9% on Celeb-DF and 90.5% on SR-DF, outperforming the other compared methods across both datasets.These are frame-level detection results.
- Evaluation on ForgeryNet: M2TR outperforms state-of-the-art methods on ForgeryNet across two-way, three-way, and 16-way classification tasks.The largest reported gain is 15.5% on the 16-way classification task.
- From Frames to Videos: Adding a temporal transformer extends M2TR into ST-M2TR for video-level detection by combining frame-level M2TR features.This addresses the temporal information available in videos.
- Ablation Study: Removing contrastive loss decreases cross-dataset accuracy by 3.8%–5.8% depending on the training and evaluation datasets.The reported decreases are 3.8% and 5.2% when training on FF++, and 5.8% and 3.0% when training on SR-DF.
6 CONCLUSION
The paper concludes that M2TR combines multi-scale transformers, frequency filters, and adaptive cross-modality fusion for Deepfake detection. It also introduces SR-DF and an evaluation system for assessing Deepfake dataset quality and diversity.
- Conclusion: M2TR uses multi-scale transformers and frequency filters to capture local inconsistencies and improve robustness against image compression.Forgery features from the RGB and frequency streams are adaptively fused through cross-modality fusion blocks.
- Conclusion: SR-DF is generated with several state-of-the-art face swapping and facial reenactment methods as a challenging Deepfake dataset.The paper also presents an evaluation system for visual quality and data diversity.