Source-linked AI summary
Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector
Xiao Guo, Xiufeng Song, Yue Zhang, Xiaohong Liu, Xiaoming Liu
TL;DR
Deepfake detection needs systems that combine reliable classification with interpretable explanations. M2F2-Det adapts CLIP through forgery prompt learning, connects its visual features to detection and an LLM, and achieves state-of-the-art results on detection and explanation generation tasks.
Problem
Existing approaches generally provide either binary deepfake scores or textual explanations, while CLIP-based and explanation-oriented methods retain limitations in dedicated forgery modeling.
Method
M2F2-Det uses Forgery Prompt Learning and a Bridge Adapter to adapt CLIP for forgery detection while connecting visual representations to an LLM for explanations.
Results
M2F2-Det achieves state-of-the-art deepfake detection across six datasets and state-of-the-art explanation generation on DD-VQA.
Takeaways & Limitations
The method simultaneously outputs detection scores and textual explanations for diverse face forgeries.
Abstract
from arXiv · showhide
Deepfake detection is a long-established research topic vital for mitigating the spread of malicious misinformation. Unlike prior methods that provide either binary classification results or textual explanations separately, we introduce a novel method capable of generating both simultaneously. Our method harnesses the multi-modal learning capability of the pre-trained CLIP and the unprecedented interpretability of large language models (LLMs) to enhance both the generalization and explainability of deepfake detection. Specifically, we introduce a multi-modal face forgery detector (M2F2-Det) that employs tailored face forgery prompt learning, incorporating the pre-trained CLIP to improve generalization to unseen forgeries. Also, M2F2-Det incorporates an LLM to provide detailed textual explanations of its detection decisions, enhancing interpretability by bridging the gap between natural language and subtle cues of facial forgeries. Empirically, we evaluate M2F2-Det on both detection and explanation generation tasks, where it achieves state-of-the-art performance, demonstrating its effectiveness in identifying and explaining diverse forgeries.
1 Michigan State University 2 Shanghai Jiao Tong University
The passage lists contact email addresses associated with Michigan State University and Shanghai Jiao Tong University.
- The listed contact addresses use the domains msu.edu and sjtu.edu.cn.
1. Introduction
M2F2-Det addresses the separation between deepfake scores and textual explanations by combining CLIP-based forgery modeling with an LLM. Its design targets generalization to diverse and unseen forgeries while producing interpretable detection outputs.
- Deepfake detectors commonly output binary scores, whereas explanation-oriented systems generate textual descriptions separately.
- M2F2-Det jointly produces fake probabilities and textual explanations through a multi-modal face forgery detector.
- Forgery Prompt Learning adapts CLIP with universal prompts and layer-wise forgery tokens for common patterns and image-dependent artifacts.
- The Bridge Adapter reuses CLIP image features, adds forgery-aware representations, and connects them to both detection and explanation generation.
- M2F2-Det achieves state-of-the-art detection across six datasets and state-of-the-art explanation generation on DD-VQA.
2. Related Works
Related work spans conventional deepfake detectors, vision-language approaches, and broader multimodal applications. M2F2-Det builds on these directions by linking deepfake representations with language generation.
- Conventional deepfake detection uses augmentation, frequency clues, disentanglement, specialized networks, and biometric analysis.
- Most conventional detectors output binary prediction scores, while DDVQA-BLIP generates textual explanations for deepfake detection.
- Multimodal language models have been applied to domains including embodied AI, document parsing, and medical diagnosis.
- M2F2-Det introduces a frequency token to align deepfake domain knowledge with a multimodal language model.
3. Method
M2F2-Det combines CLIP-based forgery prompt learning, a Bridge Adapter, and an LLM to perform binary detection and generate textual explanations. Its staged training aligns detection and visual representations with the LLM while using forgery-aware attention to guide detection.
- Preliminaries and architecture: CLIP estimates image–prompt likelihoods, while M2F2-Det adds a deepfake encoder and LLM for detection and explanation generation.The image and text encoders provide multimodal representations that support both binary classification and free-form textual responses.
- Forgery Prompt Learning: Forgery Prompt Learning combines general tokens for shared forgery patterns with image-dependent tokens for fine-grained artifacts.Layer-wise forgery tokens are inserted into Transformer encoder layers while the pretrained text-encoder weights remain frozen.
- Forgery Prompt Learning: FPL produces patch-level forged attention maps that provide spatial forgery priors for global image-level detection.The method uses text-to-image scores between CLIP text embeddings and image patches, extending prompt learning to localize forged regions.
- Bridge Adapter: The Bridge Adapter combines intermediate CLIP image features with deepfake-encoder features, then fuses them with attention maps into a refined forgery vector.Average pooling and convolution transform the fused feature map into the final representation used for deepfake detection.
- Forgery Explanation Module and training: The explanation module feeds both CLIP visual tokens and forgery-aware features to an LLM, whose outputs contain a judgment and its rationale.Training proceeds in stages: binary classification first, representation alignment second, and LLM fine-tuning with LoRA third.
4. Experiment
M2F2-Det is evaluated for detection and explanation generation across intra-dataset, inter-dataset, and DD-VQA settings, with ablations examining its forgery-aware components. It achieves strong detection and explanation results while forged attention maps and frequency-based features support its performance and interpretability.
- Intra-dataset performance: M2F2-Det achieves the best overall detection performance in FF++, surpassing TALL by 1.01% accuracy and 2.01% AUC on FF++ (c40).It also surpasses TALL by 1.41% accuracy and 1.37% AUC on Celeb-DF, and RECCE by 2.80% accuracy and 1.12% AUC on WDF.
- Inter-dataset performance: M2F2-Det improves inter-dataset AUC over FreqBlender by 0.24% on DFDC and 2.56% on FFIW, while performing worse than AUNet on DFD.The authors associate its generalization with CLIP features learned from diverse real-world web samples and its forgery-specific prompt mechanisms.
- Explanation generation performance: M2F2-Det achieves the best judgment accuracy and F1 score on DD-VQA, with 7.74% higher accuracy than DDVQA-BLIP and 4.51% higher F1 than fine-tuned LLaVA.Removing the frequency-based token HF decreases judgment accuracy by 10.12%.
- Explanation generation performance: M2F2-Det achieves the best explanation quality on DD-VQA, while its forged attention maps identify artifacts more effectively than DDVQA-BLIP’s cross-attention mechanism.The reported explanation-quality comparison uses five metrics, including CIDEr and ROUGE-L, and qualitative examples show accurate judgments and convincing explanations.
- Ablative study and analysis: Layer-wise forgery tokens and UF-prompts improve FF++ AUC by 1.54% and 1.63%, while combining both further increases the line-#2 result by 1.08% AUC.The Bridge Adapter raises the baseline by 2.77% AUC on FF++ and improves CelebDF performance by 4.93%.
- Ablative study and analysis: M2F2-Det exceeds Uni-Fake and DEFAKE by 24.18% and 19.74% AUC, respectively, while Forgery Prompt Learning exceeds CoOp and CoCoOp by 9.31% and 8.13% AUC.These comparisons are reported on FF++ (c40) and distinguish the proposed forgery-specific mechanisms from simpler CLIP-based or semantic prompt-learning approaches.
5. Conclusion
M2F2-Det is introduced for interpretable deepfake detection by combining Forgery Prompt Learning, a Bridge Adapter, and an LLM. This design connects robust visual representations with textual explanations.
- Conclusion: M2F2-Det adapts CLIP with Forgery Prompt Learning and connects its image encoder to a deepfake detection network through a Bridge Adapter.The integrated representations are also connected to an LLM to support textual interpretability.
- Conclusion: The method is designed to provide robust visual representations for detection while integrating with an LLM for enhanced interpretability.This summarizes the paper’s combined detection and explanation objective.