Source-linked AI summary
C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection
Chuangchuang Tan, Renshuai Tao, Huan Liu, Guanghua Gu, Baoyuan Wu, Yao Zhao, Yunchao Wei
TL;DR
CLIP-based deepfake detection remains insufficiently understood, particularly why its features support generalization and what detection potential they contain. C2P-CLIP analyzes CLIP’s concept-matching behavior and injects category common prompts, improving detection across benchmarks, including a 12.41% mAcc gain over UniFD on UniversalFakeDetect.
Problem
The study addresses why CLIP features enable generalizable deepfake detection through linear classification and how to further explore CLIP’s detection potential.
Method
The method decodes CLIP detection features into text to identify concept matching, then injects category common prompts into the image encoder through contrastive learning.
Results
C2P-CLIP improves detection across UniversalFakeDetect and Genimage, including a 12.41% mAcc gain over UniFD on UniversalFakeDetect.
Takeaways & Limitations
The findings indicate that CLIP detects forgery by matching similar concepts, while category prompts enhance its deepfake detection capability.
Takeaways & Limitations
The word-frequency analysis does not comprehensively analyze entire captions, resulting in incomplete information.
Abstract
from arXiv · showhide
This work focuses on AIGC detection to develop universal detectors capable of identifying various types of forgery images. Recent studies have found large pre-trained models, such as CLIP, are effective for generalizable deepfake detection along with linear classifiers. However, two critical issues remain unresolved: 1) understanding why CLIP features are effective on deepfake detection through a linear classifier; and 2) exploring the detection potential of CLIP. In this study, we delve into the underlying mechanisms of CLIP's detection capabilities by decoding its detection features into text and performing word frequency analysis. Our finding indicates that CLIP detects deepfakes by recognizing similar concepts (Fig. \ref{fig:fig1} a). Building on this insight, we introduce Category Common Prompt CLIP, called C2P-CLIP, which integrates the category common prompt into the text encoder to inject category-related concepts into the image encoder, thereby enhancing detection performance (Fig. \ref{fig:fig1} b). Our method achieves a 12.41\% improvement in detection accuracy compared to the original CLIP, without introducing additional parameters during testing. Comprehensive experiments conducted on two widely-used datasets, encompassing 20 generation models, validate the efficacy of the proposed method, demonstrating state-of-the-art performance. The code is available at \url{https://github.com/chuangchuangtan/C2P-CLIP-DeepfakeDetection}
Introduction
The paper addresses the poor generalization of deepfake detectors to unseen sources by analyzing CLIP’s detection features and introducing C2P-CLIP. It finds that CLIP detects forgeries through category-related concept matching and improves detection by injecting category common prompts into the image encoder.
- Motivation: Existing detectors often struggle with unseen deepfake sources, motivating methods that improve generalization beyond source-specific artifacts.Prior approaches include artifact representations and large pretrained models such as CLIP, with UniFd using CLIP image features for linear classification.
- CLIP Analysis: CLIP’s detection features indicate that forgery detection relies on matching similar concepts rather than inherent true-or-false semantics.Detection features are obtained by transforming CLIP image features with linear-classifier parameters and decoding them into text.
- CLIP Analysis: The study validates its concept-matching hypothesis through feature decoding, clustering visualization, and word-frequency analysis across training and unseen sources.These analyses compare decoded detection features and identify shared concepts despite frequency differences between sources.
- Method: C2P-CLIP injects category concepts into CLIP’s image encoder through the text encoder to improve real-versus-fake discrimination.The method generates captions for training images and uses Category Common Prompts to enhance CLIP’s detection ability.
- Evaluation: C2P-CLIP significantly outperforms original CLIP and achieves state-of-the-art performance across evaluations involving images generated by 20 distinct models.The evaluation reports no additional parameters during testing.
Related Work
Related work is organized into face forgery detection and AIGC detection. Face-focused methods exploit spatial, frequency, or artifact information, while AIGC detection addresses broader forgery diversity and stronger generalization demands.
- The literature is categorized into face forgery detection and AIGC detection.
- Face Forgery Detection: Face forgery detection methods commonly exploit spatial or frequency information from images, including Xception-based detection of manipulated facial images.Other approaches focus on detecting specific artifacts, although the supplied passage is truncated before detailing them.
- AIGC Detection: AIGC detection extends beyond facial forgeries to diverse scenes, creating broader Deepfake variation and greater demands for generalization.The passage contrasts these challenges with face forgery detection and introduces CNN-Spot as using data augmentation to enhance generalization, but its description is truncated.
Methodology
C2P-CLIP enhances CLIP-based generalizable deepfake detection by appending category common prompts to image captions and injecting their concepts into the image encoder. Training uses contrastive and classification losses with LoRA, while testing uses only the adapted image encoder and classifier.
- Caption Generation and Enhancement: C2P-CLIP appends category common prompts to generated image captions, then transfers the resulting classification concepts into the image encoder for deepfake detection.Captions are generated with ClipCap and enhanced using prompt pairs such as (Camera, Deepfake) or (Biden, Trump).
- Concept Injection: The method trains LoRA layers and a classifier with contrastive and classification losses to improve detection generalization across unseen sources.The image and text encoders remain frozen during training, while the losses jointly guide concept transfer and classification.
- Detection: During testing, C2P-CLIP uses only the LoRA-integrated image encoder and classifier, without adding additional parameters at inference.The LoRA parameters are fine-tuned during training and retained in the image encoder for evaluation.
- Caption Generation and Enhancement: Category common prompts are assigned by class, with P = {P_real, P_fake}, and appended to each caption according to whether the image is real or fake.The enhanced captions are used in contrastive learning to transmit category information into the image encoder.
Experiments
Experiments evaluate C2P-CLIP’s cross-model generalization on UniversalFakeDetect and GenImage using mAP and mean accuracy. On UniversalFakeDetect, C2P-CLIP achieves 93.79% accuracy and 98.66% mAP across 19 test subsets, while prompt experiments indicate performance does not depend on specific category-common prompts.
- Datasets and evaluation: C2P-CLIP is evaluated for cross-model generalization on UniversalFakeDetect and GenImage, using ProGAN and SDv1.4 as training generators, respectively.UniversalFakeDetect contains 19 test generators, while GenImage includes seven diffusion models and one GAN model, with the remaining generators used for testing.
- Implementation details: The evaluation uses mean average precision and mean accuracy, with CLIP ViT-L/14 and LoRA-based parameter-efficient fine-tuning following the UniFD baseline.Training uses Adam, a 4×10^-4 initial learning rate, batch size 128, and one epoch.
- Evaluation on UniversalFakeDetect: 93.79% and 93.00% accuracy are obtained with the category-common prompts (Trump, Biden) and (Deepfake, Camera), respectively, showing performance without reliance on specific prompts.The two prompt choices produce different but high detection accuracies in the reported experiment.
Conclusion
The study analyzes why CLIP features support deepfake detection and finds that CLIP relies on matching similar concepts rather than distinguishing true from false. Based on this finding, it introduces category common prompts to fine-tune the image encoder with manually constructed concepts and contrastive learning, improving detection performance.
- Conclusion: CLIP detection features were decoded into text and analyzed by word frequency to investigate what information supports linear-classifier deepfake detection.The analysis was designed to understand why CLIP features are effective for deepfake detection.
- Conclusion: CLIP classifies deepfakes by matching similar concepts rather than discerning true and false.The authors present this as the central conclusion of their analysis.
- Conclusion: Category common prompts manually combine category concepts with contrastive learning to fine-tune the image encoder and improve detection performance.The prompt-based approach follows directly from the conclusion about CLIP’s concept-matching behavior.