CLIPping the Deception: Adapting Vision-Language Models for Universal Deepfake Detection

1 INTRODUCTION

• The generalization challenge is that detectors trained on one category of fake images may falter on novel manipulated images and other data distributions.The paper describes limited cross-distribution generalization as a major issue for existing deepfake detectors.Source: §1 · p. 1
• The study asks whether using CLIP’s visual and textual capabilities can improve on linear probing, which uses visual features but not CLIP’s language component.It evaluates recent CLIP adaptation strategies, including Prompt Tuning, in deepfake detection.Source: §1 · pp. 1–2
• CLIP features show better real–fake separation than two supervised models in the cited t-SNE visualization.The paper notes CLIP’s ability to distinguish authentic from fake images even without specific real–fake training.Source: §1 · p. 1, Figure 1 · p. 2
• The experiments train four CLIP adaptation strategies on 200k ProGAN images and test them on images from 21 GAN-based, Diffusion-based, and Commercial generators.The strategies are Fine-tuning, Linear Probing, Prompt Tuning, and an Adapter Network.Source: §1 · p. 2
• Prompt Tuning beats the current state-of-the-art approach, and the paper also reports strong performance with only 20k real and fake training images.The introduction identifies Prompt Tuning as a leading strategy and separately highlights performance with a smaller training set.Source: §1 · p. 2
• The study also examines few-shot detection with 32 real/fake samples per LSUN object category and robustness to JPEG compression and Gaussian blurring.The introduction further reports analysis of training-set size, including performance with 20k real and fake images.Source: §1 · p. 2

2.1 Pre-trained Vision-Language Models

• CLIP and ALIGN use image–text pre-training and offer zero-shot capabilities, but downstream tasks may still require task-specific adaptation.CLIP was trained contrastively on 400 million image-caption pairs; ALIGN used one billion noisy image-text pairs.Source: §2.1 · p. 2

2.2 Transfer Learning

• Transfer-learning methods aim to adapt large vision-language models while retaining frozen encoders, including learnable textual prompts and CLIP-Adapter layers.The section notes that full fine-tuning can affect robustness to distribution shifts and gives CLIP’s 55% zero-shot accuracy on MNIST as an example of limited task performance.Source: §2.2 · pp. 2–3

2.3 Fake Image Generation and Detection

• Deepfake detectors trained on one generator may not generalize to others; CLIP-based linear probing was reported to transfer across GAN and Diffusion images after GAN-only training.Earlier approaches based on modified ResNet-50 showed differing performance on GAN- and Diffusion-generated images.Source: §2.3 · p. 3

3.1 Background

• Prior CLIP linear-probing results generalized to unseen GAN and Diffusion images, but reported weaknesses on several datasets motivated using CLIP’s broader feature space and language component.The cited weaknesses include Guided Diffusion, LDM, Deepfakes, FaceSwap, and commercial generators.Source: §3.1 · pp. 3–4, §3.2 · p. 4

3.2 Transfer Learning

• Prompt Tuning is the central adaptation strategy, alongside Linear Probing, Full Fine-tuning, and an Adapter Network.In CoOp, learnable prompt vectors are optimized while CLIP’s text and vision encoders remain frozen; the prompts place context tokens before class labels.Source: §3.2 · p. 4, §3.2.3 · p. 5
• The four strategies differ in which CLIP components are trained: Linear Probing trains a classifier over frozen image features, Full Fine-tuning trains the whole model, and the Adapter Network modifies only the vision branch.The Adapter Network keeps the rest of CLIP frozen.Source: §3.2.1 · p. 4, §3.2.2 · p. 4, §3.2 · p. 4

3.3 Generative Models Explored

• The study evaluates CLIP-based detectors across 21 datasets spanning GAN, Diffusion, and commercial generators, despite training on ProGAN alone.Glide and LDM subsets are included together and their results averaged in the tables.Source: §3.3 · p. 5

4.1 Generalization Performance

• 5.01% mAP and 6.61% average accuracy over prior SOTA across 18 synthetic image generators are reported for the proposed approaches.Prompt Tuning notably outperforms the other adaptation strategies on both metrics; Tables 2 and 3 present AP and accuracy comparisons.Source: §4.1 · p. 6, Table 2 · p. 6, Table 3 · p. 6

4.2 Effect of Transfer Learning Strategy

• Prompt Tuning (CoOp) outperforms other strategies and optimizes ≈12k parameters, versus ≈1.5k for Linear Probing, ≈427M for Fine-tuning, and ≈590k for Adapter Network.The comparison uses strategies trained on ProGAN/LSUN images; Prompt Tuning also leads in the few-shot experiments.Source: §4.2 · p. 6

4.3 Effect of Training Set Size

• Larger training sets generally yield higher scores, but differences are not significant; training-set size matters less for GAN than Diffusion or Commercial test images.The reduced datasets contain 20k, 40k, 60k, or 80k balanced real/fake images.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 Robustness to Post-processing Operations

• Increasing JPEG compression or blur severity reduces performance; Linear Probing leads across three generator families, while Fine-tuning is more robust on compressed commercial images than on GAN or Diffusion images.The models were not explicitly trained on compressed or blurred images.Source: §4.4 · p. 7, Figure 5 · p. 8

4.5 Few-shot Analysis

• With 640 training images, Prompt Tuning outperforms the other strategies on images from GAN-based, Diffusion-based, and Commercial generators.The 640 images comprise 320 real and 320 fake examples.Source: §4.5 · pp. 7–8

4.6 Performance on Commercial Tools

• On images from Midjourney-V5, Adobe Firefly, and DALL-E 3, the proposed strategies surpass previous detection methods despite using 200k rather than 720k training images.Table 6 presents the commercial-tool comparison.Source: §4.6 · p. 8, Table 5 · p. 8, Table 6 · p. 8

• The evaluation covers four transfer strategies trained on 200k ProGAN images and tested across 21 image generators.Source: §5 · p. 8
• Prompt Tuning surpasses current baselines and state-of-the-art methods, with significant improvement margins and minimal training parameters.Source: §5 · p. 8
• Strategies using both CLIP’s image and text components consistently outperform Linear Probing, which uses only visual features.Source: §5 · p. 8
• Few-shot experiments, JPEG compression, Gaussian blurring, and a 20k-image training condition are included in the robustness analysis.Source: §5 · p. 8