Raising the Bar of AI-generated Image Detection with CLIP

• Accessible generative tools expand both creative opportunities and disinformation risks, increasing demand for automatic media-authentication tools.Source: §1 · p. 1
• Detectors must generalize to unseen architectures and generation methods because real-world test images rarely match training distributions.Source: §1 · pp. 1–2
• Compression and resizing can wash out subtle forensic traces, making robustness to post-processing essential.Source: §1 · p. 1
• Prior evidence indicates training diversity helps generalization, but GAN-trained detectors perform less reliably on diffusion-generated images.Source: §1 · p. 2
• CLIP-based methods are explored as few-shot-capable alternatives for synthetic-image detection and out-of-distribution evaluation.Source: §1 · p. 2
• The proposed detector uses paired real/fake examples from one generator, with top performance from 1,000 to 10,000 pairs and minor decay at 100 or 10.Source: §1 · p. 2

• Synthetic-image detection research covers GANs and diffusion models, increasingly emphasizing invisible spatial or frequency-domain traces over visible artifacts.Source: §2 · p. 2

Spatial domain methods

• Spatial methods exploit model-specific digital fingerprints estimated from generated images, often using noise residuals.Source: §2 · p. 2
• Generalization can be pursued through augmentation, diverse categories, ensembling, local patches, and combined global-local features.Source: §2 · p. 2

• Detectors designed for GAN images struggle on diffusion images, especially after compression or resizing, and may require calibration data from the tested model.Source: §2 · p. 3

Frequency domain methods

• Frequency-domain artifacts can arise from generator up-sampling, producing regular spatial patterns and Fourier-domain peaks.Source: §2 · p. 3

Methods based on multimodal features

• Only a few studies use multimodal features for forensic detection, and prior CLIP-based OOD work relies on large fake-and-real training datasets.Source: §2 · p. 3

• The study evaluates detector behavior on unseen data from different origins and under post-processing, using a large test dataset designed to mitigate validation biases.Source: §3 · p. 3
• Test generators span GAN-based, diffusion-based, and commercial families, including DALL·E 2, DALL·E 3, Midjourney v5, and Adobe Firefly.Source: §3 · p. 3
• Real-image counterparts are selected carefully to avoid foreseeable biases in detection experiments.Source: §3 · p. 4
• Pristine and synthetic images with similar semantic content are constructed using textual descriptions extracted from a real dataset and prompts for generation.Source: §3 · p. 4
• The dataset contains 32,000 real and fake images and simulates social-network sharing through cropping, resizing, and JPEG compression.Source: §3 · p. 4
• AUC and AP are threshold-independent, whereas accuracy uses a fixed 0.5 threshold to simulate evaluation without calibration information.Source: §3 · p. 4

Detector design

• The detector generates synthetic counterparts from captions of N real images, extracts next-to-last-layer CLIP features, and trains a linear SVM on the paired vectors.Real/fake pairing is intended to reduce semantic bias, especially when N is very small.Source: §4 · p. 4

Reference set size

• AUC and AP exceed 85% across known GAN and diffusion families with only 10+10 reference images, while commercial-tool performance is substantially worse.Increasing the reference set improves performance by 5–10% up to the 10k plateau.Source: §4.1 · p. 5
• Accuracy does not always increase with N because its fixed 0.5 threshold differs from the optimal threshold for many out-of-distribution generators.AUC and AP are threshold-independent, whereas accuracy depends critically on threshold calibration.Source: §4.1 · p. 5
• With compression or resizing, AUC reaches 75–90% and accuracy 65–80% at N=10k; performance is nearly optimal at N=100, but 10+10 images are inadvisable.These results concern post-processed images simulating realistic conditions.Source: §4.1 · p. 5, Figure 3 · p. 5

Reference set content

• Reference-set performance depends on both image quantity and quality, including the specific real/fake combination and whether resizing or compression augmentation is used.These effects remain visible even with 10,000+10,000 reference images.Source: §4.2 · p. 5, Table 2 · p. 5

Generalization analysis

• Across 18 generative models, no state-of-the-art method performs uniformly well, while the CLIP-based detector consistently delivers strong out-of-distribution performance.Methods trained on GANs generally transfer within GANs but perform worse on diffusion models, with the reverse pattern for diffusion-trained methods.Source: §5 · p. 6

Robustness to perturbations

• On post-processed images, most state-of-the-art methods approach random-chance performance, whereas CLIP-based detectors retain good performance.Compression and resizing attenuate forensic traces, particularly for unknown commercial models.Source: §5 · p. 6

• Conventional detectors degrade when resizing or compression reduces subtle forensic traces, whereas the CLIP-based detector continues working well under these conditions.This behavior suggests reliance on higher-level semantic features rather than only low-level artifacts.Source: §6 · p. 6
• Table 4 compares four proposed variants using 1k or 10k real and fake training images, with optional compression/resizing augmentation, and reports dataset-average AUC.Table 5 reports AUC after random cropping, resizing, and compression.Source: Table 4 · p. 7, Table 5 · p. 7
• After attacks removing traces from synthetic images or adding them to real images, the proposed method suffers only a very limited loss while Corvi et al. degrades dramatically.The attacks were evaluated on 1,000 real and 1,000 synthetic images using AUC.Source: §6 · pp. 6–7
• The CLIP-based and Corvi detector scores are almost orthogonal, suggesting that they depend on uncorrelated features suitable for fusion.The proposed decision rule declares an image real only when both detectors agree.Source: §6 · p. 7, Figure 6 · p. 8
• The fusion improves over the proposed CLIP-based detector by +3.6% AUC and +7.4% Accuracy across GAN, Diffusion, and Commercial Tools.A smaller improvement is observed on resized or compressed images because low-level traces are more compromised.Source: §6 · p. 7
• A suitable 4× decimation removes Fourier peaks in a Stable Diffusion XL synthetic image, while autoencoder processing creates new peaks in a real image.The spectra illustrate how low-level traces can be removed from synthetic images or introduced into real images.Source: §6 · p. 6, Figure 5 · p. 8

• A simple CLIP-based detector achieves top performance across many generators and challenging conditions using only a few examples, including examples from generators not under test.Maximizing reference-feature diversity also improves performance.Source: §7 · p. 8
• Further enlarging CLIP’s pre-training set continues boosting performance, by up to 10%.Source: §7 · p. 8
• Fusion with traditional detectors is identified as a route to further performance improvements because CLIP features are largely independent of low-level forensic traces.Source: §7 · p. 8
• Table 6 shows that Corvi et al., which relies on low-level forensic traces, is severely affected by attacks, while the proposed method keeps working well across conditions.Source: Table 6 · p. 8
• Future work includes few-shot adaptation to the situation of interest and interpreting which forensic features drive detector decisions.Source: §7 · p. 8

• The appendix reports comparison methods, additional ablations and results, robustness analyses, social-network few-shot experiments, and experiments with additional synthetic generators.Source: Supplemental Material · p. 9

• Wang et al. introduced a ResNet50 CNN detector and the LSUN/ProGAN dataset later adopted for model training.Source: A. Reference methods · p. 9
• Several reference methods use specialized representations, including preserved low-level traces, local patches, learned noise patterns, or gradients from pretrained CNNs.Source: A. Reference methods · p. 9
• Corvi et al. uses strong augmentation and a large latent-diffusion dataset to improve robustness and generalization.Source: A. Reference methods · p. 9
• DIRE trains ResNet-50 classifiers on ADM or StyleGAN images, based on the premise that synthetic images reconstruct better through a pretrained model.Source: A. Reference methods · p. 9
• NPR uses residual images formed from differences between an original image and its interpolated version, with a ResNet-50 trained on four ProGAN classes.Source: A. Reference methods · p. 9
• The proposed detector extracts next-to-last-layer CLIP ViT-L/14 features and trains an SVM on a limited reference-feature set, with alternative layers and classifiers evaluated.Source: B. Additional ablation study · p. 9
• Additional experiments report AUC and balanced accuracy by generator family, including post-processed images where many methods perform near chance.Source: C. AUC vs Accuracy · pp. 9–10, Table 8 · p. 10, Table 9 · p. 10

Web examples

• Figure 9 illustrates real and synthetic images from DALL·E 3, Firefly, and Midjourney downloaded from X.The examples are arranged from left to right as real, DALL·E 3, Firefly, and Midjourney images.Source: Figure 9 · p. 10

Robustness protocol

• JPEG quality, resizing scale, and WebP compression are varied across SynthBuster images generated by six models.The evaluation reports AUC for competitive reference methods and proposed CLIP-based detectors.Source: D. Additional robustness analysis · p. 10, Figure 8 · p. 10

Robustness findings

• CLIP-based detectors with augmentation are basically insensitive to JPEG, WebP compression, and resizing.The 1k+ and 10k+ variants maintain robustness across all tested impairment conditions.Source: D. Additional robustness analysis · p. 10
• Reference methods show substantially larger degradation under compression and resizing, with Ojha becoming useless at 25% rescaling.Ojha loses up to 10 points under compression, while Corvi is unaffected by JPEG but loses performance under strong WebP compression and resizing.Source: D. Additional robustness analysis · p. 10

Dataset

• The experiment uses 500 real and 1,500 synthetic images from X, spanning DALL·E 3, Midjourney, and Firefly.Generator identities were inferred from tags and annotations on the social network.Source: E. Few-shot analysis in the wild · p. 11

Few-shot protocol

• The few-shot setting uses 10 real and 10 generated examples from one model, testing on all remaining images over 1,000 runs.Results are reported using AUC and accuracy.Source: E. Few-shot analysis in the wild · p. 11

Few-shot results

• 10+10 aligned examples provide an impressive performance boost over a much larger training set whose real and synthetic data are not aligned with the test data.The comparison uses COCO real images and latent-diffusion synthetic images for the larger training set.Source: E. Few-shot analysis in the wild · p. 11

Comparison

• Fusion of low-level and high-level features achieves more than 90% average AUC in the realistic web scenario.All methods degrade on this data, while the fusion approach remains comparable to the few-shot setting with prior information.Source: E. Few-shot analysis in the wild · p. 11

Extended evaluation

• The extended test set adds eight generators and 8,000 synthetic images for generalization analysis.Results are reported in terms of AUC and accuracy for four CLIP-based detector variants and SoTA methods.Source: F. Further generalization results · p. 11, Table 12 · p. 12

Generalization results

• The proposed method delivers consistently good AUC and accuracy across the additional generators, whereas SoTA performance varies substantially.Reference methods perform very well on some generators and very poorly on others.Source: F. Further generalization results · p. 11, Table 12 · p. 12
• 8.5% AUC and 13% Accuracy are the average gains over the best reference method on the additional data.The reported comparison aggregates performance across the extended generator set.Source: F. Further generalization results · p. 11

Post-processing robustness

• Post-processing causes dramatic losses for most competitors, sometimes reducing them to random-guess performance, while the proposed method preserves good performance.The strongest proposed variant includes augmentation in the reference data.Source: F. Further generalization results · p. 11, Table 13 · p. 12