Source-linked AI summary
DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models
Zeyang Sha, Zheng Li, Ning Yu, Yang Zhang
TL;DR
Text-to-image models can generate realistic fake images that raise misuse concerns, motivating systematic detection and attribution. The paper trains image-based and prompt-aware classifiers, analyzes prompt properties, and finds shared artifacts, model-specific fingerprints, and prompt conditions associated with higher authenticity.
Problem
Realistic fake images from text-to-image models can be misused, while prior detection and attribution efforts focused on traditional generators such as GANs.
Method
The paper trains binary detectors and multiclass attributors using image-only or image-prompt inputs, then analyzes prompt semantics and structure across four models and two datasets.
Results
Fake images share a common artifact, different source models leave distinct fingerprints, and “person” prompts or lengths from 25 to 75 produce more authentic fake images.
Takeaways & Limitations
Detection and attribution can provide empirical tools for identifying synthetic images and linking them to their source generation models.
Takeaways & Limitations
The approach cannot differentiate fake images from certain cases, and detection performance remains below its design goal on models such as GLIDE and DALL·E 2.
Abstract
from arXiv · showhide
Text-to-image generation models that generate images based on prompt descriptions have attracted an increasing amount of attention during the past few months. Despite their encouraging performance, these models raise concerns about the misuse of their generated fake images. To tackle this problem, we pioneer a systematic study on the detection and attribution of fake images generated by text-to-image generation models. Concretely, we first build a machine learning classifier to detect the fake images generated by various text-to-image generation models. We then attribute these fake images to their source models, such that model owners can be held responsible for their models' misuse. We further investigate how prompts that generate fake images affect detection and attribution. We conduct extensive experiments on four popular text-to-image generation models, including DALL$\cdot$E 2, Stable Diffusion, GLIDE, and Latent Diffusion, and two benchmark prompt-image datasets. Empirical results show that (1) fake images generated by various models can be distinguished from real ones, as there exists a common artifact shared by fake images from different models; (2) fake images can be effectively attributed to their source models, as different models leave unique fingerprints in their generated images; (3) prompts with the ``person'' topic or a length between 25 and 75 enable models to generate fake images with higher authenticity. All findings contribute to the community's insight into the threats caused by text-to-image generation models. We appeal to the community's consideration of the counterpart solutions, like ours, against the rapidly-evolving fake image generation.
1 Introduction
This work addresses the misuse of synthetic images from text-to-image models through detection, source attribution, and prompt analysis. It introduces image-only and hybrid classifiers and evaluates them across multiple models, datasets, and prompt properties.
- Text-to-image models can produce realistic synthetic images that may facilitate misinformation and threaten the art industry.
- The paper formulates three questions: distinguishing fake from real images, attributing fakes to source models, and identifying prompts linked to greater authenticity.
- Hybrid detection combines CLIP image and prompt embeddings, while image-only detection uses only the image; BLIP supplies prompts when natural prompts are unavailable.
- Fake-image attribution trains a multiclass classifier that labels images according to the text-to-image model that generated them.
- On MSCOCO, a Stable Diffusion-trained image-only detector reached 0.834 on Latent Diffusion, 0.613 on GLIDE, and 0.554 on DALL·E 2, while hybrid detection performed better.
- Prompts about “person” and prompts 25 to 75 words long were associated with fake images having higher authenticity, while noun proportion had no significant impact.
2 Preliminaries
The paper evaluates publicly available text-to-image models and benchmark prompt-image datasets. Stable Diffusion, Latent Diffusion, GLIDE, and DALL·E 2 differ in training data, architecture, and prompt handling.
- Text-to-image models generate images by denoising random noise under prompt guidance so the output matches the textual description.
- Stable Diffusion uses a CLIP text encoder and is pretrained on 512×512 images from a subset of LAION-5B.
- Latent Diffusion is pretrained on LAION-400M and also uses CLIP text encoding to guide generated images.
- GLIDE is trained on a filtered dataset of several hundred million prompt-image pairs and is not good at understanding “person” prompts because such images were removed from training.
- The experiments use MSCOCO and Flickr30k, both of which provide images paired with textual prompts.
- The study uses default settings and evaluates fake images at different resolutions because models are trained at different image sizes.
3 Fake Image Detection
The paper develops image-only and hybrid detectors for distinguishing text-to-image fakes from real images, emphasizing cross-model generalization and prompt-assisted detection. Experiments identify shared artifacts across fake images and show that hybrid detection is generally stronger and can remain effective with limited training data.
- Results: 0.5 accuracy: the forensic classifier fails to generalize from traditional generation models to text-to-image fakes.Its performance is equivalent to random guessing in all evaluated cases.
- Results: 0.871 accuracy: an image-only detector trained only on SD+MSCOCO detects LD+Flickr30k fakes despite never seeing that model-dataset combination.Detection performance varies more across generation models than across datasets; for example, SD reaches 0.913 while DALL·E 2 reaches 0.526.
- Results: 0.909 accuracy: hybrid detection with natural prompts on DALL·E 2+MSCOCO, versus 0.522 for image-only detection.With GLIDE+MSCOCO, natural prompts achieve 0.891 and BLIP-generated prompts achieve 0.838.
- Discussion: Fake images show brighter central regions and more concentrated frequency spectra than real images, revealing a shared cross-model artifact.This common artifact helps explain why fake images from different text-to-image models can be distinguished from real ones.
- Ablation Study: Higher prompt descriptiveness generally improves hybrid detection until performance stabilizes, supporting the robustness of BLIP-generated prompts.On artworks, previously trained detectors achieve 0.710 accuracy for image-only detection and 0.690 for hybrid detection.
- Takeaways: 0.830 accuracy: the hybrid detector achieves strong performance with only 500 training images, while larger datasets generally improve all detectors.For LD+MSCOCO, accuracy increases from 0.792 with 1,000 training images to 0.885 with 40,000.
- Design Goals: The detector differentiates fake images generated by text-to-image models from real images using image-only or hybrid inputs.The hybrid detector additionally uses corresponding prompts.
4 Fake Image Attribution
The paper develops image-only and hybrid multi-class attributors to identify which text-to-image model generated a fake image. Experiments show strong, dataset-agnostic attribution, model-specific fingerprints, and improved performance from prompts and larger training sets.
- Design Goals: The authors define attribution goals as tracking fake-image sources for accountability while remaining independent of prompt datasets.The attributor is trained on one prompt-image dataset and evaluated on another.
- Attributor Construction: A multi-class attributor labels images by source model, using either images alone or images paired with their prompts.The image-only version predicts real, Stable Diffusion, Latent Diffusion, or GLIDE classes.
- Evaluation Results: 0.864 accuracy is achieved by image-only attribution on images from multiple sources using MSCOCO prompts, far above the 0.25 random-guess baseline.The training set contains 80,000 images across four classes.
- Evaluation Results: Hybrid attribution outperforms image-only attribution across datasets, showing that prompts provide an additional attribution signal.This improvement is reported for hybrid attribution with and without natural prompts.
- Fingerprints and Adaptation: Distinct frequency patterns support the conjecture that generation models leave unique fingerprints, while Stable Diffusion and Latent Diffusion show similar spectra.The similarity is attributed to their use of similar algorithms.
- Impact of Training Dataset Size: 0.736 accuracy at 5,000 training images rises to 0.946 at 80,000, and hybrid attribution stabilizes with less data than image-only attribution.Hybrid attribution improves sharply from 10,000 to 20,000 images, whereas image-only attribution shows a similar improvement from 40,000 to 80,000.
- Conclusion and Limitation: The study concludes that fake images can be attributed to their sources and that the method can be adapted to unseen models, while current attribution cannot differentiate multiple unseen models individually.Unseen models are currently treated as one class.
5 Prompt Analysis
The paper analyzes how prompt semantics and structure relate to the authenticity of generated images. Prompts about people or with lengths from 25 to 75 are associated with more authentic fake images, while noun proportion shows no significant effect.
- 5 Prompt Analysis: Prompt analysis evaluates which semantic and structural prompt properties are associated with higher image authenticity.The study examines topic, prompt length, and noun proportion using generated images classified by an image-only detector.
- 5 Prompt Analysis: The top-five prompt table distinguishes subject-detail descriptions from environment-focused descriptions when identifying prompts associated with real or fake classifications.Gray cells mark subject details in Real prompts and environmental descriptions in Fake prompts.
- 5.1 Semantics Analysis: Figure 11 ranks the twenty MSCOCO topics by the proportion of generated images classified as real by the image-only detector.The ranking focuses on prompts used to generate Stable Diffusion images.
- 5.1 Semantics Analysis: “Skis” and “snowboard” rank highest under MSCOCO’s provided topics, with several animal-related topics also appearing among the top twenty.The ranking is based on the proportion of corresponding fake images classified as real.
- 5.1 Semantics Analysis: Embedding-based DBSCAN clustering identifies a highest-authenticity cluster related to the topic “person,” complementing the dataset-topic analysis.The cluster topics require manual summarization after clustering sentence-transformer prompt embeddings.
- 5.1 Semantics Analysis: Most “skis” and “snowboard” images also depict people, suggesting that the apparent topic difference reflects overlapping semantics.The paper therefore identifies “person” as more likely to produce authentic fake images.
- 5.1 Semantics Analysis: Detailed subject descriptions are associated with authentic images, whereas many prompts associated with fake classifications describe the surrounding environment.Four of the top five real prompts detail the subject, while four of the top five fake prompts describe the environment.
- 5.2 Structure Analysis: Prompts between 25 and 75 words are associated with higher authenticity, while noun proportion has no significant impact.Extremely short and extremely long prompts do not generate authentic images as often in the Stable Diffusion experiment.
6 Related Work
Earlier work studied detection and attribution mainly for traditional generators such as GANs, while text-to-image models introduced prompt-conditioned generation and remained largely unexplored. This paper positions its study as a first systematic step toward detection and attribution for these models.
- Generation Models: Text-to-image models generate images from textual prompts, unlike traditional generators that commonly synthesize images from latent codes sampled from a Gaussian distribution.Recent diffusion-based systems use noise and prompts to progressively produce clear images.
- Generation Models: Diffusion-based systems including DALL·E 2, Stable Diffusion, GLIDE, and related models have achieved state-of-the-art text-to-image generation performance.The passage contrasts these systems with earlier GAN-based approaches.
- Prior Detection and Attribution: Prior studies showed that fake images from traditional generators can share detectable defects and can be attributed to their source models.These findings motivated analogous investigation beyond GANs and other traditional generation models.
- Research Gap: Existing work focused primarily on traditional generators, leaving detection and attribution for text-to-image generation models largely unexplored.The paper presents itself as the first systematic study of this problem.
7 Conclusion
The study develops detectors and attributors for fake images from text-to-image models, finding that source models leave distinguishable fingerprints. It also identifies prompt characteristics associated with more authentic-looking generated images.
- Detection: The proposed detection framework uses image-only and hybrid detectors to distinguish fake images from real ones.The image-only detector uses images alone, while the hybrid detector incorporates additional information.
- Attribution: Image-only and hybrid attributors classify fake images according to their source text-to-image generation models.Their performance indicates that different generation models produce distinguishable image properties, interpreted as fingerprints.
- Prompt effects: Prompts about “person” and prompts with lengths from 25 to 75 generate more authentic fake images than other examined prompts.The finding covers both semantic topic and structural prompt length.
- Conclusion: The work presents a comprehensive study of detecting and attributing fake images from state-of-the-art text-to-image generation models.The authors position the study as the first comprehensive investigation of these tasks for such models.