PaperlayerView exact revision on arXiv

Source-linked AI summary

Community Forensics: Using Thousands of Generators to Train Fake Image Detectors

Jeongsoo Park, Andrew Owens

arXiv:2411.04125v2cs.CV

TL;DR

Fake-image detectors can struggle to generalize to previously unseen generators, and existing training datasets may cover too few models. Community Forensics gathers a wider range of generators to study this problem; its experiments find that increasing generator diversity improves generalization. The dataset is intended as a research resource, not as a detector for direct use in the wild.

  • Problem

    Existing fake-image detectors often struggle to generalize from training generators to unseen models, while current datasets may contain many images from relatively few generators.

    Source: §1 · p. 1

  • Method

    Community Forensics combines thousands of systematically collected open-source latent-diffusion models with selected open-source and commercial generators.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Experiments find that increasing training-data diversity improves generalization, including when additional models share latent-diffusion architecture.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The dataset provides a resource for studying generalization and data-collection strategies in generated-image detection.

    Source: §5 · p. 8

    A large portion of the dataset is diffusion-based, especially models based on Stable Diffusion.

    Source: §5 · p. 8

Abstract

from arXiv · show

One of the key challenges of detecting AI-generated images is spotting images that have been created by previously unseen generative models. We argue that the limited diversity of the training data is a major obstacle to addressing this problem, and we propose a new dataset that is significantly larger and more diverse than prior work. As part of creating this dataset, we systematically download thousands of text-to-image latent diffusion models and sample images from them. We also collect images from dozens of popular open source and commercial models. The resulting dataset contains 2.7M images that have been sampled from 4803 different models. These images collectively capture a wide range of scene content, generator architectures, and image processing settings. Using this dataset, we study the generalization abilities of fake image detectors. Our experiments suggest that detection performance improves as the number of models in the training set increases, even when these models have similar architectures. We also find that detection performance improves as the diversity of the models increases, and that our trained detectors generalize better than those trained on other datasets. The dataset can be found in https://jespark.net/projects/2024/community_forensics

1. Introduction

Detectors often fail to generalize to unseen generators, and limited generator diversity in existing datasets may contribute to that problem. Community Forensics addresses it with a broad dataset and experiments finding that more diverse training data improves generalization.

1. Introduction

  • Existing detectors can fail on images from unseen models, despite performing well on generators represented in training.Generator differences in architecture, loss, training distribution, and image processing can matter; even models with similar architectures may evade detection.Source: §1 · p. 1
  • Millions of fake images may still come from relatively few generators, limiting coverage of real-world variation and making performance harder to benchmark reliably.Cues that work on one generator set may fail on another.Source: §1 · p. 1
  • Community Forensics combines thousands of systematically collected open-source latent diffusion models with selected open-source models and commercial generators.The collection also samples varied model architectures, image processing, and image content.Source: §1 · p. 2, Figure 2 · p. 2
  • The dataset contains 4803 distinct models, roughly 34× more than the most extensive prior dataset sampling generative models.The paper presents increasing the number of generators as an underexplored axis of data diversity.Source: §1 · p. 2
  • Increasing training-data diversity improves generalization, and detectors trained on Community Forensics perform strongly on new evaluations and previously proposed benchmarks.The experiments also report that adding generators helps across architectures and that diverse architectures improve results because classifiers do not fully generalize between them.Source: §1 · p. 2

2. Related work

Prior work includes general-purpose fake-image datasets, fingerprint-based detectors, and out-of-distribution recognition methods. Community Forensics builds on this context by scaling data across generators, images, and architectures.

2. Related work

  • Earlier general-purpose datasets often sampled only a handful of models, while Community Forensics targets broader generator coverage for detection and generalization.Prior examples include datasets with fewer than 20 models each and RED116 with 116 models; concurrent user-image collection is complementary but may lack identifiable models.Source: §2 · p. 3
  • Fingerprint-based methods use spatial or spectral artifacts, but such artifacts may be eliminated by post-processing [15].The paper instead takes a data-driven approach that scales the number of models, images, and architectures.Source: §2 · p. 3
  • Out-of-distribution recognition research has explored likelihood ratios, self-supervision, model statistics, temperature scaling, and energy-based models.Prior findings emphasize diverse training data and augmentation, consistent with this paper’s reported gains from diverse generators and stronger augmentations.Source: §2 · p. 3

3. The Community Forensics Dataset

Community Forensics combines systematically collected latent-diffusion models, selected generators from other architectures, commercial models, and real-image sources. The dataset also provides evaluation data and generation practices designed to support generalization studies.

3. The Community Forensics Dataset

  • The dataset spans in-the-wild latent-diffusion models, selected models from other architectures, and commercial generators, paired with real images while preserving original formats where possible.Its coverage includes varied architectures, processing pipelines, and semantic content.Source: §3 · p. 3

3.1. Systematically collecting generative models

  • The systematic collection uses popular Hugging Face diffusers models, sampled through a common interface and ordered by download popularity.It yields 1.9M images from 4763 models, which appear to be entirely or almost entirely latent diffusion; pixel-based models incompatible with the automated pipeline were sampled manually or used for evaluation.Source: §3.1 · pp. 3–4

3.2. Collecting images from other architectures

  • The manually selected set contains 19 open-source models across GAN, pixel- and latent-diffusion, and autoregressive families, while commercial sampling covers 11 models.The manually selected models contribute 774K images; the commercial models contribute 15K.Source: §3.2 · p. 4

3.3. Collecting real images

  • Real images are sourced from a range of existing datasets, including LAION, ImageNet, COCO, FFHQ, and VISION.The real-image sources support studying how real images influence forensics models.Source: §3.3 · p. 4

3.4. Curating the evaluation set

  • The evaluation set includes 26K images from 21 models excluded from training, with generated images paired to their prompting source images.A public version uses openly licensed COCO and FFHQ images for reproducibility, while the comprehensive set may not be exactly reconstructible later because of link rot.Source: §3.4 · pp. 4–5

3.5. Generating images

  • Generation uses captions from real images for text-conditional models, balances samples across classes for class-conditional models, and saves outputs as PNG to avoid compression artifacts.Firefly outputs are saved as JPEG because its web interface only permits JPEG downloads.Source: §3.5 · p. 5

4. Experiments

The experiments evaluate binary generated-image classifiers across benchmarks, then test how model count, generator diversity, and design choices affect generalization. They also demonstrate a k-nearest-neighbor application for identifying generator types.

4.1. Training image forensics models

  • The study trains binary classifiers on 5.4M paired real and generated images and evaluates mean average precision and accuracy across five benchmarks.The classifiers use pretrained ViT and ConvNeXt backbones; the training procedure includes augmentations such as resizing, rotation, shear, and JPEG compression.Source: §4.1 · pp. 5–6

4.2. Generalization across benchmarks

  • 0.987 mAP and 89.2% accuracy on the comprehensive out-of-distribution set accompany outperformance of prior works on unseen benchmarks and near-parity on GenImage’s in-distribution set.The benchmark comparison covers several evaluation sets and previously released classifiers.Source: Table 2 · p. 6, §4.2 · p. 6, §4.1 · p. 6

4.3. Impact of model diversity

  • Performance rises with more training models, flattens beyond 1,000, and also improves on GANs and pixel-based diffusion despite latent-diffusion-only training.The model-count study samples from 3 to 3,333 generators and uses 100K training images per sampled subset.Source: §4.3 · pp. 6–7
  • Performance plateaus at approximately 27K images, while 1,000 randomly chosen models outperform 10 popular models at every image count, with a wider accuracy gap than mAP.The wider accuracy gap may indicate that model diversity helps calibrate classifier accuracy thresholds.Source: Figure 4 · p. 6, §4.3 · p. 7
  • The diverse manual set matches the latent-only set with fewer than half its images, and combining the two sets further improves performance.The manual set contains 774K images versus 1.9M in the systematic set and includes multiple generator types.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4. Analysis of design choices

  • Backbone choice has similar performance, but freezing hurts; aligned-versus-misaligned real data differs marginally, and the classifiers are more robust to transformations than prior models.GenImage is especially sensitive to blur, JPEG compression, and resizing; Ojha et al. to shear; Wang et al. perform poorly overall.Source: §4.4 · pp. 7–8, Figure 6 · p. 7, Figure 7 · p. 8

4.5. Other forensics applications

  • A k-nearest-neighbor method in the classifier’s feature space identifies the generator type used to synthesize an image.The authors present this as an example of a further forensics application enabled by the dataset.Source: §4.5 · p. 8

5. Conclusion

Community Forensics studies generalization to unseen generative models with a dataset of 4,803 models and 2.7 million images. The authors find model diversity important for generalization, while cautioning that the dataset and classifiers are not intended for direct real-world deployment.

5. Conclusion

  • The dataset contains 4,803 models and 2.7M images, and the authors report that model diversity plays a crucial role in generalization.They trained classifiers on the dataset and evaluated generalization across settings and previously proposed models and training practices.Source: §5 · p. 8
  • The authors do not intend the dataset for training classifiers used directly in the wild, where detection remains challenging and errors can have severe consequences.Examples include falsely accusing an author of creating fake images or allowing misinformation to be certified as real.Source: §5 · p. 8
  • Although the classifiers generalize to unseen models better than prior work, their error rates remain too high for many critical applications.The dataset is largely diffusion-based, especially Stable Diffusion models; the authors suggest future collection of more diverse generators.Source: §5 · p. 8

A. Other applications

The paper explores whether classifier features and nearest-neighbor classification can reveal generator types and inform generator attribution. Feature-space structure separates some generator and real-image groups, while commercial models often resemble latent diffusion models.

A. Other applications

  • A diverse set of generators and corresponding images may also support generator attribution, which seeks characteristics of the generator responsible for an image.The paper presents attribution as a potential application beyond real-or-fake forensics.Source: A. Other applications · p. 15
  • The feature space separates GANs from other groups, places most commercial models near latent diffusion models, and clusters several real datasets together.CelebA, FFHQ, and Landscapes HQ appear more isolated; the separations emerge without explicit training for this purpose.Source: A. Other applications · p. 15, Figure 8 · p. 15
  • Using k=5 and 10% of training data, nearest-neighbor classification identifies GANs, latent diffusion, and real data strongly, while pixel-based diffusion performs less well.Commercial models are predominantly classified as latent diffusion or GANs, and Stable Cascade shows similarity to latent diffusion models.Source: A. Other applications · p. 15

B. Dataset composition

The dataset records model metadata and is composed predominantly of latent diffusion models. Its evaluation set includes two HDiT variants treated separately for metrics but counted as one model.

B. Dataset composition

  • A vast majority of dataset models use the CreativeML OpenRAIL-M license.The license distribution is shown in Figure 10.Source: B. Dataset composition · p. 15
  • Model metadata records model names, categorized architectures, licenses, source real datasets, and Hugging Face tags when available.For Hugging Face models, the example metadata also logs both author and model names.Source: B. Dataset composition · p. 15
  • Most training-set models and generated images are latent diffusion, and the evaluation set includes two HDiT variants trained on different datasets.The variants are counted separately for mAP and accuracy because their training data and weights differ, but as one model in dataset model counts.Source: B. Dataset composition · p. 15

C. Training settings

The classifiers use AdamW and mixed precision, with a 52K-iteration training setting and shorter runs for large statistical analyses. The reported performance plateaus beyond roughly 3K iterations.

C. Training settings

  • Classifier training uses AdamW with a 2e-5 learning rate, 1e-2 weight decay, batch size 512, mixed precision, and cosine weight decay with 20% warmup.The main setting runs for 52K iterations.Source: C. Training settings · p. 15
  • For Figures 1 and 4, models train for 3K iterations to reduce the computational overhead of statistical analysis.The authors chose this duration because classifier performance begins to plateau at approximately that point.Source: C. Training settings · pp. 15–16, Figure 13 · p. 16
  • Table 3’s example metadata includes author and model names for Hugging Face models, plus architecture, license, source real dataset, and available tags.The dataset’s generators are predominantly latent diffusion models.Source: Table 3 · p. 16, Table 4 · p. 16
  • The dataset’s models predominantly use the CreativeML OpenRAIL-M license.Figure 10 presents the license histogram.Source: Figure 10 · p. 16

D. Example model project page

Figure 14 presents an example Hugging Face model project page, showing its model tags, download count, and sample images.

D. Example model project page

Related papers

Loading 2411.04125v2…