Source-linked AI summary
GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image
Mingjian Zhu, Hanting Chen, Qiangyu Yan, Xudong Huang, Guanyu Lin, Wei Li, Zhijun Tu, Hailin Hu, Jie Hu, Yunhe Wang
TL;DR
Photorealistic AI-generated images create disinformation risks, while existing datasets lack sufficient scale, content diversity, and coverage of advanced generators. The paper introduces GenImage, a million-scale benchmark built from diverse ImageNet classes and eight diffusion or GAN generators, and evaluates detectors through cross-generator and degraded-image tasks. Results show strong same-generator accuracy but substantial degradation across different generators, while generator similarity and image content affect generalization.
Problem
Existing fake-image datasets lack the scale, broad image content, and advanced-generator coverage needed to evaluate detectors for AI-generated images.
Method
The paper constructs GenImage from ImageNet classes, real images, and fake images generated by eight diffusion and GAN models, then evaluates detectors on cross-generator and degraded-image classification tasks.
Results
Same-generator detection exceeds 98.5% accuracy, reaching 99.9% for Stable Diffusion V1.4 and V1.5, but performance substantially degrades across different generators.
Takeaways & Limitations
GenImage supports broader detector development and evaluation through million-scale data, diverse image content and generators, and real-world-oriented robustness tasks.
Abstract
from arXiv · showhide
The extraordinary ability of generative models to generate photographic images has intensified concerns about the spread of disinformation, thereby leading to the demand for detectors capable of distinguishing between AI-generated fake images and real images. However, the lack of large datasets containing images from the most advanced image generators poses an obstacle to the development of such detectors. In this paper, we introduce the GenImage dataset, which has the following advantages: 1) Plenty of Images, including over one million pairs of AI-generated fake images and collected real images. 2) Rich Image Content, encompassing a broad range of image classes. 3) State-of-the-art Generators, synthesizing images with advanced diffusion models and GANs. The aforementioned advantages allow the detectors trained on GenImage to undergo a thorough evaluation and demonstrate strong applicability to diverse images. We conduct a comprehensive analysis of the dataset and propose two tasks for evaluating the detection method in resembling real-world scenarios. The cross-generator image classification task measures the performance of a detector trained on one generator when tested on the others. The degraded image classification task assesses the capability of the detectors in handling degraded images such as low-resolution, blurred, and compressed images. With the GenImage dataset, researchers can effectively expedite the development and evaluation of superior AI-generated image detectors in comparison to prevailing methodologies.
1 Introduction
GenImage addresses the need for broad, large-scale datasets to develop detectors distinguishing AI-generated images from real images. It expands beyond face-focused benchmarks by using diverse image content and contemporary generators.
- AI-generated photorealistic images raise disinformation concerns because they can manipulate public opinion in social domains.
- Earlier datasets primarily focused on face forgery, limiting their applicability to broader image categories.UADFV contained 241 real and 252 fake images, while ForgeryNet exceeded one million face-forgery instances.
- GenImage uses ImageNet’s 1000 classes to generate 1.3 million fake images matching ImageNet’s real-image scale.
- GenImage covers varied content, including basketball and guitar, and uses state-of-the-art diffusion generators such as Midjourney and Stable Diffusion.The dataset also includes GAN-generated images, increasing generator diversity.
2 Dataset Construction
GenImage is constructed as a million-scale, class-balanced benchmark pairing ImageNet real images with fake images generated by eight diffusion and GAN models. The generated images broadly resemble real images while retaining substantial visual variability.
- 2.1 Dataset Details: GenImage contains 2,681,167 images: 1,331,167 real and 1,350,000 fake images.Real images include 1,281,167 training images and 50,000 testing images.
- 2.1 Dataset Details: The dataset uses ImageNet’s 1000 classes and generates 1350 images per class, with nearly equal real and generated-image distributions.The class-level allocation is 1300 generated training images and 50 generated testing images per class.
- 2.2 Fake Image Generators: Eight generators produce the fake images: BigGAN, GLIDE, VQDM, Stable Diffusion V1.4, Stable Diffusion V1.5, ADM, Midjourney, and Wukong.Each generator produces nearly equal numbers of images per class, with a stated exception for Stable Diffusion V1.5.
- 2.1 Dataset Details: Generated images generally resemble ImageNet real images while varying in movement, perspective, background, and object appearance.The paper characterizes these images as having high variability and reasonableness.
- 2.2 Fake Image Generators: Midjourney V5 generates 1024 × 1024 images, Wukong generates 512 × 512 images, and Stable Diffusion models generate highly realistic 512 × 512 images.
3 GenImage Benchmark
GenImage benchmarks AI-generated image detectors across generator shifts and image degradation, revealing strong same-generator accuracy but weaker cross-generator generalization and sensitivity to resolution.
- GenImage evaluates detectors using backbone, face-forgery, and general fake-image methods, including ResNet-50, DeiT-S, Swin-T, CNNSpot, and Spec.
- Cross-Generator Image Classification: Same-generator training and testing consistently exceeds 98.5% accuracy, reaching 99.9% on Stable Diffusion V1.4 and V1.5 subsets.
- Cross-Generator Image Classification: 54.9% accuracy results when ResNet-50 trained on Stable Diffusion V1.4 is tested on Midjourney, demonstrating substantial cross-generator degradation.
- Cross-Generator Image Classification: Cross-generator evaluation measures detectors trained on one generator against testing subsets from other generators, including cross-validation across eight generators.
- Cross-Generator Image Classification: Swin-T achieves the best cross-generator result, while ResNet and DeiT-S perform similarly and have comparable computation costs and parameters.
- Degraded Image Classification: Degraded-image evaluation tests low resolutions, JPEG compression, and Gaussian blur; models perform well at 112 × 112 but worse at 64 × 64.
4 GenImage Analysis
GenImage analysis examines detector behavior across image classes, generators, and spectral characteristics. Results show that broader class coverage and more images improve generalization, while diffusion images are harder to detect than GAN images.
- 4.2 Frequency Analysis: Diffusion-generated images contain fewer visible spectral artifacts than GAN images, making them more challenging for detection.GAN artifacts appear as regular grids, whereas real and diffusion images contain few artifacts.
- 4.3 Image Class Generalization: Training on more image classes improves accuracy when evaluating across all 1000 classes.A subset of classes can generalize to other image classes, but larger subsets perform better.
- 4.3 Image Class Generalization: Increasing the number of images per class improves detection accuracy.The comparison uses 100 classes with either 128 or 1300 images per class.
- 4.4 Generator Correlation Analysis: Generators with similar architectures tend to produce stronger cross-generator generalization.Stable Diffusion V1.4, Stable Diffusion V1.5, and Wukong show strong relationships, while Midjourney is the greatest generalization challenge.
- 4.5 Image Content Generalization: A detector trained on GenImage generalizes to face and art images beyond the dataset’s broad image-class coverage.The content-generalization evaluation uses face images from LFW and art images from Laion-Art and DiffusionDB.
5 Conclusion
GenImage is introduced as a million-scale benchmark for detecting generative-model fake images. It expands dataset scale, image content, and generator selection, and evaluates detectors under cross-generator and degraded-image conditions.
- 5 Conclusion: GenImage surpasses previous datasets and benchmarks in image number, image content, and generator selection.The paper provides a detailed dataset analysis to support detector development for real-world applications.
- 5 Conclusion: The benchmark proposes cross-generator and degraded-image classification tasks to assess detector performance in realistic settings.These tasks examine generalization across generators and robustness to degraded inputs.