PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing is not always believing: Benchmarking Human and Model Perception of AI-Generated Images

Zeyu Lu, Di Huang, Lei Bai, Jingjing Qu, Chengyue Wu, Xihui Liu, Wanli Ouyang

arXiv:2304.13023v3cs.AIcs.CV

TL;DR

The paper asks whether people and detection models can distinguish realistic AI-generated images from real photographs, a question tied to the reliability of visual information. It benchmarks both using human and model evaluations and a large fake-image dataset. Humans misclassified 38.7% of images, while existing AI detection algorithms still had a 13% misclassification rate.

  • Problem

    The paper asks whether humans can reliably distinguish AI-generated images from real photographs, given concerns that false images may mislead viewers.

    Source: §1 · p. 2

  • Method

    The study establishes human and model detection benchmarks and introduces Fake2M alongside datasets for evaluating AI-image detectors.

    Source: §1 · p. 3, §2.2 · p. 3

  • Results

    Humans misclassified 38.7% of images, while existing AI detection algorithms had a 13% misclassification rate.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The authors expect Fake2M and the two benchmarks to support further research toward secure and reliable AI-generated content systems.

    Source: §6 · p. 10

    Existing models do not consistently detect fake images across varied generation settings.

    Source: §4.2.3 · p. 9

Abstract

from arXiv · show

Photos serve as a way for humans to record what they experience in their daily lives, and they are often regarded as trustworthy sources of information. However, there is a growing concern that the advancement of artificial intelligence (AI) technology may produce fake photos, which can create confusion and diminish trust in photographs. This study aims to comprehensively evaluate agents for distinguishing state-of-the-art AI-generated visual content. Our study benchmarks both human capability and cutting-edge fake image detection AI algorithms, using a newly collected large-scale fake image dataset Fake2M. In our human perception evaluation, titled HPBench, we discovered that humans struggle significantly to distinguish real photos from AI-generated ones, with a misclassification rate of 38.7%. Along with this, we conduct the model capability of AI-Generated images detection evaluation MPBench and the top-performing model from MPBench achieves a 13% failure rate under the same setting used in the human evaluation. We hope that our study can raise awareness of the potential risks of AI-generated images and facilitate further research to prevent the spread of false information. More information can refer to https://github.com/Inf-imagine/Sentry.

1 Introduction

The paper benchmarks human and AI capability to distinguish AI-generated images from real photographs. It reports that humans struggle substantially, while model performance varies with training and validation datasets.

1 Introduction

  • The study frames realistic AI-generated images as a potential source of misleading visual information, including false depictions of events.The introduction cites fake images depicting Trump’s arrest as an example.Source: §1 · pp. 2–3
  • Fake2M is introduced as a large-scale dataset of over 2 million AI-generated images from three models, intended to support detector training.The study also includes real photographs sourced from the internet.Source: §1 · pp. 2–3
  • 38.7% of images were misclassified by human participants, who were better at identifying AI-generated portraits than other generated-image types.Participants achieved 61.3% average accuracy.Source: §1 · p. 2
  • No model consistently outperforms others across training datasets, and broader training data is associated with improved overall accuracy.The same model’s accuracy also varies when validation datasets use different generation models.Source: §1 · p. 2
  • The study establishes HPBench for evaluating human discernment and MPBench for assessing detection models across 11 validation datasets.The benchmarks assess human capability and model capability, respectively.Source: §1 · p. 3

2 Dataset Collection and Generation

The paper builds image collections for human and model evaluation, then compares Fake2M with other fake-image datasets. Its human collection pairs AI-generated images with real photographs, while MPBench uses multiple training and validation datasets.

2.1 Collect Data for Human Evaluation

  • HPBench uses AI-generated and real photographs across eight categories, with generated images from Midjourney-V5 filtered to remove easily identifiable low-quality fakes.Real photos were collected from 500px and Google Images using the same text prompts as for image generation.Source: §2.1 · p. 3, Table 1 · p. 3

2.2 Collect Data for Model Evaluation

  • MPBench comprises three training fake datasets of about 2 million images and 11 validation fake datasets of about 257,000 images from multiple generative models.Training real datasets are matched to the fake datasets; validation also uses real-image datasets including ImageNet-Test, CelebA-HQ-Train, and CC3M-Val.Source: §2.2 · pp. 3–4

2.3 Comparison with Other Datasets

  • Fake2M is described as the largest and most diverse public general fake-image dataset, compared with other datasets by content type, generator type, public access, and image count.The comparison table caption defines these dataset attributes but does not supply the comparison’s cell values.Source: §2.3 · p. 4, Table 3 · p. 5

3 HPBench: Human Perception of AI-Generated Images Evaluation

HPBench evaluates human ability to distinguish AI-generated images from real photographs. Participants had substantial difficulty overall, with performance varying by background, image category, and the defects they noticed.

3.1 Evaluation Setup

  • Fifty participants completed 100-question tests without a time limit, and the study reports Accuracy, Precision, Recall, and False Omission Rate across categories.HPBench uses collected photorealistic AI-generated images and real photographs.Source: §3.1 · p. 5, §3 · p. 5, Table 5 · p. 5

3.2.1 Overall Ability to Distinguish Real and AI-generated Images

  • Participants correctly distinguished 61.3% of images overall (38.7% misclassified); they correctly identified 66.9% of real internet photos, while 44.2% of AI-generated images were incorrectly identified as real.Source: §3.2.1 · p. 6

3.2.2 Distinguishing Abilities of Participants with Various Personal Backgrounds

  • Participants with AIGC background scored 2.7% better overall, including a 3.7% advantage for identifying AI-generated images and 0.7% for real images.The analysis says AIGC experience did not significantly affect the ability to distinguish real from AI-generated images.Source: §3.2.2 · p. 6, Figure 3 · p. 6

3.2.3 Distinguishability of different photo categories

  • Accuracy ranged from 67.5% for Multiperson images to 50.8% for Object images, a 16.7% gap between the easiest and most challenging categories.The authors report that Object images are especially realistic, while Multiperson, Man, and Woman images are easier to distinguish.Source: Figure 4 · p. 6, §3.2.3 · p. 7

3.2.4 Results of the Judgment Criteria and Analysis of the AIGC Defects

  • 28% of participants cited detail problems and 17% cited smoothness problems when correctly identifying fake images; about 14% relied on intuition.The analysis reports that current state-of-the-art generators have limited fine-detail capability and often produce overly smoothed portraits.Source: §3.2.4 · p. 7, Table 6 · p. 7

4 MPBench: Model Perception of AI-Generated Images Evaluation

MPBench evaluates fake-image detectors across training and validation datasets. Results vary with the model, training data, and validation-set characteristics, and the selected best model had a 13% failure rate under the HPBench test setting.

4.1 Experiments Setup

  • MPBench trains detectors on Fake2M and evaluates them using 11 validation datasets across four training-data settings.The setup compares five models, with each setting containing matched real and fake training images.Source: §4 · p. 7, §4.1 · pp. 7–8

4.2.1 Comparative Analysis of Accuracy Across Various Models

  • No model consistently led fake-image detection across all four training settings, while ConvNext achieved the highest real-image detection accuracies in each setting.CLIP-ViT-L(LC) led fake-image detection in Settings C and D at 86.4% and 83.3%, respectively; ConvNext's real-image accuracies were 98.5%, 95.8%, 77.1%, and 99.1% across Settings A–D.Source: §4.2.1 · p. 8, Table 7 · p. 8

4.2.2 Comparative Analysis of Accuracy Across Various Training Datasets

  • Dataset Setting D generally yielded superior average performance, and CLIP-ViT-L(LC) achieved 82.0% with Setting A and 81.6% with Setting D.Setting D combines five distinct generative models; the authors suggest its diversity may improve generalization.Source: Table 7 · p. 8, §4.2.2 · p. 9

4.2.3 Comparative Analysis of Accuracy Across Various Validation Datasets

  • ConvNext-S (B+J 0.5) trained on Setting D ranged from 35.0% to 99.9% accuracy across validation datasets.The authors report that validation performance varies with the generation model, sampling method, and sampling steps.Source: Table 7 · p. 8, §4.2.3 · p. 9
  • Existing detectors do not consistently handle the broad range of generation models and hyperparameters found in realistic applications.The authors identify this as a need for future detection approaches.Source: §4.2.3 · p. 9

4.2.4 Evaluate the best model under the same setting used in HPBench.

  • 13% was the failure rate of ConvNext-S (B+J 0.5) trained on Dataset Setting D when tested under the same setting as HPBench.The model was selected as MPBench's best based on its highest total average accuracy and tested on 50 real and 50 fake images.Source: §4.2.4 · p. 9

5 Related Work

Related work describes rapid gains in image-generation quality and prior efforts to detect fake images. It highlights the growing difficulty of human detection as generated images become more realistic.

Image Generation Models

  • Diffusion-based generators have improved image fidelity and diversity over earlier GAN-based models, while autoregressive models have also achieved strong performance.The cited examples include DALL·E 2, Imagen, Stable Diffusion, IF, and Midjourney among diffusion approaches.Source: §5 · p. 9

Fake Images Generation and Detection

  • Earlier fake-image detection studies focused on GANs or smaller generators, whose lower-quality images were easier for people to identify.The passage cites prior detection works [11] [16] [26] [55] [57] [84].Source: §5 · p. 9
  • As image quality improves with recent generators, it has become increasingly difficult for humans to tell whether images were AI-generated.The passage notes that the need to detect fake images predates powerful image generators.Source: §5 · p. 9

6 Conclusion

The paper concludes that people can be deceived by current image generators, while AI detectors perform better but still misclassify images. It also provides a quick-test answer key and additional dataset-generation details.

6 Conclusion

  • AI detection algorithms outperform human discernment in the study, but still have a 13% misclassification rate.The authors present Fake2M, HPBench, and MPBench as resources for further research on secure and reliable AI-generated content systems.Source: §6 · p. 10

A Quick Test: Can you identify which ones are AI-generated images?

B.1 Dataset Configuration for Model Evaluation

  • The dataset configuration appendix describes fixed-seed, fixed-CFG-scale diffusion generation with varied sampling methods and steps, alongside StyleGAN3 data generation.StyleGAN3 images were generated from two models per dataset for faces, animal faces, and art-human faces.Source: §B.1 · p. 16

B.2 Data Content Component Analysis of Training and Validation Dataset

This section describes Fake2M and the training and validation dataset settings, including their content composition and balance.

B.2 Data Content Component Analysis of Training and Validation Dataset

  • Fake2M comprises 1M images generated with SD-V1.5Real-dpms-25, 1M with IF-V1.0-dpms++-25, and 87K with StyleGAN3.The images are generated from CC3M captions for the first two sources.Source: §B.2 · p. 17
  • Fake2M contains 82K face images, 4% of its 2M images, while MPBench validation has 60K fake and 24K real face images.The section reports no real-versus-fake image imbalance across the four training settings and MPBench.Source: §B.2 · p. 17
  • Training settings A and B each pair 1M fake images from CC3M with the first 1M real CC3M images, using different generators, and neither has content imbalance.Setting A uses SD-V1.5Real-dpms-25; setting B uses IF-V1.0-dpms++-25.Source: §B.2 · p. 17
  • Setting C pairs 87K StyleGAN3 fake images with the first 1M real CC3M images; its face-heavy imbalance was intentional for studying StyleGAN3 face fakes.Source: §B.2 · p. 17
  • Setting D includes 460K fake images from each of two generators, 87K StyleGAN3 fakes, 1M real CC3M images, and 87K real StyleGAN3 images.Fake and real face data are each 82K, and the section reports no content imbalance.Source: §B.2 · p. 17

B.3 Quality Analysis

The section describes Fake2M’s quality-score distribution and HPBench’s human-evaluation procedures, participant choices, and metrics. It reports results from both monitored and crowdsourced evaluations.

B.3 Quality Analysis

  • The average quality score across Fake2M images is 0.6, and most sub-datasets average above 0.6; the distribution includes both high- and low-quality images.The section reports Midjourneyv5-5K at 0.66 and provides score-distribution and image examples in Figures 6 and 7 and Table 12.Source: §B.3 · p. 17, Table 12 · p. 18, Figure 6 · p. 19

C.1 Procedures for HPBench

  • HPBench begins with annotator-filtered realistic AI-generated and real images across eight categories, then has 50 volunteers answer 100 questions before analysis.Source: §C.1 · pp. 19–20, Figure 8 · p. 20

C.2 Detail of Human Evaluation

  • Each diverse participant completed 100 untimed questions in a monitored setting without being told the real-to-AI image ratio; each questionnaire contained 50 images of each type.Images were sampled from databases of 244 real and 151 AI-generated images; participants could select or supply reasons for identifying an image as AI-generated.Source: §C.2 · pp. 20–21

C.3 Reason of High-quality Fifty-participant Human Evaluation

  • The authors favor a small-N design: diverse participants each complete the full survey in a quiet, controlled, monitored environment rather than noisy crowdsourcing.They cite [30] and in motivating this approach as a way to obtain high-quality human-evaluation data.Source: §C.3 · p. 21

C.4 Crowd Sourcing Human Evaluation

  • 49.9% was the accuracy across 1,085 crowd-sourced questionnaires; the authors report difficulty distinguishing high-quality AI-generated images in this setting.Participants were not asked to justify their decisions in the uncontrolled, noisy crowdsourcing setting.Source: §C.4 · p. 21

C.5 Metrics of HPBench

  • HPBench treats AI-generated images as positive samples and reports Accuracy, Precision, Recall, and False Omission Rate (FOR).Source: §C.5 · p. 21

C.6 Analysis of the AIGC defects

The section catalogs reported defects in AI-generated images, compares human score patterns across categories, and presents further MPBench and model-versus-human comparisons.

C.6 Analysis of the AIGC defects

  • The nine reported defects include malformed hands, overly smooth skin, implausible lighting, erroneous signs or text, and unrealistic details.Source: §C.6 · pp. 21–22, Figure 9 · p. 22
  • Other reported defects are blur, unrealistic color, excessive daubing, and unreasonable relationships between objects, including size or position.Source: §C.6 · p. 22

C.7 Detailed Score Distribution

  • Men’s and women’s average scores were nearly the same, at about 61%; men scored higher on Man and women on Woman, while Figure 14 breaks scores out by AIGC background.Source: §C.7 · p. 23, Figure 10 · p. 23

D.1 More Experiments on MPBench

D.2 Evaluate the best model under the same setting used in HPBench.

  • On HPBench, ConvNext-S performed better than humans on Man and Object; human peak performance was higher in other categories, but human average performance was far lower.Source: Figure 11 · p. 23, §D.2 · p. 23

D.3 Hyperparameters of the Experiments

This section points to MPBench hyperparameter tables and reviews image-generation progress and the motivation for studying human perception of realistic AI-generated photographs.

D.3 Hyperparameters of the Experiments

E.1 Image Generation

  • Diffusion and autoregressive models have advanced text-to-image synthesis, producing high-quality photorealistic images, though sampling can require time-consuming iterative processes.The section contrasts this progress with earlier GAN approaches and their narrower domains and datasets.Source: §E.1 · p. 25

E.2 Deepfake Generation and Detection

  • Realistic deepfakes can threaten trust in digital content, while improved generators make images harder to identify; this study focuses on human perception of AI-generated photographic images.Source: §E.2 · pp. 26–27

F.1 Discussion

F.1 discusses how high-quality AI-generated images deceive people while retaining detectable defects; F.2 connects that challenge to societal risks and research directions in detection, generation, and synthetic data.

F.1 Discussion

  • 38.7% of participants' answers were misclassified; image quality alone is unreliable, and inspecting visual cues took about 18 seconds per image.The cues discussed include portrait over-smoothing, object coherence, consistency, and physical laws.Source: §F.1 · p. 27
  • Suitable prompts can produce photorealistic images, but current generators retain defects and require prompt engineering and many experiments.Reported weaknesses include intricate scenes with multiple people, hand gestures, blurriness, and unrealistic physical phenomena.Source: §F.1 · pp. 27–29

F.2 Broader Impact

  • Difficulty distinguishing images raises concerns that AI-generated content could undermine photos as evidence and spread false information or harm people and organizations.The discussion calls for image-identification methods, ethical-use guidelines, and public awareness.Source: §F.2 · p. 29
  • AI-generated visuals may also offer artists, designers, and users lower-cost creation, restoration, enhancement, and personalized images.The paper describes opportunities including restoring old or damaged images and tailoring images to user preferences.Source: §F.2 · p. 29
  • The authors identify AI detection and analysis of generation failures as research directions, including improving detection systems and image generators.The failure-case analysis is proposed as inspiration for designing better generators.Source: §F.2 · p. 30
  • MPBench raises a research question about balancing detector generalization with accuracy on real images across backbone-freezing settings.CLIP-ViT-L (LC) generalizes well in MPBench but has lower real-image accuracy, while models with whole backbones unfrozen show the reverse pattern.Source: §F.2 · p. 30
  • The paper also proposes studying whether generated images can help address data imbalance and test model robustness and bias.Source: §F.2 · p. 30

F.3 Limitation

The conclusion reports that detectors outperform humans in the study but retain a considerable error rate. The authors also identify limits in dataset coverage, participant recruitment, and the security and credibility questions addressed.

F.3 Limitation

  • Fake2M training data covers three advanced generators; it excludes Midjourney V5 because the authors lacked API access and omits some model types.The authors call for greater training-data diversity and size.Source: §F.3 · p. 30
  • Validation covers advanced generative models, not other fake-image tasks such as deepfakes or low-level tasks.The authors propose expanding validation-set task diversity.Source: §F.3 · p. 30
  • HPBench recruited 50 participants with limited background attributes; its score distributions are shown overall and by gender and AIGC background.The authors identify participant diversity and sample size as areas for future improvement.Source: §F.3 · p. 30, Figure 12 · p. 32, Figure 13 · p. 33

F.4 Conclusion

  • 13% misclassification remains a challenge for AI detection algorithms, which the conclusion says distinguish authentic from fake images better than humans.The conclusion also reports that current image generators can produce high-quality images comparable to real photographs.Source: §F.4 · p. 30
  • The study leaves open why and how current generators achieve their capabilities and how to ensure AIGC image security and credibility.Source: §F.4 · p. 31
Loading 2304.13023v3…