Source-linked AI summary
Generated Faces in the Wild: Quantitative Comparison of Stable Diffusion, Midjourney and DALL-E 2
Ali Borji
TL;DR
Fine-grained evidence about face generation by modern image-synthesis systems remains limited. The paper compares Stable Diffusion, Midjourney, and DALL-E 2 by evaluating detected faces from generated scenes against real faces using FID, and finds Stable Diffusion performs best. The authors introduce a 15,076-face generated-face dataset and identify substantial remaining gaps and evaluation limitations.
Problem
Fine-grained evaluation of modern generative models on categories such as faces remains limited, despite their high image-synthesis quality.
Method
The paper synthesizes scenes with three text-to-image systems, detects and filters faces, and compares generated and real faces using repeated FID measurements.
Results
Stable Diffusion generates more realistic faces than Midjourney and DALL-E 2 according to FID, although generated faces remain substantially worse than real faces.
Takeaways & Limitations
The GFW dataset and methodology support further evaluation of generated faces, including probing potential deepfakes and societal biases.
Takeaways & Limitations
FID relies on ImageNet-trained embeddings, which may be problematic for evaluating face quality; face-specific metrics and human judgments may be better options.
Abstract
from arXiv · showhide
The field of image synthesis has made great strides in the last couple of years. Recent models are capable of generating images with astonishing quality. Fine-grained evaluation of these models on some interesting categories such as faces is still missing. Here, we conduct a quantitative comparison of three popular systems including Stable Diffusion, Midjourney, and DALL-E 2 in their ability to generate photorealistic faces in the wild. We find that Stable Diffusion generates better faces than the other systems, according to the FID score. We also introduce a dataset of generated faces in the wild dubbed GFW, including a total of 15,076 faces. Furthermore, we hope that our study spurs follow-up research in assessing the generative models and improving them. Data and code are available at data and code, respectively.
1 Introduction
Recent generative models produce high-quality images, but fine-grained evaluation of specific categories such as faces remains limited. This study focuses on evaluating photorealistic faces generated in cluttered scenes.
- Recent diffusion-based models have advanced image synthesis beyond earlier GAN and VAE approaches.These models are trained on massive datasets and have attracted substantial attention.
- Quantitative evaluation usually measures fidelity and diversity of entire scenes rather than individual objects or scene components.
- The study evaluates the quality of photorealistic faces generated in the wild, including faces in cluttered multi-object scenes.It distinguishes this task from generating portraits or evaluating portrait-specific models.
2 Comparison
The study compares Stable Diffusion, Midjourney, and DALL-E 2 by detecting faces in generated scenes and evaluating them against real faces with FID. Stable Diffusion achieves the best reported face-generation scores, although generated faces remain below real faces and exhibit recurring artifacts.
- Models: The comparison covers Stable Diffusion, Midjourney, and DALL-E 2.Stable Diffusion and DALL-E 2 generate 512 × 512 images, while Midjourney images have variable size.
- Dataset: 15,076 generated faces were collected: 8,050 from Stable Diffusion, 6,350 from Midjourney, and 676 from DALL-E 2.The faces were detected in images generated from COCO captions selected for person-related words.
- Dataset: 30,000 real faces were collected from the COCO training set and Labeled Faces in the Wild.Highly occluded, dark, blurry, dull, animal, drawn, and cartoon faces were removed, while faces with eyeglasses were retained.
- Evaluation Scheme: FID was computed between randomly selected sets of 5,000 generated and real faces, using 10 repeated runs to obtain means and standard deviations.DALL-E 2 faces were sampled with replacement because fewer than 5,000 were available; real-face FID was computed by splitting 10,000 real faces into two sets.
- Results: Stable Diffusion achieves lower FID than Midjourney and DALL-E 2, while all models remain far from real-face FID.Midjourney’s weaker performance may partly reflect surrealistic and anime-like faces.
- Results: With all systems limited to 676 faces, FID values increase but Stable Diffusion still performs best.The authors suggest that larger sets, potentially around 20,000 faces, could better distinguish the models.
- Results: Models commonly struggle with eyeglasses, eyeballs, occluded faces, profile faces, and facial symmetry.
3 Conclusion
The study finds that Stable Diffusion generates more realistic faces, while substantial evaluation gaps and methodological limitations remain. It releases data and methodology to support further investigation of deepfakes, memorization, biases, and broader evaluation settings.
- Stable Diffusion generates more realistic faces than the other evaluated systems, according to the study’s conclusion.
- Human observers can identify whether generated faces are real in most cases, indicating a large gap remains.
- The evaluation included relatively few DALL-E 2 faces because system access was limited, motivating larger future comparisons.
- Some highly realistic faces may be copied partly or wholly from training data, motivating memorization tests or training-data release.
- The dataset and methodology can support investigations of deepfakes and demographic biases through annotations such as race, gender, and age.
- FID may be problematic for face evaluation because it uses ImageNet-trained embeddings; face-specific classifiers, other metrics, human judgments, and alignment are suggested alternatives.