Source-linked AI summary
Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation
Dimitrios Christodoulou, Mads Kuhlmann-Jørgensen
TL;DR
Subjective evaluation and fragmented setups make text-to-image models hard to compare and the state of the art difficult to assess. The paper collects large-scale human preferences through a global annotator pool and ranks four models across three criteria. Its benchmark gathers over two million votes, far exceeding the annotation counts reported in existing publications.
Problem
Text-to-image quality generally requires subjective judgment, and the lack of a standardized benchmark makes models difficult to compare and the state of the art hard to assess.
Source: §1 · pp. 1–2Method
The paper uses Rapidata-enabled large-scale human feedback from a diverse global annotator pool to evaluate text-to-image models.
Source: §1 · p. 2Results
Over 2 million votes make the benchmark orders of magnitude larger than existing publications, which typically report 2–20k annotations; rankings compare models across three criteria.
Source: §1 · p. 2, §3.3 · p. 5Takeaways & Limitations
The authors say the approach provides an efficient, representative, and scalable evaluation framework and opens the door to incorporating human feedback into training.
Source: §4 · p. 7Nineteen Stable Diffusion prompts were corrupted, affecting about 1.5% of comparisons; the authors believe the overall effect is negligible.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Efficiently evaluating the performance of text-to-image models is difficult as it inherently requires subjective judgment and human preference, making it hard to compare different models and quantify the state of the art. Leveraging Rapidata's technology, we present an efficient annotation framework that sources human feedback from a diverse, global pool of annotators. Our study collected over 2 million annotations across 4,512 images, evaluating four prominent models (DALL-E 3, Flux.1, MidJourney, and Stable Diffusion) on style preference, coherence, and text-to-image alignment. We demonstrate that our approach makes it feasible to comprehensively rank image generation models based on a vast pool of annotators and show that the diverse annotator demographics reflect the world population, significantly decreasing the risk of biases.
1 Introduction
Subjective judgments and fragmented evaluation setups make text-to-image models difficult to compare and the state of the art hard to assess. The paper presents a large-scale human-feedback benchmark and ranks four prevalent models using more than two million votes.
1 Introduction
- Existing evaluations lack a standardized benchmark, making an objective overview of text-to-image model performance difficult.Human panels can be small and lack demographic information, while AI evaluators may miss human understanding and nuance.Source: §1 · pp. 1–2
- The authors introduce a global, diverse human-feedback benchmark designed for large-scale evaluation and feasible repetition as new models appear.They describe it as representative and enabled by Rapidata’s technology.Source: §1 · p. 2
- The study ranks Flux.1, DALL-E 3, Stable Diffusion, and MidJourney using votes on 4,512 generated images.The benchmark uses 282 prompts curated from existing literature.Source: §1 · p. 2
- More than 2 million votes make this benchmark orders of magnitude larger than existing publications, which typically report 2–20k annotations.The votes cover 4,512 generated images.Source: §1 · p. 2
2 Methodology
The methodology gathers pairwise human judgments across three image-generation criteria, using a global annotator pool, quality controls, and prompts assembled from complementary benchmarks. The same prompts are used to generate images from four models for comparison.
2.1 Annotation Framework
- Annotators compare image pairs on style preference, coherence, or text-to-image alignment, with a separate question for each criterion.The alignment task asks which image better reflects the caption.Source: §2.1 · p. 2
2.1.1 Interface Design
- The interface omits prompts for preference and coherence, displays them word-by-word for alignment, and includes validation tasks.Sessions group multiple tasks; preference or coherence sessions average 12.5 seconds, versus about 15 seconds for two alignment tasks.Source: §2.1.1 · pp. 2–3
2.2 Distribution
- Mobile-app task distribution provides access to annotators across geographic regions and cultural backgrounds, supporting scalability and cost-effective collection.Tasks are made available to a global pool of users.Source: §2.2 · p. 3
2.3 Quality Assurance
- Quality assurance combines minimum task times and validation questions, with unsuccessful validation potentially leading to review or disqualification.Tasks are presented in the user’s preferred device language.Source: §2.3 · pp. 3–4
2.4 Prompts
- The prompt set builds on DrawBench and adds complementary benchmarks to broaden coverage of general, complex, subjective, structural, and celebrity-description prompts.The added sources include DiffusionDB, ABC-6K, HRS-Bench, T2I-CompBench, and DALLE3-EVAL.Source: §2.4 · p. 4
2.5 Image Generation
- All four models receive the same prompts; generation uses APIs for DALL-E 3, MidJourney, and Flux.1, and Hugging Face for Stable Diffusion.This standardized setup is intended to support direct comparison of prompt interpretation and rendering.Source: §2.5 · p. 4
3 Results1
Across three criteria, Flux.1 ranked highest, while results also reveal model-specific strengths and weaknesses. The study derived rankings from over two million votes and reported annotator demographics broadly approximating world population, with regional skews.
3.1 Data
- More than 2 million votes came from 144,292 participants across 145 countries, with 26 votes collected per comparison.Across 282 prompts, four images per model and prompt yielded 27,072 comparisons per criterion.Source: §3.1 · p. 4
3.2 Ranking Algorithm
- Iterative Bradley-Terry converts completed pairwise votes into probabilistic model scores, normalized to sum to 100 because scores are arbitrary up to a common factor.Source: §3.2 · pp. 4–5
- An error corrupted 19 Stable Diffusion prompts, affecting about 1.5% of comparisons; the authors consider the overall effect negligible and general takeaways unchanged.Source: §3.2 · p. 4
3.3 Ranking
- Flux.1 led style preference (29.86), coherence (29.61), and text-to-image alignment (27.36); Stable Diffusion scored 21.99 in style, 24.09 in coherence, and 21.4 in alignment.Source: Figure 3 · p. 5, §3.3 · p. 5, Table 1 · p. 6
- Authors recommend Flux.1 as the default; DALL-E 3 may suit odd or abstract alignment prompts, MidJourney was never worst, and Stable Diffusion stayed competitive in coherence (24.09).Source: §3.3 · p. 5
3.4 Demographic Statistics
- Age coverage skewed younger but was otherwise described as quite even.Source: Figure 7 · p. 6, §3.4 · p. 6
- Gender responses were evenly split between men and women, but many selected another category or did not answer.Source: Figure 8 · p. 6, §3.4 · p. 6
- Continent-level coverage broadly tracked world population, with Africa overrepresented, Asia slightly underrepresented, and North America underrepresented against Europe's slight overrepresentation.Source: §3.4 · p. 6, Figure 6 · p. 7
4 Conclusion
The authors present an efficient, representative, scalable framework and benchmark for evaluating image-generation models. They report collecting more than 2 million responses in a few days and propose that the approach could bring human feedback into model training.
4 Conclusion
- The annotation framework and benchmark provide an efficient, representative, scalable way to comprehensively evaluate image-generation models.Source: §4 · p. 7
- In a few days, the authors cost-effectively collected more than 2 million responses, which they describe as the largest image-generation evaluation by a significant margin.The authors qualify the size comparison with “to the best of our knowledge.”Source: §4 · p. 7
- The authors say the scale and efficiency open the door to incorporating human feedback directly into image-generation training.They cite reinforcement learning from human feedback as a significant driver of progress in large language models.Source: §4 · p. 7