PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Diversity in Image Generation via Attribute-Conditional Human Evaluation

Isabela Albuquerque, Ira Ktena, Olivia Wiles, Ivana Kajić, Amal Rannen-Triki, Cristina Vasconcelos, Aida Nematzadeh

arXiv:2511.10547v1cs.CVcs.LG

TL;DR

T2I diversity evaluation remains difficult because measurement conditions are poorly standardized and existing evaluations may not explicitly control for variation. This paper introduces an attribute-specific prompt and human-evaluation framework, then applies it to compare models and automated metrics. Human evaluations rank Imagen 3 and Flux 1.1 as significantly better or not worse than the other models.

  • Problem

    Diversity metrics can conflate diversity with fidelity, and evaluation conditions often fail to specify or control the variation being assessed.

    Source: §1 · pp. 1–2

  • Method

    The framework pairs concepts with relevant attributes, elicits them through prompts, and uses a human template with counting and count-comparison subtasks.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    Human ratings identify Imagen 3 and Flux 1.1 as significantly better or not worse than all other models; Imagen 2.5 and Muse 2.2 are not significantly better than any contender.

    Source: §3.1 · p. 7

  • Takeaways & Limitations

    Attribute-specific human judgments support model comparisons, while revealing that automated evaluation is ineffective when diversity is confounded by changing aspects such as background.

    Source: §3.1 · p. 7, §3.2 · p. 9

    Automated evaluation is not effective at identifying diversity when aspects are confounded, such as when backgrounds change while the animal stays the same.

    Source: §3.2 · p. 9

Abstract

from arXiv · show

Despite advances in generation quality, current text-to-image (T2I) models often lack diversity, generating homogeneous outputs. This work introduces a framework to address the need for robust diversity evaluation in T2I models. Our framework systematically assesses diversity by evaluating individual concepts and their relevant factors of variation. Key contributions include: (1) a novel human evaluation template for nuanced diversity assessment; (2) a curated prompt set covering diverse concepts with their identified factors of variation (e.g. prompt: An image of an apple, factor of variation: color); and (3) a methodology for comparing models in terms of human annotations via binomial tests. Furthermore, we rigorously compare various image embeddings for diversity measurement. Notably, our principled approach enables ranking of T2I models by diversity, identifying categories where they particularly struggle. This research offers a robust methodology and insights, paving the way for improvements in T2I model diversity and metric development.

1. Measuring diversity in text-to-image models

The paper argues that T2I diversity evaluation needs to specify both the concept and the attribute being assessed, since existing metrics can conflate diversity with fidelity and prior evaluations often leave variation uncontrolled. It introduces an attribute-specific human-evaluation framework and uses it to assess automatic metrics and compare models.

1. Measuring diversity in text-to-image models

  • Existing diversity metrics can conflate diversity with fidelity, while prior evaluations often used prompts or human templates that did not explicitly probe diversity.The authors argue that these practices can make diversity measurements ambiguous or inconclusive.Source: §1 · pp. 1–2
  • The framework combines concept-and-attribute specification, a prompt set, and a human template that divides judgments into counting and count comparison.Its prompt set covers 86 concept–factor pairs, and the study collected 24,591 human annotations comparing five models.Source: Figure 1 · p. 1, §1 · p. 2
  • Imagen 3 and Flux 1.1 are identified as top-performing for attribute diversity among five compared T2I models.The study also reports that Vendi Score captures human diversity judgments with approximately 65% accuracy in an appropriate representation space, rising to 80% for more different model pairs.Source: §1 · p. 2

2. The three ingredients for diversity evaluation

The framework defines diversity around relevant attributes of specified concepts, builds prompts to elicit those variations, and uses a validated human template to compare them. Its golden-set results show that specifying the attribute improves annotation accuracy, with count anchoring especially helpful for larger sets.

2.1. A clearly specified problem: Diversity per attribute

  • Perfect coverage across every generative factor is intractable, so the framework measures a selected subset of salient, practically relevant factors for each concept.The authors use Gemini 1.5 M to suggest relevant variations, while noting that relevance depends on the use case.Source: §2.1 · p. 3

2.2. A systematically generated prompt set

  • LLMs generate common, visualizable concepts and associated attributes, which are manually checked to form prompts for contextually meaningful variation.Examples include apple–color, tree–species, coffee cup–material, and chair–style; five ambiguous pairs were removed.Source: §2.2 · p. 4

2.3. A validated, bespoke human evaluation template

  • The template compares image sets side by side and makes the target attribute explicit, balancing perceptual judgment, annotation fatigue, and metric requirements.Its golden set tests cases where the concept stays fixed, the attribute stays fixed, or both vary.Source: §2.3 · pp. 4–5
  • 77.9% accuracy at set size 8 with count anchoring, compared with 26.7% without aspect specification; aspect specification alone reaches 53.3%.For set size 4, the corresponding accuracies are 82.5% with the aspect specified and 30.0% without it.Source: Figure 3 · p. 5, §2.3 · p. 5
  • Annotators’ final comparisons correlate with their subset-count inferences (ρ = 0.88, p < .001), supporting count anchoring as a guide for the judgment.The golden-set count distributions generally aligned with labels, although chairs prompted as different materials were sometimes judged visually similar.Source: §2.3 · p. 5
  • Figure 4 displays counts for image sets labeled diverse or non-diverse; expected modes are 8 and 1, respectively.The analysis notes exceptions where prompted variations did not appear visually distinct to annotators.Source: §2.3 · p. 5, Figure 4 · p. 6

3. Our framework in practice

The framework is applied to compare five text-to-image models using human annotations, then to assess automatic diversity metrics against those annotations. Human rankings distinguish model pairs, while automatic rankings vary with embedding choice and conditioning.

3. Our framework in practice

  • The framework uses its human annotations as ground truth to evaluate diversity metrics and compare automatic rankings with human rankings.The Vendi Score aggregates image embeddings into set-level diversity predictions; embeddings may use images alone or be conditioned on text.Source: §3 · p. 6, §3.2 · pp. 7–8

3.1. Ranking models via human evaluation

  • The study compares Muse 2.2, Imagen 2.5, Imagen 3, DALLE3, and Flux 1.1 using 20 samples per prompt and side-by-side sets of eight images.It collected 24,591 annotations from 20 annotators, including pilot runs; agreement exceeded α = 0.8 in every case.Source: §3.1 · p. 6
  • For each model pair, ratings are aggregated first across raters by mode and then across side-by-side comparisons for each concept.The resulting human evaluation provides one comparison result per concept for that model pair.Source: §3.1 · p. 6
  • Imagen 3 and Flux 1.1 are significantly better or not worse than all other models; DALLE3 significantly surpasses Imagen 2.5 but not the other compared models.Imagen 2.5 and Muse 2.2 are not significantly better than any contender.Source: §3.1 · p. 7

3.2. Comparing autoevaluation metrics

  • All autoraters perform similarly on the full model data, but are more accurate on the roughly 24% subset with larger human-perceived diversity gaps.On the golden set, ViT performs best; that advantage does not transfer to side-by-side comparisons.Source: §3.2 · p. 8, Figure 6 · p. 9
  • Automatic metrics perform better on the subset with larger perceived gaps, indicating weaker performance when compared image sets are more similar.Inception features can identify low-diversity cases associated with mode collapse but are ineffective when a non-target attribute changes, such as background while the animal remains the same.Source: §3.2 · pp. 8–9, Figure 6 · p. 9

3.3. Ranking models with autoevaluation approaches

  • Across the tested embeddings, Imagen 3 is not worse than any other model, while Flux 1.1, Imagen 3, and DALLE3 outperform Imagen 2.5 and Muse 2.2.Inception and object-and-attribute-conditioned PALI distinguish more among the top three models; both PALI conditioning choices distinguish Imagen 2.5 from Muse 2.2.Source: §3.3 · p. 10, Figure 8 · p. 10

4. Related work

Prior T2I evaluation work uses human judgments and automated metrics, but diversity measurement has lacked consistent, targeted validation. The paper situates its framework as a way to evaluate diversity across specific attributes and compare metrics and models under varied conditions.

4. Related work

  • Benchmark composition significantly influences T2I model rankings, while human evaluation remains the gold standard.Existing benchmarks cover broad skills such as text rendering and spatial reasoning, as well as targeted skills such as numerical reasoning.Source: §4 · p. 10
  • Prior diversity metrics have often been tested only on generic prompts or simplified settings, and most studies lack human evaluation validating them.The literature includes reference-based and reference-free metrics, with the Vendi Score influencing subsequent work (Friedman and Dieng, 2022; Hemmat et al., 2024; Jalali et al., 2024; Kannen et al., 2024a).Source: §4 · p. 11
  • The paper introduces an attribute-focused prompt set and validates a human-evaluation template for gathering ground-truth diversity judgments, then compares existing diversity metrics and text-to-image models under varied conditions.The authors note that proposed metrics had not undergone thorough evaluation and were often tested only on generic prompts or in simplified settings. Most previous studies lacked human evaluation.Source: §4 · p. 11

5. Discussion

The discussion presents an attribute-specific human-evaluation framework and uses its judgments to assess model diversity and automated metrics. Imagen 3 and Flux 1.1 are identified as strong performers, while embedding choice is important to automated metric alignment with human judgments.

5. Discussion

  • Imagen 3 and Flux 1.1 are strong performers in the paper’s attribute-specific diversity ranking.The template’s explicit definition of the attribute of interest significantly improves the accuracy of human judgments.Source: §5 · p. 11
  • Vendi Score-based approaches capture human-perceived diversity with approximately 80% accuracy, and embedding-space choice is crucial to broad alignment with human judgments.Pairwise model comparisons also yield similar results when comparable statistical analysis is used.Source: §5 · p. 11
  • The framework and collected data are intended to support future work on T2I diversity and more reliable evaluation metrics.The methodology could also be used in future research to measure demographic diversity without attribute classifiers.Source: §5 · p. 11

6. Ethics Statement

The study collected human annotations from 20 participants, compensated according to the minimum wage in their geographical location.

6. Ethics Statement

  • 20 participants were compensated according to the minimum wage in their geographical location.Annotators received comprehensive instructions and could take as much time as necessary to complete the task.Source: §6 · p. 11
  • Annotators received comprehensive instructions before beginning the task.They could take as much time as necessary to complete it.Source: §6 · p. 11
  • Compensation was based on the minimum wage in each participant’s geographical location.Source: §6 · p. 11

A. Human evaluation task details

Annotators compared two image sets for diversity with respect to a specified attribute, using observed attribute variations to make a judgment. The appendix also reports the annotation count and average completion time for the final template.

A.1. Instructions

  • Annotators compared the number of distinct instances of a specified attribute in two image sets and judged which set was more diverse.The response options were left more diverse, right more diverse, equally diverse, or unable to answer.Source: §A.1 · p. 15
  • Written instructions included examples corresponding to three response options: one set more diverse, the other set more diverse, or equally diverse.Source: §A.1 · p. 15

A.2. Additional information

  • 24,591 annotations were collected, including pilot runs, and the final-template task took an average of 32 seconds.Source: §A.2 · p. 15

B. Human evaluation template

The section describes a human-evaluation template for attribute diversity and a golden set used to validate it. The appendix illustrates the concept–attribute pairs, image-generation protocol, and annotation interface.

B.1. Golden set concept-attribute pairs

  • The 10 pairs cover everyday concepts and aspects such as flower color, container material, animal background, clothing pattern, and building style.The full list of concepts and variation values is presented in Table 1.Source: §B.1 · p. 15, Table 1 · p. 16
  • The golden set contains 10 concept–aspect pairs, tested in three configurations: fixed concept with varying aspect, varying concept with fixed aspect, and both varying.Each concept contributes 24 images, for 240 images total.Source: §B.1 · p. 15
  • Golden-set images were generated with Imagen 3 using a photorealistic prompt that combines the variation value and concept, such as “a yellow begonia.”The synthetic images support comparisons of human annotators and multimodal-model autoraters on specific aspects of variation.Source: §B.1 · pp. 15–16

B.2. User interface screenshots

  • Figure 9 shows pilot templates with and without an aspect prompt; the count variant additionally asks annotators to report distinct target-attribute values in each set.Figures 10–12 show interface examples for the no-aspect, aspect, and count conditions.Source: Figure 9 · p. 17
  • Figures 10 and 11 each show a user-interface screenshot of one annotation example, for the “No aspect” and “Aspect” conditions, respectively.Source: Figure 10 · p. 18, Figure 11 · p. 18
  • A further interface screenshot illustrates the “Count” condition.In this variant, annotators specify how many distinct target-attribute values appear in the set.Source: Figure 9 · p. 17, Figure 12 · p. 18

C. Additional human evaluation results

The appendix reports the distribution of human-annotated counts across side-by-side image comparisons.

C. Additional human evaluation results

D. A deep dive on our curated Prompt set generation details

The appendix details prompt-set construction and tests how prompt-set size, embedding choice, and textual conditioning relate to diversity evaluation. It also reports qualitative examples and compute setup.

D.1. Prompt set generation

  • The prompt-generation instructions request at least 95 realistic-image cases, each pairing a main object with an aspect to evaluate, and exclude categories involving people.Examples include apple–color, book–thickness, and bridge–shape.Source: §D.1 · p. 19

D.2. On the sufficiency of the prompt set for discriminating models

  • Reducing the prompt set to 74 concepts leaves model-comparison results unchanged, while further reductions decrease significant pairwise comparisons and can obscure differences such as Imagen 3 versus Imagen 2.5.Even the smallest set does not reverse the model ordering.Source: §D.2 · p. 20, Table 2 · p. 20

E.1. Compute Usage

  • The evaluation metrics ran on TPU V3 hardware, while image generation used four TPUs.The appendix reports these compute details without specifying further resource totals.Source: §E.1 · p. 20

E.2. Performance for detecting equally diverse image sets

  • All tested metrics have AUC < 0.6 for identifying equally diverse sets; ImageNet Inception performs best, but the authors hypothesize it may struggle with confounders such as object type when measuring color diversity.Figure 14 presents the AUC comparison.Source: §E.2 · pp. 20–21, Figure 14 · p. 21

E.3. Additional qualitative results

  • The authors examine four side-by-side comparisons in which autoraters identify image groups with the highest or lowest diversity.Figure 15 visualizes these qualitative examples.Source: §E.3 · p. 21

E.4. Impact of the prompt for the multimodal embeddings

  • More related textual prompts do not consistently improve multimodal-embedding results, which the authors interpret as evidence that the embeddings are mostly vision-based.The tested templates vary in specificity and include object-and-attribute, attribute-only, object-only, and unrelated questions.Source: §E.4 · p. 21

E.5. Model ranking with autoevaluation approaches

The appendix extends embedding-based model rankings and evaluates Gemini autoraters against human judgments. Rankings vary with embedding and conditioning choices, while the latest Gemini version exceeds human accuracy on the golden-set task.

E.5. Model ranking with autoevaluation approaches

  • The authors report that embedding choice and conditioning influence model rankings.Figures 20–22 show win rates and score distributions for image models and multimodal models conditioned on attributes or on objects and attributes.Source: §E.5 · p. 22, Figure 20 · p. 25, Figure 21 · p. 26
  • Across embeddings except ImageNet ViT, Imagen3 is not worse than all other models; Flux1.1, Imagen3, and DALLE3 are not worse than Muse2.2 and Imagen2.5 across embeddings.The extent to which embeddings detect differences between the top and bottom groups varies by embedding.Source: §E.5 · pp. 21–22, Figure 17 · p. 23, Figure 18 · p. 24

E.6. Evaluating diversity using foundation models

  • Gemini v2.5 Flash achieves the highest golden-set accuracy among the three tested Gemini models and surpasses human raters on this task.The task compares diversity in side-by-side sets of eight images.Source: Figure 23 · p. 28, §E.6 · p. 28
  • Human and automatic raters perform similarly in almost all golden-set cases, with mismatches for the building–style pair.The authors hypothesize that judging architectural styles is complex and depends on annotators’ cultural backgrounds.Source: §E.6 · p. 28

F. Absence of a Diversity-Fidelity Trade-off

The evaluated models had the same average Gecko alignment score, while Imagen 3 was significantly better in diversity than Imagen 2.5 and Muse 2.2. The section frames meaningful diversity evaluation as conditional on sufficient image quality.

F. Absence of a Diversity-Fidelity Trade-off

  • A model could appear highly diverse by producing random noise, so diversity assessment assumes generated images have sufficient quality.The quality criterion requires visually coherent images without significant artifacts that capture the prompt’s salient aspects and intent.Source: F. Absence of a Diversity-Fidelity… · pp. 28–29
  • Without quality and prompt adherence, a high diversity score may indicate poor control and semantic understanding rather than a beneficial range of outputs.Source: F. Absence of a Diversity-Fidelity… · p. 29
  • Models had the same average Gecko score, while Imagen 3 was significantly better in diversity than Imagen 2.5 and Muse 2.2.Gecko measures text-to-image alignment; higher is better and 1 is the maximum. The reported alignment results were not statistically different.Source: Table 3 · p. 29, F. Absence of a Diversity-Fidelity… · p. 29
Loading 2511.10547v1…