PaperlayerView exact revision on arXiv

Source-linked AI summary

HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models

Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, Mohamed Elhoseiny

arXiv:2304.05390v2cs.CVcs.AIcs.LG

TL;DR

Existing T2I benchmarks have limited holistic coverage and often rely on subjective human evaluation. HRS-Bench introduces an automatic benchmark covering 13 skills across five categories and 50 scenarios, evaluates nine large-scale models, and finds recurring failures in object counting, visual text, and emotion grounding.

  • Problem

    Existing T2I benchmarks cover limited capabilities and rely heavily on subjective human evaluation, restricting holistic assessment.

    Source: §1 · p. 1

  • Method

    HRS-Bench evaluates 13 skills across accuracy, robustness, generalization, fairness, and bias over 50 scenarios using automatic metrics and human validation.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Evaluations of nine recent T2I models show frequent failures in desired object counting, visual text generation, and emotion-grounded images.

    Source: §1 · p. 2, §5.2 · p. 8, §5.3 · p. 8

  • Takeaways & Limitations

    HRS-Bench provides a holistic evaluation framework that identifies capability gaps in current T2I models and supports further research.

    Source: §6 · p. 9

Abstract

from arXiv · show

In recent years, Text-to-Image (T2I) models have been extensively studied, especially with the emergence of diffusion models that achieve state-of-the-art results on T2I synthesis tasks. However, existing benchmarks heavily rely on subjective human evaluation, limiting their ability to holistically assess the model's capabilities. Furthermore, there is a significant gap between efforts in developing new T2I architectures and those in evaluation. To address this, we introduce HRS-Bench, a concrete evaluation benchmark for T2I models that is Holistic, Reliable, and Scalable. Unlike existing bench-marks that focus on limited aspects, HRS-Bench measures 13 skills that can be categorized into five major categories: accuracy, robustness, generalization, fairness, and bias. In addition, HRS-Bench covers 50 scenarios, including fashion, animals, transportation, food, and clothes. We evaluate nine recent large-scale T2I models using metrics that cover a wide range of skills. A human evaluation aligned with 95% of our evaluations on average was conducted to probe the effectiveness of HRS-Bench. Our experiments demonstrate that existing models often struggle to generate images with the desired count of objects, visual text, or grounded emotions. We hope that our benchmark help ease future text-to-image generation research. The code and data are available at https://eslambakr.github.io/hrsbench.github.io

1. Introduction

Existing T2I benchmarks cover limited capabilities and often depend on subjective evaluation. HRS-Bench addresses this gap with 13 skills across five categories, 50 scenarios, automatic metrics, and evaluation of nine models.

  • Existing T2I models struggle with complex object and relationship composition, motivating evaluation beyond image fidelity [25] [37].Source: §1 · p. 1
  • HRS-Bench measures 13 skills grouped into accuracy, robustness, generalization, fairness, and bias across 50 scenarios.The benchmark includes applications such as fashion, animals, transportation, and food.Source: Figure 1 · p. 1, §1 · p. 2, Figure 2 · p. 2
  • The benchmark broadens evaluation through automatic metrics and assesses nine recent T2I models, including commercial and open-source systems.Source: §1 · p. 2
  • HRS-Bench targets capabilities that many prior evaluations omitted or explored narrowly, including creativity, fairness, anonymization, emotion-grounding, robustness, and visual-text generation.The authors note that DALL-EVAL used only 145 template-based prompts for social-bias evaluation [12].Source: §1 · p. 2
  • The evaluated models often fail to generate the desired object count, visual text, or grounded emotions.Source: §1 · p. 2

2. Revisiting Text-to-Image Benchmarks

Existing T2I benchmarks evaluate selected skills with limited coverage of models, prompts, objects, or evaluation dimensions. HRS-Bench is positioned as a broader benchmark spanning more capabilities and evaluation conditions.

DrawBench

  • DrawBench evaluates counting, composition, conflicting prompts, and writing using 200 prompts, extending assessment beyond image quality.It is described as an early effort to evaluate aspects beyond image quality.Source: §2 · p. 2

DALL-EVAL

  • DALL-EVAL covers visual reasoning and gender and racial bias, but evaluates only four models with 7k prompts, six annotators, and approximately 80 object classes.HRS-Bench covers more than 700 object classes.Source: §2 · p. 3

HE-T2I

  • HE-T2I proposes 32 benchmark aspects but evaluates only counting, shapes, and faces through human evaluation.Twenty annotators contributed to that evaluation.Source: §2 · p. 3

TISE

  • TISE evaluates positional alignment, counting, and fidelity using a collection of fidelity and alignment metrics.Its metrics include IS*, OIS, OFID, PA, and CA.Source: §2 · p. 3

3. HRS-Bench

HRS-Bench defines a broad set of T2I skills and constructs prompts across hardness levels using human-curated, template-based, and GPT-3.5-assisted procedures. Its categories cover fidelity, robustness, composition, creativity, fairness, and bias.

Accuracy skills

  • The benchmark measures counting, visual text, emotion grounding, and fidelity as distinct capabilities needed to align generated images with prompts.Visual-text evaluation targets readable text in complex scenes, while emotion evaluation targets emotion-grounded images.Source: §3.1.1 · p. 3

Robustness

  • Robustness covers invariance to paraphrases and typos, plus equivariance in spatial, attribute-specific, and action compositions.Source: §3.1.2 · p. 3

Generalization

  • Creativity is treated as out-of-distribution generation through prompts that are plausible but uncommon in standard training data.Easy, medium, and hard prompts use uncommon triplets, GPT-3.5 imaginative descriptions, and challenging counterproductive sentences.Source: §3.1.3 · p. 3, §3.2 · p. 4

Fairness and bias

  • The benchmark defines fairness as performance disparity among subgroups and studies gender and style attributes.Gender uses male and female categories, while styles include animation, real, sketch, black and white, and weather conditions.Source: §3.1.4 · p. 3

Prompt construction

  • Prompts are balanced across easy, medium, and hard levels, with half human-based and half template-based prompts; each skill uses 3k prompts.GPT-3.5 assists generation, while manual filtering and validation are used for selected prompt types.Source: §3.2 · p. 4

4. Evaluation for our Benchmark

HRS-Bench evaluates T2I models through detection-, alignment-, and miscellaneous metrics spanning composition, text, emotion, creativity, fairness, bias, and robustness-related capabilities.

Detection-Based Metrics

  • Detection-based metrics use UniDet outputs to measure object counting, spatial relations, attribute composition, and visual-text recognition.Counting uses precision, recall, and F1; spatial relations use predicted bounding-box geometry, while visual text uses CER and NED after detection and recognition.Source: §4.1 · pp. 4–5

Alignment-Based Metrics

  • The benchmark evaluates alignment through T2I, TIT, and I2I paradigms, with generated captions and augmented prompts providing alternative text-space comparisons.For TIT, an image captioner produces captions that are evaluated against the original prompt and GPT-3.5-generated paraphrases using CIDEr and BLEU.Source: §4.2 · p. 5, §4.2.2 · pp. 5–6
  • AC-T2I transforms generated images into captions, augments the reference prompt set, and scores caption–prompt similarity to evaluate T2I alignment beyond CLIP’s compositional limitations.The metric uses an image captioner followed by n-gram-based scoring with CIDEr and BLEU; the final score selects the highest prompt-caption alignment for each image.Source: §4.2.2 · pp. 5–6, §4.2.1 · p. 5

Miscellaneous

  • Fairness is measured by normalized pairwise disparities across subgroup accuracies, where lower fairness scores indicate smaller performance differences.The metric applies subgroup comparisons to gender and style fairness and uses AC-T2I for action composition.Source: §4.2.3 · p. 6
  • Creativity is assessed jointly through AC-T2I adherence and deviation from retrieved LAION training images using CLIP-based similarity.The deviation score treats similarity to nearest training images as a nearest-neighbor distance from the training dataset.Source: §4.2.4 · p. 6
  • Consistency and typo robustness compare images generated from original and perturbed prompts using visual similarity, while emotion and bias use classifiers or human ratings.Bias covers gender, race, and age, with detected facial attributes used to compute distribution skew.Source: §4.2.4 · p. 6, §4.3 · p. 6

5. Experimental Results

Across nine large-scale T2I models, HRS-Bench reveals strong weaknesses in counting, visual text, emotion grounding, and challenging composition, while models remain robust to language perturbations and largely fair. Human evaluations broadly align with the automated metrics.

Counting

  • Detailed prompts improve counting recall and F1-score, whereas Struct-Diff raises precision but sharply reduces recall and F1-score.Source: §5.2 · pp. 7–8

Fidelity and Bias

  • Fidelity remains below the accepted 80% threshold, with SD-V2 highest at 62.4% and minDALL-E lowest at 52.2%.Source: §5.2 · p. 8

Robustness and Generalization Results

  • Models remain robust to language perturbations, achieving 70%–82% alignment, with DALLE-V2 obtaining the best joint consistency and typo performance.Source: §5.3 · p. 8

Composition and Creativity

  • DALLE-V2 performs best for action composition, with CIDEr 1.4538 and human evaluation agreeing with the metric.Source: §5.3 · p. 8

Fairness and Bias Results

  • Models show negligible fairness differences but retain gender bias: CogView2’s maximum fairness gap is 3.5%, while SD-V1’s bias deviation is below 8%.DALLE-V2, CogView2, and minDALL-E average 20% gender deviation; models trained not to generate humans are excluded from the bias measure.Source: Figure 7 · p. 8, §5.4 · p. 9, Figure 9 · p. 9

Human Evaluation

  • Human evaluation correlates strongly with automated metrics, reaching 93% for counting F1, 98% for text detection, and 96% for recognition.Source: §5.5 · p. 9

6. Conclusion

HRS-Bench provides a comprehensive evaluation of T2I models across 13 skills and 50 applications, revealing weaknesses in recent large-scale models. Human evaluation confirms the benchmark’s effectiveness and reliability.

  • HRS-Bench measures 13 skills across five categories and 50 applications for holistic T2I evaluation.Source: §6 · p. 9
  • Evaluation of nine recent large-scale T2I models identifies areas where state-of-the-art systems struggle.Source: §6 · p. 9
  • Human evaluation results confirm the effectiveness and reliability of HRS-Bench.Source: §6 · p. 9
  • The benchmark is intended to support future T2I research and progress on the evaluated skills.Source: §6 · p. 9

A. Appendix

The appendix provides supplementary qualitative results, prompt-generation samples, and quantitative results.

  • The appendix contains qualitative results, prompt-generation samples, and quantitative results.Source: A. Appendix · p. 14

A.1. Qualitative results

The appendix presents ten qualitative-result figures, each showing a separate generated-image sample.

A.2. Prompt generation samples

The appendix documents prompt-generation procedures for the evaluated skills through descriptions and tables covering multiple prompt types and composition settings.

A.3. Quantitative results

The quantitative results cover counting, emotion, visual text, consistency, typos, fairness, gender bias, composition, actions, and creativity skills. Counting is additionally examined across easy, medium, and hard levels and through a prompt-detail ablation.

Related papers

Loading 2304.05390v2…