PaperlayerView exact revision on arXiv

Source-linked AI summary

GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation

Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, Deva Ramanan

arXiv:2406.13743v3cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

Text-to-visual models can produce realistic images and videos but struggle with compositional prompts involving attributes, relationships, and advanced reasoning. The paper evaluates these models and alignment metrics using GenAI-Bench, finding that VQAScore supports more effective image ranking than competing scoring methods.

  • Problem

    Compositional prompts involving attributes, relationships, and higher-order reasoning challenge text-to-visual models, while existing metrics such as CLIPScore do not reliably measure alignment for such prompts.

    Source: §1 · p. 1

  • Method

    The paper uses GenAI-Bench human ratings to evaluate image and video generators and alignment metrics, and tests black-box image selection by ranking candidates with VQAScore.

    Source: §1 · pp. 2–3

  • Results

    Ranking 3 to 9 images with VQAScore improves DALL-E 3 and SD-XL human ratings by 0.2 to 0.3 and outperforms other ranking metrics by 2x to 3x.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that VQAScore is a straightforward ranking method that improves prompt alignment for black-box generative models.

    Source: §1 · p. 3, §8 · p. 9

    VQAScore can miscount when objects are numerous, overlook fine-grained visual details, and misinterpret linguistic ambiguity.

    Source: §7 · p. 9

Abstract

from arXiv · show

While text-to-visual models now produce photo-realistic images and videos, they struggle with compositional text prompts involving attributes, relationships, and higher-order reasoning such as logic and comparison. In this work, we conduct an extensive human study on GenAI-Bench to evaluate the performance of leading image and video generation models in various aspects of compositional text-to-visual generation. We also compare automated evaluation metrics against our collected human ratings and find that VQAScore -- a metric measuring the likelihood that a VQA model views an image as accurately depicting the prompt -- significantly outperforms previous metrics such as CLIPScore. In addition, VQAScore can improve generation in a black-box manner (without finetuning) via simply ranking a few (3 to 9) candidate images. Ranking by VQAScore is 2x to 3x more effective than other scoring methods like PickScore, HPSv2, and ImageReward at improving human alignment ratings for DALL-E 3 and Stable Diffusion, especially on compositional prompts that require advanced visio-linguistic reasoning. We release a new GenAI-Rank benchmark with over 40,000 human ratings to evaluate scoring metrics on ranking images generated from the same prompt. Lastly, we discuss promising areas for improvement in VQAScore, such as addressing fine-grained visual details. We will release all human ratings (over 80,000) to facilitate scientific benchmarking of both generative models and automated metrics.

1 Introduction

The paper introduces GenAI-Bench to assess compositional text-to-visual generation and automated alignment metrics, addressing the limits of existing benchmarks and bag-of-words metrics. It reports that VQAScore improves image selection and that generated visuals still struggle with advanced reasoning prompts.

Motivation

  • Traditional benchmarks and metrics are insufficient for evaluating compositional prompts involving attribute bindings, object relationships, and logical reasoning.The paper identifies CLIPScore as a bag-of-words metric that cannot reliably measure alignment on such prompts.Source: §1 · p. 1

Evaluating text-to-visual generation

  • GenAI-Bench uses 1,600 designer-sourced prompts covering basic and advanced compositional skills, extending beyond benchmarks centered on attribute binding and object relationships.The benchmark tags prompts with evaluated skills and includes human ratings for both image and video models.Source: §1 · p. 1, Table 1 · p. 3
  • DALL-E 3 averages 4.3 on basic prompts but 3.4 on advanced prompts, indicating several discrepancies on prompts requiring higher-order reasoning.The study collects 38,400 human ratings across images and videos generated by ten models.Source: §1 · p. 2

Evaluating automated metrics

  • VQAScore significantly surpasses previous alignment metrics in correlation with human judgments.It scores the likelihood that a VQA model answers “Yes” to whether an image shows the prompt.Source: §1 · p. 2

Improving generation with VQAScore

  • Ranking 3 to 9 candidate images with VQAScore improves DALL-E 3 and SD-XL human ratings by 0.2 to 0.3 and surpasses other ranking metrics by 2x to 3x.The ranking method is black-box and requires no finetuning.Source: §1 · p. 3

Limitations and contributions

  • The authors identify fine-grained visual details and linguistic ambiguity as limitations of VQAScore.They also introduce GenAI-Rank, with over 40,000 human ratings for ranking images generated from the same prompt.Source: §1 · p. 3

2 Related Works

Prior text-to-visual benchmarks and metrics assess image quality or compositional alignment, but each has limitations in prompt complexity or alignment measurement. GenAI-Bench builds on recent compositional benchmarks by emphasizing advanced reasoning in practical user prompts.

Text-to-visual benchmarks

  • Early benchmarks use captions from existing datasets and focus on simple objects, attributes, and scenes, while HPDv2 and Pick-a-pic primarily assess image aesthetics with simpler prompts.DrawBench, PartiPrompt, and T2I-CompBench shift toward compositional text-to-image evaluation, emphasizing attribute bindings and object relationships.Source: §2 · pp. 3–4
  • GenAI-Bench extends compositional benchmarking with practical prompts that require advanced reasoning such as logic and comparison.Earlier benchmarks emphasize objects, attributes, and relations.Source: §2 · pp. 3–4

Automated metrics

  • CLIPScore measures cosine similarity between embedded images and prompts, but CLIP’s bag-of-words encoding is unreliable for compositional prompts.Other approaches use human-feedback-trained preference models or LLM prompt decomposition into simpler question-answer pairs.Source: §2 · p. 4

3 GenAI-Bench for Text-to-Visual Evaluation

GenAI-Bench combines designer-created prompts with a taxonomy spanning basic visual compositions and advanced reasoning skills. Its prompt tagging supports fine-grained evaluation across a balanced mix of skill types.

Skill taxonomy

  • The taxonomy covers basic objects, scenes, attributes, and spatial, action, or part relations, alongside advanced counting, comparison, differentiation, negation, and universality.Advanced compositions can count complex object-attribute-relation combinations or distinguish objects of the same category by varied properties.Source: §3 · p. 4

GenAI-Bench

  • The 1,600 prompts come from professional designers and exclude subjective or toxic content, celebrities, and copyrighted characters.Designers are instructed to use generic subjects such as food, vehicles, people, pets, plants, household items, and mythical creatures.Source: §3 · p. 4
  • GenAI-Bench provides over 5,000 human-verified skill tags, with about half of prompts containing only basic compositions and half combining basic and advanced skills.Each prompt is tagged with all evaluated skills, unlike prior benchmarks that provide no tags or only one or two.Source: §3 · p. 4

4 Human Evaluation via GenAI-Bench

The human evaluation rates image-text and video-text alignment for ten popular generative models using a five-point Likert scale. Ratings show that advanced reasoning prompts are harder, and that leading image models outperform open-source and video models.

Human evaluation

  • The study evaluates six text-to-image models and four text-to-video models, with three annotators rating image-text or video-text alignment on a 1-to-5 Likert scale.The scale ranges from no match to an exact match.Source: §4 · pp. 4–5

Analysis

  • Open-source and video-generative models lag behind closed-source and image-generative counterparts, while advanced prompts such as negation and comparison challenge models more.The figure reports average human alignment ratings across ten models.Source: Figure 3 · p. 5

Human evaluation

  • Inter-rater agreement reaches Krippendorff’s Alpha 0.72 for image ratings and 0.70 for video ratings.The scale makes ratings interpretable: a score near 5 means generated images almost always match prompts exactly.Source: §4 · p. 5

Analysis

  • Advanced prompts are harder for all models: DALL-E 3 scores 4.3 on basic prompts and 3.4 on advanced prompts.Models using stronger text embeddings, including DeepFloyd-IF and DALL-E 3, outperform models using CLIP text embeddings such as SD-XL.Source: §4 · p. 5

5 Evaluating Automated Metrics

The section evaluates automated alignment metrics against human ratings on GenAI-Bench and reports VQAScore as the strongest-performing metric. It defines VQAScore through a VQA model’s likelihood of answering “Yes” to whether an image depicts the prompt.

Evaluation setup

  • The authors benchmark automated alignment metrics against their collected human ratings on GenAI-Bench.The passage identifies the purpose of the evaluation but gives no additional setup details.Source: §5 · p. 5

Results

  • VQAScore achieves the strongest agreement with human ratings on GenAI-Bench images and videos, significantly surpassing CLIPScore, PickScore, and Davidsonian.The authors also report best performance across basic and advanced skills and state-of-the-art results on seven additional alignment benchmarks.Source: Table 2 · p. 6, §5 · p. 6

Automated metric

  • VQAScore measures the likelihood that a VQA model answers “Yes” to a question asking whether an image shows the prompt.The study implements it with the open-source CLIP-FlanT5 model; video-text scores are averaged across video frames.Source: §5 · p. 6

Evaluation setup

  • The evaluation compares automated metric scores with human ratings using Pearson and Kendall correlations and pairwise accuracy with tie calibration.Pairwise accuracy measures agreement on whether pairs of items score higher, lower, or tied in both human and metric ratings.Source: §5 · p. 6

6 Improving Text-to-Visual Generation

The section tests whether VQAScore can improve text-to-image generation by ranking candidate images from the same prompt. It reports gains on challenging prompts and introduces GenAI-Rank for comparing ranking metrics.

Ranking images by VQAScore

  • The black-box approach selects the highest-VQAScore image from a few candidates generated for the same prompt, without finetuning.The approach needs only an image-generation API.Source: §6 · p. 6
  • From three candidates, VQAScore can select SD-XL images that outperform DALL-E 3, and it can also improve the selection of DALL-E 3 images.The figure comparison highlights DALL-E 3 prompts involving attributes, relationships, and higher-order reasoning.Source: Figure 5 · p. 7, §6 · p. 7, Figure 7 · p. 8

GenAI-Rank

  • GenAI-Rank collects 43,200 human ratings for 14,400 DALL-E 3 and SD-XL images generated from 800 prompts.Three annotators rated nine generated images per prompt; the benchmark is intended to support evaluation of future ranking metrics.Source: §6 · p. 7

VQAScore achieves superior performance gains

  • CLIPScore ranking can lead to lower performance, while VQAScore improves consistently as more candidate images are ranked.Figure 6 reports gains and decreases relative to the Random baseline; its caption also compares VQAScore with PickScore and Davidsonian.Source: §6 · p. 7, Figure 6 · p. 7
  • With nine images, VQAScore gains 0.30 for DALL-E 3 and 0.27 for SD-XL on advanced prompts, outperforming PickScore by 2x to 3x.Advanced prompts involve complex visio-linguistic reasoning such as counting, comparison, and logic.Source: §6 · p. 7, Table 3 · p. 8

7 Goodhart’s Law Still Applies

The section examines how VQAScore may lose effectiveness when optimized as a target and identifies limits in its handling of visual details and linguistic ambiguity.

Goodhart’s Law

  • The authors connect these limitations to Goodhart’s Law and examine the possibility that over-optimizing an automated metric can reduce its effectiveness.They suggest higher-resolution VQA models and more capable language models as possible avenues for improvement.Source: §7 · p. 9

Limitations of VQAScore

  • VQAScore has three reported failure cases: miscounting many objects, overlooking fine visual details, and misinterpreting ambiguous language.Examples include confusing “two shoes” with “two pairs of shoes” and misreading “left” relative to the viewer versus an object.Source: §7 · p. 9, Figure 8 · p. 9
  • The authors recommend VQAScore as a reproducible supplement to human studies and as a more reliable alternative to CLIPScore.They characterize the limitations as mild while noting CLIPScore’s declining effectiveness.Source: §7 · p. 9

8 Conclusion

The conclusion presents a human study of compositional text-to-visual generation and automated evaluation, alongside a ranking-based method for improving black-box models. It also identifies benchmark omissions and future work on ranking efficiency.

Limitations and future work

  • GenAI-Bench does not evaluate toxicity, bias, aesthetics, or video motion, among other aspects of generative models.Future work may add mixed media, optical effects, reflection, and world knowledge; it may also explore white-box finetuning for more efficient inference.Source: §8 · p. 9

Summary

  • The paper presents a straightforward ranking-based method that improves prompt alignment for black-box generative models.The conclusion frames further research into automated evaluation as important to the scientific progression of the field.Source: §8 · p. 9

Outline

The supplement outlines additional examples and a unified taxonomy for compositional skills, then uses that taxonomy to compare benchmark coverage. Its SD-XL examples show VQAScore ranking candidate images better than CLIPScore, particularly for advanced prompts.

Outline

  • The supplement covers additional qualitative studies, GenAI-Bench’s skill taxonomy and collection, and other baseline methods.The outline identifies these as Sections A–D.Source: Outline · p. 15

A Additional Examples

  • VQAScore ranks SD-XL candidate images better than CLIPScore, particularly on prompts requiring complex visio-linguistic reasoning.Ranking by VQAScore also improves SD-XL prompt alignment using only its image-generation API.Source: Figure 9 · p. 15, A Additional Examples · p. 15

B Skill taxonomy of GenAI-Bench

  • The taxonomy separates basic compositions from advanced reasoning skills, including negation and universality, while leaving image styles for future work.The authors omit conjunction because AND is usually implied in multi-object prompts, and OR is rarely useful to designers.Source: B Skill taxonomy of GenAI-Bench · p. 16
  • Benchmark coverage is compared by mapping predefined categories or manually annotating samples to a unified skill set, counting skills above 2% as present.The authors identify arbitrary category assignments in prior benchmarks as a motivation for this unified comparison.Source: B Skill taxonomy of GenAI-Bench · p. 16

C GenAI-Bench

GenAI-Bench uses designer-created prompts to assess image and video models across compositional skills, and its results show that advanced reasoning remains challenging. The section also compares automated metrics and details baseline approaches, including their limitations on compositional prompts.

C GenAI-Bench

  • GenAI-Bench’s 1,600 prompts were collected with professional designers, who refined the skill taxonomy and wrote objective prompts for generic subjects.The designers were guided to avoid subjective phrasing, and prompts received over 5,000 human-verified skill tags with balanced basic and advanced coverage.Source: C GenAI-Bench · p. 17
  • The study evaluates six image and four video models, using all 1,600 prompts for images and a coreset of 800 for videos.The authors report over 80,000 human ratings across the study and GenAI-Rank, collected over about 800 annotator hours.Source: C GenAI-Bench · pp. 17–18
  • DALL-E 3 averages 4.3 on basic prompts and 3.4 on advanced prompts; video models score significantly lower than image models.Human ratings and VQAScores generally favor DALL-E 3 across skills except negation, and VQAScores closely match human ratings.Source: Table 6 · p. 18, C GenAI-Bench · p. 18
  • VQAScore based on CLIP-FlanT5 or GPT4-o surpasses other metrics across GenAI-Bench skills, and CLIP-FlanT5 VQAScore outperforms prior art across seven other benchmarks.Automated metrics perform much worse on advanced prompts than on basic prompts.Source: Table 7 · p. 19, Table 8 · p. 20

D Details of Baseline Methods

  • VQAScore’s CLIP-FlanT5 implementation is reported on seven additional benchmarks measuring correlation with human judgments.Table 8 uses different metrics by benchmark, including Group Score, AUROC, pairwise accuracy, and binary accuracy.Source: D Details of Baseline Methods · p. 19, Table 8 · p. 20
  • Finetuned metrics such as PickScore, ImageReward, and HPSv2 use human feedback, while the authors identify possible bottlenecks in their discriminative VLMs and training data.The passage notes that human annotations may also be noisy, especially when annotators are not well trained.Source: D Details of Baseline Methods · pp. 19–20
  • VQ2 and Davidsonian generate question-answer pairs to evaluate alignment, but their generated pairs can be irrelevant or erroneous, especially for compositional prompts.Davidsonian skips a follow-up relation question when its VQA model already answers “No” to a prerequisite object-presence question.Source: D Details of Baseline Methods · p. 20, Table 9 · p. 20

Related papers

Loading 2406.13743v3…