PaperlayerView exact revision on arXiv

Source-linked AI summary

GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment

Dhruba Ghosh, Hanna Hajishirzi, Ludwig Schmidt

arXiv:2310.11513v1cs.CVcs.LG

TL;DR

Text-to-image models are proliferating, but human evaluation is costly and common automated metrics do not provide fine-grained compositional analysis. GENEVAL addresses this gap with object-focused verification, aligns strongly with human judgments, and shows that complex spatial and attribute-binding tasks remain difficult.

  • Problem

    Human evaluation of increasingly numerous text-to-image models is costly to scale, while common automated metrics lack fine-grained compositional analysis.

  • Method

    GENEVAL uses object detection and downstream color classification to verify fine-grained properties across structured compositional tasks.

  • Results

    GENEVAL attained 83% agreement with annotators versus 88% interannotator agreement, while recent models improved overall but struggled with complex compositional tasks.

  • Takeaways & Limitations

    GENEVAL provides interpretable, fine-grained evaluation that can uncover text-to-image failure modes and inform future model development.

  • Takeaways & Limitations

    GENEVAL is limited by object-detector performance and available discriminative vision models, including restricted classes and weak generalization from photographs to visually distinct art.

Abstract

from arXiv · show

Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led to an explosion of text-to-image generative models. Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number of new models. However, most current automated evaluation metrics like FID or CLIPScore only offer a holistic measure of image quality or image-text alignment, and are unsuited for fine-grained or instance-level analysis. In this paper, we introduce GenEval, an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color. We show that current object detection models can be leveraged to evaluate text-to-image models on a variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color. We then evaluate several open-source text-to-image models and analyze their relative generative capabilities on our benchmark. We find that recent models demonstrate significant improvement on these tasks, though they are still lacking in complex capabilities such as spatial relations and attribute binding. Finally, we demonstrate how GenEval might be used to help discover existing failure modes, in order to inform development of the next generation of text-to-image models. Our code to run the GenEval framework is publicly available at https://github.com/djghosh13/geneval.

1 Introduction

GENEVAL addresses the limits of costly human evaluation and holistic automated metrics by using object-focused verification for fine-grained text-to-image assessment. It aligns strongly with human judgment while revealing persistent weaknesses in complex compositional generation.

  • Framework: GENEVAL uses object detection outputs to verify object presence, count, and relative position, then applies vision models to classify properties such as color.The framework is modular and interpretable because its verification stages provide fine-grained information about model capabilities.
  • Human agreement: 83% agreement with annotators was achieved across 6,000 fine-grained annotations over 1,200 images, compared with 88% interannotator agreement.Agreement rose to 91% on images with unanimous annotator judgments.
  • Human agreement: GENEVAL obtains greater human agreement than CLIPScore on complex tasks requiring compositional reasoning.CLIPScore produces a scalar alignment value, whereas GENEVAL verifies prompt elements before assigning a binary correctness score.
  • Model evaluation: IF-XL correctly generated 61% of images, compared with more than 50% for Stable Diffusion v2.1, while Stable Diffusion XL improved on some multi-object tasks but not counting.Larger model size improved some tasks, whereas increased pretraining time did not necessarily improve performance.
  • Model evaluation: The best tested models correctly generated only 15% of relative-position images and 35% of attribute-binding images.These results identify spatial relations and attribute binding as persistent difficulties for current text-to-image models.

2 Related work

Prior automated evaluations mainly measure image quality, image-text alignment, or holistic human preference, while specialized object-detection and VQA approaches cover narrower or less interpretable aspects. GENEVAL uses modern vision models to broaden task coverage without task-specific detector training and retains detailed per-object outputs.

  • Automated evaluation: Inception Score and FID evaluate image quality independently of the text prompt, while CLIPScore measures prompt-image embedding alignment.Reference-based metrics such as BLEU, CIDEr, and R-Precision provide alternative text-related evaluations.
  • Human preference-based evaluation: Human-preference methods train scalar predictors from manually ranked image groups, providing holistic evaluation that is costly to annotate.The underlying datasets require manual comparison of images generated from the same prompt.
  • Object detection-based evaluation: Unlike Dall-Eval, GENEVAL does not require task-specific detector training or synthetic 3D-rendered training images to obtain strong human agreement.The framework can improve as state-of-the-art vision models advance, without further finetuning.
  • Object detection-based evaluation: GENEVAL expands beyond VISOR’s spatial-reasoning focus by passing detector outputs to downstream models that predict properties such as object color.This supports a greater diversity of evaluated tasks.
  • VQA-based evaluation: Compared with VQA-based evaluation, GENEVAL provides per-object bounding boxes and confidence scores that make outputs and failure modes more interpretable.Its components can also be upgraded independently as better models become available.

3 GENEVAL: Our object-focused evaluation framework

GENEVAL decomposes prompts into objects, properties, and relations, then uses object detection and downstream vision models to verify these requirements. It scores correctness across six structured tasks and provides fine-grained explanations of deviations.

  • GENEVAL breaks prompts into object types, properties, and relations to enable fine-grained verification of image-text matching.
  • The benchmark covers single-object, two-object, counting, color, position, and attribute-binding tasks with increasing compositional demands.Two-object generation also serves as a basis for position and attribute-binding evaluation.
  • The framework uses templates filled with sampled object names, colors, numbers, and relative positions, drawing object names from the 80 MS COCO classes.The task list and prompt templates are documented in Table 1.
  • Object detection verifies object presence, counts, and relative positions, while a downstream masked-image classifier evaluates object colors.The detector supplies bounding boxes and segmentation masks for downstream verification.
  • GENEVAL averages binary image-correctness scores across images for each task and then across six tasks to produce an overall model score.Incorrect images receive descriptions of missing objects or mismatched count, position, or color.

4 Measuring alignment with human judgment

The authors compare GENEVAL with human judgments and CLIPScore using annotations from generated images. GENEVAL achieves strong overall agreement and is especially better than CLIPScore on more complex tasks, while its errors reveal concrete detector and classifier failure modes.

  • The study collects 6,000 annotations for 1,200 images generated by Stable Diffusion v2.1, IF-XL, and LAION-5B with CLIP retrieval.The evaluation compares GENEVAL with interannotator agreement and CLIPScore.
  • GENEVAL reaches 83% overall agreement with human annotators, compared with 88% interannotator agreement and 80% for CLIPScore.Threshold-tuned CLIPScore performs slightly better on single-object and color tasks, but GENEVAL performs better on the other four complex tasks.
  • Failure modes include holes incorporated into segmentation masks, merged detections for overlapping same-type objects, and out-of-distribution artistic images.These cases can mislead color classification or reduce detector accuracy.
  • A 22-point improvement over CLIPScore occurs for GENEVAL on the counting task.
  • GENEVAL provides sequential verification outputs that explain why an image is marked correct or incorrect, unlike CLIPScore’s difficult-to-interpret score differences.These explanations also support debugging when GENEVAL disagrees with human judgment.
  • On 860 unanimously judged examples, GENEVAL reaches 91% agreement with humans versus 87% for CLIPScore.

5 Benchmarking progress in recent T2I models

GENEVAL benchmarks open-source text-to-image models across compositional tasks, showing progress in recent models but persistent failures in spatial relations, counting, and attribute binding.

  • Experimental setup: The benchmark evaluates IF-XL, Stable Diffusion variants, SD-XL, and minDALL-E on 553 prompts spanning six tasks, averaging scores over four generated images per prompt.Primary results display IF-XL, SDv2.1, SDv1.5, and SD-XL 1.0.
  • Overall results: Recent models IF-XL and SD-XL significantly improve over previous models on challenging tasks, while overall scores vary by 0.01–0.02 across random seeds.Their relative ordering is consistent with CLIPScore and human annotators, except CLIPScore cannot distinguish SD-XL from IF-XL.
  • Scaling and model evolution: Stable Diffusion shows no consistent performance increase from v1.1 to v1.5 despite more training iterations, whereas v2 significantly improves with a different text encoder.SD-XL also increases scores over SDv2, although multiple architectural and training differences may contribute.
  • Failure modes: IF-XL tends to place the first-mentioned object left of the second, while SDv2.1 frequently swaps two objects’ colors during attribute binding.These patterns expose model-specific biases and failure modes in compositional generation.

6 Limitations

GENEVAL is constrained by the capabilities and training distribution of its underlying object detector and discriminative vision models.

  • 6 Limitations: GENEVAL cannot verify properties outside the object detector’s available classes and granularity, such as the number of fingers on each person’s hands.Current detectors are primarily trained or finetuned on MS-COCO, which defines a limited set of classes.
  • 6 Limitations: Object detectors trained primarily on photographs do not generalize well to visually distinct art.More powerful open-vocabulary detectors trained on broader image distributions could reduce these constraints.

7 Conclusion

GENEVAL provides object-focused, fine-grained evaluation of compositional text-to-image capabilities and aligns strongly with human judgment. Its analyses reveal difficult tasks and model failure modes that can guide future improvement.

  • 7 Conclusion: GENEVAL evaluates compositional reasoning tasks using object detection and color classification to verify fine-grained object properties.The framework is designed as an automated, object-focused evaluation method for text-to-image models.
  • 7 Conclusion: GENEVAL scores align strongly with instance-level human judgment and outperform prior approaches based on overall image-text alignment.This conclusion follows from the paper’s human study comparing GENEVAL with earlier alignment metrics.
  • 7 Conclusion: Relative position and attribute binding remain difficult for current text-to-image models, while GENEVAL can identify their failure modes.The framework is intended to support broader evaluation through additional discriminative vision models in future work.

A Further experiments

Further experiments examine human-agreement analysis and task-level GENEVAL results, including how chance agreement affects interpretation across tasks.

  • A Further experiments: Table 3 reports per-task GENEVAL scores for all evaluated models and provides the basis for additional ablations and analysis.
  • A Further experiments: Cohen’s kappa complements percent agreement by accounting for random agreement chances and ranges from −1 to 1.Task difficulty affects baseline agreement, so raw agreement rates are not directly comparable across tasks.

A.2 Evaluation parameters

The evaluation parameters tune detection and color-classification steps to better match human judgments, especially for counting, position, and attribute binding.

  • GENEVAL’s human-agreement results vary across tasks, with CLIPScore outperforming it on the single-object task because nearly all images were correct.The figure reports Cohen’s kappa results for the human study.
  • Cropping detected objects and masking backgrounds with gray jointly produce the best human agreement for color-related tasks.These steps remove distractions from other objects, particularly in attribute binding.
  • 0.9 confidence threshold raises counting-task Cohen’s kappa from 0.37 to 0.65.Five-fold validation reports 0.823 ± 0.013 agreement across validation splits.
  • A small minimum-distance threshold improves position-task alignment when objects are merged or very close together.Human annotators classify such cases as neutral or not offset; the default distance choice is sensitive to this setting.

A.3 Qualitative examples

Qualitative examples show that models can generate some difficult or unseen object pairs, while GENEVAL exposes systematic position and color-binding failures.

  • Qualitative examples: Models often struggle with structurally complicated objects and certain object combinations, although they sometimes generate rarely seen pairs successfully.A dog-and-potted-plant pair is one example that Stable Diffusion generates despite weak CLIP retrieval evidence.
  • Failure modes: GENEVAL reveals an IF-XL bias toward placing the first object to the left of the second.The failure mode is shown as a position bias in generated images.
  • Failure modes: Stable Diffusion v2.1 sometimes swaps specified colors or leaks them into the background during attribute binding.These tendencies are identified through quantitative analysis and confirmed qualitatively.
  • Prompt examples: The benchmark samples prompts from object, number, position, and color vocabularies, excluding gray and person for color-related tasks.Gray is excluded because it is visually ambiguous with black or white and is used for masked backgrounds.
  • Prompt examples: Prompt metadata records task requirements such as object class, count, and color alongside templated text.An example represents a red bicycle with count one in JSON metadata.

C.2 Image generation

The study evaluates several open-source text-to-image models and a CLIP-retrieval baseline under a fixed generation procedure without broad hyperparameter tuning.

  • The evaluated models include minDALL-E, Stable Diffusion v1 and v2, IF variants, Stable Diffusion XL, and CLIP retrieval from LAION-5B.The model set spans autoregressive and diffusion systems plus a real-image retrieval baseline.
  • IF is evaluated at three sizes, while Stable Diffusion versions and Stable Diffusion XL represent architectural and scaling changes.The IF evaluation excludes its third-stage upscaler.
  • CLIP retrieval selects the top four LAION-5B images for each prompt using CLIP image-text alignment.Retrieved images are not constrained to a fixed resolution or aspect ratio.
  • Each prompt produces four images, while generation parameters are not tuned because of compute cost and open-ended prompts.The procedure applies the same four-sample protocol to CLIP retrieval by selecting four matches.

C.3 Evaluation

GENEVAL combines object detection, geometric rules, and CLIP-based color classification with Mechanical Turk annotations to evaluate fine-grained image-text properties.

  • Object detection: Mask2Former instance segmentation supplies detected objects, bounding boxes, and masks for GENEVAL evaluation.For counting, detections above 0.9 confidence are used; other tasks use the default 0.3 threshold.
  • Relative position: Relative position is determined from object bounding-box coordinates with a size-aware distance threshold, optimized at c = 0.1.The threshold treats closely merged objects as neutral rather than visibly offset.
  • Color classification: CLIP ViT-L/14 classifies object colors from cropped, background-masked regions using cosine similarity to normalized color prompts.The prompts use three templates, and the highest-similarity color determines the prediction.
  • Baselines: CLIPScore is computed from prompt-image embedding cosine similarity and compared across several CLIP models.OpenCLIP ViT-H/14 agrees best overall with annotators, but performs significantly worse on position.
  • Human evaluation: Human evaluation asks about visible objects, counts, colors, realism, relative position, and caption fit.The study uses image-specific questions and examples through Amazon Mechanical Turk.
Loading 2310.11513v1…