PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Text-to-Visual Generation with Image-to-Text Generation

Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, Deva Ramanan

arXiv:2404.01291v2cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

The paper addresses the lack of robust metrics and standardized benchmarks for evaluating text-to-visual alignment, particularly for compositional prompts. It introduces VQAScore, which uses VQA answer likelihoods, and GenAI-Bench, a 1,600-prompt benchmark with human ratings; VQAScore outperforms prior alignment methods across image, video, and 3D settings. These tools provide an open-source alternative for evaluating complex text-to-visual generation, while currently not covering several other generative-model properties.

  • Problem

    Robust, reproducible metrics and benchmarks for image-text alignment remain limited, especially for prompts requiring compositional reasoning.

  • Method

    VQAScore computes alignment as the probability of a VQA model answering “Yes” to a question derived from the text, with CLIP-FlanT5 using bidirectional image-question encoding.

  • Results

    VQAScore outperforms nearly all prior alignment methods and achieves state-of-the-art performance across image-text benchmarks, while also extending to video-text and 3D-text evaluation.

  • Takeaways & Limitations

    GenAI-Bench and VQAScore support evaluation of text-to-visual systems using complex compositional prompts and open-source tools.

  • Takeaways & Limitations

    VQAScore currently does not evaluate toxicity, bias, aesthetics, video motion, or 3D physics.

Abstract

from arXiv · show

Despite significant progress in generative AI, comprehensive evaluation remains challenging because of the lack of effective metrics and standardized benchmarks. For instance, the widely-used CLIPScore measures the alignment between a (generated) image and text prompt, but it fails to produce reliable scores for complex prompts involving compositions of objects, attributes, and relations. One reason is that text encoders of CLIP can notoriously act as a "bag of words", conflating prompts such as "the horse is eating the grass" with "the grass is eating the horse". To address this, we introduce the VQAScore, which uses a visual-question-answering (VQA) model to produce an alignment score by computing the probability of a "Yes" answer to a simple "Does this figure show '{text}'?" question. Though simpler than prior art, VQAScore computed with off-the-shelf models produces state-of-the-art results across many (8) image-text alignment benchmarks. We also compute VQAScore with an in-house model that follows best practices in the literature. For example, we use a bidirectional image-question encoder that allows image embeddings to depend on the question being asked (and vice versa). Our in-house model, CLIP-FlanT5, outperforms even the strongest baselines that make use of the proprietary GPT-4V. Interestingly, although we train with only images, VQAScore can also align text with video and 3D models. VQAScore allows researchers to benchmark text-to-visual generation using complex texts that capture the compositional structure of real-world prompts. We introduce GenAI-Bench, a more challenging benchmark with 1,600 compositional text prompts that require parsing scenes, objects, attributes, relationships, and high-order reasoning like comparison and logic. GenAI-Bench also offers over 15,000 human ratings for leading image and video generation models such as Stable Diffusion, DALL-E 3, and Gen2.

1 Introduction

The paper addresses the lack of robust metrics and comprehensive benchmarks for evaluating image-text alignment, especially for compositional prompts. It introduces VQAScore and GenAI-Bench to evaluate text-to-visual systems across complex image, video, and 3D settings.

  • Evaluation challenges: Existing evaluation metrics struggle to measure alignment for prompts involving multiple objects, attributes, relations, counting, and logical reasoning.CLIPScore and related vision-language models can fail on compositional text, while subjective human evaluation is expensive and difficult to reproduce.
  • VQAScore: VQAScore converts text into a yes-or-no visual question and uses the VQA model’s probability of “Yes” as the image-text alignment score.The score is defined as P(“Yes”|image, “Does this figure show ‘{text}’? Please answer yes or no.”).
  • VQAScore: Open-source VQAScore models outperform nearly all prior alignment methods, including CLIPScore, human-feedback-trained models, and divide-and-conquer approaches.They also compete with methods using proprietary models such as GPT4-Vision across eight alignment benchmarks.
  • VQAScore: Bidirectional image-question encoding lets visual embeddings and question embeddings influence one another, and CLIP-FlanT5 achieves state-of-the-art alignment performance.The architecture allows the image to be interpreted differently depending on the question being asked.
  • GenAI-Bench: GenAI-Bench contains 1,600 prompts covering basic and advanced compositional skills for evaluating text-to-visual models and automated metrics.The benchmark includes over 15,000 human ratings and targets skills such as object, attribute, relation, comparison, logic, and counting.
  • Extensions: Preliminary experiments extend VQAScore to video-text and 3D-text alignment by averaging scores across sampled frames or rendered views.VQAScore significantly surpasses CLIPScore, PickScore, and GPT4-Vision-based divide-and-conquer approaches in these settings.

2 Related Works

Prior automated text-to-visual metrics rely heavily on image quality or text-image similarity, while compositional visual reasoning methods decompose complex tasks into modular subtasks.

  • CLIPScore measures cosine similarity between embedded images and text prompts, while perceptual metrics such as IS, FID, and LPIPS assess generated-image quality.
  • Visual programming methods use LLMs to decompose natural-language visual tasks into modular components that invoke expert vision-language models.

3 Image-Text Alignment Using VQAScore

VQAScore evaluates image-text alignment by asking a VQA model whether an image depicts a text-derived question and scoring the likelihood of a “Yes” answer. The paper further improves this approach with a bidirectional image-question encoder in CLIP-FlanT5.

  • VQAScore computes image-text alignment from a VQA model’s generative likelihood for answering a simple text-derived question.The method converts text into a question and uses the model’s answer probability as the alignment score.
  • The text-derived question asks whether the figure shows the described text, such as “The moon is over the cow.”
  • CLIP-FlanT5 improves VQAScore by using a bidirectional image-question encoder rather than a uni-directional decoder-only architecture.The design allows visual and question representations to influence each other.

4 Experimental Results

Across compositional matching and human-alignment benchmarks, VQAScore outperforms established, decompositional, human-feedback, and proprietary-model baselines. Its CLIP-FlanT5 variant is especially strong on advanced compositional skills, while existing benchmarks motivate GenAI-Bench.

  • Compositional image-text matching: VQAScore sets state-of-the-art performance on Winoground and EqBen, where CLIPScore and PickScore perform at chance level.The method achieves 2x to 5x higher scores than those baselines and CLIP-FlanT5 surpasses prior art including GPT4-Vision-based methods.
  • Divide-and-conquer comparison: End-to-end VQAScore significantly outperforms TIFA, VQ2, and Davidsonian when all methods use the same underlying VQA architectures.The comparison controls for the VQA model while contrasting a simple template with divide-and-conquer question generation.
  • Fine-grained compositional skills: 5x better on basic skills and 10x better on advanced skills, CLIP-FlanT5-based VQAScore surpasses CLIPScore on Winoground.Advanced skills include counting, comparison, differentiation, negation, and universality; they are harder than basic skills.
  • Agreement with human judgments: VQAScore with CLIP-FlanT5 outperforms all prior art across benchmarks measuring agreement with human judgments.Table 4 reports AUROC for DrawBench, EditBench, and COCO-T2I; pairwise accuracy for TIFA160; and binary accuracy for Pick-a-Pic.
  • Benchmark scope: Existing alignment benchmarks lack the advanced compositional structure of realistic user prompts, motivating the creation of a more challenging benchmark.The experimental section introduces this limitation as the rationale for GenAI-Bench.

5 GenAI-Bench for Text-to-Visual Evaluation

GenAI-Bench evaluates text-to-visual models with compositional prompts covering advanced reasoning skills that previous benchmarks do not fully explore. Across ten models, advanced prompts are more difficult, while VQAScore agrees with extensive human ratings.

  • Benchmark design: GenAI-Bench introduces 1,600 prompts covering compositional skills, especially advanced reasoning such as comparison, counting, differentiation, and logic.The benchmark targets real-world prompt structure and evaluates both text-to-visual models and alignment metrics.
  • Benchmark results: State-of-the-art image and video models struggle with GenAI-Bench prompts requiring higher-order compositional reasoning.The evaluated models include DALL-E 3, Stable Diffusion, Pika, and Gen2.
  • Benchmark results: Advanced prompts challenge all ten models more than basic prompts, as shown by separate averaged VQAScores.Figure 4 reports scores for basic and advanced prompts across open- and closed-source image and video models.
  • Human evaluation: 15,810 human ratings were collected for 527 prompts across ten image and video generative models.This coreset exceeds the scale of previous human evaluations; analysis of all 1,600 prompts is deferred to subsequent work.

6 Extending VQAScore to Video and 3D

VQAScore extends beyond images by evaluating text alignment for video and 3D assets through sampled frames and rendered views. It surpasses established and proprietary-model-based baselines on both modalities.

  • Evaluation setup: VQAScore evaluates text-to-video and text-to-3D alignment using sampled video frames and rendered 3D views across camera angles.The benchmark reports Pairwise accuracy, Pearson, and Kendall correlations or accuracies.
  • Results: VQAScore surpasses the GPT4-Vision-based T2VScore-A on the text-to-video benchmark.Video scores are computed by uniformly sampling frames across time and averaging their VQAScores.
  • Results: VQAScore exceeds CLIPScore, PickScore, and other popular text-to-3D metrics on StanfordT23D.The evaluation includes human ratings collected for six text-to-3D models.

7 Conclusion

The paper presents VQAScore and GenAI-Bench as tools for evaluating alignment on challenging compositional prompts across text-to-image, video, and 3D generation. It identifies a scope boundary: VQAScore does not currently assess several other generative-model qualities.

  • Limitations: VQAScore currently does not evaluate toxicity, bias, aesthetics, video motion, or 3D physics.The authors suggest these aspects may require fine-tuning on relevant data.
  • Conclusion: VQAScore surpasses current alignment metrics for evaluating text-to-image, text-to-video, and text-to-3D models.The CLIP-FlanT5 implementation is presented as a strong alternative to CLIPScore, especially for real-world compositional prompts.
  • Conclusion: GenAI-Bench evaluates text-to-visual generation and vision-language alignment with challenging compositional prompts.The benchmark is intended to support scientific evaluation of generative models.

Outline

The supplementary material covers GenAI-Bench skill definitions, benchmark construction and human validation, VQAScore computation, implementation details, and baseline methods. It organizes the evaluation around compositional reasoning skills and likelihood-based scoring.

  • D-F Methods and evaluation: The supplement reports benchmark results, baseline comparisons, and implementation details for evaluating VQAScore and CLIP-FlanT5.It includes pseudocode and comparisons against VLM metrics, human-feedback metrics, visual-programming methods, divide-and-conquer VQA methods, and GPT4-Vision approaches.
  • A Visio-Linguistic Compositional Reasoning Skills: The supplement defines and compares basic and advanced compositional reasoning skills across text-to-visual benchmarks.Skills include objects, attributes, relations, scenes, comparison, differentiation, counting, and logic.
  • A Visio-Linguistic Compositional Reasoning Skills: GenAI-Bench comprehensively covers essential skills in real-world prompts compared with existing alignment and generation benchmarks.Skill proportions are calculated after mapping or manually annotating benchmark samples.
  • B GenAI-Bench: GenAI-Bench contains 1,600 prompts created with graphic designers and evaluated through human ratings and VQAScore.A preliminary coreset uses 527 prompts across six text-to-image and four text-to-video models.
  • C Implementing VQAScore: VQAScore converts text into a yes-or-no question and computes the autoregressive likelihood of the answer conditioned on the image and question.The answer-token likelihoods can be computed in parallel because the full answer sequence is available as input.

D Training CLIP-FlanT5

This section details CLIP-FlanT5 training, ablations, and inference choices for VQAScore. Results favor VQA finetuning, larger language models, encoder-decoder FlanT5, and concise clear questions.

  • Training procedure: CLIP-FlanT5 follows LLaVA-1.5’s two-stage training recipe with a frozen CLIP ViT-L-336 encoder, a 2-layer MLP projector, captioning data, and VQA data.Stage-1 trains the projector on 558K captioning examples; the supplied passage truncates the remaining split-text procedure.
  • Ablations: Ablations compare FlanT5 and Llama-2 language models, model sizes, and finetuning on captioning alone versus captioning plus VQA data.The evaluation covers seven image-text alignment benchmarks.
  • Ablation findings: Finetuning on VQA data is crucial, whereas captioning data provides only modest gains.This observation is reported across the ablation study’s seven-benchmark aggregate.
  • Ablation findings: Scaling up language models consistently improves performance, and encoder-decoder FlanT5 significantly outperforms decoder-only Llama-2.These findings are reported for the overall seven-benchmark performance comparison.
  • Prompting and answer formats: Clear, simple question templates tend to perform best on Winoground and EqBen, while concise P(“Yes”) answers outperform longer responses.Negative answer likelihoods such as –P(“No”) often produce comparable results.
  • Prompting and answer formats: Removing the system prompt does not affect CLIP-FlanT5 but hurts LLaVA-1.5’s performance.The comparison concerns inference-time use of the chat-style system prompt.

E Details of Baseline Methods

The paper compares VQAScore with several baseline families, including discriminative metrics, human-feedback metrics, visual programming, divide-and-conquer VQA, and GPT4-Vision methods. These baselines differ in complexity, model requirements, and failure modes on compositional prompts.

  • Metrics based on vision-language models: CLIPScore and BLIPv2Score measure image-text alignment with discriminatively pretrained vision-language models, while prior analyses examine their bag-of-words behavior.CLIPScore uses the same CLIP-L-336 model as CLIP-FlanT5 and LLaVA-1.5; BLIPv2Score uses the largest BLIPv2-ViT-G ITM head.
  • Metrics finetuned on human feedback: PickScore, ImageReward, and HPSv2 finetune vision-language models with costly human feedback, but perform no better than random chance on Winoground and EqBen.The paper attributes this result as likely related to discriminative pretraining behavior and limited compositional text in finetuning data.
  • Visual programming methods: Visual programming methods invoke expert vision-language models through generated programs, yet VPEval occasionally produces erroneous or nonsensical programs for compositional texts.One reported example asks a VQA model what a person is doing while talking on the phone and expects “happily” as the answer.
  • Divide-and-conquer using VQA: Divide-and-conquer VQA methods generate modular question-answer pairs, but often produce irrelevant or erroneous questions, especially for highly compositional prompts.Davidsonian can generate nonsensical questions, and Table 16 summarizes failure cases for TIFA, VQ2, and Davidsonian.
  • Divide-and-conquer using VQA: The end-to-end VQAScore outperforms question decomposition with Davidsonian and direct answer generation when all methods use the same CLIP-FlanT5 model.This comparison is reported in the question-decomposition and answer-generation ablation.
  • GPT4-Vision-based methods: GPT4-Eval and VIEScore ask GPT4-Vision to directly generate alignment scores, whereas VQAScore requires answer likelihoods unavailable through the GPT4-Vision API.T2VScore-A is a GPT4-Vision-based divide-and-conquer method specifically designed for video-text alignment.

F Details of Alignment Benchmarks

The benchmarks evaluate agreement between automated alignment scores and human judgments across image-text, image-to-text, video-text, and 3D-text settings. They use several metrics and sampling protocols, with some datasets requiring quality filtering or restricted subsets.

  • Image-text benchmarks: TIFA160 contains 800 image-text pairs from 160 prompts and evaluates agreement with human Likert ratings using Pairwise accuracy, Pearson, and Kendall metrics.VQAScore consistently surpasses prior methods across all three meta-evaluation metrics.
  • Image-to-text benchmarks: Flickr8K-CF contains 145K binary judgments for 48K image-caption pairs, and VQAScore outperforms prior art including reference-based metrics in a reference-free setting.Each pair receives at least three binary judgments, with ratings computed from the mean proportion of “yes” annotations.
  • Video-text benchmarks: EvalCrafter evaluates 700 prompts paired with five text-to-video models using Likert ratings, with VQAScore averaged across sampled video frames.Sampling four frames achieves near-optimal performance compared with using all 36 frames.
  • 3D-text benchmarks: StanfordT23D evaluates 3D-text alignment across six text-to-3D models and 60 prompts using three human ratings per pair and uniformly sampled rendered views.A single 2x2 or 3x3 grid image often performs almost as well as averaging scores across multiple views.
  • Dataset quality caveats: The study manually filters Pic-a-pic after finding 10% inappropriate content and approximately 50% incorrect labels in a reviewed 500-sample test set.The resulting evaluation subset contains 100 prompts paired with 200 images, excluding tied labels.
  • Evaluation metrics: DrawBench, EditBench, and COCO-T2I use binary match labels and report AUROC across all possible classification thresholds.These protocols follow SeeTrue’s benchmark procedures.
  • Image-text benchmarks: Winoground uses 400 image-text pairs, while the study evaluates the higher-quality EqBen-Mini subset of 280 pairs because the official EqBen test set contains low-quality images.Both benchmarks assess alignment through image-text matching tasks.
Loading 2404.01291v2…