PaperlayerView exact revision on arXiv

Source-linked AI summary

TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering

Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, Noah A Smith

arXiv:2303.11897v3cs.CV

TL;DR

Text-to-image systems still struggle to align generated images with their prompts, creating a need for reliable and interpretable faithfulness evaluation. TIFA generates and verifies question-answer pairs, then measures whether VQA models answer them correctly from generated images. Across its benchmark study, TIFA identifies weaknesses in counting, spatial relations, and multi-object composition while correlating better with human judgments than prior metrics.

  • Problem

    Reliable automatic metrics for evaluating text-to-image faithfulness remain limited, while generated images can fail at composition, attribute binding, and visual text.

  • Method

    TIFA generates question-answer pairs from text, filters them with a QA model, and evaluates generated-image faithfulness using VQA answer accuracy.

  • Results

    TIFA v1.0 evaluations show that current models perform well on common objects, animals, and colors but struggle with multiple-object composition, spatial relations, and activity binding, while TIFA correlates better with human judgments than prior metrics.

  • Takeaways & Limitations

    TIFA v1.0 provides fine-grained evaluations that can compare models and expose specific weaknesses for future text-to-image research.

  • Takeaways & Limitations

    TIFA depends on VQA models working reasonably well, an assumption that may not hold in domains such as anime and abstract art.

Abstract

from arXiv · show

Despite thousands of researchers, engineers, and artists actively working on improving text-to-image generation models, systems often fail to produce images that accurately align with the text inputs. We introduce TIFA (Text-to-Image Faithfulness evaluation with question Answering), an automatic evaluation metric that measures the faithfulness of a generated image to its text input via visual question answering (VQA). Specifically, given a text input, we automatically generate several question-answer pairs using a language model. We calculate image faithfulness by checking whether existing VQA models can answer these questions using the generated image. TIFA is a reference-free metric that allows for fine-grained and interpretable evaluations of generated images. TIFA also has better correlations with human judgments than existing metrics. Based on this approach, we introduce TIFA v1.0, a benchmark consisting of 4K diverse text inputs and 25K questions across 12 categories (object, counting, etc.). We present a comprehensive evaluation of existing text-to-image models using TIFA v1.0 and highlight the limitations and challenges of current models. For instance, we find that current text-to-image models, despite doing well on color and material, still struggle in counting, spatial relations, and composing multiple objects. We hope our benchmark will help carefully measure the research progress in text-to-image synthesis and provide valuable insights for further research.

1. Introduction

TIFA addresses unreliable automatic evaluation of text-to-image faithfulness by decomposing prompts into verified questions and testing generated images with VQA. Its benchmark reveals that current models remain weak at compositional and relational aspects despite strengths in common visual attributes.

  • Current text-to-image models often fail to compose multiple objects, bind attributes correctly, and generate visual text.
  • Existing faithfulness metrics are unreliable because CLIPScore struggles with counting and compositional reasoning, while captioning can omit salient information.
  • TIFA generates question-answer pairs from text, filters them with a QA system, and measures generated-image faithfulness by VQA answer accuracy.
  • TIFA v1.0 contains 4K diverse text inputs and 25K questions covering 4.5K distinct elements, with pre-generated question-answer pairs and multiple VQA models.
  • Evaluation finds that current state-of-the-art models render common objects, animals, and colors well but struggle with multiple-object composition, spatial relations, and activity binding.

2. Related Work

Related work evaluates generated images through image-quality and image-text-alignment metrics, while QA-based evaluation offers a precedent for factual consistency assessment. TIFA adapts this question-based perspective to text-to-image faithfulness.

  • Prior image-generation evaluation considers both image quality and image-text alignment, alongside widely adopted image-quality metrics such as Inception Score and FID.
  • QA-based metrics have higher correlations with human judgments for factual consistency in summarization, motivating a related question-based view for image faithfulness.

3. The TIFA Metric

TIFA measures text–image faithfulness by testing whether VQA models recover answers to automatically generated questions grounded in the text. Its pipeline extracts and categorizes text elements, generates and filters questions, and averages VQA answer accuracy.

  • Metric definition: TIFA defines faithfulness as the accuracy of VQA answers to multiple-choice questions generated from the text input.The score ranges from 0 to 1 and is highest when the image covers information in the text.
  • Question generation: GPT-3 generates question-answer tuples through one in-context-learning inference after extracting elements from the text.The extracted elements include noun phrases, verbs, adjectives, adverbs, and short parse-tree spans.
  • Question generation: Elements are classified into 12 categories, enabling detailed analysis of text-to-image performance by aspect.The categories include object, activity, animal, food, counting, color, material, spatial, location, shape, attribute, and other.
  • Question generation: Each element receives two questions: a yes/no question and a question asking for the element as its answer.For counting, for example, the questions ask whether there are three dogs and how many dogs there are.
  • Question filtering: UnifiedQA filters generated questions when its answers disagree with GPT-3 or fail the required word-level F1 agreement threshold.A human evaluation found only 7 unreasonable pairs among 1,000 filtered question-answer pairs.
  • VQA evaluation: VQA models answer the filtered questions on generated images, with mPLUG-large recommended based on accuracy, human-judgment correlation, and runtime.TIFA also provides tools supporting multiple open-domain vision-language VQA models.

4. TIFA v1.0: Benchmark for Text-to-Image Generation Faithfulness

TIFA v1.0 is a diverse benchmark for evaluating text-to-image faithfulness across thousands of inputs, questions, and element categories. It supports aggregate and category-level analysis while addressing deterministic benchmark generation through a released LLaMA 2 question generator.

  • Benchmark construction: TIFA v1.0 contains 4,081 text inputs paired with 25,829 question-answer pairs for text-to-image faithfulness evaluation.Its APIs perform VQA with various state-of-the-art models, and overall scores average image-level scores across benchmark inputs.
  • Text collections: The benchmark combines 2,000 COCO validation captions with 2,081 prompts from DrawBench, PartiPrompt, and PaintSkill.The collected prompts include both captions with corresponding real images and abstract-art-oriented texts without corresponding real images.
  • Statistics and diversity: TIFA v1.0 covers 4,550 distinct elements across 12 categories, with object elements occurring most frequently.The benchmark includes counting, spatial, color, material, shape, and other categories alongside object, animal/human, attribute, activity, location, and food.
  • Statistics and diversity: Category-specific VQA accuracy measures a text-to-image model’s ability on the corresponding aspect of the input.This supports fine-grained comparisons across elements such as objects, activities, spatial relations, and counting.
  • Benchmark construction: Benchmarking with TIFA v1.0 is deterministic, but creating new benchmarks with GPT-3 may not remain deterministic if the underlying model changes.The authors release a fine-tuned LLaMA 2 7B model to generate questions for arbitrary texts more deterministically.

5. Experiments

Experiments show that TIFA correlates more strongly with human faithfulness judgments than prior metrics and provides fine-grained diagnosis of text-to-image failures. Evaluations reveal strengths in common visual elements but persistent difficulties with composition, counting, spatial relations, and increasing entity counts.

  • 5.1. Correlation with Human Judgements: Human faithfulness annotations score images from 1 to 5 according to how many prompt elements are missed or misrepresented.Annotators focus on text-to-image faithfulness rather than image quality, with objects prioritized over attributes, relations, and activities.
  • 5.1. Correlation with Human Judgements: TIFA has higher correlations with human judgments than all previous evaluation metrics across tested VQA models.TIFA with mPLUG yields the highest correlation among the evaluated VQA models.
  • 5.2. Benchmarking Text-to-Image Models: TIFA scores rise from about 60% to 75% after DALL-E is released, revealing an improvement trend across model release dates.Figure 7 reports average TIFA scores on TIFA v1.0, with release dates on the horizontal axis.
  • 5.3. Findings on Current Text-to-Image Models: VQA accuracy is higher for COCO captions than for other text inputs because COCO captions correspond to real images, whereas free-form prompts may describe unreal compositions.A cited example of such a composition is “a blue apple.”
  • 5.3. Findings on Current Text-to-Image Models: Models perform well on material, animal/human, color, and location, but obtain low accuracy on shapes, counting, spatial relations, and abstract-art notions.The evaluation also reports a substantial gap between synthesized and real images on COCO captions.
  • 5.3. Findings on Current Text-to-Image Models: Ground-truth COCO images do not receive perfect scores because captions contain crowd-worker noise and VQA models are imperfect.Real images have higher accuracy in most categories, with small differences in material, color, and location.
  • 5.3. Findings on Current Text-to-Image Models: When text inputs contain more than 5 entities, TIFA scores drop rapidly for all text-to-image models, while real-image scores remain consistent.This pattern quantitatively indicates that composing multiple objects is challenging for current models.
  • 5.4. Analysis of VQA Models: Most TIFA errors are attributed to text-to-image models rather than VQA models; for mPLUG, fewer than 25% of errors are due to VQA.Different VQA models produce similar TIFA trends, with pairwise score correlations greater than 0.6.

6. Discussion

TIFA improves faithfulness evaluation by decomposing text into questions, but its reliability depends on VQA quality and incurs greater runtime than CLIP inference.

  • TIFA decomposes text inputs into questions, unlike prior metrics that summarize images and text into a single embedding or caption.The authors hypothesize that this decomposition supports more accurate faithfulness evaluation.
  • TIFA requires VQA models to perform reasonably well, an assumption that may fail in domains such as anime and abstract art.The framework can update its VQA component or ensemble image-understanding models as stronger systems become available.
  • Answering multiple visual questions is slower than one CLIP inference, with mPLUG taking 1.6s per image without batching.The runtime may be impractical for large-scale filtering, although the authors consider benchmarking, reranking, and reinforcement learning feasible.

7. Conclusions

The paper presents TIFA and TIFA v1.0 for fine-grained text-to-image faithfulness evaluation, using VQA to expose persistent model weaknesses.

  • TIFA is an automatic VQA-based metric that is fine-grained, interpretable, and better aligned with human judgments than prior metrics.
  • TIFA v1.0 contains 4K prompts and 25K questions for evaluating text-to-image models.
  • Current image-generation models still struggle with counting, spatial relations, and composing multiple objects.
  • TIFA remains robust to different VQA models and can be upgraded with new language-model, QA, and VQA components.

A. Qualitative Examples

The qualitative examples show how TIFA turns text inputs into verified questions and compares VQA answers with the requested image content.

  • Qualitative Examples: Each example begins with a text input, followed by GPT-3-generated and UnifiedQA-filtered question-answer pairs.The examples use inputs from COCO and DrawBench and evaluate images with mPLUG.
  • Qualitative Examples: For the surfer example, questions test place, person identity, carrying a board, weather, and board color.The displayed answers identify a beach, a surfer carrying a board, a gray day, and a white board.
  • Qualitative Examples: For “A black colored banana,” the questions test banana identity and color, with the displayed answers including “yes,” “surfer,” and “black.”
  • Qualitative Examples: For “Three cats and two dogs sitting on the grass,” the questions test animal presence, the supporting surface, activity, and object counts.The displayed answers are three cats and two dogs.
  • Qualitative Examples: The benchmark table organizes results by overall score, text source, and question categories including location, color, animals or humans, and material.

B. Detailed Results

Detailed TIFA results compare text-to-image models across question categories and text sources, identifying both overall leaders and category-specific strengths.

  • Stable Diffusion v2.1 achieves the highest overall TIFA score and the highest accuracy in most categories.
  • minDALL-E leads on shape, other, and food, while VQ-Diffusion leads on color accuracy.

C.1. Likert Scale on Text-to-Image Faithfulness

The Likert-scale protocol measures text-image faithfulness by assessing whether described elements are present and correctly represented, while prioritizing objects and preserving annotator ranking preferences. Human scores were collected across generated images with multiple annotators and substantial agreement.

  • Annotation procedure: Annotators score only text-image match and may adjust scores to preserve their ranking of images generated from the same text input.Ties are allowed when the ranking preference does not distinguish images.
  • Annotation criteria: Faithfulness measures text-image match rather than image quality, using missed or misrepresented objects, attributes, relations, and context as evidence.Objects receive priority because a missing object also counts its related attributes and activities as missing.
  • Annotation criteria: Annotators treat partially recognized elements, such as a generic hat without its specified role, as partially missed or misrepresented.The guideline gives a train conductor’s hat as an example of an element that may be only half satisfied.
  • Reference scoring: The 1–5 scale ranges from a perfect prompt match to images in which no major objects are correctly presented.Intermediate scores depend on how many of n elements are missed or misrepresented, with higher scores corresponding to fewer errors.
  • Annotation procedure: The study collected 1600 annotations on 800 generated images from 160 text inputs, with each image scored by two annotators and final scores averaged.The images came from five recent text-to-image models, and images for each text input were shown together to support ranking-based judgments.
  • Agreement: Human annotation achieved Krippendorf’s α of 0.67 for Likert scores, while the separate VQA annotation achieved α of 0.88 and used majority voting after disagreement.The VQA study covered 1029 questions on 126 generated images and added a third annotator when the first two disagreed.
  • VQA annotation: VQA annotators choose answers from yes/no or four-choice questions plus “None of the above,” answering ambiguous cases according to visible image content.For example, an unclear human gender classification is marked “no” for the question “is there a man?”.

D. Common Q & A

The benchmark documentation describes TIFA’s extensibility, input distribution and bias analysis, and the question-generation workflow used to create category-specific verification questions. Examples show questions targeting objects, attributes, activities, colors, locations, and counting.

  • Extensions: TIFA can be extended into specialized benchmarks for counting, composing multiple objects, text-to-3D, and text-to-video generation.These extensions are proposed as customized versions focused on particular aspects or other generation tasks.
  • Question generation: The benchmark’s questions are pre-generated by GPT-3, while the overall approach also supports local language models such as a fine-tuned LLaMA 2 checkpoint.Evaluation does not require regenerating the TIFA v1.0 questions.
  • Input analysis: Most TIFA v1.0 text inputs contain around 10 words, and its gender distribution includes 400 perceived-male and 239 perceived-female inputs among 4K prompts.The reported gender-distribution bias comes from captions sampled from the COCO dataset.
  • Question generation: The generation prompt first extracts description elements, classifies them into categories, and then creates questions for each element.The categories include object, human, animal, food, activity, attribute, counting, color, material, spatial, location, shape, and other.
  • Question examples: A horse-and-cows example uses questions about animal presence, hay, feeding activity, and the quantity descriptor “several.”The example illustrates that question generation covers both semantic content and counting-related language.
  • Question examples: Examples generate yes/no and multiple-choice questions that test whether entities, attributes, activities, colors, and locations in descriptions appear in images.Examples include a red dog, motorcyclists outside a Polish gathering spot, and a man taking a selfie while wearing a jacket and bow tie.
Loading 2303.11897v3…