PaperlayerView exact revision on arXiv

Source-linked AI summary

What You See is What You Read? Improving Text-Image Alignment Evaluation

Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, Idan Szpektor

arXiv:2305.10400v4cs.CLcs.CV

TL;DR

Evaluating whether images and text are semantically aligned remains difficult for multimodal generation systems, especially across complex or synthetic cases. The paper introduces SeeTRUE and two evaluators—question-based VQ2 and fine-tuned VNLI—that outperform prior approaches and support misalignment localization and candidate re-ranking.

  • Problem

    Automatic evaluation of semantic alignment between images and text remains limited, while existing methods struggle with complex semantics and are often studied separately for text-to-image and image-to-text generation.

  • Method

    The paper introduces the human-labeled SeeTRUE benchmark and evaluates VQ2, a question-generation and visual-question-answering pipeline, alongside end-to-end VNLI fine-tuning.

  • Results

    VQ2 and VNLI outperform strong baselines across multiple alignment tasks; VQ2 reaches a 30.5% Winoground group score and improves evaluation of synthetic images.

  • Takeaways & Limitations

    The methods can localize specific image-text misalignments and re-rank generated image candidates for text-to-image prompts.

  • Takeaways & Limitations

    Binary alignment decisions can remain difficult even for human annotators, despite high reported agreement under comprehensive guidelines.

Abstract

from arXiv · show

Automatically determining whether a text and a corresponding image are semantically aligned is a significant challenge for vision-language models, with applications in generative text-to-image and image-to-text tasks. In this work, we study methods for automatic text-image alignment evaluation. We first introduce SeeTRUE: a comprehensive evaluation set, spanning multiple datasets from both text-to-image and image-to-text generation tasks, with human judgements for whether a given text-image pair is semantically aligned. We then describe two automatic methods to determine alignment: the first involving a pipeline based on question generation and visual question answering models, and the second employing an end-to-end classification approach by finetuning multimodal pretrained models. Both methods surpass prior approaches in various text-image alignment tasks, with significant improvements in challenging cases that involve complex composition or unnatural images. Finally, we demonstrate how our approaches can localize specific misalignments between an image and a given text, and how they can be used to automatically re-rank candidates in text-to-image generation.

1 Introduction

The paper addresses the difficulty of evaluating semantic alignment between images and text across text-to-image and image-to-text generation. It introduces SeeTRUE and two automatic evaluators that outperform strong baselines, including on compositional and synthetic-image cases.

  • Motivation: Existing alignment evaluation struggles with complex semantics because common CLIP- and BLIP-based methods encode images and text as fixed-size embeddings.The task is also commonly studied separately for text-to-image and image-to-text generation.
  • Contributions: SeeTRUE is a diverse benchmark with human alignment judgments spanning real and synthetic images and text across tasks, containing 31,855 labeled examples.It also uses an LLM-based method to generate contradicting captions during benchmark construction.
  • Methods: VQ2 evaluates alignment through text-derived questions and visual question answering, while VNLI fine-tunes a pretrained multimodal model to classify image-text alignment.Both approaches are motivated by factual-consistency evaluation methods for text pairs.
  • Results: VQ2 and VNLI outperform strong CLIP, COCA, BLIP, and OFA baselines across comprehensive SeeTRUE experiments.VQ2 improves the Winoground group score from 16% to 30.5% and also performs better on synthetic-image datasets such as DrawBench and EditBench.
  • Applications: The proposed methods can identify specific image-text misalignments and re-rank generated image candidates for a text-to-image prompt.These uses extend alignment evaluation beyond assigning a single pairwise score.

2 SeeTRUE: A Comprehensive Text-Image Alignment Benchmark

SeeTRUE is a broad image-text alignment benchmark designed to test generalization across real and synthetic data, generation tasks, and challenging contradictions. It combines standardized human labels with automatically generated contrastive captions and multiple data-construction pipelines.

  • 2 SeeTRUE: A Comprehensive Text-Image Alignment Benchmark: SeeTRUE covers four combinations of real and synthetic text and images, addressing benchmarks that mainly use natural images and lack challenging negative captions.Its scope is intended to assess alignment-model generalization across tasks.
  • 2 SeeTRUE: A Comprehensive Text-Image Alignment Benchmark: The benchmark contains 31,855 real and synthetic image-text pairs from diverse datasets and tasks.Table 1 provides a high-level overview and examples from each dataset.
  • 2.1 Datasets: The datasets include real-image pairs from SNLI-VE and Winoground, synthetic-image pairs from EditBench and newly generated COCO t2i and DrawBench sets, and synthetic-caption sets such as COCO-Con and PickaPic-Con.Winoground specifically tests compositional matching with captions differing by only a few words.
  • 2.1 Datasets: The benchmark mixes testing-only datasets with datasets containing training and test splits, enabling comparisons across training configurations.Winoground, DrawBench, and EditBench are testing-only, while SNLI-VE, COCO t2i, COCO-Con, and PickaPic-Con include training data.
  • 2.2 Human Annotation and Evaluation: Labels are standardized as binary alignment or misalignment, collapsing non-entailment and non-alignment labels into one negative class.This harmonizes datasets that originally use three-way entailment, contradiction, and neutral labels.
  • 2.2 Human Annotation and Evaluation: Human annotation asks whether an image presents all details described in the text, with three crowd-workers also describing the main misalignment for negative judgments.Annotators reached consensus in 80% of cases, with Fleiss-Kappa 0.722.
  • 2.3 ConGen: Generating Contradicting Captions by Prompting LLMs: ConGen prompts PaLM with few-shot examples to generate contradicting captions, then uses an NLI model to select the candidate with the highest contradiction score.Human agreement with the automatic labels was 94% for COCO-Con and 77% for PickaPic.

3 Methods

The paper presents two alignment evaluators: zero-shot VQ2 decomposes text-image verification into question-answer checks, while end-to-end VNLI directly predicts alignment with a fine-tuned multimodal model.

  • 3 Methods: The methods section evaluates two image-text alignment approaches against established methods using the SeeTRUE benchmark.The approaches are VQ2 and end-to-end visual natural language inference.
  • 3.1 V Q2: Zero-Shot Alignment via Question Generation and Visual Question Answering: VQ2 extracts candidate answer spans from text, generates and text-validates question-answer pairs, and scores their consistency with the image using visual question answering.Low-scoring text question-answer pairs are filtered before image validation.
  • 3.1 V Q2: Zero-Shot Alignment via Question Generation and Visual Question Answering: VQ2 computes each pair’s alignment as the VQA probability of a “yes” answer to a reformulated yes-no question, then averages the pair scores.The yes-no formulation performed better than directly asking the generated questions.
  • 3.2 End-to-end VNLI Models: The end-to-end VNLI model fine-tunes BLIP2 and PaLI-17B on 110K labeled text-image pairs formatted as “Does this image entail the description: {text}?” questions.Inference uses the relative probabilities of “yes” and “no” as the alignment score, with training data drawn from COCO-Con, PickaPic-Con, COCO t2i, and SNLI-VE.

4 Experiments

The experiments compare VQ2 and fine-tuned VNLI models with multimodal baselines on SeeTRUE and related alignment tasks. VQ2 performs strongly across datasets, especially on compositional Winoground, while fine-tuning and combining models provide complementary gains.

  • 4.1 Models and Metrics: The evaluation reports ROC AUC for binary misalignment detection and uses text, image, and group scores for Winoground.The compared systems include CLIP, CoCa, BLIP, BLIP2, OFA, TIFA, VQ2, and fine-tuned VNLI variants.
  • 4.2 Results: VQ2 is the top-performing zero-shot model across all datasets, surpassing other zero-shot baselines and most fine-tuned models.Table 2 reports ROC AUC across real and synthetic image-text evaluation sets.
  • 4.2 Results: PaLI fine-tuned on all available datasets achieves an average score of 82.9 and the best results on 3 of 7 datasets.The SNLI-VE-only variant reaches an average of 79.7 and leads on 2 of 7 datasets.
  • 4.2 Results: A simple average of VQ2 and the leading fine-tuned PaLI model produces higher performance, suggesting that the methods complement each other.Synthetic training data improves several synthetic-image datasets but does not improve COCO-Con, which contains only natural images.
  • 4.2 Results: VQ2 achieves state-of-the-art Winoground performance with a 30.5% group score, targeting visual reasoning and compositional understanding.Winoground also reports text and image scores for selecting the correct caption or image.
  • 4.2 Results: VQ2 identifies image-text contradictions by locating question-answer pairs with low alignment scores, with author-judged accuracy of 88% on Coco-Con, 74% on DrawBench, and 80% on Winoground.The evaluated contradiction consists of a question and answer intended to represent the primary discrepancy.
  • 4.2 Results: VQ2 and PaLI rankings correlate very well with human rankings of text-to-image models, with R2 > 0.92.The comparison covers DrawBench and COCO-t2i generated images and model ratings.
  • 4.2 Results: Reranking candidates with VQ2 consistently yields higher human-labeled quality scores than CLIP, while VQ2 and PaLI place brown-and-white cats above white-only cats in an example.The reranking evaluation uses DrawBench and COCO-t2i candidates.

5 Related Work

Prior work evaluates multimodal semantic alignment using visual entailment, VQA, and image-text similarity models, but these systems can miss compositional structure. SeeTRUE extends evaluation across synthetic and natural settings, while VQ2 uses multiple question-answer probes and independently scores individual image-text pairs.

  • 5 Related Work: Visual entailment adapts textual entailment to assess semantic alignment between images and text, while common vision-language models can behave like bag-of-words systems.The cited models include CLIP, CoCa, BLIP, BLIP2, and OFA.
  • 5 Related Work: VQ2 generates multiple questions probing diverse semantic aspects to improve performance on compositional tasks such as Winoground and unnatural images such as DrawBench.This approach addresses limitations attributed to shallow compositional understanding.
  • 5 Related Work: Unlike human-feedback methods such as DrawBench and DALL-Eval, the approach produces automated scores across a broader range of text-image alignments and surpasses TIFA.The paper attributes the difference from TIFA possibly to more question-answer pairs and tailored generation and answering models.
  • 5 Related Work: SeeTRUE extends cross-task alignment evaluation to synthetic images and texts in addition to natural image-text data.The work positions this extension alongside prior studies of consistency across modalities and tasks.
  • 5 Related Work: Unlike PickScore, which predicts preferences for image quality and aesthetics by comparing images, these methods independently score one image for image-text alignment.The distinction separates alignment assessment from general image preference ranking.

6 Limitations

The paper identifies difficulties in making binary image-text alignment decisions, including cases that are challenging even for humans. It addresses this with annotation guidelines and reports remaining variability in perceptions of harmful content.

  • Binary image-text alignment decisions can be difficult even for human annotators.
  • Perceptions of harmful or offensive content may vary among individuals and remain subject to personal interpretation.

7 Conclusion

The paper concludes that SeeTRUE supports image-text alignment evaluation across real and synthetic data, while VQ2 and end-to-end VNLI outperform strong baselines. It identifies future use of these evaluators to guide more aligned model outputs.

  • SeeTRUE covers real and synthetic text-image pairs across text-to-image and image-to-text tasks, including challenging generated contradictions.
  • The authors propose using automatic evaluators to guide training toward more aligned outputs, including filtering examples or providing reinforcement-learning rewards.

A.1 Dataset Supplementary Materials

The supplementary materials document SeeTRUE’s intended use, licensing, hosting, and dataset availability. They also provide example rows and distinguish newly annotated datasets from existing datasets that are not republished.

  • SeeTRUE is intended for evaluating image-text matching models and is planned for hosting on Hugging Face Datasets for open research.
  • The authors publish datasets annotated in this work but do not republish existing SNLI-VE and Winoground datasets.
  • The supplementary materials list licenses for MS COCO, EditBench, DrawBench, Pick-a-Pick, SNLI-VE, and Winoground.
  • Table 5 provides examples of SeeTRUE rows.

A.2 Human Annotation Process

The human annotation process uses qualification, instructions, and agreement checks to obtain alignment labels. Annotators compare image content with text descriptions, report primary misalignments, and mark potentially offensive material.

  • Figure 7 shows a sample annotation interface, while Table 6 presents agreement metrics for different datasets.
  • The annotation process includes qualification requirements, instructions, and evaluation of annotator agreement.
  • Annotators were screened using approval, HIT-count, location, and qualification-test requirements, with payment of 15–18 USD hourly.
  • Annotators assess whether image and text descriptions match, answer “Yes” or “No,” and report the primary issue when misaligned.
  • Annotators can flag images or text for potentially offensive bias, race-related, or NSFW content.
  • Agreement varied across datasets: COCO-Con had the highest agreement and DrawBench the lowest.

A.3 Comparing V Q2 variants

This section compares VQ2 configurations for assessing question–answer alignment against an image and reports the selected zero-shot variant. It also specifies extended answer extraction and computational costs for reproducibility.

  • A.3 Comparing V Q2 variants: VQ2 Type A answers each generated question from the image and uses an NLI model to estimate entailment between text-derived and image-derived question–answer pairs.
  • A.3 Comparing V Q2 variants: Type B answers the question from the image and uses VQA to determine whether the text-derived and image-derived answers are the same.
  • A.3 Comparing V Q2 variants: Type C reformulates each question–answer candidate as a yes-no predicate and uses the VQA model’s probability of “yes” as the alignment score.
  • A.3 Comparing V Q2 variants: Extended answer extraction adds multi-word spans, including adjectives and locations, because named entities and noun phrases alone are insufficient for short texts.
  • A.3 Comparing V Q2 variants: Type C outperforms the other VQ2 configurations on EditBench and is more efficient because it requires one VQA-model run for question–answer assessment.Table 7 compares the configurations across all EditBench categories.
  • A.4 Reproducibility: For reproducibility, BLIP2 fine-tuning updates only Q-former parameters for two epochs, while zero-shot VQ2 processing 10,000 pairs takes days for candidate generation and question generation.
Loading 2305.10400v4…