PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks

Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, Linda Ruth Petzold

arXiv:2311.01361v1cs.CVcs.CL

TL;DR

Vision-language metrics can be limited in reference availability, flexible evaluation, and handling multiple image-text pairs, while GPT-4V's evaluator role has not been systematically explored. The paper evaluates GPT-4V across varied multimodal tasks and methods, reporting promising agreement with humans alongside limitations in visual clarity assessment.

  • Problem

    Existing vision-language metrics may require references, lack free-form evaluation and explanations, and evaluate only one image-text pair at a time.

    Source: §1 · p. 1

  • Method

    The paper systematically evaluates GPT-4V across image captioning, text-to-image generation, text-guided editing, and multiple-image-to-text alignment using single-answer grading and pairwise comparison.

    Source: §1 · p. 2, §2 · p. 2, §6 · p. 13

  • Results

    GPT-4V achieves promising agreement with human evaluators across different tasks and is reported as a generally good reference-free, human-aligned evaluator.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    GPT-4V's human-aligned scores and detailed explanations support its potential as a multimodal evaluator.

    Source: §1 · p. 2

    GPT-4V may struggle to discern subtle visual-clarity differences and evaluate fine details in visual quality.

    Source: §5 · p. 12

Abstract

from arXiv · show

Automatically evaluating vision-language tasks is challenging, especially when it comes to reflecting human judgments due to limitations in accounting for fine-grained details. Although GPT-4V has shown promising results in various multi-modal tasks, leveraging GPT-4V as a generalist evaluator for these tasks has not yet been systematically explored. We comprehensively validate GPT-4V's capabilities for evaluation purposes, addressing tasks ranging from foundational image-to-text and text-to-image synthesis to high-level image-to-image translations and multi-images to text alignment. We employ two evaluation methods, single-answer grading and pairwise comparison, using GPT-4V. Notably, GPT-4V shows promising agreement with humans across various tasks and evaluation methods, demonstrating immense potential for multi-modal LLMs as evaluators. Despite limitations like restricted visual clarity grading and real-world complex reasoning, its ability to provide human-aligned scores enriched with detailed explanations is promising for universal automatic evaluator.

1 Introduction

Existing vision-language metrics can require references, lack flexible scoring and explanations, and handle only one image-text pair at a time. The paper studies GPT-4V as a generalist evaluator across vision-language tasks, reporting human-aligned evaluation while noting limits in visual clarity and complex real-world cases.

1 Introduction

  • Existing metrics can require references, lack free-form scoring and explanations, and evaluate only one image-text pair at a time.These limits motivate the question of a universal automatic evaluator for diverse tasks.Source: §1 · p. 1
  • The paper systematically studies how well GPT-4V aligns with human preferences when evaluating diverse vision-language tasks.The authors note that GPT-4V's evaluator role remains under-explored.Source: §1 · p. 2
  • The evaluation spans image captioning, text-to-image generation, and text-guided image editing.The paper also identifies multiple-image-to-text alignment among its tasks.Source: §1 · p. 2
  • GPT-4V is reported as a generally good reference-free, human-aligned evaluator that can provide fine-grained explanations, but visual clarity and complex real-world cases remain limitations.The authors identify perceptual evaluation, including vision clarity, as a challenge.Source: §1 · p. 2

2 GPT-4V-as-a-Generalist-Evaluator

The paper uses GPT-4V to assess multimodal input-output pairs through single-answer grading and pairwise comparison. The methods differ in whether candidates are scored independently or compared directly, with a scalability trade-off for pairwise evaluation.

2 GPT-4V-as-a-Generalist-Evaluator

  • GPT-4V evaluates task-specific multimodal input-output pairs using single-answer grading or pairwise comparison.The input and output modalities vary by task; for image captioning the input is an image and output is text, while synthetic image generation reverses those roles.Source: §2 · p. 2
  • Single-answer grading scores each candidate independently, while pairwise comparison directly selects the better candidate or a tie.For grading, the higher-scored candidate is preferred, and equal scores yield a tie.Source: §2 · p. 3
  • Single-answer grading scales more readily as candidate counts grow, whereas pairwise comparison enables direct comparison but has quadratic computational complexity.Single-answer grading does not directly compare candidates for the same task input.Source: §2 · p. 3
  • The evaluation compares GPT-4V and human assessments by categorizing outcomes as Answer 1, Answer 2, or Tie and measuring their alignment.For pairwise comparisons, the answer order is alternated and each comparison is evaluated twice to mitigate potential position bias.Source: §2 · p. 3

3 Evaluation

The evaluation tests GPT-4V on four multimodal tasks using single-answer grading and pairwise comparison, assessing agreement with human judgments and the challenges of each task.

3 Evaluation

  • The study tests whether GPT-4V can serve as a reasonable evaluator with explanations for multimodal tasks, conducting experiments on four different tasks to address four main questions.Source: §3 · p. 3

3.1 Image-to-Text Captioning

  • GPT-4V scores correlate significantly with human ratings on both caption subsets, outperform CLIPScore, and agree 91% across single-answer and pairwise evaluations.GPT-4V’s grading scores show greater variation than human scores, which the authors attribute to nucleus decoding.Source: §3.1 · pp. 4–5, Table 2 · p. 4, Table 1 · p. 4

3.2 Text-to-Image Generation

  • Both GPT-4V grading modes exceed 50% human agreement (33% random baseline), but cross-method agreement is weaker; visual-clarity grading and complex prompts remain difficult.The reported failure cases include prompts with multiple stringent requirements or intensive background-knowledge demands.Source: §3.2 · pp. 6, 8, Table 4 · p. 6

3.3 Text-guided Image Editing

  • GPT-4V generally correlates with human semantic-consistency ratings and shares their preference for MagicBrush over InstructPix2Pix, but sometimes misses perceptual realism.The comparison with specialized metrics is reported under the same evaluation guidelines.Source: §3.3 · p. 9, Table 5 · p. 9

3.4 Multiple Images to Text Alignment

  • For multi-image-to-text alignment, GPT-4V’s scores correlate with human scores at around 0.8, and its generated descriptions accurately cover each image separately.The evaluation uses Google Local reviews paired with two to four images.Source: §3.4 · pp. 9–10, Table 6 · p. 10

4 Related Work

Prior work develops multimodal LLMs through two broad architecture lines and explores reference-free metrics for vision-language tasks. The paper positions GPT-4V as a potential general-purpose evaluator where existing metrics do not adapt to diverse tasks and guidelines.

4 Related Work

  • Multimodal LLMs are developed either as one-stage models pretrained on mixed multimodal corpora or as two-stage models aligning vision and language before instruction tuning.The two-stage approach uses feature alignment followed by multimodal instruction tuning.Source: §4 · pp. 10–11
  • One-stage examples process images as patches or use CLIP image embeddings, while two-stage systems combine pretrained vision encoders with instruction-following language models.The passage names Fuyu and Kosmos as one-stage examples, and MiniGPT-4 and LLaVA as two-stage examples.Source: §4 · p. 11
  • Vision-language evaluation includes reference-based and reference-free metrics, but the cited metrics are not described as adapting to diverse guidelines and tasks.The paper explores GPT-4V as a universal reference-free metric aligned with human preferences across a broad domain.Source: §4 · p. 11

5 Discussion

The discussion identifies limits in evaluating complex image-generation prompts, visual clarity, and expertise-dependent content. It also reports minimal position bias in image-to-text and text-to-image evaluation, with the two tasks exhibiting different patterns.

5 Discussion

  • GPT-4V performs better on image-to-text than text-to-image, a difference the authors attribute potentially to long, cluttered generation captions mixing abstract and detailed requirements.Image-to-text outputs are described as easier to understand, more generalized, and well-formatted.Source: §5 · p. 12
  • GPT-4V may miss subtle visual-clarity differences; human evaluators penalized indistinct animal faces, while image-to-text appears less affected because it emphasizes sharpness less.Human evaluators gave the example image 25 out of 30 for clarity.Source: §5 · p. 12
  • GPT-4V gave a higher score to Synthesized Image 2 than Image 1 despite Image 1 more closely matching Canaletto’s painting style.The authors identify the evaluation as requiring specific expertise.Source: §5 · p. 12
  • Position bias was minimal in image-to-text and text-to-image; hard-negative captions with subtle noise reduced potential bias in the former, while the latter showed nearly balanced bias across positions.Table 7 describes consistency under answer-order swaps and bias toward the first or second answer.Source: §5 · pp. 12–13, Table 7 · p. 13

6 Conclusion

The paper evaluates GPT-4V across four multimodal tasks and reports promising agreement with human evaluators. It presents these findings as a foundation for an MLLM-based evaluation framework.

6 Conclusion

  • GPT-4V shows promising agreement with human evaluators across image captioning, text-to-image generation, text-guided image editing, and multiple-image-to-text alignment.The authors describe this as laying a robust foundation for an MLLM-based evaluation framework.Source: §6 · p. 13

A Image-to-Text Captioning

The appendix prompts define two image-caption evaluation formats: scoring one caption against an image and directly choosing between two captions. The single-answer prompt uses a 1–100 scale, while pairwise comparison permits a tie.

A.1 Prompt

  • Single-answer grading asks whether a caption conveys an image’s overall theme or subject, then scores its match from 1 to 100.The prompt says the caption need not describe every image detail.Source: §A.1 · p. 16
  • Pairwise comparison selects the better-aligned caption for an image or returns a tie when both appear equally good.The prompt requires the final response to be caption 0, caption 1, or Tie.Source: §A.1 · p. 16

A.2 More Showcases

The appendix shows image-to-text grading examples and gives prompts for evaluating generated images through single-answer criteria or pairwise comparison.

A.2 More Showcases

B.1 Prompt

  • Single-answer grading scores generated images for relevance (0–40), visual clarity (0–30), and object accuracy (0–30).The prompt asks whether the image matches the prompt, is clear and sharp, and accurately depicts the named objects.Source: §B.1 · p. 20
  • Pairwise comparison asks respondents to choose image 0 (left) or image 1 (right), based on which aligns most accurately with the instructions, and to respond “Tie” if both seem equally good.Source: §B.1 · p. 20

B.2 More Showcases

The appendix presents text-to-image evaluation examples, while the accompanying guideline scores semantic consistency and perceptual realism on three-level scales.

B.2 More Showcases

C.1 Prompt

  • Semantic consistency (SC) scores alignment with the provided guidance as 0, 0.5, or 1, with 1 indicating that over 75% of requirements are followed.Source: §C.1 · p. 27
  • Perceptual realism (PR) scores real-world visual convincingness as 0, 0.5, or 1, allowing some artifacts at the middle score.The guideline describes a score of 1 as generally looking real, without requiring perfection.Source: §C.1 · p. 27

C.2 More Showcases

Figures 18–20 showcase GPT-4V evaluating text-guided image editing.

C.2 More Showcases

D Multiple Images to Text Alignment

The multiple-images-to-text task evaluates whether a user review captures the images’ main content, and illustrates the evaluation with examples.

D Multiple Images to Text Alignment

  • Single-answer grading evaluates whether a user review accurately represents the uploaded images’ main context and objects, then rates relevance on a 1-100 scale, where 100 is a perfect match.Generic reviews without details, e.g., "food is great" should receive a relatively low score.Source: D Multiple Images to Text Alignment · p. 31
  • Figure 21 provides examples of GPT-4V evaluating the multiple-images-to-text task.Source: Figure 21 · p. 31
Loading 2311.01361v1…