PaperlayerView exact revision on arXiv

Source-linked AI summary

Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation

Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, Minjoon Seo

arXiv:2401.06591v1cs.CL

TL;DR

Evaluating long-form VLM responses requires checking both instruction-following and image grounding, while conventional text metrics miss granular criteria and output context. The paper trains an open-source VLM evaluator on the Perception Collection to assess responses using user-defined criteria; it reports the highest GPT-4V correlation among open-source models across eight benchmarks.

  • Problem

    Assessing long-form VLM responses requires checking instruction-following and image grounding, while text-match metrics miss granular criteria and rich output context.

    Source: §1 · p. 1

  • Method

    The authors train an open-source VLM evaluator, Prometheus-Vision, using the Perception Collection so it can assess responses against user-defined criteria.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    Across 8 benchmarks, Prometheus-Vision had the highest correlation with GPT-4V among open-source models and outperformed GPT-4 (LM judge) on 5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Prometheus-Vision supports more accessible and transparent VLM evaluation while adhering to user-defined criteria.

    Source: §7 · p. 10

    Prometheus-Vision does not perform optimally on text-rich images, and the Perception Collection is skewed toward real-world rather than text-rich images.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision

1 Introduction

Evaluating long-form VLM responses requires checking both instruction following and image grounding, while conventional metrics miss nuanced criteria and human evaluation is costly to scale. The paper introduces a fine-grained feedback dataset and evaluator VLM, reporting strong agreement with human evaluators and GPT-4V.

1 Introduction

  • Traditional text-match and edit-distance metrics miss granular criteria and cannot explain what is missing from a response.Human evaluation captures contextual and creative aspects, but is costly and inconsistent to scale.Source: §1 · p. 1, Figure 1 · p. 2
  • LM judges cannot perceive images, so adding an image-to-text model can introduce error propagation and require multiple inference calls.The paper motivates direct evaluation with VLMs instead.Source: §1 · p. 1
  • The PERCEPTION COLLECTION provides 15K fine-grained score rubrics, and the authors fine-tune LLaVA-1.5 on it to create PROMETHEUS-VISION.The dataset is designed to go beyond coarse criteria such as helpfulness, relevance, accuracy, and comprehensiveness.Source: §1 · pp. 1–2
  • Across eight benchmarks, PROMETHEUS-VISION has the highest GPT-4V correlation among open-source models and outperforms GPT-4 (LM judge) on five.On LLaVA-Bench, its Pearson correlation is 0.786 with human evaluators and 0.639 with GPT-4V.Source: §1 · p. 2
  • PROMETHEUS-VISION is presented as the first open-source VLM specialized for evaluation, with high correlation to human evaluators and GPT-4V.The authors describe it as a potential inexpensive alternative to GPT-4V evaluation.Source: §1 · p. 2

2 Related works

Prior VLM evaluation uses task-specific text-matching metrics or human assessment, while fine-grained LM judging has chiefly addressed language tasks. The paper positions its work as extending fine-grained evaluation to VLMs.

2.1 Evaluating Vision Language Models

  • Image-captioning metrics compare generated text with reference captions, while VQA commonly uses exact-match accuracy against human answers.Source: §2.1 · p. 3
  • Human evaluation captures contextual and creative response qualities that metrics miss, but cost and consistency constrain its scalability.Source: §2.1 · p. 3

2.2 Language Model as a Judge for Fine-grained Evaluation

  • Prior fine-grained LM evaluation uses core skill sets, thousands of user-defined criteria, alignment criteria, or interactive testing.Source: §2.2 · p. 3
  • The authors identify prior VLM judges as limited to coarse criteria and present PERCEPTION COLLECTION's 15K fine-grained criteria to train PROMETHEUS-VISION.Source: §2.2 · p. 3

3 The PERCEPTION COLLECTION

The PERCEPTION COLLECTION pairs multimodal response-evaluation examples with customized fine-grained rubrics, feedback, and scores. The authors construct the dataset and fine-tune LLaVA-1.5 to produce PROMETHEUS-VISION.

3 The PERCEPTION COLLECTION

  • Each example combines an image, instruction, response, customized rubric, and reference answer with language feedback and a score decision.The rubric describes the criteria and scoring decisions on a 1-to-5 scale; the reference answer represents a score of 5.Source: §3 · p. 3

3.1 PERCEPTION COLLECTION Construction

  • The four-stage construction process expands 50 seed rubrics into 15K rubrics, 30K instructions and reference answers, and 150K responses and feedback.GPT-4V generates and checks rubric variants, then augments the associated instructions, reference answers, responses, and feedback.Source: §3.1 · pp. 4–5

3.2 Fine-tuning a VLM as an Evaluator

  • Training fine-tunes LLaVA-1.5 (7B and 13B) to generate feedback as a rationale followed sequentially by a score.A fixed phrase separates feedback from the score and was found to prevent degeneration during inference.Source: §3.2 · p. 5

4 Experimental Settings

The experiments compare evaluator models by how closely their scoring decisions align with human evaluators and GPT-4V. The settings specify evaluator and response-model baselines, benchmarks, and two evaluation setups.

4.1 Protocol for Evaluating Evaluator VLMs

  • The evaluation protocol compares PROMETHEUS-VISION with human judgments and GPT-4V judgments as standards for nuanced assessment.The paper describes evaluator quality as a non-trivial problem to measure directly and uses these two indirect comparisons.Source: §4.1 · p. 5

4.2 Evaluator VLM & LM Baselines

  • Evaluator baselines include nine VLMs and four LMs, with LLaVA-1.5 captions supplied to LMs because they cannot take images as input.VLM evaluators receive the image directly; the VLM baselines include PROMETHEUS-VISION, and the LM baselines include PROMETHEUS, GPT-3.5-TURBO, and GPT-4.Source: §4.2 · p. 5

4.3 Response VLMs

  • In experiments, three VLMs sampled outputs for VLM evaluators to assess: FUYU (8B), LLAVA-1.5 (13B), and GPT-4V served as response VLMs.Source: §4.3 · p. 5

4.4 Benchmarks

  • Benchmarks cover visual instruction following, VQA, and captioning; GPT-4V adds rubrics and reference answers to instruction-following sets and rubrics to VQA and captioning sets.Instruction-following benchmarks lack inherent reference answers, whereas VQA and captioning use their original ground-truth answers as references.Source: §4.4 · pp. 5–6

4.5 Setups & Metrics

  • Setup 1 uses 45 instances with hand-crafted rubrics for human correlation and feedback-preference comparisons; Setup 2 uses 1,085 rubrics across 3,560 instances and repeated GPT-4V and evaluator scoring.Both setups report Pearson, Kendall-Tau, and Spearman correlations; Setup 1 also measures pairwise preference win rate.Source: §4.5 · p. 6

5 Experimental Results

The experiments compare PROMETHEUS-VISION with human evaluators, other evaluator models, and GPT-4V across feedback-quality judgments and multiple VLM tasks. Results show strong correlations and competitive feedback quality, alongside weaker performance on text-rich VISIT-BENCH and shorter-answer VQA settings.

5.1.1 Correlation with Human Evaluators

  • PROMETHEUS-VISION mirrors GPT-4/GPT-4V correlations on LLaVA-Bench and PERCEPTION-BENCH (0.639 and 0.870, respectively), but trails them on VISIT-BENCH.On VISIT-BENCH, it slightly outperforms GPT-3.5-TURBO and PROMETHEUS-13B.Source: §5.1.1 · pp. 6–7, Figure 3 · p. 7
  • VISIT-BENCH contains more text-rich images than the other benchmarks, and the authors attribute PROMETHEUS-VISION’s correlation disparity primarily to these dataset differences.The collection includes relatively few instruction sets for text-rich images, and the model architecture challenges their processing during inference.Source: §5.1.1 · p. 7

5.1.2 Comparison of the Quality of the Feedback

  • Human annotators rated PROMETHEUS-VISION feedback as good as or better than GPT-4V’s 57.78% of the time and GPT-4’s 45.93% of the time.The comparison covered 135 instances.Source: §5.1.2 · p. 7, Figure 4 · p. 7

5.2.1 Visual Instruction Following Benchmarks

  • Training raised correlation with GPT-4V over LLaVA-1.5 across all three visual instruction-following benchmarks and both model sizes; the 13B model also exceeded GPT-3.5-Turbo and GPT-4 on two benchmarks.The two benchmarks were LLaVA-Bench and PERCEPTION-BENCH.Source: §5.2.1 · p. 8

5.2.2 Visual Question Answering Benchmarks

  • PROMETHEUS-VISION significantly outperforms other open-source models on VQA, but its correlations are generally lower there than on visual instruction-following benchmarks.The authors attribute the difference to training data’s longer responses versus VQA’s mostly short answers.Source: §5.2.2 · pp. 8–9, Table 5 · p. 8

5.2.3 Captioning Benchmarks

  • On COCO-Captions, PROMETHEUS-VISION 13B achieves a correlation above 0.5, while most evaluators, including proprietary LMs, show low correlation.The authors present this as evidence of potential generalization to visual-language tasks beyond training data.Source: §5.2.3 · p. 9, Table 6 · p. 9

6 Analysis of Potential Biases from VLM Evaluators

The analysis finds no indiscriminate preference for longer responses by GPT-4V or PROMETHEUS-VISION in absolute grading. The models show similar evaluation patterns across two benchmarks, though potential self-enhancement from GPT-4V-augmented training remains unresolved.

6.1 Is there a Length Bias?

  • GPT-4V and PROMETHEUS-VISION do not indiscriminately favor longer responses in absolute grading, where evaluators assign scores rather than compare responses.The authors connect this finding to earlier work by Zheng et al. (2023) and Kim et al. (2023d).Source: §6.1 · p. 9, Figure 5 · p. 9

6.2 Is there a Self-Enhancement Bias?

  • Across LLaVA-Bench and PERCEPTION-BENCH, PROMETHEUS-VISION and GPT-4V show similar evaluation patterns; PROMETHEUS-VISION scores other models above its LLaVA-v1.5 backbone on LLaVA-Bench.Although it gives GPT-4V the highest score, the authors cannot determine whether this reflects GPT-4V-augmented training or GPT-4V’s performance.Source: §6.2 · p. 10, Figure 6 · p. 10
  • PERCEPTION-BENCH widens LLaVA-RLHF’s score gap from GPT-4V to 1.43 points, compared with 0.14 points on LLaVA-Bench; the authors identify it as a potential VLM-development testbed.PERCEPTION-BENCH uses fine-grained rubrics, more complex instructions, and extended responses.Source: §6.2 · p. 10

7 Conclusion

The paper extends LM-as-a-Judge to VLM-as-a-Judge, using a multimodal feedback dataset with instance-specific criteria to train an open-source evaluator that follows user-defined evaluation criteria.

7 Conclusion

  • VLM-as-a-Judge extends the LM-as-a-Judge paradigm to multimodal evaluation.The paper introduces a multimodal feedback dataset and trains Prometheus-Vision, an open-source model specialized for evaluation that can adhere to user-defined criteria.Source: §7 · p. 10
  • The Perception Collection provides unique score criteria for each instance, unlike existing multimodal datasets that do not strongly account for evaluation values users consider important.It is used to train Prometheus-Vision.Source: §7 · p. 10
  • Prometheus-Vision is presented as a route toward accessible and transparent evaluation of VLMs.The paper hopes the work will encourage further research on open-source evaluators in other modalities.Source: §7 · p. 10

Limitations

The authors identify limitations around text-rich and AI-generated images, while reporting analyses of rubric diversity and score-description decisiveness.

Limitations

  • Prometheus-Vision does not perform optimally on text-rich images, and its training collection is heavily skewed toward real-world rather than text-rich images.The authors attribute this limitation partly to the visual encoder in LLaVA-v1.5 and suggest more text-rich-image feedback data as future work.Source: Limitations · p. 10
  • The evaluation does not cover VLM outputs conditioned on AI-generated images, which the authors leave for future exploration.The stated scope boundary is the type of input image, not a reported failure on those images.Source: Limitations · p. 10
  • The authors note that publicly available multimodal feedback datasets are scarce compared with text-only datasets.They identify investigation of other feedback, preference, and critique datasets as future work.Source: Limitations · pp. 10–11

A.1 Diversity of Score Rubrics

  • Rubric keywords distinguish broad holistic perspectives in general-purpose rubrics from visual aspects and long-tail subject areas in domain-specific rubrics.The reported within-collection rubric-description ROUGE-L average is 0.31, with a left-skewed distribution.Source: §A.1 · p. 14

A.2 Decisiveness of Score Descriptions

  • Descriptions for score 1 are generally more negative, while descriptions for score 5 are more positive, suggesting score descriptions are interpolated by score.The authors interpret this pattern as supporting clear absolute scoring by models trained on the dataset.Source: §A.2 · p. 14, Figure 8 · p. 14

A.3 Length Bias of Responses per Score Provided for Training

The training data aims to reduce response-length bias across scores, and the benchmark’s score rubrics show low overlap with training rubrics.

B.1 Validity of Unseen Score Rubrics

  • 0.29 average ROUGE-L similarity separates PERCEPTION-BENCH rubric descriptions from PERCEPTION-COLLECTION, and the authors characterize train-test overlap as low.The left-skewed distribution is shown in Figure 10.Source: Figure 10 · p. 15, §B.1 · p. 15

A.3 Length Bias of Responses per Score Provided for Training

  • To counter the tendency of recent language models to give higher scores to longer responses, the collection aims for similar response lengths across scores 1–5.Responses are generated for every score level given an instruction, rubric, and reference answer.Source: §A.3 · pp. 14–15
  • Response lengths are distributed evenly across score levels, averaging 417 words; Figure 9 plots the distribution by score.This is the reported outcome of the length-balancing aim during collection construction.Source: §A.3 · p. 15

C Comparison with conventional metrics

Conventional response metrics do not account for image information and correlate less with human scores than Prometheus-Vision in the reported comparison. Prometheus-Vision also provides explanatory feedback alongside a numeric score.

C Comparison with conventional metrics

  • Conventional metrics compare responses with references without considering images, and their scores can diverge from human evaluations.The passage links this gap to varied information in images.Source: C Comparison with conventional metrics · p. 15
  • 0.674 Pearson correlation for PROMETHEUS-VISION 13B exceeds around 0.489 for METEOR, the highest-correlating conventional metric, against human scores.Table 7 reports Pearson correlations with human scores on 45 samples from three visual instruction-following benchmarks.Source: Table 7 · p. 15, C Comparison with conventional metrics · p. 15
  • Unlike conventional metrics’ simple 0-to-1 value, PROMETHEUS-VISION provides a numeric score and feedback explaining the reasons behind it.The authors say this dual output can help identify ways to improve a model’s response.Source: C Comparison with conventional metrics · p. 15

D.1 Implementation Details and Computation

This section describes Prometheus-Vision’s training and inference setup, then details a pairwise human evaluation of feedback from three evaluators.

D.1 Implementation Details and Computation

  • PROMETHEUS-VISION uses LLaVA-1.5 7B/13B; it freezes the language model and vision encoder, trains only an MLP-based alignment network for one epoch, with per-device batch size 32 and learning rate 1e-3.Source: §D.1 · p. 15
  • At inference, each Evaluator VLM receives the image, question, reference answer, rubric, and response, then generates feedback and a score.Generation used temperature 1.0, top-p 0.9, and a 2048-token maximum.Source: §D.1 · p. 17

D.2 Details in Human Evaluation

  • 9 English-proficient undergraduates compared feedback from GPT-4V, GPT-4, and PROMETHEUS-VISION 13B across 45 sampled benchmark problems.The problems came from LLaVA-Bench, VisIT-Bench, and Perception-Bench; annotations used Label Studio.Source: §D.2 · pp. 17–18

E Length Bias during Evaluation

The reported evaluator distributions show no response-length bias, while accompanying passages document rubric and evaluation prompts and qualitative comparisons.

E Length Bias during Evaluation

  • Neither GPT-4V nor PROMETHEUS-VISION 13B displays response-length bias during inference across the test sets.Figure 13 presents GPT-4V response-length distributions by score, as evaluated by both Evaluator VLMs.Source: E Length Bias during Evaluation · p. 18

F.1 Prompts for PERCEPTION COLLECTION Creation

  • The collection prompts incorporate sampled hand-crafted seed rubrics and augment general-purpose and domain-specific rubrics separately, with images included in GPT-4V calls.The prompt text itself does not include an image insertion.Source: §F.1 · p. 18

Prompt for rubric augmentation

  • The rubric-augmentation prompt asks GPT-4V to brainstorm three diverse, practical criteria and scoring rubrics for evaluating image-conditioned VLM text.It requests JSON-formatted output without surrounding text.Source: Prompt for rubric augmentation · p. 18

Prompt for refining rubric

  • The refinement prompt directs rubric changes toward image-answerable criteria and novelty when the existing rubric needs improvement.It requests a single rubric in the prescribed format.Source: Prompt for refining rubric · pp. 19–20

Prompt for generating instruction (2)

  • The instruction-generation prompt asks for a novel problem and response related to the supplied score rubric and image.The prompt fragment also supplies the rubric as an input.Source: Prompt for refining rubric · p. 20

Prompt for response and feedback (2)

F.2 Prompts for PROMETHEUS-VISION

  • PROMETHEUS-VISION’s evaluation prompt supplies the instruction, response, score-5 reference, image, rubric, and criteria for scores 1–5, then requests rubric-based feedback and an integer score.Source: §F.2 · p. 22

G Qualitative Examples

Loading 2401.06591v1…