Source-linked AI summary
MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, Lichao Sun
TL;DR
MLLM evaluation lacks multimodal benchmarks aligned with human preferences, limiting assessment of their judging utility. This paper introduces MLLM-as-a-Judge, evaluating 11 models across Scoring Evaluation, Pair Comparison, and Batch Ranking. Models align better with humans in Pair Comparison than in the other settings, but biases, hallucinations, and inconsistent judgments remain obstacles to reliable evaluation.
Problem
Existing evaluation metrics and manual annotation do not adequately provide holistic, fine-grained, consistent, and practical assessment of multimodal model outputs.
Method
The paper introduces MLLM-as-a-Judge, a benchmark comparing MLLM judgments with human annotations across three multimodal evaluation settings.
Results
MLLMs align with human judgments in Pair Comparison but perform poorly in Scoring Evaluation and Batch Ranking, while GPT-4V consistently outperforms other models.
Takeaways & Limitations
MLLMs are not yet fully reliable judges, and the benchmark supports further refinement of multimodal judging capabilities.
Takeaways & Limitations
Hallucinations occur more frequently in Batch Ranking and involve misinterpretations and retrieval errors that affect judgment accuracy and reliability.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have gained significant attention recently, showing remarkable potential in artificial general intelligence. However, assessing the utility of MLLMs presents considerable challenges, primarily due to the absence of multimodal benchmarks that align with human preferences. Drawing inspiration from the concept of LLM-as-a-Judge within LLMs, this paper introduces a novel benchmark, termed MLLM-as-a-Judge, to assess the ability of MLLMs in assisting judges across diverse modalities, encompassing three distinct tasks: Scoring Evaluation, Pair Comparison, and Batch Ranking. Our study reveals that, while MLLMs demonstrate remarkable human-like discernment in Pair Comparison, there is a significant divergence from human preferences in Scoring Evaluation and Batch Ranking. Furthermore, a closer examination reveals persistent challenges in the judgment capacities of LLMs, including diverse biases, hallucinatory responses, and inconsistencies in judgment, even in advanced models such as GPT-4V. These findings emphasize the pressing need for enhancements and further research efforts to be undertaken before regarding MLLMs as fully reliable evaluators. In light of this, we advocate for additional efforts dedicated to supporting the continuous development within the domain of MLLM functioning as judges. The code and dataset are publicly available at our project homepage: \url{https://mllm-judge.github.io/}.
1. Introduction
The paper introduces MLLM-as-a-Judge to address shortcomings in multimodal evaluation and examines MLLMs across three judging settings. MLLMs align better with human judgments in Pair Comparison than in Scoring Evaluation or Batch Ranking, while biases, hallucinations, and inconsistency remain challenges.
- Traditional text-based metrics fail to capture nuanced, multimodal, and subjective model responses, motivating more holistic evaluation.Manual annotation is more human-like but costly, potentially biased, and difficult to replicate consistently.
- MLLM-as-a-Judge is a benchmark for assessing MLLMs as judges across Scoring Evaluation, Pair Comparison, and Batch Ranking.The study curates multimodal data and compares model judgments with human annotations.
- MLLM judgments exhibit egocentric, position, and length biases, hallucinations, and inconsistent judgments, although CoT and a vision expert system mitigate some biases.The paper releases HQ and HARD datasets to support further study of alignment and hallucination-related failures.
- 11 MLLMs are evaluated across 14 datasets and three settings, with GPT-4V consistently outperforming other models.The evaluation covers models with and without vision input and compares their judgments against human preferences.
- MLLMs align with human preferences in Pair Comparison but show notable discrepancies in Scoring Evaluation and Batch Ranking, especially for reasoning tasks.The benchmark includes human annotations designed to assess these three forms of judgment.
2. MLLM-as-a-Judge: A Benchmark to Assess Vision-Language Judging Ability
The benchmark constructs multimodal image-instruction-response data, obtains MLLM judgments, and compares them with human annotations across three evaluation settings.
- The benchmark collects image-instruction pairs from 10 diverse domains and forms image-instruction-response triples from MLLM-generated outputs.The resulting dataset represents each image, instruction, and set of model responses for later comparison.
- The three settings are Scoring Evaluation, Pair Comparison with ties, and Batch Ranking of responses in descending order.Scoring assigns 1–5 ratings, while Pair Comparison identifies the better response and permits ties.
- The curated dataset contains 4,414 image-text pairs adapted from downstream task datasets for free-form response evaluation.Examples include diffusion tasks requiring models to connect images with user-specified keywords.
- Six widely used MLLMs generate approximately 17,000 responses from the image-instruction pairs, excluding overly brief or non-compliant outputs.The models include GPT-4V, Gemini, LLaVA variants, Qwen-VL-Max, and CogVLM.
- Human annotation is conducted independently by six authors with varied backgrounds, using criteria intended to reduce bias from answer length, names, and response position.The annotators are instructed to provide objective judgments without considering those factors.
3. Experiment Settings
The study evaluates MLLMs as judges across three tasks and quantifies their agreement with human annotations using task-specific similarity metrics and human validation.
- Eleven MLLMs are evaluated across Scoring Evaluation, Pair Comparison, and Batch Ranking settings.
- The evaluation includes settings without vision input, with detailed image descriptions, and with multi-step Chain-of-Thought reasoning.
- Judgment alignment uses Pearson similarity for scoring, accuracy, F1-score, and recall for pair comparison, and Normalized Levenshtein distance for batch ranking.
- Human validation examines agreement, analysis quality, and hallucinations in MLLM judgments.
- Responses are repeatedly sampled and averaged in the overall comparison, with Pearson results reported as statistically significant at p < 0.05.
4. Empirical Results and Analysis
MLLMs align most strongly with human preferences in Pair Comparison, while Scoring Evaluation and Batch Ranking remain weaker and exhibit biases, hallucinations, and consistency problems.
- GPT-4V consistently outperforms other models across tasks and settings, while Pair Comparison shows stronger human alignment than Scoring Evaluation and Batch Ranking.
- Scoring Evaluation: 0.490 is GPT-4V’s highest Scoring Evaluation similarity, compared with 0.304 for Gemini.
- Pair Comparison: 0.636 and 0.773 are GPT-4V’s Pair Comparison scores in tie and non-tie settings, respectively, with many datasets exceeding 0.8.
- Batch Ranking: 0.361 is GPT-4V’s mean Levenshtein Distance for Batch Ranking, although all MLLMs still have substantial room for improvement.
- Human Agreement: GPT-4V reaches 78% human agreement in Pair Comparison and 70% in Scoring Evaluation, while Batch Ranking alignment decreases, especially for Maths and graphic information processing.
- Multi-step CoT: Three-step CoT reduces hallucinations but does not improve human-preference alignment and can reduce judging performance, especially for Gemini.
- Vision Descriptions: Picture descriptions significantly improve traditional LLM judging performance in multimodal tasks.
- Bias and Hallucination: Models exhibit egocentric, position, and length biases; GPT-4V and Gemini increase scores for longer answers, with average gains of 0.6 and 0.75 points, respectively.
5. Related Work
Prior work established LLM-based judging and studied hallucination and bias, while multimodal settings add visual-perception challenges that require further investigation.
- LLM-as-a-Judge research developed evaluators such as JudgeLM and AUTO-J, supported by advances in Chain-of-Thought reasoning and instruction following.
- MLLM judgment research addresses vision-language hallucinations and biases including position, egocentric, and verbosity biases.
- Integrating visual perception compounds existing judgment biases and motivates further investigation.
6. Future Directions
Future work should use the benchmark’s annotations for multimodal alignment training, improve reasoning and bias control, and introduce human oversight when judgments are unreliable.
- The curated human annotations can support RLHF reward models and provide paired data for DPO training.
- More sophisticated reasoning frameworks, including multi-agent debating, may improve judging accuracy beyond longer Chain-of-Thought prompts.
- Position bias and high-score tendencies remain critical areas for improvement in MLLM judging.
- Human intervention is proposed when repeated judgments vary substantially or outputs do not comply with the required format.
7. Conclusion
The paper introduces MLLM-as-a-Judge to evaluate multimodal judging across Scoring Evaluation, Pair Comparison, and Batch Ranking, comparing model judgments with humans. Results show stronger human alignment in Pair Comparison than in Scoring Evaluation and Batch Ranking, motivating further refinement.
- MLLM-as-a-Judge evaluates multimodal judging capabilities across Scoring Evaluation, Pair Comparison, and Batch Ranking.
- Advanced MLLMs achieve substantial human recognition in Pair Comparison but perform poorly in Scoring Evaluation and Batch Ranking.
- The findings identify potential areas for future refinement and improvement of MLLMs as judges.
Impact Statement
The paper positions MLLM-as-a-Judge as a benchmark for human-aligned multimodal evaluation while acknowledging biases in both human annotation and MLLMs. It also situates the work within prior research on LLM judging, hallucination, bias, and conventional evaluation metrics.
- The benchmark is designed to move MLLMs toward judgments that align more closely with human perspectives.
- A stated limitation is bias in both human annotation and MLLMs, while ethically principled and socially beneficial systems remain future work.
- Prior work includes JudgeLM and AUTO-J, which evaluate language models through fine-tuning, pairwise comparison, and single-response assessment.
- MLLM hallucinations can contain information absent from visual or natural-language context and may stem from vision-language training misalignment.
- MLLM judging exhibits position, egocentric, and verbosity biases, with multimodal perception introducing additional complexities.
- Conventional metrics often fail on sophisticated visual-language tasks, while manual annotation is comprehensive but costly and potentially biased.
B. Detailed Benchmark Construction
The benchmark construction organizes multimodal evaluation around diverse capabilities and datasets, then collects model responses for analysis. It also records response-length distributions across the resulting dataset collection.
- The evaluation framework examines recognition, comprehension, inferential, and multilingual abilities in MLLMs.
- Ten diverse datasets are incorporated to broaden task coverage and challenge different MLLM capabilities while helping prevent bias.
- The sampling procedure randomly selects 300 images per dataset, with MathVista contributing 600 instructions covering hinted and hint-free cases.
- Four mainstream MLLMs generate responses for the initial 3,300 image-instruction pairs, producing 13,200 answers.
- The full response collection reaches 17,656 responses, after additional models are used for later datasets and the sequential Mementos dataset.
- Figure 8 depicts response-length distributions across datasets, with length on the horizontal axis and density on the vertical axis.
C. Detailed Experiment Settings
The experiments configure several MLLMs for response generation and judging, using model-specific inference settings and retries or exclusions for invalid, refused, or noncompliant outputs. Gemini also exhibits occasional multilingual output during judging, while some other models cannot reliably follow judging instructions.
- The response-generation pool includes GPT-4V, Gemini, LLaVA-1.5-13b, CogVLM, Qwen-VL-Max, and LLaVA-1.6-34b.
- Gemini uses default settings but may refuse diffusion questions, fail difficult questions, or forcibly answer them despite uncertainty.
- LLaVA-1.5-13b uses temperature 0, top-p 1, a 2,048-token limit, and beam search of 3 because it otherwise struggles with the required judge format.
- Gemini occasionally outputs Chinese or Japanese for English inputs, disrupting judge continuity despite semantic continuity after translation.
- CogVLM and LLaVA cannot follow judging instructions well in the basic Analyze-then-Judge setting, so their results are excluded from the main text.
D.1. Full Results on Judging Performance
The full results compare MLLM judging across standard and sequential-image settings, while examining output structure, human agreement, response length, consistency, and bias. Results show persistent task- and model-dependent differences in alignment and reliability.
- LLaVA-1.6-34b and Qwen-vl-max slightly outperform other models on Math, Chart, and Text scoring tasks, but neither surpasses GPT-4V.
- GPT-4V significantly outperforms other models on the sequential Mementos image dataset across batch evaluation, pair comparison, and score evaluation.The experiment uses sequential images from daily life, comics, and robotics.
- 50 randomly selected instances from three datasets are used to assess human preferences for GPT-4V’s Json Mode outputs.Three annotators label the balance between structured JSON responses and more fluid human-preferred responses.
- Annotators show minimal variance when judging whether GPT-4’s judgments are correct, but greater divergence when scoring analytical responses.The scoring discrepancies do not significantly compromise the overall integrity of the annotations.
- A decision tree for hallucination identification produces strong consensus and uniformity across annotations.The authors connect this consistency to the use of well-defined criteria.
- GPT-4V generally produces longer responses than Gemini in scoring evaluation and pair comparison, while their batch-task output lengths are comparatively similar.
- Mean Absolute Deviation measures variation in MLLM judgments across multiple responses for one image-instruction pair, and the study also examines egocentric bias.The cited analysis presents MAD as a consistency measure and reports egocentric-bias analyses in Figures 19 and 20.
G. Case Study
The case studies illustrate how MLLM judges compare, score, and rank multimodal responses using correctness, detail, explanation, and supporting evidence. In the examples, judges favor responses that provide more comprehensive reasoning while preserving accurate answers.
- An example of score comparison: The score-comparison case evaluates a response describing a toy train in a park against criteria including relevance, accuracy, comprehensiveness, creativity, and granularity.
- An example of pair comparison: The pair-comparison case contrasts two correct chart-reading answers, with one adding Steve Largent’s second-place touchdown total as supporting context.
- An example of pair comparison: In pair comparison, Judge VLM 2 selects Assistant B because its accurate answer adds comparative detail about Steve Largent.Both assistants identify Shaun Alexander as the Seattle Seahawks’ career touchdown leader with 112 touchdowns.
- An example of batch ranking: The batch-ranking example asks judges to order four answers identifying the year at the top of a magazine image.
- An example of batch ranking: For the magazine-reading batch example, both judges rank Assistant C first, followed by D, B, and A.The ranking favors detailed answers with supporting evidence over concise answers that provide no explanation.