PaperlayerView exact revision on arXiv

Source-linked AI summary

Judge Anything: MLLM as a Judge Across Any Modality

Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Zetong Zhou, Shuang Gong, Yi Gui, Yao Wan, Philip S. Yu

arXiv:2503.17489v1cs.CLcs.CV

TL;DR

Open-ended multimodal understanding and generation are difficult to evaluate consistently, and whether MLLM judges can extend reliably across modalities remains unclear. The paper introduces TASKANYTHING and JUDGEANYTHING, then reports stronger judging alignment on understanding than generation tasks, with biases and hallucinations still limiting reliability.

  • Problem

    Human evaluation of open-ended multimodal tasks is costly and inconsistent, and the extension of MLLM judging beyond text- and image-based tasks remains an open question.

    Source: §1 · pp. 1–2

  • Method

    The paper creates TASKANYTHING for MMU and MMG evaluation and JUDGEANYTHING to assess MLLM judges in pairwise and score settings against human judgments.

    Source: §1 · pp. 2–3

  • Results

    MLLM judges align more closely with human preferences on Pair Comparison than Score Evaluation and show stronger judging performance on MMU than MMG, especially struggling with video and audio generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MLLM judges only partially align with human judgment, and cross-modality biases and hallucinations continue to limit their reliability on open-ended any-to-any queries.

    Source: §1 · p. 3

Abstract

from arXiv · show

Evaluating generative foundation models on open-ended multimodal understanding (MMU) and generation (MMG) tasks across diverse modalities (e.g., images, audio, video) poses significant challenges due to the complexity of cross-modal interactions. To this end, the idea of utilizing Multimodal LLMs (MLLMs) as automated judges has emerged, with encouraging results in assessing vision-language understanding tasks. Moving further, this paper extends MLLM-as-a-Judge across modalities to a unified manner by introducing two benchmarks, TaskAnything and JudgeAnything, to respectively evaluate the overall performance and judging capabilities of MLLMs across any-to-any modality tasks. Specifically, TaskAnything evaluates the MMU and MMG capabilities across 15 any-to-any modality categories, employing 1,500 queries curated from well-established benchmarks. Furthermore, JudgeAnything evaluates the judging capabilities of 5 advanced (e.g., GPT-4o and Gemini-2.0-Flash) from the perspectives of Pair Comparison and Score Evaluation, providing a standardized testbed that incorporates human judgments and detailed rubrics. Our extensive experiments reveal that while these MLLMs show promise in assessing MMU (i.e., achieving an average of 66.55% in Pair Comparison setting and 42.79% in Score Evaluation setting), they encounter significant challenges with MMG tasks (i.e., averaging only 53.37% in Pair Comparison setting and 30.05% in Score Evaluation setting), exposing cross-modality biases and hallucination issues. To address this, we present OmniArena, an automated platform for evaluating omni-models and multimodal reward models. Our work highlights the need for fairer evaluation protocols and stronger alignment with human preferences. The source code and dataset are publicly available at: https://urrealhero.github.io/judgeanythingweb/.

1 Introduction

Human evaluation of open-ended multimodal tasks is costly and inconsistent, while the reach of MLLM judges beyond vision-language tasks remains an open question. The paper introduces benchmarks and an evaluation platform, finding that MLLM judges align better on understanding than generation tasks and remain limited by cross-modality biases and hallucinations.

1 Introduction

  • Human evaluation is time-consuming, expensive for video and audio, and prone to inconsistency because open-ended tasks lack absolute ground truths or shared criteria.Prior MLLM-as-a-Judge work has shown promise in vision-language settings, motivating the question of whether judging can extend to broader modalities.Source: §1 · pp. 1–2
  • TASKANYTHING evaluates MMU and MMG, while JUDGEANYTHING tests MLLM judges in Pair Comparison and Score Evaluation against human-annotated judgments and sample-specific checklists.The benchmarks cover 15 open-ended tasks and 1,500 queries sourced from established datasets.Source: §1 · pp. 2–3
  • Judges align more closely with human preferences on Pair Comparison than Score Evaluation and perform better on MMU than MMG, with video and audio generation particularly challenging.Gemini-1.5-Pro achieves 70.6% average Pair Comparison accuracy and 0.745 Pearson similarity on Score Evaluation in MMU tasks.Source: §1 · p. 2
  • OMNIARENA is presented as a standardized platform for evaluating omni-models and multimodal reward models using the two benchmarks.The reported initial comparison finds Gemini-1.5-Pro excels among omni-models in MMU, while ModaVerse performs best in MMG.Source: §1 · pp. 2–3
  • MLLM-as-a-Judge remains significantly limited for open-ended any-to-any queries, with cross-modality biases and hallucinations persisting despite improvements from rubrics and sample-wise checklists.The findings describe partial, rather than full, alignment with human judgment.Source: §1 · p. 3

2 TASKANYTHING and JUDGEANYTHING

The paper constructs TASKANYTHING for open-ended any-to-any evaluation and builds JUDGEANYTHING to test MLLM judges against human judgments. Its process spans benchmark curation, rubric and checklist design, judging-sample construction, and human annotation.

2.1 TASKANYTHING Construction

  • TASKANYTHING contains 1,500 open-ended queries, with 100 per task, selected or refined for quality, diversity, and natural free-form responses.Underexplored visual-to-audio queries are curated with a human-in-the-loop approach.Source: §2.1 · p. 4

2.2 Rubric and Checklist Design

  • Six principle rubrics guide sample-specific checklists, which Gemini-1.5-Pro drafts and humans manually reduce to 1–6 items.For video and audio, captions are supplied as context to refine checklists because Gemini-1.5-Pro has limited instruction-following capability in those modalities.Source: §2.2 · p. 4

2.3 Judging Sample Construction

  • Four state-of-the-art models generate 6,000 responses, which are manually reviewed before construction of score-rating and pairwise-comparison samples.Score Evaluation uses integer ratings from 1 to 5; Pair Comparison selects the better response or declares a tie.Source: §2.3 · p. 4
  • Dscore contains question-response pairs for absolute evaluation, while Dpair contains triples for comparing two different responses sampled from different models.The different-model sampling is intended to ensure diverse comparisons.Source: §2.3 · p. 4

2.4 Comparison with Human Annotations

  • Ten expert annotators provide ground-truth judgments and checklist annotations, with cross-validation for each sample and ongoing monitoring of annotator objectivity.The annotators vary in gender, age, and educational background.Source: §2.4 · p. 4, Table 2 · p. 4

2 TASKANYTHING and JUDGEANYTHING

  • The benchmarks cover 15 any-to-any combinations across text, image, video, and audio, with TASKANYTHING samples drawn from established benchmarks.State-of-the-art model responses are used to construct JUDGEANYTHING in Pair Comparison and Score Evaluation settings.Source: Figure 2 · p. 5

3 Experiments and Analysis

The experiments compare five MLLM judges with human judgments across Score Evaluation and Pair Comparison, using Overall, Rubric, and Checklist settings. Judges align better on MMU than MMG, while analyses identify modality, rubric relevance, and consistency as important qualifications.

3.1 Experimental Setup

  • The setup evaluates five advanced judges using Overall, Rubric, and human-in-the-loop Checklist baselines, with three repeated judgments aggregated for final results.Score Evaluation uses averaged scores; Pair Comparison uses majority selection.Source: §3.1 · pp. 5–6, §3 · p. 5

3.2 Quantitative Results

  • Gemini-1.5-Pro is the best evaluator, reaching 0.745 Pearson similarity in Score Evaluation and 70.60% agreement in Pair Comparison; evaluator-fusion achieves state-of-the-art alignment.Table 3 compares performance across Overall, Rubric, and Checklist blocks.Source: §3.2 · p. 6, Table 3 · p. 6
  • Across both settings, MLLM judgments align more closely with humans on MMU than MMG, while Checklist can improve alignment.The MMU–MMG difference appears in both Pair Comparison and Score Evaluation.Source: §3.2 · p. 7, Figure 3 · p. 7
  • Agreement declines across output modalities from text and images toward video and audio, while input-modality results show consistently low variance.The authors also report weaker performance on low-frequency cross-modal tasks such as image-to-audio and audio-to-video.Source: §3.2 · p. 7, Figure 4 · p. 7

3.3 In-Depth Analysis

  • Checklists can improve alignment but also mislead judges: one gun-image checklist prompted Gemini-1.5-Pro to treat the image as inherently harmful, while GPT-4o fabricated response details in MMG.The GPT-4o issue affected tasks including audio edit and video edit.Source: §3.2 · p. 7, §3.3 · p. 8
  • Fine-grained rubrics generally help Score Evaluation but can fail or harm Pair Comparison, especially when criteria poorly fit MMG tasks.In MMG, humans prioritize overall performance when rubric dimensions are less relevant, whereas judging models more often default to ties or detach rubric choices from overall preferences.Source: §3.3 · p. 9
  • Gemini-1.5-Pro has 0.9 consistency in Pair Comparison but 0.763 in Score Evaluation; Gemini-2.0-Flash exceeds 0.8 in Score Evaluation.Gemini-2.0-Flash has MCC ratios below 0.6 on some Pair Comparison rubrics, including clarity, coherence, and completeness.Source: §3.3 · p. 9, Figure 6 · p. 9

4 OMNIARENA

OMNIARENA uses TASKANYTHING and JUDGEANYTHING to assess omni-models through pairwise comparisons, with separate MMU and MMG arenas. Gemini-1.5-Pro judges matches, and ELO ratings track model performance.

OMNIARENA

  • The platform lets participants add omni-models, choose between two responses, and introduce questions for dynamic testing.Judges or users compare two responses, and the preferred outcome determines the pairwise match result.Source: §4 · p. 10

OMNIARENA Setups

  • OMNIARENA divides any-to-any tasks into MMU and MMG arenas and uses Gemini-1.5-Pro as judge based on its JUDGEANYTHING performance.ELO ratings update after each model match to provide a quantitative measure based on pairwise comparisons.Source: §4 · p. 10

Experiment Results

  • Gemini-1.5-Pro achieves the highest ELO score in the MMU arena, where specialized MMU models outperform omni-models.Next-GPT excels at MMU but underperforms at MMG; the paper attributes this discrepancy to limited control mechanisms and insufficient instruction-following ability.Source: §4 · p. 10

5 Discussion and Conclusion

The paper extends systematic MLLM-as-a-Judge assessment to 15 any-to-any MMU and MMG tasks and reports limitations and potential biases. It argues for more sophisticated cross-modal evaluation protocols that better capture human preferences.

5 Discussion and Conclusion

  • The paper assesses MLLM judges across 15 any-to-any task types in Pair Comparison and Score Evaluation, finding limitations, biases, and potential issues.It identifies the need for more sophisticated cross-modal protocols to capture nuanced human preferences.Source: §5 · p. 10
  • The authors hope the testbed will streamline evaluation, reduce dependence on human labor, and support development of more human-aligned any-to-any generative models.They call for protocols that better capture nuanced human preferences.Source: §5 · p. 10

A Related Work

  • MMU integrates information across modalities and supports tasks including visual question answering, multimodal captioning, and cross-modal retrieval.The passage describes MLLMs as using late fusion of modality features with pretrained LLMs and modality instruction tuning.Source: A Related Work · p. 18
  • Existing any-to-any benchmarks often assess MMG with Multiple-Choice QA, limiting evaluation of real-world open-ended queries.The cited benchmarks include Chen et al. (2024c), Li et al. (2024g), Ni et al. (2024), and Hong et al. (2025).Source: A Related Work · p. 18
  • MMG generates one modality from another or a mixture, and unified autoregressive models extend generation across varied modalities.Examples include text-to-image, text-to-video, image-to-video, and video-to-music.Source: A Related Work · p. 18
  • Open-ended MMG challenges ground-truth metrics, while crowdsourced evaluations can have too few votes for stable rankings.The paper describes checklist-of-thought reasoning as its approach to more unbiased, reliable, and reproducible multimodal evaluations.Source: A Related Work · p. 18
  • Prior MLLM-as-a-Judge work assessed bias and hallucinations in vision-language understanding; this work extends systematic assessment to 15 any-to-any MMU and MMG tasks.The passage also notes MLLMs used as reward models for aligning other MLLMs.Source: A Related Work · p. 19
  • Unified models use autoregressive next-token prediction with a pretrained LLM backbone to take and generate across modalities.The passage describes tokenizing continuous content into discrete tokens and exploring understanding and generation with a shared backbone.Source: A Related Work · p. 19

B.1 Benchmark Construction

The appendix describes query filtering, safety checks, and human annotation practices used in benchmark construction and review.

B.1 Benchmark Construction

  • Queries were randomly sampled from previous benchmarks and manually filtered for quality.Source: §B.1 · p. 19

B.2 Safety Checking

  • Safety checking covers NSFW content in text and multimodal data; Figure 8 summarizes the proportion of images judged safe or unsafe.Human annotators manually reviewed all videos and audio clips, classifying them as safe.Source: §B.2 · p. 19

B.3 Human Annotation Details

  • Twelve annotators—10 authors and two volunteers—worked independently, with varied backgrounds and tutorials on objective response evaluation.The paper notes annotator diversity as important for reducing bias and enhancing benchmark reliability.Source: §B.3 · p. 19

B.4 Copyright

The appendix explains the licensing choice and identifies tables detailing benchmark sources and pair-comparison choices.

B.4 Copyright

  • The authors release code, benchmark, and dataset under Creative Commons 4.0 to maintain compatibility with source benchmark and model licenses.Source: §B.4 · p. 19
  • Table 5 provides a detailed breakdown of benchmark sources and states that all samples are open-ended queries.Source: Table 5 · p. 20
  • Table 6 reports percentages of pair-comparison choices, labeling selections as first response, second response, or tie.Source: Table 6 · p. 20

B.5 ELO Rating System Details

The ELO system updates model ratings after each match using actual outcomes and expected win probabilities. In the arena, the rating-change constant K is set to 32.

B.5 ELO Rating System Details

  • PA and PB represent the expected probabilities of models 1 and 2 winning, while RA and RB are their current ELO ratings.Source: §B.5 · p. 20
  • After each comparison, the models’ ratings are updated using the difference between actual outcomes and expected win probabilities.The section introduces formulas for updating both models’ ratings after a match.Source: §B.5 · pp. 20–21
  • SA and SB encode match outcomes as 1 for a win, 0.5 for a tie, and 0 for a loss; K is set to 32.Source: §B.5 · p. 21

C Experiment Setup Details

The experiment setup describes output-evaluation rubrics and supporting materials for human annotation and multimodal understanding models. It also reports consistency figures for two Gemini judges under Overall and Rubric baselines.

C Experiment Setup Details

  • The seven evaluation dimensions are Overall, Relevance, Trustworthiness, Creativity and Novelty, Clarity, Coherence, and Completeness.They assess holistic effectiveness, prompt fit, reliability and safety, originality, understandability, logical flow, and coverage.Source: C Experiment Setup Details · p. 21

C.1 Models for Multimodal Understanding

  • The annotation materials include a screenshot and guideline, and the described interface requires applying edits before advancing to the next item.Users can move between the same task and rubric with Next and Previous.Source: §C.1 · p. 22, Figure 9 · p. 22
  • Figures 10 and 11 present Gemini-1.5-Pro and Gemini-2.0-Flash consistency data between rubrics, with Overall Choice and Overall Score from the Overall baseline and other results from the Rubric baseline.Source: Figure 10 · p. 23, Figure 11 · p. 23
  • Table 7 provides an overview of the MMU models used in the study.Source: Table 7 · p. 24

C.3 Models for Judge

The judge-model setup describes constraints in handling multimodal inputs. GPT-4o cannot process audio and visual inputs simultaneously, while open-source judges were excluded because of cross-modality and long-context limitations.

C.3 Models for Judge

  • GPT-4o cannot process audio and visual inputs simultaneously, so the setup integrates GPT-4o and GPT-4o-audio-preview.Source: §C.3 · p. 24
  • For audio-visual cross-modal tasks, the input modality is captioned into text so the response is accessible in visual or auditory form.Source: §C.3 · p. 24
  • Open-source models were not included as judges because of limitations with complex cross-modality inputs and long-context processing.The examples include interleaved audio-image data and multiple simultaneous media inputs.Source: §C.3 · p. 24

D Case Study

This section presents multimodal judging case studies across audio-to-video trustworthiness, video-to-audio creativity, and video-edit clarity, using pair comparison and score evaluation. The examples show how checklists shape evaluation across modalities, tasks, and criteria, including creativity-focused scoring and cases of hallucination or discrimination failures.

Case 1: Checklist improves Score Evaluation alignment

  • The creativity checklist frames Score Evaluation, while the example user gives a response score of 2 for identifying the bell but lacking creative expression.The checklist asks whether the description offers imaginative, original interpretations while remaining factually accurate.Source: D Case Study · p. 29, Figure 15 · p. 29

Case 2: Checklist leads to hallucination

  • A Score Evaluation trustworthiness checklist is presented in a case labeled as hallucination, asking whether the video depicts a pop sound without adding unsupported sounds or visuals.It also asks for responsible handling if the video depicts a gunshot.Source: D Case Study · p. 30, Figure 16 · p. 30

Case 3: GPT-4o discrimination failure in video-to-video task

  • A video-to-video Pair Comparison case labels GPT-4o’s performance a discrimination failure and lists relevance checks for aurora colors and preserving the original scene.The checklist asks whether the edited video depicts a predominantly red-and-yellow aurora and retains mountain silhouettes, water, and distant lights.Source: D Case Study · p. 31, Figure 17 · p. 31

Case 4: Analyzing Bias in Pair Comparison between Human and GPT-4o

  • For image-to-audio comparison, the user and checklist both choose the first track, which the user says sounds richer and more like a bass.The checklist compares bass-guitar sound, concert ambiance, and absence of misleading instruments; the user notes both tracks have bass-like sound and neither has misleading instruments.Source: D Case Study · p. 32, Figure 18 · p. 32

Cases 6–27

  • The remaining examples apply checklists to text, image, audio, and video understanding or generation, assessing dimensions such as clarity, completeness, relevance, trustworthiness, creativity, and coherence.Examples include text-to-text clarity and coherence, image-to-text completeness, audio-to-text relevance, video-to-text trustworthiness, and generation or editing tasks.Source: D Case Study · pp. 34, 36, 37, 39, 43, 46

D Case Study

  • Audio-to-video trustworthiness cases use pair comparison and score evaluation to assess whether generated video depicts a distinct pop accurately without unsupported additions or unsafe gunshot imagery.The checklist also allows added sounds or visuals when they enhance the described sound.Source: D Case Study · pp. 52–53
  • Video-to-audio creativity cases use pair comparison and score evaluation to judge emotional enhancement beyond human-like dog whimpering and cohesive audio-visual integration for audio generated from video.The prompt asks for corresponding audio based on the video’s visuals.Source: D Case Study · pp. 54–55
  • Video-edit clarity cases use pair comparison and score evaluation to assess red-and-yellow aurora coloring, smooth natural transitions, and freedom from flickering or distortion.The edit prompt asks for red and yellow aurora painting the night sky over mountain silhouettes.Source: D Case Study · p. 57
Loading 2503.17489v1…