Source-linked AI summary
Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
Tianyi Xiong, Yi Ge, Ming Li, Zuolong Zhang, Pranav Kulkarni, Kaishen Wang, Qi He, Zeying Zhu, Chenxi Liu, Ruibo Chen, Tong Zheng, Yanshuo Chen, Xiyao Wang, Renrui Zhang, Wenhu Chen, Heng Huang
TL;DR
Multimodal judges are increasingly used, but their ability to follow fine-grained, diverse criteria remains underexplored. Multi-Crit benchmarks pluralistic criteria-following using human-annotated response pairs and dedicated metrics, and finds that current judges—especially in open-ended evaluation—struggle with consistent adherence.
Problem
Existing multimodal judge benchmarks mainly provide one overall preference per response pair, leaving criterion-specific judgments and conflicts between criteria underexplored.
Source: §1 · p. 2, §3.1 · p. 3Method
Multi-Crit curates open-ended and reasoning prompts with challenging response pairs and multi-criterion human preferences, then assesses adherence, trade-off sensitivity, and conflict recognition.
Source: §1 · p. 2Results
Takeaways & Limitations
Multi-Crit provides a benchmark for systematic analysis of multimodal judges’ adherence to diverse criteria, trade-off recognition, and conflict resolution.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation.
1. Introduction
Multi-Crit addresses the underexplored question of whether multimodal judges can follow diverse evaluation criteria and identify conflicts among criterion-level judgments. Across 25 LMMs, the benchmark exposes persistent limits in pluralistic criteria-following.
1. Introduction
- Existing multimodal judge benchmarks mainly use a single overall preference label, leaving criterion-specific judgments and conflicts between criteria underexplored.The paper motivates this gap with response pairs that trade off qualities, such as factual correctness versus visual grounding.Source: §1 · p. 2
- Multi-Crit evaluates multimodal judges on diverse criteria and criterion-level judgments aligned with human evaluators.Its annotations capture distinct aspects of response quality and potential conflicts between criteria.Source: §1 · p. 2
- The curation pipeline gathers prompts for open-ended generation and verifiable reasoning, selects challenging response pairs, defines complementary criteria, and collects human annotations.The pipeline aims to ensure annotation reliability and inter-annotator consistency.Source: §1 · p. 2
- Three metrics assess pluralistic adherence (PAcc), sensitivity to criterion-level trade-offs (TOS), and recognition of criterion-level preference conflicts (CMR).Source: §1 · p. 2
- Across 25 LMMs, proprietary judges struggle with pluralistic criteria-following, open-source models lag further behind, and holistic critic fine-tuning generalizes poorly to criterion-driven evaluation.The strongest proprietary models achieve 32.78% on open-ended and 53.17% on reasoning judgments; additional analyses find reasoning fine-tuning does not enhance reasoning judgment and weakens trade-off recognition, while test-time scaling effects are inconsistent across models.Source: §1 · p. 2
- Multi-Crit is presented as a first benchmark for pluralistic multimodal judging, intended to inform more reliable and steerable evaluation.Source: §1 · p. 2
2. Related Work
Prior multimodal judge benchmarks assess overall preferences or critique correctness, while criterion-following research has only begun to address text-only judges. Multi-Crit extends pluralistic criteria-following analysis to multimodal judges.
2. Related Work
- Multimodal judges produce scalar scores or pairwise preferences, and prior work has developed both reward models with dedicated heads and generative judges with textual justifications.Multi-Crit centers on generative judges because of their flexible instruction-following abilities, while its methodology is described as broadly applicable across reward-modeling paradigms.Source: §2 · p. 2
- Existing multimodal benchmarks compare paired responses against one overall human or verifier preference, leaving diverse criteria and criterion-level conflicts unevaluated.MM-Critic additionally evaluates the correctness of generated critiques using GPT-generated scoring rubrics.Source: §2 · p. 3
- Prior criterion-following studies focus on pure-text LLM judges; this work extends the line to multimodal judges and preference conflicts.Source: §2 · p. 3
3. Multi-Crit Benchmark
Multi-Crit benchmarks multimodal judges on criterion-level preferences across open-ended and verifiable reasoning tasks. It combines curated, human-annotated response pairs with metrics for pluralistic accuracy, trade-off sensitivity, and conflict matching.
3. Multi-Crit Benchmark
- Multi-Crit provides a benchmark with diverse prompts, response pairs from 11 LMMs, and fine-grained multi-criterion human annotations.The benchmark spans open-ended and reasoning domains and introduces three metrics for criterion adherence, trade-offs, and preference conflicts.Source: §3 · p. 3
3.1. Task Formulation
- Multi-Crit extends one-dimensional response preferences to criterion-specific human preferences, allowing the same response pair to reflect conflicting judgments.Each criterion is paired with its own preferred response label.Source: §3.1 · p. 3
3.2. Prompt and Response Pair Curation
- The benchmark covers open-ended generation and verifiable reasoning, drawing prompts from diverse visual-conversation, captioning, math, STEM, coding, and puzzle domains.Candidate responses come from 11 LMMs; filtering reduces 3,538 pairs to 707 challenging pairs by removing length-imbalanced, unanimously judged, or trivially distinguishable examples.Source: §3.2 · pp. 3–4
3.4. Data Statistics
- 425 prompts and 1,425 criterion-level judgments include 315 prompts with conflicts, comprising 782 conflicting criterion pairs.Table 1 presents key benchmark statistics and Figure 3 shows prompt-source and evaluation-criteria distributions.Source: Table 1 · p. 4, §3.4 · p. 5
3.3. Criteria-Specific Preference Annotation
- Criteria are designed for practicality, specificity, and generality, while human annotation uses calibrated, cross-validated judgments and filters for agreement and criterion conflicts.Nine Ph.D. annotators contribute 289 hours; inter-annotator Cohen’s κ is 0.718 for open-ended and 0.805 for reasoning tasks.Source: §3.3 · pp. 4–5
3.5. Measuring Pluralistic Criteria Following
- PAcc measures correctness across criteria, TOS measures recognition of conflicting preferences, and CMR requires both judgments in a conflict to match human preferences.TOS reflects flexibility rather than absolute accuracy; CMR is stricter because it measures whether detected conflicts are resolved consistently with human judgments.Source: §3.5 · p. 5
4. Experiments
Experiments across 25 multimodal judges show that pluralistic criterion-following remains difficult, with proprietary models leading open-source and critic-tuned models but still showing weaknesses across tasks and criteria. The results also show that visual-grounding gains from critic fine-tuning do not reliably extend to pluralistic judgments.
4. Experiments
- The evaluation covers 25 proprietary, open-source, and fine-tuned judges, prompted to judge one response pair at a time under a single criterion.The benchmark results are reported for open-ended responses and verifiable reasoning.Source: §4 · p. 5, §4.1 · p. 5, Table 4 · p. 7
4.1. Main Results
- 32.78% and 31.77% are the highest open-ended PAcc scores, while o4-mini and GPT-5 lead reasoning with 53.17% and 45.24%, respectively.The tables report criterion-level judgments; o4-mini's open-ended PAcc is identified as the top result.Source: §4.1 · p. 6, Table 3 · p. 6, Table 4 · p. 7
- No model leads across all criteria: reasoning-task strengths differ by criterion, and GPT-4o's best trade-off sensitivity coexists with weaker conflict resolution than o4-mini.On reasoning, o4-mini leads Logic (75.58%) and Efficiency (88.07%), while o3 leads Hallucination (84.21%) and Exploration (84.62%), and Gemini-2.5-Pro leads Grounding (79.01%); GPT-4o's CMR is 55.16% versus o4-mini's 65.84%.Source: §4.1 · pp. 6–7, Figure 4 · p. 6
- Open-source models trail proprietary judges across prompt-level metrics and criterion accuracies, with larger CMR declines than accuracy declines across both task splits.CMR drops about 9.4 points in open-ended tasks (43.1 →33.7) and 18.1 points in reasoning tasks (65.8 →47.7).Source: §4.1 · p. 7
- Critic fine-tuning does not consistently improve pluralistic judgments, although Qwen-based fine-tuned judges improve visual-grounding accuracy across both splits.R1-Reward is the only Qwen-based fine-tuned judge with higher PAcc than its base model on both splits, while its trade-off and conflict improvements are inconsistent.Source: §4.1 · pp. 7–8
1. Does RLVR on multimodal reasoning improve judgment in reasoning traces?
Reasoning fine-tuning improves general reasoning performance, but the evaluated models show weaker recognition of criterion-level trade-offs and conflicts. Test-time majority voting improves the strongest model's pluralistic accuracy on both task splits, while gains are model-dependent.
1. Does RLVR on multimodal reasoning improve judgment in reasoning traces?
- GRPO-fine-tuned models gain on general reasoning tasks but show reduced trade-off sensitivity and conflict matching in reasoning judgments.Figure 5 reports results for RL-tuned reasoning models based on Qwen2.5-VL-7B.Source: §1 · p. 8, Figure 5 · p. 8
2. Does test-time scaling hold?
- 32.78→37.12 and 53.17→57.94 are o4-mini's open-ended and reasoning PAcc changes under test-time majority voting.GPT-4o shows similar gains, while other models show inconsistent trends and large variance.Source: §1 · p. 8, Figure 6 · p. 8
- The scaling test runs each judge K times, takes a majority vote for each criterion sample, and then computes pluralistic accuracy.The reported benefits are limited and model-dependent.Source: §1 · p. 8
3. What are the current boundaries of criteria following for open-source and proprietary models?
The authors compare each model group's strongest criterion accuracies with human inter-annotator agreement, excluding reasoning-efficiency because it is strongly influenced by response length. Proprietary models show a stronger reported correlation than open-source models.
3. What are the current boundaries of criteria following for open-source and proprietary models?
- Proprietary models' strongest criterion accuracies correlate with human agreement at r=0.73, p=0.024, compared with r=0.36, p=0.344 for open-source models.Figure 7 plots criterion-level top judge accuracy against inter-annotator agreement; reasoning-efficiency is excluded from the correlation analysis.Source: §3 · p. 8, Figure 7 · p. 8
- The authors interpret the weaker, less consistent open-source relationship as reflecting limited capacity to internalize human-consistent evaluation criteria.They suggest scaling high-quality human annotations could benefit open-source pluralistic criteria-following.Source: §3 · p. 8
- Reasoning-efficiency is excluded because its agreement pattern is largely influenced by response length.Source: §3 · p. 8
5. Conclusion
Multi-Crit provides a benchmark for multimodal judges using fine-grained multi-criterion human annotations and complementary pluralistic metrics. Experiments expose limitations in proprietary, open-source, and critic-tuned judges, while additional analyses probe reasoning fine-tuning, test-time scaling, and human alignment.
5. Conclusion
- Multi-Crit evaluates multimodal judges in pluralistic, criterion-level settings using human annotations and metrics for adherence, trade-off recognition, and conflict resolution.The benchmark supports systematic analysis of judges' adherence to diverse criteria, trade-off recognition, and conflict resolution.Source: §5 · p. 8
- Experiments find proprietary judges struggle particularly on open-ended tasks, open-source models lag further behind, and critic fine-tuning fails to generalize broadly from visual grounding.The conclusion also notes diminished trade-off recognition after reasoning fine-tuning, inconsistent test-time scaling, and human-aligned upper bounds for proprietary models.Source: §5 · p. 8
A. Overview of the Appendix
- The appendix overview points readers to benchmark comparisons, additional benchmark details, qualitative examples, and supplementary results.Source: A. Overview of the Appendix · p. 13
B. Comparison Against Prior Benchmarks
Multi-Crit differs from earlier benchmarks by providing multiple criterion-level human preferences for each response pair, enabling evaluation of pluralistic criterion-following. Its benchmark construction also specifies how responses are selected and how criteria and judge prompts are presented.
B. Comparison Against Prior Benchmarks
- Multi-Crit provides multiple criterion-level human annotations where earlier benchmarks assign one overall preference label, and candidate responses average 164 vs. 99 words.This enables assessment of pluralistic criteria-following and makes the benchmark more challenging.Source: B. Comparison Against Prior Benchmarks · p. 13, Table 5 · p. 13
C.1. Additional Benchmark Details
- Candidate response pairs include WildVision-Battle pairs when human preferences tie; otherwise, new responses are generated using listed LMMs.The listed generators include Gemini-2.5, GPT-4.1, GPT-4o, Qwen2.5-VL, LLaMA-3.2-Vision, InternVL3, MiMo-VL, and GLM-4.1V variants.Source: §C.1 · p. 13
- Evaluation criteria are organized by definitions, positive indicators, and negative indicators, with five criteria each for open-ended generation and verifiable reasoning.The judge prompt presents one target criterion per inference and is adapted for fine-tuned judges to match their original prompt formats.Source: §C.1 · p. 13
- Each judge inference receives the prompt, response pair, and target criterion through a template, adapted for fine-tuned judges to match their original repository formats.This adaptation is intended to avoid significant out-of-distribution judgment behaviors.Source: §C.1 · p. 13
C.2. Additional Data Statistics
Multi-Crit combines cross-model and within-model response pairs and includes preference conflicts across every criterion pair in both task splits. Its prompts span diverse open-ended and verifiable reasoning domains.
C.2. Additional Data Statistics
- Roughly 40% of response pairs in each split come from the same model, while the rest come from different models, covering both sampling and model diversity.Within-model pairs control for model identity and surface style; cross-model pairs reflect deployment scenarios with responses differing along multiple dimensions.Source: §C.2 · p. 13, Table 6 · p. 13
- Human preference conflicts occur across all criterion pairs in both splits; completeness vs. no-hallucination and no-hallucination vs. expressiveness are frequent open-ended conflicts, while exploration vs. efficiency is frequent in reasoning.Figures 8 and 9 depict preference correlations and conflict counts, respectively.Source: §C.2 · p. 13, Figure 8 · p. 16, Figure 9 · p. 16
- The benchmark defines five evaluation criteria for each task split and instructs judges to assess the target criterion exclusively in each inference.The criteria are listed in Tables 7 and 8; the inference instruction appears in Table 9.Source: Table 7 · p. 14, Table 8 · p. 15, Table 9 · p. 16
D.1. Multimodal Prompts
- Multi-Crit prompts cover captioning, storytelling, knowledge-based tasks, and text-rich understanding, as well as math, science, coding, and visual analogy.These examples span open-ended generation and verifiable reasoning use cases for LMM judges.Source: §D.1 · p. 17
D.2. Full Evaluation Instances
Multi-Crit’s examples illustrate fine-grained response differences and criterion-level preference conflicts across open-ended and verifiable reasoning tasks. In the thinking-mode analysis, smaller models improve across metrics, while gains are more limited for larger models and stronger for reasoning judgments.
D.2. Full Evaluation Instances
- The six evaluation instances demonstrate fine-grained response differences and preference conflicts within pairs across both task types.The examples include open-ended storytelling, creative-image captioning, and instruction-rich image analysis, plus math, object counting, and biological reasoning.Source: §D.2 · p. 17, Figure 10 · p. 18
- Open-ended examples show conflicts including completeness vs. grounding and expressiveness vs. clarity, including for responses sampled from the same model.The instruction-rich image-analysis pair is sampled from GPT-4.1 and contrasts background detail and hallucination with instruction-following and clarity.Source: §D.2 · p. 17, Table 10 · p. 19, Table 11 · p. 20
- Reasoning examples contrast visual grounding, logical consistency, concision, detail, and reflection, including cases where answers are both incorrect or both correct.The math traces are both incorrect; the object-counting responses and biological reasoning responses reach correct conclusions.Source: §D.2 · p. 17, Table 13 · p. 22, Table 14 · p. 23
E.1. To Think or Not?
- Thinking improves smaller 8B judges across all metrics on both splits, while larger models show more modest or mixed effects.For ∼30B models, InternVL3.5 shows no significant improvement, while Qwen3-VL improves on reasoning but slightly declines on open-ended tasks.Source: §E.1 · p. 25
- Thinking judges gain more on verifiable reasoning than open-ended tasks, and Qwen3-VL-Thinking outperforms its Instruct variant at both 8B and 32B scales.The passage contrasts these gains with domain-specific RL fine-tuning, which did not improve reasoning judgments and weakened trade-off recognition.Source: §E.1 · p. 25
E.2. Joint Multi-Criterion Judgment
Joint multi-criterion judgment affects proprietary models unevenly and usually reduces sensitivity to criterion-level trade-offs. The comparison isolates prompting format by keeping inference hyper-parameters identical to the standard single-criterion setup.
E.2. Joint Multi-Criterion Judgment
- The joint setting presents all applicable criteria together and requests criterion-level judgments in one inference, while the standard setting uses separate single-criterion inferences.Inference hyper-parameters remain identical across settings to isolate the effect of joint prompting.Source: §E.2 · p. 25, Table 17 · p. 27
- Joint prompting improves GPT-5’s pluralistic accuracy and alignment on conflict cases, but GPT-4o drops across metrics; o4-mini and Gemini-2.5-Pro have mixed open-ended effects and consistent reasoning declines.The comparison is shown in Table 18.Source: §E.2 · p. 25, Table 18 · p. 27
- Trade-off sensitivity decreases for every tested model except GPT-5 on reasoning, as joint judgments can induce inter-criterion dependencies and uniform preference directions.In the Table 19 example, o4-mini captures a completeness vs. no-hallucination conflict with separate prompts but not joint prompting.Source: §E.2 · p. 25, Table 19 · p. 28