PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaVA-Critic: Learning to Evaluate Multimodal Models

Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, Chunyuan Li

arXiv:2410.02712v2cs.CVcs.CL

TL;DR

Open LMMs had not been explored as general-purpose judges, despite the value of reliable multimodal evaluation for scalable assessment and reward signals. LLaVA-Critic pairs a critic instruction dataset with an open evaluator. Its judgments align with human and GPT-4o preferences across tasks, and its feedback improves preference learning.

  • Problem

    Open LMMs had not yet been explored as judges of multimodal model performance, while reliable evaluation can scale complex assessment and provide reward signals.

    Source: §1 · p. 1

  • Method

    The authors curate critic instruction-following data and train LLaVA-Critic, an open LMM that provides quantitative judgments and reasoning across evaluation settings.

    Source: §1 · p. 1

  • Results

    LLaVA-Critic’s pointwise scores and pairwise rankings align with human and GPT-4o preferences across multiple tasks, and its iterative DPO feedback outperforms LLaVA-RLHF’s preference signals.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    The authors release the critic data, code, checkpoints, and demo publicly to support development of general-purpose visual assistants.

    Source: §1 · p. 2

Abstract

from arXiv · show

We introduce LLaVA-Critic, the first open-source large multimodal model (LMM) designed as a generalist evaluator to assess performance across a wide range of multimodal tasks. LLaVA-Critic is trained using a high-quality critic instruction-following dataset that incorporates diverse evaluation criteria and scenarios. Our experiments demonstrate the model's effectiveness in two key areas: (1) LMM-as-a-Judge, where LLaVA-Critic provides reliable evaluation scores, performing on par with or surpassing GPT models on multiple evaluation benchmarks; and (2) Preference Learning, where it generates reward signals for preference learning, enhancing model alignment capabilities. This work underscores the potential of open-source LMMs in self-critique and evaluation, setting the stage for future research into scalable, superhuman alignment feedback mechanisms for LMMs.

1. Introduction

Reliable evaluation can reduce human effort in complex multimodal tasks and provide reward signals, but open LMMs had not yet been explored as general-purpose judges. LLaVA-Critic pairs a critic instruction dataset with an open model and reports aligned judgments and useful preference signals.

Goals

  • The work targets two uses: reliable LMM evaluation comparable to or better than GPT-4V/GPT-4o, and AI-generated reward signals for preference alignment.The authors frame the latter as reducing dependence on costly human feedback collection.Source: §1 · p. 1

Results

  • LLaVA-Critic’s scores and rankings correlate highly with commercial GPT models, and its feedback in iterative DPO outperforms LLaVA-RLHF’s preference signals.The LLaVA-RLHF reward model was trained using human feedback.Source: §1 · p. 1

Contributions

  • The 113k-sample critic dataset supports quantitative judgments with reasoning, while LLaVA-Critic extends open models to evaluation and feedback.The dataset covers 46k images and both pointwise and pairwise settings.Source: §1 · p. 1

Open-Source

  • The authors release the critic data, codebase, model checkpoints, and a trained visual-chat demo publicly.Source: §1 · p. 2

2. Related Work

Prior work uses proprietary multimodal models as judges and explores preference alignment for LMMs through human and AI feedback. LLaVA-Critic is positioned as an open generalist evaluator for these settings.

LMM-as-a-judge

  • Proprietary LMMs have judged visual-chat and detailed-caption responses using pointwise and pairwise evaluations; LLaVA-Critic is presented as an open generalist alternative with cheaper, customizable evaluation.Prometheus-Vision [14] is described as an open evaluator for specific user-designed scoring criteria.Source: §2 · p. 2

Preference learning for LMMs

  • LMM preference alignment includes human-scored rankings in LLaVA-RLHF [38], human feedback through RLHF-V [44], AI feedback through Silkie: VLFeedback [20], and self-rewarding approaches [45] [56] [55] [42].DPO [35] enables direct optimization using pairwise preference datasets.Source: §2 · p. 2

3. Data Collection

The authors curate pointwise and pairwise evaluation examples through a GPT-assisted pipeline to train a generalist critic. The resulting dataset spans diverse multimodal tasks and contains 113k samples across 46k images.

3. Data Collection

  • The GPT-assisted pipeline curates both pointwise scoring examples and pairwise rankings to equip LLaVA-Critic for general evaluation.Pointwise scoring evaluates one candidate response; pairwise ranking compares two candidates or declares a tie.Source: §3 · p. 2

3.1. Pointwise Data

  • Pointwise samples combine an image-question instruction, response, optional reference, evaluation criteria, score, and rationale; sources span eight instruction datasets.Responses come from VLFeedback’s 12 off-the-shelf LMMs, with GPT-4o responses used as reference answers.Source: §3.1 · p. 2

3.2. Pairwise Data

  • Pairwise examples draw on VLFeedback [20], RLHF [38], and RLHF-V [44], with GPT-4o generating reasons for preference judgments and 30 templates varying the evaluation prompts.Source: §3.2 · pp. 3–4
  • The training set contains 113k samples across 46k images, and its pointwise data is organized by instruction source and selected according to prompt task type.The training data comes from public instruction-following sets and does not overlap with evaluation benchmarks.Source: §3.2 · p. 4, Figure 1 · p. 4

4. LLaVA-Critic

LLaVA-Critic fine-tunes an instruction-following LMM to produce evaluation judgments and justifications. It is used both to judge model responses and to generate preference data for iterative DPO.

4.1. Model

  • The model is fine-tuned to predict pointwise scores or pairwise rankings and provide justifications, using cross-entropy loss for both judgments and explanations.Training starts from a pretrained LMM with broad instruction-following and vision-task capabilities.Source: §4.1 · p. 4

4.2. Scenario 1: LMM-as-a-Judge

  • LLaVA-Critic’s judgments align with GPT-4o or human evaluations across multimodal benchmarks for both instance-level scoring and model-level ranking.The authors describe automated evaluation as reducing labor costs.Source: §4.2 · p. 4
  • The judge evaluation covers visual chat, integrated capabilities, preferences, detailed image and video descriptions, and hallucination grounding.The cited examples include LLaVA-Bench, MM-Vet, WildVision-Bench [28], and MMHal-Bench [38].Source: §4.2 · p. 4

4.3. Scenario 2: Preference Learning

  • For preference learning, the critic scores all ordered pairs among K generated responses, aggregates relative scores into rewards, and selects the highest- and lowest-reward responses for DPO.The chosen best-worst response pair supplies the feedback data for alignment.Source: §4.3 · p. 5
  • After each DPO round, the updated LMM becomes the next checkpoint, and the critic-based feedback process repeats for further improvement.Source: §4.3 · p. 5

5. Experimental Results

LLaVA-Critic aligns with GPT-4o and human evaluators as a multimodal judge, and its preference signals improve multimodal models across benchmarks. The experiments also report gains from using the critic to select responses at inference time.

5.1. LMM-as-a-Judge

  • 0.754 average Pearson-r for LLaVA-Critic-72B versus 0.634 for LLaVA-OV-72B; LLaVA-Critic-7B reaches 0.732 and exceeds comparable open models.These are average pointwise-scoring results across seven multimodal benchmarks.Source: §5.1 · p. 5, Table 2 · p. 6
  • 73.6% average pairwise accuracy without ties for LLaVA-Critic-72B surpasses GPT-4V/4o; with ties it records 60.5% accuracy and 0.779 Kendall’s Tau.LLaVA-Critic-7B records 72.2% accuracy without ties and 0.763 Kendall’s Tau.Source: §5.1 · p. 6, Table 3 · p. 7
  • Compared with LLaVA-OneVision, LLaVA-Critic gives more accurate judgments and concrete, image-grounded justifications while largely preserving LLaVA-OV’s visual abilities.The paper also reports modest gains in visual chat performance.Source: §5.1 · p. 7
  • On MLLM-as-a-Judge, LLaVA-Critic-72B reaches 0.393 Pearson similarity with human annotators in pointwise scoring and pairwise accuracy of 57.8% with ties and 71.5% without ties.The reported results place its alignment with human evaluators near GPT-4V/4o across evaluation scenarios.Source: Table 4 · p. 7, §5.1 · p. 7

5.2. Preference Learning

  • With 9.4k prompts, LLaVA-Critic preferences yield +10.1 on LLaVA-W, +3.0 on LLaVA-Wilder, +8.8 on WildVision-Bench, and second-highest gains of +4.4 on LiveBench and +0.13 on MMHal-Bench.The paper reports that overall LLaVA-v1.5-7B capacities are largely preserved on other comprehensive benchmarks.Source: Table 5 · p. 8, §5.2 · p. 8
  • LLaVA-Critic preferences improve visual chat and reduce hallucination, outperforming other reward-model baselines on 5 of 6 benchmarks for 7B and all 6 for 72B.Preference alignment conducted solely with images also improves performance on video detailed captioning.Source: §5.2 · p. 7, Table 6 · p. 8
  • Best-of-5 response selection after iterative DPO adds +1.7 on LLaVA-W and +3.2 on LLaVA-Wilder.The selected responses come from the LLaVA-OV-7B checkpoint after three rounds of iterative DPO.Source: Table 7 · p. 8, §5.2 · p. 8

6. Conclusions

The conclusion presents LLaVA-Critic as an open-source evaluator whose scores, rankings, and preference signals support multimodal assessment and alignment. The appendix describes how pointwise and pairwise critic prompts are assembled.

6. Conclusions

  • LLaVA-Critic provides pointwise scores and pairwise rankings aligned with human and GPT-4o preferences, and reward signals that enhance visual chat beyond the LLaVA-RLHF reward model.The authors frame open-source self-critique as a step toward scalable, superhuman alignment feedback.Source: §6 · p. 8

B.1. Evaluation Prompts For Critic Training

  • Pointwise critic training reuses evaluation prompts from seven widely used multimodal benchmarks that employ GPT-as-a-judge.Source: §B.1 · p. 12
  • The pairwise prompt pool contains 30 evaluation templates designed for varied scenarios, format requirements, and evaluation criteria.Source: §B.1 · p. 12

B.2. In-domain Benchmarks for LMM-as-a-Judge

The in-domain benchmark evaluates LLaVA-Critic on responses from 13 off-the-shelf LMMs across seven multimodal benchmarks.

B.2. In-domain Benchmarks for LMM-as-a-Judge

  • The evaluation samples responses from 13 off-the-shelf LMMs spanning a wide range of visual-chat capabilities on seven multimodal benchmarks.The listed response models include GPT-4o, Claude3-Sonnet, and several open-source LMMs.Source: §B.2 · p. 12

C.1. Ablation on Data Curation

The data-curation ablation finds benefits from combining pointwise and pairwise critic data, with pointwise data and tie pairs contributing distinct advantages.

C.1. Ablation on Data Curation

  • Pointwise and pairwise critic data are mutually beneficial for training.Source: §C.1 · p. 12
  • Pointwise-only data improves pairwise ranking more than pairwise-only data improves pointwise judgments, because relative ranking lacks absolute judgments.Source: §C.1 · p. 12
  • Tie pairs further improve performance by preventing fixation on minor quality differences.Source: §C.1 · p. 12

C.2. Results on In-domain Pointwise Scoring

LLaVA-Critic aligns closely with GPT-4o in ranking model responses, with larger models and more data associated with stronger alignment. The section also describes evaluation resources and an implementation detail for MLLM-as-a-Judge results.

C.2. Results on In-domain Pointwise Scoring

  • 0.933 Kendall’s Tau is LLaVA-Critic-72B’s average alignment with GPT-4o, exceeding the LLaVA-OV-72B baseline (0.802).LLaVA-Critic-7B scores 0.911 and surpasses existing advanced LMMs of similar scale.Source: §C.2 · p. 12, Table 10 · p. 13
  • LLaVA-Critic’s alignment with GPT-4o benefits from both model scaling and data scaling.Source: §C.2 · p. 12
  • The pairwise training templates cover user-defined criteria and general evaluations without additional instructions.Source: Table 9 · p. 13

C.3. Results on MLLM-as-a-Judge

  • For MLLM-as-a-Judge results, the authors re-implemented the evaluation code because the complete pipeline was unavailable.Source: §C.3 · p. 15

C.4. Results on General Visual Tasks

Iterative DPO using LLaVA-Critic reward signals improves LLaVA-OneVision across six open-ended visual chat benchmarks. Critic training also largely preserves general visual-task capabilities, with gains on open-ended visual chat benchmarks.

C.4. Results on General Visual Tasks

  • LLaVA-Critic-7B outperforms LLaVA-OV-7B on 4 of 5 open-ended visual chat benchmarks, while Critic-72B outperforms OV-72B on all 5.Source: §C.4 · p. 15
  • Critic training largely preserves LLaVA-OneVision’s capabilities across comprehensive benchmarks, with slight improvements in visual chat performance.Source: §C.4 · p. 15

C.5. Results on Preference Learning

  • LLaVA-Critic improves LLaVA-OneVision across six open-ended visual chat benchmarks, with consistent gains per DPO round.On the LLaVA-OV-7B base model, improvement is comparable to GPT-4o while saving approximately $690 over three iterative DPO rounds.Source: §C.5 · p. 15, Figure 3 · p. 15
  • Outside visual chat, iterative DPO yields minimal benefit on other comprehensive benchmarks while largely preserving LLaVA-OneVision’s overall capabilities.Source: §C.5 · p. 15

D. More Qualitative Comparisons

Qualitative examples illustrate LLaVA-Critic’s visually grounded pairwise judgments and pointwise scoring, as well as preference alignment that changes LLaVA-OneVision’s response style.

D. More Qualitative Comparisons

Loading 2410.02712v2…