PaperlayerView exact revision on arXiv

Source-linked AI summary

LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, Ziwei Liu

arXiv:2407.12772v2cs.CLcs.CV

TL;DR

Multimodal model evaluation lacks comprehensive, standardized evidence despite the need for broad and reliable assessment. The paper introduces LMMS-EVAL, LMMS-EVAL LITE, and LIVEBENCH to address this gap through standardized, efficient, and dynamic evaluation designs. It concludes that the evaluation trilemma remains unresolved and requires trade-offs among coverage, cost, and contamination.

  • Problem

    Comprehensive and unified evaluation studies for large multimodal models remain limited, while existing protocols differ in ways that hinder transparent and reproducible comparison.

  • Method

    The paper develops LMMS-EVAL, prunes it into LMMS-EVAL LITE, and builds LIVEBENCH from continuously updated news and forum content.

  • Results

    The unified suite scales to over 50 tasks and more than 10 models, while the evaluation trilemma remains unresolved and requires trade-offs.

  • Takeaways & Limitations

    LMMS-EVAL, LMMS-EVAL LITE, and LIVEBENCH provide complementary ways to balance standardized coverage, evaluation efficiency, and resistance to contamination.

  • Takeaways & Limitations

    The paper uses a relatively simple contamination method requiring access to training data, and LIVEBENCH question-answer quality may remain below human-curated answers.

Abstract

from arXiv · show

The advances of large foundation models necessitate wide-coverage, low-cost, and zero-contamination benchmarks. Despite continuous exploration of language model evaluations, comprehensive studies on the evaluation of Large Multi-modal Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified and standardized multimodal benchmark framework with over 50 tasks and more than 10 models to promote transparent and reproducible evaluations. Although LMMS-EVAL offers comprehensive coverage, we find it still falls short in achieving low cost and zero contamination. To approach this evaluation trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that emphasizes both coverage and efficiency. Additionally, we present Multimodal LIVEBENCH that utilizes continuously updating news and online forums to assess models' generalization abilities in the wild, featuring a low-cost and zero-contamination evaluation approach. In summary, our work highlights the importance of considering the evaluation trilemma and provides practical solutions to navigate the trade-offs in evaluating large multi-modal models, paving the way for more effective and reliable benchmarking of LMMs. We opensource our codebase and maintain leaderboard of LIVEBENCH at https://github.com/EvolvingLMMs-Lab/lmms-eval and https://huggingface.co/spaces/lmms-lab/LiveBench.

1 Introduction

The paper argues that multimodal models need unified, transparent evaluation because existing LMM protocols vary substantially. It introduces LMMS-EVAL, LMMS-EVAL LITE, and LIVEBENCH to navigate trade-offs among coverage, cost, and contamination.

  • Existing LMM evaluation lacks a unified protocol, with differences in data preparation, postprocessing, and metric calculation hindering transparency and reproducibility.
  • LMMS-EVAL standardizes multimodal evaluation across over 50 tasks, more than 10 models, and around 30 variants.
  • LMMS-EVAL LITE prunes unnecessary data instances to reduce evaluation cost while maintaining evaluation quality.
  • LIVEBENCH uses recent news and forum information to provide a low-cost, generalizable evaluation of multimodal models.
  • Wide coverage, low cost, and zero contamination are difficult to achieve simultaneously in benchmarks.

2 LMMS-EVAL: A Unified Multimodal Models Evaluation Suite

LMMS-EVAL addresses inefficient and inconsistent model-by-model evaluation by providing a standardized framework for large-scale comparisons. Scaling reveals the evaluation trilemma: broad, inexpensive, and contamination-resistant assessment cannot be fully achieved at once.

  • Scaling Evaluations with a Standardized Framework: Custom evaluation pipelines require separate scripts for preprocessing, inference, and scoring, creating substantial overhead across many models and datasets.
  • Scaling Evaluations with a Standardized Framework: LMMS-EVAL unifies evaluation setups and fixes comparison procedures because perplexity-based and generation-based scoring can produce non-comparable results.
  • Scaling Evaluations with a Standardized Framework: The standardized pipeline scaled evaluation to over 10 models and more than 50 datasets, enabling broader comparisons across model capabilities.
  • The Evaluation Trilemma: Once evaluation expands to 50+ datasets, full runs become time-consuming and remain susceptible to training-time contamination.
  • The Evaluation Trilemma: The framework improves evaluation trade-offs but does not fully resolve the tension among wide coverage, low cost, and zero contamination.

3 LMMS-EVAL LITE: Affordable Evaluation with Broad Domain Coverage

LMMS-EVAL LITE is designed as a faster, broad-coverage complement to the full benchmark. It selects representative instances and evaluates whether Lite scores preserve the full benchmark’s signals across models.

  • Lite benchmark construction: The Lite set is intended to preserve absolute scores and relative model rankings while providing faster development-time signals.
  • Lite set selection: The selection objective is reformulated as a k-Center problem, and a greedy algorithm provides an efficient 2-OPT solution because exact optimization is NP-hard.
  • Lite set selection: Across six LLaVA versions, the selected subset maintains decent correlation with original benchmark scores.
  • Lite benchmark construction: LMMS-EVAL LITE selects 15 datasets across task domains to provide broad coverage at lower evaluation cost.
  • Lite benchmark construction: For datasets exceeding 1500 data points, the method selects representative instances, while retaining the full MME benchmark because its score correlation is low.
  • Score Aggregation: Aggregated Lite scores normalize each dataset’s score to a range of 100 before averaging, allowing comparison before and after pruning.

4 LIVEBENCH: From Static to Live Evaluation

LIVEBENCH addresses contamination and evaluation-cost concerns by using dynamically updated web information, while contamination analysis exposes overlap in existing multimodal benchmarks and training data. Its results suggest that current static benchmarks under-assess robust commercial-model capabilities.

  • 4.1 Probing into Multimodal Data Contamination: Over 20% overlap was found for ChartQA, VQAv2, COCO2014, and GQA, whose training sets were confirmed in LLaVA’s training data.The analysis compares benchmark data with LLaVA’s training data to identify potential contamination.
  • 4.1 Probing into Multimodal Data Contamination: Three contamination patterns were observed: duplicate images, visually similar images, and similar question structures.These patterns span both image and text overlap between training and benchmark data.
  • 4.2 Multimodal LiveBench: Traditional static benchmarks can favor open-source models despite weaker real-user experience, while dynamic arenas introduce prompt-quality, difficulty, noise, and cost challenges.Continuously updated web data also creates contamination risks for benchmarks that rely on previously available online content.
  • 4.2 Multimodal LiveBench: LIVEBENCH collects current news and forum information, organizes multimodal Q&A, verifies it with human annotators, and evaluates models using multiple judges.Its pipeline uses continuously updated web content rather than fixed benchmark questions and answers.
  • 4.2 Multimodal LiveBench: The dynamic evaluation pipeline balances cost and broad coverage, but its automatically produced Q&A may remain below the quality of human-curated answers.The authors explicitly identify this quality trade-off in the construction of the dynamic benchmark.
  • 4.2.2 Evaluation Metrics & Results on LIVEBENCH: GPT-4-series models were among the top performers, while Gemini and Claude series models outperformed open-source models; GPT-4o led particularly in recognition.LIVEBENCH requires zero-shot interpretation of continuously updated news and forum content, highlighting capabilities that existing benchmarks may not fully assess.

5 Conclusions

The paper presents three complementary responses to the LMM evaluation trilemma: standardized large-scale evaluation, a lower-cost lite benchmark, and dynamically updated evaluation intended to address contamination.

  • 5 Conclusions: The evaluation trilemma prevents simultaneously achieving wide coverage, low cost, and zero contamination.The authors frame their contributions as trade-offs rather than a complete solution to the trilemma.
  • 5 Conclusions: LMMS-EVAL provides a unified suite for standardized, large-scale LMM evaluation.It is presented as the paper’s broad evaluation framework.
  • 5 Conclusions: LMMS-EVAL LITE balances low-cost evaluation with wide coverage by pruning the full evaluation set.The lite toolkit is the efficiency-oriented component of the proposed LMMS-EVAL family.
  • 5 Conclusions: LIVEBENCH converts traditional static evaluation into a dynamic format to address potential data contamination.Its role is to provide a complementary evaluation setting within the proposed trade-off strategy.

6 Limitations

The paper identifies unresolved limitations in multimodal evaluation, including the trilemma among coverage, cost, and contamination, dependence on training data for decontamination, and risks from benchmark representativeness and web-crawled data.

  • 6 Limitations: The evaluation trilemma remains unresolved, so future work must seek better trade-offs or potentially overcome the three-way constraint.The paper assumes wide coverage, low cost, and zero contamination cannot currently be achieved simultaneously.
  • 6 Limitations: The decontamination method requires access to training data, whereas many research projects do not open-source their data.Future methods may need to rely solely on the model and improve efficiency.
  • 6 Limitations: Benchmarks may introduce biases when they do not represent diverse real-world applications and contexts.Models optimized for such benchmarks may perform poorly in practical settings.
  • 6 Limitations: Automatic evaluation cannot replace expert human assessment in specialized domains such as medical imaging.
  • 6 Limitations: LIVEBENCH uses real-world web-crawled data, creating potential data-privacy concerns and requiring cautious research use.

C Data Contamination

Image and text overlap analyses reveal substantial contamination risks in some multimodal benchmarks, especially those included in model training data or built from reused image sources.

  • C Data Contamination: ChartQA, DocVQA, COCO, and VQAv2 were included in LLaVA-NeXT training data and suffered the most image contamination.
  • C Data Contamination: Most benchmarks had image and text overlap below 10%.
  • C Data Contamination: POPE showed a high image-overlap ratio because it uses image sources from COCO.

C.1 More Qualitative Examples

Qualitative inspection illustrates contamination and near-duplicate risks across multimodal benchmarks, while the accompanying method compares selected Lite datasets with originals using embedding-based clustering and correlation analysis.

  • C.1 More Qualitative Examples: Identical images appear in LLaVAW, MathVista, and InfoVQA, with additional similar-image examples in NoCaps and MM-Vet.
  • C.1 More Qualitative Examples: Similar questions may be marked as in-domain even when images are not similar enough to establish contamination.The MathVista examples test similar domain knowledge and may help models answer benchmark questions.
  • C.1 More Qualitative Examples: The Lite-versus-original comparison calculates correlation scores after selecting datasets with k-center clustering.
  • C.1 More Qualitative Examples: CLIP, BGE, and a jointly trained LLaVA-Qwen 1.8B embedding model were used to represent multimodal data points.For LLaVA embeddings, token hidden states were averaged into one feature vector per data point.

D.2 Dataset statistics in LMMS-EVAL LITE

The LMMS-EVAL LITE analysis reports correlation and aggregation scores against the original datasets, with results visualized through a correlation graph and qualitative decontamination examples.

  • D.2 Dataset statistics in LMMS-EVAL LITE: The first LMMS-EVAL LITE version reports correlation and aggregation scores for the pruned benchmark set.
  • D.2 Dataset statistics in LMMS-EVAL LITE: Figure 7 presents additional qualitative results obtained using the decontamination tools.
  • D.2 Dataset statistics in LMMS-EVAL LITE: Figure 8 presents the correlation graph between Lite-set scores and original scores.

D.3 Curating more datasets in LMMS-EVAL LITE

This section extends LMMS-EVAL LITE with more datasets and describes its k-center-greedy selection and LIVEBENCH curation procedures. The evaluation design spans progressively challenging cognitive levels and uses model-based QA scoring and verification.

  • Curating more datasets: The expanded Lite version applies the same algorithm to additional datasets, including COCO and VQA, for broader diversity.It differs from the original LMMS-EVAL LITE by incorporating more datasets.
  • Selection algorithm: K-center-greedy selects points iteratively by adding the point farthest from the current centers until k centers are identified.The objective is to minimize the maximum distance from any vertex to its nearest cluster center.
  • LIVEBENCH curation: LIVEBENCH curates questions from raw website screenshots, generates scoring criteria, verifies candidate QAs, and scores answers on a 10-point scale.A checking model verifies QA accuracy, while a judge follows question-specific criteria.
  • Cognitive levels: The evaluation organizes news understanding into four progressively challenging levels: Concrete Recognition, Realworld Application, Analytical Understanding, and Divergent Thinking & Creation.These levels range from recognizing facts to analyzing relationships and generating creative solutions.
  • Cognitive levels: Models are evaluated on their ability to move from basic understanding toward higher-order reasoning and creative thinking.Representative examples illustrate the spectrum of cognitive demands in LiveBench-2024-09.

E.5 Case Analysis on LIVEBENCH

LIVEBENCH case analyses expose failures in multilingual understanding, numerical extraction and arithmetic, visual name matching, and instruction following. These examples illustrate concrete gaps between current LMM behavior and GPT-4o in some tasks.

  • Multilingual understanding: A model misreads Japanese and produces repeated nonsense sentences, revealing a failure case in multilingual news understanding.The example is reported in Table 15 for LLaVA-1.5-7B.
  • Analytical and numerical reasoning: GPT-4o correctly extracts closing prices and computes their average, whereas an open-source model fails at both numerical identification and arithmetic.The failure occurs in an analytical question involving financial information.
  • Visual recognition: A model confuses tennis players and opponents, misidentifies Qinwen Zheng, and struggles with small website text before hallucinating.The cited case attributes the errors to difficulty recognizing small text and failing to understand the image.
  • Instruction following: A model gives a detailed description but fails to summarize the main points, indicating difficulty following instructions effectively.The case also notes unnecessary information in the output.

F.1 Unified Evaluation Results with LMMS-EVAL

LMMS-EVAL combines additional results, model case analyses, task examples, dataset and model statistics, and LIVEBENCH website information to support unified multimodal evaluation. The section also reflects the framework’s broad experimental and implementation scope.

  • Evaluation results: LMMS-EVAL reports additional evaluation results alongside case analyses covering analytical, real-world, and concrete-recognition questions.The supplied materials identify examples involving LLaVA-NeXT-OV-72B-Chat, Qwen-VL-72B-Instruct, and LLaMA-3.2-Vision-11B-Instruct.
  • Evaluation pipeline: The section provides prompt tables for generating, scoring, checking, finalizing, and extracting information from website-based QA pairs.A judge prompt is also included for evaluation.
  • LIVEBENCH tasks: The evaluation includes examples of concrete recognition, real-world application, analytical questions, and creation questions in LIVEBENCH.These examples correspond to Tables 21 through 24.
  • Benchmark coverage: LMMS-EVAL documents task domains, ground-truth types, instance counts, splits, and model inference configurations through dataset and model statistics.The model statistics distinguish model families, model types, inference parameters, and parallel types.
  • LIVEBENCH sources: The benchmark includes a selected list of websites for LIVEBENCH, with sources categorized by primary content focus.The website list is presented in Table 28.
Loading 2407.12772v2…