PaperlayerView exact revision on arXiv

Source-linked AI summary

VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models

Haodong Duan, Xinyu Fang, Junming Yang, Xiangyu Zhao, Zerun Ma, Yuxuan Qiao, Mo Li, Tianhao Liang, Lin Zhu, Amit Agarwal, Xiaozhe Li, Shengyuan Ding, Jiazi Bu, Ziyu Liu, Zhangyang Qi, Yifei Li, Yuhang Zang, Zhe Chen, Lin Chen, Yuan Liu, Yubo Ma, Hailong Sun, Yifan Zhang, Shiyin Lu, Tack Hwa Wong, Weiyun Wang, Peiheng Zhou, Chaoyou Fu, Junbo Cui, Jixuan Chen, Enxin Song, Song Mao, Junming Lin, Xilin Wei, Jinsong Li, Zeyi Sun, Zhaowei Wang, Zicheng Zhang, Xiaoyi Dong, Junjun He, Pan Zhang, Jiaqi Wang, Dahua Lin, Kai Chen

arXiv:2407.11691v5cs.CV

TL;DR

LMM evaluation requires broad, reproducible assessment across diverse models, modalities, and benchmarks. VLMEvalKit addresses this with a unified, extensible toolkit that automates core evaluation workflows and supports specialized domains. Its evaluations report distinct performance patterns across reasoning, video, and closed-source benchmark settings, while the toolkit provides a basis for the OpenVLM Leaderboard.

  • Problem

    Evaluating LMMs across many diverse benchmarks is resource-intensive because teams must prepare data, manage environments, and assemble incomplete results.

  • Method

    VLMEvalKit provides a unified open-source framework with broad model and benchmark support, extensible interfaces, automated evaluation workflows, and generation-based scoring strategies.

  • Results

    Across supplied evaluations, proprietary APIs generally lead image reasoning, open-source models perform strongly in video understanding, and commercial APIs lead the closed-source evaluation.

  • Takeaways & Limitations

    VLMEvalKit offers a maintained foundation for reproducible multimodal evaluation and uses the resulting evaluations to support the OpenVLM Leaderboard.

Abstract

from arXiv · show

We present VLMEvalKit: an open-source toolkit for evaluating large multi-modality models based on PyTorch. The toolkit aims to provide a user-friendly and comprehensive framework for researchers and developers to evaluate existing multi-modality models and publish \textbf{reproducible} evaluation results. In VLMEvalKit, we implement over 450+ large multi-modality model configurations, including both proprietary APIs and open-source models, and support 330+ benchmarks across diverse multi-modal benchmarks. By implementing a single interface, new models can be easily added to the toolkit, while the toolkit automatically handles the remaining workloads, including data preparation, distributed inference, prediction post-processing, and metric calculation. VLMEvalKit has also evolved to a broader evaluation suite spanning video/audio, document understanding, GUI grounding, spatial reasoning, safety, scientific reasoning, and multi-turn dialogue. Based on the evaluation results obtained with the toolkit, we host the OpenVLM Leaderboard, a comprehensive leaderboard to track the progress of multi-modality learning research. The toolkit is released on https://github.com/open-compass/VLMEvalKit and is actively maintained.

1. Introduction

VLMEvalKit addresses the practical difficulty of evaluating large multi-modality models across many benchmarks by providing a comprehensive, user-friendly toolkit. It supports broad model and benchmark coverage, simplifies extensions and operation, and produces reproducible evaluations for leaderboard tracking.

  • Motivation: Evaluating LMMs across dozens of benchmarks requires extensive data preparation, environment management, and completion of missing results.These burdens can be especially difficult for small research teams.
  • Scope: VLMEvalKit supports more than 450 LMMs and over 335 multimodal benchmarks spanning image-text, video, document, GUI, spatial, safety, medical, scientific, generation, and dialogue tasks.The supported models include proprietary APIs and open-source models.
  • Usability: A straightforward design lets developers add new benchmarks or LMMs with typically one data file or one interface implementation.Users can launch evaluations across multiple models and benchmarks with a single command.
  • Usability: VLMEvalKit Lite adds local interfaces for dataset inspection, prompt debugging, prediction and judge-file review, summarized results, and leaderboard loading.These tools support debugging benchmarks, verifying prompts, and communicating evaluation results beyond batch execution.
  • Evaluation: Generation-based evaluation uses LLM choice extractors when exact matching fails, reducing the effect of response styles on multiple-choice assessment.The resulting evaluations support a comprehensive leaderboard that monitors LMM development.
  • Availability: The toolkit is publicly released under the Apache 2.0 License with source code and instructions for installation, evaluation, and further development.The repository is available at the project’s GitHub URL.

2. VLMEvalKit: Design & Features

VLMEvalKit organizes multimodal evaluation through standardized benchmark data, extensible message and model interfaces, parallel inference, reuse mechanisms, and specialized scoring strategies. Its design spans broad task domains while adding tools for inspection, debugging, and result analysis.

  • Benchmarks: Over 335 benchmarks are processed into standardized .tsv files containing multimodal evaluation samples.Samples typically include indices, questions, answers, images, and choices, while dataset classes expose a build_prompt() interface.
  • Message Interface: The multimodal message format interleaves image, video, and text content and can be extended to audio or point clouds.This provides a common representation for constructing evaluation inputs across modalities.
  • Evaluation Scope: The toolkit covers specialized evaluation domains including document, OCR, chart, safety, bias, medical, and scientific reasoning.Examples include SciDocBench, ChartX, MedQ-Bench, and SGI-Bench.
  • Evaluation Scope: Spatial and GUI grounding plus video understanding pipelines integrate benchmarks such as OSWorld-G, DA-2K, Video-MME-v2, and VideoMMMU.These benchmarks target complex agentic environments and long-context video tasks.
  • LMMs: A unified .generate() interface accepts multimodal messages and returns response strings for commercial and open-source LMMs.The interface also accepts the dataset name, while single-image models default to concatenated text and the first image.
  • Inference: Parallelized inference supports commercial APIs through multiprocessing and open-source models across multiple GPUs, with vLLM and LMDeploy acceleration for selected model series.Structured status tracking and format-aware reuse allow interrupted evaluation runs to resume using existing artifacts.
  • Evaluation Strategies: CircularEval reduces multiple-choice variance by evaluating all circular option shifts and counting a result only when every shifted question is answered correctly.The strategy is intended to make comprehension differences between models more discernible.
  • Evaluation Strategies: Open-ended benchmarks retain task-specific scoring, including GPT-4 semantic marking for subjective tasks and heuristic matching for VQA accuracy.These approaches follow the original or standard evaluation practices described for the benchmark types.

3. Evaluation Results

VLMEvalKit evaluates LMMs across broad multimodal, reasoning, video, and closed-source benchmark suites. Results show persistent reasoning gaps between commercial and open-source models, strong video performance from leading open-source models, and spatial understanding as a major bottleneck.

  • Reasoning Benchmarks: 64.8 average score places RBDash-v2.0-Thinking below Seed1.5-VL (73.3) and GPT-5 (72.9) on multimodal reasoning benchmarks.Most high-performing models exceed 80% on MathVista, but only a few exceed 50% on DynaMath.
  • Reasoning Benchmarks: Commercial APIs generally outperform open-source LMMs on multimodal reasoning, with the top four performing models all proprietary APIs.The comparison is based on the OpenVLM Reasoning Leaderboard results described in the passage.
  • Video Benchmarks: 72.7 average score makes InternVL3-78B the top reported model on video-understanding benchmarks.Qwen2.5 follows closely, while GPT-4o reaches 63.1 using only 16 frames.
  • Closed-Source Evaluation: Closed-source evaluation covers General Perception, Spatial Understanding, Infographic Understanding, MultiModal Reasoning, and MultiModal Creation.The evaluation is presented for the top-10 LMMs, ordered by average score.
  • Video Benchmarks: The video table ranks the top-10 LMMs by average score, but includes only a few commercial APIs because of resource limitations.The table covers video-understanding benchmarks through 2025.06.25.
  • Closed-Source Evaluation: 42.0 is the highest score reported for Spatial Understanding, making it a universal bottleneck across evaluated models.Doubao-Seed-2.0-Pro and Kimi-k2.5 achieve this highest score.

4. Discussion

VLMEvalKit is released as an extensible open-source evaluation toolkit with broad model and benchmark coverage. Its design supports additional modalities and future expansion, while released resources aim to advance multimodal-learning research.

  • Discussion: VLMEvalKit was released as an open-source toolkit for evaluating large multi-modality models.The discussion presents the release as the paper’s central contribution.
  • Discussion: VLMEvalKit includes over 300 LMMs and more than 100 multimodal benchmarks.The codebase emphasizes simple integration of new LMMs and benchmarks.
  • Discussion: The framework is designed to extend beyond vision by incorporating modalities such as audio.Future development focuses on expanding video and other modality support.

A. Benchmarks

This section lists supported benchmarks and organizes them into multiple categories, each with a brief introduction.

  • A. Benchmarks: Supported benchmarks are listed with brief introductions and grouped into multiple categories.The passage describes the organization of the benchmark section without naming its categories.

A.1. General Capability

The General Capability benchmarks include MMBench and SEEDBench, which evaluate broad visual-language abilities through diverse multiple-choice tasks.

  • A.1. General Capability: The MMBench Series evaluates LVLMs across 20 distinct capabilities, including MMBench-CN’s Chinese adaptation.MMBench-CN adapts MMBench questions and choices into Chinese based on GPT-4.
  • A.1. General Capability: SEEDBench contains 19,000 multiple-choice questions spanning 12 dimensions.The passage identifies SEEDBench as part of the general-capability benchmark set.

A.2. Text Recognition & Understanding

This section surveys benchmarks for text recognition and understanding in images, ranging from text-centric visual question answering to broader OCR capabilities and document analysis.

  • TextVQA evaluates visual reasoning over textual content embedded in images.Models must read image text, integrate it with visual information, and answer questions.
  • OCRVQA assesses visual question answering by reading text in images.
  • OCRBench evaluates OCR capabilities across text recognition, scene-text VQA, document VQA, key information extraction, and handwritten mathematical expression recognition.
  • DocVQA promotes purpose-driven document analysis by using extracted document content to answer human-defined tasks.

A.3. Structuralized Content Understanding

These benchmarks evaluate structured understanding of charts, multimodal tasks, visual mathematical reasoning, aesthetic perception, and low-level vision or generated-image assessment.

  • ChartQA tests chart question answering with complex visual and logical reasoning over 9,600 human-written and 23,100 machine-generated questions.The tasks include logical and arithmetic operations alongside visual chart analysis.
  • ChartQAPro extends ChartQA with broader chart-family coverage and a stricter evaluation protocol, while ChartX spans chart types and reasoning tasks.
  • MM-Vet evaluates complex multimodal tasks using a systematic structure, unified metrics, and LLM-based evaluators across question-and-answer types.
  • MathVista contains 6,141 examples from 28 existing multimodal datasets and three new datasets for visual mathematical reasoning.MathVision addresses limitations in existing visual-mathematics benchmarks with 3,040 high-quality mathematical examples.
  • AesBench measures aesthetic perception using an expert-labeled database, while Q-Bench evaluates low-level vision through perception, description, and assessment.
  • A-Bench assesses AI-generated images using 2,864 images produced by 16 text-to-image models.

A.7. Long-Context, Multi-Image & Multi-Turn

The benchmarks in this section target visually rich and long-context documents, hallucination and consistency, multilingual understanding, safety, medical and real-world reasoning, and contextual text restoration.

  • DUDE contains 6,315 visual questions averaging 5.7 images each, while MMLongBench-Doc uses 1,091 questions from 135 lengthy PDF documents.The PDFs average 47.5 pages and 21,214 tokens, emphasizing long-context document understanding.
  • HallusionBench diagnoses visual-context reasoning, visual illusions, language hallucinations, response consistency, and logical reasoning across 346 images and 1,129 questions.
  • POPE evaluates object hallucination through random, popular, and adversarial tracks totaling approximately 9k cases.
  • MMMB and Multilingual MMBench cover six languages, while MMMB spans 15 categories and 12k questions; MT-VQA covers nine languages with human annotations.Aya Vision Bench evaluates crosslingual visual question answering across diverse languages.
  • MLLMGuard provides bilingual image-text safety data, inference utilities, and lightweight evaluators, alongside medical and autonomous-driving benchmarks.
  • VCR evaluates restoration of partially obscured image text by integrating visual, textual, and contextual cues rather than relying only on OCR or masked language modeling.

A.11. Video Understanding

This section covers video understanding, spatial reasoning, GUI grounding, safety, medical image quality, physics, and scientific reasoning benchmarks.

  • MMBench-Video contains about 600 long-form YouTube videos spanning 16 domains and roughly 2,000 questions probing 26 fine-grained skills.It uses GPT-4 adjudication for scoring.
  • VSI-Bench probes visual-spatial intelligence in video through object counting, relative orientation, and ego-motion reasoning.
  • TopViewRS, ERQA, and RefSpatialBench evaluate top-view spatial reasoning, embodied grounded reasoning, and 3D-aware spatial reference understanding.
  • OSWorld-G focuses on element-level localization in real desktop screenshots, while VenusBench-GD covers web, mobile, and desktop GUI grounding.
  • VLMsAreBiased, MMSafetyBench, and Flames assess visual-task bias, multimodal safety scenarios, and Chinese value alignment respectively.
  • MedQ-Bench, HiPhO, and MMSci target medical image quality, multi-step high-school physics reasoning, and comprehension of PhD-level scientific literature.MMSci spans 72 scientific disciplines, primarily in the natural sciences.

B.1. Contributors with 3+ Major Contributions

VLMEvalKit identifies contributors with at least three significant development contributions. The listed contributors supported multiple models, benchmarks, evaluation tasks, and infrastructure improvements.

  • PhoenixZ810 supported WeMath, LogicVista, MM-AlignBench, Video-ChatGPT, Chat-UniVI, and Llama-VID.
  • Amit Agarwal supported MUIRBench, Phi-3.5, Idefics3, VILA, xGen-MM, and MVTamper-Bench.
  • Other qualified contributors supported benchmarks and models including MMVP, CVBench, CharXiv, mPLUG-Owl3, MM-IFEval, ChartMimic, mega-bench, Kimi-VL-A3B, Q-Bench, and A-Bench.Contributors also fixed Creation-MMBench evaluation problems and added vLLM support for QwenVL, Llama4, and InternVL series models.

B.2. Full Contributor List

The full contributor list records additional community members and their specific contributions to benchmarks, models, datasets, evaluation support, and documentation.

  • Contributors supported OCRBench, VCR, GMAI-MMBench, MGM-7B, Monkey, TransCoreM, AesBench, OmChat, VLM-R1, and LLaVA-XTuner.
  • Other contributions included support for MMAlaya, CogVLM, Q-Bench, A-Bench, Qwen2-VL, BLINK, TaskMeAnything-V1-ImageQA-Random, Ola, URSA-8B, and URSA-8B-PS-GR.
  • Infrastructure and maintenance work included generic video-dataset preparation, PNG saving fixes, 3DSR evaluation fixes, additional dataset cases, multimodal medical evaluation extensions, model support, and README typo corrections.
  • The list also covers SAIL-VL, Janus-Pro-1B, UCSC-VLAA-Thinker, Ristretto, VLM2Bench, VGRP-Bench, Physics, VMCBench, Taichu-VLR, and LEGO-Puzzle.
Loading 2407.11691v5…