Source-linked AI summary
SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, Ying Shan
TL;DR
MLLMs need reliable evaluation for text-rich visual comprehension, but existing benchmarks mainly target general visual understanding. SEED-Bench-2-Plus introduces a human-annotated benchmark spanning diverse text-rich categories and evaluates prominent MLLMs, revealing substantial variation in difficulty and model performance while providing resources for further research.
Problem
Existing MLLM benchmarks primarily focus on general visual comprehension, leaving comprehensive and objective evaluation of text-rich visual scenarios insufficient.
Method
SEED-Bench-2-Plus constructs human-annotated multiple-choice questions across Charts, Maps, and Webs and evaluates MLLMs using answer ranking based on choice likelihoods.
Results
Evaluation of prominent MLLMs reveals the complexity of text-rich visual data, varying difficulty across data types, and performance disparities among leading models.
Takeaways & Limitations
The benchmark, evaluation code, and leaderboard provide a supplement to existing MLLM benchmarks and support further research on text-rich visual comprehension.
Takeaways & Limitations
For Gemini-Pro-Vision, task performance is reported only when the model responds to over half of the valid data in the task.
Abstract
from arXiv · showhide
Comprehending text-rich visual content is paramount for the practical application of Multimodal Large Language Models (MLLMs), since text-rich scenarios are ubiquitous in the real world, which are characterized by the presence of extensive texts embedded within images. Recently, the advent of MLLMs with impressive versatility has raised the bar for what we can expect from MLLMs. However, their proficiency in text-rich scenarios has yet to be comprehensively and objectively assessed, since current MLLM benchmarks primarily focus on evaluating general visual comprehension. In this work, we introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating \textbf{text-rich visual comprehension} of MLLMs. Our benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs, each of which covers a wide spectrum of text-rich scenarios in the real world. These categories, due to their inherent complexity and diversity, effectively simulate real-world text-rich environments. We further conduct a thorough evaluation involving 34 prominent MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the current limitations of MLLMs in text-rich visual comprehension. We hope that our work can serve as a valuable addition to existing MLLM benchmarks, providing insightful observations and inspiring further research in the area of text-rich visual comprehension with MLLMs. The dataset and evaluation code can be accessed at https://github.com/AILab-CVC/SEED-Bench.
1. Introduction
SEED-Bench-2-Plus addresses the lack of comprehensive evaluation for MLLMs in text-rich visual scenarios by introducing a broad, human-annotated benchmark and evaluating leading models. The evaluation reveals complexity across data types and performance disparities among MLLMs.
- Motivation: Existing MLLM benchmarks primarily assess general visual comprehension, leaving text-rich visual contexts insufficiently evaluated.Text-rich comprehension requires interpreting text, visual content, and their interactions.
- Benchmark construction: SEED-Bench-2-Plus introduces 2.3K human-annotated multiple-choice questions covering text-rich visual comprehension across three broad categories and 63 specific data types.The benchmark is designed to capture a granular range of real-world text-rich scenarios.
- Findings: Evaluation of 34 prominent MLLMs reveals the complexity of text-rich visual data, varying difficulty across data types, and performance disparities among leading models.The evaluated models include GPT-4V, Gemini-Pro-Vision, and Claude-3-Opus.
- Research utility: The benchmark, evaluation code, and maintained leaderboard are intended to support further research on MLLM comprehension of text-rich scenarios.The work positions the benchmark as a supplement to SEED-Bench-2.
2. Related Work
Prior MLLM research has advanced multimodal capabilities and introduced broad evaluation benchmarks, but text-rich visual comprehension remains insufficiently assessed. SEED-Bench-2-Plus targets this gap with diverse text-rich data spanning Charts, Maps, and Webs.
- Multimodal large language models: MLLM research has expanded from image-text alignment and video understanding to multimodal image generation, while text-rich comprehension remains incompletely evaluated.This unresolved evaluation need motivates SEED-Bench-2-Plus.
- SEED-Bench-2-Plus: SEED-Bench-2-Plus focuses specifically on text-rich visual data across Charts, Maps, and Webs, represented through diverse benchmark samples.The benchmark’s ground-truth options are derived from human annotations.
- MLLM benchmarks: Existing MLLM benchmarks evaluate abilities such as visual understanding, object counting, identification, knowledgeability, and multiple ability dimensions.Examples discussed include GVT, LVLM-eHub, MME, MMBench, and MMMU.
3. SEED-Bench-2-Plus
SEED-Bench-2-Plus evaluates MLLMs on text-rich visual data across three broad categories and 63 data types. It uses human-annotated multiple-choice questions and likelihood-based answer ranking to reduce dependence on instruction following and option order.
- Benchmark Scope: SEED-Bench-2-Plus contains 2K human-annotated multiple-choice questions spanning text-rich visual scenarios.The benchmark is designed to provide broad evaluation coverage rather than focus on a single visual format.
- Benchmark Scope: The benchmark covers three broad categories—Charts, Maps, and Webs—further divided into 63 data types.Its tasks require models to interpret text, visual content, semantics, and spatial relationships within text-rich images.
- Data Source: The data source includes 20 chart types, 25 map types, and 18 web screenshot types collected from the Internet.Questions for charts and maps were generated with GPT-4V, while web questions were additionally refined by human annotators.
- Evaluation Strategy: Answer ranking selects the option with the highest likelihood under an MLLM instead of using a separate model to match free-form predictions to choices.This strategy is intended to reduce effects from instruction-following ability and multiple-choice option order.
4. Evaluation Results
Evaluation across text-rich categories reveals substantial variation in MLLM performance. GPT-4V performs best across most evaluation types, while maps and heterogeneous data types remain especially challenging.
- Main Results: GPT-4V surpasses many evaluated MLLMs and demonstrates superior performance across most evaluation types.The category-level results are reported in Table 2, with detailed results across 63 types in Tables 3–8.
- Main Results: The top-performing MLLM, GPT-4V, shows competitive results across the benchmark’s different categories.Figure 4 visualizes category rankings using darker hues for higher ranks.
- Observations: Average accuracy is below 40% for most MLLMs on text-rich data, indicating substantial remaining difficulty.The authors connect these results to the need for further advances before effective multimodal-agent use, particularly for website analysis.
- Observations: Maps are more difficult than charts and webs because they combine overlapping layers, symbols, colors, geographic details, and text.Their multidimensional structure makes it harder to interpret the full visual context.
- Observations: MLLM performance varies significantly across data types, including chart forms such as knowledge graphs and matrix diagrams.The authors attribute these discrepancies to differences in complexity and structure across data types.
5. Conclusion
SEED-Bench-2-Plus provides a broad evaluation of MLLMs on text-rich visual comprehension. Its benchmark and public evaluation resources are intended to support continued research in this area.
- Conclusion: SEED-Bench-2-Plus evaluates MLLMs in text-rich scenarios using a broad benchmark designed for this capability.
- Conclusion: The benchmark spans 2K human-annotated multiple-choice questions across three broad categories and 63 data types.
- Conclusion: The dataset, evaluation code, and maintained leaderboard are publicly available to facilitate progress in text-rich visual comprehension.