PaperlayerView exact revision on arXiv

Source-linked AI summary

MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin, Tieniu Tan

arXiv:2408.13257v3cs.CV

TL;DR

Existing MLLM benchmarks provide limited scale, model-generated annotations, or insufficient difficulty for assessing challenging real-world scenarios. MME-RealWorld builds a large, high-resolution, human-annotated benchmark and evaluates prominent models, finding that none surpasses 60% accuracy. The results leave high-resolution perception and complex real-world understanding as unresolved challenges.

  • Problem

    Existing benchmarks face limited data scale, model-based annotation quality, and insufficient task difficulty for evaluating real-world MLLM challenges.

  • Method

    MME-RealWorld collects high-resolution images and uses professional annotators and MLLM experts to produce human-checked QA pairs across real-world subtasks.

  • Results

    None of the advanced MLLMs evaluated on MME-RealWorld surpasses 60% accuracy.

  • Takeaways & Limitations

    High-resolution image perception and understanding complex real-world scenarios remain challenging for current MLLMs.

  • Takeaways & Limitations

    The evaluated MLLMs remain far from possessing world-model capability, despite receiving single-frame images for tasks such as predicting vehicle steering.

Abstract

from arXiv · show

Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has recently garnered widespread attention in the research community. However, we observe that existing benchmarks present several common barriers that make it difficult to measure the significant challenges that models face in the real world, including: 1) small data scale leads to a large performance variance; 2) reliance on model-based annotations results in restricted data quality; 3) insufficient task difficulty, especially caused by the limited image resolution. To tackle these issues, we introduce MME-RealWorld. Specifically, we collect more than $300$K images from public datasets and the Internet, filtering $13,366$ high-quality images for annotation. This involves the efforts of professional $25$ annotators and $7$ experts in MLLMs, contributing to $29,429$ question-answer pairs that cover $43$ subtasks across $5$ real-world scenarios, extremely challenging even for humans. As far as we know, MME-RealWorld is the largest manually annotated benchmark to date, featuring the highest resolution and a targeted focus on real-world applications. We further conduct a thorough evaluation involving $28$ prominent MLLMs, such as GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the most advanced models struggle with our benchmarks, where none of them reach $60\%$ accuracy. The challenges of perceiving high-resolution images and understanding complex real-world scenarios remain urgent issues to be addressed. The data and evaluation code are released at https://mme-realworld.github.io/ .

1 Introduction

MME-RealWorld addresses limitations in existing MLLM benchmarks by combining large-scale, fully human-annotated data with high-resolution images and difficult real-world tasks. Its evaluation shows that advanced models still struggle substantially on these scenarios.

  • Fewer than 10K QA pairs in many existing benchmarks can produce large evaluation fluctuations.
  • Model-generated annotations can introduce noise because the best-performing model, InternVL-2, achieves only 50% accuracy.
  • The benchmark covers five real-world domains and includes tasks designed to remain difficult even for humans.
  • None of the evaluated advanced MLLMs surpasses 60% accuracy, while tasks include counting 133 vehicles and identifying small objects in images exceeding 5000×5000 resolution.
  • MME-RealWorld-CN adds Chinese-focused data to address question-image and translation mismatches, producing 5,917 QA pairs.

2 MME-RealWorld

MME-RealWorld is constructed from challenging, high-resolution real-world imagery across multiple domains, with manually designed questions and quality controls. Its Chinese version supplements translated data with Chinese-context images and expert-reviewed annotations.

  • Instruction and Criterion: Each question has one correct option, three difficult distractors, and an additional rejection option when no answer is present.
  • The benchmark organizes high-resolution imagery into OCR, remote sensing, diagrams and tables, autonomous driving, and monitoring domains.
  • Remote Sensing: Remote sensing contributes 1,298 images and 3,738 QA pairs covering object counting, color recognition, and spatial relationships.
  • MME-RealWorld-CN: MME-RealWorld-CN combines carefully selected or professionally translated data with 300 Chinese tables, 301 Chinese diagrams, and 939 additional OCR images.
  • MME-RealWorld-CN: MME-RealWorld-CN contains 1,889 additional images and 5,917 QA pairs while retaining similar task types, image quality, and task difficulty.
  • Quality Control and Analysis: MME-RealWorld has an average resolution of 3,007,695 pixels, substantially above MME’s 975,240-pixel average.

3 Experiments

Experiments evaluate 28 MLLMs across perception and reasoning tasks, showing that high-resolution real-world evaluation exposes substantial weaknesses in visual detail perception, reasoning, and dynamic-scene understanding.

  • Experimental Setup: 28 MLLMs are evaluated across five domains, with perception results summarized using weighted and unweighted average accuracy across subtasks.The domains are OCR, remote sensing, diagram and table, monitoring, and autonomous driving.
  • Perception: InternVL-2 demonstrates the strongest overall perception ability, while performance varies substantially across tasks and domains.Its advantage is reported relative to other evaluated models, including closed-source systems.
  • Perception: 77% OCR accuracy places GPT-4o second only to Qwen2-VL, but its performance drops on more challenging real-world tasks.Gemini-1.5-Pro and GPT-4o-mini show a similar pattern of stronger OCR performance but substantially weaker results elsewhere.
  • Perception: Higher-resolution models such as Mini-Gemini-HD and SliME consistently outperform same-size models using conventional vision encoders across subtasks.The comparison highlights high-resolution image processing as important for complex real-world tasks.
  • Reasoning: Claude 3.5 Sonnet leads most reasoning domains, including a 12.6% advantage over Qwen2-VL on chart-related tasks.Even the top-ranked model remains below 45% average accuracy, while class-based accuracy does not exceed 50%.
  • MME-RealWorld-CN: Qwen2-VL and InternVL-2 outperform existing models on both Chinese perception and reasoning tasks, while GPT-4o and GPT-4o-mini drop nearly 10% in Chinese scenarios.The reported Chinese-versus-English differences are especially pronounced in reasoning tasks.
  • Fine-grained Analysis and Findings: Models frequently overproduce the “E” answer despite its annotation frequency staying below 5%, indicating difficulty identifying queried objects.Larger models tend to choose “E” conservatively, whereas smaller models often favor option “A”; InternVL-2 has a notably uniform error distribution.
  • Fine-grained Analysis and Findings: MLLMs show significant deficiencies in understanding, predicting, and reasoning about dynamic object information, including a car’s steering.The observed gap from humans persists even though the models receive single-frame images rather than video.

4 Conclusion

MME-RealWorld addresses limitations in MLLM evaluation with a large, high-resolution, human-annotated benchmark focused on complex real-world scenarios. Its evaluation reveals substantial gaps in current models’ complex image perception and understanding.

  • MME-RealWorld addresses limitations in existing MLLM evaluations, including data scale, annotation quality, and task difficulty.
  • The benchmark is described as the largest purely human-annotated dataset with the highest resolution to date, supported by 32 annotators.
  • Its QA pairs focus mainly on real-world scenarios such as autonomous driving and video surveillance, which have significant applicability.
  • The evaluation finds significant performance gaps that highlight current models’ shortcomings in complex image perception.
  • Figure 3 compares how often models output answer “E” across domains and distinguishes perception from reasoning tasks.
  • Figure 4 shows distinct error-choice behaviors: larger models favor “E”, smaller models favor “A”, while InternVL-2 has a uniform error distribution.

A Data Collection and Task Split

The benchmark uses high-resolution images from complex real-world scenes and manually constructed annotations with expert review. Its collection emphasizes difficult visual content requiring careful observation.

  • The data consists of real-world street scenes and high-resolution product advertisements with dense or difficult-to-detect text.
  • Images are manually selected from high-resolution datasets, including DIV2K and Flickr2K, while preserving complex scenes and contexts.
  • 20 volunteers annotate question-answer pairs, and 3 experts check and correct the annotations.

A.1.2 Evaluation Dimensions and Benchmark Statistics

The benchmark evaluates OCR, scene understanding, character understanding, and diagram/table interpretation through perception and reasoning tasks. These tasks use complex visual content designed to challenge rapid human interpretation.

  • OCR in the Wild: Complex-context OCR includes five perception tasks and two reasoning tasks for recognizing and understanding textual information in visual content.
  • OCR in the Wild: The five OCR perception categories cover contact information, products and advertisements, identity information, small signal text, and books, maps, and posters.
  • OCR in the Wild: OCR reasoning includes scene recognition, such as interpreting scoreboards or clocks, and character understanding involving relationships, emotions, intentions, or quantities.
  • OCR in the Wild: Some OCR tasks use overlapping image sets, so the total images counted across tasks differs from the 3,293 unique images.
  • Diagram and Table: Diagram and table data are challenging because rich content is difficult to localize and analyze quickly, even for human researchers.
  • Diagram and Table: Existing diagram and table datasets often have low resolution and limited content richness, whereas this benchmark sources more complex internet diagrams such as financial reports.
  • Diagram and Table: 20 volunteers generate perception annotations, one expert researcher generates reasoning annotations, and 3 experts review and correct them.

A.2.2 Evaluation Dimensions and Benchmark Statistics

The Diagram and Table domain contains complex, high-resolution visual data that challenges human interpretation. Its tasks separate perception of localized values from reasoning over comparisons, conditions, extrema, colors, and graphical magnitudes.

  • 2,570 images and 5,933 annotations form a Diagram and Table domain designed to be difficult even for humans.
  • Perception Tasks: Table Perception identifies values at specified table positions using the table name, horizontal coordinates, and location information.
  • Perception Tasks: Diagram Perception identifies element values or intervals using a diagram’s legend or title and specified location information.
  • Reasoning Tasks: Table Reasoning compares elements, filters rows or entries under conditions, and determines maximum or minimum values.
  • Reasoning Tasks: Diagram Reasoning distinguishes legend colors and assesses the heights of curves or bars.
  • Some remote-sensing images reach 139MB and contain rich visual detail.

A.3.1 Data Sources and Annotation Process

The benchmark draws on high-resolution remote-sensing and driving data, then uses manual question generation and expert review to construct challenging real-world tasks.

  • Remote Sensing: Remote-sensing data include fine-grained object annotations from high-resolution satellite imagery, including 37 sub-categories across five main categories in FAIR1M.FAIR1M provides 15,000 images with rotated bounding boxes and 0.3–0.8m RGB imagery.
  • Annotation: Twenty volunteers manually create remote-sensing questions and answers, followed by expert review for quality standards.The annotation procedure applies to all questions in the subsection.
  • Remote Sensing: 1,298 high-quality remote-sensing images support object counting, color recognition, and spatial-relationship tasks.These tasks contain 1,255, 1,226, and 1,257 QA pairs, respectively.
  • Autonomous Driving: Driving data span varied weather, geographic locations, viewpoints, and traffic conditions captured by onboard cameras.Sources include datasets with dense semantic, spatial, and relational annotations and wide-field-of-view stitched imagery.
  • Annotation: Professional researchers manually generate driving questions and answers from source labels, while experts review them for ambiguity, challenge, and complexity.The process is intended to ensure non-ambiguity and appropriate task difficulty.

A.4.2 Evaluation Dimensions and Benchmark Statistics

The autonomous-driving split evaluates perception and reasoning across diverse traffic scenes, with tasks covering objects, attributes, counts, intentions, interactions, and driver attention.

  • Task Dimensions: Six main autonomous-driving tasks cover object identification, attribute identification, object counting, intention prediction, interaction-relation understanding, and driver-attention understanding.The split combines three perception tasks with three reasoning tasks.
  • Perception Tasks: 1,101 images and 1,101 QA pairs support object identification of traffic elements and their quantities in front of the ego car.The task describes categories and corresponding quantities.
  • Perception Tasks: 720 QA pairs from 647 images evaluate counting of traffic elements, including novel objects such as garbage bags, dogs, and concrete blocks.The task includes cars, trucks, and traffic signals alongside less conventional driving-scene objects.
  • Reasoning Tasks: 614 QA pairs from 582 images evaluate fine-grained intention prediction for the ego vehicle, pedestrians, and vehicles.The subtasks target current or future behavior of designated traffic agents.
  • Related Data: High-resolution monitoring datasets such as VisDrone introduce viewpoint, scale, and out-of-view challenges across drone imagery.VisDrone contains 179,264 video frames and 10,209 static images.

A.5.2 Evaluation Dimensions and Benchmark Statistics

The monitoring split combines perception and reasoning tasks over varied surveillance imagery, emphasizing counting, localization, attributes, object totals, intentions, and inferred functions.

  • Perception Tasks: 1,600 monitoring images and QA pairs evaluate object counting, while zero-count cases become object-existence questions.Object existence is treated as a special case of counting.
  • Perception Tasks: 136 images and QA pairs evaluate the locations of specific vehicles, while 460 images and QA pairs evaluate vehicle and person attributes.Attribute recognition includes color and orientation perception.
  • Reasoning Tasks: Three monitoring reasoning tasks calculate totals across object types, infer an object’s next route and turn, and infer materials or functions from appearance.The tasks contain 300, 98, and 100 images and QA pairs, respectively.

B Experimental Results on All Task Splits

Across OCR, diagrams and tables, remote sensing, autonomous driving, and monitoring, results show substantial variation but persistent difficulty on high-resolution perception and real-world reasoning tasks.

  • OCR in the Wild: GPT-4o ranks first overall in OCR in the Wild, while Claude 3.5 Sonnet leads reasoning tasks.Qwen-VL and basic LLaVA achieve only slightly better than random-guessing accuracy on perception tasks.
  • Diagram and Table: Only Claude 3.5 Sonnet exceeds 60% accuracy on diagram and table reasoning, while InternLM-XComposer2.5 trails by 20%.Three models exceed 60% average accuracy overall in the diagram domain, but reasoning remains harder.
  • Remote Sensing: 30% accuracy is achieved by SliME-8B on remote-sensing counting with extremely large images, while GPT-4o-mini achieves 2%.Models performing better generally support high-resolution processing or high-resolution vision encoders.
  • Autonomous Driving: Around 30% accuracy is reached by even the most advanced models on autonomous-driving reasoning tasks, and only Claude 3.5 Sonnet exceeds 40% on average perception.Intent prediction, traffic-light recognition, and object counting remain difficult.
  • Implications: High-risk autonomous driving requires very high accuracy for practical deployment, motivating stronger 3D spatial understanding or domain-specific driving fine-tuning.The paper identifies these capabilities as necessary before effective application in this field.
  • Monitoring: 53.19 average perception accuracy is achieved by InternVL-2 on monitoring tasks versus 33.93 for GPT-4o.Qwen-VL and LLaVA perform around 20%, while no model exceeds 40% on monitoring reasoning tasks.
Loading 2408.13257v3…