MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark

1 Introduction

• MME-Industry targets the missing comprehensive assessment of MLLMs across diverse industrial sectors and specialized real-world scenarios.The introduction contrasts this gap with benchmarks focused on traffic, document understanding, and image text recognition.Source: §1 · p. 1
• The benchmark covers 21 industrial sectors and 1,050 QA processes, including power, electronics, textiles, steel, and chemicals.The introduction describes a selection process spanning over 21 sectors.Source: §1 · pp. 1–2
• Questions emphasize professional reasoning and decision-making rather than basic OCR, including tasks requiring specialized industrial knowledge.The benchmark design removes OCR-based questions and adds domain-specific questions.Source: §1 · pp. 1–2
• Industry experts validate the manually created content, and English and Chinese versions support cross-lingual research.The authors state that manual creation helps prevent leakage issues found in other datasets.Source: §1 · pp. 1–2

2.1 Multimodal Large Language Models

• MLLM development includes vision-language alignment, varied model architectures, multimodal instruction datasets, alignment techniques, and LoRA.The cited examples include CLIP, Flamingo, BLIP-2, MiniGPT-4, and LLaVA.Source: §2.1 · p. 2

2.2 MLLMs Benchmarks

• Existing benchmarks cover general multimodal understanding and specialized tasks, but comprehensive evaluation across industrial sectors and Chinese-language resources remain limited.The review names ChartQA, DocVQA, and OCRBench as domain-specific frameworks and notes only a few accessible Chinese benchmarks.Source: §2.2 · p. 2

3.1 Dataset Construction

• Domain experts curate industry-specific images and multiple-choice questions, then verify image clarity and relevance and check answers against industry standards.Expert review also removes duplicates and replaces problematic content; the content is translated into English.Source: §3.1 · p. 2

3.2 Task Categories

• The task categories span 21 industries, with questions assessing professional knowledge such as equipment, processes, materials, and safety.The listed sectors include power, electronics, textile, steel, chemical, environmental protection, machinery, finance, agriculture, medical, and others.Source: §3.2 · pp. 3–4

3.3 Evaluation Metrics

• Scores are percentage overall accuracy, computed from binary per-question scores across the benchmark’s questions.The evaluation uses the model’s complete response without additional answer extraction.Source: §3.3 · p. 4

Model Selection

• The study evaluated 10 state-of-the-art MLLMs, including GPT-4o, Claude-3.5-Sonnet, and Qwen2-VL-72B-Instruct.Taiyi and TelemM were excluded because their APIs were unavailable.Source: §4.1 · p. 4, Model Selection · p. 4, Results · p. 4

Parameter Settings

• Evaluations standardized temperature at 0.2, maximum output at 64 tokens, and top-p at 0.95; Qwen2-VL-72B-Instruct used a 256×28×28 to 1280×28×28 pixel input range.Source: Parameter Settings · p. 4

Results

• Qwen2-VL-72B-Instruct led both evaluations, scoring 78.66% in Chinese and 75.04% in English; Claude-3.5-Sonnet ranked second at 74.09% and 72.66%, respectively.The comparison reports scores with image input.Source: Results · p. 4, Table 2 · p. 5
• In Chinese, Qwen2 VL-72B reached 94% in Electronics, 92% in Light Industry, and 90% in Chemical; Education, Finance, and Building Materials averaged below 50%.Source: Results · p. 5, Table 4 · p. 6

Analysis

• Models generally performed better in technical and industrial sectors than in abstract or creative domains; Education and Finance were consistently weaker, while Environment challenged both languages.Source: Analysis · p. 6
• Qwen2-VL-72B maintained high performance across domains with little language degradation; MiniCPM-V-2.6 was weaker overall but showed potential for resource-constrained uses.Source: Analysis · p. 6
• SenseChat-Vision's Power score fell from 78% to 16% without images, illustrating the reported contribution of visual information in complex technical domains.Source: Analysis · p. 6

Results

• In English, Electronic peaked at 92% and Light Industry reached 94%, while Education and Financial remained lower-performing sectors across models.Source: Results · p. 6, Table 5 · p. 7

• MME-Industry provides 1,050 evaluation questions across 21 industries to assess MLLM capabilities and limitations in practical scenarios.Source: §5 · p. 7
• The paper reports evaluating 10 advanced MLLMs on the benchmark to characterize current capabilities in industrial applications.Source: §5 · p. 7
• Future work includes expanding dataset scale and diversity, testing more models, opening platforms to community participation, and continuously updating evaluations.Source: §5 · p. 7