PaperlayerView exact revision on arXiv

Source-linked AI summary

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yudong Liu, Yu-Hsuan Tsai, Fengji Zhang, Chenghua Lin, Wenhao Huang, Jie Fu

arXiv:2401.11944v4cs.CLcs.AIcs.CV

TL;DR

Evaluating advanced multimodal reasoning in Chinese remains a gap, despite the growth of large multimodal models. CMMMU provides a manually collected college-level Chinese benchmark across disciplines, and its results show a narrower open-/closed-source model gap in Chinese than in English. The benchmark is intended to help assess progress toward expert-level AI.

  • Problem

    Advanced LMM perception and reasoning in specialized domains remain difficult to evaluate in Chinese, while many multimodal benchmarks are predominantly English.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    CMMMU is a manually collected benchmark of 12k Chinese college-level multimodal questions covering six core disciplines.

    Source: §1 · p. 2

  • Results

    The open-/closed-source LMM gap is narrower in Chinese than in English: Yi-VL-34B achieves 36% accuracy, with a 7% gap to GPT-4V versus an 11% gap in English.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Comparing CMMMU with MMMU assesses Chinese–English reasoning disparities and helps delineate a trajectory toward expert-level AI.

    Source: §5 · p. 10

    CMMMU focuses primarily on college-level content and does not cover the full range of human knowledge and cognitive skills.

    Source: Limitations · p. 10

Abstract

from arXiv · show

As the capabilities of large multimodal models (LMMs) continue to advance, evaluating the performance of LMMs emerges as an increasing need. Additionally, there is an even larger gap in evaluating the advanced knowledge and reasoning abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to evaluate LMMs on tasks demanding college-level subject knowledge and deliberate reasoning in a Chinese context. CMMMU is inspired by and strictly follows the annotation and analysis pattern of MMMU. CMMMU includes 12k manually collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering, like its companion, MMMU. These questions span 30 subjects and comprise 39 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. CMMMU focuses on complex perception and reasoning with domain-specific knowledge in the Chinese context. We evaluate 11 open-source LLMs and one proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%, indicating a large space for improvement. CMMMU will boost the community to build the next-generation LMMs towards expert artificial intelligence and promote the democratization of LMMs by providing diverse language contexts.

1 Introduction

CMMMU is introduced as a Chinese college-level multimodal benchmark intended to assess specialized perception and reasoning and guide bilingual LMM development. Its results show low performance on complex Chinese tasks and a narrower open-/closed-source gap than in English.

1 Introduction

  • College-level exams are presented as a natural starting point for evaluating whether LLMs and LMMs can perform like expert adults in specialized domains.The motivation is the reported gap between advanced LMMs and multimodal expert AI in complex perception and reasoning.Source: §1 · p. 1
  • The benchmark addresses a shortage of Chinese evaluation benchmarks that leaves bilingual LMM development without a clear direction.The paper presents CMMMU as a way to guide bilingual LMMs toward expert-level artificial intelligence.Source: §1 · pp. 1–2
  • CMMMU contains 12k manually collected Chinese multimodal questions from college exams, quizzes, and textbooks across six core disciplines.Each question is annotated with detailed subfields and image types to help investigate which questions are difficult for LMMs.Source: §1 · p. 2
  • Yi-VL-34B achieves 36% accuracy, with a 7% gap to GPT-4V versus an 11% gap in English; GPT-4V and Qwen-VL-Plus achieve 43% and 36%, respectively.The authors also report that only a few open-source models perform notably better than random choice.Source: §1 · p. 2
  • The authors frame CMMMU as a contribution to LMM research and development and to the democratization of LMMs.They also describe it as the first Chinese Massive Multi-discipline Multimodal Understanding benchmark.Source: §1 · p. 2

2 Related Work

Prior multimodal benchmarks include task-oriented datasets and newer evaluations of complex reasoning, but most remain English-heavy. Meanwhile, many multimodal models support both Chinese and English, while relatively few have been evaluated on Chinese multimodal tasks.

2.1 Multimodal Benchmark

  • Earlier multimodal benchmarks were often task-oriented, using tasks such as visual question answering, image captioning, or information retrieval.Some work also evaluates cross-modal alignment using VQA derived from general knowledge bases.Source: §2.1 · pp. 2–3
  • Newer benchmarks target generative models and more challenging settings, including science, math reasoning, website interaction, and college-level knowledge reasoning.The authors identify the predominance of English in these datasets as a gap for comprehensive, challenging evaluation in languages such as Chinese.Source: §2.1 · p. 3

2.2 Bilingual Large Multimodal Models

  • Many multimodal models support English and Chinese through integrated bilingual language models, but only a few have been evaluated on Chinese multimodal tasks.Prior work reports that models trained only with English instruction tuning can work in Chinese zero-shot, while other models use Chinese-adapted language models and bilingual training or alignment approaches.Source: §2.2 · p. 3
  • The paper aims to measure models’ ability boundaries with comprehensive, challenging Chinese multimodal tasks, since most have been assessed only with English tasks.Source: §2.2 · p. 4

3 The CMMMU Benchmark

CMMMU is a manually curated Chinese multimodal benchmark for evaluating college-level knowledge and expert-level multimodal understanding. Its data curation, comparison with existing benchmarks, and dataset statistics define its scope and distinguishing features.

3 The CMMMU Benchmark

  • CMMMU is a manually curated benchmark for evaluating expert-level multimodal understanding across broad, college-level tasks.It is presented as the first Chinese-context multimodal question-answering benchmark and one of few that investigate complex understanding and reasoning.Source: §3 · p. 4

3.1 Data Curation Process

  • Three-stage curation uses licensed sources, qualified annotators, and author verification, while filtering out questions below college level or solvable by several models without OCR.Organizers collect at least 20 sources per subject; questions should require images and expert knowledge, and similar questions about a specific knowledge point are capped at 10.Source: §3.1 · pp. 4–5

3.2 Comparison with Existing Benchmarks

  • Unlike many benchmarks, CMMMU combines 39 image types, interleaved images and text, three question formats, and university-level subject reasoning.The comparison contrasts this knowledge depth with benchmarks that typically require common sense or simple physical or temporal reasoning.Source: §3.2 · p. 5, Table 2 · p. 5

3.3 Statistics of CMMMU

  • 12K questions span six disciplines, over 30 subjects, and 39 image types; the 11K-question test set accompanies 30%/58%/12% Easy/Medium/Hard difficulty shares.Difficulty is categorized by logical rather than intellectual difficulty.Source: §3.3 · p. 6

4 Experiments

The experiments evaluate LMMs on CMMMU in zero-shot settings and analyze benchmark performance and GPT-4V errors. GPT-4V scores 41.7% on CMMMU, versus 55.7% on MMMU.

4 Experiments

  • Models are evaluated in zero-shot settings to test raw multimodal answering ability, with default or validation-selected prompts; experiments run on NVIDIA A100 GPUs.The paper also reports some few-shot results in the Appendix.Source: §4 · p. 7

4.1 Baselines

  • The baselines include open- and closed-source LMMs and text-only LLMs; OCR tests whether external image-to-text conversion helps, while answers are extracted by rules and scored with micro-average accuracy.The evaluation also includes random-choice and topic-specific frequent-choice baselines.Source: §4.1 · pp. 7–8

4.2 Results of CMMMU

  • 41.7% is GPT-4V’s accuracy on CMMMU, versus 55.7% on MMMU, which the authors describe as evidence of weak cross-linguistic generalization.The paper characterizes CMMMU as more challenging than MMMU.Source: §4.2 · p. 8
  • On CMMMU, Yi-VL-34B narrows the disparity with GPT-4V to 7.5%, compared with 13.3% for Qwen-VL-Chat on CMMMU and 21.9% for BLIP2-FLAN-T5-XXL on MMMU.The authors characterize the disparity as relatively smaller in a Chinese context compared with MMMU.Source: §4.2 · p. 8
  • Open-source gaps are larger in Business, Science, and Tech & Engineering and on medium and hard questions, which the authors link to complex-condition reasoning.The decomposition by difficulty is described as further evidence of that reasoning-capacity disparity.Source: §4.2 · p. 8
  • Qwen-VL-Chat, Yi-VL-6B, and Yi-VL-34B are exceptions to other open-source models’ frequent-choice-level performance; Qwen-VL-Plus is weak on T/F, while gaps mainly concern MC questions.The authors suggest Qwen-VL-Plus may not understand the True/False prompt.Source: §4.2 · pp. 8–9

4.3 Error Analysis

  • 26% of GPT-4V’s analyzed errors were perceptual and 26% reasoning; errors also reflected insufficient domain knowledge, reported at 22%.Perceptual failures include misreading image symbols; reasoning failures also occur on correctly perceived problems requiring complex logic or mathematics.Source: §4.3 · p. 9
  • Rejections accounted for 12% of errors; text-comprehension, annotation, and answer-extraction errors accounted for 7%, 2%, and 5%, respectively.The paper associates these remaining errors with instruction or text logic, annotation inaccuracies, and answer-matching issues.Source: §4.3 · pp. 9–10

5 Conclusion

CMMMU assesses LMMs’ perceptual, reasoning, and domain-expertise capabilities, and compares bilingual-model performance across Chinese and English contexts.

5 Conclusion

  • CMMMU is designed to test elementary perception, intricate logical reasoning, and expertise in specific domains.The authors describe its design as tailored to rigorous evaluation of the latest LMMs.Source: §5 · p. 10
  • The paper compares performance on CMMMU and MMMU to reveal differences in advanced bilingual LMM reasoning capacity across Chinese and English contexts.Source: §5 · p. 10
  • The authors frame this assessment as helping delineate a trajectory toward AGI matching seasoned professionals’ proficiency across fields.Source: §5 · p. 10

Ethics Policy

The authors state that CMMMU data collection and use follow ethical and legal standards, with commitments to fairness, diversity, privacy, and intellectual-property protection.

Ethics Policy

  • CMMMU data collection and use are stated to comply with relevant ethical guidelines and legal regulations.Source: Ethics Policy · p. 10
  • The authors identify fairness, inclusivity, and diversity as aims for reducing dataset biases that might exacerbate societal disparities.Source: Ethics Policy · p. 10
  • The policy emphasizes privacy and intellectual-property protection as part of responsible data management.Source: Ethics Policy · p. 10

Limitations

CMMMU has bounded coverage: it focuses on college-level content rather than the full range of human knowledge and cognitive skills. Its evaluation metrics and human-experiment evidence also have stated limits.

Limitations

  • CMMMU covers college-level content, which represents only a portion of human expertise and does not encompass all human knowledge and cognitive skills.The authors describe the benchmark as comprehensive but not exhaustive.Source: Limitations · p. 10
  • The authors caution that their metrics might not fully capture advanced AI systems’ sophisticated understanding and reasoning abilities.Source: Limitations · p. 10
  • Manual experiments with contracted experts prevented the authors from providing a comprehensive human score, which they may add in future updates.Source: Limitations · p. 10

A Appendix

The appendix describes the benchmark’s disciplinary and subject coverage, image-type result decomposition, and question-format prompts. It also notes that the English prompt translations were not shown to models.

A Appendix

  • The appendix lists prompts for multiple-choice, true/false, and fill-in-the-blank questions, each instructing models to give their most likely answer when information is insufficient.Source: A Appendix · p. 14
  • CMMMU samples cover six disciplines and 30 subjects, whose proportions are shown in Figure A1.The figure caption says the subjects uniformly cover relevant expert-level domain knowledge.Source: Figure A1 · p. 14
  • The English prompt translations shown in the appendix were not part of the models’ input.Source: A Appendix · p. 14
  • Table A1 decomposes results across image types and marks the best overall and open-source model results.The caption distinguishes bold results for the best among all models and blue results for the best among open-source models.Source: A Appendix · p. 14, Table A1 · p. 15

B Case Study

The appendix’s GPT-4V case study analyzes both incorrect and correct answer examples.

B Case Study

  • The GPT-4V appendix analysis includes 150 incorrect-answer examples and 75 correct-answer examples.Source: B Case Study · p. 16

List of Case Study Figures

The figure list catalogs case-study examples of correct and erroneous cases across Art & Design, Business, Science, Health & Medicine, Humanities and Social Sciences, and Technology and Engineering. Some entries also show question prompts for selected subjects or subcategories.

Art & Design

Health & Medicine

Business

List of Case Study Figures

Humanities and Social Sciences

Technology and Engineering

Related papers

Loading 2401.11944v4…