Source-linked AI summary
CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning
Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, Hua Huang
TL;DR
CMMU addresses limited evaluation of domain-specific knowledge and question formats with a Chinese multimodal benchmark and a measure of multiple-choice position bias. Most evaluated models score lower on fill-in-the-blank and multiple-response questions than on multiple-choice questions.
Problem
Method
CMMU is a Chinese multimodal benchmark with multiple-choice, multiple-response, and fill-in-the-blank questions, paired with Positional Error Variance to quantify multiple-choice position bias.
Source: §1 · pp. 1–2Results
Most models score lower on fill-in-the-blank and multiple-response questions than on multiple-choice questions; GPT-4V leads with 30.19% validation accuracy.
Source: Results on Different Subjects and Grades · p. 6Takeaways & Limitations
The evaluation results contribute to a deeper understanding of current MLLMs across diverse and complex question formats.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose an evaluation strategy called Positional Error Variance for assessing multiple-choice questions. The strategy aims to perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs. The data and code are available at https://github.com/FlagOpen/CMMU.
1 Introduction
CMMU addresses gaps in domain-specific multimodal evaluation with a Chinese benchmark spanning subjects and question formats. It also introduces Positional Error Variance to quantify position bias in multiple-choice questions.
1 Introduction
- Existing multimodal benchmarks often emphasize common-scene knowledge, while domain-specific educational benchmarks are predominantly English-language and multiple-choice.The introduction notes that multiple-choice formats cannot evaluate text generation and may allow correct answers by guessing.Source: §1 · p. 1
- CMMU is a Chinese multimodal benchmark spanning seven subjects, with questions requiring models to combine image and text into comprehensive responses.Its coverage extends from primary school to high school.Source: §1 · p. 1
- CMMU includes multiple-choice, multiple-response, and fill-in-the-blank questions, and its evaluation finds the benchmark challenging for current MLLMs.The introduction describes the broader range of question types as a greater challenge to MLLM comprehension.Source: §1 · p. 2
- Positional Error Variance quantitatively analyzes position bias in MLLMs on multiple-choice questions.The approach cycles option positions and measures position bias.Source: §1 · p. 2
2 Related Work
Related benchmarks span visual question answering, captioning, grounding, and domain-specific evaluation, but differ in their knowledge demands and question formats. MLLM work connects image and text through varied architectural designs.
2.1 Multi-modal Benchmarks
- Visual question answering, image captioning, and visual grounding use benchmarks including VQAv2, COCO, and Visual7w, respectively.Source: §2.1 · p. 2
- Domain-specific benchmarks include ScienceQA for science curricula, MMMU for college-level knowledge, and multilingual M3Exam, while M3Exam includes images in fewer than one-third of its questions.Source: §2.1 · p. 3
2.2 Multi-modal Large Language Models
- MLLMs bridge vision and language using approaches such as BLIP-2’s Q-Former, CogVLM’s visual expert module, and projection schemes used by LLaVA, Emu2, and MiniGPT-4.Source: §2.2 · p. 3
3 The CMMU Benchmark
CMMU is a Chinese multimodal benchmark spanning seven subjects and school levels, with three question formats rather than multiple-choice alone. The section describes its preprocessing and distribution and compares it with existing datasets across five dimensions.
3 The CMMU Benchmark
- CMMU combines text-and-image questions across seven subjects, from primary through high school, in three formats: MCQ, MRQ, and FBQ.MCQs have one correct answer, MRQs can have one to four correct answers, and FBQs ask students to complete a sentence or passage.Source: §3 · p. 4
3.1 Data Pre-process
- 639 fill-in-the-blank questions were decomposed into 1,632 subquestions, 83% requiring one blank; teachers categorized questions as normal or hard.The team extracted text and images from PDFs, converted formulas to LaTeX, and manually reviewed image quality, parsing, and conversion errors.Source: §3.1 · pp. 4–5
3.2 Data Distribution
- 3,603 questions are split between validation (1,800) and test (1,803), with an 8:2 ratio of normal to hard questions.Over 70% include explanations averaging 311 characters; primary-school content is math only, while middle and high school cover seven subject types.Source: §3.2 · p. 5
3.3 Comparisons with Existing Datasets
- Across five dimensions—quantity, images, explanations, question type, and language—CMMU is described as the first Chinese benchmark for multimodal, multi-type question understanding and reasoning.Source: §3.3 · p. 5
4 Evaluation
The evaluation addresses uncertainty about whether a correct multiple-choice answer reflects knowledge or guessing, alongside the lack of quantitative measurement of position bias. It combines circular option shifts with a variance-based bias metric and applies question-type-specific scoring.
4 Evaluation
- Correct multiple-choice answers may reflect guessing rather than knowledge, and prior methods had not quantitatively measured position bias.The paper identifies preference for a particular answer position as a sign of guessing-related bias.Source: §4 · p. 5
4.1 CircularEval
- CircularEval cyclically shifts the options and scores a question as correct only when every shifted version is answered correctly.For k options, the method creates k shifted-option questions and evaluates each model response.Source: §4.1 · p. 5
4.2 Positional Error Variance
- BiasRate is the variance of option-selection probabilities among incorrectly answered questions; higher values indicate greater positional bias.The method excludes questions answered correctly across all shifts, then calculates each option's frequency across the remaining answer combinations.Source: §4.2 · p. 5
4.3 Evaluations on Different Question Types
- Multiple-response answers count as correct only when every selected option is correct, with no incorrect choices.Source: §4.3 · p. 6
- Fill-in-the-blank responses receive a binary correctness score from GPT-4, which can accept answers with similar meanings to the ground truth.Source: §4.3 · p. 6
5 Experiments
Across the evaluated models, GPT-4V achieved the highest reported overall accuracy, while multiple-response and fill-in-the-blank questions were especially difficult. Results also show differences by subject, grade, and option preference, and a GPT-4V case study identifies recurring error types.
5.1 Models
- The evaluation covered three closed-source models and seven open-source models ranging from 7b to 13b parameters.The closed-source group included GPT-4V, Gemini-Pro, and Qwen-VL-Plus; open-source models were run on NVIDIA A100 GPUs.Source: §5.1 · p. 6
5.2 Prompts and Settings
- All models were evaluated zero-shot with question-type-specific prompts and different generation settings for MCQ/MRQ versus FBQ.MCQ and MRQ used temperature 0 and a 10-token limit; FBQ used temperature 0.2 and a 128-token limit.Source: §5.2 · p. 6
Results on Different Subjects and Grades
- 30.91% test accuracy for GPT-4V was the highest among evaluated models; all three closed-source models outperformed all open-source models.GPT-4V also achieved 30.19% accuracy on the validation set; most models had lower accuracy on MRQ and FBQ than on MCQ.Source: Results on Different Subjects and Grades · p. 6
Results on Different Question Types
- Models generally performed better on middle-school than high-school questions, and politics and history accuracy exceeded physics and math accuracy.The paper attributes the subject differences to the knowledge reservoir used in politics and history versus the computation and reasoning required in physics and math.Source: Results on Different Question Types · p. 7
Position Bias Analysis
- Most models preferred one or two options, none preferred Option D most, and stronger models such as GPT-4V tended to have lower BiasRate.Source: Figure 7 · p. 6, Position Bias Analysis · p. 7
Case Study with CoT Prompts
- 27.48% of identified GPT-4V mistakes involved image misunderstanding, 35.41% misleading reasoning, and 13.03% question misunderstanding.The analysis used 500 randomly selected multiple-choice questions with a chain-of-thought prompt; one case guessed correctly but failed CircularEval.Source: Case Study with CoT Prompts · p. 7, Figure 8 · p. 7
6 Conclusion and Future Work
The conclusion presents CMMU as a Chinese benchmark for evaluating multimodal, multi-type question understanding and reasoning, and identifies broader question types and more problems as future work.