Source-linked AI summary
CMMLU: Measuring massive multitask language understanding in Chinese
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, Timothy Baldwin
TL;DR
Assessing LLM knowledge and reasoning in Chinese is difficult, especially for fluent models and context-sensitive subjects. The paper introduces CMMLU, a broad Chinese benchmark, and evaluates multilingual and Chinese-oriented models across subjects and prompting settings. Results reveal substantial weaknesses, uneven subject performance, and limited benefits from chain-of-thought prompting, while data quality has an estimated 2% noise rate.
Problem
Assessing LLM knowledge and reasoning is increasingly challenging, particularly in Chinese linguistic and cultural contexts.
Method
The paper introduces CMMLU, a 67-subject Chinese assessment suite, and evaluates multilingual and Chinese-oriented LLMs across subjects and prompting settings.
Results
Most models struggled to reach 60% accuracy against a 25% random baseline, while GPT4 achieved 71% average accuracy and performance varied substantially by subject.
Takeaways & Limitations
CMMLU identifies substantial opportunities to improve Chinese knowledge and language understanding in existing LLMs.
Takeaways & Limitations
The dataset contains around 2% noise from missing correct answers or incorrect labels.
Abstract
from arXiv · showhide
As the capabilities of large language models (LLMs) continue to advance, evaluating their performance becomes increasingly crucial and challenging. This paper aims to bridge this gap by introducing CMMLU, a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. We conduct a thorough evaluation of 18 advanced multilingual- and Chinese-oriented LLMs, assessing their performance across different subjects and settings. The results reveal that most existing LLMs struggle to achieve an average accuracy of 50%, even when provided with in-context examples and chain-of-thought prompts, whereas the random baseline stands at 25%. This highlights significant room for improvement in LLMs. Additionally, we conduct extensive experiments to identify factors impacting the models' performance and propose directions for enhancing LLMs. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.
1 INTRODUCTION
CMMLU is introduced to evaluate LLM knowledge and reasoning in Chinese linguistic and cultural contexts, addressing the difficulty of assessing fluent models. Its results show broad weaknesses, uneven subject performance, and limited benefits from common prompting strategies.
- Motivation and contribution: CMMLU evaluates advanced LLM knowledge and reasoning across 67 subjects in a Chinese linguistic and cultural context.Its subjects range from physics and mathematics to humanities and social sciences, including context-sensitive tasks that are not easily translated.
- Motivation and contribution: Numerous CMMLU tasks have China-specific answers that may not be considered correct in other regions or languages.
- Evaluation results: 71% average accuracy was achieved by GPT4, while most assessed models struggled to reach 60% against a 25% random baseline.The evaluation included GPT4, ChatGPT, and more than 20 advanced open-source multilingual and Chinese models.
- Evaluation results: All models performed unevenly across subjects, scoring comparatively higher in humanities and social sciences but lower in China-specific and STEM subjects.
- Evaluation results: Most models did not benefit from chain-of-thought prompts, while few-shot examples helped foundation models but not SFT or RLHF models.Negation reduced performance, and sub-option questions were difficult, with GPT4 losing 20% accuracy on them.
2 RELATED WORK
Existing benchmarks support broad evaluation of language-model capabilities, but major multi-task benchmarks are primarily English-focused. CMMLU extends this evaluation landscape with broader Chinese and China-specific coverage than concurrent Chinese benchmarks.
- English benchmarks: Earlier benchmarks such as GLUE and SuperGLUE primarily measure specific language skills, motivating broader evaluations for generative models.
- English benchmarks: MMLU evaluates knowledge across multiple domains and tasks using a standardized multiple-choice format based on real-world exams.
- Chinese benchmarks: Existing major benchmarks are primarily focused on English, despite the emergence of several Chinese LLM evaluation benchmarks.
- Chinese benchmarks: CMMLU adds daily-life subjects, general world knowledge, China-specific knowledge, and fully public data compared with concurrent benchmarks.Examples include Chinese food culture and driving rules, alongside world religion, world history, and global facts.
3 CMMLU
CMMLU is a broad Mandarin Chinese multitask test combining general knowledge with region-specific subjects in a standardized four-choice format. The dataset was manually collected and quality-checked, with an estimated 2% noise rate.
- Task overview: CMMLU covers humanities, social sciences, STEM, and daily-life areas, including both region-independent and China-specific questions.Examples of region-specific tasks include Chinese driving rules, food culture, and teacher qualifications.
- Data collection: Four annotators manually collected questions and answers from freely available resources, including non-public materials and OCR-processed PDFs.The collection process took around 250 hours.
- Format: Each question has four choices with exactly one correct answer, using fill-in-the-blank or direct-answer formats.Chemical formulae and mathematical expressions use a 50:50 mixture of LaTeX and plain text.
- Format: Figure 2 illustrates CMMLU prompts with few-shot examples and answers, while bracketed English translations improve readability.
- Quality check: Quality checking estimated around 2% data noise from missing correct answers or incorrect labels.The authors state this error rate does not compromise overall results because most models average below 60% accuracy.
- Statistics: CMMLU contains 11,528 questions across 67 subjects, with 5-question few-shot development sets and test sets exceeding 100 questions per subject.The subjects include 17 STEM, 13 humanities, 22 social science, and 15 other tasks; 16 tasks are China-specific.
4 EXPERIMENTS
The experiments evaluate LLMs on CMMLU using multiple-choice strategies, model groups, subjects, and prompting conditions. Results show substantial differences across models and subjects, with performance affected by language specialization, model size, prompt type, negation, and question structure.
- Setup: Next token prediction is used for most reported results because the strategy comparison found it most efficient.Commercial models use free generation, while open-source models use next token prediction; free generation is retained for some analyses.
- By model: 53.21% average accuracy makes LLaMA2-70B the best open-source multilingual model, below ChatGPT at 55.51% and GPT4 at 70.95%.Seven-billion-parameter multilingual models generally achieve nearly random performance around 25%, while Chinese-resource fine-tuning improves multilingual models.
- By model: Baichuan2-13B achieves the best overall performance among Chinese LLMs and beats ChatGPT with only 13B parameters.Several Chinese LLMs are competitive with LLaMA2-70B below 20B parameters, and model families generally improve with increasing size.
- By subject: Performance is highest in humanities and social sciences, intermediate in China-specific subjects, and lowest in STEM subjects.Across subjects, models are uneven; ancient Chinese and college actuarial science are especially challenging, while legal and moral basis is among the easiest.
- Prompting: Most models do not improve with chain-of-thought prompting, and Xverse produces 19.77% more regex-unmatched responses than with direct-answer prompting.Some models fail to explicitly generate the answer option after analysis, reducing effective evaluation accuracy.
- Few-shot prompting: One example boosts foundation models but decreases fine-tuned-model performance; additional examples generally help foundation models but show no consistent trend for fine-tuned models.The authors relate this contrast to differences in instruction-following optimization.
- Model size: LLaMA and LLaMA2 gain 5 points from 7B to 13B, while Baichuan gains 10 points; larger size increases later yield diminishing returns.The comparison also reports 7% improvement for LLaMA and 15% for LLaMA2 with a fivefold size increase.
5 CONCLUSION
The paper introduces CMMLU to assess Chinese multi-task language understanding and identifies substantial opportunities to improve existing LLMs.
- CMMLU is introduced as a benchmark for assessing multi-task language understanding in Chinese.
- Extensive analyses identify factors affecting model performance and propose actionable directions for improving LLMs.
A COMPARISON TO CONCURRENT BENCHMARKS
CMMLU is compared with concurrent Chinese benchmarks, showing greater emphasis on culture-, region-, humanities-, and social-science-related tasks. Exact string matching finds limited overlap with CEval and M3KE.
- CMMLU contains more culture-related and region-related tasks than C-Eval and M3KE, while the benchmarks share similar task types.
- CMMLU can be jointly used with C-Eval and M3KE to assess Chinese language capabilities despite differences in task distributions.
- 74 exact-string overlaps are found with CEval and 158 with M3KE after sorting choices and removing punctuation.
- CMMLU emphasizes humanities, social science, and country- or culture-specific knowledge more than CEval and M3KE, while containing fewer STEM subjects.
B CMMLU SUBJECTS
CMMLU organizes 67 subjects into a broad assessment suite with documented subject coverage, category assignments, question statistics, and examples with English translations.
- Table 6 lists all CMMLU subjects, their covered concepts, supercategories, and total question counts.
- Table 7 reports supercategory statistics, including task and question counts, subject-level count ranges, and average token lengths.
- Figure 7 visualizes question and answer token lengths for each subject.
- Table 6 summarizes all 67 subjects, marks China-specific subjects with an asterisk, and defines # Q as the total questions per subject.
- Table 8 provides CMMLU examples by category, including corresponding English translations and bolded correct choices.
D CMMLU DIFFICULTY DISTRIBUTION
CMMLU spans difficulty levels from primary school to professional subjects, and estimates within-subject difficulty by counting how many top models answer each question correctly. The analysis also cautions against attributing size-related gains solely to emergent ability.
- Difficulty distribution: CMMLU includes 5 primary-school, 10 middle/high-school, 23 college, and 29 professional subjects.
- Difficulty distribution: The difficulty analysis evaluates the top 20 models and records, for each question, how many models answer it correctly.
- Difficulty distribution: In the violin plots, the x-axis counts models answering a question correctly and the y-axis counts questions at each level; left peaks indicate greater challenge.
- Difficulty distribution: Dual peaks indicate many relatively easy and challenging questions, with fewer intermediate-difficulty questions and a gradual transition between peaks.
- Emergent ability: LLaMA-2 performance generally increases with model size across subjects, but larger training data coverage and social-science concentration prevent a simple emergent-ability attribution.
G STRATEGIES FOR ESTIMATING MODEL CHOICES
The paper compares three strategies for estimating multiple-choice model choices: next-token prediction, perplexity comparison, and free generation. Next-token prediction is most efficient, while free generation requires answer extraction and next-token prediction generally performs better.
- Strategy 1 – Next Token Prediction: Next-token prediction compares probabilities for the four choice-indicator tokens and selects the token with the highest probability.The method prompts the model with a direct answer prefix after the question and candidate choices.
- Strategy 1 – Next Token Prediction: Next-token prediction is the most efficient strategy, but models may not generate one of the choice letters without suitable prompting.Few-shot examples with expected answers are proposed to mitigate this issue.
- Strategy 2 – Perplexity Comparison: Perplexity comparison selects the candidate whose concatenated question-and-answer sequence has the lowest perplexity.Each candidate answer is appended to the full question and candidate text before a forward pass.
- Strategy 3 – Free Generation: Free generation prompts the model to generate the correct choice and then uses auxiliary matching methods to extract it from the output.The strategy allows varied prompting but can produce costly, error-prone extraction and lengthy generations.
- Choice Extraction: The extraction procedure checks direct choice outputs and applies multiple matching patterns, returning E when no unique choice is confidently identified.The matching process searches answer statements, recursively identifies choices, applies weak patterns, and checks single-choice responses.
I CORRELATION TO OTHER BENCHMARKS
The paper examines whether CMMLU performance correlates with performance on established English benchmarks spanning language understanding, reasoning, mathematics, commonsense, and coding. CMMLU shows strong correlations with four benchmarks, while PIQA’s relevance is reduced because most models score above 80%.
- Correlation Analysis: The correlation analysis uses six popular English LLMs and five benchmarks covering language understanding, commonsense reasoning, general reasoning, mathematics, and coding.The selected benchmarks are RACE, CommonSenseQA, PIQA, GSM8K, and HumanEval.
- Correlation Analysis: CMMLU demonstrates strong correlation with four benchmarks spanning mathematics, commonsense reasoning, and coding.The cited analysis reports PIQA as the exception because most models achieve scores above 80% on that task.
- Correlation Analysis: 0.88 still indicates a strong positive correlation for PIQA despite its diminished relevance in the analysis.The reduced relevance is attributed to the high scores achieved by most models on PIQA.
J.1 RESULTS OF ZERO-SHOT
This section reports zero-shot CMMLU results for language models across five subject sub-categories. The supplied passage identifies the evaluation scope but does not provide category-level scores.
- Zero-Shot Results: Table 11 reports zero-shot model accuracy on CMMLU using macro-average accuracy over subjects within each of five categories.It also defines Overall as the macro-average score across all subjects and distinguishes Base from Chat models.
J.2 THE RESULTS OF EACH SUBJECTS
The paper compares zero-shot and five-shot subject-level performance and analyzes performance distributions across CMMLU subjects. Higher-performing models show diverse abilities, whereas lower-performing models struggle across most subjects; chain-of-thought prompting usually does not improve results.
- Subject-Level Results: Higher-performing LLMs exhibit diverse abilities across tasks, while lower-performing LLMs face challenges in most subjects.The comparison uses selected models and performance distributions across all subjects.
- Chain-of-Thought Results: Chain-of-thought prompting does not improve performance for most models, although ChatGPT and ChatGLM2 gain slightly on STEM subjects while overall accuracy decreases.Some models fail to explicitly generate the answer option after analysis, causing regex matching failures.
- Subject-Level Results: Table 10 reports 0-shot and 5-shot accuracy per subject for six selected LLMs.The listed models include LLaMA2-70B, Falcon-40B, Baichuan2-13B-Chat, ChatGLM2-6B, InternLM-Chat-20B, and BatGPT-15B-sirius.
- Zero-Shot Results: Table 11 reports zero-shot accuracy using macro averages within five subject categories and across all subjects.The table distinguishes multilingual- or English-oriented models from Chinese-oriented models and identifies Base versus Chat states.