Source-linked AI summary
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, Junxian He
TL;DR
Existing benchmarks provide limited evidence about advanced LLM abilities in Chinese contexts and Chinese-specific knowledge. The paper introduces C-EVAL, a broad Chinese exam-based suite with a challenging C-EVAL HARD subset, and evaluates leading models. GPT-4 is the only model exceeding 60% average accuracy on C-EVAL, while difficult reasoning remains challenging and the benchmark does not cover all important evaluation dimensions.
Problem
Modern benchmarks primarily target English, while Chinese LLM evaluation lags and translated English tests omit knowledge relevant to Chinese users.
Method
C-EVAL uses Chinese exam questions spanning 52 disciplines and four difficulty levels, with C-EVAL HARD selecting eight especially challenging reasoning subjects.
Results
GPT-4 is the only model exceeding 60% average accuracy on C-EVAL, achieving 66.4%, while most models are near-random on C-EVAL HARD.
Takeaways & Limitations
C-EVAL can help developers analyze foundation-model strengths and limitations across multiple abilities for Chinese users.
Takeaways & Limitations
C-EVAL does not evaluate abilities such as API reasoning and calling, or dimensions beyond accuracy including safety, bias, and robustness.
Abstract
from arXiv · showhide
New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.
1 Introduction
C-EVAL addresses the limited evaluation of advanced LLM abilities in Chinese contexts by covering diverse knowledge and reasoning domains. Its results show that current models, including GPT-4, still face substantial challenges, especially on difficult reasoning tasks.
- Motivation: Modern benchmarks mainly target English, limiting understanding of LLM capabilities in other languages and Chinese-specific knowledge.The paper argues that Chinese evaluation should cover interests such as Chinese culture, history, and laws rather than simply translate English benchmarks.
- Benchmark: C-EVAL contains 13,948 multiple-choice questions across 52 disciplines and four difficulty levels, from middle school to professional.The subjects range from humanities to science and engineering.
- Benchmark: C-EVAL HARD selects particularly challenging subjects requiring advanced reasoning, including advanced mathematics and college physics.It is presented as a Chinese benchmark for highly complicated reasoning questions.
- Results: 66.4% average accuracy made GPT-4 the only evaluated model exceeding 60% on C-EVAL, while indicating substantial room for improvement.GPT-4, ChatGPT, and Claude were the top three performers despite not being specially tailored for Chinese data.
- Results: GPT-4 achieved 53.3% accuracy on C-EVAL HARD, where most models retained near-random accuracy.The results highlight the difficulty of advanced reasoning subjects.
- Implications: C-EVAL is intended to help developers analyze foundation-model strengths and limitations across multiple dimensions for Chinese users.The benchmark can be used as a whole or through subsets targeting particular abilities.
2 The C-EVAL Evaluation Suite
C-EVAL is constructed from challenging Chinese exams to measure advanced abilities across broad subjects and difficulty levels. The suite also includes a hard subset, structured data processing, explanations, and private test labels to support evaluation.
- 2.1 Design Principle: C-EVAL targets advanced world knowledge and reasoning using multiple-choice questions with clearly defined accuracy metrics.Each question has four choices and one correct answer.
- 2.1 Design Principle: The suite mitigates contamination risk by collecting mock or local exam questions rather than relying primarily on widely distributed national tests.Source documents are parsed and annotated, with complex mathematical notation converted into standardized formats.
- 2.2 Data Collection: Questions span middle-school, high-school, college, and professional levels, with college subjects selected across all 13 official undergraduate categories.Professional subjects include representative qualification exams such as physician, legal, and civil-servant tests.
- 2.2 Data Collection: C-EVAL questions are gathered mainly from online mock exams and processed from PDFs, Word documents, and web pages into structured formats.Authors manually parse questions when needed and convert complex STEM notation into standard formats.
- 2.2 Data Collection: The benchmark provides chain-of-thought explanations to facilitate potential use for reasoning evaluation.Generated explanations are manually revised before inclusion.
- 2.3 C-EVAL HARD: C-EVAL HARD comprises eight challenging mathematics, physics, and chemistry subjects requiring non-trivial reasoning abilities.Its subjects include advanced mathematics, discrete mathematics, probability and statistics, college and high-school chemistry, and college and high-school physics.
3 Experiment
C-Eval evaluates accessible Chinese-capable LLMs across zero-shot, five-shot, answer-only, and chain-of-thought settings. Results show strong differences across models, prompting strategies, model orientations, and task difficulty.
- Models and settings: Eleven accessible top-performing LLMs capable of processing Chinese are evaluated, covering English- and Chinese-oriented models of varying sizes.The evaluation reports answer-only results in zero- and five-shot settings, and five-shot chain-of-thought results.
- Overall comparison: GPT-4 is the only model exceeding 60% average accuracy, while ChatGPT trails by over 14 percentage points in both zero- and five-shot settings.Claude-v1.3 performs similarly to ChatGPT on category-wise and overall averages.
- Prompting effects: Few-shot prompting improves many models but lowers performance for GLM-130B, Bloomz-mt, ChatGLM-6B, MOSS, and Chinese-Alpaca-13B.The authors hypothesize that these instruction-tuned models may not have appropriately incorporated few-shot demonstrations during instruction tuning.
- Prompting effects: Chain-of-thought prompting does not necessarily improve average accuracy, although it raises GPT-4’s college physics accuracy from 50.6% to 60.2%.The authors attribute uneven effects to non-reasoning-intensive subjects and models’ ability to leverage chain-of-thought prompting.
- English- and Chinese-oriented models: GLM-130B trails ChatGPT by 12.3 points in STEM but by only 2.2 and 1.1 points in social science and humanities, respectively.The comparison suggests Chinese-oriented models may perform relatively well on areas involving Chinese knowledge while lagging on complex reasoning tasks.
- C-Eval Hard: On C-Eval Hard, GPT-4 reaches 53.3%, 54.9%, and 56.8% in zero-shot AO, five-shot AO, and five-shot COT settings, respectively.Only GPT-4, ChatGPT, and Claude improve by at least 10 points over a random baseline, emphasizing distinctions among models on complex tasks.
4 Related Work
Chinese benchmark coverage has expanded beyond CLUE to include exams spanning broader domains and difficulty levels, while also addressing data leakage concerns.
- Chinese benchmarks: Chinese benchmark development has lagged behind English benchmarks, with CLUE remaining the most widely used and best available Chinese standard.
- Chinese benchmarks: AGIEval and MMCU draw on Chinese entrance, qualification, and university examinations across selected domains.
- Chinese benchmarks: C-EVAL broadens domain coverage, provides four difficulty levels, and introduces C-EVAL HARD for sophisticated Chinese reasoning problems.
- Chinese benchmarks: C-EVAL primarily uses processed mock-exam documents, whereas AGIEval and MMCU collect exact questions from past examinations, helping mitigate data leakage.
5 Discussion
C-EVAL is intended to support evaluation beyond casual conversation, but the authors identify important abilities and quality dimensions that remain outside its scope.
- Discussion: C-EVAL aims to guide developers in preparing LLMs for complex scenarios beyond casual conversational bots.
- Discussion: The authors state that C-EVAL and English-language benchmarks remain imperfect for evaluating LLMs.
- Discussion: Evaluation of API reasoning and calling, safety, bias, and robustness is left for future work beyond accuracy.
- Discussion: C-EVAL includes tasks and broader categories with the number of questions listed for each task.
C Explanation Data Generation
The explanation data are generated with GPT-4 from human-written examples and then manually revised before finalization.
- Explanation Data Generation: Five human-written question-explanation pairs prompt GPT-4 to generate explanations containing each question and its correct answer.
- Explanation Data Generation: The generated explanations are manually revised to produce the final explanation data.
D Evaluation Prompts
The evaluation uses separate prompts for answer-only and chain-of-thought testing.
- Evaluation Prompts: Figure 6 presents the answer-only evaluation prompts, while Figure 7 presents the chain-of-thought evaluation prompts.
E Details of the models being evaluated
The evaluation covers a broad set of English-oriented, Chinese-oriented, multilingual, and lightweight language models. The models differ in pretraining, alignment, Chinese adaptation, quantization, and deployment characteristics.
- Model families: GPT-4 and ChatGPT are GPT-series models aligned through reinforcement learning from human feedback, while GPT-4 additionally supports image inputs and larger-scale post-training alignment.GPT-4 is described as achieving human-level performance on various benchmarks and top-10% performance on some simulated exams.
- Model families: Claude uses constitutional AI for human-intention alignment, and Claude-instant is its lighter, lower-cost, faster-inference version.
- Model families: BLOOMZ-mt combines multilingual BLOOM pretraining with multitask fine-tuning on English and machine-translated prompts, and the evaluation uses its 176B version.
- Evaluation formats: The prompts include answer-only and chain-of-thought evaluations across subjects such as physics, clinical medicine, computer architecture, and respiratory care.Figure examples show few-shot answer-only and chain-of-thought settings, while zero-shot evaluation removes the exemplars.
- Chinese-oriented models: Chinese-LLaMA, Chinese-Alpaca, and MOSS extend or adapt open models through Chinese-language pretraining, instruction tuning, alignment, or plugin augmentation.The experiments evaluate the 13B Chinese-LLaMA and Chinese-Alpaca variants and the moss-moon-003-sft version.
F Breakdown of Model Performance
Performance is reported both overall and by subject, enabling comparison of representative models across zero-shot and five-shot settings. Detailed subject-level results are provided in tables and an online leaderboard.
- Subject-level breakdown: The C-Eval website leaderboard provides detailed result breakdowns for all evaluated models.
- Subject-level breakdown: Table 9 reports zero-shot answer-only accuracy separately for each subject.
G Option Bias
The authors test whether answer-option ordering biases C-Eval results. Correct-answer proportions fluctuate only modestly around the random 25% level, and performance variance across permutations is relatively small.
- Answer distribution: Correct-answer proportions fluctuate relatively little around the random 25% level and are similar to MMLU.
- Permutation test: Five distinct answer-choice permutations are evaluated for ChatGPT, ChatGLM-6B, and ChatGLM2-6B in the zero-shot answer-only setting.Table 12 reports average accuracy within each category, overall accuracy, and variance across the five permutations.
- Permutation test: The variance of overall accuracy across different choice permutations is relatively small.
H Compute and Resources Used for Evaluation
Evaluation resources depend on model scale and access mode: released-weight models run on an 8-GPU A100 cluster, while API models run on CPUs. In most cases, inference finishes within one day.
- Compute resources: Released-weight models are evaluated on a cluster with 8 A100-80GB GPUs, while API-access models are evaluated using CPUs.The GPU cluster supports inference for BLOOMZ-mt 176B, LLaMA-65B, and GLM-130B; most evaluations finish within one day.