Source-linked AI summary
Evaluating the Performance of Large Language Models on GAOKAO Benchmark
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, Xipeng Qiu, Tianxiang Sun, Peng Li, Shiqiao Meng, Yanjun Zheng, Jun Zhan, Zhangyue Yin, Xiannian Hu, Guofeng Quan, Qixiang Wang
TL;DR
Comprehensive evaluation of LLMs remains difficult because existing benchmarks do not fully capture subjective, generative, and human-centered examination tasks. This paper introduces GAOKAO-Bench, a 2010–2022 Chinese GAOKAO benchmark with objective and subjective questions, evaluated using zero-shot prompts, human scoring, and LLM-based judging. The models achieve competitive GAOKAO performance with subject disparities, while detailed marking criteria yield LLM judgments consistent with human preferences.
Problem
Existing benchmarks mainly use objective questions and lack subjective tasks closely related to generative abilities and real-world examination complexity.
Method
The paper builds GAOKAO-Bench from 2010–2022 Chinese GAOKAO questions and evaluates LLMs with zero-shot prompts, human scoring, and criteria-guided LLM judging.
Results
LLMs achieve competitive GAOKAO scores with subject bias, and detailed human-annotated marking criteria produce LLM evaluations consistent with human preferences.
Takeaways & Limitations
GAOKAO-Bench provides an intuitive, human-aligned standard for analyzing LLM capabilities across diverse Chinese examination subjects and question types.
Takeaways & Limitations
The study does not deeply analyze model errors and cannot use human scoring for every model because of time, resource, model-development, and evaluation-cost constraints.
Abstract
from arXiv · showhide
Large Language Models(LLMs) have demonstrated remarkable performance across various natural language processing tasks; however, how to comprehensively and accurately assess their performance becomes an urgent issue to be addressed. This paper introduces GAOKAO-Bench, an intuitive benchmark that employs questions from the Chinese GAOKAO examination as test samples, including both subjective and objective questions. To align with human examination methods, we design a method based on zero-shot settings to evaluate the performance of LLMs. With human evaluation, we obtain the converted total score of LLMs, including GPT-4, ChatGPT and ERNIE-Bot.Our findings reveal that LLMs have achieved competitive scores in Chinese GAOKAO examination, while they exhibit significant performance disparities across various subjects. We also use LLMs to grade the subjective questions, and find that model scores achieve a moderate level of consistency with human scores. In conclusion, this research contributes a robust evaluation benchmark for future large language models and offers valuable insights into the advantages and limitations of such models.
1 Introduction
Existing benchmarks largely emphasize objective questions, while GAOKAO-Bench adds Chinese GAOKAO tasks spanning objective and subjective formats to evaluate LLMs more comprehensively. Initial results show competitive GAOKAO scores, subject bias, and feasible LLM-based grading when detailed criteria are provided.
- Existing benchmarks cover many tasks but largely omit subjective questions tied to generative abilities and real-world examination complexity.
- GAOKAO-Bench targets Chinese GAOKAO questions involving computation, reasoning, knowledge assessment, and writing.
- 9 subjects and 2,811 questions comprise the benchmark, including 1,781 objective and 1,030 subjective questions across multiple question types.
- LLMs achieve competitive GAOKAO scores, but all evaluated models show clear subject bias across the examination.
- With detailed human-annotated marking criteria, LLM judges show high consistency with human teachers on subjective-question evaluation.
2 GAOKAO-Bench
GAOKAO is a rigorous, comprehensive Chinese university-entrance examination covering science and humanities streams and diverse question types. GAOKAO-Bench compiles national examination content from 2010 to 2022 into an intuitive, human-aligned benchmark, with questions organized by scoring requirements and format.
- 2.1 Introduction to the GAOKAO: GAOKAO assesses Chinese high-school students applying to universities through science and humanities streams.
- 2.1 Introduction to the GAOKAO: The examination includes logical reasoning, computational analysis, knowledge-based quizzes, and written expression.
- 2.2 Dataset Description: GAOKAO-Bench includes all national GAOKAO subjects from 2010 to 2022 and is designed as an intuitive, human-aligned LLM evaluation benchmark.
- 2.2 Dataset Description: Questions are converted from PDF to JSON through automated scripting and manual annotation, with mathematical formulas represented in LATEX.
- 2.2 Dataset Description: The dataset organizes questions into subjective and objective categories according to whether human scoring is required.
- 2.2 Dataset Description: Multi-Question Choice uses multiple sub-questions, whereas Multi-Choice permits multiple correct answers for one question.
3 Experiments
The experiments evaluate multiple LLMs on GAOKAO-Bench using zero-shot prompts, converted total scores, objective-question matching, human scoring, and LLM-based judging. Results show subject-dependent performance, strong overall performance from GPT-4 and ERNIE-Bot, and strong agreement between model and human grading.
- Prompt Design: Zero-shot prompts tailored to question types specify both the task and required output format.The design aims to assess intrinsic knowledge and instruction-following alongside task completion.
- Metric: Converted total scores combine subject-level objective and subjective scoring rates with their corresponding GAOKAO subject scores.Science and humanities totals are each 750 points.
- Question Evaluation: Objective questions award points only when the model outcome matches the standard answer, while subjective questions receive scores averaged from two teachers.Objective answers are scored using regular matching; subjective responses account for both reasoning and outcome.
- Overall Performance: GPT-4 exceeds 400 converted points, ERNIE-Bot surpasses ChatGPT, and every model scores higher in humanities than sciences.The paper links the science–humanities gap to the greater reasoning and calculation demands of science subjects.
- Results: GPT-4 leads most objective subjects, ERNIE-Bot leads chemistry, politics, and history, and GPT-4 obtains the highest subjective scoring rate at 51.9%.ERNIE-Bot reaches 48.4% on subjective questions, while GPT-4-turbo judging shows markedly strong positive correlations with human scores.
4 Analysis
LLMs show substantial subject-specific differences, with stronger performance in some language and knowledge areas than in mathematics and physics. Their converted scores remain stable across recent GAOKAO years, while science scores align more closely with human scoring than humanities scores.
- 4.1 Difference in Subjects: GPT-4 exceeds 70% in English, biology, and geography but scores below 40% in mathematics and physics across subjective and objective questions.ERNIE-Bot exceeds 60% in subjective biology, history, and politics, but scores below 30% in subjective mathematics.
- 4.1 Difference in Subjects: Subject differences reflect the distinct competencies evaluated and variations in training data and human-alignment tendencies.The paper identifies language comprehension and summarization, logical reasoning, and computation as examples of differing subject demands.
- 4.2 Difference Between Subjective and Objective Questions: Subjective-question scoring is generally lower than objective-question scoring, especially when mathematics requires correct formulas and extensive reasoning steps.The passage describes this pattern as a challenge for LLMs.
- 4.3 Stable Annual Trends on the GAOKAO: Converted total scores remain stable from 2013 to 2022, indicating relative stability in GAOKAO question difficulty.The annual scores are computed after categorizing questions by year.
- 4.4 LLM as a Judge: Science-score deviations from human scoring stay below 2% of the total, compared with around 5% for humanities.The paper attributes this difference to more explicit science answers and criteria versus humanities’ semantic alignment requirements.
5 Avoid Benchmark Leakage
GAOKAO-Bench may contain questions that appeared in LLM training data, creating a benchmark-leakage concern. The authors use zero-shot evaluation, human evaluation, and yearly additions to mitigate this issue.
- 5 Avoid Benchmark Leakage: GAOKAO questions may have appeared in LLM training corpora, making benchmark leakage a plausible evaluation concern.Benchmark leakage refers to evaluation data being used during model training.
- 5 Avoid Benchmark Leakage: The benchmark covers GAOKAO questions from 2010 to 2022 and supplements them with new annual questions to reduce leakage risk.GAOKAO-Bench-2023 adds objective questions from the 2023 examination.
- 5 Avoid Benchmark Leakage: The authors regard observed scoring-rate differences between GAOKAO-Bench and GAOKAO-Bench-2023 as normal difficulty fluctuations.
6 Ablation Study
The ablation study examines whether manually annotated marking criteria improve LLM grading of subjective questions. Providing these criteria makes LLM evaluations more consistent with human preferences.
- 6 Ablation Study: GPT-4-turbo grades GPT-4, ChatGPT, and ERNIE-Bot-turbo with and without manually annotated marking criteria.The study evaluates subjective-question grading under both conditions.
- 6 Ablation Study: The study measures grading consistency using Spearman and Kendall-Tau correlations with human judgement.
- 6 Ablation Study: Marking criteria improve LLM grading alignment with human preferences.The criteria break answers into specific scoring points for each item.
7 Related Work
Existing NLP benchmarks target narrower tasks, whereas newer benchmarks seek to evaluate the broader capabilities of large language models. Related work also uses human evaluation to compare models and assess generated outputs.
- Benchmark for LLMs: Traditional benchmarks such as GLUE and SQuAD do not measure the comprehensive capabilities of LLMs.GLUE targets natural language understanding, while SQuAD targets reading comprehension.
- Benchmark for LLMs: MMLU and BIG-Bench expand evaluation toward multitask and diverse-subtask assessment of advanced LLM abilities.
- Human evaluation for LLMs: Human evaluation is presented as more aligned with real-world applications and capable of providing broader, more precise feedback than automatic evaluation.Related work includes user voting and human scoring for chatbot and generation assessment.
8 Limitations
The paper identifies two main limitations: limited error analysis and incomplete human-scored evaluation across models.
- The paper explicitly frames its evaluation and analysis as limited in scope.
- The study does not analyze LLM errors in detail, including hallucinations and reasoning mistakes.The authors attribute this limitation to constraints in time and resources.
- Human scoring was not conducted for every model because LLM development is rapid and manual evaluation is costly.Future work aims to improve reasoning-process analysis and use LLMs as a replacement for human scoring.
9 Conclusion
The conclusion presents GAOKAO-Bench as a broad evaluation standard and reports strong knowledge performance alongside weaknesses in reasoning, mathematics, and long-text Chinese comprehension.
- GAOKAO-Bench covers Chinese College Entrance Examination questions from 2010 to 2022 across subjects and question types.The dataset is intended to support an intuitive analysis of LLM advantages and gaps compared with humans.
- The evaluation uses zero-shot prediction and human evaluation to assess LLM answers to GAOKAO questions.
- LLMs perform well on knowledge-based questions but struggle with some logical reasoning, mathematical problems, and longer Chinese reading-comprehension tasks.
- LLM-as-a-Judge evaluations equipped with human-annotated marking criteria are consistent with human preference.
- The findings suggest potential applications in education and language assessment while leaving room for improvement in reasoning, calculation, and long-text comprehension.
A.1 Data Example
The appendix illustrates GAOKAO-Bench examples through a science mathematics single-choice item and a geography open-ended item with an LLM-generated response.
- The science mathematics example is a single-choice question, with an English translation added for readers’ understanding.
- The mathematics example includes a detailed solution and identifies option C as the correct answer.
- ChatGPT’s geography response proposes management, science education, visitor limits, and supporting facilities.The response emphasizes protecting relics, raising environmental awareness, limiting visitor flow, and planning facilities appropriately.
- The geography prompt requires answers in specified Analysis and Answer sections for each sub-question.
- The geography example asks for tourism-resource conservation measures for the Fengyan Ancient Terraces Ecological Museum.The site includes Qing Dynasty terraces and associated cultural relics in an approximately 800-hectare conservation area.
A.3 Model Grading Example
The model-grading example shows a structured geography-scoring prompt and a 10-point evaluation in which ChatGPT’s answer receives credit for four conservation measures.
- The grading prompt instructs a high school geography teacher to score student analyses against questions, standard answers, scoring criteria, and point allocations.
- ChatGPT’s response earns 3 points each for management and science education, plus 2 points each for visitor limitations and supporting facilities.The rationale links each response element to corresponding standard-answer criteria.
- The example awards ChatGPT a total of 10 points for the geography question.
- The benchmark reports converted subject scores and significant differences between objective and subjective question types in its tables.Table 7 presents converted total scores by subject, while Table 8 distinguishes objective questions above the line from subjective questions below it.