Source-linked AI summary
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan
TL;DR
Traditional artificial benchmarks may not accurately assess foundation models on human-level tasks, motivating a more human-centric evaluation. The paper introduces AGIEval, a benchmark based on standardized human exams, and finds broad strengths alongside weaker reasoning and calculation abilities.
Problem
Traditional benchmarks use artificial datasets and may not adequately evaluate foundation models’ general abilities on human-centric reasoning and problem-solving tasks.
Method
AGIEval evaluates foundation models on official human admission, qualification, and competition exams spanning bilingual Chinese and English tasks.
Results
The evaluation reveals strengths and limitations across understanding, knowledge, reasoning, and calculation, with GPT-4 exceeding average human performance on several exams.
Takeaways & Limitations
AGIEval provides a benchmark for evaluating foundation models on human-level cognition and real-world-relevant tasks.
Takeaways & Limitations
Calculation ability is weaker than understanding and varies by subject, with challenges in chemistry and biology involving specialized notation and symbol substitution.
Abstract
from arXiv · showhide
Evaluating the general abilities of foundation models to tackle human-level tasks is a vital aspect of their development and application in the pursuit of Artificial General Intelligence (AGI). Traditional benchmarks, which rely on artificial datasets, may not accurately represent human-level capabilities. In this paper, we introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark. Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam. This demonstrates the extraordinary performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal these models' strengths and limitations, providing valuable insights into future directions for enhancing their general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a more meaningful and robust evaluation of foundation models' performance in real-world scenarios. The data, code, and all model outputs are released in https://github.com/ruixiangcui/AGIEval.
1 Introduction
AGIEval addresses limitations of artificial benchmarks by evaluating foundation models on human-centric exams tied to cognition, reasoning, and problem-solving. The study compares several models with human performance and analyzes their capabilities and limitations.
- Traditional benchmarks rely on artificial datasets and underemphasize real-world tasks requiring human-like cognition.This can limit their usefulness for evaluating reasoning, problem-solving, and real-world applicability.
- GPT-4 scores above average human performance on the SAT, LSAT, and math competitions.
- AGIEval evaluates foundation models using official, public, and high-standard human exams spanning college admissions, law, mathematics, civil service, and qualification testing.The benchmark includes bilingual Chinese and English tasks.
- The experiments compare foundation models with human performance across multiple human-centric exams and learning settings.The study considers few-shot, zero-shot, and chain-of-thought prompting settings.
- Analyses of understanding, knowledge, reasoning, and calculation identify broad capability patterns, strengths, and limitations.The analysis examines both single-task behavior and general patterns.
- The benchmark is intended to support more reliable and effective AI systems by improving evaluation of human-level tasks.
2 Background and Related Work
Existing benchmarks assess language models across tasks but often use artificial datasets and simpler textual skills rather than complex, human-centric reasoning. AGIEval responds by using official human exams and standardized automatic evaluation for recent foundation models.
- Recent foundation models achieve strong performance, but their reasoning abilities, trustworthiness, and real-world applicability remain concerns.
- GLUE and SuperGLUE evaluate general language capabilities across diverse NLP tasks such as paraphrase identification and sentiment analysis.
- Many existing benchmarks rely on artificially curated datasets for specific machine skills and simpler textual understanding.They therefore differ from real-world problems involving human behavior and complex reasoning.
- MMLU moves toward human-centric evaluation by covering diverse subjects, but differs from AGIEval in benchmark source.AGIEval uses high-standard official admission and professional qualification exams.
- AGIEval releases questions from official human exams, supports standardized automatic evaluation, and includes experiments with ChatGPT and GPT-4.The model outputs are also released for community analysis and research.
3 Human-Centric Benchmark
AGIEval is built from standardized human exams selected for cognitive and real-world relevance, with broad coverage across disciplines and languages. Its construction uses objective questions, multiple data sources, standardized metrics, and human-performance references.
- Design Principles: The benchmark prioritizes official exams whose tasks align with human cognition, problem-solving, and real-world decision-making.Its design emphasizes challenges with practical implications for real-world situations.
- Exam Selection: The exam collection spans college admission, law-school admission, lawyer qualification, GMAT, math competitions, and civil-service testing.
- Question Formats: Subjective questions are removed, leaving multiple-choice and fill-in-the-blank items for consistent assessment.
- Data Collection: Questions are collected from public sources and existing datasets covering exams including Gaokao, SAT, LSAT, civil service, math competitions, GRE, and GMAT.
- Benchmark Construction: The benchmark contains 8,062 questions and bilingual Chinese and English tests.This design evaluates model capabilities and adaptability across languages.
- Evaluation: Multiple-choice items use classification accuracy, while fill-in-the-blank items use Exact Match and F1 metrics.
- Human Performance: Human average and top performances are reported as task-specific human-level boundaries, but provide only an approximation of test-taker abilities.
4 Evaluation of Foundation Models
The evaluation compares GPT-4, ChatGPT, Text-Davinci-003, and Vicuna across AGIEval tasks. It presents model descriptions and a comparative analysis under multiple evaluation settings.
- Models Selected: The study evaluates GPT-4, ChatGPT, Text-Davinci-003, and the open-source Vicuna-13B model.
- Models Selected: GPT-4 is described as a state-of-the-art generative pretrained transformer with enhanced performance and a broader knowledge base than its predecessors.
- Models Selected: ChatGPT is presented as an instruction-trained conversational model further tuned with reinforcement learning from human feedback.
- Models Selected: Text-Davinci-003 represents an intermediate GPT-3.5 model selected to support comparative analysis between GPT-3 and GPT-4.
- Evaluation Approach: The evaluation methodology and comparative benchmark results are presented in subsequent sections.
4.2 Experimental Setup
The benchmark evaluates models with zero-shot, few-shot, and chain-of-thought prompting across human-centric reasoning tasks, using quantitative and qualitative assessments.
- Zero-shot and Few-shot Evaluation: Zero-shot evaluation presents task questions without prior examples, testing models’ ability to reason and solve problems without explicit training.
- Zero-shot and Few-shot Evaluation: Few-shot evaluation provides a small number of same-task examples, testing rapid adaptation when direct supervision is scarce.
- Chain-of-Thought Evaluation: Chain-of-thought evaluation asks models to generate reasoning before answering, with few-shot CoT combining explanation and answer generation in one step.
- Evaluation Metrics: The study evaluates models through task-specific accuracy or Exact Match, supplemented by human judgments of understanding, knowledge, and reasoning quality.
- Implementation Details: Experiments use Azure OpenAI completion or chat-completion APIs, while ChatGPT few-shot experiments use the completion API because of conversational-pattern difficulties.
4.3 Main Results
Across 20 human-centric tasks and multiple prompting settings, GPT-4 generally leads the evaluated models, while performance varies with knowledge demands, reasoning complexity, and task design.
- Model Comparisons: GPT-4 significantly outperforms ChatGPT and Text-Davinci-003 on average across all four evaluation settings.
- Model Comparisons: ChatGPT significantly outperforms Text-Davinci-003 on tasks requiring substantial external knowledge, including geography, biology, chemistry, physics, and mathematics.
- Model Comparisons: ChatGPT slightly outperforms or matches Text-Davinci-003 on understanding-focused English and LSAT tasks across evaluation settings.
- Task Difficulty: All evaluated LLMs struggle with complex reasoning tasks such as MATH, LSAT-AR, GK-physics, and GK-Math.
- Prompting Effects: Few-shot learning generally yields only limited improvement over zero-shot learning, suggesting that current zero-shot abilities approach few-shot abilities.
- Open-Source Models: Vicuna falls short on AGIEval despite excelling on the OpenLLM leaderboard and claimed comparable ability with ChatGPT.
4.4 Analyses of Chain-of-thought Prompting (CoT)
Chain-of-thought prompting improves some tasks but harms others, with its effectiveness depending on task, backbone model, and language.
- Performance Variability: CoT substantially improves performance on English mathematical exams, including MATH, AQuA-RAT, and SAT-Math, but degrades performance on several other tasks.
- Backbone Dependency: CoT effectiveness depends on the underlying model; GPT-4 generates more illustrative reasoning processes that produce enhanced performance with CoT.
- Language Sensitivity: CoT improves LogiQA-English but degrades LogiQA-Chinese performance for both ChatGPT and GPT-4.
- Language Sensitivity: CoT boosts English math performance but reduces performance on the Chinese Gaokao math exam.
- Overall Implications: CoT effectiveness is related to task, model capability, and language, which should be considered when applying or developing prompting methods.
4.5 Qualitative Analyses of Model Capabilities
The models generally understand questions and handle simple reasoning, but remain weaker in complex reasoning, domain-specific knowledge, calculation, and robustness across contexts.
- Understanding: The models generally interpret question meanings accurately, demonstrating strong contextual understanding.
- Knowledge: The models identify correct knowledge or formulas for mathematical and logical tasks but struggle with specific law, biology, and physics knowledge.
- Reasoning: Complex reasoning remains underdeveloped, with errors involving strict logical deduction, contradictory conclusions, concept substitution, and contextual disturbance.
- Calculation: Calculation is weaker than understanding and varies by subject, with errors especially involving complex variable substitutions, mathematical symbols, chemistry, and biology.
- Strengths: The models handle simple deductions and generate general reasoning strategies, including deriving basic constraints in LSAT-AR.
- Overall assessment: These strengths coexist with substantial room for improvement in complex reasoning and domain-specific knowledge.
- Weaknesses: Long contexts can disrupt comprehension and reasoning, while insufficient commonsense or domain-specific knowledge can hinder plausible explanations and accurate answers.
5 Discussion about Future Directions
The discussion proposes improving foundation models through external knowledge, stricter reasoning, multilingual and multimodal evaluation, better metrics, and more robust reasoning across contexts.
- External knowledge: External knowledge sources, formulas, specialized knowledge bases, and knowledge-enhanced prompting could improve mathematical and domain-specific task performance.
- Reasoning: Strict complex logical reasoning could be improved with dedicated datasets, symbolic tools, and execution-based reasoning verification.
- Multilingual reasoning: Enhancing multilingual reasoning generalization is important because reasoning ability varies across languages and is relatively better for English.
- Multimodal evaluation: Multimodal evaluation should incorporate visual, auditory, or interactive tasks requiring reasoning across multiple input and output types.
- Evaluation metrics: More robust automatic metrics should capture understanding, knowledge, and reasoning while reflecting real-world task complexity.
- Reasoning robustness: Reasoning robustness should be strengthened to maintain consistency and reliability across contexts and input variations.
- Overall direction: These directions aim to improve model capabilities for complex human-centric tasks with greater accuracy and reliability.
6 Conclusion
AGIEval evaluates foundation models on official human-oriented exams, finding strong GPT-4 performance alongside limitations revealed by manual analysis.
- AGIEval uses official admission tests, qualification exams, and advanced competitions designed for human participants, including bilingual Chinese and English tasks.
- GPT-4 surpasses average human performance on LSAT, SAT, and math competitions.
- 95% accuracy on the SAT Math test and 92.5% accuracy on the Gaokao English test demonstrate GPT-4's strong performance.
- Manual analyses examine model behavior across understanding, knowledge, reasoning, and calculation, identifying strengths and limitations.
A Data Examples
The paper presents examples from Gaokao data and an SAT question paired with GPT-4's generated Chain-of-Thought reasoning process.
- Figure 4 shows examples of Gaokao data.
- Figure 5 presents an SAT example alongside a Chain-of-Thought reasoning process generated by GPT-4.
B Data Contamination Issue
The paper examines data-contamination risk by comparing GPT-4 performance on potentially contaminated and later-released Gaokao data, finding mostly minor declines outside Mathematics.
- The benchmark addresses contamination concerns by timestamping nine new Gaokao datasets and evaluating tests released after ChatGPT and GPT-4 training-data timestamps.
- GPT-4 performance experiences a minor drop without contamination for most subjects except Mathematics, while remaining close to complete-dataset performance.
- The uncontaminated set contains examples released after the training-data timestamps of ChatGPT and GPT-4.