PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Evaluation of Large Language Models

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, Xing Xie

arXiv:2307.03109v9cs.CLcs.AI

TL;DR

LLM evaluation needs a comprehensive account spanning tasks, benchmarks, and protocols, including limitations such as reasoning and robustness. This survey synthesizes these dimensions and finds that evaluation systems must adapt as models evolve.

  • Problem

    Existing LLM evaluation is fragmented across tasks, datasets, benchmarks, and protocols, limiting a comprehensive view of model capabilities and limitations.

  • Method

    The paper surveys LLM evaluation across what to evaluate, where to evaluate, and how to evaluate, covering tasks, benchmarks, protocols, and future challenges.

  • Results

    Current LLMs exhibit limitations across numerous tasks, notably reasoning and robustness, while evaluation systems need to adapt and evolve.

  • Takeaways & Limitations

    Accurate assessment of LLM capabilities and limitations requires contemporary evaluation systems that evolve alongside the models.

  • Takeaways & Limitations

    Static public benchmarks may fail to assess evolving abilities accurately and may be memorized, creating potential training-data contamination.

Abstract

from arXiv · show

Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, educations, natural and social sciences, agent applications, and other areas. Secondly, we answer the `where' and `how' questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey.

1 INTRODUCTION

LLMs’ broad capabilities and growing societal importance make rigorous evaluation essential for understanding their strengths, weaknesses, safety, reliability, and risks. This survey organizes LLM evaluation around what, where, and how to evaluate, reviewing tasks, benchmarks, protocols, findings, and future challenges.

  • Motivation: The Turing Test established machine evaluation as a longstanding objective for assessing whether responses originate from humans or machines.The introduction frames AI history as the creation and evaluation of intelligent models.
  • Motivation: LLMs have attracted substantial academic and industrial interest because they can solve diverse general and domain-specific tasks.Their performance has raised hopes that they could represent artificial general intelligence in the current era.
  • Why evaluate: Evaluation helps identify LLM strengths and weaknesses, guide human–LLM interaction, and address safety and reliability in sensitive sectors.PromptBench shows that current LLMs are sensitive to adversarial prompts, making careful prompt engineering necessary.
  • Why evaluate: Existing evaluation protocols may be insufficient because LLMs are becoming larger and developing more emergent abilities, while evaluations span many aspects without a comprehensive overview.Research has examined natural language tasks, reasoning, robustness, trustworthiness, medical applications, and ethics, but the evolving field presents continuing challenges.
  • Survey scope: The survey reviews LLM evaluation in three dimensions: what to evaluate, where to evaluate, and how to evaluate.These dimensions cover evaluation tasks, datasets and benchmarks, and evaluation processes and approaches.
  • Contributions: The paper summarizes task-level successes and failures, evaluation metrics, datasets, benchmarks, protocols, novel approaches, and future challenges, while maintaining open-source materials.The paper’s sections cover LLM and AI evaluation basics, tasks, datasets and benchmarks, evaluation methods, findings, challenges, and conclusions.

2 BACKGROUND

LLMs are advanced language models built on Transformer self-attention, generating text autoregressively and through in-context prompts. Their evaluation commonly uses validation protocols and static test sets, but these may not fully capture LLM capabilities.

  • Language Models: Language models understand and generate human language by predicting word-sequence probabilities or producing text from input.N-gram models estimate word probabilities from preceding context but face rare-word, unseen-word, overfitting, and context-capture challenges.
  • Large Language Models: LLMs extend language models with massive parameter sizes and exceptional learning capabilities, commonly using Transformer self-attention.GPT-3, InstructGPT, and GPT-4 are examples of LLMs built around this core module.
  • Large Language Models: Prompt engineering, question answering, and dialogue are common ways to interact with LLMs, with prompt design widely adopted in evaluation.In-context learning enables responses based on a supplied context or prompt.
  • Large Language Models: Autoregressive LMs predict each next token conditioned on prior context, decomposing sequence probability into position-wise probabilities to generate complete text sequences.The model predicts y from X and applies the chain rule across a sequence of length T.
  • Model Evaluation: Standard evaluation protocols include k-fold cross-validation, holdout validation, LOOCV, bootstrap, reduced set, and static validation or test sets.Deep learning models often use static sets such as ImageNet, MS COCO, GLUE, or SuperGLUE because some protocols are infeasible at large training sizes; existing protocols may not thoroughly assess LLM capabilities.

3 WHAT TO EVALUATE · 3.1 Natural Language Processing Tasks

The paper organizes LLM evaluation around diverse task categories, with natural language processing covering understanding, reasoning, generation, multilingual performance, and factuality. Across these tasks, LLMs show strong but uneven capabilities, with notable weaknesses in mathematical reasoning, semantic judgment, multilingual zero-shot performance, and factual completeness.

  • 3 WHAT TO EVALUATE: The survey classifies evaluation into NLP, robustness, ethics, biases and trustworthiness, social sciences, natural science and engineering, medical, agent, and other applications.The authors note that this taxonomy is only one possible classification and that other taxonomies exist.
  • 3.1 Natural Language Processing Tasks: Natural language processing is the dominant evaluation focus because language models were initially developed to improve language understanding and generation.The survey summarizes NLP evaluation across NLU, reasoning, NLG, and multilingual tasks.
  • 3.1.1 Natural language understanding.: LLMs perform strongly on sentiment analysis, but their language understanding remains uneven across classification, inference, semantics, and social knowledge.GLM-130B achieved 85.8% overall accuracy for miscellaneous text classification; ChatGPT outperformed GPT-3.5 on NLI, while other studies found poor NLI, nonsense-phrase, and social-knowledge performance.
  • 3.1.2 Reasoning.: ChatGPT outperforms GPT-3.5 on most arithmetic reasoning tasks, yet mathematical and symbolic reasoning remain difficult despite broader improvement in complex reasoning benchmarks.ChatGPT can match human participants on verbal insight problems, but uncertain responses contribute to weaker symbolic reasoning.
  • 3.1.3 Natural language generation.: NLG evaluation spans summarization, dialogue, translation, question answering, and open-ended generation, with results varying substantially by task and model.TNLG v2 (530B) ranked highest in both summarization scenarios, while fine-tuned Bart exceeded zero-shot ChatGPT; Claude and ChatGPT generally outperformed GPT-3.5 in dialogue.
  • 3.1.4 Multilingual tasks.: Although multilingual training enables cross-language processing, evaluations remain predominantly English-focused, and ChatGPT underperforms state-of-the-art models on most standard Arabic NLP tasks in zero-shot settings.Multilingual applications also require attention to impartiality and neutrality to mitigate English and other language biases.
  • 3.1.5 Factuality.: Factuality is important across downstream applications, yet even GPT-4 and BingChat answer correctly for more than 80% of open questions while retaining a gap of over 15% to complete coverage.The evaluation used human assessment on Natural Questions and TriviaQA.

3.2 Robustness, Ethic, Bias, and Trustworthiness

This section surveys LLM evaluation across robustness, ethics, bias, and trustworthiness, emphasizing vulnerability to unexpected or adversarial inputs, harmful information and social biases, cognitive effects, and hallucinations.

  • Robustness: Robustness evaluation examines LLM stability under unexpected inputs, especially out-of-distribution and adversarial conditions, using benchmarks including AdvGLUE, ANLI, and DDXPlus.Early work evaluated ChatGPT and other LLMs from both adversarial and OOD perspectives.
  • Robustness: PromptBench evaluated character-, word-, sentence-, and semantics-level attacks, finding that contemporary LLMs are vulnerable to adversarial prompts.The results highlight the importance of robustness when models face adversarial inputs.
  • Ethics and Bias: LLMs can internalize, spread, and potentially magnify toxic language and social biases, including stereotypes involving gender, race, religion, occupation, and ideology.Examples of toxic language include offensiveness, hate speech, and insults.
  • Trustworthiness: DecodingTrust assessed GPT-3.5 and GPT-4 across eight trustworthiness aspects, extending evaluation beyond typical robustness and ethics concerns.The listed aspects include toxicity, stereotype bias, adversarial and out-of-distribution robustness, robustness to adversarial demonstrations, and privacy.
  • Cognitive and Ethical Evaluation: Cognitive reflection tests and semantic illusion experiments found that LLMs with enhanced cognitive abilities can avoid common human intuitions and cognitive errors, exhibiting super-rational performance.These methods provide perspectives for evaluating psychological aspects, model biases, and ethical issues.
  • Trustworthiness: Hallucination evaluation targets coherent, seemingly factual outputs that contain inaccuracies or statements ungrounded in reality, helping improve training methods to reduce them.Hallucination is presented as a key issue in generated information.

3.3 Social Science

Evaluations of LLMs in social science span political analysis, computational social science, legal tasks, and psychology. Findings show useful contributions alongside substantial weaknesses, so LLMs improve productivity but cannot replace human professionals.

  • Social Science: LLMs generate meaningful responses about political ideology and significantly improve text-as-data methods in social science.These capabilities may help address scaling and measurement issues in social-science research.
  • Social Science: In computational social science classification, LLM accuracy falls below 40% on event argument extraction, character tropes, implicit hate, and empathy classification.These tasks involve complex structures or subjective expert taxonomies whose semantics differ from LLM pretraining.
  • Social Science: Zero-shot legal case-judgment summarization is mediocre and produces incomplete or meaningless text, sentence-merging errors, inconsistencies, and hallucinated information.Further improvement is necessary before legal experts can use LLMs reliably for this task.
  • Social Science: Interdisciplinary evaluation drawing on developmental and comparative psychology can deepen understanding of LLM cognition while helping mitigate technological risks.The approach explores alternative methods for evaluating LLM capabilities.
  • Social Science: LLM outputs benefit social-science work by improving efficiency and providing resources that enhance productivity, but existing models cannot completely replace human professionals.The conclusion emphasizes both practical value and the continuing need for human expertise.

3.4 Natural Science and Engineering

LLM performance in natural science and engineering varies substantially by task and difficulty. Models show strengths in basic mathematics, chemistry tasks, coding, and software engineering, but remain unreliable on advanced mathematics, vulnerability detection, and commonsense planning.

  • Mathematics: LLMs handle addition, subtraction, decimals, negative numbers, and irrational numbers relatively well, but struggle with division, exponentiation, trigonometry, and logarithms.ChatGPT and GPT-4 significantly outperform other models on the reported fundamental mathematics evaluation.
  • Mathematics: ChatGPT’s accuracy declines with cognitive complexity, falling from 83% at recognition to 10% at the highest complexity level.The intermediate levels were 62% for comprehension and 27% for application.
  • Chemistry: Chemistry evaluations report accuracy ranging from 25% to 100%, while broader benchmarking covers 8 practical chemistry tasks.The benchmark evaluates GPT-4, GPT-3.5, and Davinci-003 across those tasks.
  • Engineering: CodeGen-16B reaches about a 78% match with ChatGPT, showing that smaller task-trained models can be competitive in code generation.ChatGPT nevertheless faces challenges with some fundamental programming-language concepts.
  • Engineering: ChatGPT often surpasses human and SOTA outputs in software engineering, but fails on vulnerability detection and retrieval-based test prioritization; LLMs also struggle with commonsense planning.Fine-tuned CodeT5 performs best across considered planning domains, with the shortest inference time, while plan generalization remains limited.

3.5 Medical Applications · 3.6 Agent Applications

The paper reviews LLM evaluation in medical queries, examinations, assistance, and education, while highlighting agent applications enabled by external tool use. Medical evaluations show both promising capabilities and important reliability, resource, privacy, and safety limitations.

  • 3.5 Medical Applications: Medical applications are organized into medical queries, medical examination, and medical assistants.This categorization structures the review of LLMs’ potential impact and advantages in healthcare.
  • 3.5.1 Medical queries: Medical queries dominate evaluations, with ChatGPT producing relatively accurate information across genetics, radiation oncology physics, biomedicine, and other disciplines.Accurate, reliable answers are important for healthcare professionals and patients seeking high-quality medical information.
  • 3.5.2 Medical examination: ChatGPT showed varying accuracy on USMLE Step 1 and Step 2 question datasets, with out-of-context information identified as a limitation.These studies assessed performance using novel multiple-choice question sets.
  • 3.5.2 Medical examination: ChatGPT answers were more context-aware and demonstrated better deductive reasoning than Google search results in one medical evaluation.The comparison was reported by Sharma et al.
  • 3.5.3 Medical assistants: Medical-assistance applications include gastrointestinal disease identification, dementia diagnosis, COVID-19 literature evaluation, and broader healthcare support.Challenges include lack of originality, high input requirements, resource constraints, uncertain answers, misdiagnosis risks, and patient-privacy concerns.
  • 3.5.3 Medical assistants: 46.8% overall accuracy for GPT-3.5 and 76.4% for GPT-4 were reported when evaluating surgical clinical understanding and potential contributions to medical education.The results demonstrate a significant performance difference between the two models.
  • 3.6 Agent Applications: External tools can greatly expand LLM capabilities, and ToolLLM provides a framework for equipping open-source models with tool-use capabilities.The section also cites KOSMOS-1 as capable of understanding general patterns, following instructions, and learning from context.

3.7 Other Applications

LLM evaluations in other applications cover education, search and recommendation, personality testing, and diverse specific tasks. Results show promising capabilities alongside unresolved reliability, risk, and refinement needs.

  • Education: In education, ChatGPT generates detailed, fluent, coherent feedback, accurately assesses assignments, and supports student skill development.Its feedback was reported to surpass that of human teachers.
  • Education: 71.8% correctness made ChatGPT comparable to participating students’ average exam score, while GPT-4 subsequently achieved a score of 8.33.Educational evaluations examine automatic scoring, question generation, and learning guidance.
  • Search and recommendation: Guided ChatGPT and GPT-4 achieved competitive information-retrieval benchmark performance, outperforming supervised methods in some evaluations.A specialized model trained on 10K ChatGPT-generated examples outperformed one trained on 400K annotated MS MARCO examples in BEIR.
  • Search and recommendation: LLMs can improve recommendation accuracy and personalization by modeling user preferences, item descriptions, and contextual information, but associated risks require attention.The passage frames LLMs as increasingly important components of recommendation pipelines.
  • Personality testing: Personality evaluations found inconsistent answers, prosocial characteristics, and unreliable self-assessment, leaving uncertainty about whether responses reflect self-reflection or algorithms.The findings concern Davinci-003 and multiple language models.
  • Specific applications: Evaluations across humor, game design, model assessment, and log parsing reveal both practical potential and limitations, including difficulty capturing humor’s contextual subtleties.The studies motivate more sophisticated models and continued research.

4 WHERE TO EVALUATE: DATASETS AND BENCHMARKS

The section organizes LLM evaluation around benchmarks, compiling 46 popular benchmarks across general language tasks, specific downstream tasks, and multimodal tasks. It also highlights benchmarks for holistic assessment, robustness, specialized domains, tool use, and multimodal capabilities, while noting that the collection may not be exhaustive.

  • Benchmark categories: The survey compiles 46 popular benchmarks, grouped into general language tasks, specific downstream tasks, and multimodal tasks.The authors note that benchmarks differ in their evaluation aspects and criteria.
  • General and holistic evaluation: HELM holistically evaluates language understanding, generation, coherence, context sensitivity, common-sense reasoning, and domain-specific knowledge.This contrasts with evaluations focused on specific tasks and metrics.
  • Robustness evaluation: GLUE-X evaluates NLP robustness in out-of-distribution scenarios, while BOSS assesses out-of-distribution robustness through a benchmark collection.These benchmarks address performance beyond standard task metrics, including OOD, adversarial robustness, and fine-tuning.
  • Downstream and domain-specific tasks: MultiMedQA comprises seven medical QA datasets for evaluating LLMs’ clinical knowledge and question-answering abilities.It covers medical examinations, medical research, and consumer healthcare questions, including six existing datasets and one new dataset.
  • Downstream and domain-specific tasks: CMMLU evaluates 18 LLMs across academic disciplines, with most showing suboptimal performance in Chinese language environments.C-Eval is described as the first extensive benchmark for advanced knowledge and reasoning in Chinese foundation models.
  • Tool-augmented evaluation: API-Bank is the first benchmark explicitly designed for tool-augmented LLMs, comprising 53 API tools and 264 annotated dialogues.The associated workflow includes 568 API calls; ToolBench and MINT further target tool use and multi-turn interactions.

5 HOW TO EVALUATE

LLM evaluation is organized into automatic and human evaluation according to whether criteria can be computed automatically. Automatic methods improve efficiency and standardization, while human evaluation is needed for non-standard outputs and depends on careful evaluator design.

  • Evaluation methods: Evaluation methods divide into automatic evaluation when criteria are computable and human evaluation otherwise.This categorization is based on whether the evaluation criterion can be automatically calculated.
  • Automatic evaluation: Automatic evaluation uses standard metrics and tools, reducing human effort, subjectivity, and evaluation time while improving standardization.It is widely used in benchmarks and can support self-supervised evaluation without laborious labeling of new data.
  • Automatic evaluation: Automatic metrics assess accuracy, calibration, fairness, and robustness across task correctness, confidence agreement, group consistency, and resistance to challenging inputs.Examples include Exact Match, F1, ROUGE, ECE, DPD, EOD, ASR, and PDR.
  • Human evaluation: Human evaluation is more reliable for non-standard or open-generation tasks where automatic similarity metrics such as BERTScore are insufficient.Human participation can provide feedback closer to actual application scenarios and more comprehensive assessment.
  • Human evaluation: Reliable human evaluation requires appropriate evaluator numbers, criteria, and expertise to support representation, statistical significance, rigor, and valid domain-specific judgments.The assessment criteria include accuracy, relevance, fluency, transparency, safety, and human alignment.

6 SUMMARY

The review finds no universally best evaluation protocol, benchmark, or model, while documenting task-specific strengths and weaknesses and broader shifts toward more practical, dynamic, human-centered evaluation.

  • Key findings: No single evaluation protocol, benchmark, or model is demonstrably best across all tasks; their usefulness depends on differing characteristics and focuses.The survey therefore goes beyond identifying a single “best” benchmark or evaluation protocol.
  • Success and failure cases: LLMs are proficient at fluent text generation, language understanding, arithmetic reasoning, logical reasoning, and temporal reasoning.Reported strengths include sentiment analysis, text classification, factual-input handling, arithmetic reasoning, and logical reasoning.
  • Success and failure cases: LLMs struggle with natural-language inference, semantic similarity, fundamental-phrase evaluation, abstract reasoning, and complex contexts.They also face challenges representing human disagreements and handling linguistic contexts involving non-Latin scripts and limited resources.
  • Success and failure cases: LLMs may produce fabricated or erroneous facts, lack real-time information capabilities, and remain sensitive to adversarial prompts.These limitations affect dialogue credibility, up-to-date knowledge tasks, rapid adaptation, and robustness.
  • Evaluation trends: Evaluation is shifting from objective calculation toward human-in-the-loop testing, enabling users to identify and repair coherent failure modes.AdaVision supports user labeling for correctness, while AdaTest filters high-quality tests and organizes them into semantically related topics.
  • Evaluation trends: Evaluation is also moving from static, unified test settings toward crowdsourced, dynamic, and task-specific challenging tests.DynaBench, DynaBoard, and DynaTask use crowdworkers to create hard samples; DynamicTempLAMA constructs time-related tests, while DeepTest, CheckList, and AdaFilter generate targeted tests.

7 GRAND CHALLENGES AND OPPORTUNITIES FOR FUTURE RESEARCH

The paper identifies evaluation as an essential discipline because existing protocols cannot fully assess LLM capabilities. Future research should develop broader AGI, robustness, dynamic, trustworthy, task-general, and analytically useful evaluation systems.

  • AGI evaluation: AGI benchmarks must clarify which tasks measure AGI capabilities and address unresolved questions about human values and alternative perspectives.The paper emphasizes understanding differences between human and AGI capacities when constructing benchmarks.
  • AGI evaluation: Ideal AGI evaluation should combine standard benchmarks with open-environment behavioral tests, including real-world robot and multimodal evaluations.LLMs could serve as central controllers in robots to test behavior in real situations.
  • Robustness: LLMs require stronger robustness evaluation because equivalent prompts with different grammars or expressions can produce diverse results.The paper notes that current LLMs are not robust to varied inputs and that substantial room remains for advancement.
  • Dynamic evaluation: Static public benchmarks cannot consistently assess rapidly evolving LLM abilities, creating a need for evaluation protocols that adapt over time.Public datasets and protocols facilitate convenient community evaluation but may fail to capture changing capabilities.
  • Trustworthy evaluation: Trustworthy evaluation systems must address integrity, measurement theory, probability, and whether dynamic testing genuinely produces out-of-distribution examples.The paper highlights a scarcity of research on ensuring the trustworthiness of dynamic testing.
  • General-purpose evaluation: Evaluation systems should support tasks including value alignment, safety, verification, interdisciplinary research, and fine-tuning, while providing analysis and guidance beyond benchmark results.PandaLM illustrates open-source automated assessment for fine-tuning, and PromptBench provides robustness results on adversarial prompts.

8 CONCLUSION

The paper presents a comprehensive survey of LLM evaluation organized around what, how, and where to evaluate, while identifying persistent model limitations and challenges for future evaluation systems.

  • 8 CONCLUSION: The survey comprehensively reviews LLM evaluation tasks, protocols, and benchmarks across what, how, and where to evaluate.It aims to clarify the current status, strengths, and limitations of LLMs.
  • 8 CONCLUSION: Current LLMs show limitations across numerous tasks, notably reasoning and robustness.The conclusion also emphasizes that evaluation systems must adapt to assess models’ inherent capabilities and limitations accurately.
  • 8 CONCLUSION: Future research should address several grand challenges so LLMs can progressively improve their service to humanity.The paper frames evaluation as essential for advancing AI models, especially LLMs.

DISCLAIMER

This survey summarizes existing LLM evaluation efforts and discusses potential side effects to foster better understanding, rather than presenting all reported results as its own contributions. Because LLMs and their evaluation are continuously evolving, readers should verify current systems and recognize that the survey may omit newer limitations, papers, or benchmarks.

  • Scope and intent: The survey primarily summarizes existing LLM evaluation efforts; reported results and ethical or bias-related conclusions belong to the original studies.Its discussion of LLM side effects is intended to foster better understanding.
  • Scope and intent: Readers should use the survey as a reference while conducting experiments on current systems, because evolving services may mitigate described limitations or introduce new ones.The passage specifically identifies online services such as Claude and ChatGPT as evolving systems.
  • Scope and intent: The survey may miss new papers or benchmarks because LLM evaluation is continuously developing.The authors welcome constructive feedback and suggestions.
Loading 2307.03109v9…