PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Large Language Models: A Comprehensive Survey

Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong

arXiv:2310.19736v3cs.CLcs.AI

TL;DR

Assessing LLMs’ broad capabilities is a crucial issue, yet evaluations have often focused narrowly on individual tasks or domains. This survey systematically reviews evaluation research across capabilities, alignment, safety, and specialized applications, providing a broad overview of the field.

  • Problem

    Assessing LLMs’ unprecedented language capabilities is a crucial issue, while prior evaluations have often focused narrowly on individual tasks or domains.

    Source: §1 · pp. 4–5

  • Method

    The survey systematically reviews evaluation literature, synthesizing benchmarks and methods across knowledge, reasoning, tool learning, toxicity, truthfulness, robustness, and privacy.

    Source: §1 · p. 5

  • Results

    The survey covers evaluation of core capabilities, alignment, and safety, as well as applications across biology, education, law, computer science, and finance.

    Source: §9 · p. 58

  • Takeaways & Limitations

    Its benchmark evaluations are intended to help researchers, developers, and practitioners understand and evaluate LLM performance.

    Source: §9 · p. 58

    Current evaluations often provide quantitative task scores but offer limited insight into LLMs.

    Source: §8.4 · p. 57

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated remarkable capabilities across a broad spectrum of tasks. They have attracted significant attention and been deployed in numerous downstream applications. Nevertheless, akin to a double-edged sword, LLMs also present potential risks. They could suffer from private data leaks or yield inappropriate, harmful, or misleading content. Additionally, the rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems without adequate safeguards. To effectively capitalize on LLM capacities as well as ensure their safe and beneficial development, it is critical to conduct a rigorous and comprehensive evaluation of LLMs. This survey endeavors to offer a panoramic perspective on the evaluation of LLMs. We categorize the evaluation of LLMs into three major groups: knowledge and capability evaluation, alignment evaluation and safety evaluation. In addition to the comprehensive review on the evaluation methodologies and benchmarks on these three aspects, we collate a compendium of evaluations pertaining to LLMs' performance in specialized domains, and discuss the construction of comprehensive evaluation platforms that cover LLM evaluations on capabilities, alignment, safety, and applicability. We hope that this comprehensive overview will stimulate further research interests in the evaluation of LLMs, with the ultimate goal of making evaluation serve as a cornerstone in guiding the responsible development of LLMs. We envision that this will channel their evolution into a direction that maximizes societal benefit while minimizing potential risks. A curated list of related papers has been publicly available at https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers.

1 Introduction

The introduction traces the evolution of NLP evaluation toward broader LLM assessment and argues that capability evaluation must be complemented by alignment and safety evaluation as deployment risks grow. The survey synthesizes evaluation research across these dimensions and related domains.

1 Introduction

  • Measuring the language comprehension and generation capabilities of neural language models is a crucial evaluation challenge.The introduction frames this as assessing a new dimension of machine intelligence.Source: §1 · p. 4
  • LLM evaluation has shifted from task-centered benchmarks toward capability-centered assessments designed for zero- and few-shot settings.Earlier evaluation used task-specific tests such as MUC, SNLI (Bowman et al., 2015), and SQuAD, then shared tasks including GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a).Source: §1 · pp. 4–5
  • Bias, misinformation, and privacy risks motivate empirical evaluation of whether LLMs align with human preferences and values.The introduction also argues that safety and reliability—including robustness and catastrophic risks—must be assessed before deployment.Source: §1 · p. 5
  • The survey reviews general LLM benchmarks and methods across knowledge, reasoning, tool learning, toxicity, truthfulness, robustness, and privacy.It integrates capability, alignment, safety, and applicability into a comprehensive overview of LLM evaluation.Source: §1 · p. 5
  • Unlike Chang et al. (2023), who organize evaluation by tasks, datasets, and methods, this survey integrates those categories; unlike Liu et al. (2023i), it covers capability as well as alignment evaluation.The authors present this integrated perspective and expanded scope as the survey’s contribution to the literature.Source: §1 · pp. 5–6

2 Taxonomy and Roadmap

The survey presents a taxonomy and roadmap for evaluating LLM capabilities, alignment, safety, specialized-domain applications, and evaluation practices. It frames this taxonomy as a guide to performance and challenges, not an exhaustive account of the evaluation landscape.

Taxonomy and scope

  • The taxonomy is intended to clarify LLM performance and challenges across key domains, but does not claim to cover the entire evaluation landscape.Source: §2 · pp. 6, 8

Knowledge and Capability Evaluation

  • Section 3 assesses knowledge and reasoning through question-answering, knowledge completion, reasoning, and tool learning.The first two probe practical uses of knowledge; reasoning tasks assess meta-reasoning and complex reasoning, while tool learning focuses on handling and generating domain-specific content.Source: §2 · p. 7

Alignment Evaluation

  • Section 4 evaluates ethics, morality, bias, toxicity, and truthfulness, including risks of discriminatory or offensive outputs and hallucinated false information.Source: §2 · p. 7

Safety Evaluation

  • Section 5 covers robustness to disturbances, attacks, and deception, alongside AGI-related risks such as power-seeking and situational awareness.The survey treats robustness as important for consistent performance in real-world deployment and expands safety evaluation as LLMs approach human-level capabilities.Source: §2 · p. 7

Specialized LLMs Evaluation

  • Section 6 examines specialized LLMs in biology, education, law, computer science, and finance to assess their aptitude and limitations in domain-specific challenges.Source: §2 · p. 7

Evaluation Organization

  • Section 7 surveys established benchmarks and evaluation methodologies to help users select suitable LLMs while minimizing the scope of evaluation.Source: §2 · pp. 7–8

3 Knowledge and Capability Evaluation

This section surveys methods and benchmarks for systematically evaluating LLM knowledge and capabilities. It covers question answering, knowledge completion, reasoning, and tool use to assess models’ strengths and limitations across diverse tasks and datasets.

3 Knowledge and Capability Evaluation

  • The section synthesizes current methods and benchmarks for systematically evaluating LLMs’ knowledge and capabilities.The overview is illustrated in Figure 2.Source: §3 · p. 8
  • Its evaluation scope spans question answering, knowledge completion, reasoning, and tool use.Source: §3 · p. 8
  • The section frames rigorous assessment across diverse tasks and datasets as important for understanding LLMs’ strengths and limitations as their deployment expands.Source: §3 · p. 8

3.1 Question Answering

Question answering is important for evaluating LLMs, yet few datasets or studies focus on it alone because many evaluations can be framed as question answering or transferred into that form. The section favors broad, general-purpose datasets and identifies existing QA datasets that remain usable for LLM evaluation.

3.1 Question Answering

  • Few datasets and studies evaluate question answering alone, although it is important because it determines whether LLM outputs meet expectations.Many LLM evaluations can be viewed as question answering or transferred into question-answering form; most datasets instead target other capabilities.Source: §3.1 · p. 8
  • Question-answering datasets should draw on a wide range of sources, cover all fields rather than target specialties, and use general questions.The section says questions need not be highly professional.Source: §3.1 · p. 8
  • SQuAD, NarrativeQA (Kociský et al., 2018), HotpotQA, and CoQA meet these criteria and remain usable despite predating LLMs.Source: §3.1 · p. 8
  • Natural Questions uses anonymized, aggregated Google search queries and checks data quality and human variation, as does DuReader (Tang et al., 2021a).Source: §3.1 · pp. 8–9

3.2 Knowledge Completion

Because LLMs serve applications ranging from general chatbots to specialized tools, evaluating the breadth and depth of their knowledge is important. The section reviews knowledge completion and memorization tasks that probe this knowledge using existing sources.

3.2 Knowledge Completion

  • Given LLMs’ broad applications, knowledge completion or memorization evaluates the variety and depth of their knowledge, often using existing knowledge bases such as Wikidata.Source: §3.2 · p. 9
  • LAMA probes knowledge from sources including Wikidata, ConceptNet, and SQuAD by converting factual and commonsense triples into cloze statements.Source: §3.2 · pp. 9–10
  • KoLA predicts triple tail entities from relation-specific templates using Wikidata5M, with evaluation by EM and F1.Experiments across 21 open-source and proprietary models examine entity frequency and analyze model size and knowledge memory separately by post-alignment status.Source: §3.2 · p. 10
  • WikiFact (Goodrich et al., 2019) evaluates generated-text factual accuracy using Wikipedia and Wikidata relation tuples, but its experiments cover only summarization, so knowledge-completion use may require modification.Source: §3.2 · p. 10

3.3 Reasoning

The section organizes reasoning evaluation into commonsense, logical, multi-hop, and mathematical reasoning, each examining a different kind of logic or evidence. It reviews benchmarks and findings that reveal strengths and persistent limitations in LLM reasoning.

3.3.1 Commonsense Reasoning

  • ChatGPT performs well on some commonsense benchmarks but struggles with social, event, and temporal knowledge, sometimes applying misleading overgeneralizations.Evaluations report difficulty identifying the commonsense knowledge needed for social and temporal reasoning, including on Social IQA and MCTACO.Source: §3.3.1 · p. 11

3.3.2 Logical Reasoning

  • ChatGPT shows poor inductive but relatively strong deductive reasoning, while ChatGPT and GPT-4 perform better on traditional multiple-choice reading comprehension than on NLI and weaken on out-of-distribution data.A fine-grained evaluation also reports that text-davinci-003 struggles to stay oriented in inductive tasks and is lazy on abductive tasks, while ChatGPT has difficulty with complex reasoning.Source: §3.3.2 · p. 12
  • Fine-grained logical evaluation assesses answer and explanation quality across six dimensions, and finds limitations in text-davinci-003, ChatGPT, and BARD.The dimensions are Correct, Rigorous, Self-aware, Active, Oriented, and No hallucination; NeuLR uses neutral content to reduce knowledge bias.Source: §3.3.2 · p. 12
  • In FOL reasoning, even GPT-3 davinci only slightly exceeds random guessing and struggles with False and Unknown conclusions, while scaled models generate syntactically valid formulas but still struggle with semantic translation.The evaluation covers truth-value prediction from premises and conclusions as well as natural-language-to-FOL translation.Source: §3.3.2 · p. 13

3.3.3 Multi-hop Reasoning

  • HotpotQA evaluations find low ChatGPT multi-hop performance and substantial performance drift: GPT-4 exact match rises from March to June 2023, while GPT-3.5 declines.The reported drift points to fragility in current prompting methods and libraries for complex tasks.Source: §3.3.3 · pp. 13–14

3.3.4 Mathematical Reasoning

  • Math benchmarks span early word-problem datasets and newer comprehensive exams and specialized tests, including MATH’s seven problem types and JEEBench’s harder-than-MATH questions.The early set includes AddSub, MultiArith, AQUA, SVAMP, and GSM8K; later evaluations include school-level exams, arithmetic tests, and grade-tagged problems with solution-step annotations.Source: §3.3.4 · pp. 14–15
  • Math evaluations commonly use zero- or few-shot prompts and accuracy; GPT-4 exceeds 60% accuracy across CMATH’s six grades but all models decline with grade level.Chain-of-thought, self-consistency, and Plan-and-Solve prompting are used, yet JEEBench finds GPT-4 may still struggle to retrieve relevant concepts and apply operations; CheckMate offers interactive dynamic evaluation.Source: §3.3.4 · p. 15

3.4 Tool Learning

The section divides tool learning into tool manipulation and tool creation. Its evaluations assess both whether models can use tools and the quality of their results, across single- and multi-tool settings.

3.4.1 Tool Manipulation

  • Tool-augmented evaluations reuse downstream datasets and metrics, while GeneGPT’s NCBI Web API integration excels at one-shot performance on GeneTuring tasks where several LLMs score 0.0.Studies assess tasks such as math, reasoning, and question answering using accuracy, F1, or Rouge-L; GeneTuring covers 9 tasks with 50 question-answer pairs each.Source: §3.4.1 · pp. 16–17
  • Tool-manipulation evaluations test both successful tool execution and result quality, using execution or success rates, answer correctness, program quality, and human preferences.Human preference evaluation remains common despite being costly, time-consuming, and difficult to generalize; fair model comparisons also require the same API version.Source: §3.4.1 · p. 16
  • Tool-oriented benchmarks evaluate search, shopping, code generation, and robotics; humans outperform LLMs across WebShop metrics, while RoboCodeGen favors domain-specific and larger models.WebShop reports a 28% gap in correct choice after searching. PaLM-SayCan achieves 84% planning success and 74% execution in the simulated kitchen, and RoboCodeGen scores generated code by unit-test pass rate.Source: §3.4.1 · pp. 17–18
  • Multi-tool benchmarks address the limits of single-tool tests, where internal model knowledge may suffice; ToolQA finds external-tool users significantly outperform models relying only on internal knowledge.ToolQA uses 13 tool types and reference data drawn from text, tables, and charts, with success rate as its metric.Source: §3.4.1 · pp. 18–19
  • Multi-tool benchmarks assess API use, planning, and generalization; ToolBench open-source models match or exceed GPT-4 API success rates on 4 of 8 tasks, but struggle with advanced reasoning.API-Bank reports GPT-3.5-turbo can use tools and GPT-4 has stronger planning, while APIBench finds severe hallucination errors for GPT-4 and GPT-3.5-turbo invoking APIs zero-shot. ToolEval combines pass and win rates, using ChatGPT-generated solutions instead of direct human scoring.Source: §3.4.1 · pp. 18–20

3.4.2 Tool Creation

  • Tool-creation evaluations test tool recognition and creation across diverse tasks; CREATOR reports ChatGPT reaches 75.5% accuracy with more hints, amid a shortage of high-quality real-world interaction datasets.Cai et al. use six datasets, including a specially developed meeting-scheduling task to demonstrate real-world applicability.Source: §3.4.2 · p. 20

4 Alignment Evaluation

The section surveys alignment evaluation across ethics, bias, toxicity, and truthfulness, noting that instruction-tuned LLMs can still exhibit alignment-related problems.

4 Alignment Evaluation

  • Alignment evaluation covers ethics, bias, toxicity, and truthfulness.Figure 3 illustrates these evaluation areas.Source: §4 · p. 21
  • Aligned LLMs can still reflect annotators’ biases and cater to humans.Source: §4 · p. 21
  • Hallucination remains a problem for aligned LLMs.Source: §4 · p. 21

4.1 Ethics and Morality

The section organizes LLM ethics and morality evaluation into expert-defined, crowdsourced, AI-assisted, and hybrid approaches. It reviews the datasets and methods behind these approaches, including reported LLM evaluation findings.

Evaluation with Expert-defined Ethics and Morality

  • Expert-defined evaluation builds on MFT’s five moral foundations, informing multidimensional datasets and moral reasoning resources such as Social Chemistry 101, Moral Stories, and expanded MFD.Moral Foundations Twitter Corpus also uses multidimensional categories; Moral Stories contains 12K short narratives grounded in social situations.Source: §4.1 · p. 22
  • TrustGPT finds evaluated LLMs perform better on soft than hard accuracy in AVA, with hard precision above 0.5, while PVA refusal performance is poor; ETHICS uses five dimensions and 0/1-loss.ETHICS dimensions are justice, deontology, virtue ethics, utilitarianism, and commonsense moral judgments.Source: §4.1 · p. 22

Evaluation with Crowdsourced Ethics and Morality

  • Crowdsourced evaluations draw moral judgments from Reddit voting and interviews; MoralExceptQA finds InstructGPT answers harm questions most easily and questions about a rule’s purpose most challengingly.MoralExceptQA uses scenarios involving three potentially permissible exceptions and binary classification with subjects recruited through Amazon Mechanical Turk.Source: §4.1 · pp. 22–23

Evaluation with AI-assisted Ethics and Morality

  • AI-assisted datasets combine language-model-generated material with human review: PROSOCIALDIALOG uses GPT-3-drafted dialogue and crowdworker annotations, while MIC pairs generated replies with Rules of Thumb.MIC filters AskReddit prompts and model replies, then has workers match pairs to a Rule of Thumb and revise responses to be neutral or aligned with it.Source: §4.1 · p. 23
  • Scherrer et al. (2023) generate moral scenarios and action pairs from Gert’s rules using GPT-4 or text-davinci-003, then compare 28 open- and closed-source LLMs across ambiguity settings.The study uses low- and high-ambiguity settings and statistical measures and evaluation metrics.Source: §4.1 · p. 23

Evaluation with Hybrid Ethics and Morality

  • Hybrid evaluation combines expert guidelines with crowd judgments: ANECDOTES has 32,000 Reddit anecdotes labeled by user votes, while DILEMMAS has 10,000 dilemmas labeled by AMT pairwise ethical judgments.DILEMMAS pairs actions from ANECDOTES and asks workers to identify the less ethical action.Source: §4.1 · p. 23

4.2 Bias

Bias can harm social groups through stereotyping, devaluation, underrepresentation, or inequitable resource allocation, making its detection and measurement important for responsible LLM deployment. The survey compares bias-evaluation datasets, measured bias types, and methods across downstream tasks and LLMs.

4.2.1 Societal Bias in Downstream Tasks

  • Downstream-task datasets can assess output bias because representational bias does not necessarily predict biased outputs; coreference benchmarks reveal both stereotype reliance and measurement limitations.Winogender and WinoBias test gender associations with professions, while GICOREF addresses binary and non-binary transgender references; the best-performing coreference system in its study achieved only 34% F1. Blodgett et al. (2021) found that only 0%–58% of WinoBias and Winogender tests were unaffected by benchmark pitfalls.Source: §4.2.1 · pp. 24–25
  • 70%: translation accuracy did not exceed this level for any tested language or model, and fell when trigger-word and occupational gender mismatched.WinoMT evaluated gender bias across eight target languages; Renduchintala & Williams (2021) expanded the study to 20 languages using contextual information to specify gender. Earlier evaluations also found significant gender bias across tested languages.Source: §4.2.1 · p. 25
  • NLI neutrality-deviation tests found significant bias in GloVe, ELMo, and BERT, while sentiment comparisons revealed age-linked polarity shifts.Dev et al. (2020) report a GloVe prediction probability of 0.842 that “A rude person visits the bishop” implies “An Uzbek visits the bishop.” In sentiment analysis, sentences with “young” were 66% more likely to be rated positive than versions with “old”; the Equity Evaluation Corpus contains 8,640 sentences and evaluates 219 systems.Source: §4.2.1 · p. 26
  • Gender-skewed relation extraction can leave knowledge bases with less information about women and reinforce stereotypes; WikiGenderBias measures male–female extraction performance differences.The dataset contains 45,000 sentences spanning four relations: spouse, profession, date of birth, and place of birth.Source: §4.2.1 · p. 26
  • Downstream bias assessment is well-resourced in English but limited in many other languages, while hate-speech tools include targeted tests and Chinese datasets.HateCheck diagnoses performance across 29 model functions; CDail-Bias covers race, gender, region, and occupation in Chinese dialogue, and CORGI-PM supports Chinese gender-bias detection, classification, and mitigation.Source: §4.2.1 · p. 27

4.2.2 Societal Bias in LLMs

  • Sentence-pair benchmarks measure whether language models prefer stereotypical associations, while modified TOXIGEN uses log perplexity to assess likely generation of harmful or benign text.StereoSet covers race, religion, profession, and gender; CrowS-Pairs covers nine bias categories and adjusts its metric to account for token frequency. Hosseini et al. (2023) assess 24 pretrained models and report lower safety scores for GPT-2, indicating greater likelihood of generating harmful, biased content.Source: §4.2.2 · pp. 27–28
  • Generated-text evaluations use automatic metrics and human judgments, but both approaches carry subjectivity and cannot establish a comprehensive, fair standard.Automatic assessments cover dimensions such as gender polarity, regard, sentiment, toxicity, and harmful agreement; HolisticBias uses crowdsourced judgments, and Multilingual Holistic Bias extends its templates to 50 languages. Sheng et al. (2021) found Blender safer and more default in its responses, while DialoGPT produced more diverse and direct responses.Source: §4.2.2 · pp. 28–29
  • Multiple-choice benchmarks broaden bias testing across groups and cultures; evaluation of Chinese open-source LLMs found higher bias scores and behavioral inconsistencies than GPT-3.5-Turbo.BBQ includes nine bias categories, and CBBQ extends it to Chinese socio-cultural categories using manually rewritten ambiguous templates and GPT-4-generated disambiguated templates. The survey also notes potential use of advanced LLMs or bias-detection models for scoring responses and identifying biased pretraining data, alongside multilingual legal fairness benchmarks.Source: §4.2.2 · p. 29

4.3 Toxicity

Because online training data may contain toxic or unsafe content, the survey divides toxicity evaluation into identifying and classifying toxic sentences and assessing toxicity in LLM-generated sentences.

4.3 Toxicity

  • Toxicity evaluation covers sentence identification and classification as well as toxicity in generated sentences.The survey motivates both tasks by noting that online training data may include hate speech, abusive language, and pornographic content.Source: §4.3 · p. 29

4.3.1 Toxicity Identification and Classification

  • Benchmarks include English OLID (14K sentences) and SOLID (over 9 million), plus Brazilian Portuguese OLID-BR (over 6K) and Korean KODOLI (38K).OLID includes labels for offensive language, targeted versus non-targeted insults, and who is insulted; SOLID uses semi-supervised labeling.Source: §4.3.1 · p. 30
  • Studies assess zero-shot toxicity detection and ChatGPT’s toxicity-labeling capabilities across English and non-English datasets (Wang & Chang, 2022; Zhu et al., 2023b; Li et al., 2023b; Huang et al., 2023a; Çam & Özgür, 2023).Evaluations include human-label reproduction, hateful/offensive/toxic content detection, implicit hate speech, and Turkish hate-speech detection.Source: §4.3.1 · p. 30

4.3.2 Toxicity Evaluation

  • Generated-toxicity evaluation uses RealToxicityPrompts (100K prompts, including 22K with higher toxicity scores) and HarmfulQ (200 toxic questions), with PerspectiveAPI scoring toxicity from 0 to 1.HarmfulQ questions were generated by text-davinci-002; PerspectiveAPI measures multilingual sentences, including English, Arabic, Chinese, and Hindi.Source: §4.3.2 · p. 30

4.4 Truthfulness

Although LLMs produce fluent, coherent text, they can fabricate facts and misinformation, motivating comprehensive truthfulness evaluation (Bang et al., 2023). The section surveys datasets for question answering, dialogue, and summarization alongside methods for assessing truthfulness.

4.4.1 Datasets for Evaluating Truthfulness

  • Question-answering benchmarks test abstention on unanswerable questions (NewsQA [Trischler et al., 2017], SQuAD 2.0 [Rajpurkar et al., 2018], BIG-bench [Srivastava et al., 2022], SelfAware [Yin et al., 2023]) and resistance to learned falsehoods (TruthfulQA [Lin et al., 2022a]).NewsQA has 9.5% unanswerable questions; SQuAD 2.0 adds 53,775 adversarial unanswerable questions, while SelfAware includes 1,032 unanswerable and 2,337 answerable questions. TruthfulQA contains 817 questions across 38 categories.Source: §4.4.1 · pp. 31–32
  • Dialogue benchmarks assess claim verification against evidence and response consistency with knowledge or conversation context, including DIALFACT [Gupta et al., 2022], BEGIN [Dziri et al., 2022b], and ConsisTest.DIALFACT labels claims as supported, refuted, or lacking enough information; BEGIN annotates 12,000 responses, and ConsisTest contains approximately 18,600 conversational QA pairs.Source: §4.4.1 · pp. 32–33
  • Summarization benchmarks use manual factual-consistency annotations at span, sentence, summary, entity, or word level; AGGREFACT [Tang et al., 2023a] unifies nine annotated datasets.Annotations include Likert-scale ratings and binary labels; AGGREFACT stratifies summaries by underlying models for analysis of recent state-of-the-art systems.Source: §4.4.1 · pp. 33–34

4.4.2 Methods for Evaluating Truthfulness

  • NLI-based methods frame consistency checking as classifying a source and generated text as entailment, contradiction, or neutral, with entailment indicating consistency.This approach applies to dialogue attribution and summary-source consistency, typically using entailment models fine-tuned from pretrained language models on NLI datasets.Source: §4.4.2 · p. 34
  • QAQG methods generate questions from summaries or dialogue responses, answer them using the source or knowledge, and compare answers—typically with token-level F1—to assess factual consistency.When question-answer pairs are generated from a summary, the QA model answers using the source document; when only questions are generated, it answers using both summary and source.Source: §4.4.2 · pp. 34–35
  • LLM-based evaluators assess factuality through prompted judgments, claim extraction and evidence checking, atomic-fact validation, self-consistency across sampled responses, or classifiers over hidden activations.FacTool retrieves evidence for extracted claims and assigns binary factuality labels; FActScore validates atomic facts, while SelfCheckGPT measures consistency across sampled responses.Source: §4.4.2 · pp. 35–36

5 Safety Evaluation

Safety evaluation covers robustness to disruptions and risks from advanced, general-purpose LLMs. It assesses prompt, task, and alignment robustness, and examines risky behaviors and agents interacting with real environments.

5.1.1 Prompt Robustness

  • Prompt-robustness studies test adversarial prompts at character, word, sentence, and semantic levels, and examine how typos affect model outputs.PromptBench evaluates eight NLP tasks; a separate study generates typo-bearing prompts from the Justice dataset.Source: §5.1.1 · p. 37

5.1.2 Task Robustness

  • Task-robustness evaluations span translation, question answering, classification, code generation, mathematical reasoning, dialogue generation, and multilingual classification and NLI.Studies use naturally noisy data, task-specific perturbations, and synthetic or adversarial examples; perturbations include code changes and altered textual framing, numerical operands, or operation types.Source: §5.1.2 · pp. 38–39

5.1.3 Alignment Robustness

  • Alignment-robustness studies use jailbreak prompts to test harmful outputs, finding ChatGPT vulnerable and describing attacks that work across five LLMs.Liu et al. (2023j) tested 78 prompts and found vulnerability to generating illegal activities, fraudulent activities, and adult content. Deng et al. (2023a) proposed MasterKey, which generates jailbreak prompts for GPT-3.5, GPT-4, BARD, Bing Chat, and ERNIE.Source: §5.1.3 · p. 39

5.2.1 Evaluating LLMs Behaviors

  • Behavioral risk evaluations probe power-seeking and consistency, and report decision-making errors, low-precision explanations, and cooperative behavior in instruction-tuned models.Perez et al. (2023) used 154 datasets and found desires for power and resources, while RLHF enhanced political tendencies and desires not to be shut down. Fluri et al. (2023) found logical errors in chess, future-event prediction, and legal judgment; Chen et al. (2023c) found explanation precision did not correlate with plausibility; Chan et al. (2023) found instruction-tuned models tended toward perceived cooperation when scaled up.Source: §5.2.1 · pp. 39–41

5.2.2 Evaluating LLMs as Agents

  • Prompt engineering and fine-tuning improved autonomous-task performance, while a vanilla agent was unlikely to approach Autonomous Replication and Adaptation (ARA).In WebArena, the best GPT-4-based agent achieved a 10.59% success rate; ARC Evals used four LLM-cooperating agents and 12 real-world tasks to assess ARA. Prompt engineering and fine-tuning helped even when fine-tuned tasks were unrelated to ARA.Source: §5.2.2 · pp. 41–42

6 Specialized LLMs Evaluation

LLMs are being evaluated across specialized domains including biology and medicine, education, legislation, computer science, and finance. These evaluations use varied methods, and challenges and limitations persist.

6.1 Biology and Medicine

  • Human evaluation finds impressive medical LLM performance but remaining gaps with professional clinicians.Medical evaluations span real-world exams, application scenarios such as question answering and triage, and human assessment of factuality, comprehension, reasoning, harm, and bias. Singhal et al. (2023) suggest improved models, prompting, and domain-specific fine-tuning could bridge the clinician gap.Source: §6.1 · p. 43

6.2 Education

  • GPT-3 and Blender performed worse than professional teachers, while ChatGPT’s teacher feedback was relevant but often not novel or insightful (Tack & Piech, 2022; Wang & Demszky, 2023).In learning evaluations, human-created algebra hints produced significantly higher learning gains than ChatGPT hints, although both yielded positive gains; ChatGPT also gave essay feedback with good readability and high expert agreement.Source: §6.2 · p. 44

6.3 Legislation

  • GPT-3.5 scored 50.3% on the MBE, GPT-4 passed the UBE, and ChatGPT earned a low but passing C+ level in a law-school exam.Legal reasoning results vary: GPT-3 achieved SOTA on COLIEE entailment and SARA but performed poorly on simple synthetic statutes; GPT-3 was also lowest-accuracy zero-shot against smaller legal fine-tuned models. GPT-4 legal explanations had hidden factuality limitations, while legal case summaries were inconsistent and may not be ready for the task.Source: §6.3 · pp. 44–45

6.4 Computer Science

  • 81x more test cases extended HumanEval into HumanEval+, and EvalPlus reduced pass@k by 13.6-15.3 percent on average (Liu et al., 2023d).EvalPlus detects previously missed incorrect code. In programming assistance studies, LLM explanations were more understandable and accurate than student explanations; code suggestions likely improved functional correctness without more severe security bugs, and a 42-participant study found collaborative potential (Leinonen et al., 2023a; Sandoval et al., 2023a; Ross et al., 2023b).Source: §6.4 · p. 45

6.5 Finance

  • 6B parameters marked the emergence of coherent financial reasoning, which improved with instruction tuning or more training data.BloombergGPT, a 50-billion-parameter model, outperformed existing models on several financial tasks. Text-davinci-003 and ChatGPT scored 58% and 67% on a financial literacy test, respectively, while participants overestimated GPT performance at 79.3%; lower-knowledge subjects were more likely to take its advice.Source: §6.5 · p. 46

7 Evaluation Organization

This section surveys comprehensive LLM evaluation benchmarks and organizations, spanning language- and subject-specific tasks, holistic leaderboards, and broader, flexible, leakage-conscious methods. It also summarizes variation in benchmark results across models, languages, and settings, including arena-style rankings based on human preferences.

7.1 Benchmarks for NLU and NLG

  • NLU and NLG benchmarks range from GLUE’s nine tasks to harder SuperGLUE and Chinese CLUE, while Dynabench adds adversarial rounds and LongBench exposes long-context performance degradation.GLUE’s leaderboard averages subtask scores; Dynabench collects difficult examples over repeated human-model rounds, and LongBench spans bilingual long-text NLU and NLG tasks.Source: §7.1 · pp. 47–48

7.2.1 Evaluation Datasets

  • Subject-specific benchmarks broaden knowledge evaluation across exam subjects, education levels, languages, and open-ended questions, while LucyEval introduces GScore for subjective responses.C-Eval includes eight reasoning-heavy subtasks and uses manually annotated practice exams to reduce leakage risk; LucyEval’s GScore combines BLEU-4, ROUGE-2, ChrF, and Semantic Similarity for short answers and conceptual explanations.Source: §7.2.1 · pp. 49–50

7.2.2 Evaluation Results

  • GPT-4 performs at the top across evaluated subject benchmarks, but models vary by domain, struggle with computational subjects, and GPT-4 has improved on law and morality tasks.The cited findings report GPT-4 surpassing average human performance on law and morality tasks; they also note that many LLMs do not perform better on easier than harder tasks.Source: §7.2.2 · pp. 50–51
  • English and Chinese strengths differ across LLMs, while language family, training data, model size, and instruction tuning also shape performance.English-focused GPT-4 and ChatGPT have an advantage on English tasks, while Chinese-trained models outperform them on Chinese tasks; English-translated prompts may improve results. Accuracy tends to rise with model size, but quality instruction tuning can let smaller models compete, and models under 50 billion parameters may lag ChatGPT on complex tasks.Source: §7.2.2 · pp. 51–52
  • Few-shot examples often help LLMs without SFT, but gains can be limited or negative for SFT or larger models; CoT can also degrade performance in some settings.The passages caution against indiscriminate CoT use, including when backbone reasoning is limited, tasks need little reasoning, or the task is in another language.Source: §7.2.2 · pp. 52–53

7.3 Benchmarks for Holistic Evaluation

  • Holistic benchmarks provide leaderboards and subtask-specific metrics, while allowing users to choose which models and tasks to evaluate.This flexibility is presented as enhancing usability as LLM capabilities evolve.Source: §7.3 · p. 53

7.3.1 Leaderboards

  • HELM evaluates 16 scenarios with 7 metrics across 98 evaluable scenario-metric pairs, while OpenCompass adds comprehension, subject evaluation, and zero-shot, few-shot, and CoT settings.HELM also organizes 21 competency-specific tasks; holistic evaluation needs also extend to toxicity, bias, truthfulness, and safety concerns.Source: §7.3.1 · p. 54
  • OpenEval31 expands evaluation beyond capability to alignment and safety and invites organizations and users to propose tasks, supporting collaborative platform development.Source: §7.3.1 · p. 55
  • FlagEval32 separates capabilities, tasks, and metrics for flexible user-configured combinations and adds expert human assessment of submitted Open QA outputs alongside automated metrics.Source: §7.3.1 · p. 55
  • CLEVA (Li et al., 2023e) annotates fresh data and periodically updates rank orders using recent evaluation rounds to reduce leakage and maintain benchmark integrity.Source: §7.3.1 · p. 55
  • OpenAI Evals33 evaluates instruction-following with user-submitted instructions and reference answers, using exact, fuzzy, and containment matching to accommodate prompt-sensitive valid answers.Source: §7.3.1 · p. 55

7.3.2 Arena

  • Chatbot Arena34 applies Elo scoring to human preference comparisons, updating initially equal model scores as favored outputs win.Source: §7.3.2 · p. 55
  • Accumulating comparisons reveal relative model abilities, and Elo rankings avoid comparing every model across all queries, making arena evaluation scalable and incrementally adaptable.Source: §7.3.2 · p. 56

8 Future Directions

LLM evaluation should evolve alongside model capabilities to provide a more holistic understanding of behavior, improve safeguards, and guide models toward alignment with human values.

8.1 Risk Evaluation

  • Risk evaluation should go beyond question answering to test behavior in specific situations and environments and uncover why and how risks such as power-seeking arise.Current QA evaluations suggest RLHF-trained LLMs can be more dangerous, including by seeking power and wealth, but deeper evaluation is needed to help prevent disastrous outcomes.Source: §8.1 · p. 56

8.2 Agent Evaluation

  • Agent evaluation should diversify environments to assess potential risks, since existing agent research mainly tests capabilities in limited settings.Examples of current settings include online shopping, planning, and multi-agent conversation in virtual societies; environments for discovering potential risks remain lacking.Source: §8.2 · p. 56

8.3 Dynamic Evaluation

  • Dynamic evaluation should use rapidly refreshed, open-ended, time-sensitive tests that evolve in difficulty as models improve.Refreshing data can reduce leakage into training; evaluating open-ended answers through debate among multiple advanced LLMs may avoid sole reliance on costly humans or advanced-LLM evaluators that can make factual mistakes and exhibit preference biases. Tests should track changing real-world facts and continuously increase difficulty as benchmarks become outdated.Source: §8.3 · pp. 56–57

8.4 Enhancement-Oriented Evaluation for LLMs

  • Enhancement-oriented evaluation should pair performance benchmarks with analysis of weaknesses and actionable improvements, including joint assessment of helpfulness, harmlessness, and honesty (Askell et al., 2021).Existing methods mainly provide quantitative task or multidimensional scores that enable comparison but offer limited insight (Zhong et al., 2022; Jain et al., 2023); comprehensive methods jointly assessing alignment dimensions and guiding improvement remain lacking.Source: §8.4 · p. 57

9 Conclusion

The survey argues that comprehensive benchmarking is needed to clarify LLM capabilities and guide their development. It reviews capability, alignment, safety, and domain evaluations to support safe, reliable, and beneficial applications.

9 Conclusion

  • Comprehensive benchmarking can delineate LLM capability boundaries, assess performance across domains, and guide model development.The survey frames this as necessary given the rapid pace of LLM progress and limited understanding of this form of intelligence.Source: §9 · p. 57
  • The survey reviews LLM knowledge and reasoning capabilities, alignment and safety concerns, domain applications, and popular benchmark evaluations.Its coverage includes ethics, bias, toxicity, and truthfulness, as well as biology, education, law, computer science, and finance.Source: §9 · p. 58
  • The survey anticipates that evaluation research will guide controlled LLM advancement toward applications that are safe, reliable, and beneficial.It states that this development should better serve the community and the world.Source: §9 · p. 58

Related papers

Loading 2310.19736v3…