PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Large Language Models: A Comprehensive Survey

Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong

arXiv:2310.19736v3cs.CLcs.AI

TL;DR

LLMs’ broad capabilities and expanding deployment create a need for rigorous evaluation that also addresses harmful, misleading, privacy-related, and catastrophic risks. This survey systematically reviews evaluation across capabilities, alignment, safety, specialized domains, and platforms, concluding that it offers a comprehensive resource for understanding and evaluating LLM performance while identifying important gaps in current assessment.

  • Problem

    LLMs have broad capabilities and deployment contexts, but require rigorous evaluation to assess performance and address privacy, harmful-content, misleading-output, and catastrophic-risk concerns.

  • Method

    The survey synthesizes benchmarks and methodologies across knowledge and capability, alignment, safety, specialized domains, and evaluation organization.

  • Results

    The survey provides a comprehensive overview of LLM evaluation research spanning capabilities, alignment, safety, applications, and popular benchmarks.

  • Takeaways & Limitations

    The compiled benchmarks and evaluations support researchers, developers, and practitioners in understanding and evaluating LLM performance.

  • Takeaways & Limitations

    Current evaluations provide limited insight beyond quantitative task scores and lack comprehensive, actionable methods jointly assessing helpfulness, harmlessness, and honesty.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated remarkable capabilities across a broad spectrum of tasks. They have attracted significant attention and been deployed in numerous downstream applications. Nevertheless, akin to a double-edged sword, LLMs also present potential risks. They could suffer from private data leaks or yield inappropriate, harmful, or misleading content. Additionally, the rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems without adequate safeguards. To effectively capitalize on LLM capacities as well as ensure their safe and beneficial development, it is critical to conduct a rigorous and comprehensive evaluation of LLMs. This survey endeavors to offer a panoramic perspective on the evaluation of LLMs. We categorize the evaluation of LLMs into three major groups: knowledge and capability evaluation, alignment evaluation and safety evaluation. In addition to the comprehensive review on the evaluation methodologies and benchmarks on these three aspects, we collate a compendium of evaluations pertaining to LLMs' performance in specialized domains, and discuss the construction of comprehensive evaluation platforms that cover LLM evaluations on capabilities, alignment, safety, and applicability. We hope that this comprehensive overview will stimulate further research interests in the evaluation of LLMs, with the ultimate goal of making evaluation serve as a cornerstone in guiding the responsible development of LLMs. We envision that this will channel their evolution into a direction that maximizes societal benefit while minimizing potential risks. A curated list of related papers has been publicly available at https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers.

1 Introduction

As language models evolved into large, general-purpose systems, evaluating their expanding intelligence became increasingly important. This survey responds with a broader review of LLM evaluation methods, benchmarks, and dimensions than earlier task- or alignment-focused surveys.

  • Evaluation Landscape: Language models progressed from grammar and vocabulary tests toward shared tasks and benchmarks for increasingly general pretrained models.Examples include MUC, SemEval, CoNLL, GLUE, SuperGLUE, and XNLI.
  • Evaluation Landscape: LLMs demonstrated strong zero- and few-shot performance, prompting a shift away from traditional task-centered evaluation.
  • Motivation: The rapid deployment of capable LLMs heightened concerns about risks requiring thorough and comprehensive evaluation.
  • Research Gap: Existing benchmarks often targeted singular tasks or domains, limiting comprehensive assessment across knowledge, reasoning, tool learning, toxicity, truthfulness, robustness, and privacy.
  • Contribution: The survey synthesizes LLM evaluation across capability and alignment perspectives, integrating categories, datasets, and methods more holistically than related reviews.

2 Taxonomy and Roadmap

The survey organizes LLM evaluation into a taxonomy spanning capabilities, concerns, applications, and performance. Its roadmap covers knowledge and capability, alignment, safety, specialized-domain, and organizational evaluation.

  • Taxonomy: The taxonomy is designed to provide a structured view of LLM performance and challenges across diverse domains.
  • Taxonomy: The framework examines knowledge and reasoning capabilities, alignment concerns, safety risks, specialized applications, and evaluation organization.
  • Knowledge and Capability Evaluation: Knowledge and capability evaluation covers question answering, knowledge completion, reasoning, and tool learning.
  • Alignment and Safety Evaluation: Alignment evaluation addresses ethics, morality, bias, toxicity, and truthfulness, while safety evaluation focuses on robustness and AGI-related evaluation.
  • Specialized and Organizational Evaluation: Specialized evaluation surveys LLMs in biology, education, law, computer science, and finance, and evaluation organization reviews benchmarks supporting model selection.
  • Scope: The authors explicitly state that the taxonomy does not comprehensively cover the entire evaluation landscape.

3 Knowledge and Capability Evaluation

This section reviews evaluation of LLM knowledge and capability as models expand across applications. It synthesizes methods and benchmarks for question answering, knowledge completion, reasoning, and tool use.

  • Scope: Knowledge and capability evaluation is important for assessing LLM strengths and limitations across diverse tasks and datasets.
  • Coverage: The review spans question answering, knowledge completion, reasoning, and tool use.
  • Objective: The section aims to provide a comprehensive synthesis of current advances in methods and benchmarks for these capabilities.

3.1 Question Answering

Question answering is central to LLM evaluation because it directly affects whether outputs meet expectations, yet datasets often use question answering to test other capabilities. The section therefore emphasizes broad, general-source datasets and reviews established benchmarks.

  • Motivation: Question answering directly influences whether an LLM’s final output meets expectations, but few datasets evaluate it in isolation.
  • Dataset Criteria: Question-answering datasets should draw from broad sources and fields, with general rather than highly professional questions.
  • Benchmarks: SQuAD, NarrativeQA, HotpotQA, and CoQA are examples of datasets used to evaluate question answering in LLMs.
  • Benchmarks: Natural Questions uses anonymized aggregated Google Search queries, while DuReader accounts for data quality and human variation.
  • Overview: Figure 2 provides an overview of studies on knowledge and capability evaluation for LLMs.

3.2 Knowledge Completion

Knowledge completion evaluates the variety and depth of knowledge in LLMs by testing whether they can recover missing entities from factual and commonsense knowledge sources. Benchmarks range from LAMA-style cloze completion to KoLA’s broader knowledge memorization study, while WikiFact remains limited to summarization.

  • Knowledge completion assesses the variety and depth of knowledge required for LLMs’ multitask applications.The section frames knowledge evaluation as important because LLM utility spans general chatbots and specialized professional tools.
  • LAMA converts subject-relation-object triples from Wikidata, ConceptNet, and SQuAD into cloze statements with a missing token.These sources contain factual and commonsense knowledge represented as structured triples.
  • KoLA reconstructs knowledge triples into relation-specific templates and predicts the tail entity using Wikidata5M, EM, and F1.KoLA also examines whether knowledge-entity frequency influences evaluation results across 21 LLMs.
  • WikiFact evaluates factual accuracy with relation tuples derived from English Wikipedia and Wikidata, but its experiments are limited to text summarization.Applying WikiFact to knowledge completion may therefore require modifications to its usage.

3.3 Reasoning

The survey organizes complex reasoning evaluation into commonsense, logical, multi-hop, and mathematical reasoning, covering diverse datasets and task formats. Reported results show persistent weaknesses on socially and temporally grounded commonsense, complex logical and multi-hop tasks, while performance can vary across models and over time.

  • 3.3 Reasoning: Complex reasoning is categorized into commonsense, logical, multi-hop, and mathematical reasoning according to the logic and evidence involved.The categories distinguish different forms of reasoning used to derive conclusions or support decision-making.
  • 3.3.1 Commonsense Reasoning: Commonsense benchmarks use multiple-choice questions to test acquiring and applying commonsense knowledge, commonly reporting accuracy and F1.Datasets span different commonsense domains and are summarized in Table 1.
  • 3.3.1 Commonsense Reasoning: ChatGPT performs strongly on CommonsenseQA, PIQA, and Pep-3k, but GPT-3 and ChatGPT struggle with social, event, and temporal commonsense.The latter weakness is reported on Social IQA, HellaSWAG, and MCTACO.
  • 3.3.2 Logical Reasoning: Logical reasoning benchmarks comprise natural language inference, multiple-choice reading comprehension, and text generation datasets.NLI classifies premise-hypothesis relationships as entailment, contradiction, or neutral; reading-comprehension tasks select answers from candidates.
  • 3.3.2 Logical Reasoning: GPT-3 davinci performs only slightly above random guessing on FOLIO’s first-order-logic reasoning task and struggles with False and Unknown conclusions.GPT-3 and Codex also face difficulty translating natural-language stories into logically or semantically similar first-order logic.
  • 3.3.3 Multi-hop Reasoning: Multi-hop reasoning connects multiple facts to answer questions requiring chained inferences, with HotpotQA and HybridQA commonly evaluated by EM and F1.ChatGPT shows very low performance on sampled HotpotQA questions, and GPT-4 and GPT-3.5 exhibit substantial performance drift over time.
  • 3.3.3 Multi-hop Reasoning: Current prompting methods and libraries face limitations when handling drift in complex multi-hop tasks.The reported drift concerns changing performance on HotpotQA for GPT-4 and GPT-3.5.

3.4 Tool Learning

Tool-learning evaluations assess whether LLMs can use tools and how effectively they perform downstream tasks, while increasingly introducing multi-tool benchmarks to test genuine external-tool use. Existing evaluations span search, shopping, robotics, code generation, APIs, and specialized applications, but real-world interaction data remain limited.

  • Evaluation dimensions: Tool-learning evaluation measures both whether models can execute tools and how well they perform after tool use.Common metrics include execution pass rate, tool-operation success rate, and downstream task metrics.
  • Evaluation limitations: Human evaluation can be expensive, time-consuming, intricate, and less generalizable than automatic metrics, and model comparisons require the same API version.These constraints affect evaluations that rely on crowdsourced judgments and changing tool interfaces.
  • Specialized tool use: GeneGPT combines an LLM with the NCBI Web API and excels at one-shot GeneTuring evaluation, whereas several general LLMs often score 0.0.The evaluation covers gene naming, genome positioning, gene function analysis, and sequence alignment, but reports extraction errors.
  • Specialized tool use: RoboCodeGen evaluates 37 robot-themed function-generation tasks and finds domain-specific models generally outperform OpenAI LLMs, with performance increasing by model size.It uses unit-test pass rate and permits third-party libraries, inferred conventions, and hierarchical generation.
  • Tool-oriented models: Tool-oriented evaluations cover search engines, online shopping, and robotic environments requiring sequential planning and action execution.ALFRED includes six tasks and over 3,000 environments, while WebShop uses 12,087 instructions with human shopping paths.
  • Multi-tool benchmarks: Single-tool downstream benchmarks may overestimate tool use because some tasks can be solved from models’ internal knowledge alone.This motivates benchmarks designed specifically around external-tool interaction.
  • Multi-tool benchmarks: API-Bank evaluates tool-augmented LLMs with 53 APIs and 264 annotated dialogues, using call accuracy, ROUGE-L, and task-planning completion.The benchmark covers a workflow for tool-augmented models rather than only downstream task performance.
  • Evaluation limitations: The survey identifies a shortage of high-quality datasets representing genuine human–machine interaction in real-world scenarios.This shortage limits the representativeness of current tool-learning evaluations.

4 Alignment Evaluation

Alignment evaluation examines whether instruction-tuned LLMs behave consistently with human values and avoid problematic outputs. The survey organizes this evaluation around ethics, bias, toxicity, and truthfulness.

  • Alignment evaluation: Alignment evaluations cover ethics, bias, toxicity, and truthfulness while addressing annotator bias, human catering, and hallucination.The survey frames these dimensions as complementary ways to assess aligned LLMs.

4.1 Ethics and Morality

Ethics and morality evaluations assess whether LLMs align with ethical values and avoid outputs that deviate from ethical standards. The survey groups existing approaches by how moral categories or datasets are defined: experts, crowds, AI assistance, or hybrids.

  • Expert-defined ethics and morality: Expert-defined ethics uses academic frameworks such as Moral Foundation Theory to organize moral categories and evaluation datasets.Related datasets address politics, social sciences, social media, and multidimensional moral judgments.
  • Expert-defined ethics and morality: TrustGPT separates active value alignment from passive value alignment, evaluating social-norm judgment and answer refusal with different metrics.It uses soft and hard accuracy for active alignment and refusal proportion for passive alignment.
  • Crowdsourced ethics and morality: Crowdsourced ethics datasets derive judgments from public preferences or recruited participants without professional guidance or training.Examples include Reddit-based judgments and interview-style exception scenarios.
  • AI-assisted ethics and morality: AI-assisted ethics uses models such as GPT-3 to help draft statements or construct datasets for evaluating responses to problematic content.PROSOCIALDIALOG is a multi-turn dialogue dataset designed around socially normative responses.
  • AI-assisted ethics and morality: AI-generated ethical scenarios can be evaluated across low- and high-ambiguity settings using statistical measures and evaluation metrics.One study generates scenarios and actions with GPT-4 or text-davinci-003 and evaluates 28 open- and closed-source LLMs.
  • Hybrid ethics and morality: Hybrid ethics datasets combine expert-created guidelines with crowd-determined judgments, including normative anecdotes and ethical dilemmas.ANECDOTES contains 32,000 real-life anecdotes, while DILEMMAS contains 10,000 simple ethical dilemmas.

4.2 Bias

Bias evaluation examines harms affecting social groups and tests how biased representations or decisions appear in generated outputs and downstream tasks. The survey reviews NLI, sentiment analysis, relation extraction, coreference resolution, and machine translation benchmarks, alongside concerns about benchmark reliability.

  • Bias evaluation: Bias can involve stereotypes, group devaluation, underrepresentation, or inequitable resource allocation, and may appear in model outputs even when representations alone are inconclusive.The survey therefore emphasizes external assessment through tasks and generated content.
  • Coreference resolution: Coreference benchmarks test gender associations with professions using pronoun substitutions, stereotypical and non-stereotypical roles, and equitable reference decisions.Winogender, WinoBias, GAP, and GICOREF expand coverage of gendered and transgender-authored reference phenomena.
  • Coreference resolution: A reliability analysis found that only 0%–58% of WinoBias and Winogender tests were unaffected by identified pitfalls, including assumptions, ambiguities, and inconsistencies.These findings question whether the benchmarks effectively measure stereotyping.
  • Machine translation: WinoMT evaluates gender bias in machine translation across eight target languages using contextualized occupational references and automatic gender extraction.Reported accuracy does not exceed 70% in the described evaluation.
  • Natural language inference: NLI bias benchmarks measure deviations from neutrality across polarized adjectives and ethnic names, finding substantial bias in GloVe, ELMo, and BERT.The benchmark extends coverage beyond gender to nationality and religion and includes millions of sentence pairs.
  • Sentiment analysis: Sentiment-analysis evaluations compare otherwise matched sentences whose age-related vocabulary changes, revealing significant age bias in most evaluated algorithms.The comparative dataset derives from 121 filtered sentences drawn from blog posts and comments.
  • Relation extraction: Relation-extraction bias can produce knowledge bases with more information about male entities than female entities when profession-related relations are predicted unevenly.The survey presents this as a motivation for bias assessment in neural relation extraction.

4.3 Toxicity

The survey evaluates LLM toxicity through both toxicity identification/classification and toxicity generation, using established datasets and scoring tools.

  • Evaluation tasks: Toxicity evaluation covers identifying and classifying toxic content, as well as measuring toxicity in LLM-generated text.The survey distinguishes input-oriented toxicity tasks from evaluation of generated sentences.
  • Identification and classification datasets: OLID contains 14K Twitter sentences labeled for offensive language, targeting, and the insulted entity.Its labels distinguish offensive from non-offensive language, targeted from non-targeted insults, and individual, target, or other entities insulted.
  • Identification and classification datasets: Zero-shot studies evaluate LLM toxicity detection with Social Bias Inference Corpus, HateXplain, and Civility datasets.Other evaluations examine ChatGPT’s reproduction of human labels and detection of hateful, offensive, and toxic content.
  • Generation datasets: RealToxicityPrompts provides 100K naturally occurring prompts, including 22K with higher toxicity scores, for evaluating toxic generation.HarmfulQ adds 200 explicitly toxic questions for assessing generated answers.
  • Generation datasets: PerspectiveAPI scores toxicity from 0 to 1 and supports multilingual sentence evaluation.Higher scores indicate higher toxicity.

4.4 Truthfulness

Truthfulness evaluation addresses fabricated or misleading LLM outputs through question answering, dialogue, and summarization benchmarks, including tests of abstention and factual consistency.

  • Motivation and taxonomy: LLM fluency can coexist with fabricated facts and misinformation, motivating dedicated truthfulness evaluation.The survey organizes truthfulness datasets around question answering, dialogue, and summarization.
  • Question answering: Unanswerable-question benchmarks test whether models recognize when available information cannot support a reliable answer.These datasets target questions lacking evidence, context, consensus, or a definite answer.
  • Question answering: NewsQA contains 119,633 question-answer pairs, with 9.5% of questions lacking answers in their corresponding articles.Questions were written from headlines and summaries rather than full articles, creating evidence gaps.
  • Question answering: SQuAD 2.0 adds 53,775 adversarial unanswerable questions to test whether models know when they cannot provide reliable answers.The questions are designed to appear relevant to their context paragraphs.
  • Question answering: SelfAware contains 1,032 unanswerable and 2,337 answerable questions spanning five reasons for unanswerability.TruthfulQA instead tests avoidance of imitative falsehoods across 817 questions in 38 categories.
  • Dialogue and summarization: Dialogue and summarization evaluation uses factual-consistency benchmarks, including DIALFACT, BEGIN, and AGGREFACT, alongside NLI-based methods.AGGREFACT aggregates nine factuality-annotated datasets, while entailment models compare premises and hypotheses for consistency verification.

5 Safety Evaluation

Safety evaluation is divided into robustness assessment and risk evaluation, covering resilience to disruptions, attacks, harmful behaviors, and agentic capabilities.

  • Scope: The survey groups LLM safety evaluation into robustness assessment and risk evaluation of advanced or general-purpose models as agents.Robustness measures stability under disruptions, while risk evaluation examines behaviors and agent performance.
  • Robustness evaluation: Robustness evaluation covers prompt robustness, task robustness, and alignment robustness.These categories address stability across inputs, tasks, and alignment with human values.
  • Robustness evaluation: PromptBench attacks LLMs with dynamically generated character-, word-, sentence-, and semantic-level adversarial prompts across eight NLP tasks.Other studies test prompt typos and robustness on translation, question answering, classification, and NLI.
  • Robustness evaluation: ReCode evaluates code-generation robustness by perturbing programming problems, while related benchmarks test robustness in mathematical reasoning and multilingual tasks.The cited evaluations use HumanEval, MBPP, AOJ, ASDiv-A, MAWPS, SVAMP, and multilingual datasets.
  • Alignment robustness: Alignment robustness studies use jailbreak prompts to test whether models generate harmful or unsafe behavior and content.MasterKey generates jailbreak prompts for GPT-3.5, GPT-4, BARD, Bing Chat, and ERNIE.
  • Risk evaluation: Risk evaluation examines power-seeking, situational awareness, decision consistency, and agents’ ability to solve complex tasks in real environments.Studies construct behavioral datasets, test risky subgoals and decision patterns, and evaluate agents beyond ordinary capability benchmarks.
  • Risk evaluation: LLMs exhibit logical errors in chess, future-event prediction, and legal judgment, while prompt engineering and fine-tuning improve autonomous-task performance.The improvement is reported even when fine-tuning tasks are unrelated to autonomous replication abilities.

6 Specialized LLMs Evaluation

Specialized LLM evaluation spans medicine, education, law, computer science, and finance, combining domain benchmarks with application-focused and human assessments.

  • Overview: The survey extends LLM evaluation to specialized domains while noting that domain-specific challenges and limitations persist.Covered fields include biology and medicine, education, legislation, computer science, and finance.
  • Medicine: Medical evaluations use licensing exams, scientific-literature QA, consultation datasets, and human ratings of factuality, reasoning, harm, and bias.Human evaluation finds impressive performance but continuing gaps relative to professional clinicians.
  • Education: Educational studies evaluate teaching competence and learning support, with GPT-3 and Blender performing below professional teachers, especially for helpfulness.Human-created algebra hints produce statistically significantly higher learning gains than ChatGPT hints, while ChatGPT essay feedback shows good readability and expert agreement.
  • Legislation: GPT-4 passes the entire Uniform Bar Examination, while GPT-3.5 achieves a 50.3% correct rate on its MBE section.Other legal evaluations report strong entailment performance but poor results on simple synthetic statutory reasoning and factuality limitations in explanations and summaries.
  • Computer science: EvalPlus extends HumanEval by 81x into HumanEval+ and reduces average pass@k by 13.6–15.3% by detecting previously missed wrong code.Programming-assistance studies also find LLM explanations easier to understand and more accurate than student-created explanations.
  • Finance: BloombergGPT, a 50-billion-parameter finance model, outperforms existing models across ConvFinQA, FiQA SA, FPB, and Headline tasks.Finance studies also find coherent financial reasoning emerging at 6B parameters and report GPT accuracy of 58% and 67% on a financial literacy test.

7 Evaluation Organization

The survey organizes LLM evaluation around benchmark coverage and examines how performance varies with subjects, languages, model size, instruction tuning, and reasoning prompts. Results show strong but uneven capabilities, with important gaps in computational, multilingual, and complex reasoning tasks.

  • Benchmark coverage: GLUE evaluates nine NLU tasks plus a diagnostic dataset spanning sentence-level understanding, similarity, paraphrase, and inference.Its diagnostic set probes linguistically important phenomena such as logic and predicate-argument structure.
  • Benchmark coverage: Existing benchmarks often diverge from real-world applications: tasks easy for humans can challenge models, while benchmark success may not translate into practical usefulness.This motivates broader evaluation beyond conventional benchmark tasks.
  • Benchmark coverage: LongBench targets long-context bilingual NLU and NLG, including multi-document QA, single-document QA, and code completion, where smaller open-source models still trail commercial counterparts.It addresses the short-context bias of earlier benchmarks.
  • Subject competence: LLMs perform unevenly across subjects: GPT-4 is top-tier on evaluated benchmarks, but most models remain weak in computational disciplines such as mathematics and physics.Models may handle context and instructions well yet struggle with disciplinary concepts, formulas, symbols, and long logical chains.
  • Multilingual representation: Multilingual performance depends on language and language family: Chinese-trained models outperform GPT-4 and ChatGPT on Chinese tasks, while models struggle in non-Latin and low-resource languages.English-translated prompts can improve performance, suggesting differences in language comprehension and captured knowledge; diverse language data are therefore needed.

8 Future Directions

Future evaluation should move beyond static, question-answering benchmarks toward richer environments and analyses that expose model weaknesses, explain risky behaviors, and guide improvement.

  • 8.1 Risk Evaluation: Current risk evaluations mainly use question answering, which is insufficient for precisely assessing behaviors in specific situations or environments.The survey calls for understanding not only whether models seek power, but also why and how such behaviors occur.
  • 8.2 Environment-Based Evaluation: Risk-focused environments remain underdeveloped, while existing agent research largely studies capability execution in limited settings such as online shopping and virtual societies.The survey proposes increasing environmental diversity to support risk discovery.
  • 8.3 Dynamic Evaluation: Static benchmarks are vulnerable to data leakage and contamination because their evaluation instances can become training data for LLMs.Dynamic evaluation can update test data quickly, reducing opportunities for models to train on evaluation content.
  • 8.3 Dynamic Evaluation: Static multiple-choice benchmarks exclude open-ended questions, while LLM-based evaluators may introduce factual errors and preference biases.The survey discusses multi-model debate as a possible alternative for evaluating open-ended responses.
  • 8.3 Dynamic Evaluation: Static benchmarks assess factual knowledge that may change over time, limiting their fit for evaluating temporally evolving information.The passage identifies changing real-world knowledge as another challenge for static evaluation.
  • 8.4 Enhancement-Oriented Evaluation for LLMs: Existing evaluations primarily provide quantitative task scores, but comprehensive methods that analyze weaknesses and jointly assess helpfulness, harmlessness, and honesty remain lacking.The survey advocates enhancement-oriented evaluation that connects diagnosis of shortcomings with concrete improvement directions.

9 Conclusion

The survey argues that understanding and governing rapidly advancing LLMs requires comprehensive evaluation across capabilities, alignment, safety, and applications. It presents broad coverage and benchmark resources as guidance for safer, more reliable, and beneficial development.

  • 9 Conclusion: LLMs’ rapid progress has outpaced understanding of their capabilities, motivating comprehensive benchmarking across tasks and domains.The survey emphasizes delineating capability boundaries and understanding performance in diverse domains.
  • 9 Conclusion: The survey covers knowledge and reasoning, alignment concerns including ethics and toxicity, safety and trustworthiness, and applications in domains such as biology, education, law, computer science, and finance.It also provides popular benchmark evaluations for researchers, developers, and practitioners.
  • 9 Conclusion: The survey aims to provide clear guidance for the controlled advancement of LLM evaluations.Its stated direction is to help LLM applications become safe, reliable, and beneficial across domains.
Loading 2310.19736v3…