Holistic Evaluation of Language Models

Percy Liang; Rishi Bommasani; Tony Lee; Dimitris Tsipras; Dilara Soylu; Michihiro Yasunaga; Yian Zhang; Deepak Narayanan; Yuhuai Wu; Ananya Kumar; Benjamin Newman; Binhang Yuan; Bobby Yan; Ce Zhang; Christian Cosgrove; Christopher D. Manning; Christopher Ré; Diana Acosta-Navas; Drew A. Hudson; Eric Zelikman; Esin Durmus; Faisal Ladhak; Frieda Rong; Hongyu Ren; Huaxiu Yao; Jue Wang; Keshav Santhanam; Laurel Orr; Lucia Zheng; Mert Yuksekgonul; Mirac Suzgun; Nathan Kim; Neel Guha; Niladri Chatterji; Omar Khattab; Peter Henderson; Qian Huang; Ryan Chi; Sang Michael Xie; Shibani Santurkar; Surya Ganguli; Tatsunori Hashimoto; Thomas Icard; Tianyi Zhang; Vishrav Chaudhary; William Wang; Xuechen Li; Yifan Mai; Yuhui Zhang; Yuta Koreeda

Source-linked AI summary

Holistic Evaluation of Language Models

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, Yuta Koreeda

arXiv:2211.09110v2Nov 16, 2022cs.CLcs.AIcs.LG

TL;DR

Language models’ capabilities, limitations, and risks remain poorly understood despite their growing impact. HELM organizes evaluation around use cases and metrics, then systematically selects a benchmark subset. The resulting benchmark makes explicit both its evaluation coverage and what remains missing.

Problem
Language models’ capabilities, limitations, and risks remain poorly understood despite their growing impact and adaptability across downstream scenarios.
Source: §1 · p. 2
Method
HELM taxonomizes language-model evaluation by use cases and metrics, then systematically selects a benchmark subset reflecting explicit priorities.
Source: §9 · p. 77
Results
HELM’s benchmark makes explicit the gap between its intended evaluation space and its implemented coverage.
Source: §10 · p. 77, §9 · p. 77
Takeaways & Limitations
Benchmark performance does not universally determine a model’s practical suitability, particularly in contexts involving further specialization.
Source: §11.1 · p. 81
The extent of train-test contamination and its impact on the validity of HELM’s results remain largely unclear.
Source: §11.1 · p. 81

Abstract

from arXiv · show

Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models. First, we taxonomize the vast space of potential scenarios (i.e. use cases) and metrics (i.e. desiderata) that are of interest for LMs. Then we select a broad subset based on coverage and feasibility, noting what's missing or underrepresented (e.g. question answering for neglected English dialects, metrics for trustworthiness). Second, we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios when possible (87.5% of the time). This ensures metrics beyond accuracy don't fall to the wayside, and that trade-offs are clearly exposed. We also perform 7 targeted evaluations, based on 26 targeted scenarios, to analyze specific aspects (e.g. reasoning, disinformation). Third, we conduct a large-scale evaluation of 30 prominent language models (spanning open, limited-access, and closed models) on all 42 scenarios, 21 of which were not previously used in mainstream LM evaluation. Prior to HELM, models on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: now all 30 models have been densely benchmarked on the same core scenarios and metrics under standardized conditions. Our evaluation surfaces 25 top-level findings. For full transparency, we release all raw model prompts and completions publicly for further analysis, as well as a general modular toolkit. We intend for HELM to be a living benchmark for the community, continuously updated with new scenarios, metrics, and models.

1 Introduction

HELM combines broad scenario coverage, multi-metric measurement, and standardized evaluation of 30 language models, with public results and code, to improve transparency about their capabilities, limitations, and risks. Its findings identify strong overall performers while revealing scenario-dependent weaknesses, social risks, prompting sensitivity, and limits of evaluation and performance prediction.

1 Introduction

HELM seeks transparency through fuller characterization of language models whose broad capabilities, limitations, and risks remain poorly understood.The motivation is to improve scientific understanding and orient societal impact rather than evaluate only one aspect of these broadly adaptable models.Source: §1 · p. 2

1.1 HELM

98 of 112 core scenario–metric pairs (87.5%) are measured across 16 scenarios and seven metric categories.HELM combines an evolving taxonomy that exposes omissions with implementations selected for coverage, value, and feasibility; its English-only core spans six tasks and measures accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency together to expose trade-offs, while acknowledging contested operationalizations such as fairness. Seven targeted evaluations add 26 scenarios, including 21 new or previously non-mainstream scenarios, to isolate linguistic understanding, knowledge, reasoning, memorization and copyright, disinformation, biases, and toxicity.Source: §1.1 · pp. 3–4
96.0% average core-scenario coverage replaces 17.9% under standardized evaluation, enabling direct comparisons between language models.HELM benchmarks 30 models using the same scenarios, metrics, and 5-shot prompts; the remaining 4.0% reflects model-specific technical issues. Findings remain conditional on models’ suitability for these settings, unequal training resources, uncertain train-test contamination, and adaptation choices; more sophisticated adaptation could produce qualitatively different findings.Source: §1.1 · pp. 4–6, Figure 4 · p. 5

1.2 Empirical findings

Three metrics—accuracy, robustness, and fairness—favor text-davinci-002 across core scenarios, while non-open models show an accuracy advantage over open models on every core scenario.Anthropic-LM v4-s3 (52B) ranks in the top three on all three metrics, suggesting benefits from instruction-tuning; text-davinci-002 leads all nine core question-answering scenarios and all knowledge-intensive evaluations, while TNLG v2 (530B) shows particular strength on factual knowledge. Code models outperform text models on reasoning: code-davinci-002 achieves 52.1% accuracy on GSM8K versus 35.0% for text-davinci-002; the open/non-open gap describes only the evaluated snapshot.Source: §1.2 · pp. 6–8
72.6% standard accuracy falls to 38.9% under robustness perturbations for TNLG v2 (530B) on NarrativeQA, despite strong accuracy–robustness–fairness correlations.The most accurate model is not always most robust or fair; improving accuracy worsens calibration on HellaSwag but improves it on OpenBookQA, while OPT (175B) scores 1.506 bits per byte for White English versus 2.114 for African American English on TwitterAAE (lower is better). Core-generation bias and toxicity are broadly similar across models and low on average but potentially harmful; accuracy and efficiency show no strong overall trade-off, although larger models within families improve accuracy at higher training and inference costs, and only a subset occupy each scenario’s accuracy-efficiency Pareto frontier.Source: §1.2 · pp. 6–7
50.1% accuracy leaves OPT (175B) essentially at chance on CivilComments, illustrating weak toxicity detection despite its strong accuracy across other scenarios.Most models have similar toxicity-detection accuracy for comments mentioning Black and White individuals, but OPT (175B) drops from 51.3% standard to 8.8% robust accuracy on the Black split versus 50.8% to 24.3% on the White split; retrieval models beat classical methods and sometimes match fine-tuned neural retrievers, but trail state of the art and require computationally intensive passage-by-passage requests. Automated summarization metrics largely miss observed quality differences; generally accurate, calibrated IMDB models reveal robustness weaknesses on contrast sets from Gardner et al. (2020), and text-davinci-002 achieves only 40.8% on RAFT’s Systematic Review Inclusion split versus 97.5% from several models, including GLM (130B).Source: §1.2 · pp. 7–8
Language-modeling rankings differ from core-task rankings, while targeted evaluations reveal memorization and disinformation-generation risks.GPT-NeoX (20B), OPT (175B), BLOOM (176B), GPT-J (6B), and OPT (66B) consistently achieve the lowest bits-per-byte on The Pile, TwitterAAE, and ICE; BLiMP performance is broadly similar, but core leaders text-davinci-002 and TNLG v2 (530B) are among the weakest on irregular forms. Long copyrighted regurgitation is uncommon but more noticeable for popular books and correlates with model accuracy; large models generate realistic thesis-supporting headlines effectively, while action-encouraging disinformation results are mixed.Source: §1.2 · pp. 8–9
89.5% accuracy makes text-davinci-002 the strongest BBQ model, but the three most accurate models uniquely exhibit socially aligned biases in ambiguous contexts.The next-most-accurate models are T0++ (11B) at 48.4% and TNLG v2 (530B) at 44.9%; bias trends in disambiguated contexts are less clear. All models generate substantially more toxicity following toxic RealToxicityPrompts inputs than relatively non-toxic inputs in RealToxicityPrompts and BOLD; sensitivity to the toxicity detector and diverse human judgments remains an open question.Source: §1.2 · p. 9
74.4% ROUGE-L on NarrativeQA makes text-davinci-002 state of the art to the authors’ knowledge, surpassing UnifiedQA-v2’s 67.4% ROUGE-L.Unified evaluation surfaces new findings even when models and datasets are already publicly available.Source: §1.2 · p. 9
79.1% HellaSwag accuracy for OPT (175B) with separate 0-shot answer prompts falls to 30.2% with jointly presented choices in a single 5-shot prompt.Across scenarios and metrics, all models are sensitive to prompt formatting and the choice and number of in-context examples. Accuracy-maximizing prompting and adaptation decisions differ across models, complicating fair standardization.Source: §1.2 · p. 9
Bits-per-byte on The Pile poorly predicts downstream accuracy across model families, despite being more comparable than perplexity.Some evaluated models were trained on The Pile, whereas others were not. The authors call for reliable upstream surrogates, including for desiderata beyond accuracy, as discussed in Bommasani et al. (2021, §4.4.2).Source: §1.2 · pp. 9–10
Model scale reliably predicts accuracy within families but is not a good predictor across all models in any scenario.All models winning head-to-head accuracy comparisons at rates above 55% have at least 50B parameters, yet top-five models include Anthropic-LM v4-s3 (52B) and Cohere xlarge v20220609 (52.4B). Scaling may be less efficient than alternatives such as training with human feedback.Source: §1.2 · p. 10

1.3 Contributions

16 core scenarios, seven metrics, and seven targeted evaluations operationalize a scenario–metric taxonomy that makes benchmark priorities and limitations explicit.Metrics cover accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. Targeted evaluations examine skills and risks including knowledge, reasoning, disinformation, and copyright, introducing 21 scenarios not previously used in mainstream language-model evaluation.Source: §1.3 · p. 10
30 language models are evaluated under standardized benchmark conditions, enabling direct comparisons across many scenarios and metrics.The models comprise 10 open, 17 limited-access, and three closed models. The evaluation yields findings that both reinforce prior literature and provide new knowledge, supporting future model development and further analysis.Source: §1.3 · p. 10
A public website shares all results, underlying model predictions, and adaptation details, alongside an extensible codebase for community development.These resources support further analysis and extensions of HELM.Source: §1.3 · p. 10

2 Preliminaries

HELM organizes language-model evaluation around scenarios, adaptation procedures, and metrics. It makes the evaluation space and benchmark designers’ selections explicit.

2.1 Scenarios

Scenarios operationalize desired use cases as instances divided into a training set and one or more test sets.Each instance contains an input string and a list of reference strings annotated with evaluation-relevant properties, such as correctness or acceptability.Source: §2.1 · p. 14

2.2 Adaptation

Adaptation transforms a language model and training instances into a system that predicts new instances; HELM focuses on prompting.Prompts may include in-context training instances. Multiple-choice adaptation can present all answer choices jointly or each choice separately.Source: §2.2 · p. 14, Figure 7 · p. 15
A black-box text-to-text interface accepts prompts and decoding parameters and returns completions with prompt and completion log probabilities, without requiring internal activations or training data.This general interface covers text-only tasks while remaining agnostic to how models are constructed.Source: §2.2 · pp. 14–15

2.3 Metrics

Metrics operationalize system desiderata by evaluating the completions and probabilities produced when adapted models run on each scenario’s evaluation instances.The running example illustrates accuracy, calibration, robustness, fairness, and inference-runtime measurements.Source: §2.3 · p. 15

2.4 Roadmap

30 models are evaluated with 5-shot prompting, with each evaluation run specified by a scenario, adaptation method, and metric.HELM taxonomizes scenario and metric spaces before systematically selecting evaluation points, distinguishing its abstract aspirations from its concrete implementation.Source: §2.4 · p. 15

3 Core scenarios

HELM selects a feasible, minimal set of core scenarios organized by task, domain, and language, prioritizing user-facing tasks and covering these dimensions independently rather than exhaustively evaluating their combinations. These scenarios include toxicity detection and miscellaneous text classification, using demographic annotations to examine performance disparities and naturally occurring tasks to represent practical applications.

3 Core scenarios

Core scenarios are structured by task, domain, and language, with selection guided by coverage, minimality, user-facing relevance, and feasibility.These categories provide intuitive structure rather than atomic or unambiguous definitions; the benchmark explicitly identifies regions it does not cover.Source: §3 · pp. 15–16

3.1 Taxonomy

The task taxonomy maps ACL 2022 tracks to canonical tasks but cannot exhaust evolving language-model use cases.Canonical task choices are subjective and coarse, with important finer-grained structure inside established tasks. Deployed models introduce previously unconsidered uses, and even articulating the long tail of known applications remains open.Source: §3.1 · pp. 16–17
Domains are decomposed into genre, time period, and demographic group, while language coverage concentrates on English varieties.Datasets are simplified to a single input-based domain even when outputs differ; geography and text-production modality are omitted. Despite the world's linguistic diversity and NLP's concentration on high-resource languages, even multilingual models such as BLOOM (176B) are evaluated only in English.Source: §3.1 · pp. 17–18

3.2 Selection

Scenario selection covers tasks, domains, and English varieties independently, accepting potentially missed interactions to prioritize feasible, user-facing evaluations.Selected tasks comprise question answering, information retrieval, summarization, sentiment analysis, toxicity detection, and miscellaneous text classification; intermediary tasks and interactive dialogue are excluded, while multimodal and multilingual evaluation is deferred. Dataset availability, implementation effort, and computation constrain coverage; domains are covered through dataset selection, while targeted evaluations include national English varieties and African American English.Source: §3.2 · pp. 18–19, §3.3 · p. 19

3.3 Question answering

Question-answering datasets cover varied domains, commonsense reasoning, multidisciplinary knowledge, truthfulness, and contrast-set robustness.NaturalQuestions, evaluated open- and closed-book, NarrativeQA (Kočisky et al., 2017), and QuAC (Choi et al., 2018) cover search queries, stories, and conversational questions; HellaSwag, OpenBookQA, and TruthfulQA (Lin et al., 2021b) add commonsense inference, science reasoning, and misconception-driven truthfulness. MMLU spans 57 tasks, while BoolQ (Clark et al., 2019) and QuAC support robustness evaluation through contrast sets (Gardner et al., 2020).Source: §3.3 · p. 20

3.4 Information retrieval

Information retrieval reranks externally retrieved passages using prompted Yes/No log-probabilities for individual query–passage pairs.MS MARCO's regular track and TREC 2019 track (Craswell et al. 2020) share 9M web passages but contrast sparse relevance judgments with 43 densely annotated queries. Prompts ask whether a passage answers the query, ranking high-probability Yes answers higher; processing many passages requires repeated model calls.Source: §3.4 · p. 21

3.5 Summarization

Summarization evaluates freely generated summaries for quality, faithfulness, and extractiveness, emphasizing hallucination risks.CNN/DailyMail has largely extractive references, whereas XSUM has largely abstractive references; both cover news, leaving broader summarization diversity unrepresented and attracting evaluation-design critiques (Gehrmann et al., 2022b). Quality uses ROUGE-2 and BERTScore, while reference-free faithfulness metrics (Fabbri et al., 2022) may exploit spurious correlations (Durmus et al., 2022).Source: §3.5 · pp. 22–23

3.6 Sentiment analysis

Sentiment analysis uses binary IMDB movie-review classification, selected for its contrast set enabling robustness evaluation.Engineering constraints limit evaluation to this dataset; ratings at most 4 are negative and ratings at least 7 are positive. Its contrast set (Gardner et al., 2020) enables robustness measurement under equivariances, but broader domains and more complex sentiment tasks remain uncovered.Source: §3.6 · p. 23

3.7 Toxicity detection

Toxicity definitions can disproportionately affect marginalized groups because interpretations depend on social group membership, status, and privilege.Critiques identify reductive framing, insufficient dataset context, and annotator-dependent judgments (Sap et al., 2019a; Gordon et al., 2022).Source: §3.7 · pp. 23–24
Automated moderation could exacerbate existing harms because language models perform poorly in languages that are not dominant in machine learning.Toxicity-detection failures have contributed to human-rights violations, including the Rohingya genocide, and stressed democracies. Some failures reflect insufficient linguistic and cultural competence among human moderators.Source: §3.7 · p. 24
HELM evaluates binary toxicity classification using CivilComments (Borkan et al., 2019b), whose demographic annotations enable comparisons of performance disparities.Models are directly prompted to judge whether a passage is toxic. CivilComments contains comments from 50 English-language news sites collected during 2015–2017.Source: §3.7 · p. 24

3.8 Miscellaneous text classification

Miscellaneous classification directly predicts categorical labels, with explicit task instructions when categories are complex or lack a familiar task name.This scenario addresses the long tail of useful classification tasks beyond prominent categories such as sentiment analysis and toxicity detection.Source: §3.8 · pp. 24–25
11 naturally occurring tasks in RAFT (Alex et al., 2021) represent practical classification applications; evaluation holds out public training examples because test labels are private.Applications include banking queries, adverse drug effects, legal overruling, unfair terms of service, and complaint detection.Source: §3.8 · p. 25

4 General metrics

HELM’s general metrics cover accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency, prioritizing scalable measurements compatible with blackbox access without assumptions about model construction or broader context. Efficiency measures encompass training energy use and carbon emissions, as well as inference runtimes that distinguish deployment-specific best-case latency from standardized model comparisons.

4.1 Taxonomy; 4.2 Selection

87.5% of possible scenario–desideratum pairs are measured: 98 of 112 across 16 scenarios and seven top-level desiderata.Conference research areas inform the taxonomy; omitted pairs are generally undefined or raise measurement-validity concerns. Additional analyses address reported training efficiency and environmental impact, copyrighted-content memorization, disinformation, and adaptation sample efficiency through 5-shot prompting with example-count ablations.Source: §4.1 · pp. 25–26, §4 · p. 25, §4.2 · pp. 26–27

4.3 Accuracy

Accuracy denotes each scenario’s standard performance metric, but averaging across test instances can conceal poor performance for minority subpopulations.Metrics include exact-match accuracy for classification, word-overlap F1 for question answering, MRR and NDCG for retrieval, and ROUGE for summarization.Source: §4.3 · p. 27

4.4 Calibration and uncertainty

Calibration evaluation measures probability reliability with expected calibration error and separately tests whether confidence supports selective classification.ECE uses 10 bins containing equal numbers of probabilities. Selective classification reports accuracy on the highest-confidence C = 0.1 fraction and area under the coverage-accuracy curve across C from 0 to 1, distinguishing confidence ranking from probability calibration.Source: §4.4 · p. 28

4.5 Robustness

Robustness measures worst-case performance across input transformations, testing stability under benign changes and sensitivity when changes require different outputs.Invariance uses mild, semantics-preserving perturbations only for classification, question answering, and retrieval; equivariance uses Contrast Sets (Gardner et al., 2020) with changed targets for BoolQ and IMDB. Distribution-shift evaluation requires special annotations and training-data information, while adaptive adversarial queries and human-in-the-loop evaluation are difficult to scale.Source: §4.5 · pp. 29–30

4.6 Fairness

Fairness evaluation combines demographic counterfactual perturbations with group-specific accuracy measurements where annotations exist, while leaving appropriate dialect-use conventions unresolved.Perturbations cover Standard American English versus African American English, race, and binary gender in classification, question answering, and retrieval; they cannot capture complex differences between groups’ underlying distributions. Metadata-based disparities cover speaker gender, nationality, and spoken versus written language, plus subject gender, sex, race, religion, and disability; such metadata are unavailable for most scenarios.Source: §4.6 · pp. 30–31

4.7 Bias and stereotypes

Bias metrics quantify demographic representation and stereotypical associations in generated text, but word-count measurements can be brittle because social groups are linguistically marked differently.For binary gender and race, metrics compare mention and association rates against a uniform reference, while allowing alternative references. Unlike fairness’s task-accuracy disparities, these measures concern generation properties and apply to all core text-generation scenarios.Source: §4.7 · pp. 31–32

4.8 Toxicity

Toxicity evaluation uses the Perspective API on generated text, but scalable cross-scenario measurement lacks a precise, context-sensitive toxicity definition.The API was chosen for extensive testing and acknowledged limitations rather than presumed superiority over newer detectors. Evaluation covers all core text-generation scenarios, and released generations enable reassessment using other detectors.Source: §4.8 · pp. 32–33

4.9 Efficiency

Efficiency evaluation covers both training and inference, examining energy, carbon, and wall-clock efficiency because high costs reduce model usability and accessibility.The motivation is to assess whether accuracy improvements justify additional training or inference time and expense.Source: §4.9 · p. 33

4.9.1 Training efficiency

Training efficiency is measured through energy consumption in kWh and CO2 emissions in kilograms, accounting for accelerator number and type.Emissions additionally reflect the energy sources powering training. Training runtimes are omitted because they are rarely reported and do not capture accelerator counts.Source: §4.9.1 · p. 33
Training energy and emissions use creators’ reported values when available, otherwise hardware-and-duration estimates when sufficient information exists.Estimates incorporate GPU power, count, training duration, datacenter overhead with PUE set to 1.1, and regional carbon intensity, defaulting to the U.S. average when location is unavailable. Estimates are approximate; insufficient information prevents reliable estimates for some models, including AI21 models.Source: §4.9.1 · pp. 33–34

4.9.2 Inference efficiency

Inference energy accounting lacks public serving-hardware information, while raw per-request runtimes confound model efficiency with hardware, software, and resource contention.Contention can introduce queueing delays rather than computation time.Source: §4.9.2 · p. 34
Denoised inference runtime retains each provider’s hardware and software while removing noise from performance variation.Reported for every model, it estimates best-case query latency for end users accessing deployed interfaces.Source: §4.9.2 · p. 34
Idealized inference runtime uses uniform, optimized hardware and software to compare models’ inference efficiency.HELM reports this metric for models with publicly available architecture information, enabling efficiency-capability comparisons. Measurements use NVIDIA A100 GPUs and Megatron (Shoeybi et al., 2019).Source: §4.9.2 · p. 34
Idealized runtimes also support derived energy and CO2 metrics because hardware is controlled; denoised runtimes cannot support these calculations without provider hardware information.Source: §4.9.2 · p. 35

5 Targeted evaluations

HELM supplements user-facing evaluation with targeted analyses of language, knowledge, reasoning, memorization and copyright risks, disinformation, demographic bias, and toxicity. These analyses provide finer-grained insight into model behavior, probing disinformation through narrative generation, bias through question answering, and toxicity through prompted completions.

5.1.1 Language modeling

Language modeling uses tokenization-invariant bits per byte and includes English-variety corpora to measure performance disparities alongside broader benchmark coverage.Benchmarks include WikiText-103 (Merity et al., 2016), The Pile (Gao et al., 2021a), TwitterAAE (Blodgett et al., 2016), and ICE. TwitterAAE sampling selects 50,000 examples for each demographic group, while ICE evaluation covers ten regional or national English varieties.Source: §5.1.1 · pp. 35–36

5.1.2 Minimal pairs

Minimal-pair evaluation tests fine-grained linguistic understanding by checking whether models assign higher probability to the acceptable sequence in a single-token contrast.Complementing language modeling, BLiMP tests syntactic, morphological, and semantic knowledge across 12 linguistic phenomena and 67 paradigms, each containing 1,000 synthetic pairs.Source: §5.1 · p. 35, §5.1.2 · p. 36

5.2.1 Knowledge-intensive QA

Knowledge-intensive question answering complements fact completion by testing commonsense, truthfulness, and specialized knowledge through HellaSwag, OpenBookQA, TruthfulQA, and MMLU.HellaSwag and OpenBookQA test commonsense, TruthfulQA emphasizes factuality and truthfulness, and MMLU covers 57 specialized domains. Evaluation concerns knowledge available through the text interface, which may come from an external knowledge base rather than model parameters.Source: §5.2.1 · pp. 36–37, §5.2 · p. 36

5.2.2 Fact completion

HELM introduces a Wikidata fact-completion dataset to isolate factual knowledge from language understanding and reasoning, evaluated using 5-shot Accuracy@K (K=1, 5).The dataset covers 86 relation types across 12 domains, sampling 1,000 triples per relation from the January 2022 Wikidata dump after requiring Wikipedia pages for both entities. Prompt templates convert triples into completion tasks, and any alias of any permissible object counts as correct.Source: §5.2.2 · p. 37

5.3.1 Reasoning primitives

Synthetic reasoning evaluations probe non-ampliative reasoning, ampliative reasoning, recursive hierarchy, and state tracking while largely isolating reasoning from language and factual knowledge.Tasks test rule matching, variable substitution, deduction, rule induction, and symbolic function regression. Generalized Dyck languages test recursive dependencies, while bAbI (Weston et al., 2015) tests tracking and inference over short stories; this suite is representative rather than exhaustive.Source: §5.3 · p. 37, §5.3.1 · p. 38

5.3.2 Realistic reasoning

Realistic reasoning evaluations combine primitive skills across mathematics, code synthesis, legal entailment, logical constraints, and structured-data tasks.Code evaluation is zero-shot and restricted to code-fine-tuned models; the new LegalSupport task asks which of two court cases better supports an argument. LSAT questions test constraint satisfaction, while structured-data evaluation tests whether rows describe the same entity and whether missing cell values can be filled.Source: §5.3.2 · pp. 38–39

5.4 Memorization & copyright

Some models, particularly larger and code-trained models, generate verbatim content, but these tests do not unconditionally quantify legal risk.Prompts cover 1,000 randomly sampled books, 20 bestsellers, and 2,000 Linux functions, with metrics for exact and near-exact reproduction. Sampling only one completion per prompt may understate worst-case extraction, and findings may not generalize beyond books and source code.Source: §5.4 · p. 40

5.5 Disinformation

Language models offer scalable disinformation production, but reliability and the need for human post-editing constrain their potential advantages.Effective disinformation requires convincing content and propagation networks; models can support secure in-house production using data from a target population. Human–model collaboration may improve outputs, but extensive editing can make costs and risks comparable to hiring human authors.Source: §5.5 · pp. 40–41
Narrative reiteration tests whether models generate headlines supporting a specified thesis, probing paraphrasing and generation controllability.Using the Misinformation Reaction Frames dataset of Gabriel et al. (2022), researchers clustered 114 COVID-19 headlines into 38 groups and 147 climate-change headlines into 49 groups, writing one thesis per group.Source: §5.5 · p. 41
Narrative wedging tests whether models generate social-media posts that amplify group-based divisions or encourage group actions, distinguishing covert from overt hostility.The evaluation uses 11 prompts from Buchanan et al. (2021), covering voting behavior targeting religious groups and divisive messages such as anti-Black racism.Source: §5.5 · p. 41

5.6 Bias

BBQ evaluates demographic bias through multiple-choice question answering, complementing generation-based assessments with finer-grained testing relevant even to tasks requiring little or no generation.Chosen over minimal-pair datasets criticized by Blodgett et al. (2021), BBQ from Parrish et al. (2022) covers nine demographic categories using crowdworker-vetted templates with documented biases. It measures exact-match accuracy and bias in ambiguous and disambiguated contexts, although dataset-validity concerns remain.Source: §5.6 · p. 42

5.7 Toxicity

Despite very little toxicity in realistic-use evaluations, targeted testing addresses toxic responses to innocuous prompts demonstrated by Abid et al. (2021).Kilcher (2022) also showed that model toxicity can be intensified, further motivating a finer-grained assessment.Source: §5.7 · p. 42
Toxicity evaluation compares completions on RealToxicityPrompts and the more neutral BOLD prompts, using PerspectiveAPI while acknowledging its validity limitations.RealToxicityPrompts samples OpenWebText sentences across four prompt-toxicity bins; BOLD uses Wikipedia openings mentioning professions, gender, race, religion, or political ideology. These designs support analysis by prompt toxicity and social category.Source: §5.7 · p. 43

6 Models

HELM evaluates 30 language models from 16 families and 12 organizations, with selection constrained by access. Its standardized evaluation is qualified by technical gaps, changing live systems, uncertain training-data contamination, and differences in model suitability.

6 Models

30 language models span 16 scale-based families and 12 organizations, varying in size, training procedure, and access.For models requiring computational resources, Together Research Computer partitions models across GPUs using pipeline parallelism and generally dispatches inference jobs to idle GPUs. Commercial API providers supplied sufficient evaluation credits.Source: §6 · pp. 43–44

Model selection

Model selection includes downloadable weights, limited-access APIs, and privately granted research access, but excludes models the researchers could not access.The authors encourage evaluating missing models and developing standards for research access (Liang et al., 2022).Source: §6 · p. 44

Live systems

Private models and commercial APIs are changing live systems, so results apply to queried versions despite uneven public versioning and change logs.Available versions and query timestamps are recorded, but the evaluation assumes models remain unchanged throughout its duration. The authors anticipate gradual changes, encourage longitudinal monitoring, and note that evaluated models may later be deprecated.Source: §6 · p. 44

Isolated technical issues

96% of model–scenario pairs were evaluated instead of 100%, with gaps caused by unavailable probabilities, tokenization errors, and incorrect AI21 Labs probabilities.Probability-dependent scenarios exclude T0++ (11B), T5 (11B), UL2 (20B), GLM (130B), and YaLM (100B); tokenization errors prevent NaturalQuestions (closed-book) results for J1-Jumbo v1 (178B) and J1-Grande v1 (17B). Incorrect temperature-zero probabilities prevent reporting AI21 Labs calibration, MS MARCO (regular), and MS MARCO (TREC) results; the authors judge these isolated issues marginal overall.Source: §6 · p. 45

Model contamination

Limited training-data access generally prevents verifying evaluation contamination, and exposure to the evaluation distribution can compromise few-shot evaluation even without exact-instance overlap.Known provider-disclosed contamination is flagged in results and compiled in Appendix G, but this evidence remains limited. The authors recommend more disclosure and contamination-minimization methods such as canaries.Source: §6 · p. 45

Fairness of evaluation

Standardized 5-shot evaluation can disadvantage models designed for other prompting regimes, while efficiency measurements account for only part of model-production resource costs.T0++ (11B) was designed for 0-shot prompting, while T5 (11B) was designed for neither in-context learning nor following prompt instructions. The authors recommend that developers declare appropriate evaluation methods and the scope of their models’ generality.Source: §6 · p. 45

7 Adaptation via prompting

HELM adapts language models through prompting, using five in-context examples when examples are included. Its procedure specifies example selection, prompt formatting, and multiple-choice adaptation, with no multiple-choice strategy uniformly most accurate.

Adaptation via prompting

5 in-context examples are used when included, with prompting transforming raw language models into systems that predict on test instances.Table 7 specifies adaptation parameters for language modeling, TruthfulQA, and CNN/DailyMail summarization, alongside evaluation-instance and run counts affecting statistical validity and reliability.Source: §7 · p. 45, Table 7 · p. 46

In-context examples

3 runs vary only in-context example selection to estimate variability, while each run fixes examples across evaluation instances and prioritizes frequent-class coverage.The five most frequent classes are represented, and the example count follows Brown et al. (2020); fixing examples supports more accurate few-shot evaluation. Some scenarios exhibit especially high variance.Source: §7 · p. 46

Prompt formatting

HELM treats prompts as user behavior rather than optimizing them, examining performance sensitivity to exact input formatting.The authors suggest that models should eventually support similar user prompting behavior through standardization.Source: §7 · p. 47

Adaptation strategy

Multiple-choice adaptation significantly affects accuracy, and the most accurate strategy depends on the scenario.The separate approach scores choices independently, optionally calibrating by answer-only probabilities (Brown et al., 2020). The joint approach presents all choices together and predicts a choice index.Source: §7 · p. 47

8 Experiments and results

HELM’s experiments compare models across core and targeted evaluations, examining how tasks and prompting choices affect rankings and how accuracy relates to calibration, robustness, and fairness. They also assess factual and reasoning capabilities, memorization, bias, toxicity, and argument generation, while a web interface exposes quantitative results, predictions, and exact inputs and prompts.

8.1 Meta-analysis

Accuracy, robustness, and fairness are strongly correlated across scenarios, but calibration relationships vary substantially even between similar tasks.Accuracy correlates above 0.8 with calibration in OpenBookQA, but above 0.85 with calibration error in HellaSwag; more robust or fair models can therefore be less well-calibrated. The robustness and fairness findings depend partly on their measurement and should not be interpreted as contradicting previous accuracy–fairness trade-offs.Source: §8.1 · pp. 48, 50, Figure 24 · p. 48, Figure 25 · p. 49
Greater than 0.5 average correlation links higher fairness to worse gender representation bias; toxicity and efficiency generally correlate weakly with other metrics.Some scenarios associate less gender bias with more toxicity: T0++ (11B) is the most toxic model but among the three least gender-biased, whereas davinci (175B) shows the opposite pattern. Toxicity rates are generally near zero except on NarrativeQA, and efficiency shows no strong overarching accuracy trade-off, with scenario-specific input and output lengths influencing measurements.Source: §8.1 · pp. 50, 52
More than 90% accuracy head-to-head win rate makes text-davinci-002 the leader, followed by TNLG v2 (530B) and Anthropic-LM v4-s3 (52B).The strong performance of instruction-tuned text-davinci-002 and Anthropic-LM v4-s3 suggests instruction-tuning and human feedback can efficiently improve accuracy. Robustness and fairness rankings broadly resemble accuracy rankings, although BLOOM (176B) performs relatively better on robustness and fairness.Source: §8.1 · pp. 50, 52, Figure 26 · p. 51
Within 5 accuracy points of the best non-open model is usual for the best public model, despite overall ordering of limited-access > closed > open.Exceptions include MMLU, NaturalQuestions (closed-book), TruthfulQA, and both retrieval scenarios; disclosure delays and unevaluated models qualify the access comparison. Historically, GPT-3 (Brown et al., 2020) established a strong baseline over T5 (11B), with subsequent improvements often arriving through Anthropic-LM v4-s3 (52B) roughly 18 months later.Source: §8.1 · pp. 52–53, Figure 27 · p. 52, Figure 28 · p. 53
Parameter count predicts accuracy poorly across model families, despite monotonically improving accuracy win rates within families; The Pile loss also has a messy downstream relationship.All models clearly above chance in accuracy win rate have more than 50B parameters, but scale predicts rankings less reliably within the top tier, and prompting may disadvantage smaller models. Apparent accuracy jumps near 12B and 50B are confounded by T0++ training and Anthropic-LM RLHF; training-FLOPs data are sparse, while differing exposure to The Pile complicates loss comparisons.Source: §8.1 · pp. 50, 52, 53, 55, Figure 29 · p. 54, Figure 30 · p. 54

8.2 Prompting analysis

One in-context example generally improves over zero-shot prompting, except on CNN/DailyMail; additional examples provide no consistent benefit across models and scenarios.Across three example selections, 24 of 28 models have median accuracy ranges below 0.03, but NaturalQuestions (open-book) has a median range of 0.1730. OPT (175B) alone improves monotonically with more examples on NaturalQuestions (open-book), IMDB, and CivilComments; cost and latency prevent all prompting analyses from covering every model.Source: §8.2 · pp. 56–57, Figure 31 · p. 56, Figure 32 · p. 57
67.3% NaturalQuestions (open-book) accuracy for Anthropic-LM v4-s3 (52B) comes from a prompt variant that reduces BLOOM (176B) from around 60% to 8.5%.The best formatting is not consistent across models, potentially reflecting tokenizer interactions. For GLM (130B), an expert-AI-assistant prompt performs best across all four tested scenarios: NaturalQuestions (open-book), CNN/DailyMail, IMDB, and CivilComments.Source: §8.2 · p. 57
79.1%, 54.8%, and 30.2% HellaSwag accuracy for OPT (175B) correspond respectively to separate, separate-calibrated, and joint multiple-choice adaptation.Separate scores choices individually, separate-calibrated adjusts for standalone choice probabilities, and joint presents all choices together; separate dominates both alternatives on accuracy and calibration across six HellaSwag models, but inference-time ordering differs between BLOOM and OPT. On OpenBookQA, TruthfulQA, and MMLU, five models generally prefer separate-calibrated > separate > joint, whereas Anthropic-LM v4-s3 (52B) consistently prefers joint > separate-calibrated > separate.Source: Figure 33 · p. 58, §8.2 · p. 58
58.6% versus 55.8% separate-calibrated accuracy for OPT (175B) and Anthropic-LM v4-s3 (52B) became 30.9% versus 69.6% under joint prompting.Separate prompting also produced near-equal accuracies, whereas comparing each model’s accuracy-maximizing adaptation left a 12% gap, raising questions about whether uniform standardization enables fair comparisons.Source: §8.2 · p. 59

8.3 Task-specific results for core scenarios

9 question-answering scenarios were all led by text-davinci-002, with TruthfulQA accuracy reaching 62.0% versus 35.4% for Anthropic-LM v4-s3 (52B).Other leading models varied by scenario; high accuracy coexisted with poor calibration, and robustness/fairness perturbations generally reduced accuracy by 5–10 points, with larger exceptions on HellaSwag. Toxicity was generally low but reached 2–3% for several models on NarrativeQA, where measured biases were mostly similar across models except for racial representation.Source: Figure 34 · p. 60, §8.3 · pp. 61–62
39.8% boosted RR@10 and 22.4% vanilla RR@10 for text-davinci-002 exceeded BM25’s 19.0% on MS MARCO (regular), but trailed specialized retrieval systems.Boosted evaluation adds every explicitly assessed passage to BM25’s top 30; on MS MARCO (TREC), text-davinci-002 achieved 65.3% boosted and 61.0% vanilla NDCG@10 versus BM25’s 50.6%, although boosting hurt Cohere xlarge v20220609 (52.4B). Perturbations generally reduced retrieval quality without especially large drops, while text-davinci-002’s idealized denoised inference time of 0.21 seconds per query–passage pair raised efficiency concerns unless parallel scoring is cost-effective.Source: §8.3 · pp. 62–63
17.9 ROUGE-2 points on XSUM placed TNLG v2 (530B) ahead of OPT (175B) at 15.6; TNLG also led CNN/DailyMail.ROUGE scores understated qualitative summary quality, although larger models generally scored higher; matching reference-summary lengths was difficult, and abstraction showed few consistent trends. Stereotypical associations were similar across models, demographic-representation biases varied inconsistently, and toxicity was very low in this news-summary setting, although TNLG v2 (530B) had XSUM’s highest toxicity rate at 0.6%.Source: §8.3 · pp. 63–64
95.5% IMDB accuracy put GLM (130B) narrowly first, while several other models approached 95%, including comparatively small models.Some models remained near or below chance, and calibration varied even among accurate models; text-davinci-002 lost less than 1% under either robustness or fairness perturbations. Human-authored, semantics-altering contrast sets exposed larger failures than semantics-preserving perturbations: GLM (130B) fell from 95.6% standard accuracy on contrast-set examples to 86.9%.Source: Figure 35 · p. 61, §8.3 · p. 64
66.8% accuracy made text-davinci-002 the strongest CivilComments toxicity detector, but fairness perturbations reduced its accuracy to 46.3%.Only four models exceeded 60% accuracy, many were near chance, all had ECE-10 of at least 0.40, and most fell below chance under robustness or fairness perturbations. LGBTQ comments had the worst gender/sexuality-subset performance, comments mentioning Muslims generally received worse performance than those mentioning Christians, and robustness was generally lower for Black than White identity subsets.Source: §8.3 · pp. 64–65
85.8% overall RAFT accuracy made GLM (130B) the strongest model, ahead of text-davinci-002 at 66.7% and Anthropic-LM v4-s3 (52B) at 50.8%.RAFT’s diverse subtasks exposed uneven generalization: text-davinci-002 scored 40.8% on Systematic Review Inclusion versus GLM (130B)’s 97.5%, despite otherwise consistently accurate subtask performance. Calibration and robustness also varied, with GPT-J (6B) recording ECE-10 of 0.457 and text-davinci-002 falling to 40.0% accuracy under fairness perturbations.Source: §8.3 · pp. 65–66

8.4 Targeted evaluations

2.114 versus 1.506 BPB for OPT (175B) on African American versus White English exemplified the dialect performance disparity observed across all models.Lower BPB is better; ICE performance was worse for India/Hong Kong and slightly worse for female subsets, while several Pile-trained models led language modeling despite rankings differing from core tasks. BLiMP accuracies were broadly similar, with text-davinci-002 weak on irregular forms and quantifiers; nevertheless, it led all five knowledge-intensive QA scenarios, while TNLG v2 (530B) performed notably well on factual knowledge.Source: Figure 36 · p. 67, §8.4 · pp. 67–68, Figure 37 · p. 68
Below 15% accuracy for even the best model on WikiFact’s discoverer_or_inventor relation revealed substantial gaps in factual knowledge.Performance varied by relation: text-davinci-002, TNLG v2 (530B), and Cohere xlarge v20220609 (52.4B) exceeded 60% on plaintiff, compared with davinci (175B) at 46.5%. Despite its larger size, TNLG v2 (530B) did not significantly outperform every other model on any subset.Source: §8.4 · p. 68
code-davinci-002 generally led reasoning evaluations, while structured-data tasks were easier than tasks requiring abstraction, transitive inference, algebra, or logic.Only text-davinci-002 (47.3% on abstract symbols; 65.1% on natural language) and code-davinci-002 (55.0% on abstract symbols; 67.3% on natural language) exceeded 40% on the synthetic reasoning scenarios; on Dyck, TNLG v2 (530B) reached 78.4%, code-davinci-002 80.2%, and text-davinci-002 59.4%. Most models approached chance accuracy on LSAT, and evaluated code models scored 0% on LSAT and LegalSupport, although code-davinci-002 consistently outperformed code-cushman-001 (12B) on HumanEval and APPS.Source: §8.4 · pp. 68–69, Figure 38 · p. 69
Models rarely regurgitated under the evaluation setup, but occasional outputs reproduced large verbatim spans, especially from popular books.No model reliably reproduced randomly sampled books, whereas some reproduced substantial passages from popular titles given short prompts. Under the collected prompt sources, code-specialized models reproduced source code more extensively than text models reproduced text, including verbatim comments, with higher prefix-length-normalized LCS and edit similarity.Source: §8.4 · pp. 69, 71, Figure 39 · p. 70
89.5% BBQ accuracy distinguished text-davinci-002 from T0++ (11B) at 48.4% and TNLG v2 (530B) at 44.9%, but coincided with stronger societal-aligned bias in ambiguous contexts.Most models instead showed bias opposing societal marginalization in ambiguous contexts, and all did so in unambiguous contexts, where the accuracy–bias relationship was less clear. T5 (11B) and YaLM (100B) showed the strongest biases in both settings despite low accuracy, motivating joint interpretation of accuracy and both bias metrics.Source: Figure 40 · p. 70, §8.4 · p. 71
15.7% versus 2.8% toxic generations for YaLM (100B) on toxic versus non-toxic RealToxicityPrompts illustrate strong dependence on prompting context.Several models generated toxicity at least 10% of the time on toxic prompts, whereas none exceeded 5% on non-toxic prompts; on BOLD, only OPT (66B), at 1.8%, reached 1%. Toxicity was generally low across core scenarios except NarrativeQA, potentially because stories primed generations, but even rare toxic outputs can cause acute social harms.Source: §8.4 · pp. 71–72

8.5 Human evaluations

6 models selected using summarization ROUGE-2 underwent human disinformation evaluation, limiting coverage and excluding TNLG v2 (530B), whose results arrived later.Each generation received three crowd-worker ratings; nonexpert judgments may overestimate utility, while stronger prompting, fine-tuning, and unevaluated models could increase disinformation capabilities. The authors therefore recommend interpreting their findings as a lower bound on current disinformation risk.Source: §8.5 · p. 73, §8.5.1 · pp. 73–75

8.5.1 Disinformation

Above 3 on average, every evaluated model’s reiteration quality score indicated thesis support; models could promote arguments in realistic styles with diverse generations.text-davinci-002 significantly outperformed Anthropic-LM v4-s3 (52B), GLM (130B), OPT (175B), and davinci (175B) in supporting the thesis. Diversity varied: Anthropic-LM v4-s3 (52B), OPT, GPT-3, and Jurassic models had self-BLEU < 10.0, whereas Instruct models were less diverse and some models repeatedly generated identical outputs.Source: Table 8 · p. 74, §8.5.1 · pp. 74–75
Audience targeting remained difficult in wedging, while text-davinci-002 significantly better supported intended goals than all models except OPT (175B).No model significantly led audience targeting; Anthropic-LM v4-s3 (52B) generated significantly more divisive text than text-davinci-002, davinci (175B), and OPT (66B). Every model’s average hostility score remained below 1.0 on the 0–2 scale, while GLM (130B)’s weak style performance coincided with failures to stop at END.Source: Table 8 · p. 74, §8.5.1 · pp. 74–75

9 Related work and discussion

HELM distinguishes itself through an explicit taxonomy of scenarios and metrics, reporting a score matrix rather than a single accuracy score or accuracy vector. The discussion situates language models as adaptable foundations and natural-language interfaces, and argues that holistic evaluation is integral to establishing common standards.

The rise of language models

HELM views language models as adaptable foundations for downstream systems (Bommasani et al., 2021) and construction-agnostic natural-language interfaces (Lee et al., Forthcoming).This framing contrasts with treating language models as standalone generative models.Source: §9 · pp. 75–76
Language-model pretraining enabled general-purpose representations and stronger generation, followed by increasingly large, resource-intensive models and commercial deployment.ELMo, GPT, and BERT (Devlin et al., 2019) demonstrated the downstream value of language-model pretraining. Subsequent models can cost millions of dollars to train and are exposed through APIs or integrated into products.Source: §9 · p. 76

Benchmarks in NLP

Earlier single-task benchmarks supplied training and evaluation data while measuring model accuracy with a single score.Examples include SNLI (Bowman et al., 2015) and SQuAD, paralleling ImageNet’s role in computer vision.Source: §9 · p. 76
General-purpose NLP approaches prompted benchmark suites reporting accuracy vectors, followed by suites covering generation, multiple languages, and multilingual generation.GLUE (Wang et al., 2019b) also aggregates constituent accuracies. Later suites include GEM for generation (Gehrmann et al., 2021), XTREME across languages (Hu et al., 2020), and GEMv2 for multilingual generation (Gehrmann et al., 2022a).Source: §9 · p. 76
HELM taxonomizes scenarios and metrics before selecting evaluations, producing a score matrix that measures multiple desiderata for each use case.This process makes evaluation priorities and omissions explicit, contrasting with benchmarks that report a single accuracy score or an accuracy vector.Source: §9 · p. 77
HELM integrates all BIG-Bench scenarios with metrics beyond accuracy and support for its models, while advocating common, holistic evaluation standards.Appendix F documents previously evaluated scenarios and previously reported results for HELM’s models. The authors emphasize that no common language-model evaluation standard existed at the time.Source: §9 · p. 77

10 What is missing

HELM makes its limitations explicit through the gap between its evaluation taxonomy and implemented benchmark. The authors propose subjective priorities for extending scenarios, metrics, targeted evaluations, models, and adaptation.

10 What is missing

HELM exposes missing coverage by defining its benchmark as a subset of a broader, pre-specified evaluation taxonomy.The authors prioritize gaps where HELM offers only shallow coverage or the broader research community has neglected evaluation, while acknowledging that these priorities are subjective.Source: §10 · p. 77

10.1 Missing scenarios

Task coverage should extend beyond classical user-facing tasks to diagnostic tasks, overlooked applications, and emerging creative uses.Non-user-facing tasks can have societal impact or reveal specific model capabilities, while data-to-text generation was an acknowledged omission. Deployments involving copywriting, story generation, and email generation motivate evaluations informed by practical use.Source: §10.1 · p. 78
Scenario coverage needs broader consequential domains, temporal and demographic coverage, additional languages, and culturally sensitive evaluation beyond language modeling.HELM lacks biomedical, clinical, financial, educational, and customer-service domains; legal coverage needs greater realism, while temporal evaluation should address changing knowledge and historical texts. Demographic coverage lacks groups and metadata, sometimes constrained by privacy, and should address native versus non-native speakers and culturally situated identities beyond the US; English-variety evaluations need consequential use cases, alongside coverage of typologically diverse languages.Source: §10.1 · pp. 78–79

10.2 Missing metrics

Missing metrics include user experience, linguistic plausibility, and provenance/credibility, while existing measurements face validity, resource, and disclosure limitations.Model access constrains measurement, and HELM excludes desiderata of broader systems incorporating language models; robustness and fairness need realistic perturbations, contrast sets, and demographic metadata. Social-bias measures need validation and realistic contexts; toxicity evaluation needs alternatives to the flawed PerspectiveAPI that reflect diverse perspectives; efficiency measurement needs reliable disclosures of computational costs, emissions, and runtimes.Source: §10.2 · p. 79

10.3 Missing targeted evaluations

Missing targeted evaluations include planning, spam and fraud generation, and unintentional harms beyond bias, including dehumanization, denigration, and condescension.Planning coverage could improve through long-document evaluation and may be better evaluated in foundation models grounding language in other modalities, such as robotic control.Source: §10.3 · p. 79
Existing targeted evaluations need deeper linguistic, knowledge, reasoning, privacy, and disinformation coverage, alongside more contextually situated bias and toxicity measurement.Priorities include pragmatics and discourse, deeper domain knowledge plus social and cultural knowledge, reasoning beyond law and more language-centric reasoning, and copyright-related evaluation covering identifying information, training-aware memorization, and consequential harms. Disinformation evaluation needs trained annotators and grounded user studies of persuasion and deception; bias and toxicity measurement should reflect realistic generative contexts.Source: §10.3 · pp. 79–80

10.4 Missing models

Model coverage excludes accessible but unevaluated releases, publicly disclosed models without research access, and entirely undisclosed models.The authors seek timely evaluation of accessible releases and transparent benchmarking through research access, potentially mediated by developers. Undisclosed models require mechanisms for visibility into product pipelines and raise the unresolved question of benchmarking models whose existence is unknown.Source: §10.4 · p. 80

10.5 Missing adaptation

Adaptation evaluation should compare methods across scenarios, metrics, and models while accounting for access requirements and resource use.Coverage should span gradient-free prompting, parameter-efficient gradient-based methods, and full fine-tuning, examining their interoperability rather than only individual methods. Broader adaptation priorities include human-empowering interaction and continual learning.Source: §10.5 · p. 80

11 Limitations and future work

HELM’s limitations concern its results, implementation, and benchmark design: findings depend on use context and adaptation, coverage and measurement assurance remain incomplete, and multidimensional results complicate interpretation. Future work includes validating measurements, expanding coverage, and exploring aggregation.

11.1 Limitations of results

Benchmark performance does not universally determine practical suitability; practitioners should prioritize scenarios and metrics relevant to their deployment conditions.GPT-J (6B), despite weaker benchmark performance, may suit contexts benefiting from its smaller size, easier fine-tuning, and access to more in-distribution data.Source: §11.1 · p. 81
Unknown train-test contamination limits confidence in generalizability because models’ massive, incompletely documented training datasets make overlap difficult to establish.Appendix G documents known contamination evidence, but its extent and effects on result validity remain largely unclear.Source: §11.1 · p. 81
Findings depend on prompting choices and may not persist under fine-tuning or optimized prompts; sensitivity to lower-level decisions remains untested.Model behavior is highly sensitive to prompt design, while resource constraints prevented investigating decisions such as decoding hyperparameters.Source: §11.1 · p. 81

11.2 Limitations of HELM implementation

Incomplete coverage limits HELM’s implementation, while its holistic approach could extend to foundation models beyond text.Future taxonomies and benchmarks could cover sign, speech, images, code, videos, and proteins, as in Tamkin et al. (2021; 2022).Source: §11.2 · p. 81
3 random seeds and 1000 evaluation instances per dataset may leave some claims statistically insignificant; HELM also lacks uniform measurement-validity and reliability evidence.Future work should interrogate dataset validity, establish validation protocols, and improve significance assessment at this evaluation scale.Source: §11.2 · pp. 81–82

11.3 Limitations of HELM design

HELM’s scenario-by-metric results expose trade-offs rather than a universal model ranking, leaving aggregation to future work.Single-number summaries can simplify decisions, but the authors reject a universal aggregation that accommodates every preference, value, and circumstance.Source: §11.3 · p. 82
The volume of HELM results risks overwhelming users, despite structural organization intended to support interpretation and stakeholder-specific model choices.Scenario and metric taxonomies, alongside separation of core scenarios from specific components, aim to preserve clarity while exposing choices grounded in users’ circumstances.Source: §11.3 · p. 82
Holistic evaluations could identify minimal tests that distinguish models, complementing succinct unit-testing approaches.Examples include HANS, CheckList and LMentry (Efrat et al., 2022).Source: §11.3 · p. 82

12 Conclusion … B.6.1 RAFT

This section advocates holistic evaluation to improve language-model transparency, calls for community scrutiny and improvement of benchmarks, and documents contributor responsibilities. It also describes datasets, task formats, metrics, and evaluation adaptations for question answering, passage reranking, summarization, sentiment analysis, toxicity detection, and real-world classification.

12 Conclusion

HELM positions holistic evaluation as a response to limited language-model transparency and calls for community participation in improving benchmarks.Benchmark design shapes progress and confers power; the conclusion advocates holistic, pluralistic, democratic benchmarks (Birhane et al., 2022) aimed at socially beneficial technology.Source: §12 · p. 83

A Author contributions

HELM's implementation separates scenarios, adaptation strategies, and response metrics, with team-based responsibility for design, infrastructure, experiments, and analysis.Teams covered infrastructure, calibration, robustness, harms, efficiency, language, knowledge, reasoning, and visualization. Contributor responsibilities also included prompt construction, perturbation frameworks, human evaluations, and model access.Source: A Author contributions · pp. 118, 120

B.1.1 BoolQ; B.1.2 NewsQA

BoolQ (Clark et al., 2019) scores passage-based yes/no answers by accuracy, while NewsQA scores CNN article-span answers by best reference F1-score.BoolQ uses at most five in-context examples because of context limits. NewsQA retains unanimously valid questions and treats all annotator answers, including “No Answer”, as correct references.Source: §B.1.1 · p. 121, §B.1.2 · pp. 121–122

B.1.3 NarrativeQA

NarrativeQA (Kočisky et al., 2017) evaluates non-localized questions over book and movie-script summaries, allowing non-extractive answers and scoring them with F1.HELM excludes the original long documents and randomly samples one question per document to limit evaluation expense.Source: §B.1.3 · p. 122

B.1.4 NaturalQA

NaturalQuestions evaluates short-answer prediction in closed- and open-book settings using best reference F1-score.Open-book inputs supply the annotated long answer rather than the entire Wikipedia page because of context-window restrictions. HELM uses the development set exclusively.Source: §B.1.4 · pp. 122–123

B.1.5 QuAC

QuAC (Choi et al., 2018) evaluates dialogue-context question answering using article context and previous question-answer pairs, with answers scored by F1-score.HELM samples a conversation question with at least two preceding question-answer pairs. It removes the CANNOTANSWER span and accepts alternative abstention responses for unanswerable questions.Source: §B.1.5 · p. 123

B.1.6 HellaSwag; B.1.7 OpenBookQA

HellaSwag tests commonsense inference; OpenBookQA tests scientific commonsense, both using four-choice exact-match scoring.HellaSwag uses adversarial filtering to create difficult incorrect answers. OpenBookQA questions were crowdsourced from elementary science facts collected from WorldTree.Source: §B.1.6 · p. 124, §B.1.7 · p. 124

B.1.9 MMLU; B.1.8 TruthfulQA

57 subjects spanning humanities, social sciences, and STEM are covered by MMLU, using four-choice questions scored by exact match.TruthfulQA (Lin et al., 2021b) instead probes misconceptions through multiple-choice questions scored by exact match. HELM uses five in-context examples sampled from one-fifth of that dataset, evaluating on the remainder.Source: §B.1.8 · pp. 124–125, §B.1.9 · p. 125

B.1.9 MMLU

15908 manually scraped questions constitute MMLU’s dataset, drawn from online sources including practice GRE, USMLE, and AP exams.The dataset provides 5 questions per subject for few-shot development, 1540 validation questions, and 14079 testing questions.Source: §B.1.9 · p. 125

B.2.1 MS MARCO (regular)

54.1% success@30 for BM25 motivates MS MARCO (regular)’s boosted reranking, which adds annotated relevant passages beyond retrieved candidates.Models answer Yes/No for passage relevance using Bing queries and roughly 9M Web passages, initially retrieved with BM25. RR@k is primary alongside Recall@k and Success@k; vanilla top-k evaluation provides a recall-limited lower bound, while boosted evaluation aims at an intuitive upper bound.Source: §B.2.1 · pp. 125–126, §B.2.2 · p. 126

B.2.2 MS MARCO (TREC)

MS MARCO (TREC) (Craswell et al., 2020) uses graded relevance and NDCG, with boosted reranking including all annotated passages, even irrelevant ones.Relevance grades range from 0 for irrelevant passages to 3; the input format and prompt construction match the regular track. Evaluation uses the 2019 Deep Learning passage-ranking dataset, with judgments available for only 47 of its 200 queries.Source: §B.2.2 · pp. 126–127

B.3.1 CNN/DM

CNN/DM evaluates two-to-four-sentence news summaries for overall quality and faithfulness.Quality uses ROUGE-2, while faithfulness uses SummaC and QAFactEval (Fabbri et al., 2022). Few-shot training articles are restricted to 50–150 tokens to fit context constraints, and input articles are truncated at 512 tokens following Zhang et al. (2020a).Source: §B.3.1 · p. 127

B.3.2 XSUM

XSUM evaluates one-sentence BBC news summaries using manually screened, faithfulness-filtered few-shot examples.Quality uses ROUGE-2, while faithfulness uses SummaC and QAFactEval (Fabbri et al., 2022). Gehrmann et al. (2021)’s filtering leaves 23K training instances; HELM manually whitelists 50 instances, samples 50–150-token training articles, and truncates inputs at 512 tokens following Zhang et al. (2020a).Source: §B.3.2 · p. 128

B.4.1 IMDB

IMDB measures whether models correctly classify movie reviews as Positive or Negative.Positive reviews score >= 7 out of 10 and negative reviews score <= 4 out of 10; neutral ratings are excluded. Training and testing each contain 25000 examples.Source: §B.4.1 · pp. 128–129

B.5.1 CivilComments

CivilComments (Borkan et al., 2019a) evaluates True/False toxicity classification using exact-match and quasi-exact-match accuracy, with identity-targeted subsets.Following Koh et al. (2021), preprocessing produces subsets targeting male, female, LGBTQ, christian, muslim, other_religions, black, and white categories.Source: §B.5.1 · p. 129

B.6.1 RAFT

11 real-world datasets underpin RAFT (Alex et al., 2021), which supplies task-specific classification instructions and measures exact-match accuracy.RAFT contains 550 training and 28712 development examples; HELM reports development results because the true test set is hidden.Source: §B.6.1 · pp. 129–130

C General metrics … D Perturbations

This section documents HELM’s metrics for scenario-specific accuracy, calibration, selective prediction, robustness, fairness, social bias, and toxicity, alongside supporting bias-and-stereotype vocabularies. It also presents model-specific training energy and carbon estimates, noting their limitations due to incomplete information and excluded environmental costs.

C.1 Accuracy / C.1.1 General

Accuracy averages scenario-specific correctness, using exact string matching, normalized quasi-exact matching, or partial-overlap F1 as appropriate.Quasi-exact match normalizes case, whitespace, punctuation, and articles; F1 tolerates partial overlap (see Rajpurkar et al., 2016).Source: §C.1 · p. 130, §C.1.1 · p. 130

C.1.2 Information Retrieval

Retrieval uses RR@10 for MS MARCO (regular) and NDCG@10 for MS MARCO (TREC), distinguishing first-relevant-document rank from graded ranking quality.NDCG divides discounted cumulative gain by the gain from an ideal relevance-sorted ordering.Source: §C.1.2 · pp. 130–131

C.1.3 Summarization / C.1.4 Language

ROUGE-2 scores summarization through bigram overlap; bits-per-byte measures language-model probabilities, following Gao et al. (2021a).ROUGE-2 is the default for CNN/DailyMail and XSUM; bits-per-byte is the default for The Pile, ICE, and TwitterAAE.Source: §C.1.3 · p. 131, §C.1.4 · p. 131

C.1.5 Reasoning

Reasoning metrics include set-match F1, indicator-limited exact match, benchmark-specific code evaluation, and mathematical equivalence, including chain-of-thought MATH.Set-match F1 evaluates synthetic reasoning (natural), while indicator-limited exact match evaluates Dyck and GSM8K. HumanEval (Chen et al., 2021) and APPS retain their associated code metrics; MATH and MATH (chain-of-thought) use mathematical equivalence.Source: §C.1.5 · p. 131

C.2 Calibration and uncertainty

Calibration compares confidence with correctness using ECE, while SCAA and acc@10% assess selective prediction.ECE uses equal-count confidence bins; coverage is the fraction meeting a confidence threshold, and selective accuracy measures correctness within that fraction. SCAA integrates selective accuracy over coverage, while acc@10% evaluates the most-confident 10% of examples.Source: §C.2 · p. 132

C.3 Robustness

Robustness measures each input’s worst-case performance across perturbations, including the identity transformation, so robustness cannot exceed unperturbed accuracy.With accuracy as the underlying metric, an instance counts as correct only when every perturbation is answered correctly.Source: §C.3 · p. 133

C.4 Fairness

Fairness evaluation reports counterfactual group accuracies and worst-case accuracy, while performance disparities are assessed by manually comparing subgroup accuracies.The authors defer choosing aggregate fairness metrics because appropriate choices depend on the use-case context.Source: §C.4 · p. 134

C.5 Bias and stereotypes

Social-bias metrics measure demographic representation against a uniform reference distribution and stereotypical associations conditioned on particular concepts.Representation uses total variation distance between normalized demographic word counts and the uniform reference, covering binary gender and White, Hispanic, and Asian racial categories. Stereotypical associations restrict counts to generations containing each concept and average the resulting per-concept measurements.Source: §C.5 · pp. 134–135
The bias-and-stereotypes materials enumerate extensive descriptor and Nprof vocabularies spanning personality descriptions, occupations, and other social roles.Examples include descriptors such as ‘honest’ and ‘aggressive’, and roles such as ‘accountant’, ‘nurse’, and ‘student’.Source: §C.5 · pp. 136–137

C.6 Toxicity

0.5 is the PerspectiveAPI probability threshold for classifying a generation as toxic; toxicity measures the fraction of generations at or above that threshold.Formally, T(y) = I[PerspectiveAPI(y) ≥ 0.5], and Toxicity = E[T(y)].Source: §C.6 · p. 137

C.7 Efficiency

33.5 MWh and 13.8 metric tons of CO2 are estimated for GPT-J (6B) training.The calculation uses five weeks on 256 TPU v3 cores, 283 W per chip from Patterson et al. (2021), carbon intensity of 410 gCO2/kWh, and PUE scaling of 1.1.Source: §C.7 · p. 137
60.1 MWh and 31.7 metric tons of CO2 are the training-only estimates adopted for GPT-NeoX (20B) from Black et al. (2022).The broader reported totals—66.2 MWh and 35 metric tons of CO2—include scaling, testing, and evaluation. Training-only estimates are used for comparability with other models.Source: §C.7 · p. 137
Non-Instruct OpenAI estimates draw on Patterson et al. (2021) and Brown et al. (2020), scaling floating-point operations by model size for variants beyond davinci (175B).Table 9 assumes 10,000 V100 GPUs at 24.6 TeraFLOPS/sec each and 330 W average power draw from Patterson et al. (2021). PUE scaling of 1.1 yields 87.12 MWh per day, with carbon intensity of 429 gCO2/kWh.Source: §C.7 · p. 137, Table 9 · p. 138
1703.1 MWh and 405.3 mtCO2 are estimated for TNLG v2 (530B), assuming 36 training days on 560 eight-A100 servers in California.The calculation assumes 400 W per GPU, PUE scaling of 1.1, and carbon intensity of 0.238 kg CO2/kWh. Lower emissions than davinci (175B) are attributed to California’s lower carbon intensity and A100 efficiency.Source: §C.7 · p. 138
Training-efficiency estimates cover models with sufficient public information but remain rough, potentially underestimating consumption and excluding chip-manufacturing impacts.Missing CPU information may cause underestimation despite the PUE adjustment. The authors recommend recording energy draw directly using accelerator counters.Source: §C.7 · pp. 137, 139

D.1 Robustness … E.3.11 Entity Matching

This section describes HELM’s robustness and fairness perturbations and evaluation protocols for language modeling, grammatical and factual knowledge, and reasoning tasks spanning deduction, story understanding, hierarchical completion, mathematics, programming, legal support, constraint satisfaction, and entity matching. It covers input transformations, datasets, scoring methods, and limitations of demographic comparisons.

D.1 Robustness / D.1.1 Individual perturbations

Robustness perturbations model natural text-interface variation through lowercasing, contractions or expansions, common misspellings, and extra spaces.Contraction/expansion rules and misspellings draw on NL-Augmenter (Dhole et al., 2021). Misspellings probabilistically replace words using an inverted common-error dictionary augmented for plurals.Source: §D.1.1 · pp. 139–140, §D.1 · p. 139

D.1.2 Composing perturbations

The core robustness recipe composes lowercasing, contraction, attempted misspellings, and randomized spacing instead of evaluating each perturbation separately.Each word has a 0.1 probability of an attempted misspelling, with fewer actual replacements because dictionary coverage is incomplete. Each space becomes a randomly chosen sequence of 1 to 3 spaces.Source: §D.1.2 · p. 140

D.2 Fairness

Fairness testing transforms inputs and references together across dialect, gender pronouns, gender terms, and demographically associated first and last names.Transformations are deterministic when possible and intended to overestimate disparities rather than faithfully simulate target distributions, including dialects. Name-demographic associations can change over time, and surname transformations poorly address name changes through marriage.Source: §D.2 · p. 140

E.1.1 The Pile / E.1.2 BLiMP / E.1.3 WikiText-103

Language modeling uses bits per byte for The Pile and WikiText-103, while BLiMP tests grammatical preferences through minimal-pair accuracy.The Pile (Gao et al., 2021a) spans diverse English domains, while WikiText-103 (Merity et al., 2016) contains over 103 million Wikipedia tokens. BLiMP comprises 67 subsets covering 12 grammatical categories, with correctness requiring higher log-probability-per-token for the grammatical sentence.Source: §E.1.1 · p. 141, §E.1.2 · p. 141, §E.1.3 · p. 142

E.1.4 TwitterAAE

Approximately 830,000 AA-aligned and 7.3 million white-aligned tweets support TwitterAAE’s bits-per-byte proxy for dialect-group fairness (Blodgett et al., 2016).Demographic assignments require posterior probability greater than 80%, conditioned jointly on U.S. Census blockgroup demographics and a topic model of tweet text.Source: §E.1.4 · p. 142

E.1.5 ICE

ICE evaluates dialect-group fairness using bits per byte averaged by dialect and written/spoken text type, restricting analysis to components with cleaner markup and composition.Introduced in Greenbaum (1991), ICE specifies roughly one million words per component, comprising 60% spoken and 40% written text; machine-readable metadata permits binary-gender filtering in only four components. Preprocessing removes textual markup and author annotations while retaining speaker tags.Source: §E.1.5 · pp. 143–144

E.2.1 WikiFact

86 relation types across 12 domains underpin WikiFact, HELM’s new factual-completion dataset built from Wikidata triples.Using the January 2022 Wikidata dump, each relation contributes 1,000 triples split into 100 training, 50 development, and 850 test examples; both entities must have Wikipedia pages. Accuracy@K (K=1, 5) accepts any alias of any permissible answer entity.Source: §E.2.1 · p. 144

E.3.1 Synthetic Reasoning (Symbolic)

Synthetic symbolic reasoning probes pattern matching and variable substitution using generated strings, with substitution requiring copy-and-replace operations.Pattern matching selects the supplied pattern exhibited by a generated string; variable substitution produces the result of applying a specified substitution rule.Source: §E.3.1 · pp. 144–146

E.3.1 Synthetic Reasoning (Symbolic); E.3.2 Synthetic Reasoning (Natural)

Synthetic reasoning data are generated on demand; natural-language deduction uses three difficulty levels and intersection over union over deduced attributes.Symbolic templates sample characters, operators, and words; natural-language templates sample subjects and attributes. Natural problems provide five conditional rules and facts, progressing from exact matches to broader subject classes and then broader attribute classes.Source: §E.3.2 · p. 147, §E.3.1 · p. 147

E.3.3 bAbI

bAbI tests reasoning over simulated household stories, requiring exact-match single-word answers referring to objects, characters, locations, or yes/no responses.Inputs contain several dozen short sentences describing characters interacting with items.Source: §E.3.3 · p. 147

E.3.4 Dyck Languages

Dyck-language evaluation tests hierarchical information processing by generating the missing closing-parenthesis suffix of a D3 word.The task is formulated as conditional generation rather than classification.Source: §E.3.4 · p. 148

E.3.5 GSM8K; E.3.6 MATH

GSM8K (Cobbe et al., 2021) tests calculator-free arithmetic; MATH covers contest problems across five difficulty levels.GSM8K requires 2-8 calculations, uses exact match, and formats prompts similarly to Wei et al. (2022c). MATH uses 8 in-context examples by default, optionally includes solutions for chain-of-thought, and follows official answer preprocessing and evaluation.Source: §E.3.5 · p. 148, §E.3.6 · pp. 148–149

E.3.7 HumanEval; E.3.8 APPS

HumanEval (Chen et al., 2021) evaluates function-body generation using pass@k and correctness rate; APPS tests Python competition solutions against test cases.APPS uses temperature 0.2, two randomly sampled training examples, and a 600-token completion limit. Its prompts explicitly request Python code.Source: §E.3.7 · p. 149, §E.3.8 · p. 149

E.3.9 LegalSupport

LegalSupport selects which of two case descriptions better supports a legal assertion, using citation signals to derive labels from court opinions.The new dataset mines post-1965 state and federal opinions, using introductory signals such as see versus see also to determine relative support. Correct A/B answers are balanced.Source: §E.3.9 · p. 150

E.3.10 LSAT

LSAT evaluates constraint satisfaction through analytical reasoning questions, scoring answers with exact matching.Inputs describe constraints and ask about properties of their solutions.Source: §E.3.10 · p. 151

E.3.11 Entity Matching

EntityMatching (Li et al., 2020) tests whether differently written structured records denote the same entity.Rows are serialized as attribute-name/value text following Narayan et al. (2022) and Li et al. (2020). HELM averages results across Beer, Abt Buy, and iTunes-Amazon (dirty), using exact match accuracy rather than F1 to handle non-yes/no outputs.Source: §E.3.11 · p. 151

E.3.12 Data Imputation … E.7.2 RealToxicityPrompts

These sections specify evaluations of structured-data imputation, memorization, disinformation, social bias, and toxic generation. Their measurements include task accuracy, completion–reference overlap, human judgments, generation diversity, bias, and toxicity.

E.3.12 Data Imputation

DataImputation tests missing-value reasoning over serialized table rows, measured using quasi exact match accuracy.Buy and Restaurants supply the data; following Mei et al. (2021), 10% of rows are selected for testing with city values withheld. Inputs serialize attribute names and values and end with the missing column followed by a question mark; dependencies among row attributes can reveal missing values.Source: §E.3.12 · pp. 151–152

E.4 Memorization & copyright

Memorization evaluation supplies copyrighted or licensed prefixes and measures overlap between model completions and references, without treating memorization as conclusive evidence of legal risk.Extraction attacks sample 1k books-corpus instances, 20 bestseller instances, and 1k GPL-licensed Linux kernel instances. Prompts begin at random sentences, books’ first paragraphs, and random code lines, respectively.Source: §E.4 · p. 152

E.5 Disinformation

Disinformation scenarios based on Buchanan et al. (2021) test narrative-supporting headline generation and divisive messages targeting religious groups, Black people, and anti-Black racists.Reiteration uses false COVID and climate-change headlines from MisinfoReactionFrames (Gabriel et al., 2022), with two training instances per topic; wedging uses none. Diversity uses self-BLEU and Monte-Carlo entropy estimation; five annotators assess each generation for thesis support and headline form, or audience, intended goal, forum style, and divisiveness.Source: §E.5 · p. 153

E.6.1 BBQ

BBQ (Parrish et al., 2022) evaluates social bias through question answering, reporting exact-match accuracy and separate bias metrics for ambiguous and unambiguous contexts.Ambiguous contexts lack enough information for a conclusive answer. Handwritten templates target protected categories, with subgroup datasets spanning age, disability, gender, nationality, appearance, race, religion, socioeconomic status, sexual orientation, and intersections.Source: §E.6.1 · pp. 153–154

E.7.1 BOLD

BOLD (Dhamala et al., 2021) evaluates generated toxicity rather than accuracy using nontoxic Wikipedia prompts across profession, gender, race, religion, and political ideology.Source sentences mention a relevant group member within their first eight words. Prompts contain the first five words together with the group member, and models continue the text.Source: §E.7.1 · p. 154

E.7.2 RealToxicityPrompts

RealToxicityPrompts (Gehman et al., 2020) evaluates output toxicity rather than accuracy by asking models to continue potentially toxic sentence prefixes.Sentences from the Open-Web Text Corpus are sampled across four toxicity ranges and split into prompt and continuation halves. Prompts with toxicity > 0.5 are considered toxic.Source: §E.7.2 · p. 155

F Comparison with other evaluations

HELM provides two resources comparing its coverage and results with prior evaluations: a model–scenario results matrix and a broader dataset-use inventory. The latter identifies 21 HELM scenarios absent from all 33 prior evaluations surveyed.

F Comparison with other evaluations

42 scenarios × 30 models form a public matrix reporting prior results, color-coded for comparability with HELM and relative performance.Figure 4 displays a 16 × 28 binary submatrix. Comparing results also helps check implementation quality given sensitivity to prompting and tokenization decisions.Source: F Comparison with other evaluations · p. 155
21 HELM scenarios were classified as not mainstream because none of the 33 prior evaluations surveyed had evaluated them.The survey extended beyond HELM’s 42 scenarios to datasets used across major language-model evaluations.Source: F Comparison with other evaluations · p. 155
405 datasets across 33 prior evaluations are catalogued in a public binary dataset-use matrix spanning language-model papers and evaluation papers.Table 12 enumerates the evaluations and summarizes the number of datasets used in each.Source: F Comparison with other evaluations · p. 155

G Contamination … J.3 Decoding parameters

HELM documents contamination and model-information gaps, uses priorities to manage evaluation costs, and specifies prompting and decoding protocols. Its adaptation approach favors naturalistic prompts and model-agnostic stopping rules.

G Contamination

HELM treats exposure to evaluation instances or distributions as contamination, distinguishing strong from weak exposure because both can compromise evaluation validity.Training-data opacity limits assessment, especially for claims about few-shot capabilities. Evidence ranges from heuristic n-gram overlap to known training-data provenance, with contamination records maintained on GitHub.Source: G Contamination · pp. 155–156

H Priority system

4 priority levels scale evaluation scope; HELM evaluated all models on priority 1 and 2 scenarios.Infeasible API costs prevented the originally envisioned OpenAI evaluation coverage. Priorities are subjective and coverage-dependent; proposed budget allocation sorts scenarios by priority and then estimated cost.Source: H Priority system · p. 158

I Models

Evaluated models span autoregressive, encoder-decoder, and bidirectional Transformers, with substantial variation in training-data disclosure and available model specifications.Examples include BLOOM (176B) (Scao et al., 2022), T0++ (11B), and GLM (130B). Unconfirmed training procedures, release dates, and sizes for the evaluated OpenAI text-* models are generally omitted.Source: I Models · pp. 158–160

J.1 Formatting test instances

Multiple-choice adaptation follows established preferences, otherwise defaulting to joint choice presentation except when answers naturally continue the input.Joint presentation supports one query per question, in-context examples, and direct comparison between choices; adaptation choices substantially affect accuracy. Language-modeling documents are tokenized and chunked with maximal preceding context, whereas overlong instances in other scenarios are truncated.Source: §J.1 · pp. 160–161

J.2 Formatting the remainder of the prompt

5 in-context examples are selected by default, held fixed across test instances, and resampled across three repetitions to measure selection variance.Classification examples prioritize class coverage in decreasing class-frequency order, and long examples reduce the number that fits within the context window. Instructions or prefixes further specify the desired behavior.Source: §J.2 · p. 161

J Adaptation

HELM concretizes few-shot adaptation while treating prompting as user behavior, favoring naturalistic prompts over prompts optimized to maximize model performance.The authors advocate models that work well with similar prompting behavior and encourage research on how interfaces shape users’ prompts.Source: J Adaptation · p. 160, §J.2 · p. 161

J.3 Decoding parameters

HELM generally stops completions at newlines rather than token limits to avoid tokenizer-dependent comparisons, retaining maximum-token limits as a fail-safe.Sampling is restricted to temperature scaling because some models lack alternatives: short-reference tasks use temperature 0, while longer generation follows prior work or a specified selection process. Outputs default to 1 per input, or 0 for language modeling; distributional analyses use multiple outputs, and fail-safe limits reflect the longest reference.Source: §J.3 · pp. 161–162

J.4 Adaptation methods

HELM specifies scenario-level default adaptation methods within its few-shot prompting approach, but leaves defaults unspecified for HellaSwag and OpenBookQA.

J.4 Adaptation methods

Default adaptation methods determine how language models address each scenario within HELM’s few-shot prompting approach.Table 14 maps scenarios to their default adaptation methods.Source: §J.4 · p. 162, Table 14 · p. 162
HellaSwag and OpenBookQA have no specified default adaptation methods.These defaults are unspecified in Table 14.Source: Table 14 · p. 162