PaperlayerView exact revision on arXiv

Source-linked AI summary

OLMES: A Standard for Language Model Evaluations

Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, Hannaneh Hajishirzi

arXiv:2406.08446v2cs.CLcs.AI

TL;DR

Language-model evaluations lack a common setup, and differing choices can make performance comparisons hard to reproduce. OLMES offers an open, documented standard with justified recommendations. It distinguishes when cloze or multiple-choice formulations are useful and reports little difference between its normalization recommendation and the empirically best choice.

  • Problem

    Evaluation choices can substantially affect measured performance, and differing or incompletely reported setups make results difficult to reproduce and compare.

    Source: §1 · p. 1

  • Method

    OLMES reviews common evaluation practices and uses existing literature and experiments to develop an open, documented, practical standard with justified recommendations.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    CF is useful for assessing task knowledge in models not yet skilled at MCF, while MCF is more realistic and yields higher, more representative scores for models able to use it; recommended normalization differs little from the empirical oracle.

    Source: §3.3 · p. 6, §3.4 · p. 7

  • Takeaways & Limitations

    OLMES is intended to guide model-development evaluations and comparisons of final base models, helping unify evaluation practices.

    Source: §6 · p. 9

    The recommendations do not cover every plausible task presentation, and evaluating robustness under adversarial setups is left for future work.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Progress in AI is often demonstrated by new models claiming improved performance on tasks measuring model capabilities. Evaluating language models can be particularly challenging, as choices of how a model is evaluated on a task can lead to large changes in measured performance. There is no common standard setup, so different models are evaluated on the same tasks in different ways, leading to claims about which models perform best not being reproducible. We propose OLMES, a completely documented, practical, open standard for reproducible LLM evaluations. In developing this standard, we identify and review the varying factors in evaluation practices adopted by the community - such as details of prompt formatting, choice of in-context examples, probability normalizations, and task formulation. In particular, OLMES supports meaningful comparisons between smaller base models that require the unnatural "cloze" formulation of multiple-choice questions against larger models that can utilize the original formulation. OLMES includes well-considered, documented recommendations guided by results from existing literature as well as new experiments resolving open questions.

1 Introduction

Evaluation choices can materially change measured language-model performance, while underspecified setups make comparisons difficult to reproduce. OLMES proposes a practical, open, documented standard with justified evaluation recommendations.

1 Introduction

  • As much as an 80% accuracy difference has been reported on a task from varying formatting and in-context examples.The paper argues that differing or underspecified setups can also produce different performance conclusions across references.Source: §1 · pp. 1–2
  • OLMES is a standard intended to improve transparency and reproducibility in language-model evaluation.Its recommendations address data sampling, instance formatting, in-context examples, probability normalization, and task formulation.Source: §1 · p. 2
  • OLMES specifies evaluation details from dataset processing through task presentation and output processing to remove procedural ambiguity.The standard makes practical decisions about computation resources for easier adoption.Source: §1 · p. 2
  • OLMES documents decisions with justifications informed by prior studies and experiments, and releases prompts, examples, code, and rationales for reuse.The authors say it is straightforward to adopt in evaluation codebases such as the Eleuther LM Evaluation Harness and HELM.Source: §1 · p. 2

2 Experimental setup

The setup targets multiple-choice question answering as a useful way to assess base models during and after training, while emphasizing that scores need a clearly specified formulation. OLMES standardizes ten popular benchmark tasks and develops its choices using diverse pretrained base models.

2.1 Multiple-choice QA and LLM evaluation

  • MCQA can provide useful signals during early training and for smaller base models, but an unspecified formulation makes benchmark scores difficult to interpret and compare.The setup distinguishes MCF, which scores answer labels, from CF, which scores each answer choice using token probabilities; CF also has normalization ambiguities and cannot properly handle some “none of the above” choices.Source: §2.1 · p. 3

2.2 Targeted tasks

  • OLMES standardizes ten popular benchmark MCQA tasks spanning science, commonsense, factual knowledge, varied topics, and difficulty.MMLU alone covers 57 subjects.Source: §2.2 · p. 3

2.3 Selection of models

  • The standard was developed using 15 diverse, openly available pretrained base models ranging from 1B to 70B parameters.The selection was intended to support use during model development and comparison of final base models; evaluation setups across the field can otherwise lead to different results and conclusions.Source: §2.3 · pp. 3–4

3 Standardizing variations in evaluation

OLMES standardizes key evaluation choices, from instance formatting and few-shot examples to probability normalization and task formulation. It evaluates both CF and MCF, using whichever performs better, and specifies other implementation details for consistency.

3.1 How to format dataset instances?

  • OLMES uses a consistent “Question: <question>” prefix and “Answer:” suffix, with task-specific formats for PIQA, HELLASWAG, and WINOGRANDE.It uses the test split when labels are public, otherwise validation, and samples 1,000 instances when a dataset exceeds 1,500.Source: §3.1 · pp. 4–5

3.2 Which few-shot examples to use?

  • OLMES uses manually curated, fixed five-shot prompts for each task, with examples selected from the training set to cover the label space evenly.The five-shot limit also helps limit computational overhead; analyses cited in the passage suggest more shots generally do not meaningfully change scores.Source: §3.2 · p. 5

3.3 How to normalize LLM probabilities for CF?

  • OLMES assigns CF normalization by task: PMI for ARC-CHALLENGE, COMMONSENSEQA, and OPENBOOKQA; character normalization for five other tasks; and none for BOOLQ and WINOGRANDE.PMI adjusts for answers’ prior likelihood; character normalization was empirically best for its assigned tasks, while WINOGRANDE’s identical continuations make normalization irrelevant.Source: §3.3 · p. 6

3.4 What task formulation to use, MCF or CF?

  • Across tasks, stronger models’ MCF performance exceeds or catches up to CF, while weaker models can score near-random on MCF.For the OLMES normalization recommendations, the reported difference from the empirically best normalization is generally minimal.Source: §3.3 · p. 6, Table 3 · p. 6, §3.4 · p. 7
  • CF provides a better signal for weaker models, while MCF better represents stronger models; OLMES evaluates both and uses the better result.For OLMo-7B-0424 on MMLU, the model begins gaining MCF ability around 400 billion training tokens; for Llama3-70B on ARC-CHALLENGE, MCF scores 93.7% versus 69.0% for CF.Source: §3.4 · p. 7

3.5 Other implementation details

  • OLMES additionally specifies MMLU macro averaging, appropriate BOS tokens when needed, input limits, default precision, and other prompt details.The authors note that batch size and GPU details can flip answer choices when competing confidences are very close.Source: §3.5 · pp. 7–8

4 OLMES: Summary and results

OLMES summarizes its recommendations for reproducible evaluation, covering dataset handling, prompts, few-shot examples, CF normalization, and task formulation. Its protocol uses both MCF and CF and takes the better result.

4 OLMES: Summary and results

  • OLMES uses the test set when available, otherwise validation, and samples 1,000 instances when datasets exceed 1,500.The summary identifies dataset choice and sampling as part of the standard.Source: §4 · p. 8
  • OLMES specifies an exact prompt format and fixed, curated five-shot examples.These are listed as separate components of the standard.Source: §4 · p. 8
  • OLMES prescribes probability normalization for CF and recommendations for other evaluation details.The summary presents both as parts of the evaluation standard.Source: §4 · p. 8
  • OLMES evaluates with both MCF and CF, using the better result.This is one of the listed elements in its overall evaluation standard.Source: §4 · p. 8

5 Related work

Prior work documents many sources of variation in language-model evaluation and offers frameworks for transparency and reproducibility. OLMES provides reproducible performance scores across models and tasks.

5 Related work

  • Formatting changes, answer-option variations, and task formulation have been reported to affect evaluation scores, sometimes substantially.The passage describes these as vulnerabilities in model evaluations across prior work.Source: §5 · p. 8

model ARC_C ARC_E BoolQ CSQA HSwag MMLU OBQA PIQA SIQA WinoG average

  • Table 4 presents reproducible OLMES performance scores across models and tasks, with daggers marking use of MCF scores.The caption describes the scores as supporting comparisons across a wide range of models and tasks.Source: Table 4 · p. 9

5 Related work

  • HELM, the Hugging Face Open LLM Leaderboard, Mosaic Eval Gauntlet, Eleuther LM Evaluation Harness, and Unitxt support transparency and reproducibility.The passage notes that rationales for specific defaults are not always documented and followed in subsequent work.Source: model ARC_C ARC_E BoolQ CSQA HSwag… · p. 9

6 Discussion

OLMES combines practical evaluation recommendations with documented rationales and experiments addressing open questions. It is intended to support comparable evaluations during model development and in leaderboards and evaluation code bases.

6 Discussion

  • OLMES recommends choices for dataset formatting, in-context examples, task formulation, probability normalization, and other implementation details.The recommendations are intended to avoid unnecessary computation resources.Source: §6 · p. 9
  • Its recommendations are justified by reviews of community practices, existing studies, and experiments resolving open questions.Source: §6 · p. 9
  • OLMES evaluations can be directly compared with existing OLMES evaluations, and the standard can be applied to current leaderboards and evaluation code bases.Source: §6 · p. 9
  • OLMES is intended to provide signals during training and support comparisons of final base models.Source: §6 · p. 9
  • OLMES is ready for incorporation into evaluation code bases for broad usage and is intended to facilitate model-performance comparisons during development and model selection.Source: §6 · p. 9

Limitations

The current OLMES version focuses on evaluating multiple-choice tasks, especially during training and when comparing final base models. Extending its principles to generative tasks, chain-of-thought prompting, and other settings remains future work.

Limitations

  • The current version focuses on LLM evaluation during training and comparisons of final base models before further tuning.Evaluations of tuning effectiveness and multimodal models are identified as future directions.Source: Limitations · pp. 9–10
  • This study focuses on language-model evaluations using multiple-choice tasks.Source: Limitations · p. 10
  • Applying OLMES principles to generative tasks and chain-of-thought prompting remains an important future direction.The task suite covers science, commonsense, factual knowledge, and topics including MMLU's 57 subjects.Source: Limitations · p. 10
  • OLMES recommendations do not cover every plausible way of presenting a task.Performance is described as stable under small prompt-wording or few-shot-selection changes within the recommendations; larger differences are expected for divergences such as rare answer labels or random few-shot examples.Source: Limitations · p. 10
  • Evaluating model robustness under adversarial setups is left for future work.Source: Limitations · p. 10

Ethical considerations

The authors state that they use large-scale language models only to obtain answers to questions in commonly used multiple-choice datasets and foresee no ethical issues from their use in this research.

Ethical considerations

  • The authors report using large-scale language models only to obtain answers on commonly used multiple-choice datasets and foresee no ethical issues from that use.Source: Ethical considerations · p. 10

A FAQs

OLMES standardizes evaluation choices so reported scores can be interpreted and compared, with documented recommendations grounded in literature and experiments. Its findings distinguish which task formulations and evaluation settings are useful across model strengths and small prompt variations.

A FAQs

  • CF gives a clearer signal for weaker models, while MCF better represents task performance for stronger models.CF can help developers make inexpensive decisions early in training; MCF is a better indicator later.Source: A FAQs · p. 14
  • Five-shot prompting is an effective, universal way to convey a task, but going beyond five shots generally adds no meaningful score difference.The learning curve is more stable than with zero-shot prompting.Source: A FAQs · p. 14
  • For BOOLQ, length normalization is unnecessary because “yes” and “no” are single-token answer choices.The slightly better character-normalized performance can be an accidental effect of “yes” having one more character than “no.”Source: A FAQs · p. 14
  • OLMES aims to apply its reproducibility principles to any dataset, although the paper illustrates them using multiple-choice datasets selected to guide base-model evaluation.The authors focus on base models before further tuning, and say the principles generalize beyond the illustrated tasks.Source: A FAQs · p. 15
  • Across model releases, OLMES evaluations show larger models outperforming smaller ones within several model families, while also enabling comparisons of training and architecture improvements.The reported examples include OLMo-7B-0424 over OLMo-7B and Llama3-8B over Llama2-7B and Llama2-13B.Source: A FAQs · p. 15
  • The authors do not expect the current recommendations to need revision soon, while anticipating extensions to more benchmarks and newer models.They say the recommendations draw on a rich literature and expect the guiding principles to remain relevant in the near future.Source: A FAQs · p. 15
  • CoT prompts helped instruction-tuned Llama-3.1-8B-Instruct on MMLU but tended to lower scores for base Llama-3.1-8B.The authors focus on base models, for which these prompting strategies are less effective.Source: A FAQs · p. 16
  • Performance differences between OLMES and small prompt or example variations were generally under 1%, with the largest difference 1.4%.Averaging four variants often differed little from the original setup, not justifying four times the compute.Source: A FAQs · p. 17, Table 5 · p. 17

B Detailed CF and MCF task scores

The appendix collects detailed CF and MCF scores and explains how evaluation formulation and probability normalization differ. Prior work used both formulations, with MCF becoming more common as models learned to use it.

B Detailed CF and MCF task scores

C.1 Task formulation details

  • Evaluation practice shifted gradually from CF toward MCF as models became able to use the more natural multiple-choice formulation.Earlier work often used CF for weaker models that struggled with MCF.Source: §C.1 · p. 18

C.2 CF normalization details

  • Unlike MCF, CF requires choosing how to normalize answer-string log-likelihood, with options including token, character, and PMI normalization.Prior work used token normalization and, for some datasets, PMI-style normalization.Source: §C.2 · p. 18
  • Per-token and per-character normalization differ little when answer choices are long or similar in length, but character normalization can alter BOOLQ’s yes/no probabilities because the strings differ in length.The rationale that token normalization depends on the tokenizer is not relevant when the goal is to rank choices with the model and tokenizer fixed.Source: §C.2 · p. 18

C.2.1 Tasks that generally prefer CF

CF remains competitive on some tasks even for strong models, while MCF is more informative for stronger models on other tasks. The appendix also describes a hybrid formulation and a tokenization choice for multiple-choice labels.

C.2.1 Tasks that generally prefer CF

  • HELLASWAG and WINOGRANDE retain higher CF than MCF scores even for the strongest models, a pattern correlated with CF resembling natural-continuation language modeling.The authors note MCF scores are already close and suggest monitoring whether they eventually surpass CF as models improve.Source: §C.2.1 · pp. 18, 22
  • Table 6 compares MCF and CF by task; weaker models have near-random MCF scores, while MCF better assesses stronger models.Table 7 also reports overall averages, with the “max” average taking the better formulation per task.Source: Table 6 · p. 19, Table 7 · p. 19

C.2.2 Hybrid formulation

  • In CF, total probability can favor shorter answers; a hybrid that lists choices before scoring the answer string usually falls between CF and MCF.OLMES does not focus on the hybrid because it is not popular in evaluation standardization or model development.Source: §C.2.2 · p. 22

C.3 Tokenization of MCQA choice labels

  • OLMES prefixes answer-choice labels with a space so tokenizers can align the choice-label token with the token in the final answer.For the Llama tokenizer, the consistent token is “_A” rather than the separate “A” produced without the prefix space.Source: §C.3 · p. 22, Figure 3 · p. 22

D Extended OLMES result table

The appendix extends OLMES evaluations and documents materials and procedures for reproducing them. It also presents a reproduction comparison and examples of prompts and curated few-shot demonstrations.

D Extended OLMES result table

  • Table 13 extends OLMES evaluations to 40 models, and Table 14 adds OPENBOOKQA score variations across references to the ARC-CHALLENGE comparison.Table 14 notes that different references use setups that are sometimes unspecified, making conclusions about the best-performing models unreproducible.Source: D Extended OLMES result table · p. 22, Table 14 · p. 24

F Compute used

G Curation of 5-shot examples: considerations

model ARC_C ARC_E BoolQ CSQA HSwag MMLU OBQA PIQA SIQA WinoG average

  • Table 13’s caption describes reproducible performance scores across models and tasks, with a dagger marking use of MCF scores.Source: Table 13 · p. 23

E HELM Reproduction of MMLU

  • Figure 4 plots self-reported MMLU scores against the difference from reproduced scores; its positive trend indicates greater average overestimation at higher self-reported scores.Points above zero represent higher self-reported performance than reproduced performance.Source: E HELM Reproduction of MMLU · p. 22, Figure 4 · p. 24

H OLMES prompt formats for each task

  • The appendix provides a full five-shot ARC-CHALLENGE CF prompt and single-instance examples for both MCF and CF across ten tasks.The prompts, curated examples, and formatting code are openly available.Source: H OLMES prompt formats for each task · p. 25

Related papers

Loading 2406.08446v2…