PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Evaluate When You Don't Know How to Evaluate? Choosing Reliable Setups for Base Language Model Benchmarking

Shenbin Qian, David Samuel, Jindřich Helcl, Andrey Kutuzov, Stephan Oepen

arXiv:2610.19185v1cs.CL

TL;DR

Evaluation choices can change measured LLM performance and subsequent pretraining decisions, but guidance for selecting setups is limited. The paper compares setups using three signal criteria and finds that some choices hold across model scales while others vary with scale.

  • Problem

    LLM evaluation involves many design choices, and those choices can substantially affect measured performance and later pretraining decisions.

    Source: §1 · p. 1

  • Method

    The study compares eight evaluation aspects using monotonicity, non-randomness, and ranking consistency across checkpoints of 26 monolingual 2B models and three larger multilingual models.

    Source: §1 · p. 1, §3.2 · p. 5

  • Results

    Across scales, multiple prompts and 5-shot prompting are informative, while cloze formulation and answer probability are preferred for 2B models and multiple-choice formulation and accuracy grow more competitive at larger scales.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Prioritize choices that yield informative signals across scales, and reassess context-dependent choices as training progresses.

    Source: §5 · p. 8

    The three meta-measures are proxies for useful signals and do not guarantee that a setup measures all capabilities or data differences relevant to pretraining decisions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them. These choices can substantially affect measured performance and, consequently, decisions about subsequent pretraining. To identify effective evaluation setups, we investigate 8 aspects, such as task formulation or the number of in-context examples. We compare the setups using three widely-used signal criteria, repurposed as normative meta-measures: monotonicity, non-randomness, and ranking consistency of the evaluation scores across intermediate checkpoints. In an early pretraining scenario, we evaluate these setups on 26 monolingual 2B-parameter models trained on different types of data, using 80 tasks in four languages. We also examine the setups on three larger LLMs ($\geq$7B) in the same four languages and explore how the preferred setups change with model scale. Our results yield concrete recommendations: some choices, such as the use of multiple prompts, are consistently useful, whereas others, such as task formulation, depend on model scale. More generally, our framework serves as a recipe for the choice of evaluation setups in new scenarios.

1 Introduction

Evaluation choices can reshape measured performance and downstream pretraining decisions, yet guidance on choosing setups is limited. The paper tests evaluation setups during early pretraining and finds that some choices are robust while others depend on model scale.

1 Introduction

  • Evaluation setup decisions matter because they can shape subsequent development choices, including training-data composition, and those choices affect final model performance.The early-pretraining scenario uses small proxy models to select among candidate training datasets.Source: §1 · p. 1
  • The metaevaluation framework assesses setups using monotonicity, non-randomness, and ranking consistency across intermediate model results.The measures capture improvement with training, separation from a random baseline, and stability of model rankings in later training stages.Source: §1 · p. 1
  • The systematic study compares eight evaluation aspects across checkpoints of 26 monolingual 2B models, 80 tasks, and four languages.For 14 Norwegian multiple-choice tasks, the study adds formulations, likelihood normalizations, and soft probabilistic metrics.Source: §1 · pp. 1–2
  • Kendall’s τ = −0.6 for two Norwegian 2B evaluation setups, which largely reverse the corpus ranking and change which training data appears preferable.MADLAD-400 shifts from highest under one setup to lowest under the other, while OPUS and Tower-9B shift from bottom to top.Source: Figure 1 · p. 2
  • Multiple prompts and 5-shot prompting provide informative signals across scales, while cloze formulation and answer probability suit smaller models and multiple-choice formulation and accuracy grow more competitive at larger scales.The authors validate the scale-dependent findings on three larger multilingual LLMs.Source: §1 · p. 2

2 Related Work

Prior work has formalized signal criteria for selecting tasks and examined evaluation reliability, but this paper compares complete evaluation setups across multiple design aspects. It also extends benchmarks to expose more of those choices.

2 Related Work

  • Earlier task-selection work applied meta-measures to choose which tasks to include; this study instead uses them to select evaluation setups.FineTasks formalized monotonicity, non-randomness, ranking consistency, and signal-to-noise ratio; the paper adopts three of these measures.Source: §2 · p. 2
  • Existing reliability studies examine individual setup choices, whereas this work jointly compares complete setups across eight aspects using three meta-measures.OLMES standardizes choices by convention and accuracy, while Madaan et al. (2025) and Heineman et al. (2025a) investigate reliability choices such as continuous metrics and cloze formulation.Source: §2 · p. 2
  • Because most benchmarks offer limited variation in prompts, formulations, metrics, or decoding, the authors extend NorEval with additional evaluation options.FIN-bench-v2 already includes cloze and multiple-choice formulations with five prompts each.Source: §2 · p. 2

3 Experimental Setup

The setup compares eight evaluation choices for classification and generation tasks, then ranks complete setups and individual options using three meta-measures. Benchmarks and model checkpoints provide comparisons across four languages, with Norwegian supporting all eight aspects.

3.1.1 Task-Agnostic Aspects

  • Task-agnostic choices include 0- or 5-shot prompting, single or multi-prompt scoring, four score-rescaling methods, and task-level or category-level aggregation.Category-level aggregation gives each task category equal contribution regardless of its number of tasks.Source: §3.1.1 · p. 3

3.1.2 Classification Task Aspects

  • Classification choices vary formulation, log-probability normalization, and metric; continuous probability-based metrics can reveal gradual progress before accuracy changes.Cloze, multiple-choice, and hybrid formulations differ in whether options are shown and whether the model scores answer text or labels.Source: §3.1.2 · pp. 3–4

3.1.3 Generation Task Aspects

  • Generation tasks compare greedy decoding with random sampling, while AnsProb and CondProb are true probabilities only with no PMI normalization.With PMI normalization, AnsProb can exceed 1.Source: §3.1.3 · p. 4

3.2 Meta-Measures

  • Three meta-measures assess whether scores track training progress, separate models from chance, and preserve model rankings across checkpoints.Ranking consistency aligns models by training tokens, uses windows with at least two models still training, and excludes the first window.Source: §3.2 · pp. 4–5
  • Setups are compared by their mean percentile rank across the three meta-measures, by option rankings averaged across containing setups, and by per-option win rates.Per-option win rates compare setups differing in one aspect; ties count as half, and results are averaged across meta-measures.Source: §3.2 · p. 5

3.3 Evaluation Benchmarks

  • Benchmark support varies by language: NorEval is extended with formulations, metrics, and sampling, making Norwegian the only language covering all eight aspects.The other language benchmarks have constraints such as missing formulations or PMI-normalized accuracy.Source: §3.3 · pp. 5–6

3.4 Models

  • The study evaluates 26 same-architecture, same-setup monolingual 2B-parameter models, each trained on 100B tokens in Norwegian, Finnish, French, or Spanish, and intermediate checkpoints of three larger multilingual models.Each monolingual model has 11 checkpoints from 2B to 100B tokens; the larger models are OLMo 3 7B, Apertus 8B, and Prelude 9B.Source: §3.4 · p. 6

4 Results

The results identify evaluation choices that provide informative signals across model scales, while showing that preferences for some metrics and task formulations change as models grow. For early-pretraining models, combining independently preferred options yields a setup close to the best complete setup.

4.1 Early Pretraining Models

  • For 2B models, answer probability is the most informative metric, 5-shot prompting beats 0-shot, and cloze formulation is preferred over multiple-choice formulation.Answer probability wins 93% of pairwise comparisons across meta-measures; 5-shot prompting has a 70% win rate, and CF is selected by all methods with a 76% win rate.Source: §4.1 · p. 7
  • Choosing the top-ranked option independently for each aspect produces a 0.93 overall score and ranks 6th, versus 0.95 for the best complete setup.The tiered recipe differs from the best complete setup only in rescaling; all top five complete setups agree on high-impact choices.Source: §4.1 · p. 7

4.2 Larger-Scale LLMs

  • Across scales, answer probability with character normalization, 5-shot prompting, and mean aggregation over all tasks remain preferred, while multiple prompts outperform a single prompt.For larger LLMs, formulation and metric preferences are exceptions: all three selection methods choose hybrid formulation, and Norwegian favors accuracy.Source: §4.2 · p. 8
  • At larger scale, accuracy becomes more informative (win rate 6% →37%), while answer probability remains strongest but loses advantage (93% →63%).The authors attribute the accuracy gain primarily to improved monotonicity (9% →53%).Source: §4.2 · p. 8
  • Task-formulation preferences shift with scale: CF remains preferred overall but declines (76% →61%), while MCF improves (20% →38%).For Norwegian, option ranking and win rate prefer the hybrid formulation.Source: §4.2 · p. 8
  • Complete-setup scores are only moderately correlated across scales, with Spearman’s ρ ranging from 0.30 for Norwegian to 0.66 for Finnish; scale-dependent findings remain suggestive, not conclusive.The authors base this qualification on only three models trained with different compute budgets, leaving relatively few comparisons for ranking consistency estimates.Source: §4.2 · p. 8

5 Conclusion

The study compares evaluation setups by how much signal they provide during pretraining, using three meta-measures across 26 monolingual 2B models and validating on three larger multilingual LLMs. The results favor a tiered recipe: some choices hold across scales, while others vary with model scale.

5 Conclusion

  • The study evaluates combinations of eight setup aspects on three meta-measures using 26 monolingual 2B models in four languages, then validates on three larger multilingual LLMs.The measures are monotonicity, non-randomness, and ranking consistency.Source: §5 · p. 8
  • Multiple prompts and 5-shot prompting consistently yield more informative signals, while metric and task-formulation preferences shift with model scale.Answer probability with character normalization outperforms alternative metrics; cloze formulation and answer probability are preferred for 2B models, while accuracy and multiple-choice or hybrid formulations become increasingly informative as model performance grows.Source: §5 · p. 8

Limitations

The proposed measures are proxies for useful evaluation signals, not ground truth about which setup best measures model quality. The evidence is also bounded by how the measures are combined and by the study’s language and model coverage.

Limitations

  • The three meta-measures do not guarantee that a setup captures all factors relevant to pretraining decisions or measures broad model quality.A setup may score highly while measuring a narrow capability or missing data differences that matter downstream.Source: Limitations · p. 9
  • Equal weighting by mean percentile rank is one choice; alternative combinations could change setup rankings, especially when measures disagree.The study excludes other proposed measures, including signal-to-noise ratio, because their estimation methods are not directly comparable with this setting.Source: Limitations · p. 9
  • The study covers four European languages with differing benchmark options, and some findings may reflect individual benchmarks rather than setup choices.The larger-scale validation uses only three models, making scale-related ranking differences indicative rather than conclusive.Source: Limitations · p. 9

Ethical Considerations

The study uses publicly accessible datasets under their stated license agreements and presumes they contain no sensitive or personally identifiable information. The authors report no significant ethical risks and retain responsibility for the paper’s intellectual content, analyses, and conclusions.

Ethical Considerations

  • The study uses public datasets under their stated license agreements and presumes they contain no sensitive or personally identifiable information.On that basis, the authors deem the study to have no significant ethical risks.Source: Ethical Considerations · p. 9
  • The authors used AI tools only to improve writing and assist with coding, and retain responsibility for the intellectual content, analyses, and conclusions.Source: Ethical Considerations · p. 9

A Appendix: Inference Details

The appendix describes the inference workflow and how setup scores are converted into comparative choices. It defines three complementary selection views: the best complete setup, average option ranking, and per-option win rate.

A Appendix: Inference Details

  • Evaluations use fixed-seed few-shot examples, and each job runs one checkpoint on a single GPU; results are recorded by checkpoint, task, prompt, formulation, and shot count.Multiple-choice scores derive from answer-option log-likelihoods, with PMI also using an empty context; generative tasks use greedy decoding or sampling.Source: A Appendix: Inference Details · p. 12
  • The 2B-model evaluations took roughly 11,000 GPU hours and the three larger-model evaluations roughly 7,000 GPU hours; scoring all setups takes about five CPU minutes.Source: A Appendix: Inference Details · p. 12

B Appendix: From Meta-Measure Scores to Setup Choices

  • Setup scores aggregate task metrics through prompt aggregation, task rescaling, and task or category aggregation before the three meta-measures are computed.For 2B models, the classification matrix covers 1,488 setups and Norwegian generation covers 96.Source: B Appendix: From Meta-Measure Scores… · p. 12
  • The best complete setup maximizes the overall score, option ranking averages scores across setups containing an option, and win rate counts pairwise wins while ignoring difference size.Overall score is the mean of the three meta-measures’ percentile ranks; win-rate ties count half.Source: B Appendix: From Meta-Measure Scores… · pp. 12–13

C Appendix: Task Coverage of Evaluation Setups

A setup includes only tasks for which all of its options are available, so task coverage can vary across options.

C Appendix: Task Coverage of Evaluation Setups

  • Task coverage varies by option when some options are unavailable for certain tasks.A setup covers only tasks for which all of its options are available.Source: C Appendix: Task Coverage of… · p. 13

D Appendix: Language-Wise Results

This appendix reports language-wise results through tables of selected setup options and option-ranking scores, plus a figure of per-option win rates.

D Appendix: Language-Wise Results

E Appendix: When the Meta-Measures Disagree

For the 2B models, the meta-measures select the same option for most high-impact choices, but disagree on some aspects and more often on Norwegian generative tasks. These differences reveal trade-offs between signal properties and support combining multiple meta-measures while reporting disagreements.

E Appendix: When the Meta-Measures Disagree

  • For 2B models, the meta-measures agree on formulation, metric, number of shots, and task aggregation, but disagree on normalization and prompt aggregation.Ranking consistency provides little separation among rescaling options; disagreement occurs for four of five aspects in Norwegian generative tasks.Source: E Appendix: When the Meta-Measures… · p. 13
  • Maximum aggregation yields higher non-randomness in 80% of 672 setup pairs, while mean aggregation yields higher monotonicity in 58%.The maximum’s non-randomness advantage reflects, in part, a mechanically larger best-score-to-chance gap, rather than additional signal.Source: E Appendix: When the Meta-Measures… · p. 13
  • When meta-measures disagree, the preferred option depends on the desired signal property, and disagreements occur mainly where options have similar overall scores.A score transformation can inflate a meta-measure without increasing useful signal.Source: E Appendix: When the Meta-Measures… · p. 13
  • For Norwegian generative tasks, category-mean aggregation wins 98% of non-randomness comparisons but only 29% of monotonicity comparisons.The stronger disagreements for generative tasks motivate combining multiple meta-measures and explicitly reporting disagreements.Source: E Appendix: When the Meta-Measures… · p. 14

Related papers

Loading 2610.19185v1…