PaperlayerView exact revision on arXiv

Source-linked AI summary

Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

Evan Miller

arXiv:2411.00640v1stat.APcs.CL

TL;DR

Language-model evaluations are experiments, yet evaluation research has largely overlooked statistical analysis and planning, while reported top scores often lack significance testing. The paper develops a framework and practical recommendations for inference, variance reduction, and experiment planning; it finds that confidence intervals in [7] may be too narrow in some cases and too wide in others.

  • Problem

    Evaluation reports often emphasize the highest score without testing statistical significance, and traditional question-and-answer evals commonly lack error bars.

    Source: §1 · p. 1

  • Method

    The paper models eval questions as draws from an unseen super-population and develops statistical formulas and recommendations for inference and experiment planning.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    The authors find that confidence intervals reported in [7] are likely too narrow in some cases and too wide in others.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors recommend reporting standard errors and, when comparing models, pairwise differences, pairwise standard errors, and score correlations.

    Source: §6 · p. 10

    Pooling all answers across repeated answers to the same question gives inconsistent standard errors because those answers are not independent draws.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning. This article shows researchers with some training in statistics how to think about and analyze data from language model evaluations. Conceptualizing evaluation questions as having been drawn from an unseen super-population, we present formulas for analyzing evaluation data, measuring differences between two models, and planning an evaluation experiment. We make a number of specific recommendations for running language model evaluations and reporting experiment results in a way that minimizes statistical noise and maximizes informativeness.

1 Introduction

Language-model evaluations often emphasize top scores without testing statistical significance, leaving traditional question-and-answer evals without the confidence intervals used in some other settings. This paper develops statistical tools and reporting recommendations for analyzing noisy eval results and testing hypotheses about competing models, prompts, and hyperparameters.

1 Introduction

  • Traditional question-and-answer evals commonly report a highest score or SOTA result without testing statistical significance, and error bars remain noticeably absent.Chatbot Arena[4] popularized confidence intervals in Elo scores; the Llama 3 technical report[7] is a notable eval exception.Source: §1 · p. 1
  • The article introduces statistical inference for noisy eval data and reports that confidence intervals in [7] are likely too narrow in some cases and too wide in others.It develops an analytic framework and recommendations for computing confidence intervals and reporting eval results.Source: §1 · p. 1
  • The motivating example asks whether mixed results across MATH, HumanEval, and MGSM justify concluding that one model is generally better suited to coding and mathematical tasks.Galleon leads on MATH at 65.5% versus 63.0%, while Dreadnought leads on HumanEval and MGSM by slightly wider margins.Source: §1 · pp. 1–2
  • The paper recommends testing hypotheses about competing models, hyperparameters, and prompts while remaining agnostic about evals’ qualitative merits.Its quantitative framework is intended to inform holistic decision-making.Source: §1 · p. 2
  • The article recommends central-limit-theorem standard errors, clustered standard errors for related questions, and variance reduction through resampling or next-token probabilities.For model comparisons it recommends inference on question-level paired differences; it also discusses power analysis for evals or random subsamples.Source: §1 · p. 2
  • The paper draws on statistical theory and experimental design to help practitioners conduct inference on often-noisy eval data.It frames the analysis around a small number of conceptual assumptions.Source: §1 · p. 2

2 Analysis framework

The framework treats observed eval questions as draws from an unseen super-population, allowing inference about an underlying mean score. It distinguishes independent from clustered questions, since dependence within clusters can make naive standard errors misleading.

2 Analysis framework

  • The analysis framework models eval questions as random draws from an unseen super-population, shifting inference beyond the specific questions observed.The assumption is modified for questions drawn together in related groups.Source: §2 · p. 2

2.1 Independent questions

  • For independent questions, the observed mean estimates the super-population mean, and the Central Limit Theorem supplies a standard-error estimate when scores have finite variance and there are many questions.The authors advise reporting the standard error alongside the mean and note that bootstrapping is unnecessary unless sampling or estimation is complicated.Source: §2.1 · p. 3

2.2 Clustered questions

  • In real-world examples, clustered standard errors can be over 3X larger than naive standard errors, and the authors say reading-comprehension intervals in [7] are likely anti-conservative.The paper recommends reporting cluster count alongside question count when presenting cluster-adjusted errors.Source: Table 4 · p. 4, §2.2 · pp. 4–5, Table 3 · p. 4
  • For clustered questions, dependence violates the independent-question assumption, so naive standard errors can be inconsistent and cluster-adjusted errors are needed.The adjustment ranges between treating each cluster as one independent observation under perfect within-cluster correlation and the unclustered case under no correlation.Source: §2.2 · p. 4

3 Variance reduction

The section decomposes score uncertainty into question-selection and within-question variation, then presents ways to reduce the latter. It recommends next-token probabilities when applicable, otherwise repeated answers, and warns against changing sampling temperature just to reduce variance.

3 Variance reduction

  • The variance of the estimated mean score has additive components from question selection and within-question score variation; increasing the number of questions reduces variance, while Var(x) is immutable.The section uses the expected conditional variance as the component that can be reduced through additional strategies.Source: §3 · p. 5

3.1 Resampling

  • In the uniform-difficulty binary-score example, K = 2 reduces total variance by 1/3 and the maximum reduction from resampling is 2/3.The section recommends computing standard errors from question-level mean scores, not pooling all KN answers, which would violate independent-draw assumptions.Source: §3.1 · p. 5

3.2 Next-token probabilities

  • For an evaluation that avoids token generation, next-token probabilities can eliminate conditional variance; in the stated example, they reduce estimator variance by 2/3 versus grading one sampled answer per question.This applies to a multiple-choice prompt that elicits the answer in its first token.Source: §3.2 · p. 6

3.3 Don’t touch the thermostat!

  • Lowering sampling temperature can increase variance in conditional means or introduce bias, so it should not be used solely to reduce score variance.The section illustrates both risks with single-token true/false examples.Source: §3.3 · p. 6
  • Use next-token probabilities when available; otherwise choose K so within-question variance is small relative to question-selection variance, without changing sampling temperature to reduce variance.Next-token scoring applies when evaluation does not require token generation, such as chain-of-thought; repeated answers are an alternative for more complex interactions.Source: §3 · p. 5, §3.3 · p. 6, §3.2 · p. 6

4 Comparing models

Model comparisons require uncertainty estimates to distinguish meaningful differences from noise. The section contrasts unpaired analysis with question-level paired analysis, which can reduce variance when model scores are correlated.

4.1 Unpaired analysis

  • Unpaired analysis can test a score difference even when models used non-identical random subsets of eval questions.It uses independently reported eval scores and standard errors.Source: §4.1 · p. 7

4.2 Paired analysis

  • For shared questions, paired analysis estimates differences question by question; it can reduce variance when model scores are correlated.The paired estimate can be used with a revised standard error to compute a confidence interval and z-score.Source: §4.2 · pp. 7–8
  • MATH’s difference is statistically significant, while HumanEval and MGSM differences are indistinguishable from noise, reversing the superficial impression that Dreadnought was overall better.The table presents pairwise differences, standard errors, confidence intervals, and correlations as a supplement to main results.Source: Table 5 · p. 8, §4.2 · pp. 8–9
  • With correlation 0.5 in the uniform-score example, paired differences reduce estimator variance by 1/3, from 1/6 to 1/9.The example assumes continuous next-token-probability scores with zero conditional variance.Source: §4.2 · p. 8

5 Power analysis

The power-analysis section presents a sample-size formula for planning whether an eval can detect a hypothesized model difference. It illustrates how the formula can inform eval size and the Minimum Detectable Effect.

5 Power analysis

  • The sample-size formula relates the hypothesized difference between two models to the number of questions needed for a specified Type I error rate, Type II error rate, and Minimum Detectable Effect.The formula assumes paired analysis and accounts for the number of answer samples per model.Source: §5 · p. 9
  • At least 1,000 questions are suggested for good signaling ability in the worked example’s reasonable but fictional setting.The example targets a 3% absolute difference, 80% power, and a 5% false-positive rate.Source: §5 · pp. 9–10
  • Minimum Detectable Effect falls from 13.2% to 7.5% when KA = KB increases from 1 to 10 in the specified example.The example uses n = 198, α = 0.05, β = 0.20, σB^2 = 1/6, and ω^2 = 1/9.Source: §5 · p. 10

6 Conclusion

The conclusion presents statistical tools for analyzing eval scores and comparing models, alongside recommendations for reducing variance and reporting uncertainty. It advocates treating evals as informative experiments rather than contests to produce the largest score.

6 Conclusion

  • The article presents formulas for naive and clustered standard errors for single-model analysis, and unpaired, paired, and paired-and-clustered standard errors for two-model analysis.It also presents a sample-size formula to plan the reliably detectable difference under a given eval and resampling strategy.Source: §6 · p. 10
  • Recommendations include resampling answers, analyzing next-token probabilities, computing question-level model differences, and reporting standard errors and pairwise comparisons.The conclusion advises against adjusting sampling temperature for variance reduction.Source: §6 · p. 10
  • The authors encourage practitioners to use statistical tools to treat model evaluations as informative experiments rather than contests for the largest score.They also encourage further exploration of statistical techniques from other experimental fields.Source: §6 · p. 10

A Clustered standard errors

The appendix develops clustered standard errors through a regression framing of question scores and extends the two-sample approach to question-level score differences.

A Clustered standard errors

  • The regression framing decomposes each question score into a mean and random component, with question-level deviations and residuals represented in the model.The conditional-mean deviation is treated as a question-level fixed effect that is not separately estimated.Source: A Clustered standard errors · p. 12
  • The clustered standard-error expression uses covariates and within-cluster residual covariance; this application uses vectors of ones for the covariates.The residual covariance matrix is specified within each cluster.Source: A Clustered standard errors · p. 12
  • The two-sample version is developed by analyzing question-level score differences rather than the scores themselves.Source: A Clustered standard errors · p. 12

B Sample-size formula derivation

The derivation sets error-rate targets for a hypothetical measurement, then expresses the minimum detectable effect (MDE) and required sample size in terms of the other variables.

C Cluster-adjusted sample-size formula

The clustered sample-size formula uses cluster-adjusted variance quantities estimated from previous evaluation data. The section also describes adjustments and sampling choices for estimating those quantities.

Related papers

Loading 2411.00640v1…