tinyBenchmarks: evaluating LLMs with fewer examples

Felipe Maia Polo; Lucas Weber; Leshem Choshen; Yuekai Sun; Gongjun Xu; Mikhail Yurochkin

Source-linked AI summary

tinyBenchmarks: evaluating LLMs with fewer examples

Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, Mikhail Yurochkin

arXiv:2402.14992v2Feb 22, 2024cs.CLcs.AIcs.LGstat.ML

TL;DR

Large benchmarks make repeated LLM evaluation expensive, so the paper studies whether performance can be estimated from far fewer examples. It combines sampling and clustering strategies, especially IRT-based representations, and finds that 100 curated examples per scenario usually reproduce benchmark performance within about 2% average error. The released tiny benchmarks and IRT tools make this reduced evaluation setup available for practice.

Problem
Large benchmark evaluations require many examples and can impose substantial computational, environmental, and financial costs.
Method
The paper compares sampling and clustering strategies, including IRT representations that encode example difficulty and required abilities for selecting representative examples.
Results
100 curated examples per scenario estimate performance across four benchmarks within about 2% error on average.
Takeaways & Limitations
The approach supports cheaper, more frequent LLM evaluation and provides tiny benchmark datasets plus CPU-runnable IRT estimation tools.
Takeaways & Limitations
Performance estimation can degrade under severe distribution shifts or rapid capability increases, so curated examples and IRT parameters should be updated periodically.

Abstract

from arXiv · show

The versatility of large language models (LLMs) led to the creation of diverse benchmarks that thoroughly test a variety of language models' abilities. These benchmarks consist of tens of thousands of examples making evaluation of LLMs very expensive. In this paper, we investigate strategies to reduce the number of evaluations needed to assess the performance of an LLM on several key benchmarks. For example, we show that to accurately estimate the performance of an LLM on MMLU, a popular multiple-choice QA benchmark consisting of 14K examples, it is sufficient to evaluate this LLM on 100 curated examples. We release evaluation tools and tiny versions of popular benchmarks: Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0. Our empirical analysis demonstrates that these tools and tiny benchmarks are sufficient to reliably and efficiently reproduce the original evaluation results.

1. Introduction

Large benchmark evaluations are costly, motivating methods that estimate LLM performance from small, curated subsets. Across four benchmarks, the paper finds that 100 curated examples can estimate performance with about 2% average error.

Motivation: Benchmark evaluation can be computationally, environmentally, and financially costly, with HELM evaluations reported to exceed 4K GPU hours or $10K for APIs.Repeated evaluations for checkpoints, prompting strategies, and hyperparameters further increase costs.
Findings: On MMLU, IRT++ predicts recent LLMs’ accuracy within 1.9% using 100 examples instead of the full 14K-example benchmark.The figure compares predicted accuracy from the curated subset with true accuracy on all MMLU examples.
Approach: The paper evaluates stratified sampling, correctness-based clustering, and Item Response Theory strategies for selecting evaluation examples.IRT represents examples by latent difficulty and abilities, then uses clustering to identify robust evaluation sets and improve estimates.
Findings: 100 curated examples per scenario estimate performance across Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0 within about 2% error on average.The authors release tiny versions of these benchmarks and IRT-based estimation tools.

2. Problem statement

The paper frames efficient benchmarking as estimating each scenario’s full-benchmark performance from a small subset of examples. Scenario scores are then aggregated into the benchmark-level metric.

Benchmark structure: Benchmarks contain scenarios and possibly sub-scenarios, each composed of hundreds or thousands of examples.MMLU and HellaSwag are scenarios, while MMLU includes sub-scenarios such as marketing and elementary mathematics.
Objective: The full benchmark score averages scenario performances or uses HELM’s mean-win-rate, with sub-scenarios typically averaged equally.Unequal sub-scenario sizes require a weighted average so each sub-scenario remains equally important.
Setup: The estimation strategies assume correctness evaluations for previously evaluated LLMs on the entire benchmark are available.These correctness data support selecting examples and estimating new models’ full-benchmark performance.
Objective: For each scenario, the objective is to choose a small subset of examples that estimates a new LLM’s score on the complete example set.The subset is denoted bI_j within the scenario’s example set I_j.

3. Selecting evaluation examples

The paper selects representative examples within each scenario using stratified sampling, correctness-based clustering, or IRT-derived representations. Clustering assigns weights according to how many examples each selected anchor represents.

Stratified sampling: Stratified random sampling selects examples while preserving representation of predefined groups such as sub-scenarios.Prior work found this effective for sampling HELM examples without large losses in ranking LLMs.
Clustering: Correctness-based clustering embeds each example by previously evaluated LLMs’ correctness scores, then selects one example nearest each K-Means centroid.For a new LLM, each anchor receives a weight equal to the fraction of scenario examples assigned to its cluster.
Clustering: Correctness clustering can suffer from distribution shifts and the curse of dimensionality when many training LLMs define the embeddings.Correctness patterns may vary over time, reducing the reliability of anchors under changed distributions.
IRT-based selection: IRT representations encode each example through difficulty and the abilities required for correct responses, producing relatively low-dimensional embeddings for clustering.The resulting anchors are intended to represent examples with similar difficulty and required abilities while potentially alleviating distribution-shift problems.

4. Better performance estimation with IRT

The paper uses IRT to estimate LLM performance from a subset of observed examples, leveraging model abilities and example parameters to predict unseen correctness. It extends this estimator with a generalized combination that trades off sampling variance against IRT bias.

IRT model: IRT models correctness using latent LLM abilities, example requirements, and example-specific bias terms.The model estimates example representations from α_i and β_i parameters.
p-IRT estimator: p-IRT estimates unseen-example performance by combining observed correctness with IRT-predicted probabilities, weighted by the observed sample fraction.The observed-set weight is ˆλ = |bI_j|/|I_j|.
p-IRT estimator: p-IRT converges in probability to the conditional-expectation approximation of scenario performance as the sampled set grows, under stated assumptions.The assumptions include consistent ability estimation and bounded example parameters.
p-IRT limitations: p-IRT can deteriorate when predicted probabilities are inaccurate and does not readily support sample weighting for anchor points.Model misspecification is given as one source of inaccurate predicted probabilities.
gp-IRT estimator: gp-IRT forms a convex combination of estimators, choosing λ to balance low-bias sampling estimates against lower-variance but potentially biased IRT estimates.The heuristic increases the weight on the raw-data estimator as its variance decreases or the IRT estimator’s bias becomes more concerning.
Bounded correctness: For bounded correctness scores, the method binarizes responses using a scenario-dependent threshold so standard IRT tools can be applied.The transformed variable is ˜Y_i_l = 1[Y_i_l ≥ c].

5. Assessing evaluation strategies

The evaluation compares subset-based performance estimates with full-benchmark performance across four benchmarks, model splits, and selection strategies. Results show that small curated subsets, especially IRT-based methods, can estimate performance accurately while remaining robust to temporal and specialization shifts.

Evaluation setup: The evaluation compares each subset-based estimate with the LLM’s performance on the complete benchmark.Two train-test scenarios are used: random splits and date-based splits, with recent models reserved for testing in the latter.
Overall results: 100 examples per scenario achieve approximately 2% average performance estimation error across benchmarks and evaluated LLMs.The corresponding subset sizes are 600/29K for Open LLM Leaderboard, 100/14K for MMLU, 1000/10K for HELM, and 100/800 for AlpacaEval 2.0.
Overall results: IRT-based methods perform consistently well across benchmarks and train-test splits, and IRT++ always improves or matches its vanilla counterpart.The IRT++ adjustment adds only a few seconds to evaluation time.
Distribution shifts: Correctness-based example selection deteriorates most on specialized LLMs, whereas IRT-based anchors are only slightly affected.The specialization test uses 40 hand-picked models fine-tuned for domains such as code, biology, or finance.
Error analysis: With IRT++ and 100 MMLU examples, estimation error never exceeds 4% except for one extremely low-performing LLM.More capable models have slightly lower error, although the analysis finds no strong dependency.

6. Conclusion

The paper shows that psychometric methods can accurately assess LLM capabilities using far fewer benchmark examples, reducing evaluation costs and enabling more frequent evaluation. It also explores prompt prediction and adaptive testing, while noting that distribution shifts can limit estimation accuracy.

6. Conclusion: A fraction of benchmark examples, sometimes two orders of magnitude fewer, can accurately assess LLM capabilities using psychometric models.The approach reduces monetary, computational, and environmental evaluation costs.
6.1. Extensions: Prompt-based evaluation can predict performance across new prompt templates using only a few evaluations.The paper trains an IRT model on evaluations across LLaMA models and tests random and distribution-shifted splits.
6.1. Extensions: Adaptive IRT testing improves estimation performance, but the current implementation takes over 5 minutes to run.The method selects examples during evaluation rather than pre-selecting them beforehand.
6.2. Limitations: Severe distribution shifts can increase estimation errors when models' correctness patterns differ from those represented by the curated examples.The authors recommend periodically updating curated examples and IRT parameters with data from newer LLMs.

A. Evaluation when subscenarios have different number of samples

When benchmark scenarios contain unequal numbers of samples, the paper represents model performance as a weighted average rather than a simple average. The weighting differs by estimation strategy.

A. Evaluation when subscenarios have different number of samples: Performance for a scenario composed of multiple subscenarios is estimated as a weighted average over examples.The subscenarios are disjoint and their union forms the full scenario.
A. Evaluation when subscenarios have different number of samples: Clustering-based estimation assigns each anchor point the combined normalized weights of the examples in its cluster.Stratified random sampling instead uses equal representation of subscenarios, assigning examples weight 1/|Î_j|.

B. tinyMMLU

tinyMMLU uses 100 IRT-selected examples and associated weights, with more uniform weighting than a correctness-based alternative. Its subject weights are uneven across the 57 MMLU subjects.

B. tinyMMLU: 100 examples and IRT-derived weights form tinyMMLU's selected evaluation set.The examples were chosen based on the best test performance across random seeds, with a correctness-anchor version selected for comparison.
B. tinyMMLU: tinyMMLU has more uniform example weights than its correctness-based counterpart.Weight uniformity is measured using effective sample size, where higher uniformity supports robustness across varying correctness patterns.
B. tinyMMLU: The highest-weighted MMLU subjects are high school psychology, elementary mathematics, and professional law.Subject weights differ substantially from those produced by the correctness-based method.

C. Proof of Proposition 4.1

The proof of Proposition 4.1 bounds the difference between conditional expectations using the proportion of examples outside the estimated set and properties of the response function.

C. Proof of Proposition 4.1: The proposition's bound compares conditional expectations and scales with the fraction of examples excluded from the estimated set.The proof then invokes the response function's 1/4-Lipschitz property and the Cauchy–Schwarz inequality.

D. More details about benchmarks

The paper evaluates benchmark datasets spanning the Open LLM Leaderboard and HELM, using collections of LLMs and thousands of evaluation items. Dataset versions, scenario coverage, and leaderboard-score handling are specified.

Open LLM Leaderboard: The Open LLM Leaderboard data include 395 LLMs and approximately 29k items downloaded in January 2024.Models were filtered by MMLU score, ordered by average performance, and selected to cover all six scenarios.
HELM: The HELM Lite v1.0.0 dataset contains 37 LLMs and approximately 10k evaluation examples across 10 scenarios.Listed scenarios include OpenbookQA, MMLU, NarrativeQA, NaturalQuestions, and Math.
Score handling: Leaderboard scores may differ because the study uses each model’s last submission rather than the leaderboard’s best submission.This distinction applies to the actual scores used in the analysis.

E.1. Robustness in predicting performance in a longer time horizon

The robustness analysis tests whether IRT++ remains reliable when the test set extends farther into the future. Using 75% of the most recent data as the test set, the results do not differ considerably from the main text.

Robustness setup: IRT++ remains fairly robust when predicting new LLM performance over a longer time horizon.The extended setup places 75% of the data in the test set, corresponding to three months of future data for Open LLM Leaderboard and MMLU and six months for AlpacaEval 2.0.
Results: The longer-horizon results do not differ considerably from those reported in the main text.The analysis reports average estimation error in the test set and standard deviation across LLMs.
Evaluation measure: The extended test set uses 75% of the most recent data to evaluate average estimation error and variability across LLMs.This provides the evaluation basis for the longer-horizon robustness plots.

E.2. How costly is it for stratified random sampling beat IRT++ with larger samples?

The comparison shows that stratified random sampling generally needs more evaluation examples than IRT++ to reach comparable estimation performance. The section also examines running time, rank correlation, adaptive testing, and individual benchmark scenarios.

Sample-size comparison: 400 examples per task for the Open LLM Leaderboard match IRT++ using 100 examples per task, while MMLU random sampling requires more than 400 examples to match it.For the Open LLM Leaderboard, the comparison is 2400 total random examples versus 600 total IRT++ examples.
Sample-size comparison: On AlpacaEval, random sampling with 200 examples matches IRT++ with 100 examples.AlpacaEval has 805 examples total and requires GPT-4 for evaluation, making evaluations expensive.
Computational cost: The average running time of IRT inference for ability-parameter fitting is fairly negligible.Running time is reported as a function of the number of test examples.
Ranking performance: Rank correlation results indicate that the method can rank LLMs efficiently with tiny samples.The analysis compares true and predicted rankings, including MMLU settings with random and specialized-model test splits.
Additional analyses: Additional experiments examine adaptive testing and performance patterns across individual scenarios and benchmarks.The figures cover ARC, GSM8K, TruthfulQA, HellaSwag, MMLU, Winogrande, OpenbookQA, GSM, LegalBench, Math, MedQA, NarrativeQA, NaturalQA, and WMT14.