PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Evaluation of LLM Performance with Statistical Guarantees

Skyler Wu, Yash Nair, Emmanuel J. Candès

arXiv:2601.20251v3stat.MLcs.LG

TL;DR

Exhaustive evaluation of many LLMs on large, costly benchmark suites motivates finite-population inference with tight confidence intervals and valid frequentist coverage. FAQ uses historical factor modeling, hybrid adaptive querying, and Pro-Active Inference to select questions directly while preserving validity, achieving up to 5× higher effective sample size than uniform sampling and strong baselines at negligible overhead.

  • Problem

    Evaluating many LLMs across large, expensive benchmark suites requires estimating finite-bank accuracy under a query budget while retaining valid uncertainty coverage.

  • Method

    FAQ combines a Bayesian factor model for historical outcomes, hybrid variance-reduction and active-learning query selection, and Pro-Active Inference for coverage-preserving direct question selection.

  • Results

    FAQ achieves up to 5× higher effective sample size than uniform sampling and up to 2.4× higher effective sample size than the strongest baselines while maintaining frequentist coverage.

  • Takeaways & Limitations

    FAQ can make repeated finite-bank evaluation and monitoring more cost-effective by reusing historical data while adding fresh outcomes under limited query budgets.

  • Takeaways & Limitations

    The factor-model instantiation is illustrative rather than fundamental, and efficiency depends on regularity conditions and the quality of predictive guidance.

Abstract

from arXiv · show

Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with valid frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Proactive Active Inference -- a finite-population extension of active inference (Zrnic & Candès, 2024) that enables direct question selection while preserving coverage. With negligible overhead cost, FAQ delivers up to $5\times$ effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels: this means that it matches the CI width of uniform sampling while using up to $5\times$ fewer queries. We release our source code and our curated datasets to support reproducible evaluation and future research.

1 Introduction

FAQ frames LLM benchmarking as finite-population inference: under a fixed query budget, it seeks narrow confidence intervals for model accuracy while preserving frequentist coverage. It combines historical information, adaptive question selection, and coverage-preserving inference, reporting up to 5× query-efficiency gains with negligible overhead.

  • Motivation: Large, growing benchmark suites make exhaustive evaluation of many candidate LLMs expensive, especially when questions require expert adjudication, interaction, or long-form verification.Organizations may otherwise rely on ad hoc subsets or subjective judgments.
  • Problem formulation: FAQ estimates a new model’s accuracy over a fixed finite benchmark bank using a limited budget of adaptively selected questions.Historical outcomes may be partially observed and can inform the selection process.
  • Problem formulation: Valid frequentist coverage is essential because miscalibrated uncertainty can produce overconfidence and poor model-selection or deployment decisions.The risk is heightened when adaptive evaluation invalidates i.i.d.-based assumptions or heuristic error bars.
  • Approach: FAQ combines a Bayesian factor model for partially observed history, a hybrid variance-reduction and active-learning policy, and Pro-Active Inference for direct question selection.PAI extends active inference to select questions data-adaptively while preserving coverage.
  • Results: Up to 5× fewer queries yield confidence intervals comparable in width to competitive baselines, including uniform sampling, with little to no additional computational cost.The evaluation spans two benchmark suites and varying historical-data missingness.

2 Related Work

FAQ uses latent factor modeling to extract predictive information from incomplete historical model-question outcomes, while treating that model as an efficiency aid rather than a source of inferential validity. It combines active-learning ideas with variance reduction and differs from prior approaches in targeting finite-bank accuracy with model-free coverage.

  • Factor modeling: Probabilistic factor modeling represents partially observed binary model-question outcomes with low-dimensional latent factors, but its uncertainty guarantees depend on correct model specification.FAQ instead uses the factor model to guide querying while retaining coverage regardless of factor-model fidelity.
  • Active inference: FAQ adapts active statistical inference to benchmark banks by constructing latent features from factor models when explicit question covariates are unavailable.Unlike sequential active inference, the benchmark setting supports proactive selection from the full question pool.
  • Active learning: The hybrid policy combines active-learning scores for posterior variance reduction with an oracle-inspired variance-reduction component aimed at tight frequentist uncertainty quantification.This combines learning the model representation with reducing uncertainty about accuracy.
  • LLM evaluation: Prior LLM evaluation methods often target point estimation, ranking, selection, or benchmark design rather than model-free coverage for each model’s finite-bank accuracy.FAQ therefore compares against procedures that share its inferential target and guarantees.

3 Method

FAQ uses latent factors learned from historical outcomes to guide adaptive querying, then wraps those predictions in Pro-Active Inference for valid finite-bank confidence intervals. Its hybrid policy balances variance reduction against exploration, while the factor model remains a working approximation rather than a validity assumption.

  • Factor Model: FAQ introduces latent model and question factors because the historical outcome matrix contains no explicit covariates.A logistic link models correctness, and the resulting predictions provide uncertainty for unqueried model-question pairs.
  • Factor Model: The factor model is fit on observed historical entries, question factors are fixed for deployment, and a new model factor is initialized from historical model factors.Online updates refine the new model factor after each observed outcome.
  • Pro-Active Inference: FAQ uses the factor model only to improve query efficiency, while PAI supplies model-free coverage without assuming i.i.d. benchmark questions or correct factor-model specification.The target is the fixed finite-bank accuracy rather than a superpopulation quantity.
  • Pro-Active Inference: PAI samples questions from an adaptive distribution under a fixed budget and constructs an estimator that is unbiased for finite-bank accuracy with an asymptotic confidence interval under regularity conditions.The result follows from a martingale central limit theorem and yields approximately nominal coverage for sufficiently large problem sizes.
  • Pro-Active Inference: PAI requires sampling with replacement for the stated coverage conditions, whereas ad hoc without-replacement variants can fail to provide valid coverage.This is an explicit design constraint of the inferential procedure.
  • Hybrid Sampling: The sampling policy trades off exploitation of current variance-reduction scores against exploration that quickly improves the model-factor estimate and subsequent predictions.The hybrid policy combines these scores through normalization, mixing, tempering, and probability lower-bounding.

4 Experimental Setup

The experiments evaluate FAQ on two benchmark suites using historical model outcomes, controlled missingness, fixed query budgets, and coverage and ESS metrics. They compare FAQ with uniform sampling and active-inference baselines while auditing deployment shifts and cold-start transfer.

  • Datasets: The study uses 4.4K models and 21.6K questions across MMLU-Pro and a five-dataset reasoning suite.The suites contain 12K and 9.5K questions, respectively.
  • Experimental design: Historical models are split chronologically into 2.2K training models and 2.2K new test models, with missingness induced by complete rows and MCAR entry masking.Missingness varies through the number of fully observed rows and the observation probability on remaining entries.
  • Metrics and budgets: The evaluation targets 95% nominal coverage across ten evenly spaced budgets spanning 2.5% to 25% of each question bank.Each method produces clipped Wald intervals, and coverage is averaged over 100 seeds and 2.2K test models.
  • Baselines: Comparisons include uniform sampling, sequential active-inference variants, and post-hoc-selected baseline configurations that favor baseline performance.FAQ is tuned only on historical train/validation data, whereas baselines are selected using test models for an oracle-style upper bound.
  • FAQ procedure: FAQ fits a factor model on historical data, then adaptively samples questions with a hybrid probability policy over a fixed query budget.The algorithm initializes a prior for the new model’s latent factor and updates it after each queried outcome.
  • Metrics and budgets: ESS measures the uniform-sampling budget that would achieve the same variance as a method at the tested budget, so larger ESS indicates greater query efficiency.An ESS multiplier of m corresponds to matching uniform sampling’s CI width with roughly 1/m of the queries.
  • Additional evaluations: The study also examines progressively sparse historical data, per-model coverage under release-date and accuracy shifts, and cold-start transfer from MMLU-Pro to GPQA.The cold-start setup has zero target-suite historical data for factor-model fitting and uses transferred factors and question embeddings.

5 Empirical Results

FAQ improves effective sample size over uniform sampling and strong baselines while maintaining coverage near the nominal target, including under substantial historical-data missingness. It also shows smaller gains in cold-start transfer and a low-budget advantage over a sequential active-inference ablation.

  • Fully-observed history: With fully observed history, FAQ achieves 4–5× higher ESS than uniform sampling and 1.8–2.4× higher ESS than the strongest post-hoc baseline.Empirical coverage remains near the 95% target across budgets.
  • Partially-observed history: With 12–42% of historical data observed, FAQ substantially outperforms the strongest post-hoc baselines in ESS.With only 0.1% of historical entries observed, FAQ retains a modest but consistent ESS edge except at the two largest budgets.
  • Cold-start transfer: Cold-start FAQ reaches ESS multipliers up to 1.32× over uniform sampling and beats post-hoc baselines with 0.1% target-suite history on 8 of 10 budgets.The cold-start result uses zero target-suite historical data for factor-model fitting.
  • Cold-start transfer: The results suggest that cross-benchmark structure transferred through LLM embeddings can make FAQ useful before target-suite history accumulates.The paper frames this as a possible onboarding path for continuous monitoring.
  • Active-inference ablation: FAQ yields narrower CIs than the factor-model-matched sequential active-inference ablation at low budgets, while the ablation can match or outperform FAQ at larger budgets.The low-budget regime is the one most relevant for query-cost savings.

6 Discussion and Conclusion

FAQ combines historical-data modeling with adaptive querying to make finite-bank LLM evaluation more cost-effective while preserving model-free frequentist coverage. Its framework is flexible beyond the simple factor-model implementation and supports repeated monitoring, but extensions remain future work.

  • Discussion and Conclusion: FAQ achieves up to 5× higher ESS than uniform sampling and 2.4× higher ESS than the strongest baselines while maintaining model-free frequentist coverage.The framework operates under fixed query budgets and negligible overhead.
  • Discussion and Conclusion: FAQ reuses accumulated historical data and adds fresh outcomes, making uncertainty-quantified evaluation progressively more cost-effective for repeated model comparisons.The paper highlights checkpoints, fine-tunes, distillations, quantized models, and deployed systems as recurring evaluation targets.
  • Discussion and Conclusion: The factor-model implementation is illustrative rather than fundamental: any probability-outputting predictive model and adaptive policy satisfying PAI regularity conditions can retain valid coverage.The coverage guarantee is model-free and does not depend on the specific factor model.
  • Discussion and Conclusion: FAQ currently targets fixed benchmark banks with binary task-completion outcomes, while non-binary feedback and batched querying remain extensions for future work.The paper also suggests updating question factors after deployment waves and using drift-aware priors to improve efficiency without changing validity.
  • Theoretical Analysis: The theoretical analysis uses martingale arguments and asymptotic assumptions including variance stabilization, a Lindeberg condition, and conditional variance control.The stated assumptions support asymptotic normality and consistency of the variance estimator.

A.5 Proof of Theorem 3.2

The proof minimizes the variance of the estimator by reducing the problem to identical instantaneous optimization problems across sampling rounds. Strict convexity yields a unique oracle sampling distribution.

  • Proof of Theorem 3.2: Because the estimator is unbiased, minimizing its variance reduces to minimizing the variance of the martingale sum M_nb.The proof uses E[M_nb] = 0 and the absence of temporal dependence between sampling distributions in the oracle setup.
  • Proof of Theorem 3.2: The total variance decomposes into nonnegative instantaneous contributions, so each time step solves the same variance-minimization problem.The contribution is expressed as a sum involving p_j(1 − p_j)/q_t(j).
  • Proof of Theorem 3.2: Strict convexity of the optimization problem makes the oracle solution the unique minimizer.The positivity of q_t(j) and nonnegativity of p_j(1 − p_j) establish the relevant convexity conditions.

A.6 Derivation of (Approximate) Maximal Variance-Reducing Index

The approximate variance-reduction rule evaluates how querying each question would change the posterior variance of the target accuracy. The next index is selected to maximize this estimated reduction.

  • A.6 Derivation of (Approximate) Maximal Variance-Reducing Index: The posterior variance of θ* is approximated using the multivariate Delta method and the covariance of the updated model-factor distribution.The gradient of the target function maps uncertainty in the latent factor vector to uncertainty in θ*.
  • A.6 Derivation of (Approximate) Maximal Variance-Reducing Index: Because the next outcome is unknown, the method plugs in the current mean parameter estimate when approximating each candidate question’s variance reduction.This produces a pre-query score for every possible next index.
  • A.6 Derivation of (Approximate) Maximal Variance-Reducing Index: The selected index I_t is the question whose query is predicted to maximally reduce the approximate posterior variance of θ*.The index is chosen by maximizing the derived reduction score across candidate questions.

B Investigating Sampling With/Without Replacement

FAQ/PAI requires sampling with replacement for its stated uncertainty guarantees. An ad-hoc without-replacement modification breaks the estimator’s theoretical alignment, causing miscoverage without materially improving RMSE.

  • Sampling With/Without Replacement: With-replacement sampling aligns the draw distribution with the inverse-probability correction, supporting the martingale structure and regularity conditions behind valid inference.The sampling policy and estimator use the same q_t distribution over the full question bank.
  • Sampling With/Without Replacement: The ad-hoc without-replacement variant uses a restricted distribution but retains formulas based on q_t, so martingale differences and lower-bounded sampling probabilities are lost.This can produce unbounded inverse-probability corrections and undermine the martingale CLT argument.
  • Sampling With/Without Replacement: At moderate-to-large budgets, the ad-hoc without-replacement variant suffers substantial miscoverage, while the difference is smaller at low budgets because repeats are rarer.The reported Figure 6 experiment uses a 95% targeted nominal coverage level.
  • Sampling With/Without Replacement: The without-replacement modification yields little RMSE improvement, so the paper recommends sampling with replacement to preserve uncertainty quantification.The recommendation accepts minimal loss in point-estimation performance in exchange for theoretically guaranteed uncertainty quantification.

C Additional Experimental Details

The experiments use two benchmark suites and evaluate FAQ under varied historical-data availability, including extremely sparse and cold-start transfer settings. The implementation uses selected factor-model and FAQ hyperparameters, with released code and datasets for reproducibility.

  • Datasets: The benchmarks contain 12,032 MMLU-Pro questions and 9,574 questions aggregated from BBH, GPQA, IFEval, MATH, and MuSR.
  • Historical splits and missingness: Non-cold-start comparisons include fully observed history and missingness settings with 50, 200, or 800 fully observed rows at pobs = 0.1, plus an extremely sparse (0, 0.001) setting.
  • Historical splits and missingness: Traditional active-inference ablations use fully observed data and four extremely sparse settings with nfull-obs = 0 and pobs ranging from 0.01 to 0.00001.
  • Factor model selection: Factor-model selection sweeps latent dimension k over {8, 16, 32, 64} and weight decay λ over 10^1 through 10^-5, using AdamW for up to 2000 iterations.
  • FAQ hyperparameters: FAQ hyperparameter sweeps vary the maximum tempering exponent, tempering governor, exploration-to-exploitation governor, and uniform-mixing strength.
  • Cold-start and cross-suite transfer: Cold-start transfer fits factors on fully observed MMLU-Pro data with k = 8, λ = 1.0, and 2000 AdamW iterations before initializing the new-suite prior.
  • Reproducibility: The source code and analysis routines are publicly released for reproducing the paper’s results.

D Traditional Active Inference Ablation

The traditional active-inference ablation uses the same factor-model predictions but follows a fixed question order, revealing a budget-dependent trade-off with FAQ. FAQ is stronger at low budgets, while the traditional procedure can overtake it at larger budgets.

  • Ablation design: The ablation follows sequential active inference in the benchmarks’ native question order while retaining the same factor-model predictions as FAQ.
  • Results: At budgets below 10% of the bank, FAQ yields substantially narrower confidence intervals than the traditional ablation.
  • Results: At larger budgets, the traditional ablation eventually overtakes FAQ by similar margins.
  • Implications: The results motivate a three-stage policy combining active learning early, oracle-style variance reduction mid-budget, and traditional active inference late.
  • Mechanisms: FAQ can proactively target informative questions early, whereas traditional active inference makes label-or-skip decisions only within a fixed stream.
  • Mechanisms: FAQ’s with-replacement sampling can become less efficient at large budgets, while stream-based procedures touch each question at most once.

E Additional Results

Additional figures evaluate effective sample size, confidence-interval width, and coverage under missing historical data, per-model coverage, and cold-start coverage. These results extend the main experiments across missingness settings and confirm valid coverage in the reported cases.

  • Effective sample size: Effective sample size is compared for FAQ, the strongest post-hoc baseline, and uniform sampling across budgets and missingness settings on two benchmark suites.
  • Coverage: Coverage under missing historical data is reported for FAQ, the strongest post-hoc baseline, and uniform sampling across budgets and missingness settings.
  • Confidence intervals: Confidence-interval widths are evaluated for FAQ, the strongest post-hoc baseline, and uniform sampling under varying historical-data missingness.
  • Per-model coverage: Per-model coverage on MMLU-Pro is examined against model release date and true model accuracy at a 7.5% budget.
  • Cold-start coverage: Cold-start FAQ coverage on GPQA is compared with a strongest baseline using 0.1% target-suite historical data and with uniform sampling.
Loading 2601.20251v3…