PaperlayerView exact revision on arXiv

Source-linked AI summary

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

Sho Kawano, Zehang Richard Li, Paul A. Parker

arXiv:2609.20758v1stat.MLcs.AIcs.LGstat.APstat.ME

TL;DR

Disaggregated evaluation must estimate performance across domains, but collecting and grading outcomes is costly and direct estimates are imprecise when labels are sparse. The paper develops smoothed estimators and a design-based validation score; across benchmark and traffic studies, the estimators improve point and interval estimates with near-nominal coverage.

  • Problem

    Disaggregated evaluation seeks domain-specific means from costly samples, while direct estimators can be imprecise where few units are labeled.

    Source: §1 · pp. 1–2

  • Method

    The paper fits Bayesian Fay–Herriot smoothing to domain prediction-powered estimates, extends borrowing across a taxonomy, and proposes an approximately unbiased design-based cross-validation score.

    Source: §1 · p. 3, §3.2 · p. 7

  • Results

    The proposed estimators improve on direct estimators in point and interval estimation in both settings, with near-nominal coverage.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    A finite-population, design-based interpretation keeps domain means and interval coverage meaningful under distribution shift without a superpopulation assumption.

    Source: §1 · p. 2

    Estimating performance in an unseen domain with no labeled units requires domain-level covariates to be available.

    Source: §5 · p. 14

Abstract

from arXiv · show

Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in deployed agents. Exhaustive testing is expensive, so evaluation rests on a sample of labeled units. We treat the evaluation set as a finite population and seek accurate point and interval estimates of each domain mean. Direct estimators, including prediction-powered inference (PPI), use only a domain's own labels and are imprecise where labels are few. Small area estimation addresses this problem, and we build on it to develop an integrated workflow for estimation and validation. For estimation, we propose prediction-powered smoothing (PP-S), a Bayesian model fit to each domain's prediction-powered estimate, with an extension that borrows strength across a reporting taxonomy (PP-TS). For validation, we derive a new, approximately unbiased design-based cross-validation score for choosing among direct and smoothed estimators. We study a curated benchmark with verifiable grading and deployed agent traffic graded by humans, each with every outcome observed. In both, the proposed estimators improve on the direct estimators in point and interval estimation, with near-nominal coverage. At the same sampling budget, our score selects as well as an independent validation sample does and estimates the selected estimator's error far more accurately.

1 Introduction

Disaggregated AI evaluation needs estimates for many domains, but obtaining and grading outcomes can be costly. The paper combines finite-population estimation, smoothing across domains, and validation to address imprecision and model selection.

1 Introduction

  • Evaluation can require costly querying and grading, from automatic scoring of verifiable tasks to human grading of open-ended tasks.For deployed traffic, interactions have already occurred, leaving review and grading as the remaining costs.Source: §1 · p. 2
  • The paper treats the evaluation set as a finite population and targets each domain’s fixed mean, with intervals assessed by repeated-sample coverage.This design-based interpretation needs no superpopulation distribution assumption and remains meaningful under distribution shift.Source: §1 · p. 2
  • Direct estimators such as PPI and GREG use a domain’s own data, making them imprecise when the sampling budget is spread across many domains.Area-level Fay–Herriot models instead shrink direct estimates toward a shared regression, with more shrinkage for less precise estimates.Source: §1 · p. 2
  • Prior work leaves open how to choose among smoothing models, while existing validation approaches do not readily compare direct and model-based estimators.The paper introduces PP-S, an area-level Bayesian Fay–Herriot model of domain GREG estimates, and PP-TS, which borrows strength along a nested domain hierarchy; its DB-CV refinement replaces a conservative bias bound with an approximately unbiased score.Source: §1 · p. 3
  • The proposed smoothing improves on direct estimators in point and interval estimation across benchmark and deployed-traffic studies, with near-nominal coverage.The workflow also uses a debiased score that matches independent-sample selection while estimating the selected estimator’s error more accurately.Source: §1 · p. 3

2 The Case for Probability Sampling

The section argues that valid domain estimates require a probability sample. Sampling based on response selection can produce persistent bias and poor interval coverage that PPI does not remove.

2 The Case for Probability Sampling

  • The paper assumes a probability sample; without one, domain estimates can be biased unless selection is modeled explicitly, and that modeling assumption cannot be checked from labeled data alone.In deployed traffic, human review may be triggered by complaints, flags, or reviewer attention rather than probability sampling.Source: §2 · pp. 3–4
  • Moderate bias persists as the sampling budget grows and 95% interval coverage falls well below nominal under rejection-biased sampling; PPI does not fix the selection bias.At the 5% budget, the correlation between being labeled and the outcome is 0.029 in magnitude.Source: §2 · p. 4, Figure 1 · p. 4
  • Because neither selection nor its correction can be checked from labels alone, the paper treats probability sampling as the foundation for disaggregated evaluation.Source: §2 · p. 4

3 Methods

The methods combine design-consistent domain estimates with Bayesian smoothing and design-based cross-validation to estimate and validate performance across domains.

3.1 Direct Estimation with Auxiliary Information

  • HT estimates domain means from sampled outcomes, while PPI adds an auxiliary-based correction and GREG estimates its correction slope by pooled weighted regression.GREG accepts unit-level covariates beyond outcome-scale predictions; its slope can shrink toward zero when the auxiliary has little predictive value.Source: §3.1 · pp. 5–6

3.2 Prediction-Powered Smoothing

  • Area-level smoothing uses design-consistent domain estimates and their variances, but its normal sampling approximation and linking model can be weakest in small domains.Misspecifying the linking model can cost precision, particularly in the domains that borrow most.Source: §3.2 · p. 6
  • PP-S applies the Fay–Herriot model to GREG domain estimates, balancing each estimate against a shared regression according to its precision.Noisy domains borrow more when the linking model fits; precise estimates remain close to their direct estimates, and the estimator is design-consistent as sample sizes grow.Source: §3.2 · pp. 6–7
  • PP-TS replaces a single domain effect with effects at each taxonomy level, borrowing most strongly among domains that are closest in the hierarchy.Each level’s variance is estimated, allowing levels without residual category differences beyond covariates to receive variance near zero.Source: §3.2 · p. 7

3.3 Design-Based Validation

  • DB-CV splits each domain’s sample into probability-sample folds and adjusts cross-validation scores for fold-level and sample-level bias.The sample-level correction assumes each fit omitting one fold centers on the full-sample fit up to a small remainder.Source: §3.3 · p. 8

4 Results

Across the benchmark, prediction-powered smoothers outperform their baselines, and PP-TS outperforms all estimators with near-nominal coverage. On PRISM traffic, DB-CV selects as well as independent-sample validation and reports error closest to the oracle.

4 Results

  • The studies use a 10% stratified sample budget with proportional allocation and 100 replications, evaluating benchmark task types and PRISM’s 63 conversation-type-by-LLM domains.The benchmark compares estimators against the oracle; PRISM demonstrates a single-sample workflow and evaluates DB-CV.Source: §4 · p. 9

4.1 Estimator Comparison: Open LLM Leaderboard

  • PP-TS outperforms every direct and smoothing estimator on RMSE and interval score under all three auxiliaries, with coverage between 0.93 and 0.95.Prediction-powered smoothers also beat their baselines on both metrics under every auxiliary.Source: Table 2 · p. 11, §4.1 · p. 11
  • The largest PP-TS gains generally come from the auxiliary’s domain-mean covariate; taxonomy smoothing captures remaining structure, while intercept-only pooling barely improves on HT.The two weak auxiliaries have nearly identical unit-level correlations, yet the generalist’s domain mean lowers RMSE more than the specialist’s.Source: §4.1 · p. 11, Figure 3 · p. 12

4.2 Workflow and Validation: PRISM Agent Traffic

  • On one PRISM sample, DB-CV selects PP-S with judge and content covariates and orders all seven candidates exactly as the oracle RMSE does.Every candidate with content covariates has a lower score and narrower interval than its counterpart without them.Source: Table 3 · p. 13, §4.2 · p. 13
  • In the sparsest quarter of PRISM domains, PP-S intervals average 7 points, versus 15 for HT and 12 for GREG with all covariates.Smoothing narrows intervals most where HT is least precise; these domains also have the lowest satisfaction.Source: §4.2 · p. 13, Figure 4 · p. 14
  • DB-CV selects as well as independent-sample procedures at the same budget and is the only procedure whose reported error is close to the oracle.Naive CV and sample splits overstate the selected candidate’s error several times over.Source: §4.2 · pp. 13–14

5 Conclusion

Across the benchmark and deployed agent traffic, prediction-powered smoothing improves point and interval estimates over direct estimators while maintaining near-nominal coverage. The conclusion also identifies open questions around zero-shot evaluation and further integration of survey methods with AI evaluation.

5 Conclusion

  • PP-S improves point and interval estimates over direct estimators in both settings, with near-nominal interval coverage.The settings are a benchmark with verifiable grading and deployed agent traffic graded by humans.Source: §5 · p. 14
  • At equal sampling budget, the debiased score selects as well as two independent samples and estimates the selected estimator’s error accurately, while alternatives overstate it several times over.The independent-sample comparison uses one sample for training and the other for validation.Source: §5 · p. 14
  • The gains from smoothing depend on auxiliary information and the reporting taxonomy, and the useful structure differs by dataset.In the benchmark, shrinkage across domains alone accounts for little of the improvement.Source: §5 · p. 14
  • Smoothing may estimate performance in unseen domains without labeled units when domain-level covariates are available, but the reliability of zero-shot domain evaluation remains open.The paper identifies this as an immediate extension, not an established validation result.Source: §5 · pp. 14–15
  • Further integration could account for grader disagreement as measurement error and combine probability samples with large nonprobability samples to guard against selection bias.The paper presents these as examples of untapped potential in integrating AI evaluation and survey methodology.Source: §5 · p. 15

A Details for Design-Based Validation

The appendix derives the design-based validation score by separating variation from random fold assignment and sample selection. It shows how the adjusted score retains sample-level bias and how an additional correction yields the debiased score under a condition on training fits.

A Details for Design-Based Validation

  • The validation target is the candidate estimator’s mean squared error averaged over folds, with expectations taken over both the sampled set and its random fold split.For each fold, the held-out Horvitz–Thompson (HT) estimate is compared with a candidate fit on the remaining folds.Source: A Details for Design-Based Validation · p. 19, §A.1 · p. 19

A.1 The Bias of the Naive Score

  • The naive score has fold-level and sample-level bias because held-out estimates center on the full-sample HT estimate rather than the domain mean.The random fold split accounts for the fold-level component; the sample draw accounts for the sample-level component.Source: §A.1 · pp. 19–21

A.2 The Adjusted Score

  • The adjusted score removes fold-level bias but retains sample-level bias, which is zero only at ρ = 1/2; for HT (ρ = 1) it is too low by d, and for a no-label candidate (ρ = 0) too high by d.The adjustment estimates the fold-level terms using conditional variances and covariances.Source: §A.2 · pp. 21, 23

A.3 The Debiased Score

  • The debiased score subtracts the adjusted score’s sample-level bias, assuming each training-fold fit centers on the full-sample fit up to a remainder of order n^-1.The derivation assumes this condition for a single candidate; the cited prior derivation concerns a difference of two candidates.Source: §A.3 · p. 23

A.4 The Covariance for Each Candidate

This appendix derives the candidate-specific covariance needed by the debiased score and the covariance structure of the smoothers. It relates smoother sensitivity to posterior variance and input sampling variance, including when variance parameters are integrated over.

A.4 The Covariance for Each Candidate

  • For PPI and GREG, the candidate-specific covariance is between weighted means of labels and residuals, not the candidate’s own variance.Both the variance and covariance are estimated from the full sample using a linearization estimator for a weighted mean.Source: §A.4 · p. 24
  • With variance parameters fixed, each smoother is linear in the domain inputs plus a constant, and independent domain sampling errors determine its covariance structure.The smoothers share an area-level model form; the diagonal coefficient links input variance to posterior variance.Source: §A.4 · pp. 24–25
  • For FH and TFH the input is the HT estimate, while PP-S and PP-TS use GREG inputs, so the covariance term depends on the input estimator.For HT, input variance and covariance both equal d, reducing the covariance to posterior variance.Source: §A.4 · p. 25
  • When MCMC integrates over variance parameters, a first-order expansion and Tweedie’s formula express the smoother’s sensitivity using the posterior variance from retained draws.The formula uses the log density of the inputs under the model; with fixed variance parameters it matches the fixed-parameter result.Source: §A.4 · pp. 25–26

B Additional Details for the Benchmark Simulation

The benchmark simulation uses stratified simple random samples across 34 task types and compares estimators under three auxiliaries and sampling budgets. It reports point error, interval score, and coverage, with matched GREG-input smoothers never worse than their HT-input counterparts.

B Additional Details for the Benchmark Simulation

  • The simulation retains 34 task types after merging advanced MATH subjects and excluding a task type answered perfectly by phi-4; domain populations range from 146 to 670, with median 250.Labels are drawn by simple random sampling without replacement within each domain.Source: B Additional Details for the Benchmark… · p. 27
  • Smoothers use Gibbs sampling with one chain, retaining 2,000 draws after 1,000 burn-in iterations; point estimates are posterior means and intervals use the 2.5% and 97.5% quantiles.FH and PP-S are fit on the original scale, while TFH and PP-TS use standardized inputs.Source: B Additional Details for the Benchmark… · p. 27
  • Auxiliaries are built from about 2,200 earlier models in a compilation of about 4,400 models, with phi-4 and its fine-tuned descendants in the newer half.The earlier half is selected by release date.Source: B Additional Details for the Benchmark… · p. 27
  • At matched linking models, GREG-input smoothing is never worse than HT-input smoothing under any auxiliary or sampling budget, paired within replication.The comparison uses the same 100 samples for every estimator and auxiliary.Source: B Additional Details for the Benchmark… · p. 27
  • Tables 5–7 report RMSE, nominal 95% interval score, and empirical coverage across sampling budgets and auxiliaries, averaged over 100 replications.Domains with zero estimated variance are omitted from metrics; this affects 0.6 domains on average at 10%, 0.1 at 20%, and none at 30%.Source: B Additional Details for the Benchmark… · p. 27, Table 5 · p. 27, Table 6 · p. 28

C.1 The PRISM Data

The PRISM analysis treats each rated response as a unit and uses observed response features to estimate participant ratings. A separate demonstration uses informative selection, while the traffic judge predicts participants’ ratings from full conversations.

C.1 The PRISM Data

  • Each rated response is a unit, so a conversation can contribute several units; content covariates are recorded for every response, even when its rating is not sampled.The listed covariates include response length, whether it answers the opening prompt, and whether the participant chose it.Source: §C.1 · p. 29
  • For traffic analysis, stratified sampling depends only on domain, so participant choice affects sampling neither directly nor through the chosen indicator, which is used only as a covariate.The demonstration instead uses informative selection, with sampling probability proportional to exp(tz), where t = −0.30 and z is the standardized chosen indicator.Source: §C.1 · p. 29
  • At the 5% budget, chosen responses were sampled at 0.54 times the rate of rejected responses, with sampling-rating correlation about 0.03 in magnitude.Participants rated chosen responses 81.7 on average versus 54.1 for the rest; participant ratings differed by about 28 points between chosen and rejected responses, compared with about 5 points for judge scores.Source: §C.1 · p. 29

C.2 The Judge

  • The traffic judge predicts the participant’s 1-to-100 rating of the final response from the full conversation, returning a score and confidence; scoring all 68,371 responses cost about six dollars.The prompt was zero-shot, excluded participant attributes, and was frozen before scoring.Source: §C.2 · p. 29

C.3 Results at the 20% Budget

At the 20% traffic budget, the reported candidate ordering and validation conclusions carry over from the 10% budget. Candidate intervals had coverage close to the nominal 95% level, and interval-score gains tracked RMSE gains.

C.3 Results at the 20% Budget

  • The oracle ordering of candidates was unchanged at 20%, and the two one-sample scores and even split remained equivalent on selection; only DB-CV reported calibrated error.Table 9 presents the four validation procedures at both budgets over 100 replications.Source: §C.3 · p. 30, Table 9 · p. 30
  • The 20% comparison used 13,674 ratings, with domain sizes from 423 to 1,809 and a median of 1,129.The results are reported alongside the 10% columns of the main text.Source: §C.3 · p. 30
  • 0.943–0.964 coverage against the nominal 95% level was reported for every candidate at the 20% budget, and interval-score gains tracked RMSE gains.Table 8 compares candidates against the oracle, weighted by traffic share over 100 replications, reporting RMSE, interval score, and empirical coverage.Source: Table 8 · p. 30, §C.3 · p. 30
Loading 2609.20758v1…