PaperlayerView exact revision on arXiv

Source-linked AI summary

Prediction-Powered Inference

Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, Tijana Zrnic

arXiv:2301.09633v4stat.MLcs.AIcs.LGq-bio.QMstat.ME

TL;DR

Machine-learning predictions are plentiful but may be biased, whereas gold-standard data are valid but scarce. The paper introduces prediction-powered inference, which corrects prediction-based estimates using labeled data to produce valid intervals and p-values. It applies broadly across estimands and shows that more accurate predictions yield smaller intervals.

  • Problem

    The problem is how to use abundant, inexpensive machine-learning predictions for inference without the invalid conclusions that prediction bias can cause.

  • Method

    Prediction-powered inference uses gold-standard data to quantify prediction error, rectify an imputed estimate, and construct valid confidence sets.

  • Results

    The framework provides provably valid intervals and p-values for means, quantiles, modes, and linear and logistic regression coefficients, with benefits demonstrated across multiple scientific datasets.

  • Takeaways & Limitations

    More accurate predictions translate into smaller confidence intervals while retaining statistical validity.

  • Takeaways & Limitations

    The main theoretical treatment focuses on convex estimation and initially assumes independent identically distributed labeled and unlabeled samples from a common distribution.

Abstract

from arXiv · show

Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system. The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients, without making any assumptions on the machine-learning algorithm that supplies the predictions. Furthermore, more accurate predictions translate to smaller confidence intervals. Prediction-powered inference could enable researchers to draw valid and more data-efficient conclusions using machine learning. The benefits of prediction-powered inference are demonstrated with datasets from proteomics, astronomy, genomics, remote sensing, census analysis, and ecology.

1 Introduction

Machine-learning predictions are abundant and inexpensive but can be biased, while gold-standard measurements are scarce. Prediction-powered inference combines both to produce statistically valid inference that leverages predictions.

  • Motivation: Predictions can be generated cheaply and at high throughput, but their imperfections may produce incorrect conclusions and compound across analyses.The motivation includes applications such as protein-structure prediction and scientific modeling.
  • Motivation: Treating predictions as gold-standard measurements is fast and inexpensive, but prediction bias can invalidate conclusions.
  • Motivation: Ignoring predictions preserves statistical validity, but scarce gold-standard data limits the scope of possible discoveries.
  • Framework: Prediction-powered inference combines abundant predictions with scarce trusted data to compute statistically valid confidence intervals and p-values.
  • Framework: The framework is designed to work with any machine-learning system, supporting standardized assessment of evidence without case-by-case algorithm analysis.
  • Framework: The method uses labeled and unlabeled datasets with predictions, then quantifies prediction error on gold-standard data to adjust an imputed estimate.
  • Framework: A problem-specific rectifier and its confidence set are used to construct a prediction-powered confidence set guaranteed to contain the target with high probability.

2. Rectifier confidence set

The rectifier confidence set corrects prediction-based estimates using labeled data and yields valid inference across several estimands. Its precision improves when predictions are accurate, while the main theory focuses on convex estimation.

  • Rectifier confidence set: The rectifier is a problem-specific measure of prediction error used to quantify and correct bias in an imputed estimate.
  • Rectifier confidence set: A confidence set for the rectifier is constructed from labeled data, and each possible rectification produces the prediction-powered confidence set.
  • Scope: Prediction-powered inference provides algorithms for confidence intervals covering means, quantiles, modes, and linear and logistic regression coefficients.
  • Assumptions: The framework assumes labeled and unlabeled data are independent identically distributed samples from a common distribution, with extensions to distribution shift and finite populations.
  • Assumptions: The prediction rule is independent of the observed labeled and unlabeled data, such as when trained on separate data.
  • Mean estimation: When predictions are accurate, prediction-powered estimation can use the large unlabeled sample to reduce variance relative to the classical labeled-data estimate.
  • Mean estimation: Variance reduction makes prediction-powered confidence intervals narrower than classical intervals in a broad range of settings beyond mean estimation.
  • Related work: The technical results extend model-assisted survey-sampling ideas and relate to semiparametric inference, missing data, measurement error, and semisupervised learning.

2 Main theory: Convex estimation

Prediction-powered inference constructs valid confidence sets for convex estimation problems by correcting prediction-based estimates with prediction-error information from gold-standard data. The framework covers means, quantiles, and regression coefficients, with validity established under stated regularity conditions and without assumptions on the machine-learning model.

  • The framework targets estimands expressible as solutions to convex optimization problems, including means, quantiles, and linear and logistic regression coefficients.If minimizers are not unique, the resulting confidence set is guaranteed to contain all minimizers.
  • The rectifier measures prediction-induced bias in the imputed estimate and is estimated from gold-standard data before correcting the prediction-based confidence set.The procedure forms a confidence set for the rectifier and combines it with a set accounting for fluctuations in the unlabeled sample.
  • Theorem 1 gives valid confidence sets for nondegenerate convex estimation problems without assumptions on the data distribution or machine-learning model.The construction combines confidence sets through a Minkowski sum and has an asymptotic counterpart.
  • The theory focuses on convex estimation, while the general prediction-powered principles also extend to additional estimands and some forms of distribution shift.The convex focus reflects the broad class of estimands addressed by the framework.
  • Mean, quantile, linear-regression, and logistic-regression procedures each have prediction-powered confidence sets with valid asymptotic coverage.The mean and quantile constructions use problem-specific rectifiers; specialized regression procedures exploit their respective loss or estimator structure.
  • Quantile estimation assumes an outcome distribution without point masses to avoid a degenerate problem, although standard constructions can extend beyond this setting.The quantile rectifier is the difference between the outcome and prediction cumulative distribution functions at the candidate quantile.

3 Applications

Across proteomics, astronomy, genomics, remote sensing, census, and ecology applications, prediction-powered intervals retained statistical validity while typically being narrower than classical intervals. The experiments also show that imputation intervals can be invalid when prediction errors are ignored.

  • Application design: Across real tasks, prediction-powered intervals were compared with classical and imputation intervals; imputation failed to contain the true estimand because it ignored prediction errors.The two valid approaches were compared by interval width, while imputation was shown to be too optimistic.
  • Application scope: The applications estimate associations, population fractions, regression coefficients, and plankton growth using scarce labels alongside machine-learning predictions.Examples include PTM–IDR odds ratios, spiral-arm galaxy fractions, census regression coefficients, and phytoplankton growth.
  • Proteomics: 316 labeled observations versus 799 were required to reject an odds-ratio null in the proteomics application using prediction-powered versus classical inference.Prediction-powered intervals were smaller for all tested labeled-sample sizes, while imputation overestimated the odds ratio.
  • Gene expression quantiles: Prediction-powered intervals for all three yeast promoter-expression quantiles were much smaller than classical intervals across all labeled-sample sizes.For testing whether the median expression was at most five, prediction-powered inference required 764 examples versus 900 classically.
  • Amazon deforestation: 21 gold-standard labels versus 35 were required to reject no deforestation in the Amazon using prediction-powered versus classical inference.Prediction-powered intervals were smaller than classical intervals and retained validity, whereas imputation failed to cover the true deforestation fraction.

4 Extensions

The framework extends beyond its initial convex, identically distributed setting to general risk minimization, distribution shift, point estimation, and valid p-values. These extensions preserve inference by adapting rectification, reweighting, or optimization procedures to the target problem.

  • Scope extensions: Prediction-powered inference extends beyond i.i.d. observations and convex losses to nonconvex optimization problems and distribution shift.The paper treats arbitrary optimization problems and certain label- and covariate-shift settings.
  • Beyond convex estimation: For general risk minimization, the method can handle nonconvex losses and arbitrary parameter constraints, including mode estimation and best subset selection.If the minimizer is not unique, the method returns a set containing all minimizers.
  • Beyond convex estimation: The general optimization solution does not reduce to the convex method and is expected to be less powerful when convex problems have low-dimensional rectifiers.The convex approach remains preferable in that stated regime.
  • Distribution shift: Under known covariate shift, reweighting by the Radon–Nikodym derivative converts risk minimization on the target distribution into reweighted risk minimization on the source distribution.This permits rectifier inference using data sampled from the source distribution.
  • Distribution shift: Label-shift intervals become more conservative as the number of classes increases, and their power depends on the smallest observed class count.The label-shift treatment focuses on confidence intervals for certain linear problems.
  • Point estimation: Prediction-powered inference also yields point estimates by minimizing a rectified loss whose expected value equals the true population loss.The resulting point estimate is contained in the constructed confidence intervals.
  • P-values: Confidence-interval duality immediately extends the framework to valid prediction-powered p-values for means, quantiles, logistic regression, and linear regression.The paper gives separate validity results for each of these inferential targets.

B.1 Convex estimation

The finite-population framework applies convex estimation guarantees to several inferential targets. Its construction uses prediction errors quantified from gold-standard data rather than treating predictions as error-free outcomes.

  • General guarantee: Theorem B.1 provides a finite-population confidence-set guarantee for nondegenerate convex estimation.It fixes α ∈ (0, 1) and requires constructing Rα(θ) for each candidate θ.
  • Applications: Prediction-powered finite-population inference applies to means, quantiles, logistic regression, and linear regression.The target is defined through an appropriately chosen loss function.
  • Construction: Prediction errors are quantified using the gold-standard dataset to correct the generally biased imputed estimate.The imputation approach treats predictions as gold-standard outcomes, whereas the proposed construction accounts for their errors.
  • Construction: Finite-population inference does not require data splitting because the imputed estimate is deterministic.The finite-population rectifier is constructed without the split used in the i.i.d. setting.
  • Asymptotic counterpart: The appendix develops an asymptotic counterpart using the central limit theorem alongside nonasymptotic counterparts of the main algorithms.The asymptotic result is stated without error-budget splitting.

C.2 Algorithms with nonasymptotic validity

The nonasymptotic algorithms instantiate a general prediction-powered confidence-set recipe for means, quantiles, and logistic regression. Their validity relies on confidence bounds and explicit assumptions such as bounded observations or moments.

  • Algorithms: Nonasymptotically valid algorithms are provided for prediction-powered mean, quantile, and logistic regression inference.Each algorithm outputs a prediction-powered confidence set CPP.
  • Mean estimation: The mean algorithm uses a variance-adaptive confidence interval requiring observations bounded within a known interval.The default construction normalizes observations to [0, 1] when needed.
  • Mean estimation: The mean confidence set has valid coverage when outcomes and predictions lie almost surely in [0, B].This is the condition stated for Corollary C.1.
  • Quantile estimation: The quantile algorithm searches a grid spanning the minimum and maximum unlabeled predictions and forms a confidence set with valid coverage.The quantile target is the q-quantile.
  • Logistic regression: The logistic-regression confidence set has valid coverage when covariates are coordinatewise bounded and outcomes and predictions lie in [0, 1].The coordinatewise bound is used to bound the relevant estimating quantities.
  • Linear regression: An analogous nonasymptotic linear-regression algorithm is not recommended because direct use of the general theorem yields more conservative intervals.The refined asymptotic analysis instead reduces the problem to a one-dimensional rectifier.

D.6 Proof of Proposition 4

The proof of Proposition 4 exploits least-squares linearity to obtain stronger prediction-powered confidence intervals than the general convex-estimation theorem. It combines asymptotic regression results with the prediction-error correction.

  • Linear-regression refinement: Linearity of the least-squares estimator yields more powerful prediction-powered confidence intervals for linear regression.This avoids directly applying the more conservative general rectifier analysis.
  • Linear-regression refinement: The general nonasymptotic approach is more conservative for linear regression because it analyzes a d-dimensional rectifier rather than the refined one-dimensional version.The section notes that the refined analysis is asymptotic.
  • Estimand decomposition: The target linear-regression estimand is decomposed into the prediction-based parameter and a prediction-error correction.The correction is expressed through E[Xi(f(Xi)−Yi)].
  • Asymptotic validity: The estimator is asymptotically normal and consistent, with a consistently estimated covariance matrix.The proof obtains these properties using White’s regression results and Slutsky’s theorem.
  • Asymptotic validity: The resulting confidence intervals are constructed through a normal approximation.The confidence-bound guarantee is then combined using a union bound.

E.1 Inference with i.i.d. samples

This section reviews confidence intervals used with i.i.d. and finite-population samples, including nonasymptotic bounded-data methods and asymptotic CLT-based methods. It also situates prediction-powered inference among procedures combining labeled and unlabeled data.

  • Nonasymptotic intervals: The supermartingale is interpreted as evidence against a candidate m being the true mean, and the confidence set retains candidates with limited evidence against them.Large values of Mt(m) indicate that m is unlikely to be the true mean.
  • Asymptotic intervals: Asymptotic mean intervals rely on the central limit theorem and require finite second moments in the i.i.d. setting.The finite-population counterpart assumes a limiting sampling fraction n/N → p with p ∈ (0, 1).
  • Related procedures: Prediction-powered inference is compared with post-prediction inference and semi-supervised mean estimation as procedures combining labeled and unlabeled data.The post-prediction procedure described here has no theoretical guarantees in general.

F.1 Experimental protocol

The experiments compare prediction-powered inference with classical, imputation, post-prediction, and semi-supervised procedures across labeled-data regimes. Prediction-powered intervals dominate semi-supervised and classical intervals in the mean-estimation experiment, while validity can fail for post-prediction inference.

  • Experimental setup: The income-prediction experiments compare semi-supervised and post-prediction baselines against classical and prediction-powered approaches.The semi-supervised task estimates mean California income; the post-prediction task estimates the OLS coefficient between age and income.
  • Post-prediction inference: The post-prediction procedure does not cover at the proper rate, producing biased intervals.This comparison concerns the OLS coefficient between age and income.
  • Experimental setup: The post-prediction comparison reports interval widths as a function of n, averaging over 100 independent trials, with five interval draws shown at each n.Prediction-powered, classical, and post-prediction approaches are shown in green, gray, and red, respectively.
  • Semi-supervised mean estimation: Prediction-powered intervals dominate both semi-supervised and classical intervals for all values of n in the mean-income experiment.The semi-supervised intervals have similar width to classical intervals, whereas prediction-powered intervals are narrower in the experiment.
  • Conditions for improvement: The framework’s advantage depends on model quality and unlabeled-sample size; classical inference can outperform it when the model is insufficiently accurate or N is too small.These conditions are demonstrated in additional experiments.

G.1 Mathematical derivation

Prediction-powered intervals become tighter when predictions reduce residual outcome variance and when the unlabeled dataset is sufficiently larger than the labeled dataset. Experiments show that poor predictive performance or N smaller than n can instead favor classical inference.

  • Sample-size regime: Gains are expected when N ≫ n, whereas N ≈ n generally provides little benefit because prediction variance is typically comparable to outcome variance.The comparison follows from the variance of the prediction-powered estimator.
  • Variance comparison: Prediction-powered inference helps when Var(f(X_i) − Y_i) < Var(Y_i), meaning predictions explain away some outcome variance.This variance condition determines when prediction-powered confidence intervals are tighter than classical intervals.
  • Binary-outcome illustration: For binary outcomes with p = 0.5, model error η must be below 25% for prediction-powered inference to yield smaller intervals.When p = 0.1, the corresponding threshold is about 9.5%.
  • Binary-outcome illustration: Lower outcome variance requires lower model error for prediction-powered inference to be helpful.The binary-outcome calculation illustrates this dependence.
  • Practical condition: The main takeaway is to apply prediction-powered inference when N is preferably substantially larger than n and predictive accuracy explains enough outcome variance.The stated intuition extends beyond mean estimation to other estimation problems.
  • Inaccurate model: With a linear regression predictor in the deforestation analysis, classical inference requires n = 35 data points to reject, versus n = 40 for prediction-powered inference.The weaker predictive performance reduces prediction-powered power.
  • Small unlabeled dataset: With N = 1000 unlabeled AlphaFold data points, classical inference outperforms prediction-powered inference when n > N.For the same null hypothesis, rejection requires n = 652 classically versus n = 869 with prediction-powered inference.
  • Additional targets: The supplementary experiments extend comparisons to additional PTM odds ratios and gene-expression quantiles using prediction-powered and classical confidence intervals.Figure 6 examines ubiquitination and acetylation; Figure 7 examines q = 0.25 and q = 0.75.
Loading 2301.09633v4…