PaperlayerView exact revision on arXiv

Source-linked AI summary

Cluster-Robust Prediction-Powered Inference

David Broska, Michael Howes

arXiv:2610.09601v1stat.MEecon.EMstat.ML

TL;DR

PPI++ assumes independence across units and between labeled and unlabeled data, although clustered applications can violate these assumptions. Cluster-Robust PPI++ uses closed-form cluster-robust standard errors for inference under arbitrary within-cluster dependence, including partially labeled clusters. Cluster-Robust PPI++ achieved 93%–95% coverage in every setting but the fully labeled design with µp = 0.05 (88%), while PPI++ with i.i.d. standard errors contained θ⋆ in 25%–54% of replications.

  • Problem

    PPI++ assumes independence across units and between labeled and unlabeled data, although clustered applications can violate these assumptions.

    Source: §1 · p. 1

  • Method

    Cluster-Robust PPI++ uses closed-form cluster-robust standard errors for inference under arbitrary within-cluster dependence, including partially labeled clusters.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Cluster-Robust PPI++ achieved 93%–95% coverage in every setting but the fully labeled design with µp = 0.05 (88%), while PPI++ with i.i.d. standard errors contained θ⋆ in 25%–54% of replications.

    Source: §3.2 · p. 7

  • Takeaways & Limitations

    Precision gains depend on labeling design: in the fully labeled design, RMSE reductions were largest when labels were scarce, while gains in the partially labeled design were smaller.

    Source: §3.2 · p. 7

    The variance estimator has no small-sample correction, limiting its reliability when few clusters are labeled.

    Source: §4 · p. 8

Abstract

from arXiv · show

Data collection is often costly or logistically demanding, limiting both the questions researchers can pursue and how precisely they can answer them. Prediction-powered inference (PPI) can reduce the amount of data needed for precise parameter estimation by combining labeled data with machine learning predictions. However, ignoring dependence within clusters can produce confidence intervals that cover the true parameter less often than their nominal rate. We introduce Cluster-Robust PPI++, which provides standard errors in closed form and asymptotically valid confidence intervals under arbitrary dependence within independent clusters, requiring no bootstrap or resampling. Our central contribution is to accommodate partially labeled clusters, a common empirical setting in which clusters contain both labeled and unlabeled units. As units are dependent within clusters, partially labeled clusters violate the independence assumption of PPI++. We also show how precision increases depend on the labeling design, and derive a cluster-aware power tuning rule that minimizes asymptotic variance. In an application to television news, standard PPI++ confidence intervals have coverage below 60%, whereas Cluster-Robust PPI++ can achieve nominal 95% coverage.

1 Introduction

PPI combines labeled data with machine-learning predictions to improve the precision of parameter estimation, but standard PPI++ assumes independence that clustered data may violate. Cluster-Robust PPI++ extends inference to arbitrary within-cluster dependence, including partially labeled clusters, and its television-news application improves coverage while retaining precision gains over labels-only estimates.

1 Introduction

  • Clustered data violate PPI++'s independence assumptions because units can be correlated within clusters and labeled and unlabeled data may share clusters.Ignoring within-cluster dependence makes standard PPI++ intervals cover the truth less often than their nominal rate.Source: §1 · p. 1
  • Cluster-Robust PPI++ provides closed-form cluster-robust standard errors and asymptotically valid confidence intervals under arbitrary within-cluster dependence, without bootstrap or resampling.Its power-tuned estimator has asymptotic variance no greater than the corresponding labels-only estimator.Source: §1 · p. 1
  • Partially labeled clusters allow labels to be sampled at the unit level, unlike prior cluster-aware PPI approaches that require fully labeled or fully unlabeled clusters.Labeled and unlabeled units can coexist in the same cluster, violating PPI++'s independence assumption for the two datasets.Source: §1 · p. 1
  • Clustered labeling designs affect precision through label allocation across clusters and the degree of within-cluster dependence in outcomes and predictions.The authors derive formulas to compare designs and identify the most precise option.Source: §1 · p. 2
  • In the television-news application, Cluster-Robust PPI++ improves confidence-interval coverage over standard PPI++ while retaining precision gains over labels-only estimates.The application uses face recognition to identify politicians and evaluates fully labeled and partially labeled designs.Source: §1 · p. 2

2 Methods

The methods extend PPI++ to clustered observations, including clusters with both labeled and unlabeled units. They derive cluster-robust inference, adapt power tuning to cluster dependence, and describe how labeling designs affect precision.

2 Methods

  • Cluster-Robust PPI++ extends the mean-estimation PPI++ framework using i.i.d. clusters, with labels sampled at a common probability within each cluster.The methods discussion presents the mean as an illustrative case and notes that the framework extends to general M-estimators.Source: §2 · p. 2

2.1 The PPI++ estimator

  • PPI++ combines labeled outcomes and predictions, with its power parameter chosen to minimize asymptotic variance; setting λ = 0 yields the labeled-sample mean.The estimator's original derivations assume independent units and independent labeled and unlabeled datasets, assumptions that clustered data can violate.Source: §2.1 · p. 2

2.2 Data generating process for clustered data

  • Clusters are independent draws but may have arbitrary within-cluster dependence, and labeled and unlabeled units can occur in the same cluster.The labeling process uses cluster-level probabilities and conditionally independent Bernoulli label indicators within each cluster.Source: §2.2 · p. 3

2.3 Cluster robust standard errors

  • A plug-in asymptotic variance estimator supports confidence intervals for the PPI++ estimator under the clustered data-generating process.Theorem 1 establishes the central limit theorem as the number of clusters grows.Source: §2.3 · pp. 3–4
  • Cluster-robust variance is determined by cluster-level residual and prediction quantities, which support asymptotically valid confidence intervals.The residual sum measures cluster-level variability in labeled outcomes; the prediction quantity acts as a cluster-level control variate.Source: §2.3 · pp. 3–4

2.4 Power tuning

  • The cluster-aware power-tuned estimator is asymptotically at least as precise as the labels-only estimator under the theorem's assumptions.The tuning parameter is selected to minimize asymptotic variance using estimated cluster-level quantities.Source: §2.4 · p. 5

2.5 The PPI correlation

  • The PPI correlation depends on labeling proportions and on within-cluster correlations between outcomes and predictions.The correlation between cluster-level residuals and prediction quantities indicates when power tuning can substantially reduce variance.Source: §2.5 · p. 5

2.6 Design choices

  • Asymptotic variance depends on the number of clusters, the mean labeling probability, and the within-cluster correlation of label indicators.Design comparisons also require non-design moments describing label and prediction variability and correlation; the same number of labels can yield different precision depending on allocation.Source: §2.6 · p. 5

3 Empirical Application

The television-news application evaluates Cluster-Robust PPI++ under fully and partially labeled designs, comparing coverage and precision as labeling probability and allocation vary.

3.1 Simulation design

  • The artificial population comprises 67,062 annotated frames from 2,266 broadcasts, and the target is the prime minister’s share of sampled seconds with equal weight per frame.Broadcasts are clusters; outcome and prediction-error intraclass correlations are 0.71 and 0.50, respectively.Source: §3.1 · p. 6

3.2 Results on coverage and RMSE

  • Cluster-Robust PPI++ covers 93%–95% in all but one setting, while i.i.d.-SE PPI++ covers only 25%–54%; Figure 2 reports coverage and RMSE across both designs.With fully labeled broadcasts at µp = 0.05, Cluster-Robust PPI++ and labels-only coverage are 88% and 90%, respectively.Source: Figure 2 · p. 7, §3.2 · p. 7
  • In fully labeled designs, Cluster-Robust PPI++ cuts labels-only RMSE by about half when µp ≤ 0.15, but the gain falls to 10% at µp = 0.75.In partially labeled designs, the gain is 17% at µp = 0.05 and at most 5% for larger µp, because remaining error mostly reflects which broadcasts are sampled.Source: §3.2 · p. 7

3.3 Choosing a labeling design

  • At µp = 0.05, concentrating labels in whole broadcasts nearly doubles RMSE versus spreading labels across broadcasts; when at least half of frames are labeled, the increase is at most 7%.Figure 3 compares RMSE across mean labeling probability and within-broadcast labeling correlation.Source: §3.3 · pp. 7–8, Figure 3 · p. 8
  • Across 209 labeling designs, simulated RMSE stays within 5% of the variance formula, which reproduces the precision pattern when labels are concentrated in fewer broadcasts.The formula’s largest increase at labeling rates of at least one-half is 6%, compared with 7% in simulation; the test uses known moments, not pilot-estimated moments.Source: §3.3 · p. 8

4 Conclusion

The conclusion presents Cluster-Robust PPI++ as a way to obtain asymptotically valid inference under arbitrary within-cluster dependence, including partially labeled clusters. The television-news application illustrates coverage recovery and design-dependent precision, while several assumptions and extensions remain open.

4 Conclusion

  • Closed-form cluster-robust standard errors yield asymptotically valid confidence intervals for fully labeled, partially labeled, or fully unlabeled clusters.The method requires no bootstrap or resampling.Source: §4 · p. 8
  • In the television-news application, standard PPI++ intervals that ignore clustering under-cover, while Cluster-Robust PPI++ can restore nominal coverage.Source: §4 · p. 8
  • When labels are scarce, spreading them across many clusters can yield more precise estimates than labeling whole clusters, and the variance formulas closely predict design precision.The formulas can be used to compare labeling designs before collecting labels.Source: §4 · p. 8
  • The method has no small-sample correction, assumes one clustering level, requires data-independent label probabilities, and its variance formulas do not model labeling costs.These boundaries motivate future work on corrections, multiway clustering, active labeling, and cost-aware allocation.Source: §4 · p. 8

A Cluster standard errors for M-estimators

The appendix outlines a proof and variance-estimation route for PPI++ with clustered standard errors, then introduces the estimator and a squared-loss example.

A Cluster standard errors for M-estimators

  • The appendix proceeds from the general PPI++ estimator and data assumptions to a central limit theorem for valid cluster-robust standard errors, then discusses power tuning and PPI correlation.Source: A Cluster standard errors for… · p. 10

A.1 The PPI++ estimator

  • PPI++ is defined through a loss function ℓθ(x, y) that determines the target parameter θ⋆; the setup distinguishes labeled observations from unlabeled predictions.Source: §A.1 · p. 10
  • Under squared loss, θ⋆ is E[Y], and the PPI++ estimate is given by expression (1).Source: §A.1 · p. 10

A.2 Data generating process for clustered data

The data-generating process samples clusters independently while allowing dependence among units within a cluster. Labels are sampled at the unit level under a cluster-specific Bernoulli probability, alongside assumptions on the estimand and loss.

A.2 Data generating process for clustered data

  • Clusters are sampled i.i.d.; units may be dependent within a cluster, and cluster size may be correlated with unit features and outcomes.The estimand is assumed to be the unique minimizer of the population loss.Source: §A.2 · pp. 10–11
  • Each unit is labeled by a Bernoulli indicator with a probability shared within its cluster, and these probabilities are independent of unit features, outcomes, and cluster size.The cluster-level probabilities are themselves i.i.d., with expected value strictly between zero and one.Source: §A.2 · p. 11
  • The loss is required to satisfy convexity and regularity conditions, including twice-continuous differentiability near the target parameter.The passages also specify finite variance for sums of prediction-loss gradients.Source: §A.2 · p. 11

A.3 Cluster robust standard errors

The section derives cluster-robust asymptotic inference and considers power tuning for clustered data. It also describes a PPI-correlation quantity for mean estimation.

A.3 Cluster robust standard errors

  • Theorem 2 gives a central limit theorem for the general PPI++ estimator with a cluster-robust variance expression.The result underlies the mean-estimation result referenced in the section.Source: §A.3 · pp. 12–13

A.4 Power tuning

  • Power tuning can minimize either the trace of the asymptotic variance matrix or the variance of a selected coordinate.With a consistent tuning estimate, the power-tuned estimator supports asymptotically valid confidence intervals.Source: §A.4 · pp. 13–14
  • For GLMs, the trace- and coordinate-based tuning estimates are clipped to [0, 1] to preserve loss convexity.Source: §A.4 · p. 13

A.5 The PPI-correlation

  • The section provides more explicit formulas for ρ in the case of mean estimation.Source: §A.5 · p. 14

B.1 Proof of Theorem 2

The proof section establishes consistency and the central limit theorem for the estimator, then gives variance-design formulas and cautions about forecasting precision from small pilots.

B.1 Proof of Theorem 2

  • The proof establishes consistency of the estimator for the target parameter under the stated assumptions.It invokes the unique population-loss minimizer and convexity conditions.Source: §B.1 · p. 15
  • Theorem 2's proof completes the central limit theorem after establishing the relevant variance-estimator consistency.Source: §B.1 · p. 17

B.2 Proof of Theorem 3

  • Theorem 3's proof obtains the power-tuned central limit result by the same argument used for Theorem 2.Source: §B.2 · pp. 17–18

C Additional formulas for the PPI variance

  • Labeling design parameters and outcome- and prediction-related moments determine the estimator's variance and precision gain, enabling comparisons of labeling strategies when the moments are known.The mean labeling probability and within-cluster label correlation describe expected label volume and how labels are distributed across clusters.Source: C Additional formulas for the PPI… · pp. 18–19
  • Small pilots can yield unreliable variance forecasts; estimating within-cluster outcome covariance requires at least two labeled units in some clusters.Unreliability may appear as negative variance forecasts or estimated correlations outside [−1, 1].Source: C Additional formulas for the PPI… · p. 20

D.1 Data

The application pairs human annotations with face-recognition predictions for television-news frames. Its simulation details contrast fully labeled-cluster and partially labeled-cluster designs.

D.1 Data

  • The application combines annotations for 67,062 sampled frames with face-recognition output covering broadcasts since 2013.Annotators identified politicians in sampled frames; the system recorded recognized politicians in analyzed frames.Source: §D.1 · p. 21
  • Predictions are paired to annotations using the nearest analyzed frame within 14 frames, and 3% of annotated frames use a nearby rather than identical frame.The prediction indicates whether the system identified the sitting prime minister.Source: §D.1 · p. 21

D.2 Simulation details

  • The fully labeled design varies more in realized label share than the partially labeled design at the same expected share.At μp = 0.05, 95% of replications span 3.8%–6.3% for the fully labeled design versus 4.8%–5.2% for the partially labeled design; cluster labeling patterns also differ between designs.Source: §D.2 · p. 21

D.3 Full results

Table 1 reports coverage and RMSE for θ⋆ = 0.109 across labeling designs and mean labeling probabilities. It also includes a predictions-only estimator that uses no labels.

D.3 Full results

  • Both labels-only and predictions-only estimators use standard errors clustered by broadcast.Source: §D.3 · p. 21
  • Table 1 reports coverage of nominal 95% intervals and RMSE in percentage points across labeling designs and mean labeling probabilities µp.The target is θ⋆ = 0.109; each row is based on 2,500 replications.Source: Table 1 · p. 22
  • The predictions-only estimator uses no labels, so its values differ across rows only by simulation noise.The labels-only estimator averages observed outcomes over labeled frames; predictions-only averages predictions over all sampled frames.Source: §D.3 · p. 21, Table 1 · p. 22

E.1 Simulation design

The authors evaluate Cluster-Robust PPI++ in a regression of whether the sitting prime minister is on screen, using era and news program as predictors. Human-label coefficients are the simulation targets, and predictions-only and PPI++ estimators provide comparisons.

E.1 Simulation design

  • The regression predicts whether the sitting prime minister is on screen from era and news program, using broadcasts from NHK News 7 and TV Asahi’s Hodo Station.Era distinguishes the Suga and Kishida period from the Abe period; NHK indicates NHK News 7.Source: §E.1 · p. 23
  • The human-label coefficients are simulation targets, not tests of a substantive hypothesis; the era contrast does not isolate a new prime minister’s effect.The Suga and Kishida era coincides with the COVID-19 pandemic.Source: §E.1 · p. 23, Table 2 · p. 23
  • 13.4 percentage points is the human-label era contrast, while predictions estimate 11.1 points, 17% lower; the predicted NHK coefficient is 2.18 rather than 2.23 points.The predictions understate screen time by about the same share in both eras and by about the same number of percentage points in both programs.Source: §E.1 · p. 23
  • The evaluation reuses the earlier designs, µp values, and random draws, fitting labels-only least squares and prediction-based estimators with broadcast-clustered standard errors.PPI++ variants use the general estimator with squared-error loss and choose one λ to minimize the sum of estimated variances for the three coefficients.Source: §E.1 · p. 23

E.2 Results on coverage and RMSE

In the regression simulations, Cluster-Robust PPI++ generally achieves coverage near the nominal 95%, while PPI++ with i.i.d. standard errors substantially undercovers. Cluster-Robust PPI++ also reduces labels-only RMSE most when labels are scarce, while predictions-only performance differs by coefficient.

E.2 Results on coverage and RMSE

  • 81% era coverage occurs for both Cluster-Robust PPI++ and labels-only at µp = 0.05 in the fully labeled design, where only about 15 Suga and Kishida broadcasts are labeled.This is the single setting excepted from the reported 91%–95% era-coverage range.Source: §E.2 · p. 23
  • About half the labels-only RMSE is removed for both coefficients when µp is at most 0.15 in the fully labeled design, with the reduction shrinking to about 10% at µp = 0.75.Source: §E.2 · p. 23
  • Predictions-only intervals cover the era target in about 82% of replications but the NHK target in about 95%; their RMSE is lower than Cluster-Robust PPI++ for era at µp ≤ 0.15 and lower than every label-using estimator for NHK.Predictions alone understate the era coefficient by about 2.3 percentage points, while the NHK coefficient is nearly unbiased.Source: §E.2 · p. 24
  • 94%–96% NHK coverage across every setting and 91%–95% era coverage in every setting but one are reported for Cluster-Robust PPI++; i.i.d.-SE PPI++ coverage is 21%–51% for era and 26%–54% for NHK.The comparisons concern nominal 95% intervals; Figure 4 and Table 3 report era results, while Figure 5 and Table 4 report NHK results.Source: §E.2 · p. 23, Figure 4 · p. 24, Figure 5 · p. 24
Loading 2610.09601v1…