PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Evaluation of Language Models with Cheap Auxiliary Signals

Zhi Zhang, Lingfeng Lyu, Yue Kang, Doudou Zhou

arXiv:2608.16210v1cs.LGstat.ML

TL;DR

Conditional evaluation needs profile-level performance estimates, but gold labels are scarce and cheap signals may be biased or miscalibrated. LACE centers those signals locally to estimate conditional profiles efficiently, achieving cell-level relative efficiencies of 2.73× to 11.03× across eight benchmarks.

  • Problem

    Conditional performance profiles matter for model selection and diagnosis, but gold labels are expensive and cheap evaluators may be biased or miscalibrated.

  • Method

    LACE locally centers cheap signals within profile regions, then uses a regularized control-variate augmentation whose coefficient improves efficiency without changing the conditional estimand.

  • Results

    Cell-level relative efficiency ranges from 2.73× to 11.03× across eight benchmarks, three models, and three label budgets.

  • Takeaways & Limitations

    Cheap auxiliary signals need not be calibrated to improve data-efficient conditional performance profiles when they explain local variation in gold correctness.

  • Takeaways & Limitations

    LACE may provide little benefit when signals lack local explanatory power, and covariance estimation can be unstable with small groups and many signals.

Abstract

from arXiv · show

Aggregate accuracy hides where models succeed and fail. Estimating conditional performance profiles from gold labels alone is expensive, while cheap auxiliary signals such as LLM-judge scores, pairwise comparisons, confidence scores, and judge-disagreement features can be collected for every benchmark item but are often biased or miscalibrated. We propose LACE (Local Augmented Control-Variate Evaluation), a semi-supervised estimator for conditional LLM evaluation. The key step is local centering: after subtracting the conditional mean of a cheap signal within the target profile region, any linear augmentation has zero conditional mean and therefore cannot change the estimand. The augmentation coefficient is used only for efficiency, and a local ridge control variate combines a gold-label residual mean from the labeled subset with a cheap-signal mean from the full item pool. We prove calibration-free identification, unbiasedness for grouped profiles, local oracle optimality within centered linear augmentations, and first-order adaptivity to the estimated coefficient. The resulting gain formula is governed by a population local $R^2$, which characterizes how the efficiency attainable from the cheap signals varies across profile values. We also derive corresponding estimators for direct paired model gaps and deployment-weighted scores. We empirically evaluate the primary performance-profile estimator on MATH-500, ScienceQA, MMLU, WinoGrande, HellaSwag, TruthfulQA, GSM8K, and ARC.

1 Introduction

The paper frames conditional performance estimation as a problem requiring gold labels alongside cheap but potentially miscalibrated auxiliary signals. It introduces LACE, which uses local centering and control-variate augmentation to estimate conditional profiles efficiently, with theoretical guarantees, extensions, and empirical gains.

  • Motivation: Conditional evaluation targets model performance across profile values such as difficulty or subject, rather than aggregate accuracy alone.Profiles may be indexed by a single metadata axis or a multidimensional combination of attributes.
  • Motivation: Cheap signals—including judge scores, pairwise comparisons, confidence, and judge disagreement—can be collected broadly but may be biased, overconfident, or prompt-sensitive.The paper cautions against treating these signals as calibrated probabilities.
  • Method: LACE estimates conditional means from sparse gold outcomes and full-pool auxiliary signals using local centering and augmented control variates.The MATH-500 example targets Pr(Y_im = 1 | Z_i = z), the candidate model’s accuracy at difficulty z.
  • Theory: The method provides calibration-free identification, group-profile unbiasedness, centered-linear oracle optimality, first-order adaptivity, and a local gain formula governed by R^2_m(z).The gain formula characterizes how heterogeneous signal quality affects attainable efficiency across profile values.
  • Empirical evaluation: 2.73× to 11.03× cell-level RE was achieved across eight benchmarks, three models, and three label budgets.The paper also derives extensions for direct paired model gaps and deployment-weighted scores.

2 Related Work

Prior work establishes conditional LLM evaluation, scalable cheap evaluators, and semi-supervised variance reduction as key foundations. LACE combines these ideas for conditional targets by using profiling variables to define estimands and auxiliary signals solely for precision, with local, profile-dependent gains.

  • Conditional LLM evaluation: Conditional LLM evaluation reports performance across subjects, levels, grades, domains, tasks, scenarios, and metrics.Examples include MMLU, MATH, ScienceQA, GPQA, MMLU-Pro, BIG-bench, HELM, and the Language Model Evaluation Harness.
  • Cheap evaluators, LLM judges, and aggregation: LLM judges and pairwise preference evaluators provide scalable assessments that often correlate with human or gold-label judgments.Related systems estimate factuality or response quality more cheaply than manual evaluation, while multi-agent judging targets improved reliability.
  • Control variates, semi-supervised estimation, and local PPI: Control variates, AIPW, one-step estimators, semi-supervised regression, and prediction-powered inference combine scarce labels with abundant auxiliary information.These methods provide the methodological background for using fitted means and labeled residual corrections to reduce estimation variance.
  • Estimator structure in common notation: LACE’s contribution is a joint construction for conditional targets with multiple auxiliary signals, profile-dependent relevance, and a local vector coefficient learned from scarce labels.The centered control-variate algebra is classical, while the proposed construction adds first-order adaptivity to the estimated coefficient.
  • Estimator structure in common notation: The profiling variable Z defines E(Y | Z = z), while the auxiliary vector S is used only for precision rather than included in the conditioning event.This separates the target profile from item-level signals and avoids changing the estimand by conditioning on (Z, S).
  • Local smoothing, survey estimation, and deployment weighting: The framework supports continuous, ordered discrete, and unordered categorical profiles and yields a local gain formula involving R^2_m(z).Its continuous estimator uses kernel weights, the discrete version is a group estimator, and the ordinal version borrows strength across neighboring levels.

3 LACE: Local Augmented Control-Variate Evaluation

LACE estimates conditional performance from gold labels observed on a labeled subset and cheap signals observed for every item, using local centering so augmentation affects efficiency but not identification. A regularized local control variate combines the labeled residual mean with the full-pool signal mean, and the same centered template extends to paired model gaps and deployment-weighted scores.

  • Setup: LACE targets θ_m(z)=P(Y_im=1|Z_i=z), observing (Z_i,S_im) for every item but gold correctness Y_im only on a uniformly sampled labeled subset.The profiling covariate may be continuous, ordered discrete, or unordered categorical, and cheap signals are vectors.
  • Local weighting: Local weights smooth or group observations around the target profile value, with kernel, ordinal, and categorical rules defining the local means.Within-group weighting is the special case of the ordered-discrete rule with span parameter b_ord=1.
  • Centered augmentation: Centering cheap signals by their conditional mean preserves the target even when signals are biased, prompt-sensitive, or miscalibrated; the coefficient controls efficiency rather than identification.The local coefficient is chosen to remove the largest amount of local linear variation, using conditional signal variance and signal–outcome covariance.
  • Estimator: The LACE estimator combines a labeled residual mean with a full-pool cheap-signal mean and fits the augmentation coefficient using a ridge-regularized local covariance inversion.The ridge penalty stabilizes estimation when local covariance is ill-conditioned or labeled items are few relative to the signal dimension; uninformative signals shrink the coefficient toward zero.
  • Extensions: For paired models, LACE directly estimates Δ_ab(z)=θ_a(z)−θ_b(z) from paired outcomes and signal differences, exploiting within-item dependence that can improve efficiency over subtracting separate profiles.A direct pairwise judge feature can be appended when available.
  • Extensions: The centered augmentation template also supports deployment-weighted scores under covariate shift, using self-normalized deployment means for centering and final scoring.The deployment law reweights the profile distribution while preserving the conditional outcome-and-signal law given the covariate.

4 Theory

The theory establishes calibration-free identification and oracle optimality for locally centered augmentations, with exact unbiasedness for grouped profiles and first-order adaptivity for estimated coefficients. LACE’s local efficiency gain is governed by the conditional auxiliary-signal R^2 and extends to model gaps and deployment-weighted scores.

  • Identification and optimal augmentation: Local centering identifies θ_m(z) without calibration, and the variance-minimizing coefficient solves Σ_SS,m(z)b = Σ_SY,m(z), with minimum-norm solution β⋆_m(z) = Σ_SS,m(z)†Σ_SY,m(z).Any deterministic centered linear augmentation has conditional mean zero; the coefficient affects efficiency rather than the estimand.
  • Grouped profiles: For discrete groups, the group-weight estimator is exactly unbiased under random benchmark sampling and uniform label subsampling, with variance characterized for any fixed augmentation coefficient.Expectation and variance are taken over both benchmark sampling and label sampling.
  • Grouped profiles: The population oracle gain for a group profile is 1 / [1 − (1 − π_g)R^2_m(g)], while the realized-pool comparison is exact and cancels the common finite-population correction.The realized-pool identity is not a pointwise gain theorem for continuous profiles and does not assert equality for the feasible estimator.
  • Continuous profiles: Under local regularity and stable nuisance estimation, LACE is first-order adaptive to the estimated coefficient and achieves local oracle optimality among fixed deterministic linear augmentations.When π_0 < 1, the first-order variance minimizers solve Σ_SS,m(z)b = Σ_SY,m(z).
  • Continuous profiles: The continuous-profile efficiency gain is Gain_m(z) = 1 / [1 − (1 − π_0)R^2_m(z)]; it equals the gold-label smoother when R^2_m(z) = 0 and approaches 1 / π_0 as R^2_m(z) approaches 1.The gain is local, so efficiency varies across profile values according to local explanatory power.
  • Extensions: The same theoretical framework extends to direct paired model gaps and deployment-weighted scores under the stated regularity, covariate-shift, and weighting conditions.For gaps, the outcome and auxiliary signal are replaced by pairwise differences and corresponding features; deployment scores use a known bounded density ratio.

5 Experiments

Experiments evaluate LACE across eight benchmarks, three candidate models, and matched labeled-set budgets using multiple cheap signals and fixed shared hyperparameters. Across 72 benchmark–candidate cells, LACE substantially improves profile-estimation efficiency over gold-label-only evaluation and the strongest baseline.

  • Experimental setup: LACE is evaluated on eight benchmarks across three candidate models, with Claude Opus 4.6 providing judge-derived measurements and three pairwise anchors.The benchmarks span mathematical reasoning, commonsense, science, and general knowledge; experiments use matched random permutations with n_lab ∈ {50, 100, 200}.
  • Experimental setup: The protocol uses shared fixed settings across all benchmarks, candidates, budgets, and splits, including λ = 0.3, profile bandwidth h = h0, coefficient bandwidth b = h, and logistic regularization C = 1.The main results use no per-cell tuning or data-dependent hyperparameter fallback.
  • Scope: The empirical comparisons focus on the primary performance-profile estimand; paired-gap and deployment-weighted constructions are presented only as theoretical extensions, not separate empirical experiments.Controlled simulations separately evaluate the superpopulation gain formula and validate it pointwise in Figure 4.
  • Metrics: The primary target is the full-pool gold profile evaluated on a fixed 20-point grid spanning the 5th–95th percentiles of Z, with only labeled subsets resampled across repetitions.Profile MSE is the unweighted grid-average MSE, and relative efficiency compares paired mean MSEs against the gold-label-only estimator.
  • Results: 5.528 [5.181, 5.898] is LACE’s overall geometric-mean RE across 72 cells, versus 2.647 [2.277, 3.078] for the strongest baseline, multivariate plug-in.LACE’s budget-specific RE is 5.400, 5.761, and 5.431 for n_lab = 50, 100, 200, respectively, and it is best-performing in all 72 cells.

6 Conclusion … B.1 Proof of Proposition 1

The paper concludes that LACE uses locally centered cheap signals to improve conditional evaluation without requiring calibration, while its theory and extensions clarify when efficiency gains are possible. The appendices establish the linear-algebraic and identification foundations underlying local optimality.

  • 6 Conclusion: LACE requires cheap signals to explain local variation in gold correctness, not to be calibrated, and provides an oracle-optimal local augmentation with an interpretable gain formula.The conclusion also extends the same statistical principle to direct paired model gaps and deployment-weighted scoring.
  • 6 Conclusion: The framework extends beyond LLM evaluation whenever scarce gold outcomes, abundant proxy signals, and metadata covariates define the conditional structure of interest.Examples include clinical prediction with biomarkers and online experimentation with short-term engagement proxies.
  • 6 Conclusion: The local R2_m(z) measures potential signal usefulness for a chosen profile, but it does not select the profile; embedding projections and LSH buckets define alternative estimands.The existing weights apply unchanged to these prespecified low-dimensional or categorical profiles, while direct smoothing in the original high-dimensional embedding space remains outside the theory.
  • 6 Conclusion: A cascaded-rater design can apply cheap signals to the full pool, a more expensive rater to a random subset, and human labels to a nested random subset.The appendix presents the algebraic extension but does not claim complete optimality or adaptivity theory for the cascade.
  • 6 Conclusion: LACE improves efficiency only when cheap signals have local explanatory power; small local R2 and unstable covariance estimates can limit gains, especially with few labels and many signals.Ridge regularization and bandwidth enlargement mitigate, but do not remove, finite-sample instability.
  • A Technical Lemmas: For positive-semidefinite covariance matrices, the Moore–Penrose solution lies in the covariance range and gives the minimum-Euclidean-norm minimizer of the convex quadratic objective.The same solution set minimizes residual variance, as established by the technical lemma.
  • B.1 Proof of Proposition 1: The Proposition 1 proof shows that subtracting the centered cheap-signal term preserves the conditional mean of Y_im, while conditional application of the covariance lemma yields the variance-minimizing augmentation.The proof uses the fact that the centered shift is non-random conditional on Z_i = z and verifies the required moment conditions.

B.2 Proof of Theorem 1 · B.3 Proof of Theorem 2

The proofs establish unbiasedness through simple-random-sampling residual means and characterize oracle variance gains through local finite-population relationships. They also show that estimating the augmentation coefficient affects the estimator only through a second-order term.

  • B.2 Proof of Theorem 1: Conditional on the target group, the labeled residual mean is a without-replacement simple random-sample mean from the finite residual population.This conditional sampling identity underpins the expectation calculation for the labeled residual component.
  • B.2 Proof of Theorem 1: Taking expectation over the i.i.d. benchmark draw yields the target conditional estimand, establishing unbiasedness.The proof identifies E{bθm(g; bg)} with E(ȲT,g) and then with θm(g).
  • B.2 Proof of Theorem 1: The variance proof introduces finite-pool variance quantities to analyze the estimator’s conditional variability.The supplied proof passage begins the variance calculation by defining S2.
  • B.2 Proof of Theorem 1: The finite-pool variance identity yields the realized-pool oracle ratio used in the metrics analysis.This connects the proof’s variance identity to the realized-pool efficiency comparison.
  • B.2 Proof of Theorem 1: When the relevant finite-pool variance is positive, the common sampling factor cancels between naive and oracle conditional variances.The cancellation produces a ratio expressed through the local R^2 term.
  • B.2 Proof of Theorem 1: Substituting the local R^2 relationship into Theorem 1’s variance formula gives the stated oracle ratio and variance.The proof uses πg = ng/Mg in this substitution and concludes with the stated variance.
  • B.3 Proof of Theorem 2: Because the estimator is affine in the augmentation coefficient, coefficient perturbations can be analyzed directly.This affine structure is the starting point for the rate argument in Theorem 2.
  • B.3 Proof of Theorem 2: Cauchy–Schwarz implies that estimating the coefficient contributes only a second-order term under the stated conditions.Thus, replacing the oracle coefficient with its estimate preserves the first-order behavior described by the theorem.

C Proof of Theorem 3

The proof establishes an exact decomposition of the locally augmented estimator and derives its first-order variance through kernel-based covariance calculations. It then shows that the variance is minimized by the conditional linear projection coefficient when π0 < 1, while augmentation is first-order negligible when π0 = 1.

  • Exact decomposition: The local residual and cheap-signal means satisfy the exact identity μR,h(z) + μE,h(z) − θ(z) = Bh(z), with no remainder.This identity follows from Ri + Ei = Yi and E(Yi | Zi) = θ(Zi).
  • Asymptotic distribution: Exchangeability and independent labeled and target blocks enable a joint central limit argument for the centered local means.The proof applies the Cramér–Wold device and the Lyapunov central limit theorem after representing the sums over independent blocks.
  • Asymptotic covariance: The cross covariance of the two local mean estimators is determined by the overlapping labeled items and the conditional covariance Cov(Ri, Ei | Zi = z).Expanding over L × T eliminates contributions from distinct items and leaves the overlap terms.
  • Variance optimality: When π0 < 1, the asymptotic variance is a convex quadratic in the augmentation coefficient and is minimized by any solution of ΣSSb = ΣSY.The minimum-norm minimizer is obtained using the Moore–Penrose pseudoinverse.
  • Full-label limit: When π0 = 1, the finite-population factor makes the centered local-signal difference op(1), so augmentation does not affect the first-order variance.Specifically, nhq{¯SL(z) − ¯ST (z)} = op(1).

D Proof of Theorem 4

The proof establishes Theorem 4 by identifying the local oracle coefficient, deriving the resulting variance gain, and showing first-order equivalence between feasible and oracle estimators. It also extends the argument to fixed discrete groups through conditional counts and within-group central-limit reasoning.

  • Continuous case: m(z) = ΣSS,m(z)†ΣSY,m(z) minimizes the local quadratic f by conditional application of Lemma 1.This identifies the local oracle augmentation coefficient in the continuous case.
  • Continuous case: The feasible and oracle estimators share the same first-order limit distribution.The proof obtains this by substituting the oracle coefficient into Theorem 3's variance factor and controlling coefficient estimation error.
  • Continuous case: Gainm(z) = 1/{1 −(1 −π0)R2} follows by dividing the naive smoother's variance by the LACE variance.The naive smoother uses b = 0 and has variance factor {R(K)/fZ(z)}σ2Y,m(z).
  • Discrete groups: For a fixed discrete group g with pg > 0, the grouped CLT follows from the standard CLT for within-group sample means.Conditioning on group counts Mg and ng supports the oracle argument; replacing β⋆m(g) by bβm(g) uses a Cauchy–Schwarz bound and sample-mean rate Op((npg)^−1/2).

E Proof of Theorem 5

The proof extends Theorem 4 to paired model gaps and derives the deployment-score limit, including fixed-coefficient variance and feasibility of the estimated coefficient. It establishes that the feasible deployment estimator has the same first-order limit distribution as the oracle-coefficient estimator.

  • Paired gap: The paired-gap result follows by applying Theorem 4 with the outcome and signal replaced by paired-gap quantities.Conditional expectation identifies the paired gap as the target profile, while transferred assumptions yield the stated limit distribution and smoothing-bias control.
  • Paired gap: Conditionally on the profile value, Lemma 1 identifies the optimal paired-gap augmentation coefficient and its local coefficient of determination R^2.These quantities enter the variance factor for the paired-gap estimator.
  • Deployment score: For deployment scores, covariate-shift weighting and EP ρi = 1 yield the target-score representation used in the proof.The argument then establishes consistency of the feasible coefficient through weak-law, integrability, and Slutsky steps.
  • Deployment score: The self-normalized weighted means admit Hájek expansions, and the labeled–full-pool overlap determines the fixed-coefficient first-order variance factor.The expansions use EP rQ(Zi) = 1 and boundedness of rQ, while the overlap contributes the 1/M covariance form.
  • Deployment score: The multivariate central limit theorem gives asymptotic normality at the oracle coefficient, and the feasible estimator differs by op(n−1/2), preserving the same first-order limit.Thus coefficient estimation does not alter the deployment estimator’s first-order distribution.

F A cascaded auxiliary-rating construction

The cascaded construction uses gold outcomes nested within an expensive-rated subset and a full pool of cheap signals to form a two-stage centered estimator. Centering removes calibration requirements, while local ridge fits provide a practical implementation without claims of joint variance optimality or separate asymptotic guarantees.

  • Cascaded auxiliary-rating construction: The cascade observes cheap signals on the full pool, expensive-rater signals on a random subset, and gold outcomes on a nested random subset.The combined auxiliary vector is H_i = (E_i, S_i).
  • Cascaded auxiliary-rating construction: A two-stage centered estimator first uses expensive-rated items to reduce human-rated sampling noise, then cheap signals to reduce expensive-rated sampling noise.The construction applies profile-local means at both stages.
  • Cascaded auxiliary-rating construction: Neither auxiliary rater requires calibration because each enters through a centered difference.For equal-weight group profiles, the differences are exactly centered; normalized local kernel means satisfy the corresponding claim to first order.
  • Cascaded auxiliary-rating construction: A practical implementation fits both coefficients on labeled items using local ridge regressions of Y on H and of Y on S.The construction does not claim joint variance optimality for the cascade or establish separate adaptivity and limit-distribution theorems.

G Additional Experimental Tables · H Additional Implementation Details

The appendix provides expanded RE and pointwise-MSE results across eight benchmarks and three models, alongside implementation details for continuous and discrete profiling. It specifies kernels, smoothing, ridge settings, and continuous relaxations used in the real-data experiments.

  • G Additional Experimental Tables: Table row labels correspond to the estimator definitions in Section 5.2.This links the appendix’s tabulated comparisons to the estimator nomenclature introduced in the main paper.
  • H Additional Implementation Details: For continuous Z, naive and LACE use the same kernel, isolating the contribution of auxiliary signals in real-data experiments.All eight benchmarks use Gaussian smoothing, ordered discrete axes are jittered, discrete simulation profiles use within-group means, and Table 30 reports ordinal and exact alternatives.
  • H Additional Implementation Details: The main implementation fixes h = h0, b = h, and λ = 0.3 without enlarging the nuisance bandwidth or adapting λ.Bandwidth/ridge sensitivity and nested cross-validation are analyzed separately.
  • H Additional Implementation Details: Ordered discrete profiling axes are continuously relaxed by setting Zi = gi+Ui, with Ui ∼ Uniform(−0.4, 0.4), before Gaussian kernel smoothing.This applies to MATH-500 difficulty, ScienceQA grade, and GSM8K solution steps, using h = 1.5 × 1.06 ˆσZ M −1/5.

H.1 Robustness analyses · I Simulation study · I.1 Discrete-profile mechanism stress tests

Robustness analyses show that LACE’s gains are strongest for pairwise-anchor signals and persist across tuning choices, weaker judges and anchors, kernel choices, and sampling designs. Simulation studies stress-test local multi-signal combination and unlabeled-pool centering in ordered and categorical profiles.

  • H.1 Robustness analyses: Removing pairwise-anchor signals retains 42.1%, 47.0%, and 52.1% of full-LACE efficiency at nlab = 50, 100, and 200.This family is the largest contributor in 23/24 cells at n = 100; other signal-family removals retain 74.0–89.3%.
  • H.1 Robustness analyses: In sparse profile regions, default median RE is 4.064, 4.787, and 4.313, with 0/9 default cells below one at every budget.The sparse region comprises the five evaluation points in the lowest quartile of full-pool kernel effective sample size.
  • H.1 Robustness analyses: Within-L three-fold cross-validation keeps every method close to its fixed-hyperparameter value while selecting hyperparameters entirely within each labeled set.LACE uses b = h, and no gold outcomes outside L enter selection.
  • H.1 Robustness analyses: With jointly weaker judge and anchor models, LACE remains first overall: RE is 3.820 versus 1.899 under Llama-3.1-8B and 4.457 versus 2.281 under GPT-OSS-20B.LACE is best in 70/72 cells under Llama-3.1-8B and 71/72 under GPT-OSS-20B.
  • H.1 Robustness analyses: Jittered Gaussian smoothing has RE above one in 27/27 ordered-benchmark–candidate–budget cells, while triangular ordinal smoothing does so in 25/27, including 9/9 at n = 200.The comparison uses matched targets across alternative locality definitions.
  • I Simulation study: The simulation study uses Settings B and C as finite-sample mechanism stress tests for ordered discrete and unordered categorical profiles.They test whether local multi-signal combination and unlabeled-pool centering matter when useful signals change across groups.
  • I.1 Discrete-profile mechanism stress tests: Setting B uses five balanced ordered groups with exactly one specialist active at each level, while Setting C uses ten balanced categorical groups with one or two specialists.The designs are frozen before Monte Carlo replication, with Y, S, and L redrawn.
  • I.1 Discrete-profile mechanism stress tests: Feasible LACE and its oracle-coefficient counterpart are compared with gold-only, plug-in, cross-validation, and residual-based estimators using RE defined as a ratio of Monte Carlo mean MSEs.The experiment uses B = 300 replications and evaluates unweighted MSE across groups against θ(g).

I.2 Results … J.4 Signal Summary

The experiments show that LACE consistently improves conditional evaluation efficiency, including under discrete profiles, asymptotic bandwidth conditions, and repeated-pool superpopulation risk. The paper also documents six cheap evaluator signals based on judge scores, pairwise comparisons, self-reported confidence, and judge disagreement.

  • I.2 Results: Feasible LACE has the highest relative efficiency among feasible methods at every budget in both discrete-profile stress tests.Setting C has a larger oracle–feasible gap at nlab = 500 because ten categories leave about 50 labeled observations per category for estimating a six-dimensional coefficient.
  • I.2 Results: Group-varying signal slopes make Global CV and prediction-based plug-ins ineffective, while removing unlabeled-pool centering substantially increases error.Per-signal CV loses complementary specialists, and judge-only and multivariate plug-ins fall below the naive estimator at all budgets.
  • I.3 Gain formula in its asymptotic bandwidth regime: The shrinking-bandwidth experiment validates Theorem 4: oracle pointwise intervals contain predicted gains at 15/17, 17/17, and 16/17 points for π = 0.05, 0.10, and 0.15.Feasible pointwise intervals contain the predictions at 15/17, 16/17, and 16/17 points for the same label fractions.
  • I.4 Superpopulation risk under repeated pool resampling: The repeated-pool experiment evaluates integrated finite-sample MSE against an analytically known population target while incorporating pool sampling, label subsampling, and finite-bandwidth error.This complements rather than replaces direct validation of the asymptotic variance formula.
  • I.4 Superpopulation risk under repeated pool resampling: LACE achieves geometric-mean REs of 2.235×, 2.328×, and 1.849× at budgets 50, 100, and 200, with 2.127× overall, and wins all 72 setting–budget combinations.Per-signal CV is next-best overall at 1.161×; the largest-budget decline is compatible with budget-specific finite-sample risks and changing bandwidths.
  • J Cheap Evaluator Signal Collection: The signal-collection appendix uses API calls to a separate Opus 4.6 judge for evaluator signals, while the evaluated model supplies self-reported confidence in its original response.The appendix documents the exact prompts and procedures for K = 6 cheap evaluator signals.
  • J.1 Pointwise Judge Signal (Sjudge: Pointwise judge scores average three paraphrased prompts, and their variance forms a disagreement signal measuring prompt sensitivity rather than calibrated judge uncertainty.The disagreement feature is used as a covariate in the local ridge regression.
  • J.2 Pairwise Comparison Signal (Spair,h: Pairwise signals compare the evaluated model with weak, medium, and strong anchors using two answer-order-swapped calls per anchor, producing scores in {0, 0.25, 0.5, 0.75, 1}.The score is computed from wins and ties across the swapped comparisons.
Loading 2608.16210v1…