PaperlayerView exact revision on arXiv

Source-linked AI summary

Inferential Evaluation of Surrogate-Derived Models under Covariate Shift

Longtian Shi, Molei Liu, Doudou Zhou

arXiv:2608.15783v1stat.MLcs.LGstat.APstat.ME

TL;DR

When surrogate labels are abundant but gold outcomes are scarce and target covariates differ, evaluating a surrogate-derived model’s target operating characteristics is difficult. This paper develops a three-sample, source-aware debiased framework with cross-fitted density-ratio and threshold-kernel corrections, achieving near-oracle coverage in simulations and supporting target evaluation without gold labels.

  • Problem

    Surrogate-label discrepancies can affect a model’s gold-standard operating characteristics when target covariate distributions differ and gold outcomes are available only in small samples.

  • Method

    The paper develops cross-fitted three-sample estimators that transport source-specific outcome and surrogate residual corrections, including kernel localization near decision thresholds.

  • Results

    Full correction achieved 0.926–0.956 coverage across the six DGP-by-target cells, close to the oracle-nuisance range of 0.934–0.960.

  • Takeaways & Limitations

    Source-aware debiasing supports valid target-population inference for surrogate-derived models when target gold labels are unobserved.

  • Takeaways & Limitations

    Threshold-specific inference assumes a fixed interior threshold and bandwidth conditions requiring a shrinking neighborhood with increasing local information.

Abstract

from arXiv · show

In transfer-learning settings, a model derived from abundant surrogate labels may be deployed in a target population where gold-standard outcomes are unobserved. Evaluating its target performance is essential for determining whether decisions based on the model remain reliable, yet it is difficult when gold labels are scarce, and covariate distributions differ across data sources. We study a three-sample setting with a small gold-labeled source, a larger surrogate-labeled source, and an unlabeled target. Under conditional transportability, we evaluate the surrogate-derived model against the latent gold-standard outcome in the target population. We propose cross-fitted estimators that transport information from the two labeled sources through source-specific density ratios. We also combine outcome-regression augmentation with a kernel correction for estimating the model near a threshold, accounting for uncertainty from all three samples. We establish asymptotically linear inference for TPR and FPR, consistency and pointwise inference for the ROC curve, and asymptotically normal inference for AUC. Simulations assess bias, coverage, and sensitivity to bandwidth and relative sample sizes. A retrospective temporal validation on Chatbot Arena and a semi-synthetic ACS-Income study provide validation in real-world AI applications.

1 Introduction

The paper studies target-population evaluation of a surrogate-derived prediction model when gold outcomes are scarce, labels come from differently shifted sources, and the target has only covariates. It develops source-aware debiasing to support valid inference for operating characteristics despite nuisance-estimation and threshold-boundary errors.

  • Motivation: Gold outcomes are costly and scarce, whereas surrogate labels can be collected at scale to identify a population prediction score.Surrogates may come from proxy measurements, rules, weak supervision, or automated annotators.
  • Three-sample setting: The evaluation uses three mutually independent samples: gold-labeled and surrogate-labeled sources plus an unlabeled target with covariate shift across populations.The target supplies covariates but no gold labels, and each source may have its own covariate distribution relative to the target.
  • Identification: Under conditional outcome transportability, target performance is defined through TPR, FPR, the ROC curve, and AUC for the surrogate-derived model.The gold-labeled source identifies the gold outcome regression, while the auxiliary source identifies the surrogate-derived model and the target determines the covariate law.
  • Technical challenge: Flexible nuisance estimators can make plug-in inference unreliable because errors in the gold-outcome regression and surrogate score propagate into performance metrics.Thresholding is especially difficult: errors in the surrogate score can move observations across c, with leading contributions concentrated near the threshold.
  • Contribution: The central contribution is source-aware debiasing that corrects each first-order nuisance effect in its identifying population, enabling valid target inference without target gold labels.The method transports gold-outcome residuals from the gold-labeled source and surrogate-score corrections from the auxiliary source, with weighting near the evaluation threshold.

2 Method

The method transports gold-standard outcome information to the target using source-specific density ratios and cross-fitted augmented estimators. A kernel-based auxiliary correction addresses thresholding error, while isotonic projection and one-step inference construct ROC and AUC estimates.

  • Population identities: Source-specific density ratios transport gold-source and surrogate-source information to the target under source-support and bounded-density-ratio conditions.The ratios are w_HL(x) = p_T(x)/p_HL(x) and w_AL(x) = p_T(x)/p_AL(x).
  • Population identities: The augmented identity combines target imputation with transported gold-source residuals, leaving only a product of outcome-regression and density-ratio errors.Setting q ≡ 1 yields the corresponding identity for the target denominator p1.
  • Threshold correction: A kernel auxiliary correction offsets first-order error from replacing the true surrogate model with its estimate inside the threshold indicator.The correction concentrates on observations whose true surrogate score lies near the threshold and uses a kernel approximation to the Dirac measure.
  • Cross-fitted implementation: Cross-fitting separates nuisance training from validation across the gold-labeled, surrogate-labeled, and target samples, and combines target plug-in, gold-source, and auxiliary corrections.The gold-source terms remove first-order outcome-regression error, while the auxiliary term addresses surrogate-model error inside the indicator.
  • FPR, ROC, and AUC: ROC construction evaluates projected TPR and FPR on a threshold grid, adds deterministic endpoints, applies isotonic projection, and estimates AUC by trapezoidal integration or a scalar one-step estimator.Both AUC estimators target the same population AUC; the one-step estimator avoids requiring a distributional approximation for the entire estimated ROC curve.

3 Theoretical Results

Theoretical results establish asymptotically linear, Gaussian inference with consistent standard errors for threshold-specific TPR and FPR, fixed-FPR ROC values, and AUC. The results account for uncertainty across the gold-labeled, surrogate-labeled, and target samples, with threshold rates determined by effective auxiliary sample size.

  • Threshold-specific inference: Cross-fitted standard errors support pointwise confidence intervals for TPR and analogous Gaussian inference for FPR at fixed thresholds.The TPR standard-error estimator is ratio-consistent under the stated assumptions and nondegeneracy condition.
  • AUC inference: The scalar one-step AUC estimator is asymptotically normal with consistent standard-error estimation, without requiring a Gaussian-process limit for the full ROC curve.Its auxiliary-source influence term has variance of order one, so its contribution is of order nAL^−1.
  • Threshold-specific inference: The threshold-specific estimator has an asymptotically linear expansion and, under nondegeneracy, a standard normal limit after normalization by its standard error.The expansion separates uncertainty from all three independent samples, including the auxiliary-source correction near the threshold.
  • Threshold-specific inference: When (nALh)^−1 is negligible, the auxiliary correction does not affect the limit; when it dominates, the rate is (nALh)^−1/2, and comparable orders combine all three sources.A √nHL normalization arises when (nALh)^−1 = o(nHL^−1).
  • ROC inference: At fixed FPR levels, the ROC estimator has asymptotic inference based on source-specific influence terms that combine TPR estimation with estimation of the threshold solving the FPR equation.The section also establishes uniform consistency of the projected ROC estimator and its trapezoidal AUC.

4 Simulation Studies

Across 500-replication simulations with five-fold cross-fitting, Full generally delivers calibrated inference and lower error than fixed-score baselines. Bandwidth and nuisance-correction experiments show that variance calibration is condition-specific and both first-order correction directions are needed.

  • Study 1: Full achieved 0.926–0.952 TPR coverage across P1–P3 and had lower MSE than IW-HL in every evaluated cell.Empirical SD declined from 0.0295 →0.0155 in DGP-A and 0.0323 →0.0161 in DGP-B as sample sizes increased.
  • Study 2: Increasing C from 0.5 to 2 reduced empirical SD by 21.5%/24.5% but reduced coverage to 0.922/0.862 in DGP-A/B.For C ≤1, coverage was 0.934–0.960; mean SE fell faster than empirical SD as bandwidth increased.
  • Study 3: Full coverage was 0.926–0.956, compared with 0.100–0.296 for the plug-in estimator across six DGP-by-target cells.Full mean-SE/empirical-SD ratios were 0.927–1.036, close to the oracle-nuisance coverage range of 0.934–0.960.
  • Study 3: Both correction directions were needed because HL-only missed threshold-specific AL uncertainty, while AL-only missed substantial global-score uncertainty.The complete comparison attributes this requirement to both nuisance directions being first order.
  • Studies 1–2: Full ROC coverage was 0.882–0.954 and AUC coverage was 0.932–0.950, with AUC SE/SD of 0.974–1.006.Across 18 ROC cells, mean coverage was 0.927 for Full, 0.911 for IW-HL, and 0.919 for TSC.

5 Real Data Studies

The real-data studies provide complementary validation under natural temporal shift and revealed gold-label evaluation. Across both applications, the proposed Full estimator yields informative target-performance inference and corrections that materially improve agreement with validated outcomes.

  • Chatbot Arena: Chatbot Arena uses disjoint early HL, middle AL, and later Target cohorts to evaluate temporal-shifted target performance without artificial covariate tilting.The cohorts contain 2,816, 3,590, and 1,499 observations, respectively, and use embeddings, metadata, and A/B-swap augmentation.
  • Chatbot Arena: Full estimates target AUC at 0.848 (95% CI [0.814, 0.881]) and target TPR at FPR 0.10, 0.15, and 0.20 at 0.562, 0.640, and 0.717.The AUC interval width is 0.067, and its lower endpoint exceeds 0.80 despite two source-to-target shifts.
  • ACS-Income: In ACS-Income, Full estimates AUC at 0.8663 (95% CI [0.8216, 0.9110]), versus 0.8383 for human-correction-only and 0.7074 for plug-in.The disjoint HL/AL/Target cohorts contain 447/552/186,193 records, and Full’s interval is 0.089 wide despite only 447/552 labeled cases.
  • ACS-Income: After gold-outcome reveal, Full’s cross-estimand discrepancy is 0.0114, 92.3% smaller than plug-in’s 0.1475 and 31.5% smaller than human-correction-only’s 0.0166.Full is descriptively closest at FPR 0.10, 0.15, and 0.20, while the paired auxiliary-label correction is 0.0280 [0.0044, 0.0515] and remains positive across all six exploratory learners.

6 Discussion

The paper develops a three-sample debiased framework for evaluating surrogate-derived models against gold-standard labels in target populations, with source-aware debiasing enabling valid inference. Results from Chatbot Arena and ACS-Income support the theoretical findings and demonstrate practical potential.

  • Framework and contribution: The framework evaluates surrogate-derived predictive performance against the gold-standard label in the target population using three samples.Its central contribution is a source-aware debiasing principle that mitigates each first-order nuisance effect and enables valid inference.
  • Empirical validation: Chatbot Arena and ACS-Income results corroborate the theoretical findings and illustrate the framework’s practical potential.These real-data studies complement the extensive simulation studies.

Appendices · A Formal Assumptions

The appendices develop the paper’s formal assumptions, methodological implementation details, asymptotic results, proofs, simulation designs, and applications. The formal assumptions impose identification, regularity, nuisance-rate, threshold, ROC, and AUC conditions tailored to each inferential target.

  • Appendices: Appendix A states formal assumptions and regularity along the score-estimation path, while later appendices provide methods, asymptotics, proofs, simulations, and application documentation.Appendix B covers density-ratio estimation and FPR, ROC, and AUC constructions; Appendices C–F cover asymptotic results, proofs, simulations, and applications.
  • A Formal Assumptions: The asymptotic framework lets all three sample sizes n_g diverge jointly while fixing the number V of cross-fitting folds and treating n as a sequence index.The samples are indexed by g ∈ {HL, AL, T}, and validation-fold sizes within each source differ by at most one.
  • A.1 Conditions for TPR and FPR at a Fixed Threshold: For fixed-threshold TPR and FPR inference, Assumption A.1 requires independent, nonoverlapping samples, conditional transportability, source support, compact covariate support, bounded density ratios, positivity, and bounded outcomes and regressions.It also imposes kernel smoothness and bandwidth conditions, plus local boundedness and differentiability of target and auxiliary score densities near the threshold.
  • A.1 Conditions for TPR and FPR at a Fixed Threshold: Assumption A.2 adds cross-fitted nuisance consistency and remainder-rate control, together with local regularity along the path induced by estimated scores.The path condition controls second-order variation in derivatives and kernel-localization bias, while a sufficient conditional-density condition is established in Lemma D.4.
  • A.2 Additional Conditions for ROC and Trapezoidal AUC: For ROC and trapezoidal AUC inference, Assumption A.3 requires uniform curve regularity, a decreasing FPR curve, endpoint conditions, and a shrinking quasi-uniform threshold grid.Pointwise ROC inference additionally requires local differentiability and scale conditions bn = o(h), an = O(bn), ∆c,n = o(an), and ¯rS = op(an).
  • A.2 Additional Conditions for ROC and Trapezoidal AUC: Under Assumption A.3, isotonic projection is locally equivalent to the unprojected ROC estimator to first order, while uniform consistency and pointwise inference require distinct condition sets.The stronger bn = o(h) separates isotonic-block and localization scales; when the AL term determines an, this becomes nALh3 →∞.
  • A.3 Conditions for One-Step AUC Inference: For one-step AUC inference, Assumption A.4 requires a continuously distributed target score, smooth weighted score densities, pathwise differentiability, shrinking pilot bandwidth with nTb →∞, consistent pilot functions and density ratios, and product-rate conditions.The nuisance errors enter the scalar AUC expansion and include pilot estimation and replacement of the target outcome regression.

B Additional Methodological Details … C.1 Asymptotic Expansion for the Target Prevalence Estimator

The methodological details control auxiliary-correction remainders, estimate source-specific density ratios, construct monotone ROC/AUC estimators, and state asymptotic conditions for target prevalence inference. The procedures combine cross-fitting, kernel and isotonic corrections, and separate source-to-target weighting.

  • B.1 Remainder for the Auxiliary Correction: The remainder combines an Op(r2S,2) component with a kernel-approximation term, yielding Op(r2S,2 + h2rS,2).The two components arise from replacing µ(1)1(c) and from kernel approximation, respectively.
  • B.1 Remainder for the Auxiliary Correction: With symmetric kernels, the order-h term vanishes, and the auxiliary-correction remainder is op(an) under Assumption A.2.For compactly supported kernels, the tail term is zero for sufficiently small h; with exponentially decaying tails, it is op(h2rS,2).
  • B.2 Density Ratio Estimation Procedure: Density ratios are estimated separately by cross-fitted source-versus-target classifiers for the HL and AL sources.Each classifier uses source and target covariates outside the relevant held-out folds, and Bayes’ rule converts target-membership probabilities into ratios.
  • B.2 Density Ratio Estimation Procedure: The implementation permits calibrated probabilistic classifiers, while requiring prespecified probability clipping or weight truncation and evaluation of fitted ratios only on held-out source folds.Theoretical results use untruncated ratios under a bounded-density-ratio condition.
  • B.3 FPR, ROC, and AUC Construction: Raw TPR and FPR estimates are augmented with endpoint anchors and projected separately by least-squares isotonic regression, producing piecewise-linear monotone curves.The anchors are (FPR, TPR) = (1, 1) and (0, 0), and theoretical results use equal grid weights.
  • B.3 FPR, ROC, and AUC Construction: The projected ROC estimator is uniformly consistent, the trapezoidal AUC is consistent as grid mesh vanishes, and ROC values admit pointwise inference at fixed interior false-positive rates.Uniform confidence bands require stronger stochastic equicontinuity and uniform-in-threshold remainder conditions.
  • B.3 FPR, ROC, and AUC Construction: A scalar one-step AUC estimator targets the same population AUC as the ROC area while avoiding a process approximation for formal inference.Its augmentation uses the derivative with respect to score perturbations through weighted score densities q0 and q1.
  • C.1 Asymptotic Expansion for the Target Prevalence Estimator: Theorem C.1 formalizes the target prevalence estimator’s asymptotic behavior under Assumption A.1, requiring uniform foldwise convergence of ˆm(−k)HL to wHL in L2(PHL) and rw,HL,2rY,2 = op(bp,n).The stated rate condition uses bp,n = (n−1… as defined in the theorem passage.

C.2 Pointwise FPR and ROC/AUC Extensions … D.9 Proof of Proposition C.1

The paper extends inference from TPR to pointwise FPR, ROC, and AUC, while deriving oracle bandwidth benchmarks and proving the main asymptotic results through cross-fitting, remainder control, and three-sample central limit arguments.

  • C.2 Pointwise FPR and ROC/AUC Extensions: FPR inference follows from the TPR construction by substituting the negative-class outcome, regression, prevalence, and threshold terms.Under the stated conditions, the studentized FPR estimator converges to N(0, 1), its standard-error ratio converges to 1, and the resulting Wald interval is pointwise valid.
  • C.2 Pointwise FPR and ROC/AUC Extensions: The projected ROC estimator is consistent, and the trapezoidal AUC estimator is consistent when the AUC grid mesh Δu,n tends to zero.The ROC construction uses a generalized-inverse representation and equal-weight isotonic projection.
  • C.2 Pointwise FPR and ROC/AUC Extensions: Under Theorem 4’s conditions, the cross-fitted AUC standard error yields an asymptotically valid confidence interval.The slope-ratio consistency condition is separate from the ROC point estimator, and simultaneous ROC bands require a Gaussian-process limit whereas scalar AUC inference does not.
  • C.3 Oracle Bandwidth Details: The oracle bandwidth benchmark retains the leading kernel-localization conditional bias B2,R,nh2 and AL variance contribution VAL,R/(nALh).It is a conditional oracle comparison rather than a full conditional mean-squared-error criterion because it excludes h-independent bias, HL and target variance, and other nuisance remainders.
  • D. Proofs: The proof toolkit establishes fold-weight approximations, conditional empirical-mean bounds, ratio expansions, and a three-sample triangular-array CLT.The CLT applies when HL and target summands are bounded and the AL summand has envelope O(h−1) and variance O(h−1).
  • D.2 Proof of Theorem C.1 (Target Prevalence Estimator): The target prevalence proof uses cross-fitting and fold balance to obtain a first-order expansion, with empirical and population remainders controlled by nuisance rates and a product-rate condition.The population bias has product form, and the fold-weight replacement does not affect the first-order expansion.
  • D.3 Proof of Theorem 1: Theorem 1’s proof decomposes target, HL, and AL contributions, controls score-estimation and nuisance remainders, and obtains asymptotic normality from independent source-specific empirical sums.When s1,n(c) ≍ an, the normalized sum converges to N(0, 1), and no source must have asymptotically negligible variance.
  • D.5 Proof of Corollary C.1: The FPR proof maps the TPR argument to the negative class, preserving the remainder rates and yielding the corresponding centered influence-function expansion.The negative-class construction leaves the score and threshold indicator unchanged, while the product bias changes sign but retains the same absolute bound.

D.10 Proof of Theorem 4 and Corollary C.2 · E Additional Simulation Details and Results

The proof derives the asymptotic expansion by combining pairwise-kernel, source-correction, and prevalence expansions, then applies a triangular-array CLT and variance consistency. Its scope is pointwise for TPR, FPR, and ROC, while AUC uses a separate scalar pairwise expansion and does not yield simultaneous ROC validity.

  • D.10 Proof of Theorem 4 and Corollary C.2: The four-step proof expands the target pairwise plug-in, establishes cancellation of first-order outcome and score directions, expands prevalence, and applies a triangular-array CLT.The construction combines three source averages after the nuisance-direction cancellations.
  • D.10 Proof of Theorem 4 and Corollary C.2: Cross-fitting makes held-out gold and auxiliary records independent of fold-specific training information, while fixed fold count preserves the stated remainder rates.The argument establishes independence fold by fold before summing remainders.
  • D.10 Proof of Theorem 4 and Corollary C.2: The gold- and auxiliary-source correction remainders are op(1) under their respective nuisance-rate conditions.The displayed bounds are Op(rw,HL,2+rY,2+rΓY,2)=op(1) and Op(rw,AL,2+rS,2+rΓS,2)=op(1).
  • D.10 Proof of Theorem 4 and Corollary C.2: The prevalence estimator has expansion ˆp1 −p1 = PHL,n[wHL(Y −M)]+PT,n(M −p1)+op(ρn), enabling the final ratio expansion.The denominator is bounded away from zero, so quadratic ratio terms are op(ρn).
  • D.10 Proof of Theorem 4 and Corollary C.2: The three leading source averages are independent, satisfy a common finite envelope, and obey Lindeberg’s condition, yielding the stated Gaussian limit.Variance consistency and the Wald interval then follow from Slutsky’s theorem.
  • D.10 Proof of Theorem 4 and Corollary C.2: The proof establishes pointwise inference for TPR and FPR and local stochastic equicontinuity for ROC, but not the global process approximation required for a uniform ROC band.AUC instead relies on a scalar pairwise expansion and does not imply simultaneous validity over the ROC curve.

E.1 Data-Generating Processes · E.2 Controlled and Moderate Source Shifts · E.3 Implementation and Monte Carlo Algorithm

The simulation framework spans heterogeneous and stress-test data-generating processes, controlled and stronger source shifts, and cross-fitted implementation procedures for evaluating proposed and comparator estimators. It also specifies target-truth construction, fixed-score comparator inference, and a sparse quadratic-dictionary diagnostic.

  • E.1 Data-Generating Processes: DGP-B represents a heterogeneous target population, while unbounded Gaussian covariates intentionally stress-test empirical robustness beyond the compact-support theory.The asymptotic theory assumes compactly supported covariates for uniform rate control, whereas the simulations use unbounded Gaussian covariates.
  • E.1 Data-Generating Processes: An additional DGP-D diagnostic uses 20-dimensional sparse quadratic designs and generates Bernoulli gold and surrogate labels.The diagnostic separately specifies outcome and surrogate mechanisms before generating the corresponding labels.
  • E.2 Controlled and Moderate Source Shifts: Studies 1, 2, and 4 use controlled bounded-overlap source selection, whereas Study 3 applies a stronger bounded log-density-ratio tilt.The Study 3 construction is bounded because every coordinate of its design vector lies in [−1, 1].
  • E.2 Controlled and Moderate Source Shifts: For Study 3, logistic domain models correctly specify the log density ratio; a 10−6 probability guard was never activated, and no realized weight was truncated.Non-oracle estimators estimate source-specific weights separately within each cross-fitting training fold.
  • E.3 Implementation and Monte Carlo Algorithm: The implementation uses correctly specified logistic nuisance learners and histogram gradient boosting for sensitivity, with pointwise targets c ∈{0.2, 0.5} and ROC targets u ∈{0.1, 0.2}.ROC estimation uses a 199-point threshold grid, separate isotonic projections, and common grid-based linear interpolation.
  • E.3 Implementation and Monte Carlo Algorithm: Population truths are computed from 106 independent target draws, while fixed-score comparators use a score learner fit once on the complete independent surrogate-labeled sample.IW-HL forms transported weighted empirical ROC and AUC functionals; TSC fits a cubic Legendre score calibration standardized over target scores.
  • E.3 Implementation and Monte Carlo Algorithm: Comparator standard errors condition on the fixed score and omit surrogate-score-learning variability, so they do not provide unconditional inference for the population surrogate-derived model.The complete surrogate-labeled sample gives both comparators their strongest natural fixed-score implementation.
  • E.3 Implementation and Monte Carlo Algorithm: The DGP-D implementation expands 20 covariates into 230 degree-two terms and fits elastic-net logistic regressions with fixed settings, five-fold cross-fitting, exact controlled-shift ratios, and h = n−1/4.The dictionary contains 20 linear terms, 20 squared terms, and 190 pairwise products; no internal cross-validation or data-driven tuning is used.

E.4 Coverage and Standard-Error Summary

Coverage results distinguish stable inference from finite-sample calibration failures and show that point accuracy does not guarantee reliable reported standard errors. Bandwidth and learner choice materially affect coverage, with occasional degenerate conditional standard errors in difficult settings.

  • Finite-sample calibration: HGB coverage is 0.288–0.622 at c = 0.2, 0.770–0.886 at c = 0.5, 0.764–0.906 for ROC, and 0.788–0.922 for AUC.The pattern is consistent with nuisance errors remaining non-negligible at the inference scale, without implying that flexible learners necessarily fail.
  • Accuracy versus standard errors: Full has the most stable coverage in Study 1, whereas TSC can achieve lower RMSE alongside optimistic conditional standard errors.The table explicitly separates point accuracy from reported-SE behavior.
  • Bandwidth sensitivity: For FPR at c = 0.2, C = 2, Full versus IW-HL coverage is 0.814 versus 0.864 in DGP-A and 0.580 versus 0.676 in DGP-B.The comparison reinforces the bandwidth warning rather than being obscured by an average.
  • Degenerate standard errors: Three of 500 IW-HL replications had a degenerate zero conditional TPR SE in the difficult DGP-A/HL 0.25k cell at c = 0.2.Coverage uses all 500 replications, while studentized summaries use the 497 nondegenerate values.

E.5 Unified Study 3 Numerical Results … F.4 Cross-Fitting and Density-Ratio Estimation

The numerical studies show that Full mainly addresses standard-error underestimation, while diagnostics identify finite-sample limitations and document reproducible, cross-fitted transport construction. The applications specify cohort linkage, surrogate labeling, covariate-shift diagnostics, estimands, and density-ratio implementation details.

  • E.5 Unified Study 3 Numerical Results: Full’s SE/SD ranges from 0.927 to 1.036 across six main-display cells, while AL-only and HL-only corrections fail for different global or DGP-specific targets.The plug-in estimator’s principal failure is standard-error underestimation rather than large point-estimation bias.
  • E.6 Regularized Quadratic-Dictionary Diagnostic: Coverage is 0.922–0.946 for TPR(0.5), FPR(0.5), ROC(0.1), ROC(0.2), and AUC in the correctly specified 4×P2 quadratic-dictionary diagnostic.The experiment uses DGP-D’s complete degree-two dictionary and displays every evaluated coordinate.
  • F Application Details and Additional Results; F.1 Chatbot Arena Application: Additional Details; Data Linkage, Cohorts, and Masking: The Chatbot Arena linkage matches exact question IDs and verifies source hashes, while the declared target population retains 1,499 of 3,027 later decisive user-level comparisons (49.52%).Masking checks question, model-order, label, user, language, turn, and conversation hashes.
  • Data Linkage, Cohorts, and Masking; Natural Temporal Shift and Overlap: The surrogate label S is built from six judge votes using confidence-weighted canonical A/B orientations, with 27,482 of 32,922 X-eligible questions having all six records.A nonzero aggregate exists for 27,354 questions, and 442 individual votes use the missing-confidence fallback.
  • Natural Temporal Shift and Overlap; Content-Aware Representation: OOF domain AUC and uncapped density-ratio diagnostics assess temporal covariate differences without imposed tilting, while semantic-plus-metadata ratios summarize balance using SMDs.Domain AUC measures separability rather than proving positivity, and ratios use source-training predictions without forcing held-out means.
  • Content-Aware Representation; F Application Details and Additional Results: The primary encoder is fixed intfloat/multilingual-e5-small with 384 dimensions, separate prompt/response encoding, masked-mean pooling, L2 normalization, and deterministic 512-token truncation.Observed truncation rates are 1.01%, 6.41%, and 6.78%; dense coordinates are not individually interpreted.
  • Estimation and Validation Objects; Numerical Diagnostics and Sensitivities: Five shared folds produce held-out nuisance predictions using training folds only, while estimands distinguish population-model quantities from frozen finite-sample score validation.Full combines target plug-in information with human-label and LLM-label augmentation components.

F.5 Pointwise ROC Implementation and Diagnostics · F.6 Scalar AUC Numerical Implementation

The pointwise ROC implementation regularizes threshold-specific estimates and reports diagnostics on effective sample size, score ties, and population-score continuity. The scalar AUC implementation uses an exact tie-adjusted algorithm with analytic three-source inference and remains the primary endpoint.

  • F.5 Pointwise ROC Implementation and Diagnostics: F.5 evaluates TPR and FPR at 201 thresholds, then projects each sequence onto [0, 1] by bounded least-squares isotonic regression before inversion and interpolation.The slope-ratio pilot uses b = n−1/5.
  • F.5 Pointwise ROC Implementation and Diagnostics: F.5 computes pointwise influence values, analytic standard errors, and nominal Wald intervals according to Theorem 3 and Corollary 1.The implementation repeats the stated theorem and corollary formulas without additional application-specific modifications.
  • F.5 Pointwise ROC Implementation and Diagnostics: At target FPR u = 0.10, 0.15, 0.20, transport-weighted local kernel ESS values are 150.7, 97.3, and 64.8, respectively.The corresponding AL observation counts within | ˆmS(X) −ˆcu| ≤h are 87, 49, and 37.
  • F.5 Pointwise ROC Implementation and Diagnostics: The pooled cross-fitted Target score has 183,778 distinct values among 186,193 observations, with an ordered-pair tie probability of 1.63 × 10−7.The spline basis avoids the original coarse fitted score, but the diagnostics concern ˆmS rather than the population score m⋆S(X).
  • F.5 Pointwise ROC Implementation and Diagnostics: Because ACS covariates are largely integer- or category-valued, the implementation does not verify population-score continuity and interprets analytic intervals conditionally on stated regularity.The scalar one-step AUC remains primary because pointwise ROC results are less stable in this small-AL application.
  • F.6 Scalar AUC Numerical Implementation: F.6 computes the target pairwise AUC term exactly in O(nT log nT) time by sorting scores, using the tie-adjusted comparison κ(z, z′) = I{z > z′} + 1/2I{z = z′}.Weighted score-density pilots use a Gaussian kernel with b = n−1/5 and FFT evaluation.
  • F.6 Scalar AUC Numerical Implementation: The reported AUC adds both source corrections to the cross-fitted target pairwise term, uses the augmented prevalence denominator, and obtains standard errors from centered influence-value variances.The scalar procedure has no boundary bandwidth h and uses analytic three-source inference rather than a component bootstrap.

F.7 Cross-Fitted Target Benchmark · F.8 Exploratory Surrogate-Learner Sensitivity

The cross-fitted Target benchmark is an empirical evaluation of pooled fold-specific predictions, distinct from the population one-step estimand and unsuitable for assessing coverage. Exploratory sensitivity across six surrogate learners ranks models by AL out-of-fold log loss and finds consistent AUC improvement from auxiliary-label correction.

  • F.7 Cross-Fitted Target Benchmark: The revealed Target outcomes are used to compute empirical AUC and ROC values for pooled out-of-fold surrogate-derived models, with Target S never loaded or used.The Target AUC uses the standard tie-adjusted empirical pairwise statistic.
  • F.7 Cross-Fitted Target Benchmark: The Target benchmark pools predictions from five fold-specific finite-sample models, whereas the one-step estimator targets the population surrogate-derived model m⋆.These are different estimands: the benchmark is not the performance of one frozen deployable model.
  • F.7 Cross-Fitted Target Benchmark: Their difference is an empirical cross-estimand discrepancy, not repeated-sampling bias or a coverage assessment.
  • F.7 Cross-Fitted Target Benchmark: 0.8663 is the full scalar one-step AUC estimate, compared with 0.8549 for the cross-fitted Target benchmark, yielding an absolute discrepancy of 0.0114.The full estimate has SE 0.0228 and analytic 95% interval [0.8216, 0.9110], while the benchmark has SE = 0.0009.
  • F.7 Cross-Fitted Target Benchmark: 92.3% is the Full discrepancy reduction versus plug-in, while 31.5% is the reduction versus the human-correction-only discrepancy.The corresponding plug-in and human-correction-only discrepancies are 0.1475 and 0.0166.
  • F.8 Exploratory Surrogate-Learner Sensitivity: The six surrogate-score learners were examined after the initial Target-outcome reveal, making the analysis exploratory rather than prespecified model selection.Learners are ordered only by AL out-of-fold log loss; Target outcomes and Target-validation discrepancies do not enter the ranking.
  • F.8 Exploratory Surrogate-Learner Sensitivity: The spline-3 logistic learner has the lowest AL log loss and was therefore selected for the main presentation.
  • F.8 Exploratory Surrogate-Learner Sensitivity: For every learner, the auxiliary-label correction increases AUC.

G Further Discussion

The discussion emphasizes that surrogate-score estimation can be a first-order source of uncertainty in threshold evaluation, affecting both inference and data-collection priorities. It also clarifies the estimand’s scope, practical limitations, and directions for extending pointwise inference.

  • Implications: At a hard threshold, score-estimation error near the threshold contributes variance of order (n_AL h)^-1 rather than n^-1.A large surrogate-labeled sample alone does not justify treating the score as known when this contribution is not negligible.
  • Implications: The three-source variance components distinguish uncertainty from gold-outcome estimation, near-threshold score estimation, and target-population averaging.Their relative magnitudes can guide whether additional gold labels, surrogate labels, or target covariates would be most useful, subject to costs and nuisance-estimation quality.
  • Scope and limitations: The estimand evaluates the population surrogate-derived model under the target covariate law, not one particular finite-sample fitted model.Evaluating a specific deployed fit would require treating that fit as fixed and using a different estimand and inferential analysis.
  • Scope and limitations: Separate density ratios permit different covariate distributions across labeled sources but do not protect against violations of gold-outcome transportability.This limits the framework when the conditional transportability assumption fails.
  • Practical Considerations and Extensions: Simultaneous ROC bands, sequential monitoring, and extensions to survival or multiclass outcomes remain directions for future work.The current results provide pointwise threshold-specific and fixed-FPR ROC inference, fixed-u ROC inference, and scalar AUC inference; monitoring would require confidence bands uniform over the data-acquisition path.
Loading 2608.15783v1…