PaperlayerView exact revision on arXiv

Source-linked AI summary

Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores

Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba, Seungjin Choi, Hyunjin Shin

arXiv:2609.12386v1cs.LG

TL;DR

Label shift can undermine conformal coverage, while existing approaches do not jointly provide target alignment and adaptive interval widths. The paper derives a correction to Bayesian conformal scores from a posterior predictive tilting identity. In molecular property prediction under controlled label shift, the corrected score yields shorter intervals with comparable coverage.

  • Problem

    Under label shift, standard conformal coverage guarantees no longer apply, and existing approaches address target alignment and adaptive interval widths separately.

    Source: §1 · pp. 1–2

  • Method

    The LSA score corrects the source Bayesian score using a posterior predictive tilting identity that relates source and target predictives.

    Source: §1 · p. 2, §2.3 · p. 3

  • Results

    The LSA score produces shorter intervals than residual-based and source-based Bayesian scores, while coverage remains comparable across label-shift magnitudes.

    Source: §1 · p. 2, §4 · p. 6

  • Takeaways & Limitations

    With Bayesian Ridge Regression, the correction gives sample-adaptive, closed-form prediction intervals with negligible computational overhead.

    Source: §4 · p. 6

    The method uses pseudo-label-based density-ratio estimates and does not claim exact finite-sample validity under shift; stronger shift can worsen estimation and coverage.

    Source: §2.1 · p. 2, §3.2 · p. 5

Abstract

from arXiv · show

Conformal prediction provides distribution-free uncertainty quantification under exchangeability. However, this assumption is violated by label shift, where the marginal distribution of labels changes while the conditional distribution of inputs given labels remains stable. Under such shifts, standard conformal procedures no longer maintain their intended coverage behavior. Existing approaches address this via importance weighting. They pair the reweighting with residual-based nonconformity scores that ignore predictive uncertainty. The resulting intervals have uniform width. Bayesian conformal methods produce adaptive intervals by leveraging predictive distributions. They evaluate conformity under the source predictive, which is misaligned with the target domain under label shift. We propose the \emph{Label-Shift-Adjusted Bayesian Score} (LSA score), a nonconformity score derived from a posterior predictive tilting identity. This identity shows that the target predictive is an importance-weighted transformation of the source predictive. We use it to derive a direct correction to the Bayesian score. We evaluate the method on molecular property prediction under controlled label shift. The LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores. Coverage in the target domain remains comparable. Under stronger shift, all methods incur some coverage loss due to pseudo-label-based density-ratio estimation. The LSA score is defined for any source predictive with a tractable log-density. We instantiate it with Bayesian Ridge Regression, where the correction admits a closed form.

1. Introduction

Label shift changes the label marginal while preserving the input-given-label distribution, undermining standard conformal coverage. The paper introduces a Bayesian score correction that combines target alignment with adaptive intervals.

1. Introduction

  • Label shift changes the label marginal while leaving the conditional input distribution given labels unchanged, violating the exchangeability assumption behind standard conformal coverage guarantees.The paper notes that this shift can arise when scientific attention moves toward rare or extreme property values.Source: §1 · p. 1
  • Importance-weighting methods reweight calibration samples to match the target marginal, but their residual-based scores ignore predictive uncertainty and produce uniform-width intervals.These methods address target alignment while leaving adaptation to local difficulty unaddressed.Source: §1 · p. 1
  • Bayesian posterior-predictive nonconformity scores produce intervals of varying width, but under label shift they evaluate conformity under the source predictive distribution, inflating calibration quantiles and yielding unnecessarily wide intervals.Source: §1 · p. 2
  • The LSA score uses a posterior predictive tilting identity to correct the source Bayesian score; under Bayesian Ridge Regression, the correction yields closed-form intervals without additional computational cost.The identity expresses the target predictive as an importance-weighted transformation of the source predictive.Source: §1 · p. 2
  • The LSA score yields shorter intervals than residual-based and source-based Bayesian scores, with comparable target-domain coverage under controlled label shift.The evaluation is on molecular property prediction.Source: §1 · p. 2

2. Methods

The methods use posterior predictive tilting to correct Bayesian conformity scores for label shift, then apply weighted conformal prediction to produce target-domain intervals. The construction uses an exponential-tilt model and a Bayesian Ridge Regression instantiation with a closed-form correction.

2.1. Problem Setup

  • The method models label shift with a one-parameter exponential tilt and seeks intervals with target-domain marginal coverage near 1−αcp.The representation extractor is deterministic, and density ratios estimated from pseudo-labels mean exact finite-sample validity under shift is not claimed.Source: §2.1 · p. 2

2.2. Posterior Predictive Tilting

  • Under stated assumptions, the target posterior predictive is an importance-weighted transformation of the source predictive, using the label density ratio r(y) = pt(y)/ps(y).The assumptions include conditional label-shift invariance, absolute continuity, and a representative source predictive.Source: This is a mild condition satisfied… · pp. 2–3, §2.2 · p. 2

This is a mild condition satisfied whenever the training

  • The derivation identifies the population label-density ratio by applying Bayes’ rule to source-conditioned predictive distributions and using the source representativeness assumption.The source training data carry no information about the target label marginal.Source: This is a mild condition satisfied… · p. 3

2.3. Label-Shift-Adjusted Bayesian Score

  • The LSA score adds a log-density-ratio correction and normalization to the source Bayesian score, yielding a plug-in approximation to the target predictive’s negative log-density.The correction lowers scores for target-favored labels and raises them for target-disfavored labels.Source: §2.3 · p. 3
  • For Bayesian Ridge Regression, the Gaussian predictive and log-linear density ratio give a closed-form interval without additional computational cost over the source score.The exponential tilt shifts the predictive mean while leaving its variance unchanged.Source: §2.3 · pp. 3–4

2.4. Weighted Conformal Prediction with the LSA Score

  • Weighted conformal prediction uses calibration scores and a weighted empirical quantile; the resulting intervals are sample-adaptive around a label-shift-adjusted mean.The adjusted center is µ*(h) = µ(h) + β̂1σ^2(h), and the halfwidth is proportional to σ(h).Source: §2.4 · p. 4

3. Experiments

Across controlled molecular label-shift experiments, LSA yields shorter intervals while maintaining comparable target coverage. Under stronger negative shift, a leftward shift in its weighted score CDF explains the smaller quantile.

3.1. Experimental Setup

  • AqSolDB has 9,982 compounds; each trial allocates ∼40% to training and 20% each to weight estimation, calibration, and testing, comparing scores with shared BRR and weight machinery.The same protocol is evaluated on Lipophilicity as a second benchmark.Source: §3.1 · p. 4

3.2. Main Results

  • The qualitative trend persists with Gaussian-based density-ratio estimation, but stronger shift degrades pseudo-labels and coverage; LSA remains competitive within the estimated-weight regime.The Gaussian-based estimator preserves the log-linear form in y.Source: §3.2 · pp. 4–5
  • At β = −0.5, LSA's average interval length is 4.98 versus 5.74 for Source score and 5.82 for Residual (13% and 14% reductions); at β = 0, all achieve nominal 90% coverage.Paired t-tests on interval length yield p < 0.001 for all |β| ≥ 0.1.Source: §3.2 · p. 4, Table 1 · p. 5, Figure 1 · p. 5

3.3. Score Distribution Shift Analysis

  • As the score distribution shifts left, LSA shortens intervals without an additional coverage penalty; stronger shifts can still degrade coverage as pseudo-labels make density-ratio estimates less reliable.The same qualitative interval-length trend appears on Lipophilicity, with comparable coverage that is slightly higher for LSA.Source: §3.3 · pp. 5–6
  • At β = −0.5, LSA's weighted CDF shifts left and lowers the quantile through conformity scoring at the estimated target-aligned mean rather than the source mean.A smaller weighted quantile yields a shorter interval; at β = 0, the Source and LSA CDFs nearly overlap.Source: §3.3 · p. 5, Figure 2 · p. 6

4. Conclusion

The LSA score addresses limitations of conformal prediction under label shift, producing shorter intervals with comparable coverage while some coverage loss remains under stronger shift. The study uses a Gaussian predictive model, with richer models left as future work.

4. Conclusion

  • The LSA score corrects the standard Bayesian score using a posterior predictive tilting identity and yields sample-adaptive intervals under Bayesian Ridge Regression.Under Bayesian Ridge Regression, the intervals are closed-form with negligible computational overhead.Source: §4 · p. 6
  • The LSA score produces shorter intervals than residual-based and source-based Bayesian scores, with comparable coverage across label-shift magnitudes.As shift becomes more pronounced, all methods exhibit some coverage loss, consistent with approximation error from pseudo-label-based density-ratio estimation.Source: §4 · p. 6
  • The study uses a Gaussian predictive model; extending the approach to deep ensembles or sampling-based methods such as MC-Dropout is a natural next step.Source: §4 · p. 6

A. BRR Posterior and Predictive Distribution

  • Marginalizing over the weight vector yields the predictive mean and variance.The passage identifies H as stacking training representations row-wise and y as collecting labels.Source: A. BRR Posterior and Predictive… · p. 9

B. Full Proof of Proposition 2.1

The proof derives the target posterior predictive representation by combining label-shift invariance, Bayes’ rule, and the source-to-target label-marginal ratio.

B. Full Proof of Proposition 2.1

C. Population Tilt and the Estimated Log-Density Ratio

Under an exponential tilt model, the label density ratio is log-linear, enabling a tractable correction; in practice it is estimated using source labels and target pseudo-labels.

C. Population Tilt and the Estimated Log-Density Ratio

D. Closed-Form Derivation of ˆZ(h) and the Tilted Predictive

For Gaussian predictives and a log-linear estimated ratio, the appendix derives analytic forms for the tilted predictive and its prediction set, including a clipped implementation.

D. Closed-Form Derivation of ˆZ(h) and the Tilted Predictive

E. Gaussian Form of the Tilted Predictive

F. Score Inversion

G.1. Gaussian Mean Weight Estimation

This section evaluates Gaussian-mean density-ratio weights with the LSA score. The resulting intervals are shorter while target-domain coverage remains comparable.

G.1. Gaussian Mean Weight Estimation

  • The estimator fits separate Gaussian source and target label marginals, sharing the source variance.It constructs the estimated log-density ratio from the fitted means.Source: §G.1 · p. 12
  • Because the estimated log-density ratio is linear in y, the same normalizer and LSA construction apply with fitted coefficients.The Gaussian-mean estimator preserves the functional form used to derive equation (12).Source: §G.1 · pp. 12–13
  • Table 2 reports target coverage and average interval length at target coverage 0.9, averaged over 1,000 random seeds.Values are reported with one standard deviation; bold marks the best value per β.Source: Table 2 · p. 13
  • LSA yields shorter intervals while maintaining comparable empirical coverage with Gaussian-mean weights.The qualitative trend matches the logistic-regression results.Source: §G.1 · p. 13, Table 2 · p. 13

G.2. Additional Robustness Study on Lipophilicity

On Lipophilicity, LSA reproduces the qualitative interval-length and coverage trend observed on AqSolDB. The comparison holds on a smaller dataset with a different label range.

G.2. Additional Robustness Study on Lipophilicity

  • At β = −0.5, LSA interval length is reduced by approximately 4.6% relative to Source score and 5.3% relative to Residual.Empirical coverage remains comparable and is slightly higher for LSA.Source: §G.2 · p. 13
  • The three methods are nearly identical at β = 0, while increasing negative label shift brings consistently shorter intervals with LSA than with Residual or Source score.Coverage remains comparable across the reported comparison.Source: §G.2 · p. 13
  • The same qualitative trend appears on Lipophilicity despite its smaller size and different label range, suggesting the effect is not unique to AqSolDB.Source: §G.2 · p. 13

H. Encoder Pretraining Details

The downstream experiments use a frozen molecular BART encoder pretrained on a large unlabeled molecular corpus. Table 4 summarizes its architecture and pretraining setup.

H. Encoder Pretraining Details

Related papers

Loading 2609.12386v1…