Source-linked AI summary
Conformal Prediction under Exponential-Tilt Joint Shift
Seungjin Choi
TL;DR
Distribution shift can undermine conformal coverage, and adapting with unlabeled target inputs requires assumptions because they do not identify an arbitrary target joint distribution. The paper compares ExTRA-weighted calibration with and without tilting the source predictor. Tilting shortens sets in an informative synthetic regression setting, but coverage losses elsewhere show that the added adjustment is not reliably beneficial.
Problem
The paper studies how conformal prediction can retain intended coverage when deployment changes the data distribution, although labeled source data and unlabeled target inputs alone cannot identify an arbitrary target joint distribution.
Source: §1 · p. 1Method
The paper uses ExTRA to estimate structured joint shifts from target-input matching and conducts a controlled comparison of weighted calibration with and without predictive tilting.
Source: §1 · p. 2, §2.2 · p. 3Results
Tilting reduced mean paired length by 30.34% relative to ExTRA-WCP at (1, 1.2) in informative synthetic regression, while other settings showed coverage losses.
Source: §5.2 · p. 11, §6 · p. 16Takeaways & Limitations
Good coverage from weighted calibration alone does not ensure that adding predictive tilting preserves coverage, and choosing when to apply it without target responses remains open.
Source: §6 · p. 16Target inputs identify a joint ratio only within an assumed family, and they generally cannot determine the extent of unresolved shifts or estimation error.
Source: §6 · p. 16
Abstract
from arXiv · showhide
Conformal prediction can lose coverage when the data distribution changes after deployment. We study adaptation using labeled source data and unlabeled target inputs, allowing both the input distribution and its relationship with outcomes to change. We use Exponential Tilt Reweighting Alignment (ExTRA), introduced for classification by Maity et al. (2023), to estimate structured distribution shifts. We compare using its estimated weights in conformal calibration with additionally tilting the source predictive distribution. Shared learned predictors, estimated weights, calibration samples, and test observations isolate the effect of tilting. Existing theory gives both procedures target coverage with true weights and a common coverage bound with estimated weights. Identification calculations and an analysis of how scoring interacts with weight estimation error help explain why their performance can nevertheless differ. In a synthetic regression setting where the assumed models match the data-generating process and target inputs are informative about the shift, tilting reduces mean set length by about $30\%$ relative to weighting alone, with both methods attaining coverage near nominal. Tilting can instead cause substantial coverage losses in synthetic classification and in regression when target inputs provide little information about the response shift. Real-data experiments also show no consistent benefit. Good coverage from weighted calibration alone does not ensure that adding predictive tilting will preserve coverage. Deciding when to apply this additional adjustment using only source labels and target inputs remains an open problem.
1 Introduction
Distribution shift can undermine conformal prediction coverage, and unlabeled target inputs alone cannot identify an arbitrary target joint distribution. The paper compares weighted calibration with and without predictive tilting using ExTRA-estimated joint shifts.
1 Introduction
- Distribution changes can undermine predictive accuracy and uncertainty reliability, while labeled source data and unlabeled target inputs cannot identify an arbitrary target joint distribution without assumptions.Weighted conformal prediction uses the true target-to-source density ratio when Q ≪ P, but estimating it is difficult without target responses.Source: §1 · p. 1
- ExTRA estimates structured joint shifts by matching the target input distribution to the input distribution induced by proposed exponential tilts of the source joint distribution.For regression, conditional integration replaces the sum over classes; identification depends on the features and source distribution.Source: §1 · p. 2
- ExTRA-WCP uses the fitted ratio in weighted calibration while retaining the source score; ExTRA-WCP-T also tilts the source predictive distribution before scoring.The tilted predictor changes which responses are treated as plausible and can change prediction-set shape.Source: §1 · p. 2
- The controlled comparison holds the source predictor, fitted ratio, calibration sample, and test observations fixed to isolate predictive tilting.The paper analyzes how score-dependent misses interact with ratio-estimation error.Source: §1 · p. 2
- The study finds shorter sets at near-nominal coverage in synthetic regression, but coverage losses in synthetic classification, weak-information regression, and real-data experiments.Both procedures have target coverage with true weights and a common coverage bound with estimated weights, yet actual coverage can differ because score changes affect where misses occur.Source: §1 · p. 2
2 ExTRA and identification
The paper fits an exponential-tilt joint ratio using source and target inputs, then asks whether matching target inputs identifies that ratio within the assumed family. Matching can estimate an induced input distribution without determining a unique joint shift.
2.1 Data and shift model
- The data design separates source-model fitting, ratio estimation, calibration, and target testing, with only target inputs observed for the independent test pair.The shift model assumes a joint ratio represented by an exponential tilt of features, with correct specification meaning the true ratio belongs to that family.Source: §2.1 · pp. 2–3
2.2 Marginal matching
- ExTRA integrates the proposed joint tilt over the source conditional response distribution to obtain an induced input weight, then matches that implied input distribution to target inputs.Maximizing its objective minimizes discrepancy from the observed target input distribution without estimating a target density.Source: §2.2 · p. 3
2.3 What target inputs identify
- Matching target inputs does not necessarily identify the joint ratio: distinct joint tilts can induce the same input marginal, leaving the missing information unresolved even with unlimited samples.A penalty may select a solution but cannot supply the missing information.Source: §2.2 · p. 3, §2.3 · p. 4
- The analysis distinguishes shift representation, identification within the family, and estimation accuracy; the plug-in objective can also be affected by model, sampling, regularization, and optimization error.The paper does not derive a general regression estimation rate.Source: §2.3 · pp. 4–5, §2.2 · p. 4
3 Weighted calibration and predictive tilting
ExTRA-WCP and ExTRA-WCP-T share fitted joint weights but use different scores, with the latter tilting the fitted source predictor. Weighted ranks then determine candidate inclusion, and the two scores can have different actual coverage under estimated weights.
3 Weighted calibration and predictive tilting
- Both procedures use the same fitted joint ratio for calibration and candidate weights, but ExTRA-WCP retains the source score while ExTRA-WCP-T scores after predictive tilting.The fitted functions are fixed before calibration.Source: §3 · p. 5, Figure 1 · p. 6
3.1 Two scores from the same fitted model
- A response-dependent tilt changes relative candidate-response probabilities without refitting the source model, whereas an input-only tilt leaves the predictive distribution and score unchanged.The adjustment can be applied to suitable predictive distributions without requiring a Bayesian model.Source: §3.1 · p. 5
- Even the true tilt need not make the adjusted predictor equal the target conditional distribution, but true joint weights still guarantee target marginal coverage for a well-defined score fitted independently of calibration and testing.The guarantee does not require an exact predictive distribution.Source: §3.1 · p. 6
3.2 Candidate-dependent weighted calibration
- The weighted upper rank includes the candidate weight and calibration weights for scores at least as large, with tied scores included conservatively; candidates are retained when their rank exceeds α.ExTRA-WCP uses the source score, while ExTRA-WCP-T uses the tilted score; both use the same weights.Source: §3.2 · p. 6
- With response-dependent weights, changing a candidate response can change both its score and weight, so the weighted threshold can depend on the candidate and one cutoff may not describe the full set.For input-only weights, the candidate weight is constant across responses and the usual covariate-weighted construction is recovered.Source: §3.2 · p. 7
4 Coverage and the role of the score
Both procedures share a coverage guarantee under true weights and a common lower bound under estimated weights, but their actual target coverage can differ because their scores change which responses they miss. The section explains how ratio error, miss patterns, and candidate weights affect coverage and set size.
4.1 A common coverage bound
- With true weights both scores attain coverage at least 1 −α; with estimated weights they share a lower bound under the target, bounded by total variation from the surrogate target.The guarantee is marginal over calibration, the test pair, and score randomness, not for each input or realized calibration sample.Source: §4.1 · p. 7
4.2 Why the scores can have different coverage
- Target coverage can differ because ratio errors contribute more where misses are frequent, and undercoverage occurs only when transfer error exceeds surrogate surplus coverage.Observed target coverage alone does not separate ratio-error alignment with misses from differences in surrogate conservatism.Source: §4.2 · p. 8
- For two response modes, transfer error depends on the mode-specific ratio error weighted by the procedure’s conditional miss probability within each mode.When input marginals match, understating a mode’s probability contributes positively to transfer error if misses occur more often in that mode; undercoverage additionally requires the error to exceed the score’s surplus coverage.Source: §4.2 · p. 8
- At a fixed input, a positive response-mode tilt favors the positive mode relative to the source score, while an interaction tilt lets that adjustment vary with input.This score comparison alone does not determine coverage because calibration scores, weights, and candidate thresholds also change.Source: §4.2 · p. 9
- A finite predictive normalizer does not ensure finite regression-set length: a linear response tilt can make both score choices yield infinite-length sets at an input.Candidate weights may exceed the inclusion threshold even when response features are bounded, so regression sets require checking across the full response space.Source: §4.2 · p. 9
5 Experiments
Across experiments, ExTRA-WCP-T’s gains depend on the shift and the information available in target inputs: it substantially shortens synthetic regression sets in an informative, correctly specified setting, but can lose coverage elsewhere. Real-data experiments show no consistent benefit from tilting.
5 Experiments
- Experiments compare weighted calibration with and without predictive tilting using shared predictors, fitted ratios, calibration samples, and test observations.Synthetic settings use 30 replications; real-data standard errors describe variation across eight source partitions of fixed datasets.Source: §5 · p. 9, §5.5 · p. 15
5.1 Synthetic classification
- Tilting lowers synthetic classification coverage from ExTRA-WCP’s 0.924 to 0.814, despite both methods reaching 0.903 with true weights.The paired WCP-T minus WCP coverage difference is −0.110 (95% interval [−0.164, −0.057]).Source: Table 1 · p. 10, §5.1 · p. 10
5.2 Regression under covariate and response shifts
- The regression design identifies the response-shift coefficient b from target inputs when η > 0, but this depends on a substantive feature restriction.An added c v sign(y) interaction can make distinct tilt coefficients imply the same input marginal, which target inputs alone cannot distinguish.Source: §5.2 · p. 11
- At nominal coverage 0.90 and shift (1, 1.2), ExTRA-WCP-T’s mean paired set length is 30.34% shorter than ExTRA-WCP’s, with coverage difference −0.0052.Source: §5.2 · p. 11, Table 2 · p. 11
5.3 A response shift that depends on the noise input
- 34.24% shorter mean sets with a coverage difference of −0.0014: in the response–noise interaction setting, the full tilted model covers 0.899 versus 0.892 for separable ExTRA-WCP-T.The full model’s TV error is 0.041 rather than 0.384; the result supports representing the interaction, not superiority over other estimators that can represent it.Source: Table 3 · p. 13, §5.3 · p. 13
5.4 Identification sensitivity and ratio error
- As mode information weakens, ExTRA’s coverage deteriorates before nonidentification: at η = 0.25, ExTRA-WCP and WCP-T cover 0.819 and 0.730, versus 0.727 and 0.493 at zero signal.At η = 0.25, the two-stage baseline covers 0.904 and 0.874, showing that some weak-positive-signal loss is estimator-specific; at η = 0, target inputs provide no information about b.Source: §5.4 · p. 13, Figure 3 · p. 14
5.5 Classification and regression on real data
- Tilting has no consistent real-data benefit: it shrinks Gas sets but undercovers in batches 9 and 10, while Crime shows no consistent length reduction.In Gas, coverage falls from 0.911 to 0.740 in batch 9 and from 0.913 to 0.830 in batch 10; all methods under-cover in Crime’s South region.Source: Figure 4 · p. 15, §5.5 · p. 15
6 Discussion and conclusion
Tilting can shorten prediction sets at near-nominal coverage in synthetic regression when models are correctly specified and target inputs are informative, but other settings show coverage losses. Actual coverage depends on how score misses align with ratio errors, and choosing when to tilt without target responses remains open.
6 Discussion and conclusion
- Tilting shortens sets at near-nominal coverage in correctly specified synthetic regression with informative target inputs, but classification, weak-signal regression, and real data show possible coverage losses.The separable study shows the regression result with either ratio estimator, while the interaction study shows the value of representing the shift.Source: §6 · p. 16
- Actual coverage depends on where ratio errors and score misses coincide, as well as surplus coverage under the surrogate target.Source: §6 · p. 16
- Target inputs identify a ratio only within an assumed family, so choosing when to tilt without target responses remains open.Matching target inputs cannot determine the extent of unresolved shifts in general.Source: §6 · p. 16
A Identification in the regression experiments
The regression identification calculations distinguish shifts identifiable from target inputs within a specified family from shifts that target inputs cannot distinguish. Identification depends on the family’s restrictions and, in the separable case, on signal strength.
A Identification in the regression experiments
- Within the stated family and source-model conditions, the target input marginal uniquely determines the tilt parameters, including in the separable case.The source response-mode probability must vary with V.Source: A Identification in the regression… · p. 19
- For separable shifts, target inputs contain no information about the response-shift coefficient when η = 0.This explains why the sensitivity study varies η.Source: A Identification in the regression… · p. 19
- A different ambiguity arises when the response shift interacts with V, the input controlling source mode probabilities.The section introduces this as a distinct interaction family.Source: A Identification in the regression… · p. 19
The parameter triples
Allowing a response-shift interaction with V creates distinct joint shifts that can induce the same target input marginal. The identification conclusion therefore depends on excluding that interaction from the fitted family.
The parameter triples
- Distinct parameter triples can induce the same target input marginal while their joint ratios differ.Under the stated transformation, their conditional positive-mode probabilities are complementary.Source: The parameter triples · pp. 19–20
- At η = 1, (1, 1.2, 0) and (1, 0.8, −3) yield identical input marginals but different joint ratios.The experimental separable and U-interaction families omit V sign(Y), excluding the second shift.Source: The parameter triples · p. 20
- Allowing both interactions creates indistinguishable joint shifts, so matching target inputs cannot establish that an excluded interaction is absent.Source: The parameter triples · p. 20
B Essential experimental details
The experimental details describe a two-stage shift estimator and the fitting and evaluation choices used for synthetic and real-data comparisons. The WCP and WCP-T procedures share predictors, calibration samples, and evaluation data, isolating the effect of predictive tilting.
Synthetic experiments
- Synthetic regression fits the shift and response models jointly or separably, with multiple initializations for separable ExTRA and a distinct interaction experiment.The interaction experiment adds an input-dependent response coefficient, which the unchanged two-stage estimator cannot represent.Source: B Essential experimental details · pp. 20–22
Two-stage estimation
- The two-stage baseline estimates input reweighting and response-mode frequencies separately, then uses its fitted tilt in the same WCP and WCP-T procedures.It estimates mode frequency from target inputs using BBSE and requires distinct within-mode predictor means; those means coincide at η = 0.Source: B Essential experimental details · pp. 21–22
Real-data fitting and evaluation
- Real-data comparisons use separate training, shift-estimation, calibration, and target-fold evaluation data, with dataset-specific predictors and tilt features.The fixed datasets and partitioning schemes do not exactly follow the independent population sampling model of the coverage theorem.Source: B Essential experimental details · p. 23