PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformal Prediction with Stable Measurement Mechanisms

Seungjin Choi

arXiv:2610.34204v1stat.MEstat.ML

TL;DR

When target outcomes are unavailable, shifts can undermine conformal interval coverage, and observed inputs alone generally do not determine the needed correction. The paper uses a stable measurement distribution conditional on outcome and context to adapt intervals, then analyzes estimation costs and compares coverage and length. Synthetic comparisons show gains over covariate weighting and pooled label adjustment, while Air Quality results vary across controlled and calendar-based shifts.

  • Problem

    When target outcomes are unavailable, observed input changes generally do not determine how to correct conformal predictions, and continuous outcome distributions can be difficult to distinguish from finite measurement samples.

    Source: §1 · pp. 2–3

  • Method

    The paper uses stability of the measurement distribution given outcome and context to fit outcome corrections and adapt conformal intervals.

    Source: §1 · pp. 2–3

  • Results

    Synthetic comparisons show improved coverage over covariate weighting and shorter intervals than pooled label correction, while controlled Air Quality shifts gain coverage with longer intervals and calendar-based shifts show substantial undercoverage.

    Source: §5 · p. 23

  • Takeaways & Limitations

    Sharing a correction reduces fitting error when outcome changes are similar across contexts, while sufficiently different changes favor separate corrections.

    Source: §5 · p. 23

    Finite target data and weak measurements can make outcome changes difficult to distinguish, and matching target inputs alone cannot verify measurement stability.

    Source: §2.2 · p. 9

Abstract

from arXiv · show

In applications such as air-quality monitoring, inexpensive sensor readings can help predict pollutant concentrations measured by a separate reference instrument. We study how to adapt conformal prediction intervals to a new population when reference outcomes are available in the source data, but only sensor readings and context, such as weather conditions, are available in the target. We assume that the measurement distribution at a given outcome and context remains stable, while context frequencies and outcome distributions may change. This model is a particular form of causal mechanism shift. Its statistical restriction has been studied as sparse joint shift. Using established marginal matching, predictive adjustment, and weighted conformal calibration, we examine what this restriction contributes to coverage and interval length for continuous outcomes. Gaussian analyses show how measurement stability can identify outcome changes and how retaining context can shorten prediction intervals. For Gaussian models with known source mechanisms, we derive exact coverage formulas in the large-calibration limit while keeping the unlabeled target sample finite. These formulas explain when sharing a correction across contexts is preferable to fitting separate corrections, and why estimating outcome variability is particularly sensitive to weakly informative measurements. Experiments on synthetic Gaussian data show a coverage gain over covariate weighting at the same mean length and a length reduction of about $8\%$ relative to pooled label adjustment at comparable coverage. On the Air Quality dataset, controlled shifts yield coverage gains with longer intervals, while spring-to-autumn and year-over-year transfers show substantial undercoverage.

1 Introduction

The paper studies conformal prediction intervals when target outcomes are unavailable but a measurement process remains stable conditional on outcome and context. It analyzes how this restriction supports adaptation, what it costs to estimate corrections, and when retaining context helps.

1 Introduction

  • The model holds the measurement distribution given outcome and context fixed while allowing context frequencies and outcome distributions to change.Readings may include measurement noise, and their marginal distribution can still change across populations.Source: §1 · p. 2
  • Unlike covariate or label shift, the restriction preserves only the measurement distribution given outcome and context, so adaptation depends on measurement information and correction accuracy.A stable sensor response does not imply a stable outcome prediction from the sensor.Source: §1 · p. 2
  • The restriction is sparse joint shift (SJS) (Chen et al., 2022; Tasche, 2024), while prior identification and correction work covers categorical outcomes and this paper studies continuous-outcome intervals.The context and proxy are the selected and remaining feature blocks, respectively.Source: §1 · pp. 2–3
  • Separate context-specific corrections can improve coverage when their benefit outweighs their extra estimation error.The analysis distinguishes using context for prediction from fitting a separate shift in each context.Source: §1 · p. 3
  • Gaussian coverage formulas quantify finite-target-sample gaps even as calibration size tends to infinity, and show variance estimation is especially sensitive to weak proxy information.The scalar calculation also applies to label shift, the no-context special case.Source: §1 · pp. 3–4
  • The study compares estimated-mechanism coverage and length against covariate weighting and pooled label-shift adjustment, and conditional stability requires justification in each application.Sensor aging, recalibration, or unmeasured influences can change the measurement process.Source: §1 · pp. 2, 4

2 Mechanism shift models and identification

This section models target adaptation under stable proxy measurement mechanisms, examining how unlabeled target inputs can support outcome correction and when that correction is identifiable. It contrasts this restriction with covariate shift, showing that identical target input distributions can correspond to different target outcome means and cannot by themselves establish measurement stability.

2.1 Prediction data and the stable measurement model

  • When the proxy distribution given outcome and context is stable, target context and outcome distributions may change, but instrument stability alone does not ensure this condition if the reference outcome is measured with error.The target provides context and proxy readings, not outcomes; the shared conditional measurement density describes possible readings at fixed outcome and context.Source: §2.1 · pp. 4–5

2.2 Marginal matching and identification

  • Marginal matching selects an outcome correction by matching its implied input distribution to unlabeled target inputs, and injectivity of the measurement map makes that correction unique.At the population level, uniqueness requires distinct admissible outcome distributions to induce distinct sensor distributions within almost every target context; misspecification can make a good input fit misleading about outcomes.Source: §2.2 · pp. 7–8
  • Weakly informative measurements can make finite-sample outcome correction difficult even when the measurement map is injective, and matching inputs cannot verify that measurement stability holds.Small sensor loading relative to noise makes distinct outcome distributions produce similar sensor distributions; estimating the source measurement density adds uncertainty.Source: §2.2 · pp. 8–9

2.3 Estimation and predictive adjustment

  • A conditional exponential tilt represents outcome changes, while predictive adjustment reweights the source predictive distribution by the joint density ratio; context-only factors cancel for conditional prediction but remain in calibration weights.The tilt can shift conditional means and, with an outcome-squared term, variances. Predictive adjustment changes prediction-set size but does not guarantee shorter sets.Source: §2.3 · pp. 9–10

2.4 What the measurement restriction rules out

  • A Gaussian example shows that matching the same target sensor distribution under a broader joint-tilt model can yield different outcome means, so input matching alone does not determine the outcome correction.The two shift assumptions resolve the ambiguity by fixing different conditional distributions; the stable measurement model fixes the proxy distribution given outcome and context.Source: §2.4 · pp. 10–11
  • Stable measurement implies EQ(Y | C = c) = µZ,c/λ, whereas covariate shift implies EQ(Y | C = c) = λµZ,c/σ²Z.The restrictions require βZ,c = 0 and βY,c = 0, respectively; input matching then gives βY,c = µZ,c/λ and βZ,c = µZ,c/σ²Z.Source: §2.4 · p. 11
  • With λ = 1, τ² = 1, and µZ,c = 2, both models give target sensor distribution N(2, 2), but outcome means of 2 and 1, respectively.At the same observed reading z, the predictive means are z/2+1 under stable measurement and z/2 under covariate shift.Source: §2.4 · p. 11
  • Either restriction selects a unique correction in the example, but perfect input matching cannot distinguish the models or establish measurement stability.The ambiguity arises from allowing both tilts in (13); a more restricted joint-tilt family can be identifiable.Source: §2.4 · p. 11

3 Conformal calibration, context, and estimation error

This section separates the predictive benefit of retaining context from the cost of estimating target corrections. It analyzes weighted calibration, Gaussian coverage loss, and when shared or separate context-specific corrections are preferable.

3 Conformal calibration, context, and estimation error

  • The section evaluates how context can shorten intervals while separating that benefit from correction-estimation costs and weighted-calibration error.It recalls weighted-conformal guarantees before analyzing context and Gaussian correction fitting.Source: §3 · p. 11

3.1 Weighted calibration with oracle and estimated ratios

  • Weighted calibration with fitted ratios bounds coverage error by finite-quantile discretization and discrepancy between the target distribution and the distribution represented by the weights.With exact weighted ranks, only the total-variation discrepancy term remains.Source: §3.1 · p. 13

3.2 Prediction benefit of context

  • When context explains outcome-mean variation beyond the sensor, intervals using context attain minimum expected length while intervals using only the sensor are strictly longer.This result assumes a Gaussian conditional outcome distribution with constant variance.Source: §3.2 · pp. 14–15

3.3 Coverage with oracle and fitted Gaussian corrections

  • With known source mechanisms and infinite calibration data, fitted corrections still have a finite-m coverage gap, with additional loss from estimating variance, especially at weak measurement signal.The exact formulas keep the unlabeled target sample finite; the leading variance-estimation loss includes a term of order 1/(mκ^2).Source: §3.3 · pp. 15, 17

3.4 Shared versus separate corrections across contexts

  • Sharing one correction reduces estimation cost but adds a penalty when context-specific changes differ; similar changes favor sharing, while sufficiently different changes favor separate fits.The comparison does not guarantee coverage for a data-selected fit.Source: §3.4 · p. 18

4 Experiments

The experiments compare stable-measurement conformal adjustment with covariate- and label-shift methods, then examine correction sharing, finite-target-sample effects, and Air Quality transfers. SMM-T improves synthetic coverage or interval length in the reported comparisons, while Air Quality results vary by shift design.

4 Experiments

  • Air Quality results summarize day splits from one dependent time series and do not establish a time-series coverage guarantee.The study uses 50 day splits per design.Source: §4 · p. 19, Table 2 · p. 22

4.1 Prediction benefit of measurement stability and context

  • 0.8996 coverage at mean length 2.521: SMM-T with separate corrections beats Covariate-S at the same length and is about 8% shorter than Pooled label-T at comparable coverage.Predictive adjustment also shortens intervals relative to SMM-S with the same weights; a shared correction fails when contexts change in opposite directions.Source: §4.1 · p. 19, Table 1 · p. 20
  • Population residual quantiles give lengths 2.5150 with context and 2.7408 without it, an 8.2% reduction.The unequal target predictive variances fall outside Proposition 1’s common-variance setting, so these intervals are not claimed to attain its optimality bound.Source: §4.1 · p. 19
  • In a non-Gaussian check, SMM-T obtains 0.8982 coverage and length 2.902, versus 0.8714 and 3.070 for Covariate-S.The source outcome shapes, as well as context frequencies, affect calibration.Source: §4.1 · p. 20

4.2 Sharing corrections and finite calibration

  • Sharing gives higher coverage when context-specific changes are similar; separate corrections avoid sharing bias when those changes differ.At expected counts, the curves cross at η = 0.1931, close to the first-order value 0.194.Source: §4.2 · p. 20, Figure 2 · p. 21
  • With λ = 0.25 and true variance one, increasing calibration size from 1500 to 96000 moves coverage from 0.8359 to 0.8384, toward the 0.8386 large-calibration limit rather than 0.90.At variance 1.5, the corresponding coverages are 0.8636, 0.8708, and 0.8713.Source: §4.2 · p. 20

4.3 Air Quality under controlled and calendar-based shifts

  • 0.901 coverage versus 0.885 for Covariate-S under controlled context-and-outcome shift, with mean length increasing from 0.75 to 0.85.The paired coverage difference is 0.0163 (split Monte Carlo SE 0.0036).Source: §4.3 · p. 22, Table 2 · p. 22
  • Spring-to-autumn adaptation reaches 0.708 coverage, below Standard CP at 0.751 and Covariate-S at 0.733; year-over-year coverage is only 0.491.The calendar-based splits do not enforce measurement stability, and the year-over-year comparison uses unequal date ranges.Source: §4.3 · pp. 21–22

5 Conclusion

The conclusion separates the benefits of context and stable-measurement adjustment from the costs and limits of estimating corrections. Synthetic comparisons are favorable in specified settings, while Air Quality transfer results expose substantial undercoverage and unresolved scope limits.

5 Conclusion

  • Sharing reduces fitting error when changes are similar, whereas sufficiently different changes favor separate corrections; weak measurements amplify the coverage cost of variance estimation.For known source mechanisms, Gaussian formulas quantify coverage gaps from a finite unlabeled target sample even as calibration size tends to infinity.Source: §5 · pp. 22–23
  • Synthetic comparisons show SMM-T improves coverage over covariate weighting and shortens intervals relative to a correctly specified pooled label correction that omits context.This combines the reported synthetic comparison outcomes without extending them beyond those settings.Source: §5 · p. 23
  • Controlled Air Quality shifts yield coverage gains with longer intervals, while calendar-based shifts show substantial undercoverage, including worse coverage for spring-to-autumn adaptation.Stable physical sensors need not satisfy the restriction conditional on a noisy outcome or coarsened context, and matching target inputs cannot verify the restriction.Source: §5 · p. 23
  • Exact weighted ranks give finite-sample marginal coverage with the true ratio under stated sampling assumptions, but FWQ and fitted ratios add approximation and distributional error.Coverage analysis for estimated measurement mechanisms, broader outcome families, and dependent observations remains future work.Source: §5 · p. 23

A Proofs and supplementary calculations

The supplement organizes proofs in the order of the main-text results and supplies exact Gaussian evaluation formulas and full statements for context-fitting results.

A Proofs and supplementary calculations

  • The supplementary calculations follow the main-text result order and include exact Gaussian evaluation formulas and full context-fitting statements.The identification lemma is proved immediately after its main-text statement.Source: A Proofs and supplementary calculations · p. 26

A.1 Proof of Lemma 2 (finite-quantile discretization)

The proofs establish a finite-quantile bound using weighted exchangeability, then transfer fitted-weight coverage to the target distribution through a total-variation argument.

A.1 Proof of Lemma 2 (finite-quantile discretization)

  • Weighted exchangeability identifies each observation’s probability of being the target draw; bounding the failure indices establishes the finite-quantile result.If deleting a target index leaves no calibration weight, the procedure returns the outcome and covers it; otherwise failure requires its score to exceed the weighted calibration quantile.Source: §A.1 · p. 26

A.2 Proof of Corollary 1 (fitted finite quantiles)

  • With the fitted weight function fixed, the finite-quantile bound applies under its induced distribution; changing the test distribution to the target can reduce coverage by at most their total-variation distance.The proof applies the estimated-weight transfer argument of Lei and Cand`es (2021) to the joint prediction pair.Source: §A.2 · p. 26

A.3 Proof of Proposition 1 (context and minimum interval length)

For Gaussian outcomes, concavity bounds coverage at a given expected interval length; retaining context can strictly shorten intervals when conditional predictive means vary across contexts.

A.3 Proof of Proposition 1 (context and minimum interval length)

  • For length ℓ, Gaussian coverage is f(ℓ) = 2Φ{ℓ/(2σ)} − 1, an increasing, strictly concave function.The proof uses the greatest-probability property of an interval centered at the normal mean, then applies Jensen’s inequality to random interval lengths.Source: §A.3 · pp. 26–27
  • Constant-length intervals centered at µpred(C, Z) attain the coverage bound.Their length is 2σzα.Source: §A.3 · p. 27
  • When VarQ{µpred(C, Z) | Z} > 0 on a set of positive probability, a Z-only interval has strictly greater expected length at the required coverage.The proof establishes strictness either from nonconstant interval lengths or from the inability of a Z-measurable midpoint to match the conditional mean almost surely.Source: §A.3 · p. 27

A.4 Proof of Proposition 2 (exact coverage and oracle gap)

The proof derives limiting coverage by averaging over the finite target fitting sample after calibration grows large. It also examines fixed-variance fitting and clipping effects on coverage.

A.4 Proof of Proposition 2 (exact coverage and oracle gap)

  • The exact finite-m calculations average the coverage over fitted parameters, with the oracle gap defined relative to 1 − α.The derivation introduces Hv and averages over U to obtain Cm and Δm.Source: §A.4 · pp. 27–28
  • As calibration grows, both weighted conformal rules converge to the true target coverage of I∞, conditional on fitted parameters and then averaged over the fitting sample.Under the fitted target distribution, the standardized predictive residual is normal, and the weighted quantile converges to zα.Source: §A.4 · p. 27
  • For fixed fitted variance v0 = v, the added variance from fitting the mean at finite m makes coverage strictly below 1 − α.The proof also states that the centered mean error and variance pivot have distributions independent of µ.Source: §A.4 · p. 28
  • For α = 0.10, m = 100, λ = 0.5, v = τ² = 1, and bounds [0.99, 10], clipping gives Cm = 0.9103 and Δm = −0.0103.The example shows overcoverage even though the true variance lies inside the clipping bounds.Source: §A.4 · p. 28

A.5 Proof of Proposition 3 (leading coverage gaps)

The proof analyzes leading coverage losses from estimating outcome parameters, showing that weak measurements particularly impair variance estimation and that variance-estimation error locally reduces average coverage.

A.5 Proof of Proposition 3 (leading coverage gaps)

  • As λ approaches zero, information about the outcome mean decreases as λ², while information about outcome variance decreases as λ⁴.At zero loading, the sensor contains no information about either parameter.Source: §A.5 · p. 28
  • Negative coverage curvature means small, mean-zero variance-estimation errors reduce average coverage to second order.Near the true variance, underestimation loses more coverage than an equal overestimation gains; this local effect does not require global concavity.Source: §A.5 · p. 29
  • Using the unbiased sample variance removes the variance-estimator-bias contribution without changing the leading variance term.The expansion separates contributions from bias, mean estimation, and curvature with respect to the variance estimate.Source: §A.5 · p. 30
  • In the numerical example, unbiased variance estimation gives coverage 0.8407, while the exact expression is advised when weak measurements cause large errors or frequent boundary estimates.The expansion holds at fixed κ as m grows; projection places the unprojected estimate at the lower bound 0.05 with probability 0.0380.Source: §A.5 · p. 30

A.6 Statement and proof of Corollary 2 (context-specific fitting cost)

The corollary gives context-specific coverage costs when outcome means, and optionally variances, are fitted separately. Its marginal loss depends on measurement information, while rare contexts can still have poor individual coverage.

A.6 Statement and proof of Corollary 2 (context-specific fitting cost)

  • With known outcome variances, separate mean fitting yields large-calibration coverage averaged over target fitting observations and contexts.The setup uses context-specific Gaussian outcomes and sensor measurements, with target context probabilities πc.Source: §A.6 · p. 30
  • For fixed K, positive context probabilities and signal-to-noise ratios, the corollary gives first-order coverage gaps for separate mean-only and mean-and-variance fits.It averages over random context counts in the regime n → ∞ before m → ∞; a prespecified fallback applies if any Mc < 2.Source: §A.6 · pp. 30–31
  • The leading marginal location loss is proportional to 1/(mκc), because the context probability cancels, although rare contexts can have poor individual coverage.Each context receives about mπc target observations, and its within-context loss is proportional to 1/(mπcκc).Source: §A.6 · p. 31
  • The prespecified fallback does not affect the displayed first-order terms because the probability of Mc < 2 decays exponentially in m.The fraction is defined as zero when Mc < 2.Source: §A.6 · p. 31

A.7 Statement and proof of Proposition 4 (shared versus separate corrections)

Proposition 4 compares shared and separate location corrections under Gaussian measurement models, showing that separate fitting helps when context-specific changes outweigh its additional estimation cost. The experimental implementation describes the Gaussian comparisons, a non-Gaussian check, and finite-calibration studies.

A.7 Statement and proof of Proposition 4 (shared versus separate corrections)

  • Separate corrections help when context-specific outcome changes exceed the additional estimation cost of fitting K parameters instead of one.The local comparison considers differences between contexts on the order of estimation error.Source: §A.7 · p. 32

B Experimental implementation

  • The Gaussian comparisons use context-specific source fits, separate or precision-weighted shared corrections, and covariate-weighting and pooled label-adjustment comparators.The pooled label adjustment predicts from sensor readings alone; covariate weights use context and sensor features.Source: §B.1 · p. 33

B.1 Gaussian comparisons and non-Gaussian check

  • The non-Gaussian check combines Gaussian and mixture outcome fits with a shared linear sensor regression, comparing separate tilts with covariate weights.It evaluates conditional coverage and length by numerical integration.Source: §B.1 · p. 33

B.2 Sharing corrections and finite calibration

  • The sharing study compares shared and separate corrections across target context counts, while the scalar calibration study contrasts empirical-quantile and exact population-quantile coverage.The scalar study uses 1,000 independent normal and chi-square fitting pivots, with calibration samples nested over n.Source: §B.2 · pp. 33–34

B.3 Air Quality data and evaluation

The Air Quality evaluation uses log carbon monoxide outcomes, sensor measurements, and temperature and humidity contexts. It assesses controlled and calendar-based transfers with separate source, target-fitting, and evaluation samples, while noting that the day splits do not establish temporal independence or independent validation of design choices.

B.3 Air Quality data and evaluation

  • The outcome is log CO(GT), with five log-transformed sensors and warm and humid contexts; absolute humidity is not used as a predictor.Records coded −200 in specified sensor or weather variables are excluded, and positive CO is required.Source: §B.3 · p. 34
  • Controlled shifts allocate eligible days to source training and calibration and target fitting and evaluation, with three designs varying selection by context and outcome.The selection designs use ξ = 0, ξ = 0.8s_H + 0.4s_T, and ξ = 0.25s_T + 1.2s_Hℓ(y).Source: §B.3 · p. 34
  • Calendar-based transfers include weekdays to weekends, spring to autumn 2004, and March–April 2004 to March–April 2005.Source days are split 60/40 for training and calibration, and target days 50/50 for fitting and evaluation.Source: §B.3 · p. 34
  • Day splits separate sample roles but do not establish temporal independence or independent validation of design choices.Source: §B.3 · p. 34

Related papers

Loading 2610.34204v1…