PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformal Calibration Transfer

Achref Doula

arXiv:2609.10737v1cs.LGstat.ML

TL;DR

Calibration labels may exist only in a source space while prediction sets are needed in a paired target space without labeled target calibration. TCC transports source calibration into the target space, then uses label-free mismatch correction through TCC-KS or weighted-TCC. Across corrupted-image and cross-modal datasets, it provides reliable coverage transfer and diagnostics for correction choice.

  • Problem

    Calibration labels are available in Xs, but valid prediction sets are needed in Xt without labeled target calibration data.

  • Method

    TCC transports labeled source calibration using paired data, then applies TCC-KS or weighted-TCC to correct residual target mismatch using unlabeled inputs.

  • Results

    Across CIFAR-100-C, Tiny-ImageNet-C, and SEN12MS, TCC transfers coverage without target labels; TCC-KS protects larger-mismatch cases and weighted-TCC is competitive with stable weights.

  • Takeaways & Limitations

    Label-free diagnostics can guide whether conservative KS correction or efficiency-oriented reweighting is more trustworthy at deployment.

  • Takeaways & Limitations

    TCC assumes unlabeled paired observations; noisy pairing can degrade transport, increase mismatch, and destabilize weighted correction.

Abstract

from arXiv · show

Conformal prediction converts point predictions into set-valued predictions with coverage guarantees under exchangeability between calibration and deployment data. We study conformal calibration transfer, where this requirement fails because labeled calibration is available only in a source space, while prediction sets are needed in a target space linked to the source through unlabeled paired observations (e.g., paired modalities or sensor changes). We propose Transported Conformal Calibration (TCC): we transport labeled source calibration into the target space using the paired data, and then correct residual post-transport mismatch using only unlabeled target inputs. We instantiate this correction with two complementary methods: TCC-KS, which uses a label-free uncertainty surrogate to detect mismatch and adjust calibration conservatively, and weighted-TCC, which reweights transported calibration toward the target domain for improved efficiency when weights are stable. We provide finite-sample target-domain coverage guarantees that adapt to an observable measure of mismatch. Across CIFAR-100-C, Tiny-ImageNet-C, and SEN12MS, we show reliable target-domain coverage transfer without labeled target calibration data, with label-free diagnostics that predict when correction is needed.

1. Introduction

The paper addresses conformal calibration when labels exist only in a source space but prediction sets are required in a paired target space. TCC transports source calibration and corrects residual mismatch using unlabeled target inputs.

  • Motivation: TCC targets deployments where source labels cannot directly calibrate prediction sets required in a different target space.Unlabeled paired observations connect the spaces, but no labeled target calibration set is available.
  • Approach: Paired observations enable a transport map that moves labeled source calibration inputs into the target space while retaining their labels.The transported examples provide pseudo-target calibration data for target-space scores.
  • Challenge: Transport alone can under-cover because transported and true target distributions may differ, while unlabeled target inputs cannot directly reveal true-label score mismatch.The paper argues that nontrivial label-free validity requires structural assumptions and label-free signals tied to score difficulty.
  • Correction mechanisms: TCC-KS uses a label-free uncertainty surrogate and one-sided KS discrepancy to reduce the internal level when mismatch is detected.Small mismatch preserves tightness; larger mismatch triggers intentional conservatism.
  • Correction mechanisms: Weighted-TCC reweights transported calibration toward the target distribution and is intended to improve efficiency when residual weights are stable.Effective-sample-size diagnostics indicate when the reweighting is trustworthy.
  • Empirical findings: Across corrupted-image and cross-modal remote-sensing datasets, TCC transfers coverage without target labels, with diagnostics indicating when correction is needed.TCC-KS protects coverage under larger mismatch, while weighted-TCC is competitive when residual weights remain stable.

2. Related Work

Prior work provides conformal coverage under exchangeability, shift-aware weighting, and domain transport, but does not directly solve paired cross-space calibration transfer. TCC instead uses transport to move labeled calibration information before correcting residual mismatch.

  • Conformal prediction under shift: Conformal prediction offers finite-sample marginal coverage, while shift-aware methods address non-exchangeability primarily within a shared input space.These approaches do not directly address labels in Xs with coverage required in Xt.
  • Domain transport: Domain adaptation methods learn mappings or aligned representations mainly for predictive accuracy or feature alignment rather than prediction-set coverage guarantees.TCC uses the mapping specifically to transport labeled calibration information into the target space.

3. Transported Conformal Calibration (TCC)

TCC transports labeled source calibration into the target space, then applies label-free corrections for residual mismatch. TCC-KS offers conservative validity protection, while weighted-TCC offers efficiency when residual weighting is reliable.

  • 3.1. Problem Formulation: TCC learns a transport map from paired data, preserves labels on transported calibration examples, and applies split conformal calibration in the target space.Transport is imperfect, so the resulting calibration may not match the true target score distribution.
  • 3.3. TCC-KS: KS-Corrected Transported Calibration: TCC-KS computes a one-sided KS discrepancy on a label-free uncertainty surrogate and adjusts the internal level to α⋆=max(0, α−δ+).The procedure uses disjoint unlabeled splits and a finite-sample inflation bound for the observable discrepancy.
  • 3.3. TCC-KS: KS-Corrected Transported Calibration: Small δ+ keeps TCC-KS tight, whereas large δ+ lowers α⋆ and intentionally produces conservative sets.The correction targets under-coverage when the target is harder than the transported domain; opposite deviation requires no correction.
  • 3.4. Weighted-TCC: Importance-Weighted Correction: Weighted-TCC performs transport-then-reweight using residual density ratios from unlabeled samples, improving efficiency when weights are stable.Effective sample size is used as a label-free diagnostic because concentrated weights can undermine reliability.
  • 3.6. Predictive Alignment: Predictive alignment during transport reduces measured δ+ and improves efficiency for both correction mechanisms.The alignment regularizer matches target-model predictive outputs on paired observations.
  • 3.5.1. AN IMPOSSIBILITY RESULT: Without structural assumptions, uniformly valid label-free procedures are near-vacuous; TCC obtains nontrivial guarantees through surrogate-control assumptions and diagnostics.The framework’s guarantees also rely on sample splitting or cross-fitting among transport, calibration, and mismatch estimation.
  • 3.5.1. AN IMPOSSIBILITY RESULT: TCC-KS and weighted-TCC provide complementary deployment choices: conservative protection under larger mismatch versus efficiency under stable residual weights.The diagnostics δ+ and ESS indicate which correction is likely to be reliable.

4. Evaluation

The evaluation tests label-free conformal calibration transfer across corrupted-image and cross-modal shifts, measuring target coverage, prediction-set size, and diagnostics for mismatch and weight stability. TCC-KS protects coverage under larger residual mismatch, while weighted-TCC is efficient when transported weights remain stable.

  • 4.1. Overall Performance and a Severe-Shift Stress Test: On average, Transported CP and weighted-TCC were near nominal on CIFAR-100-C and Tiny-ImageNet-C, while TCC-KS added a safety margin at moderate set-size cost.On SAR→RGB, TCC-KS achieved near-perfect coverage with average set size approximately 2.5.
  • 4.1. Overall Performance and a Severe-Shift Stress Test: Across severe-shift stress tests, TCC-KS restores coverage after Transported CP and weighted-TCC under-cover, while WCP remains more conservative with substantially larger sets.The stress test covers shot noise and motion blur at severities 4–5; TCC-KS improves validity with a moderate set-size increase.
  • 4.2. Effect of Transport Quality: As transport quality improved, δ+ decreased and ESS% increased, making TCC-KS sets tighter and weighted-TCC more credible.On Tiny-ImageNet-C, +KL changed δ+ from 0.1006 to 0.0148 and ESS% from 48.4 to 91.8 at epoch 150; weighted-TCC reached 0.900 coverage with size 69.71.
  • 4.3. Assumption Validation and Stress Regimes: Across 328 configurations, transport-only calibration under-covered by more than 1 percentage point in 11 cases, and TCC-KS recovered valid coverage in all 11 with 97.8% mean coverage and 1.82× inflation.The under-coverage cases averaged 87.9% versus 89.9% for oracle calibration before correction.
  • 4.3. Assumption Validation and Stress Regimes: The unlabeled δ+ diagnostic correlated 0.772 with set-size inflation and 0.734 with safety margin, supporting its use as a deployment signal for mismatch severity.In mild or certified corruption regimes, the median slack proxy was at most 9.4 × 10^-3 and worst-case slack was bounded by 4.44 × 10^-2.
  • 4.3. Assumption Validation and Stress Regimes: In the SAR→RGB stress regime, large δ+ caused TCC-KS to tighten to α⋆ = 0, yielding conservative behavior under persistent cross-modal mismatch.The result is consistent with the method’s role as a high-mismatch safety guardrail.

5. Discussion

TCC separates calibration transport from residual-mismatch correction, offering a conservative validity guardrail and an efficiency-oriented reweighting alternative with label-free deployment diagnostics.

  • Correction mechanisms: TCC-KS uses a label-free uncertainty discrepancy to adjust the calibration level and provide a target-domain coverage safeguard.The adjustment becomes stronger as residual mismatch grows, while remaining close to transported split conformal when mismatch is mild.
  • Correction mechanisms: weighted-TCC reweights transported calibration scores toward the target and can tighten prediction sets when residual shift resembles covariate shift and weights are stable.Its effective-sample-size diagnostic indicates when reweighting is likely to be reliable.
  • Deployment diagnostics: Label-free diagnostics support operational choice: use TCC-KS when validity is paramount and consider weighted-TCC when effective sample size is sufficiently high.Predictive entropy produces similar qualitative diagnostic trends as the least-confidence surrogate in a sensitivity check.
  • Scope and limitations: Imperfect pairing can degrade transport, increase residual mismatch, and make TCC-KS more conservative or weighted-TCC weights less stable.The framework assumes access to unlabeled paired observations linking source and target inputs.
  • Scope and limitations: The guarantees are marginal, and sample splitting can reduce effective sample size when unlabeled data are scarce.Shifts not reflected in the surrogate uncertainty statistic remain difficult to detect without target labels.

6. Conclusion

TCC transfers labeled source calibration into a target space through unlabeled pairs, then corrects residual mismatch with two complementary label-free methods.

  • Conclusion: TCC transports source calibration, applies label-free residual-mismatch correction, and combines a conservative guardrail with efficiency-oriented reweighting.The framework targets corrupted-image benchmarks and cross-modal remote sensing without labeled target calibration data.
  • Conclusion: Across shift severities, TCC provides reliable label-free coverage transfer and actionable deployment signals for selecting correction strength.TCC-KS uses a mismatch certificate, while weighted-TCC uses weight-stability diagnostics.

Impact Statement

Without structural assumptions linking observable target signals to true-label scores, distribution-free target validity without labels can require nearly vacuous prediction sets.

  • A.1. Proof of Theorem 3.1: Theorem 3.1 shows that uniformly valid prediction sets without labeled target data can be essentially vacuous when target conditionals are unrestricted.The worst-case expected set size is at least (1 −α)|Y| for a fixed target marginal.
  • A.1. Proof of Theorem 3.1: The impossibility result motivates explicit proxy assumptions that trade unrestricted distribution-free validity for nontrivial prediction sets.Procedures cannot guarantee substantially smaller-than-|Y| sets over the full class without additional assumptions.
  • A.3. Sufficient condition: A surrogate uncertainty statistic enables nontrivial sets only through an assumption connecting surrogate mismatch to true-label score mismatch.Assumption A2 provides this proxy relationship, while A2′ is a sufficient condition for zero slack.
  • A.3. Sufficient condition: Under A2′, the one-sided true-score CDF gap is bounded by the one-sided surrogate gap, δS ≤δT.This transfers a label-free discrepancy bound into a score-mismatch bound.
  • A.4. Quantile inflation: A one-sided CDF gap δ yields quantile inflation control, so transported calibration quantiles remain usable after accounting for mismatch.The quantile relationship follows from bounding the transported score CDF by the target score CDF plus δ.

A.5. Proof of Theorem 3.2

Theorem 3.2 converts a label-free surrogate mismatch estimate into a finite-sample target-coverage guarantee, while weighted-TCC obtains validity under covariate-shift and weight assumptions.

  • TCC-KS guarantee: Theorem 3.2 establishes a high-probability target-coverage bound for TCC-KS using the adjusted level α⋆ and the estimated mismatch certificate δ+.In the ideal ε = 0 case, nominal 1 −α coverage holds whenever δ+ ≤α.
  • TCC-KS guarantee: The proof bounds true-label score mismatch by the observable surrogate gap, then applies split-conformal validity to transported calibration scores.The argument combines Assumption A2 with a high-probability bound on the empirical surrogate discrepancy.
  • weighted-TCC guarantee: weighted-TCC achieves target-domain marginal coverage under covariate shift when oracle density ratios between target and transported inputs are available.Its proof uses weighted quantiles and a posterior argument for identifying the target draw among transported calibration points.
  • weighted-TCC guarantee: The weighted construction is practically sharp when residual shift is close to covariate shift, but estimated weights may concentrate and yield small effective sample size.This makes weighted-TCC complementary to the validity-oriented TCC-KS guardrail.
  • Experimental protocol: Experiments use corrupted-image and SAR-to-RGB settings with disjoint data splits, target predictors, LAC scores, and learned transport maps.The protocol evaluates CIFAR-100 and Tiny-ImageNet corruptions alongside paired SAR-to-RGB tiles.

B.8. Evaluation Metrics and Diagnostics

The evaluation uses target-held-out coverage and prediction-set size, supplemented by diagnostics for surrogate alignment, KS mismatch, and weighted-calibration stability. Figures summarize per-condition behavior across corrupted-image settings and compare methods against nominal coverage and Oracle CP.

  • Target empirical coverage and average prediction-set size are evaluated on held-out labeled target data, with labels used only for evaluation.
  • The KS diagnostic reports δ+ and its resulting worst-case miscoverage bound, while αbnd is explicitly treated as a conservative certificate.
  • Weighted-TCC stability is assessed using effective sample size computed from estimated calibration-to-target weights.
  • Surrogate validity is audited through Spearman rank correlation between the label-free surrogate and the true-label nonconformity score, plus bin-wise monotonicity diagnostics.
  • For corrupted-image benchmarks, metrics are macro-averaged across target conditions, while Figure 3 shows per-condition coverage distributions and Figure 4 shows corresponding set-size distributions.
  • Figures and condition-specific tables evaluate corruption settings across severities using ℓ1+KL transport at epoch 150, with coverage, set size, and δ+ reported.

D. Detailed Results for Experiment 2

Experiment 2 reports results across transport epochs and losses for CIFAR-100-C, Tiny-ImageNet-C, and SAR→RGB. The evaluation summarizes coverage, prediction-set size, and TCC-KS mismatch diagnostics across corruption conditions and severities.

  • Results are reported for multiple transport epochs and losses across CIFAR-100-C, Tiny-ImageNet-C, and SAR→RGB.
  • CIFAR-100-C and Tiny-ImageNet-C results are macro-averaged over four corruption types, while SAR→RGB has no corruption severities.
  • Across the reported settings, empirical marginal coverage and average prediction-set size are evaluated, with δ+ reported for TCC-KS.

E. Additional Cross-Modal Benchmark: SUN RGB-D

SUN RGB-D extends TCC evaluation to a real cross-modal RGB-to-depth classification benchmark using unlabeled paired observations for transport. The results show conservative coverage recovery by TCC-KS and improved coverage from weighted-TCC when weights are stable.

  • SUN RGB-D evaluates 19-class indoor-scene classification with RGB as the source domain and depth as the target domain.
  • The protocol learns transport from unlabeled paired RGB-depth observations, calibrates on transported labeled source examples, and evaluates target-domain coverage.
  • δ+ = 0.067 and ESS = 79.3 at the best transport checkpoint; Transported CP under-covers, while TCC-KS restores coverage above nominal with moderate set-size growth.
  • Weighted-TCC stays close to Transported CP and improves coverage when weights are stable, whereas WCP is more conservative and produces larger sets.

F. Robustness to Surrogate Choice: Predictive Entropy

The robustness study replaces least-confidence with predictive entropy as TCC-KS’s uncertainty surrogate and finds essentially unchanged behavior. Across the broader mismatch analysis, δ+ predicts conservativeness and identifies under-coverage regimes without target labels.

  • Robustness to Surrogate Choice: Swapping least-confidence for entropy leaves TCC-KS behavior essentially unchanged across corrupted-image and SAR→RGB settings.
  • Robustness to Surrogate Choice: Across all tested settings, surrogate-induced δ+ differences remain below 0.004, with stable tightened levels, coverage, and set sizes.
  • Diagnostic Validation: TCC-KS recovered coverage in all 11 identified under-coverage cases, while δ+ predicted method behavior across 328 configurations.
  • Distribution of Mismatch and Under-Coverage Recovery: The 11 under-coverage cases occurred at early-to-mid epochs under severe corruptions; transport-only coverage was 87.9%, versus 97.9% after TCC-KS.
  • Diagnostic Validation: δ+ correlated with set-size inflation at 0.772 and with TCC-KS’s safety margin over oracle coverage at 0.734.
  • Diagnostic Validation: TCC-KS coverage exceeded oracle coverage in 327 of 328 configurations, or 99.7%.

G.3. Dataset-Specific Patterns

Mismatch severity varies across datasets, and TCC-KS provides the strongest validity protection while weighted-TCC is more efficient when residual weights are stable. Training and diagnostic monitoring help practitioners select corrections and identify when transport or labeled target calibration may be needed.

  • Dataset-specific mismatch: SAR→RGB has the largest mismatch, with mean δ+ = 0.544, 3.31× inflation, and 87.5% of configurations showing high mismatch.It nevertheless has zero under-coverage cases because high early mismatch triggers strong conservativeness.
  • Dataset-specific mismatch: CIFAR-100-C shows mild mismatch, whereas Tiny-ImageNet-C is intermediate, with mean δ+ values of 0.054 and 0.121, respectively.The corresponding mean inflation values are 1.39× and 1.90×.
  • Transport improvement: Transport quality improves over training: δ+ decreases rapidly from epochs 5–20, continues falling through epoch 80, and stabilizes by epoch 150.The +KL predictive alignment objective accelerates this decrease.
  • Operational guidance: Practitioners can reduce mismatch by training longer, using better objectives, or increasing model capacity before accepting conservative fallback.The proposed workflow monitors δ+ and ESS% before choosing a correction mechanism.
  • Overall correction performance: TCC-KS achieves 99.7% validity with zero under-coverage cases across 328 configurations, while weighted-TCC achieves 91.8% validity with 1.24× mean inflation when weights are stable.TCC-KS has 1.67× mean inflation overall; weighted-TCC is more efficient when ESS% > 70% and δ+ ≤ α.
  • Operational guidance: The operating guidance is empirically calibrated to the evaluated datasets and model families rather than intended as a universal prescription.Only δ+ ≤ α is directly tied to the finite-sample coverage certificate; other severity bands are heuristics based on observed inflation.
Loading 2609.10737v1…