Source-linked AI summary
Missingness-Aware Conformal Prediction Under Cross-Hospital Distribution Shift
Liang You, Dongwen Ou, Hengyu Shi, Siyuan Dai
TL;DR
Marginal conformal coverage does not ensure coverage within groups defined by missingness, and missingness patterns can differ across hospitals. The paper selects a measurement on an independent sample and calibrates separately by its availability. It finds pooled gains on selected groups, but those gains do not consistently carry over to broader groups or within-hospital evaluation.
Problem
Marginal conformal coverage averages across patients and does not ensure coverage within groups defined by missingness, whose patterns and errors can differ across hospitals.
Source: §1 · p. 1Method
The procedure selects a site-varying measurement on an independent sample, partitions patients by its availability, and applies Mondrian calibration within each group.
Source: §1 · p. 2, §3.2 · p. 4Results
Missingness-aware calibration reduced the average worst-group coverage gap on selected groups in all six settings, with a median reduction of 1.9 percentage points, but gains did not extend uniformly.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
Pooled improvement alone cannot establish better within-hospital coverage, even when calibration groups are fixed.
Source: §6 · p. 10The retrospective study uses information unavailable at admission, assumes independent calibration observations, and includes care units from only one MIMIC-IV institution.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Clinical measurements are recorded for some patients but not others, at rates that differ across hospitals, and marginal conformal coverage does not ensure coverage within groups defined by missingness. We propose a missingness-aware conformal calibration procedure for mortality prediction under cross-hospital distribution shift. It selects a measurement on an independent sample, groups patients by whether that measurement is recorded, and applies Mondrian calibration within each group, so no calibration outcome is reused. We evaluate the procedure across hospitals in eICU and across care units within one MIMIC-IV hospital, using three predictors. Relative to pooled calibration, it reduces the average worst-group coverage gap on its selected groups in all six settings, with a median reduction of 1.9 percentage points; paired site-bootstrap intervals exclude zero in five. These gains do not extend uniformly. Calibration by predicted risk achieves smaller gaps on a broader panel of missingness groups, and when eICU hospitals are evaluated separately, the gain shrinks for all three predictors and reverses in sign for one. We explain this discrepancy with a hospital-level decomposition. Pooling reweights hospitals through a covariance between group shares and coverage errors, and lets errors of opposite sign cancel: weighting explains the reversal, and cancellation accounts for most of the attenuation for the other two predictors. Constructed population distributions show that pooled and within-hospital evaluations can rank calibration methods oppositely even without sampling noise. Pooled improvement alone therefore cannot establish better coverage within hospitals, even when the calibration groups are fixed.
1 Introduction
Marginal conformal coverage can hide undercoverage in missingness-defined groups, motivating a calibration procedure that partitions by measurement availability. Pooled gains on selected groups do not establish better within-hospital coverage.
1 Introduction
- Because missingness rates and within-group errors can both shift across hospitals, marginal, within-group, and within-site coverage are distinct targets.A pooled average can meet its target while patients lacking a measurement remain undercovered.Source: §1 · pp. 1–2
- The procedure selects a measurement on an independent sample, partitions patients by its availability, and applies Mondrian calibration without reusing calibration outcomes.Its selected groups have the usual Mondrian guarantee under exchangeability.Source: §1 · p. 2
- 1.9 percentage points was the median reduction in average worst-group coverage gap on selected groups versus pooled calibration across six dataset–predictor settings.The gains did not extend uniformly to broader groups or separate-hospital evaluations.Source: §1 · p. 2
- Risk calibration achieved smaller gaps on a broader missingness panel, while separate eICU hospital evaluation shrank gains for all predictors and reversed the XGBoost comparison.These patterns show that results depend on the audited groups and evaluation level.Source: §1 · p. 2
- Pooling reweights hospitals through covariance between group shares and coverage errors; weighting explains the XGBoost reversal, while cancellation removes most LR and MLP gains.Constructed populations also show that aggregation alone can reverse method rankings without sampling noise.Source: §1 · p. 2
2 Related work
Related work distinguishes Mondrian coverage guarantees and missingness-based grouping from conditional-coverage methods, adaptive scores, and approaches to evaluating coverage across sites.
2 Related work
- Mondrian methods calibrate within fixed categories under suitable exchangeability conditions, while prior missingness-mask coverage work uses imputation and augmentation [Zaffran et al., 2023].The paper studies binary clinical prediction across sites and distinguishes calibration groups from evaluation groups.Source: §2 · p. 2
- Conditional-coverage methods address different validity targets, while this paper holds the score fixed to isolate the effect of calibration groups.Its score-tree comparator is not the full procedure of Martinez Gil et al. [2024].Source: §2 · p. 3
- Coverage-evaluation work includes learned assessment criteria and hierarchical methods, but the paper’s hospital comparisons are descriptive rather than hierarchical coverage guarantees.It focuses on fixed clinical groups, hospital weights, and aggregation order.Source: §2 · p. 3
- Other distribution-shift approaches reweight scores, localize predictions, or adapt calibration online; the paper’s mask-frequency comparator is not Fan et al.’s method [2026].That method combines imputation, masking, and weighting for mask-conditional coverage.Source: §2 · p. 3
3 Missingness-aware conformal calibration
The method fixes a grouping rule independently of calibration, then applies conformal calibration within groups. Its coverage guarantee depends on exchangeability, and its selection heuristic is not an optimization of coverage error.
3.1 Setup
- The setup defines a missingness mask, a fixed mortality predictor, and a nonconformity score used to form prediction sets that may be empty, single-label, or two-label.The predictor, preprocessing, and grouping rule are independent of the calibration sample for the stated theoretical results.Source: §3.1 · p. 3
- Pooled split conformal uses the target-coverage order statistic, while Mondrian calibration applies that rule separately to scores in each fixed group.An empty calibration group receives an infinite threshold and hence both labels.Source: §3.1 · p. 3
3.2 Selecting the partition without reusing calibration data
- The procedure selects an eligible measurement with the most variable missingness across selection sites, using a sample from sites disjoint from training and calibration.Eligibility requires nonconstant training availability and selection-sample missingness between 10% and 90%, with at least 100 records in each state.Source: §3.2 · p. 4
- Fixing the partition before observing calibration outcomes preserves the Mondrian group guarantee under within-group exchangeability, but that guarantee does not hold under between-site shift.The study therefore evaluates coverage empirically under site shift.Source: §3.2 · p. 4
- The selector is a heuristic targeting site-dependent availability, not an optimal partition; separate calibration removes between-group score differences but not a group’s shift across sites.Missingness frequencies identify neither source of error and do not rank candidate partitions by actual errors.Source: §3.2 · p. 4
4 Pooled versus within-site coverage
Pooled and within-site coverage are different targets because pooling uses group-specific site weights and aggregation can cancel errors across sites. The section formalizes these effects and shows that population-level method rankings can reverse without sampling noise.
Aggregation schemes
- Pooling weights each site by its share of the evaluated group, while the within-site gap averages site-level worst-group gaps.Common-weight schemes separate weighting from the order of aggregation.Source: §4 · pp. 4–5
Decomposition
- The weighting term reflects covariance between group shares and coverage errors, and vanishes when shares or errors are constant across sites.With equal site weights, it also includes the change from patient to equal weights.Source: §4 · p. 5
- Cancellation measures how opposite-signed site errors offset, while the final term captures sites having different worst groups.For each method, Aw ≤ Dw ≤ Bw, but differences between methods can have either sign.Source: §4 · p. 5
Conditions and scope
- Varying group proportions are necessary but not sufficient for a weighting-driven reversal; coverage errors and their association with those proportions also matter.The selector chooses measurements whose availability varies across sites but does not maximize the test-site covariance.Source: §4 · p. 5
Population examples
- Constructed two-hospital, two-group populations show opposite pooled and within-site rankings even without calibration-to-test shift.The examples establish possible mechanisms, not the cause of any particular clinical reversal.Source: §4 · p. 6
5 Clinical evaluation
The clinical evaluation compares conformal calibration rules across eICU hospitals and MIMIC-IV care units using three predictors. Missingness-aware calibration improves selected-group gaps against pooling, but its advantage does not extend uniformly to broader groups or individual hospitals.
5.1 Cohorts and study design
- The evaluation uses eICU records from 82 hospitals and MIMIC-IV patients from six care units within one hospital.It tests LR, XGBoost, and an MLP using first-day ICU information; assignments separate training, selection, calibration, and testing roles.Source: §5.1 · p. 6
5.2 Comparators and evaluation targets
- Seven calibration rules share predictions and differ in score grouping or weighting, while the selected gap and panel gap assess selected groups and a broader missingness audit.The audit panel covers both missingness states across many measurements, but not combinations of missing variables or demographic attributes.Source: §5.2 · pp. 6–7
5.3 Gains on the selected groups
- 0.34–3.13 percentage-point reductions, with a 1.87-point median, were observed in mean selected gap versus pooling across all six settings.Five of six paired site-bootstrap intervals exclude zero after adjustment; the exception is eICU XGBoost, where the rules produce different sets for only 1.03% of test patients on average.Source: Table 1 · p. 7, §5.3 · p. 7
5.4 Do the gains extend to other missingness groups?
- Risk terciles have smaller panel gaps than missingness calibration for every dataset and predictor, with larger average sets in the reported XGBoost examples.Binary risk groups also have smaller panel gaps in all six settings; label-only calibration does not reduce selected gaps relative to pooling.Source: §5.4 · pp. 7–8
5.5 Do the gains extend to individual hospitals?
- Within eICU hospitals, missingness-minus-pooled gap differences are −0.0045, +0.0023, and −0.0004 for LR, XGBoost, and MLP, respectively.All adjusted intervals include zero; Table 2 attributes the XGBoost reversal to weighting and most LR and MLP attenuation to cancellation.Source: Table 2 · p. 9, §5.5 · p. 9
5.6 Do the gains extend to outcome labels?
- 16.1% of patients who died were covered by eICU XGBoost missingness calibration despite overall coverage of 0.895.Joint risk-and-outcome calibration raises death coverage to 0.908 in eICU but returns both labels for 63.7% of patients.Source: §5.6 · p. 9
6 Discussion and conclusion
The discussion distinguishes the guarantee for selected missingness groups from evidence of reliability within hospitals. It cautions against inferring site-level coverage from pooled gains and identifies study limitations.
Conclusion
- Missingness-aware calibration improves pooled coverage on selected groups in the studied settings, but pooled improvement alone does not establish better within-hospital coverage.Risk grouping achieves smaller gaps on the broader panel, while missingness grouping retains a coverage guarantee under exchangeability that risk grouping generally lacks.Source: §6 · p. 10
- Multi-site evaluations should report audited groups, hospital weights, and aggregation order alongside outcome coverage and prediction-set composition.Source: §6 · p. 10
Limitations
- The study is retrospective, uses information unavailable at admission, and its coverage bounds assume independent calibration observations rather than within-hospital dependence.MIMIC-IV covers units of one institution, the selector is a heuristic, and the clinical reversal is not statistically significant.Source: §6 · p. 10
AI-use statement
The authors report that AI tools assisted with writing and grammar checks.
AI-use statement
- The authors state they are responsible for verifying the results, citations, and final manuscript.Source: AI-use statement · p. 10
Ethics statement
This retrospective study uses restricted clinical databases and does not evaluate clinical deployment or patient benefit.
Ethics statement
- Clinical database access requires credentialing and is subject to data-use restrictions.Individual patient data, identifiers, and predictions are not released.Source: Ethics statement · p. 10
- The study does not evaluate clinical deployment or patient benefit.Empty prediction sets would require a separate operational policy.Source: Ethics statement · p. 10
- Missingness may reflect care processes and unequal access, but calibrating its groups establishes neither demographic fairness nor grounds for withholding measurements.The authors claim no study-specific institutional approval or exemption.Source: Ethics statement · p. 10
Reproducibility statement
The paper defines its coverage quantities and evaluation assignments, while noting that restricted clinical records limit independent reproduction to authorized database users.
Reproducibility statement
- Independent reproduction requires authorized access because individual clinical records and predictions are withheld under the databases’ access restrictions.The appendices specify preprocessing, model settings, selection, weighting, uncertainty summaries, and controlled distributions.Source: Reproducibility statement · p. 10
A Terminology and notation
- Coverage means the prediction set contains the true outcome, and the selected gap is the largest absolute deviation from 0.9 across the two selected-measurement states.The notation also distinguishes group shortfall and excess, overall coverage, and survivor and death coverage.Source: A Terminology and notation · p. 14
- Sites are hospitals in eICU or care units in MIMIC-IV, and assignments divide them among predictor fitting, grouping selection, calibration, and testing.The 28 assignments and three predictors yield 84 fitted predictors, but assignments share sites and are not independent replications.Source: A Terminology and notation · p. 14
- LR is logistic regression, XGBoost is a gradient-boosted tree ensemble, and the MLP is a one-hidden-layer feedforward neural network.AUROC measures ranking of predicted risks between deaths and survivors.Source: A Terminology and notation · p. 14
B.1 Setting and result
The section bounds expected group coverage under pooled and groupwise calibration, then gives simultaneous realized-coverage intervals and their limits. It also distinguishes pooled from within-site aggregation through cancellation and worst-group switching.
B.1 Setting and result
- Pooled calibration’s group-coverage error is bounded by ηk + δk + 1/(n + 1) + a, combining pooling discrepancy, deployment shift, sample size, and score atoms.The bound assumes iid calibration examples and an independent deployment example.Source: §B.1 · pp. 14–15
- Groupwise calibration replaces the pooling term ηk with a larger sample-size term 1/(nk + 1), but neither rule controls deployment shift δk.Comparing the bounds does not rank actual coverage errors; they bound expected coverage rather than expected absolute error conditional on calibration data.Source: §B.1 · p. 15
- Separate calibration removes the pooling discrepancy but cannot stabilize a group’s score distribution under site shift; if that distribution shifts arbitrarily, δk can make the bound uninformative.When there is no deployment shift, δk = 0 and the usual expected-coverage bound remains.Source: §B.1 · p. 15, §B.2 · p. 16
- Score atoms account for ties, which can cause overcoverage; tree ensembles produce tied scores, so the bounds retain atom terms.Inclusive threshold comparisons preserve the lower bound.Source: §B.1 · pp. 14–15
B.3 Simultaneous bounds for realized group coverage
- Simultaneous intervals bound realized coverage across calibrated groups with probability at least 1 − β, using groupwise order-statistic quantiles and a union bound.The argument does not require independence across groups.Source: §B.3 · p. 16
- At target 0.9 and nk = 1000, realized-coverage standard deviation is about 0.0095 versus a 0.0010 bias term; the intervals do not cover arbitrary audit groups or within-hospital dependence.Unknown shift and atom terms can also make the intervals uninformative.Source: §B.3 · p. 16
C Pooled and within-site aggregation
- For fixed site weights, A averages signed errors before taking the largest absolute group error, B averages each site’s worst-group error, and D lies between them.B − A separates cancellation of signed errors from worst-group switching.Source: C Pooled and within-site aggregation · p. 16
C.1 Population example
A constructed population example shows that pooled and within-hospital evaluations can favor different calibration methods. The clinical decomposition attributes observed differences to weighting and cancellation, alongside implementation and uncertainty boundaries.
C.1 Population example
- Changing only hospital weights to w0 = 0.8 makes pooled aggregation favor pooling while within-hospital aggregation favors mask calibration.Without deployment change, mask calibration has exact coverage in both mask states, yet each hospital has a gap of 0.1; the example uses exact population quantities.Source: §C.1 · p. 17, Table 3 · p. 17
C.2 Clinical decomposition
- In the clinical decomposition, cancellation contributes +0.01135 for LR and +0.00631 for MLP, while weighting produces the XGBoost sign reversal.The table reports paired missingness-minus-pooling differences over ten eICU assignments; the contributions are arithmetic, not causal hospital processes.Source: §C.2 · p. 18, Table 4 · p. 18
- For XGBoost, replacing pooled weights with common patient weights moves the comparison from −0.00515 to +0.00192, and the covariance term reproduces the pooled signed error.The weighting term is not the difference between mean covariances because taking the maximum is nonlinear.Source: §C.2 · p. 18, Table 5 · p. 18
D Implementation details
- The selector chooses a measurement with cross-site variation in missingness from an independent selection sample, then fixes groups before calibration.Preprocessing derives group indicators from raw records before imputation; the described predictors use training-only model settings and selection rules.Source: D Implementation details · pp. 18–19
- Under a 200-epoch cap, the mean within-hospital MLP difference is +0.0024 rather than −0.0004, making the sign of this small difference sensitive to optimization.All reported MLP results use fits that met the stopping rule; that does not establish a global minimum.Source: D Implementation details · p. 19
D.1 Paired uncertainty for the headline improvement
- The pooled selected-gap improvement is estimated with paired site resampling, but the six MIMIC-IV units cannot support strong inference about a wider hospital population.The intervals are conditional descriptive approximations and do not refit predictors or regenerate calibration samples.Source: §D.1 · p. 20, Table 6 · p. 20, D Implementation details · p. 20
D.2 Sensitivity to evaluation-group size
Risk-tercile calibration is favored across all six comparisons when evaluation groups are size-filtered or errors are standardized. Additional diagnostics show that few prediction sets change, while panel-gap estimates remain descriptive and the conditional-conformal comparison tests a prespecified change on smaller samples.
D.2 Sensitivity to evaluation-group size
- All six mean comparisons favor risk terciles under both group-size cutoffs and on the standardized-error scale.The standardized measure is a descriptive scale check, not a corrected gap, significance test, or simultaneous confidence bound.Source: §D.2 · p. 20, Table 7 · p. 21
E Additional results
- The additional-results tables report coverage, set size, selected gap, and panel gap across calibration rules, including risk groups split at the selection-sample median.The score-tree and mask-weighting rows are stated controls, not implementations of other published missing-data methods.Source: E Additional results · pp. 22–23
E.1 Coverage diagnostics
- The maximum panel gap is an empirical diagnostic, not a precise estimate of the largest population error, because selecting among overlapping groups adds uncertainty.The analyses provide neither patient-level iid confidence intervals for clustered data nor a claim of simultaneous empirical validity.Source: §E.1 · p. 23
- For eICU XGBoost, only 1.03% of patients changed prediction sets on average, although group-specific coverage could differ.The range across assignments was 0.01–3.91%; it describes variation, not uncertainty.Source: §E.1 · p. 23, Table 9 · p. 23
- The direction table reports selected-state shortfall and excess alongside panel gap, while set-type proportions describe prediction behavior rather than a clinical action policy.The conditional-conformal comparison tests one prespecified basis extension on identical 2,000-patient samples, not an optimized function class or full-test-sample result.Source: §E.1 · pp. 23–25, Table 12 · p. 25
F Outcome coverage and score transformations
The section examines outcome-specific coverage and probability-scale sensitivity, then explores learned thresholds beyond risk. It reports low death coverage despite near-target overall coverage, a joint-calibration trade-off, and limited evidence that adding missingness improves thresholds beyond predicted risk.
F Outcome coverage and score transformations
- An absolute coverage gap can reflect overcoverage rather than improved shortfall: eICU XGBoost has the same mean selected shortfall, 0.0169, under missingness and pooled calibration.Mean excess coverage is 0.0092 for missingness calibration and 0.0159 for pooled calibration.Source: F Outcome coverage and score… · p. 26
- At α = 0.1, eICU XGBoost death coverage is 0.161 while overall coverage is 0.895 and survivor coverage is 0.970.Across α = 0.05, 0.1, and 0.2, death coverage is 0.470, 0.161, and 0.044, respectively.Source: F Outcome coverage and score… · p. 26
- Joint calibration raises XGBoost low-risk death coverage from 0.000 to 0.916 in eICU and from 0.015 to 0.892 in MIMIC-IV.It returns both labels for 63.7% of eICU and 61.1% of MIMIC-IV patients on average, and reduces panel shortfall relative to label-only but not risk-only calibration.Source: F Outcome coverage and score… · p. 28
- After an odds multiplier of 1/4, 1, or 4, eICU XGBoost death coverage is 0.024, 0.161, or 0.593 while overall coverage stays within 0.894–0.896.The odds shifts preserve risk ranking and AUROC but can change the joint ordering of survivor and death scores.Source: F Outcome coverage and score… · pp. 27–28, Figure 2 · p. 29
G Exploratory calibration beyond risk
- The exploratory threshold learners estimate the 0.9 score quantile from risk alone or risk plus measurement masks, then validate a weighted combination using site-mean pinball loss.Residual calibration requires exchangeability of calibration and test residuals; it does not establish group or hospital validity under the studied shifts.Source: G Exploratory calibration beyond risk · p. 32
- Validation selects no mask correction in 21 of 28 assignments, and in eICU the full-sample mask learners make the same sets as the risk learner.In MIMIC-IV, nine of 18 learners use mask splits; the mean fraction of patients with different sets is 0.0169.Source: Table 21 · p. 32, G Exploratory calibration beyond risk · p. 33
- The binary 0.9 score quantile changes with mask only when conditional mortality crosses 0.1 or 0.9, so additional outcome information can leave it unchanged.Even threshold changes need not cross either candidate score and alter the resulting sets.Source: G Exploratory calibration beyond risk · p. 33
- In controlled examples, the validation rule selected the mask learner in all 20 quantile-changing repetitions and the risk learner in all 40 others.The changing case had mean population coverage 0.99 and mean size 1.5, versus coverage 1 and size 2 for risk alone; this tests a specific signal, not general power.Source: G Exploratory calibration beyond risk · p. 33