PaperlayerView exact revision on arXiv

Source-linked AI summary

Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage

Ying Jin, Zhimei Ren

arXiv:2403.03868v3stat.MEmath.STstat.MLstat.TH

TL;DR

Data-driven selection can make marginal conformal coverage insufficient for the test units practitioners care about. JOMI constructs prediction sets with selection-conditional coverage and efficient implementations for several selection rules. In the reported applications, vanilla conformal prediction can miscover selected units, while JOMI achieves valid coverage.

  • Problem

    Practitioners often select specific test units for downstream decisions, but marginal validity does not ensure the promised coverage for those units.

    Source: §1 · pp. 1–2

  • Method

    JOMI provides selection-conditional prediction sets for arbitrary selection rules invariant to calibration-unit permutations, with efficient instantiations for several practical rules.

    Source: §1.2 · p. 3, §4.2 · p. 10, §4.3 · p. 11

  • Results

    Across the reported scenarios, vanilla conformal prediction can miscover selected units, whereas JOMI achieves valid selection-conditional coverage.

    Source: §5.1 · p. 13, §5.2 · p. 14, §6.2 · p. 17

  • Takeaways & Limitations

    Practitioners can retain their selection rule while obtaining coverage tailored to selected units, without modeling assumptions under the stated exchangeability condition.

    Source: §1 · p. 2, §1.2 · p. 3

    In constrained optimization experiments, FCR can be much lower than selection-conditional coverage because it is zero when the selection set is empty.

    Source: §5.3 · p. 15

Abstract

from arXiv · show

Conformal prediction builds marginally valid prediction intervals that cover the unknown outcome of a randomly drawn test point with a prescribed probability. However, in practice, data-driven methods are often used to identify specific test unit(s) of interest, requiring uncertainty quantification tailored to these focal units. In such cases, marginally valid conformal prediction intervals may fail to provide valid coverage for the focal unit(s) due to selection bias. This paper presents a general framework for constructing a prediction set with finite-sample exact coverage, conditional on the unit being selected by a given procedure. The general form of our method accommodates arbitrary selection rules that are invariant to the permutation of the calibration units, and generalizes Mondrian Conformal Prediction to multiple test units and non-equivariant classifiers. We also work out computationally efficient implementation of our framework for a number of realistic selection rules, including top-K selection, optimization-based selection, selection based on conformal p-values, and selection based on properties of preliminary conformal prediction sets. The performance of our methods is demonstrated via applications in drug discovery and health risk prediction.

1 Introduction

Conformal prediction provides marginal coverage, but data-driven selection can leave the units of interest without the promised coverage. The paper introduces JOMI, which uses a reference set to support selection-conditional prediction and develops implementations for several practical selection rules.

1 Introduction

  • Marginal coverage may not hold for selected units, even when practitioners target them for decisions in drug discovery, diagnosis, and healthcare management.The paper motivates coverage conditional on a unit being of interest.Source: §1 · pp. 1–2

1.1 Exchangeability via reference sets

  • Selection breaks the exchangeability between calibration data and a test point, complicating coverage guarantees for data-dependent selection events.The distributions of nonconformity scores conditional on selection can be complex.Source: §1.1 · p. 2
  • JOMI constructs a reference set of calibration points that remain exchangeable with the test point conditional on selection, enabling calibrated uncertainty quantification for selected units.Its framework accommodates selection rules invariant to permutations of the calibration data.Source: §1.1 · p. 2, Figure 1 · p. 3

1.2 Preview of contributions

  • Under exchangeability, JOMI provides selection-conditional coverage for arbitrary calibration-permutation-invariant selection rules, without modeling assumptions.The conditioning event can be as granular as selecting a unit and a specified number of units.Source: §1.2 · p. 3
  • The paper develops efficient implementations for covariate-dependent, conformal p-value-based, and preliminary-prediction-set selection, and demonstrates applications in drug discovery and health risk prediction.For finite outcome spaces, the stated worst-case complexity is O(|Y|mn) times that of the selection rule.Source: §1.2 · p. 3

2 Problem setup

The setup defines coverage conditional on selection, with a taxonomy allowing the conditioning event to be specified at different levels of detail. It distinguishes this guarantee from false coverage rate control, which can conceal poor coverage in particular selection events or units.

2.1 Selection-conditional coverage

  • Selection-conditional coverage evaluates whether a test unit’s prediction set covers its outcome given that the unit is selected and the selection set belongs to a specified taxonomy.Taxonomies can condition on selection size or on a particular selected set.Source: §2.1 · p. 4

2.2 Relations among notions of selective coverage

  • Strong selection-conditional coverage across a partition of selection events implies weak coverage conditional only on a unit being selected.The weak guarantee need not imply FCR control in general.Source: §2.2 · pp. 4–5
  • Strong selection-conditional coverage over disjoint taxonomies with fixed selection sizes implies false coverage rate control for a corresponding choice of taxonomy.The resulting FCR is bounded by α times the probability of a nonempty selection, and hence by α.Source: §2.2 · p. 5
  • When selection size varies, conditioning on its size may shrink the reference set and potentially produce wider or unstable prediction intervals.The paper notes this concern especially when selection is highly variable.Source: §2.2 · p. 5
  • FCR averages over selection events, including empty selections, and over selected units, so it can mask poor coverage for nonempty selections or particular units.For a single test unit, marginal validity can give FCR control without resolving the selection issue.Source: §2.2 · p. 5

3 JOMI: a unified framework

JOMI restores selection-conditional coverage by calibrating against reference sets that remain exchangeable with a selected test point. Its guarantees rely on exchangeability and a selection rule invariant to calibration-unit permutations.

3.1 Warm-up: split conformal prediction

  • Split conformal prediction calibrates hypothetical outcomes against calibration scores and achieves finite-sample coverage under exchangeability, but selection can break the needed exchangeability.Conditional on selection, calibration points and the test point are no longer exchangeable.Source: §3.1 · p. 6

3.2 Conformal inference via a reference set

  • JOMI uses calibration points that, after swapping with a hypothesized test point, preserve the same selection event to form a reference set.The reference set supplies calibration scores for the selected unit.Source: §3.2 · pp. 6–7
  • With continuous nonconformity scores and a reference set of moderate size, the basic prediction set has near-exact coverage; added randomness yields exact coverage.The random variables are independent uniform draws.Source: §3.2 · p. 7

3.3 Theoretical guarantees

  • Under calibration-permutation-invariant selection and the stated exchangeability condition, Theorem 1 confirms conditional validity of the JOMI prediction sets.The randomized construction achieves validity through exchangeability conditional on the selection event.Source: §3.3 · pp. 7–8

4 Computationally tractable instances

Direct construction can be costly when the outcome space is large, so the paper derives efficient implementations for structured selection rules. These cover covariate-based, conformal-p-value, and preliminary-prediction-set selection.

4 Computationally tractable instances

  • For finite outcome spaces, direct implementation costs at most O(|Y|mn) times selection-rule complexity; special structures enable efficient computation beyond small |Y|.The section develops implementations for covariate-only, conformal-p-value, and prediction-set-based selection.Source: §4 · p. 8

4.1 Covariate-dependent selection rules

  • Covariate-only rules include top-K, quantile, and calibration-label-free optimization; JOMI costs O(mn·CS), with O(max{m, n}) reference-set shortcuts for rules (1)–(3).Optimization-based selection can include black-box resource allocation, provided it does not use calibration labels.Source: §4.1 · pp. 8–9

4.2 Selection based on conformal p-values

  • For conformal-p-value selection, JOMI handles shared calibration data and data-dependent thresholds with exact coverage, while the p-values control type-I error for detecting large outcomes.The framework covers threshold rules including fixed and Benjamini-Hochberg thresholds.Source: §4.2 · p. 10

4.3 Selection based on conformal prediction sets

  • For preliminary-prediction-set selection, JOMI's selective set is a superset of its general prediction set and has tight empirical coverage, but may not preserve preliminary-set constraints.The authors describe this constraint mismatch as desired when selection is fixed and inference is decoupled from selection.Source: §4.3 · p. 12

5 Application to drug discovery

The paper evaluates JOMI in drug-property prediction across top-K, conformal-selection, and cost-constrained scenarios, with additional drug-target-interaction results deferred to the supplement. Across these scenarios, JOMI provides valid or near-exact selection-conditional coverage, while vanilla conformal prediction can be miscalibrated for selected units.

5 Application to drug discovery

  • Across the three main-text scenarios, JOMI methods provide valid or near-exact selection-conditional coverage, while vanilla conformal prediction can be miscalibrated and BY is often conservative.The application section reports both shorter intervals in some cases where vanilla CP is under-confident and interval inflation to attain coverage in others.Source: §5 · p. 12, §5.1 · p. 13, §5.2 · pp. 14–15
  • The study applies JOMI to drug-property classification and considers top-K, conformal-selection, and cost-constrained selection; drug-target-interaction regression results are deferred to the supplement.The drug-property task predicts whether a candidate binds a specified disease target from encoded chemical structure.Source: §5 · p. 12, §5.3 · p. 15

5.1 Top-K selection

  • With APS scores, vanilla CP is over-confident for bottom-ranked and under-confident for top-ranked units, while JOMI_rand removes JOMI’s discretization gap; at α = 0.8, bottom-ranked units can receive empty vanilla sets.With the binary score, JOMI remains valid and vanilla CP’s top-versus-bottom miscalibration reverses.Source: §5.1 · pp. 13–14

5.2 Conformal selection

  • JOMI and JOMI_rand maintain valid coverage across conformal-selection scenarios; empty selections can lower FCR, while JOMI usually yields smaller sets than Bao et al.’s heuristic and BY is conservative.JOMI with APS can retain a discretization gap; randomized JOMI and the binary-score results do not show that gap.Source: §5.2 · pp. 14–15

5.3 Selection with constraints

  • Under cost constraints, JOMI methods yield near-exact coverage, whereas vanilla CP’s calibration depends on the score; reference-set size is positively correlated with the number selected.BY remains overly conservative in this scenario.Source: §5.3 · p. 15
  • In constrained optimization, FCR can be zero when no units are selected, so controlled FCR need not ensure anticipated coverage when the selected set is nonempty.The paper notes that vanilla CP’s empirical FCR can be much lower than its selection-conditional coverage in this task.Source: §5.3 · p. 15

6 Application to health risk prediction

The health-risk application tests JOMI on ICU-stay prediction under three clinical selection rules. Across these settings, JOMI achieves exact selection-conditional coverage, while preliminary-set selection can under-cover and JOMI’s sets can grow or become segmented.

6 Application to health risk prediction

  • The health-risk study considers budget-constrained allocation, selection by short preliminary prediction sets, and selection by preliminary upper bounds below a threshold.The application uses ICU-stay length as the outcome and evaluates the rules with MIMIC-IV data.Source: §6 · pp. 15–16

6.1 Selection with constraints

  • JOMI achieves exact coverage for selected patients under budget-constrained ICU allocation, whereas vanilla conformal prediction is over-confident and BY is overly conservative.The experiment uses MIMIC-IV ICU-stay data and an approximate solver for the NP-hard budgeted selection problem; JOMI validity does not require an exact optimization solution.Source: §6.1 · p. 16, §6 · p. 16

6.2 Selecting small-sized prediction sets

  • Selecting patients with preliminary intervals of at most 5 days can under-cover; JOMI restores exact coverage by inflating its prediction sets.The evaluated JOMI superset is almost always a single interval.Source: §6.2 · p. 17, Figure 7 · p. 17

6.3 Selecting prediction sets below a threshold

  • Selecting patients whose preliminary interval upper bound is below 6 days can under-cover at small α; JOMI achieves exact coverage but may return multiple segments at large α.The comparison averages results across 500 independent runs.Source: §6.3 · p. 17, Figure 8 · p. 17

A.1 Literature overview

  • JOMI targets coverage conditional on selection, unlike marginal conformal coverage, and its predictive-inference use of exchangeability distinguishes it from post-selection inference for parameters.The framework decouples selection from uncertainty quantification and permits selection rules depending jointly on all test units.Source: §A.1 · p. 21

A.2 Connection to Mondrian Conformal Prediction

  • When selection is permutation-equivariant and uses a binary classifier, JOMI reduces to Mondrian conformal prediction; more generally, it handles asymmetric rules and multiple test points.JOMI constructs exchangeable subgroups by jointly considering calibration and test points.Source: §A.2 · p. 22

A.3 Comparison to the Benjamini-Yekutieli (BY) correction

  • BY empirically provides selection-conditional coverage but can be overly conservative, producing uninformative binary sets or intervals twice as long as JOMI’s.BY adjusts the nominal level using only the selection-set size, while JOMI uses a reference set to target uncertainty to the selection.Source: §A.3 · p. 22

B Additional results on covariate-dependent selection

These supplementary results give tailored reference-set constructions for covariate-dependent selection and report additional drug-property prediction experiments. For top-K and quantile rules, the constructions can simplify reference-set computation.

B.1 Top-K selection

  • Top-K selection uses a common reference set for selected units, reducing reference-set construction to O(max(m, n)).The selected units’ scores are assumed distinct almost surely; the common reference set is independent of the selection taxonomy.Source: §B.1 · p. 23

B.2 Selection based on joint quantiles

  • For selection above a joint calibration-and-test score quantile, a shared reference set reduces construction complexity to O(max(m, n)).The quantile threshold is invariant to permutations of calibration and test scores.Source: §B.2 · p. 23

B.2.1 Selection based on calibration quantiles

  • For selection above a calibration-score quantile, the reference set consists of calibration units above that quantile and can be computed in O(max(m, n)).The same reference set applies to each selected test unit and matches the general construction for any specified selection taxonomy.Source: §B.2.1 · p. 23

C Additional experiment results

  • The supplement reports additional drug-property prediction results for APS and binary scores, including several selection rules and drug-target interaction settings.The listed experiments include APS results for top-K selection in calibration and test data and calibration-referenced selection.Source: C Additional experiment results · p. 24, §C.1 · p. 24

C.1.1 Top-K selection in mixed sample

In drug-property prediction, JOMI and its randomized version attain valid selection-conditional coverage under both mixed-sample top-K and calibration-referenced selection. Vanilla conformal prediction can miss the target coverage for selected units.

C.1.1 Top-K selection in mixed sample

  • Under mixed-sample top-K selection, JOMI and JOMI_rand achieve valid selection-conditional coverage, while vanilla conformal prediction has a large coverage gap.Randomization eliminates the discretization-related coverage gap and reduces prediction-set sizes; at α = 0.8, vanilla conformal prediction can use empty sets for selected units while attaining marginal coverage.Source: §C.1.1 · p. 24, Figure 9 · p. 24

C.1.2 Calibration-referenced selection

  • With calibration-referenced selection, JOMI’s coverage is valid and exact with randomization, while vanilla conformal prediction fails for units selected by predicted affinity.The selected test units’ predicted affinities are compared against the K-th highest or lowest calibration prediction.Source: §C.1.2 · pp. 24–25, Figure 10 · p. 25

C.2 Additional results for binary score in drug property prediction

With the binary score, JOMI’s methods achieve valid selection-conditional coverage across the reported drug-property prediction configurations, while vanilla conformal prediction can be over- or under-confident.

C.2 Additional results for binary score in drug property prediction

  • Across selection rules (i)–(iii), JOMI’s methods achieve valid coverage with the binary score, and nonrandomized JOMI has a smaller coverage gap than with APS.Figures 11–13 report selection-conditional coverage and prediction-set sizes for the binary score.Source: §C.2 · p. 25, Figure 11 · p. 25

C.3 Additional results for empirical FCR in drug property prediction

Across drug-property and drug-target-interaction experiments, JOMI provides coverage tailored to selected units, including under heterogeneous thresholds and budget-constrained selection. Empirical FCR can be misleading when selections are empty, while stable, nonempty selection sets support tight FCR control.

C.3 Additional results for empirical FCR in drug property prediction

  • For fixed-size top-K tasks, JOMI’s empirical FCR is consistent with control implied by selection-conditional coverage; FCR remains near nominal under conformal selection with stable set size.The experiments use APS and binary conformity scores for top-K tasks at α = 0.1.Source: §C.3 · p. 25, Figure 14 · p. 27, Figure 15 · p. 27
  • Under constrained optimization, JOMI’s empirical FCR is close to nominal, but vanilla CP’s lower FCR can reflect empty selections and mask poor coverage when selections are nonempty.This illustrates why FCR can be an overly optimistic error metric.Source: §C.3 · pp. 25, 27, Figure 16 · p. 27

C.4.1 Conformalized selection for heterogeneous cn+j’s

  • With heterogeneous conformal-selection thresholds, JOMI achieves near-exact coverage across q and α, while vanilla conformal prediction has exceedingly high miscoverage.Two-segment prediction sets occur at very low frequency across the experiments.Source: Figure 17 · p. 28, §C.4.1 · p. 28, Figure 18 · p. 28

C.4.2 Selection under budget constraint

  • JOMI yields exact coverage for focal drug-target pairs under budget-constrained selection, while vanilla conformal prediction has much higher miscoverage.The selected-unit count is relatively stable, but the reference-set size may vary.Source: Figure 19 · p. 29, §C.4.2 · p. 29
  • When a drug-target selection set may be empty, JOMI’s empirical FCR can be lower than its selection-conditional coverage.Related empirical FCR results are reported for two tasks.Source: §C.4.2 · p. 29

C.5 Additional results for drug-target-interaction prediction

  • Additional drug-target-interaction experiments cover covariate-dependent top-K selection among test units, mixed calibration and test samples, and calibration-referenced thresholds.The three selection-rule variants are identified as (i), (ii), and (iii).Source: §C.5 · p. 29

C.5.1 Top-K selection for fixed K

Across three top-K selection settings, vanilla conformal prediction can be miscalibrated for focal units, whereas JOMI’s methods provide near-exact or valid coverage. BY is conservative, and its intervals do not adapt to whether the selected units have high or low predicted affinities.

C.5.1 Top-K selection for fixed K

  • BY is overly conservative and returns the same interval width for highest- and lowest-ranked selections, whereas JOMI adapts uncertainty quantification to the selection.BY depends on the selection-set size, not the nature of the ranking.Source: §C.5.1 · p. 30
  • JOMI methods achieve near-exact coverage across K and α, while vanilla conformal prediction is overconfident for highest-affinity pairs and underconfident for lowest-affinity pairs.In this setting, higher-predicted-affinity pairs have greater intrinsic uncertainty and need longer intervals for a given coverage level.Source: §C.5.1 · pp. 29–30, Figure 20 · p. 30

C.5.2 Top-K selection in mixed sample

  • For top-K units selected across calibration and test data, vanilla CP is miscalibrated and BY is conservative, while JOMI methods yield valid coverage.Vanilla CP is overconfident for highest-prediction units and underconfident for lowest-prediction units.Source: Figure 21 · p. 30, §C.5.2 · p. 30

C.5.3 Calibration-referenced selection

  • For calibration-referenced selection, vanilla CP is not calibrated, BY is overly conservative, and JOMI methods achieve near-exact coverage across K.Selection is based on predicted affinities above or below calibration-data thresholds.Source: §C.5.3 · pp. 30–31, Figure 22 · p. 31

C.6 Additional results for empirical FCR in drug target interaction

In drug-target-interaction experiments, empirical FCR is reported for conformalized selection and constrained optimization. The observed relation between FCR and selection-conditional coverage depends in part on whether selections can be empty.

C.6 Additional results for empirical FCR in drug target interaction

  • Conformalized selection produces empirical FCR below selection-conditional coverage, illustrating FCR’s limitations as an error metric.The passage relates this pattern to the possibility that the selection set is empty.Source: Figure 23 · p. 31, §C.6 · p. 31
  • Under constrained optimization, empirical FCR remains tightly below nominal in these experiments, where the selection set is always nonempty.Source: Figure 24 · p. 31, §C.6 · p. 31

C.7 Additional results for empirical FCR in health risk prediction

In health-risk prediction, stable, nonempty selections coincide with tightly controlled empirical FCR. The accompanying proofs relate strong selection-conditional coverage to weaker coverage and FCR control under specified disjoint-taxonomy conditions.

C.7 Additional results for empirical FCR in health risk prediction

  • Health-risk prediction experiments show tightly controlled empirical FCR when selection-set sizes are stable and nonempty.Figure 25 also reports preliminary prediction-interval length and lower bound across nominal miscoverage levels.Source: Figure 25 · p. 32, §C.7 · p. 32

D.1 Proof of Proposition 1

  • A disjoint partition of selection events lets strong selection-conditional coverage imply weak selection-conditional coverage.The proof uses the disjoint sets’ union covering the full set.Source: §D.1 · p. 32

D.2 Proof of Proposition 2

  • For a disjoint taxonomy partition whose members fix selection-set size, strong selection-conditional coverage supports the FCR-control proof.The proof uses each taxonomy member’s fixed cardinality.Source: §D.2 · pp. 32–33

D.3 Proof of Proposition 3

These proofs establish the stated coverage, FCR, and conformal p-value validity claims using exchangeability and swap-based arguments. They also characterize p-value selection rules and describe conditions under which calibration splitting preserves selection-conditional validity.

D.3 Proof of Proposition 3

  • The two-unit construction has weak selection-conditional coverage 1−α for each unit but FCR 2α/(1+α)>α.It therefore violates FCR control despite satisfying weak selection-conditional coverage.Source: §D.3 · p. 33

D.4 Proof of Theorem 1

  • Conditional on the unordered realized data, exchangeability makes assignments equally likely, while the selected indices and scores are fixed, supporting the proof of Theorem 1’s two statements.The proof conditions on the unordered data and uses the resulting uniform assignments to establish the theorem’s statements.Source: §D.4 · pp. 34–36

D.5 Proof of Proposition 4

  • The Proposition 4 proof reduces the claim for a selected unit to separate cases y≤c_n+j and y>c_n+j.For each case, it compares the swap-based selected-index set with the conformal set.Source: §D.5 · p. 36

D.5.1 Case 1: y ≤c_n+j.

  • For y≤c_n+j, the proof establishes both set inclusions and thus equality between the swap-based selected-index set and the conformal set.It obtains the inclusions by comparing stopping times under swaps, considering whether Y_i≤c_i or Y_i>c_i.Source: §D.5 · p. 36, §D.5.1 · pp. 37, 39

D.5.2 Case 2: y > c_n+j.

  • The supplied proof establishes τ_swap(i,j)(y) = τ^(j)(0,0): it shows τ^(j)(0,0) ≤ τ_swap(i,j)(y) and τ_swap(i,j)(y) ≤ τ^(j)(0,0), then combines them.Source: §D.5.2 · p. 39

D.5.2 Case 2: y > cn+j.

  • For y > c_n+j, the proof establishes the reverse inclusion by showing the relevant stopping times coincide, completing the set inclusion argument.The proof obtains matching stopping times for the (0,1) and (0,0) cases before concluding the inclusion.Source: §D.5.2 · pp. 40–41

D.6 Proof of Proposition 5

  • Proposition 5 is proved by splitting candidate y values into below, above, and within the η− to η+ range, with the middle case immediate.For values below η− and above η+, the proof uses separate expressions for the reference set based on whether Si falls below or above η− or η.Source: §D.6 · pp. 41–42

D.7 Proof of Proposition 6

  • For top-K selection, the proof shows the proxy and original stopping times coincide when the test score exceeds the top-K threshold, yielding equality of the resulting sets.It then establishes both inclusions between the fixed and test-unit reference sets.Source: §D.7 · p. 42

D.8 Proof of Proposition 7

  • For conformal selection, the proxy stopping time equals the original when the test score exceeds it, and the proof uses this to establish equality of the reference sets.The argument establishes corresponding stopping-time relations after swapping units and then proves both set inclusions.Source: §D.8 · p. 43

E.1 Connection between selection-conditional coverage and FCR control

  • Under the stated asymptotic conditions, FCR coincides with selection-conditional miscoverage.The selection rule must be asymptotically close to a threshold function, and the prediction set must have the specified conformal form.Source: §E.1 · pp. 43, 45

E.2 The conformal p-values

  • Under Theorem 1’s conditions, the conformal p-values and their randomized counterparts satisfy the stated validity property; added randomness can produce sharper p-values.The randomized construction uses independent uniform random variables.Source: §E.2 · p. 45

E.3 Additional results on the conformal p-value-based procedures

  • Fixed-threshold and BH selection on conformal p-values are each equivalent to thresholding corresponding transformed scores.The fixed-threshold rule uses T, while the BH rule uses T^BH.Source: §E.3 · pp. 46–47

E.4 Prediction set construction by further sample splitting

  • Further calibration splitting usually does not help without changing selection, but it yields valid selection-conditional sets for rules based only on calibration data and focal features.Examples include fixed-threshold conformal p-values and selection based on preliminary prediction sets; the selection rule is applied using one calibration split.Source: §E.4 · pp. 47–48

E.5 Analysis of calibration sample size

The analysis characterizes how selection affects the available calibration sample size. For threshold-based reference sets with a convergent threshold and continuous tail probability, their size converges to the corresponding population tail probability.

E.5 Analysis of calibration sample size

  • Accounting for selection can reduce calibration data and affect prediction-set size and stability, especially when selecting only the test unit with the largest predicted value.That case amounts to inference in a distributional tail, where only a limited number of calibration samples may be available.Source: §E.5 · p. 48
  • For threshold-based reference sets, the calibration fraction converges almost surely to G(q*) when the threshold converges to q* and G is continuous.Here G(t) = P(h(X) ≥ t), for a fixed feature function h.Source: §E.5 · p. 48
  • The analyzed class includes quantile-based and conformal p-value-based selection rules, including fixed-threshold and BH-threshold variants.Their thresholds converge to a population quantile or a common constant, respectively.Source: §E.5 · p. 48
Loading 2403.03868v3…