PaperlayerView exact revision on arXiv

Source-linked AI summary

Occupancy-based Quantile Risk Control

Zihao Shi, Huajun Xi, Bingyi Jing, Hongxin Wei

arXiv:2609.03104v1stat.MLcs.LG

TL;DR

Quantile risk control broadens conformal risk control beyond expected loss, but existing approaches can be conservative or lack finite-sample guarantees. OQRC partitions losses into occupancy bins to construct tight finite-sample bounds. Across evaluations it narrows the risk gap, including a reported 78.64% reduction on MS COCO under VaR-Interval.

  • Problem

    Quantile Risk Control covers quantile-based risks, but existing methods can be overly conservative or lack rigorous finite-sample guarantees.

    Source: §1 · p. 1

  • Method

    OQRC partitions the loss space using ordered calibration losses, estimates test-loss occupancy in the resulting intervals, and bounds risk by interval maxima to select λ.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across extensive evaluations OQRC significantly narrows RiskGap versus alternatives; on MS COCO under VaR-Interval, RiskGap is reduced by 78.64%.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    OQRC also addresses unconditional quantile risk control, enabling users to control prediction risk without configuring δ.

    Source: §7 · p. 9

    The method assumes i.i.d. data, while distribution shifts remain an uninvestigated setting.

    Source: §7 · p. 9

Abstract

from arXiv · show

Conformal risk control is an emerging framework for the safe deployment of machine learning models with finite-sample guarantees. To accommodate a broader class of risk notions, quantile risk control extends this framework to quantile-based risk measures. However, existing methods either suffer from excessive conservatism or lack rigorous finite-sample guarantees. To address these limitations, we introduce Occupancy-based Quantile Risk Control (OQRC), a novel method that provides tight risk control bounds with finite-sample validity. Our key idea is to formulate risk control as a finite-occupancy problem by partitioning the loss space with the ordered calibration losses. Specifically, we estimate the distribution of test losses across the resulting bins and upper-bound the risk by the maximum loss attained within each bin. We then select the parameter $\lambda$ such that this upper bound does not exceed a predefined threshold $\alpha$ with high probability $1-\delta$. Theoretically, we establish a finite-sample guarantee showing that OQRC yields tight risk control bounds that converge to the optimal bounds at a provable rate of $\mathcal{O}_ p(n^{-1/2})$. Extensive experiments demonstrate the effectiveness of our method, reducing the risk gap by up to 78.64\% on common benchmarks.

1 Introduction

Existing quantile risk-control methods can be conservative or lack finite-sample guarantees. OQRC uses a finite-occupancy formulation to provide tight finite-sample bounds, with experiments reporting smaller risk gaps than prior methods.

1 Introduction

  • Quantile Risk Control extends expected-loss control to quantile risks, but can overestimate prediction risk; an L-statistics approach offers tight bounds only asymptotically.The paper motivates a method combining tight bounds with finite-sample guarantees.Source: §1 · p. 1
  • OQRC partitions the loss space using ordered calibration losses, estimates test-loss occupancy in each interval, and bounds risk using each interval’s maximum loss.It selects λ so the upper bound stays below α with probability at least 1 − δ; the bounds converge to the optimal bounds at Op(n−1/2).Source: §1 · p. 2
  • 78.64%: OQRC reduces RiskGap on multi-label image classification using false-negative proportion as the loss, compared with alternative approaches.Experiments span seven real-world datasets, three tasks, and VaR, CVaR, and VaR-Interval risk measures.Source: §1 · p. 2
  • The authors report robust performance across hyperparameter configurations and model architectures, and applicability to unconditional risk control.The contribution summary also reports finite-sample quantile risk control and convergence of bounds to the optimum.Source: §1 · p. 2

2 Preliminaries

The setup defines prediction losses and the goal of selecting a model parameter that controls a risk measure on the loss distribution. The section introduces quantile-based risk measures, calibration assumptions, and the trade-off between valid control and informative predictions.

2 Preliminaries

  • The setup treats λ as a tunable model parameter and defines risk control as choosing it so a risk measure on the loss distribution meets the target α at confidence 1 − δ.Calibration losses are held out to select λ; the calibration and test losses are assumed i.i.d. and non-increasing and left-continuous across λ.Source: §2 · p. 3
  • A valid risk need not be minimized: selecting every pixel can minimize missed-tumor risk while making the prediction uninformative.Once the risk-control objective is met, a risk closer to α is preferred because it yields predictions with more useful information.Source: §2 · p. 3
  • Quantile-based risk measures weight the quantile function, with expected loss, β-VaR, β-CVaR, and VaR-Interval as examples.Expected loss averages across quantiles; β-VaR uses the loss at β, while β-CVaR and VaR-Interval average over specified quantile regions.Source: §2 · p. 3, Figure 2 · p. 3
  • Prior QRC uses a one-sided test statistic to construct a lower confidence bound for the loss distribution, but can overestimate prediction risk.The passage identifies this approach as Snell et al. [9].Source: §2 · p. 3
  • The L-statistics approach derives tighter bounds by adding a correction to empirical risk measures, but its guarantees are asymptotic.The passage identifies the approach as Chen et al. [12] and L-statistics as [13].Source: §2 · p. 3

3 Method

OQRC reformulates quantile risk control as a finite-occupancy problem, using calibration losses to partition the loss space and construct an upper confidence bound. It selects λ to control risk while establishing finite-sample validity and convergence of its bounds to the optimum.

3.1 Upper Confidence Bound via Partitioned Occupancy

  • OQRC partitions the loss space at ordered calibration losses, then uses the resulting bins to build an upper confidence bound on risk.Each test loss is assigned the maximum loss in its bin before the overall risk is computed.Source: §3.1 · p. 4
  • Under the stated assumption, the occupancy counts of test losses across calibration-induced bins are uniformly distributed over their possible count vectors.This distribution is independent of the actual loss values.Source: §3.1 · p. 4
  • The resulting upper-risk bound is valid at confidence level 1−δ for every λ.Computing the upper-risk CDF becomes demanding as m and n grow, motivating the Monte Carlo approximation.Source: §3.1 · p. 5

3.2 Monte-Carlo Approach for OQRC

  • Monte Carlo sampling approximates the upper-risk CDF, and its approximation converges uniformly to the target CDF.The sampler uses a stars-and-bars construction to draw occupancy vectors.Source: §3.2 · p. 5

3.3 Other theoretical results on OQRC

  • Theorem 3.5 gives a risk-level and confidence-level guarantee for the selected λ.Under the density bounds stated in Theorem 3.6, OQRC’s risk bounds converge to the optimal bounds at O_p(n^-1/2).Source: §3.3 · p. 5

4 Experiments

Experiments evaluate OQRC across three tasks and compare its coverage, RiskGap, and AvgSize with baseline methods. Results report tighter control with valid coverage, including under varying hyperparameters and model architectures.

4.1 Experimental setups

  • Experiments use three tasks, task-specific models, and methods; validity is assessed by coverage exceeding 1 − δ, while smaller RiskGap and AvgSize indicate tighter control when valid.The setup uses 200 calibration and 500 test examples, and averages results over 1,000 repetitions.Source: §4.1 · pp. 6–7

4.2 Main results

  • On MS COCO under VaR-Interval, OQRC reduces RiskGap by 78.64% and AvgSize by 38.80% versus Two-sided BJ while maintaining valid coverage.CDRC-L fails the 0.9 coverage requirement in all three tasks; OQRC's empirical coverage modestly exceeds 0.9.Source: §4.2 · p. 7
  • Across varying δ values, OQRC reduces RiskGap by 82.54% and AvgSize by 44.66% on average under the [0.85, 0.95]-VaR-Interval measure.The comparison uses MS COCO; Figure 3 compares RiskGap in its top panels and AvgSize in its bottom panels.Source: §4.2 · pp. 7–8, Figure 3 · p. 7
  • Across model architectures, OQRC lowers average RiskGap from 0.347 to 0.075 and AvgSize from 15.226 to 6.849 on MS COCO under VaR-Interval.Figure 4 compares performance across architectures; the results support the method's model-agnostic nature.Source: §4.2 · p. 8, Figure 4 · p. 8

5 Discussion

The discussion extends OQRC to unconditional quantile risk control and relates its framework to conformal risk control. It also describes the limiting case as the number of intervals grows without bound.

Unconditional quantile risk control

  • The guarantee takes expectation over the randomness of calibration and test losses.Unconditional control addresses Rψ(FX) ≤ α; prior training-free methods for unconditional QBRM control were lacking.Source: §5 · p. 8
  • OQRC's unconditional procedure achieves E[Rtest(λ*)] ≤ α, with empirical risk strictly below α across three tasks and varied target levels.Table 2 reports the difference between Risk Level and Empirical Risk; positive values indicate valid control.Source: §5 · p. 8, Table 2 · p. 9

Relation to conformal risk control

  • Setting m = 1 and using expected loss recovers the conformal risk control procedure as a special case of Theorem 5.1.Under this setting, the construction uses (n + 1)-dimensional one-hot vectors.Source: §5 · p. 8

The limiting case as m →∞

  • As m approaches infinity, OQRC controls risk over the entire loss distribution and yields unconditional quantile risk control Rψ(FX; λ) ≤ α.The paper refers to Appendix F for detailed analysis.Source: §5 · p. 9

6 Related Work

Related work covers distribution-free uncertainty quantification, quantile risk control, and training-based risk control. The paper positions OQRC as a post-hoc approach with finite-sample guarantees and theoretically tight control.

Distribution-free uncertainty quantification

  • Conformal prediction and its extensions provide distribution-free uncertainty quantification through prediction sets and risk control, with applications across language models, vision, and medicine.The paper proposes controlling risk notions beyond expected loss.Source: §6 · p. 9

Quantile risk control

  • Prior quantile risk control can be conservative [9], while tighter L-statistics bounds [12] rely on asymptotic guarantees rather than finite-sample validity.The paper states that OQRC is the first quantile risk control method to provide both finite-sample guarantees and theoretically tight control.Source: §6 · p. 9

Other works on risk control

  • Training-based approaches formulate risk as a learning objective, whereas OQRC is a post-hoc method that does not train the model and is computationally more efficient.Source: §6 · p. 9

7 Conclusion

OQRC is a training-free method for finite-sample quantile risk control, with bounds that converge to optimal bounds. The conclusion also identifies the i.i.d. assumption as a limitation and leaves distribution shifts for future work.

7 Conclusion

  • OQRC provides finite-sample quantile risk control, and its risk-control bounds converge to the optimal bounds.The conclusion describes the method as training-free.Source: §7 · p. 9
  • The approach also enables users to control prediction risk without configuring δ.Source: §7 · p. 9
  • The method is established under an i.i.d. assumption; distribution shifts are left for future work.Source: §7 · p. 9

A.1 Proof of Theorem 3.1

The proof counts arrangements of calibration and test-loss labels to analyze occupancy, then uses an ordering of augmented outputs to handle ties in the losses. It also draws on conformal p-values and relaxes the NoTies assumption.

A.1 Proof of Theorem 3.1

  • The proof adapts conformal p-value methodology to risk control and relaxes the earlier NoTies assumption.Source: §A.1 · p. 14
  • The proof counts label arrangements under uniformly distributed permutations, accounting for repeated labels when counting distinct sequences.Source: §A.1 · p. 14
  • A strict total order on augmented outputs lets the proof construct intervals and establish the result even when losses contain ties.Source: §A.1 · pp. 14–15

A.2 Proof of Theorem 3.2

These appendices establish the finite-sample upper bound, relate ordered samples to occupancy vectors, and compare the exact OQRC risk with optimal test risk. The proofs also state the connection to prior work and the regularity used in the comparison.

A.2 Proof of Theorem 3.2

  • Theorem 3.2 uses the uniform rank of occupancy upper risks to show that R+ is a valid (1 −δ) upper confidence bound for Rtest.Source: §A.2 · p. 15

A.3 Proof of Proposition 3.3

  • Proposition 3.3 maps uniformly distributed ordered indices bijectively to occupancy vectors using the stars-and-bars construction.Source: §A.3 · p. 15

A.4 Proof of Theorem 3.5

  • The proof of Theorem 3.5 is stated to be identical to the proof of Theorem A.1 in Bates et al. [18].Source: §A.4 · p. 15

A.5 Proof of Theorem 3.6

  • Theorem 3.6 compares OQRC and optimal test risk using a coupling of ordered occupancy-grid values with uniform order statistics and the Lipschitz property of quantile risk.The quantile risk is 1-Lipschitz with respect to the supremum norm of sorted losses.Source: §A.5 · pp. 16–17

A.6 Proof of Theorem 5.1

The proof derives an unconditional quantile risk guarantee by taking expectations of an occupancy-based upper bound. Selecting λ* under the stated condition yields unconditional quantile risk control.

A.6 Proof of Theorem 5.1

  • The proof takes expectation of the inequality bounding Rtest by Φ+ over the joint randomness of calibration and test losses.Source: §A.6 · p. 17
  • It rewrites the expectation using the occupancy vector induced by the test samples.Source: §A.6 · p. 17
  • Choosing λ* to satisfy the stated condition yields unconditional quantile risk control.Source: §A.6 · p. 17

B Other methods

The section contrasts quantile-risk control approaches, outlines OQRC’s Monte Carlo bound and calibration algorithms, and reports a condition on the tightness theory’s coverage behavior.

B.1 Methods in Quantile Risk Control

  • Quantile risk control methods vary by one-sided test statistic, including KS, Berk-Jones, OrderStats, and truncated Berk-Jones variants.The OrderStats and truncated Berk-Jones methods select indices using the quantile-risk parameters.Source: §B.1 · p. 18

B.2 Conformal Distortion Risk Control via L-statistics

  • CDRC-L uses L-statistics to estimate empirical risk uncertainty and add a correction term, but its guarantee requires calibration-set size to approach infinity.The resulting risk-control behavior is indeterminate for finite sample sizes.Source: §B.2 · p. 18

C Algorithms of OQRC

  • Algorithm 1 estimates an upper bound at confidence level 1 − δ by Monte Carlo sampling test-sample allocations across calibration-loss bins.Its inputs include calibration losses, test-set size, bounds, weighting function, and Monte Carlo replicates.Source: C Algorithms of OQRC · pp. 18–19
  • Algorithm 2 sorts candidate λ values and returns the first whose estimated risk bound is at most α, or FAIL if none qualifies.The calibration procedure uses Algorithm 1 to obtain the bound for each candidate.Source: C Algorithms of OQRC · p. 19

D Tightness of OQRC

  • OQRC’s tightness theory relies on loss functions with ideal properties; for non-continuous FNP lacking a density bounded below, coverage deviates from 1 − δ.On a synthetic dataset, coverage converges to 1 − δ under both risk measures as calibration size increases; tighter theory for more general losses is left to future work.Source: D Tightness of OQRC · p. 19

E Implementation details

The implementation uses a non-increasing false negative proportion loss, evaluates coverage and conservatism alongside prediction-set size, and reports consistent results across configurations. It also describes OQRC’s large-test-set regime.

E.1 Loss function details

  • False negative proportion (FNP) is the loss; analogous output sets use pixels for tumor segmentation and emotion labels for text recognition, and FNP is non-increasing in λ.The passage also introduces FNP for multi-label classification prediction sets.Source: §E.1 · pp. 19–20

E.2 Training details

  • Training uses PraNet-related task details for segmentation, four named architectures for multi-label classification, and a pre-fine-tuned BERT model for emotion recognition.The supplied training details specify 1,450 segmentation training images, 118,287 multi-label images, and the named emotion-recognition model.Source: §E.2 · p. 20

E.3 Evaluation metrics

  • Coverage measures the empirical probability that test risk stays below α, while RiskGap and AvgSize measure conservatism when the method is valid.For valid methods, larger RiskGap or AvgSize indicates greater conservatism; smaller values indicate tighter control.Source: §E.3 · pp. 20–21

F Asymptotic Behavior in the Large Test-Set Regime

  • As test-set size m tends to infinity with calibration size n fixed, OQRC’s objective shifts from finite-test-set risk control to risk control over the full loss distribution.The empirical test-loss distribution converges to the loss distribution, and the corresponding empirical risk approaches the distributional risk.Source: F Asymptotic Behavior in the Large… · p. 21
  • In the large-test-set regime, Monte Carlo replicates use simplex vectors; the spacing vector q from sorted uniform draws follows Dirichlet(1, . . . , 1).The Monte Carlo implementation is correspondingly simplified when m → ∞.Source: F Asymptotic Behavior in the Large… · p. 21

G.1 Extended results across diverse hyperparameter configurations

  • OQRC maintains required coverage while significantly reducing RiskGap and AvgSize versus baselines across datasets, risk measures, and hyperparameter configurations.Figures 6 to 14 report Cov, RiskGap, and AvgSize across three tasks and three risk measures.Source: §G.1 · p. 21

G.2 Extended results across diverse model architectures

This section reports extended OQRC results across diverse model architectures and includes checklist declarations about research involving human subjects or crowdsourcing and the role of LLMs in method development.

G.2 Extended results across diverse model architectures

  • Across model architectures, OQRC significantly narrows RiskGap and AvgSize versus baselines while achieving valid coverage.The comparisons cover [0.85, 0.95]-VaR-Interval, 0.9-CVaR, and 0.8-VaR; the figures show architecture comparisons for VaR-Interval, 0.8-CVaR, and 0.8-VaR.Source: §G.2 · p. 22, Figure 15 · p. 26, Figure 16 · p. 26

NeurIPS Paper Checklist

  • The checklist says proofs are in Appendix A, limitations are discussed in Section 7, and societal impacts are addressed in the introduction.Source: NeurIPS Paper Checklist · pp. 28, 31
  • The checklist reports results in mean and standard error format, directs readers to Appendix E for experimental settings and compute resources, and says the code will be released.It also states that implementation details are provided to reproduce the results.Source: NeurIPS Paper Checklist · pp. 29–31
  • The paper marks crowdsourcing, human-subject research, and related IRB approval questions as not applicable because the work involves no human-subject research.The checklist also states that the work does not involve crowdsourcing.Source: NeurIPS Paper Checklist · p. 33
  • The paper marks LLM-use declaration as not applicable because core method development does not involve LLMs as an important, original, or non-standard component.Source: NeurIPS Paper Checklist · p. 33
Loading 2609.03104v1…