PaperlayerView exact revision on arXiv

Source-linked AI summary

Nested Conformal Prediction Sets for Classification with Applications to Probation Data

Arun K. Kuchibhotla, Richard A. Berk

arXiv:2104.09358v1stat.MEstat.AP

TL;DR

The paper asks how accurately statistical-learning risk forecasts perform when those forecasts shape consequential probation decisions. It augments confusion-table assessments with nested conformal prediction sets and a localized conformal modification. The authors report that these approaches can differ dramatically from standard uncertainty measures and argue for using prediction sets, with localized conformal methods matching confusion tables more closely when possible.

  • Problem

    Standard confusion tables provide aggregate uncertainty measures for risk forecasts, while individual decisions require information about whether forecasts are correct and how uncertain they are.

    Source: §2 · p. 6

  • Method

    The paper augments confusion-table assessments with nested conformal prediction sets and develops a localized conformal modification for categorical outcomes.

    Source: §1 · p. 2

  • Results

    Nested conformal and standard uncertainty measures can differ dramatically in concept and output, while localized conformal methods can match confusion tables more closely when possible.

    Source: §1 · p. 2, §6 · p. 25

  • Takeaways & Limitations

    Prediction sets retain uncertainty information, allow precision–uncertainty tradeoffs, and can better represent cases where multiple outcome classes have similar probabilities.

    Source: §6 · p. 25

    Randomly splitting the data into disjoint training and assessment sets introduced additional uncertainty and noticeable variation across repeated analyses, although it made no material difference here.

    Source: §3 · p. 8

Abstract

from arXiv · show

Risk assessments to help inform criminal justice decisions have been used in the United States since the 1920s. Over the past several years, statistical learning risk algorithms have been introduced amid much controversy about fairness, transparency and accuracy. In this paper, we focus on accuracy for a large department of probation and parole that is considering a major revision of its current, statistical learning risk methods. Because the content of each offender's supervision is substantially shaped by a forecast of subsequent conduct, forecasts have real consequences. Here we consider the probability that risk forecasts are correct. We augment standard statistical learning estimates of forecasting uncertainty (i.e., confusion tables) with uncertainty estimates from nested conformal prediction sets. In a demonstration of concept using data from the department of probation and parole, we show that the standard uncertainty measures and uncertainty measures from nested conformal prediction sets can differ dramatically in concept and output. We also provide a modification of nested conformal called the localized conformal method to match confusion tables more closely when possible. A strong case can be made favoring the nested and localized conformal approach. As best we can tell, our formulation of such comparisons and consequent recommendations is novel.

1 Introduction

Statistical-learning risk assessments are being reconsidered amid concerns about accuracy and the consequences of forecasts for probation decisions. The paper compares conventional confusion-table uncertainty with nested conformal prediction sets, which provide finite-sample coverage guarantees and can represent multiple possible outcomes.

1 Introduction

  • The paper presents a demonstration of concept in proprietary probation data because a major revision of an existing statistical-learning risk procedure is under consideration.The study focuses on accuracy while acknowledging broader controversy about fairness and transparency.Source: §1 · p. 2
  • Confusion tables provide aggregate, out-of-sample accuracy estimates but only indirectly indicate how a forecast will perform for a particular offender.Conditioning on the forecast also constrains the assessment and omits important sources of uncertainty.Source: §1 · p. 2
  • Nested conformal prediction evaluates sets of possible outcome classes rather than only the single most likely outcome.For categorical responses, it identifies a best prediction set and estimates the probability that the set contains the true outcome.Source: §1 · p. 2
  • Conformal prediction offers finite-sample accuracy guarantees, while prediction sets for particular offender types have asymptotic uncertainty guarantees.The method can construct prediction sets using offenders’ predictor values.Source: §1 · p. 2
  • The paper compares nested conformal prediction with confusion tables and develops an alternative categorical method intended to corroborate confusion-table inferences.The broader analysis proceeds from the policy setting and conformal-prediction background to comparisons, an alternative method, and recommendations for practice.Source: §1 · p. 2

2 The Policy Setting

The probation department used statistical learning to allocate supervision intensity, but forecast accuracy had direct consequences for offenders and agency resources. The section compares conventional confusion-table measures with the practical uncertainties and tradeoffs surrounding individual risk forecasts.

Policy evaluation

  • A randomized experiment found effectively identical re-arrest proportions under standard and low-intensity supervision, supporting reduced supervision for offenders forecasted to be arrest-free.Low-intensity supervision increased average caseloads from about 150 to 400 offenders and reduced annual office visits from 4.5 to 2.4.Source: §2 · p. 3

Policy setting

  • Forecasts shaped supervision intensity, with low-risk offenders receiving less restrictive monitoring and savings redirected toward more intensive supervision.The department’s stated goal was the least restrictive supervision consistent with public safety, and forecasted outcome classes could affect offenders’ experiences and the agency’s budget.Source: §2 · pp. 3–4

Data

  • The department studied 102,555 probationers over two years, classifying outcomes as violent arrest, nonviolent arrest, or no arrest using 17 administrative-record predictors.The data were split into exchangeable training and test sets, with the test set containing 51,277 offenders.Source: §2 · pp. 4–5

Risk algorithm

  • Stochastic gradient boosting trained on D1 and evaluated on D2 produced the out-of-sample confusion table for the three outcome classes.Predictor weighting was introduced to approximate stakeholder-specified asymmetric error costs.Source: §2 · p. 5, Table 1 · p. 6

Accuracy measures

  • Forecasting error ranged from 0.19 to 0.69 across predicted classes, while classification errors were roughly 40% for all three true outcome classes.The highest forecasting error was inflated by costly misclassification of violence-prone offenders as posing no public-safety threat; lowering that cost ratio would increase overlooked violence-prone offenders.Source: §2 · pp. 6–7
  • The algorithm improved forecasting error over marginal-distribution baselines, reducing no-arrest error to 19% versus 41% and improving nonviolent- and violent-arrest forecasts to 47% and 69% from 68% and 91%.These improvements reflect tradeoffs induced by stakeholder cost ratios, especially the effort to avoid missing violent offenders.Source: §2 · p. 7

3 Some Background for Conformal Prediction

The section introduces conformal prediction as a way to quantify uncertainty for individual categorical forecasts, contrasting its prediction sets and guarantees with conventional single-outcome forecasting.

  • Unlike unconditional guarantees averaged across future offenders, conditional guarantees target offenders sharing a specific predictor configuration.Source: §3 · p. 9
  • Nested conformal prediction is emphasized because it applies to categorical and numeric responses, while conditional finite-sample guarantees are generally impossible and require large samples for approximation.Source: §3 · p. 10
  • For numeric outcomes, oracle prediction regions retain values whose conditional density exceeds a threshold chosen to achieve the desired coverage, but the true density is unknown in practice.Source: §3 · pp. 10–11
  • Estimating the density alone cannot guarantee coverage, so conformal calibration adjusts the estimated threshold to recover the target coverage.Source: §3 · p. 11

4 Nested Conformal Prediction Sets for Categorical Y

This section develops nested conformal prediction sets for categorical outcomes, establishing finite-sample coverage and defining conformal scores while illustrating the method on probation data. It examines the resulting uncertainty information, including tradeoffs in precision and certainty and the motivation for localized conformal regions defined by forecasted class.

4 Nested Conformal Prediction Sets for Categorical Y

  • For categorical outcomes, increasing a probability threshold changes included classes in abrupt steps, so the largest threshold satisfying the target coverage can be conservative or even yield an empty set.With probabilities 0.43, 0.35, and 0.22 and α = 0.30, the selected oracle threshold is 0.35; higher thresholds can reduce coverage below 0.70 or eliminate all classes.Source: §4 · pp. 12–14, Table 2 · p. 13

Calibration

  • The naive categorical prediction set lacks a coverage guarantee when class probabilities are estimated, whereas direct conformal calibration targets the prediction set and supplies the guarantee.Exact finite-sample validity uses the adjusted quantile with the 1 + 1/|D2| increment.Source: §4 · p. 14, Calibration · p. 18

Conformal Scores

  • Conformal scores are the largest provisional miscoverage levels whose naive prediction sets contain the observed class, and nesting makes these scores calibratable across outcome classes.For the illustrative probabilities, the scores are s(x,0) = 1, s(x,1) = 0.57, and s(x,2) = 0.22.Source: Conformal Scores · p. 15

Calibration

  • Under exchangeability, Algorithm 1 produces prediction sets with coverage at least 1 − α.The split-conformal procedure trains on D1, computes scores on D2, calibrates a quantile, and returns the prediction set.Source: Calibration · pp. 17–18
  • In probation data, nonconformity scores group cases by whether the observed class has the highest, second-highest, or lowest fitted probability, with a large zero spike consistent with classifier signal.At α = 0.05, γ(α) = 0.26 produces larger sets for 0.95 coverage; at α = 0.30, γ(α) = 0.60 favors smaller sets for 0.70 coverage.Source: Calibration · pp. 19–20, Table 5 · p. 19
  • Choosing α requires stakeholder tradeoffs between smaller prediction sets with lower coverage and larger sets with greater uncertainty, especially when violence forecasts have different consequences.These choices should be resolved before deploying a risk algorithm.Source: Calibration · p. 20
  • Conditional forecasts may better reflect individual-level subgroup differences and fairness concerns, but their coverage guarantees are only asymptotic.This matters when probation decisions are made one individual at a time.Source: Calibration · p. 20
  • Confusion tables estimate the probability that the true outcome matches the classifier’s best forecast, whereas conformal inference conveys more complete information.Confusion tables condition on forecasted classes, while nested conformal sets do not and use the same threshold for all subjects.Source: Calibration · p. 20
  • Nested conformal inference provides finite-sample-valid, classifier-agnostic estimates and can adapt forecasts to subgroups defined by predictor values.This flexibility addresses heterogeneity among convicted offenders and broadens the range of appropriate applications.Source: Calibration · p. 20
  • At α = 0.2, a confusion table can support a single “NoArrest” prediction at the required probability while nested conformal prediction reports multiple outcomes for the same subject.These methods can therefore produce conflicting conclusions for the same required probability.Source: Calibration · p. 20
  • Localized nested conformal prediction regions address discrepancies by estimating sets separately for groups defined by their forecasted class.The paper turns next to this localized approach.Source: Calibration · p. 21

5 Localized Conformal Prediction

Localized conformal prediction applies nested conformal inference separately within classifier-forecast subsets, aligning prediction sets more closely with confusion-table practice while retaining coverage guarantees. In the probation application, its precision varies by outcome class and confidence level because localized and nested methods provide different guarantees.

Localized conformal method

  • Localized conformal prediction applies the nested conformal procedure separately to test subsets defined by the classifier’s forecasted outcome class.For the probation data, the procedure is applied separately to NoArrest, NonViolent, and Violent forecast subsets.Source: §5 · p. 21
  • The localized procedure selects among three forecast-specific thresholds and reports the prediction set corresponding to the classifier’s forecasted class.The construction uses pC0,α(x), pC1,α(x), or pC2,α(x) according to the forecasted class.Source: §5 · p. 21

Guarantees

  • Localized prediction sets retain a finite-sample unconditional coverage guarantee regardless of classifier accuracy, while being more selective than applying nested conformal inference to all test data.The localized set is constructed from forecast-specific subsets, whereas the complete-data nested method uses all test observations together.Source: §5 · pp. 21–22

Comparison with confusion tables

  • Confusion tables fix prediction sets to one class and therefore produce different coverage levels across forecasts, including 81% for NoArrest and 53% for NonViolent forecasts.These values arise from estimated miscoverage probabilities of 0.19 and 0.47, respectively.Source: §5 · p. 22

Example comparison

  • At 0.95 confidence, localized sets are smaller than nested sets for NoArrest forecasts, whereas at 0.70 confidence the four examples show no difference.Singleton prediction sets can simplify subsequent probation decisions.Source: §5 · p. 22, Table 6 · p. 23

Application to probation data

  • Across the broader comparison, precision differences are inconsistent: localization performs better for NoArrest forecasts, while nested conformal is often superior for the two less common arrest outcomes.The authors note that violent-crime arrests are especially affected by smaller localized subsets, false positives, and stronger per-subset guarantees.Source: Table 7 · p. 23, §5 · p. 24

6 Conclusions

The conclusions argue that conformal prediction sets complement confusion tables by representing individual-level uncertainty and enabling explicit precision–coverage tradeoffs. Their practical value remains bounded by data quality, exchangeability, algorithmic skill, implementation tradeoffs, and stakeholder acceptance.

Implications

  • Conformal prediction sets can better align risk forecasts with stakeholder needs by representing uncertain individual outcomes as sets rather than forced single-class predictions.Singleton sets indicate relatively definitive assessments, while multi-class sets preserve near-tied outcome probabilities.Source: §6 · p. 25
  • Confusion tables remain useful for evaluating aggregate algorithmic performance, including classification error rates, even though they provide less individual-level uncertainty information.The paper presents conformal sets as complementary to rather than replacements for aggregate evaluation.Source: §6 · pp. 24–25
  • Conformal methods let users control coverage and trade it against precision, while nested sets provide finite-sample guarantees and conditional forecasts can have asymptotic guarantees.Localized sets can additionally condition on the classifier’s forecasted outcome class when the policy requires refined forecasts.Source: §6 · p. 25

Limitations

  • The approach depends on exchangeable data and effective forecasting, while administrative-data quality and depth can vary across jurisdictions and criminal-justice decisions.The authors state that credible arguments must justify exchangeability for the available data.Source: §6 · p. 25
  • Risk algorithms involve feasibility, accuracy, speed, ease-of-use, and tuning tradeoffs that remain beyond the paper’s scope.The authors identify proper tuning as essential.Source: §6 · p. 25
  • Whether criminal-justice decision makers and stakeholders will accept conformal methods remains unresolved amid broader concerns about algorithmic risk forecasting, fairness, and transparency.The paper identifies substantial educational work ahead.Source: §6 · p. 25

Related papers

Loading 2104.09358v1…