PaperlayerView exact revision on arXiv

Source-linked AI summary

SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions

Liang You, Hengyu Shi, Dongwen Ou

arXiv:2610.00523v1cs.LGstat.AP

TL;DR

Marginal conformal coverage can conceal uneven coverage across simplex-valued predictions, while the field lacks a benchmark for comparing wrapper behavior in that setting. SimplexUQ evaluates existing wrappers using controlled synthetic and frozen-predictor real tasks. Its results show that grouping often improves disparity for a fixed map, but the best wrapper depends on the task and evaluation groups.

  • Problem

    A global threshold can preserve marginal coverage while distributing coverage unevenly across simplex prediction space, and no benchmark had compared wrappers for this allocation problem.

    Source: §1 · pp. 1–2

  • Method

    SimplexUQ benchmarks existing wrappers across six synthetic regimes and six frozen-predictor real tasks, fixing predictor, score, and response-free stratification map while varying the wrapper.

    Source: §1 · p. 2, §5 · p. 4, §6 · p. 5

  • Results

    Mondrian has lower mean disparity than Global in 23 of 24 fixed-map task–map cells, but no single wrapper leads across all tasks and protocols.

    Source: §8 · p. 7

  • Takeaways & Limitations

    Wrapper choice should rest on held-out diagnostic comparison because effective repairs depend on the heterogeneity pattern and evaluation groups.

    Source: §8 · p. 7, §10 · p. 9

    Results and deployment recommendations depend on predictor quality, score geometry, evaluation map, data source, and application; some archived configurations also lack explicit map provenance.

    Source: §10 · p. 9

Abstract

from arXiv · show

Conformal prediction guarantees marginal coverage, but a single calibration threshold can still spread that coverage unevenly, over-covering easy regions and under-covering hard ones. SimplexUQ is, to our knowledge, the first benchmark and reproducible protocol for measuring this allocation problem on simplex-valued predictions; it compares existing conformal wrappers rather than proposing a new one. Its task suite, SimplexTasks-12, combines six controlled synthetic regimes with six frozen-predictor real tasks spanning class probabilities, topic mixtures, spectral abundances, cell-type fractions, age distributions, and emotion mixtures. Each comparison fixes the predictor, score, and response-free stratification map, varies only the wrapper, and reports marginal coverage, worst-stratum coverage, max disparity, and within-task radius and compute. Global calibration can look valid while failing badly: on CIFAR-10 it attains 0.900 marginal coverage but only 0.542 in the worst entropy stratum, and Mondrian calibration raises that stratum to 0.886 while reducing max disparity from 0.358 to 0.022. No wrapper dominates, however. Under smooth synthetic heterogeneity, several repairs are competitive; fixed-map analyses show that rankings depend on the evaluation groups and protocol; and in a 12-task comparison, Mondrian has lower disparity on its single target partition for all 12 tasks, whereas BatchMVP has lower disparity over overlapping groups on five. These are empirical comparisons, not new coverage guarantees. A controlled predictor-bias sweep shows that removing predictor bias only partly reduces global-threshold disparity. We release task cards, result provenance, permitted derived arrays, and rebuild instructions, and treat wrapper selection as a diagnostic comparison rather than a universal ranking.

1 Introduction

Simplex-valued predictions need uncertainty evaluation beyond marginal coverage because a global threshold can over-cover easy regions and under-cover hard ones. SimplexUQ benchmarks existing wrappers with controlled comparisons and explicit allocation diagnostics.

1 Introduction

  • Conformal prediction guarantees marginal coverage under exchangeability, but one global threshold can over-cover easy regions and under-cover hard ones.The paper identifies coverage allocation, not marginal coverage alone, as a primary evaluation target.Source: §1 · p. 1
  • SimplexUQ introduces a benchmark for measuring coverage allocation on simplex-valued outputs, rather than proposing a new wrapper.The gap between marginal and subgroup coverage motivates the work; the benchmark is the stated novelty.Source: §1 · p. 2
  • SimplexTasks-12 combines six controlled synthetic regimes with six frozen-predictor real tasks.The suite is released with task cards, permitted derived arrays, and rebuild scripts.Source: §1 · p. 2
  • Each comparison fixes the predictor, score, and response-free stratification map, varies only the wrapper, and reports coverage, disparity, radius, and compute.The reported metrics include marginal coverage, max disparity, worst-stratum coverage, and within-task radius.Source: §1 · p. 2
  • The benchmark assumes exchangeability and applies the same pre-declared map to calibration and test points; deployment claims remain dependent on predictor, data source, and application.Source: §1 · p. 2

2 Related Work

Prior work supplies conformal validity, heterogeneity-aware wrappers, and simplex-specific modeling approaches, but not a benchmark centered on allocation across simplex prediction space. SimplexUQ contributes a controlled evaluation protocol and benchmark for that gap.

Conformal wrappers for heterogeneous prediction

  • Existing conformal wrappers address non-uniform difficulty through grouped calibration or local residual normalization, alongside full and jackknife+ variants.The paper benchmarks these as wrapper families on fixed simplex predictors rather than proposing a new wrapper.Source: §2 · p. 2

Simplex-valued outputs and prediction-space allocation

  • Compositional conformal methods include Dirichlet-based regions [12] and credal sets over class distributions [13], while conformal fairness studies coverage across protected-attribute groups [14] [16].Source: §2 · p. 2

Evaluation as the artifact

  • SimplexUQ follows benchmark infrastructure work such as WILDS, Dynabench, and HELM [17] [18] [19], standardizing twelve tasks and allocation diagnostics for simplex-valued uncertainty.The paper states that existing conformal benchmarks emphasize scalar regression or classification.Source: §2 · p. 2
  • All compared wrappers use the same frozen predictor output and score, differing only in threshold calibration.Source: Table 1 · p. 3

3 Background and Setup

The setup holds the predictor and score fixed while evaluating conformal wrappers on simplex-valued targets. Split conformal provides the marginal-coverage baseline, but its shared threshold does not adapt to variation in residual difficulty.

3 Background and Setup

  • For input X, target Y and fixed-predictor output U lie on the simplex, and the nonconformity score R = d(Y, U) induces prediction sets.The setup uses Aitchison distance for continuous compositions and total variation for classification-style tasks.Source: §3 · p. 3
  • Split conformal calibrates a threshold from calibration residuals using the empirical (1 − α) quantile.Source: §3 · p. 3
  • Under exchangeability, the split-conformal set attains marginal coverage at level 1 − α, but applies the same threshold even when residual distributions vary across the simplex.Source: §3 · p. 3
  • The comparison focuses on coverage allocation and wrapper selection, holding score design, set geometry, and high-dimensional simplex geometry fixed.Source: §3 · p. 3

4 Strategy Taxonomy and Diagnostic Foundations

The strategy taxonomy groups wrappers by how they calibrate thresholds and the heterogeneity pattern they target. The propositions show why marginal validity alone cannot ensure reliable coverage allocation, and why normalization choices can introduce distinct failures.

4 Strategy Taxonomy and Diagnostic Foundations

  • The taxonomy distinguishes global thresholds, per-stratum thresholds, local-scale normalization, and diagnostic approximate-normalization or weighting methods.Each family targets a different heterogeneity pattern and can fail for a different reason.Source: §4 · p. 3

4.1 Three failure modes and main propositions

  • A pooled global threshold can over-cover one stratum and under-cover another despite marginal calibration.The construction uses a two-stratum model with differing residual distributions.Source: §4.1 · pp. 3–4
  • Same-sample kNN scale fitting and quantile calibration can break the finite-sample split-conformal guarantee.The construction uses the same calibration residuals to estimate scales and calibrate the quantile.Source: §4.1 · p. 4
  • Independent training-residual normalization can retain marginal validity while hard-stratum coverage tends to zero, showing that pooled validity does not ensure good allocation.Source: §4.1 · p. 4

5 Evaluation Protocol

The protocol measures coverage allocation within fixed, response-free prediction-space strata while holding predictor, score, and stratification map constant. Radius is an efficiency proxy that must be interpreted within task and alongside coverage and allocation.

5 Evaluation Protocol

  • Strata use pre-declared entropy- or boundary-based rules, with task-specific partitions where a natural grouping exists.The response-free map is applied to test points to compute empirical coverage within each stratum.Source: §5 · p. 4
  • Max disparity is the primary allocation diagnostic, complemented by marginal coverage as a validity check and worst-stratum coverage as a failure floor.The protocol also reports mean radius as an efficiency proxy.Source: §5 · p. 4
  • On D2, a global threshold under-covers boundary-adjacent strata despite near-nominal pooled coverage, while Mondrian and TwoStage both improve allocation.The profile does not establish unique superiority of normalization.Source: Figure 1 · p. 5
  • The fixed-map series reruns Global, Mondrian, and BatchMVP on released frozen arrays, applying each predictor-output map to both calibration and test points.Archived runs cover all wrappers but include configurations with differing scales or partitions, so not every archived ranking isolates wrapper effects.Source: §5 · p. 5
  • A smaller radius indicates an efficiency gain only when coverage and allocation are comparable, and radius measures radial extent rather than simplex volume.Radius comparisons are meaningful within a task, where predictor, score, and simplex dimension are shared; the coverage tolerance is a screening rule, not a finite-sample validity guarantee.Source: §5 · p. 5

6 Synthetic Validation

The synthetic suite isolates distinct heterogeneity mechanisms to test how conformal wrappers allocate coverage. Results vary by regime: several repairs compete under smooth scale heterogeneity, while aligned groups favor group-wise calibration and some settings remain only partly repairable.

6 Synthetic Validation

  • The six regimes isolate homogeneous behavior, smooth scale heterogeneity, aligned discrete groups, bias-type counterexamples, and high-dimensional failure.D5 adds heavy tails to smooth scale heterogeneity; D6 raises simplex dimension.Source: §6 · p. 5
  • D1 acts as a control; normalization and grouped calibration improve on Global in D2/D5, aligned D3 favors grouped calibration, and D4 remains only partially repairable.Paired intervals do not support a uniquely superior normalization family; in D2 Mondrian outperforms TwoStage, while its difference from FullCP is unresolved.Source: §6 · p. 5, Figure 2 · p. 6
  • D6 shows large global disparity, but its daggered FullCP and Jackknife+ entries use a smaller exact-reference run and are not direct main-scale comparisons.Figure 2 reports max disparity in each cell.Source: §6 · p. 5, Figure 2 · p. 6

7 Real Benchmark Suite

The real benchmark applies conformal wrappers to six tasks with fixed, realistic simplex-valued predictors, isolating calibration behavior from predictor development. The suite includes archived task-specific comparisons, repeated split evaluation, and reproducibility artifacts.

7 Real Benchmark Suite

  • The six real tasks cover vision, NLP, remote sensing, genomics, and demographics, with a realistic simplex-valued predictor fixed within each task.Only the conformal wrapper varies, separating calibration behavior from predictor development.Source: §7 · p. 6
  • CIFAR-10, Samson, and UTKFace provide grouped-repair examples; Topics and AffectiveText test that pattern’s limits, while PBMC is a semi-synthetic sensitivity case.Together with the synthetic suite, these tasks form SimplexTasks-12.Source: §7 · p. 6
  • The artifact includes task cards, permitted derived (Y, U) arrays, and table and figure scripts, with rebuild instructions and documented limitations for restricted assets.Source: §7 · p. 6
  • Evaluation uses cached fixed predictors and repeated random calibration/test splits; most real tasks use 50 splits, while AffectiveText and PBMC use 200.Evaluation is CPU-only after predictor outputs are cached; FullCP costs about 3.25 seconds per repetition on full-scale CIFAR-10.Source: §7 · p. 6
  • Archived disparity results favor grouped repair on CIFAR-10, Samson, and UTKFace, while Topics and AffectiveText differ under their archived protocols.PBMC auxiliary-method cells use five-stratum runs while other displayed methods use two-stratum main runs, so those cells do not support cross-protocol rankings.Source: Figure 3 · p. 7

8 Benchmark Results

Global split conformal prediction reaches near-nominal marginal coverage across the real tasks but shows substantial disparity, and wrapper rankings vary by task, evaluation groups, and protocol. Grouped calibration can sharply reduce disparity on some tasks, while different methods favor single partitions versus overlapping groups.

8 Benchmark Results

  • 0.899–0.904 marginal coverage coexists with max disparity of at least 0.098 on every real task, reaching 0.358 on CIFAR-10 and 0.490 on UTKFace, where worst-stratum coverage is 0.410.The homogeneous control D1 is the expected exception.Source: §8 · p. 7
  • Mondrian reduces max disparity from 0.358 to 0.022 on CIFAR-10, from 0.138 to 0.025 on Samson, and from 0.490 to 0.021 on UTKFace.On Topics and AffectiveText, its disparity is 0.276 and 0.356, respectively, the highest among non-weighted wrappers; these archived runs do not separate heterogeneity structure from protocol effects.Source: §8 · p. 7
  • Mondrian has lower mean disparity than Global in 23 of 24 fixed-map task–map cells, with 21 paired intervals excluding zero; Topics with entropy bins favors Global.Its advantage is relative to the chosen map, and no single wrapper leads across all tasks and protocols.Source: §8 · p. 7
  • Figure 4 shows within-task disparity versus mean score radius from the summary-table sources; radius measures radial extent, not volume, and smaller values are preferable only alongside acceptable coverage and comparable protocols.Source: Figure 4 · p. 8
  • The inverse-scale Weighted rule fails the 0.88 coverage screen on every real task, with coverage from 0.130 to 0.770, and is treated as a diagnostic failure case.Source: §8 · p. 8
  • On all 12 tasks, Mondrian has lower default-partition disparity than BatchMVP, while BatchMVP has lower overlapping-family disparity on five tasks.BatchMVP passes the 0.88 coverage screen on every task but over-covers on CIFAR-10 (0.953) and Samson (0.962); paired intervals are not multiplicity-adjusted.Source: §8 · p. 8

9 Strategy-Selection Framework

The framework offers a benchmark-derived heuristic for choosing wrappers based on measured task heterogeneity. Its recommendations are starting points for held-out comparison, not a validated universal rule.

9 Strategy-Selection Framework

  • Coarse aligned heterogeneity points to grouped calibration; smooth heterogeneity calls for comparing normalization with grouping; overlapping target groups call for a multivalid comparison.The heuristic is applied only after diagnostics are computed under the pre-declared map.Source: §9 · p. 8
  • When no wrapper meets the application’s allocation and radius criteria, investigate the predictor or score rather than forcing a recommendation.The controlled predictor-bias experiment supports this step but does not guarantee that improving the predictor repairs allocation.Source: §9 · pp. 8–9
  • Table 3 gives starting points for held-out comparison, not guarantees or a validated universal selection rule.Recommendations must be re-derived after a material predictor change.Source: Table 3 · p. 9

10 Limitations, Future Work, and Conclusion

Marginal coverage alone is insufficient for simplex-valued conformal uncertainty: global CP can remain near nominal while worst-stratum coverage falls to 0.410. Wrapper choice depends on heterogeneity and evaluation groups, so the paper recommends reproducible held-out diagnostic comparison.

10 Limitations, Future Work, and Conclusion

  • Results depend on predictor quality, score geometry, and evaluation map; frozen-predictor recommendations should not transfer to materially different predictors without rerunning diagnostics.The theory identifies failure modes, not an optimal selection rule; some reference runs are not like-for-like method comparisons.Source: §10 · p. 9
  • Proposed extensions include score sweeps, shift-aware tasks, high-K scaling, theory of failure-mode dominance, and training-time feedback for predictors.Released arrays, task cards, and rebuild scripts are intended to make these extensions inexpensive to attempt.Source: §10 · p. 9
  • 0.410 worst-stratum coverage can accompany near-nominal global CP, making marginal coverage insufficient and motivating held-out diagnostic comparison of wrappers.SimplexTasks-12 makes that comparison reproducible.Source: §10 · p. 9

11 Funding and Competing Interests.

The authors report no external funding or competing interests. The accompanying appendices document code, data, reproducibility materials, and theoretical claims about validity and coverage allocation, and specify benchmark construction and interpretation limits while testing sensitivity across stratifications and paired synthetic comparisons.

11 Funding and Competing Interests.

  • No external funding was received, and the authors declare no competing interests.Source: §11 · p. 10

A Code and Data Availability

  • The code repository provides benchmark runners, configurations, tests, and figure scripts; the dataset artifact includes task cards, derived arrays, metadata, and archived results.Source-specific restrictions are documented, and restricted raw assets are not mirrored.Source: A Code and Data Availability · p. 12

B.1 Appendix roadmap

  • The supplement provides source-result snapshots, SHA256 manifests, an audit-and-rebuild script, and rebuild instructions, while noting limits of protocol and predictor reproduction.Current runners use fixed response-free maps; AffectiveText uses frozen outputs, and its open fallback is a different predictor.Source: §B.1 · p. 12

B.2 Formal proof setup

  • The proofs reduce claims to scalar nonconformity scores and use the conservative split-conformal quantile convention, conditioning on fitted objects while requiring calibration and test exchangeability.Source: §B.2 · p. 12

B.5 Approximation result: when local normalization reduces disparity

  • Local normalization bounds conditional coverage error by ρ + L [η + (q⋆+ η)ε] when normalized tails are stable and scale and quantile estimates are accurate.The result assumes a scale model, locally Lipschitz conditional residual cdfs, and estimates independent of the test point.Source: §B.5 · pp. 17–18

B.3 Standard split-validity results used in the paper

  • Split conformal and independently fitted, fixed-scale normalization retain marginal validity; fixed group-wise calibration is valid within groups, but normalization does not ensure conditional validity.Group-wise validity requires exchangeability after conditioning on the fixed response-free map, and small groups can yield large or infinite thresholds and inefficient sets.Source: §B.3 · pp. 13–14, §B.6 · p. 18

B.6 Claims supported by this appendix

  • The appendix’s claim summary distinguishes marginal validity, within-group validity, and allocation performance, emphasizing that normalization methods need not prevent hard-stratum failure.It states that local normalization reduces disparity under normalized-tail stability and accurate scale estimates.Source: §B.6 · p. 18

B.4 Formal proofs for the three main propositions

  • A global threshold can be marginally calibrated yet misallocate coverage across strata; a one-shot normalization counterexample achieves coverage exactly 1/2 < 1 −α.Training-residual normalization can remain marginally valid but fail on hard strata when its scale is overfit; the standard split-validity proof does not automatically apply to calibration-residual-fitted one-shot scales.Source: §B.4 · pp. 15–16, §B.6 · p. 18

B.7 Synthetic data-generating processes

  • Synthetic regimes vary residual scale, prediction-space group structure, predictor bias, noise tails, and simplex dimension; D6 uses larger calibration samples to avoid undersampling alone driving high variance.Fresh data are generated for each repetition with separate train, calibration, scale-estimation, and test subsets.Source: §B.7 · p. 19

B.8 Real-data benchmark distributions, preprocessing, and predictors

  • Real tasks fix target and predictor arrays before conformal evaluation, so comparisons vary the uncertainty wrapper rather than repeatedly retraining predictors.The appendix also documents task-specific predictors and cached outputs; the benchmark’s real-task results are based on those frozen artifacts.Source: §B.8 · pp. 19–20

B.10 Omitted cells and computational constraints

  • Exact-method omissions reflect computational cost or undefined quantities, while inverse-local-scale Weighted CP is unstable and should be read as a diagnostic failure case.Oracle entries require a meaningful residual-scale oracle, and train-residual normalization is undefined for fixed deconvolution pipelines such as PBMC.Source: §B.10 · pp. 20–21

B.11 Runtime notes

  • At D6 benchmark scale, exact and leave-one-out methods were too slow; FullCP and Jackknife+ appear only as smaller-run references (ntrain=250, ncal=250, ntest=1000, 50 repetitions).Representative FullCP runtimes vary by configuration, and split, partition, and normalization-based methods are typically cheaper; these are not hardware-independent complexity comparisons.Source: §B.11 · pp. 21–22, Table 16 · p. 28

B.13 Real-task stratification sensitivity

  • Across four fixed maps, Mondrian was best valid on five of six real tasks; AffectiveText instead favored Global under dominant-component bins.PBMC’s archived mixture-generation runs varied with both generated data and realized strata, so they are descriptive rather than a controlled concentration-only comparison.Source: §B.13 · pp. 22–23, Table 6 · p. 23, Figure 8 · p. 24

C.1 Terminology and interpretation

  • Marginal validity concerns average coverage under exchangeability, whereas allocation measures coverage differences across prediction-defined subsets; radii are score thresholds, not comparable volumes across tasks.Mondrian uses one partition; multivalid methods target a supplied family of potentially overlapping groups.Source: §C.1 · p. 29

C.2 Experiment series and provenance

  • The original benchmark and supplementary sensitivity experiments are distinct series, so comparisons across them are descriptive unless pairing is established within a specified experiment.The sweeps use fixed maps computed from response-free prediction arrays before splitting, with saved result snapshots and a SHA256 manifest.Source: §C.2 · p. 29

C.3 Paired synthetic comparisons

  • In D2, normalization improves on Global, but Mondrian has lower disparity than TwoStage; D2 and D5 comparisons leave several repairs competitive rather than identifying a unique winner.The TwoStage–FullCP difference in D2 and the TwoStage–Mondrian and TrainRes–Jackknife+ comparisons in D5 are unresolved by intervals containing zero.Source: §C.3 · p. 30, Table 18 · p. 30

C.4 Group-wise and multivalid evaluation

Across group-wise, fixed-map, and predictor-bias analyses, wrapper rankings depend on the evaluation groups and objective. Mondrian leads on the single target partition across tasks, while BatchMVP performs better on some overlapping-group comparisons; removing predictor bias only partly reduces global disparity.

C.4 Group-wise and multivalid evaluation

  • Mondrian has lower mean Dg than BatchMVP on all 12 tasks, while BatchMVP has lower mean DH on five; against Global, BatchMVP lowers mean Dg on 10/12 tasks and mean DH on 11/12.D2’s overlap difference is small and its interval includes zero; these comparisons do not establish universal superiority.Source: §C.4 · pp. 30–31, Table 19 · p. 31, Table 20 · p. 32
  • BatchMVP mean coverage ranges from 0.885 to 0.962, and although all tasks pass the 0.88 screen, CIFAR-10 and Samson overcover relative to 0.90.The respective reported coverage values are 0.953 for CIFAR-10 and 0.962 for Samson.Source: §C.4 · p. 31, Table 19 · p. 31

C.5 Fixed-map definition and count sweeps

  • Mondrian has lower mean disparity in 23/24 fixed-map cells, with 21 paired intervals excluding zero in its favor; Topics–entropy favors Global.Topics–KMeans and AffectiveText–dominant intervals include zero; this separate released-array experiment is not the all-method comparison in Table 6.Source: §C.5 · p. 32, Table 21 · p. 33
  • With requested group counts of 2, 3, 5, or 10, CIFAR-10, Samson, and UTKFace favor Mondrian throughout, while Topics favors Global under entropy bins and AffectiveText changes ordering with granularity.PBMC ties reduce realized groups below requested counts, so the analysis reports realized counts and minimum calibration support.Source: §C.5 · p. 33, Table 22 · p. 34

C.6 Controlled predictor-quality experiment

  • The bias sweep pairs samples but recomputes boundary maps at each predictor level, so the population partition is not held fixed.Residual scale heterogeneity remains, and allocation is not monotone at every intermediate bias level; this is controlled evidence, not a guarantee about retraining real-world predictors.Source: §C.6 · p. 34
  • Removing predictor bias lowers Global disparity from 0.151 to 0.128 and raises worst-stratum coverage from 0.750 to 0.773, while Mondrian disparity changes from 0.051 to 0.050.The paired changes are −0.0236 [−0.0312, −0.0165] for Global disparity, +0.0230 [0.0154, 0.0307] for Global worst-stratum coverage, and −0.0008 [−0.0034, 0.0018] for Mondrian disparity.Source: §C.6 · p. 34, Table 23 · p. 34

Related papers

Loading 2610.00523v1…