PaperlayerView exact revision on arXiv

Source-linked AI summary

TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy

Tianqi Zhao, Jia Guo, Yan Shuo Tan, Qiong Zhang

arXiv:2610.13654v1cs.LG

TL;DR

Tabular foundation models are often evaluated by predictive performance, but the reliability of their posterior predictive distributions is less characterized. TabReliBench evaluates reliability across classification and regression, with benchmark comparisons and dataset-level analyses. Results show that rankings shift across benchmarks and metrics, and that marginal coverage, subgroup behavior, and interval efficiency can diverge.

  • Problem

    Tabular benchmarks have primarily compared aggregated predictive performance, leaving the reliability properties of many TFMs’ posterior predictive distributions less characterized.

    Source: §1 · pp. 1–2

  • Method

    TabReliBench evaluates modern TFMs and classical methods on classification and regression datasets across predictive, calibration, coverage, and interval-efficiency measures, with dataset-level analyses.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    TFM rankings change across benchmarks and metrics, and models with similar predictive performance can differ in calibration and interval efficiency.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Reliable tabular prediction requires jointly evaluating predictive accuracy, marginal calibration, subgroup reliability, and interval efficiency for each target dataset and decision.

    Source: §6 · p. 11

    The evidence is limited to the benchmark datasets, fitted configurations, and reliability criteria considered, and worst-slab coverage does not guarantee pointwise conditional coverage.

    Source: §6 · p. 11

Abstract

from arXiv · show

Tabular foundation models (TFMs) are increasingly popular for their strong predictive performance. Many also provide posterior predictive distributions (PPDs), whose reliability remains less studied. We assess reliability through point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency. We then introduce TabReliBench, a large classification and regression benchmark for these comparisons and dataset-level analyses. We find that TFMs' rankings change between TabArena and our benchmark and across reliability metrics. Regression TFMs usually achieve near-nominal marginal coverage and retain narrow intervals, but this favorable marginal behavior does not imply valid conditional coverage. Dataset-level analyses show that classification difficulty often accompanies poor calibration, whereas regression predictive performance is not a reliable indicator of calibration. Ensembling improves predictive performance but does not consistently improve calibration of PPDs; TFMs also benefit little from conformal post-processing, which does not reliably resolve subgroup gaps. Reliable TFM selection therefore requires multidimensional evaluation beyond aggregate rankings. Code is available at https://github.com/Tianqi-Zhao/TabReliBench.

1 Introduction

Existing tabular benchmarks mainly rank predictive performance, while the reliability of TFMs’ posterior predictive distributions remains less characterized. TabReliBench broadens evaluation to multiple reliability dimensions, dataset-level variation, and interventions such as ensembling and conformal post-processing.

Motivation

  • Benchmarks such as TabArena primarily compare aggregated predictive performance, leaving the properties of many TFMs’ posterior predictive distributions uncharacterized.Evaluating those distributions calls for a broader view of reliability than predictive performance alone.Source: §1 · p. 1
  • Predictive accuracy, calibration, coverage, and interval efficiency provide complementary rather than interchangeable evidence about reliability.A favorable proper score combines calibration and sharpness but does not reveal how a model balances them.Source: §1 · p. 1
  • Dataset-level analyses are needed to distinguish shared dataset difficulty from model-specific advantages, a question that typical tabular leaderboards have too few datasets to address.Prior meta-feature analyses on TabArena found limited held-out predictability of model-family performance gaps.Source: §1 · p. 2
  • How ensembling and conformal post-processing affect prediction, marginal coverage, subgroup coverage, and interval width across current TFMs remains less well understood.The paper’s contributions include matched studies of both interventions.Source: §1 · p. 2

Contributions

  • TabReliBench evaluates modern TFMs and classical methods on 160 classification and 81 regression datasets across prediction, calibration, coverage, and interval efficiency.The benchmark also compares results with TabArena and analyzes dataset-level reliability and reliability interventions.Source: §1 · p. 2

2 TabReliBench: A Reliability Benchmark

TabReliBench expands the dataset base for reliability comparisons and dataset-level analyses, using paired splits and a multidimensional evaluation profile across classification and regression.

Benchmark motivation

  • TabArena’s 51 datasets—38 classification and 13 regression—were too few for dataset-level regression models with many dataset-level features, motivating a larger reliability benchmark.Additional train–test splits can improve within-dataset precision but cannot increase the cross-dataset sample size.Source: §2 · p. 2

Benchmark curation

  • The benchmark contains 160 classification and 81 regression datasets, evaluated on five separately generated train–test repetitions with shared observations and partitions across models.At most 10,000 observations are sampled per task; repetitions reduce sensitivity to any single partition while datasets remain the analysis unit.Source: §2 · p. 2

Models under evaluation

  • The comparison includes 11 TFMs and classical methods, but only nine TFMs provide regression point predictions and seven provide regression predictive distributions.BART serves as a probabilistic reference, while RealMLP and XGBoost provide neural-network and gradient-boosted-tree baselines.Source: §2 · p. 2

Evaluation metrics

  • Each dataset–model pair receives a profile spanning point prediction, distributional quality, marginal calibration, classwise or subgroup calibration, and interval sharpness.The metrics include accuracy and R2, Brier score and CRPS, calibration diagnostics, and normalized mean width, which is interpreted only at comparable coverage.Source: §2 · p. 3, Table 1 · p. 3

3 Reliability Beyond Aggregate Rankings

TFM rankings vary across benchmarks and reliability metrics, while regression coverage can look favorable marginally yet differ across feature regions. Dataset differences account for much of score variation, though coverage diagnostics show stronger dataset–model and split-related variation.

3.1 Rankings Across Benchmarks and Metrics

  • The strongest TFMs exchange positions across benchmarks and metrics, although TFMs rank ahead of classical baselines across most reported metrics.LimiX-2 leads mean rank for classification accuracy and regression R2; TabPFNv3.5 for the Brier score, classwise ECE, and CRPS; Causilo for confidence ECE and regression calibration metrics.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 Marginal Coverage Masks Local Gaps

  • Regression TFM intervals are generally close to nominal in marginal and worst-slab coverage, but lower worst-slab coverage does not establish conditional coverage.For every configuration WSC is below marginal coverage; BART reaches near-target WSC through marginal overcoverage and wider intervals, while XGBoost undercovers on both measures despite the widest intervals.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 Decomposing Score Variation

  • Most variation among TFMs comes from dataset differences, while calibration scores—especially coverage errors—show larger dataset–model interactions.Adding classical methods increases model and interaction contributions in regression, so dataset dominance is most pronounced among the more homogeneous TFMs.Source: §3.3 · p. 5
  • Prediction and distributional score differences are more repeatable across splits than coverage errors, with coverage diagnostics—especially WSC—containing substantial partition-related variation.R2 correlations vary markedly across split pairs.Source: §3.3 · p. 5

4 Predicting Reliability

This section tests whether dataset meta-features predict common score shifts and relative TFM differences, then relates dataset properties to predictive and calibration outcomes. Common shifts are more predictable than relative advantages, while classification and regression show distinct calibration signals.

4.1 Average & Relative Performance

  • Using training-only dataset meta-features, the analysis predicts common score shifts and model-specific deviations separately, with ridge regression as the primary predictor and Extra-Trees as a sensitivity analysis.Meta-features summarize dataset properties and are averaged across five training splits; held-out performance is assessed by repeated 5-fold cross-validation over datasets.Source: §4 · p. 5, §4.1 · pp. 5–7
  • Meta-features predict average shifts in accuracy, Brier score, R2, and normalized CRPS, but provide little information about relative TFM differences; PIT–KS and coverage-error shifts remain difficult to predict.Direct feature–response summaries provide the largest gains for average shifts; auxiliary fitted-model summaries add comparatively little.Source: Figure 4 · p. 6, §4.1 · p. 7

4.2 Meta-features Associated with Reliability

  • Class overlap accompanies lower accuracy, worse Brier scores, and larger calibration errors; neighborhood disagreement correlates with accuracy at ρ = −0.78 and confidence ECE at ρ = 0.64.Training size is associated with lower ECE but only weakly with accuracy and Brier score.Source: §4.2 · p. 7, Figure 5 · p. 7
  • Regression prediction and calibration show different meta-feature associations: feature–target dependence accompanies better prediction, while sample size and target non-normality associate with coverage error and PIT–KS.The analysis recommends evaluating coverage and PIT calibration directly, especially when training samples per feature are few, rather than inferring reliability from R2 or normalized CRPS.Source: §4.2 · pp. 7–8, Figure 6 · p. 8

5 Improving Reliability

The section evaluates whether larger ensembles and conformal post-processing improve TFM reliability. Ensembles broadly improve predictive metrics but not direct calibration, while conformal methods can align marginal coverage without consistently resolving subgroup undercoverage.

5.1 Ensemble Gains are Metric-dependent

  • Larger ensembles broadly improve predictive metrics, especially at small to moderate sizes: accuracy rises for all 11 classification TFMs, while R2 improves for 6 of 9 regression TFMs.Brier score improves for all 11 classification TFMs and normalized CRPS for all seven distributional regression TFMs; gains are often front-loaded.Source: §5.1 · p. 8, Figure 7 · p. 9
  • Larger ensembles do not reliably improve direct calibration diagnostics: classwise ECE improves for 5 of 11 classification models and PIT–KS for only three of seven distributional regression models.Coverage–width changes differ across TFMs, and movement toward 90% coverage can accompany wider intervals.Source: §5.1 · pp. 8–9, Figure 7 · p. 9

5.2 Conformal Gains Leave Subgroup Gaps

  • SCP-CQR changes TFMs’ marginal coverage and widths modestly because their intervals are already near target, while BART’s overcoverage moves close to 90% with narrower intervals.All seven TFMs remain narrower than BART after calibration; worst-slab coverage remains heterogeneous and often decreases.Source: §5.2 · p. 9, Figure 8 · p. 10
  • Across the benchmark, no post-processor consistently reduces worst-slab coverage error across models and nominal levels; some coverage gains from PCP and occasionally CalLCP or RLCP involve infinite intervals.Worst-slab coverage is not equivalent to pointwise conditional coverage.Source: §5.2 · p. 10, Figure 9 · p. 10
  • Conformal post-processing can align marginal coverage, but neither global nor covariate-adaptive calibration consistently eliminates worst-slab undercoverage; interval width and degenerate outputs also matter.Covariate-adaptive corrections do not consistently move worst-slab coverage toward 90%, and some methods produce infinite intervals at higher targets.Source: §5.2 · pp. 9–10, Figure 9 · p. 10, Figure 8 · p. 10

6 Discussion & Conclusion

TFM reliability cannot be summarized by one leaderboard: rankings vary across benchmarks and metrics, and similar predictive performance can accompany different calibration and interval efficiency. Dataset properties explain more score variation than average TFM differences, but meta-features predict general performance more readily than the best individual model.

6 Discussion & Conclusion

  • TFM rankings change across benchmarks and reliability metrics, and models with similar predictive performance can differ in calibration and interval efficiency.These comparisons show why reliability cannot be reduced to a single leaderboard.Source: §6 · p. 10
  • Most score variation is associated with datasets rather than average differences among TFMs, and meta-features predict general performance more readily than which TFM will perform best.This distinction is reported for dataset-level analyses of TFM outcomes.Source: §6 · p. 11
  • Local class overlap accompanies poorer classification accuracy and calibration, whereas regression prediction and calibration are associated with different dataset properties.Larger ensembles improve accuracy and proper scores most consistently, but direct calibration measures change less uniformly.Source: §6 · p. 11
  • Conformal calibration can correct substantial marginal miscalibration, particularly for BART, but neither global nor covariate-adaptive methods consistently remove worst-slab undercoverage.Interval width and degenerate outputs remain part of the comparison.Source: §6 · p. 11
  • The evidence is limited to the benchmark datasets, fitted configurations, and reliability criteria considered, and worst-slab coverage does not guarantee pointwise conditional coverage.The conclusion is that accuracy, marginal calibration, subgroup reliability, and interval efficiency answer different questions.Source: §6 · p. 11

Supplementary Materials

The supplementary material situates the benchmark among work on distributional reliability, dataset-dependent model performance, and conformal calibration. It documents dataset selection and repeated train–test sampling, including shared partitions for model comparisons.

A Additional Related Work

  • Related work on distributional regression and conditional density estimation motivates extending reliability evaluation to both classification and regression, with dataset-level analyses of general performance and relative model advantages.The supplementary material also describes evaluation of conformal methods as post-processors using marginal error, subgroup error, width, and degenerate sets.Source: A Additional Related Work · p. 13

B.1.1 Dataset Curation and Target Processing

  • The benchmark contains 160 classification and 81 regression datasets curated from TabArena and eligible OpenML tasks, with restrictions on sample size, columns, class count, and task type.Tasks use default targets; missing classification labels and nonfinite regression responses are discarded, and class labels are consistently mapped for probability evaluation.Source: §B.1.1 · p. 13

B.1.2 Sampling and Train–Test Splits

  • Each of five repetitions samples at most 10,000 observations and uses an approximately 50/50 train–test split, with stratification for classification.Models share the same observations and outer partition; TFMs use the training partition as context and BART fits on that partition.Source: §B.1.2 · p. 13

B.1.3 Complete Dataset Catalog

The supplementary material catalogs benchmark datasets, evaluated models, and implementation choices. It standardizes comparisons through shared partitions and an eight-member TFM ensemble budget while retaining model-specific inference procedures and common distributional evaluation choices.

B.1.3 Complete Dataset Catalog

  • Table 2 lists the OpenML dataset identifiers that define the benchmark.The catalog is presented as the complete dataset list.Source: §B.1.3 · p. 13, Table 2 · p. 14

B.2.1 Model Overview

  • The comparison includes 11 classification TFMs, nine regression TFMs, and five conventional configurations; seven TFMs provide regression distributions, while LimiX-2 and TabFM enter only point-prediction comparisons.Table 3 identifies the evaluated checkpoints, implementations, and prediction types.Source: §B.2.1 · p. 14, Table 3 · p. 14
  • The main benchmark uses eight members of each TFM’s native ensemble, fixed pretrained weights, and no supervised fine-tuning or dataset-specific hyperparameter search.Regression distributions are represented on a 199-level quantile grid, monotonized by cumulative maximum, and linearly interpolated; native point predictions are retained.Source: §B.2.1 · pp. 14–15, §B.2.2 · p. 15

B.2.2 Tabular Foundation Models

  • TFM implementations mostly retain checkpoint-specific defaults, with input adaptations including TabPFN categorical handling, numeric conversion of all-missing TabICL columns, and context-only category encoding for TabDPT.LimiX-2 uses no-retrieval recipes and selects eight classification pipelines to maximize preprocessing diversity; its regression recipe already supplies eight pipelines.Source: §B.2.2 · p. 15

B.2.3 Conventional Baselines

  • BART uses 1,000 post-burn-in draws across four chains; regression predictive draws include observation noise, and their mean supplies the point prediction.For multiclass tasks, BART fits one probit outcome per class and normalizes probabilities within each posterior draw.Source: §B.2.3 · p. 16
  • RealMLP and XGBoost share an 80/20 internal validation split, select checkpoints by validation loss, and use 50 configurations with a 256-stage cap for HPO.The budgets standardize candidate counts and maximum training stages, not wall-clock time or computational cost; both predict the common quantile grid for regression.Source: §B.2.3 · p. 16
  • RealMLP regression replaces its standard point objective with mean pinball loss on the common quantile grid, while XGBoost regression uses one output tree per quantile.XGBoost-HPO uses a prespecified TabArena-based search space, with the learning-rate lower bound raised to 0.02 under the common 256-round cap.Source: §B.2.3 · p. 16, Table 5 · p. 16

B.3 Computational Environment

  • The main benchmark ran on NVIDIA A100 GPUs, conformal experiments on an RTX 5090 GPU, and ensemble-size experiments on CPU and A100 cluster nodes.Software dependencies and model-specific installation instructions are provided in the repository.Source: §B.3 · pp. 16–17

C Metric Definitions and Aggregation

This section defines a multidimensional profile of prediction, distributional quality, calibration, coverage, and interval width, then describes how metrics and comparisons are aggregated. Results show that method ordering varies across criteria, and regression coverage profiles involve trade-offs that separate dataset medians cannot fully resolve.

C.3 Coverage, Worst-Slab Coverage, and Width

  • Worst-slab coverage selects a supported feature slab using search observations, then evaluates that subgroup on held-out observations; total coverage error sums marginal and worst-slab absolute deviations.The associated interval width remains the overall evaluation-set width.Source: §C.3 · pp. 17–18

C.4 Aggregation, Ranks, and Uncertainty Intervals

  • Scores are averaged over available repetitions within each dataset before ranks are computed, and mean ranks weight datasets equally.Bootstrap intervals resample paired dataset identifiers and are pointwise, reflecting dataset resampling rather than new train–test splits.Source: §C.4 · p. 18

D.1 Complete Rank Comparisons

  • Complete rank comparisons average within-dataset five-split scores and then ranks across datasets, with paired dataset bootstrap intervals.Tables 6–11 contain the numerical comparisons; their intervals are pointwise and do not account for related dataset-family dependence.Source: §D.1 · p. 18, Table 6 · p. 19, Table 7 · p. 19

D.2 Absolute Prediction and Calibration Profiles

  • Regression calibration diagnostics do not always rank methods alike, so proper scores and direct calibration diagnostics provide complementary information about predictive distributions.The absolute profiles summarize median dataset-level five-split means; regression CRPS is normalized by training-response standard deviation.Source: §D.2 · p. 19, Figure 10 · p. 21
  • Classification accuracy and Brier score, and regression R2 and CRPS, are ranked within datasets before averaging across datasets; CRPS is training-scale normalized for cross-dataset profiles.Tables 12–13 report absolute-profile medians and interquartile ranges; Figure 10 also presents calibration diagnostics.Source: §D.2 · p. 19, Table 12 · p. 21, Figure 10 · p. 21

D.3 Coverage–Width Profiles at Additional Nominal Levels

  • At 95%, 85%, and 80% nominal coverage, TabPFNv3.5 has higher marginal and worst-slab coverage than TabPFNv3, with slightly wider intervals.Separate medians of coverage and width do not establish width rankings at equal achieved coverage.Source: §D.3 · p. 22, Figure 11 · p. 22

D.4 Score Decomposition with Conventional Baselines

  • Adding conventional methods preserves the classification result but makes regression model and dataset–model components larger, with distinct methods showing more recurring interactions.The TFM-only regression comparison is comparatively homogeneous; adding BART, RealMLP, and XGBoost raises split-pair correlations for regression calibration metrics.Source: §D.4 · p. 23, Figure 12 · p. 23

D.5 Persistent Interactions versus Split Variation

The split-level analysis separates recurring dataset–model interactions from variation tied to particular splits, with results differing across diagnostics. The section also specifies a held-out component-prediction analysis and its meta-feature and cross-validation setup.

D.5 Persistent Interactions versus Split Variation

  • The split-level decomposition separates dataset, model, dataset–model interaction, shared split, and residual model–split terms.The exact identity needs no stochastic assumptions, but persistent interaction estimates additionally use a working model; residual variation is not identified as pure measurement error.Source: §D.5 · pp. 23–24
  • The variance-component estimates describe persistence and resplit variation conditional on observed samples, not variability across independently collected datasets.They rely on a working model with mean-zero, mutually uncorrelated random effects and homogeneous component variances; negative estimates are truncated at zero.Source: §D.5 · pp. 23–24
  • Brier and PIT–KS interactions remain largely persistent, whereas much of the apparent worst-slab and total-coverage interaction is attributed to split variation.PIT–KS remains highly repeatable after split averaging, while worst-slab coverage remains unstable; the variance-ratio and split-pair diagnostics are complementary.Source: §D.5 · p. 24, Table 14 · p. 24

E.1 Held-Out Component Targets

  • Component prediction evaluates held-out error as separate average-shift and relative-deviation terms, with positive gains indicating improvement over the training-fold baseline.Baselines and targets are re-estimated within each fold.Source: §E.1 · p. 26

E.2 Meta-Feature Groups

  • The component-prediction catalog groups training-data meta-features into cumulative information sets.The table identifies the cumulative sets, while the detailed counts are not included in these passages.Source: Table 15 · p. 26

E.3 Meta-Predictor Configuration and Cross-Validation

  • Meta-prediction uses five shuffled outer folds across five repetitions, with datasets as sampling units and no dataset shared across training and validation/test folds.Ridge penalties are selected in inner folds; Extra-Trees uses fixed settings and predicts average shifts and model-specific deviations.Source: §E.3 · p. 26

E.4 Additional Meta-Prediction Results

Additional analyses find that meta-features predict average score shifts better than model-specific deviations and that individual-score predictability varies by task and metric. The section also catalogs the training-data summaries used in these analyses.

E.4.1 Sensitivity to the Models and Meta-Predictor

  • Across models and meta-predictors, meta-features predict average score shifts better than differences among models, while model-specific deviations remain weakly predictable.Adding classical methods exposes additional calibration signal.Source: §E.4.1 · p. 27, Figure 16 · p. 27

E.4.2 Individual-Model Score Prediction

  • Individual-model score predictability is strongest for classification scores and regression predictive metrics; PIT–KS is less predictable.Out-of-fold R2 ranges are 0.41–0.69 for classification scores, 0.10–0.87 for regression R2, 0.14–0.82 for normalized CRPS, and 0.04–0.50 for PIT–KS.Source: §E.4.2 · p. 27, Figure 17 · p. 28

E.5.1 Overview and Shared Conventions

  • The catalog uses 62 classification and 59 regression meta-features, computed from training data and organized into four cumulative information groups.Those groups are covariate-only, response-only, direct feature–response, and auxiliary predictive meta-features.Source: §E.5.1 · p. 27, Table 16 · p. 27

E.5.2 Covariate-Only Meta-Features (X)

  • The 27 covariate-only meta-features describe dataset size, feature types, marginal distributions, dependence, and unsupervised geometry.They are shared by classification and regression.Source: §E.5.2 · p. 28, Table 17 · p. 29

E.5.3 Response-Only Meta-Features (Y )

  • Response-only features add six classification and nine regression summaries, while direct feature–response features add 16 and nine, respectively.The cumulative dimensions after those groups are 33 and 36, then 49 and 45, for classification and regression, respectively.Source: §E.5.3 · p. 29, §E.5.4 · p. 29, Table 18 · p. 29

E.5.4 Feature–Response Meta-Features (XY )

  • Direct feature–response summaries include mutual-information concentration, entropy, and weak-signal measures based on feature–response associations.The mutual-information estimates use three nearest neighbors.Source: §E.5.4 · p. 30, Table 19 · p. 30

E.5.5 Auxiliary Meta-Features (Aux)

  • Auxiliary features include fitted-model prediction scores and summaries of threshold trees, tree shape, and hierarchical shrinkage.The auxiliary group contributes 13 classification and 14 regression features.Source: §E.5.5 · pp. 30–31

F Dataset Associations

The association analyses identify dataset characteristics linked to TFM performance, with patterns differing between classification and regression and across models. The appendix describes pooled and individual-model correlation procedures, which are exploratory rather than causal or predictive.

F.1 Association Protocol

  • Pooled analyses correlate training-data meta-features with median model scores across datasets using Spearman correlations for four classification and six regression metrics.The pooled responses weight reporting configurations equally; normalized CRPS uses each split’s training-response standard deviation.Source: §F.1 · p. 31
  • The analysis excludes auxiliary-predictor outputs and supervised tree summaries, leaving 49 classification and 45 regression meta-features eligible.For each metric, the five eligible features with the largest absolute correlations are selected, subject to task-specific complete-pair and variability requirements.Source: §F.1 · p. 31
  • Benjamini–Hochberg correction is applied over eligible tests, but the associations remain exploratory and do not establish causality or held-out predictability.They also do not account for dependence among related datasets.Source: §F.1 · p. 31

F.2 Individual-Model Association Atlas

  • Individual-model displays select each model’s five strongest eligible non-auxiliary feature associations, so a displayed feature may reflect a model-specific pattern.The authors describe these displays as descriptive, not as cross-validated meta-prediction or tests of between-model differences.Source: §F.2 · p. 31
  • Class overlap has consistent associations with classification accuracy, Brier score, and both ECE variants; in regression, feature–target mutual information and feature outliers or skewness track predictive performance.Training size is more specifically associated with classification calibration, while training size and samples per feature are the most stable regression correlates of PIT–KS and coverage errors; regression calibration associations vary across models.Source: §F.2 · p. 32

G Conformal Methods and Additional Results

The conformal experiments compare interval coverage and efficiency across methods and base models. Marginal-error reductions are more consistent than subgroup-error reductions, while some methods incur infinite intervals.

G.1 Experimental Protocol

  • The experiment evaluates seven distributional TFMs and BART on shared held-out data, with paired comparisons aggregated by dataset.A shared split assigns auxiliary, calibration, and held-out blocks; WSC subgroup selection uses one held-out half and evaluation the other.Source: §G.1 · p. 35

G.2.1 Global Split Conformal Methods

  • Global split-conformal methods calibrate absolute, normalized, signed-quantile, or PIT-based scores using a finite-sample calibration quantile.SCP-CQR applies its threshold to the base quantile interval, and the threshold may be negative, shrinking that interval.Source: §G.2.1 · p. 40

G.2.2 Localized Conformal Methods

  • Localized methods use covariate-dependent weights or score adjustments to adapt conformal thresholds around each query.CalLCP and RLCP localize calibration; SLCP estimates a local score correction on auxiliary data and calibrates its residual on a separate block.Source: §G.2.2 · p. 40

G.2.3 Learned Conditional Adaptation

  • Learned conditional-adaptation methods estimate score quantiles from covariates and then apply conformal correction on calibration data.RCP uses a distributional regressor for signed CQR scores, while PCP follows its absolute-residual construction; the original response predictor remains frozen.Source: §G.2.3 · p. 41

G.3 Coverage Reliability

  • SCP-CQR tracks marginal coverage targets, but worst-slab coverage remains more heterogeneous; across post-processors, marginal-error reductions are more consistent than subgroup-error reductions.Across 32 base-model–level combinations, global split-conformal variants, CalLCP, and RLCP reduce marginal error significantly in 24–27 cases; no method significantly reduces WSC error in more than 10.Source: Figure 28 · p. 41, §G.3 · pp. 41–42, Figure 29 · p. 42
  • SCP-CQR reduces marginal error throughout, but increases BART’s worst-slab error at 80–90% coverage; PCP increases marginal error in 29 of 32 combinations.SCP-ABS increases WSC error in 30 combinations, and no post-processor significantly reduces WSC error in more than 10.Source: §G.3 · p. 42

G.4 Efficiency and Degenerate Outputs

  • PCP’s infinite-interval rate rises from 14.3% at 80% target coverage to 44.1% at 95%, while SCP-CQR remains at zero.Any positive infinite-interval rate makes arithmetic mean width infinite; negative signed-CQR thresholds can also produce empty prediction sets, retained as uncovered observations with zero width.Source: §G.4 · pp. 49–50

H.1 Evaluation Protocol

  • The ensemble sweep varies estimator count across five values, using five seeds and common datasets for each task and metric.It covers 11 classification TFMs on 160 datasets and nine regression TFMs on 81 datasets, with medians computed after averaging seeds within datasets.Source: §H.1 · p. 51

H.2.1 Median-Metric Results

Ensembling generally improves predictive metrics, but calibration responses are less consistent across models and metrics. Coverage–width trajectories further show that reaching nominal coverage can involve wider intervals and need not improve subgroup coverage.

H.2.1 Median-Metric Results

  • Across ensemble-size sweeps, accuracy and the proper scores (Brier and normalized CRPS) generally improve, mostly at small-to-moderate sizes; median R2 is mixed, and larger ensembles do not uniformly improve calibration.Source: §H.2.1 · p. 51, Figure 42 · p. 52, Figure 43 · p. 52

H.2.2 Coverage–Width Results

  • As ensemble size changes, movements toward nominal coverage can accompany wider intervals, and marginal and worst-slab coverage need not move in the same direction.The coverage–width trajectories summarize medians across the same 81 regression datasets at 90% nominal coverage.Source: Figure 44 · p. 53, §H.2.2 · p. 53

Related papers

Loading 2610.13654v1…