Source-linked AI summary
Reaching the Tail: Calibration Diversity Drives Conformal Coverage under Data Scarcity
Donald Aadithiyan
TL;DR
Multi-horizon rare-event forecasting is constrained by scarce labeled events and conformal methods' exchangeability assumption under autocorrelated macroeconomic data. The paper combines inter-temporal forecasting with calibration diagnostics, identifies calibration diversity as the main coverage correlate, and tests a diversity-maximizing selector. The selector improves six-month coverage but remains short of the 90% target under honest scoring.
Problem
Rare-event, multi-horizon macroeconomic forecasting has scarce labels, while standard conformal uncertainty quantification assumes exchangeability violated by autocorrelation.
Method
The paper evaluates a two-stage recession-forecasting framework with inter-temporal conditioning, horizon-specific calibration diagnostics, and a diversity-maximizing calibration selector.
Results
The selector is the only tested strategy that improves long-horizon coverage, while Mondrian, shift-robust, and extreme-value alternatives fail to close the gap.
Takeaways & Limitations
Calibration-set diversity deserves attention alongside calibration algorithms when designing conformal predictors under scarcity, although diversity is necessary rather than sufficient.
Takeaways & Limitations
Six-month coverage remains unresolved under honest scoring, and the full forecasting pipeline is U.S.-only despite mechanism replication across five countries.
Abstract
from arXiv · showhide
Multi-horizon rare-event forecasting is hard under long macroeconomic series' data constraints: labeled events are scarce, and standard uncertainty quantification assumes an exchangeability that autocorrelation violates. A controlled ablation shows an apparent rare-event threshold for Adaptive Conformal Inference instead reflects calibration-set size. Across 200 random calibration sets, support width of the nonconformity-score distribution explains up to 85% of coverage variance versus 2% for rare-event count; the same, not the same magnitude, replicates across synthetic conditions and five countries (five-country Spearman $ρ$ 0.45-0.66 vs. 0.02-0.23). A diversity-maximizing selector built on this is the only strategy tested that improves long-horizon coverage (67.8% to 81.4% at six months); Mondrian, shift-robust, and extreme-value alternatives fail to close it. Mondrian even worsens coverage under oracle labels. A compact proposition explains why: coverage deficit reflects how closely the calibration set's upper quantile reaches the test distribution's. Diversity is necessary, not sufficient. Demonstrated on a two-stage U.S. recession-forecasting framework with RegressorChain, whether six-month coverage reaches 90% under honest scoring remains open, a question this paper quantifies rather than resolves.
1 Introduction
The paper frames multi-horizon rare-event forecasting as difficult because scarce labels and temporal dependence challenge both prediction and conformal uncertainty quantification. It argues that calibration-set diversity, rather than rare-event count, drives coverage and motivates a diversity-maximizing selector tested in a multi-horizon recession framework.
- Motivation: Rare-event forecasting combines scarce labels, autocorrelated series, and multiple forecast horizons, with U.S. recession prediction providing a useful test setting.Recessions occur in fewer than 15% of months, while policymakers need probability estimates at several horizons.
- Approach: The study evaluates a practical framework combining inter-temporal conditioning with horizon-specific calibration diagnostics rather than resolving the setting's three properties generally.Its diagnostic finding is supported by a selector, theory, and testbed.
- Contributions: The diversity-maximizing selector is the only tested strategy reported to meaningfully improve long-horizon coverage, while incurring wider intervals.The selector is presented alongside its deployment trade-off.
- Core finding: A fixed-size calibration ablation shows that calibration-set diversity, not rare-event count, predicts conformal coverage and falsifies an apparent rare-event threshold.The comparison is designed to separate calibration-set composition from event frequency.
- Comparisons: Mondrian, a shift-robust controller, and extreme-value tail fitting fail to close the coverage gap, with Mondrian worsening coverage even under oracle labels.The selector alone closes the gap across synthetic conditions and five countries.
- Theory and testbed: A proposition explains that diversity helps but is insufficient because coverage deficit depends on how closely the calibration set's upper quantile reaches the test distribution's.The recession testbed uses a RegressorChain for multi-horizon forecasting.
2 Related Work
Prior recession-forecasting work uses leading indicators, probit models, tree ensembles, penalized logistic regression, and neural sequence models, generally without inter-temporal conditioning or uncertainty quantification.
- Prior forecasting methods: Classical recession forecasting used yield curve spreads and probit models based on credit and yield signals.These approaches are identified as earlier macroeconomic forecasting methods.
- Prior forecasting methods: Later work applied tree-based ensembles, penalized logistic regression, and neural sequence models to recession forecasting.The passage lists these as subsequent modeling approaches.
- Gap: The cited prior work generally lacks inter-temporal conditioning or uncertainty quantification.This positions the paper's framework relative to the described literature.
3 Methodology
The methodology combines a two-stage, multi-horizon recession-forecasting pipeline with ACI and a fixed-budget selector that targets calibration-score support. Its central design rationale is that support width approximates, but does not guarantee, reaching the test distribution’s upper nonconformity quantile.
- Point Prediction Engine: The two-stage pipeline forecasts 12 macroeconomic indicators, then converts those forecasts into horizon-conditioned recession probabilities with a RegressorChain.Stage 1 uses hybrid Prophet-XGBoost and ARIMA-XGBoost models; Stage 2 uses a stacking ensemble across horizons.
- Data and leakage control: All lag, rolling, and autocorrelation features use strictly causal windows, while preprocessing and feature selection are fit only before January 2020.This prevents future information from entering the training features.
- Conformal calibration: ACI sets interval half-width from the empirical (1−α) quantile of calibration nonconformity scores and updates its miscoverage target online.The nonconformity score is the absolute forecast error |y_t − ŷ_t|; the method addresses exchangeability violations from autocorrelated macroeconomic series.
- Diversity-Optimal Conformal Calibration: For a fixed calibration budget N, the proposed selector chooses months maximizing pooled-score support width, p95−p5, rather than using recent months or rare-event counts.Extreme-tail months maximize this quantity for a fixed score set and count, making the selection exact rather than merely greedy.
- Diversity-Optimal Conformal Calibration: The selector overlaps heavily with rare-event-maximizing selection on this series, so random-sampling analysis is needed to distinguish diversity from rare-event composition.The overlap alone does not establish that the two quantities are different.
- Diversity-Optimal Conformal Calibration: The calibration quantile’s shortfall from the test distribution’s corresponding quantile governs coverage deficit, while support width is only a proxy for that reach.Thus, targeting extreme tails is necessary but not sufficient: a wide calibration set can still miss the precise test-period extreme.
4 Results and Discussion
Controlled fixed-size ablations show that calibration-set composition, especially nonconformity-score diversity, predicts conformal coverage better than rare-event count. A diversity-maximizing selector improves six-month coverage, but standard alternatives fail to close the gap and honest six-month coverage remains unresolved.
- The Drivers of Conformal Coverage under Data Scarcity: Coverage rose gradually when rare-event count varied at fixed N=254, removing the apparent threshold caused by changing window size.At 20–40% training fractions, window size and rare-event count changed together; fixing size removed that confound.
- The Drivers of Conformal Coverage under Data Scarcity: R2=0.85 for support width versus R2=0.02 for rare-event count at six months across 200 random N=254 calibration subsets.Support width was defined as p95−p5; the diversity relationship also held across shorter horizons, with little variance to explain at current and one month.
- The Drivers of Conformal Coverage under Data Scarcity: 67.8% to 81.4%: diversity-maximizing selection was the only tested strategy to improve six-month coverage.The gain came with wider intervals, increasing from 10.6 to 32.6 points, and coverage still remained below the 90% target.
- The Drivers of Conformal Coverage under Data Scarcity: 84.75% to 96.61%: out-of-fold scoring produced a similar six-month gain, but its confidence interval still spanned the 90% target.The in-sample result likewise remained statistically indistinguishable from 90% at this sample size.
- The Drivers of Conformal Coverage under Data Scarcity: Mondrian reduced six-month coverage from 67.8% to 61.0% even with oracle labels, while PID-conformal and extreme-value tail fitting also failed to close the gap.Per-regime quantiles stop borrowing the wide scores pooled ACI relies on because the test window contains 63 expansion months and 2 rare-event months.
- Generality Across Conditions: Across five countries, diversity correlated with coverage at Spearman ρ=0.45–0.66 versus 0.02–0.23 for rare-event count.Across seven synthetic scenarios, the corresponding ranges were 0.42–0.68 versus 0.02–0.45; the advantage was strongest in the low-frequency, moderate-magnitude recession regime.
- The Drivers of Conformal Coverage under Data Scarcity: The six-month calibration quantile, 3.01, fell far below the test quantile, 27.06, predicting 57.6% coverage.This quantile mismatch explains the coverage deficit, while sampling noise and ACI online updates likely account for the residual gap.
- Limitations: The full pipeline is U.S.-only, and six-month coverage remains unresolved under honest scoring with only 65 post-2020 observations.The mechanism replicated across five countries, but extension to shorter or noisier records was not tested.
5 Conclusion
The paper argues that calibration-set diversity, rather than rare-event count, drives conformal coverage in data-scarce forecasting. A diversity-maximizing selector improves long-horizon coverage, though six-month tail reach remains unresolved.
- Calibration-set diversity, not rare-event count, is the paper’s true driver of conformal coverage under scarcity.
- The diversity-maximizing selector outperforms Mondrian, PID-conformal, and extreme-value calibration, and is the only tested strategy improving long-horizon coverage.
- Practitioners should prioritize calibration-set composition and diversity alongside calibration algorithms when designing conformal predictors.
- Whether the calibration set reaches the relevant tail at six months remains an open question.
A Which Components Are Essential
The central calibration-diversity finding is independent of the forecasting architecture, while the forecasting pipeline and RegressorChain serve the separate goal of producing strong point predictions.
- The calibration-diversity finding does not depend on the forecasting pipeline or RegressorChain.Synthetic scores without a forecasting model and a probit-model check reproduce the same qualitative pattern.
- Synthetic conditions directly test the diversity-over-rare-count relationship without any forecasting model in the loop.
- A probit-model comparison provides an independent architecture check, although its specific test window was uninformative.
- RegressorChain’s lower six-month MAE is specific to point-prediction quality and is not required for the calibration-diversity result.
B Glossary of Terms
The glossary defines the paper’s forecasting, calibration, uncertainty, dependence, and evaluation terminology. Together, these terms describe how calibration scores are formed, adapted, compared, and assessed under temporal dependence.
- Forecasting and evaluation: RegressorChain predicts related horizons sequentially, feeding earlier forecasts into later-horizon predictions.
- Calibration and uncertainty: A nonconformity score measures forecast error, which conformal methods use to construct future prediction intervals.
- Calibration and uncertainty: Support width measures the distance between the 95th and 5th percentiles of calibration scores, capturing calm and turbulent periods.
- Calibration and uncertainty: Adaptive Conformal Inference changes interval width over time according to recent coverage outcomes and distribution shifts.
- Calibration and uncertainty: Mondrian calibration fits separate calibrations for different classes or regimes instead of pooling all observations.
- Calibration and uncertainty: PID-conformal adjusts interval width using feedback from current and accumulated coverage error.
- Calibration and uncertainty: Quantile reach asks whether calibration extremes are large enough to match extreme test-time errors, even when overall diversity is high.
- Statistical and resampling tools: Spearman ρ measures rank-order association without assuming a straight-line relationship.