Sequential Confidence Sets for Coverage-Constrained Conformal Model Selection

1. Introduction

• Coverage and efficiency play different roles: a narrow set is not useful if its pipeline undercovers, and failure to reject infeasibility does not certify feasibility.A scalar trade-off can favor an under-covering method, while an infeasible narrow method may remain under consideration near the coverage boundary.Source: §1 · p. 2
• The target is the set of minimum-cost pipelines satisfying multiple prefix-average conditional miscoverage constraints at each time.Both the feasible set and the identity of its optimizer may change over time.Source: §1 · p. 2
• CC-SMCS separates certified and possible feasibility, then projects simultaneous coverage confidence sequences onto the constrained argmin.The rectangular projection has a closed-form rule computable in O(MR) time.Source: §1 · p. 2, outcome-dependent efficiency objectives · p. 3
• With probability at least 1−δ, the method covers constrained-optimal pipelines simultaneously over time without stationarity, mixing, or exchangeability assumptions, and remains valid under data-dependent stopping.The guarantee is stated under prequential measurability and bounded miscoverage increments.Source: §1 · p. 3
• The target is method comparison, not predictive coverage after selection.Source: §1 · p. 3

outcome-dependent efficiency objectives.

• Near the coverage boundary, a known strict cost advantage may still fail to permit high-probability singleton identification under uniform oracle coverage.The paper also extends the construction to delayed multi-horizon feedback and outcome-dependent efficiency objectives.Source: outcome-dependent efficiency objectives · p. 3, §1 · p. 3

• Classical MCS uses asymptotic confidence for a fixed sample, while Sequential MCS provides time-uniform coverage for scalar conditional-average losses; this work minimizes an observed objective over a time-varying feasible set.Source: §2 · pp. 3–4
• Online conformal and multi-model methods construct or deploy prediction sets, whereas CC-SMCS compares pipelines under monitored coverage constraints.Source: §2 · p. 4
• Unlike constrained best-arm identification of a fixed optimizer, this framework evaluates adaptive pipelines on a common outcome stream, covers current constrained optimizers over time, and allows excluded pipelines to re-enter.Source: §2 · p. 4
• Conformal model-selection work addresses validity after selection or learned reliability, while this paper targets sequential comparison under prefix-average conditional coverage constraints.Source: §2 · p. 4

3. Setup and Target

• Each candidate is a complete conformal procedure, potentially adaptive to the past, whose internal state and time-t output must be measurable before outcome Y_t is issued.Candidates may differ in forecaster, score, calibration window, nominal level, update rule, hyperparameters, or randomization.Source: §3 · p. 5
• The monitored constraints bound exposure-weighted prefix averages of conditional miscoverage by tolerated rates, including subgroup-specific averages when exposure selects a regime.This target differs from marginal coverage and is weaker than pointwise conditional coverage.Source: §3 · pp. 5–6

3.1. Observed prequential efficiency

• The objective is a predictable, observed set cost, such as interval length or classification-set cardinality, so uncertainty is isolated to feasibility.Outcome-dependent efficiency losses are addressed by adding confidence sequences for pairwise objective differences.Source: §3.1 · p. 6
• The dynamic feasible and oracle sets may change over time, and the target assumes neither a true data-generating candidate nor stationarity.Source: §3.1 · pp. 6–7
• The confidence set has finite-sample time-uniform coverage, so repeated inspection or data-dependent stopping does not invalidate its guarantee.Source: §3.1 · p. 7

4.1. Simultaneous confidence sequences for coverage risk

• Empirical-Bernstein confidence sequences are combined across method–constraint pairs to obtain simultaneous coverage-risk bounds.The construction uses predictable estimates and fixed weights for the pairwise error allocation.Source: §4.1 · pp. 7–8

4.2. Certified and possibly feasible methods

• A method is ruled infeasible when a constraint’s lower confidence bound is positive; certified feasibility requires every value in every constraint interval to be nonpositive.A method may remain possibly feasible when the intervals have not established either exclusion or certification.Source: §4.2 · p. 8

4.3. Projection onto the constrained argmin

• The rectangular confidence region admits a closed-form projection onto the constrained argmin for fixed objective values.The paper defines the operational CC-SMCS by this deterministic rule and summarizes the full procedure in Algorithm 1.Source: §4.3 · p. 9

5. Theoretical Guarantees

• Assumption 5.1 fixes the method and constraint counts, tolerances, confidence level, error allocations, objective definition, and mixture parameter before evaluation.It also requires the reported cost denominator to be positive.Source: §5 · p. 10
• For each method and constraint, the prediction set, exposure, cost, and weight are measurable before the outcome and miss indicators are observed.The outcomes and resulting miss indicators are measurable at the current time.Source: §5 · p. 10

5.1. Time-uniform coverage and optional stopping

• Theorem 5.2 states the main coverage result under Assumption 5.1 using confidence sequences from (10) with fixed parameters.The supplied theorem passage does not state the guarantee's probability or scope.Source: §5 · p. 9, §5.1 · p. 10

allocations ηmr = δπmr. Then

• Theorem 5.2's simultaneous-in-time guarantee needs no extra correction for repeated inspection or data-dependent stopping.The claim applies to any stopping time T.Source: allocations ηmr = δπmr. Then · p. 10

5.2. Contraction of the oracle confidence set

• Under persistent exposure, the confidence-set and oracle sets eventually agree if every non-oracle method has a constraint violation margin or a strictly cheaper feasible competitor with a margin.The stated condition requires some ζ > 0 to hold for all sufficiently large times.Source: §5.2 · pp. 10–11

5.3. The coverage boundary

• At the coverage boundary, a strict cost advantage does not determine the oracle because feasibility of the cheaper candidate depends on whether p ≤ τ.The example compares intervals with costs 1/2 and 1 and miscoverage probabilities p and 0.Source: §5.3 · p. 11
• If certification of p ≤ τ has probability at most η whenever p > τ, its probability at p = τ is also at most η.For a model-set rule satisfying (5), first excluding candidate 2 is such a safe certification time with η = δ.Source: §5.3 · p. 11
• A unique lower-cost oracle does not ensure high-probability singleton identification under uniform oracle coverage; near the boundary, identification needs order (τ − p)^−2 observations.The lower bound applies as p approaches τ for fixed threshold and error levels.Source: §5.3 · p. 11
• A prespecified tolerance εr > 0 creates a strict feasibility margin when p ≤ αr and τr = αr + εr, enabling eventual certification under persistent exposure.The stated margin is p − τr ≤ −εr.Source: §5.3 · p. 12

6.1. Simulations

• The simulations use a shared variation process and point predictor across three oracle-dynamics settings, with 3,000 observations and 1,000 replications each.The settings assess oracle retention and adaptation to time-varying oracle models.Source: §6.1 · p. 12

6.1.1. Data Generating Process

• The simulations include stationary, structural-change, and linear-drift settings; in the stationary setting M4 is the fixed oracle on the coverage boundary.The structural break occurs after t0 = 400; the linear-drift setting changes forecast-error scale gradually.Source: §6.1.1 · p. 13

6.1.2. Simulation Results

• Table 1 reports oracle-retention rates close to one, decreasing mean confidence radii, and rare certification of the boundary method.The population oracle set is nonempty at every evaluated time in all replications.Source: §6.1.2 · p. 13, Table 1 · p. 14
• CC-SMCS retains M4 in the stationary simulation, excludes it as undercoverage accumulates after oracle changes, and tracks the changing oracle under structural break or drift.Under gradual drift, M6's inclusion frequency first decreases and then increases as it becomes the constrained oracle; Weak-SMCS follows different paths using standardized Winkler-score differences.Source: Figure 1 · p. 14, §6.1.2 · p. 15

6.2. Empirical Study

• The empirical study applies DLinear, PatchTST, and TimesNet with six conformal interval methods across four real-world time-series datasets.Figure 2 shows DLinear results at 95% nominal coverage with δ = 0.10.Source: §6.2 · p. 15, Figure 2 · p. 16
• Across the empirical applications, CQR can be excluded despite high coverage when its efficiency cost is too high, while ACI, WCP, and Split CP can re-enter model sets.The reported re-entry examples are French Electricity Price for ACI, Electricity Load for WCP, and Weather for Split CP.Source: §6.2 · p. 16

7. Conclusion

• CC-SMCS separates feasibility from efficiency, provides finite-sample time-uniform oracle coverage, and has theory for oracle recovery and limits at the coverage boundary.Simulations and real-data examples illustrate responses of model sets to changes in coverage and interval width.Source: §7 · p. 17

Appendix A. Notation and Algorithm

• The notation distinguishes an inner confidence set for feasibility from outer confidence sets for possible feasibility and the constrained optimizer.The passages identify three reported sets and describe the optimizer set as an outer confidence set.Source: Appendix A · p. 21

Appendix B.1. Empirical-Bernstein confidence sequence

• The empirical-Bernstein confidence-sequence construction allocates error ηmr = δπmr across method–constraint pairs and uses a union bound over those pairs.The proof passages construct a nonnegative supermartingale and apply Ville's inequality.Source: Moreover, for every λ ∈[0, 1), with… · pp. 24, 25, 28

Appendix B.2. Exact rectangular projection

• The exact operational projection relies on coordinatewise freedom within a rectangular confidence region; a nonrectangular region need not preserve the formula.The proof chooses coordinate values independently inside the rectangle.Source: Appendix B.2 · p. 30

Appendix B.3. Main time-uniform theorem

• With probability at least 1 − δ, the theorem’s three set inclusions hold simultaneously at every time.The proof applies the simultaneous confidence event at an arbitrary time and then uses its time-uniform probability bound.Source: Appendix B.3 · pp. 30–31

almost surely. Under the conditions of Theorem 5.2,

• Under Theorem 5.2’s conditions, the oracle-coverage guarantee also holds at any finite stopping time.The corollary applies the event that guarantees the inclusions at every deterministic time.Source: Appendix B.3 · p. 31, almost surely. Under the conditions of… · p. 31

confidence sequence based on the gamma-exponential mixture boundary satisfies

• Persistent exposure and uniform margins make every non-oracle method eventually excludable as the confidence radii vanish.The stated contraction argument assumes finitely many methods and constraints.Source: confidence sequence based on the… · p. 34

Appendix B.5. Certification and oracle identification

• The boundary result limits safe certification: a method uniquely optimal just below the coverage threshold requires order Δ^-2 observations for high-probability singleton identification.This lower bound applies for fixed δ and β.Source: Appendix B.5 · pp. 35–36

Appendix C. The Weak Sequential Model Confidence Set

• Weak-SMCS identifies methods not worse than every competitor using predictably standardized pairwise loss differences; its target may change over time.The implementation uses Winkler score differences, which are conditionally bounded after predictable normalization.Source: Appendix C · pp. 36–38
• The dynamic Weak-SMCS checks whether the joint confidence region permits each method to be no worse than every competitor, allowing previously excluded methods to re-enter.The appendix uses this pointwise-in-time version rather than the running-intersection counterpart.Source: Appendix C · pp. 38–39
• Weak-SMCS and CC-SMCS can follow different paths because they target score-based superiority and coverage-constrained interval width, respectively.The appendix includes Weak-SMCS as a score-based benchmark in simulations.Source: Appendix C · p. 39

Appendix D. The setting of ρ

• The tuning procedure prespecifies ρ⋆ from the forecasting horizon using a deterministic Hoeffding reference intrinsic-time path, rather than realized empirical-Bernstein time.Once selected, ρ⋆ is fixed before outcomes are observed.Source: Appendix D · pp. 40–42

Appendix E.3. Connection Between the Parameters and the Three Simulations

• In the stationary simulation, M4 is the unique constrained oracle: it lies on the coverage boundary and is the narrowest method satisfying coverage.M1–M3 violate the constraint, while M5–M6 are conservative and wider under the stated width ordering.Source: Appendix E.3 · p. 47
• After a structural break, M4 loses feasibility at t = 401, and the post-break oracle is M5 or M6 depending on M5’s accumulated coverage margin.Source: Appendix E.3 · p. 47
• Under gradual drift, M4 soon loses feasibility, M5 becomes oracle first, and M6 takes over after M5’s accumulated coverage margin is exhausted.The stated scale thresholds are approximately t = 1500 for M5 and t = 3000 for M6.Source: Appendix E.3 · p. 47

Appendix E.4. Additional Results

• For δ = 0.25, CC-SMCS retains the current oracle as it changes from M4 to M5 and then M6, while Weak-SMCS tends to retain under-covering methods longer after oracle changes.In the drift simulation, cumulative coverage means M5 can remain feasible after its scale falls below the true scale; Figure E.3b marks the structural break.Source: Appendix E.4 · p. 48, Figure E.3 · p. 49

Appendix F.1. Datasets

• The empirical evaluation uses four hourly or 10-minute univariate series: French electricity price, electricity load, weather temperature, and Hackberry wind generation.The Weather series is regularized to a 10-minute grid, with nine missing values filled from the previous day’s same time.Source: Appendix F.1 · pp. 48, 50

Appendix F.2. Point Forecasting

• The common forecasting framework uses DLinear, PatchTST, and TimesNet for one-step-ahead forecasts with chronological 12:4:2:2 train-validation-calibration-evaluation splits.Inputs use only observations before the target time, and standardization uses training-sample statistics.Source: Appendix F.2 · p. 51

Appendix F.3. Conformal Prediction Methods

• Split CP keeps its calibration radius fixed, whereas WCP updates a recency-weighted residual window after each response.Source: Appendix F.3 · pp. 53–54
• ACI adjusts its miscoverage level from past coverage errors, while AgACI aggregates multiple ACI experts’ endpoints using Bernstein Online Aggregation.Source: Appendix F.3 · p. 54
• CQR uses calibrated quantile intervals that may contract, while EnbPI forms sequential intervals from bootstrap ensembles and out-of-bag residuals without a separate calibration set.EnbPI updates its residual window after each test response.Source: Appendix F.3 · pp. 55–57

Appendix F.4. Additional results

• On French electricity prices with PatchTST, Split CP and EnbPI enter the certified set first, while Split CP and later CQR are excluded for their larger interval widths.CQR enters after its cumulative coverage improves; the reported model-set paths also show methods can re-enter when relative efficiency improves.Source: Appendix F.4 · pp. 57–58, Figure F.5 · p. 61
• On Electricity Load with PatchTST, CQR becomes infeasible as cumulative coverage stays below the 95% nominal level and leaves the possible set.The DLinear and TimesNet patterns instead evolve with candidate coverage and interval widths.Source: Appendix F.4 · p. 58, Figure F.5 · p. 61, Figure F.6 · p. 62
• Across DLinear, PatchTST, and TimesNet on Weather, CC-SMCS generally excludes wide-interval methods once a narrower method is certified feasible, while retaining adequately covered, narrower methods.The reported pattern spans Figures F.5c, F.6c, F.7c, and F.8c.Source: Appendix F.4 · p. 58, Figure F.5 · p. 61, Figure F.6 · p. 62
• On Wind Power, the certified set stays empty until near the prediction-period end under DLinear and PatchTST; Split CP enters first but has the largest cumulative mean interval width.It is excluded only after EnbPI becomes [the passage ends before specifying the condition].Source: Appendix F.4 · p. 58
• With TimesNet, Split CP and EnbPI are certified earlier and wider methods are excluded earlier than under DLinear or PatchTST, where weak coverage–width separation delays exclusions.The contrast shows that informativeness depends on both the conformal calibration method and the forecasting backbone.Source: Appendix F.4 · p. 59, Figure F.4 · p. 60, Figure F.5 · p. 61

Appendix G.1. Delayed multi-horizon feedback

• For delayed h-step feedback, forecast origins are partitioned into h residue classes, with a separate confidence sequence for every method, constraint, horizon, and class.Within each class, an outcome is resolved by the next origin in that same class, and error probability is allocated across streams.Source: Appendix G.1 · p. 59

Corollary Appendix G.1. If the streamwise allocations satisfy P

• The delayed construction retains probability-at-least-1−δ coverage of the constrained target simultaneously over calendar times and monitored horizons, using only resolved forecast origins.On the joint stream event, certified methods are truly feasible and truly feasible methods are possibly feasible.Source: Corollary Appendix G.1. If the… · pp. 59, 69, 70

Appendix G.2. Outcome-Dependent Efficiency Objectives

• For an adapted loss observed after each outcome, the extension adds confidence sequences for pairwise objective differences to the feasibility confidence event.The pairwise differences use methods evaluated on the same outcomes; their confidence sequences can rely on predictable bounds or an appropriate conditional sub-ψ process.Source: Appendix G.2 · p. 70
• On the joint event, every true constrained optimizer remains in the outcome-dependent model set at all times.A certified competitor is truly feasible, so its objective difference from a true optimizer cannot make the lower confidence bound positive.Source: Appendix G.2 · p. 70
• The resulting set is generally an outer set, not the exact projection of a coherent joint objective region.The method combines pairwise objective-difference confidence sequences with the feasibility event.Source: Appendix G.2 · p. 70