PaperlayerView exact revision on arXiv

Source-linked AI summary

State-Matched Sequence Conformal Prediction

Ruihang Han, Li-Hsiang Lin

arXiv:2610.19484v1stat.MLcs.LG

TL;DR

Sequential conformal prediction must choose which past errors inform uncertainty for the current target, and recency can be inadequate when comparable states recur at nonadjacent times. SMSCP matches pre-target local-linear states to select calibration scores, then applies the ordinary unweighted conformal rule. Its reported results favor SMSCP on several real-data comparisons, while showing that gains depend on state relevance and temporal patterns.

  • Problem

    Sequential conformal prediction must identify which past observations are relevant to the current target, since temporal closeness can be a poor signal and similar observed features need not imply similar target-score quantiles.

    Source: §1 · p. 3

  • Method

    SMSCP builds an interpretable one-sided local-linear state from pre-target information, matches past states to select calibration scores, and applies the ordinary unweighted conformal order statistic.

    Source: §1 · p. 3

  • Results

    SMSCP has lower mean interval score than recency, weighted, adaptive, and HopCPT comparators, and coverage closer to nominal 0.90 than ResCP on five of six benchmarks, while ResCP has lower mean interval score on all six.

    Source: §1 · p. 4

  • Takeaways & Limitations

    State matching is most effective when score-relevant states recur, and its gains weaken when the state misses relevant information or switching is abrupt.

    Source: §1 · p. 4

    Full-history predictive conditional validity additionally requires the state-information discrepancy to vanish; nearby states must also have comparable target score quantiles under the stated relevance condition.

    Source: §3 · p. 9

Abstract

from arXiv · show

Sequential conformal prediction needs to decide which past errors should calibrate uncertainty for the current target. Recency and online error feedback are useful when recent observations remain relevant, but the recent past can be a poor reference when comparable states recur at nonadjacent times, with the recurrence pattern generally unknown. We propose State-Matched Sequence Conformal Prediction (SMSCP), which constructs an explicit one-sided local-linear state from pre-target information, uses state similarity to select a matched reference set of past calibration scores, and then applies the ordinary unweighted conformal order statistic. In contrast to existing approaches, SMSCP uses state similarity to identify calibration observations relevant to the target score quantile. This motivates a theoretical analysis of whether nearby states have comparable target score quantiles under temporal dependence. We establish control of the resulting conformal cutoff and characterize predictive information not captured by the chosen state. Simulations show that state matching is most effective when score-relevant states recur across nonadjacent times, while revealing settings in which its advantage weakens or reverses. Across solar irradiance and bike-demand benchmarks, SMSCP improves interval score over recency, weighted, adaptive, and HopCPT comparators, and achieves coverage closer to the nominal 0.90 level than ResCP on five of six benchmarks.

1 Introduction

SMSCP makes calibration relevance target-specific by matching pre-target states and applying the ordinary conformal rule to the selected past scores. Its theory analyzes state-quantile relevance under temporal dependence, and the reported results favor SMSCP on several benchmarks while identifying settings where gains weaken or reverse.

1 Introduction

  • SMSCP targets comparable target-score quantiles across states because similar observed contexts can correspond to different residual regimes.Unlike methods that retrieve or reweight residuals through observed or learned representations, it asks whether pre-target states imply comparable target-score quantiles.Source: §1 · pp. 2–3
  • The theory bounds cutoff error through state matching, dependent sampling, and finite-k order-statistic terms, while measuring predictive information omitted by the chosen state.Full-history predictive conditional validity additionally requires the state-information discrepancy to vanish.Source: §1 · pp. 3–4
  • On five of six benchmarks, SMSCP coverage is closer to nominal 0.90 than ResCP, while ResCP has lower mean interval score on all six.SMSCP also has lower mean interval score than recency, weighted, adaptive, and HopCPT comparators.Source: §1 · p. 4
  • Empirical gains are clearest when score-relevant states recur and weaken when the state misses relevant information or switching is abrupt.The paper does not claim universal dominance.Source: §1 · p. 4
  • SMSCP frames sequential calibration as state matching, using interpretable pre-target states to define relevance beyond temporal recency.It selects a matched reference set, then applies the ordinary unweighted conformal order statistic without changing the predictor, score, or quantile rule.Source: §1 · p. 4

2 Method

SMSCP uses a rolling pool of past calibration scores and selects relevant scores by matching one-sided local-linear states. The state summarizes local response level and slopes; a validation block selects the state history length and matching fraction.

2.1 Problem setup

  • The eligible calibration pool contains past indices with the required pre-target predictor and state, and calibration uses their scores to inform the current target score quantile.When all required quantities are available, the eligible pool equals the rolling candidate window.Source: §2.1 · p. 5

2.2 State construction

  • For fixed history length h, SMSCP represents each time point with one-sided local-linear coefficients that summarize local response level and slopes against pre-target inputs.Neither the response at that time nor future observations enter the state fit.Source: §2.2 · pp. 5–6

2.3 Conformal prediction based on matched states

  • SMSCP compares calibration and target states using regularized Mahalanobis distance, selects a matched reference set, and applies the ordinary unweighted conformal order statistic to its scores.The distance accounts for scale and correlation among state coordinates; when ρ = 1, calibration reduces to the eligible rolling pool.Source: §2.3 · pp. 6–7

2.4 Hyperparameter selection

  • A validation block selects history length h and matching fraction ρ by minimizing mean interval score, then the selected pair is fixed for deployment.Smaller ρ retains fewer, closer states; larger ρ uses more scores while admitting more distant states.Source: §2.4 · p. 8

3 Theoretical Properties of SMSCP

The theory bounds SMSCP’s state-indexed cutoff error using state–quantile relevance, uniform local concentration under temporal dependence, and local quantile regularity. Approximation to the full-history target additionally depends on predictive information omitted by the state.

Assumptions

  • State–quantile relevance requires the target score quantile to be stable across recurrent states and vary smoothly with state distance, not the full score distributions to match.The condition does not require a local-linear response model or constant score quantiles over calendar time.Source: §3 · pp. 9–10
  • Uniform local concentration must hold over data-dependent neighborhoods, while local quantile regularity converts CDF error into quantile error.The concentration requirement is stated at the empirical-process level rather than tied to one dependence model.Source: §3 · pp. 10–11

Theorem 1

  • SMSCP’s cutoff error is bounded by state-matching, sampling-under-dependence, and finite-k order-statistic terms.The bound concerns the state-indexed target quantile; it does not by itself establish validity relative to the full pre-target history.Source: §3 · pp. 11–12
  • Smaller k can improve locality, whereas larger k reduces the finite-k term and typically stabilizes empirical quantile estimation.The additional small-perturbation conditions in Theorem 1 control the empirical order statistic and are not algorithm restrictions.Source: §3 · p. 12

State-information discrepancy

  • The state-information discrepancy measures predictive information in the full pre-target history omitted by the chosen state, and full-history approximation depends on it.The theorem controls estimation of q(Z_t); predictive conditional validity additionally requires the discrepancy to vanish.Source: §3 · pp. 9, 12, 13

Asymptotic coverage

  • Predictive conditional coverage converges to nominal when the discrepancy and local-concentration error vanish and k grows.Stationary absolute regularity with suitable mixing decay and k growth is one sufficient route to the required local concentration.Source: §3 · p. 13

4 Experiments

Experiments compare SMSCP with non-neural and learned-retrieval methods across simulations and solar and bike benchmarks. State matching is strongest when score-relevant states recur, while its advantage can weaken or reverse in other settings.

4.1 Setup

  • The setup evaluates nominal 0.90 coverage using shared nonanticipating point predictions and scores in the non-neural panel, with separate HopCPT and ResCP comparisons.State inputs vary by application, while state matching and conformal calibration rules remain unchanged; tuning uses a chronological validation block before test evaluation.Source: §4.1 · pp. 13–15

4.2 Simulation results

  • In all four HomoSin and DualFreq cells, SMSCP beats six non-neural comparators in mean interval score, with paired 95% Monte Carlo intervals strictly favoring SMSCP.Intervals are narrower and coverage is at or above nominal, though somewhat conservative; HeteroSin-small is mixed, HeteroSin-large has slight undercoverage (0.898), and ARCH(1) results are not consistent across sizes or HopCPT comparisons.Source: §4.2 · pp. 15–16

4.3 Real-data benchmark: solar irradiance

  • At all five solar stations, SMSCP has lower mean interval score than each of six non-neural comparators, with coverage from 0.909 to 0.920.All 30 paired intervals favor SMSCP; width reductions are not uniform, so coverage and width are interpreted separately.Source: §4.3 · p. 18, Table 2 · p. 19

4.4 Real-data benchmark: bike sharing

  • On bike demand, SMSCP has coverage 0.894 and mean interval score 419.6; its paired interval-score advantage over ECI is 199.5 [95% interval: 190.1, 220.1].Against HopCPT, SMSCP has lower mean interval score and coverage closer to nominal 0.90; against ResCP, coverage is closer on five of six benchmarks, while ResCP has lower interval score on all six.Source: §4.4 · pp. 19–20

5 Conclusion

  • The conclusion attributes the strongest gains to recurring score-relevant states, with weaker gains when recency or observed inputs suffice, the state omits relevant information, or switching is abrupt.The authors describe a coverage–interval-score trade-off against ResCP rather than universal dominance.Source: §5 · p. 20

tion 3

  • The appendix states that detailed proofs are omitted from the preprint, while theorem and corollary statements and reported numerical results are unchanged.Source: tion 3 · p. 22

A.1 Cutoff bound: main argument

  • The cutoff bound transfers state–quantile relevance through local concentration and quantile regularity to the empirical order statistic, with a finite-k correction of at most 2/k.Very small matched sets can put the order statistic near the sample maximum, which local quantile behavior alone cannot control.Source: §A.1 · pp. 22–23

A.2 Predictive coverage: main argument

  • The predictive-coverage argument combines the cutoff bound with the state-information discrepancy and the continuity modulus of the score distribution.Source: §A.2 · p. 23

A.3 A prediction-error scale model implying the state assumptions

A prediction-error scale model gives a sufficient link between the constructed state and the target score quantile. Further sufficient conditions show how dependence, recurrence, validation selection, and the algorithmic procedure fit within the theory.

A.3 A prediction-error scale model implying the state assumptions

  • Under a common conditional law for scaled absolute errors, the score quantile is proportional to the state-dependent scale, yielding the required quantile regularity.This scale model is sufficient, not an additional requirement for the main theorem; serially correlated errors can still satisfy it when correlation does not change the conditional magnitude quantile beyond Zt.Source: §A.3 · pp. 23–24
  • When the conditional prediction-error scale is not represented by Zt, the full-history and state-indexed quantiles can differ, as in ARCH/GARCH-type models.In those models, the full-history quantile is proportional to current conditional volatility; this is the failure mode measured by the state-information discrepancy.Source: §A.3 · p. 24

A.4 A standard β-mixing route to uniform local concentration

  • A stationary β-mixing sequence with a uniform VC law and deviation ζn = op(k/n) provides one sufficient route to the required uniform local concentration.The lower bound k/n on empirical neighborhood mass converts uniform deviation into local concentration; the theorem itself requires the concentration condition, not β-mixing specifically.Source: §A.4 · p. 25

A.5 Primitive sufficient condition for a shrinking matched radius

  • A small-ball recurrence condition places order k historical states within radius of order (k/n)^(1/d), enabling the matched radius to shrink under the stated metric and concentration conditions.This is a sufficient setting, not a claim about every nonstationary sequence; the experiment’s fixed finite ρ-grids are not claimed to satisfy the asymptotic regime.Source: §A.5 · pp. 26–27

A.6 Finite-grid validation selection

  • Finite-grid validation selection inherits candidate-specific bounds through a union bound, provided the selected pair satisfies the other conditions of the relevant theorem.The algorithm splits data into training, validation, calibration, and test blocks, then applies the selected tuning pair to test targets.Source: §A.6 · p. 27, §A.7 · p. 28

B Additional Experimental Results

The additional experiments probe settings where state matching helps, adds little, or can lose to online or learned-retrieval methods, and report complete real-data results.

B Additional Experimental Results

  • State matching helps most when the constructed state captures recurrent score-relevant structure, adds little when recency already suffices, and can lose under abrupt switching.ACI-tuned selects its learning rate by mean validation interval score; at test start αt resets to 0.10, is clipped to [0.001, 0.30], and updates after each outcome.Source: B Additional Experimental Results · p. 28
  • The real-data sections provide complete station-level solar results and a common-target Bike Sharing analysis.The abstract reports that SMSCP improves interval score over recency, weighted, adaptive, and HopCPT comparators, with coverage closer to nominal 0.90 than ResCP on five of six benchmarks.Source: B Additional Experimental Results · p. 29

B.1 Complete core simulation results

The core simulations show the clearest SMSCP gains in recurrent periodic designs, while outcomes vary in other settings. The learned-retrieval comparison supports competitiveness when the state captures recurrent relevance, not universal dominance.

B.1 Complete core simulation results

  • All four HomoSin/DualFreq cells have lower SMSCP mean interval score than each of six non-neural comparators, with empirical coverage at or above nominal and narrower mean intervals.HeteroSin-small is mixed; HeteroSin-large favors SMSCP but has slight undercoverage, while abrupt-switching MarkovSwitch favors ECI in interval score.Source: §B.1 · p. 29

B.2 Direct learned-retrieval comparison

  • SMSCP is competitive with HopCPT when the state captures recurrent relevance, while abrupt switching can favor HopCPT in interval score.The comparison averages three fixed HopCPT initializations within each of 50 data realizations before paired inference.Source: §B.2 · pp. 29–30

B.3 Additional simulation designs

Additional simulations show that state matching has limited incremental value when recency or observed covariates provide useful information. A controlled comparison also finds that the constructed state is not universally better than matching on original features.

B.3 Additional simulation designs

  • In AR1, SMSCP is essentially tied with Tuned-Recent and ACI-tuned, while Rolling and Tuned-NexCP have slightly lower interval score; CovARX also favors several alternatives modestly.WindowAR remains close to recency and weighted alternatives, testing longer-memory lag structure alongside CovARX’s observed exogenous covariates.Source: §B.3 · p. 31

B.4 State-space versus feature-space matching

  • For WindowAR, Feature-k scores 2.6282 versus 2.6439 for State-k, with paired Feature-minus-State difference −0.0156 (95% CI [−0.0190, −0.0122]).The comparison holds the calibration pool, reference-set size, predictor, score, conformal order statistic, Mahalanobis metric, and covariance regularization fixed.Source: §B.4 · p. 32
  • The constructed state is one interpretable matching implementation, not a universal improvement over original features; its value depends on nearby states implying comparable target score quantiles.For CovARX, Feature-k’s mean interval score is 2.2043 versus 2.2205 for State-k.Source: §B.4 · pp. 32–33

B.5 Solar benchmark: detailed results

Across station–phase combinations, SMSCP has lower interval score than most comparators, while results still show station-specific coverage–width trade-offs. The benchmark uses chronologically ordered solar irradiance data and a pre-target local-linear level–slope state.

B.5 Solar benchmark: detailed results

  • The five-station benchmark uses chronological 30-minute NSRDB series, with 45,308 indices in the test tail after training, validation, and candidate-window blocks.Daylight targets are evaluated after restricting the rolling candidate window to past daylight points with valid states.Source: §B.5 · pp. 33–34
  • SMSCP uses the one-sided local-linear level–slope state, with history length h selected from {5, 7, 9, 11, 13} and tuning performed on validation data before test.The offset is standardized as in the general state construction.Source: §B.5 · p. 34
  • Across 20 station–phase combinations, SMSCP has lower interval score than Rolling, Recent-k, Tuned-Recent, and Tuned-NexCP in all 20, and than ECI in 16.The section notes station-specific coverage–width trade-offs rather than uniform interval narrowing.Source: §B.5 · p. 34
  • Table 8 reports station-level absolute results, with width and interval score measured in W/m2.Source: Table 8 · p. 35

B.6 Bike Sharing benchmark: detailed results

On the common Bike Sharing test targets, SMSCP has lower interval score than each of seven listed comparators, with the result direction unchanged under a 14-day block sensitivity analysis. The evaluation uses exact previous-day and previous-week inputs and reports substantial HopCPT initialization variability.

B.6 Bike Sharing benchmark: detailed results

  • Bike Sharing preserves the observed 17,379-row sequence despite 75 gaps longer than one hour, and state lags use exact elapsed calendar time.The maximum adjacent-timestamp gap is 37 hours; rows are neither inserted nor deleted.Source: §B.6 · p. 35
  • The chronological split reserves 2,160 observations for training history, 2,160 for validation, and 8,760 for calibration before the test tail.All methods are compared on the same 4,169 valid SMSCP test targets.Source: §B.6 · pp. 35, 37
  • The state uses exact previous-day and previous-week rental counts, fitted as a three-dimensional local-linear state from information available before the target.Missing daily or weekly lags are not imputed.Source: §B.6 · p. 36
  • All seven comparator-minus-SMSCP interval-score differences have strictly positive paired 7-day bootstrap intervals on the 4,169 common test targets.The 14-day block-length sensitivity analysis leaves the point estimates unchanged, and all seven comparator intervals remain strictly positive.Source: Table 11 · p. 37, §B.6 · p. 38, Table 14 · p. 38
  • Paired time-series inference averages HopCPT across 50 initializations within each target, then uses 500 moving-block-bootstrap replications with seven-calendar-day blocks.A 14-day sensitivity analysis uses 3,859 admissible starting blocks containing 217–336 common targets.Source: §B.6 · pp. 37–38
  • Across 50 HopCPT initializations, mean coverage was 0.870 (SD 0.029) and mean interval score was 467.7 (median 443.0; SD 85.3); 8 reached at least 0.90 coverage.Sixteen initializations had lower interval score than SMSCP and 34 had higher interval score; these summaries describe initialization variability, not time-series sampling confidence intervals.Source: §B.6 · p. 38

B.7 ResCP real-data comparison

On five of six benchmarks, SMSCP’s empirical coverage point estimate is closer to nominal 0.90, but ResCP has lower mean interval score on all six. The appendix also documents limits on interpreting computational diagnostics and specifies the simulation setup.

B.7 ResCP real-data comparison

  • The real-data ResCP comparison aligns the frozen point predictor, chronological outer candidate window, and evaluation targets, but retains ResCP’s reservoir sampling, signed residuals, and asymmetric intervals.Solar profile selection uses three prespecified configurations on validation data; Bike uses the generic configuration without dataset-specific profile search.Source: §B.7 · p. 39
  • ResCP has lower mean interval score on all six benchmarks, while SMSCP coverage is closer to 0.90 at five and ResCP is slightly closer at Phoenix.The coverage comparison is descriptive, not a formal test of calibration error, and higher empirical coverage is not treated as inherently preferable.Source: §B.7 · p. 40
  • For Bike, the 14-day block sensitivity gives ∆IS = −167.1 with 95% interval [−207.5, −141.7], compared with [−199.0, −146.3] under 7-day blocks.Changing block length changes the bootstrap interval, not the point estimate.Source: §B.7 · p. 40

B.8 Computational diagnostics

  • The execution timers are not controlled cross-method hardware benchmarks, so they document neural-training burden rather than establish a universal runtime ranking.HopCPT times are per neural initialization, whereas SMSCP times are median wall time for its full path per data realization.Source: §B.8 · p. 40, Table 16 · p. 40

C Data-Generating Processes

  • The simulation designs use a rolling lag-one-correlation predictor based on the 20 most recent responses and absolute prediction error as the score.The core designs use small and large splits, while additional designs are evaluated only under the large split.Source: C Data-Generating Processes · p. 41

C.1 Core simulation designs

The core simulations vary temporal structure and score-scale behavior while constructing one-sided local-linear states from pre-target inputs. Additional designs use lagged responses or exogenous covariates as state inputs.

C.1 Core simulation designs

  • Core time-index designs summarize the preceding h observations with a one-sided local-linear level–slope state, using chronological time as the input.Source: §C.1 · p. 41
  • DualFreq combines two sinusoidal components with noise and uses the same strictly pre-target time-index state as HomoSin.The unit-spaced time parameterization is used at both sample sizes.Source: §C.1 · p. 41
  • In HeteroSin, response scale varies with phase, while the time-index state built from local level and slope captures score-scale variation only partially.Source: §C.1 · p. 41
  • ARCH(1) tests a case where the level–slope state may miss volatility information relevant to the absolute prediction-error quantile.Its state is constructed by locally regressing the raw response on a pre-target volatility-related input.Source: §C.1 · p. 42
  • MarkovSwitch uses a two-state Markov chain with 0.95 probability of remaining in the same state, creating abrupt transitions as a boundary case for smoothly recurring-state matching.Source: §C.1 · p. 42

C.2 Additional simulation designs

  • An additional design constructs a three-dimensional state from centered lagged-response offsets scaled by their local empirical standard deviation.The corresponding input pair is (Yt−5, Yt−6), with a numerical floor of 10−4 for the local standard deviations.Source: §C.2 · pp. 42–43
  • CovARX uses observed pre-target exogenous inputs (X1,t, X2,t) to fit a three-dimensional state.The local scaling and numerical floor follow the preceding specification.Source: §C.2 · p. 43

Related papers

Loading 2610.19484v1…