PaperlayerView exact revision on arXiv

Source-linked AI summary

CALQA: Conformal Screening of Forecast-Ensemble Configurations from One Mergeable State

Xiufeng Liu, Ruyu Liu, Yusen Liu, Huan Huo, Per Sieverts Nielsen

arXiv:2610.13249v1cs.DB

TL;DR

Exact calibration of post-hoc ensemble weights scores a group's rows for each candidate, making large configuration searches costly. CALQA stores mergeable block-score histograms over a weight net and screens candidates without reading rows. Its two-stage search nearly matches exact search on held-out blocks at substantially lower time, while coverage guarantees depend on how the selected weights are deployed.

  • Problem

    Exact split-conformal calibration scores every calibration row for each candidate weight, while existing summaries or stored counts do not support unseen continuous weights with the needed group and window queries.

    Source: §III · p. 3

  • Method

    CALQA builds a fixed-size integer aggregate of block-score histograms at net weight vectors, merges or subtracts states, and uses the result to screen candidates before exact re-checks.

    Source: §I · pp. 1–2, §VII · p. 12

  • Results

    On five public panels, the two-stage plan matched exact search at 99.95% of its held-out block-Winkler score and ran 1.3–59 times faster; on two real panels it was 44–54 times faster at 99.98–100.01% of exact search's held-out score.

    Source: §I · p. 2

  • Takeaways & Limitations

    CALQA is a screen that ranks candidates for exact re-checking, and its state saves time rather than memory.

    Source: §VII · p. 12

    Coverage for a data-selected weight vector is stated for i.i.d. blocks with union-bound slack when deployed with the state's shift; a pick deployed with its exact shift is not covered by Corollary 2.

    Source: C. Coverage for Each Post-Hoc Weight… · p. 6

Abstract

from arXiv · show

AutoML systems fit ensemble weights over model libraries; a search can score, per category, market or time pane, thousands of weightings of the quantile-averaged (Vincentized) ensemble for the narrowest conformally calibrated interval; exact calibration scores a group's rows per candidate. We present CALQA, a fixed-size integer aggregate per group and pane that stores histograms of normalised block scores at a net of weight vectors. One GROUP BY builds it; it merges by addition, retracts deletions by subtraction and scores any candidate without reading a row. We prove that its score never falls below the exact split-conformal shift and exceeds it by at most the net distance plus a grid gap; that coverage carries over to each fixed weight vector and, for i.i.d. blocks and up to a union-bound slack, to a searched weight vector deployed with the state's shift; and how small any such state can be. The state is thus a screen: it ranks all candidates, and exact recomputation re-checks the top 100. On five public panels (604 groups, 10,000 candidates each) this two-stage plan matches exact search on held-out blocks (99.95% of its Winkler score), 1.3-59 times faster. On two real panels of about $10^8$ rows (smart meters, retail sales) it is 44-54 times faster at 99.98-100.01% of that score and beats sample screens with the same re-check in regret at less total time. Picks deployed with their exact shift need a held-out coverage check. The state does not pay off for small or greedy searches, off-net weights at ample memory or full twelve-member support.

I. INTRODUCTION

Post-hoc searches over quantile-averaged forecast ensembles need group-level conformal calibration, but exact shifts require scoring calibration rows for every candidate. CALQA stores block-score histograms at weight-net points, enabling mergeable screening with coverage guarantees and exact re-checks for leading candidates.

I. INTRODUCTION

  • Row-level coverage can overstate block coverage: on electricity-price panels, uncalibrated intervals covered 0.813–0.823 of rows but only 0.296–0.334 of daily blocks.Exact calibration scores each group's held-out calibration rows for every candidate, so searching M candidates across G groups can require GM passes over rows.Source: §I · p. 1
  • CALQA stores histograms of normalised block scores at a finite net of weight vectors and scores candidates using a net threshold plus half their ℓ1 distance to that net point.The state is built with one GROUP BY, merges by addition, and retracts deleted entities by subtraction.Source: §I · pp. 1–2
  • On 604 public groups, top-100 re-check matched exact search at 99.95% of its held-out block-Winkler score, 1.3–59 times faster; two real panels ran 44–54 times faster at 99.98–100.01%.The real panels contained about 10^8 rows each; the comparison was against exact search.Source: §I · p. 2
  • The state shift never falls below the exact split-conformal shift and exceeds it by at most the net distance plus a grid gap, making it a safe ranking screen before row-level re-checks.The state ranks candidates; exact recomputation then needs to re-check only the leading candidates.Source: §I · p. 2
  • For each fixed weight vector, CALQA's shift preserves the weighted conformal coverage bound; for data-selected weights, coverage holds for i.i.d. blocks with a union-bound slack.The guarantee applies when the selected vector is deployed with the state's shift; picks deployed with their exact shift are not covered by this result.Source: §I · p. 2
  • The paper also gives a merge-closure lower bound for deterministic dominating states, with order ϵ^−(1+d/2) coordinates on a d-dimensional simplex face when the shift window is sufficiently long.For shorter shift windows, the stated order is ϵ^−d/2.Source: §I · p. 2

II. RELATED WORK

Prior work addresses forecast combination, conformal calibration, and mergeable data summaries, but does not index calibration summaries by a continuous ensemble weight vector. The paper frames the resulting what-if query as returning conservative, tight shifts for unseen weights over unions of cells at cost independent of their row counts.

II. RELATED WORK

  • Quantile averaging can be sharper than linear pooling, yet averaged quantiles may be overconfident; fitted corrections require refitting per configuration, while the Bates–Granger formula predicts error variance rather than coverage.Prior conformal methods calibrate a fixed predictor or aggregate scores over a weight grid or function class with a uniform slack.Source: §II · p. 2
  • Mergeable quantile sketches and federated conformal summaries cover one score stream or model, while data-cube aggregates support updates but stored per-configuration counts require configurations declared in advance.The paper's target is post-hoc weights of one aggregate, including roll-ups, complements, and sliding windows.Source: §II · p. 3

III. PRELIMINARIES AND PROBLEM

  • A block is a nonempty sequence of rows, and its score is the selected order statistic of row scores; with the all-rows rule, a block is covered only when every row is covered.The exact shift is a weighted score threshold; with unit weights it is the split-conformal order statistic.Source: §III · p. 3
  • The workload searches candidate weights for the minimum mean normalised interval width plus twice the conformal shift; exact evaluation scores every calibration row for every candidate.The stated query covers unions of cells and weight vectors unknown at ingest, with cost that grows with cell count rather than rows held.Source: §III · pp. 3–4

IV. THE CALQA STATE

CALQA's net-score state stores integer histograms of block scores at fixed net points and queries those histograms to screen weights not known at ingest. Its construction supports conservative thresholds, bounded excess, and exact merging or subtraction.

A. State and Query

  • The state stores per-net-point block-score histogram counts over grid bins, plus a total-weight entry; the query finds each net point's threshold and returns the minimum threshold plus half the ℓ1 distance.A finite grid rounds each net-point threshold up to its next grid value, accounting for the grid-gap term.Source: A. State and Query · p. 4
  • For a DE quarter with 54 calibration days, 84 net points and 65 bins require 84 · 65 + 1 integers, or 21,844 bytes, regardless of the number of days.The query returns an excess of at most 1/3 plus a grid gap for N6; merging quarters adds their histograms.Source: A. State and Query · p. 4
  • Under the scale rule, row and block scores change by at most half the ℓ1 weight distance; conservative quantisation rounds a weighted score threshold up to a grid point.These properties underlie the net-distance and grid-gap terms in the state guarantee.Source: A. State and Query · p. 4
  • Theorem 1 bounds every queried shift between the exact shift and that shift plus the nearest-net distance and grid gap; the state also merges by addition and retracts subsets by subtraction.The state size depends on the net and grid, not on the number of blocks, block sizes, or queried weights.Source: A. State and Query · p. 4

B. SQL Realisation

The SQL realization builds CALQA from rows using net-point scores, block aggregation, grid bins, and grouped counts. The resulting screen ranks candidates, and row-based re-checks refine the leading candidates.

B. SQL Realisation

  • SQL builds the state by scoring rows at net points, taking block scores, assigning grid bins, and counting by group, pane, net point, and bin.Roll-ups sum cell states; complements subtract a group from the total, and deletes subtract the deleted blocks' increments.Source: B. SQL Realisation · p. 5
  • The query costs O(|C||N|J + M|N|K) without touching rows, versus O(nrows|N|K) for building the state.With 4-byte counts, the state uses 4(|N|(J + 1) + 1) bytes; K = 4, g = 6, and J = 64 require 21,844 bytes.Source: B. SQL Realisation · p. 5
  • The screen ranks candidates by an upper-bound score, then exactly re-checks the k highest-ranked candidates; the two-stage pick is no worse than the state-only pick with exact re-checks.A returned pick deployed with its exact shift is not covered by Corollary 2.Source: B. SQL Realisation · p. 5

C. Coverage for Each Post-Hoc Weight Vector

Under B0–B3, the coverage bound for each fixed post-hoc weight vector is controlled by total-variation gaps; with exchangeable blocks, those gaps vanish and coverage is at least 1−α.

C. Coverage for Each Post-Hoc Weight Vector

  • The bounds assume B0–B3, including setup fixed independently of block values and weights fixed or independent of the test data.Theorem 2 also permits pane or decayed weights through the state under its stated conditions.Source: C. Coverage for Each Post-Hoc Weight… · pp. 5–6
  • For every fixed w, coverage is at least 1−α−G_w, 1−α−G_A, and 1−α−G_Z, with these bounds ordered as stated.Theorem 2 applies to w on or off the net.Source: C. Coverage for Each Post-Hoc Weight… · p. 6
  • Under exchangeable blocks, coverage is at least 1−α for each w in the simplex and all weights satisfying B1.With λ = 1, this is the split-conformal guarantee for each post-hoc w.Source: C. Coverage for Each Post-Hoc Weight… · p. 6
  • The coverage gaps arise through weighted-swap bounds and maps of block scores; under exchangeability, every gap is zero.The score-sufficient image is unchanged by the stated affine transformations when the scale floor does not bind.Source: C. Coverage for Each Post-Hoc Weight… · p. 6
  • For i.i.d. blocks, data-selected weights can be covered using a union-bound slack over the net; at stated calibration sizes, the slack ranges from 0.016 (M4 Daily) to 0.129 (DE).For the stated 0.1 target, the rank is finite only for n ≥406 (427 for |N| = 126).Source: C. Coverage for Each Post-Hoc Weight… · p. 6

V. STATE SIZE: OPTIMAL IN EXPONENT WITHIN THE NET-HISTOGRAM CLASS

The section derives lower bounds for mergeable states that safely approximate conformal shifts, then shows that net-histogram states match their own lower bound in exponent. The resulting coordinate costs make full-support net histograms impractical at K=12, while restricted pair or triple families remain possible.

V. STATE SIZE: OPTIMAL IN EXPONENT WITHIN THE NET-HISTOGRAM CLASS

  • Merge closure forces valid, tight states to recover threshold counts up to an ϵ-shift, enabling a fuzzy-boundary counting lower-bound argument.The section applies this to deterministic states that dominate the exact shift; coverage-valid samples and randomized summaries are outside that class.Source: §V · pp. 6–7
  • For one weight vector, a one-point net histogram with grid spacing ϵ uses ⌈B/ϵ⌉+2 coordinates and is ϵ-tight.Theorem 4(c) shows this grid size is necessary for the net-histogram class.Source: §V · p. 7
  • For net-histogram states on one d-face, the required coordinate count is Θ_d(Bϵ^-(1+d)), matching the construction’s exponent.The upper bound uses a lattice net and uniform grid with r+h≤ϵ; the lower bound forces an ϵ-covering net and sufficiently fine grid.Source: §V · p. 7
  • For general states, the lower-bound exponent is 1+d/2, leaving a d/2 exponent gap against the net-histogram class’s 1+d.Whether a mergeable state outside the net-histogram class closes this gap remains open.Source: §V · p. 7
  • For K=12 and ϵ=0.1, the net-histogram class requires 878,906,251 coordinates for full support, versus 3,565 for pairs and 15,841 for triples.The paper concludes that full support is out of reach for this class, while the corresponding 5,631-coordinate general integer-additive state is not excluded.Source: §V · p. 7

VI. EXPERIMENTS

The experiments evaluate CALQA’s search quality, efficiency, and operating boundaries across public and real forecasting panels. They also examine memory regimes, scaling, temporal behavior, larger model pools, and deletion.

VI. EXPERIMENTS

  • The evaluation covers end-to-end search quality, equal-byte summary comparisons, build/merge/query costs, forward-in-time behavior, twelve-member families, and deletion.It includes two real panels of about 10^8 rows and compares random, realistic, declared, and neighbourhood weights at equal bytes.Source: §VI · p. 7
  • Table III lists the forecasting panels and member subsets used in the experiments.The panels include M4, five day-ahead price markets, and the German day-ahead panel; the member pools use subsets K=4 and K=6, with twelve members also studied.Source: Table III · p. 8
  • The section’s closing Table XI condenses the experimental findings into guidance on when to use the state and when not to.The experiments include twelve-member families and deletion tests.Source: §VI · pp. 7–8

A. Experimental Setup

The experiments use public forecasting panels with fixed model pools, block-based calibration and test splits, and multiple candidate-search strategies. The two-stage CALQA plan re-checks screened candidates exactly, achieving near-exact held-out performance with lower runtime in the reported search comparisons.

A. Experimental Setup

  • The panels use twelve-member pools with 19 quantile levels, while the main experiments use K=4 and K=6 subsets and block coverage as the target.M4 blocks are forecast origins; EPF and DE blocks are delivery days, and the pools contain no neural or foundation models.Source: A. Experimental Setup · p. 8
  • The setup uses N6 for K=4 and N4 for K=6, with 64 score-grid points placed using pilot blocks excluded from calibration and test.Baselines include exact per-configuration conformal search, samples, stored configuration counts, and PAC variants; timings use a 2-core virtual machine.Source: A. Experimental Setup · p. 8
  • Across 84 panel × K × n cells, the net state’s block coverage is 0.908–0.974 and exact per-w conformal’s is 0.887–0.910.Random block splits make blocks exchangeable, so these coverage results check the implementation; no shift falls below the exact one.Source: A. Experimental Setup · p. 8

B. End-to-End Configuration Search

  • 99.95% of exact search’s held-out block-Winkler score is achieved by the two-stage plan, which runs 1.3–59 times faster across families.The comparison covers 604 groups and 10,000 candidates per group; the state alone reaches 100.41% of exact search’s score.Source: Table IV · p. 8, B. End-to-End Configuration Search · p. 9
  • On the largest M4 Daily group, screening 10^4 candidates takes 42 ms and the two-stage plan 118 ms, versus 6.9 s for exact float16 search.The state build is paid once; in DuckDB, screening is 13–2,301 times faster than exact SQL, with identical NumPy picks in every group.Source: B. End-to-End Configuration Search · p. 9
  • With the exact shift, searched picks cover 0.84–0.90 of held-out blocks, below the 0.9 target; the authors recommend a held-out coverage check.With the state’s shift, its picks cover 0.88–0.92, but the stated i.i.d.-block rank correction is finite only for n≥406 or n≥427, depending on net size.Source: B. End-to-End Configuration Search · p. 9

C. At Scale: Smart-Meter and Retail Panels of 108 Rows

On the CER and M5 panels, the top-100 re-check nearly matches exact search while taking substantially less time. Across memory budgets, weight sets and query workloads, CALQA’s advantage depends on the regime and the number of configurations queried.

C. At Scale: Smart-Meter and Retail Panels of 108 Rows

D. Equal-Bytes Regime Map and Nets

  • At 4 and 8 KB, the net state is the narrowest reliable summary in 9 of 10 settings and tied in the other; from 128 KB onward, it loses in all 10.At 16 KB it is best or tied everywhere; at 32–64 KB it mostly ties or loses as its net-inflation floor remains above samples approaching exact width.Source: Table VI · p. 10, D. Equal-Bytes Regime Map and Nets · p. 10

E. Cost of Build, Merge and Query

  • At M = 10^4, SQL state queries cost 3.89–6.82 µs per configuration, 413–4,039 times less than SQL exact recomputation.The state query stays within a factor of two from one to 10^3 configurations; at M = 1, engine overhead dominates.Source: Figure 4 · p. 11, E. Cost of Build, Merge and Query · p. 11

D. Equal-Bytes Regime Map and Nets

  • Off-net stacking weights and drifting refits favor the equal-bytes sample in at least 7 of 10 settings, while declared net weights favor the state.The top-two subsets favor the state in 10 of 10 settings; declared stacking weights do so in 10 of 10 once added as net points.Source: Table VII · p. 10, D. Equal-Bytes Regime Map and Nets · p. 11

E. Cost of Build, Merge and Query

  • For many-group and pane queries, a second cube level removes the one-block-cell roll-up disadvantage, and every state, merge and query matches across engines.The dense state becomes smaller than rows only from tens of blocks per cell.Source: Table VIII · p. 11, E. Cost of Build, Merge and Query · p. 11
  • On real CER data, 104 configurations take 0.07 s from the NumPy state versus 163 s for vectorised float16 recomputation; a six-month window slides by merge and retraction in 6 ms versus 2.95 s to rebuild.The CER state is built from 97,920,480 non-pilot rows in 3.97 s by NumPy.Source: E. Cost of Build, Merge and Query · p. 11

F. Forward in Time

Forward-in-time evaluation finds that decay can improve the share of week–weight cases meeting a coverage threshold, with a smaller effective sample. Twelve-member support remains costly at full support, while subtraction makes entity deletions exact; the paper concludes that CALQA is a screen with practical scope limits.

F. Forward in Time

  • On 406 forward test steps, the net state covers 0.908–0.934 of test blocks, compared with 0.885–0.906 for exact per-weight conformal; no theorem certifies these measurements.The decay selected during the first 203 steps was λ = 1 in all four settings.Source: F. Forward in Time · pp. 11–12, Table IX · p. 12

G. Twelve Members and Deletion

  • For twelve-member families, the width ratio on full support remains 1.385–1.774 even at 346.6 KB, whereas on pairs it falls to 1.027–1.045 at 12 members.At equal bytes, the block sample is narrower in 22 of 25 restricted panel–family cells.Source: Table X · p. 12, G. Twelve Members and Deletion · p. 12

F. Forward in Time

  • Using λ = 0.95 lowers the share of week–weight cells below 0.8 coverage from 0.021–0.162 to 0.007–0.059, at an effective sample size of 39–195 blocks.Across 1,333,344 checks, no decayed or unweighted net shift fell below the exact weighted shift; the decayed-state check is empirical because it uses floating point.Source: F. Forward in Time · p. 12

G. Twelve Members and Deletion

  • Subtracting deleted entities gives exactly the remainder’s state, and its width ratio moves by at most 0.027; a pre-built sample can only drop deleted blocks without rescanning.At 16 KB the best sample is wider than the state at every deletion fraction in three settings, and from ϕ = 0.25 or 0.5 on DE and M4 Daily.Source: Figure 5 · p. 12, G. Twelve Members and Deletion · p. 12

VII. CONCLUSION

  • CALQA saves time rather than memory, and the paper identifies adaptive recalibration under drift, the lower-bound exponent gap and randomized summaries as open problems.The conclusion says the state does not pay off for small or greedy searches, off-net weights at ample memory, or full twelve-member support.Source: §VII · p. 12

APPENDIX A PROOFS

The appendix proves the score’s Lipschitz transfer and conservative quantisation, then uses those properties to establish state validity, mergeability and coverage guarantees. Its coverage argument transfers weighted conformal bounds through blockwise transformations and an exchangeability-based invariance step.

A.1 Proof of Lemma 1 (Lipschitz transfer)

  • The block score changes by at most half the ℓ1 distance between weight vectors after normalisation by a scale dominating member spread.The order statistic inherits the same perturbation bound.Source: §A.1 · p. 13

A.2 Proof of Lemma 2 (weighted conservative quantisation)

  • Weighted cumulative counts are nondecreasing and right-continuous, so the first threshold reaching the required mass is a score jump point; grid rounding adds its grid gap.If the required mass exceeds total weight, both threshold sets are empty.Source: §A.2 · p. 13

A.3 Proof of Theorem 1 (state)

  • The state threshold dominates the exact shift, while its excess is bounded by the net distance plus the grid gap.Per-block histogram increments add associatively, supporting merge, complement, decay and exact integer aggregation.Source: §A.3 · p. 13

A.4 Proof of Theorem 2 and Corollary 1 (coverage)

  • For fixed weights, pointwise dominance reduces miscoverage under the state shift to the weighted conformal swap bound, with blockwise score transformations preserving that bound.The proof uses the threshold event and total-variation contraction under measurable transformations.Source: §A.4 · p. 13
  • The coverage argument is invariant to a common outcome shift and positive scale, and exchangeability makes the corresponding gaps vanish.Source: §A.4 · p. 14

A.5 Proof of Corollary 2 (data-selected w)

The appendix bounds coverage for a searched weight vector using a uniform empirical-CDF deviation over the net. It also proves merge-closure lower bounds by constructing row multisets whose states must remain distinguishable.

A.5 Proof of Corollary 2 (data-selected w)

  • For i.i.d. blocks, a net-wide DKW bound controls the coverage of a data-selected weight, with a union-bound slack depending on net size and calibration count.The proof first selects the net point attaining the state threshold and applies the empirical-CDF deviation bound.Source: §A.5 · p. 14

A.6 Proof of Lemma 3 (gadgets and merge closure)

  • Merge closure forces distinct multisets with different counts below a threshold to have distinct states when the state is valid and tight.The proof adds common low- and high-score rows to construct equal-size inputs whose correct shifts differ beyond the tightness allowance.Source: §A.6 · p. 14

A.7 Proof of Theorem 3 (general lower bound)

  • The general lower bound constructs a tangent-plane lattice whose threshold-incidence matrix has identity diagonal blocks, then uses injectivity to bound the state dimension.The lattice parity argument separates thresholds, and a kernel-vector contradiction yields the dimension bound.Source: §A.7 · p. 14

A.8 Proof of Lemma 4 (support-preserving rounding)

The proof rounds weights to a denominator-g grid while preserving their support, then bounds the resulting L1 error and its worst case over support size.

A.8 Proof of Lemma 4 (support-preserving rounding)

  • Rounding up the r largest fractional parts produces a denominator-g vector u that sums to one and has support contained in the original support F.When r=0, the rounded vector equals w.Source: §A.8 · p. 14
  • The rounding error satisfies ∥w − u∥1 ≤ 2r(p − r)/(pg), using that the r largest fractional parts sum to at least r^2/p.The bound follows from a lower bound on the sum of the selected fractional parts.Source: §A.8 · pp. 14–15
  • The error bound is maximized at r = ⌊p/2⌋, giving p/(2g) for even p and (p^2 − 1)/(2pg) for odd p.Both parity-specific maxima increase with p.Source: §A.8 · p. 15

A.9 Proof of Theorem 4 (net histograms)

The proof establishes bounds for the net-histogram screen by constructing instances that expose distance and grid-gap terms, and by bounding net size and sparse-support approximation error.

A.9 Proof of Theorem 4 (net histograms)

  • The proof constructs instances in which the score includes a distance-to-net term and, separately, a grid-gap term γ_G(x).For identical rows, it gives r*_m = x and t_m = x + γ_G(x).Source: §A.9 · p. 15
  • For weights supported on at most p coordinates, rounding to the net moves block scores by at most r/2 and yields the stated score sandwich when (J − 1)h ≥ B + r.With g = ⌈(d + 1)/ϵ⌉, the proof bounds r by d + 1.Source: §A.9 · p. 15
  • The covering argument gives |N|(2ϵ)^d ≥ 1, while disjoint neighborhoods around equal-size faces yield a stronger lower bound on net size.The state has |N|(|G| + 1) + 1 coordinates.Source: §A.9 · p. 15
  • The resulting state-size bound is Θ_d(Bϵ^−(1+d)) under the proof's choices of g and J.The proof uses |N| ≤ (2(d + 1)/ϵ)^d/d! and J ≤ 5B/ϵ.Source: §A.9 · p. 15

Related papers

Loading 2610.13249v1…