PaperlayerView exact revision on arXiv

Source-linked AI summary

Approximate Design-Based Intervals for Downsampled Cross-Sectional Market Aggregates: A Randomized Design for Bandwidth-Constrained Financial Data Pipelines

Minmin Zeng

arXiv:2610.08801v1q-fin.RMcs.CE

TL;DR

Target-blind Top-k does not identify the OI-weighted aggregate from retained outcomes alone without outcome bounds, limiting error quantification for downsampled panels. The paper replaces deterministic retention with probability sampling and uses Hájek estimation and design-based variance methods to construct approximate intervals. At 10% retention in Hong Kong, Top-k has lower MAE than Poisson–Hájek (1.33 vs. 2.45), but its error is not auditable from retained data alone.

  • Problem

    Target-blind Top-k does not identify the OI-weighted aggregate from retained outcomes alone without outcome bounds, limiting error quantification for downsampled panels.

    Source: §1 · p. 2

  • Method

    The paper replaces deterministic retention with probability sampling and uses Hájek estimation and design-based variance methods to construct approximate intervals.

    Source: §1 · p. 2, §3.2 · p. 7

  • Results

    At 10% retention in Hong Kong, Top-k has lower MAE than Poisson–Hájek (1.33 vs. 2.45), but its error is not auditable from retained data alone.

    Source: §7 · p. 19

  • Takeaways & Limitations

    Inclusion probabilities should match the target’s influence function, and a single OI-proportional subsample is not optimal for all aggregates.

    Source: §7 · p. 19

    The intervals are first-order approximations without finite-sample or exact asymptotic coverage guarantees, and the reported variance construction applies to Poisson sampling.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Financial institutions routinely downsample cross-sectional options panels to meet bandwidth and cost constraints. The industry default---deterministic Top-$k$ selection by open interest---provides no estimate of the aggregation error it introduces, rendering downstream risk metrics unauditable. We make the identification problem precise. For deterministic rules whose selected set depends only on auxiliary variables and not on the outcome (target-blind rules, of which Top-$k$-by-open-interest is one), the target is not identified from the retained outcomes alone unless bounds or modelling assumptions are imposed; under outcome bounds it is partially identified, with a sharp worst-case interval we characterise in closed form. We propose replacing Top-$k$ with probability sampling whose inclusion probabilities are proportional to open interest, paired with the Hajek ratio estimator and a linearised design-based variance estimator, yielding approximate design-based intervals around every aggregate. Experiments on 80 US trading sessions (211 tickers/day) and 83 sessions of Hong Kong options (131 tickers/day) show that for OI-weighted implied volatility the randomized design reduces mean absolute error by 35% (MAE 5.31 vs. 8.15 at 10% retention; paired $t=6.68$, Cohen's $d=0.75$, moving-block-bootstrap 95% CI $[1.82, 3.96]$), and the sharp worst-case interval available to Top-$k$ is 2.7--6.4$\times$ wider than the sampling interval under oracle bounds, and 9--26$\times$ wider under feasible bounds. Empirical coverage is 91.9% against a nominal 95%. We further document that inclusion probabilities must match the target's influence function, motivating a design-aware framework for production pipelines.

1 Introduction

The paper formalizes why target-blind Top-k downsampling leaves aggregate error unauditable and proposes probability sampling with design-based intervals. It connects survey-sampling methods to financial pipelines and shows that accuracy depends on the market and the aggregate being estimated.

1 Introduction

  • The study targets a second compression stage, reducing an approximately 210-row per-underlying panel to about 21 rows at 10% retention.Intraday refreshes, multiple markets, terminals, and downstream models multiply pipeline volume; per-record licensing can add an incentive to compress.Source: Table 1 · p. 2, §1 · p. 2
  • Target-blind Top-k does not identify the OI-weighted target without outcome bounds; with bounds, its sharp worst-case interval is 2.7–6.4 × wider than the sampling interval.The two intervals are not interchangeable: the former is worst-case valid, while the latter is only nominally 95%.Source: §1 · pp. 2–3
  • At 10% retention, randomized sampling reduces US OI-weighted implied-volatility MAE by 35% versus Top-k (5.31 vs. 8.15), while Top-k performs better in Hong Kong at low-to-moderate retention (1.32 vs. 2.42).The Hong Kong ranking reverses only at high retention; the paper traces the reversal to omitted open-interest share and the retained-minus-omitted outcome gap.Source: §1 · p. 3
  • Inclusion probabilities should track the target’s influence function: OI-proportional sampling suits the OI-weighted target, while other aggregates call for different designs.The equal-weight mean and dispersion have influence-proportional oracle optima; uniform sampling is a feasible outcome-blind proxy for the mean and is near-optimal for the median.Source: §1 · p. 3

Related work.

  • The paper applies classical survey-sampling estimators and design-based variance methods to financial pipelines, where deterministic Top-k is the default.It relates the identification issue to deterministic eviction in randomized KV-cache work [2] and distinguishes its financial estimators, optimal-design theory, and multi-market evidence.Source: Related work · pp. 3–4

2 Problem Formulation

The paper studies estimating finite-panel aggregates after a pipeline retains only a subset of instruments, with full auxiliary information but outcomes observed only for retained units. For target-blind deterministic selection, retained outcomes alone may not identify the target; outcome bounds instead yield a sharp worst-case interval.

2.1 Setting

  • The downstream system observes retained outcomes and auxiliary information, then estimates a scalar aggregate of the full panel.The paper studies OI-weighted and equal-weight means, dispersion, and the median.Source: §2.1 · p. 4
  • The comparison conditions on the full auxiliary vector and population size; the unobserved quantities are the outcomes for omitted units.This gives Top-k the auxiliary information it needs to rank open interest.Source: §2.1 · p. 4
  • A design-based interval takes coverage over the sampling design while holding the finite population fixed.The proposed intervals are first-order linearisation normal approximations, not exact intervals or error certificates; measured coverage is 89–94% at nominal 95%.Source: §2.1 · p. 4

2.2 Target-blind deterministic selection

  • A deterministic rule is target-blind when its retained set depends only on auxiliary variables, not on the outcomes.Top-k by open interest, round-robin, sector-quota, and liquidity-threshold rules are examples.Source: §2.2 · p. 4
  • Without bounds on omitted outcomes, target-blind selection leaves the OI-weighted target and equal-weight mean with an identified set equal to all real numbers.The result also extends to dispersion and, under a count condition, the median.Source: §2.2 · p. 5
  • Determinism alone does not preclude finite bounds: retaining the observed minimum and maximum ensures omitted outcomes lie between them.The impossibility result depends on target-blindness together with the absence of a priori outcome bounds.Source: §2.2 · p. 5

2.3 Partial identification under bounds

  • With outcomes constrained to [a, b], Top-k has a sharp worst-case interval whose width is (b − a)W_Sc/W, the omitted share of auxiliary mass times the outcome range.Both endpoints are attained, and the width depends on the auxiliary mass left behind.Source: §2.3 · p. 5

3 Method

The method matches probability-sampling designs to a common expected budget, estimates aggregates with design-based estimators, and constructs approximate intervals. It also derives how inclusion probabilities should align with each target’s influence function.

3.1 Probability designs with a matched budget

  • The water-filled Poisson PPS schedule sets π_i = min(1, λw_i) so expected sample size equals the target budget n, including when probabilities are capped at one.The paper also evaluates fixed-size systematic PPS with the same first-order inclusion probabilities; it draws exactly n records but Poisson sampling does not enforce a hard cap.Source: §3.1 · pp. 5–6

3.2 Estimators and design variances

  • For the OI-weighted ratio target, the method uses the Hájek estimator and a first-order linearised design-variance estimator.The interval is estimate ± z_0.975 times the estimated standard error, using a first-order normal approximation rather than guaranteed coverage.Source: §3.2 · pp. 6–7
  • For the equal-weight mean, the method uses a Hájek mean, while the median is estimated by an inverse-probability-weighted quantile.The median’s design-matched multiplier approximation perturbs design weights and uses the standard deviation across replicates as its standard error.Source: §3.2 · p. 6
  • The variance construction in Equations (5)–(6) applies to independent Poisson sampling; systematic PPS needs joint-inclusion information or replication-based variance estimation.Uniform Bernoulli and stratified designs also have standard design-based variance estimators, but are not tabulated.Source: §3.2 · pp. 6–7

3.3 Matching the design to the target

  • Under Poisson sampling, the first-order variance-optimal allocation makes inclusion probabilities proportional to the absolute influence function, subject to probability bounds.A positive inclusion floor prevents zero-probability units; the constrained solution uses two-sided water-filling, with probabilities floored or capped where necessary.Source: §3.3 · p. 7
  • OI-proportional sampling aligns with the OI-weighted target, whereas the equal-weight mean’s oracle optimum is influence-proportional and uniform sampling is a feasible outcome-blind alternative.The oracle allocation depends on the unknown target or outcomes, so it is not directly feasible in a production pipeline.Source: §3.3 · p. 7
  • The median’s influence is approximately constant in magnitude, while dispersion’s varies with distance from the cross-sectional mean; their suitable designs therefore differ from a universal OI-proportional schedule.For dispersion, the influence-proportional optimum oversamples names far from the mean; uniform sampling is a feasible baseline, not the theoretical optimum.Source: §3.3 · pp. 7–8

4 Experimental Setup

The experiments use US and Hong Kong options panels to compare five downsampling methods at matched retention budgets. They evaluate point accuracy, interval calibration, and performance across market-stress regimes.

4.1 Data

  • 80 US sessions and 82 Hong Kong sessions provide the evaluated panels; China A-shares are excluded because their daily panels average only 6.9 records.The US and Hong Kong panels average 210.7 and 131.4 underlyings per session or day, respectively.Source: §4.1 · p. 8

4.2 Methods Compared

  • Five methods are compared: Top-k, uniform, stratified, Poisson PPS-Hájek, and fixed-size systematic PPS-Hájek.All are calibrated to the same expected budget; intervals are reported for Poisson PPS because its variance is estimable from first-order inclusion probabilities alone.Source: §4.2 · p. 8

4.3 Retention Rates and Repetitions

  • Five retention rates, from 5% to 50%, are tested; each stochastic method uses 200 random seeds per trading day, while Top-k is deterministic.Reported MAE is averaged across seeds and days.Source: §4.3 · p. 9

4.4 Evaluation Metrics

  • Evaluation covers full-panel MAE, Poisson-Hájek interval coverage and width, serial-dependence-robust tests, and MAE, coverage, and width across market-shock terciles.Stress terciles use the absolute day-over-day change in cross-sectional mean ATM implied volatility, not within-day cross-sectional spread.Source: §4.4 · pp. 9–10, Table 2 · p. 10

5 Results

On US OI-weighted implied volatility, probability-proportional designs beat Top-k at every retention rate, while Poisson intervals remain approximate and under-cover. Results also show that design–target alignment, market differences, and estimator choice affect accuracy and interval calibration.

5.1 Flagship Result: OI-Weighted IV

  • Aligned Poisson intervals cover 89.0–94.3%, below nominal; Top-k’s worst-case interval is 2.7–6.4× wider under oracle bounds and 9–26× wider under feasible bounds.The width comparison is not a coverage comparison: both Top-k intervals are worst-case valid, whereas the Poisson interval targets nominal 95% coverage and under-covers empirically.Source: §5.1 · pp. 11–13
  • Both probability-proportional designs beat Top-k at every US retention rate; fixed-size PPS’s nominal MAE edge is sensitive to unit ordering and should not be treated as a ranking.Poisson PPS is used for intervals because its variance estimator uses first-order inclusion probabilities alone.Source: Figure 1 · p. 13

5.2 Design–Target Alignment

  • At US 10%, OI-proportional Poisson-Hajek is not best for unweighted targets: Top-k has lower median-PCR MAE (0.107 vs. 0.262), and stratified sampling leads for equal-weight VRP.Only the OI-weighted target calls for size-proportional inclusion; oracle optima for the equal-weight mean and dispersion are outcome-dependent and infeasible.Source: Table 4 · p. 11, §5.2 · p. 13

5.3 A Feasible Uniform Correction Consistent with Design–Target Alignment

  • Unif-PH beats OI-PH in point accuracy and interval calibration at every retention rate in both markets.At US 10%, MAE falls from 4.41 to 2.98 and coverage rises from 83.9% to 91.6%; in HK, MAE falls from 3.78 to 2.32 and coverage rises from 70.0% to 90.3%. Uniform is a feasible proxy, not the influence-proportional oracle optimum.Source: §5.3 · pp. 13–14

5.4 Cross-Market Comparison

  • At 10%, US and HK OI-weighted IV have opposite point-accuracy rankings; Poisson coverage is 91.9% in the US and 90.2% in HK.Top-k is shown with a worst-case identification width rather than coverage, so the two guarantees are not directly comparable.Source: §5.4 · pp. 14–15, Table 7 · p. 15

5.5 Known Total: Horvitz–Thompson versus H´ajek

  • HT improves calibration but has roughly twice H´ajek’s MAE (US 10%: 10.462 vs. 5.305); H´ajek is retained for headline point accuracy.With the population total known, HT is design-unbiased; H´ajek re-estimates that total and trades calibration for lower point error.Source: §5.5 · p. 15, Table 9 · p. 17

5.6 What Drives the HK Reversal

  • At 10%, the retained–omitted outcome gap (US 14.74 vs. HK 3.56; 4.14×) outweighs omitted OI share (55.5% vs. 37.7%; 1.47×) in the Top-k error decomposition.Concentration determines only the omitted-weight factor; error also depends on the outcome gap between retained and omitted names.Source: §5.6 · p. 16

5.7 Stress Regime Stability

  • US OI-weighted IV MAE and coverage stay near-flat across stress regimes, but coverage remains below nominal; equal-weight VRP PH MAE exceeds stratified MAE in every regime.The regimes are terciles of the absolute day-over-day change in cross-sectional mean ATM IV.Source: Table 5 · p. 12, §5.7 · p. 16

6 Related Work

The paper situates its contribution in survey sampling, partial identification, approximate query processing, and model-risk management. It applies established sampling tools to bandwidth-constrained financial aggregates, where known weights make computable worst-case bounds possible.

Design-based inference for finite populations

  • The contribution is the application of established finite-population sampling methods to pipeline retention, where known weights enable computable worst-case bounds for target-blind selection.The paper uses known auxiliary totals for calibration but does not pursue a full GREG construction.Source: §6 · pp. 17, 19

Fixed-size unequal-probability selection

  • Poisson sampling supports interval construction because its variance is unbiasedly estimable from first-order inclusion probabilities alone.The paper contrasts this with systematic PPS, whose second-order inclusion probabilities complicate closed-form variance estimation.Source: §6 · p. 17

Variance estimation for non-smooth functionals

  • For medians and quantiles, the paper uses a design-matched multiplier bootstrap because these non-smooth functionals require linearisation or resampling.The multiplier includes a √1 −πi factor so certainty units receive no perturbation; the testing protocol uses moving-block bootstrap, Newey–West standard errors, and Benjamini–Hochberg control.Source: §6 · pp. 17–18

Approximate query processing and sketching

  • Unlike sampling-based query systems with a known design, the paper studies reductions imposed upstream by bandwidth or licensing constraints on daily cross-sections with known weights.The paper relates its impossibility argument to deterministic KV-cache eviction [2] and adapts it to finite-population market aggregates.Source: §6 · pp. 18–19

Optimal subsampling and coresets

  • The optimal-allocation principle assigns inclusion probabilities according to influence or leverage, and the paper tests this alignment across four aggregate targets.Unlike coreset work, it seeks computable interval widths rather than a worst-case multiplicative approximation guarantee.Source: §6 · p. 18

Partial identification and model risk

  • With outcome bounds, omitted mass yields a sharp identified interval, but bounds taken from discarded data are an oracle quantity unavailable to a production consumer.This connects the paper to partial-identification methods and to model-risk concerns about unquantified uncertainty in data choices.Source: §6 · p. 18

7 Discussion

The discussion frames probability-sampling intervals as an auditability benefit rather than a guarantee of better point accuracy. It recommends matching sampling designs to targets while emphasizing calibration and validation limits in the studied data.

Auditability vs. point accuracy

  • Top-k can outperform on point accuracy, yet its feasible-bound worst-case width is 188.4 versus a Poisson–Hajek mean interval width of 10.98 with 90.4% coverage.The comparison is a wide worst-case-valid range versus a narrower, approximately calibrated interval, not a claim of equal-reliability dominance.Source: §7 · p. 19

Design-aware pipelines

  • Pipelines computing multiple aggregates face a multi-objective design problem because no single inclusion-probability schedule is optimal for every target.The paper suggests target-specific parallel subsamples or a compromise design and recommends documenting which target each subsample serves.Source: §7 · p. 19

Implications for model risk management

  • Design-based intervals make downsampling uncertainty explicit for risk management; the reported US OI-weighted IV interval width is 24.56 at 10% retention.The width quantifies uncertainty contributed by downsampling, subject to the approximation caveat.Source: §7 · p. 19

Relation to KV-cache eviction intervals

  • The study transfers a target-blind non-identification argument from deterministic KV-cache eviction [2] to financial aggregates, while adding market validation and design-alignment analysis.It also tests robustness across stress regimes.Source: §7 · p. 19

Study scope

  • Evidence covers roughly four months in one macro regime and end-of-day snapshots, leaving cross-regime and intraday coverage calibration unvalidated.Several secondary retention-rate comparisons are underpowered; a longer backtest spanning a full market cycle is proposed.Source: §7 · p. 20

Interval calibration

  • Linearised variance intervals under-cover at every tested retention rate for OI-weighted IV, with the largest shortfall at the smallest effective sample sizes.Coverage ranges from 89.0% to 94.3% in the US and 84.9% to 94.3% in Hong Kong as retention rises from 5% to 50%.Source: §7 · p. 20

Variance for non-smooth targets

  • The corrected multiplier perturbs design weights by √1 −πi so certainty units remain unperturbed, matching the Poisson design variance.The naive multiplier omits the (1 −πi) factor and assigns non-zero variance to certainty units.Source: §7 · p. 20

Bootstrap and validation limits

  • Correcting the multiplier bootstrap lowers measured coverage, and the paper says dispersion intervals are not usable at low retention as they stand.For US dispersion at 30% retention, coverage falls from 87.0% to 77.2%; the study excludes China A-shares for small, ETF-only panels and does not quantify the gap between oracle and feasible pilot designs.Source: §7 · p. 20

8 Conclusion

The conclusion argues that target-blind Top-k does not identify the OI-weighted aggregate without valid outcome bounds, while probability sampling provides approximate intervals. It emphasizes that accuracy depends on target alignment and that interval calibration must be checked rather than assumed.

8 Conclusion

  • Top-k does not identify the OI-weighted aggregate from retained outcomes alone without valid bounds; Poisson sampling with Hajek estimation yields approximate intervals and reduced MAE on matched targets.The conclusion reports MAE reductions of up to 36%, while the point-accuracy ranking depends on concentration and design-target alignment and reverses in Hong Kong.Source: §8 · p. 21
  • Inclusion probabilities should match the target functional: OI-proportional sampling is suboptimal for equal-weight and median statistics.The paper recommends selecting or documenting pipeline designs relative to downstream aggregates.Source: §8 · p. 21
  • Interval calibration cannot be assumed across targets or retention rates; dispersion misses nominal coverage in the sample and reaches 42.4% in Hong Kong at the most aggressive compression.The intervals are diagnostics to validate per target, not guaranteed coverage.Source: §8 · p. 21

A Full Proofs

  • The appendix supplies rigorous proofs of target-blind non-identification and optimal allocation for Poisson designs, with the target-blind scope restriction essential.Source: A Full Proofs · p. 23

A.1 Proof of Proposition 2.4

For target-blind deterministic selection, retained data do not identify the target when unobserved outcomes are unrestricted. Bounds or target-specific conditions can instead yield partial identification.

A.1 Proof of Proposition 2.4

  • Without bounds, changing an unobserved outcome leaves the retained data unchanged while making the OI-weighted target unbounded.The argument fixes N, the auxiliary vector, and retained values; it applies specifically to target-blind rules.Source: §A.1 · p. 23
  • Any uniformly valid interval based on retained data must therefore have infinite width for this target-blind setting.The conclusion is worst-case over populations consistent with the retained data, not a claim about a particular sampling distribution.Source: §A.1 · p. 23
  • For the unweighted median, the identified set is unbounded when at least ceil(N/2) records are missing, but bounded when fewer are missing.The median depends on the count of missing records, not their open-interest weight; when bounded, interval endpoints come from retained order statistics.Source: §A.1 · p. 24
  • With bounded outcomes, all four targets have finite sharp intervals, and the OI-weighted case has a closed-form interval.The OI-weighted target and equal-weight mean are affine in each unobserved value; dispersion is unbounded above without outcome bounds but bounded below by zero.Source: §A.1 · p. 24

A.2 Proof of Proposition 3.1

The proof derives an inclusion-probability schedule by minimizing a first-order design-variance proxy under a budget and probability bounds. The solution allocates in proportion to influence, subject to floor and ceiling constraints.

A.2 Proof of Proposition 3.1

  • The variance-minimizing interior schedule is proportional to each unit’s absolute influence, with probabilities clipped at the floor and ceiling.Units with sufficiently large influence are sampled with certainty; zero-influence units remain at the floor.Source: §A.2 · p. 25
  • Two-sided water-filling chooses the schedule slope to meet the expected-sample-size budget while accounting jointly for floor and certainty strata.The proof uses bisection; a one-sided promotion argument is valid only in the special case where no unit is in the lower-bound stratum.Source: §A.2 · pp. 25–26
  • When the budget exceeds what positive-influence units can absorb, those units receive certainty and the remaining budget can be distributed among zero-influence units.If all influences are zero, every feasible design is optimal; the usual non-degenerate case has a water-filling root.Source: §A.2 · p. 26
  • With influence constant within strata, the allocation reduces to Neyman allocation.The result generalizes stratum-level allocation to per-unit influence functions.Source: §A.2 · p. 26
  • Because influence may depend on the unknown target, feasible implementations can use a pilot estimate or robust surrogate.The paper reports that coarse feasible approximations recover most of the predicted gain.Source: §A.2 · p. 26

B Complete Result Matrix

The result matrix spans two markets, four targets, and five retention rates; the extended verification tests design-alignment predictions for median PCR and IV dispersion. The reported findings show that the predicted alternatives outperform OI-proportional sampling in those tests.

B Complete Result Matrix

  • The complete result matrix covers two markets, four aggregates, and five retention rates, reporting MAE and Poisson-Hajek interval diagnostics.The US and Hong Kong panels average 210.7 and 131.4 units, respectively, across 80 and 83 trading days.Source: B Complete Result Matrix · p. 27, §B.1 · p. 27, §B.2 · p. 29

OI-Weighted IV (US)

  • The US matrix includes OI-weighted IV, equal-weight VRP, median PCR, and IV dispersion, with paired Top-k versus Poisson-Hajek tests reported separately.The supplied passages identify table coverage but do not provide the matrix’s cell values.Source: Table 12 · p. 27, Table 14 · p. 27, Table 13 · p. 27

OI-Weighted IV (HK)

C Complete Stress-Regime Breakdown

D.1 Median PCR

D.2 IV Dispersion

D.1 Median PCR

  • At US 5% retention, median-PCR MAE falls from 0.418 to 0.166 under Unif-PH versus OI-PH; Unif-PH wins in all 10 cells.The reported paired tests have p < 10^-4 and Cohen’s d from 1.0 to 3.9.Source: §D.1 · p. 33, Table 36 · p. 33

D.2 IV Dispersion

  • At US 10% retention, the oracle influence design cuts IV-dispersion MAE from 6.60 to 2.94, a 55% reduction; at HK 10%, it falls from 4.65 to 1.89, a 59% reduction.The oracle design beats OI-PH in all 10 market-rate cells, but is not directly implementable because it uses full-panel values.Source: §D.2 · p. 34, Table 37 · p. 34

D.3 Summary Across All Four Targets

Across the four targets and 40 market-target-rate cells, the predicted-optimal design beats the OI-proportional baseline. The appendix also documents inclusion-floor implementation effects and interval diagnostics for equal-weight VRP.

D.3 Summary Across All Four Targets

  • Across all 40 market-target-rate cells with directional predictions, the predicted-optimal design beats the OI-proportional baseline.The summary covers OI-weighted IV, equal-weight VRP, median PCR, and IV dispersion; the ideal influence-proportional design is not always feasible.Source: §D.3 · p. 34

D.4 Inclusion-Floor Diagnostics

  • At US 10% retention, ceiling clipping removes 136.8 expected units and leaves the oracle design 7.7% below nominal sample size.The ceiling removes 2,333.8 expected units at 50% retention, leaving the oracle 26.1% below nominal; the main results use budget-matched bisection instead.Source: §D.4 · p. 35
  • The inclusion floor binds on at most 0.11% of US unit-days and 0.07% of HK unit-days, and changing the floor moves oracle MAE by less than Monte Carlo noise.The floor prevents zero inclusion probabilities; the diagnostics conclude the reported oracle results are not floor artefacts.Source: §D.4 · pp. 35–36

D.5 Equal-Weight VRP: Interval Diagnostics

  • For equal-weight VRP, uniform sampling improves coverage at every rate in both markets and narrows intervals; at HK 5%, coverage rises from 59.2% to 86.0%.Uniform sampling is a feasible proxy rather than the influence-proportional oracle.Source: Table 40 · p. 36, §D.5 · p. 37, Table 41 · p. 37

E Ordering Sensitivity of Fixed-Size Systematic PPS

Systematic PPS’s realized error depends on unit ordering, so the reported Fixed-PPS versus Poisson-PPS accuracy ranking is not robust. Randomizing the unit permutation makes the production results interpretable, while outcome-sorted ordering illustrates a deployment-infeasible alternative.

E Ordering Sensitivity of Fixed-Size Systematic PPS

  • The production implementation randomizes the unit permutation on every replicate, averaging over the permutation distribution across 200 replicates per day.Systematic PPS’s second-order inclusion structure, and therefore realized error, depends on unit order.Source: E Ordering Sensitivity of Fixed-Size… · p. 37
  • In 9 of 10 market × rate cells, ordering spread exceeded the Poisson-minus-Fixed MAE margin; at HK ρ = 10%, it was 0.69 against 0.07.The paper withdraws the claim that fixed-size PPS is more accurate than Poisson PPS; the choice should instead reflect the structural trade-off between a hard bandwidth cap and cheaper unbiased variance estimation.Source: E Ordering Sensitivity of Fixed-Size… · p. 38, Table 42 · p. 38
  • Outcome-sorted iv desc roughly halves US MAE at ρ = 10% (5.10 → 2.36), but requires sorting by the unavailable target and violates target-blindness.A production input order correlated with the outcome can therefore produce error properties materially different from the randomized-order results.Source: E Ordering Sensitivity of Fixed-Size… · p. 38
  • Fixed-size systematic PPS should randomize its unit permutation or use a design whose second-order structure does not depend on sweep order.If neither is done, the ordering should be reported as part of the design; pivotal or cube sampling are alternatives.Source: E Ordering Sensitivity of Fixed-Size… · pp. 37–38

F.1 Data Schema

The appendix documents the record schema, analysis scripts, random-seed protocol, runtime environment, target construction, and data provenance. It specifies the analytical panel and computational choices needed to interpret and reproduce the reported analyses.

F.1 Data Schema

  • Daily trading sessions are stored as JSON files with a date, market, record count, and per-underlying data; experiments use only the schema’s four core fields.Missing values are stored as JSON null, while other ingested vendor fields are not consumed by the reported experiments.Source: §F.1 · p. 39, Table 43 · p. 39

F.2 Script Inventory

  • The unified experiment script is the source of the paper’s numbers, while table and figure-generation scripts derive outputs from its results.The earlier pipeline is retained for audit but superseded after a main-table/appendix discrepancy.Source: §F.2 · p. 40

F.3 Random Seed Protocol

  • Each trading day uses 200 replications, with a moving-block bootstrap for paired mean differences and a nested bootstrap for median and dispersion standard errors.The paired-difference bootstrap uses 2,000 replicates and block length L = max(2, ⌈T^1/3⌉); the nested bootstrap uses B = 60 resamples.Source: §F.3 · pp. 40–41
  • The four probability designs share a per-day random stream for paired comparisons, and stable market-date keys make the seed schedule reproducible across processes and machines.Sharing a stream reduces between-design simulation noise; stable keys replaced process-randomized Python hashing that had caused irreproducible results and earlier table disagreements.Source: §F.3 · pp. 40–41

F.4 Runtime Environment

  • The experiments run on Python 3.11 with NumPy 1.26 and SciPy 1.11; the main experiment takes about 20–40 minutes on a laptop-class machine.The reported environment is macOS on Apple silicon, single-process, with no GPU required; design-aware scripts take about 5–15 minutes each.Source: §F.4 · p. 41

F.5 Variable Construction

  • The four daily targets include OI-weighted ATM IV, equal-weight VRP, median put-call ratio, and equal-weight ATM-IV dispersion.OI-weighted IV uses total open interest as its weight; VRP and dispersion have uniform target weights and are design-misaligned with OI-proportional sampling, while median PCR uses an inverse-probability-weighted quantile.Source: §F.5 · p. 42
  • The target panel requires present ATM IV and positive open interest, uses nearest-expiry ATM IV, and applies no winsorization or outlier removal.Nearest-expiry time to expiry has a median of 7 days and range of 3–37; target-specific fields must also be non-null, and days with fewer than five surviving units are skipped.Source: §F.5 · pp. 42–43

F.6 Data Provenance

  • The data are institutional daily end-of-day US and Hong Kong options snapshots from 2026-03-20 through 2026-07-24; ratios are invariant to common positive scale changes, but not arbitrary nonlinear transformations.A nonlinear transformation of the outcome changes the estimands and coverage, while one of the design weight changes sampling probabilities and target weights.Source: §F.6 · p. 43
Loading 2610.08801v1…