PaperlayerView exact revision on arXiv

Source-linked AI summary

When 10,000 Windows Are Not 10,000 Tests: Auditing Statistical Confidence in Sliding-Window Time-Series Classification

Xinze Shi, Litian Zhang, Binrui Shi

arXiv:2609.30721v1cs.LG

TL;DR

Overlapping windows create many test rows without making neighboring predictions independent, even under subject-disjoint evaluation. The paper maps three generalization targets to aggregation and dependence-robust inference; at 75% overlap, HARTH Accuracy intervals include zero while Macro-F1 favors MiniROCKET, whereas WISDM favors MiniROCKET on both metrics.

  • Problem

    Overlapping test windows reuse observations and remain nested within recordings and subjects, so subject-disjoint evaluation does not make test-window uncertainty independent.

    Source: §I · p. 1

  • Method

    The audit maps observed-record, new-session, and new-subject claims to explicit aggregation rules and matched dependence-robust inference using frozen predictions.

    Source: §I · p. 1

  • Results

    At 75% overlap, HARTH paired Accuracy-difference intervals include zero while Macro-F1 favors MiniROCKET by 13.2–18.0 points; WISDM favors MiniROCKET on both metrics across targets.

    Source: C. Metric Choice Changes the… · p. 6

  • Takeaways & Limitations

    The workflow distinguishes additional predictions from independent evidence and makes target- and metric-sensitive conclusions inspectable from paired prediction files.

    Source: §VIII · p. 8

    Validation covers two wearable HAR datasets and two classifier families, does not fully propagate training randomness across overlapping folds, and assumes within-session stationarity as an approximation.

    Source: §VII · p. 8

Abstract

from arXiv · show

Sliding-window classifiers are often evaluated on thousands of overlapping test windows, even though neighboring predictions share observations and remain nested within recordings and subjects. Subject-disjoint evaluation prevents one form of leakage but does not make those test windows independent. We present a practical audit that maps three claims - performance on observed recordings, future recordings from observed subjects, and unseen subjects - to explicit aggregation rules and established dependence-robust inference. At 75% overlap, controlled simulations give 16.9% Type-I error for IID observed-record inference and 7.2% for session-centered Bartlett-HAC: a substantial improvement with residual miscalibration. Audits of frozen WISDM and HARTH predictions show that nearly fourfold growth in test rows yields only 1.75-1.94-fold variance-equivalent information growth. At that overlap, fixed-record paired Accuracy-difference intervals are 1.22-1.66 times the IID widths; this inflation is not universal at zero overlap. On HARTH, paired Accuracy-difference intervals include zero across three overlap settings, whereas Macro-F1 favors MiniROCKET. Independent recomputation, common-session checks, class-level results, and separately seeded calibration make the audit's scope and limitations inspectable. The resulting workflow distinguishes additional predictions from additional independent evidence.

I. INTRODUCTION

Overlapping windows multiply test rows without multiplying independent evidence, and subject-disjoint evaluation does not resolve uncertainty from dependent test windows. The paper maps distinct generalization claims to aggregation and inference choices, using established methods in an executable audit workflow.

I. INTRODUCTION

  • At 75% overlap, a recording yields approximately four times as many test rows as non-overlapping segmentation, but adjacent windows reuse samples and are not four times as many independent tests.Predictions also remain nested within recordings and subjects.Source: §I · p. 1, Figure 1 · p. 2
  • Subject-disjoint evaluation addresses one form of leakage but does not justify IID intervals for dependent test windows.Prior work motivates subject-disjoint evaluation, while the uncertainty among dependent test windows remains unresolved once predictions are generated.Source: §I · p. 1, §II · p. 1
  • Observed-window, future-recording-from-the-same-people, and new-person comparisons are distinct targets requiring explicit aggregation and inference choices.The contribution is an executable mapping from a sliding-window claim to its aggregation rule, inference unit, eligibility checks, and report.Source: §I · p. 1
  • The audit separates temporal, session, and subject variation, checks calibration with the real-data estimator, and reports Accuracy and Macro-F1 from frozen WISDM and HARTH predictions.Its contribution is the claim-to-analysis workflow and associated evidence checks.Source: §I · p. 1

II. RELATED WORK

  • Established HAC, block-bootstrap, cluster-robust, and hierarchical-bootstrap methods are components of the audit; its contribution is the executable target-to-analysis mapping, not a new generic variance estimator.Prior uncertainty work primarily concerns variability across training/test splits, whereas this paper conditions on frozen out-of-fold predictions and examines evidence in the dependent test sequence.Source: §II · pp. 1–2

III. TARGETS AND AUDIT PROTOCOL

The protocol compares classifiers on the same eligible windows using paired correctness differences, preserving the matched structure of the test data.

A. Data hierarchy and paired contrasts

  • For Accuracy, the paired contrast is the matched difference between the two classifiers’ window-level correctness indicators.The setup indexes subjects, sessions, and time-ordered windows, with both classifiers evaluated on the same eligible windows.Source: A. Data hierarchy and paired contrasts · p. 2
  • Pairing retains covariance on shared test windows that separate estimates of the two metric variances would discard.Positive contrast values favor classifier A.Source: A. Data hierarchy and paired contrasts · p. 2
  • Before inference, the audit requires matching subject, session, time, label, fold, and window-boundary keys for both models.Source: A. Data hierarchy and paired contrasts · p. 2

B. Three evaluation targets

The three targets differ in what is held fixed and how observations are weighted: observed windows, future sessions from observed subjects, or new subjects.

B. Three evaluation targets

  • For the observed-record target, sessions and subjects are fixed and observed windows are pooled; uncertainty concerns a finite temporally dependent realization within those sessions.Source: B. Three evaluation targets · p. 2
  • For the new-session target, sessions receive equal weight while sessions from the same subject remain dependent; inference uses a subject-clustered sandwich calculation on session means.For nonlinear metrics, session confusion matrices are normalized by window count before aggregation and paired subject-cluster resampling.Source: B. Three evaluation targets · p. 2
  • For the new-subject target, subjects receive equal weight and inference is paired across subjects, so longer recordings do not give a subject more weight.Source: B. Three evaluation targets · p. 2

C. Observed-record HAC inference

Observed-record Accuracy inference estimates temporal sampling variance conditional on the observed sessions, using session-centered paired contrasts and within-session lag products.

C. Observed-record HAC inference

  • Observed-record Accuracy contrasts are centered within sessions so between-session difficulty is not mistaken for within-session temporal dependence.For this target, observed sessions and their process-level mean differences are conditioned on, excluding between-session mean heterogeneity from conditional temporal sampling variance.Source: C. Observed-record HAC inference · p. 3
  • The Bartlett-HAC calculation uses lag products that never cross sessions, with lags defined on retained window starts and a length-based bandwidth plus sensitivity settings.The Newey–West/Bartlett construction is specialized to paired sequences separated by session.Source: C. Observed-record HAC inference · p. 3
  • Observed-record Macro-F1 uses a confusion-matrix influence-function HAC, with a session-aware block bootstrap that recomputes Macro-F1 as a sensitivity check.The other targets aggregate normalized confusion matrices or subject-specific Macro-F1 differences rather than averaging per-window F1 scores.Source: C. Observed-record HAC inference · p. 3
  • Fixed-record HAC intervals use a normal critical value, while new-session Accuracy and equal-subject intervals use finite-sample t references and new-session Macro-F1 uses paired subject-cluster percentile resampling.HARTH has 22 subject clusters, and small-cluster problems remain unresolved for sandwich and percentile-bootstrap intervals.Source: C. Observed-record HAC inference · p. 3

D. Audit outputs and interpretation

The audit reports variance-equivalent information and target-aligned uncertainty alongside diagnostics, while making target choices and input eligibility explicit. It retains IID inference as a reference rather than an automatic model-selection rule.

D. Audit outputs and interpretation

  • Information growth is variance-equivalent relative growth, not a universal absolute effective sample size; the report also gives target-aligned/IID width ratios and interval-disagreement flags.It includes global and session-centered autocorrelation and bandwidth sensitivity.Source: D. Audit outputs and interpretation · p. 4

E. From a scientific claim to an executable audit

  • The audit declares the target, metric, and difference direction before inference, rejects mismatched or invalid inputs, and preserves fixed class indexing for Macro-F1 resampling.Checks include duplicate or missing window keys, invalid boundaries, non-monotone order, and subjects appearing in multiple test folds.Source: E. From a scientific claim to an… · p. 4
  • IID inference remains a diagnostic reference: disagreement with target-aligned intervals flags an uncertainty or target issue, not an automatic winner.The minimum report-fields table specifies what the dependence-aware audit should report.Source: E. From a scientific claim to an… · p. 4, Table II · p. 4

IV. EXPERIMENTAL DESIGN

The experiments calibrate dependence-robust inference under controlled overlap and apply the audit to frozen wearable-sensor predictions. The real-data design fixes subject-disjoint folds and recording boundaries while varying test-window overlap.

A. Controlled calibration

  • The controlled null experiments use paired correctness sequences with known accuracies and dependence at subject, session, raw-sample, and model-pair levels, testing 0%, 50%, and 75% overlap.Both marginal accuracies are 0.8 under the null; conditions also vary dependence structure, nonzero differences, subject count, and paired-model correlation.Source: A. Controlled calibration · p. 4

B. Real data, models, and frozen comparisons

  • WISDM uses phone accelerometer data and HARTH uses thigh accelerometer data; session boundaries and window eligibility are defined separately for each dataset.Windows cannot cross session boundaries; WISDM windows are single-activity, while HARTH 10-s windows require a majority label covering at least 0.8 of the window.Source: B. Real data, models, and frozen… · p. 4
  • After eligibility checks, 933 WISDM sessions contribute at every overlap, while HARTH contributes 393, 403, and 409 sessions at 0%, 50%, and 75% overlap.The frozen boundary policy yields 934 WISDM and 591 HARTH segments.Source: B. Real data, models, and frozen… · p. 4, Table III · p. 4
  • The comparison pairs a feature-based Random Forest with MiniROCKET and validation-selected ridge using subject-level five-fold out-of-fold predictions and fixed 10-s test windows.Test overlap is 0%, 50%, or 75%; WISDM also has a prespecified frozen 5-s sensitivity analysis, with the same raw test sessions and held-out people across overlap settings.Source: B. Real data, models, and frozen… · p. 4
  • Frozen prediction files are checked for matched rows, unique keys, valid window geometry, monotone session time, and fold/subject isolation before inference.Seeds, folds, feature settings, model parameters, and ridge search ranges are recorded in manifests.Source: B. Real data, models, and frozen… · p. 4
  • At 0–50% overlap, session-centered Bartlett-HAC remains close to nominal for fixed-record inference; for new subjects, equal-subject paired inference uses the matched sampling unit.Window-level IID and within-session HAC remain severely target-misaligned for the new-subject target.Source: Figure 3 · p. 5

C. Frozen comparisons and sensitivity grid

The calibration and frozen-data analyses show that inference depends on matching the uncertainty method to the target and accounting for dependence among windows. More test rows at dense overlap add substantially less variance-equivalent information, and target-aligned intervals can be wider than IID intervals.

C. Frozen comparisons and sensitivity grid

  • The frozen comparisons keep predictions and evaluation settings fixed across overlaps, and the sensitivity grid varies bandwidth, Macro-F1 inference, subject count, session heterogeneity, and cross-model correlation.The historical HAC recalculation reused earlier simulation seeds, so the authors separately locked rules before generating independently seeded tests.Source: C. Frozen comparisons and sensitivity… · p. 5

A. Target alignment determines calibration

  • At 75% overlap, IID Type-I error rises to 16.9%, while session-centered Bartlett-HAC yields 7.2%; HAC reduces but does not eliminate anti-conservatism.The comparison concerns fixed-record inference; at 0% and 50% overlap, the formal HAC procedure remains close to nominal.Source: A. Target alignment determines… · p. 5

B. Overlap multiplies rows faster than information

  • At 75% overlap, nominal test-window counts grow 3.93–3.97-fold, while variance-equivalent information grows only 1.94-fold for WISDM 5 s, 1.87-fold for WISDM 10 s, and 1.75-fold for HARTH 10 s.These ratios describe evaluation information, not overlap's predictive value.Source: B. Overlap multiplies rows faster than… · p. 5, Figure 4 · p. 5
  • At dense overlap, paired Accuracy-difference intervals are 1.22–1.66× wider under target-aligned inference than under IID inference.The table reports fixed-record paired Accuracy-difference interval widths in percentage points at 75% overlap.Source: B. Overlap multiplies rows faster than… · p. 5, Table IV · p. 6, Figure 5 · p. 6
  • At 0% overlap, session-centering narrows intervals while the dependence factor widens them: WHAC/WIID is 0.92 for WISDM 5 s and 0.84 for WISDM 10 s.Adjacent non-overlapping windows can still be temporally dependent; interval inflation therefore varies with dataset, window length, label persistence, model errors, and session composition.Source: B. Overlap multiplies rows faster than… · p. 6

C. Metric Choice Changes the Scientific Conclusion

On HARTH, Accuracy-difference intervals include zero at 75% overlap, while Macro-F1 favors MiniROCKET; class-level results show that this advantage is concentrated in particular activities. The section distinguishes metric weighting from target weighting and checks whether overlap results depend on the contributing-session set.

C. Metric Choice Changes the Scientific Conclusion

  • At 75% overlap, HARTH paired Accuracy-difference intervals include zero for all three targets, while Macro-F1 favors MiniROCKET by 13.2–18.0 points; WISDM favors MiniROCKET on both metrics and all targets.The WISDM result also holds for the frozen 5-s sensitivity.Source: C. Metric Choice Changes the… · p. 6
  • Accuracy weights eligible windows, Macro-F1 weights classes equally, and new-session or new-subject targets separately equalize sessions or people.For nonlinear metrics, resampling must preserve the target aggregation: new-session analysis averages session-normalized confusion matrices within subject-cluster resamples, while new-subject inference averages subject-specific metric differences.Source: C. Metric Choice Changes the… · p. 6
  • Independent recomputation found HARTH pooled and equal-subject Accuracy differences of +0.073918641805 and +0.070878431669 percentage points, respectively, both rounding to +0.07.Unequal subject window counts do not force distinct displayed means.Source: C. Metric Choice Changes the… · p. 6
  • Across HARTH overlap settings, every paired Accuracy-difference interval includes zero and every Macro-F1-difference interval favors MiniROCKET, although point differences vary.The added 0%-to-75% grid positions were not rejected coarse-grid windows, and their mean purity was slightly higher than that of shared windows.Source: C. Metric Choice Changes the… · p. 6, Table V · p. 6
  • At 75% overlap, HARTH information growth is 1.750128 on 393 common sessions versus 1.751439 originally, but the equal-session Accuracy difference shifts from −2.25 [−5.87, +1.37] to −1.49 [−4.51, +1.52] points.The original information-growth calculation used 393/403/409 contributing sessions.Source: C. Metric Choice Changes the… · p. 6
  • At 75% overlap, sitting supplies 52.04% of HARTH windows, while stair ascent/descent supply 1.08%/0.59%; those two stair classes contribute 7.34 points of MiniROCKET’s 13.21-point pooled Macro-F1 advantage.RF has 672 more correct sitting predictions and 415 more for seated cycling, partly offsetting MiniROCKET’s gains elsewhere; the class-specific tradeoff does not imply every minority class benefits equally.Source: C. Metric Choice Changes the… · p. 7

D. Robustness and additional calibration

Separately seeded calibration continues to show reduced but residual HAC miscalibration at dense overlap, while target-misaligned methods have excessive null rejection under hierarchical dependence. Additional checks compare interval methods, examine bandwidth sensitivity, and mark the limits of the length-stress simulations.

D. Robustness and additional calibration

  • In the historical WISDM 5-s length-stress test at 75% overlap, HAC Type-I error is 8.9% versus IID 18.2%, with HAC coverage of 91.2%.The new length-stress result is 6.7%, differing from the historical 8.85% by 2.12 combined MCSEs; the two rates remain separate Monte Carlo outcomes.Source: D. Robustness and additional calibration · p. 7
  • For WISDM 5-s at 75% overlap, the influence-function Macro-F1 interval and a 1,999-replicate session-aware block bootstrap are nearly identical.Bandwidth sensitivity changes interval widths but not the reported real-data model conclusions.Source: D. Robustness and additional calibration · p. 7
  • At 0/50/75% overlap, independently calibrated IID rejection rates are 5.2/8.7/16.9% and formal HAC rates are 5.5/6.4/7.9%.In the hierarchical design, IID/HAC/subject-t reject 73.2/66.3/4.6%.Source: D. Robustness and additional calibration · p. 7
  • With a true +2-point difference, formal HAC power in balanced AR(1) designs is 67.7/85.6/93.9%, with corresponding coverage of 93.1/94.6/92.8%.Hierarchical equal-subject power is 14.1% with 94.9% coverage; higher rejection from target-misaligned methods is not a power advantage when null rejection is excessive.Source: D. Robustness and additional calibration · p. 7
  • At 75% overlap, the per-session bandwidth rule and default produce the same rejection counts in new WISDM 5-s/HARTH 10-s length-template stress tests: 6.7%/6.6%.The 2K0 = 6 floor dominates; this does not show that per-session bandwidth cures residual miscalibration, so the original default is retained.Source: D. Robustness and additional calibration · p. 7
  • Rule-selection/calibration independence remains unverified, and the new length-template simulations reproduce lengths but not HARTH purity gaps or class transitions.The separately seeded test sets used fixed rules.Source: D. Robustness and additional calibration · p. 8

VI. DISCUSSION AND REPORTING GUIDANCE

Sound uncertainty reporting begins by declaring the target population, metric aggregation, highest independent unit, and dependence structure. The audit distinguishes observed-record, future-recording, and unseen-subject claims, with different scope and reporting requirements.

VI. DISCUSSION AND REPORTING GUIDANCE

  • A p-value or confidence interval requires an explicit target population, metric aggregation, highest independent unit, and dependence structure.Figure 2 links these declarations to the audit.Source: §VI · p. 8
  • Without subject identifiers, analyses are limited to fixed records or recording level; one recording per subject cannot identify a same-subject new-session target.A single series supports only a conditional temporal target without additional population assumptions.Source: §VI · p. 8
  • For claims about unseen people, new-subject inference should be primary; fixed-record inference is only a conditional precision reference, not a lower bound for other targets.Intervals crossing zero do not establish equivalence.Source: §VI · p. 8
  • Reporting should include Table II fields, eligibility counts, and sensitivity analyses, while preserving pairing, target weights, and the complete confusion matrix during resampling.The supplied passage specifies the reporting requirements but does not provide Table II's fields.Source: §VI · p. 8
  • Denser segmentation may improve latency, temporal coverage, or point prediction, but it does not create an equal amount of independent evaluation evidence.Reporting nominal and information growth together makes that distinction visible.Source: §VI · p. 8

VII. LIMITATIONS

The audit’s validation and inference have important scope boundaries, while the conclusion emphasizes reproducible checks of targets, aggregation, independent units, and metric-specific uncertainty. Its numerical factors are not universal, and its frozen-prediction analysis does not fully capture training randomness.

VII. LIMITATIONS

  • Validation covers two wearable HAR datasets and two classifier families, so numerical factors are not universal; broader domains and models remain future work.HAC is bandwidth-dependent and conditional on observed sessions, and strong overlap or uneven recording lengths can leave residual anti-conservatism.Source: §VII · p. 8
  • The evaluation uses frozen OOF predictions and does not fully propagate training randomness across overlapping folds,.Within-session stationarity is an approximation; abrupt regime changes motivate segmentation or block-based sensitivity checks.Source: §VII · p. 8

VIII. CONCLUSION

  • Reliable reporting requires the target population, aggregation rule, highest independent unit, and metric-specific uncertainty method; the reproducible audit can expose target- and metric-sensitive conclusions.The checks are reproducible from paired prediction files.Source: §VIII · p. 8
Loading 2609.30721v1…