PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Pre-Filtering for Context-Adaptive Sensor Fusion: A Four-Role FFT-GDCB Integration for High-Stakes Decision Systems

Oleg Miroshnichenko

arXiv:2608.22023v1cs.LG

TL;DR

Periodic structure contaminates innovation and feature regressions, causing calibration to model structural modes rather than genuine uncertainty or demand sensitivity. The paper introduces a four-role FFT pre-filter that whitens measurements, cleans covariance-regression inputs, adds spectral bandit context, and deseasonalises supervised-regression features. Across six pre-registered domains, the pipeline receives a PROVES verdict, while its use remains bounded by stationarity, identifiable cycles, and specific failure modes.

  • Problem

    Periodic contamination in calibration inputs can distort covariance models and sensitivity coefficients across sensor-fusion and revenue-management systems.

  • Method

    A four-role FFT preprocessing layer performs whitening, innovation cleaning, spectral context generation, and feature deseasonalisation before downstream calibration and decision components.

  • Results

    All six pre-registered domains return a PROVES verdict under the shared evaluation protocol, with clinical covariance improvement reaching 13.5×.

  • Takeaways & Limitations

    The pipeline provides a unified preprocessing layer whose outputs support Kalman calibration, GDCB scaling, bandit regime selection, and sensitivity-coefficient regression.

Abstract

from arXiv · show

Context-adaptive Kalman filters calibrate their noise covariance matrices Q and R from innovation residuals via online regression. When the underlying sensor or signal carries periodic structure -- mechanical LiDAR rotation harmonics, engine vibration, ground multipath, weekly and annual demand cycles, dosing-interval rhythms, weekly media-buying cadence -- the regression input is contaminated and the fitted covariance models structural modes rather than genuine state uncertainty. We introduce a four-role FFT pre-filter that solves this problem at $O(N\log N)$ cost and serves three additional roles "for free": (i) it whitens coloured noise before the Kalman update, restoring the optimality assumption; (ii) it cleans innovations before covariance regression, preventing periodic contamination of $\hat{R}$ and $\hat{Q}$; (iii) it generates spectral context features that enrich the downstream bandit's regime-selection state; (iv) it deseasonalises the input feature vector before any supervised regression that produces a sensitivity coefficient (beta, dose offset, bid modifier). We position the algorithm inside the Gated Decoupled Compositional Bandits (GDCB) family, where it acts as a preprocessing layer for the supervised scaler. The single $O(N\log N)$ FFT call thereby serves four downstream consumers, fits in <0.1% of the sensor-fusion or pricing-pipeline compute budget, and is a drop-in addition with no changes to the Kalman filter, bandit, or runtime composition operator. We summarise empirical validation across six independent domains (rocket descent, autonomous-vehicle tracking, short-term rental pricing, clinical drug dosing, airline fare distribution, and ad-operations bid calibration), all returning a PROVES verdict under a pre-registered evaluation protocol.

1 Introduction

The paper presents a unified four-role FFT pre-filter for periodic contamination in sensor-fusion and revenue-management calibration. One O(N log N) transform supports whitening, covariance cleaning, spectral context generation, and feature deseasonalisation without modifying downstream components.

  • Motivation: Periodic structure contaminates calibration regressions, causing covariance estimates to absorb structural modes and sensitivity coefficients to shrink.Examples include LiDAR harmonics, vibration, dosing rhythms, booking cycles, annual demand, and advertising cadence.
  • Architecture: One FFT call produces four outputs: measurement whitening, covariance-regression cleaning, spectral bandit context, and feature deseasonalisation.The architecture feeds off-the-shelf Kalman, GDCB, bandit, and revenue components.
  • Roles 1–2: Role 1 applies an empirical PSD-based whitening filter before the Kalman update, restoring approximately white residuals and asymptotic optimality.Role 2 notches spectral peaks from innovations before fitting R and Q regressions.
  • Roles 3–4: Role 3 appends named-band spectral energies to bandit contexts, enabling regime-aware arm selection beyond time-domain state features.Role 4 feeds only aperiodic regressor components into sensitivity-coefficient regression.
  • Integrated outputs: The four-role view also exposes periodic components, spectral energies, and seasonal components as useful outputs for monitoring, bandit context, and forecasting.The paper frames this as an integrated calibration substrate rather than a single-purpose periodic-noise filter.
  • Validation: All six pre-registered domains return a PROVES verdict under the shared evaluation protocol.The validation spans three sensor-fusion domains and three revenue-management domains.

2 Background

The background establishes that Kalman optimality depends on correctly characterized noise, while real sensor and revenue signals commonly violate whiteness through periodic structure. Classical adaptive covariance methods estimate from innovations but do not remove that periodic structure first.

  • Kalman assumptions: Under the linear-Gaussian model, the Kalman filter is minimum-mean-squared-error optimal with known moments, and innovations are white with covariance S_t.The normalized innovation squared is χ^2-distributed under the stated assumptions.
  • Assumption failure: Real measurements often contain harmonics, vibration, multipath, or calendar cycles, breaking the white-noise assumption and miscalibrating the Kalman gain.The paper also states that NIS tests then no longer detect model violations reliably.
  • Prior methods: Classical adaptive methods estimate Q and R from innovation residuals but do not remove periodic structure before computing sample variance.The paper positions FFT cleaning as addressing this unhandled contamination.

3 The Four-Role FFT Pipeline

The four-role FFT pipeline detects and removes periodic structure while reusing the same spectral decomposition for whitening, covariance calibration, bandit context, and feature deseasonalisation. Its notch stage restores an approximately flat spectrum and reduces contaminated innovation variance by about sevenfold, but notch filtering is unsuitable for broadband noise and deseasonalisation requires care when the dependent variable shares the regressor’s cycles.

  • FFT notch algorithm: An N-point rfft provides O(N log N) access to each innovation channel’s spectrum, where peaks exceeding a noise-floor threshold are candidates for notching.Known frequencies can be added to the notch mask, and aliased frequencies must be specified below Nyquist.
  • Roles 2–3: The cleaned innovation νaper feeds covariance ridge regression, while discarded periodic content supplies structural-mode amplitude trajectories and spectral context features.Growing peak amplitudes can signal incipient bearing wear or flutter onset.
  • Role 2 result: 0.20 to 0.029 m2 (∼7× improvement) after notching the synthetic innovation contaminated by 15 and 20 Hz harmonics.The post-notch spectrum is approximately flat; the white-noise floor is σ=0.18, with harmonic amplitudes 0.41 and 0.23.
  • Limitations and ordering: Notch filtering is inappropriate for broadband, non-peak-concentrated noise, while mixed broadband-and-harmonic spectra should be whitened before notching.The recommended order makes harmonics stand out above the µ + 3σ threshold after flattening the noise floor.
  • Role 3 — Spectral context features: The pipeline appends named-band spectral energies and summary statistics to bandit contexts, enabling regime-aware arm selection beyond time-domain context alone.High spectral entropy can indicate an off-distribution autonomous-vehicle scenario, while centroid shifts can signal rocket structural transitions.
  • Role 4 — Feature deseasonalisation: Per-feature FFT notching targets regressor contamination in multivariate ridge regression and recovers sensitivity coefficients closer to demand-driven ground truth under identifiable-cycle and cycle-orthogonality conditions.If the dependent variable shares the regressor’s cycles, deseasonalising only the regressor can shift the estimated coefficient toward zero.

4 When to Use Which Role

The four roles are signal-class specific rather than interchangeable, so role selection depends on the type of periodic structure being processed.

  • Role selection varies by signal class, and the four roles are not interchangeable.

5 Computational Budget

The four-role FFT is designed as a low-cost preprocessing call whose computational burden is practical across the target deployment regimes.

  • O(W log W) is the per-channel cost of each FFT call.
  • Table 2 reports realistic computational costs for the six target domains.
  • ∼700 multiply-accumulate operations let a 100-sample FFT fit trivially in a 100 Hz GNC loop on a Cortex-M4.
  • There is no realistic deployment regime in which the four-role FFT is too expensive.Daily-data implementations cost microseconds in NumPy.

6 Six-Domain Empirical Validation

The pre-registered validation covered six domains and seven conditions, with every condition receiving a PROVES verdict under the shared protocol.

  • The validation used one pre-registered evaluation protocol across six domains, with detailed tables, plots, and code available in the companion repository.
  • All seven evaluation conditions return a PROVES verdict under the pre-registered evaluation protocol.The AV domain contributes separate dry and rain conditions, producing seven bars across six domains.
  • The three sensor-fusion domains meet or exceed theoretical bR improvement ceilings, with clinical reaching 13.5× versus a theoretical 5.94×.The clinical excess is attributed to tracking-error variance removed incidentally by Role 2 but omitted from the theory.
  • The three revenue-management domains recover all 4/4 ridge-regression coefficients closer to the demand-driven ground truth.The result spans STR, airline, and ad-operations domains rather than only STR.
  • The protocol defines PROVES using factor ≥2.5× for Role 2, or mean improvement > 10% with at least 3/4 parameters closer for Role 4.All six domains pass these protocol thresholds.

7 Discussion

The four-role FFT view consolidates whitening, covariance calibration, spectral context generation, and deseasonalisation into one preprocessing step. Its applicability depends on stationarity, harmonic structure, and known or detectable cycle frequencies.

  • Four-role integration: A single FFT call replaces four separate moving parts with one shared O(N log N) subroutine, reducing integration burden to a single point.The four roles are whitening, covariance calibration, context featurisation, and deseasonalisation.
  • Assumptions: The pipeline assumes stationarity over calibration window W, harmonic spectral structure, and known or detectable cycle frequencies.These assumptions define the operating boundary of the four-role proposition.
  • Open problems: A fixed τ=3 peak threshold is a false-positive heuristic rather than exact Type-I/Type-II error control.An adaptive threshold based on the white-noise null is identified as an improvement.
  • Open problems: Wavelet packet decomposition may better localise non-stationary rocket transitions and autonomous-vehicle scenario changes, but matched-compute comparison remains open.The paper names wavelets as an alternative when FFT assumptions fail.
  • Open problems: The Role 4 seasonal blending parameter γ_seas is set manually, leaving data-driven cross-validation as an open problem.The runtime composition is f_i = f_aper i + γ_seas f_seas i.
  • Open problems: The formal treatment of context-dependent stochastic gates remains open because Role 3 spectral features should also enter the pre-execution gate policy.The current discussion distinguishes bandit context from gate policy.

8 Conclusion

The paper concludes that one FFT call can produce four operational outputs for calibration and downstream decision systems. Across six domains, the pipeline receives a PROVES verdict while remaining a drop-in preprocessing step with negligible compute cost.

  • Conclusion: A single FFT call whitens coloured noise, cleans innovations for covariance regression, generates spectral bandit context, and deseasonalises regression features.These four products serve Kalman updates, covariance calibration, bandit regime selection, and sensitivity-coefficient regression.
  • Conclusion: Across six independent domains, the pipeline returns a PROVES verdict against pre-registered metrics.The domains span aerospace GNC, autonomous vehicles, clinical drug monitoring, and three revenue-management classes.
  • Conclusion: The FFT preprocessing step has negligible deployment cost and requires only a single drop-in integration point.The paper contrasts this with maintaining four separate pipeline components.
  • Conclusion: The method reframes periodic contamination as structured signal whose decomposition produces four independent operational outputs.The resulting design is presented as an integrated calibration substrate rather than an isolated filtering trick.

A Implementation Reference

The implementation reference describes an FFT-based notch and whitening pipeline using real FFTs, power spectral density thresholds, optional known frequencies, and inverse transforms. Role 4 applies the same deseasonalisation logic per parameter.

  • Implementation: The five-step notch filter is implemented in approximately 20 lines of NumPy.The reference includes the whitening function and the evaluation harness location.
  • Notch detection: The filter computes Y with rfft, estimates PSD as |Y|^2/N, and identifies peaks above psd.mean() + τ·psd.std().The peak mask can be augmented with known frequencies.
  • Notch detection: Known frequencies are iterated as f0 values and converted into a peak mask using bandwidth bw around each frequency.The reference uses a condition based on |freq - f0| < bw.
  • Reconstruction: The aperiodic spectrum Y_a removes detected peaks, while the periodic spectrum Y_p retains them before inverse FFT reconstruction.The output uses irfft with n=N.
  • Deseasonalisation: Role 4 uses a per-parameter known-frequency dictionary, including weekly, monthly, quarterly, and annual cycles for selected coefficients.Examples include β1 occ_sensitivity with {1/7, 1/365} and β3 gap_discount with {1/30, 1/91}.
  • Aliasing: Aliased harmonics are mapped into the Nyquist interval using f_mod = f % fs and min(f_mod, fs - f_mod).The aliased frequencies are then used as known_aliased inputs.
  • Whitening: The whitening implementation applies rfft to z, multiplies by inverse square-root PSD weights with ε, and reconstructs using irfft.The function signature is whiten(z, psd, eps=1e-6).
  • Evaluation: The full six-domain evaluation harness is located at ml/HITL_GDBK/B/evaluate.py.
Loading 2608.22023v1…