PaperlayerView exact revision on arXiv

Source-linked AI summary

Surprised by Attention: Predictable Query Dynamics for Time Series Anomaly Detection

Kadir-Kaan Özer, René Ebeling, Markus Enzweiler

arXiv:2603.12916v3cs.LGcs.AI

TL;DR

Multivariate anomalies can alter cross-channel dependencies while leaving amplitudes plausible, limiting residual-only detection. AxonAD couples reconstruction with history-only prediction of attention queries, using their mismatch alongside reconstruction error. It improves ranking and localization on proprietary telemetry and TSB-AD, while evaluation remains bounded by nominal-only training and an offline alignment procedure that introduces lookahead for benchmark localization.

  • Problem

    Residual-based detectors can miss cross-channel dependency shifts when flexible sequence models reconstruct individual signals plausibly despite altered coordination.

  • Method

    AxonAD couples a bidirectional reconstruction pathway with a history-only predictor trained by masked cosine loss against an EMA target encoder, then combines standardized reconstruction and query-mismatch scores.

  • Results

    AxonAD improves threshold-free ranking and temporal localization over strong baselines across proprietary telemetry and TSB-AD; on telemetry, AUC-PR is 2.2× the next-best baseline and Event-F1 is 1.6×.

  • Takeaways & Limitations

    Query prediction and combining query mismatch with reconstruction are necessary for the strongest aggregate performance, supporting sensitivity to structural shifts while retaining amplitude-level detection.

  • Takeaways & Limitations

    Training and robust scoring statistics use nominal windows only, while offline TSB-AD localization assigns centered scores and effectively introduces window-dependent lookahead.

Abstract

from arXiv · show

Multivariate time series anomalies often manifest as shifts in cross-channel dependencies rather than simple amplitude excursions. In autonomous driving, for instance, a steering command might be internally consistent but decouple from the resulting lateral acceleration. Residual-based detectors can miss such anomalies when flexible sequence models still reconstruct signals plausibly despite altered coordination. We introduce AxonAD, an unsupervised detector that treats multi-head attention query evolution as a short horizon predictable process. A gradient-updated reconstruction pathway is coupled with a history-only predictor that forecasts future query vectors from past context. This is trained via a masked predictor-target objective against an exponential moving average (EMA) target encoder. At inference, reconstruction error is combined with a tail-aggregated query mismatch score, which measures cosine deviation between predicted and target queries on recent timesteps. This dual approach provides sensitivity to structural dependency shifts while retaining amplitude-level detection. On proprietary in-vehicle telemetry with interval annotations and on the TSB-AD multi-variate suite (17 datasets, 180 series) with threshold-free and range-aware metrics, AxonAD improves ranking quality and temporal localization over strong baselines. Ablations confirm that query prediction and combined scoring are the primary drivers of the observed gains. Code is available at the URL https://github.com/iis-esslingen/AxonAD.

1 Introduction

AxonAD targets coordination breaks that residual-based detectors may miss by modeling attention-query evolution as a predictable process. It combines query mismatch with reconstruction error and improves ranking and temporal localization across the evaluated settings.

  • Motivation: Coordination breaks can leave individual channels within plausible ranges while changing cross-channel dependencies.Examples include steering commands decoupling from lateral acceleration or throttle position decoupling from engine torque.
  • Motivation: Residual-based detectors may miss dependency shifts because flexible sequence models can reconstruct each channel accurately despite altered coordination.Low reconstruction error therefore does not guarantee preservation of the full dependency structure.
  • Core idea: Under stationary nominal dynamics, attention query vectors should evolve predictably over short horizons, whereas structural anomalies can disrupt this predictability.Query mismatch is therefore complementary to reconstruction error when amplitudes remain plausible.
  • Method: AxonAD couples bidirectional reconstruction with a history-only predictor trained using masked cosine loss against an EMA target encoder.The method treats query vectors as a temporally predictable signal rather than a one-shot routing decision.
  • Method: At inference, reconstruction error and query mismatch are robustly standardized on nominal training data and summed into the final anomaly score.This combines amplitude-level and structural-dependency signals.
  • Evaluation: Across proprietary telemetry and the TSB-AD multivariate suite, AxonAD improves threshold-free ranking and temporal localization relative to strong baselines.Ablations identify query prediction and score combination as the primary drivers of the gains.

2 Related Work

Prior anomaly detectors commonly score isolation, density, decomposition, reconstruction, forecasting, attention, or value-residual signals. AxonAD instead makes attention-query predictability over time both its predictive objective and inference signal.

  • Classical methods: Isolation, density, neighborhood, decomposition, clustering, histogram, and copula methods capture separation, local geometry, low-rank structure, or density surrogates.These approaches do not directly capture the full contextual dependency dynamics described here.
  • Deep detectors: Deep detectors typically learn nominal dynamics through reconstruction or forecasting and score residual magnitude or likelihood deviation.Residual scoring can miss dependency shifts when per-channel values remain plausible.
  • Attention-based methods: Attention-based detectors have scored association discrepancies, sensor relations, attention weights, or value residuals.AxonAD differs by scoring what the model is about to attend to through query-vector predictability.
  • AxonAD: AxonAD uses reconstruction and query mismatch signals, with the default pipeline excluding attention-divergence KL tail scoring.The overview distinguishes Qrec, predicted queries, EMA target queries, drec, and dq.
  • AxonAD: Unlike common predictive objectives on values or latent states, AxonAD supervises directly in query space so training and inference use the same cosine distance.The objective is stabilized with EMA target networks and masking.

3 Model Architecture

AxonAD combines bidirectional reconstruction with a history-only predictor of future multi-head query vectors, using an EMA target encoder for stable masked supervision. Reconstruction and recent-tail query mismatch are robustly standardized and summed into one anomaly score.

  • Overview: The model takes fixed-length windows and produces reconstructions alongside reconstruction and query-mismatch signals for final scoring.The two signals are combined after robust standardization.
  • Overview: AxonAD has gradient-updated reconstruction, history-only query prediction, and an EMA target encoder providing stable supervision targets.The reconstruction pathway uses bidirectional self attention, while the predictive pathway forecasts future queries from shifted embeddings.
  • Shared representation: The shared embedding feeds both reconstruction attention and the predictive branch, while the online encoder forms multi-head queries, keys, and values.The embedding uses a linear projection with learnable positional bias followed by layer normalization.
  • Online reconstruction pathway: The reconstruction pathway applies full-window multi-head self attention, feedforward processing, and a linear output head before computing mean squared ℓ2 reconstruction error.This signal measures average reconstruction discrepancy across timesteps.
  • Predictive attention pathway: The causal predictive pathway uses only past context to forecast future query vectors, targeting queries rather than keys or values.The history-only shift ensures predictions at timestep τ depend only on embeddings available up to τ −s.
  • EMA target and masked training: Masked training compares predicted and EMA target queries with cosine loss, updating the predictor while stopping gradients through the target.The EMA encoder tracks online parameters without gradient updates, and masking samples contiguous future timesteps.
  • Query mismatch and final score: At inference, query mismatch averages cosine deviations between predicted and EMA target queries over valid recent timesteps, then combines with reconstruction error.The tail index enforces the history constraint and recent-timestep focus; robust median/IQR standardization uses nominal training windows.
  • Query mismatch and final score: The additive score detects anomalies that elevate reconstruction error, query mismatch, or both, separating amplitude and coordination anomalies from nominal windows.The schematic uses a diagonal constant-score contour to capture complementary anomaly types.

4 Experimental Setup

AxonAD is evaluated under strictly unsupervised protocols on proprietary telemetry and the 17-dataset TSB-AD multivariate suite. The setup uses chronological, anomaly-free training partitions, fixed cross-dataset configuration, official metrics, and explicit latency alignment for offline evaluation.

  • Evaluation protocol: Evaluation covers proprietary in-vehicle telemetry and the TSB-AD multivariate suite spanning 17 datasets and 180 series.
  • Evaluation protocol: Training is strictly unsupervised, with parameters and robust scoring statistics fit only on nominal training windows and labels reserved for evaluation.Telemetry labels are excluded from hyperparameter selection, thresholding, postprocessing, and early stopping.
  • Datasets and split: The proprietary stream contains 80,000 timesteps and 19 continuous channels, with 30 annotated anomaly intervals and a chronological split at timestep 40,000.The first anomaly occurs at index 43,410, leaving training and validation anomaly free.
  • Scoring and alignment: Window scores use only samples through timestep t for real-time detection, but offline TSB-AD scores are assigned to window centers, introducing a lookahead of floor((T − 1)/2) steps.
  • Baselines and metrics: The benchmark compares classical, reconstruction, forecasting, variational, and Transformer-based detectors using threshold-free and range-aware metrics.Metrics include AUC-ROC, AUC-PR, VUS-ROC, VUS-PR, PA-F1, Event-F1, Range-F1, and Affiliation-F1.
  • Model configuration: A single configuration is used across datasets, including a causal dilated predictor, EMA target encoder with momentum m=0.9, masked query supervision, and reconstruction-plus-query-loss training.Unless stated otherwise, results use T=100, D=128, 8 attention heads, forecast horizon s=1, and tail length k=10.

5 Results

AxonAD substantially improves anomaly ranking and temporal localization on proprietary telemetry and generalizes to the TSB-AD multivariate benchmark. The gains are broadly distributed across series and are associated with query prediction and combined scoring.

  • Proprietary telemetry: AUC-PR reaches 0.285 on proprietary telemetry versus 0.128 for the next-best method, SISVAE.
  • Proprietary telemetry: Event-F1 is 0.420 versus 0.255 and Range-F1 is 0.328 versus 0.262, showing stronger temporal localization than the next-best method.
  • TSB-AD benchmark: On TSB-AD, AxonAD has the highest mean AUC-PR of 0.437, VUS-PR of 0.493, and Range-F1 of 0.471.M2N2 leads on PA-F1, while VASP and OmniAnomaly are competitive on Affiliation-F1.
  • Cross-series analysis: AxonAD wins on a clear majority of the 180 series against every baseline, with all paired Wilcoxon signed-rank tests yielding p < 10^-4.

6 Ablation Studies

Ablations show that query prediction, cosine-based score combination, moderate EMA momentum, and short forecast horizons drive AxonAD’s ranking and localization performance. Query mismatch complements reconstruction because the two signals capture distinct anomalous-window regimes.

  • Scoring components: Removing the query branch reduces VUS-PR by 0.055 and Event-F1 by 0.117, while query-only scoring reduces AUC-PR by 0.145 and AUC-ROC by 0.097.The cosine-based combined score provides the strongest aggregate behavior across metric families.
  • Scoring components: Replacing cosine query mismatch with MSE produces a similar performance drop, indicating that the cosine formulation matters beyond combining two signals.
  • EMA and masking: Removing EMA or using extreme momentum values reduces performance, supporting default momentum m = 0.9 as a balance between target stability and responsiveness.EMA 0 reduces AUC-PR by 0.024 and Event-F1 by 0.051; EMA 0.99 incurs an AUC-PR penalty of 0.048.
  • Prediction target: Predicting query vectors outperforms predicting keys, values, attention maps, or hidden states across ranking and localization metrics.
  • Parameter sensitivity: Performance peaks at s = 1 with AUC-PR 0.545, while larger horizons increase prediction difficulty and score variance; tail-length ranking remains stable for k ∈ {3, 5, 10, 20}.Range-F1 peaks at k = 10, suggesting that k primarily controls temporal smoothing.
  • Mechanistic diagnostics: Query deviation correlates with tail KL divergence at median ρ = 0.677, and high-query-mismatch/low-reconstruction windows occur alongside the reverse regime.The two signals have median correlation ρ = 0.211, and combining them improves AUC-PR over the best single component in 8 of 20 series.
  • Runtime: AxonAD has 334 s fitting cost but scores each window in 0.069 ms, below OmniAnomaly at 0.190 ms and Isolation Forest at 0.461 ms.The fitting cost reflects iterative training and is amortized at deployment.

7 Conclusion

AxonAD monitors the short-horizon predictability of attention queries alongside reconstruction to detect dependency shifts in multivariate telemetry. It improves ranking and localization across vehicle telemetry and TSB-AD, with low inference latency and ablations supporting query prediction plus combined scoring.

  • AxonAD couples bidirectional reconstruction with masked EMA distillation in query space, producing query mismatch alongside reconstruction residuals.
  • AxonAD improves proprietary-telemetry AUC-PR by 2.2× and Event-F1 by 1.6× over the next best baseline.
  • On TSB-AD, AxonAD leads on threshold-free ranking and range-aware localization across 17 datasets and 180 series.
  • Ablations establish that query prediction outperforms alternative targets and that combining reconstruction with query mismatch is necessary for best aggregate performance.
  • Per-window inference latency is 0.069 ms, and the evaluation reports no label-based threshold tuning.

S1 Additional Tables

The supplementary tables document the TSB-AD benchmark scope, proprietary-dataset protocol, and pairwise baseline comparisons, including reported deltas and statistical tests.

  • Table S1 covers 17 TSB-AD multivariate datasets and 180 time series, reporting mean ± standard deviation over evaluated series.
  • Table S2 reports the TSB-AD multivariate benchmark protocol on the proprietary dataset using mean ± standard deviation over four random seeds.
  • Table S3 compares each baseline with AxonAD using win-rate, mean and median performance deltas, Wilcoxon p-values, and 95% confidence intervals.Negative deltas indicate that the baseline is worse than AxonAD in the export.
  • Table S4 continues the pairwise comparison for VUS-PR, R-based-F1, and Affiliation-F1, while several other metrics are not shown in the export.
Loading 2603.12916v3…