PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Multi-Teacher Scoring Principles for Unsupervised Time Series Anomaly Detection

Dongchan Cho, Jiho Han, Juwon Hwang, Changhee Min, Minsang Kim, Namsoon Jung

arXiv:2610.29063v1cs.LGcs.AI

TL;DR

Unsupervised TSAD has no anomaly labels to define a shared target, while heterogeneous detectors provide complementary scoring principles. AnoMix distills pairwise preferences from their normalized training scores into one student; across benchmarks it outperforms individual detectors and score ensembles, and its zero-shot variant remains effective on unseen datasets.

  • Problem

    Unsupervised TSAD lacks anomaly labels for a shared target, while heterogeneous detectors use different anomaly criteria and score scales.

    Source: §1 · p. 1

  • Method

    AnoMix independently normalizes frozen teachers’ training scores and uses CoRank to distill their consensus pairwise anomaly preferences into one student.

    Source: §1 · p. 2

  • Results

    AnoMix outperforms individual detectors and score-ensemble baselines, while AnoMix-ZS remains effective on unseen datasets and can compete with larger pretrained time-series foundation models.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The results support transferring complementary teacher scoring principles into a single lightweight student rather than merely combining teacher scores.

    Source: §5 · p. 10

    A weak teacher pool or correlated teacher failures can produce unreliable consensus supervision, and clean training splits or nearly uniform scores may leave few meaningful pairs.

    Source: Q Limitations and Future Work · p. 32

Abstract

from arXiv · show

Time series anomalies take diverse forms, and individual detectors encode different anomaly scoring principles that rarely cover all anomaly types. Recent unsupervised time-series anomaly detectors provide complementary scoring signals, yet deploying them as an ensemble preserves the inference and maintenance cost of every individual detector. We study whether heterogeneous teacher detectors can be distilled into a single lightweight detector in the unsupervised setting, where no anomaly labels are available to define a shared target. AnoMix addresses this by extracting weak pairwise supervision from the training-set anomaly scores of frozen teachers. It normalizes each teacher score sequence independently and trains the student with CoRank, a multi-teacher ranking-distillation objective that converts teacher score gaps into pairwise supervision through teacher consensus. Across seven benchmark datasets, AnoMix successfully integrates complementary teacher scoring principles into a single deployable model, improving over individual detectors and static score ensembles while using a compact student with under 80k parameters on the benchmark settings. A zero-shot variant further shows that the distilled ranking signal transfers to unseen datasets and can outperform large pretrained time-series foundation models for anomaly detection.

1 Introduction

Unsupervised TSAD lacks a shared anomaly criterion, and heterogeneous detectors capture complementary abnormality patterns. AnoMix distills their training-set score comparisons into one student rather than fitting incompatible teacher score scales.

1 Introduction

  • Different TSAD detectors encode complementary anomaly criteria, while no single detector dominates across datasets.An ensemble can exploit this complementarity but retains every detector’s inference cost and maintenance overhead.Source: §1 · p. 1
  • In unsupervised TSAD, continuous teacher scores are detector-specific and unverified, making direct regression to multiple score scales structurally mismatched.There are no anomaly labels to anchor a shared target.Source: §1 · p. 2
  • AnoMix uses pairwise anomaly preferences from frozen teachers, with independently normalized scores and CoRank consensus, to train a compact student without pointwise score regression.CoRank derives comparison directions and confidence from teacher-level score gaps.Source: §1 · p. 2
  • Across TSAD benchmarks and zero-shot cross-dataset evaluation, AnoMix and AnoMix-ZS perform strongly against individual detectors, static ensembles, and pretrained foundation-model baselines.The contribution frames these results as evidence that multi-teacher preferences can be distilled into one lightweight student.Source: §1 · p. 2

2 Related Work

Prior distillation and ranking work transfers outputs, representations, or relations, but generally assumes a defined transfer target or target ordering. AnoMix addresses label-free transfer from heterogeneous TSAD teachers by constructing the pairwise supervision itself.

2 Related Work

  • Knowledge distillation has expanded from softened outputs to features, representations, and sample relations, while multi-teacher methods aggregate supervision from several teachers.For anomaly detection, prior distillation has focused mainly on image settings, often using one teacher and teacher–student discrepancy as the anomaly signal.Source: §2 · p. 2
  • Time-series anomaly-detection distillation remains relatively underexplored, and prior work does not address label-free transfer from heterogeneous TSAD teachers to one student.AnomalyLLM studies single-teacher feature mimicry and discrepancy-based detection [29].Source: §2 · p. 3
  • Existing ranking objectives optimize an available ordering or relation, whereas AnoMix constructs pairwise supervision from heterogeneous teacher score sequences without labels or a single calibrated teacher.Examples include RankNet, max-margin ranking, ListNet, and differentiable ranking [33].Source: §2 · p. 3

3 Methodology

AnoMix trains a single student from frozen teacher scores on unlabeled training windows, converting teacher-level score gaps into confidence-weighted pairwise preferences. At inference, the teachers are discarded and the student produces pointwise anomaly scores.

3 Methodology

  • Frozen teachers score the unlabeled training series, and CoRank selects informative pairs whose consensus directions and confidence guide student learning.The student learns pairwise ordering through a confidence-weighted loss.Source: §3 · p. 3

3.1 Problem Setup

  • The setup uses unlabeled multivariate training windows, frozen pre-trained teachers, and a student that predicts one scalar score for each window endpoint.No anomaly labels are used; normalized teacher scores provide the training supervision, and the student alone scores test windows.Source: §3.1 · pp. 3–4

3.2 Teacher Score Normalization

  • Each teacher’s scores are independently min–max normalized using training statistics, preserving within-teacher ordering while making score-gap ranges comparable for pairwise supervision.Normalization does not make the scores a reliable shared regression target.Source: §3.2 · p. 4

3.3 Student Architecture

  • The lightweight student is a causal temporal convolutional network with one scalar output for the final timestep of each input window.The design separates the distillation method’s effect from gains due to a complex student architecture.Source: §3.3 · p. 4

3.4 CoRank Loss

  • CoRank retains pairs with meaningful teacher score gaps, aggregates signed gaps into a consensus direction, and weights each comparison by teacher agreement.A high confidence weight indicates agreement among teacher gaps, not correctness; batches with no retained pairs contribute zero loss.Source: §3.4 · pp. 4–5

3.5 Training and Inference

  • During training, the student optimizes on batches paired with normalized teacher scores; at inference, teachers are discarded and the student outputs pointwise anomaly scores.Pairwise training therefore yields a standard pointwise score for each test window.Source: §3.5 · p. 5

4 Experiments

Experiments on seven real-world multivariate benchmarks compare AnoMix with individual detectors, static score ensembles, and distillation variants. AnoMix leads overall, while ablations, teacher-pool analyses, and held-out transfer tests examine its student, CoRank, and zero-shot generalization.

4.1 Setup

  • The evaluation covers seven real-world multivariate datasets and centers on V-ROC and V-PR among six threshold-independent metrics; the fixed teacher set is SARAD, CrossAD, and OracleAD.The benchmark student is a TCN with fewer than 80k parameters and is trained without anomaly labels.Source: §4.1 · pp. 5–6

4.2 Main Results

  • AnoMix achieves the strongest overall average rank across seven benchmarks, turning dataset-dependent detector complementarity into a single student.Table 1 reports results across 14 dataset-metric values.Source: Table 1 · p. 6, §4.2 · p. 6
  • With the teacher pool fixed, AnoMix achieves a substantially better average rank than static score-fusion baselines, including the strongest fusion rule, RRF.The gaps are especially large on PSM, SWaT, SMD, and SMAP.Source: Table 2 · p. 6, §4.2 · p. 7

4.3 Ablation Studies

  • TCN achieves 73.93% macro V-PR, above Transformer (62.51%), Dilated-RNN (58.90%), and LSTM (55.63%) under the same CoRank objective.The comparison uses the same endpoint-scoring template across backbones.Source: Figure 2 · p. 7, §4.3 · p. 7
  • CoRank reaches 73.93% macro V-PR, exceeding RankNet (68.01%), Hinge (67.75%), ListNet (64.60%), Spearman (68.27%), and MeanRank (69.46%).Score regression reaches 67.97%; decomposition gives 69.48% with added consensus confidence, 73.54% with consensus support, and 73.93% for full CoRank.Source: Figure 2 · p. 7, §4.3 · pp. 7–8

4.2 Main Results

  • Improvements vary by dataset and metric: gains are largest on PSM, SMD, and especially SMAP, small on near-ceiling Exathlon, and larger on V-PR than V-ROC.Source: §4.2 · p. 7

4.4 Distillation Analysis

  • LOO-1 is substantially weaker than multi-teacher variants; All-10 ranks best overall but degrades PSM substantially, while Sel-3 remains close to stronger LOO variants.Across five non-saturated datasets, AnoMix has higher V-PR and lower average Spearman correlation with teacher scores than score-ensemble baselines.Source: Table 3 · p. 8, §4.4 · pp. 8–9, Figure 3 · p. 8

4.5 Zero-Shot Cross-Dataset Detection

  • In strict zero-shot transfer, AnoMix-ZS achieves the best average rank (1.64) with 83k parameters, versus foundation baselines ranging from 1.84M to 710M parameters.It is trained on six datasets and evaluated on a held-out target without observing that target or its teacher scores during training.Source: Table 4 · p. 9, §4.5 · p. 9

5 Conclusion

AnoMix distills pairwise anomaly preferences from heterogeneous teachers into a single student, and the experiments indicate that it outperforms individual detectors and score ensembles. The section also describes CoRank’s supervision, the student architectures and pipeline, and visual comparisons of test-time scores.

5 Conclusion

  • AnoMix outperforms individual detectors and score-ensemble baselines across benchmarks, while zero-shot AnoMix-ZS remains effective on unseen datasets and can compete with larger pretrained time-series foundation models.Teacher-set and correlation analyses indicate that AnoMix learns beyond static ensemble behavior.Source: §5 · p. 10

A Rationale for Multi-Teacher Ranking Distillation

  • CoRank converts heterogeneous teacher score gaps into pairwise supervision, selecting informative pairs before aggregation and discounting directional conflict instead of fitting teacher-specific score scales.Its consensus direction is compatible with the student’s scalar output, and its conflict weight is not an estimate of teacher correctness.Source: A Rationale for Multi-Teacher Ranking… · p. 15

B Student Architectures

  • The benchmark student has under 80k parameters, while AnoMix-ZS uses a channel-independent encoder for variable-dimensional transfer and has 83k parameters independent of the number of variables.For each target dataset, AnoMix-ZS trains on the other six datasets without target-dataset fine-tuning, target teacher scores, or target training data.Source: B Student Architectures · p. 16

C Pseudocode

  • Algorithm 1 separates AnoMix into offline teacher-score preprocessing, single-head student training with CoRank, and teacher-free inference.Teacher scores are normalized on the training data before pairwise supervision is constructed for student training.Source: C Pseudocode · p. 17

D Visualization

  • Across seven benchmark datasets, AnoMix often assigns high test-time scores to labeled anomaly segments without simply copying any single teacher’s scores.Figure S1 displays normalized scores for SARAD, CrossAD, OracleAD, and AnoMix, with anomaly segments shaded in red.Source: D Visualization · p. 18, Figure S1 · p. 18

E Full Results

The full-results section presents five-seed mean results across seven benchmark datasets for PR, ROC, and F1-based metric families. The supplied passages identify the metrics and tables but do not provide cell values or performance comparisons.

E Full Results

  • Table S1 reports five-seed mean VUS-PR, AUC-PR, and R-AUC-PR results across the seven benchmark datasets.AUC-PR is especially informative under severe class imbalance; range-based R-AUC-PR relaxes exact timestamp matching.Source: E Full Results · p. 19, Table S1 · p. 19
  • Table S2 reports five-seed mean VUS-ROC, AUC-ROC, and R-AUC-ROC results across the seven benchmark datasets.AUC-ROC summarizes global ranking quality, while range-based R-AUC-ROC credits predictions within a tolerance window around anomaly segments.Source: E Full Results · p. 19, Table S2 · p. 19
  • Table S3 reports five-seed mean point-wise F1, Event-F1, and Aff-F results across the seven benchmark datasets.The passages name these metrics but do not report their values.Source: E Full Results · p. 19, Table S3 · p. 19

F Datasets

The paper evaluates seven multivariate time-series anomaly detection benchmark datasets selected to span differing dataset sizes, anomaly ratios, event counts, and numbers of variates. The first six come from TSB-AD and SWAN comes from TAB.

F Datasets

  • The seven benchmark datasets were selected to cover varied training-set sizes, anomaly ratios, event counts, and numbers of variates.Subsets from the same benchmark source are treated as separate benchmark instances because they differ in dimensionality and data distribution.Source: F Datasets · p. 20, Table S4 · p. 20
  • The first six datasets are sourced from TSB-AD, while SWAN is sourced from TAB.TSB-AD supplies standardized preprocessing, train/test splits, and anomaly labels.Source: F Datasets · p. 20
  • The benchmarks span server monitoring, water-treatment sensors, distributed computing, satellite telemetry, wearable accelerometers, and solar magnetic-field monitoring.The described datasets include PSM, SWaT, Exathlon, SMD, SMAP, Daphnet, and SWAN, with anomalies tied to incidents, attacks, failures, or domain-specific events.Source: F Datasets · p. 20

G Metrics

The paper reports threshold-independent PR and ROC metrics to assess different aspects of time-series anomaly detection. These metrics summarize score performance across thresholds rather than at one chosen threshold.

G Metrics

  • Threshold-independent PR and ROC metrics cover pointwise discrimination, tolerance to small temporal offsets, and ranking quality across the full score range.They evaluate anomaly-score quality across possible thresholds rather than one threshold choice.Source: G Metrics · p. 21
  • AUC-PR and AUC-ROC measure standard pointwise curve areas, while R-AUC-PR and R-AUC-ROC credit predictions within a tolerance window around anomaly segments.AUC-PR is especially informative under severe class imbalance; AUC-ROC summarizes global ranking quality.Source: G Metrics · p. 21
  • VUS-PR and VUS-ROC integrate range-based AUC over both score thresholds and tolerance-window sizes.This provides a broader threshold-free view than standard or fixed-range AUC.Source: G Metrics · p. 21

H Baselines and Experimental Protocol

The protocol compares unsupervised detectors, score-ensemble and zero-shot baselines under shared evaluation controls. Its fixed Sel-3 teacher pool is presented as a compact, representative setting, with leave-one-out rankings assessing its cross-dataset competitiveness.

H Baselines and Experimental Protocol

  • Benchmark detectors include stochastic or inconsistency-based reconstruction, association discrepancy, and forecasting-style prediction deviations; pretrained MOMENT keeps its backbone frozen while its reconstruction head is fine-tuned for 2 epochs per target training split.Source: H Baselines and Experimental Protocol · p. 22
  • All baselines and teachers were rerun on common files and fixed splits without anomaly labels, using the same TSB-AD metric implementation.Five-run comparisons use five fixed seeds; range-based and VUS metrics use an evaluation window of 100.Source: H Baselines and Experimental Protocol · p. 23

I.1 Teacher-Set Selection

  • Sel-3 combines SARAD, CrossAD, and OracleAD as a compact representative pool, not a claimed optimum.Leave-one-out rankings place SARAD and CrossAD among the top two teachers on six of seven target datasets; OracleAD or CATCH typically fills third place.Source: §I.1 · p. 24, Table S6 · p. 24

I.2 Teacher-Pool Robustness

Teacher-pool stress tests show that AnoMix can outperform both the best constituent teacher and a normalized mean ensemble across several pool types, but weak teachers with sparse pair support can fail. The tested pools broaden evidence beyond Sel-3 without establishing robustness to shared systematic teacher errors.

I.2 Teacher-Pool Robustness

  • 9.65%pt: HML-3 improves over its best teacher, while Bottom-3 has only 3.9% pair support and falls below both comparators.AnoMix also exceeds both comparators with medium-quality and highly correlated pools; the reported pools are fixed rather than random subsets.Source: §I.2 · pp. 24–25, Table S7 · p. 25
  • Sel-3 provides broader pair support and higher student performance than Corr-3, consistent with complementary teachers supplying informative gaps on different pairs.The comparisons do not isolate teacher quality from coverage, and unanimous teacher agreement can still be wrong.Source: §I.2 · p. 25

J Implementation Details

AnoMix uses a compact single-head TCN trained on cached teacher supervision, with sensitivity analyses identifying useful ranges for its main hyperparameters. Training losses decrease stably across six datasets, and student-only inference has low reported cost, although the measurements exclude offline teacher computation and training.

J Implementation Details

  • The single-head TCN has under 80k parameters on the benchmark settings and uses a 127-step receptive field; training uses no anomaly labels.The model is trained for 10 epochs with AdamW and CoRank threshold δ = 0.1; student variability is conditional on fixed teacher-score caches.Source: J Implementation Details · p. 26
  • δ ≥ 0.2 reduces macro performance by roughly seven points, while window size and batch size are largely flat at moderate values.A window size of 10 collapses performance; the reported new-dataset ranges are L ∈ [64, 127], δ ∈ [0.05, 0.1], and batch size 32–64.Source: J Implementation Details · p. 26, Figure S2 · p. 26

K Optimization

  • CoRank loss decreases monotonically over 10 epochs on all six benchmark datasets, with narrow across-seed bands.The figure reports five-seed means and ±1 standard deviation.Source: K Optimization · p. 27, Figure S3 · p. 27

L Model Complexity and Inference Efficiency

  • 8.65M FLOPs per window and 9.59 MB of peak inference memory correspond to 127,000× fewer FLOPs than MOMENT and 93× lower peak memory than OracleAD.These measurements use a standardized synthetic input and describe single-model inference.Source: L Model Complexity and Inference… · p. 28, Table S9 · p. 28
  • The reported inference savings exclude teacher acquisition, scoring and caching, and student training, so they do not establish lower end-to-end training cost.The measurements concern deployment with the student alone.Source: L Model Complexity and Inference… · p. 28

M Additional Ablations

Additional analyses examine score normalization, pair selection, teacher alignment, and broader deployment considerations. Min–max normalization and a shared threshold perform well in the reported ablations, while confidence filtering lowers performance on SWaT and deployment risks remain domain-dependent.

M Additional Ablations

  • Min–max normalization attains the highest seven-dataset macro VUS-PR, with z-score and rank within roughly one point.The normalization comparison is reported as five-seed averages.Source: M Additional Ablations · p. 29, Figure S4 · p. 29
  • The fixed δ = 0.1 matches or slightly exceeds teacher-specific thresholds on seven-dataset macro VUS-PR.The alternatives use each teacher’s mean or 25th-percentile training-score gap as its threshold.Source: M Additional Ablations · p. 29, Figure S4 · p. 29

N Teacher Alignment and Consensus Confidence

P Broader Impact

  • AnoMix is presented as a way to consolidate detector signals without anomaly labels, but false-positive and false-negative impacts vary by deployment domain.The authors call for domain-specific validation, threshold calibration, human oversight, and continuous monitoring.Source: P Broader Impact · p. 32

Q Limitations and Future Work

AnoMix’s supervision can become unreliable when teachers are weak or share errors, and scarce score gaps can limit the available pairwise training signal.

Q Limitations and Future Work

  • Weak target-domain teachers can pass unreliable supervision, while correlated teacher failures can make CoRank confidently reinforce incorrect pairwise preferences.The authors suggest automatic teacher-pool construction and diversity-aware selection as possible future responses.Source: Q Limitations and Future Work · p. 32
  • Very clean training splits or nearly uniform teacher scores may leave too few meaningful gaps for pairwise anomaly-preference supervision.The authors suggest synthetic anomaly injection as one possible way to enrich pairwise training signals.Source: Q Limitations and Future Work · p. 32
  • Proposed future directions include label-free model-selection criteria, source-data selection, and synthetic anomaly injection to address supervision and teacher-pool weaknesses.The authors also identify automatic teacher-pool construction and diversity-aware teacher selection.Source: Q Limitations and Future Work · p. 32
Loading 2610.29063v1…