PaperlayerView exact revision on arXiv

Source-linked AI summary

READ-Bench: Benchmarking Historical Instance Retrieval for Time-Series Diagnosis

Gerardo Pastrana, Haojun Li, Dhruv Mehta, Anoushka Vyas, Sina Khoshfetrat Pakazad, Henrik Ohlsson, John Paparrizos

arXiv:2609.32123v2cs.AI

TL;DR

Time-series diagnosis has limited direct evaluation of historical-case retrieval, although relevant cases may differ in signal shape. READ-Bench evaluates fault-level retrieval across 12 datasets and compares retrieval stages and methods. The study finds no detectable pretrained-embedding advantage for search alone, while its combined system improves over its search stage on all 12 datasets.

  • Problem

    Historical-case retrieval is rarely used in time-series diagnosis and is often evaluated only indirectly, despite relevance depending on shared fault type rather than signal similarity.

    Source: §1 · pp. 1–2

  • Method

    READ-Bench defines fault- and event-level relevance across 12 datasets and systematically compares retrieval methods, fusion, and reranking under varied corpus conditions.

    Source: §1 · pp. 2–3

  • Results

    Pretrained embeddings show no advantage over strong classical and symbolic baselines, while the combined system improves over its search stage on all 12 datasets.

    Source: §1 · p. 3, §5 · pp. 9–10

  • Takeaways & Limitations

    Reranking is a reliable addition across retrieval pools, including under normal-series pollution, whereas fusion helps only in select settings and language-model reranking has almost no detectable effect.

    Source: §5 · p. 10

    Evaluation is bounded by source-label fidelity and categorization granularity, which may obscure subtle cascading failures or multi-fault interactions.

    Source: §5 · p. 10

Abstract

from arXiv · show

Time-series diagnostic systems rarely rely on retrieving relevant historical cases, and when they do, retrieval is evaluated only indirectly through downstream prediction. We introduce READ-Bench, a benchmark for historical-case retrieval across 12 diagnostic datasets, centered on multivariate time series, that defines relevance by shared fault or event type rather than signal shape, so visually different traces of the same fault count as relevant while similar-looking traces of different faults do not. Treating retrieval as a base retriever followed by a reranker, we evaluate classical distances, symbolic retrievers, self-supervised and foundation-model embedders, and their fusion, plus label-aware and language-model rerankers, under one protocol that varies supervision, pollution, and corpus scale with significance testing. Under a common channel-independent interface, pretrained representations offer no statistically detectable advantage over strong classical and symbolic baselines for search alone. The decisive factor is a small amount of resolved-case supervision at reranking, namely a Gaussian-process reranker that propagates a few neighbor labels in embedding space, which helps far more than more sophisticated representations or language-model reasoning and holds under pollution and at full corpus scale. Guided by these findings, we fuse a normal-residual-scored embedder with a dynamic time warping leg via reciprocal-rank fusion, then rerank with the Gaussian-process reranker, improving NDCG@10 over its own search stage on all 12 datasets, by +0.11 from reranking and +0.16 over the strongest single base retriever.

1 INTRODUCTION

Time-series diagnosis can benefit from retrieving historical cases, but diagnostic relevance depends on shared fault or event type rather than signal similarity. READ-Bench enables direct comparison of retrieval methods and introduces a combined system that improves over its search stage across all 12 datasets.

1 INTRODUCTION

  • Diagnostic relevance is not equivalent to signal similarity: one fault can produce different traces, while similar traces can have unrelated causes.End-to-end prediction alone cannot isolate retrieval quality from how retrieved cases are used downstream.Source: §1 · p. 2
  • READ-Bench provides explicit fault- and event-level relevance judgments across 12 datasets spanning five diagnostic domains.It includes multivariate and complementary univariate evaluations.Source: §1 · p. 2
  • The study compares classical, symbolic, and pretrained retrieval methods, fusion strategies, and rerankers under varying corpus composition and scale.This addresses limited cross-domain comparisons, especially when relevant instances differ in signal shape.Source: §1 · pp. 2–3
  • The combined retrieval component improves NDCG@10 over its underlying search stage on all 12 datasets and outperforms single-method retrievers across five diagnostic domains.It combines representation-based search, dynamic-time-warping search, and label-aware Gaussian-process reranking.Source: §1 · p. 3

2 RELATED WORK

Prior work applies retrieval to time-series tasks and studies representations, ranking methods, and benchmarks, but evaluation often remains tied to downstream tasks or restricted settings. READ-Bench addresses the need for broader diagnostic retrieval comparisons when relevant cases can differ in signal shape.

2 RELATED WORK

  • Many retrieval-augmented time-series systems are evaluated on final tasks, making retrieval quality difficult to separate from downstream use of retrieved instances.Examples span anomaly detection, PHM diagnosis, motion-series reasoning, and forecasting.Source: §2 · p. 3
  • Existing time-series retrieval methods use raw distances, symbolic representations, learned embeddings, metadata, and ranking or fusion strategies.For multivariate series, channel combination and irrelevant channels can affect similarity.Source: §2 · p. 3
  • TSRBench covers similar-series and industrial-incident retrieval, but its open similar-series setting ties relevance closely to shape and its multivariate diagnostic setting covers one telecom domain.TS-Haystack instead evaluates sparse-event retrieval and reasoning in long time-series contexts, not corpus ranking of historical cases.Source: §2 · p. 3
  • Prior evaluations leave limited cross-domain comparisons of diagnostic retrieval, particularly when same-fault instances differ in signal shape.The cited benchmark and retrieval studies provide related frameworks and methods, but do not collectively address this scope.Source: §2 · p. 3

3 READ-BENCH

READ-Bench formalizes historical-case retrieval around fault-type relevance, constructs a varied 12-dataset benchmark, and evaluates retrieval as a staged pipeline. It compares representations, fusion, and reranking under a shared protocol with explicit metrics and significance testing.

3.1 PROBLEM FORMULATION

  • Retrieval ranks corpus windows for a query, with a reranker reordering a first-stage candidate pool; relevance requires a shared non-normal fault type.Queries are anomalous, normal windows are never relevant, and query labels are withheld from retrieval methods.Source: §3.1 · p. 4

3.2 BENCHMARK CONSTRUCTION

  • The benchmark draws 12 multivariate datasets from varied community and industrial sources, spanning domains, channel counts, window lengths, and fault-class distributions.Windows and labels follow dataset conventions where available; otherwise labeling uses a majority rule with a relaxation that recovers short faults.Source: §3.2 · pp. 4–5
  • Queries come from test splits and corpus items from disjoint train splits; corpus composition controls normal-window pollution at roughly 0%, 10%, and 20%.Anomalies of other fault types and normal windows serve as distinct distractors, while stratified sampling preserves class imbalance.Source: §3.2 · p. 5

3.3 METHODS

  • Base retrievers include raw-window distances, symbolic methods, and pretrained embedders; fusion combines complementary rankings, including with reciprocal-rank fusion.Normal-Residual scoring removes the expected-if-normal embedding component and compares residual directions; it requires normal windows but no fault labels.Source: §3.3 · pp. 5–6
  • Rerankers include label-free feedback, query-local label-aware methods, Gaussian-process classification, and language-model rerankers; supervised classifiers serve as a reference ceiling.The Gaussian-process reranker uses labels from the candidate pool and ranks candidates by the probability of sharing the query’s inferred class.Source: §3.3 · p. 6

3.4 EVALUATION PROTOCOL

  • Evaluation reports P@K, HR@K, and NDCG@K, with Friedman and Nemenyi tests across 12 datasets to distinguish significant differences from noise.Per-dataset metrics are descriptive robustness checks rather than formal hypothesis tests.Source: §3.4 · p. 7

4 EXPERIMENTS AND ANALYSIS

Across the benchmark, label-free retrieval methods cluster on a plateau, while label-aware reranking is the most reliable lift. Combining normal-residual scoring, DTW-I fusion, and GPC yields the strongest system.

4.1 BASE RETRIEVERS

  • The top four label-free retrievers are statistically indistinguishable and far below the supervised reference; their differences are smaller than variation across datasets.Only the weakest method, SBD-D, separates from the rest in the significance test.Source: §4.1 · p. 7

4.2 NORMAL-RESIDUAL SCORING AND ROBUSTNESS TO POLLUTION

  • NR roughly halves degradation under pollution and lifts clean accuracy for every wrapped embedder except TiRex, which is essentially unchanged.MantisV2 + NR is the strongest single base retriever on clean accuracy and pollution robustness.Source: §4.2 · p. 8, Figure 3 · p. 9

4.3 FUSION AND RERANKING

  • RRF fusion with DTW-I gives small, consistent, mostly-on-plateau gains, providing a stronger starting pool without lifting fusion alone off the plateau.The best fusion leg for every embedder is DTW-I.Source: §4.3 · p. 8
  • GPC significantly improves every tested pool and narrows, but does not close, the supervised-reference gap; label-free reranking harms most pools.Language-model reranking barely moves NDCG@10, while direct fault-type classification without retrieval scores near the random floor.Source: §4.3 · p. 9, Figure 4 · p. 9

4.4 COMBINING FUSION AND RERANKING

  • NDCG@10 reaches 0.687/0.640/0.619 at ρ=0/10/20% for CHARM + NR | DTW-I + GPC, strongest at every level.NR scoring carries the system clear of the plain-embedding stack, which reaches 0.631; fusion adds a smaller increment.Source: §4.4 · p. 9

5 CONCLUSION

READ-Bench finds that pretrained embeddings offer no advantage over strong classical and symbolic baselines, while small amounts of additional information, especially label data at reranking, are decisive. Reranking remains reliable under normal-series pollution, whereas score fusion helps selectively and language-model reranking has almost no detectable effect.

5 CONCLUSION

  • Pretrained embeddings show no advantage over strong classical and symbolic baselines, while small amounts of label data or normal periods are a decisive lever.The evaluation is bounded by source-label fidelity and categorization granularity, which may obscure subtle cascading failures or multi-fault interactions.Source: §5 · pp. 9–10
  • Reranking and Gaussian-process reranking improve the strongest retrieval pool, including under normal-series pollution.Source: §5 · p. 10
  • Score fusion helps only in select settings, while language-model reranking of a fixed pool shows almost no detectable effect.Source: §5 · p. 10

ETHICS STATEMENT

READ-Bench uses publicly available time-series datasets and is intended to support human operators rather than replace their judgment.

ETHICS STATEMENT

  • READ-Bench is assembled entirely from publicly available time-series datasets released by their originators for research use.Source: ETHICS STATEMENT · p. 10
  • The benchmark redistributes curated queries, corpora, and relevance judgments while retaining each source’s original license and attribution.Source: ETHICS STATEMENT · p. 10
  • Retrieved evidence should be reviewed by a qualified expert before it informs a decision.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

Generative AI tools were used only to refine and polish the paper’s writing; the authors carried out the research work and take responsibility for the final content.

AI USE STATEMENT

  • Generative AI tools were used to improve readability, tighten phrasing, and suggest a title and keywords.Source: AI USE STATEMENT · p. 10
  • The authors did not use generative AI to create data, develop models or hypotheses, design experiments, implement methods, or interpret results.Source: AI USE STATEMENT · p. 10
  • The authors reviewed all AI-assisted text and take responsibility for the paper’s final content, claims, and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper specifies its task, methods, metrics, significance testing, and compute settings, and provides the benchmark as a versioned public dataset with a leaderboard.

REPRODUCIBILITY STATEMENT

  • READ-Bench specifies its task, relevance definition, windowing rules, dataset curation, labeling, and split protocols.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The paper documents method definitions, evaluation metrics, significance testing, hyperparameters, and compute budget.Source: REPRODUCIBILITY STATEMENT · p. 10
  • All splits and subsampling use a fixed seed, and the benchmark is publicly available as a versioned dataset with a leaderboard and submission protocol.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors state that code for building corpora, running retrieval methods, and reproducing every table and figure will be released.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DATASET DETAILS

This appendix describes READ-Bench’s twelve datasets and how they are prepared for retrieval evaluation.

A.1 DATASET DETAILS

  • The appendix details the twelve datasets included in READ-Bench.Source: §A.1 · p. 16
  • It covers how the datasets were curated and how queries and retrieval corpora are constructed.Source: §A.1 · p. 16
  • It explains how each dataset’s native annotations are converted into fault-type labels.Source: §A.1 · p. 16

A.1.1 CURATION PROTOCOL

READ-Bench applies a common curation pipeline while preserving dataset-specific conventions, including a univariate reference dataset.

A.1.1 CURATION PROTOCOL

  • Datasets retain their source channels and label vocabularies, with existing windows kept or series segmented into fixed-length windows.Source: §A.1.1 · p. 16
  • Most datasets fill missing readings within each series, while Petrobras 3W drops affected rows rather than interpolating them.Most sources forward-fill gaps, back-fill leading gaps, and set wholly absent channels to zero; Petrobras 3W also converts out-of-range sentinels to missing.Source: §A.1.1 · p. 17
  • CTSR is the sole univariate dataset, providing a shape-driven reference because diagnostic relevance in multivariate data can concentrate in a subset of channels.Its corpus is sampled to retain at least ten same-class entries per query for stable scores.Source: §A.1.1 · p. 17

A.1.2 QUERY AND CORPUS CONSTRUCTION

The protocol holds out anomalous test windows as queries and constructs training-split corpora with controlled normal-window pollution across datasets of varied difficulty.

A.1.2 QUERY AND CORPUS CONSTRUCTION

  • Queries are anomalous test-split windows, while retrieval corpora come from the training split and contain anomalies alone or mixed with normal windows.The three pollution levels are roughly 0%, 10%, and 20% normal windows; a fixed seed and coarser-than-window split unit prevent query or near-duplicate overlap.Source: §A.1.2 · p. 17
  • Table 3 records dataset composition, corpus sizes at each pollution level, query counts, same-class entries per query, and class imbalance.Source: Table 3 · p. 17
  • The datasets vary widely in channel count, window length, and class balance, with some datasets also varying in channel count or length internally.Channel count ranges from 1 to 664, window length from 16 to 1024 timesteps, and some rare classes have only one corpus window.Source: §A.1.2 · p. 17

A.1.3 LABEL CONSTRUCTION

READ-Bench resolves window-level fault types using dataset-specific annotation rules and merges separate onset codes when they denote the same underlying fault.

A.1.3 LABEL CONSTRUCTION

  • Most datasets assign a window’s fault type by the majority of its per-timestep labels, while some use native labels or a short-fault coverage rule.MIT-BIH applies majority labeling over beats; brief injected events can qualify when a fault occupies most of its own extent even without a window majority.Source: §A.1.3 · pp. 17–18
  • Separate codes for a fault’s transient onset and steady phase are merged before labeling so windows receive the underlying fault’s label.Tennessee Eastman is given as an example of this annotation pattern.Source: §A.1.3 · p. 18
  • Table 4 identifies each dataset’s label source, window-label rule, and split unit used to keep queries separate from the corpus.Source: Table 4 · p. 18

A.2 METHOD SUITE FORMAL DEFINITIONS

This appendix formalizes the retrieval, fusion, reranking, and evaluation methods, including statistical tests and implementation settings. It also documents evaluation configurations and per-dataset and macro-averaged metrics across pollution levels, corpus scales, and retrieval-and-reranker pipelines.

A.2.1 TRIVIAL BASELINES; A.2.2 RAW-WINDOW DISTANCES

  • Random and majority rankings provide label-blind baselines, while raw-window distances convert dissimilarity to descending scores and accommodate unequal shapes through padding, interpolation, or native alignment.DTW permits one-to-many local alignment, unlike lock-step measures.Source: §A.2.1 · p. 20, §A.2.2 · p. 20

A.2.3 EMBEDDING-SPACE SCORERS

  • Normal-Residual scoring compares normalized embedding residuals after subtracting a local mean of k=30 normal-window embeddings, measuring similarity relative to expected normal operation.A k-sweep across 12 datasets found k=1 consistently weakest and a mid-range k (≈30) best; averaging neighbors reduces noise.Source: §A.2.3 · p. 21

A.2.4 SYMBOLIC BAG-OF-WORDS RETRIEVERS; A.2.5 FOUNDATION-MODEL EMBEDDERS

  • Symbolic retrievers score per-channel SAX or SFA words with BM25, while frozen foundation encoders produce fixed-dimensional embeddings through channel pooling and length adaptation.SAX uses piecewise-aggregate approximation and Gaussian breakpoints; SFA discretizes low-frequency DFT coefficients with multiple-coefficient binning.Source: §A.2.4 · p. 21, §A.2.5 · p. 21

A.2.6 SUPERVISED CLASSIFIERS; A.2.7 SCORE FUSION

  • MR-Hydra and RDST score candidates using query posterior mass on their class, while fusion combines two base scorers by reciprocal rank, weighted sum, or a two-step cascade.The fusion grid pairs four foundation embedders with five shape or symbolic legs.Source: §A.2.6 · p. 22, §A.2.7 · p. 22

A.2.8 RERANKERS

  • Pool-based rerankers adjust retrieved candidates using embedding-centroid feedback, label-neighbor purity, query-conditioned purity, majority voting, or Gaussian-process class probabilities.The GPC is fit on labeled neighbors and scores whether each candidate shares the query’s class.Source: §A.2.8 · pp. 22–23

A.2.9 LANGUAGE-MODEL RERANKERS

  • Four language-model rerankers reorder the top-20 CHARM pool using candidate labels but not the query label; pointwise methods score pairs, whereas listwise methods return a complete ordering.Claude + TSAD ranks deseasonalized residuals rather than raw series, and all rerankers use the same evaluation metrics as other methods.Source: §A.2.9 · p. 23

A.3 EVALUATION METRIC DEFINITIONS; A.4 STATISTICAL TESTING DEFINITIONS

  • Evaluation averages P@K, HR@K, and NDCG@K over queries; unlike set-based P@K and HR@K, NDCG@K is sensitive to ranking order, with Friedman and Nemenyi tests comparing methods across 12 datasets.The Nemenyi test follows a significant Friedman result and uses family-wise significance level α=0.05.Source: §A.3 · pp. 23–24, §A.4 · p. 24

A.5 HYPERPARAMETERS AND COMPUTE BUDGET

  • Implementation settings document fixed and tuned choices, evaluate retrieval at K∈{1, 3, 5, 10, 20}, and specify compute hardware and method-specific inference configurations.Foundation-model embeddings are frozen and channel-mean-pooled; the listed experiments ran on a single node with two NVIDIA A100-SXM4-80GB GPUs.Source: §A.5 · pp. 24–25

A.6 LANGUAGE-MODEL RERANKER PROMPTS

  • Toto-1.0-QA and ChatTS score query-candidate pairs, Claude ranks a pool jointly, and Claude + TSAD uses deseasonalized residuals; a separate baseline compares queries with one labeled example per class.Reranking prompts append candidate labels as neutral metadata, while the query’s own label is not shown.Source: §A.6 · pp. 26–28

A.7.1 FULL BASE-RETRIEVER METRIC GRID

  • Table 8 reports a full base-retriever metric grid at ρ=0%, macro-averaged over 12 datasets; σ is the cross-dataset standard deviation of NDCG@10, and supervised rows are classification references, not retrieval systems.Source: Table 8 · p. 29

A.7.2 SCALING TO THE FULL CORPUS

  • On uncapped corpora reaching over a million windows, label-free embedders remain close with no uniform winner, while absolute scores drop and several distance, symbolic, and supervised methods do not scale.Five datasets were already at or below the cap; comparisons of capped and full-corpus scores cover all 12 datasets.Source: Table 9 · p. 29, §A.7.2 · p. 29

A.7.3 NORMAL-RESIDUAL SCORING BY EMBEDDER

  • Normal-residual scoring roughly halves pollution degradation for every wrapped embedder and improves clean accuracy except for essentially unchanged TiRex; MantisV2 + NR is the strongest single base retriever.TiRex’s clean NDCG@10 changes from 0.483 to 0.481.Source: §A.7.3 · p. 29, Table 10 · p. 30

A.7.4 CHOOSING THE FUSION LEG AND MODE; A.7.5 COMPLETE FUSION GRID

  • Across seven embedder blocks, reciprocal-rank fusion with DTW-I is the joint fusion optimum; the complete grid also identifies MantisV2 + NR as the strongest fused pool.RRF is best on every leg except SBD-D, which is strongest under weighted sum but never reaches the top cell.Source: Table 11 · p. 30, §A.7.4 · p. 30, §A.7.5 · p. 30

A.7.6 RERANKING A FIXED POOL; A.7.7 COMPLETE RERANKER GRID

  • GPC gives the largest gain on most fixed pools and its gains hold across embedders and pollution; MajVote leads on CHARM + NR and TiRex + NR pools.MantisV2 + NR + GPC reaches NDCG@10 of 0.645 versus 0.604 for MantisV2 + ED + GPC.Source: Table 13 · p. 31, §A.7.6 · p. 31, Table 14 · p. 31

A.7.8 COMPOSING FUSION AND RERANKING

  • In the 12-dataset macro average, CHARM + NR | DTW-I + GPC is strongest at every pollution level; fusion and GPC stack close to additively, and normal-residual scoring lifts these stacks above plain-embedding counterparts.Source: §A.7.8 · p. 31, Table 15 · p. 32

A.8 COMPLETE PER-DATASET RESULTS

  • The appendix reports per-dataset and macro-averaged metric grids for each non-fusion method across three pollution levels, covering precision, hit rate, NDCG cutoffs, and macro-F1.Fusion combinations are reported separately in Appendix A.7.Source: §A.8 · p. 32, Table 16 · p. 32
  • The appendix reports metric grids for DTW-I paired with GPC, MajVote, QPurity, PRF, and Purity.Each grid covers per-dataset and macro-averaged results at each pollution level.Source: Table 42 · p. 45, Table 43 · p. 46, Table 44 · p. 46
  • MantisV2 is evaluated with GPC, MajVote, QPurity, PRF, and Purity in separate metric grids.The grids report per-dataset and macro-averaged results at each pollution level.Source: Table 47 · p. 48, Table 48 · p. 48, Table 49 · p. 49
  • MantisV2 + NR is evaluated with GPC, MajVote, QPurity, PRF, and Purity in separate metric grids.Each grid includes per-dataset and macro-averaged results at each pollution level.Source: Table 52 · p. 50, Table 53 · p. 51, Table 54 · p. 51
  • TiRex is evaluated with GPC, MajVote, QPurity, PRF, and Purity in separate metric grids.The grids show per-dataset and macro-averaged metrics at each pollution level.Source: Table 57 · p. 53, Table 58 · p. 53, Table 59 · p. 54
  • TiRex + NR is evaluated with GPC, MajVote, QPurity, PRF, and Purity in separate metric grids.Each grid covers per-dataset and macro-averaged metrics at each pollution level.Source: Table 62 · p. 55, Table 63 · p. 56, Table 64 · p. 56
  • Chronos-2 + NR is evaluated with GPC, MajVote, QPurity, PRF, and Purity in separate metric grids.The grids report per-dataset and macro-averaged results at each pollution level.Source: Table 67 · p. 58, Table 68 · p. 58, Table 69 · p. 59
  • Standalone metric grids are also provided for MR-Hydra and RDST.Both tables report per-dataset and macro-averaged metrics at each pollution level.Source: Table 72 · p. 60, Table 73 · p. 61

Related papers

Loading 2609.32123v2…