Source-linked AI summary
TSDS-Toolbox: A Toolbox for Measuring Time-Series Dataset Similarity
Yen-Ku Liu, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt
TL;DR
Time-series dataset similarity is difficult to benchmark consistently, limiting reliable comparison and evaluation for source-dataset selection. TSDS-Toolbox provides a unified, extensible framework for reproducible benchmarking and downstream evaluation, showing that no single similarity method aligns with performance across all tasks.
Problem
Time-series dataset similarity methods lack consistent, extensible benchmarking and downstream evaluation despite supporting source-dataset selection and cross-domain analysis.
Method
TSDS-Toolbox uses a configuration-driven, modular pipeline to compute dataset similarities, apply reducers, and evaluate downstream tasks under standardized settings.
Results
No single similarity method consistently aligns with downstream performance across tasks, with metric and reducer effectiveness depending on the task and underlying distance.
Takeaways & Limitations
Systematic evaluation is needed when comparing time-series similarity methods and selecting source datasets for downstream tasks.
Abstract
from arXiv · showhide
The rapid advancement of artificial intelligence (AI) has significantly accelerated research in time-series analysis, particularly in forecasting, classification, and generation tasks. Recent models, especially foundation models, benefit from time-series dataset similarity due to its significant role in source dataset selection for fine-tuning. However, many existing implementations for benchmarking time-series dataset similarity methods are fragmented and difficult to extend. To address this, we present a unified framework, the Time-Series Dataset Similarity Toolbox (TSDS-Toolbox). Our work enables (1) systematic and reproducible comparisons of time-series dataset similarity methods; (2) flexible extensibility for users to add customized datasets, similarity methods, and downstream time-series tasks; and (3) consistent evaluation of both dataset-level and series-level similarity methods through integrated time-series dataset reducers. The effectiveness of TSDS-Toolbox is validated through comprehensive experiments under diverse experimental settings. Our toolbox is publicly available.
1 Introduction
Time-series dataset similarity supports source-dataset selection, visualization, benchmark curation, and analysis of foundation-model generalization. TSDS-Toolbox addresses fragmented evaluation through unified benchmarking, standardized downstream pipelines, and modular, configuration-driven experiments.
- Motivation: Time series represent temporally evolving systems across finance, healthcare, speech processing, and climate science, motivating dedicated resources for downstream research [4] [24] [40].Existing resources target classification [5], anomaly detection, and foundation-model development [3] [10].
- Motivation: Dataset similarity supports source-dataset selection, visualization and inspection, benchmark curation, and analysis of foundation-model generalization [12] [35] [41].In transfer-learning and adaptation, closer source datasets are more likely to provide useful fine-tuning signals.
- Problem: Existing toolboxes emphasize individual-series similarity or general tasks, leaving dataset-level metrics evaluated under inconsistent settings and requiring repeated integration and standardization [23] [31] [35].Researchers must repeatedly integrate metrics, standardize dataset formats, and build downstream evaluation workflows.
- Contributions: TSDS-Toolbox provides a unified benchmarking interface for fair comparison of time-series dataset similarity methods under consistent experimental settings.This interface is one of the toolbox’s stated contributions.
- Contributions: Standardized downstream evaluation pipelines assess practical utility, while modular, configuration-driven design supports reproducible experiments and future extensions.The contributions jointly target practical evaluation, reproducibility, and extensibility.
2 Related Work
Prior work primarily measures similarity between individual time series using sequence-level distances, while dataset-level comparison can use reducers to create representative sequences before computing similarity. Existing time-series benchmarks and toolboxes support several modeling tasks but do not systematically benchmark dataset-similarity metrics.
- Time-Series Similarity: Sequence-level distances such as DTW and LCSS compare individual time series rather than entire datasets [9].These methods are commonly used to study time-series similarity.
- Time-Series Similarity: Dataset-level comparison can combine distances with reducers such as DBA and PCA to obtain compact representative sequences before computing similarity [14].This provides a practical route from sequence-level comparisons to dataset-level similarity.
- Benchmarks and Toolboxes: Existing benchmarks and toolboxes support forecasting, classification, generation, and representation-learning model development and evaluation [19] [23], but not systematic benchmarking of dataset-similarity metrics.The gap motivates a unified framework focused on time-series dataset similarity.
3 Preliminaries
This section formalizes time-series dataset similarity for datasets containing potentially variable-length sequences and distinguishes dataset-level from reducer-based series-level metrics.
- 3 Preliminaries: Time-series dataset similarity defines a dissimilarity function s(X,Y), where smaller values indicate higher similarity.The datasets contain sequences that may have different lengths, with observations in R^d.
- 3 Preliminaries: Dataset-level metrics directly compute the dissimilarity as s_m(X,Y) = m(X,Y).
- 3 Preliminaries: Reducer-based series-level metrics first map the datasets before computing similarity.
4 Time-series Dataset Similarity Toolbox
TSDS-Toolbox provides a configuration-driven pipeline for computing and evaluating time-series dataset similarity across heterogeneous datasets, similarity methods, reducers, and downstream tasks. Its four-layer design unifies data preparation, similarity computation, task-based evaluation, and visual and quantitative analysis.
- 4.1 Overall Pipeline: The configuration-driven pipeline supports similarity-only analysis with pairwise distance matrices and similarity evaluation with downstream-task performance matrices.Users configure dataset formats, preprocessing, similarity methods, reducers, evaluation tasks, and foundation-model adapters.
- 4.2 Data Layer: The Data Layer converts datasets from formats including GluonTS, CSV, NPY, and pandas DataFrames into a unified representation, then applies consistent preprocessing.Supported preprocessing includes sampling, fixed-length window extraction, reshaping, train-validation splitting, and optional normalization for similarity and downstream evaluation.
- 4.3 Similarity Layer: The Similarity Layer supports dataset-level metrics and reducer-based series-level metrics within one pipeline, including Wasserstein Distance, Maximum Mean Discrepancy, Optimal Transport, Match-and-Deform [2] [7] [26], and DTW, Euclidean Distance, and LCSS.DBA and PCA reduce datasets to representative sequences before series-level comparison; Table 1 summarizes method support and constraints.
- 4.4 Evaluation Layer: The Evaluation Layer tests whether similarity scores reflect downstream behavior through out-of-distribution and transfer-learning settings for classification and forecasting.Transfer learning fine-tunes on a source dataset, adapts using a small target-reference split, and evaluates on a target-inference split.
- 4.5 Analysis Layer: The Analysis Layer uses heatmaps, network graphs, and Pearson correlation to relate similarity structures to downstream task errors.These analyses support comparison of distance-matrix structure and alignment with downstream behavior.
5 Experiments
TSDS-Toolbox is evaluated end-to-end across 25 diverse GluonTS datasets using all supported similarity methods and downstream pipelines. Results show that similarity–task alignment and reducer effectiveness depend on the downstream task and underlying series-level distance.
- Experimental setup: Experiments cover 25 GluonTS datasets spanning traffic, weather, electricity, exchange rates, tourism, and public health, using 100 sampled windows of length 100 per dataset.The pipeline applies z-score normalization and removes flat sequences before computing similarity scores.
- Similarity evaluation: No similarity metric consistently achieves the highest correlation across tasks; in Time-MoE OOD forecasting, MAD [26] performs strongest, closely followed by WSD [7].Pearson correlation is measured between dataset-distance scores and downstream task losses.
- Reducer analysis: DBA achieves stronger correlations in most reducer settings, especially for classification and Time-MoE forecasting, whereas PCA remains competitive for Lag-Llama [32] OOD forecasting.Table 3 compares DBA and PCA across base distance metrics; reducer effectiveness depends on both downstream task and underlying series-level distance.
- Similarity evaluation: Reducer-based DTW and ED remain competitive, while LCSS and MMD show weaker alignment.
6 Conclusion
TSDS-Toolbox is a unified, modular framework for standardized benchmarking of time-series dataset similarity methods. Its experiments show that no single similarity method consistently aligns with downstream performance across all tasks, underscoring the need for systematic evaluation.
- 6 Conclusion: TSDS-Toolbox provides standardized benchmarking for time-series dataset similarity methods.It is presented as a unified toolbox for evaluating similarity methods under standardized settings.
- 6 Conclusion: The framework integrates similarity computation, downstream evaluation, and correlation-based analysis.Its modular and extensible design combines these components within one evaluation framework.
- 6 Conclusion: No single similarity method consistently aligns with downstream performance across all tasks.This finding motivates systematic evaluation rather than reliance on a single similarity method.