Source-linked AI summary
DISC: Domain-Integrated Similarity Cost for Time-Series Dataset
Yen-Ku Liu, Hongjie Chen, Franck Dernoncourt, Ryan Rossi
TL;DR
Time-series similarity metrics capture different relationships, and their alignment with downstream utility varies across tasks, making source-target transfer assessment difficult without exhaustive evaluation. DISC learns task-specific combinations of metrics from calibration transfer observations and then scores pairs from similarity values alone. On the reported split, it raises average test correlation by about 10% across four task settings.
Problem
Similarity metrics capture different dataset relationships and vary in alignment with downstream utility, while exhaustive source-target evaluation is costly.
Source: §I · pp. 1–2Method
DISC learns task-specific metric mixtures from calibration transfer observations and scores source-target pairs using similarity features alone after calibration.
Source: §I · p. 2, D. Training Objective and Procedure · p. 5Results
About 10% higher average test correlation across four task settings is reported for DISC on the evaluated split.
Source: §I · p. 2Takeaways & Limitations
The reported findings support treating source-target transfer alignment as task-dependent rather than relying on one fixed similarity metric.
Source: §VI · p. 9Short, flat, or low-diversity sampled windows can weaken single-metric correlation.
Source: §VI · p. 9
Abstract
from arXiv · showhide
The growth of time-series datasets and foundation models has made dataset comparison a central part of time-series workflows. Assessing source-target transfer before model adaptation requires similarity estimates that can summarize transfer-related dataset relationships without exhaustive source-target evaluation over every candidate pair. Many metrics have been proposed for measuring similarity between time-series datasets, but they capture different dataset relationships and vary in how well they align with downstream utility across tasks. We evaluate this gap by measuring the correlation between source-target similarity scores and downstream forecasting errors in out-of-distribution and transfer-learning settings. We propose Domain-Integrated Similarity Cost (DISC), a learnable aggregation framework that uses calibration transfer observations to select, weight, and combine similarity metrics for different downstream tasks. By integrating complementary metric values, DISC better distinguishes source-target dataset differences that matter for downstream behavior. After calibration, DISC scores source-target pairs using only their similarity-metric values and yields higher alignment than single-metric references on the reported split. These results show that time-series dataset similarity is task-dependent, and that learned metric integration provides a stronger proxy for downstream transfer behavior than manually selecting one similarity metric.
I. INTRODUCTION
The paper treats time-series dataset similarity as task-dependent and proposes DISC to combine metric evidence using calibration transfer observations. On the reported split, DISC raises average test correlation by about 10% across four forecasting task settings.
I. INTRODUCTION
- Similarity metrics capture different dataset relationships, so which view is informative can depend on the downstream task [25].The paper names marginal distribution, temporal shape, representative pattern, and alignment structure as distinct relationship views.Source: §I · p. 1
- DBA+LCSS reveals mobility and temporal-shape similarities that WSD and MAD leave less clear, while MMD rejects a semantically mismatched group ranked close by the other metrics.These examples motivate integrating complementary evidence rather than relying on a single metric.Source: Figure 1 · p. 1, §I · p. 2
- Because exhaustive source-target transfer evaluation is costly, similarity can prioritize evaluations and should align with downstream errors across zero-shot and fine-tuned settings.The paper frames this as a correlation-alignment problem and does not expect one metric to dominate all tasks.Source: §I · p. 2
- DISC uses calibration transfer observations to learn task-specific metric mixtures, then scores source-target pairs using similarity values alone.A rank-based prior modulates a base score formed from standardized metric values.Source: §I · p. 2
- About 10% higher average test correlation across four task settings is reported for DISC on the evaluated split.The evaluation covers out-of-distribution and transfer-learning forecasting settings.Source: §I · p. 2
- The learned metric weights are presented as an interpretable view of which dataset relationships are useful for a given model and task setting.DISC is described as flexible and task-adaptable through learnable aggregation parameters.Source: §I · p. 2
II. RELATED WORK
Prior work offers dataset-level distances and reducer-based sequence comparisons, whose suitability can vary across datasets and tasks. The preliminaries formalize these approaches alongside pairwise metric matrices and directed source-to-target errors.
II. RELATED WORK
- Existing metric families capture different dataset relationships, and their suitability can vary across datasets and downstream tasks.Dataset-level distances compare empirical collections; series-level distances can be applied after reducing each dataset to a representative sequence.Source: §II · p. 2
- Foundation models and dataset-reuse methods motivate evaluating similarity by alignment with downstream forecasting behavior rather than mathematical or visual plausibility alone.The related-work discussion links source-target compatibility with adaptation value or harmful bias.Source: §II · p. 2
III. PRELIMINARIES
- Reducer-based metrics summarize each dataset as a representative sequence before applying a series-level distance, while dataset-level metrics compare datasets directly.The abstraction covers both configuration classes used in the experiments.Source: §III · p. 3
- For each candidate method, pairwise distances form a dataset-universe matrix, while downstream errors are directed from source dataset Xa to target dataset Xb.The error entry E(Xa, Xb) represents using Xa as source and evaluating on Xb as target.Source: §III · p. 3
IV. DISC: DOMAIN-INTEGRATED SIMILARITY COST
DISC learns how to select, weight, and combine similarity metrics for a downstream task rather than assuming one metric is consistently reliable. Its source-target formulation produces a score for an ordered pair and seeks monotone alignment with transfer error.
IV. DISC: DOMAIN-INTEGRATED SIMILARITY COST
- DISC learns task-specific selection, weighting, and combination of candidate time-series similarity metrics rather than assuming one metric is always reliable.The section presents a base mixture, a rank-based prior, and a learned gate as the components of the approach.Source: §IV · p. 3
A. Problem Setting
- The input comprises source-target distance matrices and downstream error entries over a shared dataset universe, and the output is a scalar score for each ordered pair.The score is expressed as a base similarity score modulated by a gated prior.Source: A. Problem Setting · p. 4
- DISC uses ordered pairs because transfer is directed: training on X and evaluating on Y need not yield the same error as reversing the pair.The learned score is intended to reflect this directed source-to-target relationship.Source: A. Problem Setting · p. 4
- For a fixed source, a useful distance should tend to be larger for targets with larger downstream error, without requiring exact error prediction.This monotone association motivates positive alignment between score magnitude and source-to-target transfer error.Source: A. Problem Setting · p. 4
B. Base and Prior Metric Selection
DISC uses standardized metric values for its base score and source-wise percentile ranks for a separate prior signal. The two mixtures can select and weight different metrics, preserving magnitude and relative-position information.
B. Base and Prior Metric Selection
- Standardized metric values form the base score, while source-wise percentile ranks form a separate prior signal over relative source-target evidence.The percentile rank indicates a target’s relative distance from a source and is less sensitive to scale.Source: B. Base and Prior Metric Selection · p. 4
- The base mixture selects and weights metrics using standardized values to define the primary similarity score.Its learned weights are produced by a softmax over base logits, with temperature controlling mixture concentration.Source: B. Base and Prior Metric Selection · p. 4
- The prior mixture independently selects and weights metrics using source-wise ranks, allowing it to contribute relative evidence from different metrics than the base mixture.Separating the mixtures lets one set of metrics define the main distance while another supplies source-wise relative evidence.Source: B. Base and Prior Metric Selection · p. 4
C. Gated Integration
DISC combines its standardized-value base score with rank-based prior evidence through a learned multiplicative gate. The gate’s strength, threshold, and sharpness control how prior evidence modulates the base score.
C. Gated Integration
- The rank-based prior signal multiplicatively modulates the base score rather than serving as a standalone additive distance.The multiplier starts at one and changes the base score’s magnitude according to the learned prior signal.Source: C. Gated Integration · p. 4
- The gate’s modulation strength, threshold, and sharpness determine the magnitude and rank level of its interaction and whether the transition is gradual or sharp.These parameters let DISC learn whether a task benefits from broad prior modulation or a sharper response to high-prior, distant pairs.Source: C. Gated Integration · p. 4
- Because the base score is not constrained to be positive, the gate is a learned multiplicative modulation, not a standalone monotone penalty.The gate sharpness is bounded to [1, 55] in the final run.Source: C. Gated Integration · p. 5
D. Training Objective and Procedure
DISC parameters are calibrated against downstream errors using source-wise and global Pearson alignment. Training optimizes the task-specific aggregation, selects a checkpoint by validation source-wise correlation, then freezes it for test scoring from similarity features alone.
D. Training Objective and Procedure
- Downstream pair errors calibrate the task-specific metric weights and gate parameters, while the learned DISC score for a pair requires only similarity features.The training objective combines average within-source correlation with a smaller global-pair alignment term, weighted by λ = 0.10.Source: D. Training Objective and Procedure · p. 5
- Entropy regularization encourages sparse metric usage, while an overlap penalty discourages the base and prior mixtures from selecting the same metrics.The reported coefficients are η = 0.006 for entropy and µ = 0.02 for overlap; parameters are optimized with Adam.Source: D. Training Objective and Procedure · p. 5
- Training optimizes DISC parameters on training datasets, selects the checkpoint with the best validation source-wise correlation, then freezes it and reports test-split source-wise correlation.Training proceeds independently for each downstream task; validation uses the same criterion as reporting, with validation datasets replacing test datasets.Source: D. Training Objective and Procedure · p. 5
V. EXPERIMENTS
The experiments ask whether single metrics predict downstream errors consistently, whether DISC aligns better than the best single metric, and whether learned aggregation beats non-learned search. Evaluation uses a shared four-task split, with results presented in correlation tables whose supplied captions do not include cell values.
V. EXPERIMENTS
- The experiments also ask whether single similarity metrics consistently predict downstream errors.The shared split controls dataset-selection effects across tasks.Source: §V · p. 5, A. Datasets and Splits · p. 6
- The experiments test whether DISC aligns more highly with downstream errors than the best single metric and whether learned aggregation outperforms non-learned parameter search.The correlation tables compare single metrics and DISC against OT as the reference, but the supplied passages contain no table cell values.Source: §V · p. 5, Table II · p. 6, Table III · p. 6
A. Datasets and Splits
- The evaluation uses 46 GluonTS datasets represented by 100 sampled windows of length 16, split into 24 training, 8 validation, and 14 test datasets.The same split is shared across four forecasting tasks; the test datasets define 14 source rows.Source: A. Datasets and Splits · p. 5
B. Downstream Tasks
The study evaluates two foundation models across out-of-distribution and transfer-learning forecasting settings, then compares similarity configurations that represent different views of time-series datasets.
B. Downstream Tasks
- With 46 datasets, each foundation model is evaluated on 2,070 directed source-target pairs per protocol, across four model-and-protocol tasks.OOD evaluates targets without target adaptation; transfer learning adapts each source model using target-side data before evaluation.Source: B. Downstream Tasks · p. 6
C. Similarity Metrics
- The comparison tests whether complementary dataset-relationship views vary in usefulness across models, tasks, and adaptation settings.DBA and PCA provide representative sequences rather than standalone similarity metrics.Source: C. Similarity Metrics · p. 6
- Four dataset-level metrics capture distributional or matching relationships, while DBA and PCA reducers feed six series-level distance configurations.The series-level configurations are DBA-DTW, DBA-ED, DBA-LCSS, PCA-DTW, PCA-ED, and PCA-LCSS.Source: C. Similarity Metrics · p. 6
D. Evaluation Protocol
The evaluation measures source-conditioned correlation between similarity distances and downstream forecasting errors. Single-metric performance varies by task, while DISC has the strongest reported average alignment.
D. Evaluation Protocol
- For each test source, Pearson correlation compares its distances to the other test datasets with the corresponding downstream errors; the reported score averages 14 source-level correlations.Each source-level correlation uses 13 target points, and higher positive values indicate stronger alignment.Source: D. Evaluation Protocol · p. 6
- The source-wise measure includes target forecasting difficulty in observed source-target behavior and is not a direct source-selection benchmark.Figure 3 illustrates fixed-source examples; upward trends indicate that larger DISC scores tend to accompany larger errors.Source: D. Evaluation Protocol · p. 7, Figure 3 · p. 7
E. RQ1: Single Similarity Configurations Are Task-Dependent
- The strongest single metric varies by task: MAD for OOD-Time-MoE, DBA-DTW and DBA-ED for OOD-Lag-Llama, PCA-ED for TL-Time-MoE, and MMD for TL-Lag-Llama.Optimal Transport is the reference metric in the reported tables.Source: E. RQ1: Single Similarity… · p. 7
- DISC reaches 0.598 average correlation, above the per-task single-metric oracle at 0.543 and PCA-ED’s 0.403 average.The comparison is based on the reported split.Source: E. RQ1: Single Similarity… · p. 7, Table IV · p. 8
F. RQ2: DISC Shows Gains Through Task-Specific Aggregation
On the reported split, DISC improves on the best single metric across four forecasting tasks, with source-wise examples showing local score-error alignment. Its learned aggregation also outperforms an extensive manual search.
F. RQ2: DISC Shows Gains Through Task-Specific Aggregation
- DISC exceeds the best single metric in all four task columns, with a 10.29% average relative change.The comparison uses one shared train-validation-test split; the test score averages 14 source-row evaluations across four settings.Source: F. RQ2: DISC Shows Gains Through… · p. 8
- Figure 3’s four illustrative fixed-source panels show that targets with larger DISC scores generally incur larger forecasting errors.The panels use sources chosen for the strongest source-wise correlation, while reported RQ2 scores average alignment over all 14 source rows.Source: F. RQ2: DISC Shows Gains Through… · p. 8
- The learned base and prior mixtures differ by task, combining metric values and metric ranks rather than improving one particular metric.The prior mixture supplies a rank-based gate signal that modulates the base score.Source: F. RQ2: DISC Shows Gains Through… · p. 8
G. RQ3: Learned Aggregation Versus Manual Search
- Manual search evaluates approximately 7.9 million configurations per task yet averages 0.291 test correlation, versus DISC’s 0.598 under the same protocol.Manual search remains below DISC on every task; its OOD-Time-MoE result is negative, so a relative gain is not reported.Source: G. RQ3: Learned Aggregation Versus… · p. 9, Table VI · p. 9
H. Discussion: Dataset Characteristics and Metric Reliability
The discussion links weak single-metric explanations to smooth, flat, or low-diversity sampled representations and notes that sampling can miss seasonality. The conclusion reports task-dependent gains from learned aggregation on the evaluated split.
H. Discussion: Dataset Characteristics and Metric Reliability
- Overly smooth, flat, or low-diversity sampled windows can limit how well similarity metrics explain source-target relationships.The diagnostics use N = 100 sampled windows of length L = 16 per dataset; Table VII summarizes low- and high-correlation groups.Source: H. Discussion: Dataset Characteristics… · p. 9, Table VII · p. 9
- Similarity metrics are not interchangeable: rank information can complement standardized values, while short windows may miss seasonality or overrepresent flat regimes.The discussion notes that Table IV’s best-single-metric-per-task reference is selected from test results, a choice unavailable before evaluation.Source: H. Discussion: Dataset Characteristics… · p. 9
VI. CONCLUSION
- Across four forecasting tasks with Time-MoE and Lag-Llama on the reported split, DISC beats the best single metric in every task and raises average correlation by 10.29%.The conclusion presents learned aggregation as a practical task-aware score for source-target transfer assessment.Source: §VI · p. 9