Source-linked AI summary
BITS: Rethinking Fair and Comprehensive Evaluation for Irregular Time Series Forecasting
Kangjia Yan, Linfeng Wang, Tianen Shen, Xiangfei Qiu, Ruitong Zhang, Hao Miao, Jilin Hu, Chenjuan Guo, Bin Yang, Christian S. Jensen
TL;DR
Irregular forecasting lacks a unified benchmark with broad datasets, consistent protocols, and metrics beyond point-wise errors. BITS provides a standardized evaluation framework across datasets, model types, and metrics; its experiments show that model performance varies with irregularity and metric choice. The benchmark does not exhaustively represent all real-world irregular scenarios or emerging architectures.
Problem
Irregular forecasting evaluations use limited datasets, inconsistent protocols, and predominantly error-based metrics, complicating fair comparisons across diverse settings.
Source: §1 · p. 1Method
BITS standardizes evaluation across eleven datasets from nine domains, characterizing irregularity and supporting diverse forecasting models through a unified pipeline and multiple metric types.
Source: §1 · pp. 2–3, §3.1 · p. 5Results
Model performance varies across irregularity characteristics, and different evaluation metrics can yield different assessments of model performance.
Source: §1 · p. 3, §5 · p. 10Takeaways & Limitations
The benchmark's multi-dimensional evaluation shows that model comparisons depend on both irregularity characteristics and the metrics used.
Source: §1 · p. 3, §4.2.3 · p. 8BITS cannot exhaustively represent all real-world irregular scenarios or emerging architectures.
Source: §A.7 · p. 19
Abstract
from arXiv · showhide
Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing evaluations are often conducted on a limited set of datasets with inconsistent experimental protocols and predominantly error-based metrics, rendering it difficult to compare and assess methods fairly and comprehensively across diverse settings. To eliminate these limitations and accelerate progress, we propose BITS, a standardized, reproducible, and extensible benchmark for advancing research on irregular time series forecasting. BITS covers eleven datasets from nine domains with diverse irregularity characteristics, and it characterizes the datasets according to their missing rate, missing pattern complexity, sampling irregularity, and skewness. Further, it offers a unified pipeline for data preprocessing, model integration and evaluation, and reporting. It accommodates regular and irregular time series forecasting methods, including time series foundation models, under consistent settings, incorporating both error-based and non-error-based evaluation metrics. Findings include that method performance varies substantially across irregularity characteristics, with no single modeling strategy consistently dominating. We also find that using error-based or non-error-based metrics can yield different model rankings, highlighting the need for multi-dimensional evaluation. The code can be found at https://anonymous.4open.science/r/BITS-8F2E/.
1 INTRODUCTION
Irregular forecasting methods have advanced, but evaluations lack representative datasets, consistent protocols and comprehensive metrics. BITS addresses these gaps with broader coverage and a standardized benchmark.
1 INTRODUCTION
- Existing evaluations often use few datasets and may overlook common missingness patterns or temporal order, risking limited coverage and information leakage.Sample-aware splitting can be necessary for sparse datasets, but using it when temporal splitting is feasible may put future observations in training and earlier ones in testing.Source: §1 · pp. 1–2
- Inconsistent normalization and input-output windows hinder fair comparisons, while point-wise errors alone can miss whether forecasts capture temporal dynamics.Near-constant forecasts may obtain relatively low MSE despite failing to capture important temporal patterns.Source: §1 · p. 2
- BITS combines eleven datasets from nine domains with irregularity-aware splitting, standardized evaluation, and error-based and non-error-based metrics.Time-aware splitting preserves temporal order when observation counts allow it, while sample-aware splitting remains for sparse datasets.Source: §1 · pp. 2–3
- BITS also supports regular and irregular forecasting methods, including time series foundation models, and reports performance variation across irregularity characteristics and metrics.Different metrics can lead to different assessments of model performance.Source: §1 · p. 3
2 RELATED WORK
Prior irregular forecasting work spans several modeling strategies, while existing benchmarks largely target regular forecasting or cover only parts of irregular evaluation. BITS is positioned as a unified framework across these gaps.
2 RELATED WORK
- Irregular forecasting methods include imputation, temporal segmentation and aggregation, multi-scale, continuous-time, structured-format, and sampling-context approaches.Examples include mTAN for imputation, Warpformer for multi-scale modeling, NeuralFlows for continuous-time dynamics, and GRU-D for sampling context.Source: §2 · p. 3
- Regular forecasting benchmarks provide standardized datasets and pipelines, but offer limited support for irregular time series forecasting.The passage names BasicTS, Monash, TSLib, TFB, and TSFM-Bench among these benchmarks.Source: §2 · p. 3
- Existing irregular benchmarks lack a unified framework jointly covering diverse datasets, irregularity analysis, multiple forecasting paradigms, scalable pipelines, and multidimensional metrics.BITS is designed to bridge these gaps through systematic and unified evaluation.Source: §2 · p. 4
3 BITS BENCHMARK FRAMEWORK
BITS organizes irregular forecasting evaluation around diverse datasets, broad baseline coverage, multidimensional metrics, and a unified pipeline. The framework standardizes conditions while supporting varied models and reproducible reporting.
3.1 DATASETS DETAILS
- BITS collects eleven datasets from nine domains and characterizes irregularity by missing rate, missing-pattern complexity, sampling irregularity, and skewness.For datasets with substantially varying sample durations or observation counts, the authors select samples and variables with comparable temporal spans and observation counts.Source: §3.1 · pp. 4–5
3.2 COMPARED BASELINES
- The benchmark compares six categories of irregular forecasting methods alongside multivariate forecasting methods and time series foundation models.The categories include imputation, temporal segmentation and aggregation, multi-scale, differential-equation, data-format transformation, and sampling-context modeling.Source: §3.2 · p. 5
3.3 EVALUATION METRICS
- BITS evaluates normalized and denormalized predictions with error metrics and complementary measures of value accuracy, pattern similarity, and temporal structure.Additional metrics include MAPE, SMAPE, RMSE, WAPE, MSMAPE, PCC, R2, DA, DTW, CPN, and IER.Source: §3.3 · p. 5
3.4 UNIFIED PIPELINE
- The unified pipeline standardizes data preprocessing and splitting, model integration, evaluation settings, and experimental reporting.Its layers record logs, runtime, resource use, and environment information, and summarize results across datasets and metrics.Source: §3.4 · p. 6
4 EXPERIMENTS
Across BITS, model rankings vary by metric and irregularity setting: TiWeaver, TFMixer, and ASTGI lead overall, but no strategy dominates every dataset. The experiments also show why comparisons across models and metrics need consistent evaluation.
4.1 EXPERIMENTAL SETUP
- Experiments use a 6:2:2 train-validation-test split, customized lookback and prediction lengths, and sample-aware splits for PhysioNet, USHCN, and APTC versus time-aware splits elsewhere.Models are trained with MSE loss and Adam; reported performance is the best found through hyperparameter searches.Source: §4.1 · p. 6
4.2.1 PERFORMANCE COMPARISON ON IRREGULAR TIME SERIES FORECASTING
- TiWeaver and ASTGI achieve the lowest average MSE rank, while TFMixer ranks first in DTW, highlighting strong overall performance from segmentation and data-format transformation methods.Performance nevertheless varies within these categories, and regular models sometimes outperform specialized irregular methods.Source: Figure 4 · p. 7, §4.2.1 · p. 7
4.2.2 IRREGULAR TIME SERIES FORECASTING WITH TIME SERIES FOUNDATION MODEL
- Zero-shot TSFMs generally underperform specialized irregular methods, particularly on highly sparse PhysioNet and USHCN; MIRA remains competitive on CESNET, ClusterTrace, FNSPID, and Seabirds.The authors note that ClusterTrace has high sampling irregularity but relatively low missingness, while PhysioNet and USHCN have severe missingness and complex patterns.Source: Table 3 · p. 8, §4.2.2 · p. 8
4.2.3 CASE STUDY ON THE LIMITATION OF ERROR-BASED METRICS
- On USHCN, Warpformer and APN achieve lower MSE than NeuralFlows and mTAN, respectively, while the latter models have lower DTW and better capture temporal patterns.These examples show that lower point-wise errors need not mean better temporal-shape modeling.Source: Figure 5 · p. 8, §4.2.3 · p. 8
4.2.4 COMPARISON OF SIX CATEGORIES OF IRREGULAR FORECASTING MODELS
- Temporal segmentation and aggregation perform best under most missing rates, while data-format transformation methods also perform competitively at high missing rates; sampling-context-aware modeling performs best on GDELT.The authors suggest that using available observations may be more beneficial than reconstructing fine-grained temporal information under severe missingness.Source: Figure 6 · p. 9, §4.2.4 · p. 9
4.2.5 PERFORMANCE ON DIFFERENT IRREGULARITY CHARACTERISTICS
- No model performs best across all irregularity settings: APN leads on PhysioNet, ASTGI on CESNET, ClusterTrace, and GDELT, TiWeaver on CESNET, and TFMixer on Pamap2.The authors conclude that strong overall performance does not imply universal superiority.Source: Figure 7 · p. 9, §4.2.5 · p. 9
5 CONCLUSION
BITS provides a multi-domain benchmark and standardized pipeline for comparing irregular time-series forecasting methods. Its experiments find that performance varies across irregularity characteristics and evaluation metrics, while multiple modeling approaches show strengths and limitations.
5 CONCLUSION
- BITS combines eleven multi-domain datasets with diverse irregularity characteristics and a standardized, scalable pipeline for preprocessing, splitting, modeling, evaluation, and reporting.The benchmark also incorporates both error-based and non-error-based metrics.Source: §5 · pp. 9–10
- Model performance varies across irregularity characteristics; TiWeaver and TFMixer are consistently strong overall, ASTGI is competitive, and regular models sometimes outperform specialized irregular methods.Time-series foundation models also show limitations under severe missingness and complex missing patterns; error-based metrics alone may not fully reflect forecasting performance.Source: §5 · p. 10
- The reproducibility materials describe standardized benchmark procedures and point to detailed dataset, preprocessing, metric, irregularity, and experimental-setting descriptions.The paper states that an anonymous implementation is provided.Source: REPRODUCIBILITY STATEMENT · p. 10
A.1.1 IRREGULAR TIME SERIES
Regular time series have uniformly spaced timestamps, whereas irregular series have non-uniform intervals. Irregularity also includes variable-specific observation times and missing values on a shared timeline.
A.1.1 IRREGULAR TIME SERIES
- A series is regular when consecutive timestamps are uniformly spaced, and irregular when their intervals are non-uniform or vary substantially.The passage denotes the timestamps by t_i and the uniform interval by Δt.Source: §A.1.1 · p. 13
- Irregular observations include variable-level asynchronous sampling, where each variable may have its own observation times.The passage introduces this as one of two common forms of irregularity.Source: §A.1.1 · p. 13
- A second form is instance-level missingness: on a shared timeline, some variables may be unobserved at particular time steps.A masking matrix is used to represent which values are observed.Source: §A.1.1 · p. 13
- Irregular time series arise in domains including medicine, geoscience, and industrial monitoring, where asynchronous sensors, event-driven sampling, and device failures occur.Source: §A.1.1 · p. 13
A.1.2 IRREGULAR TIME SERIES FORECASTING
Irregular time-series forecasting predicts future values from observations with non-uniform timing. The forecasting setup includes observation masks and time gaps, and models must account for irregular intervals and align asynchronously sampled variables.
A.1.2 IRREGULAR TIME SERIES FORECASTING
- Irregular time-series forecasting predicts values over a future horizon H from an observed multivariate series.The forecasting model takes the observed series, its masks, and the time gaps as inputs.Source: §A.1.2 · p. 13
- Models must account for non-uniform temporal intervals, for example by encoding time gaps to avoid assuming uniform sampling.The passage gives exponential decay, continuous-time embeddings, and neural ODE dynamics as examples of temporal encoding approaches.Source: §A.1.2 · p. 13
- When variables are sampled asynchronously, models must align their representations by aggregating information with respect to time.Source: §A.1.2 · p. 13
A.2 DATASETS
The section describes datasets adapted for irregular forecasting across financial, event, biological, health, activity, climate, network, environmental, industrial, and vehicle-sensing settings. Each description specifies selected series or variables and forecasting windows.
A.2 DATASETS
- The health, activity, and climate datasets cover ICU signals, wearable movement, and station-based climate conditions, respectively.PhysioNet uses 24 observed hours; Human Activity uses 6,000 observed milliseconds; USHCN uses 24 observed months, with the section specifying their respective prediction windows.Source: §A.2 · pp. 13–14
- The remaining datasets describe network traffic, wearable activity, air quality, cluster workloads, and vehicle-road sensing for forecasting.CESNET, Pamap2, EPA-Air, ClusterTrace, and APTC each specify selected series or variables and dataset-specific observed and prediction windows.Source: §A.2 · p. 14
- FNSPID pairs irregularly timed financial news with stock-price variables; the benchmark selects 10 series and 6 variables for forecasting.It uses 730 observed days to predict fluctuations and sentiment 30, 180, or 365 days ahead.Source: §A.2 · p. 14
- GDELT captures irregularly timed global events; the benchmark selects 6 series and 4 variables to forecast event intensity and regional stability.The observed period is 365 days, with prediction horizons of 30, 90, or 180 days.Source: §A.2 · pp. 14–15
- Seabirds uses GPS and accelerometer records from movement and behavior, with 108 series and 3 variables selected for forecasting.The setup observes 1,440 minutes and predicts 60, 360, or 720 minutes ahead.Source: §A.2 · p. 15
- Table 4 presents statistics for the evaluation datasets.Source: Table 4 · p. 15
A.3 METRICS
The section defines error-based and non-error-based measures for assessing forecast accuracy, trend and shape, structural changes, and complexity.
A.3 METRICS
- Error-based metrics quantify prediction error, and lower values indicate better performance.The section lists MSE, MAE, MAPE, SMAPE, RMSE, WAPE, and MSMAPE.Source: §A.3 · p. 16
- PCC measures linear association and R2 measures how well predictions account for true-series variance; higher values indicate better trend or variability capture.PCC ranges from −1 to 1, while R2 ranges from −∞ to 1.Source: §A.3 · p. 16
- DA scores correct change direction, while DTW compares overall series shape under nonlinear time alignment.Higher DA and lower DTW indicate better performance.Source: §A.3 · p. 16
- CPN measures change-point-count difference, while IER compares predicted and true-series complexity and information content.Smaller CPN values and IER values closer to 1 indicate better performance.Source: §A.3 · p. 16
A.4 IRREGULAR TIME SERIES CHARACTERISTICS
This block describes BITS’s irregularity measures, model-efficiency findings, performance-comparison resources, leaderboard aggregation, and scope limitation.
A.4 IRREGULAR TIME SERIES CHARACTERISTICS
- BITS measures missing rate from the original observation mask, missing-pattern complexity with missing entropy, sampling irregularity with interval coefficient of variation, and skewness by averaging variable-level values.These characteristics capture incompleteness, missing-pattern structure, sampling-time irregularity, and distributional asymmetry, respectively.Source: §A.4 · pp. 16–17
A.5.1 MODEL EFFICIENCY ANALYSIS
- APN has the lowest MSE and fastest inference among temporal segmentation and aggregation methods on USHCN, with moderate memory use.Across methods, accuracy, latency, and memory should be considered jointly; greater complexity does not necessarily mean better accuracy or faster inference.Source: §A.5.1 · pp. 17–18
A.6 LEADERBOARD
- The leaderboard aggregates per-task MSE and DTW ranks and relative errors against GRU-D; lower values indicate better overall performance.Warpformer appears in available per-task rankings but is excluded from the overall leaderboard because Pamap2 results are missing.Source: §A.6 · p. 18, Figure 10 · p. 24
A.5.2 FULL RESULTS OF PERFORMANCE COMPARISON ON IRREGULAR TIME SERIES FORECASTING
- Tables 5–8 report model comparisons using MSE and MAE, including comparisons between forecasting models and time-series foundation models.The supplied passages identify the tables’ metrics and scope but do not provide their result values.Source: Table 5 · p. 19, Table 6 · p. 20, Table 7 · p. 21
A.7 LIMITATIONS AND FUTURE WORK
- BITS cannot exhaustively represent all real-world irregular scenarios or emerging architectures, and future work will expand its datasets and baselines.Source: §A.7 · p. 19