PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond MSE: Rethinking the Evaluation Metric and Benchmarking for Irregular Time Series Forecasting

Rongwen Li, Haixin Xie, Xiao Wang, Changjian Chen

arXiv:2608.17293v1cs.LGcs.AI

TL;DR

Irregular forecasting research has emphasized model design while relying on MSE, which can reflect timestamp sampling rather than continuous-time predictive performance. This paper proposes importance-weighted CSE and finds it recovers continuous-time risk more accurately than MSE in benchmark experiments.

  • Problem

    Evaluation metrics for irregular time-series forecasting remain insufficiently studied despite extensive progress in model design and reliance on observed-timestamp MSE or MAE.

  • Method

    The paper proposes Continuous-time Squared Error (CSE), using self-normalized importance weighting to estimate continuous-time risk from irregular timestamps.

  • Results

    CSE recovers continuous-time risk more accurately than MSE under non-uniform sampling, while MSE alone may not fully reflect real-world continuous-time predictive performance.

  • Takeaways & Limitations

    Observation-point MSE can estimate sampling-induced observation-distribution risk rather than continuous-time risk, making CSE a relevant evaluation metric for irregular forecasting.

  • Takeaways & Limitations

    The theoretical analysis assumes conditionally independent future timestamps with a sampling density bounded below by a positive constant.

Abstract

from arXiv · show

Existing research on irregular time-series forecasting has primarily focused on model design, while evaluation metrics remain insufficiently studied. Existing benchmarks typically use mean squared error (MSE) as the evaluation metric. We show that, in irregular forecasting, MSE is determined not only by the model prediction but also by the sample-specific timestamp sampling distributions, leading to a biased assessment of the models' continuous-time predictive performance. To address this issue, we propose the Continuous-time Squared Error (CSE), which employs importance weighting to eliminate the influence of the timestamp sampling distributions. We further theoretically prove that CSE's asymptotic estimation error with respect to continuous-time risk is no greater than that of MSE. Finally, we construct a systematic benchmark covering synthetic, semi-synthetic, and eight real-world datasets to validate the effectiveness of CSE and systematically evaluate models' continuous-time predictive performance. Experiments show that CSE can recover continuous-time risk more accurately than MSE, while relying solely on MSE may not fully reflect models' continuous-time predictive performance in real-world scenarios. Our code can be obtained at https://github.com/hnu-vis/ITS-Bench.

1. Introduction

Irregular forecasting research has advanced model design faster than evaluation metrics, while MSE can conflate predictive performance with sample-specific timestamp distributions. The paper proposes CSE and a systematic benchmark to evaluate continuous-time predictive performance more faithfully.

  • Motivation: Irregular time series have non-uniform timestamps and sample-specific sampling patterns, motivating continuous-time evaluation across the future interval.Such data arise in healthcare, environmental monitoring, and human activity analysis.
  • Motivation: Existing irregular forecasting emphasizes continuous-time, attention-based, graph-based, and patch-based model design, but evaluation metrics remain insufficiently studied.Common metrics such as MSE and MAE were developed primarily for regularly sampled forecasting.
  • Evaluation bias: MSE estimates observation-time risk weighted by p(t | X), so sample-specific timestamp distributions can bias assessments of continuous-time predictive performance.Model A can obtain lower MSE than Model B despite Model B having lower continuous-time error because sampling is denser in [0.0, 0.4].
  • Proposed metric: CSE uses importance weighting to reweight errors by the observation-time density, reducing dense-region contributions and increasing sparse-region contributions.The paper proves that CSE’s asymptotic estimation error is no greater than MSE’s.
  • Benchmark: The benchmark covers synthetic, semi-synthetic, and real-world datasets to validate CSE and systematically evaluate models’ continuous-time predictive performance.Synthetic and semi-synthetic datasets provide true continuous or regular trajectories for evaluation.

2. Related Work

Prior irregular time-series research has emphasized continuous-time model design and benchmark construction, while usually evaluating predictions with MSE or MAE at observed future timestamps. This work instead examines the evaluation protocol, including how irregular sampling and query-time distributions affect model comparison and continuous-time risk assessment.

  • Modeling irregular time series: Irregular time-series models address sparse, asynchronous, and nonuniform observations by modeling continuous-time dynamics with tools such as Latent ODE, Neural CDE, and Continuous Recurrent Unit.These methods handle arbitrary time intervals through ordinary differential equations, controlled differential equations, or continuous state transitions.
  • Evaluation protocols: Existing studies typically evaluate predictions using MSE or MAE at observed future timestamps, rather than studying the evaluation protocol itself.The paper investigates how irregular sampling affects model comparison and selection.
  • Benchmarking: Prior benchmarks have expanded data resources, software frameworks, datasets, tasks, and experimental standardization for irregular time series.Examples include MIMIC-IV, PYRREGULAR, and Time-IMM, which support unified tasks, analytical tools, or multimodal multivariate benchmarks.
  • Benchmarking: Physiome-ODE provides controlled ODE-generated trajectories for more discriminative irregular forecasting comparisons but still evaluates predictions with MSE at sampled query times.It therefore does not examine dependence on the query-time distribution or recovery of risk over the entire future interval.

3. Preliminaries

The section formalizes irregular time-series forecasting from historical observations to latent-process predictions over future query times. It also shows that standard MSE evaluates errors at sampled timestamps, making the objective depend on the conditional timestamp distribution as well as prediction quality.

  • Problem setup: Each dataset comprises S irregular time series, with the s-th sequence represented by historical timestamp–observation pairs Os.The observations are denoted Os = {(qs,i, xs,i)}Hs.
  • Problem setup: Given historical observations Os, forecasting model fθ estimates the latent process over a future interval T and outputs X̂s(t) = fθ(Os, t) at query time t.Xs(t) denotes the ground-truth latent process.
  • Evaluation observations: Test datasets provide ground-truth future values only at finite timestamps {ts,j}Lj=1 sampled from the conditional distribution p(t | Xs).Thus, evaluation uses observations generated according to the process-dependent timestamp sampling mechanism.
  • MSE evaluation: MSE averages errors within each trajectory and weights trajectories equally, effectively estimating risk under the observed future-timestamp distribution.Consequently, densely sampled temporal regions receive greater evaluation weight than sparsely sampled regions, so MSE depends on p(t | X) as well as model errors.

4. Continuous-Time Risk for Irregular Forecasting

This section defines a continuous-time risk that evaluates forecasting over the entire future interval independently of sample-specific timestamp sampling. It develops CSE via self-normalized importance sampling, establishes its asymptotic non-inferiority to MSE, and decomposes evaluation discrepancies into sampling and temporal effects.

  • Continuous-time risk: Rct assigns equal weight to each unit of time and is independent of the sampling distribution of observed timestamps.It characterizes forecasting capability over the entire continuous-time process.
  • CSE construction: CSE estimates Rct from irregular observations by using the conditional timestamp density for importance weighting and self-normalization.The weighting decreases contributions from densely sampled regions and increases those from sparsely sampled regions; self-normalization limits finite-sample variance from extreme weights.
  • CSE construction: Leave-one-out Gaussian kernel density estimation reduces an evaluation timestamp’s influence on its own estimated temporal density.Excluding the current point prevents additional kernel mass at its location and produces a more robust local density estimate.
  • Asymptotic analysis: Under Assumption 4.1, CSE’s asymptotic estimation error for continuous-time risk is no greater than MSE’s, with equality under uniform timestamp sampling.When observation-distribution risk differs from continuous-time risk, CSE has strictly smaller asymptotic error.
  • Evaluation discrepancy decomposition: The global-time risk removes sample-specific sampling effects while retaining population-level temporal nonuniformity, enabling separate estimation of both discrepancy components through GSE.Gsamp captures sample-dependent sampling, whereas Gtime captures population-level temporal nonuniformity; the decomposition supports analysis of their effects on model scores and rankings.

5. Benchmark Protocol

The benchmark combines synthetic, semi-synthetic, and real-world irregular time-series datasets with representative models under unified training and evaluation protocols. It varies test-time sampling distributions while holding training data and predictions fixed to isolate metric effects.

  • Synthetic and Semi-Synthetic Data: The benchmark includes two fully synthetic and two semi-synthetic datasets designed to provide directly computable continuous-time risks.Synthetic-Regime varies forecasting difficulty from smooth, low-amplitude behavior to high-frequency periodic behavior, while Synthetic-Multiscale combines trends, multiscale periodicity, and local transients.
  • Synthetic and Semi-Synthetic Data: Sampling nonuniformity is controlled by α ∈{0, 0.3, 0.5, 0.9, 0.99}, while training data and model predictions remain identical across conditions.Only the test-time distribution changes, isolating the effect of the evaluation protocol.
  • Real-World Datasets: Eight real-world irregular time-series datasets span different application domains and sampling patterns, including climate, ICU, and location-sensor data.USHCN provides long-term weather-station observations, MIMIC-III sparse ICU records, and HumanActivity irregular location-sensor measurements.
  • Representative Models: Eleven representative models cover recurrent or continuous-time, set-, attention-, Transformer-, graph-, and hypergraph-based temporal modeling paradigms.The listed models include GRU-D, NeuralFlow, SeFT, mTAN, Warpformer, GraFITi, HyperIMTS, tPatchGNN, and ASTGI.
  • Training and Evaluation Protocol: All models use identical data splits and input information, train with MSE over observed timestamps, and compute MSE, GSE, and CSE from one test-prediction set per run.Model selection uses validation MSE, while remaining training and hyperparameter settings follow public implementations; metric differences therefore arise solely from the evaluation protocol.

6. Experiments

Experiments show that CSE more accurately recovers continuous-time risks and model rankings than MSE under non-uniform sampling. On eight real-world datasets, MSE and CSE can yield different scores, rankings, and model selections because sampling-dependence and temporal-distribution gaps vary across datasets and models.

  • Controlled experiments: Controlled experiments evaluate CSE and MSE using five models across two synthetic and two semi-synthetic datasets with complete future ground truth.Continuous-time risk is computed by dense temporal integration for synthetic data and over complete future trajectories for semi-synthetic data.
  • Controlled experiments: 80.6% and 34.7% are the MSE and CSE relative errors on Synthetic-Regime when α = 0.9, respectively.Under uniform sampling, both metrics accurately recover continuous-time risk; increasing α increases MSE’s discrepancy while CSE maintains lower estimation error.
  • Controlled experiments: At α = 0.9, CSE exactly recovers the Rct ranking of all five models on Synthetic-Regime and Weather, whereas MSE differs on both datasets.This supports CSE’s more accurate recovery of continuous-time risk and model rankings under non-uniform test sampling.
  • Real-world evaluation: Five of eight real-world datasets exhibit pairwise rank inversions when replacing MSE with CSE, and the best-performing model changes on MIMIC-III and RepoHealth.MIMIC-III and StudentLife each have three inversions, GDELT and USHCN each have two, and RepoHealth has one.
  • Real-world evaluation: GraFITi ranks first on three datasets under MSE but one under CSE, while HyperIMTS increases from two to three first-place rankings and APN from one to two.GraFITi retains the best average rank, but MSE and CSE do not provide fully consistent conclusions about relative model advantages.
  • Discrepancy analysis: The sampling-dependence and temporal-distribution gaps jointly influence MSE–CSE discrepancies, with their relative contributions varying across datasets and models.GDELT and RepoHealth are primarily affected by sampling-dependence, CESNET by temporal-distribution, and USHCN by both components; model-specific error patterns can alter rankings.

7. Conclusion

The paper reframes irregular time-series forecasting evaluation from a continuous-time perspective, showing that observation-point MSE and MAE estimate sampling-distribution-induced risk rather than continuous-time risk. It proposes CSE, based on self-normalized importance weighting, to estimate continuous-time risk and provides a theoretical proof of its validity.

  • Conclusion: Observation-point MSE and MAE estimate observation-distribution risk induced by the sampling distribution, not continuous-time risk over the full future interval.This creates an objective mismatch between common evaluation metrics and continuous-time predictive performance.
  • Conclusion: The proposed Continuous-Time Squared Error (CSE) addresses this mismatch by estimating continuous-time risk through self-normalized importance weighting.
  • Conclusion: The paper theoretically proves the validity of the CSE-based continuous-time risk estimation approach.

A. Proof of Theorem 1

Under the positivity, uniform-consistency, and conditional i.i.d. timestamp assumptions, the CSE estimator consistently recovers continuous-time risk, with asymptotic estimation error no greater than MSE's. Equality occurs when observed and continuous-time risks coincide, while finite-sample self-normalization may remain biased.

  • Importance-weighted estimator: The proof introduces oracle importance-sampling terms and their estimated counterparts for constructing the CSE estimator.These terms provide the importance-weighted basis for the trajectory-level CSE estimator.
  • Trajectory-level convergence: Conditional i.i.d. future timestamps and the strong law yield convergence of the importance-weighted trajectory-level quantities.The proof combines the conditional strong law with the preceding weight-consistency result.
  • Joint convergence: Because |T| > 0, the continuous mapping theorem establishes convergence of the trajectory-level CSE estimator.Uniform assumptions over test trajectories and a further strong-law argument extend convergence to the joint estimator.
  • Theorem conclusion: CSE's asymptotic estimation error is no greater than MSE's, with both errors zero when p(t | X) = 1/|T| almost everywhere.Equality also holds whenever R_obs = R_ct, whereas the inequality is strict when R_obs ≠ R_ct.
  • Theorem conclusion: The self-normalized CSE estimator may be biased at finite sample sizes, although the theorem establishes consistency and asymptotic estimation error.The finite-sample caveat does not invalidate the asymptotic result.

B. Derivation of the Evaluation Discrepancy

The evaluation discrepancy is derived by separating a sampling-dependence gap from a temporal-distribution gap. The resulting terms distinguish sample-dependent sampling effects from population-level temporal non-uniformity.

  • The derivation identifies a sampling-dependence gap in the evaluation discrepancy.
  • For the temporal-distribution gap, the derivation introduces U ∼Unif(T ) and uses EU[|T |pT (U)] = 1.
  • Gsamp captures sample-dependent sampling, whereas Gtime captures population-level temporal non-uniformity.

C. Global-Time Squared Error · D. Benchmark Details

This section defines Global-Time Squared Error (GSE) using dataset-level marginal timestamp weighting and positions it as an intermediate reference between MSE and CSE. It also reports benchmark results across four real-world datasets, averaged over three random seeds.

  • D. Benchmark Details: Table 3 presents complete MSE and CSE results on CESNET, FNSPID, GDELT, and HumanActivity.The table covers four named datasets and reports both evaluation metrics.
  • D. Benchmark Details: The reported benchmark values are mean ± standard deviation over three random seeds.Standard deviations are rounded to three decimal places.
  • D. Benchmark Details: The benchmark table reports variability alongside metric means, with standard deviations rounded to three decimal places.This reporting convention applies to the complete MSE and CSE results described for the four datasets.
  • C. Global-Time Squared Error: The weighting ratio uses the dataset-level target density over the trajectory-specific proposal density.The supplied expression identifies b_pT(t_s,i) and b_p(t_s,i | X_s) as the numerator and denominator, respectively.
  • C. Global-Time Squared Error: GSE evaluates each trajectory under the dataset-level marginal timestamp distribution, removing sample-specific sampling effects while retaining population-level temporal non-uniformity.The marginal timestamp density is estimated by averaging trajectory-specific conditional densities.
  • C. Global-Time Squared Error: GSE serves as the empirical counterpart of R_global and an intermediate reference between MSE and CSE.Its construction uses b_p(t | X_s) as the proposal density and b_pT(t) as the target density.

D.1. Synthetic and Semi-Synthetic Data … E.1. Complete Results on Synthetic and Semi-Synthetic Data

The benchmark combines controlled synthetic and semi-synthetic settings with diverse real-world datasets, standardized model comparisons, and a unified evaluation protocol. Results under strong sampling non-uniformity show that CSE better recovers continuous-time risk and preserves model rankings more reliably than MSE.

  • D.1. Synthetic and Semi-Synthetic Data: The controlled benchmark uses two synthetic and two semi-synthetic datasets, with 512/128/128 train/validation/test trajectories and 16 historical inputs.Training and validation use 30 future target points, while each test realization contains 128 future query timestamps.
  • D.1. Synthetic and Semi-Synthetic Data: The semi-synthetic ETTm1 and Weather datasets retain 16 historical points and sample 30 future points for training and validation from 96-history, 512-future grids.Test timestamps are resampled under α ∈{0, 0.3, 0.5, 0.9, 0.99}, with 20 shared resamples per setting.
  • D.2. Real-World Datasets: The real-world benchmark spans eight datasets across healthcare, climate, human sensing, finance, software engineering, international events, and network systems.Their irregularity arises from event-triggered logging, activity-dependent collection, operational constraints, human scheduling, missing observations, and system jitter.
  • D.3. Baselines and Hyperparameters: The study compares eleven representative models spanning recurrent, continuous-time, set-based, attention-based, graph-based, and pre-alignment forecasting paradigms.Each model uses the same configuration across all real-world datasets to ensure consistent comparisons.
  • D.4. Training and Evaluation Details: All models use Adam with masked observation-point MSE, train for 100 epochs, and retain the checkpoint with the lowest validation MSE.The learning rate is 10^-3 for all models; NeuralFlow and mTAN use weight decay 10^-4, while other real-world baselines use none.
  • D.4. Training and Evaluation Details: Experiments are independently repeated with three random seeds, and results are reported as the mean and standard deviation across runs.Each controlled sampling-shift checkpoint is additionally evaluated with 20 independent timestamp resamples for every sampling strength α.
  • E. Additional Experimental Results: Under increasing sampling non-uniformity, CSE remains substantially closer to Rct than MSE because inverse-density weighting compensates for temporal concentration.The correction becomes increasingly beneficial as the mismatch between observed and uniform temporal distributions grows, despite possible finite-sample variability.
  • E.1. Complete Results on Synthetic and Semi-Synthetic Data: At α = 0.9, CSE recovers the same ranking of all five models as Rct on all four datasets, whereas MSE produces at least one ranking inversion.Different models are affected differently because their prediction errors vary over time, so temporal-distribution correction improves ranking fidelity.

E.2. Complete Results on Real-World Datasets

Complete real-world results show that CSE and CAE can differ systematically from MSE and MAE because timestamp sampling interacts with model-specific temporal error patterns. These effects vary across datasets and models, while most model–dataset combinations remain stable across random seeds.

  • Evaluation results: Tables 3–6 report complete MSE, CSE, MAE, and CAE results on eight real-world datasets, including standard deviations over three random seeds.Best and second-best mean values are bold and underlined, respectively; superscripts indicate ranks from unrounded means.
  • MSE–CSE comparison: CSE is neither a fixed-direction nor fixed-ratio transformation of MSE across real-world datasets.CSE is generally lower on CESNET, FNSPID, GDELT, and HumanActivity, whereas the opposite trend is more common on MIMIC-III, RepoHealth, and USHCN.
  • MSE–CSE comparison: Within a dataset, the magnitude of the MSE-to-CSE change varies across models, indicating interaction between sampling distributions and model-specific temporal error patterns.This systematic metric effect is distinct from training variability: most model–dataset combinations remain relatively stable across seeds, although RepoHealth and USHCN show larger variation for several models.
  • MAE–CAE comparison: The MAE–CAE comparison shows the same dataset- and model-dependent correction pattern and produces local rank changes.On MIMIC-III, the best-performing model changes from GraFITi under MAE to HyperIMTS under CAE.
Loading 2608.17293v1…