Beyond MSE: Rethinking the Evaluation Metric and Benchmarking for Irregular Time Series Forecasting

Introduction

• Irregular time series have non-uniform timestamps and sample-specific sampling patterns, while evaluation metrics have received less study than forecasting model design.The ideal forecasting metric measures continuous-time discrepancy between predicted and true trajectories over the future interval.Source: §1 · p. 1
• In Figure 1, Model B has lower continuous-time error but Model A lower MSE because [0.0, 0.4] is oversampled; MSE reflects sample-specific observation-time risk, not continuous-time risk.The observation-time risk weights prediction errors by the sample-specific timestamp density p(t | X), confounding evaluation with sampling patterns.Source: Figure 1 · p. 1, §1 · pp. 1–2
• CSE uses importance weighting to estimate continuous-time risk, and its asymptotic estimation error is no greater than MSE’s; Continuous-time Absolute Error (CAE) similarly debiases MAE.CSE reduces dense-region contributions and increases sparse-region contributions according to the observation-time density.Source: §1 · p. 2
• CSE recovers continuous-time risk more accurately than MSE on synthetic and semi-synthetic data, while MSE alone may misrepresent real-world performance; both identified gaps shape their discrepancies.The benchmark findings attribute MSE–CSE differences jointly to sampling-dependence and temporal-distribution gaps.Source: §1 · p. 2
• The benchmark covers synthetic, semi-synthetic, and real-world datasets; synthetic and semi-synthetic data provide true continuous or regular trajectories for validating CSE and analyzing metric discrepancies.The benchmark is designed to evaluate models’ continuous-time predictive performance.Source: §1 · p. 2

2.1. Irregular Time Series Forecasting

• Prior models either represent continuous latent dynamics or directly process irregular observations, while newer methods combine multiscale representations and graph structures.Examples include Latent ODE, Neural CDE, and Continuous Recurrent Unit (Kidger et al., 2020; Schirmer et al., 2022); SeFT, mTAN, and Raindrop (Horn et al., 2020; Shukla & Marlin, 2021; Zhang et al., 2022); and Warpformer and GraFITi (Zhang et al., 2023).Source: §2.1 · p. 2
• Existing studies focus on model design and typically use MSE or MAE at observed future timestamps, whereas this work examines how irregular sampling affects model comparison and selection.Source: §2.1 · p. 2

2.2. Benchmarks and Evaluation for Irregular Time Series

• Prior benchmarks standardize data, tools, tasks, and experiments: MIMIC-IV covers sparse clinical series, PYRREGULAR provides data structures and classification settings, and Time-IMM adds multimodal datasets.Physiome-ODE supplies controlled ODE-generated trajectories for comparing forecasting models (Bui et al., 2024; Spinnato & Landi, 2025; Chang et al., 2025; Kl¨otergens et al., 2025).Source: §2.2 · pp. 2–3
• Physiome-ODE is the closest prior work, but retains sampled-query MSE and does not test query-time dependence or whole-interval risk; this work proposes CSE and separates sampling-dependence from temporal-distribution gaps.Existing benchmarks mainly improve datasets, tasks, and experimental standardization rather than the evaluation objective itself (Kl¨otergens et al., 2025).Source: §2.2 · p. 3

3.1. Irregular Time Series Forecasting

• A forecasting model uses each sequence’s historical observations to estimate its latent process at future query times, while test ground truth is available only at finite future timestamps sampled from p(t | Xs).Source: §3.1 · p. 3

3.2. MSE and MAE under Irregular Sampling

• MSE gives greater evaluation weight to densely observed future regions and less to sparsely observed regions, so its objective depends on both prediction errors and p(t | X).Existing studies typically compute MSE and MAE at actually observed future timestamps.Source: §3.2 · p. 3

4.1. Continuous-Time Risk

• Rct weights every unit of future time equally, independently of actual timestamp sampling, and measures predictive error over the entire continuous-time process.It is adopted as the target measure because MSE’s observation-distribution risk depends on the trajectory-specific sampling distribution p(t | X).Source: §4.1 · p. 3

4.2. Estimating Continuous-Time Risk

• CSE estimates continuous-time risk from irregular observations using self-normalized importance weights based on estimated sampling densities, reducing dense-region contributions and increasing sparse-region contributions.The method uses leave-one-out Gaussian kernel density estimation to limit an evaluation timestamp’s influence on its own density estimate, then averages trajectory-level estimates equally; the same construction yields CAE under absolute error.Source: §4.2 · p. 4
• As the number of trajectories and minimum observations per trajectory grow, CSE’s asymptotic error is no greater than MSE’s, and is strictly smaller when observation-distribution risk differs from continuous-time risk.The metrics are asymptotically equivalent when timestamps are uniformly distributed, under the stated assumptions.Source: §4.2 · p. 4

4.3. Sources of Evaluation Discrepancy

• The evaluation discrepancy separates into Gsamp, driven by sample-specific sampling, and Gtime, driven by population-level temporal nonuniformity; GSE estimates the intermediate global-time risk.Gsamp is zero when sampling is independent of trajectory, while Gtime is zero under uniform population-level sampling; the decomposition supports separately quantifying their effects on scores and rankings.Source: §4.3 · pp. 4–5

5.1. Datasets

• Synthetic-Regime and Synthetic-Multiscale provide analytical continuous ground truth, while semi-synthetic ETTm1 and Weather samples use complete future grids as reference risk.Test-time nonuniformity is varied by α ∈ {0, 0.3, 0.5, 0.9, 0.99} while training data and model predictions remain fixed, isolating evaluation-protocol effects.Source: §5.1 · p. 5
• The benchmark covers eight irregular datasets: USHCN, MIMIC-III, HumanActivity, GDELT, RepoHealth, StudentLife, FNSPID, and CESNET.They span climate, clinical, activity-sensor, event, software, mobile-sensing, stock, and network-traffic data; preprocessing and splits follow public settings.Source: §5.1 · p. 5

5.2. Baselines

• Eleven baseline models span recurrent or continuous-time, set/attention/Transformer, graph/hypergraph, and pre-alignment or patch-aggregation paradigms.The models include GRU-D and NeuralFlow (Che et al., 2016; Bilos et al., 2021), SeFT, mTAN, and Warpformer (Horn et al., 2020; Shukla & Marlin, 2021; Zhang et al., 2023), GraFITi, HyperIMTS, tPatchGNN, and ASTGI (Li et al., 2025; Liu et al., 2026a), and KAFNet and APN (Zhou et al., 2026; Liu et al., 2026b).Source: §5.2 · p. 6

5.3. Training and Evaluation Protocol

• All models train on observed-timestamp MSE and use validation MSE for selection; MSE, GSE, and CSE are computed from one test-prediction set per run.Test-timestamp density estimates are shared across models per dataset, bGsamp and bGtime analyze discrepancy sources, and experiments use five random seeds.Source: §5.3 · p. 6

6.3. How Do the Sampling-Dependence and Temporal-Distribution Gaps Influence MSE–CSE Discrepancies?

• Gap contributions varied by dataset: sampling-dependence averaged 5.9% on GDELT and −5.8% on RepoHealth, while the temporal-distribution gap averaged 3.3% on CESNET.On USHCN, the sampling-dependence and temporal-distribution gaps were −7.8% and −2.2%, respectively; both gaps were comparatively small on the remaining datasets.Source: Figure 2 · p. 7, §6.3 · p. 8

6.1. Can CSE Better Recover Continuous-Time Risk and Rankings than MSE?

• On Synthetic-Regime and Weather at α = 0.9, CSE exactly matched the continuous-time-risk ranking of all five models, whereas MSE matched neither ranking.The five models were SeFT, GRU-D, GraFITi, tPatchGNN, and KAFNet.Source: §6.1 · p. 6, Table 2 · p. 7

6.2. Does MSE Fully Reflect Continuous-Time Predictive Performance on Real-World Data?

• CSE is not a fixed-direction or fixed-ratio correction of MSE, and the metrics yield different model-level conclusions despite GraFITi retaining the best average rank.GraFITi ranked first on three datasets under MSE but one under CSE; HyperIMTS increased from two to three first-place rankings and APN from one to two.Source: §6.2 · p. 7
• Five of eight real-world datasets showed pairwise rank inversions under CSE versus MSE, and the best-performing model changed on MIMIC-III and RepoHealth.MIMIC-III and StudentLife had three inversions each, GDELT and USHCN had two each, and RepoHealth had one.Source: §6.2 · p. 7, Figure 3 · p. 8

6.3. How Do the Sampling-Dependence and Temporal-Distribution Gaps Influence MSE–CSE Discrepancies?

• Within a dataset, discrepancy components generally had similar signs across models, but model-specific magnitudes could change rankings, as when MIMIC-III’s best model shifted from GraFITi under MSE to HyperIMTS under CSE.GraFITi’s sampling-dependence gap was −2.2% versus HyperIMTS’s +0.8%; on RepoHealth, GraFITi’s more negative gap than APN’s erased its MSE advantage, making APN best under CSE.Source: §6.3 · p. 8

6.1. Can CSE Better Recover Continuous-Time Risk and Rankings than MSE?

• At α = 0.9 on Synthetic-Regime, MSE’s relative error was 80.6% versus 34.7% for CSE; CSE’s error also remained substantially lower at α = 0.99.Under uniform sampling, both metrics accurately recover continuous-time risk, but MSE’s discrepancy grows as sampling becomes less uniform.Source: §6.1 · p. 6, Figure 4 · p. 8

7. Conclusion

• Observation-point MSE and MAE estimate sampling-induced observation-distribution risk rather than continuous-time risk over the entire future interval.The paper identifies this as an objective mismatch in evaluating irregular forecasts.Source: §7 · p. 8
• CSE estimates continuous-time risk through self-normalized importance weighting, with asymptotic estimation error no greater than MSE.The paper proposes CSE as a way to mitigate the mismatch between observation-distribution and continuous-time risk.Source: §7 · p. 8
• The risk discrepancy is decomposed into a sampling-dependence gap and a temporal-distribution gap, characterizing how sampling structures affect model evaluation.Source: §7 · p. 8
• A benchmark spanning synthetic, semi-synthetic, and real-world datasets validates CSE and systematically evaluates models’ continuous-time predictive performance.Source: §7 · p. 8

A. Proof of Theorem 1

• The proof introduces oracle importance-sampling terms and estimated counterparts to construct the trajectory-level CSE estimator.It then applies the continuous mapping theorem to that estimator.Source: A. Proof of Theorem 1 · pp. 10–11
• Under Assumption 1, the timestamp density is bounded below and its estimate is uniformly consistent, enabling the conditional strong-law argument.Conditional on each trajectory, future timestamps are independently and identically distributed according to its sampling density.Source: A. Proof of Theorem 1 · p. 10
• For the joint limit, Assumption 1 holds uniformly over test trajectories, which are mutually independent with bounded trajectory-level risks.A further strong-law application combines the trajectory-level limits.Source: A. Proof of Theorem 1 · p. 11

• The derivation separately considers a sampling-dependence gap.Source: B. Derivation of the Evaluation… · p. 11
• For the temporal-distribution gap, the derivation takes U ∼Unif(T ) and uses EU[|T |pT (U)] = 1.Source: B. Derivation of the Evaluation… · p. 11
• Gsamp captures the effect of sample-dependent sampling, whereas Gtime captures population-level temporal non-uniformity.Source: B. Derivation of the Evaluation… · p. 11

C. Global-Time Squared Error

• The marginal timestamp density is estimated by averaging trajectory-specific conditional densities.Source: C. Global-Time Squared Error · p. 11
• GSE uses bp(t | Xs) as the proposal density and bpT(t) as the target density.The supplied passages identify these densities in the definition of GSE but do not provide a legible complete formula.Source: C. Global-Time Squared Error · pp. 11–12
• Table 3 reports complete MSE and CSE results for CESNET, FNSPID, GDELT, and HumanActivity.Values are mean ± standard deviation over three random seeds, with standard deviations rounded to three decimal places; the supplied caption gives no result values.Source: Table 3 · p. 12
• The passage displays bpT(ts,i) and bp(ts,i | Xs) as terms, but does not establish that they form a GSE weighting ratio or specify their interpretation.Source: C. Global-Time Squared Error · p. 12
• GSE evaluates each trajectory under the dataset-level marginal timestamp distribution, removing sample-specific sampling effects while retaining population-level temporal non-uniformity.This makes GSE an empirical counterpart of Rglobal and an intermediate reference between MSE and CSE.Source: C. Global-Time Squared Error · p. 12

D.1. Synthetic and Semi-Synthetic Data

• The controlled benchmark includes two synthetic datasets and semi-synthetic ETTm1 and Weather, with 16 historical inputs and 128 test queries per trajectory.Each dataset has 512 training, 128 validation, and 128 test trajectories; test timestamps are resampled at five α levels, with 20 shared resamples per level.Source: §D.1 · pp. 12–13

D.2. Real-World Datasets

• The real-world benchmark spans eight datasets across healthcare, climate, human sensing, finance, software engineering, international events, and network systems, with varied irregularity mechanisms.Only numerical time-series modalities are used.Source: §D.2 · p. 13

D.3. Baselines and Hyperparameters

• Eleven representative recurrent, continuous-time, set-based, attention-based, graph-based, and pre-alignment models are compared using consistent configurations across real-world datasets.Source: §D.3 · p. 14

D.4. Training and Evaluation Details

• All models use Adam with masked observation-point MSE, train for 100 epochs, and retain the checkpoint with lowest validation MSE; experiments use three seeds.Controlled experiments use uniformly sampled training and validation target timestamps, and each sampling-shift evaluation uses 20 independent timestamp resamples per α.Source: §D.4 · pp. 15–16

E.1. Complete Results on Synthetic and Semi-Synthetic Data

• As sampling became more non-uniform, MSE generally deviated further from Rct, whereas inverse-density-weighted CSE remained substantially closer under strong non-uniformity.Under uniform sampling, CSE reduces to MSE; the advantage of correction grows with mismatch between observed and uniform temporal distributions.Source: Figure 5 · p. 16, §E.1 · p. 17

D.4. Training and Evaluation Details

• CSE’s trajectory-specific timestamp densities are estimated with leave-one-out Gaussian kernels, and trajectory errors are normalized independently before equal-weight averaging.Estimated densities are lower-bounded by 10^-8; padding and timestamps with no observed target variables are excluded from density estimation and metric computation.Source: §D.4 · pp. 16–17

E.1. Complete Results on Synthetic and Semi-Synthetic Data

• At α = 0.9, CSE recovered Rct’s exact five-model ranking on all four controlled datasets, while MSE produced at least one ranking inversion.The result shows temporal sampling can affect models differently because their prediction errors vary over time.Source: Table 7 · p. 17, §E.1 · p. 17

E.2. Complete Results on Real-World Datasets

• Complete MSE, CSE, MAE, and CAE results cover eight real-world datasets and report means, standard deviations, and ranks.Tables 3–6 show standard deviations over three random seeds; bold and underlined values mark the best and second-best means, while superscripts give ranks based on unrounded means.Source: §E.2 · p. 17
• CSE is generally lower than MSE on CESNET, FNSPID, GDELT, and HumanActivity, while the opposite trend is more common on MIMIC-III, RepoHealth, and USHCN.The size of the change varies across models within a dataset, consistent with sampling distributions interacting with model-specific temporal error patterns rather than applying a dataset-level scaling factor.Source: §E.2 · p. 17
• Most model–dataset combinations remain relatively stable across seeds, but RepoHealth and USHCN show larger variation for several models.The reported standard deviations help distinguish the systematic metric effect from training variability.Source: §E.2 · p. 18
• MAE-to-CAE corrections vary by dataset and model and can change rankings, indicating that observation-point metrics may not fully characterize continuous-time performance.The best MIMIC-III model changes from GraFITi under MAE to HyperIMTS under CAE; complete real-world trajectories are unavailable, so the results do not establish which metric is exact.Source: §E.2 · p. 18