PaperlayerView exact revision on arXiv

Source-linked AI summary

CTIA-ODE: Continuous-Time Inter-Variable Attention for Long-Term Time Series Forecasting

Chengyi Pan, Huangqimei Zheng, Hao Xue

arXiv:2610.19831v2cs.LG

TL;DR

Discrete-time architectures can impose rigid alignment when incorporating heterogeneous exogenous variables, limiting continuous interaction with evolving effects. CTIA-ODE embeds asymmetric inter-variable attention in an ODE solver and adds trend and outlier stabilization. Across eight benchmarks, it is reported to achieve state-of-the-art forecasting performance with a favorable accuracy-efficiency trade-off.

  • Problem

    Discrete-time forecasting architectures may force heterogeneous endogenous and exogenous variables into rigid temporal alignment, limiting continuous interaction without explicit alignment.

    Source: §1 · pp. 2–3

  • Method

    CTIA-ODE embeds continuous-time inter-variable attention in an ODE solver so an evolving endogenous trajectory queries static exogenous context, alongside LRA and ORHL stabilization.

    Source: §1 · p. 3, §3.3 · p. 5, §5 · p. 9

  • Results

    Across eight benchmarks and diverse datasets and horizons, CTIA-ODE achieves state-of-the-art performance with a favorable accuracy-efficiency trade-off, compact memory, and low inference latency.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The framework offers continuous-time asymmetric exogenous interaction without explicit step-by-step alignment, with reported efficiency advantages particularly under long-horizon forecasting.

    Source: §1 · p. 3

    The current encoder assumes regularly sampled historical observations, so adapting it to irregularly sampled data with missing values remains a challenge.

    Source: L LIMITATIONS AND FUTURE WORK · p. 23

Abstract

from arXiv · show

Although multivariate time series forecasting has broad application prospects, effectively incorporating exogenous variables remains challenging. Prevailing discrete-time architectures often impose rigid temporal alignment across heterogeneous variables, which may limit their ability to capture continuously evolving external effects. To address this limitation, we introduce CTIA-ODE, a continuous-time forecasting framework centered on Continuous-Time Inter-Variable Attention (CTIA). Specifically, CTIA is embedded within an ordinary differential equation (ODE) solver, enabling the continuously evolving endogenous trajectory to dynamically query a static exogenous context along the integration path without requiring explicit temporal alignment. To improve the stability of the continuous latent dynamics, we further introduce a dual-stabilization strategy: architecturally, a Linear Residual Augmentation (LRA) module provides a macroscopic trend anchor, while an Outlier-Robust Hybrid Loss (ORHL) mitigates the influence of heavy-tailed anomalies during optimization. Extensive experiments across eight real-world datasets demonstrate that CTIA-ODE achieves state-of-the-art forecasting performance across diverse datasets and prediction horizons, while maintaining a favorable accuracy-efficiency trade-off with a compact memory footprint and low inference latency, particularly under long-horizon forecasting settings. Anonymous code is available at https://anonymous.4open.science/r/CTIA-ODE-15A6/.

1 INTRODUCTION

The introduction frames exogenous information and evolving latent representations as challenges for multivariate forecasting. CTIA-ODE uses continuous-time asymmetric interactions and dual stabilization, and is reported to achieve state-of-the-art performance across eight benchmarks with favorable efficiency.

1 INTRODUCTION

  • Forecasting systems may need exogenous variables because external shocks such as holidays, macroeconomic indicators, and extreme weather can drive severe endogenous fluctuations.The introduction argues that ignoring external variables caps predictive potential for systemic state shifts.Source: §1 · p. 1
  • The paper examines hidden-state evolution and latent-space organization as complementary views of continuous dynamics, in a context where complex dependency models may overfit and over-smooth high-frequency information.Figure 1(c–d) is cited for examining both temporal evolution and latent-space organization.Source: §1 · p. 2
  • Discrete architectures struggle to model continuous interactions between evolving endogenous trajectories and heterogeneous exogenous information without explicit temporal alignment.Forced step-by-step alignment can add computational redundancy and unnecessary interference between primary and auxiliary variables.Source: §1 · pp. 2–3
  • CTIA-ODE lets an evolving endogenous trajectory query static exogenous context along an ODE integration path, avoiding explicit step-by-step alignment.The paper also claims this mechanism avoids quadratic temporal complexity.Source: §1 · p. 3
  • The dual-stabilization strategy pairs LRA’s macroscopic trend anchor with ORHL’s mitigation of heavy-tailed anomalies during optimization.ORHL is also described as suppressing gradient spikes and improving learned vector-field robustness.Source: §1 · p. 3
  • CTIA-ODE achieves state-of-the-art forecasting performance across eight real-world benchmarks, with compact memory and low inference latency, particularly for long horizons.The introduction also reports a favorable accuracy-efficiency trade-off.Source: §1 · p. 3

2 RELATED WORK

The related work contrasts continuous-time forecasting with discrete models and identifies robustness and heterogeneous-variable interactions as coupled challenges. CTIA-ODE combines asymmetric exogenous-context interaction with trend anchoring and bounded-gradient loss.

2.1 CONTINUOUS-TIME FORECASTING MODELS

  • CTIA-ODE decouples trend learning from the continuous branch, using LRA and zero-initialization to maintain stable vector fields and prevent error amplification over long trajectories.The section contrasts this design with continuous-time models that may suffer representation collapse when one solver handles global trends and high-frequency spatial dynamics.Source: §2.1 · p. 3

2.2 ROBUST MULTIVARIATE DEPENDENCIES

  • CTIA selectively absorbs exogenous context without symmetric cross-talk, while ORHL combines localized quadratic convergence with a bounded linear penalty for large errors.The related work says the bounded gradients enhance robustness to heavy-tailed sensor anomalies and help mitigate solver instability.Source: §2.2 · p. 4

3 METHODOLOGY

The methodology formulates multivariate prediction using shared auxiliary context and known future time features, then processes it through continuous-time interaction, refinement, and residual augmentation. Its components address alignment, latent-state stability, trend anchoring, and outlier-robust optimization.

3.1 PROBLEM FORMULATION

  • The forecasting objective maps historical observations and shared auxiliary context to future multivariate values, using known future time features.One variable is designated primary, while the remaining M −1 variables form the shared auxiliary context.Source: §3.1 · p. 4

3.2 STRUCTURE OVERVIEW

  • The framework normalizes and encodes inputs, evolves hidden states with an ODE solver, refines the terminal state, and adds a parallel LRA prediction before denormalizing.Figure 2 identifies RK4 integration, Stable Refinement, and LRA’s prediction baseline as framework components.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 CONTINUOUS-TIME INTEGRAL ATTENTION

  • CTIA evaluates cross-variable dependencies continuously: static exogenous keys and values provide global context that the evolving endogenous state queries to adapt its trajectory.An initial asymmetric fusion constructs the starting endogenous state before integration.Source: §3.3 · pp. 4–5

3.4 STABLE REFINEMENT

  • Stable Refinement filters accumulated terminal-state numerical noise while preserving temporal details through a weight-sharing, two-iteration recurrent design.Each iteration applies self-attention and an MLP with residual connections and layer normalization.Source: §3.4 · p. 5

3.5 LINEAR RESIDUAL AUGMENTATION

  • LRA separates historical fluctuations into trend and seasonal components, maps them to the forecast horizon, and adds future-time effects fused with the refined latent state.Learnable scalar weights add these parallel outputs to the primary nonlinear prediction.Source: §3.5 · p. 6

3.6 OUTLIER-ROBUST HYBRID LOSS

  • ORHL applies a piecewise penalty to forecast residuals; large anomalies receive a bounded gradient, limiting loss signals entering the continuous branch.Within |e_i| ≤ δ, the loss combines quadratic curvature with a scaled ℓ1 penalty; for |e_i| > δ, the gradient saturates at ±(δ + λ).Source: §3.6 · p. 6

4 EXPERIMENTS

Experiments evaluate CTIA-ODE on eight real-world benchmarks against nine forecasting models, then assess its components, efficiency, and robustness. The reported results show strong benchmark performance, including long-horizon stability and a favorable accuracy-efficiency trade-off.

4.1 EXPERIMENTAL SETTINGS

  • Experiments cover eight benchmarks, nine state-of-the-art baselines, MSE and MAE metrics, and prediction horizons T = 96, 192, 336, and 720.The datasets span energy, meteorological, and other domains; the look-back window is L = 96.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 MAIN RESULTS (RQ1)

  • CTIA-ODE achieves the best performance on 44 of 80 metrics, versus 15 for TimeXer and 12 for PatchTST.On ETTm1, its MSE is 0.472 at T = 720 and 0.326 at T = 96.Source: Table 1 · p. 7, §4.2 · p. 7
  • On Traffic, CTIA-ODE achieves the lowest MAE across all prediction horizons, while processing exogenous variables asymmetrically to filter heterogeneous noise.The authors also report mitigating negative transfer on heterogeneous datasets including Exchange, Weather, and Traffic.Source: §4.2 · pp. 7–8

4.3 ABLATION STUDY (RQ2)

  • Replacing ORHL with pure MSE or pure MAE degrades performance: MSE overpenalizes heavy-tailed anomalies, whereas MAE converges more slowly and reaches suboptimal optima.The authors report that the hybrid objective mitigates both issues.Source: §4.3 · p. 8
  • At T = 720, removing LRA yields a nearly 16% MSE increase on ETTm1; removing continuous dynamics raises errors on complex high-frequency variation.The authors attribute macroscopic trend stability to LRA and local dynamics to the ODE branch; removing exogenous interactions also degrades results, particularly on Traffic.Source: §4.3 · p. 8

4.4 MODEL EFFICIENCY AND IN-DEPTH MECHANISTIC ANALYSIS (RQ3)

  • CTIA-ODE lies on the Electricity accuracy-efficiency Pareto frontier, with error 0.205 at T = 720 and inference speed of 6.8 ms per sample.The figure compares latency and forecasting error, with peak GPU memory represented by bubble size; the model operates within 4GB VRAM without OOM errors.Source: §4.4 · pp. 8–9, Figure 3 · p. 9
  • The Looped Stable Block performs best at K = 2; on Weather at T = 96, MSE rises from 0.154 at K = 2 to 0.161 at K = 5.The authors attribute degradation at K ≥ 3 to over-smoothing.Source: §4.4 · p. 8, Figure 3 · p. 9
  • Under heavy-tailed perturbations, ORHL maintains stable forecasting accuracy across contamination levels, unlike standard MSE, whose quadratic penalty amplifies localized gradient spikes.The figure examines varying outlier contamination ratios on ETTm1.Source: §4.4 · p. 9, Figure 3 · p. 9

5 CONCLUSION

CTIA-ODE uses asymmetric continuous-time attention to query static exogenous context without step-by-step temporal alignment. Across eight benchmarks, it reports state-of-the-art forecasting performance and a favorable accuracy-efficiency trade-off.

5 CONCLUSION

  • CTIA-ODE lets the evolving endogenous trajectory query static exogenous context without explicit step-by-step temporal alignment.Its dual stabilization strategy combines LRA trend anchoring with ORHL, which bounds prediction gradients and reduces the influence of heavy-tailed anomalies during optimization.Source: §5 · pp. 9–10
  • Across eight benchmarks, CTIA-ODE achieves state-of-the-art performance across datasets and forecasting horizons, with compact memory use and low inference latency.The authors identify extending CTIA to irregularly sampled multivariate time series and event-driven forecasting as future work.Source: §5 · p. 10

AI USE STATEMENT

The authors report using an LLM exclusively for copy-editing and proofreading, and state that they reviewed its revisions and retain responsibility for the paper.

AI USE STATEMENT

  • An LLM assisted only with sentence structure, grammar, fluency, and readability; the authors reviewed revisions and assume responsibility for the scientific content and final presentation.The statement describes copy-editing and proofreading assistance.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

The paper states that its time-series forecasting methodology uses no sensitive data and presents no potential ethical risks.

ETHICS STATEMENT

  • The authors state that the research involves no sensitive data and presents no potential ethical risks.They base this statement on the research's focus on time-series forecasting methodology.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The paper places its formal model in the main text and directs readers to supplementary material for implementation and evaluation details. Its experiments use eight real-world multivariate datasets and a shared exogenous-context strategy for forecasting all variables.

REPRODUCIBILITY STATEMENT

  • The formal model is defined in the main text, while supplementary material provides dataset specifications, metrics, experimental configurations, and source code.The eight benchmarks are summarized in Table 3.Source: REPRODUCIBILITY STATEMENT · p. 11, Table 3 · p. 15

A DETAILED DATASET DESCRIPTIONS

  • The eight benchmarks span domains and data characteristics including periodic electricity use, high-dimensional traffic, heterogeneous weather variables, non-periodic exchange rates, and noisy high-frequency ETT series.The datasets include 321 electricity clients, 862 traffic sensors, and 21 weather indicators.Source: A DETAILED DATASET DESCRIPTIONS · p. 15, Table 3 · p. 15
  • Forecasts and metrics cover all M variables, with the remaining M −1 variables projected into a shared static exogenous context queried by every channel.MSE and MAE average errors across all test samples, prediction steps, and variables; lower values indicate better accuracy.Source: A DETAILED DATASET DESCRIPTIONS · p. 15

B ROBUSTNESS TO IRREGULAR AND MISALIGNED EXOGENOUS CONTEXTS

On ETTm2 at T = 96, CTIA-ODE remains comparatively stable when exogenous observations are masked or shifted, unlike discrete baselines. The section also reports choices and analyses of the compute setup, complexity, ODE solver, ORHL parameters, and initialization.

B ROBUSTNESS TO IRREGULAR AND MISALIGNED EXOGENOUS CONTEXTS

  • The perturbation test keeps ETTm2’s Oil Temperature pristine and masks 30% of the six exogenous load features or shifts them backward by two steps.The comparison includes iTransformer and TimeXer.Source: B ROBUSTNESS TO IRREGULAR AND… · p. 16
  • CTIA-ODE’s MSE rises 3.0% under masking and 4.2% under shifting, versus iTransformer’s 23.0% drop under phase shift and TimeXer’s 11.0% error increase with missing values.The perturbation comparison is on ETTm2 at T = 96; Table 4 reports degradation relative to the pristine setting.Source: B ROBUSTNESS TO IRREGULAR AND… · p. 16, Table 4 · p. 16

C.1 ENVIRONMENT AND HARDWARE

  • Experiments use an NVIDIA GPU with 4GB VRAM, an Intel Core i7-9750H CPU, 16GB RAM, Python 3.8.10, PyTorch 1.12.0, CUDA 11.6, and torchdiffeq 0.2.5.Dataset-specific hyperparameters are tabulated in Table 5; splits are 6:2:2 for ETT and 7:1:2 for other datasets, with early stopping after 12 unimproved validation epochs.Source: §C.1 · p. 16, §C.2 · p. 16, Table 5 · p. 17

D COMPUTATIONAL COMPLEXITY ANALYSIS

  • CTIA-ODE’s stated complexity is O(M · L · D) + O(S(M · D^2 + M · Kex · D)) + O(K · M^2 · D), avoiding the O(L^2) temporal bottleneck.The analysis uses S = 5 integration steps, K = 2 unrolling iterations, and D = 128; the O(M^2) term is described as manageable.Source: D COMPUTATIONAL COMPLEXITY ANALYSIS · p. 17

E.1 IMPACT OF DIFFERENT ODE SOLVERS

  • RK4 yields the lowest forecasting error on both evaluated datasets, while Euler shows notable MSE degradation relative to the higher-order solvers.The sensitivity analysis compares Euler, Midpoint, and RK4 at T = 720.Source: §E.1 · p. 17, Table 6 · p. 18

E.2 SENSITIVITY TO ORHL HYPERPARAMETERS (δ AND λ)

  • Among tested ORHL settings, δ = 1.0 and λ = 5.0 perform best, balancing local optimization stability with rejection of macro-outliers.The tested values are δ ∈ {0.5, 1.0, 2.0} and λ ∈ {1.0, 5.0, 10.0}; the comparison is on Weather at T = 720.Source: §E.2 · p. 18, Table 7 · p. 18

E.3 EFFECT OF ZERO INITIALIZATION

  • Zero-initializing the base vector-field projection makes the attention contribution the dominant initial forcing term and produces smoother early convergence than Xavier initialization.The scaling parameter α is initialized to 0.1; Figure 4 contrasts the training loss trajectories.Source: §E.3 · p. 18

F EXTENDED VISUALIZATIONS OF FORECASTING RESULTS

At Weather’s extreme horizon T = 720, the visual comparisons contrast CTIA-ODE’s continuous forecasts with noise-prone Transformer and rigid-amplitude linear predictions. The section also explains ORHL’s piecewise penalty for residuals, including bounded gradients for large deviations.

F EXTENDED VISUALIZATIONS OF FORECASTING RESULTS

  • Zero initialization yields smoother early training loss than standard random initialization, which triggers stiffness and stalls convergence.Figure 4 compares the initialization strategies for the ODE state network.Source: Figure 4 · p. 19
  • At Weather T = 720, CTIA-ODE preserves temporal continuity against iTransformer’s high-frequency noise and tracks the ground-truth envelope better than DLinear’s fixed-amplitude oscillations.Figure 5’s top and bottom panels show the respective comparisons; CTIA-ODE captures decaying amplitude and nonlinear trends despite initial phase shifts.Source: F EXTENDED VISUALIZATIONS OF… · pp. 18–20, Figure 5 · p. 19

G THEORETICAL ANALYSIS OF THE OUTLIER-ROBUST HYBRID LOSS (ORHL)

  • ORHL is introduced to improve optimization stability for continuous-time ODE networks trained on noisy, heavy-tailed series, where large outliers can distort the learned vector field under standard quadratic objectives.The loss is applied to the aggregated final prediction and its forecasting residual.Source: G THEORETICAL ANALYSIS OF THE… · p. 20, §G.1 · p. 20

G.1 MATHEMATICAL FORMULATION

  • ORHL combines quadratic and L1 penalties when |ei| ≤ δ, then switches to a bounded linear penalty when |ei| > δ to limit large-residual gradients.The threshold δ separates the regions, and λ scales the linear term.Source: §G.1 · p. 20

G.2 GRADIENT BOUNDEDNESS AND OPTIMIZATION ROBUSTNESS

The section explains how ORHL bounds loss gradients to support robust adjoint integration, then details the CTIA-ODE implementation, its stable refinement block, baseline settings, and look-back-window results.

G.2 GRADIENT BOUNDEDNESS AND OPTIMIZATION ROBUSTNESS

  • With δ = 1.0 and λ = 5.0, ORHL clamps the maximum loss-gradient norm at 6.0, limiting gradients passed through the continuous branch without globally bounding the ODE vector field.The bounded loss gradient provides a well-conditioned initial state for adjoint integration and improves optimization robustness with large residuals.Source: §G.2 · p. 20

H PYTORCH-STYLE PSEUDOCODE FOR CTIA-ODE

  • The forward pass computes LRA trend and seasonal forecasts, encodes and splits primary and auxiliary variables, precomputes exogenous keys and values once, then aggregates nonlinear, time, and linear forecasts.The pseudocode also includes a looped stable block before nonlinear forecasting.Source: H PYTORCH-STYLE PSEUDOCODE FOR CTIA-ODE · p. 21

I DETAILED MATRIX FORMULATION OF THE LOOPED STABLE BLOCK

  • The Looped Stable Block refines Neural ODE outputs through K unrolled self-attention layers along the variable dimension, with Layer Normalization, residual connections, and a position-wise FFN.Setting K = 2 bounds refinement complexity at O(2 · M^2 · D), avoiding the stated deep spatial Transformer memory bottlenecks.Source: I DETAILED MATRIX FORMULATION OF THE… · pp. 21–22

J DETAILED BASELINE CONFIGURATIONS

  • The nine baselines use official repositories or the standardized Time-Series-Library, recommended architectural defaults, and a shared learning-rate and batch-size grid search.The configurations are intended to provide consistent, reasonably tuned comparisons.Source: J DETAILED BASELINE CONFIGURATIONS · pp. 22–23

K IMPACT OF LOOK-BACK WINDOW SIZE (L)

  • CTIA-ODE’s MSE and MAE decrease monotonically as look-back length grows on Weather and ETTh1 at T = 720, while Autoformer degrades and reaches OOM at L = 720.The authors attribute CTIA-ODE’s pattern to continuous-time integration filtering noise and extracting low-frequency patterns from longer histories.Source: K IMPACT OF LOOK-BACK WINDOW SIZE (L) · p. 23, Table 8 · p. 23

L LIMITATIONS AND FUTURE WORK

The authors identify limits involving regularly sampled inputs, dataset-specific training without zero-shot transfer, and reliance on RK4 for highly chaotic or turbulent systems.

L LIMITATIONS AND FUTURE WORK

  • The encoder’s fixed-interval 1D convolutional embeddings constrain CTIA-ODE to regularly sampled histories; irregular observations with missing values remain a challenge.The authors say addressing this would require replacing the fixed-step encoder with continuous-time variants such as ODE-RNNs or neural flow models.Source: L LIMITATIONS AND FUTURE WORK · p. 23
  • CTIA-ODE is trained from scratch on individual datasets and currently lacks the broad-scale transferability of emerging foundation models.The authors propose pretraining its continuous-time modules on diverse, large-scale time-series corpora and fine-tuning for unseen tasks.Source: L LIMITATIONS AND FUTURE WORK · p. 23
  • RK4 may not capture the full complexity of highly chaotic or turbulent systems, such as high-frequency financial ticks, where adaptive-step solvers may be needed.Adaptive solver conditions can complicate training speed and loss stability; the authors propose exploring dynamic ORHL thresholding.Source: L LIMITATIONS AND FUTURE WORK · pp. 23–24

Related papers

Loading 2610.19831v2…