PaperlayerView exact revision on arXiv

Source-linked AI summary

CoRA: Boosting Time Series Foundation Models for Multivariate Forecasting through Correlation-aware Adapter

Hanyin Cheng, Xingjian Wu, Yang Shu, Zhongwen Rao, Lujia Pan, Bin Yang, Chenjuan Guo

arXiv:2603.21828v1cs.LGcs.AI

TL;DR

Many TSFMs neglect or incompletely model channel correlations, which matter for multivariate forecasting. CoRA is a lightweight adapter that models multiple correlation types during fine-tuning; experiments report improved forecasting performance on real-world datasets.

  • Problem

    Many TSFMs focus on temporal dependencies and neglect channel correlations or fail to consider their different aspects, although such correlations matter for multivariate forecasting.

    Source: §1 · pp. 1–2

  • Method

    CoRA is a lightweight fine-tuning adapter that combines low-rank Time-Varying and Time-Invariant correlation components with contrastive learning for heterogeneous and partial correlations.

    Source: §1 · p. 3, §4.1 · p. 5

  • Results

    CoRA improves TSFMs' multivariate forecasting performance in experiments on 10 real-world datasets.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    CoRA captures dynamic, heterogeneous, and partial correlations with linear complexity in the number of channels during inference.

    Source: §1 · p. 3

    The polynomial approximation result is stated for time series that are locally stationary.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Most existing Time Series Foundation Models (TSFMs) use channel independent modeling and focus on capturing and generalizing temporal dependencies, while neglecting the correlations among channels or overlooking the different aspects of correlations. However, these correlations play a vital role in Multivariate time series forecasting. To address this, we propose a CoRrelation-aware Adapter (CoRA), a lightweight plug-and-play method that requires only fine-tuning with TSFMs and is able to capture different types of correlations, so as to improve forecast performance. Specifically, to reduce complexity, we innovatively decompose the correlation matrix into low-rank Time-Varying and Time-Invariant components. For the Time-Varying component, we further design learnable polynomials to learn dynamic correlations by capturing trends or periodic patterns. To learn positive and negative correlations that appear only among some channels, we introduce a novel dual contrastive learning method that identifies correlations through projection layers, regulated by a Heterogeneous-Partial contrastive loss during training, without introducing additional complexity in the inference stage. Extensive experiments on 10 real-world datasets demonstrate that CoRA can improve TSFMs in multivariate forecasting performance.

1 INTRODUCTION

TSFMs often neglect channel relationships or model them incompletely, although multivariate forecasting can benefit from correlations that are dynamic, heterogeneous, and partial. CoRA addresses this with a lightweight fine-tuning plugin, and experiments on 10 real-world datasets show improved multivariate forecasting performance.

1 INTRODUCTION

  • Channel correlations can improve multivariate forecasting and include dynamic, heterogeneous, and partial relationships.Dynamic correlations vary over time; heterogeneous correlations distinguish positive and negative interactions; partial correlations occur only among some channels.Source: §1 · p. 1
  • Existing TSFMs and correlation-aware models do not comprehensively model these relationships: some miss changing correlations over time, while others interact across all channels without explicitly handling heterogeneous or partial correlations.The introduction notes these shortcomings for TTM, UniTS, and Moirai.Source: §1 · p. 2
  • The paper identifies a need for a lightweight plugin for downstream TSFM fine-tuning that captures multiple correlation types without the complexity or training requirements of existing approaches.It contrasts this need with methods that can have O(N^2) time complexity, require model redesign, or require additional pre-training or training from scratch.Source: §1 · p. 2
  • CoRA combines Dynamic Correlation Estimation with Heterogeneous-Partial Correlation Contrastive Learning to model dynamic, heterogeneous, and partial correlations.DCE uses low-rank Time-Varying and Time-Invariant components and learnable polynomials; HPCL uses projection layers and a contrastive loss to learn heterogeneous and partial correlations.Source: §1 · pp. 2–3
  • Experiments on 10 real-world datasets show that CoRA improves TSFMs' multivariate forecasting performance.The plugin is designed to capture the three correlation types without re-pre-training the TSFMs.Source: §1 · p. 3

2 RELATED WORK

The related work reviews TSFMs and channel-correlation methods, noting that complex inter-channel relationships remain under-explored and existing approaches may lack comprehensive modeling or carry substantial complexity.

2.1 FOUNDATION MODELS FOR TIME SERIES FORECASTING

  • Most TSFMs use channel-independent strategies, while only a few model relatively simple inter-channel relationships; the effects of more complex correlations in TSFMs remain under-explored.The section distinguishes LLM-based models from time-series pre-trained models.Source: §2.1 · p. 3

2.2 CORRELATION OF CHANNELS IN TIME SERIES FORECASTING

  • Channel-correlation models use MLPs, GNNs, or Transformers, while plugins such as LIFT and CCM learn correlations through local relationships or clustering and dedicated prediction heads.The section describes channel mixing, message passing, and treating channels as tokens as examples of model approaches.Source: §2.2 · p. 3
  • The reviewed correlation methods either lack comprehensive correlation modeling capabilities or have substantial complexity.Source: §2.2 · p. 3

3 PRELIMINARIES

The preliminaries frame forecasting as predicting future steps from multichannel time-series data and describe adapting a pre-trained TSFM during downstream fine-tuning.

3 PRELIMINARIES

  • The forecasting setup uses a multivariate time series with N channels and a look-back window L to predict future F steps.Source: §3 · p. 3
  • A pre-trained TSFM F processes downstream input data during fine-tuning, with CoRA augmenting it to form an adapted model F*.Source: §3 · p. 4
  • During fine-tuning, the TSFM generates initial predictions and extracts an intermediate series representation; at inference, the adapted model predicts on unseen test data.Source: §3 · p. 4

4 METHODOLOGY

CoRA is a lightweight plug-in that combines dynamic correlation estimation, positive/negative representation spaces, contrastive learning, and fusion to enhance TSFM predictions. Its correlation modeling separates changing and stable components, while training-only contrastive operations keep inference linear in the number of channels.

4 METHODOLOGY

  • CoRA uses TSFM inputs, predictions, and representations in four stages—dynamic correlation estimation, heterogeneous division, H-PCorr contrastive learning, and fusion—to generate enhanced predictions.The original and new predictions are gated and added together.Source: §4 · p. 4

4.1 DYNAMIC CORRELATION ESTIMATION

  • DCE decomposes learnable correlations into low-rank time-varying Qt and time-invariant V components, alongside rule-based correlation R, for parameter-efficient estimation.The decomposition is described as functionally equivalent to conventional additive decomposition.Source: §4.1 · pp. 4–5

4.1.1 LEARNABLE TIME-AWARE POLYNOMIALS

  • Learnable time-aware polynomials estimate Qt by combining a shared learnable basis with coefficients that vary over time.A mapping from TSFM representations estimates the coefficients rather than the entire time-varying component.Source: §4.1.1 · p. 5

4.1.2 TIME-VARYING AND TIME-INVARIANT COMPOSITION

  • Global learnable vectors capture stable channel dependencies in V, while Pearson correlation initializes the final estimate, which combines rule-based and learnable correlations.The learnable components include both time-varying Qt and time-invariant V.Source: §4.1.2 · pp. 5–6

4.2 HETEROGENEOUS CORRELATION DIVISION

  • The channel-aware projector uses contextual information to adjust channel weights and project representations into positive and negative correlation spaces.The division module does not itself disentangle heterogeneous correlations; that separation is guided by the next contrastive-learning stage.Source: §4.2 · p. 6

4.3 HETEROGENEOUS PARTIAL CORRELATION CONTRASTIVE LEARNING

  • H-PCorr contrastive learning uses DCE correlations and divided representations to adaptively learn partial correlations within each heterogeneous space.It supports fine-grained channel interactions without adding inference burden.Source: §4.3 · p. 6

4.4 HETEROGENEOUS FUSION AND PREDICTION

  • Fusion projects and combines the positive- and negative-space representations for prediction, with a learned gated weight for each channel.The fusion uses a convex combination because channels may require different degrees of interaction or independence.Source: §4.4 · p. 7

4.5 COMPLEXITY ANALYSIS

  • DCE and H-PCorr contrastive learning have O(N^2) complexity during training, while inference uses only heterogeneous division at O(N).The division module has O(N) complexity.Source: §4.5 · p. 7

5 THEORETICAL ANALYSIS

The analysis argues that the time-varying/time-invariant composition preserves conventional additive-decomposition behavior with lower complexity, while polynomial fitting approximates dynamic correlations with an error bound that shrinks as degree rises.

5.1 THE SIGNIFICANCE OF TIME-VARYING AND TIME-INVARIANT COMPOSITION

  • The time-varying/time-invariant decomposition retains conventional additive-decomposition behavior while reducing parameter complexity.Source: §5.1 · pp. 7–8

5.2 THE FITTING ABILITY OF TIME-AWARE POLYNOMIALS

  • For locally stationary series with a smooth correlation function of basis q, a high-order polynomial can approximate the underlying correlation, with error bound decreasing as degree K increases.Source: §5.2 · p. 8
  • Selecting K balances model effectiveness against computational efficiency for dynamic-correlation estimation.Source: §5.2 · p. 8

6 EXPERIMENT

Experiments compare CoRA-equipped TSFMs with baselines, other correlation plugins, and ablations, and assess forecasting performance and efficiency. CoRA improves fine-tuned TSFM results, while adding little time or parameter overhead, including as channel counts rise.

6.1 EXPERIMENTAL DETAILS

  • The evaluation covers ten forecasting benchmarks across multiple domains and compares TSFMs using MSE and MAE.Baselines include three time-series LLM-based models and three pretrained time-series models.Source: §6.1 · p. 8

6.2 MAIN RESULTS

  • CoRA improves average and horizon-specific results over fine-tuning without it for both LLM-based and pretrained TSFMs, including in 5% few-shot settings.The comparison also reports better performance for a CoRA-tuned channel-independent TTM than for a channel-dependent TTM fine-tuned without CoRA.Source: §6.2 · p. 8, Table 1 · p. 9

6.3 COMPARISON WITH OTHER CORRELATION PLUGINS

  • With GPT4TS, UniTime, and Timer at H=96, Figure 4 compares CoRA with LIFT and C-LoRA in 5% few-shot fine-tuning; limited training samples degrade LIFT and C-LoRA, while CoRA is TSFM-specific.CoRA learns multiple correlations from TSFM representations, allowing it to leverage their predictive capabilities.Source: §6.3 · p. 8, Figure 4 · p. 9

6.4 ABLATION STUDIES

  • The ablation reports the best performance when all three modules are combined; adding either DCE or HD to HPCL further improves performance.The comparison uses naive replacements for DCE and HD: series-level Pearson correlation and a single-branch projection, respectively.Source: §6.4 · p. 9, Table 2 · p. 9

6.5 MODEL ANALYSIS

  • CoRA adds no significant time or parameter overhead, and its efficiency shows no noticeable degradation versus the backbone as channel count rises from 7 to 321.The analysis uses ETTm2 (N=7), Weather (N=21), and Electricity (N=321), with look-back window and horizon both set to 96; inference is especially efficient.Source: §6.5 · p. 10, Figure 5 · p. 10
  • The experiments also report MSE results for different data percentages and sensitivity analyses for polynomial degree, decomposition size, and projection-layer counts.The sensitivity analyses are referenced in the appendix; no outcomes are given in these passages.Source: Table 3 · p. 10, §6.5 · p. 10

7 CONCLUSION

The conclusion presents CoRA as a lightweight TSFM adapter for modeling three types of channel correlations. It combines a dynamic-correlation module with contrastive learning and reports improved forecasting performance on real-world datasets.

  • CoRA is a lightweight adapter intended to improve TSFM predictions by considering three types of correlation relationships.Source: §7 · p. 10
  • Time-varying and time-invariant composition with learnable time-aware polynomials models dynamic correlations with regular patterns.Source: §7 · p. 10
  • Heterogeneous-partial contrastive learning decouples heterogeneous correlation while adaptively learning partial correlations among channels.Source: §7 · p. 10
  • Experiments on real-world datasets show that CoRA can improve TSFM forecasting performance.Source: §7 · p. 10

ETHICS STATEMENT

The work uses public benchmark datasets without personally identifiable information and involved no human subjects. The adapter is designed for beneficial applications in system reliability and safety monitoring.

REPRODUCIBILITY STATEMENT

The reproducibility statement says the results can be reproduced and that model code is released. The accompanying definitions characterize dynamic, heterogeneous, and partial correlations in multivariate series.

REPRODUCIBILITY STATEMENT

A DEFINITIONS OF THE THREE CORRELATIONS

B.1 TRAINING PHASE

CoRA estimates channel correlations with a time-varying and time-invariant decomposition, using time-aware polynomials to fit changing correlations. Its training and inference complexity are analyzed separately.

B.1 TRAINING PHASE

  • The training phase includes Dynamic Correlation Estimation, Heterogeneous Division, and H-PCorr contrastive learning, with the stated complexity of each module.The listed time complexities are O(PKNM + PNd^2), O(lN^2 + NM^2 + MN^2), O(NP^2d^2), and O(PdN^2) for the respective operations described.Source: §B.1 · p. 17
  • Training complexity is O(N^2) with O(N) space when focusing on channel count; inference complexity is O(N) with O(1) space.The training analysis gives total complexity O(dN^2 + Nd^2) and space O(d + N); inference retains only projectors in the Heterogeneous Division and Fusion modules.Source: §B.1 · p. 17, §B.2 · p. 17

C.1 THE SIGNIFICANCE OF TIME-VARYING AND TIME-INVARIANT COMPOSITION

  • CoRA decomposes learnable correlations into time-invariant and time-varying components, which contain stable and changing information, respectively.The decomposition is motivated by locally stationary time series and is described as analogous to conventional additive decomposition.Source: §C.1 · p. 18

C.2 THE FITTING ABILITY OF TIME-AWARE POLYNOMIALS

  • Time-aware polynomials fit dynamic correlations because trends and seasonality make their changes regular; under local stationarity and smoothness, higher polynomial degree reduces fitting error.The paper states that the underlying correlation matrix can be approximated with a high-order polynomial.Source: §C.2 · p. 18

D.1 DATASETS

The experiments use ten forecasting benchmarks, compare selected pretrained and LLM-based forecasting models, and follow a shared evaluation and optimization setup.

D.1 DATASETS

  • Ten forecasting benchmarks include ETT, Electricity, Traffic, Solar, Weather, AQShunyi, and ZafNoo datasets.ETT has four subsets; the datasets cover settings such as hourly electricity consumption, freeway sensors, solar generation, and meteorological indicators.Source: §D.1 · p. 20, Table 4 · p. 20

D.2 BASELINES

  • The selected baselines include pretrained Timer, TTM, and Moment, alongside LLM-based CALF, GPT4TS, and UniTime.The paper refers readers to Table 5 for model descriptions.Source: §D.2 · p. 20, Table 5 · p. 20

D.3 IMPLEMENTATION DETAILS

  • Experiments use TSFM-Bench for unified evaluation, omit the “Drop Last” trick for fair comparison, and train CoRA with MSE loss and ADAM using PyTorch on an NVIDIA Tesla-A800 GPU.Experiments use Python 3.10.Source: §D.3 · p. 21

E FULL RESULTS

CoRA is compared with other adapters across few-shot sampling rates, and the paper also reports hyperparameter sensitivity and heterogeneous-space visualizations.

E FULL RESULTS

  • Tables 6 and 7 report MSE and MAE for LLM-based and pretrained models, respectively, across forecasting horizons 96, 192, 336, and 720.The captions say better results are highlighted in bold but do not provide cell values here.Source: Table 6 · p. 22, Table 7 · p. 23

F.1 COMPARISON IN DIFFERENT SAMPLING RATE

  • CoRA consistently outperforms LIFT and C-LoRA with TTM at H=96 across few-shot sampling rates, especially at lower rates.The comparison uses sampling rates of 5%, 10%, 15%, 20%, and 25%; the authors attribute CoRA’s advantage to comprehensive correlation modeling and efficient TSFM use.Source: §F.1 · p. 24, Table 8 · p. 24

F.2 HYPERPARAMETER SENSITIVITY

  • With TTM at H=96, polynomial degree K is robust; decomposition size M need not grow rapidly with channel count, and projector depth balances fitting against generalization.The authors commonly choose K values of 3 or 4 and projector layer counts of 3 or 5.Source: §F.2 · p. 24, Figure 6 · p. 24

F.3 VISUALIZATION OF HETEROGENEOUS SPACES.

  • Weather visualizations show positive, negative, and independent channel relationships changing across three consecutive 64-point intervals.Across the intervals, the reported positive and negative channel pairs differ; the authors conclude the visualization captures DCorr, HCorr, and PCorr.Source: §F.3 · pp. 24–25, Figure 7 · p. 25
Loading 2603.21828v1…