PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaRNN: Adaptive Learning and Forecasting of Time Series

Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, Chongjun Wang

arXiv:2108.04443v2cs.LGcs.AI

TL;DR

Non-stationary time series create Temporal Covariate Shift because distributions change over time, while modeling this shift remains underexplored. AdaRNN characterizes temporal distributions and matches them with an adaptive RNN model. Across classification and regression tasks, it reports 2.6% higher accuracy and 9.0% lower RMSE than state-of-the-art baselines.

  • Problem

    Temporal Covariate Shift arises when time-series distributions change across periods, creating a distribution-shift problem that existing methods do not explicitly model.

  • Method

    AdaRNN sequentially applies Temporal Distribution Characterization to identify diverse periods and Temporal Distribution Matching to reduce their divergence while retaining temporal dependencies.

  • Results

    2.6% higher classification accuracy and 9.0% lower regression RMSE than state-of-the-art baselines were reported across multiple time-series tasks.

  • Takeaways & Limitations

    AdaRNN provides a general adaptive framework compatible with multiple RNN structures, distribution distances, and Transformer extension.

  • Takeaways & Limitations

    A naïve pair-specific importance network is impractical because early hidden states are weakly meaningful and learning separate weights for every domain pair is expensive.

Abstract

from arXiv · show

Time series has wide applications in the real world and is known to be difficult to forecast. Since its statistical properties change over time, its distribution also changes temporally, which will cause severe distribution shift problem to existing methods. However, it remains unexplored to model the time series in the distribution perspective. In this paper, we term this as Temporal Covariate Shift (TCS). This paper proposes Adaptive RNNs (AdaRNN) to tackle the TCS problem by building an adaptive model that generalizes well on the unseen test data. AdaRNN is sequentially composed of two novel algorithms. First, we propose Temporal Distribution Characterization to better characterize the distribution information in the TS. Second, we propose Temporal Distribution Matching to reduce the distribution mismatch in TS to learn the adaptive TS model. AdaRNN is a general framework with flexible distribution distances integrated. Experiments on human activity recognition, air quality prediction, and financial analysis show that AdaRNN outperforms the latest methods by a classification accuracy of 2.6% and significantly reduces the RMSE by 9.0%. We also show that the temporal distribution matching algorithm can be extended in Transformer structure to boost its performance.

1 INTRODUCTION

Time-series distributions change over time, creating Temporal Covariate Shift that challenges existing methods. AdaRNN addresses this by characterizing temporal distributions and matching them to learn adaptive models.

  • Motivation: Time-series statistical properties and distributions change over time, making forecasting difficult across real-world applications.Applications include financial, medical, weather, renewable-energy, and air-quality analysis.
  • Problem: Temporal Covariate Shift occurs when marginal distributions vary across periods while conditional distributions remain unchanged, violating existing methods’ IID assumption.The resulting distribution shift can impair performance and generalization on unseen test data.
  • Problem: The paper identifies two challenges: characterizing distributions across varied periods and matching them while preserving temporal dependencies.These challenges motivate a distribution-based perspective for non-stationary time series.
  • Approach: AdaRNN combines Temporal Distribution Characterization, which identifies diverse periods, with Temporal Distribution Matching, which reduces cross-period distribution divergence using an RNN.The framework learns an accurate and adaptive prediction model for TCS.
  • Results: 2.6% accuracy and 9.0% RMSE improvements over state-of-the-art baselines were reported for classification and regression tasks, respectively.Experiments covered human activity recognition, air quality prediction, household power consumption, and stock price prediction.

2 RELATED WORK

Prior time-series methods use distances, engineered features, ensembles, segmentation, clustering, or domain adaptation, but each has limitations for changing and complex distributions.

  • Traditional methods: Distance-based methods compare raw-series segments using metrics such as Euclidean distance or Dynamic Time Warping.They measure similarity between segments rather than explicitly matching temporal distributions.
  • Traditional methods: Feature-based methods capture global or local patterns through manually extracted or learned features.Their reliance on feature construction can limit scalability and flexibility.
  • Traditional methods: Ensemble methods combine multiple weak classifiers to improve predictive performance.This approach is distinct from distribution-based adaptation.
  • Segmentation and clustering: Time-series segmentation separates series into pieces for pattern discovery, whereas clustering groups similar series segments.Segmentation methods generally do not use distribution matching and therefore cannot directly adapt to this problem.
  • Distribution adaptation: Domain adaptation reduces train–test distribution divergence through instance re-weighting or feature transfer, while domain generalization learns invariant models across multiple domains.These approaches motivate related distribution-shift solutions for time-series prediction.

3 PROBLEM FORMULATION

The paper formulates prediction under Temporal Covariate Shift, where distributions change across time periods but conditional relationships remain stable. The model must discover those periods and exploit their shared knowledge for future prediction.

  • Prediction task: The prediction task maps labeled time-series segments x_i to labels y_i for future r-step-ahead segments.When r = 1, the task is one-step prediction.
  • Temporal Covariate Shift: Existing approaches assume all segments share one distribution, but the formulation allows both training and test distributions to differ over time.This setting is intended to represent more realistic non-stationary applications.
  • Temporal Covariate Shift: Temporal Covariate Shift partitions a time series into K periods whose marginal distributions differ while their conditional label distributions remain equal.Segments within each period share the same joint distribution.
  • Objective: The goal is to automatically discover the unknown periods and learn a prediction model that exploits commonality across them for precise future predictions.Shared economic laws and patterns illustrate the common knowledge that may persist despite changing financial factors.
  • Challenges and assumptions: Period discovery is difficult because both the number of periods and their boundaries are unknown, creating a large search space that can affect final generalization.The formulation assumes future test segments belong to one period whose marginal distribution differs from training periods while the conditional distribution remains the same.

4 OUR PROPOSED MODEL: ADARNN

AdaRNN addresses temporal covariate shift by first identifying diverse time-series periods, then matching their distributions while preserving temporal dependencies. Its training combines prediction loss with distribution matching and learns hidden-state importance through boosting-based updates.

  • Framework: AdaRNN combines Temporal Distribution Characterization (TDC) and Temporal Distribution Matching (TDM) to build a generalized time-series prediction model.TDC identifies periods that characterize distribution information, while TDM matches their distributions.
  • Temporal Distribution Characterization: TDC searches for periods with maximally diverse distributions, motivated by maximum entropy and the goal of handling worst-case temporal covariate shift.The optimization maximizes averaged pairwise distribution distances while constraining period sizes and the number of periods.
  • Temporal Distribution Characterization: Because the TDC optimization is generally intractable, AdaRNN uses greedy splitting over evenly divided time-series parts and searches candidate values of K.For each K, splitting points are selected by maximizing distribution distance between the resulting segments.
  • Temporal Distribution Matching: TDM matches distributions across learned periods using RNN hidden states, combining prediction loss with a regularization term based on pairwise period distances.The prediction loss uses labeled segments and a task loss such as MSE; λ controls the trade-off between prediction and matching.
  • Temporal Distribution Matching: To capture temporal dependencies, TDM assigns normalized importance weights to hidden states and adaptively matches RNN-cell distributions between period pairs.The importance vector weights the hidden states inside the RNN and dynamically reduces cross-period distribution divergence.
  • Boosting-Based Importance Evaluation: AdaRNN pre-trains network parameters before learning hidden-state importance, then updates importance through a boosting procedure based on changes in cross-domain distribution distance.Weights increase when the distribution distance worsens between epochs and otherwise remain unchanged.

5 EXPERIMENTS

Experiments evaluate AdaRNN across time-series classification and prediction settings using shared splits, tuned hyperparameters, and multiple baselines. On the UCI activity task, AdaRNN achieves the strongest reported classification performance.

  • 5.1 Setup: AdaRNN is evaluated on four real-world datasets, using GRUs as the primary recurrent cells.The datasets cover human activity recognition, air quality prediction, household power consumption, and stock price prediction.
  • 5.1 Setup: The comparison includes traditional time-series models, recent time-series models, Transformer, and adapted domain adaptation or generalization methods.The adapted methods are denoted MMD-RNN and DANN-RNN.
  • 5.1 Setup: All methods use the same train-validation-test split and periods after TDC, with final results averaged over five random runs.Hyperparameters are tuned on the validation set; the main results use MMD as the distribution distance.
  • 5.2 Human activity recognition: The UCI activity dataset contains smartphone sensor recordings from thirty volunteers performing six activities, with 7,352 training and 2,947 test instances.The training and testing sets have different distributions, so temporal covariate shift is present.
  • 5.2 Human activity recognition: AdaRNN achieves the best performance on all UCI activity recognition metrics, outperforming DANN by 2.56% in accuracy and 3.07% in F1 score.The reported metrics include accuracy, precision, recall, F1, and area under curve.

5.4 Household electric power consumption

The experiments report AdaRNN results for household power consumption alongside air quality and stock prediction comparisons, then examine how its distribution-characterization and matching components affect performance. AdaRNN achieves the best reported household power-consumption result and benefits from distribution-aware design choices.

  • 5.4 Household electric power consumption: The household power dataset contains 2,075,260 one-minute measurements collected between 2006 and 2010.After removing null data, 2,062,346 measurements remain; the train-validation-test ratio is 6:2:2.
  • 5.4 Household electric power consumption: AdaRNN achieves the best household power-consumption RMSE among the compared methods.The result is reported in the right part of Table 3.
  • 5.5 Stock price prediction: AdaRNN outperforms other comparison methods on all stock-price prediction metrics and achieves the best IC values.Its RankIC is also higher than that of the other methods.
  • 5.6.1 Temporal distribution characterization: The best temporal-distribution-characterization result uses K=3 or K=5 for the two air-quality stations, while larger K values reduce performance.The results indicate that K matters for characterizing distributions in non-stationary time series.
  • 5.6.1 Temporal distribution characterization: TDC achieves the best RMSE when periods are split with the largest distribution distance rather than randomly or with similar distributions.This pattern also holds for other tested K values.
  • 5.6.2 Temporal distribution matching: Pre-training improves both naïve and boosting-based TDM variants, while pre-training with boosting achieves the best performance.All four TDM variants reportedly outperform STRIPE in the comparison described.

5.7 Further analysis

Further analyses show that AdaRNN remains effective across prediction horizons, distribution distances, and evaluation settings, while retaining practical training and inference costs.

  • Distribution-distance robustness: AdaRNN is robust to four distribution distances—MMD, CORAL, Cosine, and domain-adversarial distance—while achieving the best performance.The evaluation covers three air-quality stations.
  • Multi-step prediction: AdaRNN achieves the best performance for multi-step air quality prediction across 6, 12, 18, and 24-hour horizons.Performance worsens for all methods as prediction steps increase.
  • Efficiency: AdaRNN converges quickly during training, although its autoregressive RNN structure requires more training time than traditional RNNs.Its training time remains comparable to GRU and STRIPE, while inference time matches existing RNN-based models.
  • Prediction quality: AdaRNN produces fewer prediction mistakes than DANN and MMD in the Dongsi station analysis.The comparison includes prediction-versus-ground-truth plots and absolute prediction errors.

6 EXTENSIONS

The paper extends adaptive importance weighting beyond RNNs by incorporating it into Transformer self-attention representations, forming AdaTransformer.

  • Transformer extension: AdaTransformer adds adaptive importance weights α to the hidden representations of each Transformer self-attention block.The extension is motivated by the claimed greater power of Transformer models compared with RNN-based models.
  • Future direction: The authors identify deeper Transformer extensions as a direction for achieving better performance in future work.The paper states that further improvements may be possible.
  • Transformer extension: Table 5 reports results comparing vanilla and adaptive Transformer models.The pilot study is presented as evidence for extending AdaRNN’s adaptive mechanism to Transformers.

7 CONCLUSIONS AND FUTURE WORK

The paper frames non-stationary time-series forecasting as a temporal covariate shift problem and proposes AdaRNN to improve generalization through temporal distribution modeling.

  • Conclusions: The paper studies temporal covariate shift in non-stationary time series as a realistic and challenging forecasting problem.The problem arises from temporally changing distribution information in time series.
  • Conclusions: AdaRNN combines temporal distribution characterization with temporal distribution matching to learn an adaptive RNN model.The first stage characterizes time-series distributions, while the second builds the generalized model through distribution matching.
  • Future work: The authors plan to extend AdaRNN more deeply to Transformers and integrate its two stages into an end-to-end network.These directions target better performance and easier training, respectively.

A DISTRIBUTION DISTANCE

The paper evaluates several distribution distances for temporal distribution matching, spanning similarity, mean-embedding, covariance, and adversarial neural-network formulations.

  • Distance choices: AdaRNN uses cosine, MMD, CORAL, and domain-adversarial discrepancy as alternative distribution distances.These four distances are adopted from prior domain adaptation and domain generalization work.
  • Cosine distance: Cosine distance measures similarity between two distributions using their normalized inner product.Its formulation is d_cosine(h_s, h_t) = 1 − ⟨h_s, h_t⟩ / (||h_s|| · ||h_t||).
  • MMD: MMD is a non-parametric distance between distributions based on differences between sample mean embeddings in an RKHS.Its empirical estimate uses a kernel function and the sample counts n_s and n_t.
  • CORAL: CORAL measures distribution distance through differences between second-order statistics, represented by covariance matrices.The feature dimension is q, with C_s and C_t denoting the two covariance matrices.
  • Domain-adversarial discrepancy: Domain-adversarial discrepancy parameterizes domain discrepancy with a neural network and introduces a domain discriminator D.The associated objective is defined through the adversarial domain-discrimination formulation.
  • Distance comparison: Table 6 reports results for different distribution distances used in temporal distribution characterization.The table provides the corresponding distance-comparison evaluation.

B DIFFERENT DISTANCES IN TDC

The TDC algorithm is evaluated with multiple distribution distances against random and opposite domain splits. It generally performs best when producing the largest distribution distance and remains effective across distance functions.

  • TDC is compared with a random domain split and an opposite split using K = 2.The opposite split uses the smaller distribution distance.
  • TDC achieves the best RMSE and MAE with the largest distribution distance under most distance functions.
  • With cosine distance, TDC obtains the second-largest distribution distance but still achieves the best RMSE and MAE.
  • The results indicate that TDC effectively characterizes distribution information and is agnostic to the distance function.

C STOCK PRICE PREDICTION: METRICS

The stock-price evaluation uses correlation-based metrics for predicted and actual returns, including rank-based variants and information-ratio measures. Higher values indicate better model performance.

  • Information coefficient (IC) measures the correlation between predicted and actual stock returns and ranges from −1.0 to 1.0.An IC of +1.0 indicates a perfect linear relationship, while 0.0 indicates no linear relationship.
  • RankIC measures the correlation between factor ranks from period t−1 and stock-return ranks from period t.
  • The information ratio (IR), also called ICIR, is the portfolio’s mean excess return divided by tracking error.
  • RankIC, RankICIR, RawIC, and RawICIR extend the evaluation to robustness against permutations and performance on unnormalized data.Higher values indicate better model performance.
Loading 2108.04443v2…