PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Replay-based Continuous Learning Framework for Spatio-Temporal Prediction on Streaming Data

Hao Miao, Yan Zhao, Chenjuan Guo, Bin Yang, Kai Zheng, Feiteng Huang, Jiandong Xie, Christian S. Jensen

arXiv:2404.14999v1cs.DBcs.LG

TL;DR

Streaming spatio-temporal prediction must learn from arriving data while preserving previously learned knowledge. URCL combines replay-based mixup with an autoencoder and STSimSiam representation learning; real-data experiments report improved prediction accuracy over the state of the art.

  • Problem

    Streaming spatio-temporal prediction must learn from arriving data while preserving knowledge acquired from earlier sequences.

    Source: §I · p. 2, §III · p. 3

  • Method

    URCL mixes current observations with replay-buffer samples and couples a spatio-temporal prediction autoencoder with STSimSiam, mutual information maximization, and five augmentations.

    Source: §I · p. 2, A. Framework Overview · p. 4

  • Results

    Real-data experiments offer evidence that URCL improves on the state of the art in prediction accuracy.

    Source: §VI · p. 12

  • Takeaways & Limitations

    Removing STMixup, RMIR sampling, data augmentation, or GraphCL increases MAE and RMSE on both base and incremental sets.

    Source: 3) Ablation Study · p. 11

Abstract

from arXiv · show

The widespread deployment of wireless and mobile devices results in a proliferation of spatio-temporal data that is used in applications, e.g., traffic prediction, human mobility mining, and air quality prediction, where spatio-temporal prediction is often essential to enable safety, predictability, or reliability. Many recent proposals that target deep learning for spatio-temporal prediction suffer from so-called catastrophic forgetting, where previously learned knowledge is entirely forgotten when new data arrives. Such proposals may experience deteriorating prediction performance when applied in settings where data streams into the system. To enable spatio-temporal prediction on streaming data, we propose a unified replay-based continuous learning framework. The framework includes a replay buffer of previously learned samples that are fused with training data using a spatio-temporal mixup mechanism in order to preserve historical knowledge effectively, thus avoiding catastrophic forgetting. To enable holistic representation preservation, the framework also integrates a general spatio-temporal autoencoder with a carefully designed spatio-temporal simple siamese (STSimSiam) network that aims to ensure prediction accuracy and avoid holistic feature loss by means of mutual information maximization. The framework further encompasses five spatio-temporal data augmentation methods to enhance the performance of STSimSiam. Extensive experiments on real data offer insight into the effectiveness of the proposed framework.

I. INTRODUCTION

Streaming spatio-temporal prediction must adapt to changing data while retaining historical knowledge and spatial-temporal patterns. URCL addresses this with replay-based learning, a spatio-temporal prediction autoencoder, and STSimSiam representation learning.

I. INTRODUCTION

  • Static prediction models assume datasets share a distribution, but concept drift can cause performance degradation, motivating continuous learning for streaming data.Existing continuous-learning techniques for vision and language cannot be directly applied because spatio-temporal prediction must capture spatial dependencies and temporal correlations.Source: §I · pp. 1–2
  • URCL fuses current observations with replay-buffer samples through STMixup to alleviate catastrophic forgetting in streaming prediction.The buffer stores a subset of previously learned observations; a ranking-based retrieval strategy selects representative samples.Source: §I · p. 2
  • URCL pairs an STEncoder and STDecoder in a spatio-temporal prediction network to capture complex correlations across diverse data and applications.The introduction motivates this architecture as a way to avoid converting separate customized static models into continuous versions.Source: §I · p. 2
  • STSimSiam uses mutual information maximization to preserve holistic features, alongside five augmentation methods tailored to spatio-temporal data.Its encoders are shared with the prediction network.Source: §I · p. 2

A. Spatio-Temporal Data Prediction

  • Prior spatio-temporal methods include statistical, CNN, and graph-neural-network approaches, but existing methods do not support streaming and suffer catastrophic forgetting.Statistical models have limited capacity for complex spatial and temporal correlations.Source: A. Spatio-Temporal Data Prediction · pp. 2–3

B. Continuous Learning

Continuous learning incrementally learns a sequence of tasks while transferring knowledge without catastrophic forgetting. This paper frames streaming spatio-temporal prediction as learning from successive data sequences while preserving prior knowledge.

B. Continuous Learning

  • Continuous learning incrementally learns a sequence of tasks with knowledge transfer and without catastrophic forgetting, extending acquired knowledge from a data stream for future learning.It is also called lifelong or incremental learning.Source: B. Continuous Learning · p. 3
  • Replay-based, regularization-based, and architecture-based methods are common deep continuous-learning approaches, but most are designed for vision or language rather than spatio-temporal prediction.Prior streaming traffic studies use node-incremental settings, whereas this work keeps the sensor count fixed and varies instances over time.Source: B. Continuous Learning · p. 3

III. PROBLEM STATEMENT

  • The SSTP problem learns to predict N future observations from a current observation and M−1 preceding observations while preserving knowledge from earlier sequences.The data stream comprises sequences from successive time periods over a sensor network.Source: §III · p. 3

IV. METHODOLOGY

URCL organizes streaming spatio-temporal prediction into data integration, continuous representation learning, and prediction. It combines current and replay-buffer observations, learns holistic representations with STCRL, and uses an encoder-decoder prediction stage.

A. Framework Overview

  • URCL comprises data integration, spatio-temporal continuous representation learning (STCRL), and spatio-temporal prediction modules.The framework overview identifies these as its three major modules.Source: A. Framework Overview · p. 3, §IV · p. 3

B. Data Integration

  • RMIR selects historical observations from a replay buffer, and STMixup integrates them with current-stream observations to accumulate knowledge and alleviate catastrophic forgetting.The buffer stores a subset of previously learned observations.Source: A. Framework Overview · p. 3, B. Data Integration · p. 4

A. Framework Overview

  • STCRL uses five spatio-temporal augmentations and a self-supervised STSimSiam network to learn holistic representations through mutual information maximization.Two randomly selected augmentations are encoded by parameter-sharing STEncoders followed by a projection MLP.Source: A. Framework Overview · p. 4
  • The prediction module encodes original data with an STEncoder shared with STCRL, stores learned features in replay memory, and feeds them to an STDecoder for prediction.Source: A. Framework Overview · p. 4

1) RMIR Sampling Method:

RMIR selects replay-buffer observations that are vulnerable to parameter updates and similar to current observations. STMixup interpolates current and historical observations, which are then used for continuous representation learning.

1) RMIR Sampling Method:

  • RMIR first retrieves observations expected to suffer the greatest loss increase after a parameter update, then selects replay samples similar to current observations by Pearson correlation.The initial candidate set is larger than the final selection; the similarity step is motivated by temporal patterns such as trend and periodicity.Source: 1) RMIR Sampling Method · p. 4

2) STMixup Mechanism:

  • STMixup interpolates current observations with replay-buffer observations, creating mixed training examples intended to revisit historical instances and reduce overfitting.The mechanism applies interpolation to observation-ground-truth pairs, following Vicinal Risk Minimization.Source: 2) STMixup Mechanism · pp. 4–5

C. Spatio-temporal Continuous Representation Learning

  • STCRL feeds STMixup observations into two randomly augmented views and an STSimSiam network, maximizing mutual information between their representations for holistic feature preservation.Source: C. Spatio-temporal Continuous… · p. 5

1) Spatio-Temporal Data Augmentation:

The augmentation scheme uses four spatially oriented methods and one temporally oriented method to perturb integrated observations. The methods alter nodes, edges, subgraphs, or temporal sequences while aiming to preserve useful semantics or correlations.

1) Spatio-Temporal Data Augmentation:

  • The five augmentations are DropNodes (DN), DropEdge (DE), SubGraph (SG), AddEdge (AE), and TimeShifting (TS), spanning spatial and temporal perturbations.The paper motivates these methods by the complex spatial and temporal correlations in spatio-temporal data.Source: 1) Spatio-Temporal Data Augmentation · p. 5
  • DN randomly discards nodes and masks their adjacency entries, with the aim of improving robustness to missing data such as sensor or communication failures.The discarded nodes are selected according to a specified distribution.Source: 1) Spatio-Temporal Data Augmentation · p. 5
  • DE samples edges and deletes those below a weight threshold, which is intended to retain important edge connections.Source: 1) Spatio-Temporal Data Augmentation · p. 5
  • SG uses random walks to sample a subgraph for preserving network semantics, while AE connects selected distant node pairs using their feature similarity as edge weights.AE targets global spatial correlations by connecting distant nodes with similar features.Source: 1) Spatio-Temporal Data Augmentation · pp. 5–6
  • TS randomly applies time slicing, time warping, or time flipping to transform observations in the temporal domain.Time slicing extracts a continuous subsequence, and time warping interpolates it back to the original sequence length.Source: 1) Spatio-Temporal Data Augmentation · p. 6

2) STSimSiam Network:

STSimSiam learns representations from two randomly augmented observations using two STEncoders and a projection MLP. Its similarity-based objective maximizes mutual information between the paired representations.

2) STSimSiam Network:

  • STSimSiam uses two STEncoders to encode augmented observations and a projection head to map one latent embedding into the other’s space.The encoders share parameters, and the described implementation uses GraphWaveNet as the spatio-temporal network.Source: 2) STSimSiam Network · p. 6
  • The network maximizes mutual information between representations of paired augmentations using a GraphCL loss based on cosine similarity.The loss is computed across augmented pairs in a minibatch and uses a symmetric similarity function.Source: 2) STSimSiam Network · p. 6
  • A stop-gradient operation blocks gradients from one branch to prevent a trivial solution in STSimSiam.Source: 2) STSimSiam Network · p. 6

D. Spatio-Temporal Prediction

The prediction network pairs an encoder that learns spatio-temporal representations from interpolated observations with a decoder that produces predictions. The encoder shares parameters with the STEncoder in STSimSiam, while newly learned data are stored in the replay buffer.

D. Spatio-Temporal Prediction

  • Interpolated observations Xmix pass through STEncoder to learn high-dimensional representations of complex spatio-temporal dependencies, then STDecoder maps them to predictions.The STEncoder shares parameters with the STEncoder in STSimSiam.Source: D. Spatio-Temporal Prediction · p. 7
  • Newly learned data are stored in the replay buffer during the prediction process.Source: D. Spatio-Temporal Prediction · p. 7
  • GraphWaveNet is reorganized into an STEncoder–STDecoder architecture, while stacked MLPs serve as the decoder for prediction networks lacking one.The framework is also studied with DCRNN and GeoMAN; those studies show adaptation to different prediction models.Source: D. Spatio-Temporal Prediction · p. 7

1) STEncoder:

The STEncoder combines temporal and spatial feature extraction: a gated temporal convolution network learns temporal correlations, and graph convolution captures spatial correlations. Its graph construction considers both local geographic connections and adaptive spatial relationships.

1) STEncoder:

  • An MLP maps Xmix into a high-dimensional latent space, followed by gated temporal convolution for temporal correlations and graph convolution for spatial correlations.Source: 1) STEncoder · p. 7
  • The encoder uses spectral graph convolution on a sensor network, with the adjacency matrix augmented by self-connections and normalized.The graph convolution applies a learnable weight matrix and activation function.Source: 1) STEncoder · p. 7
  • Diffusion graph convolution models graph signals through K finite steps, using transition-matrix power series and forward and backward diffusion for directed graphs.Source: 1) STEncoder · pp. 7–8
  • A local spatial graph connects geographically adjacent nodes, while a learnable self-adaptive adjacency matrix addresses global spatial correlations such as POI similarity.The final graph convolution combines local and global spatial correlations; if the sensor network is unknown, only the self-adaptive adjacency matrix is used.Source: 1) STEncoder · pp. 7–8
  • Dilated causal convolution is used for long-term temporal correlations and parallel computation, while the Gated TCN uses an output gate to model complex temporal correlations.The gate combines two parallel TCN layers; the described gated form uses an element-wise product of activated projections.Source: 1) STEncoder · pp. 7–8

2) STDecoder:

The STDecoder projects learned representations into future predictions using stacked feed-forward layers and ReLU. URCL trains with prediction and graph-contrastive losses, using replayed observations, spatio-temporal mixup, and data augmentation in its training process.

2) STDecoder:

  • The STDecoder applies stacked feed-forward layers followed by ReLU to project learned representations into future predictions.Source: 2) STDecoder · p. 8

E. Overall Objective Function

  • Training samples replay-buffer observations, fuses them with the current batch using STMixup, augments the data, learns holistic features, predicts, and updates model parameters.The algorithm identifies data preprocessing and training as lines 2–3 and 4–12, respectively.Source: E. Overall Objective Function · p. 8
  • URCL combines prediction-task loss with GraphCL self-supervised loss for holistic feature learning.The prediction objective uses mean absolute error for each data stream.Source: E. Overall Objective Function · p. 8

V. EXPERIMENTAL EVALUATION

The experiments use four graph-based urban spatio-temporal datasets covering traffic speed and traffic flow prediction. Evaluation methods include statistical and neural spatio-temporal prediction models.

1) Datasets:

  • Experiments use METR-LA and PEMS-BAY for traffic speed prediction, and PEMS04 and PEMS08 for traffic flow prediction.Source: 1) Datasets · p. 8

2) Evaluation Methods.:

3) Metrics:

The evaluation uses MAE and RMSE to measure prediction accuracy and examines URCL’s performance across streaming-training strategies and datasets. URCL outperforms baselines in most overall-accuracy cases and remains relatively stable across base and incremental sets.

3) Metrics:

  • MAE and RMSE measure prediction error, with lower values indicating greater accuracy; the evaluation also measures training and inference time.The metrics are computed over M test samples using predictions and ground truth.Source: 3) Metrics · p. 9

4) Other Implementation Details:

  • The streaming sequence uses 30% of each dataset as the base set and divides the remainder into four equal, sequential incremental sets.Training begins on the base set, then retrains from the last learned model on subsequent sets.Source: 4) Other Implementation Details · p. 9

2) Overall Accuracy:

  • URCL performs better than the best baseline by up to 36.0% in MAE and 34.1% in RMSE across most cases.It leads on most traffic-speed and traffic-flow dataset cases, with exceptions on METR-LA I4 and PEMS08 I1.Source: Table III · p. 10, 2) Overall Accuracy · p. 11

1) Performance of Training on Streaming Data:

  • URCL outperforms other methods on both datasets, while OneFitAll and FinetuneST deteriorate on incremental sets and URCL remains relatively stable.OneFitAll trains on the base set only; FinetuneST repeatedly fine-tunes on incremental sets.Source: 1) Performance of Training on… · pp. 10–11

3) Ablation Study:

Ablations show that each tested URCL component contributes to prediction accuracy, and results support using different backbones. The framework also shows favorable efficiency and faster convergence on incremental sets.

3) Ablation Study:

  • Removing STMixup, RMIR sampling, augmentation, or GraphCL raises MAE and RMSE; removing augmentation is worst in most cases, with URCL reductions up to 41.5% and 47.8%.The reductions are in MAE and RMSE, respectively, compared with the variant without spatiotemporal augmentation.Source: 3) Ablation Study · p. 11

4) Effect of Different Backbones:

5) Efficiency:

  • On PEMS04, URCL trains faster than DCRNN and has inference time far below DCRNN’s and comparable to other baselines.Other baselines except DCRNN train faster, but perform worse than URCL in prediction accuracy.Source: Figure 7 · p. 12, 5) Efficiency · p. 12

6) Convergence Analysis of Training:

VI. CONCLUSION:

  • The framework combines replay-buffer STMixUp, a spatiotemporal SimSiam network with a prediction autoencoder, and five augmentation methods; real-data experiments report improved prediction accuracy over the state of the art.The SimSiam and autoencoder coupling extracts holistic features through mutual information maximization.Source: §VI · p. 12

Related papers

Loading 2404.14999v1…