PaperlayerView exact revision on arXiv

Source-linked AI summary

LSTTN: A Long-Short Term Transformer-based Spatio-temporal Neural Network for Traffic Flow Forecasting

Qinyao Luo, Silu He, Xing Han, Yuhan Wang, Haifeng Li

arXiv:2403.16495v1cs.LGcs.AIcs.SI

TL;DR

Many traffic forecasting methods use short histories, limiting access to long-term trends and periodicity. LSTTN learns compact long-history representations and combines long- and short-term features; experiments report better performance than baseline models across prediction horizons.

  • Problem

    Many traffic forecasting methods use short histories, while extracting key temporal information and compact representations from long histories remains challenging.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    LSTTN uses a masked-subseries Transformer to learn long-history representations, extracts long-term trends and periodicity, and fuses them with short-term features.

    Source: §1 · p. 2, §3.5 · p. 6, §5 · p. 13

  • Results

    LSTTN outperforms baseline models for all prediction horizons in experiments on four real-world datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LSTTN is presented as a pluggable framework compatible with STGNN models for integrating long- and short-term historical features.

    Source: §5 · p. 13

    Compared with STGNNs that learn only short-term features, LSTTN requires longer training time and more parameters, which may affect real-world adoption.

    Source: §5 · p. 13

Abstract

from arXiv · show

Accurate traffic forecasting is a fundamental problem in intelligent transportation systems and learning long-range traffic representations with key information through spatiotemporal graph neural networks (STGNNs) is a basic assumption of current traffic flow prediction models. However, due to structural limitations, existing STGNNs can only utilize short-range traffic flow data; therefore, the models cannot adequately learn the complex trends and periodic features in traffic flow. Besides, it is challenging to extract the key temporal information from the long historical traffic series and obtain a compact representation. To solve the above problems, we propose a novel LSTTN (Long-Short Term Transformer-based Network) framework comprehensively considering the long- and short-term features in historical traffic flow. First, we employ a masked subseries Transformer to infer the content of masked subseries from a small portion of unmasked subseries and their temporal context in a pretraining manner, forcing the model to efficiently learn compressed and contextual subseries temporal representations from long historical series. Then, based on the learned representations, long-term trend is extracted by using stacked 1D dilated convolution layers, and periodic features are extracted by dynamic graph convolution layers. For the difficulties in making time-step level prediction, LSTTN adopts a short-term trend extractor to learn fine-grained short-term temporal features. Finally, LSTTN fuses the long-term trend, periodic features and short-term features to obtain the prediction results. Experiments on four real-world datasets show that in 60-minute-ahead long-term forecasting, the LSTTN model achieves a minimum improvement of 5.63\% and a maximum improvement of 16.78\% over baseline models. The source code is available at https://github.com/GeoX-Lab/LSTTN.

1. Introduction

Many traffic forecasting models use short histories, which can miss long-term trends and periodicity; LSTTN combines long- and short-term temporal features. Its experiments report better performance than baseline models across all prediction horizons.

1. Introduction

  • Accurate traffic forecasts can help drivers plan routes and transport departments arrange personnel to guide traffic in congested areas.Source: §1 · p. 1
  • Many existing models use short historical series, which may not capture the rich temporal features needed to distinguish future trends across locations.In the example, six hours of history are insufficient when two sensors have similar short-term trends and values.Source: §1 · pp. 1–2
  • LSTTN integrates long- and short-term features, using masked-subseries pretraining and convolutional and graph components to extract long-range representations, trends, and periodicity.The Transformer is pretrained through a mask-reconstruction task to learn compressed, contextual subseries representations.Source: §1 · p. 2
  • LSTTN outperforms baseline models across all prediction horizons, and prediction visualizations show robustness to missing data and abrupt changes.The evaluation covers four real-world datasets.Source: §1 · p. 2

2. Related Work

Traffic forecasting methods include statistical approaches and data-driven spatial-temporal graph neural networks. Related work traces the move from statistical models to STGNN architectures and learning paradigms, while noting the challenge of using long histories to capture trends and periodicity.

2. Related Work

  • Traffic flow forecasting methods are grouped into statistical-based methods and data-driven spatial-temporal graph neural network methods.Source: §2 · p. 2

2.1. Statistical-based Tra

  • Statistical approaches include historical averages, ARIMA, and Kalman filtering, each relying on a distinct temporal modeling strategy or data assumption.Historical averages use same-time-of-day values, ARIMA models differenced stationary series, and Kalman filtering models dynamic-system states.Source: §2.1 · p. 2

2.2. STGNN-based Traffic Flow Forecasting Methods

  • STGNNs combine graph-based spatial modeling with temporal models, addressing road-network dependencies that ordinary time-series methods often omit.Road networks are naturally represented as graphs, which capture topological relationships between roads.Source: §2.1 · p. 2, §2.2.1 · pp. 2–3

2.2.1. Structural Design of STGNN-based Tra

  • RNN-, CNN-, and Transformer-based STGNNs model temporal dependencies differently, but short inputs and structural limits can hinder long-range traffic-pattern learning.RNNs have difficulty capturing long-range dependencies; CNNs use temporal convolutions, while Transformers can connect distant time steps but cited traffic models still use short histories.Source: §2.2.1 · p. 3

2.2.2. Learning Paradigm of STGNN-based Tra

  • STGNNs mainly use supervised future-flow labels, while self-supervised and contrastive approaches add training signals; STEP uses long-range representations but does not sufficiently model long-term trends and periodicity.Source: §2.2.2 · p. 4

3. Methodology

LSTTN extends traffic forecasting beyond short input histories by combining long-range temporal representations with fine-grained short-term features. Its pipeline reconstructs masked subseries, extracts long-term trends and periodic patterns, and fuses them with short-term trends for prediction.

3.1. Problem Definition

  • LSTTN frames forecasting as predicting F future steps from historical traffic features and adjacency, while seeking temporal patterns from a longer history than the short histories common in prior methods.The problem definition gives H = 12 as an example of a short historical input and distinguishes the long history Xlong from Xshort.Source: §3.1 · p. 4

3.2. LSTTN Framework

  • The framework learns subseries representations from long histories, extracts long-term trends and periodic features from them, and models local short-term trends directly from the last subseries.These feature streams are fused for the final prediction.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Masked Sub-series Transformer

  • The masked subseries Transformer uses equal-length subseries as input units and 75% random masking, then reconstructs the long series from unmasked representations and learned mask tokens.Its self-supervised loss is calculated only on masked subseries; the Transformer encoder serves as the representation learner.Source: §3.3 · p. 5

3.4. Long-term Trend Extractor

  • Stacked 1D dilated convolution layers extract long-term trends from subseries representations, with a receptive field that grows exponentially as layers are added.The stated design captures long-range features efficiently while avoiding problems such as gradient vanishing.Source: Figure 4 · p. 5, §3.4 · p. 6

3.5. Periodicity Extractor

  • The periodicity extractor applies spatial-based graph convolution to corresponding previous-week and previous-day representations, combining graph-defined and hidden spatial dependencies.The hidden dependencies are represented by a self-adaptive neighbor matrix when graph structure is unavailable.Source: §3.5 · p. 6, Figure 5 · p. 6

3.6. Short-term Trend Extractor

  • Because subseries representations are too coarse for time-step-level prediction, an existing STGNN extracts fine-grained short-term trends from the recent historical series.The paper names Graph WaveNet, DCRNN, and STGCN as examples of suitable STGNNs.Source: §3.6 · pp. 6–7

3.7. Feature Fusion

  • The model concatenates long-term trend, periodic, and short-term trend features for prediction and trains the forecast output with an L1 loss.The target is the traffic flow over the F future time steps.Source: §3.7 · p. 7

4.1. Datasets

The evaluation uses four public traffic datasets with differing sensor coverage and time windows, split using dataset-specific train/validation/test ratios.

  • METR-LA has speeds from 207 Los Angeles County freeway sensors (March–June 2012), while PEMS-BAY has speeds from 325 Bay Area highway sensors (January–May 2017).Source: §4.1 · p. 7
  • PEMS04 and PEMS08 contain flow, occupancy, and speed from 307 and 170 California sensors, respectively, covering January–February and July–August 2018.Source: §4.1 · p. 7
  • The four public datasets use train/validation/test splits of 7:2:1 for METR-LA and PEMS-BAY and 6:2:2 for PEMS04 and PEMS08.Source: §4.1 · p. 7

4.2. Experimental Setup

The experimental setup evaluates predictions with three error metrics, compares LSTTN with nine traffic forecasting models, and specifies the hardware and training settings.

4.2.1. Evaluation Metrics

  • Evaluation uses RMSE, MAE, and MAPE to compare predicted sequences with ground truth, omitting missing data from all three metrics.Source: §4.2.1 · p. 7

4.2.2. Baseline Models

  • The comparison includes five widely adopted baselines: DCRNN, STGCN, Graph WaveNet, GMAN, and MTGNN.Source: §4.2.2 · p. 7
  • Four more recently proposed baselines are DDSTGCN, STID, DSTET, and DAGN.Source: §4.2.2 · p. 7

4.2.3. Experimental Environment and Hyperparameter Settings

  • Table 1 reports statistics for the four datasets.Source: Table 1 · p. 8
  • Experiments use a two-week input sequence (4032 five-minute steps), batch size 32, 100 training iterations, Adam, and a multistep learning-rate scheduler.The experiments run on Linux with an Intel Xeon Gold 5218R CPU and NVIDIA RTX A6000 GPU; Graph WaveNet is the STGNN implementation.Source: §4.2.3 · pp. 7–8

4.3. Experimental Results

Results favor LSTTN on three datasets, with some gains growing at longer horizons; ablations, hyperparameter choices, visualizations, and execution costs characterize its performance and trade-offs.

4.3.1. Prediction Accuracy

  • On METR-LA, LSTTN beats all nine baselines across metrics and horizons; at 60 minutes, its gains range from 5.63% RMSE on PEMS04 to 16.78% MAPE on PEMS-BAY.Improvements also grow with prediction horizon on three datasets.Source: §4.3.1 · p. 8, Table 2 · p. 9

4.3.2. Ablation Studies

  • Removing any of the three extractors worsens performance, with long-term trend gains increasing at longer horizons and periodicity contributing less.The long-term trend ablation gap reaches approximately 17% at 60 minutes on MAPE; without short-term extraction, coarse subseries representations cannot capture fine-grained fluctuations.Source: §4.3.2 · p. 8, Table 3 · p. 9

4.3.3. Hyperparameter Analysis

  • The selected subseries-representation dimension is 64, while hidden dimensions of 4 are selected for the long-term trend and periodicity modules.Dimension 32 underrepresents contextual and temporal information, while 96 or 128 risks overfitting; larger module hidden dimensions also reduce performance, with the authors attributing this to overfitting.Source: §4.3.3 · p. 10, Figure 6 · p. 10, Figure 7 · p. 10

4.3.4. Visualization

  • Across forecast horizons, LSTTN responds to abrupt traffic changes with less delay than Graph WaveNet, with the authors associating this advantage with periodicity modeling.Both models lag sudden changes, and the lag grows at longer horizons; the comparison visualizes 15-, 30-, and 60-minute predictions.Source: Figure 8 · p. 11, Figure 9 · p. 11, §4.3.4 · p. 12
  • When missing intervals exceed a short-term STGNN’s input window, LSTTN still produces rough predictions from long-term trends and periodic information.The visualized examples include a spiky pattern at node 91 and a rapid-rise-then-slow-decrease trend at node 196.Source: §4.3.4 · p. 12, Figure 10 · p. 12

4.3.5. Execution cost analysis

  • LSTTN uses more parameters and training time than the compared models, with over 96% of its training time spent on the masked subseries Transformer.The authors attribute the additional cost to the long-sequence Transformer, long-term and periodicity extractors, and feature-fusion training.Source: §4.3.5 · p. 13

5. Conclusion and Future Work

LSTTN combines long- and short-term traffic features, with its design described as pluggable for STGNNs. Experiments support the value of its long-term features, while the authors note costs and design questions that remain for future work.

5. Conclusion and Future Work

  • The authors describe LSTTN as pluggable and compatible with all STGNN models, supporting its potential use as a deployment plugin.They present this compatibility as making deployment to application platforms easier.Source: §5 · p. 13
  • Experiments report better performance than baseline models across real-world datasets and prediction horizons, with ablations supporting each learning module.Prediction visualizations also show faster perception of abrupt changes and predictions despite missing data.Source: §5 · p. 13
  • LSTTN learns subseries representations with a masked subseries Transformer, then extracts long-term trends and periodicity and fuses them with short-term features.The long-term trend uses stacked 1D dilated convolution; periodicity uses spatial-based graph convolution.Source: §5 · p. 13
  • LSTTN requires longer training time and more parameters than short-term-only STGNNs such as Graph WaveNet, potentially affecting real-world adoption.The authors also identify user-friendliness, alternative module and fusion designs, dynamic spatial dependencies, and interpretability as areas for further work.Source: §5 · p. 13

CRediT authorship contribution statement

The contribution statement credits authors with conceptualization, methodology, writing, data curation, software, validation, supervision, and funding acquisition.

CRediT authorship contribution statement

Related papers

Loading 2403.16495v1…