PaperlayerView exact revision on arXiv

Source-linked AI summary

TPLLM: A Traffic Prediction Framework Based on Pretrained Large Language Models

Yilong Ren, Yue Chen, Shuai Liu, Boyue Wang, Haiyang Yu, Zhiyong Cui

arXiv:2403.02221v2cs.LG

TL;DR

High collection and storage costs make accurate, generalizable traffic prediction with limited historical data challenging. TPLLM adapts pretrained LLMs to traffic sequences and road-network information. It outperforms other baselines in full-sample and few-shot prediction, including a 10% training-set setting.

  • Problem

    Traffic prediction models often need long training histories, but the cost of collecting and storing traffic data makes comprehensive datasets difficult to obtain, leaving accurate, generalizable prediction with limited historical data challenging.

    Source: §I · p. 1

  • Method

    TPLLM shapes multivariate traffic sequences into LLM-compatible inputs, incorporates road-graph spatial information, and fine-tunes a pretrained LLM with LoRA.

    Source: §I · p. 2

  • Results

    TPLLM outperforms other baselines in full-sample and few-shot prediction, with less metric degradation in the 10% training-set few-shot setting.

    Source: §V · p. 10

  • Takeaways & Limitations

    The few-shot results support pretrained LLM capability for small-sample traffic prediction tasks close to realistic scenarios.

    Source: §V · p. 10

Abstract

from arXiv · show

Traffic prediction constitutes a pivotal facet within the purview of Intelligent Transportation Systems (ITS), and the attainment of highly precise predictions holds profound significance for efficacious traffic management. The precision of prevailing deep learning-driven traffic prediction models typically sees an upward trend with a rise in the volume of training data. However, the procurement of comprehensive spatiotemporal datasets for traffic is often fraught with challenges, primarily stemming from the substantial costs associated with data collection and retention. Consequently, developing a model that can achieve accurate predictions and good generalization ability in areas with limited historical traffic data is a challenging problem. It is noteworthy that the rapidly advancing pretrained Large Language Models (LLMs) of recent years have demonstrated exceptional proficiency in cross-modality knowledge transfer and few-shot learning. Recognizing the sequential nature of traffic data, similar to language, we introduce TPLLM, a novel traffic prediction framework leveraging LLMs. In this framework, we construct a sequence embedding layer based on Convolutional Neural Networks (CNNs) and a graph embedding layer based on Graph Convolutional Networks (GCNs) to extract sequence features and spatial features, respectively. These are subsequently integrated to form inputs that are suitable for LLMs. A Low-Rank Adaptation (LoRA) fine-tuning approach is applied to TPLLM, thereby facilitating efficient learning and minimizing computational demands. Experiments on two real-world datasets demonstrate that TPLLM exhibits commendable performance in both full-sample and few-shot prediction scenarios, effectively supporting the development of ITS in regions with scarce historical traffic data.

I. INTRODUCTION

Traffic prediction is important to ITS, but data collection costs and task-specific training make accurate, generalizable prediction difficult where historical data are scarce. TPLLM adapts pretrained LLMs to traffic data and is reported to perform well in both full-sample and few-shot settings.

I. INTRODUCTION

  • Traffic prediction is an ITS core functionality, and accurate traffic-flow forecasts can support timely congestion alerts and reduced travel time and emissions.The cited example links alerts to drivers avoiding congested routes.Source: §I · p. 1
  • Comprehensive traffic datasets are costly to collect and store, while deep-learning models often need long histories and may overfit task-specific training.These constraints make accurate, generalizable prediction with limited historical data challenging.Source: §I · p. 1
  • TPLLM shapes multivariate traffic sequences into LLM-compatible inputs to exploit pretrained knowledge, while adding graph-structured road-network information for spatial features.Its embedding module uses CNNs and GCNs, and the LLM output is used to generate predictions.Source: §I · p. 2
  • The reported experiments find TPLLM effective for regular traffic prediction and equally capable in few-shot prediction.The paper says the results validate applying pretrained LLM prior knowledge to traffic prediction.Source: §I · p. 2
  • LoRA is used to reduce training costs while maintaining fine-tuning quality and preserving the pretrained model’s existing knowledge.The paper also describes it as improving efficiency with limited data.Source: §I · p. 2
  • Experiments in sufficient- and limited-data scenarios validate that pretrained LLM prior knowledge can be applied to traffic prediction.The paper frames these scenarios as relevant to realistic situations.Source: §I · p. 2

II. RELATED WORK

Related work covers traffic prediction methods and pretrained LLMs, including their few-shot capabilities and their proposed use for sequence prediction.

A. Traffic Prediction

  • Early statistical and conventional machine-learning approaches treat traffic as a simple time series, while deep models combine temporal and spatial feature extractors.Graph-based methods such as STGCN, ASTGCN, and STSGCN capture spatiotemporal relationships, but typically require substantial historical training data.Source: A. Traffic Prediction · p. 3

B. Pretrained LLMs

  • Scaling pretrained language models has been associated with improved performance and few-shot capabilities in larger models.The related-work discussion describes GPT, GLM, LLaMA, and BERT as examples of pretrained model families with differing architectures and uses.Source: B. Pretrained LLMs · p. 3
  • The paper treats time-series prediction as generative prediction of future sequences and therefore considers decoder-based generative LLMs, such as GPT, applicable.Source: B. Pretrained LLMs · p. 3

C. LLMs for Time-series Prediction

Prior LLM applications chiefly address general time-series tasks, while few traffic-focused studies leave graph-structured road features unaddressed. The paper introduces TPLLM and frames traffic prediction as mapping historical sensor sequences and road adjacency to future sequences.

C. LLMs for Time-series Prediction

  • Pretrained LLM applications have focused mainly on general time-series processing, with relatively few studies addressing traffic prediction.Earlier traffic work applied LLMs to imputation or prediction, but the cited prediction method did not account for graph-structured road-network features.Source: C. LLMs for Time-series Prediction · pp. 3–4
  • The paper proposes a traffic-prediction framework based on pretrained LLMs in response to traffic data’s complex spatiotemporal characteristics.Source: C. LLMs for Time-series Prediction · p. 4

A. Task description

  • The task description represents the road network as a graph and treats each sensor’s traffic measurements as node features.The graph includes sensor nodes, edges between connected nodes, and an adjacency matrix denoting connectivity.Source: A. Task description · p. 4
  • The prediction model takes the last T historical steps and the graph adjacency matrix as input, and outputs a sequence of future traffic values.The task description aims to construct the model from a pretrained LLM for regular and small-sample prediction.Source: A. Task description · p. 4

B. Overview of the TPLLM

TPLLM uses a pretrained Transformer-based LLM to analyze and predict traffic spatiotemporal data. Its input embeddings combine CNN-derived sequence features with GCN-derived spatial features, then map them to the LLM’s embedding size.

B. Overview of the TPLLM

  • The framework uses a pretrained Transformer-based LLM backbone and an input embedding module designed to account for temporal and spatial traffic features.The sequence layer uses a CNN; the graph layer uses a GCN.Source: B. Overview of the TPLLM · p. 4

C. Input Embedding

  • A GCN-based graph embedding extracts spatial information from the road network’s graph structure.The graph embedding has F channels.Source: C. Input Embedding · p. 5
  • A 1-D CNN sequence embedding extracts features from the traffic time series.The sequence embedding uses F filters.Source: C. Input Embedding · p. 5
  • The input embedding combines graph and sequence features, maps the temporal dimension to the LLM’s embedding size, and takes the last feature channel as the LLM input.The resulting Emb(X) has shape N×D.Source: C. Input Embedding · p. 5

D. Pretrained LLM and PEFT

TPLLM applies parameter-efficient fine-tuning to adapt a pretrained LLM to traffic prediction without retraining all its parameters. LoRA adds trainable low-rank matrices in Transformer blocks, and the LLM output is mapped to the prediction horizon.

D. Pretrained LLM and PEFT

  • TPLLM uses LoRA to adapt the pretrained LLM for traffic prediction with lower computational resources than full retraining.The motivation is that retraining the entire LLM requires computational resources deemed too large to be acceptable.Source: D. Pretrained LLM and PEFT · p. 5
  • LoRA injects trainable rank-decomposition matrices B and C into each Transformer block to reduce the number of trainable parameters.The method applies LoRA to the attention layers’ Query and Key; B is zero-initialized and C is Gaussian-initialized.Source: D. Pretrained LLM and PEFT · p. 5
  • The rank r controls how much information the LoRA module can contain, with r ≪ min(d, k).For Query and Key, B ∈ R^(d×r) and C ∈ R^(r×k).Source: D. Pretrained LLM and PEFT · p. 5
  • The LLM output is passed through a linear layer to match the desired prediction dimension T′.The LLM with LoRA processes Emb(X) to produce Emb(H).Source: D. Pretrained LLM and PEFT · p. 5

E. Loss Function

TPLLM uses robust L1 loss for training because traffic datasets can contain outliers, including those caused by sensor failures. The experiments cover full-sample and few-shot prediction, LoRA-rank sensitivity, and ablations.

E. Loss Function

  • Robust L1 loss, equivalent here to Mean Absolute Error (MAE), is selected because traffic datasets can contain outliers such as those caused by sensor failures.The paper identifies MAE as TPLLM’s loss function.Source: E. Loss Function · p. 6

IV. EXPERIMENTS

  • The experiments include full-sample and few-shot prediction, LoRA-rank sensitivity analysis, and ablations.These experiments are intended to assess pretrained LLM cross-modal knowledge transfer and few-shot learning for traffic prediction.Source: §IV · p. 6

A. Datasets

Experiments use the PeMS04 and PeMS08 datasets, focusing on traffic flow measured by sensors on California roads. The setup specifies prediction horizons, the GPT-2 backbone with LoRA, and MAE, RMSE, and MAPE evaluation metrics.

A. Datasets

  • The experiments use PeMS04 and PeMS08, datasets collected by sensors on major California roads at five-minute intervals, and focus on traffic flow.The datasets include traffic flow, occupancy, and velocity; the study selects traffic flow as its prediction target.Source: A. Datasets · p. 6
  • The setup uses one hour of historical traffic flow to predict traffic at 15-minute, 30-minute, and one-hour horizons.The corresponding input length is 12 steps, and output lengths are 3, 6, and 12 steps.Source: A. Datasets · p. 6

B. Settings

  • The framework uses GPT-2 as its base LLM, with LoRA making about 0.95% of its parameters trainable.The paper notes that other pretrained LLMs can be substituted.Source: B. Settings · p. 6

C. Evaluation Metrics

  • Performance is evaluated using Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE).The paper describes metrics for individual predictions.Source: C. Evaluation Metrics · p. 6

D. Baselines

TPLLM outperforms the baselines across full-sample traffic predictions on both datasets and all reported metrics. The section also describes the comparison models and evaluation setup.

D. Baselines

  • LSTM captures only temporal dependencies, while STGCN, ASTGCN, and STSGCN use spatiotemporal graph-based approaches.STGCN combines temporal gated and spatial graph convolution; ASTGCN uses attention, and STSGCN connects the same nodes at adjacent times.Source: D. Baselines · pp. 6–7

E. Full-sample Prediction

  • TPLLM performs better than baselines on all metrics for 15-minute, 30-minute, and 1-hour predictions across both datasets.Table III reports these full-sample results.Source: Table III · p. 7, E. Full-sample Prediction · p. 7
  • Full-sample data were split chronologically into training, validation, and test sets in proportions of 60%, 20%, and 20%.The reported comparison uses the best results across different LoRA rank values.Source: E. Full-sample Prediction · p. 7

F. Few-shot Prediction

With only 10% of the full-sample training-set size, TPLLM outperforms baselines on both datasets and all metrics. Sampled visualizations show good prediction accuracy and trend tracking across weekday and weekend traffic.

F. Few-shot Prediction

  • Few-shot training uses 10% of the full-sample training-set size, with validation and test sets unchanged from the full-sample experiment.The split proportions are 6%, 20%, and 20% of the dataset, respectively.Source: F. Few-shot Prediction · p. 7
  • TPLLM outperforms baselines on all metrics in few-shot predictions on both datasets, with most metrics changing least relative to full-sample results.Some few-shot metrics also outperform baselines in the full-sample experiment.Source: F. Few-shot Prediction · p. 7, Table IV · p. 8

G. Visualization of Full-sample and Few-shot Prediction

H. Ablation Study

Ablations indicate that the sequence embedding layer, graph embedding layer, and LoRA each contribute positively to prediction performance. Sensitivity tests find relatively flat MAE curves across LoRA ranks, supporting smaller ranks to save computation.

H. Ablation Study

  • Removing the sequence embedding layer produces the worst results in all but full-sample PeMS04, indicating its especially strong impact on accuracy.The authors attribute this to traffic data's time-series nature and the 1-D CNN layer's sequence-feature extraction.Source: H. Ablation Study · p. 8
  • Removing the graph embedding layer decreases accuracy in every experiment, consistent with GCNs extracting spatial features that help prediction in complex road networks.Source: H. Ablation Study · p. 8
  • Removing LoRA decreases accuracy, while the frozen LLM retains some zero-shot capability and its prior knowledge remains effective for traffic prediction.Without LoRA, the pretrained LLM parameters are completely frozen and cannot learn new knowledge.Source: H. Ablation Study · p. 8
  • The full TPLLM framework outperforms its three degraded variants in full-sample and few-shot evaluations on both datasets.Table V reports average evaluation metrics across prediction steps within one hour.Source: Table V · p. 8, H. Ablation Study · p. 8

I. Sensitivity to rank of LoRA

  • MAE curves are relatively flat across LoRA ranks; smaller ranks can therefore reduce learnable-matrix size and computational demands while retaining effective predictions.Optimal ranks are 48 for both PeMS04 settings and full-sample PeMS08, and 32 for few-shot PeMS08.Source: §I · p. 9, Figure 7 · p. 9

V. CONCLUSIONS

The paper presents TPLLM for traffic prediction with limited historical data, reporting that it outperforms other baselines in full-sample and few-shot settings. It also reports that smaller LoRA ranks can save computational resources and that the sequence embedding layer, graph embedding layer, and LoRA positively affect prediction accuracy.

  • TPLLM outperforms other baselines in full-sample and 10% training-set few-shot prediction, with less metric degradation in the few-shot setting.Source: §V · p. 10
  • A smaller LoRA rank can provide sufficiently accurate predictions while saving computational resources.The reported prediction accuracy is less affected by LoRA rank.Source: §V · p. 10
  • The sequence embedding layer, graph embedding layer, and LoRA each have a positive effect on prediction accuracy in the ablation study.Source: §V · p. 10
  • Future work will design embeddings that consider more traffic-affecting factors to improve prediction accuracy.The authors also plan to explore PEFT techniques suited to spatiotemporal prediction and interpretable LLM knowledge learning.Source: §V · p. 10
Loading 2403.02221v2…