Source-linked AI summary
A Dynamic Fusion Large Language Model for Traffic Flow Prediction
Xue Qiu, Jianli Xiao
TL;DR
Traffic flow prediction requires models that capture both temporal dynamics and spatial topology, while existing neural and LLM-based approaches have important limitations. DF-LLM combines multi-scale spatiotemporal embedding, graph-based fusion, and a pretrained LLM backbone, achieving competitive forecasting performance across four real-world datasets. The results support its potential for traffic prediction across diverse scenarios and network scales.
Problem
Traffic flow prediction methods have limited ability to model complex spatiotemporal dependencies and traffic-network topology, while LLMs also face language–traffic structural and semantic disparities.
Method
DF-LLM combines spatiotemporal embedding, graph-based fusion with residual connections, differentiated parameter adaptation, and context aggregation attention around a pretrained LLM backbone.
Results
DF-LLM achieves competitive forecasting performance compared with existing methods across four real-world datasets and adapts stably to diverse traffic scenarios and network scales.
Takeaways & Limitations
The results indicate that combining graph convolution, multi-scale spatiotemporal features, and an LLM backbone is a viable approach for traffic flow forecasting.
Abstract
from arXiv · showhide
Traffic flow prediction is a core supporting technology for intelligent transportation systems. It uses historical data to infer future traffic dynamics in specific areas, thereby helping to alleviate congestion and improve resource allocation efficiency. Traditional neural networks struggle to break through accuracy limits due to their reliance on singular feature modeling, while large language models (LLMs) suffer from insufficient capture of spatial topological information and mining spatiotemporal correlation. This study proposes a Dynamic Fusion Large Language Model (DF-LLM) for traffic flow prediction. The model incorporates three core components: spatiotemporal embedding module, spatiotemporal fusion module, and LLM backbone. The spatiotemporal embedding module enables synergistic representation of multi-scale spatiotemporal features. The spatiotemporal fusion module integrates spatial topology and dynamic dependencies via graph convolution. The LLM backbone adopts a differentiated parameter adaptation strategy to balance training efficiency and traffic data adaptability. Additionally, it introduces a context aggregation attention module to strengthens global dependencies. More importantly, the LLM backbone takes the residual connections to mitigate the gradient vanishing in deep networks. Experiments show that DF-LLM has achieved better performance by comparing the metrics on all the four datasets.
1 Introduction
Traffic flow prediction supports traffic management and resource allocation, but existing neural and LLM-based methods have difficulty modeling complex spatiotemporal dependencies and traffic-network topology. DF-LLM addresses these challenges by combining graph-based spatial modeling with a pretrained Transformer backbone and reports competitive or superior forecasting performance.
- Traffic flow prediction enables forecasting future traffic conditions to support traffic management, congestion alleviation, and resource allocation.
- Traditional models struggle with complex spatiotemporal dependencies, irregular non-Euclidean topology, and multi-scale periodic patterns in traffic data.
- LLM-based approaches mainly emphasize temporal information while neglecting traffic-network topology, and language–traffic disparities hinder knowledge transfer.
- DF-LLM integrates graph-based spatial modeling with a pretrained Transformer backbone, residual connections, and differentiated parameter adaptation for traffic flow prediction.
- DF-LLM achieves competitive or superior forecasting performance compared with existing advanced methods.
2 Proposed Model
DF-LLM combines multi-scale spatiotemporal embeddings, graph-based fusion, and a selectively adapted GPT-2 backbone for traffic-flow forecasting. Its pipeline preserves and integrates spatial, temporal, and global contextual information before regression prediction.
- 2 Proposed Model: DF-LLM processes historical traffic data through multi-scale embedding, spatiotemporal fusion, an LLM backbone, and regression prediction.The embedding module produces initial, temporal, and spatial representations; fusion and the backbone refine them before future-flow prediction.
- 2.1 Spatiotemporal Embedding Module: The embedding module uses parallel initial, temporal, and spatial branches to represent basic features, periodic patterns, and spatial topology.Daily and weekly indices support temporal representations, while adaptive spatial embedding captures road-segment topological correlations.
- 2.2 Spatiotemporal Fusion Module: Residual 1×1 convolution fuses the three embeddings from 3D′ to D′ while preserving spatiotemporal features and improving training stability.The residual projection is described as mitigating gradient vanishing during dimensionality reduction.
- 2.2 Spatiotemporal Fusion Module: A two-layer GCN aggregates neighboring road-segment features using an adjacency matrix, producing fused representations that encode spatial topological dependencies.Residual access to the original fused features is retained, and the resulting representation is passed to the LLM backbone.
- 2.3 LLM Backbone: GPT-2 models temporal dependencies through differentiated parameter adaptation and a Context Aggregation MHA module that integrates global spatiotemporal information.Early blocks freeze attention and feedforward layers while training normalization; later blocks unfreeze attention while keeping feedforward layers frozen.
3 Experiments and Results
Experiments across four public traffic datasets show that DF-LLM provides competitive forecasting performance, adapts to diverse traffic patterns, and benefits substantially from its residual connections and other proposed modules.
- Main results: DF-LLM achieves competitive forecasting performance across PEMS04, PEMS08, METR-LA, and PEMS-BAY against deep learning and LLM-based baselines.Table 1 reports the evaluation across all four datasets.
- Main results: 18.96 (MAE), 30.76 (RMSE), and 12.87% (MAPE) are reported on PEMS04, while METR-LA results are 3.12 (MAE), 6.27 (RMSE), and 8.19% (MAPE).The PEMS04 values outperform most baselines, and the METR-LA results surpass ST-LLM and GCNGPT.
- Multi-scenario adaptability analysis: DF-LLM closely follows ground-truth traffic values across four datasets, capturing fluctuations, periodic patterns, congestion conditions, and large-scale traffic dynamics.The visual analysis combines Fig. 2 with the quantitative results to assess diverse traffic scenarios and network scales.
- Ablation study: Removing Context Aggregation MHA, GCN, or residual connections degrades performance, with residual-connection removal causing the largest decline.The ablation study attributes particular importance to residual connections for stabilizing model training.
4 Conclusion
The paper concludes that DF-LLM combines graph convolution, spatiotemporal representations, and a pretrained LLM backbone for traffic forecasting. Across four real-world datasets, it delivers competitive performance and stable adaptability, while future work targets dynamic spatiotemporal modeling and multimodal traffic data.
- 4 Conclusion: DF-LLM integrates graph convolution, spatiotemporal embedding, Context Aggregation MHA, and differentiated parameter adaptation with a pretrained LLM backbone.The framework is designed for spatiotemporal traffic forecasting.
- 4 Conclusion: Experiments on four real-world datasets show competitive forecasting performance, while ablation studies validate the effectiveness of the proposed components.The conclusion also reports stable adaptability across diverse traffic scenarios and network scales.
- 4 Conclusion: Future work will explore dynamic spatiotemporal modeling and multimodal traffic data to improve generalization and adaptability.