Source-linked AI summary
AirFormer: Predicting Nationwide Air Quality in China with Transformers
Yuxuan Liang, Yutong Xia, Songyu Ke, Yiwei Wang, Qingsong Wen, Junbo Zhang, Yu Zheng, Roger Zimmermann
TL;DR
Air pollution creates substantial health and environmental concerns, making fine-grained air-quality forecasting important in China. AirFormer combines deterministic spatio-temporal Transformer modeling with stochastic latent variables for nationwide prediction, reducing prediction errors by 4.6-8.2% compared with prior methods.
Problem
Air pollution harms human health and the environment, while existing monitoring motivates forecasting air quality across China.
Method
AirFormer decouples prediction into deterministic spatial-temporal modeling with DS-MSA and CT-MSA, followed by a stochastic latent-variable stage for uncertainty.
Results
4.6-8.2%: AirFormer reduces prediction errors compared with prior methods.
Takeaways & Limitations
Nationwide forecasting across thousands of locations provides more useful information to the public and supplies more data samples for model training.
Abstract
from arXiv · showhide
Air pollution is a crucial issue affecting human health and livelihoods, as well as one of the barriers to economic and social growth. Forecasting air quality has become an increasingly important endeavor with significant social impacts, especially in emerging countries like China. In this paper, we present a novel Transformer architecture termed AirFormer to collectively predict nationwide air quality in China, with an unprecedented fine spatial granularity covering thousands of locations. AirFormer decouples the learning process into two stages -- 1) a bottom-up deterministic stage that contains two new types of self-attention mechanisms to efficiently learn spatio-temporal representations; 2) a top-down stochastic stage with latent variables to capture the intrinsic uncertainty of air quality data. We evaluate AirFormer with 4-year data from 1,085 stations in the Chinese Mainland. Compared to the state-of-the-art model, AirFormer reduces prediction errors by 5%~8% on 72-hour future predictions. Our source code is available at https://github.com/yoshall/airformer.
Introduction
Air quality forecasting matters for health, policy, and society, but nationwide fine-grained prediction must address transformer efficiency and intrinsic uncertainty. AirFormer targets these challenges with efficient spatial-temporal attention and stochastic latent variables.
- Motivation: Air pollution harms human health and the environment, with emerging countries such as China particularly affected.The supplied passage reports seven million annual fatalities worldwide and widespread exposure above WHO-recommended levels.
- Motivation: Hourly monitoring stations provide pollutant time series, while forecasting can support health protection and government policy-making.Measurements include PM2.5, PM10, and NO2; forecasting can inform decisions such as whether to travel outdoors.
- Research gap: Prior air-quality prediction largely operated at city scale, whereas recent work began extending collective prediction toward nationwide station networks.Classical and data-driven approaches were constrained by computational expense and commonly used dozens of stations.
- Research gap: Transformers can capture short- and long-term spatial interactions and dynamically changing location correlations, but standard MSA costs O(N^2) in the number of stations.This quadratic expense becomes difficult to afford for fine-grained datasets with many stations.
- Research gap: Future air-quality readings are uncertain because observations may be inaccurate or missing and emissions and policies can be unpredictable.Earlier deterministic approaches often do not capture this uncertainty in large-scale air-quality data.
- AirFormer: AirFormer addresses nationwide prediction by decoupling learning into deterministic and stochastic stages with efficient attention mechanisms and latent variables.The approach uses domain knowledge, spatial and temporal MSA variants, and latent variables learned from deterministic hidden states.
- AirFormer: AirFormer uses dartboard spatial attention to model nearby locations finely and distant stations coarsely, organizing surrounding stations into regional representations.The supplied figure description states that regions are formed by circles and lines before attention uses the query station and regional keys and values.
Preliminary
The preliminary formulation represents station measurements over time and defines transformer self-attention alongside variational latent-variable modeling. These components establish the prediction task and uncertainty-modeling foundation.
- Problem Formulation: At each time t, N stations provide D measurements, including pollutants and external factors such as weather and wind speed.Each x_ij denotes the j-th measurement at station i.
- Problem Formulation: Given the past T time steps, the model predicts D′ measurement types over the next τ steps.The mapping is expressed as F(·) → Y_1:τ from historical data X_1:T.
- Multi-head Self-Attention: Multi-head self-attention lets each sequence token gather messages from other tokens through learned queries, keys, and values.The projections use learnable W_q, W_k, and W_v parameters across N_h heads.
- Multi-head Self-Attention: Standard self-attention has quadratic computational complexity with respect to sequence length S.This complexity motivates efficient attention mechanisms for long station sequences.
- Variational Models with Latent Variables: Variational autoencoders model p(x) with an unobserved latent variable z and parameterized distribution p_θ(x|z)p_θ(z).The marginal distribution integrates over the latent variable.
- Variational Models with Latent Variables: Because the marginal integral is usually intractable, VAE optimization uses an approximate posterior and the evidence lower bound.The ELBO combines a KL-divergence term with an expected log-likelihood term, while Gaussian prior and posterior distributions encode uncertainty.
Methodology
AirFormer separates nationwide air-quality forecasting into deterministic spatio-temporal representation learning and stochastic uncertainty modeling. Its deterministic stage uses efficient spatial and temporal attention, while latent variables model uncertainty and support prediction.
- Deterministic stage: AirFormer transforms historical readings with an MLP and processes them through L AirFormer blocks for deterministic spatio-temporal representations.
- Stochastic stage: AirFormer generates latent variables at each level, conditioning lower-level variables on higher-level variables so upper levels provide larger contextual receptive fields.The generation model predicts the next time step from past observations, while the inference model approximates the posterior using current and previous steps.
- Deterministic stage: DS-MSA captures spatial interactions with linear complexity by projecting each station’s surroundings into M regions before applying multi-head self-attention.The projection uses a station-specific assignment matrix and excludes regions without stations during attention.
- Deterministic stage: CT-MSA captures temporal dependencies within non-overlapping local windows, increases window sizes across blocks, and masks future positions to preserve causality.Its computational cost is O(TWC), where W is the window size, and learnable absolute position encoding is added.
- Prediction and optimization: Predictions use deterministic and stochastic hidden states at time T through an MLP, trained jointly with prediction loss and the negative ELBO.The prediction loss is L1, while the ELBO term includes reconstruction likelihood and KL divergence between prior and posterior.
Experiments
Experiments use a four-year nationwide dataset and evaluate AirFormer against classical, STGNN, and attention-based baselines using MAE, RMSE, and sudden-change errors. AirFormer performs best overall, while ablations support its spatial, temporal, stochastic, and positional components.
- Dataset and Evaluation: The dataset contains hourly air-quality records from 1,976 stations, with experiments retaining 1,085 stations whose PM2.5 missing rate is below 20%.Predictions use the past 72 hours to forecast PM2.5 for the next 72 hours.
- Dataset and Evaluation: AirFormer is compared with history average, vector autoregression, STGNN variants, and attention-based baselines using MAE and RMSE, where smaller values indicate better performance.Sudden changes are PM2.5 values above 75 µg/m3 that change by more than ±20 µg/m3 within three hours.
- Model Comparison: 8.2%, 7.5% and 5.3% lower MAE than DeepAir are achieved for 24-, 48- and 72-hour predictions, respectively, while sudden-change errors improve by at least 7.3% in MAE and 13.1% in RMSE.The reported model comparison averages results over five runs and finds AirFormer significantly better than competing baselines at level 0.01.
- Spatial Modeling: DS-MSA lowers errors across future horizons while running 39% faster than vanilla MSA and 21% faster than local MSA.Removing DS-MSA significantly degrades MAE, while the 50-200 dartboard setting is selected as the speed-accuracy default.
- Temporal Modeling: Temporal modules outperform removing CT-MSA, MSA-based variants outperform WaveNet, and causal local-window attention improves over standard MSA across future steps.These comparisons support the necessity of temporal modeling and the design choices in CT-MSA.
- Stochastic Modeling and Position Encoding: Latent variables reduce sudden-change MAE by 4.5% with 149 additional seconds per training epoch, while removing either spatial or temporal position encoding degrades performance across horizons.The improvement from spatial position encoding is slightly higher than that from temporal position encoding.
Related Works
Air-quality prediction research spans physics-based dispersion models and data-driven approaches, with each relying on different sources of information and modeling strategies.
- Prior Approaches: Prior work includes classical dispersion models and data-driven models for forecasting air quality.The related-work discussion presents both as established lines of research.
Conclusion and Future Work
The study presents a transformer model for nationwide air-quality prediction across thousands of Chinese locations, combining spatio-temporal learning with uncertainty measurement. It reports 4.6-8.2% lower prediction errors than prior methods and identifies online learning and public deployment as future directions.
- 4.6-8.2% lower prediction errors are reported compared to prior methods.
- The model collectively forecasts air quality among thousands of locations in China.
- Its design combines transformers’ spatio-temporal learning capabilities with stochastic latent spaces for uncertainty measurement.
- Future work will explore online learning and deployment for public use.