PaperlayerView exact revision on arXiv

Source-linked AI summary

Shift Aware Transfer Learning with Adaptive Dual-Encoder Fusion for PM Forecasting in Data-Limited Environments

Shahab Band, Hamed Mohammadi

arXiv:2608.14456v1cs.AI

TL;DR

PM2.5 forecasting is challenging with limited target observations and source–target shifts. The paper uses adaptive dual-encoder transfer learning to preserve target-specific representations, outperforming target-only and direct-transfer baselines under the investigated U.S.–Taiwan shift.

  • Problem

    Under source–target domain shifts, forecasting methods need to preserve target-specific information rather than force all features into a common representation.

  • Method

    The study develops an adaptive dual-encoder framework combining a source encoder pretrained on U.S. monitoring data with target-domain representation learning.

  • Results

    The trainable-source dual-encoder achieved the best overall performance, while dual-encoder fusion outperformed target-only and direct-transfer baselines under the investigated U.S.–Taiwan shift.

  • Takeaways & Limitations

    Source knowledge is most effective when target-specific information is preserved and the transferred encoder adapts during target-domain training.

  • Takeaways & Limitations

    The framework does not provide calibrated uncertainty estimates for operational prediction, though these could be incorporated.

Abstract

from arXiv · show

Short-horizon forecasting of fine particulate matter (PM2.5) remains difficult when observations from the target domain are limited and the statistical properties of the source and target domains differ. In these settings, models trained only on local data may not capture complex temporal dynamics, while direct transfer learning can result in negative transfer. This study develops a shift-aware dual-encoder transfer framework that combines source-domain knowledge with target-specific representation learning. The source encoder was pretrained using hourly observations from 10 U.S. monitoring locations. The framework was then adapted and evaluated using two years of hourly observations from 77 stations in Taiwan under a chronological train-validation-test protocol. Among the four principal baselines, the frozen-source dual-encoder model achieved the best performance, with MSE = 21.8960, MAE = 3.1597, and R^2 = 0.8725. This corresponds to an MSE reduction of approximately 7.1% relative to TL-v1 and 4.1% relative to TL-v2. The ablation analysis showed that removing the Taiwan-specific branch caused the largest decline in performance. Allowing the source encoder to adapt produced the best overall result, with MSE = 21.6575, MAE = 3.1383, and R^2 = 0.8739. SHAP analysis indicated that predictions were driven mainly by recent PM2.5 observations and meteorological variables related to pollutant transport and dispersion. These results suggest that source-domain knowledge is most effective when target-specific information is preserved and the transferred representation is allowed to adapt under target supervision.

1 Introduction

Short-horizon PM₂.₅ forecasting is challenging with limited target observations and source–target distribution shift, which can cause negative transfer. The study addresses this challenge with a shift-aware dual-encoder framework that preserves target-specific information while transferring source knowledge.

  • Motivation: Limited observations and differing training and deployment characteristics reduce deep-learning effectiveness for PM₂.₅ forecasting.These conditions motivate transfer learning for data-limited target domains.
  • Transfer challenge: Source–target differences in meteorology, emissions, topography, and monitoring environments can cause negative transfer during conventional fine-tuning.Source-specific characteristics may be only partly relevant to the target domain.
  • Transfer challenge: A shared latent representation may suppress target-specific information because pollutant persistence can transfer while local meteorological and geographical processes remain region-specific.The introduction motivates retaining transferable knowledge while learning locally relevant patterns from target data.
  • Framework: The proposed shift-aware dual-encoder framework maintains separate source-derived and target-specific representations and combines them through supervised fusion.The source encoder transfers temporal knowledge from a data-rich U.S. domain, while the target encoder captures Taiwan-specific relationships.
  • Study objectives: The study compares transfer strategies, conducts component-level ablations, evaluates multiple predictive behaviors, and tests whether source representations should adapt under target supervision.These objectives assess target-only learning, direct transfer, dual-encoder fusion, encoder contributions, staged optimization, and interpretability.

2 Materials and Methods · 2.1 Study Area and Datasets · 2.2 Data Preprocessing and Sequence Construction

The study combines U.S. source observations for source-encoder pretraining with two years of hourly Taiwan observations from 77 monitoring stations. Taiwan samples use 24-hour causal input sequences for one-step-ahead PM₂.₅ prediction and chronological, leakage-preventing data partitioning.

  • 2.1 Study Area and Datasets: The Taiwan target dataset comprises hourly observations from 77 regulatory monitoring stations collected between January 1, 2018, and December 31, 2019.The stations provide broad spatial coverage across the island rather than representing a single monitoring location.
  • 2.1 Study Area and Datasets: The final predictor set includes 15 pollutant and meteorological variables, including PM₂.₅, PM₁₀, gases, rainfall, humidity, wind direction, and wind speed.Before imputation, the reshaped hourly dataset contained 0.1227% missing values.
  • 2.1 Study Area and Datasets: The source domain contains hourly PM₂.₅ forecasting data from 10 U.S. monitoring locations, primarily in Florida and nearby urban and coastal environments.The source data span April 17, 2020, to December 31, 2023, and are used exclusively to pretrain the source encoder.
  • 2.2 Data Preprocessing and Sequence Construction: U.S. source observations were resampled to hourly averages, separated into continuous segments at temporal gaps, and converted into memory-mapped sequence arrays.The resulting source sequence dataset contains 8,336 input-target pairs.
  • 2.2 Data Preprocessing and Sequence Construction: Feature normalization used MinMax scaling fitted exclusively on the training subset, with the resulting transformation applied to validation and test data.This procedure keeps scaling parameters restricted to the training data.

2.3 Problem Formulation

The problem concerns one-step-ahead PM2.5 forecasting with limited target supervision and distribution shift across space and time. It formulates selective transfer as a dual-encoder approach that preserves target-specific information while reusing source knowledge.

  • 2.3.1 Forecasting Task: The task forecasts future PM2.5 concentration from multivariate observations under limited reliable target supervision and distribution shift.
  • 2.3.1 Forecasting Task: The experimental target data comprise two years of hourly multi-station Taiwan observations, with validation and test samples occurring after training to prevent temporal leakage.
  • 2.3.1 Forecasting Task: The forecasting setup uses an hourly sampling interval, a look-back window of L= 24, and a forecasting horizon of H= 1.
  • 2.3.2 Distribution Shift and Selective Representation Transfer: Air-quality distributions vary across domains because of climatology, station location, regional conditions, emission mixtures, aerosol chemistry, and boundary-layer behavior.
  • 2.3.2 Distribution Shift and Selective Representation Transfer: Temporal covariate shift can also occur within the target domain, making direct transfer unreliable.
  • 2.3.2 Distribution Shift and Selective Representation Transfer: The practical objective is selective reuse of source knowledge rather than indiscriminate transfer of all source information.
  • 2.3.2 Distribution Shift and Selective Representation Transfer: The dual-encoder fusion model combines target-side and source-side latent representations to address domain differences.
  • 2.3.2 Distribution Shift and Selective Representation Transfer: Freezing the source encoder preserves a stable source prior, whereas adaptive fine-tuning can correct source-domain bias between U.S. and Taiwan.

2.4 Proposed Shift-Aware Dual-Encoder Framework

The proposed framework transfers temporal knowledge through a source encoder while preserving target-specific dynamics through a separate target encoder. The two representations remain separate during feature extraction and are fused only at the prediction stage.

  • Dual-encoder architecture: The framework processes target inputs through complementary source and target branches that capture transferable temporal patterns and domain-specific dynamics.A supervised fusion head combines the resulting latent representations to generate the PM₂.₅ forecast.
  • Source encoder: The source branch uses an LSTM pretrained on the source dataset and remains frozen during main target-domain training.An adaptive-source variant instead updates the pretrained encoder during target adaptation.
  • Target encoder: The target branch is initialized from an LSTM trained exclusively on Taiwan observations and optimized to learn target-specific temporal relationships.These dynamics may not be captured by the transferred source representation.
  • Input adaptation and fusion: A trainable adapter maps Taiwan inputs into the pretrained source model’s feature space, while the original and adapted sequences follow parallel target and source paths.MMD-based latent alignment is applied during training, and an adaptive fusion gate combines the branch representations for forecasting at t + H.
  • Representation fusion: The architecture keeps source and target representations separate during feature extraction and concatenates them only before the prediction head.This preserves transferable information without losing target-specific temporal characteristics.

2.5 Training Strategy

The fusion framework uses staged optimization to preserve pretrained source knowledge while gradually adapting the Taiwan-specific encoder. Ablation variants isolate the contributions of each branch, staged unfreezing, and source-encoder trainability.

  • Training objective: The training objective combines prediction loss with L2 weight regularization.
  • Ablation design: Four additional variants remove the source branch, remove the Taiwan branch, replace staged unfreezing with simultaneous optimization, or allow source-encoder updates.These variants assess the source branch, target branch, staged optimization strategy, and source-encoder trainability.
  • Staged optimization: Training first optimizes the fusion head with both encoders frozen, then progressively unfreezes and fine-tunes the Taiwan encoder.The final Taiwan LSTM layer is unfrozen first, followed by both Taiwan LSTM layers at a lower learning rate.
  • Source-encoder adaptation: The primary configuration keeps the source encoder frozen, allowing it to update only in the Trainable Source Encoder ablation.This preserves the pretrained source representation during primary adaptation while permitting source-side updates in the ablation.

2.6 Validation and Experimental Design · 2.7 Evaluation Metrics

The study compared progressively sophisticated transfer-learning strategies and architectural ablations under an identical chronological Taiwan evaluation protocol. Performance was assessed with four regression metrics alongside diagnostic analyses of pollution-level behavior, prediction distributions, residuals, and interpretability.

  • 2.6 Validation and Experimental Design: Four forecasting strategies ranged from the Taiwan-only TW-Scratch baseline to direct transfer, staged fine-tuning, and the proposed Fusion/Frozen TG-DEF model.TW-Scratch used a two-layer LSTM trained exclusively on Taiwan data; TL-v1 used direct U.S.-pretrained transfer, TL-v2 added an adapter and staged unfreezing, and Fusion/Frozen TG-DEF combined frozen source and adapter components with a Taiwan-specific encoder and supervised fusion head.
  • 2.6 Validation and Experimental Design: Four ablation variants isolated the effects of removing the source branch, removing the Taiwan branch, replacing staged optimization, and keeping the source encoder trainable.The ablations were designed to isolate individual architectural and optimization components.
  • 2.6 Validation and Experimental Design: All models used the same Taiwan sequence dataset, chronological train–validation–test partition, look-back window, forecasting horizon, feature dimensionality, and fixed random seed.Performance was assessed on the pooled Taiwan test set under an identical evaluation protocol.
  • 2.7 Evaluation Metrics: Point forecasts were evaluated using Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and the coefficient of determination (R²).These were the four adopted regression metrics.
  • 2.7 Evaluation Metrics: Average error metrics may conceal performance differences across pollution levels, motivating additional diagnostic analyses.The evaluation therefore examined behavior beyond aggregate point-forecast errors.
  • 2.7 Evaluation Metrics: Diagnostic analyses included quantile-stratified MAE, observed-versus-predicted density plots, and residual distribution analysis.These analyses addressed performance across pollution quantiles, prediction distributions, and residual behavior.
  • 2.7 Evaluation Metrics: SHAP-based interpretability was used to examine predictive behavior and physical consistency.SHAP analysis formed part of the additional diagnostic evaluation.

2.8 Implementation and Training Details · 2.9 Reproducibility

The study specifies its TensorFlow/Keras training configurations, staged optimization settings, and reproducibility procedures. Experiments used fixed-seed chronological evaluation, with preprocessing fitted only on training data to prevent information leakage.

  • 2.8 Implementation and Training Details: Source models used L = 24, H = 1, Adam optimization, batch size 128, maximum 12 epochs, and early stopping patience four.These settings define the source-model training protocol.
  • 2.8 Implementation and Training Details: TL-v2 used batch size 256, patience five epochs, and staged learning rates of 10^-3, 5×10^-4, and 10^-4.Its adapter network had hidden dimension 32, dropout 0.10, and sigmoid activation.
  • 2.8 Implementation and Training Details: TG-DEF and its ablation variants used batch size 256 and early stopping patience four epochs.When staged optimization was used, training proceeded through three consecutive stages.
  • 2.8 Implementation and Training Details: Staged TG-DEF optimization used 12, 10, and 10 epochs with learning rates of 10^-3, 3×10^-4, and 10^-4, respectively.The fusion head used 64 hidden units, dropout 0.20, and a linear output layer.
  • 2.9 Reproducibility: Implementation used TensorFlow/Keras and NumPy memmap arrays for efficient sequence storage and processing.The study also reports dataset definitions, temporal coverage, feature sets, sequence construction, partitions, hyperparameters, and metrics.
  • 2.9 Reproducibility: MinMax scaling was fitted exclusively on training data and applied unchanged to validation and test sets to prevent information leakage.SHAP was used for post hoc interpretation of trained models.
  • 2.9 Reproducibility: All experiments used a fixed random seed of 42, so reported results reflect fixed-seed chronological evaluations rather than multi-seed averages or confidence intervals.The study explicitly limits interpretation of the reported results in this way.

3 Results

Frozen-source fusion outperformed direct-transfer baselines on the chronological Taiwan test set, while ablations showed that target-specific information was essential and source adaptation yielded the best aggregate performance. Error-distribution analyses and SHAP attributions further characterized model behavior across concentrations and predictors.

  • Aggregate comparison: MSE = 21.8960, MAE = 3.1597, and R² = 0.8725 were the best principal-baseline results for frozen-source TG-DEF.Its MSE was approximately 7.1% lower than TL-v1 and 4.1% lower than TL-v2; results came from a single fixed-seed run.
  • Prediction agreement: The fusion model showed tighter concentration around the identity line and fewer deviations at higher PM₂.₅ concentrations than target-only and direct-transfer models.TW-Scratch showed range compression, while TL-v1 and TL-v2 showed greater off-diagonal dispersion.
  • Residual analysis: Direct-transfer baselines had broader residual distributions and heavier tails, whereas TG-DEF residuals were more tightly centered around zero.This pattern indicates fewer large overpredictions or underpredictions, although residual analysis cannot determine the improvement mechanism.
  • Concentration-stratified error: Upper-quantile errors increased for all models, but frozen-source TG-DEF maintained lower MAE than direct-transfer baselines at the highest concentration levels.The fusion model therefore showed lower upper-tail error than the direct-transfer baselines.
  • Interpretability: Recent lagged PM₂.₅ values dominated SHAP importance, while wind-related features and relative humidity also showed important or nonlinear relationships with predictions.SHAP was used to assess consistency with physical understanding, not to establish direct causal effects.
  • Ablation analysis: MSE rose from 21.8960 to 23.6055 and R² fell from 0.8725 to 0.8626 when the Taiwan-specific branch was removed.The trainable-source variant achieved the best aggregate performance: MSE = 21.6575, MAE = 3.1383, and R² = 0.8739.

4 Discussion

The discussion concludes that dual-encoder transfer is most effective when target-specific representations are preserved and source knowledge adapts to domain shift. It also identifies deployment limitations and directions for broader, uncertainty-aware evaluation.

  • Transfer effectiveness: Dual-encoder fusion outperformed conventional direct transfer under the investigated U.S.–Taiwan domain shift, with frozen-source TG-DEF strongest among principal baselines.The trainable-source ablation produced the best overall aggregate result.
  • Transfer effectiveness: Removing the Taiwan-specific branch caused the largest accuracy reduction, emphasizing the importance of preserving target-domain information during transfer.The findings support combining target-specific representation learning with source knowledge that remains adaptable during target-domain training.
  • Mechanistic interpretation: The framework preserves separate source-derived and target-specific representations and learns their complementary contributions during supervised adaptation.Adaptive updating accommodates source–target mismatch by allowing transferred features to adjust to the target distribution.
  • Practical implications: Adaptive transfer is more suitable than rigid knowledge reuse when source and target domains differ, but operational deployment requires continuous performance monitoring.Changing meteorological conditions, emission patterns, and other environmental drivers may reduce predictive accuracy over time.
  • Limitations: The findings are limited by a single random seed, pooled test data, non-overlapping regional and temporal datasets, added computational cost, and deterministic one-hour-ahead evaluation.The evaluation also excludes uncertainty calibration and comprehensive tail-specific comparisons across ablation variants.
  • Future research: Future work should test broader source-target settings and pollution regimes, incorporate probabilistic forecasting and continual adaptation, and evaluate flexible fusion and robustness under extreme conditions.Suggested extensions include quantile regression or conformal prediction, attention-based or gated fusion, larger environmental foundation models, multi-seed experiments, and tail-focused evaluation.

5 Conclusion

The study presents an adaptive dual-encoder framework that combines source-derived and target-specific representations for PM₂.₅ forecasting under source-target mismatch and limited target data. Results indicate that preserving target information and adapting transferred knowledge improve transfer effectiveness.

  • 5 Conclusion: The framework combines source-derived and target-specific representations to address limited target-domain data while retaining information unique to the target environment.It is designed for short-horizon PM₂.₅ forecasting under source-target mismatch.
  • 5 Conclusion: The frozen-source TG-DEF model outperformed target-only and direct-transfer baselines, while the trainable-source ablation achieved the best overall aggregate performance.These results favor dual-encoder transfer over conventional alternatives and support source-encoder adaptation.
  • 5 Conclusion: Removing the Taiwan-side branch caused the largest decline in predictive accuracy, underscoring the importance of preserving target-domain information during transfer.The ablation isolates the target-specific branch as the most consequential component.
  • 5 Conclusion: Allowing the source encoder to adapt during target-domain training produced the best overall performance, whereas fully freezing the transferred representation was less effective.Transfer effectiveness therefore depends on how source knowledge is integrated with and adapted to the target domain.
  • 5 Conclusion: The framework offers a practical approach to PM₂.₅ forecasting in data-limited settings where source-target distribution differences limit conventional transfer methods.Its stated application is forecasting under distributional mismatch and constrained target observations.

Data and Code Availability

The Taiwan and U.S. air-quality observations came from publicly available monitoring sources and were processed into the sequence datasets used in the study. Processed data, trained-model metadata, and analysis code are available from the corresponding author upon reasonable request, subject to provider terms, pending stable repository arrangements.

  • Taiwan and U.S. air-quality observations were obtained from publicly available monitoring sources and processed into the paper’s sequence datasets.
  • A permanent repository link should be included before final publication if the materials are deposited in a public repository, because stable links and redistribution permissions are not yet established.
  • Processed data, trained-model metadata, and analysis code are available from the corresponding author upon reasonable request, subject to the original data providers’ terms.
Loading 2608.14456v1…