PaperlayerView exact revision on arXiv

Source-linked AI summary

Initialization Is Critical: Advancing Federated Short-Term Load Forecasting under Load Heterogeneity via Model Initialization

Jianing Chen, Vajiheh Farhadi, Yan Li, Thomas La Porta

arXiv:2608.27791v1cs.LGeess.SY

TL;DR

Federated STLF protects raw smart-meter data but remains vulnerable to structured client heterogeneity, including differing exogenous responses and temporal load profiles. The paper introduces pretrained global initialization and SLIAvg sequential local initialization, and experiments show reduced client drift, improved convergence, and lower forecasting errors.

  • Problem

    Structured heterogeneity in clients’ load data, including different exogenous responses and temporal load profiles, can degrade federated STLF forecasting performance.

  • Method

    The paper uses auxiliary public load data for pretrained global initialization and proposes SLIAvg for progressively adapted local initialization within communication rounds.

  • Results

    Experiments on real load data with two representative forecasting models show reduced client drift, improved convergence behavior, and better predictive performance.

  • Takeaways & Limitations

    Model initialization provides a global and local strategy for promoting more consistent federated updates in heterogeneous STLF.

Abstract

from arXiv · show

Short-term load forecasting (STLF) provides essential information for numerous applications in modern power systems. However, accurate STLF often relies on fine-grained smart-meter data from distributed users, raising increasing concerns about data privacy. Federated learning (FL) has therefore emerged as a promising privacy-preserving paradigm for STLF. Nevertheless, this paper reveals structured heterogeneity in clients' load data. Specifically, clients exhibit different responses to exogenous factors and distinct temporal load profiles, which can degrade forecasting performance in FL. To mitigate these issues, this paper studies the role of model initialization in federated STLF, and proposes two initialization strategies from global and local perspectives. For global model initialization, when auxiliary public load data are available, a pretrained initialization strategy is developed to initialize the global model before federated training, thereby reducing client drift during the training process. For local model initialization, we propose SLIAvg, a sequential local initialization strategy that promotes a more consistent training process by allowing participating clients to start from progressively adapted models within each communication round. Since the proposed strategies only modify the initialization process, they are compatible with most existing FL frameworks and privacy-enhancing techniques. Experiments on real smart-meter data with two representative forecasting architectures demonstrate that the proposed strategies effectively improve forecasting performance, as evidenced by reduced client drift, improved convergence behavior, and lower forecasting errors.

I. INTRODUCTION

Federated learning enables privacy-preserving STLF, but structured differences in users’ load data can make local updates inconsistent and degrade forecasting. This paper analyzes those differences and proposes global pretrained and local sequential initialization strategies to improve federated training.

  • Federated learning trains a global forecasting model from distributed local updates without centralizing raw load data.
  • Data heterogeneity remains a major federated STLF challenge because users differ in consumption behavior, responses to exogenous factors, and temporal load profiles.
  • Random global initialization can produce inconsistent early local updates that amplify through aggregation and lead to severe client drift.
  • Global model initialization: Pretrained initialization uses auxiliary public load data to initialize the global model before federated training, reducing client drift and improving forecasting accuracy.
  • Local model initialization: SLIAvg progressively adapts participating clients’ initial models within each communication round to promote smoother cross-client transitions and more consistent local updates.
  • Evaluation and compatibility: Experiments on real smart-meter data with two forecasting architectures evaluate the strategies alongside existing federated STLF methods and privacy-enhancing techniques.The initialization changes are compatible with advanced architectures, aggregation rules, personalization methods, secure aggregation, and differential privacy.

III. STRUCTURED LOAD HETEROGENEITY AND CLIENT DRIFT IN FEDERATED STLF

Federated STLF data exhibit structured non-IID behavior beyond conventional distribution skew: users respond differently to exogenous factors and have different temporal load patterns. These differences create divergent local objectives and client drift, complicating global optimization.

  • Conventional federated heterogeneity is described through feature-distribution or target-distribution skew, which can create discrepancies among users’ local objectives.
  • STLF exhibits two structured and persistent heterogeneity forms: heterogeneous responses to exogenous factors and temporal heterogeneity across users.
  • Heterogeneous responses to exogenous factors: Users can respond differently to the same exogenous inputs because their underlying load-generation mechanisms differ.
  • Heterogeneous responses to exogenous factors: A linear response model is fitted using each user’s local data to quantify heterogeneous responses to exogenous factors.
  • Heterogeneous responses to exogenous factors: User-specific coefficients βk and δk,c quantify average sensitivity to continuous exogenous variables and average load shifts relative to a reference weather category.

2) Temporal heterogeneity:

Temporal heterogeneity reflects user-specific differences in load timing and evolution despite shared short-term periodicity, and MMD measures these distributional differences across users.

  • Users may share overall short-term periodicity while differing in peak timing, transition duration, and stochastic load-response timing.
  • A user’s temporal load profile is represented through temporal load windows of length L describing its evolution up to time t.
  • Maximum mean discrepancy (MMD) compares distributions of temporal load windows across users to quantify temporal heterogeneity.
  • Larger MMD values indicate greater discrepancy between users’ temporal load patterns, including differences in peaks, transitions, and stochasticity.
  • Temporal and exogenous-response heterogeneity jointly produce structured non-IID behavior, creating discrepancies among clients’ local objectives that underpin client drift.

B. Client Drift in Federated STLF

Client drift is the deviation of heterogeneous local updates from the global optimization direction, causing aggregated FedAvg trajectories to diverge from centralized optimization and harming the final model.

  • Client drift is the systematic deviation of local model updates from the global optimization direction, making accurate predictive-model training difficult.
  • Under severe drift, persistent differences between local and global gradients make aggregated updates misaligned with the global objective’s descent direction.
  • In FedAvg, clients start from one server model, perform multiple private-data updates, and return models for server-side aggregation.
  • With heterogeneous objectives, local updates move toward different client-specific optima, so aggregation can follow an averaged local solution rather than the centralized optimum.
  • FedAvg and variants such as FedAdam remain insufficient under heterogeneous data, where biased aggregated updates can degrade the final global model.

IV. ADVANCING FEDERATED STLF VIA MODEL INITIALIZATION UNDER LOAD HETEROGENEITY

The paper addresses client drift through model initialization at both global and local levels, using pretraining when public load data exist and drift-aware initialization otherwise.

  • Model initialization is proposed as a way to alleviate client drift through pretrained global initialization and sequential local initialization.
  • Random initialization can place the model in an uninformative optimization region where heterogeneous clients follow substantially different descent directions.
  • The client-drift measure Dloc quantifies divergence of local models around their weighted average after one communication round.
  • When auxiliary public load data are available, pretraining captures common temporal dynamics and supplies a more informative initial global model.
  • The pretrained strategy changes only the initial global parameters, without adding communication rounds or changing the client-side training protocol.

B. Sequential Local Initialization without Pretraining Data

SLIAvg sequentially passes progressively adapted local models among randomly ordered clients, then aggregates all resulting models to reduce order bias and terminal-endpoint dominance.

  • Sequential local initialization without pretraining data: SLIAvg forwards each client’s locally updated model as the next participating client’s initialization through server coordination.
  • Sequential local initialization without pretraining data: Progressive adaptation carries earlier representations forward, smooths transitions among local updates, and is expected to improve consistency across heterogeneous clients.
  • FedAvg Aggregation for Avoiding Terminal-Endpoint Dominance: The proposed procedure aggregates the resulting local models after all selected clients complete local training.
  • Randomized Client Ordering: SLIAvg randomly shuffles participating clients each round because sequential initialization makes client position affect its starting model.
  • FedAvg Aggregation for Avoiding Terminal-Endpoint Dominance: Using only the last local model as the next global model would increasingly favor later clients and dilute representations learned from earlier clients.
  • FedAvg Aggregation for Avoiding Terminal-Endpoint Dominance: FedAvg aggregation combines all sequentially trained local models, assigning relatively larger weights to earlier updates to mitigate terminal-endpoint dominance.

V. NUMERICAL EXAMPLES

The numerical experiments use the Low Carbon London smart-meter dataset and a federated short-term load forecasting setup with half-hourly observations and sliding-window prediction.

  • Experiments use the public Low Carbon London dataset containing smart-meter measurements and detailed weather information.
  • The federated setup includes K clients, participating-client sampling, communication rounds, local steps, local sample sizes, and server-side aggregation.
  • Load data are recorded at half-hour resolution, yielding 48 observations per day.
  • Thirty clients are selected, with training data from July 1, 2013, to January 15, 2014, and testing data from January 16 to February 16, 2014.
  • A sliding window of 48 half-hourly observations predicts the current load from the preceding day.
  • Experiments are implemented in Python 3.12 and PyTorch 2.11 on a workstation with two NVIDIA L40S GPUs.

A. Data Heterogeneity Analysis

The adopted load dataset exhibits structured client heterogeneity in both responses to exogenous weather factors and temporal load-profile distributions.

  • Exogenous-factor heterogeneity: Client-specific responses to continuous and categorical weather factors vary across users under shared exogenous conditions.
  • Exogenous-factor heterogeneity: Temperature and dew-point response coefficients show especially clear cross-client variability.
  • Exogenous-factor heterogeneity: Negative temperature coefficients indicate that lower temperature is generally associated with higher electricity consumption, consistent with heating-driven demand in London.
  • Exogenous-factor heterogeneity: Categorical weather coefficients show that clients experience different load shifts under the same weather condition relative to clear-day conditions.
  • Temporal heterogeneity: The pairwise MMD heatmap contains large values for several client pairs, indicating distinct temporal load behaviors.

B. Comparisons of Different Training Strategies

The comparison evaluates pretrained and sequential-initialization strategies using two forecasting architectures, controlled randomness, and MSE and MAE test metrics.

  • Experimental settings: The experiments compare a Transformer and a dual-channel LSTM to assess whether the proposed strategies generalize across forecasting architectures.
  • Experimental settings: For the Transformer, exogenous features enter the encoder and historical load enters the decoder, while the LSTM uses separate channels whose representations are fused.
  • Experimental settings: The random seed is fixed to 42 for weight initialization, data shuffling, and client sampling, reducing stochastic variation between comparable runs.
  • Evaluation metrics: Performance is reported using mean squared error and mean absolute error, based on the best test performance achieved during training.

1) Evaluation of Pretrained Initialization with Auxiliary Public Load Data:

Pretrained global initialization and sequential local initialization are evaluated against federated baselines, with results linking initialization to forecasting accuracy, convergence, and client drift.

  • Pretrained initialization: Pretraining improves best MSE/MAE from 0.1363/0.1993 to 0.0953/0.1658 for Transformer and from 0.0897/0.1599 to 0.0753/0.1496 for LSTM models.
  • Pretrained initialization: When auxiliary pretraining data exclude federated-training data, increasing pretraining beyond five epochs can degrade performance, with both models worsening at 20 epochs.
  • Pretrained initialization: Pretrained initialization accelerates convergence during early communication rounds and achieves better forecasting performance.
  • Client drift: Pretrained initialization produces lower client drift than random initialization in most later rounds, especially during the middle training stage.
  • Sequential local initialization: SLIAvg sequentially initializes local models and averages their resulting models to form the next global model.
  • Sequential local initialization: SLIAvg reduces overall MSE and MAE from 0.0720 and 0.1491 with FedAvg to 0.0633 and 0.1375, respectively.

C. Privacy Discussion

The proposed initialization strategies remain compatible with federated learning’s privacy-preserving setting when auxiliary data are privacy-compliant and raw client data remain local. SLIAvg trades additional training latency for forecasting accuracy and remains compatible with privacy-enhancing techniques.

  • Pretrained Initialization: Auxiliary load data for pretrained initialization must be anonymized and authorized when contributed by users.The auxiliary data are assumed to be privacy-compliant.
  • SLIAvg Privacy: Raw load data remain local throughout SLIAvg training, preserving its privacy-compatible federated setting.Additional privacy-enhancing techniques can be incorporated when formal privacy guarantees are required.
  • SLIAvg Cost: SLIAvg introduces extra training latency because initialization models are synchronized sequentially through the server.The added training time becomes more pronounced as more clients participate in a round.
  • SLIAvg Cost: The latency–accuracy trade-off is considered justified in STLF when forecasting accuracy is prioritized over training speed.This cost does not alter FL’s decentralized paradigm.
  • Future Work: Future work will target reducing SLIAvg training latency while preserving its benefits.
Loading 2608.27791v1…