PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Importance and Applicability of Pre-Training for Federated Learning

Hong-You Chen, Cheng-Hao Tu, Ziwei Li, Han-Wei Shen, Wei-Lun Chao

arXiv:2206.11488v3cs.LGcs.AIcs.CV

TL;DR

Federated learning often uses random initialization, while pre-training’s applicability and effects in FL—particularly under non-IID client data—remain questions for study. The paper systematically evaluates pre-training and alternatives using synthetic data or clients’ data. Across visual recognition benchmarks, pre-training improves FL and narrows its accuracy gap with centralized learning, while making aggregation more stable but not alleviating local model drifting.

  • Problem

    The effects of pre-training in FL, where networks are mostly randomly initialized and non-IID data can reduce FEDAVG accuracy, remain underexplored.

    Source: §1 · p. 1

  • Method

    The paper systematically studies pre-training for FL across visual recognition tasks, including pre-training with synthetic data and clients’ data.

    Source: §1 · p. 2

  • Results

    Across visual recognition tasks, pre-training improves FEDAVG and largely closes its accuracy gap with centralized learning, especially under more challenging non-IID conditions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pre-training makes global aggregation more stable and enables global models under different client-data conditions to converge to the same loss basin.

    Source: §1 · p. 2, §6 · p. 9

    Pre-training seems not to alleviate local model drifting.

    Source: §6 · p. 8

Abstract

from arXiv · show

Pre-training is prevalent in nowadays deep learning to improve the learned model's performance. However, in the literature on federated learning (FL), neural networks are mostly initialized with random weights. These attract our interest in conducting a systematic study to explore pre-training for FL. Across multiple visual recognition benchmarks, we found that pre-training can not only improve FL, but also close its accuracy gap to the counterpart centralized learning, especially in the challenging cases of non-IID clients' data. To make our findings applicable to situations where pre-trained models are not directly available, we explore pre-training with synthetic data or even with clients' data in a decentralized manner, and found that they can already improve FL notably. Interestingly, many of the techniques we explore are complementary to each other to further boost the performance, and we view this as a critical result toward scaling up deep FL for real-world applications. We conclude our paper with an attempt to understand the effect of pre-training on FL. We found that pre-training enables the learned global models under different clients' data conditions to converge to the same loss basin, and makes global aggregation in FL more stable. Nevertheless, pre-training seems to not alleviate local model drifting, a fundamental problem in FL under non-IID data.

1 INTRODUCTION

The paper studies model initialization, a rarely explored dimension of federated learning, asking whether pre-training improves FEDAVG, particularly with non-IID client data. Across visual recognition tasks, pre-training improves FEDAVG and largely closes its accuracy gap with centralized learning; the paper also explores synthetic and client-data pre-training and analyzes its effects.

1 INTRODUCTION

  • The study systematically examines pre-training for FL using visual recognition as its running example and multiple application scenarios.Its scope includes image classification and semantic segmentation on Cityscapes.Source: §1 · pp. 1–2
  • Pre-training consistently improves FEDAVG across visual recognition tasks, with larger relative gains under more challenging non-IID settings and a largely closed accuracy gap with centralized learning.The paper also reports that pre-training improves advanced FL methods, while diminishing their gain over FEDAVG.Source: §1 · p. 2, Figure 1 · p. 2
  • Pre-training with synthetic data, including the proposed Fractal Pair Similarity scheme, improves FEDAVG even when real server data are unavailable.The approach is intended for settings where pre-trained models are also unavailable.Source: §1 · p. 2
  • The study tests self-supervised pre-training on clients’ data followed by supervised learning in a federated setting, finding it effective and compatible with available pre-trained weights.This combination further boosts performance.Source: §1 · p. 2
  • Pre-training does not seem to alleviate local model drifting, but makes global aggregation more stable and enables global models under different data conditions to converge to the same loss basin.The paper identifies reduced sensitivity to aggregation coefficients as one reason aggregation yields a stronger global model.Source: §1 · p. 2

2 RELATED WORK

Prior FL research has sought to improve FEDAVG through changes to global aggregation and local training, especially for non-IID data. Pre-training is common in other domains but its effects on FL remain largely unexplored, motivating this study of initialization.

2 RELATED WORK

  • FL methods have modified global aggregation and local training to address FEDAVG’s accuracy drop under non-IID data.Approaches include model matching, ensembles, distillation, server optimization, regularization, and corrected local updates.Source: §2 · pp. 2–3
  • Few FL studies examine initialization, and the cited prior uses of pre-trained models did not analyze their effects systematically.The paper characterizes its own study as more systematic and comprehensive.Source: §2 · p. 3
  • Although pre-training has been widely studied across domains, its effects on FL remain largely unexplored.Prior analyses examined effects such as robustness, loss landscapes, and forgetting in other settings.Source: §2 · p. 3

3 BACKGROUND: FEDERATED LEARNING

Federated learning distributes training data across clients, so the centralized empirical-risk objective cannot be solved directly. FEDAVG alternates local training with server aggregation, while client updates can drift apart and reduce accuracy.

3 BACKGROUND: FEDERATED LEARNING

  • The data are partitioned among clients, and the global objective is empirical risk over their aggregated training set.Each client’s local empirical risk is defined on its own data.Source: §3 · p. 3
  • FEDAVG relaxes the centralized objective by iterating parallel client-side local training and server-side aggregation.Aggregation averages local model parameters element-wise.Source: §3 · p. 3
  • When local models drift away from one another and the previous global model, the aggregated model can deviate from the centralized objective’s solution and suffer an accuracy drop.Local training aims to minimize each client’s empirical risk, often using several epochs of SGD.Source: §3 · p. 3

4 PRE-TRAINING IS APPLICABLE TO AND IMPORTANT FOR FL

The study examines pre-training across settings where real pre-trained weights or data may be available and settings where they are not. Pre-training improves FEDAVG, with larger gains in more challenging conditions and reduced accuracy gaps to centralized learning.

4.1 PRE-TRAINING SCENARIOS IN THE CONTEXT OF FL

  • The paper studies pre-training both when a server has real weights or data and when it has neither, using synthetic data in the latter scenario.The authors frame this as relevant to data-scarce, costly, or privacy-critical domains.Source: §4.1 · p. 4

4.2 EXPERIMENTAL SETUP AND IMPLEMENTATION DETAILS

  • Experiments cover four image-classification datasets and Cityscapes segmentation, using random or pre-trained initialization and task-appropriate models and metrics.The setup also describes synthetic fractal pre-training and the FEDAVG training protocol.Source: §4.2 · p. 5

4.3 EMPIRICAL STUDY ON CIFAR-10 AND TINY-IMAGENET

  • Pre-training improves FEDAVG and narrows its accuracy gap to centralized learning; on CIFAR-10 the gap falls from 9.0 without pre-training to 2.5 with ImageNet-pre-trained weights.The comparison uses α = 0.3; pre-training also improves centralized learning, but the reported gap reduction indicates greater benefits for FL.Source: §4.3 · p. 6, Table 2 · p. 6, Table 3 · p. 6
  • Pre-training yields robust gains across federated settings, with larger gains at smaller α or more clients, and helps larger models benefit from increased depth and width.Figure 3 varies one setting at a time from 10 clients, α = 0.3, full participation, and 5 local epochs; Figure 5 studies ResNet depth and width on CIFAR-10 at α = 0.3.Source: §4.3 · pp. 6–7, Figure 3 · p. 6, Figure 5 · p. 7
  • Real and synthetic pre-training consistently boost FedProx, FedDyn, and MOON; with real pre-training, these methods and FEDAVG perform similarly.The comparison uses the default federated setting.Source: §4.3 · p. 6, Figure 4 · p. 6

4.4 EMPIRICAL STUDY ON LARGE-SCALE AND REAL DATA SPLITS

  • On realistic non-IID splits, FPS reduces the centralized–federated gap and improves FEDAVG by > 5% on iNaturalist and > 20% on Cityscapes.The reported FPS gains exceed those of centralized learning.Source: §4.4 · p. 7, Table 4 · p. 7, Table 5 · p. 7

4.5 ON PRE-TRAINING WITH SYNTHETIC DATA

  • FPS outperforms the compared synthetic pre-training baselines, using same-code fractal images as positive pairs in similarity learning.The authors attribute its effectiveness to geometrically meaningful pairs; multi-label fractal training does not outperform random initialization.Source: §4.1 · p. 4, Figure 2 · p. 4, §4.5 · p. 7
  • On chest X-ray diagnosis, FPS achieves 74.8% test accuracy versus 69.1% with ImageNet pre-training, a reported 5.9% advantage for synthetic pre-training.Random, FPS, and ImageNet initialization achieve 68.9%, 74.8%, and 69.1%, respectively; this illustrates potential usefulness when accessible pre-training data come from a distant domain.Source: §4.5 · p. 7

5 FEDERATED SELF-PRE-TRAINING CAN IMPROVE FL

The paper applies federated self-supervised pre-training on clients’ decentralized data before supervised FEDAVG. This procedure improves FEDAVG from random initialization without external data and can also improve on pre-trained weights.

5 FEDERATED SELF-PRE-TRAINING CAN IMPROVE FL

  • Federated self-pre-training first applies self-supervised learning to clients’ unlabeled data, then runs standard FEDAVG; it can start from random or pre-trained weights.The self-supervised stage uses a federated algorithm for 50 rounds.Source: §5 · p. 8
  • Self-pre-training significantly improves randomly initialized FEDAVG without external data, and the gain is not explained simply by running 50 extra FEDAVG rounds.The extended FEDAVG baseline without self-pre-training did not improve.Source: §5 · p. 8, Table 7 · p. 8
  • Self-pre-training can also improve FEDAVG initialized with pre-trained weights.The authors present this result as further evidence for the importance of initialization.Source: §5 · p. 8, Table 7 · p. 8

6 AN ATTEMPT TO UNDERSTAND THE EFFECTS, AND CONCLUSION

This section analyzes how real-data pre-training changes FedAvg’s training dynamics and aggregation. It concludes that pre-training largely bridges the accuracy gap between federated and centralized learning.

Analysis on the training dynamics

  • The accuracy decomposition separates initial performance, local-training gain, and aggregation gain; negative local gain reflects local models drifting from the previous global model.The decomposition is used to analyze FedAvg’s training dynamics.Source: §6 · p. 8
  • The training-dynamics analysis uses CIFAR-10 and Tiny-ImageNet with M = 10 and α = 0.3; Figure 6 summarizes Equation 3 values, showing test accuracy for each dataset and pre-training combination using global model θ̄(t).Source: §6 · p. 8
  • Pre-training does not appear to alleviate local model drifting, but it yields larger global-aggregation gains, especially in early rounds.Both FedAvg variants show notable negative local-training gains.Source: §6 · p. 8

Analysis on global aggregation

  • Across sampled convex combinations, pre-training produces a smaller 95% confidence interval for test loss, indicating a lower-variance aggregation loss surface.The authors say this helps explain the smaller accuracy gap between simple averaging and optimal convex aggregation.Source: Figure 8 · p. 9, §6 · p. 9
  • The authors measure aggregation robustness by comparing FedAvg’s data-size-weighted average with an accuracy-maximizing convex aggregation of local models.The convex weights are optimized over the simplex using SGD.Source: §6 · p. 9
  • The simple weight average trails optimal convex aggregation, but this accuracy gap is much smaller with pre-training.Figure 7 compares test accuracy for the global model and optimal convex aggregation across datasets and pre-training conditions.Source: Figure 7 · p. 9, §6 · p. 9

Analysis on the loss surface

  • Pre-training brings global models from different client-data conditions, including IID, into the same lower-loss basin; without it, models settle in isolated, often poor basins.The authors offer this as another explanation for why pre-training improves FL.Source: Figure 9 · p. 9, §6 · p. 9

Conclusion

  • The paper concludes that pre-training largely bridges the accuracy gap between FL and centralized learning and offers new insights into FL.The authors describe the study as a systematic investigation of pre-training for FL.Source: §6 · p. 9

A ADDITIONAL DETAILS OF FRACTAL PRE-TRAINING

Fractal Pair Similarity (FPS) replaces supervised labels tied to a limited set of fractal codes with self-supervised pairs generated from shared codes. The appendix details how the fractals and pairs are made and how SimSiam and MoCo-V2 use them for pre-training.

A.1.1 BACKGROUND: SUPERVISED FRACTAL PRE-TRAINING

  • Earlier supervised fractal pre-training treats an IFS code as a class label and trains on images rendered from sampled codes.IFS rendering repeatedly applies probability-weighted affine transformations to points; a code can produce geometrically similar but varied fractals.Source: §A.1.1 · pp. 17–18

A.1.2 OUR APPROACH: FRACTAL PAIR SIMILARITY (FPS)

  • FPS trades intra-class variation for code diversity by sampling multiple IFS codes per image and treating the resulting images as unlabeled examples.Because different code sets define different classes, the method can sample new code combinations on the fly.Source: §A.1.2 · p. 18
  • Each FPS positive pair uses the same IFS codes but different fractal shapes, colors, and placements, providing object-level variation alongside image augmentation.The method samples 100K codes in advance and uniformly selects I codes to generate each pair.Source: Figure 10 · p. 17, §A.1.2 · p. 18

A.2 SELF-SUPERVISED LEARNING APPROACHES

  • SimSiam learns similarity between paired-image features, whereas MoCo-V2 uses contrastive loss with negative keys and a reusable feature dictionary.MoCo-V2 also applies a momentum update to its key encoder for training stability.Source: §A.2 · pp. 18–19
  • The experiments pre-train ResNet-18 or ResNet-20 for 100 epochs, with one million generated image pairs per epoch for FPS.After pre-training, they retain the query encoder in MoCo-V2 and the encoder in SimSiam.Source: §A.2 · p. 19

B.1 DATASETS, FL SETTINGS, AND HYPERPARAMETERS

This appendix records the federated training setup, the procedure for finding convex client-weight combinations, and the configuration and scope caveat for federated self-pre-training.

B.1 DATASETS, FL SETTINGS, AND HYPERPARAMETERS

  • FEDAVG runs for 100 rounds with five local epochs and weight decay 1e−4, while image preprocessing and Cityscapes crop settings vary by dataset.The experiments reserve 2% of training data for validation-based hyperparameter selection.Source: §B.1 · pp. 19–20, Table 9 · p. 20

B.2 OPTIMAL CONVEX AGGREGATION

  • Optimal convex aggregation weights are selected by optimizing nonnegative, ℓ1-normalized client weights on the global test set from multiple initializations.The search uses SGD for 20 epochs, and returns the best test-accuracy result.Source: §B.2 · p. 20

B.3 FEDERATED SELF-PRE-TRAINING

  • Federated self-pre-training uses a clustering-based self-supervised method for 50 rounds before supervised FEDAVG, but preliminary trials hardly scaled to iNaturalist or Cityscapes.The authors also report that validation accuracy measured by k-nearest neighbors hardly improved.Source: §B.3 · p. 20

C.1 SCOPE AND MORE EXPERIMENTS IN NLP

The paper focuses on computer vision, including realistic and challenging iNaturalist and Cityscapes tasks, and adds two NLP experiments. On Sent140 and Shakespeare, the reported pattern is that pre-training helps FL more than centralized learning and largely closes their gap, without alleviating local-model drift.

C.1 SCOPE AND MORE EXPERIMENTS IN NLP

  • The study focuses on computer vision and extends beyond simple CIFAR classification to iNaturalist and Cityscapes, described as realistic and challenging even in centralized learning.The authors characterize results on these datasets using real or synthetic pre-training as valuable to the FL community.Source: §C.1 · pp. 20–21
  • To check whether the computer-vision observations also hold for NLP, the authors test pre-trained DistilBERT on Sent140 and an LSTM pre-trained on Penn Treebank for Shakespeare next-character prediction.Sent140 has 660K clients and 1.6M samples; the Shakespeare version used has 606 clients.Source: §C.1 · p. 21
  • On Sent140 and Shakespeare, pre-training helps FL more than centralized learning and largely closes their gap, even though local-model drift remains.Figure 11 is described as showing the same trend on both NLP tasks.Source: §C.1 · p. 21, Figure 11 · p. 21

C.2 MORE ANALYSIS ON GLOBAL AGGREGATION

Additional aggregation analyses find that pre-training lowers loss-surface variance without reducing local-model drift. The synthetic-pre-training comparison reports that FPS outperforms its baselines across datasets and self-supervised algorithms.

C.2 MORE ANALYSIS ON GLOBAL AGGREGATION

  • Pre-training produces a lower-variance loss surface but does not reduce local-model drift in the additional iNaturalist-GEO-3K and Cityscapes analysis.Figure 13 plots global-model test loss and a 95% confidence interval for test losses of sampled convex combinations.Source: §C.2 · p. 22, Figure 13 · p. 22

C.3 MORE COMPARISONS ON SYNTHETIC PRE-TRAINING

  • Across three datasets and two self-supervised algorithms, FPS consistently outperforms the synthetic-pre-training baselines.FPS plus SimSiam provides the strongest overall performance, and FPS plus MoCo-V2 shows a similar conclusion.Source: §C.3 · p. 22

C.4 DISCUSSIONS ON DIFFERENT FEDERATED SETTINGS

Across varied federated settings, pre-training improves aggregated global accuracy and is more robust to stronger non-IID data, while local client shift persists.

C.4 DISCUSSIONS ON DIFFERENT FEDERATED SETTINGS

  • Across learning-rate schedules, pre-training leaves client shift largely unchanged but improves aggregated global accuracy and consistently lowers loss variance.The main-text schedule decays the learning rate by 0.1 every 30 rounds; it performed better than decay-by-0.95-every-round and no-decay schedules.Source: §C.4 · pp. 22–23
  • With more clients, performance drops as smaller client datasets invite overfitting, but real-data and FPS pre-training still help; pre-training is also more robust to stronger non-IID data.Smaller Dirichlet α values correspond to more non-IID settings; random initialization degrades sharply there.Source: §C.4 · p. 23
  • Participation rate and local epochs are not very sensitive once sufficiently large (e.g., participation above 30% and more than five epochs), and changing epochs does not close the pre-training gap.Using either fewer or more local epochs does not close the gap between pre-training and random initialization.Source: §C.4 · p. 23

C.5 ANALYSIS WITH FPS

  • With FPS, pre-training does not alleviate local model drift but stabilizes global aggregation, as shown by smaller gains from optimal convex combinations and lower loss variance.The FPS analysis reports the same pattern across CIFAR-10 and Tiny-ImageNet.Source: §C.5 · p. 25

D.1 POTENTIAL NEGATIVE SOCIETAL IMPACTS

  • Collecting real pre-training data can introduce bias or undermine privacy if the collection process is not carefully designed; synthetic data avoids using real data and can still improve FL.The authors locate these concerns in data collection rather than the learning algorithms.Source: §D.1 · p. 25

D.2 COMPUTATION RESOURCES

  • For 32 × 32 images, pre-training took about one day and FL about eight hours; 224 × 224 experiments took two to three days and one day, respectively.Cityscapes training with an A6000 GPU took about two days.Source: §D.2 · p. 25

D.3 MORE DISCUSSION ON INITIALIZATION AND PRE-TRAINED MODELS IN FL

  • Client-specific sparse parameter masks target communication and computation efficiency, an orthogonal problem to initialization; another approach freezes pre-trained models and learns class prototypes.The appendix also reports a smaller gap between FEDAVG and optimal convex aggregation with pre-training.Source: §D.3 · pp. 25–26, Figure 17 · p. 26

D.4 SYNTHETIC DATA FOR FL

The discussion focuses on synthetic image data for pre-training FL models, reports FPS’s superiority among evaluated methods, and distinguishes server-side generation from federated generator training.

D.4 SYNTHETIC DATA FOR FL

  • The paper focuses on generating synthetic images at the server for pre-training the global model in FL.The authors describe synthetic-data pre-training as a relatively new and fast-growing area in centralized learning.Source: §D.4 · p. 26
  • FPS outperforms other evaluated synthetic pre-training methods, including one based on random StyleGANs (Baradad et al., 2021; Karras et al., 2019).The paper suggests synthetic-data pre-training for modalities beyond vision as a promising direction for federated learning.Source: §D.4 · p. 26
  • FedSyn trains a generator using clients’ data, unlike this paper’s server-side synthetic-data generation without access to client data.Source: §D.4 · p. 26
Loading 2206.11488v3…