PaperlayerView exact revision on arXiv

Source-linked AI summary

Divergence-aware Federated Self-Supervised Learning

Weiming Zhuang, Yonggang Wen, Shuai Zhang

arXiv:2204.04385v1cs.LGcs.AIcs.CVcs.DC

TL;DR

Decentralized unlabeled images are often unavailable for centralized SSL, while existing federated SSL work leaves key Siamese-network components insufficiently understood. The paper studies those components in a generalized FedSSL framework and introduces FedEMA, which adaptively updates client models using a divergence-measured EMA of the global model. The authors report that FedEMA outperforms existing methods across a wide range of settings.

  • Problem

    Decentralized images may remain unavailable centrally for privacy reasons, and existing federated Siamese-network SSL studies do not provide deep insight into its fundamental building blocks.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces a generalized FedSSL framework and proposes FedEMA, which updates client models using an EMA of the global model with a divergence-measured decay rate.

    Source: §1 · p. 2, §4 · p. 7

  • Results

    The study finds that stop-gradient is not always essential in FedSSL, target networks are essential, and retaining client-local knowledge benefits non-IID performance; FedEMA outperforms existing methods across a wide range of settings.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The study offers insights into FedSSL components and reports that retaining local knowledge is beneficial for non-IID data.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

Self-supervised learning (SSL) is capable of learning remarkable representations from centrally available data. Recent works further implement federated learning with SSL to learn from rapidly growing decentralized unlabeled images (e.g., from cameras and phones), often resulted from privacy constraints. Extensive attention has been paid to SSL approaches based on Siamese networks. However, such an effort has not yet revealed deep insights into various fundamental building blocks for the federated self-supervised learning (FedSSL) architecture. We aim to fill in this gap via in-depth empirical study and propose a new method to tackle the non-independently and identically distributed (non-IID) data problem of decentralized data. Firstly, we introduce a generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods. In this framework, a server coordinates multiple clients to conduct SSL training and periodically updates local models of clients with the aggregated global model. Using the framework, our study uncovers unique insights of FedSSL: 1) stop-gradient operation, previously reported to be essential, is not always necessary in FedSSL; 2) retaining local knowledge of clients in FedSSL is particularly beneficial for non-IID data. Inspired by the insights, we then propose a new approach for model update, Federated Divergence-aware Exponential Moving Average update (FedEMA). FedEMA updates local models of clients adaptively using EMA of the global model, where the decay rate is dynamically measured by model divergence. Extensive experiments demonstrate that FedEMA outperforms existing methods by 3-4% on linear evaluation. We hope that this work will provide useful insights for future research.

1 INTRODUCTION

The paper studies Siamese-network FedSSL through a generalized framework, identifies how its components behave, and proposes FedEMA for decentralized non-IID data. FedEMA adaptively updates client models using an EMA of the global model, with performance reported above existing methods across a wide range of settings.

1 INTRODUCTION

  • Decentralized images may remain unavailable centrally because of privacy constraints, and training SSL separately at each edge performs poorly on mostly non-IID data.Such images can include sensitive personal information, while learning from them may benefit downstream tasks in the same scenarios.Source: §1 · p. 1
  • Prior federated SSL studies focus on individual Siamese-network methods, leaving the fundamental building blocks of FedSSL insufficiently understood.Examples include FedCA based on SimCLR and FedU based on BYOL.Source: §1 · p. 2
  • The generalized framework supports existing Siamese-network SSL methods: clients train online and target networks, while a server aggregates online networks and updates client online networks.The framework focuses primarily on cross-silo federated learning with stateful, highly available clients.Source: §1 · p. 2
  • The study finds that stop-gradient is not always essential in FedSSL, target networks are essential, and retaining client-local knowledge benefits non-IID performance.Across four popular SSL methods, FedBYOL performs best and FedSimSiam worst.Source: §1 · p. 2
  • FedEMA outperforms existing methods in a wide range of settings by adapting client updates with an EMA of the global network.Its EMA decay rate is dynamically measured by the divergence between global and online networks.Source: §1 · p. 2

2 RELATED WORK

Related work covers self-supervised representation learning, federated learning, and federated unsupervised learning. The paper positions its framework as a way to study Siamese-network components for purely unlabeled decentralized data.

2 RELATED WORK

  • Contrastive SSL uses negative pairs to prevent collapse, whereas non-contrastive methods can maintain performance using only positive pairs; both can perform poorly on decentralized non-IID data.The paper analyzes their similarities and differences in proposing a generalized FedSSL framework.Source: §2 · p. 2
  • Federated learning trains across decentralized parties without transmitting raw data to a central server.Source: §2 · p. 2
  • The paper frames FedEMA as a FedSSL extension that inherits the framework and overrides its model-update method.The related-work discussion also notes that prior personalized FL methods interpolate global and local models, while differing from FedEMA in motivation, scenario, and decay-rate measurement.Source: §2 · pp. 2–3
  • Prior federated unsupervised approaches address parts of the problem, but the cited work either neglects non-IID data, raises potential privacy risks, or does not explain why BYOL works best.The paper focuses on purely unlabeled decentralized data and investigates fundamental components as SSL methods evolve.Source: §2 · p. 3

3 AN EMPIRICAL STUDY OF FEDERATED SELF-SUPERVISED LEARNING

The section defines the FedSSL problem and a generalized framework for studying Siamese-network methods across decentralized data. Its empirical comparisons identify differences among methods and examine how their components affect learning, particularly under non-IID data.

3.1 PROBLEM DEFINITION

  • FedSSL addresses learning a generalized representation from decentralized, unlabeled client data that may be non-IID and limited in categories.The problem definition notes that an individual party may be unable to obtain a good representation on its own.Source: §3.1 · p. 3

3.2 GENERALIZED FRAMEWORK

  • The generalized framework has clients train Siamese networks and a server aggregate online networks and communicate the global model; it accommodates differing SSL-method components.The framework describes local training, model aggregation, and model communication as its three key operations.Source: §3.2 · pp. 3–4

3.3 EXPERIMENTAL SETUP

  • Experiments use CIFAR-10 and CIFAR-100 with label heterogeneity to simulate non-IID data, and assess representation quality by training a linear classifier on frozen representations.The default implementation uses five clients, five local epochs, and 100 rounds.Source: §3.3 · p. 4

3.4 ALGORITHM COMPARISONS

  • FedBYOL performs best and FedSimSiam worst among the non-contrastive FedSSL methods compared on non-IID CIFAR datasets.The comparison motivates further analysis of the fundamental components of non-contrastive methods.Source: §3.4 · p. 5, Table 1 · p. 5

3.5 IMPACT OF FACTORS OF NON-CONTRASTIVE METHODS

  • Without predictors, FedBYOL and FedSimSiam can barely learn, indicating the predictor's vital role in these non-contrastive methods.Figure 2 reports the drop in kNN testing accuracy without predictors.Source: §3.5 · p. 5, Figure 2 · p. 5
  • Stop-gradient is not essential for FedBYOL, which remains competitive without it, whereas FedSimSiam collapses without stop-gradient.The paper attributes the difference to FedBYOL's distinct online and target encoders versus SimSiam's shared weights.Source: §3.5 · p. 5, Figure 2 · p. 5
  • EMA is not essential but improves performance, while updating both encoders with the global model lowers performance, especially at 2 or 4 classes per client.The authors hypothesize that retaining local knowledge in the target encoder is particularly beneficial when client data are highly skewed.Source: §3.5 · pp. 5–6, Table 2 · p. 6
  • The target encoder needs knowledge from the online encoder: FedBYOL can barely learn without EMA, while updating both encoders with the global model yields comparable results.The paper argues that the target encoder provides contrastive targets and that competitive FedSSL methods update it using online-encoder knowledge.Source: §3.5 · p. 6, Figure 3 · p. 6

4 DIVERGENCE-AWARE DYNAMIC MOVING AVERAGE UPDATE

FedEMA adapts how clients combine local and global knowledge by updating their online networks with an EMA of the global model whose decay rate depends on model divergence. The method is motivated by retaining more local knowledge when divergence is large and incorporating more global knowledge when it is small.

4 DIVERGENCE-AWARE DYNAMIC MOVING AVERAGE UPDATE

  • Unlike FedBYOL’s direct global-model update, FedEMA fuses local and global knowledge through an EMA update.FedBYOL is used as the baseline for optimizing the model update operation.Source: §4 · p. 6
  • FedEMA updates clients’ online encoder and predictor through EMA of the global model, with the decay rate dynamically measured by divergence between global and online encoders.The encoder and predictor updates are given separately; λ scales the divergence used to measure μ, and λ = 0 makes FedEMA equivalent to FedBYOL.Source: §4 · p. 7
  • FedEMA proposes a personalized autoscaler that calculates a separate λ_k for each client because divergence varies with data and federated-learning settings.The paper notes that data characteristics are unknown before training because the data are unlabeled.Source: §4 · p. 7
  • FedEMA is designed to retain more local knowledge when divergence is large and incorporate more global knowledge when divergence is small.The paper states that retaining local knowledge is more beneficial when divergence is large, while using more global knowledge helps improve generalization when divergence is small.Source: §4 · pp. 7–8
  • FedEMA outperforms other methods in linear-probing comparisons on CIFAR datasets and in semi-supervised comparisons on non-IID CIFAR datasets.The supplied table captions identify the evaluations as top-1 accuracy comparisons; the semi-supervised evaluation uses 1% and 10% labeled data.Source: Table 3 · p. 8, Table 4 · p. 8

5 EVALUATION

Evaluation compares FedEMA with federated and standalone alternatives in linear and semi-supervised learning, then examines its update components and hyperparameters. FedEMA performs strongly across the reported CIFAR settings, while the ablations assess how divergence-aware updating behaves under different conditions.

5 EVALUATION

  • The evaluation covers linear evaluation and semi-supervised learning, with ablations of important hyperparameters.The algorithm comparisons include standalone BYOL, FedCA, FedBYOL, FedU, and centralized BYOL upper bounds.Source: §5 · p. 8, §5.1 · p. 8

5.1 ALGORITHM COMPARISONS

  • FedEMA is more than 3% higher than existing methods in most CIFAR linear-evaluation settings.The comparison is under linear probing on CIFAR datasets; the reported autoscaler achieves results similar to λ = 0.8.Source: §5.1 · p. 8
  • FedEMA consistently outperforms other methods on non-IID CIFAR semi-supervised evaluations using 1% and 10% labeled data.The autoscaler also outperforms manually selected λ = 1.Source: §5.1 · p. 8

5.2 ABLATION STUDIES

  • Updating both the online encoder and predictor gives the best performance; updating only the predictor converges faster.Updating only either component also leads to better performance, and results are shown in Figure 5.Source: §5.2 · p. 9, Figure 5 · p. 9
  • Divergence between the global and online encoders decreases gradually during training, allowing clients to update at different paces according to their local-data divergence.The authors interpret larger early divergence as a reason to use more local knowledge at the start of training.Source: §5.2 · p. 9, Figure 6 · p. 9
  • For CIFAR settings, λ values between 0 and 1 outperform FedBYOL, whereas λ > 1 causes a significant performance drop.The authors attribute the drop to μ = 1 at the start of training, so no aggregated global network is used; they caution that suitable λ depends on data and FL hyperparameters.Source: §5.2 · p. 9, Figure 7 · p. 9
  • FedEMA outperforms constant-μ alternatives for the online encoder, predictor, or both, although a good fixed μ can beat FedBYOL.The comparison tests the necessity of dynamic EMA.Source: §5.2 · p. 9, Figure 7 · p. 9
  • With autoscaling, FedEMA consistently outperforms FedBYOL across CIFAR-10 settings ranging from 2 to 10 classes per client.These experiments compare different non-IID levels.Source: §5.2 · p. 9, Figure 7 · p. 9

6 CONCLUSION

The conclusion presents the generalized FedSSL framework and its empirical insights as the basis for FedEMA, which addresses non-IID data challenges and outperforms existing methods across a wide range of settings.

6 CONCLUSION

  • The paper reports empirical insights into FedSSL using a newly introduced generalized framework.The framework is presented as the basis for the study’s findings.Source: §6 · p. 9
  • FedEMA is proposed to address non-IID data challenges, and experiments and ablations show it outperforms existing methods in a wide range of settings.The authors plan to implement FedSSL and FedEMA on larger-scale datasets.Source: §6 · p. 10

7 REPRODUCIBILITY STATEMENT

The reproducibility statement lists the experimental information provided in the paper and notes a plan to open-source the code.

7 REPRODUCIBILITY STATEMENT

  • The paper provides experimental setups and appendix details covering datasets, transformations, architectures, training, and default settings, and says code will be open-sourced in the future.Non-default settings and hyperparameters are also indicated where applicable.Source: §7 · p. 10

A DIFFERENCES OF SELF-SUPERVISED LEARNING METHODS

The appendix distinguishes four SSL methods by predictors and encoder weight sharing, then documents the experimental data and augmentation setup. Supplementary comparisons report FedEMA outperforming other methods in non-IID settings and FedBYOL in client-subsampling settings.

A DIFFERENCES OF SELF-SUPERVISED LEARNING METHODS

B EXPERIMENTAL DETAILS

  • The experimental-details section introduces datasets, network architecture, and training and evaluation setups.The supplied passage gives no further specifics for this subsection.Source: B EXPERIMENTAL DETAILS · p. 14

B.1 DATA

  • CIFAR-10 and CIFAR-100 training sets are split equally across clients, with non-IID severity controlled by the number of classes per client.The default uses five clients and two classes per client for CIFAR-10, or 20 for CIFAR-100; two augmentations feed the online and target networks.Source: §B.1 · p. 14
  • FedEMA outperforms other methods on non-IID CIFAR datasets, while it consistently beats FedBYOL in both client-subsampling settings.Table 5 compares top-1 accuracy under linear evaluation; Table 6 uses 5/20 and 8/80 client selection per round.Source: Table 5 · p. 15, Table 6 · p. 15

B.2 NETWORK ARCHITECTURE

The network description specifies the predictor and encoder architectures, including modifications to the ResNet implementation. Supplementary results compare FedEMA and other methods under semi-supervised evaluation and across CIFAR-100 non-IID levels.

B.2 NETWORK ARCHITECTURE

  • The predictor is a two-layer MLP, beginning with a 4096-neuron fully connected layer and ending with a 2048-neuron fully connected layer.A one-dimensional batch-normalization layer and ReLU activation intervene.Source: §B.2 · p. 15
  • ResNet-18 is the default encoder architecture for most experiments, with ResNet-50 results also reported.Source: §B.2 · p. 15
  • FedEMA outperforms all other methods in semi-supervised learning on non-IID CIFAR datasets using ResNet-50.Source: Table 7 · p. 16
  • Table 8 compares top-1 accuracy across CIFAR-100 non-IID levels, varying the number of classes per client.Source: Table 8 · p. 16
  • The ResNet implementation uses a 3 × 3 first-convolution kernel, 4 × 4 average pooling, and a two-layer MLP in place of the final linear layer.These differ from the cited PyTorch implementation's 7 × 7 convolution, adaptive average pooling, and final linear layer.Source: §B.2 · p. 16

B.3 TRAINING AND EVALUATION DETAILS

The section describes training and evaluation protocols and supplementary analyses of FedEMA. These experiments compare methods across training rounds and batch sizes, while noting the cross-silo setting and its boundary at very large client scale.

B.3 TRAINING AND EVALUATION DETAILS

  • Training uses SGD, an initial learning rate of 0.032 with cosine annealing, and by default 100 rounds, five local epochs, batch size 128, and five clients.The implementation uses PyTorch and NCCL communication; FedEMA defaults to λ = 1 or autoscaler τ = 0.7.Source: §B.3 · pp. 16–17

C ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS

  • Table 9 compares FedBYOL without EMA and stop-gradient against updating both online and target encoders, with the former described as barely learning.The supplied result passage says updating both encoders achieves comparable results.Source: Table 9 · p. 17, §B.3 · p. 17
  • FedEMA outperforms FedBYOL across training rounds on non-IID CIFAR-10, and the batch-size comparison cautions against batches that are too small or too large.The Figure 9 caption states consistent outperformance across total rounds; Table 10 reports the batch-size comparison.Source: Figure 9 · p. 17, Table 10 · p. 18

B.3 TRAINING AND EVALUATION DETAILS

  • The paper primarily studies cross-silo FL with stateful, highly available clients; at millions of stateless clients, FedEMA degrades to FedBYOL without local states.The authors state that client states can be cached on the server in the smaller cross-device settings considered by most studies.Source: §B.3 · p. 17
  • Linear evaluation trains a new fully connected layer on a frozen encoder, while semi-supervised evaluation fine-tunes the model using 1% or 10% of training data.The respective protocols use 200 and 100 epochs.Source: §B.3 · p. 17

C ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS

C.1 MORE EXPERIMENTAL RESULTS

Additional comparisons report FedEMA's performance under non-IID CIFAR settings, client subsampling, and semi-supervised evaluation. Further tables compare FedSimSiam, FedBYOL, and FedBYOL variants.

C.1 MORE EXPERIMENTAL RESULTS

  • FedEMA outperforms all other methods on non-IID CIFAR datasets under linear evaluation with ResNet-18 and ResNet-50.The reported settings use λ = 0.8 for ResNet-18 and λ = 1 for ResNet-50.Source: §C.1 · p. 18
  • With client subsampling, FedEMA consistently outperforms FedBYOL when selecting 5 of 20 or 8 of 80 clients per round.The experiments use ResNet-18, with 400 rounds and five local epochs in the 5/20 setting, and 800 rounds and two local epochs in the 8/80 setting.Source: §C.1 · p. 18
  • FedEMA consistently outperforms all other methods in the supplementary ResNet-50 semi-supervised results.Source: §C.1 · p. 18
  • Tables 8 and 9 compare FedSimSiam, FedBYOL, and FedBYOL variants to extend the earlier empirical analyses.Source: §C.1 · p. 18

C.2 FURTHER ANALYSIS

Further analysis tests FedEMA under different training and data settings. Results highlight trade-offs in communication versus computation and the importance of per-client data amount.

C.2 FURTHER ANALYSIS

  • Autoscaler values τ in (0, 1) generally outperform FedBYOL (τ = 0), while τ = 1 performs worse when global knowledge is neglected.Values τ ∈ [0.5, 1) are generally better than other values, and tuning τ can improve CIFAR-100 performance.Source: §C.2 · p. 18
  • With FedEMA, 40 rounds and 800 computation epochs reach 80% accuracy at E = 20, versus 80 rounds and 400 epochs at E = 5.Increasing local epochs reduces communication rounds but generally raises computation cost; E = 1 does not reach 80% in 100 rounds.Source: Table 11 · p. 19, §C.2 · p. 19
  • Increasing data amount per client improves performance significantly, whereas increasing the number of clients at the same per-client data amount does not.The comparison varies client data amount and client count while keeping total data fixed.Source: Table 12 · p. 19, §C.2 · p. 19
  • Among constant EMA settings, µo = 0.5 and µp = 0.3 perform best, while µo = 0.9 and µp = 0.9 yield 54.52%.FedEMA consistently outperforms the tested constant-EMA methods and FedBYOL.Source: §C.2 · p. 19
  • FedEMA consistently outperforms FedBYOL as communication rounds increase on non-IID CIFAR-10.Both methods improve with more rounds; FedEMA uses λ = 0.5 in these experiments.Source: §C.2 · p. 19
  • Batch sizes B = 128 and B = 256 perform similarly and outperform other tested sizes; FedEMA beats FedBYOL at every tested size.These experiments use the non-IID CIFAR-10 setting with autoscaler τ = 0.7.Source: §C.2 · p. 19

Related papers

Loading 2204.04385v1…