PaperlayerView exact revision on arXiv

Source-linked AI summary

SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training

Enmao Diao, Jie Ding, Vahid Tarokh

arXiv:2106.01432v4cs.LG

TL;DR

Most FL work assumes clients have ground-truth labels, although practical clients may lack the expertise or resources to label their data. SemiFL alternates labeled-server fine-tuning with global-model pseudo-label generation, and reports improved SSFL performance alongside competitive results against state-of-the-art FL and SSL.

  • Problem

    Most existing FL work assumes clients have ground-truth labels, but in practical settings clients may lack the expertise or resources to label task-specific data.

    Source: §1 · p. 1

  • Method

    SemiFL alternates fine-tuning the global model with server-labeled data and generating pseudo-labels with the global model for unlabeled clients.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    SemiFL outperforms existing SSFL methods and performs competitively with state-of-the-art FL and SSL; on CIFAR10, performance rises from 42% to 88% with 250 labeled data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SemiFL addresses SSFL with unlabeled clients in a communication-efficient setting where clients train multiple local epochs alongside a server with a small labeled set.

    Source: §1 · p. 2

    Updating the aggregated client model with labeled server data delays computation time.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Federated Learning allows the training of machine learning models by using the computation and private data resources of many distributed clients. Most existing results on Federated Learning (FL) assume the clients have ground-truth labels. However, in many practical scenarios, clients may be unable to label task-specific data due to a lack of expertise or resource. We propose SemiFL to address the problem of combining communication-efficient FL such as FedAvg with Semi-Supervised Learning (SSL). In SemiFL, clients have completely unlabeled data and can train multiple local epochs to reduce communication costs, while the server has a small amount of labeled data. We provide a theoretical understanding of the success of data augmentation-based SSL methods to illustrate the bottleneck of a vanilla combination of communication-efficient FL with SSL. To address this issue, we propose alternate training to `fine-tune global model with labeled data' and `generate pseudo-labels with the global model.' We conduct extensive experiments and demonstrate that our approach significantly improves the performance of a labeled server with unlabeled clients training with multiple local epochs. Moreover, our method outperforms many existing SSFL baselines and performs competitively with the state-of-the-art FL and SSL results.

1 Introduction

SemiFL targets FL settings where a small labeled server must learn with completely unlabeled clients, a regime that complicates combining SSL with multi-epoch communication-efficient FL. The paper analyzes augmentation-based SSL and proposes alternating server and client training.

1 Introduction

  • FL aggregates locally trained model parameters into a global inference model without sharing private local data.Source: §1 · p. 1
  • Many practical clients may lack the expertise to label task-specific data, motivating a server with labeled data to leverage unlabeled clients for supervised FL.Source: §1 · pp. 1–2
  • Alternate training fine-tunes the global model with labeled data and generates pseudo-labels with the global model to maintain pseudo-label quality.The server and unlabeled clients are trained in an alternating manner.Source: §1 · p. 2
  • SemiFL combines a small labeled server with completely unlabeled clients that train multiple local epochs to reduce communication costs.The authors identify difficulty in combining communication-efficient FedAvg with state-of-the-art SSL methods.Source: §1 · p. 2
  • The paper theoretically analyzes strong data augmentation in SSL to explain its success and the bottleneck of a vanilla combination with communication-efficient FL.Source: §1 · p. 2
  • 30% improvement over existing SSFL methods; SemiFL is also only 1% and 2% away from state-of-the-art FL and SSL results, respectively, with 4000 labeled data on CIFAR10.The introduction also reports performance increasing from 42% to 88% with 250 labeled data and from 77% to 93% accuracy with 4000 labeled data on CIFAR10.Source: §1 · p. 2

2 Related Work

The related-work section situates SemiFL among federated optimization methods, semi-supervised learning techniques, and prior semi-supervised federated approaches.

Federated Learning

  • FedAvg allows local clients to train multiple epochs, while FedProx, FL counterparts of Batch Normalization, momentum methods, and FedOpt address other aspects of federated training.Source: §2 · p. 3

Semi-Supervised Learning

  • SSL covers partially labeled data with much more unlabeled data; its approaches include self-training, pseudo-labeling, consistency regularization, and strong data augmentation.Source: §2 · p. 3

Semi-Supervised Federated Learning (SSFL)

  • Prior SSFL work includes separating model parameters for labeled servers and unlabeled clients [27] and parallel training with group-wise reweighting [28].The section also notes SSFL applications in specific domains [29].Source: §2 · p. 3

3 Method

SemiFL addresses the difficulty of combining semi-supervised learning with communication-efficient federated learning when clients have unlabeled data. Its theoretical analysis motivates alternate training, and its algorithm alternates server updates with client pseudo-labeling and local training.

3.1 Problem

  • FixMatch-style SSL is difficult to combine with FedAvg’s multiple local epochs because its training batches mix labeled and unlabeled data at a tuned ratio, while FixMatch works with batch-wise FedSGD aggregation.The paper identifies this as a bottleneck for communication-efficient SSFL.Source: §3.1 · p. 3

3.2 Theoretical Analysis of Strong Data augmentation for SSL

  • Under adequate transmission, strong augmentation lets high-confidence unlabeled examples cover undertrained prediction regimes; kernel-classifier risk bounds indicate a critical unlabeled-data region for improvement over labeled-only learning.The bound depends on data dimension, conditional-label smoothness, class separability, and transmission inadequacy.Source: §3.2 · p. 4

3.3 Alternate Training

  • Alternate training addresses degrading pseudo-label quality by fine-tuning the aggregated global model on server labels, then generating client pseudo-labels once when clients receive it.The paper contrasts this sequence with parallel server/client training and pseudo-labeling every batch during local training.Source: §3.3 · pp. 5–6

3.4 The SemiFL Algorithm

  • At each round, the server updates its model using labeled data and sends it to active clients, which construct high-confidence pseudo-labeled datasets from the received global model.The described server update uses supervised loss; clients use a shared confidence threshold for the high-confidence data.Source: §3.4 · p. 6
  • Clients train for E local epochs using FixMatch-style and Mixup-inspired data and losses, then transmit their models for server aggregation.Client updates include strong data augmentation and a Mixup hyperparameter.Source: §3.4 · pp. 6–7
  • The server averages received client models and, after training, further fine-tunes the aggregate with its supervised data.Source: §3.4 · p. 8

4 Experiments

Across the experiments, SemiFL improves on the labeled-server-only baseline, significantly outperforms existing SSFL methods, and performs close to state-of-the-art FL and SSL methods. Its alternate-training components improve pseudo-label quality when clients have completely unlabeled data and train multiple local epochs.

4 Experiments

  • Experiments use 100 clients, a 0.1 activity rate, IID and Non-IID partitions, and CIFAR10, SVHN, and CIFAR100 datasets.The experiments use four random seeds; server labeled-data counts vary by dataset.Source: §4 · p. 8
  • SemiFL improves the labeled server, significantly outperforms existing SSFL methods, and performs close to state-of-the-art FL and SSL methods.The comparison covers baselines, SSL, FL, and SSFL methods.Source: §4 · pp. 8–9, Table 1 · p. 9

4.1 Quality of Pseudo Labeling

  • The CIFAR10 ablations identify both global-model fine-tuning with labeled data and pseudo-label generation from the global model as critical components of alternate training.The experiments also ablate static Batch Normalization, local training epochs, Mixup, and global SGD momentum.Source: §4 · p. 9, Table 2 · p. 10
  • With completely unlabeled clients training multiple local epochs, the combination of global-model fine-tuning and pseudo-label generation produces much better pseudo-label quality.Figure 6 measures pseudo-label accuracy, threshold accuracy, and label ratio on CIFAR10.Source: §4.1 · p. 9, Figure 6 · p. 10

5 Conclusion

SemiFL targets SSFL with unlabeled clients through alternate training of the labeled server and clients. Experiments show improved labeled-server performance and competitive results against state-of-the-art centralized SSL and fully supervised FL methods.

5 Conclusion

  • SemiFL alternates server fine-tuning with labeled data and client pseudo-label generation from the global model.The framework addresses communication-efficient SSFL for clients with unlabeled data.Source: §5 · p. 10
  • The paper’s theoretical analysis of strong data augmentation for SSL illustrates a bottleneck in vanilla combinations of communication-efficient FL and SSL.Source: §5 · p. 10
  • Experiments report significant improvement for a labeled server with unlabeled clients and competitive performance with state-of-the-art centralized SSL and fully supervised FL.The authors present the framework as a practical approach that extends the scope of FL applications.Source: §5 · p. 10

Checklist

The paper frames SSFL as seeking performance near centralized SSL while substantially exceeding partially supervised learning. It describes static Batch Normalization for FL and reports that server data can update its global inference statistics without degrading performance.

Checklist

  • The checklist points to supplementary sections for training details, theoretical assumptions and proofs, and work limitations.It also reports source code in supplementary material, publicly available datasets, and error bars or standard errors for experiments.Source: Checklist · pp. 13–14

A Performance Goal

  • The stated SSFL goal is to perform similarly to state-of-the-art SSL while significantly outperforming partially supervised learning and existing SSFL methods.The paper expresses this target as FSL ≳ SSL ≳ SSFL ≫ PSL.Source: A Performance Goal · p. 15

B Static Batch Normalization

  • Static Batch Normalization standardizes each batch using batch-wise statistics rather than tracking running statistics with momentum.The paper reports sBN accelerates convergence and improves FedAvg performance relative to other normalization forms.Source: B Static Batch Normalization · p. 15
  • For SemiFL, server data can update global sBN inference statistics instead of querying every client each communication round; using server data alone does not degrade performance.The comparison of server-only and server-plus-client statistics is reported for CIFAR10.Source: B Static Batch Normalization · p. 15, Table 3 · p. 15

C.1 Experimental setup

The appendix specifies experimental hyperparameters and the losses used for labeled-server and unlabeled-client training. It also identifies SVHN and CIFAR100 results and the server labeled-data counts shown for those datasets.

C.1 Experimental setup

  • Experiments use SGD, cosine learning-rate decay, local learning rate 0.03, local momentum 0.9, confidence threshold 0.95, and Mixup parameter 0.75.The table lists the experiment hyperparameters.Source: §C.1 · p. 16, Table 4 · p. 16
  • Unlabeled-client training combines FixMatch’s consistency-regularization and pseudo-labeling loss with a Mixup-adapted loss; the latter moderately improves performance.Pseudo-labels come from weakly augmented data, while training uses strongly augmented data.Source: §C.1 · p. 16

C.2 SVHN and CIFAR100

C.3 Ablation studies

The ablations examine local training epochs, Mixup, and global SGD momentum. Fewer local epochs hurt performance, Mixup improves CIFAR10 accuracy, and global momentum provides a marginal gain.

C.3 Ablation studies

  • Fewer local training epochs significantly hurt performance because convergence is slower.The ablation concerns CIFAR10 with 4,000 labeled examples at the server.Source: §C.3 · p. 17, Table 5 · p. 17
  • Mixup improves CIFAR10 accuracy by around 2%.The authors find it beneficial alongside strong data augmentation when training on unlabeled data.Source: §C.3 · p. 17
  • Global SGD momentum marginally improves the result.The ablation is reported for CIFAR10 with 4,000 labeled examples at the server.Source: §C.3 · p. 17, Table 5 · p. 17
  • Figure 9 covers CIFAR10 with 4,000 server-labeled examples under IID and non-IID K=2 data partitions.The caption identifies the settings but does not state comparative outcomes.Source: Figure 9 · p. 18

D Theoretical Analysis of Strong Data Augmentation for SSL

This background defines classification through prediction risk and semi-supervised learning through the use of unlabeled data alongside labeled data. The latter setting is especially relevant when unlabeled examples greatly outnumber labeled ones.

D.1 Background of Classification

  • Classification seeks a classifier whose probability of misclassifying an example is small.The background introduces m(x) as the conditional probability of the label given x and gives logistic regression as an example.Source: §D.1 · p. 18
  • When the underlying conditional probability is unknown, a classifier can be trained by first estimating that probability from observed training data.The passages describe training examples as IID draws from the same distribution as the prediction data.Source: §D.1 · p. 18

D.2 Background of Semi-Supervised Learning

  • Semi-supervised learning uses labeled and unlabeled observations, with the unlabeled sample potentially drawn from a different distribution.The setting of interest has many more unlabeled than labeled examples and seeks to improve on a classifier trained using labeled data alone.Source: §D.2 · p. 18

D.3 A new perspective of Semi-Supervised Learning

The section motivates strong augmentation as a way to use high-confidence pseudo-labels to train on data near regimes that labeled examples may insufficiently cover. It develops a kernel-based analysis under an adequate-transmission condition and supporting assumptions.

D.3 A new perspective of Semi-Supervised Learning

  • Strong augmentation maps high-quality examples to lower-quality versions, which are trained with high-confidence pseudo-labels to add observations near difficult regimes such as decision boundaries.RandAugment examples include shearing and contrast adjustment, with greater distortion magnitude producing stronger augmentation.Source: §D.3 · p. 19, Figure 10 · p. 19
  • The theory explains strong-augmentation SSL through adequate transmission: augmented data from high-confidence unlabeled examples cover test-relevant regimes and transmit reliable information there.The analysis is restricted to nonparametric kernel-based classification rather than SSL in full generality.Source: §D.3 · pp. 19, 21
  • The generic procedure selects high-confidence unlabeled examples, pseudo-labels and strongly augments them, then trains with those examples together with labeled data.The setup assumes a test distribution matching the labeled-data distribution; the unlabeled distribution may differ.Source: §D.3 · pp. 19–20
  • Assumption A1 requires a nontrivial tail of unlabeled examples with high-confidence predictions, with s quantifying the sufficiency of that tail.For logistic regression, the passage gives an exponential-tail example where the sufficiency function is proportional to δ.Source: §D.3 · p. 20
  • Assumptions A2 and A3 require augmented high-confidence examples to preserve label information and cover test-relevant regions, respectively.Together, they express adequate transmission: augmented data can represent regimes of interest and be approximated as labeled data for supervised training.Source: §D.3 · pp. 20–21
  • Assumptions A4 and A5 constrain task difficulty and smoothness: the margin condition reflects class separability, while the regression function cannot be too bumpy.The boundedness of the labeled inputs is included for technical convenience.Source: §D.3 · p. 21
  • Assumptions A6 and A7 require an adequately accurate initial estimate in high-margin regions and impose additional technical parameter conditions.The paper relates the initial-estimate condition to SSL training following a preceding round that uses labeled data.Source: §D.3 · p. 21

D.4 Main result

The main result gives an explicit convergence rate for strong-augmentation SSL using unlabeled data and identifies when its risk bound can substantially improve on labeled-only learning. The improvement depends on task and adequate-transmission parameters.

D.4 Main result

  • Theorem 1 provides an explicit convergence rate for SSL classification risk as a function of unlabeled sample size n_u.The theorem applies under Assumptions A1–A7, and its constant C does not depend on sample size.Source: §D.4 · p. 21
  • When the margin parameter α is large, the SSL risk rate can reach n_u^-1.The paper characterizes this as a relatively easy classification setting.Source: §D.4 · p. 22
  • When tail sufficiency s or coverage v is large, the rate is approximately n_u^-(α+1)/(v+s), reflecting fewer high-confidence examples transmitted to benefit classification.The paper says larger s or v corresponds to fewer such examples.Source: §D.4 · p. 22
  • When v=s=0, the SSL rate exponent matches an existing classification result, while the labeled-only risk is bounded by C′n_l^-q(α+1)/{q(α+3)+d}.The labeled-only classifier uses n_l labeled examples.Source: §D.4 · p. 22
  • Inequality (14) identifies a region of n_u where strong-augmentation SSL can significantly improve classification, depending on q, d, α, s, and v.The comparison is between the risk bounds for SSL and the labeled-only classifier; the supplied passages do not give the complete inequality in readable form.Source: §D.4 · p. 22

D.5 Proof of Theorem 1

The proof bounds classification risk through estimation error, then controls the estimation error using bias–variance decomposition and concentration inequalities. It uses adequate-transmission conditions and tuning choices to limit the influence of pseudo-labeled data.

D.5 Proof of Theorem 1

  • The proof decomposes estimation error into bias and variance terms, bounding them with concentration inequalities and carefully chosen δ and kernel bandwidth.It uses adequate-transmission conditions to derive the convergence rate from labeled and pseudo-labeled data.Source: §D.5 · p. 22
  • The proof bounds classification error through the event that |m(X)−m̂(X)| exceeds a chosen threshold under Assumption (A4).This connects the risk bound to the estimation error of the learned function.Source: §D.5 · p. 22
  • The expected pseudo-label disagreement is bounded by 2δ_n + 2c_8n^-r.The proof tunes δ_n so the bias introduced by pseudo-labeled data has a diminishing influence on the risk rate.Source: §D.5 · pp. 22, 24

Related papers

Loading 2106.01432v4…