PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning

Wonyong Jeong, Jaehong Yoon, Eunho Yang, Sung Ju Hwang

arXiv:2006.12097v3cs.LGstat.ML

TL;DR

Federated Semi-Supervised Learning addresses federated training when client data is partly or completely unlabeled, including cases where labels remain only at the server. FedMatch tackles these scenarios with inter-client consistency and parameter decomposition, and experiments report improvements over local SSL and naive federated-SSL baselines.

  • Problem

    FSSL addresses federated learning when client data is partly labeled or completely unlabeled because labeling can be costly or require expert knowledge.

  • Method

    FedMatch combines inter-client consistency loss with parameter decomposition to separate supervised and unsupervised learning.

  • Results

    FedMatch outperforms local SSL and naive Fed-SSL baselines across labels-at-client and labels-at-server scenarios with i.i.d. and non-i.i.d. data.

  • Takeaways & Limitations

    FedMatch provides a federated semi-supervised approach for both partly labeled clients and server-only labeled data.

  • Takeaways & Limitations

    The labels-at-client setting assumes users annotate only a small portion of local data, leaving most data unlabeled.

Abstract

from arXiv · show

While existing federated learning approaches mostly require that clients have fully-labeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to the requirement of expert knowledge. Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated Semi-Supervised Learning (FSSL). In this work, we study two essential scenarios of FSSL based on the location of the labeled data. The first scenario considers a conventional case where clients have both labeled and unlabeled data (labels-at-client), and the second scenario considers a more challenging case, where the labeled data is only available at the server (labels-at-server). We then propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch). FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data. Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning. The code is available at https://github.com/wyjeong/FedMatch.

1 INTRODUCTION

Federated Semi-Supervised Learning addresses federated settings where client data is partly or mostly unlabeled. FedMatch combines inter-client consistency with parameter decomposition to handle these settings and outperforms local SSL and naive federated SSL baselines.

  • Federated learning collaboratively trains a global model while keeping data privately accessible at local clients.This decentralization supports data privacy, security, and access rights.
  • Existing federated learning methods commonly assume fully labeled local data, an unrealistic requirement when users cannot or do not label all device data.The resulting majority-unlabeled setting motivates Federated Semi-Supervised Learning.
  • FSSL studies federated learning with deficient supervision, including partly labeled local data and completely unlabeled local data.
  • FedMatch enforces consistency across predictions from multiple client models and decomposes parameters for supervised and unsupervised learning.The decomposition targets disjoint learning across labeled and unlabeled data.
  • FedMatch significantly outperforms local SSL and naive federated-SSL combinations across labels-at-client and labels-at-server settings with both non-i.i.d. and i.i.d. data.

2 PROBLEM DEFINITION

The paper formalizes Federated Semi-Supervised Learning by combining federated privacy and coordination with semi-supervised learning, then distinguishes scenarios by where labeled data resides. Standard federated rounds aggregate client updates, while FSSL distributes unlabeled data privately and may place labeled data at clients or the server.

  • Federated Learning: Federated learning coordinates multiple clients to learn a global model without sharing their privately collected data.At each round, selected local models are initialized from global weights, trained, aggregated, and rebroadcast.
  • Semi-Supervised Learning: Semi-supervised learning uses a small labeled set S and a larger unlabeled set U with supervised and unsupervised loss terms.The objective is ℓ_final(θ) = ℓ_s(θ) + ℓ_u(θ).
  • Federated Semi-Supervised Learning: FSSL addresses federated datasets whose client-side examples may lack accompanying labels.The unlabeled data is privately distributed across K clients.
  • FSSL Scenarios: The Labels-at-Server scenario keeps supervised labels at the server and supplies local clients with unlabeled data.The two scenarios differ according to labeled-data availability at clients or the server.
  • FSSL Scenarios: The Labels-at-Client scenario provides clients with both labeled and unlabeled data, while the server aggregates and redistributes client updates.

3 FEDERATED MATCHING

FedMatch combines inter-client consistency regularization with parameter decomposition for disjoint supervised and unsupervised learning. The method also uses helper agents, pseudo-labeling, sparsity, and transmitted parameter differences.

  • 3.1 INTER-CLIENT CONSISTENCY LOSS: FedMatch introduces inter-client consistency loss to regularize predictions from models trained at multiple clients toward the same output.
  • 3.1 INTER-CLIENT CONSISTENCY LOSS: Helper agents are selected from other clients by model similarity, frozen locally, and broadcast by the server each communication round.
  • 3.1 INTER-CLIENT CONSISTENCY LOSS: The final consistency term combines inter-client consistency with data-level consistency using RandAugment and agreement-based pseudo-labeling.
  • 3.1 INTER-CLIENT CONSISTENCY LOSS: Low-confidence predictions below threshold τ are discarded before standard cross-entropy minimization with the pseudo-label.
  • 3.2 PARAMETER DECOMPOSITION FOR DISJOINT LEARNING: FedMatch decomposes model parameters into supervised σ and unsupervised ψ sets, training ψ while keeping σ fixed during unsupervised learning.
  • 3.2 PARAMETER DECOMPOSITION FOR DISJOINT LEARNING: The decomposition is intended to preserve labeled-data knowledge, reduce inter-task interference, support disjoint learning, and lower communication costs through sparse parameter differences.

4 LABELS-AT-CLIENT SCENARIO

In the labels-at-client scenario, users annotate a small local subset while the remaining data stays unlabeled. Clients perform semi-supervised learning, and the server aggregates the resulting parameter updates.

  • 4 LABELS-AT-CLIENT SCENARIO: Labels-at-Client assumes that users annotate 5% of their local data, leaving the remainder unlabeled.
  • 4 LABELS-AT-CLIENT SCENARIO: Clients train on both labeled and unlabeled data, while the server only aggregates client updates and redistributes the aggregated parameters.
  • 4 LABELS-AT-CLIENT SCENARIO: Active local models minimize ℓfinal(θla) = ℓs(θla) + ℓu(θla) on their labeled and unlabeled subsets, respectively.
  • 4 LABELS-AT-CLIENT SCENARIO: FedMatch clients learn supervised and unsupervised parameter components locally, then send both learned components to the server.
  • 4 LABELS-AT-CLIENT SCENARIO: The server separately aggregates the components, retrieves top-H helper agents using model similarity, and transmits aggregated parameters in the next round.

5 LABELS-AT-SERVER SCENARIO

In the labels-at-server scenario, labeled data remains at the server while clients possess only unlabeled data. The global model learns supervised parameters centrally, and active clients learn unsupervised parameters locally.

  • 5 LABELS-AT-SERVER SCENARIO: Labels-at-Server places the supervised labels exclusively at the server, while local clients work with unlabeled data.
  • 5 LABELS-AT-SERVER SCENARIO: Before broadcasting its parameters, the global model performs supervised learning on the server’s labeled dataset by minimizing ℓs(θG).
  • 5 LABELS-AT-SERVER SCENARIO: At each round, active clients perform unsupervised learning solely by minimizing ℓu(θla) on their unlabeled data.
  • 5 LABELS-AT-SERVER SCENARIO: FedMatch has the server learn σ, clients learn ψ, and clients transmit their learned ψ updates back to the server after local training.
  • 5 LABELS-AT-SERVER SCENARIO: The server embeds local models by similarity and creates a KD-Tree to support rapid helper-agent selection.

6 EXPERIMENTS

Experiments evaluate FedMatch across batch and streaming federated semi-supervised settings, including labels-at-client and the more challenging labels-at-server scenario. FedMatch consistently outperforms local and naive federated semi-supervised baselines while addressing interference and communication costs.

  • Experimental Setup: FedMatch is evaluated on Batch-IID, Batch-NonIID, and Streaming-NonIID tasks under labels-at-client and labels-at-server scenarios.The batch experiments use 100 clients for 200 rounds, while Streaming-NonIID uses 10 clients for 100 rounds.
  • Batch-IID & NonIID Results: FedMatch outperforms all naive federated semi-supervised baselines across batch tasks and both label-location scenarios.Under labels-at-server, naive combinations deteriorate after a communication round because shared parameters cause interference between labeled and unlabeled learning.
  • Streaming-NonIID Results: FedMatch outperforms local-SSL and naive federated semi-supervised models on Streaming-NonIID, with margins of 4-15%p except against fully supervised models.It also achieves significantly smaller average communication cost than all baselines.
  • Ablation Study: Ablations show that inter-client consistency improves consistency across models, while parameter decomposition alone still outperforms base models.Removing either decomposed variable causes a substantial performance drop, with a larger degeneration when σ is removed.
  • Additional Analysis: Parameter decomposition preserves labeled-set knowledge by separating supervised and unsupervised learning, mitigating the inter-task interference seen in baseline models.FedMatch also improves consistently as labels per class increase from 1 to 5, 10, and 20, while FedProx-UDA/FixMatch can degrade from 5 to 10.

7 RELATED WORK

The related work situates FedMatch at the intersection of federated learning and consistency-based semi-supervised learning. Prior federated methods aggregate local weights, while SSL methods enforce prediction consistency under input perturbations.

  • Federated Learning: FedAvg weighted-averages local weights by training-set size, FedProx adds proximal regularization with uniform averaging, and FedMA matches hidden elements layer by layer.These methods represent alternative strategies for aggregating or aligning client models.
  • Semi-Supervised Learning: Consistency regularization assumes class semantics remain unchanged under input transformations and enforces matching predictions across perturbed versions of an example.The cited extensions perturb inputs adversarially, through dropout, or through data augmentation.

8 CONCLUSION

The conclusion frames FSSL around partly labeled clients or server-only labels and presents FedMatch as a solution combining inter-client consistency with parameter decomposition. Experiments report superiority over local SSL and naive federated-SSL combinations across diverse scenarios.

  • Conclusion: The paper defines two FSSL scenarios: partly labeled client data and completely unlabeled client data with supervised labels available only at the server.They are named Labels-at-Client and Labels-at-Server, respectively.
  • Conclusion: FedMatch combines inter-client consistency, which aligns models trained at different clients, with parameter decomposition for labeled and unlabeled learning.The decomposition is described as supporting reliable-knowledge preservation, lower communication costs, and disjoint learning.
  • Conclusion: Across diverse and realistic scenarios, FedMatch significantly outperforms local semi-supervised methods and naive federated-learning plus semi-supervised-learning combinations.The authors identify adaptation to completely unlabeled data streams from pretrained client models as future work.

A.2 NETWORK ARCHITECTURE

Table 3 presents the network architecture used for the ResNet-9 model.

  • Network Architecture: The section identifies the network architecture as ResNet-9.

A.3 TRAINING DETAILS

The experiments use controlled labeled/unlabeled partitions across batch, streaming, and class-imbalanced settings, with training and communication analyses for FedMatch. Results examine convergence, helper-agent transmission, label counts, and client participation using multiple architectures.

  • Training setup: SGD training starts at learning rate 1e-3, uses adaptive decay, and applies L2 weight decay of 1e-4.The learning rate is reduced by a factor of 3 after every 5 epochs without consecutive validation-loss improvement.
  • Dataset configuration: The dataset is partitioned into labeled set S and unlabeled set U, with U distributed across K clients and optionally split into T streaming steps.Class-imbalanced tasks additionally control each client's number of instances per class.
  • Communication efficiency: FedMatch communication transmits sparse parameter differences ∆ψ and ∆σ in both server-to-client and client-to-server directions.Element-wise thresholding discards nearly unchanged values so only meaningful updates are transmitted.
  • Communication analysis: Communication costs gradually decrease during training across batch and streaming tasks in both labels-at-client and labels-at-server scenarios.The authors attribute this trend to separate parameters learning supervised and unsupervised tasks effectively.
  • Communication analysis: Transmitting H=2 helper agents does not significantly increase total server-to-client cost, while model reconstruction remains possible without meaningful information loss.This is attributed to parameter decomposition and efficient subtraction.
  • Additional experiments: The study also evaluates label counts per class, client participation, and global accuracy with an AlexNet-Like architecture.Table 6 varies available-client fraction and labeled-data accessibility across batch and streaming tasks.

B.3 EXPERIMENTS ON REAL-WORLD DATASET

FedMatch is evaluated on a real-world COVID-19 radiography dataset and with a smaller, different backbone architecture. Across these experiments, the method shows stronger accuracy, faster training, and stable performance, while increasing client participation slightly improves all models.

  • COVID-19 Radiography Dataset: FedMatch outperforms naive federated semi-supervised baselines by around 4%p-10%p in both labeled-data scenarios on the COVID-19 dataset.The experiment uses 10 clients, five labeled examples per class per client, and 100 training rounds.
  • COVID-19 Radiography Dataset: On the COVID-19 dataset, FedMatch trains faster than the baseline models and shows more stable test-accuracy behavior.The comparison includes FedProx-UDA and FixMatch-style naive combinations.
  • Architecture analysis: FedMatch is tested with an AlexNet-Like backbone to assess performance on a smaller and different architecture.The architecture uses convolutional layers with 64, 128, and 256 filters followed by two fully connected layers.
  • Client participation: Increasing the participating-client fraction from 0.05 to 0.10 or 0.20 slightly improves the performance of all models.The study reports no extraordinary phenomenon associated with the fraction of clients per communication round.
Loading 2006.12097v3…