PaperlayerView exact revision on arXiv

Source-linked AI summary

When does the student surpass the teacher? Federated Semi-supervised Learning with Teacher-Student EMA

Jessica Zhao, Sayan Ghosh, Akash Bharadwaj, Chih-Yao Ma

arXiv:2301.10114v1cs.LG

TL;DR

Federated semi-supervised learning seeks to use client-side unlabeled data while navigating privacy, statelessness, and communication constraints. FedSwitch uses a teacher-student EMA framework with local teacher adaptation and adaptive pseudo-label selection, and is reported to achieve state-of-the-art performance on CIFAR-10 and Fashion MNIST.

  • Problem

    Federated semi-supervised learning seeks to use client-side unlabeled data, while existing approaches can compromise privacy or assume stateful clients and EMA synchronization poses communication and generalization trade-offs.

    Source: §1 · pp. 1–2

  • Method

    FedSwitch uses federated teacher-student EMA with local teacher adaptation and selects teacher or student pseudo-labels using a KL-divergence metric.

    Source: §1 · p. 2

  • Results

    FedSwitch is reported to achieve state-of-the-art performance on CIFAR-10 and Fashion MNIST in federated image classification.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper reports that FedSwitch supports privacy and client statelessness with tractable communication cost in practical federated settings.

    Source: §4 · p. 5, §7 · p. 9

    Federated teacher-student EMA can add two-way model-transfer overhead, increasing wall-clock time to reach a target performance.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Semi-Supervised Learning (SSL) has received extensive attention in the domain of computer vision, leading to development of promising approaches such as FixMatch. In scenarios where training data is decentralized and resides on client devices, SSL must be integrated with privacy-aware training techniques such as Federated Learning. We consider the problem of federated image classification and study the performance and privacy challenges with existing federated SSL (FSSL) approaches. Firstly, we note that even state-of-the-art FSSL algorithms can trivially compromise client privacy and other real-world constraints such as client statelessness and communication cost. Secondly, we observe that it is challenging to integrate EMA (Exponential Moving Average) updates into the federated setting, which comes at a trade-off between performance and communication cost. We propose a novel approach FedSwitch, that improves privacy as well as generalization performance through Exponential Moving Average (EMA) updates. FedSwitch utilizes a federated semi-supervised teacher-student EMA framework with two features - local teacher adaptation and adaptive switching between teacher and student for pseudo-label generation. Our proposed approach outperforms the state-of-the-art on federated image classification, can be adapted to real-world constraints, and achieves good generalization performance with minimal communication cost overhead.

1. Introduction

Federated semi-supervised learning uses client-side unlabeled data, but existing approaches may conflict with privacy and stateless-client requirements. FedSwitch combines local teacher adaptation with adaptive teacher/student pseudo-labeling and is reported to achieve state-of-the-art results on CIFAR-10 and Fashion MNIST.

1. Introduction

  • Existing FSSL approaches can rely on assumptions that compromise privacy or require stateful clients, limiting fit with industrial privacy controls.The introduction identifies FedMatch as an example of privacy-relaxing design.Source: §1 · p. 1
  • Federated teacher-student EMA is difficult to integrate because model placement and synchronization frequency trade off communication cost against generalization performance.Unlike centralized EMA updates after every batch, federated updates require design choices about server and client models and weight communication.Source: §1 · pp. 1–2
  • FedSwitch achieves state-of-the-art federated image-classification performance on CIFAR-10 and Fashion MNIST, improving over FedMatch, FedRGD and FedProx-FixMatch on CIFAR-10.The introduction reports the dataset-level result without giving numerical scores.Source: §1 · p. 2
  • FedSwitch combines local teacher adaptation with choosing either teacher or student pseudo-labels using a KL-divergence metric.The paper presents this as its federated teacher-student EMA mechanism.Source: §1 · p. 2

2. Related Work

Related work covers federated optimization and privacy or communication improvements, semi-supervised consistency methods, and approaches that combine federated and semi-supervised learning. The cited FSSL methods differ in how they regularize or aggregate client learning, with FedMatch noted to violate practical privacy constraints.

2.1. Federated Learning

  • Federated learning builds on FedAvg, with FedProx adding a proximal term; other work targets privacy or communication costs.The cited examples include differentially private training and communication reduction.Source: §2.1 · p. 2

2.2. Semi-Supervised Learning

  • Semi-supervised methods combine input augmentation or perturbation with consistency regularization to encourage invariant predictions across semantically identical transformed inputs.Examples include adversarial perturbations, mixed inputs, and feature-space prototypes.Source: §2.2 · p. 2

2.3. Federated Semi-Supervised Learning

  • FedMatch uses inter-client consistency but violates practical privacy constraints, while FedRGD groups client updates to reduce gradient diversity.FedRGD replaces FedAvg-based averaging with grouping-based averaging.Source: §2.3 · p. 2

3. Preliminaries

Practical FSSL must account for privacy, client statelessness, and communication costs. The paper adapts teacher-student EMA to federated training and describes communication-saving designs for that setting.

3.1. Real-world considerations for practical FSSL

  • Existing FSSL approaches have practical gaps: FedMatch does not adequately provide client privacy, FedSEAL/FedCon assume stateful clients, and none use teacher-student EMA.The comparison concerns privacy guarantees similar to standard supervised FL and client statelessness.Source: Table 1 · p. 3
  • Stateless cross-device FSSL clients retain no information from earlier rounds, but FedCon and FedSEAL rely on client-side information across rounds.The section notes that many cross-device clients participate only once, making statelessness a deployment desideratum.Source: §3.1 · p. 3

3.2. FSSL with teacher-student EMA models

  • Teacher-student EMA uses a separate teacher to generate pseudo-labels and updates it slowly, which can stabilize pseudo-labels and improve generalization.The paper says this is the first demonstration of EMA effectiveness in FSSL.Source: §3.2 · p. 3
  • Federated EMA maintains local teacher models alongside global models, updating local teachers from local students and the global teacher from the aggregated global student.The global teacher is sent to clients to keep local and global teachers synchronized.Source: §3.2 · p. 3

3.3. Practical challenges with EMA models in FL

  • Sending teacher and student models in both directions adds communication overhead and can increase production wall-clock time to reach a target performance.TS-Server EMA avoids sending local teachers to the server: clients send student updates, the server aggregates them and EMA-updates the global teacher, which clients use for pseudo-labels.Source: §3.3 · p. 4, Figure 1 · p. 4

4. Proposed Method: FedSwitch

FedSwitch modifies teacher-student EMA with local teacher adaptation and adaptive switching between teacher and student for pseudo-label generation. It uses KL-divergence relative to an IIDness prior to choose the pseudo-label source.

4. Proposed Method: FedSwitch

  • The method’s two modifications to TS-Server EMA are local teacher adaptation and adaptive teacher-student switching.The paper evaluates their effects on generalization with ablation studies.Source: §4 · p. 4
  • FedSwitch updates the local teacher from the student after every batch, retaining EMA smoothing without sending teacher updates to the server.This avoids increasing client-to-server transport cost.Source: §4 · p. 5
  • FedSwitch addresses non-IID clients, where using only the EMA teacher for pseudo-labels is often sub-optimal.The teacher acts as a temporal ensemble that stabilizes pseudo-label quality.Source: §4 · p. 5
  • The server compares teacher and student prediction-distribution KL-divergences with β and selects the model closer to that IIDness prior for pseudo-labels.If the teacher is closer, both models are sent next round; if the student is closer, only the student is sent.Source: §4 · pp. 5–6

5. Experiments

The experiments use CIFAR-10 and Fashion MNIST under labels-at-client and labels-at-server configurations. They vary client data imbalance and describe the training setups, tuning, and KL-based measurements.

5.1. Datasets

  • CIFAR-10 uses 56,000 training, 2,000 validation, and 2,000 test images; Fashion MNIST uses 63,000, 3,500, and 3,500, respectively.Both datasets use labels-at-client and labels-at-server configurations; CIFAR-10 also covers batch-IID and batch-nonIID splits.Source: §5.1 · p. 6

5.2. Experimental setup

  • CIFAR-10 experiments use 100 clients with five participating each round, while Fashion MNIST uses 10 clients with all participating per round.Fashion MNIST streams unlabeled data over 10 training rounds and uses weak flip-and-shift augmentation.Source: §5.2 · p. 6

5.3. Hyper-parameters

  • The authors tune six hyperparameter types and run five seeded trials per method, recording the average results.The tuned parameters include learning rate, pseudo-label threshold, unlabeled-loss weight, weight decay, optimizer momentum, and EMA decay where applicable.Source: §5.3 · p. 6

5.4. Performance under increasing non-IIDness

  • Non-IIDness is simulated with Dirichlet α values from 0.01 to 100, and the analysis measures client-averaged KL divergence and baseline pseudo-label-to-ground-truth KL ratios.Lower α represents more extreme non-IIDness; larger α distributes samples more uniformly across clients.Source: §5.4 · p. 7

6. Experimental Results

The experiments compare FedSwitch with federated SSL and teacher-student EMA baselines on CIFAR-10 and Fashion MNIST. They also examine performance as client data become more non-IID, including how pseudo-label distributions respond to that imbalance.

6.1. Results on CIFAR-10 Dataset

  • FedSwitch outperforms FedMatch, FedRGD, TS-Server EMA, and TS-Client EMA on CIFAR-10.Performance is generally higher on Batch-IID than Batch-NonIID for nearly all methods except FedMatch; under labels-at-server, weak augmentations avoid collapse or premature convergence unlike FedMatch.Source: §6.1 · p. 7, Table 2 · p. 8

6.2. Results on Fashion MNIST Dataset

  • On Fashion MNIST, FedSwitch improves accuracy by 11% over FedMatch for labels-at-client and test accuracy by 2% over FedProx-FixMatch for labels-at-server.It also outperforms FedProx-FixMatch under labels-at-client.Source: §6.2 · p. 7, Table 3 · p. 8

6.3. Analysis of performance under Non-IIDness

  • As client data become more imbalanced, TS-Server EMA pseudo-label KL divergence saturates around 0.75, while FedSwitch remains relatively robust and adapts predictions to client distributions.FedSwitch uses local student and teacher adaptation and switches between them for pseudo-label generation.Source: §6.3 · p. 9, Figure 4 · p. 9

7. Conclusion

The conclusion presents FedSwitch as a federated teacher-student EMA framework for image classification, where integrating EMA requires synchronization across global and client-local models. It reports superior performance on CIFAR-10 and Fashion MNIST, with tractable communication cost and practical privacy and statelessness properties.

7. Conclusion

  • Integrating federated EMA requires synchronization between global and client-local student and teacher models; FedSwitch also satisfies privacy and client-statelessness requirements for practical settings.Source: §7 · p. 9
  • On CIFAR-10 batched-nonIID and Fashion MNIST streaming-nonIID, FedSwitch performs better at fewer global rounds and is more stable than baseline approaches.Source: Figure 5 · p. 9
  • Across CIFAR-10 and Fashion MNIST, FedSwitch achieves performance superior to existing state-of-the-art approaches while using teacher-student EMA updates with tractable communication cost.Source: §7 · p. 9

Related papers

Loading 2301.10114v1…