Source-linked AI summary
Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, Koji Yamamoto
TL;DR
Typical federated learning incurs communication overhead that grows with model size, motivating a scalable approach that preserves benchmark performance. DS-FL exchanges model outputs over unlabeled open data, and experiments show similar or higher accuracy with substantially lower communication costs.
Problem
Typical FL periodically exchanges model parameters, so communication overhead scales with model size and can bottleneck practical learning on wireless mobile devices.
Method
DS-FL exchanges local model outputs, uses them to label shared unlabeled open data, and retrains local models on the resulting augmented dataset; ERA sharpens aggregated logits.
Results
DS-FL with ERA achieves similar or superior accuracy to benchmark FL while reducing communication costs by up to 99%.
Takeaways & Limitations
Output exchange over unlabeled open data provides a communication-efficient alternative to parameter-exchange FL, with ERA also improving convergence and robustness under non-IID data.
Takeaways & Limitations
DS-FL must address higher global-logit entropy caused by non-IID data and corrupted logits or noisy open data, which can reduce model performance.
Abstract
from arXiv · showhide
This study develops a federated learning (FL) framework overcoming largely incremental communication costs due to model sizes in typical frameworks without compromising model performance. To this end, based on the idea of leveraging an unlabeled open dataset, we propose a distillation-based semi-supervised FL (DS-FL) algorithm that exchanges the outputs of local models among mobile devices, instead of model parameter exchange employed by the typical frameworks. In DS-FL, the communication cost depends only on the output dimensions of the models and does not scale up according to the model size. The exchanged model outputs are used to label each sample of the open dataset, which creates an additionally labeled dataset. Based on the new dataset, local models are further trained, and model performance is enhanced owing to the data augmentation effect. We further highlight that in DS-FL, the heterogeneity of the devices' dataset leads to ambiguous of each data sample and lowing of the training convergence. To prevent this, we propose entropy reduction averaging, where the aggregated model outputs are intentionally sharpened. Moreover, extensive experiments show that DS-FL reduces communication costs up to 99% relative to those of the FL benchmark while achieving similar or higher classification accuracy.
1 INTRODUCTION
The paper proposes DS-FL to address communication overhead that grows with model size while preserving benchmark-level performance under non-IID data. It exchanges model outputs and uses unlabeled open data for distillation, with ERA reducing ambiguity from heterogeneous logits.
- Typical FL parameter exchange incurs communication overhead that scales with model size, limiting large models on bandwidth-constrained mobile networks.
- The paper frames its goal as achieving model-size-scalable communication efficiency while maintaining performance comparable to benchmark FL.
- DS-FL exchanges local logits, aggregates them into global teacher knowledge, and transfers that knowledge to local student models; communication depends only on output dimension.
- Federated distillation can perform poorly under non-IID data because global logits retain information similar to local labels, making retraining nearly identical to local training.
- Unlike federated distillation, DS-FL labels shared unlabeled open data with teacher knowledge, creating additional labeled data for local retraining and data augmentation.
- ERA intentionally reduces global-logit entropy, producing faster convergence and higher classification accuracy than an FL benchmark under non-IID distributions.
- The study excludes additional requirements such as performance under unbalanced or massively distributed data.
2 DISTILLATION-BASED SEMI-SUPERVISED FEDERATED LEARNING METHOD
The method replaces parameter exchange with model-output exchange and distills aggregated logits through shared data. DS-FL uses unlabeled data to create new labels, while ERA addresses ambiguous aggregated logits under non-IID distributions.
- The proposed DS-FL targets communication efficiency while seeking similar or higher model performance than benchmark schemes.
- Benchmark 1: Federated Learning with model parameter exchange: Benchmark FL exchanges model parameters or gradients through iterative client updates, server aggregation, and global-model broadcast.
- Benchmark 2: Federated Distillation: Federated distillation exchanges per-class logits, aggregates them as teacher knowledge, and retrains each client using attached labels and broadcast logits.
- Benchmark 2: Federated Distillation: Under strong non-IID data, local models can overfit one-hot labels, causing averaged global logits to resemble those labels.
- Benchmark 2: Federated Distillation: In MNIST, strong non-IID data makes FD’s class-“7” probability resemble class “0” more closely than under IID or weak non-IID data.
- Proposed DS-FL: DS-FL uses global logits to assign classes to shared unlabeled samples, creating a new labeled dataset for further local-model training.
3 ENTROPY REDUCTION AGGREGATION
ERA intentionally reduces the entropy of aggregated global logits to address ambiguous knowledge caused by heterogeneous non-IID client outputs. It sharpens the global logit before distillation, with temperature-controlled softmax processing, and is evaluated against simple aggregation.
- Motivation: ERA intentionally reduces the entropy of global logits to improve training under heterogeneous non-IID data distributions.High-entropy global logits can fail to identify the class of an unlabeled sample and hinder training.
- Motivation: Under non-IID data, simple averaging produces higher global-logit entropy than under IID data, particularly during early training.The entropy exceeds 2.0 early in training, approximately the upper limit for ten-class classification, making the resulting knowledge uninformative about class identity.
- Procedure: The DS-FL procedure initializes and distributes open data, updates local models, uploads local logits, aggregates them with ERA or SA, broadcasts the global logit, and performs distillation.Steps 1–6 repeat for multiple training rounds.
- Procedure: ERA applies a temperature-controlled softmax transformation to averaged local logits, producing a sharper global logit.The paper describes the resulting output as having lower entropy; ERA uses T = 0.1 in the experimental evaluation.
- Procedure: Softmax temperature controls entropy: higher temperature increases output entropy, whereas lower temperature decreases it.ERA uses a lower temperature than the original knowledge-distillation example, which uses T = 20.
- Comparison: ERA is compared with simple aggregation, which averages uploaded local logits without the entropy-reduction transformation.The baseline is denoted SA in the paper.
4 EXPERIMENTAL EVALUATION
Experiments across image and text classification evaluate communication efficiency, accuracy, convergence, and robustness under non-IID data and attack conditions. DS-FL with ERA generally matches or exceeds FL accuracy while reducing communication costs and improving convergence.
- Communication efficiency: DS-FL communication costs are smaller than FL costs and do not depend on model size, although they exceed FD costs because DS-FL uploads per-sample logits.The FD cost is reported as 100 times smaller than DS-FL because FD uploads per-class logits while DS-FL uploads per-sample logits.
- Communication efficiency: Across MNIST, Fashion-MNIST, IMDb, and Reuters, DS-FL outperforms FL in cumulative communication cost while achieving comparable accuracy.For IMDb and Reuters, FD accuracy is 23.3% and 39.0% lower than FL, respectively.
- Convergence: ERA reaches convergence with lower cumulative communication cost than SA while obtaining almost the same accuracy on IMDb and Reuters.The reported difference is attributed to ERA’s acceleration effect.
- Communication efficiency: 99.0% lower ComU@65% and 99.4% lower ComU@75% are achieved by DS-FL with ERA than FL on Fashion-MNIST.On Reuters, ERA achieves 99.4% lower ComU@65% and higher Top-Accuracy than FL.
- Classification accuracy: DS-FL with ERA achieves similar or superior Top-Accuracy to FL across tasks, except IMDb, where it is 4.7% lower than FL and 18.6% higher than FD.On MNIST, ERA reaches 98.5% Top-Accuracy when Io = 40,000, only 0.2% below FL; it exceeds FL on Fashion-MNIST and Reuters.
- Robustness: ERA maintains Top-Accuracy as noisy classes increase, whereas DS-FL with SA and FL decline under IID label noise.For noisy open data, ERA’s Top-Accuracy decrease is smaller than SA’s, and its entropy increase is also smaller.
5 CONCLUSION
DS-FL exchanges model outputs over an unlabeled open dataset to scale communication independently of model size while preserving accuracy. ERA further improves communication efficiency and robustness under non-IID data, though broader data settings remain future work.
- DS-FL exchanges model outputs over an unlabeled open dataset, labels that data, and retrains local models for augmentation.The approach is designed to avoid communication overhead scaling with model size.
- ERA aggregates logits to accelerate convergence and improve robustness against heterogeneous outputs under non-IID data.The method intentionally reduces entropy in the aggregated global logit.
- DS-FL with ERA outperformed benchmark FL in communication cost and robustness while achieving similar or superior accuracy.
- Future work will consider device characteristics, previously uploaded logits, and unbalanced, massively distributed non-IID data.The reliability of clients and control of uploaded-logit influence remain unresolved.