Source-linked AI summary
Preservation of the Global Knowledge by Not-True Distillation in Federated Learning
Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, Se-Young Yun
TL;DR
Federated learning struggles with heterogeneous client data, and the paper identifies forgetting of prior and out-of-local-distribution knowledge as a related problem. It proposes FedNTD, which distills the global model’s predictions for not-true classes during local training. Across various setups, FedNTD achieves state-of-the-art performance without compromising data privacy or adding communication costs.
Problem
Data heterogeneity degrades federated learning performance because clients’ local distributions fail to represent the overall global distribution; the paper examines forgetting as a related concern.
Method
FedNTD performs local-side distillation using the global model’s predictions only for not-true classes, preserving knowledge outside the local distribution.
Results
FedNTD consistently outperforms baselines on all evaluated setups and achieves state-of-the-art results in most cases.
Takeaways & Limitations
Knowledge preservation addresses forgetting in local training while retaining data privacy and avoiding additional communication requirements.
Takeaways & Limitations
Maintaining knowledge outside the local distribution can make local models more prone to share a bias present in the global model.
Abstract
from arXiv · showhide
In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that tackling down forgetting will relieve the data heterogeneity problem. To this end, we propose a novel and effective algorithm, Federated Not-True Distillation (FedNTD), which preserves the global perspective on locally available data only for the not-true classes. In the experiments, FedNTD shows state-of-the-art performance on various setups without compromising data privacy or incurring additional communication costs.
1 Introduction
Federated learning trains a global model by aggregating privacy-preserving local updates, but heterogeneous client distributions make convergence difficult. The paper frames this difficulty as forgetting across rounds and proposes FedNTD to preserve global knowledge during local training.
- Federated learning setting: Federated learning aggregates independently trained client models into a global model without directly accessing clients’ private data.Most computation occurs on client devices, while the server aggregates and redistributes model parameters.
- Data heterogeneity: Data heterogeneity occurs because clients’ local distributions do not represent the overall global distribution, degrading federated learning performance.The paper also relates resolving heterogeneity to greater robustness against partial participation and faster convergence.
- Forgetting in federated learning: Distributional shifts across communication rounds make the global model’s predictions inconsistent and cause forgetting of previously learned class knowledge.The authors test this conjecture by tracking class-wise accuracy as communication rounds proceed.
- Forgetting in federated learning: Local training preferentially forgets global knowledge outside the local distribution, which simple averaging cannot recover.This connects local forgetting to the global model’s difficulty preserving previous knowledge.
- FedNTD: FedNTD preserves the global model’s perspective on locally available data only for not-true classes, aiming to mitigate forgetting and data heterogeneity.The paper reports state-of-the-art performance across various setups without compromising privacy or adding communication costs.
2 Forgetting in Federated Learning
Experiments show that forgetting intensifies with data heterogeneity and is concentrated on knowledge outside each client’s local distribution. The analysis motivates preserving that knowledge to improve alignment between local and global updates.
- 2.1 Global Model Prediction Consistency: In non-IID federated learning, some classes that the previous global model predicted well can suffer significant accuracy drops after updates.IID server predictions remain comparatively even across rounds, whereas NIID predictions are highly inconsistent.
- 2.2 Knowledge Outside of Local Distribution: The out-local distribution assigns higher proportions to classes underrepresented in a client’s local dataset, representing knowledge that the local distribution cannot capture.When the in-local distribution is uniform, the out-local distribution is uniform as well.
- 2.2 Knowledge Outside of Local Distribution: Local training fits models to the in-local distribution while accuracy on the out-local distribution significantly drops, and the aggregated global model also degrades there.The measurements use global and sampled local models across communication rounds.
- 2.3 Forgetting and Local Drift: Preserving out-local knowledge guides local gradients toward the global gradient and reduces gradient diversity.The paper defines gradient diversity Λ as a measure of local-gradient dissimilarity relative to the global function.
3 FedNTD: Federated Not-True Distillation
FedNTD preserves the global model’s perspective during local training by distilling only not-true-class logits, balancing new local knowledge with retained global knowledge.
- Core idea: FedNTD combines cross-entropy with a not-true distillation loss during local-side training.The method preserves the global view only for not-true classes while learning from true-class signals in local labeled data.
- Not-true distillation: The not-true distillation loss matches softened prediction vectors after excluding the true-class logits.Ignoring the true-class logits makes the distillation gradient for that class zero.
- Training procedure: The local algorithm broadcasts the global model, updates sampled clients for local epochs and batches, then aggregates uploaded parameters by averaging.The supplied algorithm passages describe the client-side loop and server aggregation sequence.
- Interpretation: The paper reports that matching true-class and not-true-class logits corresponds to losses on in-local and out-local distributions.This proposition explains why not-true distillation preserves knowledge outside the local distribution.
- Loss design: β controls the trade-off between acquiring local knowledge and preserving previous knowledge on the out-local distribution.The paper interprets this balance through the stability-plasticity dilemma.
4 Experiment
FedNTD is evaluated across multiple datasets and heterogeneous partition strategies, where it consistently outperforms the listed baselines while reducing forgetting without extra privacy or communication burdens.
- Experimental setup: Experiments cover MNIST, CIFAR-10, CIFAR-100, and CINIC-10 with 100 clients, or 200 clients for CINIC-10.Clients are randomly sampled at ratios of 0.1, or 0.05 for CINIC-10.
- Experimental setup: The evaluation uses Sharding and Latent Dirichlet Allocation partitions to vary statistical and distributional heterogeneity.Sharding varies the number of shards per user, while LDA varies class distributions and client dataset sizes.
- Performance: FedNTD consistently outperforms baselines across all setups and achieves state-of-the-art results in most cases.The comparison includes FedAvg and existing methods reported in Table 1.
- Performance: Lower forgetting measure F accompanies better federated-learning performance in the reported experiments.The paper defines smaller F as less forgetting of previous knowledge and reports a close relationship between forgetting and performance.
- Practical comparison: FedNTD avoids the statefulness, additional communication cost, or auxiliary data required by some prior approaches.The paper presents this as a comparison concerning privacy and communication burden.
5 Knowledge preservation of FedNTD
FedNTD’s knowledge preservation benefits heterogeneous federated learning by improving semantic alignment and stabilizing local-model weight divergence, with empirical support from CIFAR-10 analyses.
- Weight Alignment: Weight alignment is assessed through neurons’ class preferences and the proportion matched between two models.A neuron’s class preference is the class producing its largest average activation.
- Weight Alignment: FedNTD significantly enhances weight alignment over FedAvg in non-IID cases while showing little difference in IID settings.Alignment measures the proportion of neurons whose class preferences match across models.
- Distributional perspective: The paper argues that a uniform underlying distribution is the most robust choice when local distributions may be generated arbitrarily.This argument motivates using class-wise global accuracy as an approximation to the global model’s underlying distribution.
- Weight Divergence: The analysis relates weight divergence between global and local models to the distance between their underlying distributions.Figure 8 reports a strong correlation between these two distances over CIFAR-10 experiments.
- Weight Divergence: FedNTD stabilizes weight divergence by providing a better starting point for local training.The paper links this stabilization to knowledge preservation and more even class prediction.
6 Related Work
Prior federated-learning methods address heterogeneity through local regularization, server aggregation changes, forgetting prevention, or knowledge distillation, while FedNTD preserves global knowledge during local training.
- Federated learning: FedAvg aggregates locally trained model parameters by weighted averaging, but heterogeneous data can produce sub-optimal performance.Related methods modify local training or server-side aggregation to address this problem.
- Federated learning: FedNTD belongs to the local-side approach because it preserves global knowledge during local training.The paper contrasts this with server-side modifications that improve aggregation efficacy.
- Forgetting view: FedCurv treats each client as a task and regulates local parameter changes, but requires communicating parameter-wise importance across clients.The paper instead focuses on class-wise forgetting using not-true logits from local data.
- Knowledge distillation: Knowledge-distillation approaches commonly use unlabeled auxiliary data, which may be unavailable, while data-free variants require additional heavy computation.These methods aim to extract knowledge from ensembles of local models or generated pseudo-data.
7 Conclusion
The paper identifies forgetting as a major federated-learning concern and proposes FedNTD to preserve knowledge outside local distributions without additional requirements. It reports benefits while noting that preserving global knowledge can also preserve global-model bias.
- 7 Conclusion: FedNTD applies local-side distillation only to not-true classes to prevent forgetting outside the local distribution.The method is presented as simple, effective, and requiring no additional requirements compared with previous approaches.
- 7 Conclusion: The study links forgetting of knowledge outside local distributions to unstable global convergence in federated learning.
- 7 Conclusion: Preserving knowledge outside local distributions may make locally trained models more prone to retain a biased global model tendency.The paper identifies this as a limitation to consider for machine-learning practitioners.
Checklist
The checklist records the paper’s reproducibility, ethics, scope, and theoretical-result disclosures, including supplemental materials, appendix details, public datasets, and an MIT-licensed codebase.
- Checklist: The authors state that the paper’s claims accurately reflect its contributions and scope and that limitations and potential negative societal impacts are discussed as broader impact.
- Checklist: The paper states that theoretical results include their assumptions, detailed notations in Appendix A, and complete proofs.
- Checklist: Reproduction materials and training details are reported in the supplemental material and appendix, including code, data, instructions, splits, and hyperparameters.
- Checklist: The main experiments report standard deviations, and compute resources are documented, including one Titan-RTX and one RTX 2080Ti GPU.
- Checklist: The experiments use public datasets and cite existing assets, discuss licenses, and identify the code license as MIT.
- Checklist: The checklist marks human-subject procedures and participant compensation as not applicable.
B.2 Datasets
The experiments use four benchmark datasets and two strategies to create heterogeneous client data distributions. The section also situates FedNTD among federated-distillation and continual-learning methods and describes its resource requirements and class-wise evaluation.
- B.2 Datasets: The experiments use MNIST, CIFAR-10, CIFAR-100, and CINIC-10, with cropping, flipping, normalization, and selected Cutout augmentation.
- B.2 Datasets: The implementation uses momentum SGD locally, with learning-rate decay, weight decay, and algorithm-specific hyperparameter selection.
- B.2 Datasets: Client heterogeneity is induced through Sharding and Latent Dirichlet Allocation strategies.Sharding heterogeneity increases as shards per user decreases, while LDA heterogeneity increases as concentration parameter α decreases.
- B.2 Datasets: Existing federated-distillation methods use additional local information or auxiliary data, whereas FedNTD is described as having neither constraint.
- B.2 Datasets: The paper compares additional resource requirements with FedAvg and uses normalized class-wise test accuracy to examine prediction consistency.
- B.2 Datasets: Continual-learning work is presented as related context because training on heterogeneous task sequences can cause catastrophic forgetting.
F Experiment Table with Standard Deviation
This section provides experiment-table captions for accuracy and forgetting measurements on CIFAR-10 under Sharding and LDA settings, alongside broader benchmark accuracy results with standard deviations.
- F Experiment Table with Standard Deviation: Table 7 reports Accuracy@1 (%) on MNIST, CIFAR-10, CIFAR-100, and CINIC-10, with standard deviations and comparisons to FedAvg.
- F Experiment Table with Standard Deviation: Tables 8 and 10 report CIFAR-10 Accuracy@1 under Sharding with s = 2 and include forgetting F in parentheses.
- F Experiment Table with Standard Deviation: Tables 9 and 11 report CIFAR-10 Accuracy@1 under LDA with α = 0.1 and include forgetting F in parentheses.
G.3 Results on ResNet-10 Model
An additional experiment evaluates FedNTD with the ResNet-10 architecture, whose parameter count is about ten times larger than the main experimental model.
- ResNet-10 contains about 10x more parameters than the 2-conv + 2-fc model used in the main experiments.
H Comparison to KD
The comparison experiments examine how not-true distillation affects local learning, global performance, forgetting, feature alignment, and robustness across heterogeneous federated settings. FedNTD generally preserves global knowledge while maintaining local learning and shows consistent gains under severe heterogeneity, though its hyperparameters and distillation loss require care.
- Comparison to KD: Moving the loss toward LNTD significantly improves performance by separating preservation of out-of-local-distribution knowledge from acquisition of local true-class knowledge.
- Comparison to KD: KD improves server performance but produces much lower local performance, whereas FedNTD achieves much higher local performance without hindering local learning.Personalized performance is evaluated on test samples matching each local client's label distribution.
- Comparison to KD: FedNTD's improvement is attributed to a better trade-off between acquiring new local knowledge and preserving old global knowledge.
- Personalized performance of FL methods: SCAFFOLD demonstrates that lower local performance does not necessarily imply worse server convergence, with significantly lower local performance but considerably improved global convergence.
- Comparison to FedAlign: FedNTD consistently improves performance under severe heterogeneity, whereas FedAlign improves performance only at some settings and suffers as heterogeneity increases.
- Feature and loss-landscape analysis: FedNTD also smooths the local loss landscape and produces better-aligned features in both IID and NIID settings.The feature-alignment comparison visualizes last-layer neurons' class preferences.
- Hyperparameter analysis: Too small τ significantly reduces accuracy, while FedNTD is not very sensitive to β.
- Distillation-loss analysis: The MSE variant improves accuracy and reduces forgetting as β increases but can diverge, unlike the original softmax and KL-divergence formulation.