Source-linked AI summary
Federated Mutual Learning
Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, Chao Wu
TL;DR
Federated learning must handle heterogeneous data, objectives, and model requirements while keeping data decentralized. FML combines collaborative generalized-model training with independently trained personalized and customized client models. The paper reports better performance than other federated-learning methods in typical settings and benefits across different models and tasks.
Problem
Federated learning faces data, objective, and model heterogeneity: Non-IID data, differing server and client goals, and clients' need for task- and scene-specific models.
Method
FML trains a generalized model collaboratively while clients continuously train private personalized models through mutual knowledge distillation and can customize their architectures.
Results
FML outperforms other federated-learning methods in typical settings and can handle data, objective, and model heterogeneity.
Takeaways & Limitations
Non-IID data can serve clients personally, while clients with different models and tasks can benefit from federated mutual learning.
Takeaways & Limitations
FML abandons sample-count weighting, treating clients equally because exposing sample counts raises privacy concerns and unequal weighting raises fairness concerns.
Abstract
from arXiv · showhide
Federated learning (FL) enables collaboratively training deep learning models on decentralized data. However, there are three types of heterogeneities in FL setting bringing about distinctive challenges to the canonical federated learning algorithm (FedAvg). First, due to the Non-IIDness of data, the global shared model may perform worse than local models that solely trained on their private data; Second, the objective of center server and clients may be different, where center server seeks for a generalized model whereas client pursue a personalized model, and clients may run different tasks; Third, clients may need to design their customized model for various scenes and tasks; In this work, we present a novel federated learning paradigm, named Federated Mutual Leaning (FML), dealing with the three heterogeneities. FML allows clients training a generalized model collaboratively and a personalized model independently, and designing their private customized models. Thus, the Non-IIDness of data is no longer a bug but a feature that clients can be personally served better. The experiments show that FML can achieve better performance than alternatives in typical FL setting, and clients can be benefited from FML with different models and tasks.
1 Introduction
Federated learning faces data, objective, and model heterogeneity: clients' data distributions differ, server and clients may pursue different goals, and clients may need distinct architectures. FML addresses these heterogeneities by supporting personalized and customized client models alongside collaborative generalized-model training.
- Data Heterogeneity: Federated learning keeps data decentralized for privacy, but client data are typically Non-IID rather than identically distributed.This heterogeneity can reduce accuracy through weight divergence during model-weight averaging.
- Objective Heterogeneity: The server seeks a generalized model, whereas clients may seek personalized models and may perform different classification tasks.FedAvg compromises client personalization while attempting consensus across clients.
- Model Heterogeneity: Clients may require customized models because their hardware, local-data representations, and tasks differ.Local models also involve privacy concerns because they are private client property.
- Federated Mutual Learning: FML enables clients to train personalized models, benefit from collaboration across similar but different tasks, and design customized models.It treats Non-IID data as potentially beneficial for serving clients personally rather than only as a source of global-model difficulty.
2 Related Work
Related work addresses federated learning's data, model, and objective heterogeneities through data sharing, theoretical analysis, separate models, distillation, and personalization. These approaches leave limitations around privacy, architecture control, or personalization under global optimization.
- Data Heterogeneity: Federated learning differs from distributed data-center training because client data remain locally fixed and inaccessible to others.This preserves privacy but produces Non-IID and unbalanced distributions that make training harder and can reduce accuracy.
- Data Heterogeneity: Prior work improves Non-IID training with shared data, privacy-preserving distillation or generation, and convergence analyses for FedAvg.These approaches target weight divergence and convergence under heterogeneous data.
- Model Heterogeneity: MOCHA and ARUBA support separate-model or meta-learning analyses, but model architectures remain controlled by the central server.Knowledge-distillation frameworks enable independently designed models, though the cited passage presents them as prior alternatives to FML.
- Objective Heterogeneity: Personalization studies report that some clients may not benefit from the global model and that optimizing only global accuracy can hinder personalization.Small local datasets can also cause the global model to overfit local data and weaken personalization ability.
3 Preliminaries
Typical federated learning trains a shared model by optimizing local objectives and aggregating client weights, but Non-IID data and differing objectives make direct local-model averaging problematic. Knowledge distillation and deep mutual learning provide mechanisms for transferring information between models, including models with different architectures.
- Typical Federated Learning Setup: FedAvg trains one shared model by repeatedly updating client copies on private data and aggregating their weights at the server.The global model is optimized over decentralized data through local gradient updates followed by weighted averaging.
- Typical Federated Learning Setup: Non-IID client distributions can make local objectives differ from the global objective, causing direct aggregation to face weight divergence and accuracy reduction.The expected local loss need not equal the global loss when client distributions differ from the joint distribution.
- Knowledge Distillation: Knowledge distillation transfers information from a powerful teacher to a lighter student through cross-entropy and KL-divergence terms.Teacher predictions provide soft targets that can act as a regularizer for the student.
- Knowledge Distillation: FML replaces one-way teacher-student transfer with deep mutual learning, allowing two models to train simultaneously while transferring knowledge in both directions.The two models optimize their own losses while encouraging prediction consensus, and they may use different architectures.
- Knowledge Distillation: In FML, each client trains a meme model and a personalized model that learn mutually during local updates.The figure describes two client-side models trained over private data, with mutual learning performed for several epochs.
4 Methodology
FML rethinks federated learning by jointly training generalized and personalized models, while allowing different clients to share partial models and use customized architectures. Its mutual-learning process transfers knowledge bidirectionally between server-linked meme models and private personalized models.
- Rethinking Federated Learning: FML addresses objective heterogeneity by reconsidering federated learning as joint training of generalized and personalized models.The generalized model fits the joint distribution, while personalized models fit each client’s distribution.
- Rethinking Federated Learning: FML allows clients to share partial models, such as encoders or other modules, rather than requiring a complete end-to-end model.This supports clients with different local objectives and architectures.
- Federated Mutual Learning: Each client maintains a meme model for global knowledge transfer and a personalized model designed for its own data and task.Clients train these models mutually through knowledge distillation instead of training the personalized model directly on the global model.
- Federated Mutual Learning: During local updates, deep mutual learning transfers knowledge in both directions between meme and personalized models over private data.The local and meme losses combine supervised classification terms with bidirectional KL-divergence terms controlled by α and β.
- Federated Mutual Learning: After local training, clients send meme models to the server, which averages them into the next global model.The global model learns a generalized representation, while personalized models continue training privately and distilling knowledge from meme models.
5 Experiments
The experiments evaluate FML on image-classification datasets under IID and increasingly difficult Non-IID settings, first against FedAvg and FedProx and then across heterogeneous models and tasks. FML performs better in most typical settings and supports personalized models and different tasks through shared knowledge.
- Experimental Setup: The study evaluates FML with MLP, LeNet5, CNN1, and CNN2 on MNIST, CIFAR10, and CIFAR100.Experiments use five clients in simulated IID and three Non-IID settings, with Non-IID(3) being the most extreme.
- Typical FL Settings: FML outperforms FedAvg and FedProx in most tested global-model settings across IID and Non-IID data.Global-model accuracy decreases as data heterogeneity becomes more difficult.
- Typical FL Settings: In Non-IID training, FML follows a steadier trajectory than the severe oscillations observed for FedAvg and FedProx.Figure 3 attributes the regularizing effect to the DKL loss item in deep mutual learning.
- Data Heterogeneity: On private validation data, FML trains personalized models intended to fit each client’s distribution rather than relying only on the generalized global model.Figure 4 reports that FML rises rapidly and stabilizes at a high level, improving both stability and accuracy relative to the baselines.
- Model and Objective Heterogeneity: FML supports clients using different architectures and tasks through a shared model or representation.Experiments compare personalized models with independent training and use LeNet5 and CNN1 for CIFAR10 and CIFAR100 tasks with a shared CNN2 representation.
6 Discussion
The discussion highlights model-capacity spillovers, dynamic loss weighting, and privacy and fairness constraints in FML. It also frames customized local models and client-level treatment as important design considerations.
- Models with low capabilities can improve through interaction with a higher-capability model, while one badly trained model has little effect on others.This phenomenon is described as the catfish effect.
- Dynamic proportions of cross-entropy and KL losses can improve both global and local model performance during training.The discussion reports that these proportions matter significantly at different training stages.
- FML treats customized local models as private property that should be protected from being stolen.
- FML abandons client sample counts so the server does not learn potentially sensitive quantities and clients receive equal treatment.The change addresses both privacy concerns and fairness concerns arising from unequal sample counts.
Ethical Impact
FML is presented as a potential basis for cross-disciplinary data cooperation, especially in cross-silo settings. Its suitability is narrower for cross-device settings where clients may not need customized models.
- Federated learning is framed as a business model for cross-disciplinary enterprise-level data cooperation and globalization.
- Cross-silo federated learning can benefit from FML because clients may have sufficient data and require customized local models.
- Cross-device federated learning might not suit FML when clients do not require customized models.