Source-linked AI summary
Federated Meta-Learning with Fast Convergence and Efficient Communication
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, Xiuqiang He
TL;DR
Federated learning must train across privacy-sensitive, distributed, and heterogeneous mobile-device data without collecting raw data centrally. This paper proposes FedMeta, which shares a parameterized algorithm rather than a global model and evaluates it on federated and industrial recommendation datasets. FedMeta reports faster convergence, 2.82-4.33 times lower required communication cost, and 3.23%-14.84% higher final accuracies than FedAvg.
Problem
Federated learning faces statistical and systematic challenges when collaboratively training models across privacy-sensitive, distributed, and personalized mobile-device data.
Method
FedMeta shares a parameterized meta-learner, treating clients as tasks and using meta-learning algorithms including MAML and Meta-SGD.
Results
FedMeta improves accuracy, convergence speed, and communication cost across federated datasets, and outperforms stand-alone and unified federated models in industrial recommendation.
Takeaways & Limitations
Sharing a parameterized algorithm preserves client privacy without collecting raw data and provides a flexible alternative to sharing a global model.
Takeaways & Limitations
The current work leaves model-attack privacy analysis and online APP-recommendation deployment for future work.
Abstract
from arXiv · showhide
Statistical and systematic challenges in collaboratively training machine learning models across distributed networks of mobile devices have been the bottlenecks in the real-world application of federated learning. In this work, we show that meta-learning is a natural choice to handle these issues, and propose a federated meta-learning framework FedMeta, where a parameterized algorithm (or meta-learner) is shared, instead of a global model in previous approaches. We conduct an extensive empirical evaluation on LEAF datasets and a real-world production dataset, and demonstrate that FedMeta achieves a reduction in required communication cost by 2.82-4.33 times with faster convergence, and an increase in accuracy by 3.23%-14.84% as compared to Federated Averaging (FedAvg) which is a leading optimization algorithm in federated learning. Moreover, FedMeta preserves user privacy since only the parameterized algorithm is transmitted between mobile devices and central servers, and no raw data is collected onto the servers.
1 Introduction
FedMeta applies meta-learning to federated learning, sharing a parameterized algorithm rather than a global model to address privacy, non-IID, and personalized client data. The framework is evaluated against FedAvg for accuracy, computation, and communication costs.
- Federated learning keeps raw, privacy-sensitive client data decentralized while collaboratively training a shared model on a central server.
- FedMeta shares a parameterized algorithm that clients use for local model training, while query-set results update the algorithm on the server.
- Non-IID and highly personalized decentralized data motivate treating clients as tasks within a federated meta-learning framework.
- FedMeta supports more flexible sharing than a global model because each client can fast-train a task-specific model from the shared algorithm.
- FedMeta integrates MAML and Meta-SGD, and experiments compare them with FedAvg on accuracy, computation cost, and communication cost.
2 Related Work
Prior meta-learning work studied unified task formats, while federated optimization methods targeted global-model training. FedMeta instead examines meta-learning in real-world federated settings by treating clients as tasks.
- Initialization-based meta-learning learns model initializations that adapt to new tasks with limited samples after a small number of gradient descents.
- Earlier meta-learning studies generally assumed tasks with a unified form, leaving variation in real-world federated tasks less explored.
- Federated optimization methods improve convergence or system overhead but generally aim to learn one large global model across clients.
- FedMeta treats each client as a task and trains a well-initialized model for rapid adaptation rather than a global model ingesting all tasks.
- The framework includes MAML, FOMAML, and Meta-SGD, allowing model-agnostic meta-learning for non-convex problems.
3 Federated Meta-Learning
The federated meta-learning procedure meta-trains a parameterized algorithm across task-specific support and query sets. MAML and Meta-SGD are implemented as running examples for server-client training.
- FedMeta implements meta-learning algorithms in a federated setting, with MAML and Meta-SGD presented as running examples.
- Meta-learning updates a parameterized algorithm A_ϕ across tasks so it can quickly train models for new tasks.
- Each task contains disjoint support and query sets; the algorithm trains a model on support data and evaluates it on query data.
- The query-set test loss measures the algorithm’s training ability and drives its outer update to minimize that loss.
- On the server, each episode samples clients and distributes θ for MAML or (θ, α) for Meta-SGD.
10 end
The client stage samples local support and query data as part of the federated meta-learning procedure.
- Each client samples a support set and a query set for local meta-learning.
18 Return gu to server
FedMeta incorporates meta-learning into federated training by collaboratively optimizing a shared algorithm or initialization from client data. Clients perform task-specific training locally, while the server updates the shared parameters using clients’ test losses without collecting raw data.
- Meta-training: Meta-training proceeds episodically, with batches of tasks sampled from a task distribution and the algorithm optimized using their query performance.This is the objective underlying the MAML-style meta-learning procedure.
- Client-side training: Each client trains a task-specific model on its support set and evaluates it on a disjoint query set.The query loss is used as the outer-loop signal for optimizing the meta-learner.
- Meta-SGD: Meta-SGD jointly learns the initialization θ and a coordinate-wise inner learning-rate vector α.Both θ and α are updated in the outer loop using gradients of the query loss.
- FedMeta framework: FedMeta meta-trains an algorithm across client data rather than directly maintaining only a shared model.In MAML, the shared initialization is adapted into task-specific models through inner-loop updates.
- Federated communication: Communication rounds correspond to meta-learning episodes, with the server collecting client query losses to update the shared initialization.For Meta-SGD, α is transmitted as part of the algorithm parameters; raw client data remains local.
4 Experiments
FedMeta is evaluated on LEAF benchmarks and a production recommendation task against federated-learning baselines, measuring accuracy, convergence, system overhead, fairness, and adaptation to new clients. Across these experiments, FedMeta generally converges faster and achieves higher accuracy, with lower communication overhead and task-dependent fairness and computation trade-offs.
- Evaluation Scheme: FedMeta is evaluated on LEAF datasets and a real-world industrial recommendation task, with clients divided into training, validation, and testing groups to assess generalization to new clients.Each client’s data is divided into support and query sets, and the support fraction p is varied to study adaptation with limited data.
- LEAF Datasets: FedMeta methods achieve higher final accuracy with faster and more stable convergence than FedAvg on the LEAF datasets.MAML and Meta-SGD have similar performance on FEMNIST and Shakespeare, while Meta-SGD performs significantly better than MAML on Sent140.
- System Overhead: 2.82-4.33 times: FedMeta reduces the required communication cost for reaching target accuracies across the evaluated LEAF tasks.FOMAML has the lowest computation cost for FEMNIST and Sent140, whereas FedAvg is about 5 times cheaper than MAML and Meta-SGD for Shakespeare.
- Fairness and Production Task: FedMeta’s fairness effects vary by task: it improves fairness for FEMNIST, remains comparable or sacrifices fairness for higher mean accuracy on language modeling tasks, and yields higher-accuracy production recommendation models than baselines.In the production task, MAML + NN and Meta-SGD + NN generally outperform the baselines, while Meta-SGD trains models with only 100 gradient steps.
5 Conclusion and Future Work
FedMeta addresses statistical and systematic challenges in federated learning, improving accuracy, convergence speed, and communication cost across federated datasets. The work also validates FedMeta in industrial recommendation while identifying privacy attacks and online deployment as future directions.
- FedMeta achieves improvements in accuracy, convergence speed, and communication cost across a suite of federated datasets.
- FedMeta outperforms both stand-alone models and unified federated-learning models in an industrial recommendation scenario.
- The authors plan to study whether sharing a meta-learner provides additional privacy advantages against model attacks.
- The authors also plan online deployment for APP recommendation, which requires further engineering work.
A.1 Datasets and Models
The evaluation uses four datasets spanning image classification, language modeling, sentiment classification, and industrial recommendation. Each task uses a task-specific model and federated client structure.
- FEMNIST is a 62-class image-classification task partitioned by digit or character writer, using a convolutional neural network.
- Shakespeare is a 53-class next-character prediction task in which each speaking role is treated as a separate client.
- Sent140 is a binary sentiment-classification task built from tweets annotated using emoticons, with a two-layer LSTM classifier.
- The production dataset contains 2,400 services and 9,369 clients for an industrial recommendation task.
A.2 Implementation Details
The implementation uses TensorFlow and Adam across FedAvg and FedMeta variants, with client sampling, weighted server aggregation, and tuned learning rates.
- FedAvg, FedAvg(Meta), and FedMeta with MAML, FOMAML, and MetaSGD are implemented in TensorFlow.
- Adam is used as the local optimizer for all evaluated approaches.
- Clients are uniformly sampled each communication round, and server updates weight local models by local data-point counts.
- The numbers of active clients per round are 4 for FEMNIST, 50 for Shakespeare, and 60 for Sent140.
- Table 4 specifies the learning-rate setup for the LEAF experiments.
A.3 Additional Experiments on LEAF
Additional experiments examine FedMeta across support fractions, recommendation settings, convergence episodes, and training steps. The results emphasize stronger low-data adaptation and more efficient model training.
- Support fraction experiments: At 90% support, the convergence curves of FedAvg(Meta) and FedMeta almost coincide, narrowing their gap relative to lower support.
- Support fraction experiments: Meta-learning methods have greater advantage in the low-data regime than in medium- or large-data regimes.
- Convergence comparison: Meta-SGD converges faster than MAML, while the other three methods converge within 20000 episodes.
- Convergence comparison: Meta-SGD + NN outperforms the best baseline NN after 4000 episodes.
- Training efficiency: After 10000 steps, LR and NN converge, while NN remains below Meta-SGD + NN and LR is below or marginally above Meta-SGD + LR.
- Training efficiency: Meta-SGD trains models with only 100 steps, making it more efficient than training models from scratch with non-parametric optimization.