Source-linked AI summary
Practical One-Shot Federated Learning for Cross-Silo Setting
Qinbin Li, Bingsheng He, Dawn Song
TL;DR
Existing federated learning approaches often require many communication rounds, while one-shot methods have limited model support and lack privacy guarantees. FedKT uses knowledge transfer to support arbitrary classification models and flexible differential privacy, and experiments report stronger accuracy with one communication round.
Problem
Existing one-shot federated learning methods support only specific models and provide no privacy guarantees, limiting their practical use in cross-silo settings.
Method
FedKT uses a two-tier knowledge-transfer framework with public unlabelled data to learn classification models in a single communication round and provide differential privacy options.
Results
FedKT achieves much better accuracy than other state-of-the-art federated learning algorithms with a single communication round across different models and tasks.
Takeaways & Limitations
FedKT is presented as a practical one-shot solution for model-based sharing in cross-silo federated learning.
Takeaways & Limitations
The privacy analysis is data-dependent, so publishing the resulting privacy budget may create potential privacy breaches.
Abstract
from arXiv · showhide
Federated learning enables multiple parties to collaboratively learn a model without exchanging their data. While most existing federated learning algorithms need many rounds to converge, one-shot federated learning (i.e., federated learning with a single communication round) is a promising approach to make federated learning applicable in cross-silo setting in practice. However, existing one-shot algorithms only support specific models and do not provide any privacy guarantees, which significantly limit the applications in practice. In this paper, we propose a practical one-shot federated learning algorithm named FedKT. By utilizing the knowledge transfer technique, FedKT can be applied to any classification models and can flexibly achieve differential privacy guarantees. Our experiments on various tasks show that FedKT can significantly outperform the other state-of-the-art federated learning algorithms with a single communication round.
1 Introduction
Existing federated learning methods rely on iterative communication, creating practical problems for cross-silo settings. FedKT addresses these constraints with one-shot knowledge transfer, broad model support, privacy mechanisms, and strong experimental accuracy.
- Motivation: Cross-silo federated learning makes multi-round participation impractical and may expose models to inference attacks.Organizations cannot always sustain repeated training, while attacks can occur across rounds.
- Motivation: One-shot federated learning uses a single communication round, making model markets and other cross-silo applications more suitable.Parties can upload or sell local models for collective final-model training.
- Research gap: Existing one-shot methods are usually specialized for particular model architectures and provide no privacy guarantees.These limitations are especially important when models may be sold to unknown parties, including attackers.
- FedKT: FedKT is a knowledge-transfer-based one-shot algorithm designed to support any classification model.Its two-tier knowledge-transfer framework uses unlabelled public data and covers both differentiable and non-differentiable models.
- FedKT: FedKT provides differentially private versions and experimentally achieves much better accuracy than other federated learning algorithms with one communication round.The paper reports experiments across various models and tasks.
2 Background and Related Work
The background introduces knowledge transfer, federated learning variants, differential privacy, and prior one-shot methods. It positions FedKT as a one-shot approach that transfers voting information and targets privacy and model-flexibility gaps.
- Knowledge Transfer: Knowledge transfer can compress an ensemble of models into a single model, as illustrated by PATE's teacher-voting and student-training process.PATE applies noise to vote counts to obtain differential privacy and supports any classification model, but it was not designed for federated learning.
- Federated Knowledge Transfer: Prior federated knowledge-transfer methods differ in data requirements, objectives, or setting, and generally operate iteratively.Some require public labeled data, while others address poisoning, edge-device limits, or server-side model improvement.
- Federated Knowledge Transfer: FedKT transfers voting counts rather than prediction vectors and is designed to provide differential privacy with a theoretical privacy-loss bound.The paper contrasts this design with existing iterative studies that transfer logits.
- Differential Privacy: Differential privacy limits how much an output probability can depend on whether an individual record is included in the input.The paper introduces randomized mechanisms, adjacent inputs, privacy loss, and moments accounting as the privacy framework.
- Differential Privacy: Party-level differential privacy protects whether an entire party participated in federated learning, rather than protecting only a single record.This distinction is presented as especially relevant to federated settings.
3 Our Approach
FedKT uses two-tier knowledge transfer over public unlabelled data: parties train student models, and the server trains a final model from their predictions. The framework supports multiple privacy levels and reduces communication relative to multi-round FedAvg.
- Privacy settings: FedKT offers L0, L1, and L2 privacy settings, placing no mechanism, server-side noise, or party-side noise on the relevant model outputs.L1 protects a published final model, while L2 protects models transferred from parties to a curious server.
- Server-side knowledge transfer: The server ensembles student-model predictions on the public dataset and uses the resulting labels to train the final model.This repeats knowledge transfer at the server side.
- Consistent voting: Consistent voting counts class predictions only when the student models of a party agree, ignoring predictions from parties that are not confident.The final vote count combines party-level counts across parties.
- Party-side knowledge transfer: FedKT partitions each party’s local data and trains student models through knowledge transfer on public data labeled by local predictions.The student models are sent to the server for aggregation.
- Communication overhead: FedKT’s communication size is nsM, compared with 2nMr for FedAvg over r rounds, so FedKT uses less communication when r > s/2.M is the size of each model, and s is the number of partitions per party.
4 Data-Dependent Privacy Analysis of FedKT
The privacy analysis tracks FedKT’s data-dependent privacy loss under party-level and partition-level differential privacy. It derives guarantees for server- and party-noise settings, while noting that the reported privacy budget can itself create disclosure concerns.
- Privacy accounting: For L1, the analysis targets party-level differential privacy, protecting whether a party participated in federated learning rather than an individual record.The moments accountant method is used to track privacy loss.
- FedKT-L1: 2sγ, 0 is the party-level differential privacy guarantee for the server-side mechanism in FedKT-L1.The guarantee depends on the number of partitions s and privacy parameter γ.
- Composition: If each party’s student models satisfy (εi, δ)-differential privacy, the final model satisfies (maxi εi, δ)-differential privacy.The result follows from parallel composition across parties.
- Limitation: FedKT’s privacy analysis is data-dependent, so publishing the resulting privacy budget may create potential privacy breaches.The paper reports data-dependent budgets experimentally and identifies tighter accounting as future work.
5 Experiments
FedKT is evaluated across diverse datasets, model architectures, baselines, privacy settings, public-dataset sizes, and communication-round regimes. The experiments report strong single-round accuracy, comparable private performance, robustness to reduced public data, and improved communication efficiency when combined with FedProx.
- Experimental setup: Experiments cover random forests, GBDTs, MLPs, and CNNs across Adult, cod-rna, MNIST, and extended SVHN.The default setup uses 50 parties for Adult and cod-rna and 10 parties for MNIST and SVHN, with heterogeneous partitions simulated using a Dirichlet distribution.
- Baselines: FedKT is compared with eight baselines, including iterative federated methods, one-shot PNFM, and centralized PATE and XGBoost upper bounds.FedDF and PATE also use an unlabeled public dataset, while PATE and XGBoost are centralized reference methods.
- Single-round effectiveness: FedKT learns all studied models and achieves about 6.5% higher accuracy than FedDF in a single communication round.FedKT also outperforms PNFM by about 25% accuracy on MNIST, while its accuracy is close to PATE and XGBoost.
- Privacy: With a privacy budget less than 10, differentially private FedKT retains accuracy comparable to the non-private version without requiring model-architecture knowledge.The privacy experiments vary γ and the number of queries for FedKT-L1 and FedKT-L2.
- Public-dataset size: Using 20% of the public dataset decreases accuracy by no more than 1% on cod-rna and 2% on MNIST, while Adult accuracy is almost unchanged.The 20% subsets contain 1807 cod-rna examples and 1000 MNIST examples.
- Multiple rounds: FedKT-Prox reaches 87% accuracy in about 11 rounds, whereas the other approaches require at least 32 rounds.FedKT can initialize a global model before iterative federated learning continues.
6 Conclusions
The paper presents FedKT as a practical one-shot solution for cross-silo federated learning, combining broad model support with privacy guarantees. Experiments report higher accuracy than other state-of-the-art algorithms with one communication round, while private FedKT remains comparable to its non-private version.
- FedKT is proposed as a practical one-shot federated learning solution for the cross-silo setting.
- FedKT can learn different models with much better accuracy than other state-of-the-art algorithms using a single communication round.
- Differentially private FedKT achieves accuracy comparable to the non-differentially private version with a modest privacy budget.
- The appendix covers privacy analysis, experimental settings, additional results, hyper-parameter effects, public-dataset size, and a multi-round extension.
A Privacy Analysis of FedKT
This section analyzes FedKT's privacy loss using the moments accountant method and supporting lemmas. The analysis derives party-level privacy from the sensitivity of party-side noise injection and establishes bounds for the relevant mechanisms.
- FedKT-L1 and FedKT-L2 privacy loss are analyzed using the moments accountant method.
- The analysis uses prior theorems and lemmas for differentially private mechanisms, outcome probabilities, and label-score vectors.
- Theorem 1 implies that the mechanism satisfies (2sγ, 0) party-level differential privacy.
- Party-side noise injection has sensitivity 2, and combining the supporting lemmas yields the privacy theorem.
A.3 Example-Level Differential Privacy Analysis of FedKT-L1
The example-level privacy analysis of FedKT-L1 examines how changing one original record affects teacher models, vote counts, and the student model. Its privacy behavior depends on the number of affected partitions and whether consistent voting is used.
- Changing one original example influences only one party and one teacher model in each party partition.
- The student model remains unchanged when the teachers' top-2 vote counts differ by at least 2, absent consistent voting.
- If z partitions can contain queries whose top-2 vote counts differ by at most 1, changing one record can alter at most z student models.
- The example-level privacy of FedKT-L1 is the same as party-level privacy when z = s and is usually the same with consistent voting.
A.4 Party-Level Differential Privacy Analysis of FedKT-L2
The party-level privacy analysis of FedKT-L2 relates privacy to the number of subsets in each party partition. Although the mechanism satisfies a formal bound, the resulting privacy loss can become large as that number increases.
- For t subsets per party partition, the party-side mechanism satisfies (2tγ, 0) party-level differential privacy.
- Changing an entire local dataset affects t teachers and changes vote counts by at most 2t, giving sensitivity 2t.
- The moments bound for party-level privacy depends on t, γ, q, and l under the stated condition on q.
- Party-level privacy loss for FedKT-L2 can be quite large, and increasing t to tighten one error bound also enlarges the privacy bound.
B.1 Additional Details of Experimental Settings
Experiments evaluate FedKT across datasets, baselines, partition settings, and iterative-training initialization. FedKT-based training improves accuracy and communication efficiency, while larger partition counts add overhead with little further accuracy gain.
- Experimental setup: Experiments use Adult, cod-rna, MNIST, and SVHN, comparing FedKT with eight baselines under tuned training settings.FedKT, SOLO, and PATE use one local epoch; other federated-learning baselines are tuned over learning rates and local epochs.
- Number of partitions: For FedKT-L0, the authors suggest setting s to 2 when users do not want to tune the parameter.For private variants, smaller s values are recommended because privacy loss increases as s increases.
- Number of teachers: FedKT achieves its best performance with five teachers per partition, while larger teacher counts reduce each subset's size and may weaken predictions.The authors suggest setting t to 5 when parameter tuning is not desired.
- Iterative training: FedKT-Prox achieves much higher accuracy than FedAvg and FedProx and is more communication-efficient overall.FedKT is used as an initialization step before iterative training, with results evaluated on SVHN as communication rounds or communication size increase.
B.4 Differential Privacy
The privacy experiments compare private FedKT variants with non-private performance and examine privacy accounting across datasets and partition settings. Results indicate that modest privacy budgets preserve comparable accuracy, while moments accounting can substantially tighten the reported privacy loss.
- Private variants: FedKT-L1 and FedKT-L2 achieve accuracy comparable to the non-private version with a modest privacy budget.The comparison covers heterogeneous and homogeneous data partitions on Adult and cod-rna.
- Experimental settings: The experiments use Adult and cod-rna for privacy evaluation, with privacy-specific partition and subset counts reported for each variant.FedKT-L1 uses one partition and five subsets; FedKT-L2 uses one partition and 25 subsets, with 20 parties for FedKT-L2.
- Privacy accounting: Moments accounting generally yields tighter privacy loss than advanced composition.For cod-rna with homogeneous partitioning, γ = 0.1, and a 1% query fraction, the reported ε is approximately 11.2 versus 20.2 under advanced composition.