Federated Model Distillation with Noise-Free Differential Privacy

Motivation

• Prior centralized differential privacy requires a trusted aggregator and can struggle to converge with few parties, while local approaches often support only shallow models or incur high communication and privacy costs [Geyer et al., 2017; McMahan et al., 2018; Wang et al., 2019b; Zhao et al., 2020; Bhowmick et al., 2018; Sun et al., 2020a].The cited large-scale local approach reports MNIST (ϵ = 500) and CIFAR-10 (ϵ = 5000); another requires sending model parameters individually to the cloud.Source: §1 · p. 1
• Conventional federated learning requires shared model weights, limiting collaboration to homogeneous architectures and exposing models to white-box inference attacks while increasing communication overhead and privacy cost.Model weights are usually higher-dimensional than predictions.Source: §1 · p. 1
• FedMD shares predictions on an unlabeled public set [Li and Wang, 2019], but predictions may leak local data [Papernot et al., 2017], and adding noise trades privacy budget against model performance [Bhowmick et al., 2018].The introduction identifies no reasonable privacy guarantee for sharing model predictions in federated learning.Source: §1 · pp. 1–2

Contributions

• FEDMD-NFDP combines federated model distillation with a noise-free differential privacy mechanism that protects each party without explicitly adding noise.The framework shares predictions rather than model weights.Source: §1 · pp. 1–2
• NFDP guarantees (ϵ, δ)-differential privacy under replacement and without-replacement sampling, avoiding explicit noise addition and privacy-cost explosion; this differs from sampling-based privacy amplification [Balle et al., 2018].The paper formally proves the guarantee for both sampling strategies.Source: §1 · p. 2
• FEDMD-NFDP achieves comparable utility on benchmark datasets across IID and non-IID distributions and heterogeneous architectures using only a few randomly sampled private samples.The experiments validate the framework across these settings.Source: §1 · p. 2

Differential Privacy

• In federated learning with an untrusted server, parties apply local differential privacy before publishing data, providing stronger protection than central differential privacy [Abadi et al., 2016].The local-versus-global distinction depends on whether the server is trusted.Source: §2 · p. 2
• A randomized mechanism satisfies (ε, δ)-differential privacy when its output likelihoods for neighboring inputs meet the stated bound.The passage introduces this condition for any measurable subset of outputs.Source: §2 · p. 2
• Record-level DP bounds how much a record’s presence or absence can affect output likelihood, with δ allowing unlikely responses outside the pure ε-DP criterion.Each party can apply the randomized mechanism individually to ensure record-level LDP.Source: §2 · p. 2

• FEDMD-NFDP supports heterogeneous local architectures by sharing model predictions on unlabeled public data, with random data sampling providing differential privacy for released knowledge.Source: §3 · p. 2
• Unlike FEDMD-NFDP, FEDMD-LDP explicitly adds noise before release, and its privacy cost accumulates with shared-knowledge dimension and communication rounds.Source: §3 · p. 2
• Training starts with each party updating its model on a randomly sampled local subset without collaboration, then proceeds through repeated collaboration rounds.Source: §3 · p. 3
• In each collaboration round, parties share public-data predictions, the server averages them into soft labels, and parties train first on public data (Digest) and then their local subset (Revisit).Source: §3 · p. 3
• The algorithm can also implement FEDMD-LDP by perturbing each released prediction with differentially private Laplace or Gaussian noise.Source: §3 · p. 3

4 Theoretical Analysis

• Theorem 1 establishes an (ϵ, δ)-differential privacy guarantee for sampling without replacement from n records, with subsample size k.The theorem states a parameterized guarantee, but its expression is split across the supplied passages.Source: §4 · p. 3
• Theorem 2 establishes an (ϵ, δ)-differential privacy guarantee for sampling with replacement from n records, with subsample size k.The theorem states a parameterized guarantee, but its expression is split across the supplied passages.Source: §4 · p. 3
• For n > 0 and 0 < k ≤ n, sampling with replacement is consistently more private than sampling without replacement.The comparison applies to Algorithm 1.Source: §4 · p. 3
• Sampling once and applying differential-privacy post-processing removes privacy dependence on the number of public-dataset queries in FEDMD.The passage attributes this property to NFDP’s random sampling once and the post-processing property of differential privacy.Source: §4 · p. 3

Experimental Evaluation

• On MNIST/FEMNIST, parties classify letters from one writer in non-IID settings, versus randomly drawn private data in IID settings.MNIST is public data, while private data is a FEMNIST subset [Caldas et al., 2018].Source: §5 · p. 3
• The evaluation plots party-level ϵ and δ against party count and accuracy against party count, with k = 3 and 100,000 queries over 20 rounds.Each round uses 5,000 public-dataset queries; private-data size n is 28,800 for FEMNIST and 3,000 for CIFAR-10.Source: Figure 2 · p. 4
• On CIFAR-10/CIFAR-100, IID parties classify subclasses, while non-IID parties hold one subclass per superclass and classify generic test images into superclasses.CIFAR-10 is public data; private data is a CIFAR-100 subset, whose subclasses belong to superclasses [Li and Wang, 2019].Source: §5 · pp. 3–4
• Parties use two- or three-layer neural networks, initialized from the same in-domain pretrained model; FEMNIST and CIFAR-10 experiments finish within an hour at 10 parties.The experiments use PyTorch and one NVIDIA Tesla V100 GPU; the selected settings are R = 20, T1 = 20, T2 = 2, and T3 = 1.Source: §5 · p. 4
• Experiments report that FEDMD-NFDP largely reduces communication cost without degrading utility.Each of 20 communication rounds uses a randomly selected public-data subset of size 5,000.Source: §5 · p. 4

5.1 Baselines

• The comparison omits FedAvg because it delivers utility similar to the Centralized framework.Source: §5.1 · p. 4
• FedMD-NP follows FedMD’s local training and public-data distillation process but provides no privacy guarantee.Parties train on their private data, collaborate on public-data distillation, and use aggregation feedback to update local models.Source: §5.1 · p. 4
• The Centralized framework pools all parties’ private data on a server to train a global model, serving as a utility upper bound.Source: §5.1 · p. 4
• FEDMD-LDP requires each party to add Gaussian noise locally before releasing model knowledge to ensure (ϵ, δ)-DP.Source: §5.1 · p. 4

Evaluation on privacy budget ϵ.

• With k = 3, increasing the number of parties raises ϵ and δ, but at N = 10 their log10 scales remain near -2 for CIFAR-10 and -3 for FEMNIST.The analysis assumes a fixed total private-data size divided among parties; it notes that independently collected data would not shrink each party’s local dataset as participation grows.Source: Evaluation on privacy budget ϵ · pp. 4–5, Evaluation on δ · p. 5

Evaluation on model convergence.

• All parties converge to decent performance within 20 communication rounds, reducing communication cost; FEMNIST performs slightly better than CIFAR-10.Figure 3 shows the parties’ accuracy trajectories.Source: Figure 3 · p. 5, Evaluation on model convergence · p. 5

Evaluation on number of parities.

• With a higher privacy budget, FEDMD-NFDP can outperform FEDMD-NP and be comparable to the centralized approach; at (ϵ,δ) with k = 16, it remains competitive on CIFAR-10.The authors report that prior differential-privacy federated-learning methods could not achieve comparable CIFAR-10 performance at such a small ϵ; they also report improved utility with more parties and good CIFAR-10 performance with 10 parties.Source: Table 2 · p. 5, Evaluation on number of parities · p. 6

Evaluation on σ in FEDMD-LDP.

• At the same ϵ and δ as FEDMD-NFDP, FEDMD-LDP requires σ with a log10 scale of 4–7, making predictions close to random guess; preserving utility requires a meaningless privacy guarantee.The paper attributes the large noise requirement to the many queries and communication rounds under sequential composition.Source: Evaluation on σ in FEDMD-LDP · p. 5

Evaluation on distillation approaches.

• Across distillation approaches, results differ little overall, but argmax is recommended for its slightly better performance and lower communication cost than softmax or logits.Argmax sends an integer per query, while softmax and logits send float vectors.Source: Evaluation on distillation approaches · p. 5

Evaluation on IID and Non-IID distributions.

• FEDMD-NFDP performs well on non-IID data, with FEMNIST doing better than CIFAR-10 because CIFAR-10 training uses only one subclass and poorly trains local models for other classes.The authors describe non-IID tasks as more challenging than IID tasks because each party has incomplete class information.Source: Table 4 · p. 6, Evaluation on IID and Non-IID… · p. 6

Evaluation on number of parities.

• With replacement sampling uses less privacy budget than without replacement sampling for the same subset size, with little performance difference, so the authors recommend sampling with replacement.Increasing the number of sampled examples raises the required privacy budget when the number of parties is fixed.Source: Table 2 · p. 5, Figure 4 · p. 6, Evaluation on number of parities · p. 6

Diversity of public dataset.

• Public-data utility needs evaluation as its distribution diverges from private data, including no-overlap domains; GAN-generated data may ease real-data acquisition and storage limits.The passage identifies public-data distributions that match or differ from private training data and calls for examining how the gap affects distillation.Source: Diversity of public dataset · p. 6

Diversity of local models.

• FEDMD-NFDP accommodates local models with different structures, sizes, and numerical precision, but equal-weight prediction averaging may hurt utility under extreme model or data heterogeneity.This flexibility can benefit IoT edge devices with diverse hardware and computing resources; weighted averaging may better quantify local-model contributions and boost utility.Source: Diversity of local models · pp. 6–7, Weighted aggregation · p. 7

Limitations.

• NFDP needs adequately sized local datasets—a one-label dataset offers no privacy—and suits large-data MLaaS, low-diversity large datasets, and tasks using only a small data fraction.FedMD [Vinyals et al., 2016; Snell et al., 2017], traditional one-shot learning, and few-shot learning are examples of the last case.Source: Limitations · p. 7

7.1 Differential Privacy

• Prior FL privacy work uses trusted-party centralized DP [Geyer et al., 2017; McMahan et al., 2018], local gradient randomization [Sun et al., 2020a], or DDP with cryptography [Lyu, 2020].Source: §7.1 · p. 7

7.2 Knowledge Distillation

• Knowledge distillation [Hinton et al., 2015] transfers class probabilities from large or ensemble DNNs to smaller models with marginal utility loss and enables prediction-based knowledge sharing.Source: §7.2 · p. 7
• Ensemble-teacher distillation appears in prior work [Hamm et al., 2016; Papernot et al., 2017; Wang et al., 2019a; Sun et al., 2020b]; PATE [Papernot et al., 2017] uses trusted aggregation and DP labeling of public data.PATE then trains a student semi-supervisedly; this paper's focus differs because PATE requires a trusted aggregator to aggregate teacher predictions and conduct DP mechanisms.Source: §7.2 · p. 7

7.3 Federated Learning

• FL enables parties to train jointly without exposing private data; FedSGD sends every SGD update, whereas FedAVG batches local iterations before sending updates for greater communication efficiency [McMahan et al., 2017].Source: §7.3 · p. 7
• FedMD [Li and Wang, 2019] aligns models on public data before private training, while Cronus [Chang et al., 2019] jointly trains with public soft labels and private data; neither provides theoretical privacy guarantees.Both average logits on unlabeled public data, reducing communication costs without significantly sacrificing utility.Source: §7.3 · p. 7

8 Conclusion

• NFDP with both replacement and without-replacement sampling formally ensures (ϵ, δ)-differential privacy without explicit noise addition.The framework is presented as avoiding privacy-cost explosion issues in previous works.Source: §8 · p. 7
• Empirical results across datasets, settings, and heterogeneous architectures show comparable utility using only a few randomly sampled private samples per party.The authors describe these results as confirming the framework’s effectiveness and superiority.Source: §8 · p. 7
• Future directions include applying NFDP to semi-supervised, pre-training, meta-learning, and few-shot learning, improving sampling through advanced data analysis, and supporting NLP, graph analysis, and medical diagnosis.These are proposed directions rather than evaluated results.Source: §8 · p. 7

NFDP privacy guarantees

• The appendix states differential-privacy guarantees for both sampling without replacement and sampling with replacement.The theorem statements give parameterized guarantees, but the displayed formulas are incomplete in the supplied passages.Source: Appendix · p. 9

Proof approach

• The proofs analyze neighboring datasets in both directions and partition sampled subsets into shared and differing portions.The without-replacement proof explicitly considers both cases in which one neighboring dataset contains an additional sample.Source: Appendix · p. 9

Comparison of sampling methods

• For any n > 0 and 0 < k ≤ n, sampling with replacement is consistently more private than sampling without replacement.The appendix says this comparison follows by induction over the inequalities for the two guarantees.Source: Appendix · p. 10