Federated Learning with Bayesian Differential Privacy

Aleksei Triastcyn; Boi Faltings

Source-linked AI summary

Federated Learning with Bayesian Differential Privacy

Aleksei Triastcyn, Boi Faltings

arXiv:1911.10071v1Nov 22, 2019cs.LGcs.CRcs.DCstat.ML

TL;DR

Federated learning lacks formal privacy guarantees, while standard differential privacy can impose a poor privacy-utility trade-off in deep learning. The paper adapts Bayesian differential privacy and privacy accounting to federated learning, including joint client- and instance-level accounting. Experiments report tighter budgets, with ε below 1 for client privacy and ε = 0.1 for instance privacy while maintaining reasonably high accuracy.

Problem
Federated learning lacks theoretical privacy guarantees, and combining it with differential privacy can produce a poor privacy-utility trade-off for complex deep learning models.
Method
The paper adapts Bayesian differential privacy accounting to federated learning and introduces variants for instance privacy and joint client- and instance-level accounting.
Results
ε can be kept below 1 for client privacy and close to ε = 0.1 for instance privacy while maintaining reasonably high accuracy.
Takeaways & Limitations
Bayesian differential privacy requires less noise for the same privacy guarantees, enabling fewer communication rounds and tighter privacy budgets in federated learning.
Takeaways & Limitations
The approach assumes users come from a population with similarly distributed data, such as medical images, emails, or motion sensor data.

Abstract

from arXiv · show

We consider the problem of reinforcing federated learning with formal privacy guarantees. We propose to employ Bayesian differential privacy, a relaxation of differential privacy for similarly distributed data, to provide sharper privacy loss bounds. We adapt the Bayesian privacy accounting method to the federated setting and suggest multiple improvements for more efficient privacy budgeting at different levels. Our experiments show significant advantage over the state-of-the-art differential privacy bounds for federated learning on image classification tasks, including a medical application, bringing the privacy budget below 1 at the client level, and below 0.1 at the instance level. Lower amounts of noise also benefit the model accuracy and reduce the number of communication rounds.

I. INTRODUCTION

Federated learning keeps user data on devices but lacks differential privacy’s formal guarantees, while conventional differentially private federated learning can yield weak privacy-utility trade-offs. The paper proposes Bayesian differential privacy and federated privacy accounting to obtain tighter guarantees under distributional assumptions.

Motivation: Federated learning trains models without copying user data, but its local-update and server-aggregation design does not itself provide theoretical privacy guarantees.Clients update models locally, and the server aggregates those updates.
Motivation: Differentially private federated learning can provide only weak guarantees unless the user population is exceedingly large, with ε close to 10 potentially permitting 99.99% attack accuracy.The passage contrasts this with the stronger guarantees sought by the paper.
Proposed approach: Bayesian differential privacy calibrates noise to the data distribution, yielding tighter guarantees than differential privacy for datasets drawn from the same distribution.Full distributional knowledge is unnecessary because required statistics can be estimated from data.
Contributions: The paper adapts Bayesian differential privacy to federated learning, including practical non-i.i.d. settings, and develops joint accounting for client-level and instance-level privacy.The joint method estimates both privacy levels simultaneously and securely.
Related work: The work addresses a privacy-utility trade-off in deep learning by combining federated learning with a relaxation that retains meaningful privacy accounting and probabilistic interpretation.Bayesian differential privacy is presented as a more permitting assumption than several alternative relaxations.

III. PRELIMINARIES

The paper establishes notation for adjacent datasets, examples, gradients, outcomes, and federated updates, then defines differential privacy, privacy loss, Gaussian mechanisms, and Rényi divergence.

Notation: Adjacent datasets D and D′ differ in one example, with individual examples denoted x or x_i and the differing example denoted x′.Whenever possible, the paper writes D′ = D ∪{x′}.
Notation: The paper distinguishes non-private gradients g and g′ from private learning outcomes w, and uses u_i and U for individual and collective federated updates.Expectations over data and learning outcomes are denoted E_x and E_w, respectively.
Privacy definitions: Differential privacy constrains outcome probabilities of a randomized algorithm on any pair of adjacent datasets.The formal definition applies to every outcome set S.
Privacy definitions: Privacy loss is defined for an algorithm outcome and two datasets, with the privacy-loss random variable distributed by sampling the outcome under one dataset.This links privacy loss to divergence-based analysis.
Mechanisms: The Gaussian mechanism provides approximate differential privacy using noise calibrated to the L2-sensitivity of the queried function.The sensitivity is the maximum norm difference between outputs on neighboring datasets.

B. Setting

The framework represents iterative learning and federated communication rounds as noisy transformations of non-private outcomes. It accumulates per-iteration privacy costs to derive overall privacy-loss bounds, with sampling providing amplification.

B. Setting: Each learning iteration produces a non-private outcome, while each federated communication round produces one non-private update for every client.The same framework covers general iterative learning and federated settings.
B. Setting: Private outcomes are generated from non-private outcomes through an additive noise mechanism and then used to initialize the next iteration or communication round.The formulation allows conditional outcome distributions based on datasets or client updates.
B. Setting: Subsampling data or participating clients amplifies privacy because outcomes depend only on the sampled batch or participating-user updates.The paper denotes these sampled sets B(t) and U(t).
B. Setting: A per-iteration privacy cost c_t accumulates over training and is converted into ε, δ bounds using concentration inequalities.The accounting workflow generalizes the moments accountant and is not tied to one learning algorithm.

C. Motivation

The paper motivates alternative privacy definitions by showing that familiar differential privacy budgets can still permit highly accurate attacks. It frames posterior change and attack accuracy as more interpretable ways to assess guarantees.

C. Motivation: Differential privacy’s complexity can lead to misunderstanding or overestimation of its guarantees, motivating a more interpretable evaluation example.The paper focuses on changes in an attacker’s posterior after observing the private model.
C. Motivation: Models approaching non-private accuracy often use ε > 2 and sometimes ε > 5, while real-world applications can exceed ε > 10.These budgets may appear numerically small despite their implications for attack accuracy.
C. Motivation: In the income example, an attacker compares likelihoods and posteriors under datasets with and without an unusually high-income individual.The attacker uses a flat prior and observes a privatized mean income.
C. Motivation: The privacy bound limits the likelihood ratio to e^ε0, but a sufficiently small re^-ε0 term can still yield a high probability of a correct attacker guess.This connects the formal bound to posterior attack accuracy.
C. Motivation: For ε = 5, the attack’s upper-bound accuracy reaches 99.33%, and more informed priors could make these figures even larger.The paper presents this as evidence that such guarantees may not be reassuring.

IV. BAYESIAN DIFFERENTIAL PRIVACY

Bayesian differential privacy adapts privacy accounting to data distributions and iterative learning, using tighter bounds under exchangeability and related assumptions. The method estimates per-iteration privacy costs and composes them into overall ε, δ guarantees.

Definition and assumptions: BDP incorporates randomness from an additional example drawn from the data distribution when bounding privacy loss.Its definition is close to traditional DP but accounts for the randomness of x′.
Definition and assumptions: BDP may underestimate actual privacy loss when the data distribution is non-stationary.The guarantee does not cover every scenario supported by traditional DP.
Privacy accounting: The accounting framework assigns each iteration a privacy cost c_t(λ), then sums these costs across the learning process.The accumulated costs are converted into ε, δ bounds using concentration inequalities.
Privacy accounting: Theorem 2 provides an efficient accounting algorithm based on Rényi divergence and expectation over data rather than the privacy-loss random variable.The procedure generalizes the moments accountant and is not tied to a specific learning algorithm when it can be mapped to the framework.
Estimation: Because the true data prior is usually unavailable, the method estimates privacy costs from samples and uses an upper confidence bound to account for estimator error.The resulting estimator overestimates the true privacy cost with probability 1−δ′ under the stated continuity condition.
Gaussian mechanism: For the subsampled Gaussian mechanism, privacy cost depends on the noise parameter σ and sampling probability q.The outcome distribution is modeled as a mixture of two Gaussians, and the resulting cost is expressed using the binomial distribution.

V. FEDERATED LEARNING WITH BAYESIAN DIFFERENTIAL PRIVACY

The paper extends Bayesian differential privacy and its accounting method to federated learning. It targets client-level privacy, examines instance-level privacy, and combines both for FedSGD.

V. Federated Learning with Bayesian Differential Privacy: The framework adapts Bayesian differential privacy to client-level privacy in federated learning.Client-level privacy is the level most frequently addressed in the literature.
V. Federated Learning with Bayesian Differential Privacy: The paper also explores instance-level privacy and proposes joint accounting of instance-level and client-level privacy for FedSGD.The stated goal is to provide a trade-off between utility and privacy guarantees.

A. Client privacy

For client privacy, the paper retains the Gaussian mechanism used in classic DP but replaces moments-accountant composition with Bayesian accounting. Individual client update distributions support non-i.i.d. federated settings.

Client privacy: Client-level privacy bounds the influence of any single user on the model-parameter distribution.The objective is to hide whether a single user participated in training.
Client privacy: The method uses clipped client updates and Gaussian noise, as in classic DP, while applying Bayesian accounting instead of the moments accountant.The classic mechanism clips updates to an L2 threshold C and adds noise with variance C^2σ^2.
Client privacy: The Bayesian method is expected to gain an advantage when data are similarly distributed across users and their updates strongly agree.The paper motivates this advantage through the relationship between similar data distributions and update agreement.
Client privacy: Individual client update distributions make the approach applicable to non-i.i.d. federated settings.Client i sends an update sampled from its own distribution p_i(u), and users participate with probabilities α_i.
Client privacy: The accounting procedure combines the Gaussian mechanism, user sampling, and an estimator to upper-bound each round's privacy cost with high probability.The accumulated costs are then converted into ε, δ using the advanced-composition theorems.

B. Instance privacy

The paper distinguishes instance privacy from client privacy and develops tighter instance-level accounting using subsampled Gaussian noise. This distinction matters especially when individual records require stronger protection than participation itself.

Instance privacy: Client-level DP bounds the influence of a user, but the same guarantee is not automatically meaningful for individual instances under BDP.The paper notes that client-level and instance-level privacy can differ substantially in federated settings.
Instance privacy: Instance privacy is especially important for medical federated learning, where hospital participation may be public but patient data must remain protected.It also provides an additional protection layer when the curator is untrusted.
Instance privacy: The method applies a subsampled Gaussian noise mechanism to gradient computation on user devices for tighter instance-level guarantees.The noise parameter σ and sampling probability q may vary across the considered settings.
Instance privacy: The paper presents two possible accounting schemes for instance privacy.The supplied passage identifies the alternatives but does not describe their procedures.

1) Sequential accounting:

The paper develops sequential, parallel, and joint privacy accounting for federated learning, balancing tighter guarantees against communication, disclosure, and convergence costs.

1) Sequential accounting:: Sequential accounting combines privacy costs across clients and rounds, with part of the accounting performed locally and part on the server.The overall cost matches centralized training using sampling probability q = B_i/N, where N is the total number of data points and B_i is the local batch size.
1) Sequential accounting:: Sequential accounting can reveal the total dataset size to participating clients and bounds the shared model rather than each individual update.Individual updates therefore require a separate local privacy bound.
2) Parallel accounting:: Parallel accounting uses local sampling probability q = B_i/N_i and parallel composition across independent client updates within each round.The server aggregates the maximum privacy cost over users, then sums maximum costs across rounds because updates depend on previous rounds.
2) Parallel accounting:: Parallel accounting may require more communication rounds to reach the same quality solution under the same privacy guarantee, especially with non-identically distributed data.Its per-round maximum replaces a sum over clients, but local dataset sizes can produce higher individual privacy costs.
C. Joint privacy: Joint accounting re-counts client-side noise toward the client-level guarantee instead of adding further server noise.This preserves tight instance- and client-level guarantees while keeping convergence nearly at the client-privacy-only level; the method is currently usable with FedSGD rather than FedAvg.

VI. EVALUATION

The evaluation tests Bayesian and differential privacy accounting on federated image classification using MNIST and a clinically labeled retinal-disease dataset. It compares privacy guarantees and accuracy across client distributions, with accounting bounds estimated from sampled updates or gradients.

A. Experimental setup: The experiments use MNIST and APTOS 2019, a retinal-image dataset with clinician labels for diabetic retinopathy severity.MNIST contains 60,000 training and 10,000 test examples; APTOS contains 3,662 fundus photographs and has more serious privacy-leak implications than common benchmark datasets.
A. Experimental setup: The federated setup includes 100, 1,000, or 10,000 clients coordinated by a server, with each client holding a subset of data.MNIST clients hold 600 examples, while APTOS uses data augmentation; testing uses the official MNIST split and the first 500 APTOS samples.
A. Experimental setup: MNIST uses a four-layer CNN, whereas APTOS uses ImageNet-pretrained ResNet-50 with only its final fully connected layer retrained.The study limits communication rounds to 300 or fewer and uses FedSGD because joint accounting is not compatible with FedAvg.
A. Experimental setup: The MNIST experiments compare i.i.d. and non-i.i.d. client data distributions, including a shard-based non-i.i.d. split.With only 100 clients, some clients may be distinguishable by their digit distributions, potentially making reasonable privacy bounds costly in accuracy.
A. Experimental setup: Privacy accounting compares moments-accountant bounds for DP with Bayesian bounds estimated from sampled updates or gradients and Chernoff's inequality.The evaluation reports accuracy together with privacy pairs (ε, δ) on both non-i.i.d. and i.i.d. MNIST settings.

B. Client privacy

The paper evaluates Bayesian accounting for client privacy and compares sequential with parallel accounting for instance privacy. Bayesian accounting improves privacy guarantees across datasets and settings, while sequential accounting is more efficient when users can know the total dataset size.

Client-level accounting: Bayesian accounting provides substantial privacy gains across all evaluated datasets and settings.The paper reports that exponentiating ε yields an approximately 400-fold stronger guarantee in one comparison.
Client-level accounting: 10% higher accuracy is achieved in non-i.i.d. MNIST with 100 clients using the Bayesian method.The gain is attributed in the paper to the greater difficulty of the non-i.i.d. learning setting.
Client-level accounting: Privacy accounting runs on the server without computation or communication overhead from users in these experiments.The accounting code is executed on the server.
Instance-level accounting: Parallel accounting produces faster ε growth than sequential accounting across the evaluated client settings.The comparison covers 100 and 1000 clients and uses the change in ε relative to its initial value.
Instance-level accounting: Sequential accounting is preferred when communicating total dataset size to users is acceptable.Parallel accounting remains applicable for personal privacy with an untrusted curator, but requires more noise for reasonable guarantees.

D. Joint privacy

The paper studies joint accounting for instance-level and client-level privacy in federated learning. Joint accounting preserves accuracy much better than separate accounting and supports strong privacy budgets, although tighter guarantees rely on assumptions about data distributions.

D. Joint privacy: Joint accounting performs nearly as well as non-private or client-only training in the evaluated i.i.d. and non-i.i.d. settings.Separate accounting completely fails in the i.i.d. experiment because of excessive noise at both levels, while joint accounting reaches similar performance after slightly more rounds in the non-i.i.d. setting.
D. Joint privacy: The approach adds computation overhead on user devices because instance and joint accounting require multiple gradient calculations.Communication overhead is described as negligible because only one floating-point number is added to user messages.
D. Joint privacy: Joint accounting introduces instance and client privacy using only instance-level noise.The technique is proposed to obtain guarantees at both privacy levels jointly.
D. Joint privacy: ε = 0.1 instance privacy and ε = 1 client privacy are maintained with reasonably high accuracy in the paper’s strong-privacy evaluation.On APTOS, accuracy is 6% below the non-private baseline; on MNIST, accuracy exceeds 70% i.i.d. and 60% non-i.i.d.
D. Joint privacy: Tighter privacy guarantees may be inapplicable for non-stationary data distributions or datasets with non-exchangeable samples.The paper identifies automatic detection and mitigation of these scenarios as future work.