PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Differentially Private Recurrent Language Models

H. Brendan McMahan, Daniel Ramage, Kunal Talwar, Li Zhang

arXiv:1710.06963v3cs.LG

TL;DR

The paper asks whether large recurrent language models can protect whole users' language histories without sacrificing predictive quality. It applies user-level differential privacy through a noised federated averaging algorithm and reports comparable private and non-private LSTM performance, with additional computation as the main cost.

  • Problem

    Language modeling needs domain-matched data that can contain highly sensitive information, while protecting individual examples is insufficient when users contribute many thousands of examples.

    Source: §1 · p. 1

  • Method

    The paper adds user-level differential privacy to federated averaging, using grouped user updates and the moments accountant to support large-step model updates.

    Source: §1 · p. 2, §2 · p. 5

  • Results

    17.5% accuracy is maintained at (4.6, 10^-9)-differential privacy, with roughly 60× additional computation compared with non-private training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Large recurrent language models can achieve strong user-level privacy with negligible utility loss when the dataset is sufficiently large, shifting the main cost to computation.

    Source: §1 · p. 2, §4 · p. 9

    The approach begins from random initialization, although initializing from a public dataset could mitigate its additional computational cost.

    Source: §1 · p. 2

Abstract

from arXiv · show

We demonstrate that it is possible to train large recurrent language models with user-level differential privacy guarantees with only a negligible cost in predictive accuracy. Our work builds on recent advances in the training of deep networks on user-partitioned data and privacy accounting for stochastic gradient descent. In particular, we add user-level privacy protection to the federated averaging algorithm, which makes "large step" updates from user-level data. Our work demonstrates that given a dataset with a sufficiently large number of users (a requirement easily met by even small internet-scale datasets), achieving differential privacy comes at the cost of increased computation, rather than in decreased utility as in most prior work. We find that our private LSTM language models are quantitatively and qualitatively similar to un-noised models when trained on a large dataset.

1 INTRODUCTION

The paper addresses the challenge of protecting sensitive language data at the user level while preserving language-model quality. It introduces user-adjacent differential privacy and a noised federated averaging method, demonstrating comparable accuracy with increased computation.

1 INTRODUCTION

  • The work targets language modeling settings where domain-matched data improves utility but may contain passwords, messages, search queries, or identifying phrases.The motivating application is next-word prediction for mobile keyboards, although experiments use a public dataset for reproducibility and comparison.Source: §1 · p. 1
  • The proposed noised federated averaging algorithm groups multiple SGD updates into large-step model updates and uses the moments accountant for privacy accounting.This approach adapts federated averaging to user-level differential privacy while building on analysis developed for differentially private SGD.Source: §1 · p. 2
  • 17.5% accuracy is matched under (4.6, 10^-9)-differential privacy, but training requires roughly 60× more computation than the non-private baseline.The baseline reaches 17.5% in 4120 rounds, while the private model reaches the same accuracy in 4980 rounds while processing about 5000 users per round.Source: §1 · p. 2
  • The paper reports that privacy can be paid for through computation rather than utility loss when the dataset contains sufficiently many users.On a larger dataset with 10^8 users, the reported guarantee would improve to (1.2, 10^-9).Source: §1 · p. 2
  • User-level privacy protects entire user histories because one person may contribute many thousands of training examples, unlike example-level privacy.The paper defines user-adjacent datasets by adding or removing all examples associated with one user.Source: §1 · pp. 1–2
  • The privacy guarantee makes the presence or absence of any individual user's data have an imperceptible effect on model parameters, ruling out memorization in a strong information-theoretic sense.This protects against inference of whether a specific user's data was included, even with arbitrary auxiliary information.Source: §1 · p. 3

2 ALGORITHMS FOR USER-LEVEL DIFFERENTIALLY PRIVATE TRAINING

The paper extends federated training with user-level differential privacy by sampling, clipping, estimating, and noising per-user updates, with privacy tracked by the moments accountant. Its design exploits large user populations and larger per-user updates to make privacy primarily a computational rather than utility cost.

  • FedAvg performs extra local computation to create larger per-user updates, which are more resistant to noise and can reduce the number of privacy-costly rounds.Source: §2 · p. 3
  • DP-FedAvg and DP-FedSGD sample users, clip per-user updates, estimate their average, and add Gaussian noise before updating the shared model.Source: §2 · p. 3
  • Random user sampling is essential for the moments accountant, while bounded-sensitivity estimators address variable sample sizes in weighted averages.Source: §2 · pp. 3–4
  • Clipping bounds update sensitivity, but its parameter trades clipping bias against the noise required for privacy and therefore must be tuned.Source: §2 · pp. 4–5
  • The moments accountant gives tighter composition bounds for sampled Gaussian mechanisms, and large datasets permit many optimization rounds at modest additional privacy cost.Source: §2 · pp. 5–6
  • At 5000 rounds, Table 2 reports privacy bounds and AccuracyTop1 for different noise and clipping settings, while Figure 1 compares noised models with a non-private baseline.Source: Table 2 · p. 6, Figure 1 · p. 6

3 EXPERIMENTAL RESULTS

Experiments show that DP-FedAvg can train LSTM language models with little predictive-accuracy loss when enough users are sampled, while privacy mainly increases computation. Parameter studies examine estimators, clipping, noise, training schedules, and qualitative prediction effects.

Comparing DP and non-DP models

  • FedAvg offers better privacy-utility trade-offs than FedSGD, and conclusions extend to larger dictionaries with little additional parameter tuning.The larger models incur substantially greater size but preserve the reported empirical pattern.Source: §3 · p. 6

Building towards DP: sampling, estimators, clipping, and noise

  • Changing from fixed to variable user sampling and switching estimators does not significantly affect convergence, so the fixed-denominator estimator ˜ff is used thereafter.The comparison uses an expected 100 users per round.Source: §3 · p. 7
  • Clipping choices in S ∈ [10, 20] have at most a small effect on FedAvg convergence, while larger-user-round experiments reduce sensitivity and improve privacy guarantees at greater computational cost.With ˜C = 1250, S = 15, and σ = 0.012, the method achieves (1.97, 10−9)-differential privacy after 3000 rounds; models trained with ˜C = 100 and 1250 have almost identical accuracy curves.Source: §3 · pp. 7–8
  • Noise σ = 0.024 causes almost no early convergence loss, but later noise above σ = 0.012 slightly reduces convergence; the privacy-required σ = 0.2 destroys accuracy at ˜C = 100.At S = 20 and ˜C = 100, the estimator sensitivity is 0.2, implying σ = 0.2 for noise scale z = 1.Source: §3 · p. 7, Figure 3 · p. 7

Comparing DP and non-DP models

  • −0.13% AccuracyTop1: with σ = 0.003, S = 15, and ˜C = 5000, the private model achieves (4.6, 10−9)-differential privacy while nearly matching the baseline.Training on a different corpus was observed to cost −2.50% in AccuracyTop1.Source: §3 · p. 8

Adjusting noise and clipping as training progresses

  • Adjusting clipping and noise after 4885 rounds improves later training: S = 7.9 and σ = 0.0048 outperform the initial S = 10–12.6 and σ = 0.006–0.0076 settings.As training progresses, fixed noise levels eventually become detrimental, motivating smaller clipping and noise at unchanged per-round privacy cost.Source: §3 · p. 8, Figure 4 · p. 8

Comparing DP and non-DP models

  • Changing user count or noise produces little qualitative prediction difference until noise compromises model quality, when the model becomes as head-word-biased as similarly accurate less-noisy models.Table 3 counts how many top-10 predictions fall among the 10, 50, or 100 most frequent corpus words.Source: Table 3 · p. 8, §3 · p. 9

4 CONCLUSIONS

The work shows that large neural networks, including complex LSTM language models, can be trained with user-level differential privacy at negligible utility loss, with added computation as the main cost.

4 CONCLUSIONS

  • User-level private training of large LSTM language models achieves negligible utility loss on a realistic dataset, while requiring additional computation.The approach provides significant privacy guarantees for real-world applications, though reducing computational overhead remains open.Source: §4 · p. 9

A ADDITIONAL PROOFS

The proofs establish privacy by combining bounded-sensitivity estimators with random user sampling, clipped updates, Gaussian noise, and composable moments accounting.

A ADDITIONAL PROOFS

  • The estimator proof bounds sensitivity by limiting numerator changes for the fixed-denominator estimator and the normalized clipped update for the clipped-denominator estimator.For neighboring datasets differing by one user, the update contribution is bounded by S.Source: A ADDITIONAL PROOFS · p. 11
  • Random user sampling and bounded per-user updates allow privacy-loss moments to be bounded by a sampled Gaussian mechanism.The bound assumes each update has L2 norm at most S and uses noise scale σ/S with sampling probability q.Source: A ADDITIONAL PROOFS · p. 11
  • The privacy proof composes per-step moments across optimization rounds because the algorithm uses fixed noise variance and data-independent i.i.d. Gaussian noise.The resulting accounting combines the moments bounds with the sensitivity bounds of the estimators.Source: A ADDITIONAL PROOFS · p. 11

B EXPERIMENT DETAILS

The experiments use a compact word-level LSTM and a large, user-partitioned Reddit dataset, with controlled architectural, initialization, evaluation, and reproducibility choices.

B EXPERIMENT DETAILS

  • The vocabulary is a preselected dictionary of the 10,000 most frequent normalized words, with remaining words mapped to UNK.Private vocabulary discovery is left outside the experimental setup.Source: B EXPERIMENT DETAILS · p. 11
  • The language model maps words to 96-dimensional embeddings and combines them with a 256-dimensional recurrent state before softmax scoring over the vocabulary.Sequences use implicit beginning-of-sequence and end-of-sequence tokens.Source: B EXPERIMENT DETAILS · pp. 11–12
  • The model ties input and output embeddings, fixes embedding norms at 1.0, and contains 1.35M trainable parameters.Weight tying reduces model size by about 40% for a small decrease in model quality, while fixed norms were found to improve convergence time.Source: B EXPERIMENT DETAILS · p. 12
  • Experiments train from random initialization on private data, although public-data pretraining followed by private refinement is identified as a likely practical approach.Training purely on private data is described as the strictly harder setting.Source: B EXPERIMENT DETAILS · p. 12
  • AccuracyTop1 measures whether the highest-probability predicted word is correct, counting out-of-dictionary targets as errors for fair dictionary comparisons.The evaluation also considers Top3, Top5, and perplexity across tasks and corpora.Source: B EXPERIMENT DETAILS · p. 12
  • The Reddit training set contains 763,430 users, each retained with exactly 1,600 tokens after removing users with fewer tokens.The construction limits any one author’s contribution while avoiding excessive per-user overhead from very small users.Source: B EXPERIMENT DETAILS · p. 12
  • The public dataset was selected to resemble private next-word-prediction data while supporting reproducibility, non-private comparisons, and inspection beyond aggregate statistics.The running application is next-word prediction for mobile keyboards.Source: B EXPERIMENT DETAILS · p. 12

C SUPPLEMENTARY PLOTS

The supplementary plots examine estimator choice, user counts, clipping, noise tradeoffs, and FedSGD behavior under specified sampling and clipping settings.

C SUPPLEMENTARY PLOTS

  • Fixed and variable user sampling, together with fixed-denominator and clipped-denominator estimators, are compared for estimating per-round updates.The variable scheme samples users independently with probability q for an expected 100 users, while fixed sampling uses exactly 100 users.Source: Figure 5 · p. 13
  • The effect of C is evaluated for FedAvg with the exact estimator and without noise or clipping.This plot isolates the user-count setting from privacy noise and clipping effects.Source: Figure 6 · p. 13
  • Training with expected 100 versus 1,250 users per round is compared using flat clipping at S = 15.0 and per-coordinate noise with σ = 0.012.The comparison varies the number of users processed per round under fixed clipping and noise settings.Source: Figure 7 · p. 13
  • Noise-versus-clipping tradeoffs are compared while holding z fixed, using S = 20 and σ = 0.012 as a baseline across randomly initialized and resumed training.The resumed-training comparison starts from a model snapshot after 4,885 initial rounds.Source: Figure 8 · p. 13
  • The supplementary plots also examine clipping effects on FedSGD and on models with 20,000- and 30,000-token dictionaries.For FedSGD, the cited setting uses 50 users per round and learning rate η = 6, with a smaller clipping level than FedAvg.Source: Figure 9 · p. 14, C SUPPLEMENTARY PLOTS · p. 14, Figure 11 · p. 14

D ADDITIONAL EXPERIMENTS

Additional experiments compare private training behavior across optimization methods, dictionary sizes, and update regularization. FedAvg remains comparatively robust, while FedSGD requires substantially more iterations and larger dictionaries provide only modest accuracy gains.

Experiments with SGD

  • FedSGD is more sensitive to noise than FedAvg, likely because its updates have smaller magnitudes.Figure 10 evaluates noised FedSGD updates with clipping level S = 20.Source: Figure 10 · p. 14

Models with larger dictionaries

  • Noised-update effects were also examined for models with larger dictionaries clipped at S = 20.The supplied figure passages identify the comparison but do not report a specific outcome.Source: Figure 12 · p. 14

Experiments with SGD

  • FedSGD requires significantly more iterations than FedAvg to reach equivalent accuracy, increasing privacy cost from additional rounds.With 763,430 users, privacy changes from (3.81, 10^-9)-DP after 3,000 rounds to (8.92, 10^-9)-DP after 20,000 rounds.Source: D ADDITIONAL EXPERIMENTS · p. 14

Models with larger dictionaries

  • Larger 20,000- and 30,000-token dictionaries yield only modest accuracy improvements without requiring different clipping or noise parameters.These models use FedAvg with η = 6, 1,600 tokens per user, and 100 expected users per round.Source: D ADDITIONAL EXPERIMENTS · p. 14

Other experiments

  • Adding an explicit L2 penalty to each user's model update produced no positive effect.The penalty was intended to reduce the need for clipping by favoring updates with smaller L2 norms.Source: D ADDITIONAL EXPERIMENTS · p. 14

Related papers

Loading 1710.06963v3…