PaperlayerView exact revision on arXiv

Source-linked AI summary

Recovering Private Text in Federated Learning of Language Models

Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, Danqi Chen

arXiv:2205.08514v2cs.CLcs.CRcs.LG

TL;DR

Prior work showed that eavesdroppers could recover private data from federated-learning gradients, but text recovery remained limited to unrealistically small batches. FILM identifies words from gradients and reconstructs text, recovering significant portions of sentences from batches of up to 128; freezing embeddings can defend against the attack when fine-tuning a public pretrained language model.

  • Problem

    Text recovery from federated-learning gradients remained limited to unrealistically small batch sizes, despite demonstrated recovery of private data in other settings.

    Source: §1 · p. 2

  • Method

    FILM identifies private words from gradients, then reconstructs sentences with beam search and prior-based token reordering.

    Source: §1 · p. 2, §4.3 · p. 5, §4.4 · p. 5

  • Results

    Up to 128 sentences: FILM recovers significant portions of sentences in large batches, with high-fidelity single-sentence and iterative multi-sentence recovery also reported.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Freezing word embeddings can defend against FILM with little utility loss when training starts from a pretrained language model.

    Source: §1 · p. 2, §8 · p. 10

    The attack was not evaluated on other NLP tasks or batches beyond 128, and selecting sentences for multi-sentence recovery remains open.

    Source: §8 · p. 10

Abstract

from arXiv · show

Federated learning allows distributed users to collaboratively train a model while keeping each user's data private. Recently, a growing body of work has demonstrated that an eavesdropping attacker can effectively recover image data from gradients transmitted during federated learning. However, little progress has been made in recovering text data. In this paper, we present a novel attack method FILM for federated learning of language models (LMs). For the first time, we show the feasibility of recovering text from large batch sizes of up to 128 sentences. Unlike image-recovery methods that are optimized to match gradients, we take a distinct approach that first identifies a set of words from gradients and then directly reconstructs sentences based on beam search and a prior-based reordering strategy. We conduct the FILM attack on several large-scale datasets and show that it can successfully reconstruct single sentences with high fidelity for large batch sizes and even multiple sentences if applied iteratively. We evaluate three defense methods: gradient pruning, DPSGD, and a simple approach to freeze word embeddings that we propose. We show that both gradient pruning and DPSGD lead to a significant drop in utility. However, if we fine-tune a public pre-trained LM on private text without updating word embeddings, it can effectively defend the attack with minimal data utility loss. Together, we hope that our results can encourage the community to rethink the privacy concerns of LM training and its standard practices in the future.

1 Introduction

Federated learning keeps client data local, but prior gradient attacks had limited success recovering text at realistic batch sizes. FILM reconstructs sentences from language-model training gradients, and its evaluations show recovery from batches of up to 128 sentences alongside a pretrained-embedding defense with minimal utility loss.

1 Introduction

  • Federated learning lets participants train a shared model without exchanging private data, with updates sent from clients to a central server for aggregation.The paper notes applications such as virtual keyboards and hospital analysis of electronic health records.Source: §1 · p. 1
  • Prior text-recovery success remained limited to unrealistically small batch sizes, motivating an attack that can recover private text from language-model training.The paper identifies language models’ memorization capability as a potential source of information leakage.Source: §1 · p. 2
  • FILM recovers a set of words from embedding gradients, builds candidate sentences with beam search, then reorders tokens using language priors and gradient information.The method can be applied iteratively to recover multiple sentences from one batch.Source: §1 · p. 2
  • Up to 128 sentences per batch, FILM enabled high-fidelity recovery of a single sentence; repeated applications recovered 1/3 of sentences from a batch of 16.The evaluation also reports recovery of significant portions of sentences for batches up to 128.Source: §1 · p. 2
  • Freezing word embeddings in a public pretrained LM effectively defends against FILM with minimal utility loss, while training from scratch this way causes more utility loss.Gradient pruning and DPSGD also substantially reduce utility.Source: §1 · p. 2

2 Related work

Prior gradient-inversion work established image recovery and early text leakage, but text methods had limited scope or stronger threat assumptions. FILM builds on embedding-gradient leakage and language-model memorization to target sentence recovery across federated-learning batches.

Data recovery from gradients

Defenses against gradient inversion attacks for text data

  • Cryptographic defenses can secure federated training against gradient inversion but may slow training and require special setup; noise and gradient pruning can hurt model accuracy.The cited text also describes InstaHide and its extension to language understanding tasks.Source: §2 · p. 3

3 Preliminaries

The preliminaries define language modeling and federated learning, then specify FILM’s eavesdropping threat model and the attacker’s sentence-recovery objective. The accompanying pipeline illustration depicts recovery from a batch of two sentences.

3 Preliminaries

  • The preliminaries establish language modeling, federated learning, and the threat model as the background for FILM.The section introduces the relevant background and threat model for the proposed attack.Source: §3 · p. 3

3.1 Language Modeling

  • Language modeling estimates a sequence’s probability, with neural models using embeddings and prior-token-conditioned hidden representations to predict the next token.The described models include RNNs and Transformers.Source: §3.1 · p. 3

3.2 Federated Learning

  • In federated learning, clients compute gradients on randomly sampled private-data batches and share updates for server aggregation; clients may start from random or public pretrained models.Figure 2 illustrates FILM’s batch-of-two pipeline: extract a bag of words, generate candidates with beam search, then reorder phrases and words.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Threat Model

  • The attacker can eavesdrop on client-server communications and inspect gradients and model parameters, aiming to recover at least one sentence from a private batch.The paper measures attack strength by similarity to the corresponding private sentence.Source: §3.3 · p. 4

4 Proposed Attack: FILM

FILM extracts words from gradients, reconstructs sentences with beam search, and reorders them using a prior that combines language likelihood and gradient information. The method supports single-sentence recovery from large batches and can be iterated to recover multiple sentences.

4.1 Overview

  • The attacker can inspect gradients at any training step, and the passage notes that attacks at larger steps may perform more strongly.Source: §4.1 · p. 4
  • The pipeline recovers single sentences from large training batches and can be repeated to build candidate recoveries for multiple sentences.The overview describes batches of up to 128 sentences; the multi-sentence extension repeats beam search with an additional n-gram penalty, then reorders candidates.Source: §4.1 · pp. 4–5, §4.5 · p. 6

4.2 Bag-of-Words Extraction

  • FILM first recovers a set of words from non-zero word-embedding gradient rows, which identifies words but not their frequencies.Non-zero position-embedding gradient rows also reveal the maximum sentence length in the batch.Source: §4.2 · p. 5

4.3 Beam Search for Sentence Reconstruction

  • Beam search starts from capitalized words, expands the top k likely continuations, and returns the highest-probability sentence at a specified length.The described implementation uses a 40-word length and adds a repeated n-gram penalty; with n = 2, performance is comparable to the frequency-informed oracle scenario.Source: §4.3 · p. 5

4.4 Prior-Guided Token Reordering

  • The prior score combines sentence likelihood and gradient norm to guide phrase-wise and token-wise reordering of the beam-search output.The two reordering stages select candidates by this score; phrase reordering and token reordering each terminate after 200 steps.Source: §4.4 · pp. 5–6

4.5 Recovering Multiple Sentences

  • To recover multiple sentences, FILM repeats beam search with an n-gram penalty informed by earlier searches, then reorders the resulting candidates.Source: §4.5 · p. 6

5 Defending Against the FILM Attack

Freezing word embeddings prevents FILM from recovering its initial bag of words. Among the tested settings, continuing training from a public pre-trained LM while freezing embeddings offers the best privacy-utility trade-off.

5 Defending Against the FILM Attack

  • Gradient pruning and DPSGD are less effective defenses against FILM and suffer substantial utility loss.Source: §5 · p. 6
  • Freezing word embeddings prevents recovery of the bag of words, blocking FILM’s critical first stage and leaving later stages no better than vocabulary-wide beam search.The same word-embedding-gradient step is also used by previously proposed gradient inversion attacks for smaller batch sizes.Source: §5 · p. 6
  • Continuing from a public pre-trained LM while freezing embeddings provides the best privacy-utility trade-off among the two settings considered.Training from scratch causes more utility loss because its word embeddings are randomly initialized.Source: §5 · p. 6

6 Experiments

Experiments show FILM recovers text across large training batches and iteratively recovers portions of multiple sentences, while tested defenses trade off attack resistance and model utility.

6.1 Setup

  • Experiments use GPT-2 base on WikiText-103 and Enron Email, with recovery assessed by ROUGE overlap and named-entity recovery ratio.The reported subsets contain 203,456 WikiText-103 sentences and 31,797 Enron Email sentences.Source: §6.1 · pp. 6–7

6.2 Performance of Our Attack

  • At batch size 1, recovery scores are near 1; as batch size grows, several scores decline, while Enron Email is more susceptible and FILM recovers portions of batches up to 128 sentences.Average F-scores across 20 mini-batches also weaken with larger batches and are stronger on Enron Email.Source: §6.2 · pp. 7–8, Figure 3 · p. 8
  • At larger batch sizes, prior gradient-matching methods degrade sharply, whereas FILM recovers a large portion of sentences at batch size 128.Zhu et al. (2019) fails to recover the original sentence at batch size 1 in the reported comparison.Source: §6.2 · pp. 7–8
  • Average ROUGE-L rises from 0.41 before fine-tuning to 0.53 after training on WikiText-103, while pre-trained models initially outperform randomly initialized models and their gap narrows over time.The authors attribute the stronger attack to model memorization, rather than only the language prior.Source: §6.2 · p. 8

6.3 Evaluation of Defenses

  • Gradient pruning preserves token precision at 1, while recall falls as pruning increases; FILM still recovers over 90% of tokens at prune ratio 0.9999.Pruning removes some embedding-gradient entries, making the corresponding tokens unrecoverable.Source: Table 4 · p. 9, §6.3 · p. 9
  • Under DPSGD, attack performance falls as epsilon decreases, but perplexity worsens; FILM retrieves fewer than half the tokens when epsilon is below 15.The attack uses a threshold to distinguish likely token gradients from noise.Source: Table 4 · p. 9, §6.3 · p. 9

6.2 Performance of Our Attack

  • With 100 beam-search repetitions at batch size 16, FILM recovers roughly 34% of the batch as sentences with ROUGE-L above 0.25.The resulting 100 generations still require a way to select good recoveries; the table defines recall over original sentences and precision over generated sentences.Source: §6.2 · p. 9, Table 3 · p. 9

6.3 Evaluation of Defenses

  • Freezing word embeddings prevents token recovery, but significantly harms utility when training from scratch and has negligible utility impact with a pre-trained model.The reported utility comparison uses perplexity for frozen and unfrozen GPT-2 models.Source: §6.3 · p. 9, Table 5 · p. 9

7 Analysis

The analysis finds that larger beams improve recovery but increase computation, with a beam size around 32 offering the best performance–efficiency trade-off. Repeated 2-grams are best penalized, β = 1 is optimal for prior-based scoring, and token reordering helps most at small batch sizes.

7 Analysis

  • A beam size around 32 gives the best trade-off between attack performance and efficiency as larger beams improve recovery but require more computation.The 2-gram penalty performs closest to the oracle case, where word frequencies are known.Source: §7 · p. 10, Figure 5 · p. 10
  • Repeated 2-gram penalties perform best; no penalty performs significantly worse, and n = 2 is closest to the oracle case.Source: §7 · p. 10, Figure 5 · p. 10
  • β = 1 is optimal for prior-based scoring, balancing perplexity and gradient-norm guidance across batch sizes.The score is Perplexity(x) + β · Gradient Norm(x).Source: §7 · p. 10, Figure 6 · p. 10
  • Token reordering improves ROUGE-L most for small batch sizes, which the authors hypothesize is because beam-search results are closer to the ground-truth sentence.Source: §7 · p. 10, Figure 6 · p. 10

8 Discussion and Future Work

FILM recovers multiple sentences from gradients in federated language-model training, including with batches up to 128, and uses language-model knowledge to reconstruct sentences. Freezing word embeddings can defend against FILM when fine-tuning a pretrained model, with little utility loss; the paper also identifies limits in task and batch-size coverage.

8 Discussion and Future Work

  • FILM recovers multiple sentences from federated language-model gradients and works with Transformer-based models and batches up to 128.The authors identify language-model memorization as a key source of reconstruction power.Source: §8 · p. 10
  • Freezing word embeddings effectively defends against FILM when starting from a pretrained language model, with little utility loss.Source: §8 · p. 10
  • The attack was not evaluated on other NLP tasks or batches beyond 128, and the authors leave sentence selection in multi-sentence recovery open.Freezing embeddings during pretraining can harm utility, leaving effective defense in that setting unresolved.Source: §8 · p. 10
  • The authors warn FILM may support stronger attacks and call for defenses that provide meaningful privacy guarantees for federated language-model training.Source: §8 · p. 10

Checklist

The appendices describe the beam-search procedure, evaluation preprocessing, and how FILM differs from prior attacks. The checklist also reports reproducibility and evaluation disclosures, while noting that Enron Email has no license information.

Checklist

  • The checklist reports that code and experiment materials are provided in the supplement, training details and compute resources are documented, and error bars appear in Figure 3.Source: Checklist · p. 14
  • The checklist says existing assets are cited, but no license information is available for the Enron Email dataset.Source: Checklist · p. 14

A Beam Search Algorithm

  • Beam search takes a bag of words, maximum sentence length, beam size, prompts, a trained language model, and an n-gram penalty, and outputs candidate sentences.Source: A Beam Search Algorithm · p. 15

B.1 Preprocessing details for WikiText-103 and Enron Email

  • Evaluation sentences were limited to fewer than 40 tokens and five unknown tokens, perplexity below 50, and at least one named entity; longer paragraphs were left for future work.The evaluation set contains 2,000 examples from each dataset.Source: §B.1 · p. 15

B.2 Comparison with Previous Attacks

  • FILM is distinguished from prior attacks by avoiding end-to-end optimization, handling larger batch sizes, and targeting autoregressive language modeling.Source: §B.2 · p. 15, Table 6 · p. 15

B.3 More Results of DLG (Zhu et al. 2019) and TAG (Deng et al. 2021)

The appendix reports that FILM recovers more sentence data than prior methods across datasets and batch sizes, while earlier gradient-matching approaches are sensitive to batch size and initialization. It also presents reconstruction metrics and best-sentence F-scores across batch sizes.

B.3 More Results of DLG (Zhu et al. 2019) and TAG (Deng et al. 2021)

  • Zhu et al. (2019)’s gradient-matching attack fails across batch sizes and succeeds only with initialization one or two words from the original sentence.Tables 7 and 8 report batch-size and initialization results, respectively.Source: §B.3 · p. 16, Table 7 · p. 16, Table 8 · p. 16
  • FILM recovers more sentence data than prior methods across datasets and batch sizes, with a marked ROUGE-2 gap indicating better word ordering.Table 9 compares reconstructions across datasets, prior methods, and batch sizes.Source: Table 9 · p. 16, §B.3 · p. 16

B.4 Quantitative Results for Different Batch Sizes

B.5 More Results For Attacking the Pre-trained Model

For a pretrained model, FILM reconstructs better sentences as training iterations increase, while attacks without private-data training are relatively weak. Random initialization yields lower overall attack quality than pretraining.

B.5 More Results For Attacking the Pre-trained Model

  • Without private-data training, FILM is relatively weak across batch sizes, indicating that model memorization provides a stronger attack than the pretrained model's language prior alone.Table 12 reports the best-case reconstruction across 10 evaluated batches for each batch size.Source: §B.5 · p. 17, Table 12 · p. 18
  • FILM recovers better sentences as training iterations increase for both pretrained and randomly initialized models, though random initialization produces lower overall attack quality.The pretrained-model results use batch size 1; the table captions describe the recovered phrases and words across iterations.Source: Table 11 · p. 17, §B.5 · p. 17, Table 13 · p. 18

B.6 Qualitative Results with Randomly Initialized Models

  • Randomly initialized models yield lower overall attack quality than pretrained models as training iterations increase.Table 13 illustrates recovered phrases and words for batch size 1.Source: Table 13 · p. 18, §B.6 · p. 18

B.7 Ablation Study for Training Configurations

The ablations find similar attack performance across evaluated training-set sizes and slightly better reconstruction with larger initial learning rates. The reordering score reflects lower gradient norms and perplexities for original training sentences, while the bag of words particularly helps with named entities.

B.7 Ablation Study for Training Configurations

  • Attack performance is similar across training sets of 5,000 to 200,000 examples, while larger initial learning rates slightly improve ROUGE and recover more sentence components despite similar final perplexities.The learning-rate comparison fixes batch size at 16 and dataset size at 200,000, and uses early stopping.Source: Figure 8 · p. 18, §B.7 · pp. 18–19, Table 14 · p. 19

B.8 Justification for the Scoring Function in the Reordering Stage

  • Original training sentences usually have lower gradient norms and perplexities than slightly altered versions, motivating both terms in the reordering-stage scoring function.Figure 9 compares five original sentences with variants produced by swapping, deleting, or inserting a token.Source: §B.8 · p. 19, Figure 9 · p. 20

B.9 How much does the bag of words help?

  • With a bag of words, beam search is significantly stronger for small batches and reconstructs significantly more named entities even at larger batch sizes.Figure 10 compares beam search with and without the bag of words across training iterations and batch sizes, and includes NERR at 90,000 iterations.Source: §B.9 · p. 19, Figure 10 · p. 20

Related papers

Loading 2205.08514v2…