PaperlayerView exact revision on arXiv

Source-linked AI summary

TAG: Gradient Attack on Transformer-based Language Models

Jieren Deng, Yijue Wang, Ji Li, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, Caiwen Ding

arXiv:2103.06819v6cs.CR

TL;DR

Gradient leakage can reveal private training data, yet its mechanism in Transformer-based language models had not been systematically studied. TAG formulates the attack and reconstructs local training data from shared gradients; experiments report improved recovery over prior methods across model and weight settings.

  • Problem

    Gradient leakage has been shown for images, but the mechanism and evaluation of attacks on Transformer-based language models had not been systematically established.

    Source: §1 · p. 1

  • Method

    TAG formulates gradient attacks on Transformer language models and reconstructs local training data by matching adversary gradients to shared gradients.

    Source: §1 · p. 2

  • Results

    TAG works across more weight distributions and achieves 1.5× Recover Rate and 2.5× ROUGE-2 over prior methods without ground-truth labels.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The reported experiments indicate that Transformer gradients can reveal private text, including information such as names.

    Source: §7 · p. 9

    The formulation assumes the adversary can access shared gradients and the current global model, and considers random or pretrained initialization.

    Source: §3.2 · pp. 3–4

Abstract

from arXiv · show

Although federated learning has increasingly gained attention in terms of effectively utilizing local devices for data privacy enhancement, recent studies show that publicly shared gradients in the training process can reveal the private training images (gradient leakage) to a third-party in computer vision. We have, however, no systematic understanding of the gradient leakage mechanism on the Transformer based language models. In this paper, as the first attempt, we formulate the gradient attack problem on the Transformer-based language models and propose a gradient attack algorithm, TAG, to reconstruct the local training data. We develop a set of metrics to evaluate the effectiveness of the proposed attack algorithm quantitatively. Experimental results on Transformer, TinyBERT$_{4}$, TinyBERT$_{6}$, BERT$_{BASE}$, and BERT$_{LARGE}$ using GLUE benchmark show that TAG works well on more weight distributions in reconstructing training data and achieves 1.5$\times$ recover rate and 2.5$\times$ ROUGE-2 over prior methods without the need of ground truth label. TAG can obtain up to 90$\%$ data by attacking gradients in CoLA dataset. In addition, TAG has a stronger adversary on large models, small dictionary size, and small input length. We hope the proposed TAG will shed some light on the privacy leakage problem in Transformer-based NLP models.

1 Introduction

Shared gradients can expose private training data, but gradient attacks on Transformer language models lacked systematic evaluation. TAG reconstructs training data across weight settings, with stronger attacks reported for larger models and smaller dictionaries and input sequences.

1 Introduction

  • Prior gradient-attack work lacked generalizability across weight distributions and systematic NLP evaluation, including analysis across Transformer attention heads and architectures.Source: §1 · p. 1
  • TAG reconstructs private training data by updating dummy inputs to match the gradients received from a distributed-learning participant.Source: §1 · p. 1, Figure 1 · p. 2
  • TAG is proposed as a general attack for Transformer-based language models and works on both pretrained and normally initialized models.Source: §1 · p. 2
  • TAG achieves 1.5× Recover Rate and 2.5× ROUGE-2 over prior methods while quantitatively evaluating NLP gradient attacks.The evaluation uses Recover Rate, ROUGE-1, ROUGE-2, ROUGE-L, and runtime.Source: §1 · p. 2
  • Larger models, smaller vocabulary dictionaries, and shorter input sequences are reported as riskier for private-data leakage.Source: §1 · p. 2

2 Related Work

Related work frames gradient leakage within broader privacy attacks and distributed training. It distinguishes gradient-based reconstruction from model inversion and reviews prior attacks and their evaluation gaps.

2.1 Privacy leakage problem

  • Gradient-based privacy attacks recover private training data by exploiting model gradients, unlike model inversion, which uses correlations between data and model outputs.Source: §2.1 · p. 2

2.2 Distributed learning

  • In synchronous SGD, a server aggregates gradients shared by local devices and returns the aggregate for the next training round.Source: §2.2 · p. 2

2.3 Prior arts on gradients-based attack

  • Prior attacks include word identification and gradient-matching reconstruction, but DLG evaluation did not thoroughly cover weight settings, networks, or training stages; Transformer attacks lacked benchmark datasets and standard metrics.Source: §2.3 · p. 3

3 Approach

The approach formulates recovery of private language tokens from shared gradients and develops TAG to optimize dummy data against those gradients. Its distance function combines gradient norms to address variation across weight initializations.

3.1 Transformer-based NLP models

  • Transformer-based models use scaled dot-product self-attention, and multi-head attention computes it in embedding subspaces before concatenating the outputs.The attention calculation uses query, key, and value representations with a scaling factor.Source: §3.1 · p. 3

3.2 NLP gradient attack formulation

  • The attack assumes an adversary cannot access local training data directly but can access shared gradients and the current global model, and seeks to recover training tokens and labels.The formulation considers attacks at any training stage, including randomly initialized and pretrained models.Source: §3.2 · pp. 3–4

3.3.1 Recovered token initialization

  • The recovery procedure randomly initializes dummy input and label, then seeks dummy data whose gradient is close to the ground-truth gradient.Source: §3.3.1 · p. 4

3.3.2 Distance function for gradient matching

  • TAG uses a combined L2 and L1 gradient distance because Euclidean distance can be dominated by large gradients when normal initialization produces mostly near-zero gradients.The layer-specific coefficient decreases with layer order, weighting gradient differences in front layers more heavily because they are closer to the private input.Source: §3.3.2 · p. 4

3.4 The framework of the algorithm

  • TAG iteratively initializes dummy input and label, computes their gradient, minimizes its distance from the shared gradient, and updates the dummy data until a stopping condition is met.Stopping occurs at the preset iteration limit, 200 iterations, or when the number of recovered ground-truth tokens stops changing.Source: §3.4 · p. 4, §3.3.1 · p. 4

4 Experimental setup

The experiments evaluate TAG on GLUE tasks with varied sentence lengths and across Transformer, TinyBERT, and BERT model sizes. Evaluation uses token recovery, text-overlap, and runtime metrics.

4 Experimental setup

  • Experiments ran on servers with Intel Xeon Gold 5218 CPUs, 504 GB memory, and eight NVIDIA Quadro RTX 6000 GPUs.The software stack used PyTorch 1.5.1, Python 3.6, and CUDA 10.2.Source: §4 · pp. 4–5

4.1 Datasets

  • Experiments use CoLA, SST-2, and RTE, whose typical sentence lengths are about 5–15, 10–30, and 50–100 words, respectively.The paper describes TAG as data agnostic and applicable to text inputs from any benchmark or source.Source: §4.1 · p. 5

4.2 Model settings

  • The study tests a basic Transformer, TinyBERT4, TinyBERT6, BERTBASE, and BERTLARGE with differing layer counts, hidden sizes, feed-forward sizes, and attention heads.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 Experiment parameters settings

  • For each task, the experiments use BertAdam with learning rate 0.05 and a maximum of 1,000 iterations per sentence recovery.Source: §4.3 · p. 5

4.4 Experiment evaluation

  • The evaluation measures maximum token Recover Rate, ROUGE-1, ROUGE-2, ROUGE-L text similarity, and average attack runtime.ROUGE-1 and ROUGE-2 measure unigram and bigram overlap; ROUGE-L measures the longest matching token subsequence.Source: §4.4 · pp. 5–6
  • Figure 2 tracks average loss across four model variants and CoLA, SST-2, and RTE; loss decreases during the first 200 iterations and then stabilizes.Source: Figure 2 · p. 6

5 Result Analysis and Visualization

TAG outperforms DLG on the reported text-recovery metrics, while results also show that recovery improves over iterations and larger models leak more information. The visualizations track gradient convergence, token-embedding similarity, and recovery of a CoLA sentence.

5 Result Analysis and Visualization

  • The gradient-distance loss decreases continuously across the evaluated models, with stable, distinct convergence especially during the first 200 iterations.The loss is normalized on a log scale between 0.8 and 1.2.Source: §5 · p. 6
  • 0.93 cosine similarity: dimension-reduced dummy and ground-truth token embeddings reach this value after 200 iterations, up from 0.42 at iteration 5.The Figure 3 visualization uses PCA to plot the dimension-reduced embeddings of TinyBERT4 on CoLA.Source: §5 · p. 6, Figure 3 · p. 7
  • BERT_LARGE leaks more information than smaller models, but its recovery runtime is higher: 3× TinyBERT4’s runtime, while BERTBASE takes 2.5× TinyBERT4’s.The comparison averages Recover Rate, ROUGE-1, ROUGE-2, ROUGE-L, and runtime across CoLA, SST-2, and RTE.Source: §5 · pp. 6, 8, Table 2 · p. 7
  • 88.89% Recover Rate: TAG recovers 8 of 9 tokens in the CoLA sentence example by 200 iterations, after recovering 7 of 9 at 50 iterations.The Figure 4 caption identifies the example as a sentence from CoLA.Source: §5 · p. 6, Figure 4 · p. 8
  • About 2.7× ROUGE-2 and 1.5× ROUGE-1, ROUGE-L, and Recover Rate: TAG outperforms DLG on the reported comparison.Figure 5 normalizes the four metrics to DLG; Table 3 reports up to 2× as many recovered words on sample texts.Source: Table 3 · p. 7, Figure 5 · p. 8, §5 · p. 8

6 Ablation Studies

The ablations examine how TAG’s recovery varies with weight distributions, pre-trained weights, sentence length, and vocabulary size. Recovery is stronger with some settings, including pre-trained weights, shorter CoLA inputs, and a smaller vocabulary.

6.1 Effects of weight distributions

  • At standard deviation 0.01, TAG recovers half the words, versus one of three at 0.03; uniform initialization yields more recovery with a larger range.These weight-distribution experiments use TinyBERT6 and CoLA.Source: §6.1 · p. 8

6.2 Effects of weights from pre-trained model

  • 1.5× better recovery on CoLA with pre-trained weights than with normal initialization; the pre-trained model performs better overall across the comparison.The authors suggest pre-trained weights may contain more information from training.Source: §6.2 · p. 9

6.3 Performance on different datasets

  • 3× better ROUGE-1 and ROUGE-2 on CoLA than RTE, and 2.5× better ROUGE-L.The comparison uses CoLA inputs within ten words and RTE inputs over 50 words.Source: §6.3 · p. 9

6.4 Effects of vocabulary dictionary

  • A vocabulary of 21,128 tokens yields more than 1.6× improvement over a 30,522-token vocabulary across all evaluation metrics.The vocabulary comparison uses TinyBERT6 on CoLA.Source: §6.4 · p. 9

7 Conclusion

The paper proposes TAG to recover private text from Transformer gradients and evaluates the attack quantitatively. Across several Transformer-based models and GLUE, TAG outperforms prior methods on recovery metrics, including without ground-truth labels.

7 Conclusion

  • 1.5× Recover Rate and 2.5× ROUGE-2 over prior methods, without ground-truth labels.Experiments cover Transformer, TinyBERT4, TinyBERT6, BERTBASE, and BERTLARGE on GLUE.Source: §7 · p. 9
  • Up to 88.9% of tokens and 0.93 cosine similarity in token embeddings are recovered from private training data on CoLA.Source: §7 · p. 9
  • TAG recovers private training data across more weight distributions in experiments on five Transformer-based model variants.The tested models are Transformer, TinyBERT4, TinyBERT6, BERTBASE, and BERTLARGE.Source: §7 · p. 9

Related papers

Loading 2103.06819v6…