PaperlayerView exact revision on arXiv

Source-linked AI summary

Information Leakage in Embedding Models

Congzheng Song, Ananth Raghunathan

arXiv:2004.00053v2cs.LGcs.CLcs.CRstat.ML

TL;DR

Embedding models are intended to preserve useful semantics, but the paper asks whether they also expose sensitive information from their inputs. It systematically attacks word and sentence embeddings through inversion, attribute inference, and membership inference, finding substantial leakage and partial mitigation through adversarial training.

  • Problem

    Embedding models may capture and expose sensitive input information beyond the semantic information needed for downstream tasks.

  • Method

    The paper systematically evaluates inversion, sensitive-attribute inference, and membership-inference attacks across widely used word and sentence embedding models, and tests adversarial-training defenses.

  • Results

    The attacks recover sentence words with high precision and recall, improve authorship classification by 1.5–3× with 10–50 labeled sentences per author, and achieve a 30% membership advantage over random guessing.

  • Takeaways & Limitations

    Embeddings should not be treated as merely real-valued vectors when inputs are sensitive, because they can leak words, authorship, and membership information.

  • Takeaways & Limitations

    Differentially private training is left for future work because the over-10-million-parameter word-embedding matrices make training and hyperparameter tuning computationally infeasible.

Abstract

from arXiv · show

Embeddings are functions that map raw input data to low-dimensional vector representations, while preserving important semantic information about the inputs. Pre-training embeddings on a large amount of unlabeled data and fine-tuning them for downstream tasks is now a de facto standard in achieving state of the art learning in many domains. We demonstrate that embeddings, in addition to encoding generic semantics, often also present a vector that leaks sensitive information about the input data. We develop three classes of attacks to systematically study information that might be leaked by embeddings. First, embedding vectors can be inverted to partially recover some of the input data. As an example, we show that our attacks on popular sentence embeddings recover between 50\%--70\% of the input words (F1 scores of 0.5--0.7). Second, embeddings may reveal sensitive attributes inherent in inputs and independent of the underlying semantic task at hand. Attributes such as authorship of text can be easily extracted by training an inference model on just a handful of labeled embedding vectors. Third, embedding models leak moderate amount of membership information for infrequent training data inputs. We extensively evaluate our attacks on various state-of-the-art embedding models in the text domain. We also propose and evaluate defenses that can prevent the leakage to some extent at a minor cost in utility.

1 INTRODUCTION

The paper initiates a systematic study of privacy leakage in embedding models, showing that embeddings can expose input words, sensitive attributes, and training-data membership. It evaluates attacks across widely used word and sentence embeddings and tests adversarial-training defenses.

  • Motivation: Embeddings are widely used to transfer information from large unlabeled datasets to downstream tasks, motivating scrutiny of what sensitive information they retain.The paper frames embeddings as more than semantic representations because they are computed from potentially sensitive inputs and shared for downstream use.
  • Attack scope: The study examines three leakage classes: recovering input words, inferring sensitive attributes such as authorship, and inferring training-data membership.These attacks target different information exposed through embedding vectors rather than only the intended downstream semantics.
  • Inversion attacks: 60%+ precision and recall are achieved by inversion techniques that recover significant portions of sentence inputs under black-box and white-box access.The reported result concerns recovery of exact input words rather than merely abstract semantics.
  • Sensitive-attribute inference: 1.5–3× improvement over prior stylometry methods is obtained for authorship classification using 10 to 50 labeled sentences per author.The comparison covers hundreds of authors and embedding models trained with different learning paradigms.
  • Membership inference: 30% improvement over random guessing is achieved for membership inference on both word and sentence embeddings, though the threat is weaker than the other attacks.The paper nevertheless characterizes membership inference as a viable privacy threat.
  • Defenses: Adversarial training reduces inversion and attribute-inference attacks by 30% and 80%, respectively, at a minor utility cost.The authors evaluate these techniques as mitigations rather than treating reduced utility alone as the explanation for lower leakage.

2 PRELIMINARIES

The paper describes how word and sentence embeddings map text into vectors and how threat models give adversaries access to embeddings, auxiliary data, or embedding-model interfaces. It organizes leakage into inversion, attribute inference, and membership inference attacks.

  • Word embeddings: Word embeddings map vocabulary words to vectors, with nearby vectors representing words with similar meanings.Word2Vec, FastText, and GloVe are cited as common unsupervised word-embedding models.
  • Sentence embeddings: Sentence embeddings map variable-length word sequences to fixed-size vectors through neural networks, using reductions such as last-state or mean pooling.They are commonly trained on large unlabeled corpora; dual encoders learn from paired context sentences and negative samples.
  • Sentence embeddings: Dual-encoder training maximizes similarity between embeddings of context sentences while minimizing similarity to randomly sampled negative sentences.The objective predicts the correct context sentence from the positive and negative candidates conditioned on the other context sentence.
  • Threat model: The threat model includes a potentially sensitive training dataset, a white-box or black-box embedding model, target embeddings, and auxiliary labeled or unlabeled data.Auxiliary labeled data may be limited and same-distribution, while unlabeled text is comparatively cheap to collect.
  • Attack taxonomy: Attackers may recover text, infer sensitive attributes, or determine whether an input and its context were used to train the embedding model.The taxonomy distinguishes reconstruction, attribute inference, and membership inference according to the attacker's target.
  • Attack taxonomy: Sensitive-attribute inference is studied with only 10–50 labeled examples per class, where transfer from embeddings may outperform direct supervised learning from inputs.The adversary trains a classifier on labeled auxiliary data to infer a discrete attribute from a target embedding.

3 EMBEDDING INVERSION ATTACKS

Embedding inversion attacks recover sets of words from target embeddings under both white-box and black-box access. The methods use continuous optimization, learned mappings, or auxiliary-data inversion models to expose information retained in embeddings.

  • 3 EMBEDDING INVERSION ATTACKS: Embedding inversion targets recovery of the input text from its embedding, focusing on word sets rather than exact word order.Exact sequence recovery remains open; the section analyzes short texts and unordered word recovery.
  • 3.1 White-box Inversion: White-box inversion relaxes discrete word selection into continuous variables, enabling gradient-based optimization against the embedding model.Softmax attention converts position-wise variables into relaxed word vectors, then argmax recovers words.
  • 3.1 White-box Inversion: For deep embeddings, a learned mapping first predicts a lower-layer representation before optimization minimizes distance to that representation.A linear least-squares mapping works reasonably well in practice.
  • 3.2 Black-box Inversion: Black-box inversion trains an auxiliary-data model to map observed embeddings directly to the set of input words.The inversion model learns word-set probabilities from queried embedding–word-set pairs and predicts the target words from its embedding.
  • 3.2 Black-box Inversion: Multi-label classification predicts each vocabulary word independently, while multiset prediction models an ordered sequence of word predictions conditioned on previously predicted words.The multiset formulation is reported to improve the precision-recall trade-off over independent multi-label classification.

4 ATTRIBUTE INFERENCE ATTACKS

Attribute inference tests whether embeddings expose sensitive input attributes unrelated to their downstream task. An adversary uses a small labeled auxiliary set to train a classifier, leveraging similarity patterns learned from large unlabeled corpora.

  • 4 ATTRIBUTE INFERENCE ATTACKS: Sensitive attributes such as authorship can be inferred by treating attribute prediction as a downstream classification task on embeddings.The adversary trains a classifier on embeddings labeled with the sensitive attribute and applies it to target embeddings.
  • 4 ATTRIBUTE INFERENCE ATTACKS: The attack is designed for limited-label settings, restricting the adversary to a small auxiliary dataset of embeddings and sensitive-attribute labels.This setting is intended to reflect the cost and scarcity of labeled sensitive data.
  • 4 ATTRIBUTE INFERENCE ATTACKS: Contrastive training makes embeddings similar for data in the same context and dissimilar for negative samples, enabling latent-class attributes to be inferred when they correlate with context.Unsupervised pre-training on much larger unlabeled data can capture similarities that limited labeled data alone may not reveal.

5 MEMBERSHIP INFERENCE ATTACKS

Membership inference measures whether embedding-model training data is exposed through model outputs. The attacks use similarity scores for contexts, sentence pairs, or aggregates, with optional learned metrics customized from auxiliary membership labels.

  • 5 MEMBERSHIP INFERENCE ATTACKS: Membership inference asks whether a context used by an unsupervised embedding model belonged to its training data.Contexts may be sliding windows of words or pairs of sentences rather than conventional labeled examples.
  • 5.1 Word Embeddings: Word-level attacks average similarity scores between a central word and neighboring words, then threshold the average to decide membership.The method applies to words trained in sliding windows and can use measures such as cosine similarity.
  • 5.2 Sentence Embeddings: Sentence-level attacks use the similarity of two contextual sentence embeddings as the membership decision score.Sentences appearing together during training are expected to be more similar than sentences not used together.
  • 5.3 Aggregate-level Membership Inference: Aggregate-level attacks infer whether a user contributed training data by averaging similarity scores across consecutive sentence embeddings.This targets user-level participation when models train on aggregated input logs.
  • 5 MEMBERSHIP INFERENCE ATTACKS: A learned similarity metric projects embeddings with W_m before computing similarity, using auxiliary membership labels to customize the inference score.The learned metric is trained with a binary cross-entropy objective.

6 EXPERIMENTAL EVALUATION

The evaluation tests inversion, sensitive-attribute inference, and membership inference across word and sentence embedding models, then examines layer effects and inversion defenses. Results show substantial recoverability and attribute leakage, while membership leakage is strongest for infrequent data and can be mitigated through adversarial training.

  • Experimental setup: The evaluation uses locally trained word and sentence embeddings plus pretrained BERT and ALBERT sentence embeddings, with precision, recall, and F1 measuring inversion recovery.Word embeddings use Wikipedia data, while sentence embeddings use BookCorpus data and LSTM or Transformer architectures.
  • Embedding inversion: More than half of target inputs can be recovered on nearly all models with Equation 7, with only little performance loss when the mapping uses cross-domain auxiliary data.Equation 7 improves all performance scores over Equation 5.
  • Embedding inversion: Higher BERT and ALBERT layers sharply reduce direct inversion performance, whereas learning a mapping makes the decline across layers much less significant.Figure 2 compares precision and recall from layers 0 through 12.
  • Embedding inversion: LMSP provides a better precision–recall balance and higher F1 scores than LMLC, which achieves high precision but low recall in black-box inversion.The reported difference is attributed to LMLC’s inversion model favoring auxiliary-data words.
  • Sensitive attribute inference: Dual-encoder LSTM and Transformer embeddings achieve the strongest authorship inference across labeled-data and author-count settings, outperforming TextCNN and other pretrained embeddings.The experiments evaluate top-5 accuracy while varying labeled sentences per author and the number of authors.
  • Membership inference: Membership advantage is below 0.1 for frequent word-embedding inputs but rises to roughly 0.3 as frequency decreases; book-level sentence attacks likewise reach 0.3 for infrequent sentences.Context-level sentence membership inference remains below 0.1 across frequency ranges, while learning-based similarity increases book-level advantage.

7 DEFENSES

The paper evaluates adversarial training as a defense against embedding inversion and sensitive-attribute inference, balancing attack performance against downstream utility. It reduces both attacks, with authorship mitigation producing relatively stable utility and inversion mitigation affecting LSTM models more strongly.

  • Adversarial training: Adversarial training jointly optimizes the embedding model and an adversary, making the embedding model minimize sensitive-information predictability while preserving its main objective.Gradient reversal efficiently optimizes both minimax objectives.
  • Inversion defense: As λw increases, adversary performance against inversion drops for all models, while white-box inversion attacks decline most significantly.LSTM models require larger λw than Transformer models for similar mitigation.
  • Inversion defense: Utility scores drop more drastically for LSTM than Transformer embedding models because LSTM uses a larger λw.The comparison concerns downstream benchmark utility under inversion defense.
  • Sensitive-attribute defense: As λs increases, authorship-inference performance drops significantly for attacks trained with 10 and 50 labeled examples per author.The result holds for both inference attack models evaluated in Figure 7.
  • Sensitive-attribute defense: Nearly all utility scores on four sentence-analysis benchmarks remain stable across different λs values, indicating different adversary tasks can affect utility differently.Removing input-word information is reported to be harder and more utility-costly than removing authorship.

8 RELATED WORK

Related work covers privacy leakage in supervised and generative models, representation inversion, membership inference, and privacy-preserving training. This paper extends these concerns to unsupervised text embeddings and notes that differential privacy remains computationally infeasible for its large embedding matrices.

  • Privacy in deep representations: Prior work studies membership inference, memorization, and sensitive-attribute leakage in supervised, generative, and language models.This paper instead focuses on unsupervised text embedding models and considers both sensitive attributes and raw input leakage.
  • Inverting deep representations: Unlike prior work focused on structural patterns in language-model inputs, this work assumes no input-text structures and still recovers a substantial portion of the data.The authors describe their attack taxonomy as broader in scope than the concurrently related work.
  • Inverting deep representations: The paper’s inversion attacks target exact raw input text from embedding vectors rather than class representatives reconstructed from classifier outputs.The methods are tailored to unsupervised text embeddings produced by recurrent neural networks and Transformers.
  • Differential Privacy: Differential privacy clips and noises instance-level gradients but does not defend against sensitive-attribute inference based on aggregate training-data properties.The paper leaves efficient differentially private embedding training for future work because its word-embedding matrices exceed 10 million parameters.

9 CONCLUSIONS

The paper shows that embedding vectors can leak raw text, authorship, and membership information, and that adversarial training partially mitigates these attacks with a minor utility cost. The results motivate caution when embeddings represent sensitive inputs.

  • Conclusions: Embedding vectors can be inverted to recover sentence words with high precision and recall, reveal authorship from a few labeled examples, and leak moderate membership information for infrequent data.Membership leakage is measured using similarity scores from embedding vectors in context.
  • Conclusions: Adversarial training partially mitigates the attacks at the cost of a minor decrease in utility.The paper presents this as its proposed defense against information leakage.
  • Implications: When inputs are sensitive, embeddings should not be treated as merely vectors of real numbers because they may encode sensitive information that can be exfiltrated.The caution follows the paper’s conclusion that embeddings encode both useful semantics and sensitive input information.
Loading 2004.00053v2…