PaperlayerView exact revision on arXiv

Source-linked AI summary

Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence

Haoran Li, Mingshi Xu, Yangqiu Song

arXiv:2305.03010v1cs.CLcs.CR

TL;DR

Sentence embeddings can expose input text during downstream use, raising privacy concerns. GEIA treats embeddings as initial token representations for a decoder, and experiments show it can generate coherent, contextually similar sequences that outperform prior inversion attacks in classification metrics.

  • Problem

    Sentence embeddings used in downstream applications may disclose original sentences without authorization, creating privacy risks.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    GEIA trains a decoder to reconstruct ordered input sequences by using a sentence embedding as the initial token representation.

    Source: §1 · pp. 1–2, §3.5 · p. 4

  • Results

    GEIA outperforms previous embedding inversion attacks in classification metrics and generates coherent, contextually similar sentences, sometimes verbatim.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments indicate that current state-of-the-art sentence embedding models are vulnerable to GEIA.

    Source: §1 · p. 2, §5 · p. 8

    The attacker model is less capable of recovering exact domain-specific tokens, and attacks are more successful on PersonaChat than on QNLI.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Sentence-level representations are beneficial for various natural language processing tasks. It is commonly believed that vector representations can capture rich linguistic properties. Currently, large language models (LMs) achieve state-of-the-art performance on sentence embedding. However, some recent works suggest that vector representations from LMs can cause information leakage. In this work, we further investigate the information leakage issue and propose a generative embedding inversion attack (GEIA) that aims to reconstruct input sequences based only on their sentence embeddings. Given the black-box access to a language model, we treat sentence embeddings as initial tokens' representations and train or fine-tune a powerful decoder model to decode the whole sequences directly. We conduct extensive experiments to demonstrate that our generative inversion attack outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as the original inputs.

1 Introduction

Sentence embeddings used in downstream NLP tasks can carry sensitive information, while earlier inversion attacks recover only partial or unordered words. GEIA reframes inversion as generation, reconstructing ordered sequences across varied embedding models and outperforming earlier attacks on classification metrics.

1 Introduction

  • Sentence embeddings used for downstream tasks can disclose sensitive query information, creating privacy risks.The paper notes that unintended disclosure may violate legislation and hinder people from contributing data to service models.Source: §1 · p. 1
  • Earlier inversion attacks recover partial keywords or unordered word sets, while full-sentence recovery exposes additional semantic information and enables subsequent attacks.The paper says recovered sentences can be used for attribute inference or converted to bag-of-words for prior inversion methods.Source: §1 · p. 1
  • GEIA generates ordered, mostly coherent sequences—sometimes verbatim—and outperforms earlier inversion attacks on classification metrics.Experiments include Sentence-BERT, SimCSE-BERT/SimCSE-RoBERTa, Sentence-T5, and MPNet.Source: §1 · p. 2
  • GEIA treats inversion as generation rather than classification, enabling recovery of ordered input sequences from sentence embeddings.The overview contrasts prior word-set prediction with reconstruction of actual input sequences.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • GEIA is designed to adapt across embedding architectures and training approaches, including BERT-to-T5 models, contrastive learning, and siamese networks.Source: §1 · p. 2

2 Related Works

Sentence-embedding research moved from predictive and autoencoding approaches to transformer-based models, while privacy work established that representations can expose private information. Prior inversion methods predict partial or unordered words; GEIA instead targets ordered sentence generation.

2 Related Works

  • Sentence representations developed from surrounding-sentence prediction and autoencoding to transformer architectures, including Sentence-BERT, SimCSE, and Sentence-T5.Source: §2 · p. 2
  • Prior privacy attacks recover training data, infer attributes, or extract partial patterns and unordered word sets from language-model representations.The passage describes training-data extraction and gradient leakage alongside attacks on sentence embeddings.Source: §2 · p. 2
  • GEIA projects sentence embeddings to the attacker’s token-representation dimension, concatenates them with token representations during training, and uses them as initial representations at inference.Source: Figure 2 · p. 3
  • Earlier embedding inversion methods frame recovery as classification and cannot reconstruct input sequences; GEIA reformulates it as generation and also outperforms them on classification metrics.Source: §2 · p. 3

3 Embedding Inversion Attacks on LMs

Sentence embeddings may expose sensitive input text, while prior inversion methods recover unordered token sets rather than sentences. GEIA uses a generative decoder with black-box access to reconstruct ordered sequences, and experiments report stronger classification performance and useful sentence recovery.

3.1 Motivation

  • Sentence embeddings can disclose original sentences in settings such as sensitive-domain search, where unintended disclosure may violate legislation and depress service use.The attack formulation assumes a frozen victim model and an auxiliary dataset with a distribution similar to the attack data.Source: §3.1 · p. 3, §3.2 · p. 3

3.4 Limitations of Previous Approaches

  • Earlier black-box attacks train MLC or MSP to predict token sets, but set prediction loses order and repetition and may reverse sentence meaning.Even the exact set {Alice, likes, Bob} can yield the wrong meaning, “Bob likes Alice.”Source: §3.3 · p. 4, §3.4 · p. 4

3.5 Generative Embedding Inversion Attacks

  • GEIA treats the sentence embedding as the decoder’s initial token representation, then generates words using the embedding and prior context.A fully connected layer aligns embedding size with the attacker model’s token representation when needed; the victim model remains frozen.Source: §3.5 · p. 4

4.1 Experimental Settings

  • Experiments compare five frozen victim embedding models on PersonaChat and QNLI, with classification, token-informativeness, sentence-similarity, and fluency metrics.GEIA is trained as a GPT-2 attacker from random weights, and beam search is used for sentence recovery.Source: §4.1 · pp. 5–6

4.2 Evaluation on Classification

  • GEIA achieves the highest recall and F1 on five victim models across both datasets, except MSP has slightly higher F1 on SimCSE-RoBERTa for QNLI.On PersonaChat, GEIA’s F1 is around 0.58, compared with around 0.4 for MSP and an average no more than 0.3 for MLC.Source: §4.2 · p. 6

4.3 Precision-Recall Trade-off of MLC

  • MLC often has high precision but low recall, and raising its threshold sharply reduces precision for only a minor recall increase.Most precision-recall curves have small areas, which the authors interpret as evidence that MLC is not a good classification model.Source: §4.3 · p. 6

4.4 What Types of Tokens are Inverted?

  • GEIA recovers around 40% of named entities on PersonaChat and around 15% on QNLI, while MLC and MSP recover less than 3% in all situations.The authors conclude that GEIA recovers sensitive content while the previous baselines fail to capture informative entities.Source: §4.4 · p. 7

4.5 Evaluation on Generation

  • GEIA’s PersonaChat outputs have ROUGE-1 of 0.59–0.72 and BLEU-1 of 0.35–0.46 across five victim models, with higher embedding similarity and lower perplexity than baselines.The Figure 4 examples show GEIA recovering informative words, while baseline outputs are mostly stop words and punctuation.Source: Table 4 · p. 7, §4.5 · p. 8, §4.6 · p. 8

5 Conclusion

The paper proposes generative embedding inversion to recover original text sequences from sentence embeddings and reports that existing attacks have limitations. Its experiments disclose privacy risks, while defenses remain understudied.

5 Conclusion

  • GEIA is proposed to better recover original text sequences from sentence embeddings, addressing limitations of existing inversion attacks.Source: §5 · p. 8
  • Extensive experiments demonstrate the inability of previous inversion attacks and disclose privacy risks from the proposed attack.Source: §5 · p. 8
  • Defenses against embedding inversion attacks are not well studied, motivating a call for effective defenses with minor costs.Source: §5 · p. 9

Limitations

The attack has difficulty recovering exact domain-specific tokens, with lower success on QNLI than PersonaChat and repeated failure to recover the location “Fresno.”

Limitations

  • The attacker’s main limitation is recovering exact domain-specific tokens.PersonaChat conversations contain almost no expert knowledge, whereas QNLI includes Wikipedia question-answer pairs with more domain-specific named entities.Source: Limitations · p. 9
  • On average, PersonaChat F1 scores are 0.1∼0.2 larger than QNLI scores in Table 1.Across Tables 1, 3, and 10, attacks on PersonaChat are more successful than attacks on QNLI.Source: Limitations · p. 9
  • GEIA fails to recover the exact location “Fresno” 7 out of 10 times, producing a result similar to “What was the population of the city in 2010?”The example illustrates difficulty identifying the exact city name.Source: Limitations · p. 9

Ethical Considerations

The paper frames its work as exposing privacy vulnerabilities in sentence embeddings and calls for awareness and defenses. Its experiments use public datasets without personal identifiable information.

Ethical Considerations

  • The work substantially reveals potential privacy vulnerabilities of current LM-based sentence embedding models during inference.The authors hope to raise awareness of privacy leakage and call for defenses against it.Source: Ethical Considerations · p. 9
  • The experiments use publicly available PersonaChat and QNLI datasets, and report that no personal identifiable information is used or revealed.Anonymity was applied during data collection.Source: Ethical Considerations · p. 9
  • The experiments use sentence embedding models from their original GitHub repositories with given weights.The authors say they will test future open-source models with improved privacy protection.Source: Ethical Considerations · p. 9

A Training Details

The attacker is trained by conditioning a decoder on sentence embeddings represented as initial token inputs, then predicting the original sequence. The experiments specify attacker settings and use several pretrained victim embedding models.

A Training Details

  • GEIA uses randomly initialized GPT-2 medium (345M) with beam search size 5, Adam at learning rate 3e-4, and batch size 64.MLC uses a one-layer neural network; MSP uses a unidirectional GRU with 10 time steps, and all methods share a byte-pair-encoding tokenizer.Source: A Training Details · p. 12
  • GEIA projects victim sentence embeddings to the attacker token-representation dimension, concatenates them before token representations, and trains with a language-modeling objective.The sentence embedding therefore serves as the initial representation followed by representations of the original tokens.Source: A Training Details · p. 12
  • The input pairs the aligned sentence embedding with token representations, while the target sequence is the original tokens followed by <eos>.At the first step, the attacker predicts the first token from the sentence embedding alone; at the final step, it predicts <eos>.Source: A Training Details · p. 12
  • The experiments use Stanza for named-entity extraction, NLTK for BLEU, Huggingface Evaluate for ROUGE, and sklearn for micro-averaged precision, recall, and F1.The attacker was trained for 10 epochs on two NVIDIA GeForce RTX 3090 GPUs, taking around 7 hours.Source: A Training Details · p. 12

B Details of Victim Embedding Models

  • Victim models include Sentence-RoBERTa “all-roberta-largev1” (355M), two SimCSE variants (340M and 355M), and Sentence-T5 “sentence-t5-large” (770M).The selected Sentence-RoBERTa is described as outperforming SBERT; SimCSE is described as outperforming SBERT and SRoBERTa, and Sentence-T5 as achieving then-state-of-the-art sentence embedding performance.Source: B Details of Victim Embedding Models · p. 12
  • The experiments also use MPNet “all-mpnet-base-v1” (110M) as a victim embedding model.The subsection describes MPNet’s objective as combining masked and permuted language modeling.Source: B Details of Victim Embedding Models · p. 12

C Evaluations on Exact Match Ratio and Edit Distance

GEIA recovers some verbatim PersonaChat inputs but few exact QNLI sequences, with larger edit distances on QNLI. The evaluation uses exact-match ratio and edit distance to assess sequence recovery.

C Evaluations on Exact Match Ratio and Edit Distance

D Evaluation on Decoding Methods

Beam search generally outperforms Nucleus Sampling across generation and classification metrics on two datasets, supporting its use in the experiments.

D Evaluation on Decoding Methods

E Evaluations on Models’ Sizes

GEIA remains effective across embedding-model scales and different attacker models, although the reported effects of attacker capacity and pre-training vary.

E Evaluations on Models’ Sizes

  • ST5base tends to be most vulnerable to GEIA, while larger ST5 models are more robust; increasing attacker capacity generally increases victim-model vulnerability.The size evaluation uses ST5 models from 220M to 11B and GPT-2medium and GPT-2large attackers on PersonaChat.Source: E Evaluations on Models’ Sizes · pp. 13–14

F Evaluations on Different Attacker Models and Initializations

  • All evaluated attackers outperform the previous baselines on classification, but pre-training does not consistently improve GEIA performance.With beam search, pre-trained T5large-lm-adapt outperforms T5large-random, while GPT-2medium-random defeats GPT-2medium and GPT-2large.Source: F Evaluations on Different Attacker… · p. 14
  • Increasing GPT-2 and OPT attacker sizes helps improve GEIA, while pre-training helps attackers model token probabilities and improves sampling-based decoding.Source: F Evaluations on Different Attacker… · p. 14

G More on Case Studies

Across the illustrated cases, GEIA recovers relevant content and produces coherent sentences, whereas the earlier MLC and MSP attacks perform worse.

G More on Case Studies

  • On PC 2, GEIA correctly inverts all three hobbies.Source: G More on Case Studies · p. 14
  • GEIA works much better than MLC and MSP in the illustrated cases, with both decoding methods recovering relevant words and generating coherent sentences.MLC often returns meaningless stop words, while MSP recovers some informative words such as “US” and “population” on QNLI.Source: G More on Case Studies · p. 14

H GEIA on More Datasets

On five additional datasets, GEIA mostly outperforms MLC and MSP, but performance varies with data scale, domain, and content informativeness.

H GEIA on More Datasets

  • The evaluation applies GEIA with SimCSE-BERT embeddings to ABCD, MNLI, MultiWOZ, SST-2, and WMT16.Source: H GEIA on More Datasets · p. 14
  • GEIA mostly outperforms MLC and MSP on recall and F1 for classification and informativeness, with SST-2 as an exception.Source: H GEIA on More Datasets · p. 14, Table 8 · p. 15, Table 9 · p. 15
  • On SST-2, GEIA frequently generates meaningless stop-word repetitions; the authors attribute this to insufficient training data and poor generalization by randomly initialized GPT-2.The dataset has 67,349 sentences.Source: H GEIA on More Datasets · p. 15
  • Across seven datasets, inversion performance depends on data scale, domain, and content informativeness; with a reasonable amount of training data, GEIA can exceed previous baselines on classification and informativeness.GEIA also changes inversion from classification to generation and can recover ordered sequences.Source: H GEIA on More Datasets · p. 15

Related papers

Loading 2305.03010v1…