PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Embeddings Reveal (Almost) As Much As Text

John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, Alexander M. Rush

arXiv:2310.06816v1cs.CLcs.LG

TL;DR

The paper asks whether dense text embeddings allow private information, including full input text, to be reconstructed. It frames inversion as controlled generation and introduces iterative correction and re-embedding, recovering 92% of 32-token inputs exactly and exposing sensitive clinical information.

  • Problem

    The paper investigates whether text can be fully reconstructed from embeddings, posing a privacy threat when adversaries access embeddings and the embedding model.

  • Method

    Vec2Text iteratively refines a text hypothesis by re-embedding it and correcting it toward a fixed point near the target embedding.

  • Results

    92% of 32-token text inputs were recovered exactly, while the method also extracted full names and other critical information from clinical notes.

  • Takeaways & Limitations

    Embeddings can leak sensitive information similarly to raw text and should receive comparable privacy protections.

  • Takeaways & Limitations

    The method has been evaluated for exact recovery mainly up to 32 tokens, with only some information recovered up to 128 tokens; longer embeddings remain unexplored.

Abstract

from arXiv · show

How much private information do text embeddings reveal about the original text? We investigate the problem of embedding \textit{inversion}, reconstructing the full text represented in dense text embeddings. We frame the problem as controlled generation: generating text that, when reembedded, is close to a fixed point in latent space. We find that although a naïve model conditioned on the embedding performs poorly, a multi-step method that iteratively corrects and re-embeds text is able to recover $92\%$ of $32\text{-token}$ text inputs exactly. We train our model to decode text embeddings from two state-of-the-art embedding models, and also show that our model can recover important personal information (full names) from a dataset of clinical notes. Our code is available on Github: \href{https://github.com/jxmorris12/vec2text}{github.com/jxmorris12/vec2text}.

1 Introduction

The paper asks whether dense text embeddings can be inverted to recover private source text, and introduces Vec2Text, an iterative correction method that substantially reconstructs embedded text and sensitive information.

  • Earlier text-inversion work recovered only approximate bags of words from embeddings of shallow networks.The paper targets full text reconstruction rather than approximate lexical recovery.
  • A privacy risk arises when an attacker has vector-database access and text-embedding pairs from the embedding model.Such access could enable learning a function that reproduces text from embeddings.
  • Vec2Text frames inversion as controlled generation, iteratively updating text hypotheses using differences between hypothesis and target embeddings.The system generates hypotheses, re-embeds them, and applies discrete corrections toward the target embedding.
  • 92% of 32-token inputs are recovered exactly, with BLEU 97.3, using a state-of-the-art black-box encoder.The method is evaluated on web documents and recovers near-perfectly despite black-box access.
  • 89% of full names are recovered from embedded clinical notes in MIMIC.This demonstrates recovery of sensitive personal information from clinical-note embeddings.
  • The authors conclude that embeddings should receive protections comparable to raw text because they leak similar amounts of sensitive information.This conclusion is explicitly framed around anonymization and protection requirements for dense embeddings.

2 Overview: Embedding Inversion

Text embedding inversion seeks to recover an unknown text sequence from its embedding by finding text whose embedding is maximally similar to the target.

  • A text encoder maps a token sequence x ∈ V^n to a fixed-length embedding vector e ∈ R^d.Embeddings encode notions of semantic similarity and support search, clustering, and classification.
  • The inversion objective is to recover text x̂ whose embedding has maximal similarity to the ground-truth embedding under encoder ϕ.The paper formalizes recovery as optimization over candidate text sequences.
  • The threat model assumes black-box query access to ϕ, allowing an attacker to re-embed hypotheses and compute cosine similarity with the target.A cosine similarity of 1 is treated as identifying the original text because collisions are assumed rare.

3 Method: Vec2Text

Vec2Text improves text-embedding inversion by replacing direct generation with iterative self-correction: it re-embeds hypotheses, uses their discrepancy from the target, and searches among sequence-level corrections.

  • Base model: A learned conditional language model approximates the infeasible enumeration of all text sequences by modeling p(x | e; θ).The model amortizes the combinatorial inversion optimization into neural-network parameters.
  • Black-box setting: Unlike gradient-based controlled generation, textual inversion operates with queries to ϕ but without access to its gradients.This makes the method applicable to black-box embedding models.
  • Iterative correction: Vec2Text differs from direct inversion by iteratively refining a hypothesis after computing its new embedding ˆe(t).Each correction uses the target embedding, the current hypothesis, and its re-embedded representation.
  • Conditioning: The correction model receives the target embedding, hypothesis embedding, and their difference alongside the hypothesis token embeddings.Small MLPs project embedding vectors into sequences suitable for a transformer encoder.
  • Training: The encoder-decoder transformer is trained with standard language-modeling loss to generate corrected text hypotheses.Its backbone is conditioned on the previous output and embedding information.
  • Inference: Beam search retains the best unique sequence continuations by measuring their embedding-space distance to the target.Token decoding is greedy within each candidate, while sequence-level search considers b × b continuations.

4 Experimental Setup

Vec2Text is evaluated by inverting embeddings from two state-of-the-art models across in-domain and out-of-domain datasets, using iterative correction and standard reconstruction metrics.

  • Models and datasets: Vec2Text is trained to invert GTR-base and text-embeddings-ada-002 embeddings.Both embedding families are described as among the highest-performing models on the MTEB benchmark.
  • Models and datasets: The evaluation includes Natural Questions, MS MARCO, MIMIC-III clinical notes, and 15 BEIR datasets.Training uses Wikipedia passages for GTR-base and MSMARCO examples for the OpenAI models.
  • Baselines: The baseline predicts text from an embedding without correction steps.Additional comparisons include a bag-of-words model and reranked unconditional generations.
  • Metrics: Reconstruction is measured with BLEU, Token F1, and exact-match, alongside embedding similarity.These metrics compare reconstructed text with the ground truth and assess similarity in embedding space.
  • Inference: Inference uses greedy token-level decoding, while iterative sequence-level beam search accepts generations closer to the ground-truth embedding.Models are initialized from T5-base and use a projection sequence length of 16.
  • Evaluation design: The study also examines alternative initial hypotheses and reports in-domain and out-of-domain reconstruction tables.Table 1 covers in-domain datasets, while Table 2 covers BEIR out-of-domain performance.

5 Results

Vec2Text substantially reconstructs text both in-domain and across BEIR datasets, and it recovers substantial personal information from embedded clinical notes.

  • 5.1 Reconstruction: In-Domain: Vec2Text outperforms baselines on all in-domain metrics, with more correction rounds helping but producing diminishing returns.It recovers 77% of BLEU in five correction rounds, while sequence-level beam search increases exact-match scores by 2 to 6 times.
  • 5.2 Reconstruction: Out-of-Domain: 66% of Quora examples are recovered exactly, while all BEIR datasets achieve Token F1 of at least 41 and cosine similarity of at least 0.95.Reconstructions generally have average length error below three tokens, and accuracy tends to decrease as input length increases.
  • 5.3 Case study: MIMIC: 89% of clinical-note full names are recovered, alongside 94% of first names and 95% of last names.The notes are drawn from a pseudo-reidentified MIMIC-III release containing inserted fake names.
  • 5.3 Case study: MIMIC: 26% of 32-token clinical notes are reconstructed exactly, while other outputs remain semantically close and generally preserve syntax.The examples show slightly garbled entities and reordered or altered medical phrasing.
  • 5.3 Case study: MIMIC: Clinical Event entities are reconstructed most accurately, whereas Detailed Description entities are reconstructed least accurately.Detailed Description includes specific terminology and multi-word medical events.

6 Defending against inversion attacks

The defense study adds Gaussian noise to embeddings and evaluates the trade-off between retrieval utility and reconstruction accuracy across noise levels.

  • Defense mechanism: Gaussian noise is added directly to each embedding to defend against inversion while preserving nearest-neighbor retrieval utility.The noise amount is controlled by λ, and the study measures retrieval and reconstruction under varying levels.
  • Results: Figure 2 compares retrieval performance and reconstruction accuracy as Gaussian noise increases.The complete results are reported across BEIR tasks in Appendix A.2.
  • Evaluation: Mean NDCG@10 across 15 BEIR retrieval tasks measures retrieval performance in the defense experiment.The simulation uses GTR-base and a self-corrective model with 10 correction steps.
  • Results: At λ = 10^-1, retrieval performance is preserved while BLEU drops by 10%.This level provides the reported utility–reconstruction trade-off in the experiment.
  • Results: At noise level 0.01, retrieval performance is degraded by 2% while reconstruction falls to 13% of the original BLEU.Additional noise severely impacts both retrieval performance and reconstruction accuracy.
  • Feedback ablation: Figure 3 compares self-correction with and without reembedding the latest hypothesis, showing that embedding feedback improves iterative recovery.Without feedback, the model can still edit text but quickly plateaus.

7 Analysis

The analysis shows that iterative embedding feedback substantially improves reconstruction, while embedding similarity tracks textual similarity and poor initializations can be overcome.

  • Feedback ablation: 52.0% of examples were exact matches after 50 feedback-guided correction rounds, versus 4.2% without feedback.The feedback model began at 1.5% exact matches, while the no-feedback model quickly plateaued.
  • Iterative correction: Training covers hypotheses across a wide range of embedding similarities, supporting repeated corrections toward the ground-truth embedding.At test time, the model can push text from about 0.9 embedding similarity toward 1.0 without leaving the training range.
  • Embedding geometry: Cosine similarity and BLEU score are strongly correlated on reconstructed Natural Questions inputs.Few samples have high cosine similarity but low BLEU, suggesting that closer adherence to embedding geometry is associated with better textual recovery.
  • Initialization: The mean GTR-base training output has cosine similarity 0.924 with the true embedding.Figure 4 reports the distribution of cos(e, ϕ(x(0))) over the training data.
  • Initialization: After 20 correction steps, the model can recover from unhelpful initializations, including random token sequences.This indicates that the model can focus on the target embedding when the initial hypothesis is not useful.

8 Related work

The paper situates textual embedding inversion alongside visual representation inversion, privacy-leakage studies, gradient leakage, and text autoencoders, while emphasizing iterative reconstruction from frozen encoders.

  • Deep representation inversion: Visual inversion research reconstructs input images from final-layer logits, motivating analogous investigation of information retained in textual embeddings.The related work includes visual detail recovery from deep representations and facial reconstruction from embeddings.
  • Text embedding inversion: Prior text-inversion work recovered approximate bags of words from shallow-network embeddings, whereas this work targets full text reconstruction.The paper also distinguishes its paragraph-length setting from approaches focused on sentence-length queries.
  • Privacy leakage: Other text studies analyze token distributions, decode queries in one step, or recover sensitive information without exact-text reconstruction.These approaches differ from the paper’s direct focus on full reconstruction and privacy implications.
  • Gradient leakage: Gradient leakage differs because gradients are relatively high-resolution, while textual inversion uses only a single dense embedding vector.The paper frames single-vector recovery as the more difficult problem.
  • Text autoencoders: Text autoencoders and retrieval decoders map encoder outputs back toward text, but this work inverts embeddings from a frozen, pre-trained encoder.The distinction is between jointly learned reconstruction settings and inversion of an already trained encoder.

9 Conclusion

Vec2Text iteratively corrects and re-embeds text toward a fixed point in latent space, recovering much of the original text and exposing sensitive information in clinical notes.

  • Method: Vec2Text iteratively corrects and re-embeds text based on a fixed point in latent space.This is the paper’s proposed multi-step reconstruction method.
  • Reconstruction: 92% of 32-token text inputs are recovered exactly from their embeddings.The result demonstrates that text embeddings reveal much of the original text.
  • Privacy implications: The model extracts critical clinical information from clinical notes, highlighting privacy implications in medicine.The conclusion connects reconstruction capability to sensitive-domain data protection.
  • Privacy implications: Embeddings and raw data are described as leaking similar amounts of sensitive information.The authors therefore characterize dense embeddings as highly sensitive private data.

10 Limitations

The paper’s limitations concern non-adaptive attack settings, limited search, untested long-text scalability, and reliance on black-box access to the original embedder.

  • Adaptive attacks and defenses: The evaluated defense setting adds noise to embeddings, but reconstruction modules were trained on un-noised embeddings.Adaptive attacks and defenses remain a future-work direction.
  • Search thoroughness: Search is limited to 50 rounds and sequence beam width 8, although reconstruction improves monotonically with more searching.More rounds, wider beams, or more sophisticated search algorithms could find additional exact matches.
  • Scalability to long text: Most exact recovery is demonstrated only up to 32 tokens, with some information recovered at 128 tokens.The limits of inversion for the thousands-of-tokens inputs supported by popular embedding models remain untested.
  • Access to embedding model: The threat model assumes black-box access to the embedding model and requires querying it at every refinement step.An imitation embedder could potentially reduce queries to the true model.

A Appendix

The appendix examines how word frequency relates to prediction correctness and reports retrieval and reconstruction results under Gaussian noise for 32-token inputs. Correctness generally improves with training-data frequency, while some unseen words remain recoverable.

  • Word frequency: More frequent training-data words are generally predicted more correctly, although some words unseen during training are also recovered.The authors hypothesize this occurs because test tokens were seen during training, enabling reconstruction of unseen words from seen tokens.
  • Word frequency: Figure 6 plots correct and incorrect ground-truth word predictions against word frequency in the training data.Orange indicates correct predictions and blue indicates incorrect predictions.
  • Word frequency: Peaks between 10^4 and 10^5 in the frequency plot come from the frequently appearing characters (, −, and ).
  • Noise evaluation: BEIR results report GTR-base retrieval performance under varying Gaussian-noise levels, with inputs limited to 32 tokens.Because some corpora are longer on average, baseline (λ = 0) NDCG@10 values are lower than typically reported.
  • Noise evaluation: The appendix presents retrieval and reconstruction performance across varying noise levels λ.
Loading 2310.06816v1…