PaperlayerView exact revision on arXiv

Source-linked AI summary

Shadow Queries for Private Retrieval in Vector Databases

Xinguo Feng, Zhongkui Ma, Zihan Wang, Chuan Yan, Guowei Yang, Alsharif Abuadbba, Guangdong Bai

arXiv:2609.04767v1cs.AI

TL;DR

Embedding inversion attacks threaten private text stored as vector embeddings, while existing noise and scaling defenses offer limited privacy-utility balance. SHAQ generates diverse shadow queries, stores their embeddings instead of document embeddings, and reports stronger privacy with preserved retrieval utility across IR benchmarks.

  • Problem

    Embedding inversion attacks can reconstruct private document text from stored embeddings, while existing defenses struggle to balance privacy and retrieval utility.

  • Method

    SHAQ uses a generative language model to create diverse shadow queries, encodes them, and stores their embeddings in place of original document embeddings.

  • Results

    SHAQ consistently outperforms existing countermeasures, reaching a recovery rate as low as 0.2104 and reducing recovered tokens by up to 19.50% while maintaining retrieval performance near undefended systems.

  • Takeaways & Limitations

    Semantic decomposition and embedding decoupling provide an effective alternative to directly modifying embeddings for defending against embedding inversion attacks.

Abstract

from arXiv · show

Large language models (LLMs) increasingly rely on information retrieval (IR) systems, such as Retrieval-Augmented Generation (RAG), to incorporate domain-specific knowledge without costly re-training. These systems often store pre-computed document embeddings in cloud-based vector databases. However, such embeddings are vulnerable to embedding inversion attacks (EIAs), which can reconstruct their underlying text. Existing defenses, such as adding noise or scaling embeddings, often provide limited privacy or significantly reduce retrieval utility. We propose SHAQ (shadow query generation), a semantic-decomposition and embedding-decoupling defense against EIAs. SHAQ is based on the insight that EIAs rely on the strong coupling between an embedding and its original text. Instead of storing document embeddings directly, SHAQ uses a generative language model to create diverse shadow queries that capture different semantic aspects of each document. These queries are then encoded and stored in place of the original document embeddings, thereby decomposing document semantics and decoupling stored embeddings from the source text. Experiments across diverse IR datasets show that SHAQ substantially improves privacy while preserving retrieval utility, achieving a recovery rate as low as 0.2104, defending up to 19.50% more tokens than baseline defenses, and reaching up to 0.7967 MAP@10 with up to 5.53% utility improvement. These results demonstrate that semantic decomposition and embedding decoupling provide an effective alternative to directly modifying embeddings for defending against EIAs.

I. INTRODUCTION

SHAQ addresses embedding inversion attacks by replacing directly stored document embeddings with embeddings of diverse shadow queries, aiming to decouple stored representations from source text while preserving retrieval utility.

  • Embedding inversion attacks exploit semantic coupling between stored embeddings and underlying text to reconstruct private documents.
  • Existing noise and scaling defenses struggle to balance privacy with retrieval utility, with scaling removable through vector normalization.
  • SHAQ generates shadow queries covering diverse document facets and stores their embeddings instead of original document embeddings.
  • 0.2104 is SHAQ’s lowest reported recovery rate, measured with ROUGE-1 under EIA evaluation.
  • 19.50% fewer recovered tokens and NDCG up to 0.8524 show improved defense while maintaining retrieval performance near an undefended system.
  • SHAQ is presented as generic and model-agnostic, supporting variable-length documents and different embedding models through an offline pipeline.

II. BACKGROUND

The background motivates dense retrieval for semantic matching and frames embedding inversion attacks as a privacy risk requiring defenses that preserve retrieval effectiveness.

  • Sparse retrievers rely on lexical overlap, whereas embedding-based dense retrievers represent semantic similarity between queries and documents.
  • Embedding inversion attacks reconstruct original text by exploiting connections between embeddings and encoded content.
  • Noise injection can impair retrieval when perturbations are increased, while secret scaling can be bypassed through normalization.
  • The threat model places pre-computed document embeddings in a third-party cloud database accessible to an honest-but-curious adversary.

B. Overview

SHAQ decomposes each document into diverse shadow queries and indexes their embeddings, replacing direct document embeddings while retaining mappings for retrieval.

  • SHAQ generates plausible user queries that cover different semantic facets of each target document.
  • A crafted prompt and K-Means clustering guide generation toward diverse shadow-query outputs.
  • The generated shadow queries are encoded into embeddings that replace document embeddings in the vector database.
  • Randomized indexing and a mapping from shadow queries to documents support retrieval after embedding decoupling.

D. Embedding Indexing

The indexing stage converts selected shadow queries into searchable embeddings, records their document associations, and builds a shuffled vector database for retrieval.

  • The indexing procedure takes an embedding model, shadow queries, documents, and a selection count, producing mappings and a shadow-query embedding database.
  • For each selected shadow query, SHAQ computes an embedding, maps it to the source document ID, and appends it to the vector database.
  • Replacing document embeddings means an inversion attack no longer directly reconstructs the original document from an observed stored embedding.
  • The evaluation includes retrieval utility, defense efficacy, and ablations involving variable-length documents and another embedding model.

A. Experimental Settings

The experiments evaluate SHAQ against state-of-the-art embedding inversion attacks and defenses using dense retrieval across diverse BEIR datasets. The setup standardizes datasets, attack models, document lengths, and sampled queries for comparable evaluation.

  • Attack and defenses: The evaluation uses the state-of-the-art vec2text attack against document embeddings in modern dense retrievers.The attack models are trained on Natural Questions or MS MARCO embedding-text pairs and focus on GTR-T5 embeddings.
  • Dense retrieval: The retrieval task ranks corpus documents by cosine similarity or dot product between query and stored document embeddings.The top-k similar documents are returned as relevant candidates.
  • Datasets: The study evaluates SHAQ on diverse BEIR datasets spanning scientific, financial, and open-domain question-answering domains.Development or test splits are used to reduce potential information leakage from attack-model training.
  • Datasets: The BEIR evaluation excludes datasets with non-binary relevance labels or excessively many relevant documents per query.The study randomly samples 100 queries from each evaluated dataset.
  • Preprocessing: All sampled documents are truncated to 32 tokens to match the fixed output length of the vec2text attack model.This preprocessing occurs before shadow-query generation.

3) Evaluation Metrics:

The evaluation measures retrieval utility, defense efficacy, and the mechanisms underlying SHAQ’s semantic decomposition and embedding decoupling. It also specifies the SHAQ generation configuration and compares utility across no defense, embedding noise, and SHAQ.

  • Evaluation metrics: Retrieval utility is measured with NDCG, MAP, Recall, Precision, and Accuracy using top-k evaluation with k = 10.All utility metrics range from 0 to 1, with higher values indicating better retrieval performance.
  • SHAQ settings: SHAQ uses a generative language model and K-Means clustering to produce diverse shadow queries covering different document facets.The generated queries are encoded for indexing in place of document embeddings.
  • Semantic decomposition: The proof-of-concept evaluates semantic decomposition through embedding-level similarity and lexical-level token coverage.These measures assess how much document content the collection of shadow queries captures.
  • Retrieval utility: The retrieval utility comparison covers no defense, embedding noise, and SHAQ using the metrics reported in Table III.The table’s annotations distinguish utility changes from the undefended system and the better defense result.
  • Defense efficacy: The defense-efficacy comparison reports privacy-risk metrics in Table IV, where lower values indicate better defense efficacy.Its annotations also mark changes relative to no defense and the stronger defense between embedding noise and SHAQ.
  • Semantic decomposition: The decomposition analysis finds that shadow queries capture most of the document’s semantic content.This result supports the semantic decomposition property of SHAQ.

2) Embedding Decoupling:

SHAQ preserves retrieval utility by retrieving through shadow-query embeddings mapped back to original documents. Across diverse datasets, it achieves utility broadly comparable to undefended retrieval while reducing embedding inversion risk.

  • Retrieval procedure: SHAQ retrieves documents by ranking shadow-query embeddings and mapping the top-10 results back to original document IDs.Successful retrieval occurs when the mapped documents are relevant to the query.
  • Utility results: SHAQ’s utility variation from the undefended system ranges from −0.0747 to +0.0382 for NDCG and −0.0778 to +0.0390 for MAP.The corresponding ranges are −0.1100 to +0.0640 for Recall, −0.0110 to +0.0110 for Precision, and −0.1100 to +0.0600 for Accuracy.
  • Utility results: SHAQ maintains competitive retrieval effectiveness across diverse domains, generally remaining on par with undefended retrieval or embedding noise.The reported conclusion is that SHAQ preserves retrieval utility despite replacing document embeddings.
  • Defense evaluation: The defense-efficacy evaluation compares reconstructed text with its original document using 100 indexed embeddings per defense setting.For SHAQ, the sampled attack targets are shadow-query embeddings linked to corresponding documents.
  • Overall result: SHAQ offers a strong privacy-utility trade-off by reducing embedding-inversion risk while maintaining retrieval performance comparable to existing baselines.The paper characterizes this as a practical defense for embeddings in vector databases.
  • Ablations: SHAQ maintains robust retrieval utility for untruncated documents and under an alternative embedding similarity measure.These ablations support generality across variable-length documents and model-agnostic similarity choices.

F. Adaptive Attack

The adaptive-attack evaluation tests whether SHAQ remains effective when attackers know the defense, its parameters, and the generation model. The broader contribution replaces direct embedding modification with semantic decomposition and embedding decoupling.

  • Adaptive attack: Under an adaptive attack with extensive knowledge of SHAQ, attack efficacy is limited to as low as 0.2083 in R-1.The attacker attempts to recover all shadow queries and then reconstruct the original document.
  • Embedding inversion attacks: Embedding inversion attacks reconstruct original text from observed embeddings and therefore pose a data-privacy threat to embedding-based retrieval systems.The attacks may use optimization or generative language models conditioned on embeddings.
  • Existing defenses: Existing defenses modify document embeddings with noise or secret scaling, but noise is constrained by utility and scaling can be removed through vector normalization.SHAQ instead replaces document embeddings with shadow-query embeddings.
  • Related generation methods: Unlike utility-focused query-generation methods such as doc2query and query2doc, SHAQ specifically targets document privacy in dense retrievers.Its query-generation pipeline is designed for privacy rather than retrieval improvement or training-data augmentation.
  • Conclusion: SHAQ breaks the embedding-text connection while preserving document semantic facets in generated shadow queries.This combines embedding decoupling for privacy with semantic decomposition for retrieval.

APPENDIX

The appendix specifies the shadow-query prompt, illustrates semantic coverage, and lists the diverse IR datasets used in evaluation.

  • The prompt asks for concise, natural queries that reflect different perspectives and avoid semantic repetition.
  • Generated shadow queries cover much of the target document through compact facts or plausible retrieval-time questions.
  • The example uses queries spanning TSPAN7 expression, tumor burden, bone-marrow homing, adhesion, migration, prognosis, and proliferation in multiple myeloma.
  • The evaluation includes scientific, open-domain, web, biomedical, financial, argument, and paraphrase retrieval datasets.

E. Excluded Datasets from Evaluation

Three datasets are excluded from the BEIR-based evaluation, with exclusions summarized by dataset statistics and stated reasons.

  • The listed exclusion reasons are many relevant documents per query, non-binary relevance scores, and failure to meet the standard IR-dataset definition.
  • Table V summarizes data splits and statistical details for the excluded datasets.

F. Hyperparameter Search

The hyperparameter search selects 10 final shadow queries per document and compares this setting with tuned embedding-noise levels, while documenting evaluation scope limitations.

  • Shadow-query count: nk = 10 yields the best utility–privacy trade-off among the tested shadow-query counts.
  • Noise level: σ = 0.01 provides utility comparable to SHAQ with nk = 10, whereas larger noise values further degrade retrieval utility.
  • Evaluation scope: The main evaluation truncates documents to 32 tokens because the vec2text attack model reconstructs fixed-length 32-token texts.
  • Embedding-model ablation: On the DPR embedding model, all three scenarios lose retrieval performance relative to GTR-T5, but remain comparable to one another.

J. Adaptive attack

The adaptive attack assumes complete knowledge of SHAQ and attempts to regroup shadow-query embeddings before reconstructing documents, yet recovery remains low.

  • Threat model: The adaptive attacker knows SHAQ, its shadow-query count, the generation LLM, and the defense mechanism’s relevant details.
  • Attack procedure: The attacker locates the other nk − 1 shadow-query embeddings using cosine similarity, then prompts an LLM to reconstruct the document.
  • Results: Adaptive-attack recovery remains low, with BLEU ranging from 0.0151 to 0.1446 and R-1 reported as low in the evaluation.
  • Results: SHAQ withstands the challenging adaptive attack despite the attacker possessing complete knowledge of the applied defense.
Loading 2609.04767v1…