Privacy Implications of Retrieval-Based Language Models

1 Introduction

• The paper addresses limited evidence about how external datastores affect the privacy of retrieval-based language models.Source: §1 · p. 2
• The study examines kNN-LMs in public API deployments where a domain-specific private datastore may improve task performance but also contain sensitive information.Personal-use systems without an exploitable attack channel are outside the study’s scope.Source: §1 · p. 2
• Private inference-time datastores increase privacy risk over parametric language models, and adversaries can recover sensitive datapoints.The authors advise against storing sensitive information in the datastore.Source: §1 · p. 2
• The findings may not apply to retrieval-based architectures such as RETRO and Atlas, which differ substantially from kNN-LMs.Source: §1 · p. 2
• For targeted information, sanitizing the datastore and encoder training process eliminates privacy risks with minimal utility loss.Separate key and query encoders offer more flexibility in balancing privacy and performance.Source: §1 · p. 2
• For untargeted information, public-data datastore and encoder-training strategies yield modest improvements, but mitigation remains challenging.This information is impractical to identify and remove from the data.Source: §1 · p. 2

2.1 Nearest Neighbor Language Models

• A kNN-LM augments a standard language model with a datastore of precomputed tokens to improve performance.Source: §2.1 · p. 2
• Encoders map contexts to fixed-length vectors; kNN-LMs usually share an encoder for datastore keys and query prompts.Source: §2.1 · pp. 2–3
• The datastore stores each encoded context as a key and its ground-truth next word as the value, with an index enabling retrieval.Source: §2.1 · p. 3
• At inference, the encoded prompt retrieves its nearest datastore neighbors; their distance-weighted next-token distribution is interpolated with the language model’s prediction.Source: §2.1 · p. 3

2.2 Data Extraction Attacks

• Data-extraction attacks generate candidate reconstructions by prompting a model, then rank candidates by their likelihood of being memorized text.Source: §2.2 · p. 3
• Although extraction risks in parametric models are established, risks and potential benefits for retrieval-based models such as kNN-LMs remain poorly understood.Source: §2.2 · p. 3

2.3 Problem Formulation

• The deployment scenario is a provider serving a kNN-LM as an API using its domain-specific private data alongside public data.Source: §2.3 · p. 3
• Providers choose datastore composition and encoder training data to balance held-out private-data perplexity against disclosure to attackers.Source: §2.3 · p. 3

3.1 Privacy Measurements

• Targeted attacks count reconstructed private PIIs, while untargeted attacks assess reconstruction of entire training examples using ROUGE-L.A candidate is a good untargeted reconstruction when its ROUGE-L score against the closest private example exceeds 0.5.Source: §3.1 · p. 4

3.2 Evaluation Setup

• The evaluation uses Enron emails as Dprivate, WikiText-103 as Dpublic, and GPT-2 base as Encpublic before private fine-tuning.The study searches k from 64 to 2048 and λ from 0.1 to 0.5 for best validation perplexity.Source: §3.2 · p. 4
• A private datastore with Encpublic yields greater utility than Encprivate alone but increases leakage; Encprivate with Dprivate achieves the highest utility and privacy cost.Encprivate improves perplexity from 30.28 to 20.63 while increasing extraction risk.Source: §3.2 · pp. 4–5, Table 1 · p. 5

4.1 Sanization of Datastore and Encoders

• Datastore sanitization removes privacy-sensitive text segments using end-of-text tokens, dummy text, or randomly selected public phrases.The encoder can also be sanitized by fine-tuning Encpublic on a dataset with sensitive information removed.Source: §4.1 · p. 5

4.2 Decoupling Key and Query Encoders

• Separate encoders allow Encpublic to encode queries while Encprivate encodes keys, aiming to protect prompts while retaining useful datastore matches.The section presents encoder separation as a way to manage the privacy–utility trade-off for targeted risks.Source: §4.2 · pp. 5–6, §4 · p. 5

4.3 Experimental Results

• Sanitizing both encoder and datastore eliminates targeted PII extraction; public-information replacement gives perplexity 16.38 versus 16.12 for the non-sanitized private model.Among the three sanitization methods, replacing PII with random public information gives the highest utility.Source: §4.3 · p. 6
• A non-sanitized query encoder leaves risk high, whereas a sanitized query encoder with a non-sanitized key encoder eliminates risk while maintaining high utility.Separate key and query encoders can improve utility over using the same sanitized encoder for both.Source: §4.3 · p. 6, Table 2 · p. 6
• Increasing λ lowers perplexity but raises privacy risk, while increasing k improves perplexity and reduces risk.The k sweep uses Encprivate for both encoders and Dprivate as the datastore; Figure 2 plots perplexity and reconstructed-PII risk.Source: §4.3 · p. 6, Figure 2 · p. 6

5 Mitigations Against Untargeted Risks

• The methods are preliminary: opacity in the definition of privacy makes untargeted risks harder to mitigate, and substantial challenges remain.Source: §5 · p. 7

5.1 Methods

• The proposed approaches leverage public data because private datastore content is a primary source of leakage and sanitizing it is challenging for untargeted risks.Source: §5.1 · p. 7
• Adding public datapoints to the datastore may reduce privacy risk, while mixed public/private encoder fine-tuning is proposed to bridge the distribution gap and improve retrieval accuracy.The gap can arise because public datastore data, such as Web Crawl, differs from private encoder-training data, such as email conversations; the mixing ratio is determined empirically.Source: §5.1 · p. 7

5.2 Experimental Results

• A public datastore lowers privacy risk, but sharply reduces utility; adding a few private examples may suit settings with stricter utility needs and looser privacy constraints.Using a privately fine-tuned encoder with a public datastore also yields slightly lower risk than the standard language model with that encoder, attributed to interpolation with non-sensitive information.Source: §5.2 · p. 7
• Separate key and query encoders with a public datastore offer a better privacy-utility balance; private keys and public queries significantly reduce extraction risk with only a slight utility cost.Mixed public/private encoder fine-tuning reduces privacy risk and improves perplexity from 21.46 to 21.10, with the reported best perplexity at a public-data ratio of 0.05.Source: §5.2 · pp. 7–8

6.1 Retrieval-based Language Models

• Retrieval-based language models retrieve documents using an input query and incorporate them as additional information in the final prediction.Prior work reports their use for language modeling and open-domain question answering.Source: §6.1 · p. 8

6.2 Privacy Risks in Language Models

• Prior studies show that parametric language models can memorize training data and expose sensitive text, while proposed mitigations include differential privacy and anonymization.The passages cite Carlini et al. (2019), Thakkar et al. (2021), Zhang et al. (2021), Huang et al. (2022), Lehman et al. (2021), and Pan et al. (2020) on memorization or sensitive information.Source: §6.2 · p. 8
• This study is presented as the first investigation of privacy risks in retrieval-based language models; Arora et al. (2022) study information-retrieval privacy but not this model context.Source: §6.2 · p. 8

7 Conclusion

• Private datastores increase targeted and untargeted privacy risks in kNN-LMs compared with parametric models trained on private data.Source: §7 · p. 8
• For targeted attacks, sanitizing the datastore and encoders and decoupling key and query encoders eliminate privacy risks without sacrificing utility, with perplexity 16.38 vs. 16.12.Source: §7 · p. 8
• For untargeted attacks, a public datastore and mixed public/private encoder training reduce privacy risk while reducing utility by 24.1%, with perplexity 21.12 vs. 16.12.Source: §7 · p. 8

Limitations

• The findings are scoped to kNN-LMs; their applicability to RETRO and ATLAS remains an open research question.Source: Limitations · p. 9
• Combining the proposed approaches with differential privacy is suggested as future work toward better trade-offs for untargeted-risk mitigation.Source: Limitations · p. 9
• The paper calls for further study of privacy implications in other retrieval-based model variants, including RETRO and ATLAS.Source: Limitations · p. 9

A.1 Untargeted Attack

• The untargeted attack generates candidate reconstructions by prompting a trained model, then ranks candidates using membership metrics such as calibrated perplexity.The study uses calibrated perplexity, described as the most effective membership metric among those tested by Carlini et al. (2021).Source: §A.1 · p. 11
• For candidate generation, the retrieval-augmented LM receives contexts formed from randomly selected chunks of Common Crawl.The attack adapts the candidate-generation step to the retrieval-augmented LM’s sentence-completion API.Source: §A.1 · p. 11

A.2 Targeted Attack

• Because an untargeted attack can recover an entire sentence, the study also targets smaller sensitive segments that may identify a person.Targeted attacks customize prompts to the type of information sought.Source: §A.2 · p. 11

A.2.1 Definition of Personal Identifiers

• The study focuses on three PII types: email addresses, telephone numbers, and URLs.PII can include other identifying information, and its exact definition varies across jurisdictions and regulations.Source: §A.2.1 · p. 11

A.2.2 The Attack

• The targeted attack reconstructs PII values rather than the relationship between identifiers and their owners; public information can support a later linkage attack.The cited alternative threat model requires additional information about whether the owners are present in the dataset.Source: §A.2.2 · pp. 11–12

• Regular expressions identify telephone numbers, email addresses, and URLs in the Enron Email training dataset for targeted attacks.Table 6 reports counts for these identifier types.Source: B Experimental details · p. 12
• Targeted prompts use common preceding contexts for telephone numbers, email addresses, and URLs.Table 5 gives example prompts.Source: B Experimental details · p. 12
• The experiments generate 100,000 candidates for untargeted attacks and 10,000 for targeted attacks, using beam search with a repetition penalty of 0.75.Source: B Experimental details · p. 12