Privacy-Preserving In-Context Learning for Large Language Models

1 INTRODUCTION

• A malicious user can craft prompts that bypass system instructions and extract sensitive information contained in in-context exemplars.The passage gives clinical records as an example of sensitive data at risk.Source: §1 · p. 1, Figure 1 · p. 1
• DP-ICL partitions exemplars into disjoint subsets, prompts an LLM with each subset and the query, then privately aggregates the responses.The approach uses parallel inference and releases an aggregate that does not overly rely on any single exemplar.Source: §1 · pp. 1–2, Figure 2 · p. 2
• For classification, DP-ICL privately releases the majority-vote class; for generation, it uses embedding-space or keyword-space aggregation.ESA privatizes aggregated semantic embeddings, while KSA privately selects frequent keywords.Source: §1 · p. 2
• 95.80% accuracy at ε = 3 on SST-2 matches all non-private baselines; document summarization's average ROUGE scores degrade approximately 1% at ε = 1.The evaluation covers four text-classification datasets, DocVQA, and SAMsum.Source: §1 · p. 2

2 BACKGROUND: PRIVACY RISKS OF IN-CONTEXT LEARNING

• In-context learning appends k query-answer exemplars to a query, then uses the combined prompt to generate an answer.The paper says exemplars help the LLM identify the relevant input-output mapping.Source: §2 · p. 2
• Prompt-leaking attacks can extract proprietary prompts, while other studies examine secret-information leakage and membership inference targeting ICL.Membership inference can potentially reveal whether a particular record was part of training data.Source: §2 · p. 3
• Differential privacy bounds output-probability differences for datasets differing in one data point; for ICL, the paper applies this to changes in private exemplars.The stated purpose is to make outputs similar across such changes, limiting reproduction of private exemplars.Source: §2 · p. 3

3 DIFFERENTIALLY PRIVATE IN-CONTEXT LEARNING

• DP-ICL partitions private exemplars into disjoint subsets, pairs each with the query, obtains LLM answers, and privately aggregates those answers; Poisson subsampling can reduce costs and amplify privacy.Subsampling independently includes each exemplar with probability q.Source: §3 · p. 3

3.1 PRIVATE AGGREGATION FOR TEXT CLASSIFICATION

• RNM-Gaussian adds independent Gaussian noise to class-vote counts and releases the class with the highest noisy count.The method aggregates one-shot predictions from each exemplar-query pair into a voting histogram.Source: §3.1 · p. 4

3.2 PRIVATE AGGREGATION FOR LANGUAGE GENERATION

• Autoregressive generation creates an enormous, sparse sentence-voting space, while token-wise voting composes privacy loss; DP-ICL therefore targets lower-dimensional semantic representations.The design maps generated outputs into representations that preserve relative semantic meaning.Source: §3.2 · pp. 4–5

3.2.1 EMBEDDING SPACE AGGREGATION (ESA)

• ESA embeds generated sentences, privately estimates their mean, then selects a zero-shot candidate nearest that privatized mean; its performance depends on embedding-to-text mapping quality.The embedding-to-text mapping remains an active research direction.Source: §3.2.1 · pp. 5–6

3.2.2 KEYWORD SPACE AGGREGATION (KSA)

• KSA privately selects frequent keywords—using joint EM for moderately large domains or PTR for very large or infinite ones—then re-prompts the LLM to reconstruct a sentence.Joint EM releases the ranking of token counts, while PTR does not.Source: §3.2.2 · p. 6

4.1 DP-ICL FOR TEXT CLASSIFICATION

• A 0.04% accuracy drop on SST-2 at ε = 3 with GPT-3 Babbage accompanies comparable performance to non-private ICL across classification tasks; at ε = 1, AGNews can exceed zero-shot by over 20%.The study also reports that GPT-3 Davinci exceeds Babbage by ∼3.1% on SST-2 at ε = 1.Source: §4.1 · pp. 6–7, Table 2 · p. 7

4.2 DP-ICL FOR DOCUMENT QUESTIONS ANSWERING

• At ε = 1 on DocVQA, ESA's ROUGE-1 decrease from non-private aggregation is 0.19%, while KSA by PTR declines by less than 3%.KSA significantly surpasses ESA and even standard 1-shot ICL; the authors attribute the ESA drop mainly to embedding information loss and low-quality zero-shot candidates.Source: Table 3 · p. 7, §4.2 · pp. 7–8

4.3 DP-ICL FOR DIALOG SUMMARIZATION

• At ε = 1 on SAMSum, keyword by joint EM improves over zero-shot by approximately 4.5% ROUGE-1, 3.0% ROUGE-2, and 3.4% ROUGE-L, averaging ∼1% below non-private aggregation.Joint EM outperforms keyword by PTR, which the authors link primarily to releasing frequency order; non-private aggregation methods perform worse than 4-shot ICL predictions.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 ABLATION STUDIES

• As ensembles grow from 10 to 100, ESA and keyword by PTR improve, while keyword by joint EM fluctuates and reaches high performance with 30 ensembles.More ensembles raise subsampling rates and aggregation noise, but may also yield a more reliable consensus.Source: §4.4 · p. 9
• KSA by joint EM shows only marginal performance degradation up to 1,000 queries, despite increasing query counts raising noise and strengthening privacy protection.The tested query counts are 1, 10, 100, 1,000, and 10,000; the authors identify the reported robustness up to 1,000 as evidence of suitability for real-world applications.Source: §4.4 · p. 9

Differentially Private Language Models

• Prior differentially private language-model research primarily improves DP-SGD training, adding noise to gradients; scaling LLMs makes fine-tuning more challenging and this route less practical (Li et al., 2022; Yu et al., 2022; Bu et al., 2023; He et al., 2023; Abadi et al., 2016).The paper compares DP-ICL and DP-SGD in Appendix C.Source: §5 · p. 9

Concurrent works on Differentially Private In-Context Learning

• Duan et al. (2023a) privately labels public data for demonstrations, whereas Tang et al. (2023) privately generates exemplars through prompting; neither approach relies on the same data setup as DP-ICL.The paper states that its own approach does not rely on public data.Source: §5 · p. 9
• The paper reports competitive DP-ICL performance on SAMSum and DocVQA, tasks it contrasts with prior concurrent approaches limited to small label-space tasks such as classification and word extraction.The authors characterize SAMSum and DocVQA as more complex and challenging language-generation tasks.Source: §5 · p. 9

6 DISCUSSION AND FUTURE WORKS

• DP-ICL uses a unified sample-and-aggregate framework, with private aggregation instantiations for text classification and language generation.The framework partitions private exemplars into disjoint subsets and aggregates the resulting model responses.Source: §6 · p. 9

A.1 TEXT CLASSIFICATION VIA RNM-GAUSSIAN

• RNM-Gaussian classifies by adding Gaussian noise to the histogram of class votes and reporting its top bin.The algorithm partitions the private data, prompts the model for each subset, and converts outputs to one-hot class vectors.Source: §A.1 · p. 14

A.2 LANGUAGE GENERATION VIA EMBEDDING SPACE AGGREGATION (ESA)

• ESA maps generated sentences into embedding space, privatizes their mean, then returns the public candidate sentence with maximum cosine similarity.Its algorithm first obtains responses from disjoint exemplar subsets and projects those responses into embeddings.Source: §A.2 · p. 14

A.3 LANGUAGE GENERATION VIA KEYWORD SPACE AGGREGATION (KSA)

• KSA aggregates generated text by counting how often tokens appear across the subset responses, then privately selects keywords.The algorithm offers JointEM and PTR for selection.Source: §A.3 · p. 14, §A.3.1 · p. 15

A.3.1 KSA VIA JOINT EXPONENTIAL MECHANISM.

• KSA-JEM initializes a structure from a dialogue-word histogram, but directly histogramming all exemplar outputs would violate privacy.The histogram supports efficient sampling with the joint exponential mechanism.Source: §A.3.1 · pp. 14, 16

A.3.2 KSA VIA PROPOSE-TEST-RELEASE (KSA-PTR)

• KSA-PTR releases exact top-k tokens when the count gap is sufficiently large, and privately tests whether the gap exceeds 2.It can choose k privately by selecting the gap-maximizing k with the exponential mechanism.Source: §A.3.2 · p. 16

B PRIVACY ANALYSIS

• Differential privacy bounds how much an algorithm’s output can change between neighboring datasets, and its guarantees persist under post-processing.Privacy costs from multiple adaptively composed mechanisms accumulate; the basic composition theorem adds their ε and δ values.Source: B PRIVACY ANALYSIS · p. 17

B.1 TEXT CLASSIFICATION VIA RNM-GAUSSIAN

• RNM-Gaussian is (ε, δ)-differentially private with σ = 2√(2 log(1.25/δ))/ε.Its proof uses the sensitivity of the voting histogram and the Gaussian mechanism.Source: §B.1 · p. 17

B.2 EMBEDDING SPACE AGGREGATION (ESA)

• ESA’s embedding aggregation steps are (ε, δ)-DP under the stated Gaussian-noise setting; selecting the public candidate by cosine similarity is post-processing.The proof relies on unit-norm text embeddings and ℓ2 sensitivity 1.Source: §B.2 · p. 18

B.3 LANGUAGE GENERATION VIA KEYWORD SPACE AGGREGATION (KSA)

• For KSA-JEM and KSA-PTR, the analysis uses RDP or approximate RDP to track privacy costs under composition, then converts the guarantee to standard DP.The exponential mechanism is ε-DP; privately choosing k uses the histogram-gap utility with sensitivity 2.Source: §B.3 · pp. 18–19, §B.3.1 · p. 19

B.3.2 KSA VIA PROPOSE-TEST-RELEASE (KSA-PTR)

• KSA-PTR is δ-approximate α/(2σ²)-RDP; when the count gap exceeds 2, releasing the exact top-k tokens has no privacy cost.The algorithm otherwise terminates when its noisy threshold test does not pass.Source: §B.3.2 · p. 19

B.3.3 PRIVACY AMPLIFICATION BY SUBSAMPLING FOR APPROXIMATE RDP

• Poisson subsampling converts a δ-approximate RDP guarantee into a δq-approximate guarantee with an amplified RDP bound.The theorem specifies the tightest possible amplification bound for mechanisms with the stated RDP guarantee and sampling rate q.Source: §B.3.3 · p. 19

C DP-ICL ENABLES PRIVATE PREDICTION

• DP-ICL takes a private-prediction approach, avoiding DP-SGD’s hyperparameter-tuning and engineering burdens while remaining compatible with black-box LLM APIs.The paper also says removing an individual’s data from the exemplar database enables the right to be forgotten.Source: C DP-ICL ENABLES PRIVATE PREDICTION · p. 21
• 1.2% is the reported SST-2 enhancement at ε = 3 over earlier DP-SGD methods, corresponding to over 20% lower relative error rate.The table caption reports significantly lower performance degradation for DP-ICL at ε = 3 and 8.Source: Table 5 · p. 21, C DP-ICL ENABLES PRIVATE PREDICTION · p. 21
• DP-ICL in the Table 5 comparison answers at most 10,000 queries, whereas DP-SGD has no such query limit.The paper notes a sentence-level versus phrase-level training-data discrepancy in the SST-2 comparison.Source: C DP-ICL ENABLES PRIVATE PREDICTION · p. 21

D.1 TEXT CLASSIFICATION

• For text classification, experiments use GPT-3 label prompts with logit bias 100 and temperature 0.0 across four datasets, with δ = 1e−5.The dataset sizes are each below 10,000 training examples.Source: §D.1 · p. 23

D.2.1 DOCUMENT QUESTIONS ANSWERING

• Document question answering assumes accurate OCR tokens and evaluates OpenLLaMA-13B on 100 randomly selected validation examples.The training data contains 221,329 examples; the task uses ROUGE-1, BLEU, and Levenshtein similarity.Source: §D.2.1 · pp. 23–24

D.2.2 DIALOG SUMMARIZATION Task.

• Dialogue summarization uses SAMsum with 14,732 training examples and 100 test examples, evaluates ROUGE-1, ROUGE-2, and ROUGE-L, and uses GPT-3 Davinci.The prompt is modified to request word suggestions for the two keyword-space aggregation methods.Source: §D.2.2 · pp. 24–25

E.1 EFFECTIVENESS OVER VARIOUS QUERIES

• With ε = 3 and 10 ensembles, query-count increases cause less than 2% performance degradation through 10,000 queries; drops are larger on AGNews and TREC.The authors hypothesize that the larger drops relate to those datasets’ four and six classes, respectively.Source: §E.1 · p. 26, Figure 6 · p. 26

E.2 EFFECTIVENESS OF VARIOUS MODELS

• DP-ICL accuracy on SST-2 rose from 73.31% with Ada to 95.11% with Davinci at ε = 1, while zero-shot accuracy rose from 86.24% to 94.15%.Across all datasets, performance improved as the evaluated GPT-3 models became more advanced; Table 8 reports text-classification results.Source: §E.2 · pp. 26–27, Table 8 · p. 26

E.3 EFFECTIVENESS OVER THE NUMBER OF IN-CONTEXT EXEMPLARS.

• With ε = ∞, performance remained stable across shot counts, but 8-shot and 16-shot setups incurred drops of about 10% and 20%, respectively, attributed to added noise.The ablation tested 1, 2, 4, 8, and 16 exemplars with an ensemble size of 10; more shots required a higher subsampling rate.Source: §E.3 · p. 27, Figure 7 · p. 27

E.4 EFFECTIVENESS OVER SUBSAMPLING RATES

• At a subsampling rate of 0.5 × 10^-2, performance was satisfactory and 100 SST-2 predictions cost $0.0945; higher rates raised cost without significant performance gains.The evaluation used 100 test examples per dataset; the rate corresponds to 10 exemplars after subsampling.Source: §E.4 · p. 27, Figure 8 · p. 28

F ADDITIONAL EVALUATIONS ON LANGUAGE GENERATION

• Language-generation ablations cover document question answering and dialog summarization, with results reported in Appendices F.1 and F.2, respectively.Source: F ADDITIONAL EVALUATIONS ON LANGUAGE… · p. 29

• Performance with keyword-space aggregation via PTR remains stable up to 10^6 queries, supporting a strong-performance guarantee for at least 10^5 queries.The evaluation identifies PTR as consistently outperforming embedding-space aggregation; Figure 9 shows performance across query counts.Source: §F.1 · p. 29, Figure 9 · p. 29

• More public candidates improve embedding-space aggregation performance, but also require more API queries and increase overall cost.Figure 10 evaluates performance across candidate counts.Source: §F.2 · p. 29, Figure 10 · p. 29
• Figure 11 presents the full ablation results for varying ensemble counts.Source: §F.2 · p. 29, Figure 11 · p. 30
• Figure 12 presents the full ablation results for varying query counts.Source: §F.2 · p. 30, Figure 12 · p. 30