PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation

Xinyu Tang, Richard Shin, Huseyin A. Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, Robert Sim

arXiv:2309.11765v2cs.LGcs.CR

TL;DR

Using private examples as LLM demonstrations creates risks of adversarial access and regurgitation. The paper generates DP synthetic demonstrations for ICL and reports substantial gains over 0-shot at ε = 1, with performance competitive with a non-private solution.

  • Problem

    Using private examples in LLM prompts raises privacy risks because an adversary may access or infer information about them.

    Source: §1 · p. 1

  • Method

    The paper generates synthetic few-shot demonstrations from private data under differential privacy and uses them for ICL.

    Source: §7 · p. 9

  • Results

    At ε = 1, the approach substantially improves on 0-shot performance and is competitive with the non-private solution.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    Instruction-only synthetic demonstrations can substantially improve zero-shot performance in some applications without using private dataset examples.

    Source: §5.2 · p. 7

    The algorithm resamples private demonstrations for each generated token; fixing them once would lose privacy amplification by subsampling and require much higher noise.

    Source: §7 · p. 9

Abstract

from arXiv · show

We study the problem of in-context learning (ICL) with large language models (LLMs) on private datasets. This scenario poses privacy risks, as LLMs may leak or regurgitate the private examples demonstrated in the prompt. We propose a novel algorithm that generates synthetic few-shot demonstrations from the private dataset with formal differential privacy (DP) guarantees, and show empirically that it can achieve effective ICL. We conduct extensive experiments on standard benchmarks and compare our algorithm with non-private ICL and zero-shot solutions. Our results demonstrate that our algorithm can achieve competitive performance with strong privacy levels. These results open up new possibilities for ICL with privacy protection for a broad range of applications.

1 INTRODUCTION

Using private examples in LLM prompts can expose them through adversarial access or regurgitation, while removing PII may not prevent linkage. The paper proposes DP synthetic demonstrations for effective ICL and also examines demonstrations generated without private data.

1 INTRODUCTION

  • Prompted private examples risk direct adversarial access or regurgitation, and PII removal may not prevent linkage using auxiliary information.The motivating question is whether effective ICL can be achieved while protecting prompt data with differential privacy.Source: §1 · pp. 1–2

1.1 OUR CONTRIBUTIONS

  • The proposed algorithm privately generates synthetic few-shot demonstrations from private data for use in ICL.It aggregates generation across disjoint subsets to provide DP guarantees.Source: §1.1 · p. 2
  • For TREC, GPT-3 Babbage achieves 50.7% accuracy at ε = 1, versus 35.4% for fully private 0-shot and 50.6% for the non-private solution.Source: §1.1 · p. 2
  • On AGNews, instruction-only synthetic 4-shot demonstrations achieve 68.0% accuracy, compared with 69.3% for the non-private solution and 47.9% for 0-shot.This setting uses no private dataset examples.Source: §1.1 · pp. 2–3

2 PRELIMINARIES

ICL adapts a pretrained model by conditioning it on task demonstrations rather than changing its weights. The paper situates its approach among DP techniques that privately aggregate information from disjoint data subsets.

2.1 PROMPTING AND IN-CONTEXT LEARNING

  • ICL adapts a task through demonstrations in the prompt without modifying the pretrained model’s weights.Source: §2.1 · p. 3

2.2 DIFFERENTIAL PRIVACY

  • DP-SGD adds noise to clipped per-sample gradients during training, whereas PATE privately aggregates predictions from teacher models trained on disjoint subsets.Source: §2.2 · p. 3
  • The proposed approach generates examples with an LLM without fine-tuning, privately aggregating generation across disjoint private-data subsets in a PATE-like approach.Source: §2.2 · p. 3

3 PROBLEM DEFINITION, THREAT MODEL, AND NOTATIONS

The problem is to answer ICL queries using demonstrations drawn from a private dataset while protecting its records under differential privacy. The proposed solution generates private synthetic demonstrations for reuse across queries.

3 PROBLEM DEFINITION, THREAT MODEL, AND NOTATIONS

  • The task input includes a private dataset of records, a pretrained language model, and a query with demonstrations drawn from that dataset.Source: §3 · p. 3
  • The model’s output must be differentially private for all queries, including an unlimited number of queries within the specified privacy budget.The formulation covers both fixed label sets and open-form labels.Source: §3 · p. 3
  • The threat model protects dataset examples from an adversary seeking direct access to them or seeking to infer private information.Source: §3 · p. 3
  • The method generates synthetic demonstrations from the private-data distribution under (ε, δ)-DP, then relies on post-processing to protect outputs across infinitely many queries.For fixed-label tasks, the prompt construction function can draw from examples of the requested label; for open-form labels, it uses the full dataset.Source: §3 · p. 4

4 PROPOSED METHOD

The framework generates DP synthetic few-shot examples before deployment, then uses them as demonstrations at inference without additional privacy cost. Its token-by-token generation algorithm privately aggregates subset-conditioned next-token probabilities, and its analysis establishes DP through subsampling and composition.

4 PROPOSED METHOD

  • The framework first generates DP synthetic few-shot examples and then uses them as inference demonstrations, with no additional privacy cost in the second step.The generation can be performed offline before deployment.Source: §4 · p. 4

4.1 ALGORITHM FOR DP FEW-SHOT GENERATION

  • At each token step, the algorithm samples MN private examples, partitions them into M disjoint subsets of N, and privately aggregates their next-token probabilities.Generation continues until EOS or the maximum token limit; the algorithm considers Gaussian and Report-Noisy-Max mechanisms.Source: §4.1 · p. 4
  • Restricting candidate tokens to the instruction-only top-K vocabulary can reduce noise effects before private aggregation.The algorithm requires a trusted LM during generation; the paper proposes using a trusted LM offline and a different, potentially untrusted LM for inference.Source: §4.1 · pp. 4–5

4.2 PRIVACY ANALYSIS

  • Theorem 4.2 establishes that Algorithm 1 is (ε, δ)-DP through per-iteration guarantees, subsampling amplification, and composition across Tmax iterations.Source: §4.2 · p. 5

5 EXPERIMENTS

Across the tested tasks, DP-generated demonstrations improve on fully private baselines and are generally competitive with non-private ICL. The ablations show how shot count, model size, privacy mechanism, and generation settings shape these results.

5.1 IMPLEMENTATION SETTINGS

  • Experiments cover AGNews, TREC, DBPedia, and MIT Movies genre and director slot filling, using 4-shot demonstrations for the main results.Main results use GPT-3 Babbage; the datasets include classification and information extraction tasks.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • For 4-shot ICL on downstream-task test sets, the private solution at ε = 1, 2, 4, and 8 generally provides substantial gains over fully private baselines and achieves competitive results with non-private cases (ε = ∞).The non-private baseline reports the maximum of results from Alg. 1 with σ = 0 and randomly selected 4-shot private-data demonstrations.Source: Table 1 · p. 6
  • With top-100 public vocabulary reduction, choices of M and N show little variation; without it, variation is higher, and matching the reduced-vocabulary performance requires more LM queries per token.For a fixed MN, N = 1 gives the best signal-to-noise ratio; without vocabulary reduction, the best performance is generally at the highest MN and N = 1.Source: §5.2 · p. 7

5.3 ABLATION STUDIES

  • Across DBPedia shot counts, the private approach improves on the instruction-only baseline and is competitive with non-private methods at 1, 2, and 4 shots, but random private examples perform better at 8 and 12 shots.Reducing noise does not bridge the gap at larger shot counts, which the authors identify as an opportunity for improvement.Source: Table 2 · p. 7, §5.3 · p. 7
  • As model size increases, instruction-only synthetic demonstrations improve performance; the private approach adds gains through Curie, while Davinci's instruction-only solution matches the non-private solution.The authors note that Davinci can generate useful demonstrations from its existing knowledge despite poor zero-shot performance.Source: Table 3 · p. 8, §5.3 · p. 8
  • Gaussian and Exponential mechanisms yield generally competitive DBPedia performance, with different generation and privacy trade-offs.The Exponential mechanism provides pure DP but only selects the highest noisy-probability token; Gaussian noise supports sampling methods such as nucleus sampling.Source: Table 4 · p. 8, §5.3 · p. 8
  • Membership-inference attacks show significant leakage for non-private ICL, while DP-generated few-shot samples reduce the attack to random guess.The paper also discusses the algorithm's monetary cost and provides generated-sample examples in appendices.Source: §5.3 · p. 8

6 RELATED WORK

Prior privacy-preserving ICL approaches use private ensembling, sometimes spending privacy per query or requiring public data. This work instead generates private synthetic demonstrations without public data or private fine-tuning, while remaining focused on few-shot ICL.

  • Wu et al. (2024) privately construct consensus over queries with disjoint demonstrations, but expend privacy budget per query, limiting the number of queries for a given budget; this approach supports unbounded queries.Source: §6 · pp. 8–9
  • Duan et al. (2023) privately label unlabeled public data for prompting, whereas this framework needs no public data and extends beyond text classification.The authors position their approach for settings where similar-distribution public data may be a strong assumption, including healthcare and industrial applications.Source: §6 · p. 9
  • The method privately aggregates LLM generation probabilities from disjoint private-data subsets to create synthetic demonstrations, without fine-tuning the LLM.The authors distinguish this from private inference methods that require non-private training on private data for domain adaptation.Source: §6 · p. 9
  • The lightweight approach is suited to ICL when few-shot demonstrations suffice; the authors consider private fine-tuning more effective for applications requiring large volumes of synthetic data.Source: §6 · p. 9

7 CONCLUSION AND FUTURE WORK

The paper introduces a differentially private algorithm that generates synthetic few-shot demonstrations from private data, and reports minimal loss in ICL performance. It also finds that self-generated demonstrations can improve zero-shot performance; the algorithm’s token-by-token resampling is a weakness.

7 CONCLUSION AND FUTURE WORK

  • The algorithm generates synthetic few-shot demonstrations from private data for ICL while guaranteeing differential privacy with respect to private examples.Empirical studies show formal privacy protection with minimal loss in ICL performance.Source: §7 · p. 9
  • LLM-generated demonstrations can significantly improve zero-shot performance when subsequently used for ICL.Source: §7 · p. 9
  • The algorithm resamples private demonstrations for each generated token, a weakness the authors identify for future improvement.Fixing demonstrations for an entire synthetic sample would require much more noise because it would not benefit from privacy amplification by subsampling.Source: §7 · p. 9

A PROOF OF THEOREM 4.2

The appendix analyzes the algorithm’s privacy by bounding per-iteration mechanisms, accounting for subsampling, and composing privacy guarantees across iterations. It also notes sampling qualifications, numerical-composition limits, and where task-specific prompt functions are documented.

A PROOF OF THEOREM 4.2

  • Privacy curves enable tighter composition guarantees than advanced composition, including numerical composition for adaptive compositions.The cited approach supports curves for Gaussian, Laplace, and subsampled Gaussian mechanisms.Source: A PROOF OF THEOREM 4.2 · p. 13
  • For Gaussian noise, the analysis uses subsampled Gaussian privacy curves per iteration and composes them across Tmax iterations; subsequent processing is post-processing.The proof overview states that per-iteration privacy follows from bounding ℓ2-sensitivity.Source: §2 · p. 14, A PROOF OF THEOREM 4.2 · p. 14

2. As we are adding

  • The Exponential-noise variant uses Report-Noisy-Max to obtain a pure-DP per-iteration guarantee for a sampled subset, then amplifies privacy through subsampling and composes across Tmax iterations.The analysis focuses on w rather than the private-generation term used in the Gaussian analysis.Source: §2 · p. 14
  • Numerical composition means the final privacy parameters have no closed-form expressions; an analytical expression is available but does not give the tight upper bound.Source: §2 · p. 14
  • The Gaussian analysis assumes Poisson subsampling, while experiments use sampling without replacement, which the authors describe as approximately the same.For the Report-Noisy-Max variant, the analysis applies to sampling without replacement under substitution neighboring datasets.Source: §2 · p. 14

B PROMPT CONSTRUCTION FUNCTIONS FOR EACH TASK IN SECTION 5

C DATASETS USED IN SECTION 5

The experiments use classification and information-extraction datasets, with prompt-construction functions provided for each task. The ICL prompt formats follow Zhao et al. (2021).

C DATASETS USED IN SECTION 5

  • AG News has four labels and 30,000 training and 1,900 test samples per class; DBpedia has 14 topics and 40,000 training and 5,000 test samples per class.Source: C DATASETS USED IN SECTION 5 · pp. 14, 16
  • TREC contains 5,500 training and 500 test questions across six non-uniform answer types; MIT-G and MIT-D each have 100 test samples for information extraction.MIT-G has 2,953 training samples and MIT-D has 1,561.Source: C DATASETS USED IN SECTION 5 · pp. 14, 16
  • Tables 5 and 6 provide task-specific prompt construction functions for text-classification and information-extraction tasks, respectively.The prompts begin with an instruction, include a demonstration, and then expect generation under the specified label.Source: Table 5 · p. 15, Table 6 · p. 15

D PROMPT FORMAT DURING ICL FOLLOWING PRIOR WORK ZHAO ET AL. (2021)

E HYPERPARAMETER SEARCH

The appendix provides the hyperparameters used for the main results and reports dataset-specific hyperparameter searches. The main-results setup uses the OpenAI API to select the top 100 tokens when RVP is enabled.

E HYPERPARAMETER SEARCH

F EMPIRICAL PRIVACY EVALUATION BY MEMBERSHIP INFERENCE ATTACKS

Membership-inference attacks against DP-generated demonstrations remained close to random guessing, including at ε=8. The section also estimates the one-time cost of generating demonstrations before deployment.

F EMPIRICAL PRIVACY EVALUATION BY MEMBERSHIP INFERENCE ATTACKS

  • The evaluation used DBPedia, Babbage, and 1-shot ICL, averaging attack AUC across trials for ε values 1, 2, 4, and 8.For DP generation, 50 member and 50 non-member samples were queried; the non-private comparison used actual member samples in the prompt.Source: F EMPIRICAL PRIVACY EVALUATION BY… · p. 18
  • Even at ε=8, membership-inference attack AUC remained close to random guessing, while attacks using actual private samples were successful.The reported result for actual samples is 81.84 at ε=∞.Source: F EMPIRICAL PRIVACY EVALUATION BY… · pp. 18–19

G MONETARY COST

  • The one-shot DBPedia generation cost was estimated across Ada, Babbage, Curie, and Davinci models as a one-time predeployment expense.The estimate considers API calls per generated token and prompt length, which can be approximated from training-example and instruction token lengths.Source: G MONETARY COST · p. 19, Table 16 · p. 19

H DEMONSTRATIONS

The DBPedia examples illustrate the quality of DP-generated demonstrations: Babbage generations are generally coherent and fluent, with occasional repetition in longer outputs.

H DEMONSTRATIONS

  • Babbage’s synthetic DBPedia generations maintain reasonable coherence and fluency despite the model being comparatively smaller and less advanced.Examples of DP few-shot generations at ε=4 are provided for DBPedia.Source: H DEMONSTRATIONS · p. 19, Table 17 · p. 20
  • The lengths of synthetic generations generally align with samples in their respective training datasets.Source: H DEMONSTRATIONS · p. 19
  • Longer generations occasionally repeat their last sentence; the authors suggest more capable instruction-following models or refined instructions may mitigate this.Source: H DEMONSTRATIONS · p. 19
Loading 2309.11765v2…