PaperlayerView exact revision on arXiv

Source-linked AI summary

LatticeGen: A Cooperative Framework which Hides Generated Text in a Lattice for Privacy-Aware Generation on Cloud

Mengke Zhang, Tianxing He, Tianle Wang, Lu Mi, Fatemehsadat Mireshghallah, Binyi Chen, Hao Wang, Yulia Tsvetkov

arXiv:2309.17157v5cs.CL

TL;DR

In cloud-based prompted generation, the server controls generation and sees prompts and outputs, limiting users’ ability to keep generated text private. LatticeGen lets the client sample true and noise tokens while the server performs inference; experiments report that more than 50% of the semantic remains hidden under strong attacks, with generation quality and speed degraded.

  • Problem

    In cloud-based prompted generation, the server controls generation and can access both prompts and outputs, leaving users no option to keep generated text private.

    Source: §1 · p. 1

  • Method

    LatticeGen has the server perform most computation while the client samples true and noise tokens and hides the true sequence in a noised lattice.

    Source: §3 · p. 3, §3.1 · p. 4

  • Results

    More than 50% of the semantic remains unknown to the server under strong attacks, as measured by BERTScore.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    LatticeGen demonstrates a cooperative generation framework in which the server still does most computation without knowing exactly what is generated.

    Source: §7 · p. 9

    LatticeGen sacrifices generation quality and speed for obfuscation, and alleviating quality degradation with larger G increases inference computation exponentially.

    Source: §8 · p. 9

Abstract

from arXiv · show

In the current user-server interaction paradigm of prompted generation with large language models (LLM) on cloud, the server fully controls the generation process, which leaves zero options for users who want to keep the generated text to themselves. We propose LatticeGen, a cooperative framework in which the server still handles most of the computation while the user controls the sampling operation. The key idea is that the true generated sequence is mixed with noise tokens by the user and hidden in a noised lattice. Considering potential attacks from a hypothetically malicious server and how the user can defend against it, we propose the repeated beam-search attack and the mixing noise scheme. In our experiments we apply LatticeGen to protect both prompt and generation. It is shown that while the noised lattice degrades generation quality, LatticeGen successfully protects the true generation to a remarkable degree under strong attacks (more than 50% of the semantic remains hidden as measured by BERTScore).

1 Introduction

Cloud-based prompted generation leaves the server access to both prompts and generated text, although outputs can affect users’ private decisions. LatticeGen gives the client control over sampling while the server performs most computation, and the paper evaluates defenses against a malicious server.

1 Introduction

  • In cloud-based prompted generation, the server fully controls generation and can access prompts and outputs, leaving users no option to keep generated text private.The paper notes that generated text may inform real-life private decisions, such as restaurant choices or writing inspiration.Source: §1 · p. 1
  • LatticeGen is a cooperative client–server protocol in which the server performs most computation while the client controls token sampling.The client mixes the true sequence with noise tokens in a lattice sent to the server.Source: §1 · p. 1
  • The paper proposes a repeated beam-search attack and a mixing noise scheme as defense against a hypothetically malicious server.The experiments apply LatticeGen to creative writing and report that the noised lattice degrades generation quality.Source: §1 · pp. 1–2
  • More than 50% of the semantic remains unknown to a malicious server under strong attacks, as measured by BERTScore.The noised lattice also degrades generation quality.Source: §1 · p. 2

2 Motivation and Preliminaries

The paper argues that generated text, like prompts, needs obfuscation because it can inform private decisions. It places a third-party client between user and server, and introduces lattices and the LLG format as preliminaries for LatticeGen.

2.1 Generated Text (also) Needs Obfuscation

  • Generated text and prompts are available to the server as raw text, while outputs can affect users’ private decisions; LatticeGen aims to obfuscate generated text.The protocol can be placed in a third-party client so users can keep a simple interface and need not handle protocol complexity.Source: §2.1 · p. 2

2.2 LatticeGen as a Third-Party Client

  • A third-party client can take the user’s prompt, run the privacy-aware protocol with the server, and return the generation without exposing protocol complexity to the user.The paper suggests open-sourcing the client so researchers and users can vet it.Source: §2.2 · p. 2

2.3 Preliminaries

  • An N-lattice holds N token options per time-step, while the LLG format linearizes the lattice and appends a G-token tail for next-token prediction.Larger G can improve generation quality by reducing history noise, but requires the server to enumerate N^G combinations; the paper also describes finetuning on noised lattices.Source: §2.3 · pp. 2–3

3 LatticeGen

LatticeGen divides generation between a server that performs inference and a client that samples true and noise tokens. The resulting noised lattice blurs earlier token history while retaining the exact last G tokens, trading generation performance for protection.

3 LatticeGen

  • LatticeGen keeps most computation on the server while the client samples one true token and N −1 noise tokens, so only the client knows which token is true.Both parties obtain the same noised lattice, and the client may also noise the prompt by treating it as a generation prefix.Source: §3 · p. 3, §3.1 · p. 4

3.1 Protocol

  • The server processes all NG potential G-gram tails and returns next-token distributions; the client selects the true-tail distribution, samples noise tokens, and shuffles the list before sending it back.A simple synonym scheme samples noise from nearby embeddings, but the text says this scheme is vulnerable to malicious-server attacks.Source: §3.1 · p. 4

3.2 Comparison with Standard LM: History Noised While Locally Sharp

  • LatticeGen blurs earlier token history through the noised lattice, strengthening protection as noise-token count rises but degrading LM performance.It preserves the exact last G tokens; in the worst case of zero use of nonimmediate history, it is at least as strong as a (G + 1)-gram LM.Source: §3.2 · p. 5

4 Attack and Defense

This section formalizes attack strength against a malicious server and develops beam-search attacks alongside client-side noise defenses.

4 Attack and Defense

  • Max-true-ratio has a 1/N lower bound, while exact-match true-ratio misses semantic similarity, motivating max-BERTScore as an embedding-based leakage measure.Source: §4 · p. 5

4.1 The Repeated Beam-Search Attack

  • Simple synonym noise is vulnerable to beam search because noise tokens lack direct links, leaving the true sequence with higher log-probability than noise combinations.Source: §4.1 · p. 5
  • Parallel noise can hide the true sequence from ordinary beam search, but RBS can remove its likely high-probability noise hypotheses and expose the true sequence.Source: §4.1 · p. 6
  • Repeated beam-search (RBS) performs N−1 beam searches, removes each resulting hypothesis from the lattice, then treats the remaining sequence as the final hypothesis.The attack is proposed as a more powerful variant of beam search.Source: §4.1 · p. 6

4.2 The Mixing Noise Scheme for Defense

  • Mixing noise samples from the true sequence’s next-token distribution at a random mix-ratio, branching the true sequence into noise paths while retaining independent noise sampling otherwise.Source: §4.2 · p. 6
  • With a tuned mix-ratio, the mixing noise scheme achieves the best max-true-ratio under repeated beam-search attack.Figure 3 illustrates the noise schemes under beam-search attack.Source: Figure 3 · p. 5, §4.2 · p. 6

5 Experiments

The experiments evaluate LatticeGen on creative writing, finding that its mixing scheme conceals substantial generation semantics under repeated beam search while trading off quality and computation. The evaluation also compares noise schemes, lattice granularity, and a non-lattice baseline.

5.1 Experiment Setting

  • The evaluation uses OPT-1.3B and Llama2-7B with WritingPrompts, comprising 10,000 training samples and 200/500 development/evaluation samples.The models were selected because of limited computing resources and as a proof of concept; interaction protocols were simulated on one machine.Source: §5.1 · pp. 6–7

5.2 Experiment Results

  • Under RBS attack, recommended mixing yields max-true-ratio/BERTScore close to or below 65%/55% for N = 2; N = 3 improves protection to 50%/40%, with worse generation quality.For N = 2, the result indicates around half the semantics are hidden and is close to the theoretical best max-true-ratio (N = 50%).Source: §5.2 · p. 7

5.1 Experiment Setting

  • Because LatticeGen noises word history, the authors do not recommend it for tasks requiring high consistency or factuality.The study focuses on creative writing; prompts and stories average 29 and 674 tokens, respectively.Source: §5.1 · p. 7

5.2 Experiment Results

  • The vanilla model has the best PPL and PMI but provides zero obfuscation, while LatticeGen degrades generation quality in exchange for protection.Table 1 reports the main results when LatticeGen is applied to both generation and prompt.Source: §5.2 · p. 7, Table 1 · p. 7
  • The synonym scheme is defenseless against BS, while the parallel scheme achieves true-ratio below 20% against BS but remains vulnerable to RBS.The mixing scheme is the authors’ recommended scheme and achieves the best protection under RBS.Source: §5.2 · p. 7
  • Using 4-gram rather than unigram units substantially alleviates quality degradation, especially PPL, while larger G-grams increase computation exponentially.Figure 5 compares perplexity for OPT-1.3B and Llama-7B-HF across G-gram units.Source: §5.2 · p. 7, Figure 5 · p. 8
  • Without a lattice, synonym noise applied to 100% or 50% of tokens drastically degrades PPL and PMI; LatticeGen offers a quality–privacy trade-off.In the non-lattice baseline, the client sends a synonym of each true token back to the server.Source: §5.2 · pp. 7–8
  • On a single A40 GPU, LatticeGen with 4-gram, N = 2 units is 4.76 times slower than the vanilla model.The authors expect the slowdown to be less significant on servers with stronger computing power.Source: §5.2 · p. 8

6 Related Work

Prior privacy-aware NLP work mostly addresses training or inference privacy, with much of it focused on NLU. The paper positions LatticeGen as applying lattice structure to privacy-aware generation and contrasts it with prompt-only anonymization.

6 Related Work

  • Prior privacy-aware NLP work mostly protects data for training or inference, and most studies focus on NLU tasks (Feyisetan et al., 2020; Xu et al., 2021).The passage also cites Qu et al. (2021), McMahan et al. (2017), Huang et al. (2020), and Yue et al. (2023) on privacy-aware NLP and training-data protection.Source: §6 · p. 8
  • The authors position their work as the first to consider decoding-time privacy for cloud LLM prompted generation and to use lattice structure for privacy-aware generation.The latter novelty claim is completed by the related-work passage fragment on privacy-aware generation.Source: §6 · pp. 8–9
  • Unlike prompt anonymization that replaces named entities and later de-anonymizes them, LatticeGen can noise all token types, especially generated tokens.The cited anonymization approach is Chen et al. (2023).Source: §6 · p. 9

7 Conclusion

LatticeGen lets the server perform most generation computation without knowing exactly what is generated, using a noised lattice and cooperative client–server protocol. It trades generation quality and inference speed for obfuscation, while leaving more than 50% of generation semantics unknown to the server by BERTScore.

7 Conclusion

  • LatticeGen keeps the server responsible for most generation computation while withholding the exact generated text from it.The paper attributes this to a noised lattice structure and a cooperative generation protocol between server and client.Source: §7 · p. 9
  • More than 50% of the generation semantics remain unknown to a malicious server by BERTScore, despite degradation in generation quality and inference speed.The reported protection uses LatticeGen with the proposed mixing noise scheme.Source: §7 · p. 9
  • The framework’s obfuscation comes with costs to both generation quality and inference speed.Source: §7 · p. 9

8 Limitations

LatticeGen trades generation quality and speed for obfuscation, and increasing the G-gram unit can alleviate quality degradation while making inference computation grow exponentially. The paper’s protection scope centers on users facing repeated beam-search attacks from the server, while its current implementation also requires a separate finetuned OPT model for each lattice configuration.

8 Limitations

  • Larger G-gram units can alleviate quality degradation to some degree, but make inference computation grow exponentially.The authors suggest strategically selecting a small portion of NG grams as future work.Source: §8 · p. 9
  • The study focuses on protecting users against server-side repeated beam-search attacks, leaving other attacks such as human inspection outside its scope.The authors also note that sharing generation control with the client could endanger the server, for example through jailbreaking.Source: §8 · p. 9
  • The current implementation finetunes a separate OPT model for every lattice configuration, making it space-unfriendly.A unified linearized-lattice format for one LLM to process different configurations is proposed as future work.Source: §8 · p. 9

9 Broader Impact

The paper argues that prompts and generated text need privacy protection because they can affect users’ private decisions, while current LLM-provider data-use and retention policies are insufficient. LatticeGen is proposed as a protocol for privacy-aware generation with controlled obfuscation.

9 Broader Impact

  • Prompts and generated text are transparent to the server in the current interaction paradigm, leaving users no option to keep generated text private.Source: §9 · p. 9
  • The paper argues that provider data-use and retention policies do not provide sufficient privacy protection.Source: §9 · p. 9
  • LatticeGen is proposed as privacy-aware generation with a controlled level of obfuscation.The authors hope the work raises awareness of privacy considerations for generated content.Source: §9 · p. 9

A Model Training and Inference with Lattice (Server)

The server is finetuned and performs inference with linearized lattices containing true and noise tokens. The appendices describe prompt incorporation, communication cost, a non-lattice baseline, and related work.

A Model Training and Inference with Lattice (Server)

  • Training constructs a lattice with a true token from one sample and noise from a randomly selected parallel sample, shuffles tokens, and uses P selected true tokens as next-token targets.The described training setup uses N=2 and G=1 for simplicity; the authors state that larger configurations follow similarly.Source: A Model Training and Inference with… · p. 14
  • Reusing saved hidden states supports efficient inference, but the server must still enumerate and infer NG G-gram combinations, the stated major source of slowdown.The saved hidden states are reused from the previous time step.Source: A Model Training and Inference with… · p. 14

B Auxiliary Framework Description

B.1 Incorporating the Prompt (Client)

  • To incorporate the prompt, the client uses prompt tokens as the true tokens for prompt time steps while continuing the other protocol steps, including noise generation.This embeds and noises the prompt in the lattice.Source: §B.1 · p. 14

B.2 Communication Cost

  • Communication requires sending NG length-|V| vectors per time step, but agreeing on a sampler such as top-k with k=50 can reduce transmission to logits and indices for the top 50 tokens.Source: §B.2 · p. 14

B.3 The Non-Lattice Baseline

  • The non-lattice baseline trains on inputs with some tokens noised and predicts the next true token, without including true tokens in the input.Its training objective is described as similar to lattice finetuning, except for excluding true tokens from the input.Source: §B.3 · p. 14

C Related Work

  • The paper contrasts LatticeGen’s cooperative lattice approach with local-DP text-noising methods, whose adaptation to privacy-aware autoregressive generation is unclear (Meehan et al., 2022).Source: C Related Work · p. 15

D Auxiliary Results

On a single A40 GPU, LatticeGen with bigram units runs 2x slower than PM, though the slowdown may be smaller on more powerful servers. The discussion of industry practices argues users cannot assume their data will be deleted immediately and presents LatticeGen as an algorithmic, cooperative approach to privacy protection.

D Auxiliary Results

  • Figure 8 is described as a different-prompt bigram example, while its caption specifies 4-gram, N=2, and notes that the prompt is also noised.The caption also identifies noise tokens mixed from previous true tokens.Source: D Auxiliary Results · p. 15, Figure 8 · p. 17
  • 2x slowdown: on a single A40 GPU, LatticeGen with bigram units (N = 2) runs slower than PM, though the slowdown should be less significant on more powerful servers.The comparison refers to Table 2, §D.Source: D Auxiliary Results · p. 15

E The Current Privacy Protection Practices in Industry

  • The paper says current LLM-provider data policies are insufficient, citing past consumer ChatGPT data use and sharing and Bard activity retention for at least three months.The passage presents these as examples of providers' data usage and retention practices.Source: E The Current Privacy Protection… · p. 15
  • Reported incidents include Samsung employees sharing sensitive code with ChatGPT and some Bard conversations being mistakenly indexed and accessed through Google Search.The passage describes the Samsung disclosure as reportedly occurring.Source: E The Current Privacy Protection… · p. 16
  • Despite improved provider security practices, users cannot assume sent or received data will be deleted immediately and completely; LatticeGen instead uses an algorithmic, cooperative approach to give users privacy control.The passage contrasts this approach with relying on regulations.Source: E The Current Privacy Protection… · p. 16
Loading 2309.17157v5…