PaperlayerView exact revision on arXiv

Source-linked AI summary

No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices

Qi Pang, Shengyuan Hu, Wenting Zheng, Virginia Smith

arXiv:2402.16187v3cs.CRcs.CLcs.LG

TL;DR

LLM watermarks can help detect generated content, but common design choices can also expose them to removal and spoofing attacks. The paper studies these attacks and proposes defenses and guidelines; its results show trade-offs between resistance to watermark stealing and vulnerability to removal.

  • Problem

    LLM watermarks are intended to make generated content detectable, but it remains an open question whether public detection APIs can be made available without exposing systems to attacks.

    Source: §1 · p. 1, §6 · p. 7

  • Method

    The authors analyze removal and spoofing attacks that exploit watermark robustness, multiple keys, and public detection APIs, and propose a differential-privacy-inspired defense with complementary safeguards.

    Source: §1 · pp. 1–2, §6.4 · p. 9

  • Results

    With three keys, watermark-removal attack ASR is 36.2% and watermark stealing-based spoofing ASR is 13.0%, illustrating a trade-off as more keys reduce stealing-based spoofing but increase removal vulnerability.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    Choosing more keys to resist watermark stealing can increase removal risk, so limiting user query rates can help mitigate both attacks.

    Source: §5.2 · p. 7

    The study focuses on three PRF-based robust watermarks; attacking semantics-based watermarks is outside its scope.

    Source: §7 · p. 10

Abstract

from arXiv · show

Advances in generative models have made it possible for AI-generated text, code, and images to mirror human-generated content in many applications. Watermarking, a technique that aims to embed information in the output of a model to verify its source, is useful for mitigating the misuse of such AI-generated content. However, we show that common design choices in LLM watermarking schemes make the resulting systems surprisingly susceptible to attack -- leading to fundamental trade-offs in robustness, utility, and usability. To navigate these trade-offs, we rigorously study a set of simple yet effective attacks on common watermarking systems, and propose guidelines and defenses for LLM watermarking in practice.

1 Introduction

The paper examines how common LLM watermarking design choices can enable simple removal and spoofing attacks, despite their intended benefits. It studies these attacks and proposes defenses and design guidelines.

1 Introduction

  • The study focuses on robustness, multiple watermark keys, and public detection APIs as design choices that can make existing watermarking systems vulnerable.These choices are intended to resist removal, prevent watermark stealing, or make verification accessible.Source: §1 · p. 1
  • The paper studies watermark-removal attacks and spoofing attacks that embed a target watermark in potentially toxic content, making it appear to come from a specific LLM.The attacks exploit common watermarking design choices that have clear benefits but can also create vulnerabilities.Source: §1 · p. 1
  • The authors examine trade-offs among watermark robustness, utility, and usability, then propose defenses and general guidelines for future LLM watermarking systems.They evaluate three state-of-the-art watermarks [17] [46] [20] with LLAMA-2-7B [38] and OPT-1.3B [45].Source: §1 · pp. 1–2
  • Robustness can enable piggyback spoofing that makes watermarked text toxic or inaccurate through small edits, while multiple keys and public APIs create additional attack risks.The introduction notes that one token can render an entire sentence inaccurate and that differential-privacy techniques can help counter API-based spoofing.Source: §1 · p. 2

2 Related Work

The related work covers watermarking as a way to detect LLM-generated text, prior removal and spoofing attacks, and how this paper differs from those approaches.

2 Related Work

  • LLM watermarking embeds invisible signals in generated content that can be verified with a secret key, supporting detection of AI-generated text amid concerns about misuse.The surveyed goals include robustness to removal, resistance to watermark stealing, and easy detection of candidate text.Source: §2 · pp. 2–3
  • Prior removal methods use paraphrasing, often requiring additional training or a high-quality oracle, whereas this work studies scalable attacks tied to watermark design choices.Earlier work demonstrated watermark removal using a local LLM [19] [44].Source: §2 · p. 3
  • Earlier spoofing attacks estimate a watermark pattern from large amounts of queried or collected tokens, while this work studies attacks that do not require pattern estimation.For example, [35] requires 1 million queries, and [15] [11] assume access to millions of watermarked tokens.Source: §2 · p. 3

3 Preliminaries

The preliminaries define the paper’s language-model and watermarking setup, then specify the attacker’s objectives and capabilities for the studied attack families.

3 Preliminaries

  • An unwatermarked language model predicts a distribution for the next token from the preceding token sequence and samples the next token from that distribution.The notation distinguishes the original model, watermarked model, secret key, token sequence, and vocabulary.Source: §3 · p. 3
  • The paper studies decoding-based watermarks KGW [17], Unigram [46], and Exp [20], which perturb the model’s next-token distribution according to a secret key.Watermark detection returns a statistical score for the null hypothesis that the input sequence is independent of that key.Source: §3 · p. 3

3.1 Threat Model

  • Removal attacks seek high-quality output without a watermark, while spoofing attacks seek harmful or incorrect output carrying the victim organization’s watermark.The threat model examines attacks exploiting robustness, multiple keys, or public detection APIs, with different capabilities for each.Source: §3.1 · p. 4
  • Piggyback spoofing assumes O(1) queries to the target watermarked model and the ability to edit its generated sentence.Edits may include inserting or substituting tokens.Source: §3.1 · p. 4
  • For multi-key removal, the attacker is assumed to make O(l) queries for a sentence of length l; the setting considers multiple keys used to improve resistance to watermark stealing [20] [17] [15] [11] [35].For public-API attacks, the model assumes O(l) detection queries, returned confidence scores, and access to the top five tokens and their probabilities at each position [26] [32].Source: §3.1 · p. 4

4 Attacking Robust Watermarks

Robust watermarks can preserve detection after small edits, which attackers exploit to add toxic or inaccurate content. Evaluation finds effective attacks, and the discussion cautions against treating robustness alone as proof of authenticity.

4 Attacking Robust Watermarks

  • A more robust watermark makes piggyback spoofing easier by allowing more toxic tokens to be inserted, creating a trade-off between resistance to edits and spoofing feasibility.The attack begins with one query for watermarked text and then edits it, including token insertion or inaccurate rewriting with another LLM.Source: §4 · pp. 4–5

4.1 Evaluation

  • >20% median portion of toxic tokens can be inserted into watermarked text, while the resulting content has median moderation confidence above 0.8 across the studied schemes.Before attack, average moderation confidence is around 0.01; the figure reports the toxic-token insertion evaluation.Source: Figure 1 · p. 5, §4.1 · p. 5
  • 94.17% of GPT4-edited piggyback results retained a z-score above the default threshold of 4, and 92% of 100 sampled results were fluent and inaccurate or opposite in meaning.The reported attack success rate exceeds 90%.Source: §4.1 · p. 5

4.2 Discussion

  • Robust watermarks are not suitable as proof of content authenticity alone; the authors suggest pairing robustness with measures such as signature-based fragile watermarks.They note that even one toxic or incorrect token can make content harmful or inaccurate, while robust detection may not identify the attacker’s token.Source: §4.2 · pp. 5–6

5 Attacking Stealing-Resistant Watermarks

Using multiple watermark keys can impede watermark stealing but also enable watermark removal. The evaluation and discussion show that key count creates a trade-off, with query-rate limits offered as a mitigation.

5 Attacking Stealing-Resistant Watermarks

  • The removal attack estimates the unwatermarked output distribution by querying the same prompt repeatedly, then sampling from observed token frequencies.The paper connects this strategy to watermark unbiasedness and says it can preserve generated-content quality.Source: §5 · p. 6

5.1 Evaluation

  • 91% ASR with one key falls to 13% with three, while more than 7 keys brings stealing ASR close to zero but removal ASR above 97%.Figure 2 compares the attacks at an FPR of 1e-3.Source: §5.1 · pp. 6–7, Figure 2 · p. 7
  • More keys improve removal-attack output quality; seven keys suffice for content comparable to unwatermarked output, with the pattern consistent across watermark schemes and models.Source: §5.1 · p. 7

5.2 Discussion

  • The key-count trade-off has no trivial sweet spot; limiting query rates can reduce attack success, and the authors suggest defense in depth.Their suggested measures include anomaly detection and user identification verification.Source: §5.2 · p. 7

6 Attacking Watermark Detection APIs

The section examines how public watermark-detection APIs can expose systems to removal and spoofing attacks, then evaluates a differential-privacy-inspired defense. The defense substantially reduces spoofing while retaining high detection accuracy, but is less effective against removal attacks.

6 Attacking Watermark Detection APIs

  • Whether to make detection APIs public remains an open question: public access aids detection, but acknowledged oracle-attack risks motivate examining its trade-offs.The section introduces API-exploiting attacks and considers defenses to mitigate them.Source: §6 · p. 7

6.1 Attack Procedures

  • Removal attackers query token replacements and use their probabilities and detection scores to choose tokens that reduce watermark evidence while preserving output quality.The described setup considers the top L = 5 tokens; for KGW and Unigram with γ = 0.5, the chance of finding an unwatermarked candidate is 1 − γ^L.Source: §6.1 · p. 8
  • Spoofing attackers choose tokens with higher detection confidence, so synthesized content can pass the threshold without every token carrying a watermark.The procedure can generate harmful content using a local model.Source: §6.1 · p. 8

6.2 Evaluation

  • API-based removal reduces KGW detection to a median z-score of 1.43, below the threshold of 4, while PPL stays close to watermarked output (6.17 vs. 6.28).For Exp, sampling based on token probabilities and detection scores improves output quality relative to the other two watermarks.Source: §6.2 · p. 8
  • API-based spoofing raises detection confidence for content not generated by the watermarked LLM, with high success rates and reasonable query counts.The authors report consistent results on OPT-1.3B; Table 2 covers ASR and average queries per token on LLAMA-2-7B.Source: §6.2 · p. 8, Table 2 · p. 8

6.3 Defending Detection with Differential Privacy

  • At σ = 4, DP detection lowers KGW/LLAMA-2-7B accuracy from 98.2% to 97.2% while spoofing ASR falls to 0%.The authors report consistent results for Unigram and Exp watermarks and OPT-1.3B in Appendix L.Source: Figure 4 · p. 9, §6.3 · p. 9

6.4 Discussion

  • Public detection APIs enable both spoofing and removal; DP is less effective against removal, while rate limits and user-identity checks are proposed as complementary safeguards.The authors also note that multi-token substitutions may support spoofing but require many more API queries.Source: §6.4 · p. 9

7 Discussion, Limitation & Future Work

The discussion bounds the studied attacks to particular watermark designs and distinguishes them from several related design choices. It identifies trade-offs involving context width and detection aggregation, and points to other watermark families and image watermarking as future directions.

Generalizability of our attacks

  • The study focuses on three popular PRF-based robust watermarks; attacks on semantics-based watermarks are outside its scope and left as future work.The authors describe the selected PRF-based set as a natural target given its popularity and formal guarantees.Source: §7 · p. 10

Related watermarking schemes

  • The removal attacks exploiting multiple keys do not directly generalize to a cited undetectable watermark scheme that uses pseudorandom error-correcting codes instead of multiple keys.The authors identify attack opportunities on such schemes as a future direction.Source: §7 · p. 10

Future work

  • The authors suggest image watermarking as a future direction, noting that attacks must preserve domain-specific meaning and quality constraints.They describe exploring fundamental trade-offs for image watermarks as promising future work.Source: §7 · p. 10

Trade-offs of watermark context width

  • Multiple keys mitigate watermark stealing but introduce removal attack vectors, whereas increasing KGW context width raises stealing resistance while reducing robustness.Prior work reported successful stealing even at h = 4; the paper states its attacks exploit properties orthogonal to context width.Source: §7 · p. 10

The influence of how detection proceeds with multiple keys

  • The removal attack exploiting multiple keys does not depend on the detector’s aggregation method, though different aggregation can shift detection performance and the sweet spot for key count.The evaluated scenarios use min/max aggregation; harmonic-mean p-value aggregation is cited as a more robust alternative.Source: §7 · p. 10

Changing to p-values in KGW and Unigram

  • The paper presents z-statistics for KGW and Unigram while expecting no result impact from switching to monotonic p-values; additional p-value results appear in Appendix M.P-values are used for Exp in the paper.Source: §7 · p. 10

8 Conclusion

Common LLM watermarking design choices can improve robustness, resistance to watermark stealing, and public detection while also enabling removal or spoofing attacks. The paper offers guidelines, but how to navigate watermarking trade-offs depends on the application.

8 Conclusion

  • Watermarking design choices that improve robustness, resistance to watermark stealing, or public detection can also enable attacks that remove watermarks or damage a model’s reputation.The authors propose guidelines and possible defenses based on theoretical and empirical analysis.Source: §8 · p. 10

A Broader Impacts

  • The proposed guidelines are rough proposals because the best way to navigate trade-offs among utility, usability, robustness, and attack susceptibility depends on the application.Choosing strategies for specific applications remains future work.Source: A Broader Impacts · p. 14

B Attacker’s Motivation

  • Removal attacks could let a student or journalist claim originality, or let a malicious service reuse a victim’s watermarked model outputs and potentially infringe its IP.Piggyback spoofing could also falsely associate fake news, incorrect facts, or malicious code with a benign LLM provider.Source: B Attacker’s Motivation · p. 14

C Watermarking Schemes & Hyper-Parameters

  • The setup covers KGW, Unigram, and Exp, and evaluates sentence quality with perplexity using GPT-3 as the oracle model.Unigram’s global green-red split is described as twice as robust as KGW but more prone to leaking the watermark pattern; Exp uses multiple keys and a Gumbel trick for a distortion-free guarantee in expectation.Source: C Watermarking Schemes &… · pp. 14–16

D Attack Feasibility Analysis of Piggyback Spoofing Exploiting Robustness

The section bounds how many tokens can be inserted while keeping an edited Unigram watermark’s expected z-score above its detection threshold. It links watermark robustness to piggyback spoofing risk.

D Attack Feasibility Analysis of Piggyback Spoofing Exploiting Robustness

  • Theorem 1 gives a sufficient insertion bound for an edited Unigram sequence’s expected z-score to remain above threshold T.The analysis concerns a watermarked sequence of length l, detection z-score z, green-list portion γ, and s inserted tokens.Source: D Attack Feasibility Analysis of… · p. 16
  • The analysis provides a tight bound on the maximum number of edits, making piggyback attack feasibility easier to assess than prior analysis of z-score changes for a specified edit count.The section expresses the attack’s editing-distance bound in terms of the watermark’s sequence and detection quantities.Source: D Attack Feasibility Analysis of… · p. 17
  • A stronger watermark allows more toxic tokens to be inserted in a piggyback attack, and even one toxic token can make content harmful or inaccurate.The authors state that this trade-off applies to robust watermarking schemes and that such spoofing may be extremely difficult to detect.Source: D Attack Feasibility Analysis of… · p. 17

E Validation of Theorem 1

The validation reports that most observed insertion portions satisfy Theorem 1’s prediction for Unigram. Piggyback examples show factual edits alongside watermark detections that retain high z-scores.

E Validation of Theorem 1

F Piggyback Attack Examples

  • A piggyback example changes Usain Bolt’s Olympic-gold count from eight to three while the reported detection z-score is 5.58.The corresponding watermarked example reports a z-score of 6.15.Source: F Piggyback Attack Examples · p. 18

G Additional Results of Piggyback Spoofing Attack

Additional experiments extend piggyback-spoofing findings to OPT-1.3B and multiple watermark/model combinations. The key-number analysis bounds recovery of unwatermarked tokens, while the Exp analysis supplies a separate bound for its sampling scheme.

G Additional Results of Piggyback Spoofing Attack

H Watermark Key Number Analysis for Watermark-Removal Attacks Exploiting the Use of Multiple Watermark Keys

  • At n = 13, γ = 0.5, and c = 3, the estimated probability of recovering the highest-probability unwatermarked token is 0.71.The passage says this implies watermark removal from over 71% of tokens using a small number of observations under different keys.Source: H Watermark Key Number Analysis for… · p. 20
  • The KGW and Unigram probability bound assumes greedy sampling; multinomial sampling is left for future analysis because a tight softmax-variance bound is unavailable.The authors nevertheless report that empirical attacks generate high-quality unwatermarked content with multinomial sampling.Source: H Watermark Key Number Analysis for… · pp. 20–21

I Probability Bound of Unwatermarked Token Estimation for Exp

  • For Exp, the analysis bounds recovery of the highest-probability unwatermarked token from observations under different keys.The proof uses the Gumbel trick: each independent key yields that token with its original probability p.Source: I Probability Bound of Unwatermarked… · p. 21

J Additional Results of Watermark-Removal Attacks Exploiting the use of Multiple Watermark Keys

Additional evaluations find that multiple-key watermark removal generalizes across three watermark schemes and two models, while key choices also affect stealing vulnerability. Exp is removable with 13 queries per token, and query-rate limits reduce stealing success.

J Additional Results of Watermark-Removal Attacks Exploiting the use of Multiple Watermark Keys

  • Multiple-key watermark-removal results generalize across KGW, Unigram, and Exp on LLAMA-2-7B and OPT-1.3B.The additional results report that Exp is also easily removed using multiple queries to estimate unwatermarked-token distributions.Source: J Additional Results of… · p. 21, Figure 11 · p. 22, Figure 12 · p. 23
  • More keys mitigate watermark stealing but increase vulnerability to removal; three keys are the observed best balance, though attack success remains non-negligible.The authors recommend also limiting potentially malicious users’ capabilities.Source: J Additional Results of… · pp. 21–22
  • Limiting attacker query rates significantly decreases watermark-stealing success, motivating complementary defenses such as anomaly detection and user verification.Figure 10 reports KGW stealing results on LLAMA-2-7B using three keys at varying query counts.Source: J Additional Results of… · p. 22, Figure 10 · p. 22
  • For Exp, 13 queries per token bring the resulting p-value close to unwatermarked content and significantly away from the watermarked p-value, while removal retains high output quality.The attack produces perplexity comparable to unwatermarked content when the number of queries under different keys exceeds 13.Source: J Additional Results of… · p. 22, Figure 12 · p. 23, Figure 15 · p. 23
  • Figures 12 and 15 report watermark-removal results for the Exp watermark [20] with multiple watermark keys, using LLAMA-2-7B and OPT-1.3B models, respectively.Source: Figure 12 · p. 23, Figure 15 · p. 23

K Additional Results of Attacks Exploiting Detection APIs

On OPT-1.3B, attacks exploiting the detection API reproduce the LLAMA-2-7B findings, with attack success rates above 75% using around three queries per token.

K Additional Results of Attacks Exploiting Detection APIs

L Additional Results of DP Defense

This section presents additional evaluations of differential-privacy detection and KGW context width, alongside checklist information on limitations, societal impacts, and human-subjects research.

L Additional Results of DP Defense

M Additional Results of Larger Context Width and Using P-Values

NeurIPS Paper Checklist

  • The checklist acknowledges imperfect attack-defense trade-offs and recommends defense-in-depth, including query rate limiting and anomaly detection; it reports no error bars but says repeated runs were consistent.The authors also state that results were consistent across different watermarks and models.Source: NeurIPS Paper Checklist · pp. 27, 29, 30

10. Broader Impacts

  • The authors report discussing both positive and potential negative societal impacts in Section 8.Source: §10 · p. 31
  • The work does not involve crowdsourcing or human subjects, so the participant-instructions and compensation item is marked NA.The paper also marks the IRB-approval item NA, citing the absence of human-subject research.Source: §10 · pp. 32–33
Loading 2402.16187v3…