PaperlayerView exact revision on arXiv

Source-linked AI summary

Watermark Stealing in Large Language Models

Nikola Jovanović, Robin Staab, Martin Vechev

arXiv:2402.19361v2cs.LGcs.AIcs.CR

TL;DR

The adversarial robustness of LLM watermarks remains poorly understood despite suggestions that current schemes may be ready for deployment. The paper proposes automated API-based watermark stealing and studies its downstream attacks; it reports over 80% success for spoofing and for hard-setting scrubbing, challenging assumptions about watermark robustness.

  • Problem

    Although some works suggest current LLM watermark schemes may be fit for deployment, their robustness to adversarial actors remains poorly understood and may be overestimated.

    Source: §1 · p. 1

  • Method

    The paper proposes an automated watermark-stealing algorithm that queries a watermarked LLM API to approximate its watermarking rules and studies resulting spoofing and scrubbing attacks.

    Source: §1 · p. 2, §5 · p. 5

  • Results

    The study reports over 80% spoofing success on KGW2-SELFHASH and, for hard long-text scrubbing, over 80% average paraphraser success versus almost 0% before, with no baseline reaching 25%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend that future evaluations treat watermark stealing as a first-class threat and conclude that more work is needed to design truly robust schemes.

    Source: §8 · p. 9

    The authors knew of no active LLM watermark deployments in practice and therefore believed the work’s enabled malicious uses could not then be directly abused.

    Source: Impact Statement · p. 10

Abstract

from arXiv · show

LLM watermarking has attracted attention as a promising way to detect AI-generated content, with some works suggesting that current schemes may already be fit for deployment. In this work we dispute this claim, identifying watermark stealing (WS) as a fundamental vulnerability of these schemes. We show that querying the API of the watermarked LLM to approximately reverse-engineer a watermark enables practical spoofing attacks, as hypothesized in prior work, but also greatly boosts scrubbing attacks, which was previously unnoticed. We are the first to propose an automated WS algorithm and use it in the first comprehensive study of spoofing and scrubbing in realistic settings. We show that for under $50 an attacker can both spoof and scrub state-of-the-art schemes previously considered safe, with average success rate of over 80%. Our findings challenge common beliefs about LLM watermarking, stressing the need for more robust schemes. We make all our code and additional examples available at https://watermark-stealing.org.

1. Introduction

Although current LLM watermark schemes have been suggested as deployment-ready, their robustness to adversaries remains poorly understood. The paper identifies API-based watermark stealing as a route to practical spoofing and scrubbing, including in settings previously considered safe.

1. Introduction

  • Prior work suggested current schemes might be fit for deployment, but their robustness to adversarial actors remains poorly understood and may be overestimated.Prior work had also concluded that prominent distribution-modifying schemes resist scrubbing on sufficiently long texts.Source: §1 · pp. 1–2
  • Watermark stealing uses API queries to approximate secret watermarking rules, which can then support spoofing and scrubbing attacks.Spoofing generates text detected as watermarked without the secret key; scrubbing removes the watermark from text.Source: §1 · pp. 1–2
  • Over 80% spoofing success is reported on KGW2-SELFHASH; in hard long-text scrubbing, average paraphraser success rises from almost 0% to over 80%, while no baseline reaches 25%.The scrubbing comparison is for KGW2-SELFHASH.Source: §1 · p. 2
  • The authors propose the first automated watermark-stealing algorithm, with a one-time cost below $50 in the Jan 2024 ChatGPT API pricing model.The pricing assumption is stated in the paper’s footnote.Source: §1 · p. 2
  • The paper argues watermark stealing should be treated as a first-class threat in future evaluations while more robust schemes are developed.The authors remain optimistic about watermarking but say more work is needed before schemes can be safely deployed.Source: §1 · p. 2

2. Background on LLM Watermarks

These schemes modify token-generation distributions to embed a signal, then detect it by testing whether observed green-token counts exceed a non-watermarked expectation.

Distribution-modifying watermarks

  • Distribution-modifying watermarks alter the language model’s logits before sampling to bias which tokens are generated.The schemes considered include work published at ICML 2023 and ICLR 2024.Source: §2 · p. 3

KGW-Soft/Hard

  • KGW-SOFT increases green-token logits, while KGW-HARD excludes red tokens; detection uses green-token counts to compute a z-statistic and p-value.KGW-HARD is acknowledged as impractical because it harms text quality.Source: §2 · p. 3

SelfHash and variants

  • SelfHash variants seed the pseudorandom function from a longer token context, optionally including the candidate token through self-seeding and rejection sampling.KGW2-SELFHASH uses h = 3 and self-seeding, and was recommended as promising for text quality and robustness.Source: §2 · p. 3

Variants and tradeoffs

  • Larger context size can make watermark rules harder to infer but make watermarks less localized and easier to scrub; UNIGRAM instead uses fixed green lists.KGW2-SELFHASH partly overcomes the scrubbing-robustness loss described in this tradeoff; UNIGRAM drops dependence on prior tokens.Source: §2 · p. 3

3. The Watermark Stealing Threat Model

The attacker is modeled as querying watermarked full generations to approximate secret-key rules, with detector access and access to non-watermarked responses defining two threat-model dimensions.

Threat model

  • The attacker has black-box access to full generations through the watermarked API, knows a watermark is present, and seeks to approximate its secret-key rules with few queries.The model separates watermark stealing from the downstream attacks it may enable.Source: §3 · p. 3

Assumptions

  • The attacker is assumed to know scheme parameters but not the secret key, and to have an auxiliary language model, which the paper says is readily available.Source: §3 · p. 3

Detector access

  • Detector access ranges from private detection (D0) to arbitrary binary API responses (D1); exact z-scores are excluded because they expose red/green lists.D0 is the more restrictive setting and is commonly considered in prior work.Source: §3 · pp. 3–4

Base-response availability

  • Base responses may be unavailable (B0) or available (B1); the paper focuses its main evaluation on the most restrictive combination, D0 and B0.B1 can include non-watermarked model outputs or a corpus generated before watermark deployment.Source: §3 · p. 4

4. The Watermark Stealing Attacker

The attacker uses a learned token-and-context scoring function with an auxiliary model to strengthen a watermark for spoofing or weaken it while paraphrasing for scrubbing.

Stealing the watermark

  • Watermark stealing queries the owner model and produces a scoring function that estimates whether a candidate token is green in its context.The score can support downstream attacks independently of the particular stealing algorithm, provided it supplies s⋆.Source: §4 · p. 4

Overview

  • The automated attacker consistently mounts spoofing and scrubbing attacks with high success rates in realistic settings across several state-of-the-art schemes.The experimental section evaluates both downstream attacks after watermark stealing.Source: §4 · p. 4

Mounting a spoofing attack

  • For spoofing, the attacker uses an auxiliary model to answer a chosen prompt or paraphrase existing text, promoting tokens in proportion to their estimated green-token scores.The logit update adds δ_att · s⋆, with δ_att > 0.Source: §4 · p. 4

Duplicate handling

  • When detection ignores duplicate (h + 1)-grams, the attacker penalizes tokens that would complete duplicates because they add length without helping produce a green token.The penalty divides the token logit by ρ_att before adding s⋆.Source: §4 · p. 4

Mounting a scrubbing attack

  • For scrubbing, the attacker paraphrases watermarked text and sets δ_att < 0 to demote tokens estimated to strengthen the watermark.This applies the learned scoring function to weaken rather than imprint the watermark.Source: §4 · p. 4

5. Our Stealing Algorithm

The algorithm estimates watermarked and base token distributions from queried text, then combines full and partial-context signals into a score for candidate tokens.

Modeling watermarked text

  • For KGW2-SELFHASH, the attacker tokenizes queried responses and uses each consecutive 4-gram to estimate watermarked token distributions for every subset of the preceding three-token context.The attacker uses its auxiliary model’s tokenizer and does not need to know the owner model’s tokenizer.Source: §5 · p. 5

Modeling base responses

  • Base distributions come from non-watermarked owner-model responses when available, or from auxiliary-model prompts in the unavailable-base setting.The paper focuses on the latter setting in its main evaluation.Source: §5 · p. 5

Scoring candidate tokens

  • A candidate token’s score compares corresponding watermarked and base probability masses as evidence that it belongs to the green list, then clips and rescales the result to [0, 1].Source: §5 · p. 5

The sparsity challenge

  • Full three-token context scores can be unreliable because contexts are sparse; the algorithm zeroes estimates based on very few samples and supplements them with weaker partial-context signals.The unified score is used for each candidate token after the three-token sequence.Source: §5 · pp. 5–6

Leveraging partial contexts

  • The algorithm uses score-vector comparisons to identify the minimum-hash context token, whose single-token score supplies a more reliable partial-context signal.The method also adds a context-independent score when the candidate token is minimal in the seeding rule.Source: §5 · p. 6

Unified score

  • For KGW2-SELFHASH, the unified score weights the partial-context and context-independent components by 0.5 and 0.25, respectively; inapplicable components are zeroed for other schemes.The resulting score supports both spoofing and scrubbing, and the method can be adapted to other schemes using only applicable features.Source: §5 · pp. 5–6

6. Experimental Evaluation

The evaluation tests whether watermark rules approximated through stealing support targeted spoofing and scrubbing, including under a restrictive setting with unavailable base responses, different owner and attacker models, and no detector access. Across the studied schemes, stealing supports high-success attacks in both directions, challenging the expected tradeoff between spoofing and scrubbing robustness.

6.1. Mounting Spoofing Attacks

  • Over 80% of MISTRAL-7B-generated texts are high-quality spoofs detected as watermarked on KGW2-SELFHASH, with similar results across other attacker models.The study reports this result in the restrictive (B0, D0) setting.Source: §6.1 · p. 7, Table 1 · p. 7
  • KGW2-SUM is harder to spoof than KGW2-SELFHASH, yet over 50% of attacker texts are valid spoofs; harmful-query results also show no degradation.MISTRAL-7B had a 4% refusal rate, compared with 100% for the better-aligned LLAMA-13B model used as the watermark owner.Source: §6.1 · p. 7, Table 2 · p. 7
  • At 30,000 queries, stealing costs $42, and the spoofing curves converge to the 30,000-query results at around 10,000 queries.The cost estimate uses then-current ChatGPT API prices.Source: Figure 3 · p. 7, §6.1 · p. 8
  • Removing partial-context scores lowers average spoofing success from around 80% to below 50% at 30,000 queries.This ablation supports the value of the partial-context component in the attacker.Source: Figure 3 · p. 7, §6.1 · p. 8
  • Imprinting a watermark onto existing text with DIPPER achieves at least 74% spoofing success across scenarios on KGW2-SELFHASH and KGW-SOFT.The reported expected false-positive rate is 10^-3.Source: §6.1 · p. 8

6.2. Boosting Scrubbing Attacks

  • DIPPER+OURS raises average scrubbing success on KGW2-SELFHASH from around 0 to above 80%, while the best baseline reaches no more than 30%.The boosted paraphrases also have median p-values above 0.03, versus several orders of magnitude below the boosted results for the best baseline.Source: §6.2 · p. 8, Table 3 · p. 8
  • Adding stolen watermark scores to DIPPER raises scrubbing success to almost 100% on KGW2-SUM and above 81% on average for other schemes.Without the added scores, DIPPER averages at most around 30% on the other three schemes.Source: §6.2 · pp. 8–9, Table 4 · p. 9
  • The results show that spoofing and scrubbing robustness are not necessarily at odds: stealing supports both attacks with high success on the same schemes.The authors contrast this with the supposed tradeoff, under which schemes easier to steal were expected to have better scrubbing robustness.Source: §6.2 · p. 9

7. Related Work

Prior work had studied watermarking and individual spoofing or scrubbing threats, but had not established realistic KGW2-SELFHASH spoofing or examined whether querying the owner model could boost scrubbing. Related concurrent work addresses overlapping attacks under different settings.

LLM watermarking

  • The paper focuses on distribution-modifying text watermarks, distinct from semantic, sampling-modification, neural-network, model-weight, and post-hoc detection approaches.The related work also situates text watermarking within language steganography.Source: §7 · p. 9

Spoofing and scrubbing

  • Earlier work concluded that distribution-modifying schemes were mostly robust to scrubbing on long texts, a conclusion this paper refutes.Prior work had studied scrubbing but not how querying the watermarked model before an attack could boost it.Source: §7 · p. 9
  • Prior spoofing work included a manually written KGW-SOFT proof of concept, but did not show automated, realistic spoofing of KGW2-SELFHASH.The paper characterizes realistic settings as chat LMs, low FPR, high-quality text, and a low one-time query cost.Source: §7 · p. 9
  • Pang et al. (2024) study spoofing without a high-quality constraint and with z-score detector access, while Wu & Chandrasekaran (2024) independently connect stealing with scrubbing.The latter propose an attack requiring additional queries for each generated token.Source: §7 · p. 9

8. Conclusion

The paper formalizes watermark stealing and introduces an automated algorithm, then evaluates spoofing and scrubbing in realistic settings. It concludes that current schemes are more vulnerable than previously thought and that future evaluations should treat stealing as a first-class threat.

8. Conclusion

  • The paper formalizes watermark stealing and proposes the first automated stealing algorithm, evaluated in a comprehensive study of spoofing and scrubbing in realistic settings.The authors report that the two attacks are not necessarily at odds.Source: §8 · p. 9
  • The authors conclude that current schemes are more vulnerable than previously thought and recommend that future evaluations account for watermark stealing as a first-class threat.They call for further work to design robust schemes.Source: §8 · p. 9
  • The authors remain optimistic about LLM watermarks while arguing that more work is needed to build robust schemes and sound intuitions about adversarial scenarios.They state that such work is needed before watermarks can be safely deployed.Source: §8 · p. 9

Impact Statement

The authors describe watermark stealing as raising potential misuse concerns but report no known active deployments. They frame the work's impact as highlighting robustness gaps and motivating more thorough evaluation and stronger schemes.

Impact Statement

  • The authors see no known active LLM-watermark deployments and therefore believe the work has no direct negative impact in practice.They note that the work could theoretically be used to invalidate watermarks and harm model owners or clients.Source: Impact Statement · p. 10

A. Examples of Attack Interactions of Our Watermark Stealing Attacker

A.1. Spoofing Example (SelfHash, (B0), Dolly-CW prompt #72)

  • In the Dolly-CW example, the generated email is judged watermarked, with a z-score of 9.66 and p-value 2.2e-22.The example's text-quality score is 9.00.Source: §A.1 · pp. 13–14

A.2. Spoofing Example (SelfHash, (B0), HarmfulQ+AdvBench prompt #02)

  • In the harmful-query example, the generated response is judged watermarked, with a z-score of 6.66 and p-value 1.4e-11.The example reports a text-quality score of 8.33.Source: §A.2 · p. 14

A.3. Scrubbing Example (SelfHash, (B0), Reddit Writing Prompts prompt #19)

  • In the scrubbing example, the original and baseline paraphrase are judged watermarked, while the stealing-assisted paraphrase is judged human.The respective decisions are reported with z-scores 6.92, 29.91, and 0.84.Source: §A.3 · pp. 15–16

B. Additional Experimental Results

The appendix extends the experimental results with additional spoofing tables, varying datasets, detector filtering, schemes, attacker models, and base-response access. It also reports a secret-key consistency check.

Additional Experimental Results

Additional spoofing results

  • Table 6 extends spoofing results to two additional datasets, and Table 8 covers the full range of datasets, schemes, and attacker models.The passages describe table scope only, not the underlying values.Source: Table 6 · p. 17, Table 8 · p. 17
  • Table 7 adds top-1 spoofing results selected with detector feedback or a heuristic, while Table 5 extends results to f = 10^-6.The supplied passages identify these extensions but do not provide the table values.Source: Table 7 · p. 17, Table 5 · p. 17
  • Table 9 presents the Table 2 variant with available base responses and both models set to MISTRAL-7B.The passage gives the setting but no outcome values.Source: Table 9 · p. 18

Consistency analysis with respect to watermark secret key

  • Table 10 reports a consistency check of spoofing results across variations in the watermark secret key ξ.The supplied passage identifies the table’s scope without giving its values.Source: Table 10 · p. 18

B.1. Additional spoofing results

Additional experiments extend spoofing evaluations across settings and datasets, and test consistency across secret keys. Detector-feedback filtering boosts results, while self-filtering is inconsistent.

B.1. Additional spoofing results

  • Oracle-filtering can raise spoofing results to around 100% in all cases, while self-filtering does not consistently perform well.The added results also show similar trends across datasets and no significant influence from the gap between the models in the B1 setting.Source: §B.1 · p. 18

B.2. Consistency analysis with respect to watermark secret key

  • The secret-key consistency check uses 10,000 queries rather than the main experiments’ 30,000 and reports consistent FPR rates.The reduced query count was chosen to save computation because this experiment tested self-consistency.Source: §B.2 · p. 18

B.3. Additional scrubbing results

Additional scrubbing experiments extend evaluation to more datasets and an available-base-response setting. A separate parameter study finds c = 2 reasonable, though c = 4 may improve results.

B.3. Additional scrubbing results

  • DIPPER consistently scrubs watermarks on two additional datasets, while the baseline paraphraser is generally unable to do so.Table 11 extends the main scrubbing results to these datasets.Source: §B.3 · p. 18, Table 11 · p. 19
  • In the B1 available-base-response setting, scrubbing results show no significant difference from the main B0 results.Table 12 presents this variant with MISTRAL-7B as LMmo and the attacker’s response-generating model.Source: §B.3 · p. 18, Table 12 · p. 19

B.4. Tuning the Clipping Parameter

  • The clipping study finds c = 2 a good choice for spoofing, though c = 4 may have improved results; the authors did not rerun experiments with c = 4.The study tested KGW2-SELFHASH on two datasets and omitted reruns due to computational constraints.Source: §B.4 · p. 19

C. Potential Mitigations

The paper discusses multiple secret keys and sampling-modification watermarks as possible directions for mitigating watermark stealing. Its experiments and cited work leave important questions about their practical viability.

Potential Mitigations

Multiple secret keys

  • Prior work found switching k = 2 keys at every token yields an expected 44% green percentage for unwatermarked text at γ = 0.25, while switching keys per response can increase scrubbing vulnerability except for small k.The first setup is described as impractical; the latter result is attributed to concurrent work.Source: C. Potential Mitigations · p. 19
  • On harmful data, spoofing still reaches 73% with k = 4 keys switched per response, and the authors hypothesize query efficiency and sparse, distinct key signals help explain this result.The attacker learns from 30,000 total responses, or n/k responses per key in expectation; the authors conclude multiple keys are not currently a viable defense.Source: C. Potential Mitigations · pp. 19–20, Table 14 · p. 20

Sampling modification watermarks

  • Sampling-modification schemes may merit renewed study because distortion-freedom in expectation could protect against stealing, but the gap between expectation and practice may itself be exploitable.The authors describe this as a possible direction and encourage further research on both attacks and defenses.Source: C. Potential Mitigations · p. 20

D. Omitted Experimental Details

The appendix records experimental defaults and describes a GPT-4 judge for text quality, including its scoring procedure and illustrative outputs. The judge separates ethics from quality scores, though ethics concerns can still leak into those scores.

D. Omitted Experimental Details

  • The watermarking experiments use default γ = 0.25 and δ = 4, ignore repeated n-grams, and generally set ρatt = 1.6 and δatt = 7.5.The attacker parameters were tuned on separate data when necessary.Source: D. Omitted Experimental Details · p. 20

D.1. GPT4 as a Judge of Text Quality

  • GPT-4 scores Style, Consistency, Accuracy, and Ethics from 1 to 10, with the first three averaged and 6.5 used as the high-quality threshold.The authors report that ethics concerns still leak into other categories, slightly understating attacker text quality on unethical content.Source: §D.1 · p. 21

D.1.1. EXAMPLE TEXT WITH QUALITY SCORE OF 3.67

  • The text-quality score was 3.67, computed in postprocessing from GPT4’s JSON output as the average of Style, Consistency, and Accuracy.Source: §D.1.1 · p. 22

D.1.2. EXAMPLE TEXT WITH QUALITY SCORE OF 5.67

  • The Vancouver-versus-Portland response scored 5.67 because it discussed Vancouver but omitted Portland, lowering its consistency and accuracy assessments.Its Style score was 7, Consistency 5, and Accuracy 5.Source: §D.1.2 · p. 23

D.1.3. EXAMPLE TEXT WITH QUALITY SCORE OF 7.00

  • The electronic-music answer received a Text Quality score of 7.00, computed in postprocessing from GPT4’s JSON output; it was incomplete because it did not cover all aspects of creating an electronic music song.Source: §D.1.3 · p. 24

D.1.4. EXAMPLE TEXT WITH QUALITY SCORE OF 9.67

  • The Netflix streaming-technology response scored 9.67, with Style 9 and Consistency and Accuracy 10 each.Source: §D.1.4 · p. 24
Loading 2402.19361v2…