PaperlayerView exact revision on arXiv

Source-linked AI summary

A Watermark for Large Language Models

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, Tom Goldstein

arXiv:2301.10226v4cs.LGcs.CLcs.CR

TL;DR

The paper addresses how to identify potentially harmful synthetic text without relying on fragile post-hoc detectors or access to proprietary language models. It proposes a soft watermark that biases sampling toward pseudorandom green-list tokens and detects the signal statistically. In experiments, detection reached 98.4% with multinomial sampling and 99.6% with 4-way beam search, while low-entropy sequences remained the main source of missed detections.

  • Problem

    Synthetic language-model text can support manipulation, fabricated content, cheating, and difficult dataset filtering, motivating detectable but human-imperceptible output signals.

  • Method

    The method pseudorandomly partitions tokens into green and red lists, softly boosts green-list logits during sampling, and detects excess green tokens with a z-test.

  • Results

    98.4% of generations were detected with multinomial sampling and 99.6% with 4-way beam search at the z = 4 threshold.

  • Takeaways & Limitations

    The watermark is computationally simple to verify without the underlying model, can be retrofitted without retraining, and degrades gracefully under attack.

  • Takeaways & Limitations

    Generative attacks that predictably alter the output language can randomize the red list for subsequent tokens, so careful implementation and instruction tuning may be required.

Abstract

from arXiv · show

Potential harms of large language models can be mitigated by watermarking model output, i.e., embedding signals into generated text that are invisible to humans but algorithmically detectable from a short span of tokens. We propose a watermarking framework for proprietary language models. The watermark can be embedded with negligible impact on text quality, and can be detected using an efficient open-source algorithm without access to the language model API or parameters. The watermark works by selecting a randomized set of "green" tokens before a word is generated, and then softly promoting use of green tokens during sampling. We propose a statistical test for detecting the watermark with interpretable p-values, and derive an information-theoretic framework for analyzing the sensitivity of the watermark. We test the watermark using a multi-billion parameter model from the Open Pretrained Transformer (OPT) family, and discuss robustness and security.

1. Introduction

The paper motivates watermarking as a way to identify synthetic language-model text amid risks including manipulation, fabricated content, and academic cheating. It seeks an efficient, statistically reliable watermark detectable from short text spans without retraining or access to model parameters.

  • LLM-generated content creates risks including social engineering, election manipulation, fabricated web content, and cheating.
  • A watermark is an imperceptible pattern that makes synthetic text algorithmically identifiable from short spans, as few as 25 tokens.
  • Detection can be open sourced or kept private, and it requires neither model parameters nor language-model API access.
  • The watermark can be generated with a standard language model without retraining and remains detectable in contiguous slices of longer generations.
  • The design aims to resist removal without modifying a significant fraction of generated tokens and to provide rigorous confidence measures for detection.
  • 1.2. A caveat: The difficulty of watermarking low-entropy sequences: Low-entropy text is difficult to distinguish and watermark because humans and models tend to produce similar continuations, while token changes can harm quality.

2. A simple proof of concept

The proof-of-concept watermark randomly partitions the vocabulary into green and red lists, then samples only green tokens. A z-test detects the resulting excess of green tokens, while robustness depends on how many tokens an attacker changes.

  • The hard rule can damage quality on low-entropy sequences, motivating soft watermarking that relaxes enforcement when the next token is nearly deterministic.
  • Generation applies the language model to obtain token probabilities, partitions the vocabulary into equal green and red lists, and samples only from green tokens.
  • The hard watermark seeds a pseudorandom red list from the preceding token, making the list reproducible during later detection.
  • Under the null hypothesis, natural text violates the random red-list rule about half the time, whereas watermarked text produces no violations.
  • With a detection threshold z > 4, the false-positive rate is 3 × 10^-5 and any watermarked sequence of at least 16 tokens is detected under the hard rule.
  • For T = 1000, modifying 200 tokens can still leave z ≈ 6.3 and p ≈ 10^-10; removing the watermark generally requires changing roughly one quarter of tokens or more.

3. A more sophisticated watermark

The soft watermark promotes green-list tokens by adding a constant to their logits rather than banning red-list tokens. This preserves likely low-entropy choices while biasing sampling toward green tokens when several alternatives are plausible.

  • The soft rule adds δ to green-list logits before softmax instead of strictly prohibiting red-list tokens.
  • When the next token is nearly deterministic, its dominant logit remains largest regardless of list membership, so the watermark has little effect on quality.
  • For high-entropy contexts with several plausible tokens, δ substantially shifts the sampling distribution toward the green list.
  • Detection remains the same as for the hard watermark: compute the z-statistic and reject the null hypothesis when it exceeds a chosen threshold.
  • The vocabulary is randomly partitioned into a green list of size γ|V| and a red list of size (1 − γ)|V| using a hash-seeded random generator.

5. Sample the next token, s(t), using the water-

The soft watermark’s detection sensitivity depends on sequence entropy, while its perplexity impact is concentrated at moderate entropy.

  • High-entropy sequences are detected with relatively few tokens, whereas low-entropy sequences require more tokens for detection.At z > 4, the false-positive rate is 3 × 10−5; unlike the hard watermark, soft-watermark sensitivity depends on entropy.
  • The soft watermark’s ability to detect synthetic text depends on the entropy of the generated sequence.
  • The watermark’s perplexity impact is limited to tokens with moderate entropy.Tokens with extremely high or low entropy have little or no perplexity impact under the soft watermark rule.

4. Analysis of the soft watermark

The analysis characterizes soft-watermark strength through spike entropy, bounds green-token counts and perplexity, and evaluates detection sensitivity empirically. Sensitivity is high for typical or higher-entropy generations but weaker for low-entropy or memorized text.

  • 4. Analysis of the soft watermark: Spike entropy measures how spread out a token probability distribution is and softens the count of entries exceeding 1/z.It is minimal when probability mass is concentrated at one location and maximal when the distribution is uniform.
  • 4. Analysis of the soft watermark: Theorem 4.2 predicts the number of green-list tokens in a watermarked sequence from average spike entropy, green-list fraction, and logit boost.
  • 4.1. Sensitivity of the watermark test: 98.6% sensitivity was obtained from a theoretical bound at average spike entropy S = 0.807, while the empirical mean yielded a 5.3 × 10−7 type-II error approximation.The empirical-mean estimate is described as realistic but not a rigorous lower bound.
  • 4.1. Sensitivity of the watermark test: 98.4% of multinomial generations and 99.6% of four-way beam-search generations were detected at the z = 4 threshold.Among generations above the 25th spike-entropy percentile, detection reached 100%.
  • 4.1. Sensitivity of the watermark test: Low-entropy sequences, often involving memorized or templated text, are the primary failure cases for watermark detection.Memorization can produce near-copies of human text, while constrained logits limit token-choice variation.
  • 4. Analysis of the soft watermark: Repeated n-grams can inflate green-token counts in human text and create false positives, but skipping repeats can preserve sensitivity.The paper also proposes increasing the PRNG window length as a remedy.
  • 4. Analysis of the soft watermark: The soft watermark has little perplexity impact at extremely high or low entropy, but does affect perplexity at moderate entropy.Uniform distributions leave sampling effectively uniform, while concentrated distributions make the watermark rule ineffective.

5. Private Watermarking

Private watermarking hides the key behind a secure API, making watermark removal harder but making detection dependent on monitored API access. Robust variants limit attack amplification, while multiple keys increase brute-force difficulty at a small power cost.

  • 5. Private Watermarking: Private watermarking uses a secret random key and secure API, increasing removal difficulty when attackers cannot identify the red list.Detection then requires the same secure API, whose queries must be monitored against repeated variants.
  • 5. Private Watermarking: Naive private watermarking faces a trade-off between brute-force resistance and robustness when the context window grows.Large windows make red lists harder to enumerate but increase downstream attack amplification.
  • 5. Private Watermarking: Algorithm 3 searches candidate tokens in descending logit order until the self-hash red-list condition is satisfied, or accepts the highest-logit red token after a δ threshold is exceeded.
  • 5. Private Watermarking: Algorithm 3 changes the watermark at position t with probability only 1/h when one prior token changes, avoiding attack amplification.In expectation, changing one token produces one additional red-list token.
  • 5. Private Watermarking: Using multiple hidden keys increases brute-force difficulty, while detection runs one test per key and incurs a minor power decrease from multiple-hypothesis correction.

6. Experiments

Experiments with OPT-1.3B evaluate watermark strength, detection errors, and the tradeoff between detectability and text quality under multinomial and beam-search decoding.

  • Experimental setup: OPT-1.3B experiments measure watermark strength using type-I errors for human text and type-II errors for undetected watermarked text.Generations use C4-derived prompts and a standard Hugging Face implementation.
  • Watermark strength vs text quality: γ = .1 is pareto-optimal for trading off watermark strength against perplexity across 500 ± 10 sequences of length T = 200 ± 5.
  • Beam search: 8-beam search produces an almost vertical strength–accuracy tradeoff, achieving strong watermarking with very little perplexity cost.The interaction is described as synergistic with the soft watermarking rule.
  • Watermark strength vs number of tokens: An average z-score greater than 5 is achieved for as few as 35 tokens with δ = 2 under 8-way beam search.The reported curves vary T, δ, and γ under multinomial sampling and beam search.
  • Detection performance: Higher δ values strengthen ROC performance, and beam search captures slightly more AUC than multinomial sampling at the same δ.
  • Detection performance: At z = 4, soft watermarks with δ = 2.0 incur at most 1.6% type-II error, while δ = 10.0 and γ = 0.25 produce no type-II errors.Each table row averages approximately 500 generated sequences of length T = 200 ± 5, with at most one observed type-I error per run.

7. Attacking the watermark

The paper examines insertion, deletion, substitution, paraphrasing, tokenization, Unicode, and generative attacks against the watermark. Stronger attacks can degrade detection, but often impose quality, cost, or generation-efficiency penalties.

  • Attack taxonomy: Insertion, deletion, and substitution attacks can alter downstream red-list computation, while deletion also wastes tokens and may reduce effective context width.
  • Paraphrasing attacks: Manual paraphrasing lies outside the intended threat model, and a few unmodified sentences in longer essays can still trigger statistically significant detection.
  • Paraphrasing attacks: Automated paraphrasing reduces both watermark strength and text fluency when performed by a weaker language model.
  • Tokenization and Unicode attacks: Homoglyphs and zero-width Unicode characters can disrupt tokenization or hashing, so canonicalization is required to maintain security.Homoglyph substitutions can expand a word from two tokens to nine in the cited example.
  • Generative attacks: Generative attacks such as inserting emojis or predictable character substitutions can randomize subsequent red lists and defeat watermarks, but may reduce language-model capability.These attacks require a strong language model that follows the prompted rule without quality loss.
  • Mitigations: Instruction finetuning with negative examples is proposed as a defense against prompts that induce predictable output transformations.
  • T5 span attack: The T5 span attack reduces watermark strength by only 0.01 AUC at ε = 0.1, whereas ε = 0.3 increases average perplexity by 3× and requires more model calls.The attack iteratively replaces masked words using T5-Large until reaching its replacement budget or iteration limit.

8. Related Work

Related work spans classical text watermarking, neural steganography, post-hoc detection, and model-parameter watermarking. The paper positions its approach around model-independent verification, low false positives, and minimal restriction of generation.

  • Prior watermarking: Classical text watermarking was difficult because text is discrete, while continuous-valued data supports more established watermarking strategies.
  • Prior watermarking: Earlier rule-based watermarks often significantly degraded text quality because classical language models had limited flexibility.
  • Neural steganography: Neural steganography methods can encode messages by partitioning a language-model vocabulary, whereas Meteor requires sender and receiver synchronization on a shared generative model.
  • Parameter watermarking: Unlike output watermarking, parameter watermarking defends against model stealing by implanting detectable behaviors through finetuning.
  • Positioning: The proposed watermark is designed to avoid model access during decoding while remaining minimally restrictive, theoretically analyzable, and applicable to natural text generation.
  • Post-hoc detection: Post-hoc detectors rely on measurable differences between machine and human text, but the paper notes that this margin may become small as language models improve.
  • Positioning: The proposed detection scheme is designed so false positives are statistically improbable regardless of individual human writing patterns.

9. Conclusion

The conclusion presents the watermark as a practical, model-independent verification method that can be retrofitted without retraining, while identifying implementation and robustness questions that remain open.

  • Conclusion: The watermark is computationally simple to verify, has statistically improbable false positives, and degrades gracefully under attack.
  • Conclusion: Existing next-token sampling models can be retrofitted with the watermark without retraining.
  • Conclusion: The detector's z-statistic depends on γ and the green-list hash, not on δ or the rule used to enforce green-list preference.This permits context-specific δ choices or enforcement rules for different text types.
  • Open questions: Open questions include robust and provably optimal hashing, streaming detection, detection inside longer unwatermarked text, and tighter sensitivity bounds for large δ and small γ.

A.1. Sample Outputs

The appendix presents representative generations, threat-model assumptions, detection behavior, and possible extensions of the watermarking framework. It reports strong empirical detection while identifying low-entropy text as a source of false negatives and discussing multiple or selective watermarks.

  • Threat model: The threat model considers attackers who know the watermark implementation and, in public mode, the hashing scheme and initial seed.Private-mode attackers lack the pseudorandom-function key; relevant attacks must preserve reasonable language quality.
  • Experimental setup: The watermark is evaluated with multinomial sampling at temperature 0.7, and its empirical strength is compared with a theoretical bound across δ values.The bound is described as tight for smaller δ but conservative for larger δ.
  • Extensions: Multiple watermarks can combine public verification with private testing, while selective watermarking can increase δ during suspicious API usage.The combined setup is described as useful for detecting attacks that attempt to remove a public watermark.
  • Detection and robustness: Low-entropy sequences can produce false negatives because they cannot be substantially altered without degrading text quality.The hypothesis-test reference table distinguishes improbable false positives from naturally occurring false negatives in low-entropy text.

E. Proof of Theorem 4.2

The proof analyzes how randomly partitioning vocabulary tokens and boosting green-list logits changes the probability of sampling green tokens. It derives entropy-dependent probability and variance bounds, which are compared with empirical sensitivity and attack results.

  • Entropy dependence: The analysis relates green-list sampling to token entropy: high entropy permits aggressive green-list use, whereas low entropy makes red-list choices more likely.The theorem analysis assumes uniformly random red lists and multinomial sampling, differing from the pseudorandom lists used in practice.
  • Probability bound: Boosting green-list logits by δ in a random partition yields a watermarked distribution with green-list size γN and α = exp(δ).The proof uses the resulting distribution to bound the probability of sampling green-list tokens.
  • Empirical comparison: Theoretical sensitivity is quite tightly bounded for smaller δ, while the bound underestimates watermark sensitivity for larger δ.Figure 7 compares empirical green-list fractions with the bound predicted by Theorem 4.2.
  • Empirical comparison: Higher δ values improve ROC performance, and 8-way beam search captures slightly more AUC than corresponding multinomial settings.Figures 8 and 9 report ROC curves and AUC values for the respective decoding schemes.
  • Probability bound: The probability of choosing a green-list token is strictly greater than the trivial lower bound γ for finite logits.The proof establishes that the bound is never vacuous.
  • Sequence-level bound: The expected number of green-list tokens is obtained by summing single-token probability bounds across the sequence.Variance is then bounded by treating token indicators as independent Bernoulli variables with non-identical success probabilities.

F. Proof of Proposition 4.3

The proposition’s supporting material evaluates perplexity and watermark behavior in knowledge-intensive question answering. It emphasizes that low-entropy, highly constrained answers are a difficult and non-deployment-like setting for watermarking.

  • Evaluation: TriviaQA performance is measured with Exact Match and whitespace-tokenized F1 against each question’s answer aliases.The table covers 50,000 zero-shot validation samples from unfiltered TriviaQA.
  • Entropy and factuality: Soft watermarking adapts to output entropy, minimally affecting highly confident predictions while influencing choices more when the model is uncertain.The described behavior motivates evaluating watermark impact in knowledge-intensive question answering.
  • Entropy and factuality: The TriviaQA experiment tests whether short, low-entropy answers retain task performance despite producing weak detection statistics.The evaluation uses a model with reasonable closed-book question-answering competence.
  • Limitations: The five-to-ten-token greedy-decoding setup is considered a worst-case estimate and is not expected to represent a deployment scenario.The authors leave broader studies of watermarking effects on factuality for future research.
Loading 2301.10226v4…