Source-linked AI summary
On the Reliability of Watermarks for Large Language Models
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, Tom Goldstein
TL;DR
The paper asks whether watermarks remain reliable after machine-generated text is paraphrased, rewritten, or mixed into longer human-written documents. It re-evaluates watermark generation and detection under these realistic modifications and compares watermarking with other detectors. Watermarks remain detectable under these attacks, and the study finds watermarking more robust than the compared post-hoc methods as text length increases.
Problem
The paper investigates whether watermarking can reliably identify machine-generated text after humans or language models modify it or mix it into larger documents.
Method
The study empirically evaluates watermark generation and detection under machine paraphrasing, human rewriting, copy-paste embedding, and comparisons with other detection approaches.
Results
Watermarking remains detectable after strong paraphrasing and human rewriting, and is more robust than other post-hoc detection methods across the studied scenarios.
Takeaways & Limitations
Watermark reliability is best characterized by how detection scales with observed text length, with favorable sample complexity supporting reliable detection in realistic settings.
Takeaways & Limitations
The original LeftHash scheme can expose its greenlist rules to third-party observers, motivating more secure hashing schemes for API-based deployment.
Abstract
from arXiv · showhide
As LLMs become commonplace, machine-generated text has the potential to flood the internet with spam, social media bots, and valueless content. Watermarking is a simple and effective strategy for mitigating such harms by enabling the detection and documentation of LLM-generated text. Yet a crucial question remains: How reliable is watermarking in realistic settings in the wild? There, watermarked text may be modified to suit a user's needs, or entirely rewritten to avoid detection. We study the robustness of watermarked text after it is re-written by humans, paraphrased by a non-watermarked LLM, or mixed into a longer hand-written document. We find that watermarks remain detectable even after human and machine paraphrasing. While these attacks dilute the strength of the watermark, paraphrases are statistically likely to leak n-grams or even longer fragments of the original text, resulting in high-confidence detections when enough tokens are observed. For example, after strong human paraphrasing the watermark is detectable after observing 800 tokens on average, when setting a 1e-5 false positive rate. We also consider a range of new detection schemes that are sensitive to short spans of watermarked text embedded inside a large document, and we compare the robustness of watermarking to other kinds of detectors.
1 INTRODUCTION
The paper examines whether watermarking remains reliable when machine-generated text is paraphrased, rewritten, mixed with human text, or embedded in larger documents. It finds that watermark detection remains highly reliable when enough tokens are observed.
- Watermarking marks generated text imperceptibly, enabling detectors to distinguish machine-generated from human-written text with high accuracy on relatively short fragments.
- Realistic modifications include mixing synthetic text into larger documents, revising or rephrasing parts, and using other language models to rephrase entire passages.
- ROC-AUC remains > 0.85 at T = 200 tokens and > 0.9 at T = 600 tokens after paraphrasing by GPT-3.5 and purpose-built models.
- A 600-token human-written passage containing 150 watermarked tokens achieves AUC above 0.95 under the Copy-Paste scenario.
- After human rewriting intended to remove the watermark, detection usually remains possible after about 800 observed tokens at a 1e−5 false positive rate.
- The study evaluates watermark reliability by measuring required machine-generated text and behavior as sequence length increases, rather than relying only on accuracy for one text distribution.
2 AN OVERVIEW OF MACHINE-GENERATED TEXT DETECTION
Machine-generated text detection includes post-hoc systems that analyze completed text and proactive systems that require intervention during generation. Proactive approaches are generally more robust but require adoption by the model owner.
- Post-hoc detectors operate without interaction during text generation, whereas proactive detectors require an action during generation.
- Watermarking requires the model owner to embed a hidden signal into outgoing text, placing it among proactive detection approaches.
- Statistical outlier detectors use signals including entropy, perplexity, n-gram frequencies, or differences in model-assigned probabilities for nearby text sequences.
- Retrieval detection stores all text generated by a model and later matches samples against that database.
- Retrieval can remain reliable under broad modifications such as strong paraphrases, but it requires action by the model owner.
3 HOW TO IMPROVE WATERMARK RELIABILITY?
The paper improves watermark reliability through design choices for generation, hashing, and detection. It introduces SelfHash and WinMax to make watermarking harder to discover and better suited to detecting embedded watermarked spans.
- A watermark combines a hash, context width h, greenlist fraction γ, and logit bias δ, then detects the signal by recomputing greenlists and counting greenlist token positions.
- Sampling from modified logits biases generation toward greenlisted tokens, creating the statistical pattern used for detection.
- A large z-score indicates that a sequence is likely to be watermarked.
- Improved hashing schemes: LeftHash uses one preceding token, but this allows an observer to infer greenlists from unlikely subsequent words when the scheme is hidden behind an API.
- Improved hashing schemes: SelfHash increases effective context width by including the token being generated and uses h = 4 with a minimum-based seed-selection method.
- Windowed detection: WinMax searches windows within a document because the global z-score is weakened when unwatermarked text surrounds a watermarked sub-region.
4 EVALUATING WATERMARKING IN THE WILD
The study evaluates watermark reliability under machine paraphrasing, copy-paste insertion, and human rewriting, emphasizing detection as a function of observed text length. Watermarks weaken under attack but recover with more tokens, and outperform alternative detectors particularly for copy-paste attacks.
- Experimental scope: Watermark robustness is tested on embedded, paraphrased, and human-rewritten text under a threat model where users know a watermark exists but not its parameters.The experiments use controlled token lengths and publicly available language models, including GPT-3.5-turbo and Dipper.
- Machine paraphrasing: Unattacked LeftHash and SelfHash detection exceed 0.999 AUC at the evaluated token lengths.
- Machine paraphrasing: At 600 tokens, GPT and Dipper paraphrasing reduce AUC by less than 0.1 points while shortening outputs to approximately 300 and 500 tokens, respectively.
- Copy-paste attacks: At 600 tokens, copy-paste attacks retaining 25% watermark text reduce AUC below 0.85, though detectability continues to grow with text length.
- Human paraphrasing: After strong human paraphrasing, watermark evidence mounts and writers are clearly detected on average after 800 tokens.Human paraphrases exceeded the expert-quality P-SP threshold of 0.7.
- Comparative detection: Watermarking outperforms Retrieval and RADAR under copy-paste attacks, while Retrieval matches or exceeds watermarking on GPT and Dipper paraphrases.Under strong copy-paste attacks, non-watermarking methods can decline as surrounding unwatermarked text increasingly dominates.
5 CONCLUSIONS
The conclusion presents watermarking as the most reliable approach studied across realistic text modifications and emphasizes reliability as a function of text length.
- Across strong machine paraphrasing, human rewriting, and embedded-text scenarios, watermarking is the most reliable approach evaluated.
- Even human writers cannot reliably remove watermarks when detection uses 1000 words.
- Watermark reliability improves with text length, unlike retrieval and loss-based detection, which show no strong improvement as text grows.
6 REPRODUCIBILITY STATEMENT
The paper reports that its experimental setup is documented to support reproducibility and extension.
- All key experimental details are provided, with additional setup information referenced in the Appendix.
- The models and data sources are publicly available, and the implementation uses commodity CPUs and GPUs with open source software.
7 ETHICS STATEMENT AND DISCUSSION OF SOCIETAL IMPACTS
The discussion describes watermarking as a promising harm-reduction technology while stressing ethical oversight, false-positive risks, and the limits of adversarial robustness.
- Ethics: The human study received Exempt status from the Institutional Review Board and documents its setup, evaluation procedure, and participant compensation.
- Societal implications: Watermarking is presented as reliable in everyday scenarios involving human or model modification and insertion into human-written text.
- Limitations: Sufficiently motivated attackers can break watermarks using generative attacks, so the reported reliability is not universal.
- Deployment considerations: Deployment requires weighing low false-positive rates against detection sensitivity because false positives can be extremely harmful.
A APPENDIX
The appendix supplements the main results with extended visualizations, ablation studies, and additional methodological and experimental details.
- The appendix provides extended visualizations that supplement the results presented in the main body.
- It includes ablation studies to examine additional aspects of the paper’s experimental findings.
- Additional methodological and experimental details are organized through a subsection table of contents.
1. What about the White-Box Setting?
Human–machine text separation can use post-hoc detectors or proactive systems. Proactive systems are generally more robust but require adoption by the model owner.
- Text detection approaches broadly divide into post-hoc systems and proactive systems.
8. GPT Attack Prompt Ablation
The appendix examines watermark security, quality, reliability, and generalization across attacks, hashing choices, metrics, models, and datasets. It finds that present-day paraphrasers may not remove watermarks reliably, while robustness depends on scheme, context width, entropy, and evaluation domain.
- A.1 WHAT ABOUT THE WHITE-BOX SETTING?: Without white-box access to the hashing scheme, removing a watermark requires avoiding all original n-grams, including recycled long words or phrases.
- A.2 RELATIONSHIP TO THEORETICAL RESULTS ON (IM)POSSIBILITY OF DETECTION: Watermarks can remain detectable after paraphrasing because current strong models may be insufficient against weaker model outputs.
- A.3 ON THE RELIABILITY OF P-VALUES: The analysis evaluates empirical false-positive and true-positive rates rather than relying on p-values, while testing the effect of repeated n-grams.
- A.4 UTILIZING BETTER QUALITY METRICS: P-SP was retained as the sole semantic-similarity metric because its comparisons subsumed the insights from MAUVE and coherence, while output-quality evaluation remains open.
- A.5 HASHING SCHEME EXTENDED ABLATION: Larger context widths make detection reliability depend substantially on the hashing scheme, motivating comparisons of alternative pseudo-random hash functions.
- A.5 HASHING SCHEME EXTENDED ABLATION: Skip and Min schemes can preserve robustness at larger context widths, with Min-SelfHash at h = 4 matching the diversity of Additive-LeftHash at h = 1.
- A.6 EFFECT OF WATERMARKS ON UTILITY: TRIVIAQA: The SelfHash watermark minimally impacts utility in short, low-entropy TriviaQA generations, where watermark strength is correspondingly low.
- A.7 DATASETS AND MODELS ABLATION: Github has lower watermark scores because restrictive code syntax reduces next-token entropy, while post-paraphrase detectability generalizes better to domains with similar syntactic flexibility.