PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Adaptive Attacks against Watermarks for Language Models

Abdulrahman Diaa, Toluwani Aremu, Nils Lukas

arXiv:2410.02440v2cs.CRcs.AI

TL;DR

Existing watermark robustness tests overlook attackers who know watermarking algorithms, leaving a gap in evaluating whether quality-preserving evasion is possible. The paper formulates robustness as an optimization objective and adaptively tunes paraphrasers, finding that they evade all surveyed watermarks at negligible quality impact, including against unseen watermarks. These results support testing watermark robustness with adaptively optimized attacks, while the study does not evaluate adaptive defenses or domains beyond text generation.

  • Problem

    Existing LLM watermark robustness evaluations test only non-adaptive attackers who lack knowledge of the watermarking algorithms.

  • Method

    The paper formulates watermark robustness as an objective function and uses preference-based optimization to adaptively tune paraphrasing attacks against known watermark algorithms.

  • Results

    Adaptive attacks evade all surveyed watermarks with negligible quality impact, achieving at least 96.7% evasion at costs no greater than $10.

  • Takeaways & Limitations

    Watermark robustness testing should include adaptively optimized attacks, which can remain effective against unseen watermarks even in non-adaptive evaluation.

  • Takeaways & Limitations

    The study does not evaluate adaptive defenses and focuses exclusively on text generation rather than other domains or interacting defenses.

Abstract

from arXiv · show

Large Language Models (LLMs) can be misused to spread unwanted content at scale. Content watermarking deters misuse by hiding messages in content, enabling its detection using a secret watermarking key. Robustness is a core security property, stating that evading detection requires (significant) degradation of the content's quality. Many LLM watermarking methods have been proposed, but robustness is tested only against non-adaptive attackers who lack knowledge of the watermarking method and can find only suboptimal attacks. We formulate watermark robustness as an objective function and use preference-based optimization to tune adaptive attacks against the specific watermarking method. Our evaluation shows that (i) adaptive attacks evade detection against all surveyed watermarks, (ii) training against any watermark succeeds in evading unseen watermarks, and (iii) optimization-based attacks are cost-effective. Our findings underscore the need to test robustness against adaptively tuned attacks. We release our adaptively optimized paraphrasers at https://github.com/nilslukas/ada-wm-evasion.

1. Introduction

LLM watermarking can help detect harmful generated content, but existing robustness tests assume attackers lack knowledge of watermarking algorithms. This paper proposes adaptively optimized attacks and reports high-evasion, quality-preserving results across surveyed watermarks.

  • Motivation: Adaptive attacks target watermarking because hidden messages enable detection of generated content while perturbations can preserve quality.Watermarking is intended to reduce misuse such as spam, misinformation, and phishing by supporting detection.
  • Motivation: Existing watermark robustness evaluations use non-adaptive attackers, leaving methods vulnerable when watermarking algorithms are known.Robustness requires that evading detection significantly degrades text quality, but prior tests do not identify the strongest method-specific attack.
  • Method: The paper adaptively optimizes paraphrasing attacks against known watermarking algorithms through curated preference datasets.The approach addresses discrete-text optimization and attackers’ limited computational resources.
  • Results: Over 96% evasion was achieved against every surveyed watermark with negligible impact on text quality using less than 7 GPU hours.The attacks remained Pareto optimal even when transferred to unseen watermarks in the non-adaptive setting.
  • Contributions: The contributions include small adaptive paraphrasers, transfer to unseen watermarks, and publicly released attack models for robustness research.The evaluated paraphrasers span 0.5–7 billion parameters, and attacks against models with 46× more parameters require less than seven GPU hours.

2. Background

The paper formalizes LLM generation, watermark verification, robustness, and attacker access. It studies no-box, offline attacks that know watermark algorithms but not secret keys or verification outputs.

  • LLM generation: LLMs model next-token probabilities with a neural network and generate text autoregressively, making reward optimization difficult in the discrete token domain.Direct backpropagation cannot pass through token sampling steps.
  • Watermarking: A watermark hides a message in generated text and later verifies it using a secret key and a p-value for rejecting chance agreement.The watermarking system comprises KEYGEN, EMBED, and VERIFY algorithms.
  • Watermark robustness: Robustness requires high-quality paraphrases to retain the watermark with high probability, formalized through verification and text-quality thresholds.The definition uses an (ε, δ)-robustness condition over randomized paraphrasing methods.
  • Evasion attacks: Adaptive attackers know KEYGEN, EMBED, and VERIFY algorithm descriptions, whereas non-adaptive attackers lack that knowledge.The study focuses on no-box, offline attacks in both adaptive and non-adaptive settings.
  • Surveyed methods: The evaluation covers four state-of-the-art watermarking methods using token-selection, green-list, bit-string, and inverse-sampling mechanisms.The methods are Exp, Dist-Shift, Binary, and Inverse; additional methods are evaluated in an appendix.

3. Threat Model

The threat model considers providers watermarking API-generated text and constrained attackers seeking undetectable, high-quality paraphrases. Attackers lack watermarked samples, verification access, and unlimited compute but may know watermark algorithms.

  • Deployment: The deployment scenario is a provider-controlled LLM API whose untrustworthy users generate harmful content without detection.The model is assumed to be accessible to many users through a black-box interface.
  • Provider capabilities and goals: The provider aims for a quality-preserving, robust watermark that detects generated text at a low false-positive rate.The provider controls model deployment, watermark embedding, and verification.
  • Attacker capabilities: Attackers are no-box and offline, know watermark algorithms, and lack secret keys, provider models, and watermarked samples.They use open-weight surrogate models and cannot train LLMs from scratch.
  • Attacker goals: The attacker seeks text without a watermark while maintaining high quality, evaluated with quality metrics and a provider verification threshold.Lower verification thresholds make successful evasion more likely.
  • Threat-model motivation: If inexpensive attacks succeed in this constrained setting, more capable attackers cannot be expected to face stronger protection from the watermark.The study reports that such attacks exist, are inexpensive, and require no access to watermarked samples.

4. Related Work

Prior LLM watermark attacks are non-adaptive, handcrafted, black-box, or online and often require watermarked samples or verification access. This work instead studies optimizable adaptive attacks against no-box, offline settings.

  • Prior attacks: The paper differs from prior LLM attacks by optimizing adaptive paraphrasers that remain effective against unseen watermarks without verification access or collected watermarked samples.Earlier approaches include non-adaptive DIPPER, handcrafted attacks, black-box attacks, and online verification attacks.
  • Positioning: Unlike impossibility results requiring quality and perturbation oracles, this approach finds a single-step evasion perturbation at computational cost no greater than $10.The paper emphasizes feasibility and efficiency against constrained attackers.

5. Conceptual Approach

The paper formulates watermark robustness as an objective for adaptively fine-tuning paraphrasers, using surrogate watermarking keys to optimize evasion while preserving quality. It then curates preference data with best-of-N sampling and uses preference optimization to train the attack.

  • 5. Conceptual Approach: The attacker uses an open-weight surrogate model to generate surrogate keys because the provider’s model parameters and random seeds are unknown.Optimization therefore averages over many random seeds and relies on knowledge of KEYGEN, EMBED, and VERIFY rather than the provider’s secret key.
  • 5.1. Robustness as an Objective Function: Adaptive attacks optimize paraphraser parameters for high-probability watermark evasion while preserving text quality against surrogate watermarked samples.The objective samples random seeds, messages, and queries, then evaluates paraphrases using surrogate watermarking keys and quality thresholds.
  • 5.1. Robustness as an Objective Function: Optimization is difficult because text is discrete, generation is not directly backpropagatable, and VERIFY may be nondifferentiable under limited compute.The paper identifies reinforcement learning as applicable but potentially compute-intensive and unstable.
  • 5.2. Preference Dataset Curation: Algorithm 1 samples known watermarking methods, queries, messages, and surrogate keys before partitioning paraphrases into positive and negative sets.For each positive sample, the procedure selects a negative sample and adds the resulting comparison to dataset D.
  • 5.2. Preference Dataset Curation: The method curates preference triplets by generating N paraphrases, labeling high-quality watermark-free outputs positive and watermark-retaining or low-quality outputs negative.Each selected positive is paired with a corresponding negative example and the original watermarked text in the preference dataset.

6. Evaluation

The evaluation measures collection cost, attack settings, and robustness across surveyed watermarks, models, and quality metrics. Adaptively optimized attacks achieve high evasion with preserved quality, transfer to unseen watermarks, and remain effective with substantially smaller attacker models.

  • 6.1. Preference Dataset Collection: Less than 5 GPU hours are required to curate 7,000 preference samples for Dist-Shift, with total attack costs below $10 USD at current AWS rates.Estimated curation time is approximately 1.5 GPU hours for Dist-Shift and 0.5 GPU hours for Inverse before quality and detection overhead.
  • 6.2. Ablation Studies: Adaptive and non-adaptive settings are compared across Llama2-13b and Llama3.1-70b targets, Llama2 and Qwen2.5 attackers, four watermark methods, hyperparameters, and false-positive thresholds.The ablation design includes Exp, Dist-Shift, Inverse, and Binary watermarking methods.
  • 6. Evaluation: The evaluation defines evasion as failure to retain a watermark and reports LLM-Judge as the main quality metric alongside LLM-CoT, LLM-Compare, Mauve, and perplexity.Unless otherwise specified, watermark retention is tested at p-value threshold ρ = 0.01.
  • 6.3. Experimental Results: Against the Exp watermark with a Llama3.1-70b provider target, adaptive paraphraser attacks using models of at most 7b parameters are Pareto-optimal, despite being smaller than the 11b DIPPER model.The paper reports that adaptive attacks outperform much larger closed-source systems such as GPT-4o at evading watermark detection.
  • 6.3. Experimental Results: 97.3% evasion at 0.846 text quality is achieved by adaptively fine-tuned Qwen2.5-7b, while adaptive attacks overall reach at least 96.6% evasion.Training on one watermark transfers effectively to unseen watermarks, although the attack trained on all four performs slightly worse than the Exp-trained attack.
  • 6.3. Experimental Results: Optimized non-adaptive attacks reach 90.9% to 97.6% evasion, exceeding vanilla model-based paraphrasers while maintaining text quality near or above ChatGPT3.5.Baseline contraction, swapping, and synonym attacks achieve less than 20% evasion, whereas vanilla Llama2-7b and ChatGPT3.5 reach 61.8% and 86.1%, respectively.
  • 6.3. Experimental Results: Increasing Qwen2.5 paraphraser size from 0.5b to 7b strictly improves text quality at similar evasion rates, while smaller target models permit higher evasion and quality.The same qualitative pattern is reported across target-model evaluations.
  • 6.3. Experimental Results: Adaptively tuned paraphrasing raises expected detection p-values substantially relative to non-optimized paraphrasing as sequence length increases.At around 500 tokens, expected p-values are 0.10 for non-optimized Llama2-7b and 0.4 for the adaptively tuned model.

7. Discussion

The discussion shows that adaptively optimized, resource-constrained attacks undermine current watermark robustness, including against larger models and unseen watermarks. It also identifies learnable attack signals, reports low computational costs, and notes important evaluation boundaries.

  • Effectiveness of Adaptive Attacks: Less than seven GPU hours lets Llama2-7b evade detection from larger Llama3.1-70b models, while the attacks also transfer to unseen watermarks.The authors frame these results as evidence that current robustness claims do not hold against restricted attackers.
  • Analysis: Adaptive attacks exploit token-level watermark structure by avoiding repeated sequences, preserving quality through targeted replacements, and calibrating edit distance and lexical diversity.These signals are presented as explanations for both adaptive effectiveness and cross-watermark transfer.
  • Attack Runtime: Dataset curation takes less than 5 GPU hours and model optimization about 2 GPU hours, costing under $10 with on-demand GPU pricing.The reported cost applies to curating 7,000 samples and optimizing a Llama2-7b model at 16-bit precision.
  • Restricted Attackers: The study focuses on restricted attackers with limited resources and no secret-key or watermarked-sample access, while online access to VERIFY could further improve attack effectiveness.The reported robustness conclusions therefore concern offline, resource-constrained settings rather than the stronger online attacker.
  • Limitations: The authors did not evaluate adaptive defenses, other generation domains, or interactions between watermarking and defenses such as safety alignment and content filtering.They also caution that LLM-as-a-Judge is noisy and may not align with human judgment.

8. Conclusion

The conclusion finds that current LLM watermarks are not robust against adaptively optimized attacks. Resource-constrained attackers can evade detection cheaply with smaller open-weight models, motivating adaptive attackers in future robustness testing.

  • 8. Conclusion: At least 96.7% evasion is achieved for at most $10 with models 46× smaller than the provider’s model.The attacks also outperform surveyed alternatives in non-adaptive settings, including paraphrasing with larger models such as GPT4o.
  • 8. Conclusion: Future defenses must consider adaptive attackers when testing watermark robustness.This recommendation follows the conclusion that existing security claims fail against resource-constrained attackers.

Impact Statement

The work highlights societal and responsible-deployment implications of adaptive attacks that undermine current LLM watermarking methods. It releases the research openly while acknowledging potential misuse and emphasizing the need for stronger content authentication.

  • Adaptive attacks using limited computational resources can undermine current LLM watermarking methods, with implications for responsible AI deployment.
  • Watermarking weaknesses matter because major AI providers increasingly use watermarking to address LLM-generated misinformation and online spam.
  • The authors release their methods, findings, and code despite potential misuse, because current watermark deployments remain experimental.
  • The release aims to encourage more robust watermarking and content authentication methods that resist adaptive attacks.

A.1. Quality Metrics

The study evaluates text quality with multiple surrogate metrics because comprehensive human evaluation is impractical. These metrics compare watermarked or attacked responses against suitable baselines, with higher or lower values preferred depending on the metric.

  • Quality Metrics: Multiple surrogate metrics approximate human evaluation because aggregating scores from many human evaluators is impractical.The benchmark includes LLM-Judge, LLM-CoT, LLM-Compare, Mauve, and perplexity.
  • Quality Metrics: LLM-Judge scores response accuracy, detail, and typographical, grammatical, and lexical correctness on a 100-point scale, where higher is better.Llama3-8B-Instruct evaluates responses greedily for deterministic results.
  • Quality Metrics: LLM-CoT uses chain-of-thought reasoning to evaluate response quality across accuracy, detail, and typographical, grammatical, and lexical correctness.Its rating is also produced by Llama3-8B-Instruct on a 1–100 scale.
  • Quality Metrics: LLM-Comparator compares attacked or watermarked responses with non-watermarked baselines, encoding the outcome as 0, 0.5, or 1.A score of 0 means the non-watermarked response is better, 0.5 indicates a tie, and 1 means the attacked or watermarked response is better.
  • Quality Metrics: Table 3 compares median text-quality metrics for vanilla Llama2-7b with the strongest adaptive and non-adaptive attacks, all capped at 7b parameters.The table is designed to assess quality while controlling the maximum attacker model size.

A.2. Prompt-set Curation

This appendix curates prompts, preference data, and evaluation settings for testing adaptive paraphrasing attacks across watermarks. It also reports ablations and analyses showing that optimized attacks preserve quality while reducing watermark evidence through lower overlap and higher token edits.

  • A.2. Prompt-set Curation: The evaluation uses 296 prompts, while training uses 1,000 synthetic, non-overlapping prompts spanning diverse topics.Training prompts are generated by combining topic titles produced by ChatGPT-4.
  • A.3. Preference-data Curation: Each training prompt yields watermarked outputs, 16 paraphrases per input, and preference pairs filtered by Algorithm 1.Larger models produce more successful paraphrases, although every model receives the same number of paraphrases.
  • A.3. Preference-data Curation: Expected evasion increases with more paraphrases but saturates near 16, while larger Qwen2.5 models generally achieve higher rates at equal budgets.The 1.5b model is an exception, outperforming the 3b model under the same paraphrase count.
  • A.4. Baseline Testing against other Watermarks: Against newer watermarks, the Exp-trained attack transfers to unseen methods, while Unigram receives a separately tuned attack under a 256-token evaluation limit.Only the adaptive attacks are on the Pareto front for SynthID, Exp, and Unigram; GPT-4o appears on the front only for SIR and KGW.
  • A.5. Additional Statistics: Adaptive paraphrasers produce slightly shorter texts, but their GPT-Judge quality remains similar to unattacked and baseline paraphrasing methods.Word substitution and deletion substantially reduce quality, unlike the optimized paraphrasers.
  • A.5. Additional Statistics: Adaptive paraphrasers lower watermark scores most strongly, typically centering near -1 versus approximately 5 for unattacked watermarked text and approximately 1 for untuned paraphrases.Top-token distributions remain broadly similar across GPT-4o, baseline Qwen2.5-3b, and tuned Qwen2.5-3b paraphrases.
  • A.7. Detailed Textual Analysis: The adaptive paraphraser achieves the lowest N-gram overlap and combines higher token edit distance with higher evasion than baseline Qwen2.5-3b.These patterns suggest optimization targets token-level differences while preserving text quality.
  • A.10. Additional Ablation Studies: Across Dist-Shift and Exp, adaptive attacks outperform surveyed attacks at every tested false-positive threshold, reaching 80% and 77% evasion at 10% FPR.The evaluation uses thresholds ρ ∈ {0.01, 0.025, 0.05, 0.075, 0.1}.
Loading 2410.02440v2…