PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Learnability of Watermarks for Language Models

Chenchen Gu, Xiang Lisa Li, Percy Liang, Tatsunori Hashimoto

arXiv:2312.04469v3cs.LGcs.CLcs.CR

TL;DR

Watermarking can statistically detect model-generated text, but existing methods depend on specialized decoding, leaving open whether models can learn watermarks directly. The paper uses watermark distillation to train students to imitate decoding-watermarked teachers. Models learn watermarks with high detectability, although low-distortion settings are harder to learn and fine-tuning on normal text can remove the capability.

  • Problem

    Existing watermarking relies on specialized decoding, and the paper investigates whether language models can instead learn to generate watermarked text directly.

    Source: §1 · p. 1

  • Method

    Watermark distillation trains a student to imitate a teacher model that applies decoding-based watermarking, using distribution matching or watermarked samples.

    Source: §1 · p. 2

  • Results

    Higher-distortion settings were learned by both methods with median p-values below 0.0001; lower-distortion watermarks were harder and less sample efficient, but remained detectable relative to the 0.5 baseline.

    Source: §1 · p. 2, §9 · p. 10

  • Takeaways & Limitations

    Learned watermarks work with standard decoding, a step toward watermarking open models, while their learnability also raises the possibility of spoofing attacks.

    Source: §1 · p. 2, §6 · p. 8, §9 · p. 10

    Fine-tuning on normal, non-watermarked text can remove weights-based watermarking capabilities, leaving robustness to fine-tuning as an unresolved challenge.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Watermarking of language model outputs enables statistical detection of model-generated text, which can mitigate harms and misuses of language models. Existing watermarking strategies operate by altering the decoder of an existing language model. In this paper, we ask whether language models can directly learn to generate watermarked text, which would have significant implications for the real-world deployment of watermarks. First, learned watermarks could be used to build open models that naturally generate watermarked text, enabling watermarking for open models, where users can control the decoding procedure. Second, if watermarking is used to determine the provenance of generated text, an adversary can hurt the reputation of a victim model by spoofing its watermark and generating damaging watermarked text. To investigate the learnability of watermarks, we propose watermark distillation, which trains a student model to behave like a teacher model that uses decoding-based watermarking. We test our approach on three decoding-based watermarking strategies and various hyperparameter settings, finding that models can learn to generate watermarked text with high detectability. We also find limitations to learnability, including the loss of watermarking capabilities under fine-tuning on normal text and high sample complexity when learning low-distortion watermarks.

1 INTRODUCTION

The paper asks whether models can learn watermarks that previously required specialized decoding, with implications for open-model watermarking and spoofing. Watermark distillation achieves high detectability, but low-distortion watermarks are harder to learn and fine-tuning on normal text can remove learned capabilities.

1 INTRODUCTION

  • Little was known about whether models could learn watermarks, despite their proposed use for detecting generated text and tracing its model provenance.The cited provenance use case is attributed to Abdelnabi & Fritz (2021) and Kuditipudi et al. (2023).Source: §1 · p. 1
  • Weights-based watermarking lets open models generate watermarked text with standard decoding, rather than relying on a specialized decoder.The paper also reports that fine-tuning on normal text can remove this capability.Source: §1 · p. 1, Figure 1 · p. 2
  • Fine-tuning weights-based watermarks on normal text can remove the watermarking capability, leaving robustness to fine-tuning as an open challenge.This is a limitation for the open-model setting discussed in the introduction.Source: §1 · p. 2
  • Learned watermarks can enable spoofing: an attacker can use a victim model’s signal to attribute damaging text to that model.The paper demonstrates a proof-of-concept attack on an instruction-following chat model.Source: §1 · p. 2
  • Watermark distillation trains a student to imitate a teacher using decoding-based watermarking, either by matching next-token distributions or fine-tuning on watermarked samples.The paper evaluates both approaches across three decoding-based strategies.Source: §1 · p. 2
  • Higher-distortion settings were learned by both distillation methods with median p-values below 0.0001; lower-distortion settings were harder but remained detectable relative to the 0.5 baseline.The experiments cover KGW, Aar, and KTH watermarks.Source: §1 · p. 2

2 BACKGROUND AND NOTATION: DECODING-BASED WATERMARKING

The section defines decoding-based watermarks as modifications to a language model’s next-token distribution that embed a detectable signal, with detection testing against a null hypothesis. It then describes three strategies: KGW biases green-list tokens, Aar favors high hash-score tokens, and KTH aligns text to a key sequence.

2 BACKGROUND AND NOTATION: DECODING-BASED WATERMARKING

  • A decoding-based watermark modifies the model’s next-token distribution using a key; its detector tests whether text is independent of that keyed strategy and reports a p-value.A sufficiently small p-value rejects the null hypothesis and detects the text as watermarked.Source: §2 · pp. 2–3

2.1 KGW: GREEN LIST BIAS

  • KGW hashes preceding tokens to split the vocabulary into green and red lists, then adds bias δ to green-list logits; larger δ increases detectability and distortion.The green-list proportion is controlled by γ, and k determines how many preceding tokens are hashed.Source: §2.1 · p. 3

2.2 AAR: BOOSTING CONTINUOUS HASH SCORES

  • Aar hashes the previous k tokens to score vocabulary items and selects using both hash score and original probability; larger k reduces repetition but weakens edit robustness.High hash scores in observed text correspond to a stronger detected signal.Source: §2.2 · p. 3

2.3 KTH: ROBUST SEQUENCE ALIGNMENT

  • KTH uses scores from a key sequence to select tokens, with possible sequence shifts; its detector aligns text to the key using a minimum-Levenshtein-distance statistic.A lower, more negative statistic indicates a stronger watermark, and the p-value resolution depends on the reference sample count.Source: §2.3 · p. 4

3 METHODS

The methods train a student model to generate watermarked text naturally from its weights, rather than applying a special decoding-time algorithm. The paper presents logit-based distillation, which matches teacher distributions, and sampling-based distillation, which trains on teacher-generated watermarked text.

3 METHODS

  • The goal is a student whose sampling distribution generates watermarked text with detection p-values similar to a decoding-watermarked teacher, using only the student’s learned weights.The paper frames both proposed methods as watermark distillation.Source: §3 · p. 4

3.1 LOGIT-BASED WATERMARK DISTILLATION

  • Logit-based distillation minimizes mean KL divergence between the teacher’s watermark-adjusted next-token distribution and the student’s distribution over dataset prefixes.The teacher is frozen; the dataset need not contain watermarked text, but teacher and student need aligned tokenizers and vocabularies.Source: §3.1 · p. 4

3.2 SAMPLING-BASED WATERMARK DISTILLATION

  • Sampling-based distillation generates watermarked completions from prompts, then fine-tunes the student on those samples with standard language-model cross-entropy.Unlike logit-based distillation, teacher and student need not share a tokenizer or vocabulary, but generating the samples makes it less efficient.Source: §3.2 · p. 5

4 EXPERIMENTAL SETUP

The experimental setup compares logit-based and sampling-based distillation across three watermark strategies and varying distortion settings. It evaluates detectability and generation quality against decoding-watermarked teachers and an unwatermarked student baseline.

4.1 WATERMARKING STRATEGIES AND HYPERPARAMETERS

  • The experiments test KGW, Aar, and KTH across hyperparameter settings chosen to vary watermark-induced distortion.The tested settings include KGW k = {0, 1, 2} and δ = {1, 2}, Aar k = {2, 3, 4}, and KTH s = {1, 2, 4, 256}.Source: §4.1 · p. 5

4.2 TRAINING

  • Both distillation approaches are tested for each decoding-based watermarking strategy, primarily using Llama 2 7B as teacher and student.Sampling-based experiments also test a Llama 2 7B teacher with a Pythia 1.4B student; sampling distillation is described as fairly robust to domain shifts.Source: §4.2 · p. 5

4.3 EVALUATION AND METRICS

  • Evaluation uses 5,000 200-token completions from 50-token prompts, with standard sampling at temperature 1 for the main results.The prompts are drawn from the RealNewsLike subset of C4; additional datasets and decoding settings are evaluated separately.Source: §4.3 · p. 6
  • Detectability is measured by median detection p-values and AUROC, while generation quality uses perplexity and seq-rep-3 for repetition.For KTH, the median test statistic provides finer distinctions below the p-value lower bound; AUROC compares equal numbers of same-length model- and human-generated texts.Source: §4.3 · p. 6

4 EXPERIMENTAL SETUP

  • The comparisons place each distilled student against its decoding-watermarked teacher, with the base student serving as the generation-quality baseline.The intended comparison covers both watermark detectability and generation quality.Source: §4 · p. 5, §4.3 · p. 6

5 RESULTS

Watermark distillation learns higher-distortion watermarks with high detectability, while lower-distortion settings are harder and less sample-efficient to learn. Learned watermarks also remain detectable across decoding changes and moderate text edits, with limits to the generation-quality/detectability tradeoff.

5 RESULTS

  • Higher-distortion watermarks are successfully learned with small p-values and high detectability, while lower-distortion settings remain learnable but are harder and less sample-efficient.For lower-distortion settings, more training samples and steps reduce p-values, with no sign of convergence.Source: §5 · p. 6, Table 1 · p. 7, Figure 2 · p. 8
  • Watermark distillation achieves a less optimal generation-quality/detectability tradeoff than decoding-based watermarking across KGW, KTH, and Aar.For KGW and KTH, distillation has slightly to moderately higher perplexity and similar or larger p-values; for Aar, it has similar or lower seq-rep-3 but larger p-values.Source: §5 · p. 7
  • The better-performing distillation method varies by watermark: sampling-based distillation has higher detectability and lower perplexity than logit-based distillation on KTH, while logit-based distillation yields less repetitiveness on Aar.The authors hypothesize that training on complete watermarked sequences versus next-token predictions on prefixes may explain these differences.Source: §5 · p. 7

5.1 ROBUSTNESS TO CHANGES IN DECODING PARAMETERS

  • Weights-based watermarking retains small detection p-values across tested decoding algorithms and parameters, with smaller p-values as temperature or nucleus threshold decreases and the smallest under greedy decoding.The tested settings include nucleus sampling, temperature-based sampling, and greedy decoding.Source: Table 2 · p. 8, §5.1 · p. 8

5.2 ROBUSTNESS TO TEXT EDITS

  • All three watermark types are robust to moderate random edits of about 20–30% of tokens, while KTH is significantly more robust than KGW and Aar at edit proportions up to about 60–70%.The KTH comparison is consistent with findings by Kuditipudi et al. (2023).Source: §5.2 · p. 8

6 WATERMARKING FOR OPEN MODELS

Weights-based watermarking works with standard decoding algorithms, including when users change decoding parameters, and the paper reports robustness to moderate text edits. Further fine-tuning on normal text removes the watermark, although some open-model use cases may not require fine-tuning robustness.

6 WATERMARKING FOR OPEN MODELS

  • Weights-based watermarking works under standard decoding algorithms and remains robust to changes in decoding parameters, supporting it as a first step toward watermarking open models.Open-model users can change decoding algorithms, and their inference libraries may not implement specialized watermarking.Source: §6 · pp. 8–9
  • Weights-based watermarks are robust to moderate random edits, but further fine-tuning on normal, non-watermarked text removes the learned watermarking capability.The text-edit figure examines generations with varying proportions of randomly edited tokens; the fine-tuning result is shown for distilled Llama 2 7B models fine-tuned on OpenWebText.Source: §6 · p. 9, Figure 4 · p. 9, Figure 3 · p. 9
  • Weights-based watermarking may suit open models unlikely to be fine-tuned further, and it simplifies deployment by requiring only different model weights rather than a specialized decoder.The paper gives RLHF-trained instruction-following chat models as an example of open models unlikely to be fine-tuned by users.Source: §6 · p. 9

7 SPOOFING ATTACKS

The spoofing experiment tests whether an adversary can learn a victim model’s watermark from its outputs and use it on harmful requests. Harmful responses from the distilled adversary retained a detectable watermark, creating a risk of misattribution.

7 SPOOFING ATTACKS

  • Sampling-based watermark distillation fits this attack setting because it requires only generated samples from the victim model.The attack uses watermarked samples from Llama 2-Chat 7B after refusals are filtered, then fine-tunes Alpaca-7B.Source: §7 · p. 9
  • The proof of concept targets a safety-trained Llama 2-Chat 7B victim that tends to refuse harmful requests, training an adversary to answer them with its watermark.The victim uses KGW with k = 1, γ = 0.25, and δ = 2.Source: §7 · p. 9
  • The distilled adversary had a 71% harmful response rate, versus 0% for the victim and 80% for base Alpaca-7B.The experiment used HarmfulQ toxic questions and GPT-4 annotations.Source: §7 · p. 9
  • Among harmful adversary responses, the median watermark detection p-value was 0.002, making false attribution to the victim plausible.The median generation length was 593 tokens.Source: §7 · p. 10

8 RELATED WORK

The related work distinguishes text watermarking from post-hoc detection and API watermarking, and places this paper alongside work on watermark removal and spoofing attacks.

8 RELATED WORK

  • Post-hoc methods detect generated text without changing generation, whereas watermarking embeds a detectable signal by modifying the generation process.The approaches may also be used together.Source: §8 · p. 10
  • Text-watermarking research includes editing existing text and decoding-based methods, with semantic, multi-bit, and public/private-key variants.The passage also notes work detecting whether training data contained watermarked text.Source: §8 · p. 10
  • Prior spoofing attacks exploit KGW’s green-list bias (Sadasivan et al., 2023; Jovanović et al., 2024) or watermark robustness and public detection APIs (Pang et al., 2024).This paper studies spoofing through sampling-based watermark distillation.Source: §8 · p. 10
  • API watermarking detects whether a suspect model was distilled from a victim API, while text watermarking detects whether text was model-generated.The two approaches address different objects of detection.Source: §8 · p. 10

9 CONCLUSION

The paper finds that models can learn to generate watermarked text using standard decoding, while lower-distortion watermarks are harder to learn. It identifies fine-tuning robustness and spoofing defenses as open challenges.

9 CONCLUSION

  • Logit-based and sampling-based watermark distillation let models naturally generate watermarked text with standard decoding, but lower-distortion watermarks are harder and less sample-efficient to learn.These findings address a technical challenge toward watermarking open models and raise the possibility of spoofing attacks.Source: §9 · p. 10
  • Further fine-tuning robustness remains unresolved; the paper calls for work to make weights-based watermarks more robust to fine-tuning.The cited future-work discussion frames this as a challenge for robust watermarking of open models.Source: §9 · p. 10
  • Across 200-token slices of harmful responses, the median watermark detection p-value was 0.04.This is distinct from the median p-value reported across harmful responses in the main spoofing result.Source: §9 · p. 10

ETHICS STATEMENT

The paper argues that public awareness of spoofing limitations should temper trust in watermarks for assigning provenance or blame, while preserving their use for statistical detection of LM-generated text.

ETHICS STATEMENT

  • The authors argue that disclosure of spoofing risks can reduce harm if watermark detection is not treated as proof of a specific model’s authorship.The passage presents this as a reason to inform the public about spoofing and watermark limitations.Source: ETHICS STATEMENT · p. 11
  • Watermarking may still support statistical detection of LM-generated text for tasks such as identifying policy infractions.The passage gives finding policy violations as an example use.Source: ETHICS STATEMENT · p. 11
  • The stated risk is reputational harm when damaging text bearing a victim model’s watermark is mistaken for that model’s output.The passage specifically cautions against trusting watermarks to assign provenance or blame.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The appendix provides reproducibility resources, formal details for three decoding-based watermarking strategies, and training configurations for both distillation methods.

REPRODUCIBILITY STATEMENT

  • Code, reproduction scripts, and trained model weights are released for reproducing the experiments.The main experimental setup and evaluation procedure are described in §4; other experimental details are in the appendix.Source: REPRODUCIBILITY STATEMENT · p. 11

A ADDITIONAL DETAILS ON WATERMARKING STRATEGIES

A.1 KGW

  • KGW’s key-parameterized hash maps the previous k tokens to a green-list mask, and its detector evaluates the green-list count against a binomial reference distribution.For k = 0, the hash returns a fixed green list regardless of previous tokens.Source: §A.1 · p. 16

A.2 AAR

  • Aar hashes the previous k tokens to assign scores across the vocabulary, and its detector evaluates a statistic against a gamma reference distribution.The hash scores are uniformly distributed on [0, 1].Source: §A.2 · p. 16

A.3 KTH

  • KTH uses a sequence of random keys, permits evenly spaced key shifts across generations, and detects watermarks by aligning text to the key sequence.Its detection p-values are bounded below by 1/(T + 1), where T is the reference-distribution sample count.Source: §A.3 · pp. 16–17

B.1 LOGIT-BASED WATERMARK DISTILLATION TRAINING DETAILS

  • Logit-based distillation training uses Llama 2 7B on OpenWebText for 5,000 steps with 512-token sequences and a maximal learning rate of 1e-5.Each run took approximately six hours on four NVIDIA A100 80GB GPUs.Source: §B.1 · p. 17

B.2 SAMPLING-BASED WATERMARK DISTILLATION TRAINING DETAILS

  • Sampling-based distillation fine-tunes Llama 2 7B for one epoch of 5,000 steps on watermarked samples, using 256-token sequences and a maximal learning rate of 1e-5.Each run took approximately four hours on four NVIDIA A100 80GB GPUs.Source: §B.2 · p. 17

C PYTHIA SAMPLING-BASED WATERMARK DISTILLATION EXPERIMENTS

Sampling-based distillation learns watermarks in a smaller Pythia model despite tokenizer and size differences, while additional experiments examine training volume, mixed keys, and evaluation domains. The appendix also specifies setups for text-edit, continued-fine-tuning, spoofing, and harmfulness evaluations.

C PYTHIA SAMPLING-BASED WATERMARK DISTILLATION EXPERIMENTS

  • Higher-distortion watermarks are learned by sampling-based distillation in Pythia 1.4B, while lower-distortion settings are learned less strongly; the method works across different tokenizer and model sizes.Lower-distortion settings remain learnable, with p-values and AUROC values differing from the non-watermarked baseline of 0.5.Source: Table 3 · p. 18, C PYTHIA SAMPLING-BASED WATERMARK… · p. 18

D SAMPLE COMPLEXITY OF WATERMARK DISTILLATION

  • More training tokens strengthen learned watermarks, but even smaller training amounts yield detection p-values below the non-watermarked baseline of 0.5.The authors characterize sample complexity as continuous: additional samples and steps help but are not always crucial, and requirements vary by watermark and hyperparameters.Source: Figure 5 · p. 19, D SAMPLE COMPLEXITY OF WATERMARK… · p. 19

E MIXING TWO KEYS FOR SAMPLING-BASED WATERMARK DISTILLATION

  • Training on two watermark keys hinders learning but does not prevent it; models learn signals for both keys, while multiple-testing correction reduces detection power and raises false negatives.Table 4 reports detection p-values and KTH test statistics for single-key versus two-key training.Source: E MIXING TWO KEYS FOR SAMPLING-BASED… · pp. 19–20, Table 4 · p. 20

F ADDITIONAL DATASETS

  • Wikipedia and arXiv evaluations exhibit trends similar to the C4 RealNewsLike evaluation, indicating relative robustness to domain shifts.The appendix reports Llama 2 7B results on both datasets and Pythia 1.4B sampling-based results on both datasets.Source: F ADDITIONAL DATASETS · p. 20, Table 5 · p. 20, Table 6 · p. 21

G ROBUSTNESS TO TEXT EDITS EXPERIMENT DETAILS

  • The text-edit evaluation deletes a proportion of generation tokens, inserts random replacements to restore length, and measures median detection p-values across edit proportions.The tested edit proportions range from 0 to 0.8.Source: G ROBUSTNESS TO TEXT EDITS EXPERIMENT… · p. 22

H CONTINUED FINETUNING DETAILS

  • The continued-fine-tuning experiment trains distilled Llama 2 7B models on OpenWebText and tracks median detection p-values at checkpoints through 2,500 steps.Checkpoints are saved every 500 steps and evaluated using 200-token completions.Source: H CONTINUED FINETUNING DETAILS · p. 22

I.1 SPOOFING ATTACK TRAINING DETAILS

  • For the spoofing attack setup, Alpaca-7B is fine-tuned on KGW-watermarked Llama 2-Chat responses after keyword filtering removes refusals.The training set contains 52,000 samples generated from Alpaca instructions.Source: §I.1 · p. 23

I.2 HARMFULNESS EVALUATION DETAILS

  • Harmfulness evaluation uses 200 toxic questions and GPT-4 annotations of whether each response provides information enabling harmful behavior.Responses are generated with temperature 1 sampling; the models are not explicitly prompted to refuse harmful requests.Source: §I.2 · p. 23
Loading 2312.04469v3…