PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Watermark LLM-generated Text via Reinforcement Learning

Xiaojun Xu, Yuanshun Yao, Yang Liu

arXiv:2403.10553v1cs.LGcs.AIcs.CR

TL;DR

Fixed-model watermarking relies on output-level distortions, which constrain accuracy and robustness and limit adaptability. This paper embeds watermarks through LLM finetuning and co-trains the model with a detector using reinforcement learning. The authors report improved detection, robustness, and adaptability, with an AUC over 0.99 after training on large perturbations.

  • Problem

    Fixed-model token-level watermarks must limit output distortion to preserve readability, constraining accuracy and leaving them vulnerable to paraphrasing and difficult to adapt.

    Source: §1 · p. 1

  • Method

    The framework finetunes LLM weights to embed a watermark while reinforcement learning co-trains the LLM and a paired detector to make outputs detectable.

    Source: §8 · p. 8

  • Results

    AUC over 0.99 is reported when large perturbations are incorporated into training, alongside near-perfect detection, stronger performance than token-level methods, and robustness to small perturbations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports adapting watermarks to new attacks and open-sourcing watermarked models without releasing an unwatermarked version.

    Source: §1 · p. 2

    The method requires finetuning and relevant training data, and detection costs more than statistical tests; integrating watermarking with standard alignment can mitigate the first two issues.

    Source: §8 · p. 8

Abstract

from arXiv · show

We study how to watermark LLM outputs, i.e. embedding algorithmically detectable signals into LLM-generated text to track misuse. Unlike the current mainstream methods that work with a fixed LLM, we expand the watermark design space by including the LLM tuning stage in the watermark pipeline. While prior works focus on token-level watermark that embeds signals into the output, we design a model-level watermark that embeds signals into the LLM weights, and such signals can be detected by a paired detector. We propose a co-training framework based on reinforcement learning that iteratively (1) trains a detector to detect the generated watermarked text and (2) tunes the LLM to generate text easily detectable by the detector while keeping its normal utility. We empirically show that our watermarks are more accurate, robust, and adaptable (to new attacks). It also allows watermarked model open-sourcing. In addition, if used together with alignment, the extra overhead introduced is low - only training an extra reward model (i.e. our detector). We hope our work can bring more effort into studying a broader watermark design that is not limited to working with a fixed LLM. We open-source the code: https://github.com/xiaojunxu/learning-to-watermark-llm .

1. Introduction

The paper addresses limits of fixed-model, token-level watermarking by embedding detectable signals through LLM finetuning. It co-trains the LLM and a paired detector, reporting improved detection and adaptability alongside trade-offs in training and detection costs.

1. Introduction

  • Fixed-model, token-level methods constrain watermark signals to readable output distortions, which can limit accuracy, increase vulnerability to paraphrasing, and restrict adaptation and watermarked-model release.Post-hoc watermarking can require releasing the unwatermarked LLM as well if practitioners want to open-source the watermarked version.Source: §1 · p. 1
  • The framework finetunes LLM weights to embed a watermark and co-trains a paired language-model detector, using reinforcement learning to make generated text detectable while preserving readability.The detector can serve as the reward model; the RLHF objective is described as utility-preserving.Source: §1 · pp. 1–2
  • Watermarking is data-driven and can adapt to new attacks by incorporating adversarial text into training; generation needs no special watermarking operation, and the watermarked model can be open-sourced without its unwatermarked counterpart.The authors also describe robustness to paraphrasing as an advantage of not relying on low-level output distortion.Source: §1 · p. 2
  • The approach may cost more during instruction tuning, requires relevant tuning data, and has higher detection overhead than fixed-model methods.Combining watermarking with standard alignment can mitigate the first two costs; detection involves language-model inference rather than simple statistical tests.Source: §1 · p. 2
  • AUC over 0.99 is reported after large perturbations are included in training; the framework also achieves near-perfect detection, outperforms token-level methods, and withstands small perturbations.The authors describe the large-perturbation result as evidence of adaptability to new attacks.Source: §1 · p. 2

2. Preliminary

The preliminary section introduces RLHF as a process for aligning an LLM with human preferences. It describes training a reward model and optimizing the LLM policy with PPO and a KL penalty.

2. Preliminary

  • RLHF first trains a reward model to score whether a completion is desirable to humans, using a dataset of prompt and preference-completion pairs.The paper identifies rejected and chosen completions based on human preference as the dataset elements.Source: §2 · pp. 2–3
  • A KL-divergence term is added to the reward to limit deviation from the reference model and avoid out-of-distribution problems.The paper gives the regularized reward as r_total(x, y; θ) = r(x, y) − ηKL(π_θ(a_t|s_t), π_ref(a_t|s_t)).Source: §2 · p. 2
  • PPO then optimizes the LLM policy using the trained reward model and the original model as reference.The objective includes a KL term; β is the KL reward coefficient and γ is the strength of the KL penalty.Source: §2 · p. 3

3. Scenario and Goal

The paper assumes a service-provider setting with resources to finetune an LLM and collect relevant data. Its goal is a paired detector that distinguishes the watermarked model’s outputs while preserving the original model’s utility.

3. Scenario and Goal

  • The scenario assumes an LLM service provider can finetune its model, collect relevant data, and seek accurate detection at reasonable cost without harming normal-task utility.The detection target includes text from humans and other LLMs as well as outputs from the provider’s model.Source: §3 · p. 3
  • The goal is to finetune an original LLM into a watermarked model paired with a detector that scores whether a prompt-response pair came from that model.The detector has the same architecture as a reward model.Source: §3 · p. 3
  • The desired detector scores watermarked-model outputs highly and outputs from other sources poorly, while the watermarked model retains utility close to the original.The text describes preserving utility as keeping the generated output close to that of the original LLM.Source: §3 · p. 3

4. Reinforcement Learning-based Watermark

The framework co-trains a detector and a watermarked LLM, using detector scores to guide LLM tuning while regularization limits deviation from the original model. It alternates LLM and detector updates, and can combine watermarking with standard alignment.

4.1. Overview

  • The detector serves as the RLHF reward model, guiding the watermarked LLM toward high-scoring text while a KL penalty limits deviation from the original model.The detector distinguishes generated watermarked text from non-watermarked text.Source: §4.1 · p. 3

4.2. Algorithm

  • The algorithm alternates PPO updates to the LLM with detector updates that classify watermarked text against non-watermarked text.The LLM update uses the detector's score as its reward; the detector is updated while the LLM is fixed.Source: §4.2 · p. 4
  • Detection uses a forward pass to score prompt-output pairs, with the threshold chosen to reach a specified true positive rate.A high score indicates the output is likely generated by the watermarked LLM.Source: §4.2 · p. 4

4.3. Combining with Alignment

  • With alignment, the reward blends alignment and detector scores, while the added cost is training an extra reward model and running inference on it.The authors describe offline RL co-training as computationally expensive and recommend combining it with standard alignment to reduce additional overhead.Source: §4.3 · p. 4

5. Experiments

The experiments evaluate watermark detection and task utility across two LLMs and two tasks, then test robustness to substitutions and paraphrases and transfer to another LLM. They report better detection at comparable utility and robustness gains from adversarial training.

5.1. Setting

  • Experiments use OPT-1.3B and Llama2-7B for C4 prompt completion and PKU safety alignment, measuring watermark-detection AUC on 1K prompts alongside original-task utility and comparison with watermark baselines.Inference-time watermark methods do not fine-tune the LLM; they are generated on the pretrained model for C4 and on the aligned model after RLHF on PKU.Source: §5.1 · pp. 4–5

5.2. Main Results

  • At similar utility, our watermarks generally detect better than baselines; on OPT-1.3B C4, ITS and EXP detection is slightly better where our logPPL is much better.The paper attributes ITS and EXP’s utility drop on C4 to their inability to use the same sampling setting.Source: Table 1 · p. 5, §5.2 · p. 5
  • Training the detector without fine-tuning the LLM performs much worse, while watermark fine-tuning does not negatively impact utility.The authors suggest semantic-level signals may affect utility less negatively than lower-level token signals.Source: §5.2 · p. 5

5.3. Word Substitution Attacks

  • ITS and EXP outperform the method on substitutions, but adversarial training yields almost no AUC loss at 50% token replacement.Substituted training responses are labeled as still watermarked, and the detector is tested on unseen substituted responses.Source: §5.3 · p. 6, Figure 2 · p. 6

5.4. Paraphrasing Attacks

  • Under Pegasus paraphrasing, the method beats baselines even at low strength; adversarial training further strengthens robustness.The paper attributes this robustness to model-level, semantic-level changes rather than replacing specific tokens.Source: §5.4 · p. 6, Figure 3 · p. 7
  • Training with Pegasus paraphrases also improves robustness on DIPPER paraphrases, demonstrating adaptation across these two attacks.The reported cross-attack test uses a model adversarially trained on Pegasus-paraphrased responses and tested on DIPPER-paraphrased responses.Source: §5.4 · p. 6, Figure 4 · p. 7

5.5. Detecting Text Generated by Another LLM

  • Human-only training leads to minor out-of-distribution issues on another LLM’s text, while adding that text to training recovers detection accuracy.The compared training variants are Ours (H) and Ours (H+L).Source: §5.5 · p. 7

6. Related Work

Related work covers output-space watermarking, model-level watermarking, reinforcement-learning fine-tuning, and general LLM-text detection. The paper distinguishes its target—texts from a specific watermarked model—from general LLM-generated text detection.

6. Related Work

7. Discussion

The discussion identifies open questions about detector transfer across model updates and whether non-RL fine-tuning can reduce costs. It also suggests the data-driven watermark may be less vulnerable to spoofing than fixed-model approaches.

7. Discussion

  • Whether a detector paired with one LLM can detect outputs from related models after weight updates remains unclear.The question includes models in the same family, trained on the same dataset, or iterated from the same pipeline.Source: §7 · p. 8
  • Whether non-RL fine-tuning such as DPO can achieve high detection accuracy more efficiently remains an open question given RLHF's high costs.Source: §7 · p. 8
  • Because its watermark is data-driven rather than algorithmically generated and transparent to attackers, the method might be less vulnerable to spoofing than fixed-model approaches (Kirchenbauer et al., 2023a; Sadasivan et al., 2023).Source: §7 · p. 8

8. Conclusion and Limitation

The paper presents a reinforcement-learning pipeline that co-trains a detector and LLM, reporting watermarks that are more accurate, robust, and adaptable to new attacks. It also notes costs and data requirements, with the first two limitations mitigable through alignment integration.

8. Conclusion and Limitation

  • The proposed pipeline alternates between fine-tuning an LLM to generate detectable text and training a paired detector; the authors report improved accuracy, robustness, and adaptability to new attacks, plus support for open-sourcing.Source: §8 · p. 8
  • Fine-tuning may raise computational cost, the data-driven method requires relevant training data, and detection costs more than statistical tests; alignment integration can mitigate the first two issues.Source: §8 · p. 8

A. Examples of Watermarked Text

The appendix points to examples of model outputs with and without watermark across two models and two tasks.

A. Examples of Watermarked Text

B. Full Results of Word Substitution Attacks

Table 7 presents the full detection results for word-substitution attacks.

C. Full Results of Paraphrasing Attacks

The section reports full detection results for paraphrasing attacks in Tables 8 and 9. The supplied captions identify example outputs and the settings varied in the attack tables, but not their numerical results.

Loading 2403.10553v1…