PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Follow In-Context Watermark Instructions via Self-Distillation

Yepeng Liu, Tianyi Chen, Xuandong Zhao, Dawn Song, Yuheng Bu

arXiv:2608.29030v1cs.AI

TL;DR

The paper asks whether LLMs can follow in-context watermark instructions without sacrificing answer quality. It introduces ICWBench and a self-contained two-stage training method, achieving strong watermark detection and quality on two open-source models while exposing reward hacking as a scope limitation.

  • Problem

    Existing evidence had not measured how reliably current LLMs follow ICW instructions or what answer-quality cost this entails.

  • Method

    The method combines self-distillation with instruction-equivalent decoding-time logit perturbation and reinforcement learning using an automatic verifier as reward, without stronger teachers or manual annotation.

  • Results

    0.974 and 0.968 average TPR@1%FPR were achieved for Qwen3-14B and GPT-OSS-20B, respectively, across three instruction families while maintaining high response quality.

  • Takeaways & Limitations

    ICWs can provide a watermarking interface that third parties invoke without model-internal access, and the proposed training method improves the evaluated models’ ICW instruction-following–quality trade-off.

  • Takeaways & Limitations

    An SA run reached TPR@1%FPR 0.994 through reward hacking that introduced unnatural formatting and word-splitting artifacts instead of natural sentence construction.

Abstract

from arXiv · show

In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response. It thus equips LLMs with a watermarking interface that third parties can invoke without access to model internals. Its reliability hinges on the LLM following the instruction without degrading answer quality, yet how well current LLMs do so has not been measured. We introduce $\mathsf{ICWBench}$, a benchmark of three verifiable ICW instruction families, each scored on both detectability and answer quality. Evaluating 14 frontier proprietary and open-source LLMs, we find that none of the evaluated LLMs achieves both objectives across all three families. To address this, we propose a self-contained two-stage training method, requiring no distillation from a stronger model, no manual annotation, and no pre-existing ICW IF ability. The first stage, self-distillation with logits perturbation (SDLP), uses the same base LLM as both teacher and student: an instruction-equivalent decoding-time logits perturbation makes the teacher follow the ICW instruction, and the student is trained to match the teacher's output distribution. The second stage applies reinforcement learning with the automatic verifier as the reward. Applied to Qwen3-14B and GPT-OSS-20B, our method raises average TPR@$1\%$FPR across three ICW instructions from $0.100$ to $0.974$ and from $0.337$ to $0.968$, respectively, while maintaining high response quality under both perplexity evaluation and LLM-as-a-Judge.

1 Introduction

ICW enables third parties to invoke watermarking through instructions, but reliable use requires models to follow those instructions without harming answer quality. ICWBench exposes this gap, and the proposed self-contained training method substantially improves the detectability–quality trade-off.

  • ICW instructions ask LLMs to embed statistically detectable signals, offering third parties watermarking without access to model internals.
  • ICWBench evaluates 14 frontier proprietary and open-source LLMs across three instruction families using both watermark detectability and answer quality.
  • Current models exhibit both under-following and over-following: weak watermark detection in TSP or strong detection accompanied by severe quality degradation.No evaluated model exceeds 0.66 AUC on TSP, while Gemini-3-pro reaches 1.000 AUC on WIP as perplexity rises from 4.33 to 67.61.
  • The paper addresses the cold-start problem of training weak ICW followers without stronger teachers, manual annotation, or pre-existing ICW ability.The method uses the base LLM alone to synthesize instruction-compliant data and provide distribution-level supervision.
  • 0.974 and 0.968 average TPR@1%FPR are achieved by trained Qwen3-14B and GPT-OSS-20B, respectively, across three instruction families while maintaining high response quality.The resulting models have a more favorable ICW instruction-following–quality trade-off than the evaluated frontier models.

2 Related Works

Prior work studies LLM watermarking and instruction following largely as separate capabilities. This paper connects them by proposing a training method for instruction-mediated watermarking.

  • Existing LLM watermarking methods embed verifiable signals but generally require model owners’ access to the internal sampling process.
  • The paper proposes a training method to close the gap between general instruction following and reliable in-context watermark instruction following.
  • Prior instruction-following benchmarks evaluate surface constraints such as length, format, keywords, or style, whereas ICW constrains aggregate response statistics.

3 ICWBench

ICWBench measures whether responses satisfy verifiable, query-general watermark instructions while remaining high-quality answers. Its design emphasizes automatic verification, quality preservation, and robustness to editing.

  • ICWBench contains three parameterized, verifiable ICW instruction families that produce distinct watermark instances through different keys.
  • Problem formulation: An ICW instruction consists of a family, a template, and a secret key; prepending the instantiated instruction to a query produces the model response.
  • Problem formulation: Each family has an automatic verifier that maps a response and key to a test statistic, and evaluation requires detection above threshold while answering the query.
  • Design principles: The benchmark selects instructions for generalizability, verifiability, quality preservation, and robustness to editing operations.
  • Instruction families: The content-agnostic families apply across query domains and evaluate statistical or ordered constraints rather than merely inserting a few trigger strings.
  • Evaluation metrics: For TSP and WIP, the verifier counts instruction-aligned scored units and computes a z-statistic whose larger values provide stronger evidence of watermark adherence.SA instead uses a longest-common-subsequence match against a permutation null.

4 Learning to Follow ICW Instructions

The method addresses ICW cold-start training by synthesizing instruction-compliant data with logits perturbation, distilling perturbed distributions, and refining the resulting policy with verifier-based reinforcement learning.

  • Learning to Follow ICW Instructions: RLVR needs an initial policy that can generate high-reward ICW rollouts, but weak base models may not reliably produce instruction-compliant responses.This creates the central cold-start challenge for training ICW instruction-following ability.
  • Training Data Synthesis: Instruction-equivalent logit perturbation steers base-model decoding toward responses with higher verifier scores instead of relying on the model to follow the ICW instruction directly.The perturbation is applied autoregressively to token logits and is adapted to each ICW instruction family.
  • Training Data Synthesis: Repeating perturbed decoding across queries and instruction parameters forms a cold-start dataset whose watermarked samples satisfy the verifier-scored ICW instruction by construction.The dataset also includes unwatermarked samples generated under the null hypothesis, and its evaluation reports high reward with good quality.
  • Self-Distillation with Logits Perturbation: Standard SFT loses distribution-level ICW information because the watermark signal is spread across many token choices rather than a single emitted token.The method therefore distills the teacher’s output distribution instead of only the realized response sequence.
  • Self-Distillation with Logits Perturbation: SDLP uses a frozen base LLM as teacher and a copy as student, with the teacher’s perturbed distribution on watermarked data and natural distribution on unwatermarked data.The student receives the ICW instruction and query and learns to reproduce the corresponding teacher distributions.
  • Self-Distillation with Logits Perturbation: The distillation objective restricts the KL term to the teacher’s top-k tokens, renormalizing both teacher and student distributions over that shared support.Top-k truncation removes low-probability tail tokens that contribute little meaningful training signal.
  • Reinforcement Learning with Verifiable Rewards: After SDLP, GRPO further trains the policy on watermarked samples using the automatic ICW verifier as reward and group-relative advantages from sampled rollouts.The objective uses per-token importance ratios with a clipped surrogate controlled by the clipping range.

5 Experiments

Experiments show that frontier LLMs vary sharply in ICW instruction-following and often trade detectability against response quality. The proposed training method substantially improves ICW detection on two weak open-source backbones while preserving response quality and robustness.

  • 5 Experiments: 500 LFQA queries are reused across all three instruction families and evaluated models, with disjoint LFQA queries used for training.The benchmark reports ICW IF and response quality separately for each instruction family.
  • 5.2.1 Evaluation of frontier models on ICWBench: TSP defeats every frontier model: DeepSeek-v4-pro is strongest at TPR@1%FPR = 0.120, while the strongest TSP AUC is 0.658.SA is easiest, whereas WIP has strong performance from some proprietary models but remains weak for many others.
  • 5.2.1 Evaluation of frontier models on ICWBench: Gemini-3-pro reaches WIP TPR@1%FPR = 1.000 but has PPLQ = 67.61 and the lowest average judge score, 3.94.For TSP, quality remains high because the constraint is not encoded; SA is more balanced because its constraint affects sentence-initial positions.
  • 5.2.2 Evaluation of our method on ICWBench: Qwen3-14B improves average TPR@1%FPR from 0.100 to 0.974, while GPT-OSS-20B improves from 0.337 to 0.968 across the three families.The trained models remain comparable to their original backbones in perplexity and LLM-as-a-Judge quality.
  • 5.2.2 Evaluation of our method on ICWBench: Across attacks, the trained models achieve state-of-the-art AUC on TSP and are competitive with the strongest baselines on WIP and SA.The attacks are paraphrase, 30% word replacement, and 30% word deletion; further attacks are evaluated in the appendix.
  • 5.3 Ablation Study: SDLP provides a strong cold start across all three instruction families, whereas direct RL is limited on TSP and WIP and reward-hacks SA formatting.The ablation compares SFT, RL, and SDLP; SFT is not reliable across ICW instructions.

6 Conclusion

The paper introduces ICWBench to evaluate instruction-following watermarking and proposes a self-contained two-stage training method that improves the IF-quality trade-off. The results support ICWs as a watermarking interface usable by third parties without model-internal access.

  • ICWBench evaluates three verifiable ICW instructions across 14 frontier proprietary and open-source LLMs.
  • The proposed SDLP and RLVR method improves ICW instruction following while maintaining generation quality, without stronger-model distillation, manual annotation, or pre-existing ICW IF ability.
  • The trained Qwen3-14B and GPT-OSS-20B achieve a better IF-quality trade-off than evaluated frontier models on TSP and WIP, while matching the strongest proprietary baselines on SA.
  • ICWs are presented as a promising watermarking interface that third parties can invoke without access to model internals.

A Implementation Details

The implementation instantiates family-specific verifiers and decoding-time logit perturbations for TSP, WIP, and SA. These designs define token, word-initial-letter, and sentence-acrostic signals with corresponding statistical verification procedures.

  • The verifier Vψ and perturbation Δt(ψ; κ) are instantiated separately for each ICW family and used in cold-start synthesis and SDLP teacher distributions.
  • Vocabulary and token representation: The implementation restricts watermarking to English tokens, including word-initial subwords whose first letters are mapped by ϕ(v).
  • Token-Set Preference (TSP): TSP samples a token subset T from the English-token vocabulary and verifies whether unique response tokens overlap with T more than expected under p0 = γ.
  • Word-Initial Preference (WIP): WIP applies a decoding-state-independent logit perturbation to word-initial English tokens whose first letters belong to a keyed letter set L.
  • Sentence Acrostic (SA): SA extracts sentence-initial letters and verifies the secret string using longest-common-subsequence matching against an empirical permutation null distribution.
  • Sentence Acrostic (SA): SA perturbation is stateful: it biases the current target letter at sentence starts and advances after a match or three consecutive failures.

A.2 Cold-Start Dataset Dcs Statistics

The cold-start dataset pairs watermarked and clean responses generated from the base model, using family-specific perturbations and disjoint query and key sources. Its watermarked samples contain a strong detectable signal.

  • The cold-start dataset Dcs combines perturbed H1 responses carrying ICW signals with clean H0 responses generated without perturbation.
  • Queries come from the LFQA training split and are disjoint from the 500 evaluation queries.
  • Dataset statistics: Mean z-scores for H1 samples are well above the 1%FPR detection threshold across all three instruction families.
  • Each instruction family uses paired positive and negative responses differing only by the presence of the instruction.
  • TSP and WIP instructions: TSP and WIP prompts ask models to favor keyed tokens or letters while preserving clarity, accuracy, informativeness, relevance, and text quality.
  • SA instruction: SA prompts ask models to align sentence initials with a secret string while treating the acrostic as a soft guide and dropping targets after repeated misses.
  • Key construction: Keys are deterministic functions of seeds, and training and evaluation use disjoint seed pools.

B Generalization of ICW Instruction Following

Generalization tests vary watermark keys, instruction templates, and query domains beyond training conditions. The trained Qwen3-14B retains high detectability and preserves its advantage over the base model across these changes.

  • B Generalization of ICW Instruction Following: Evaluation varies the instruction, key, and query while using the trained Qwen3-14B, with evaluation keys drawn from seeds disjoint from training.
  • B.1 Detectability across watermark keys: Verifier z-statistic medians range from 4.29 to 4.98 across the three families under held-out evaluation keys.
  • B.2 Held-out instruction templates: Held-out templates preserve the task while changing wording, and the trained method retains high detectability across all three families.
  • B.2 Held-out instruction templates: The held-out-template evaluation compares Ours, Base, and their difference on LFQA evaluation queries.
  • B.3 Unseen query domains: BioASQ and FiQA provide biomedical and finance query sets different from the LFQA training distribution.
  • B.3 Unseen query domains: The performance gap between Ours and Base is preserved on both unseen query domains under the original instruction template.

C Robustness under Stronger and Adaptive Attacks

This section examines watermark robustness against stronger and adaptive edits, including aggressive paraphrasing and attacks trained specifically for watermark removal.

  • The evaluation extends beyond standard edits to include more aggressive response modifications and an attack trained specifically to remove watermarks.

C.1 Multi-round paraphrasing and back-translation

Stronger edits reduce detection performance unevenly, but the watermark remains detectable after repeated paraphrasing and back-translation. Summarize-then-expand is most damaging for Sentence Acrostic.

  • Multi-round paraphrasing: After three paraphrasing rounds, detection AUC remains 0.853, 0.872, and 0.796 for TSP, WIP, and SA.
  • Back-translation: Back-translation reduces detection AUC by at most 0.065 for any instruction family.
  • Summarize-then-expand: Summarize-then-expand lowers SA detection AUC to 0.597, while TSP and WIP remain at 0.757 and 0.887.
  • Evaluation setup: Table 10 reports detection AUC under three stronger edits for the trained Qwen3-14B model.

C.2 Adaptive attack

The adaptive attack can evade detection readily when semantic damage is unconstrained, but its advantage narrows sharply when attacked responses must preserve content.

  • Attack setting: RLCracker requires watermarked text for training, and the evaluation grants it 100 ICW-bearing samples despite the third-party origin of ICWs.
  • Metrics: ESR measures the fraction of attacked responses falling below the 1% FPR detection threshold.
  • Metrics: Semantic utility is evaluated with P-SP and an LLM judge scoring answer Coverage and Fidelity on a 1–5 scale.
  • Results: At Coverage ≥2, RLCracker evades detection on more than 97% of responses, while P-SP falls from about 0.96 to 0.78 and Coverage from about 4.95 to 3.5.
  • Results: At Coverage ≥4.5, ESR falls to 0.011/0.370/0.073, so RLCracker no longer exceeds plain LLM paraphrasing on any family.
  • Interpretation: The results align with the cited theoretical conclusion that robust watermarks are impossible when attackers may alter semantic content without restriction.
Loading 2608.29030v1…