PaperlayerView exact revision on arXiv

Source-linked AI summary

TextSeal2: Distortion-Free LLM Watermarking with Copula Mixing and Memoryless Races

Tom Sander, Pierre Fernandez, Itai Gat, Haiming Bao, Jiaxin Shi, Jason Park, Valeriu Lacatusu, Sylvestre-Alvise Rebuffi, Hongyan Chang, Hady Elsahar, Tomáš Souček, Tuan Tran, Surya Parimi, Alexandre Mourachko

arXiv:2610.40613v1cs.CRcs.AIcs.CL

TL;DR

TextSeal’s detector averages scores across two possible keys, weakening its detection signal. TextSeal2 merges keys through a Gaussian copula and uses a residual race for speculative decoding; it preserves the target output law and acceptance while improving detection over TextSeal and textGrain in the reported speculative comparison.

  • Problem

    TextSeal’s detector cannot identify which of its two keys generated a token and averages both scores, halving the signal and reducing signal-to-noise ratio.

    Source: §1 · p. 3

  • Method

    TextSeal2 merges two or more keys into a virtual key through AutoMark’s Gaussian copula and uses a race on draft-race leftovers for speculative decoding.

    Source: §1 · p. 3, §3.2 · p. 6, §5 · p. 11

  • Results

    At 200 tokens under speculative decoding, TextSeal2 detects 89% of answers versus 71% for TextSeal and 67% for textGrain at TextSeal’s Self-BLEU.

    Source: §4.3 · p. 10

  • Takeaways & Limitations

    TextSeal2 preserves the target’s output law and block-verification acceptance while carrying the watermark through speculative decoding.

    Source: §3.2 · p. 7, §5 · p. 11

    The evaluation covers three domains and three open-weight models at temperatures 0.7–1; the speculative implementation is unbatched, so wall-clock speedups are not measured.

    Source: §5 · p. 11

Abstract

from arXiv · show

We present TextSeal2, a distortion-free watermark for large language models that keeps its signal under speculative decoding: tokens that replace rejected drafts remain watermarked with the same keys, and the acceptance rate is unchanged. It also preserves output diversity and adds no measurable inference latency. Its predecessor, TextSeal, restores diversity in the Gumbel-max scheme by routing each token to one of two keys via a coin flip. This is suboptimal because the detector averages the scores from both keys without knowing which key was used. TextSeal2 instead merges two or more keys into a single virtual key and couples it to fresh noise through AutoMark's Gaussian copula. Any number of keys, including publicly shared keys, can be merged this way without diluting detection. Under speculative decoding, TextSeal2 verifies keyed drafts with one more exponential race, run on the leftovers of the draft races. By memorylessness, this race reproduces block verification, so the output follows the target model exactly. We evaluate TextSeal2 on three LLMs. At equal output token diversity, TextSeal2 detects 8 to 22 percentage points more 100-token answers than TextSeal at a false-positive rate of $10^{-3}$, and is comparable to textGrain while requiring no post-logits optimization. Under speculative decoding, it detects 18 and 22 percentage points more 200-token answers than TextSeal and textGrain, respectively, at the same acceptance rate.

1 Introduction

TextSeal2 addresses the loss of detection signal caused by TextSeal’s unknown coin-flip key routing while aiming to preserve distribution, diversity, latency, and speculative-decoding acceptance. It merges keys through a copula and uses a memoryless residual race for speculative decoding.

1 Introduction

  • The introduction motivates watermarking as a keyed statistical signal for detecting generated text, citing misuse concerns and the EU AI Act’s marking requirement.It also notes deployment in several language-model systems.Source: §1 · p. 1
  • The paper frames deployed watermarking around preserving the model distribution and fixed-key diversity, minimizing latency, and retaining speculative-decoding acceptance.It defines distortion-free schemes as those whose next-token distribution, averaged over keys, equals the model’s.Source: §1 · p. 1
  • TextSeal’s detector averages scores across keys because it cannot identify the key used for each token, halving the signal and reducing signal-to-noise ratio.TextSeal2 is presented as retaining TextSeal’s two keys while addressing this detection gap.Source: §1 · p. 3
  • TextSeal2 merges two keys into one virtual key through AutoMark’s copula, allowing additional keys, including a public key, without diluting detection.Each key retains its role, such as supporting public detection.Source: §1 · p. 3
  • A race on draft-race leftovers reproduces block verification: output remains distortion-free, acceptance is unchanged, and detection gains 28 to 33 percentage points over an independent-key last token.The method uses the memorylessness of exponential races to choose the first rejected draft token and the bonus token.Source: §1 · p. 3
  • The copula strength ρ controls entropy removal, which increases strictly with ρ; the copula comes close to the bound that detection evidence cannot exceed removed entropy.The passage identifies ρ as playing the role of textGrain’s entropy budget β.Source: §1 · p. 3

2 Background & Related work

The background describes keyed Gumbel-max sampling and its detector, then contrasts watermark designs that add randomness or couple keys differently. It also explains repeated-window handling and speculative-decoding verification.

2 Background & Related work

  • Gumbel-max uses key-derived pseudo-random values to form an exponential race whose winner is distributed according to the model’s next-token probabilities.The detector scores observed tokens from recomputed keyed values; under the null, the scores are Exp(1) and their sum is Gamma-distributed.Source: §2 · p. 3

Repeated windows.

  • Repeated windows can make a keyed race repeat; context masking restores fresh randomness, while ResidualRace continues the race from its leftovers to preserve single-sequence distortion freedom.Detection scores each (window, token) pair once to avoid biasing the test.Source: Repeated windows · p. 3
  • TextSeal randomizes Gumbel-max by routing each token to one of two keys, but its detector must fuse both keys’ scores without knowing which key generated the token.The keys also support speculative decoding and public detection, with one key public and the other private.Source: Repeated windows · p. 4
  • AutoMark adds fresh Gaussian noise through a copula; its strength ρ ranges from unwatermarked sampling at 0 to deterministic keyed Gumbel-max at 1 while preserving uniform values.This preserves distortion-free sampling across the strength range.Source: Repeated windows · p. 4
  • textGrain uses keyed Gumbel costs and Sinkhorn iterations to solve an entropy-regularized optimal-transport coupling, but those iterations lie on the decoding critical path.The passage reports 1–3 ms per token in the authors’ implementation and 0.2–0.4 ms in a minimal version.Source: Repeated windows · p. 4
  • Speculative decoding proposes draft blocks for target-model verification; block verification chooses jointly and preserves the target’s output law while decoding at least as many tokens in expectation as other guaranteed procedures.Token-level verification accepts drafts with probability min(1, p_x/q_x) and samples a residual replacement at the first rejection.Source: Repeated windows · p. 4

3 The TextSeal2 watermark

TextSeal2 merges keyed values into a virtual key and uses memoryless exponential races to watermark speculative decoding without changing the target distribution or block-verification acceptance. Its strength controls entropy removal, while detection uses matched key scores.

3.1 Copula mixing of the two keys

  • Copula mixing merges two or more keys into a virtual key, so every token carries the signal scored by the detector without diluting full-detector power.The construction is distortion-free for every coupling strength; keys can include a publishable key for public detection.Source: §3.1 · p. 5

3.2 Speculative decoding through memoryless races

  • A residual race on draft-race leftovers selects the stop position and emitted token, preserving the target law and block-verification acceptance.Kept drafts retain their draft-race signal, and the emitted token carries the relay race’s signal.Source: Figure 3 · p. 6, §3.2 · p. 7

3.3 Controlling the entropy removed

  • The coupling strength ρ controls entropy removal, increasing strictly from none at ρ = 0 to the token’s full entropy at ρ = 1.A fixed ρ removes nearly the same entropy fraction across many-token distribution shapes; the copula attains 97% of optimal per-token signal-to-noise ratio on real Qwen3.5-9B distributions in the weak-watermark limit.Source: §3.3 · p. 7

3.4 Detection and localization

  • The detector sums scores for distinct window-token pairs; private and public scores are Exp(1) under H0 for every coupling strength.Localization tests overlapping windows at dyadic lengths and greedily extracts the most significant spans.Source: §3.4 · p. 8

4 Experiments

At equal diversity under speculative decoding, TextSeal2 detects more answers than the compared watermarking methods while maintaining block-verification acceptance. Without speculative decoding, it also improves detection over TextSeal, with robustness, latency, and extended-task results reported across the experiments.

4.1 Setup

  • The evaluation generates ELI5 answers with three LLMs and compares methods at TextSeal’s Self-BLEU; speculative decoding uses eight drafts per round for Qwen3.5-9B.Detection is the fraction of answers declared watermarked at a nominal false-positive rate of 10^-3.Source: §4.1 · pp. 8–9

4.2 Without speculative decoding

  • Entropy weighting improves every method by 4–23 percentage points at 200 tokens without changing their ranking.Figure 5 reports gains of 5–18 points at 100 tokens and 4–23 at 200 tokens.Source: Figure 5 · p. 9, §4.2 · p. 10
  • At TextSeal’s Self-BLEU, TextSeal2 detects 10, 22, and 8 percentage points more 100-token answers than TextSeal on Llama-3.1-8B, Qwen3.5-9B, and Qwen3.5-27B, respectively.Against textGrain, differences are +11, +1, and −5 points on those models, respectively.Source: §4.2 · p. 10
  • Under random word substitutions, TextSeal2 retains more detections than TextSeal and as many as textGrain; LLM paraphrasing removes almost every watermark.Its decoding step is within 0.6% of TextSeal’s once key-only work overlaps the forward pass.Source: §4.2 · p. 10
  • On 500 MBPP test problems, TextSeal2 detects 55% of answers at 256 tokens, versus 39% for TextSeal, 46% for textGrain, and 50% for SynthID-Text.For repeated windows in long traces, continuing races from leftovers raises detection by 3.8 points on ELI5 at 256 tokens and 7.6 points on MBPP at 512 tokens, without measurable distortion.Source: §4.2 · p. 10

4.3 With speculative decoding

  • At equal Self-BLEU, TextSeal2 detects 59% / 89% of 100 / 200-token answers under speculative decoding, versus 31% / 71% for TextSeal and 31% / 67% for textGrain.It leads every baseline that keeps acceptance by 15–41 percentage points at 200 tokens; textGrain’s own recipe comes closest but emits 5% fewer tokens per round.Source: §4.3 · p. 10

5 Conclusion

TextSeal2’s conclusion reports distortion-free generation, preserved diversity and speculative-decoding acceptance, and near-plain-sampling latency. The appendices establish copula mixing’s distributional property, exponential-race memorylessness, and the block-verification framework it relies on.

5 Conclusion

  • TextSeal2 is exactly distortion-free, preserves diversity under a fixed key, stays within 0.9% of plain sampling when key-only work overlaps the forward pass, and maintains speculative-decoding acceptance.Its detection is comparable to textGrain at 100 tokens across the three named models, without textGrain’s per-token optimisation.Source: §5 · p. 11

A Proofs

  • The appendix assumes unused keyed values are i.i.d. uniform and independent of fresh copula noise when establishing these proofs.Its copula-mixing lemma establishes distortion-freedom for every coupling strength.Source: A Proofs · p. 14, §A.1 · p. 14

A.1 Copula mixing

  • For any coupling strength ρ, the copula-mixed values remain i.i.d. uniform, so their race emits each token according to its model probability.The proof uses independent Gaussian components whose variances sum to one.Source: §A.1 · p. 14

A.2 Exponential races and their leftovers

  • An exponential race emits token v with probability qv, and after the winner and time are known, losing lanes retain independent Exp(1) leftovers.Tokens with qv = 0 never win and their values remain untouched.Source: §A.2 · p. 14

A.3 Block verification

  • Block verification jointly determines how many drafts to keep and the replacement or bonus token, allowing likely later drafts to rescue unlikely earlier ones.For one draft, it reduces to token-level verification.Source: §A.3 · pp. 14–15

A.4 Exactness of the residual race

The residual race uses leftover draft-race values to reproduce block verification while carrying watermark signal. Repeated-window rules preserve the whole-text distribution and keep detection tests valid.

A.4 Exactness of the residual race

  • Given the drafts, the residual race reproduces block verification’s outcome law, so acceptance is preserved and the generated text follows the target’s law.This result assumes independent keyed values at positions and rounds, plus independent copula noise.Source: §A.4 · p. 15
  • The relay races on draft-race leftovers, choosing both the stopping position and final token; kept drafts retain their draft-race signal, and detection needs no round boundaries.The detector sums token scores as before.Source: §A.4 · p. 16
  • Racing on raw lane lengths is biased: in the one-draft example, it emits the third token with probability 15/17 instead of the residual race’s 3/4.The raw-lane race also stops at position 1 with probability 85/144, versus 2/3 for the residual race.Source: §A.4 · p. 16

A.5 Repeated windows

  • Repeated windows use fresh values after a window has already been raced, preserving the model’s law for the whole answer rather than introducing dependencies between repeated tokens.The detector scores each window-token pair once; under the null, the sum follows Gamma(n, 1).Source: §A.5 · pp. 16–17

A.6 Acceptance

The acceptance analysis bounds what valid draft verification can achieve and shows that the residual race inherits block verification’s multi-draft advantage. At one position, its acceptance matches the raw test’s bound.

A.6 Acceptance

  • For one draft, the residual race keeps the draft with probability 1 − TV(⃗p1,⃗q1), and no scheme preserving the output distribution does better.This is the one-position acceptance bound.Source: §A.6 · p. 17
  • At one position, acceptance is bounded by 1 − E[TV(˜p, ˜q)] ≤ 1 − TV(⃗p,⃗q), with strictness in the second inequality when a token’s watermarked probability difference takes both signs.The residual race with N = 1 attains the second bound, as does the raw test with an independent coin.Source: §A.6 · p. 17
  • For N > 1, block verification decodes at least as many tokens in expectation as any valid draft-verification algorithm, and the residual race inherits this guarantee.The cited result identifies block verification as the comparison class.Source: §A.6 · p. 17

A.7 Entropy removed

The coupling strength ρ calibrates how much token entropy the key removes while preserving the token law. Its continuous, strictly increasing relationship to removed entropy provides a unique setting for each target budget.

A.7 Entropy removed

  • For positive-entropy distributions, ρ maps continuously and strictly increasingly from zero removed entropy to the token’s full entropy, giving a unique coupling for every target budget.A prefix-dependent setting keeps the token law and leaves the detector unchanged.Source: §A.7 · pp. 17–18
  • Under speculative decoding, ρ may depend on the draft distribution but not the target distribution, because the drafter fixes its race inputs before the target computes its distribution..Source: §A.7 · p. 18

Setup.

The section frames watermarking as a trade-off between detection signal and token diversity, measured through entropy removed by coupling. It bounds detection evidence by that cost and assesses how closely the copula approaches the bound.

Setup.

  • The coupling’s per-token cost is mutual information between the keyed Gaussian and emitted token, while its signal-to-noise ratio compares keyed-score shift with null standard deviation.The null score has mean µ0 and standard deviation σ0.Source: Setup · p. 18
  • Every distortion-free coupling yields less than one nat of detection evidence per nat of entropy removed.The expected log-likelihood ratio per token measures evidence accumulation, and the bound follows from data processing.Source: Setup · p. 18
  • The best coupling under a fixed entropy-removal budget is an exponential tilt, or entropic optimal transport, which textGrain approximates per token.The optimization maximizes expected score subject to the token marginal and mutual-information cost.Source: Setup · p. 19
  • The copula’s efficiency reaches equality only when the score is increasing and affine in the Gaussian and the token distribution is uniform on its support.Extending the normal-maximum function to non-integer inputs agrees with an exact computation within 0.001 on uniform, Zipf, and peaked distributions.Source: Setup · p. 19
  • At finite strength, the copula reaches 91–100% of the optimal signal-to-noise ratio at ρ = 0.5, 0.8 and 0.95; its weak-limit efficiency averages 0.966 across 5,169 Qwen3.5-9B positions.The weak-limit average is 0.971 when weighted by each position’s capacity.Source: Setup · p. 19

C Entropy removed by a constant coupling

A fixed coupling strength removes a nearly shape-independent fraction of token entropy on many-token synthetic distributions. The appendix also gives a residual-race algorithm for one round of speculative decoding.

C Entropy removed by a constant coupling

  • At ρ = 0.8, the copula removes 0.47–0.49 of entropy on many-token shapes, and at ρ = 0.9 it removes 0.63–0.65.The removed fraction is about 0.1 lower for two equiprobable tokens; real Qwen3.5-9B distributions are assessed by efficiency instead.Source: C Entropy removed by a constant coupling · pp. 19–20

D.1 One round in pseudocode

  • The appendix’s one-round algorithm forms draft races, obtains target distributions in one forward pass, then uses a residual race for block verification.It returns the accepted draft prefix followed by the replacement token, or samples directly from the next-position target law without speculation.Source: §D.1 · p. 20
  • Key-only work for position i can run alongside the drafter’s forward pass, while verification and relay work scale linearly with (N + 1)|V| values.The algorithm explicitly zeros the draft’s own lane mass to prevent a rounding residue from winning the race.Source: §D.1 · p. 20

D.2 Post-logits latency

TextSeal2’s post-logits work matches TextSeal’s and is cheaper than shared top-p filtering. Its additional key-only work can overlap the model forward pass, keeping full decoding latency close to TextSeal.

D.2 Post-logits latency

  • After logits are known, TextSeal2 costs the same as TextSeal—one vocabulary pass—and is cheaper than top-p filtering required by every method, watermarked or not, according to Table 4’s post-logits timing.At batch size 1, even textGrain’s minimal version does 8–22 times TextSeal2’s post-logits work.Source: §D.2 · p. 21
  • With key-only work overlapped on a side CUDA stream, TextSeal2 is within 0.3 ms (0.6%) of TextSeal and within 0.9% of unwatermarked sampling.Table 5 reports full Qwen3.5-9B decoding-step latency.Source: §D.2 · p. 21, Table 5 · p. 21
  • Running the extra key-only work sequentially after the forward pass costs 0.2 ms at batch size 1 and 2.9 ms (5%) at batch size 64.TextSeal2 needs both keys’ PRF values and Gaussian transforms at every token, unlike TextSeal’s single key.Source: §D.2 · p. 21

E PRF resolution

The 13-bit PRF measurably distorts the race, especially at high-entropy positions. TextSeal2 instead uses a 53-bit PRF, with reported distortion tests passed under speculative decoding and score agreement with the single-key copula.

E PRF resolution

  • With continuous uniform values, mean −ln p_x − H(p) is −0.03 ± 0.12 millinats per token, versus −1.00 ± 0.14 millinats with the 13-bit PRF.TextSeal2 uses a 53-bit PRF and passes both distortion tests under speculative decoding.Source: E PRF resolution · p. 21
  • The 13-bit PRF’s bias is −42 millinats at positions with 101–1000 candidates, compared with −0.3 millinats at positions with 2–10 candidates.These groups account for 0.9% and 52% of positions, respectively.Source: E PRF resolution · p. 21
  • With the 53-bit PRF, TextSeal2 and the single-key copula agree in per-answer mean scores across all three models at ρ = 0.8 and 0.9.The differences range from −0.006 to +0.011, with every 95% interval containing zero; the 13-bit PRF also correlates the two keys’ values slightly.Source: E PRF resolution · p. 22

F Additional results

Additional experiments examine multi-key merging, lower-entropy settings, entropy-weighted detection, and keys with different window lengths. They also test repeated windows and speculative decoding, including AIME results.

Four keys

  • Four merged keys detected 63% / 91% of 100 / 200-token answers, versus 60% / 92% with two keys.The results use TextSeal’s 13-bit PRF; the paper reports that additional keys cost the full detector no detection power.Source: F Additional results · p. 22

Temperature 0.7

  • At temperature 0.7, TextSeal2 detected 22% / 56% of 100 / 200-token answers, versus 9% / 37% for TextSeal at similar Self-BLEU.textGrain detected 22% / 55% at Self-BLEU 27.3.Source: F Additional results · p. 22

Entropy-weighted detection

  • Entropy weighting improved every method by 4–23 percentage points at 200 tokens, while the two copulas remained level without speculative decoding.The table reports a false-positive rate of 10^-3 and Self-BLEU over 32 answers to each of 100 questions.Source: Table 6 · p. 22

Keys with different windows

  • With a six-key ladder, merging keys with different window lengths detected 98% of answers, versus 22% when keys were routed by TextSeal’s coin flip.The ladder also matched two-key TextSeal2 on clean Qwen3.5-9B text at equal Self-BLEU; pooled edit-rate estimates were unbiased up to 20% substitutions, while single-answer estimates remained informative to about 5%. At 8–12% substitutions, ladder detection fell 15–27 points relative to fixed short windows.Source: F Additional results · p. 23

Repeated windows in long answers

  • Repeated windows occurred at only 2.3% of positions in long-form prose, where every tested scheme detected 98–100% of answers within 256 tokens and 100% from 512 tokens.Repeated windows were more common in code (11% of positions) and thinking traces (16–34%), where leftover continuation helps.Source: F Additional results · pp. 23–24

200-token diversity–detection trade-off

  • Figure 6 extends the diversity–detection comparison to 200-token answers and includes a speculative-decoding panel.The caption gives the scope and protocol but no specific outcome values.Source: Figure 6 · p. 23

Draft counts

  • Table 9 compares draft counts of 4, 8 and 16 at matched TextSeal Self-BLEU; only textGrain’s own recipe lowered tokens per round.Its reductions were 0.08 / 0.14 / 0.18 for the three draft counts, respectively; the table also reports a shared-reference distortion-test deviation at N = 16.Source: Table 9 · p. 24

AIME under speculative decoding

  • 30% of AIME answers were detected by TextSeal2 at 2048 tokens, versus 23% for the coin flip, 21% for SynthID-Text, 17% for textGrain and 9% for AutoMark’s copula.The comparison is interpolated at TextSeal’s Self-BLEU of 46.1; TextSeal2’s lead was significant over all but the coin flip.Source: Figure 8 · p. 25

G Experimental details

The experiments specify generation, detection, and distortion protocols, then test speculative decoding, diversity, computation, and reproducibility. They also state a broader-impact boundary for watermark evidence.

Speculative decoding

  • Speculative decoding used Qwen3.5-0.8B to draft for Qwen3.5-9B, with block verification and N = 8 as the main setting.TextSeal2 ran a draft race followed by a residual race using copula noise drawn once per position; the procedure tracked used window-key pairs and rolled back the key–value cache to the kept prefix.Source: G Experimental details · p. 26

Detection

  • Under speculative decoding, TextSeal2’s last tokens averaged 1.34 under the virtual key, versus 1.42 for kept drafts; AutoMark’s last tokens averaged 0.84 under the draft key.Last tokens made up 36–37% of text in the reported setting, with 8 drafts per round.Source: G Experimental details · p. 26

Calibration

  • At 10^-3 and above, every detector’s false-positive rate was within a factor of 1.3 of nominal in the calibration experiment.Calibration used 10 keys and 10,000 unwatermarked Qwen3.5-9B answers.Source: Figure 9 · p. 27

Diversity

  • With distinct-4-gram diversity, textGrain detected 66% of answers versus 62% for the copulas at TextSeal’s diversity without speculative decoding.Under Self-BLEU, textGrain was level with TextSeal2; under speculative decoding, TextSeal2 still led.Source: G Experimental details · p. 27, Figure 10 · p. 28

Distortion tests

  • The residual-race simulation matched block verification’s law (χ2 test p = 0.15), while racing on raw lane lengths did not (χ2 = 3194 on 401 degrees of freedom).Across baseline simulations, the first emitted token followed the target law and kept-draft counts followed native block verification, except textGrain’s own recipe kept fewer drafts.Source: G Experimental details · p. 28

Compute

  • The unbatched, unoptimised speculative implementation took about 0.11 seconds per token, while the residual-race decision took 0.33 ms per token versus 0.38 ms for native block verification.The reported implementation throughput was three times slower than plain target decoding; the authors say this affects experimental throughput, not reported rates, because block verification is exact.Source: G Experimental details · pp. 28–29

textGrain reimplementation

  • The textGrain results carry an implementation caveat because its report released no code and OpenAI’s implementation and hyperparameters may differ.The authors implemented the described algorithms and validated their kernels against a direct implementation and toy distributions.Source: G Experimental details · p. 29

Reproducibility

  • The authors state that generation and evaluation scripts, fixed seeds, prompt lists, threshold calibration, and per-answer outputs will be released with the paper.Source: G Experimental details · p. 29

Related papers

Loading 2610.40613v1…