PaperlayerView exact revision on arXiv

Source-linked AI summary

Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs

Zhihao Wu, Gracia Gong, Qinglin Zhu, Yudong Chen, Runcong Zhao

arXiv:2605.30501v1cs.CL

TL;DR

Watermarking faces a gap in multi-provider settings, where users can combine independently watermarked models. The paper proves that linear ensembling recovers the unwatermarked distribution and introduces WASH to handle heterogeneous ensembles, with experiments showing near-complete detection failure and practical efficiency gains. The findings identify coordinated watermarking as necessary for reliable detection under model mixing.

  • Problem

    Watermarking research assumes adversaries access only one watermarked model, although users can access multiple frontier LLMs whose independent perturbations may be averaged.

  • Method

    The paper proves linear-ensemble convergence and introduces WASH, using fluency-aware routing and context re-synchronisation for heterogeneous model aggregation.

  • Results

    WASH renders watermarks statistically undetectable with 3 models while improving generation quality by 27.5% and running 6× faster than state-of-the-art removal baselines.

  • Takeaways & Limitations

    Reliable watermark detection when users mix models requires coordinated watermarking or a common signal shared across providers.

  • Takeaways & Limitations

    Robustness against model mixing is constrained by the need for coordinated watermarking among providers.

Abstract

from arXiv · show

Watermarking embeds statistical signatures in AI-generated text for detection and attribution. We reveal a fundamental vulnerability: when users access multiple models (today's reality), watermarks trivially fail. Watermarks perturb output distributions away from the original, and in competitive markets, these perturbations are typically independent across providers. We theoretically prove that averaging output probability distributions recovers the unwatermarked distribution with up to a second-order error term. Empirically, simply averaging 3-5 models cancels out these perturbations. We introduce WASH (Watermark Attenuation via Statistical Hybridisation), which solves practical challenges in ensemble generation: vocabulary misalignment and tokenisation differences across heterogeneous models. Experiments across six watermarking schemes and three LLMs show that averaging across 3 models suppresses detection z-scores from 5-300 to below 2 (below the detection threshold of 4) and reduces TPR at 5% FPR to below 50%, while improving quality by 27.5% and running 6 times faster than the best baseline on the long sequence generation. Our results suggest that robust AI-text detection via watermarking requires either accepting this fundamental vulnerability or unprecedented coordination among model providers.

1. Introduction

The paper identifies a structural weakness in watermarking: independently perturbed output distributions can be averaged across models to recover the unwatermarked consensus. WASH addresses heterogeneous vocabularies and tokenisers, making this attack practical while substantially reducing detection.

  • Motivation: Current watermarking assumes adversaries access only one watermarked model, despite users’ access to many frontier LLMs.This multi-provider setting enables averaging across independently watermarked models.
  • Core vulnerability: Independent watermark perturbations cancel under linear ensembling, recovering the consensus distribution with convergence rate O(1/N).The theoretical result applies to unbiased watermarking with independent per-model perturbations.
  • WASH: 27.5% quality improvement and 6× faster runtime distinguish WASH from state-of-the-art removal baselines.The reported efficiency result uses parallel inference and response caching to achieve constant-time per-token complexity regardless of ensemble size N.
  • Implication: The paper concludes that robust detection under model mixing requires coordinated watermarking with a common signal shared across providers.Without such coordination, independently configured watermark perturbations are vulnerable to averaging.
  • WASH: WASH uses fluency-aware routing and context re-synchronisation to aggregate probabilities across heterogeneous vocabularies while preserving semantic integrity.The method targets vocabulary mismatch and tokenisation differences that complicate direct distribution averaging.
  • Empirical findings: With 3 models, WASH suppresses generation-time detection z-scores from 5-300 to below 2 and lowers final-text TPR@5%FPR below 50%.The evaluation covers six watermarking schemes, three LLMs, and two complementary detection settings.

2. Methodology

The method models each watermarked model as a provider-specific perturbation of a shared consensus distribution, then averages independent output distributions to attenuate watermark signals. WASH extends this ensemble approach to heterogeneous vocabularies through union aggregation, fluency-aware routing, and context re-synchronisation while providing convergence guarantees.

  • Consensus and perturbation model: Independent provider-specific perturbations are treated as uncoordinated noise around a shared consensus distribution.The consensus absorbs shared model bias, while each model contributes a provider-specific perturbation that primarily represents its watermark signal.
  • Linear ensemble: WASH recovers the consensus distribution by uniformly averaging the output probabilities of N independent models.The aggregated distribution is used recursively during autoregressive generation.
  • Theoretical guarantee: Under unbiased, bounded, independent perturbations with bounded expected variance, averaging concentrates around the consensus distribution with a second-order systematic error.The stochastic deviation is controlled by concentration, while softmax nonlinearity leaves a residual bias dominated by second-order terms.
  • Vocabulary mismatch: Heterogeneous ensembles create vocabulary mismatches because the vocabulary intersection can exclude semantic ground-truth tokens.Restricting generation to shared tokens creates an expressivity bottleneck, especially for entities and technical terms absent from at least one model.
  • Fluency-aware routing: WASH uses union aggregation and fluency-aware routing to handle tokens unavailable to some models while preserving autoregressive context compatibility.When a token lies outside the shared intersection, subsequent sampling is restricted to specialist models that can process the committed context.

3. Experiment

The experiments evaluate WASH across multiple watermarking schemes, models, detection settings, quality benchmarks, and efficiency comparisons. WASH suppresses detection while preserving stronger utility and practical inference efficiency than competing removal methods.

  • Experimental Setup: Experiments cover three LLMs, six watermarking schemes, generation-time and final-text detection, quality benchmarks, and inference efficiency.
  • Watermark Removal Effectiveness: N = 3 suppresses z-scores below the detection threshold across all tested cases, while N = 5 provides additional safety for aggressive schemes.Individual watermarked models produce z-scores from 5.2 to 304.0; the threshold is z < 4.
  • Watermark Removal Effectiveness: WASH keeps native-detector TPR@5% FPR below 43% across schemes and outperforms RandomWalk on 4 of 5 schemes.
  • Quality, Semantic Integrity, and Efficiency: WASH achieves quality comparable to or better than De-mark in 7 of 9 benchmark settings and outperforms ToBlend across all settings.
  • Quality, Semantic Integrity, and Efficiency: WASH operates at 1.0×–2.4× baseline cost, providing roughly a 6× speedup over competing methods on long-form generation.De-mark exceeds 30× latency on GSM8K and SQuAD, while ToBlend incurs approximately 12× overhead.
  • Ablation and Analysis of Fluency-Aware Routing: Naive averaging suppresses detection but degrades GSM8K accuracy, whereas WASH’s sparse local routing preserves low detection and stronger utility.
  • Ablation and Analysis of Fluency-Aware Routing: Routed tokens account for less than 3.2% of generation, and final detection scores remain far below the watermark threshold.Routing is more often triggered by ordinary lexical units with tokenizer mismatches than by sustained specialist spans.

4. Related Works

Related work develops watermarking schemes that preserve distributional quality or improve resilience, alongside attacks that paraphrase, rewrite, or combine outputs. These approaches commonly depend on detector access or assumptions about the generation process.

  • Watermarking research includes green/red-list, distortion-free, unbiased, semantic-aware, and maximal-coupling approaches.
  • Watermark-removal attacks use paraphrasing, multiple-key voting, local rewriting, and quality-filtered candidate selection.
  • Many prior removal methods assume access to an oracle verifier, limiting their applicability outside detector-available scenarios.

5. Conclusion

The paper argues that distributional watermarking is structurally vulnerable to linear ensembling. WASH addresses heterogeneous vocabularies, and the results indicate that robust detection requires coordination among model providers.

  • Averaging outputs from independent models cancels watermark perturbations, making current distributional watermarking schemes structurally vulnerable to linear ensembling.
  • WASH uses fluency-aware routing to aggregate probabilities across heterogeneous models despite vocabulary mismatches.
  • With as few as three models, WASH makes watermarks statistically undetectable while preserving generation quality and practical inference efficiency.The reported detection condition is z < 2.
  • Reliable detection in a competitive marketplace is presented as unattainable without standardised watermark keys among model providers.

Impact Statement

The paper frames watermark robustness as a societal and technical concern, then reports that coordinated signals across heterogeneous models can partially mitigate ensemble attacks. It therefore highlights cooperation among providers as relevant to provenance mechanisms.

  • AI-generated-content detection is motivated by societal trust, content authenticity, and intellectual-property protection.
  • Simple signal coordination across heterogeneous models can partially mitigate the effectiveness of ensemble attacks.
  • The findings suggest that robust watermarking depends on cooperative approaches beyond isolated model-level defences.

A. Proof of the Main Theorem

The theorem shows that averaging independently perturbed model distributions approaches the consensus distribution with a concentration term and a second-order perturbation error. The proof combines concentration bounds with bias control under bounded, centered perturbations.

  • Theorem A.1 bounds the aggregated distribution’s ℓ∞ distance from the consensus distribution with high probability.The result applies for any fixed context and δ > 0.
  • Shift invariance of softmax permits centering each perturbation without changing the resulting model distribution.The centering operation preserves the relevant variance properties under the stated assumptions.
  • The proof separates concentration around the mean from the bias term in the aggregation error.Hoeffding’s inequality controls the concentration term because the model probabilities are independent and bounded.
  • Bounded perturbations allow exponential and normalization inequalities to produce upper and lower bounds on each perturbed distribution.The analysis assumes |δ_i(v)| ≤ ξ ≪ 1 and uses standard inequalities for |y| ≤ 1.
  • The final bound combines concentration and bias estimates, with vocabulary size entering through the union-bound term.The bound includes 2 log(2|V|/δ)/N and 2η², where η² bounds expected weighted perturbation variance.

B. Extension of the Theoretical Result to Grouped Watermarking Settings

The theoretical guarantee extends to providers grouped by shared perturbation structure. Within groups, common bias creates an irreducible term, while conditional independence controls idiosyncratic variation.

  • The analysis assumes full provider independence in its main setting, whereas real providers may share toolkits, agreements, or model families.These shared sources induce statistical dependence among perturbation vectors.
  • The grouped theorem extends convergence when providers form independent clusters with conditional independence within each cluster.The extension allows models in a group to share a common perturbation component.
  • The grouped setting partitions N providers into disjoint groups and assigns each group a latent variable governing shared dependence.Across groups the collections are mutually independent, while within groups perturbations are conditionally independent given the group variable.
  • Each perturbation is decomposed into a group-specific bias and an idiosyncratic component with bounded magnitude and conditional variance.The shared bias is defined through b_g, while ε_i captures provider-specific variation whose weighted variance is bounded by η².
  • Theorem B.2 bounds aggregation error by concentration, idiosyncratic variance, and the irreducible group bias B(x).The proof derives this result by conditioning on group variables, applying concentration, and bounding the second-order idiosyncratic term.
  • The proof uses conditional independence to apply Hoeffding’s inequality and then combines the resulting terms with the group-consensus bias.The final derivation substitutes the concentration, idiosyncratic-effect, and bias bounds.

C. Robustness to Biased Watermark Perturbations

Experiments stress-test the zero-mean perturbation assumption using a biased green-red-list watermark. Stronger bias improves detectability but sharply reduces task accuracy, while WASH remains below the detection threshold.

  • Stronger watermark bias makes the baseline easier to detect but rapidly harms task performance.Accuracy falls from 0.443 to 0.023 as the biased perturbation becomes stronger.
  • WASH keeps detector z-scores at or below 4 across bias values and improves accuracy over the corresponding biased baseline.The experiments evaluate both GSM8K accuracy and watermark detection z-score.
  • Shared bias leaves an irreducible convergence term, explaining why WASH detection z-scores do not vanish completely.Despite this residual term, the z-scores remain below the stated detection threshold.

D.1. Implementation Details

The implementation evaluates generation-time detection with z-scores and final-text detection with native sequence detectors. Detection confidence is categorized using explicit z-score thresholds.

  • Generation-time attacks use the z-score protocol to quantify watermark signal strength.High-confidence identification is z > 10, low-confidence identification is 4 < z ≤ 10, and no detection is z ≤ 4.
  • Final-text rewrite attacks use native sequence detectors because perturbation detection on a small set of generated tokens is incompatible.Sequences are generated on C4 using 30-token prompts followed by 200 generated tokens before detection.

D.2. Detailed Experiment Results

Ensemble size strongly reduces watermark detection signals, especially when both watermark schemes and base models vary. Fixed-base ensembles retain residual signal because shared model bias persists, while such ensembles are uncommon in practice.

  • Larger ensembles produce stronger watermark-signal decay across mixed watermark schemes and base models.The experiments sampled multiple ensemble combinations and repeated detection runs for stable z-scores.
  • ≈150 to ≈10: Llama3.1-8B’s detection z-score fell at N = 5 in the fixed-base setting.This empirical decay mirrors the predicted O(1/N) scaling.
  • Residual detection signal remains in fixed-base ensembles because the shared systematic bias δsys persists across identical base models.The mixed-base setting achieves more complete cancellation than the fixed-base setting.
  • The fixed-base scenario is rarely available to end users because providers seldom expose multiple watermarked versions of the same model.
  • Table 9 reports watermark-identification outcomes for mixtures of watermark schemes and base models, distinguishing high-confidence, low-confidence, and absent identifications.

E. Fluency-Aware Routing Example

Fluency-Aware Routing handles heterogeneous tokenisation during three-model watermarked generation. It routes differently tokenised spans through compatible specialist models and then re-synchronises them.

  • Fluency-Aware Routing Example: Three watermarked models—Llama-Aar, Ministral-DIPMark, and Qwen-KGW—generate the C4 completion and MMLU Law reasoning examples.
  • Fluency-Aware Routing Example: Routing is triggered mainly when models tokenise the same word differently.
  • Fluency-Aware Routing Example: For “negligent,” Ministral uses [‘negl’, ‘igent’], whereas Llama and Qwen retain a complete word token.
  • Fluency-Aware Routing Example: The routed span is completed only by specialist models sharing its tokenisation scheme, then re-synchronised with each model’s tokenizer.
  • Fluency-Aware Routing Example: Figure 4 presents complete generation flows for C4 completion and MMLU Law reasoning, including the intervening routing step.
Loading 2605.30501v1…