PaperlayerView exact revision on arXiv

Source-linked AI summary

WoE Wrote It? Watermarking Mixture-of-Experts LLMs for Black-Box Text Provenance

Jona te Lintelo, Lichao Wu, Stjepan Picek

arXiv:2608.29151v1cs.CR

TL;DR

Post-theft attribution is difficult because adversaries controlling stolen weights can bypass inference-time watermarking and modify deployment or models. WoE embeds a green-list bias in selected MoE expert weights and reconstructs routing traces from suspect text for black-box detection. Across eight architectures, it achieves strong detection with limited utility loss and retains signal under adaptation, extraction, and paraphrasing, subject to evidence-length and sparsity constraints.

  • Problem

    Model theft lets adversaries bypass inference-time watermarks, creating a need for black-box provenance from suspect text without access to their deployment or weights.

  • Method

    WoE derives a secret green list, learns its bias in selected expert weights through LoRA, and detects the signal by reconstructing routing trajectories from suspect text.

  • Results

    Across eight diverse MoE architectures, WoE achieves 90.1% average TPR at 1% FPR, reaches 94.9% on 300-token generations, and averages a 1.7-percentage-point accuracy decline.

  • Takeaways & Limitations

    WoE retains substantial detectability under sustained SFT, transfers through model extraction, and remains partially detectable after text-level perturbations.

  • Takeaways & Limitations

    Sequences shorter than approximately 50 tokens may contain too few target-expert activations for reliable detection, while targeting more experts can increase trigger frequency at a utility cost.

Abstract

from arXiv · show

Large Language Model (LLM) watermarks provide a mechanism for text provenance, enabling model owners to identify machine-generated content and attribute it to a specific watermarked model. However, current LLM watermarking approaches predominantly rely on inference-time sampler methods and focus their analysis on dense models. Inference-time methods are only effective when the text is explicitly generated via the model owner's controlled API; they fail in a post-compromise scenario. An adversary who steals or leaks the model weights gains complete control over inference and can simply run an unmodified sampler, bypassing the watermark and preventing post-theft attribution. In this work, we introduce Watermarking of Experts (WoE), a novel black-box text provenance method that leverages the unique structural properties of sparse Mixture-of-Experts (MoE) models. WoE biases the vocabulary of specific experts and shifts the watermark signal embedding away from unenforceable inference wrappers. This approach ensures the watermark remains intrinsic to the model parameters, enabling defenders to attribute text generated by stolen weights, leaked checkpoints, and secondary dense models distilled from the stolen architecture without needing access to the adversary's deployment or weights. We evaluate WoE across eight MoE models, demonstrating successful watermark detection from suspect text, achieving an average true positive rate of 90.1% at a 1% false positive rate, reaching up to 94.9%, while largely preserving general model utility. Furthermore, WoE remains detectable under adversarial supervised fine-tuning, model extraction, and output-level paraphrasing, forcing malicious actors into a trade-off in which weakening the attribution signal requires additional model adaptation or text-rewriting operations, or compromises the utility of the resulting output.

1. Introduction

Existing watermarking methods are vulnerable after model theft because inference-time controls can be bypassed, motivating WoE, which embeds provenance signals into selected MoE experts. Across eight MoE models, WoE detects suspect text while largely preserving utility and remaining resilient to several post-theft attacks.

  • Model theft removes owner control over inference, allowing adversaries to alter decoding, disable hooks, and fine-tune stolen weights before deployment.
  • Inference-time watermark wrappers depend on a controlled API, while zero-access black-box provenance for stolen weights remains underexplored.
  • WoE embeds an owner-specific vocabulary bias into selected expert weights, making the signal intrinsic to the MoE model rather than its generation-time sampler.
  • WoE creates a secret-seed green list, injects localized LoRA-based bias into selected experts, and detects overrepresented green-list tokens along reconstructed routing traces.
  • 90.1% average TPR at 1% FPR, reaching 94.9% for 300-token sequences, was achieved across eight MoE models with an average accuracy drop of 1.7 percentage points.
  • WoE was evaluated against supervised fine-tuning and semantic paraphrasing, exposing trade-offs among watermark evasion, attack cost, and model utility.

2. Background

LLM watermarking embeds detectable signals either during generation or in model weights, while MoE architectures achieve scalable computation through sparse expert routing. These architectural properties provide the context for expert-specific watermarking.

  • LLM watermarks support attribution by embedding statistically detectable signals into generated text.
  • Inference-Time Watermarking: Inference-time methods alter next-token sampling, commonly by boosting tokens from a context-dependent green list.
  • Weight-Embedded Watermarking: Weight-embedded methods modify models so watermark signals persist without requiring a particular sampling procedure, including distillation-based and direct parameter modifications.
  • Mixture-of-Experts Architectures: MoE architectures replace dense feed-forward networks with independent experts and activate only a sparse subset for each input token, reducing per-token computation.
  • Mixture-of-Experts Architectures: The router computes logits over experts and uses top-k selection, while non-selected experts remain inactive and selected outputs are aggregated.
  • Mixture-of-Experts Architectures: Shared-expert MoE variants add experts that bypass top-k routing and remain active for every token, unlike standard sparse MoE architectures.

3. Threat Model

The threat model considers adversaries with white-box access to watermarked MoE weights who seek to deploy, adapt, perturb, or distill them. Defenders have only suspect text and must perform zero-access black-box attribution.

  • The scenario covers stolen weights and unauthorized distillation, where adversaries control the weights and inference pipeline and can disable generation-time watermarks.
  • Attacker Capabilities: Attackers are assumed to know the WoE algorithm and possess the full watermarked checkpoint, but not the defender’s secret seed or an identical pre-watermarked checkpoint.
  • Attacker Capabilities: With white-box access, adversaries can inspect and modify weights, change decoding, deploy arbitrary infrastructure, paraphrase outputs, fine-tune to scrub the watermark, or distill a smaller model.
  • Defender Capabilities: The defender has no access to the adversary’s infrastructure, backend, or modified or distilled weights, and observes only suspect text from public interfaces or products.

4. Structural MoE Watermarking

WoE embeds a watermark in selected MoE expert weights rather than relying on inference-time wrappers, using sparse routing to localize green-list bias. Detection reconstructs routed positions with the original model and tests green-list hits against an empirical threshold.

  • WoE embeds a model-specific watermark in selected expert weights, leveraging sparse routing instead of continuously perturbing the global token distribution.The payload consists of a pseudorandom green list and selected layer–expert pairs; LoRA applies the localized bias.
  • 4.1. Payload Creation: The payload uses a secret seed K to derive green list G, while target configuration C specifies the sparse layer–expert pathways carrying the bias.The green-list fraction is fixed at γ = 0.25 across evaluations.
  • 4.1. Payload Creation: WoE selects three target layers at quartile positions and one median-frequency expert per selected layer to balance activation coverage with utility.Targeting several experts in one layer can overlap contributions without increasing distinct detector positions proportionally.
  • 4.2. Watermark Injection: LoRA adapters are applied to targeted expert feed-forward matrices and merged before deployment, while global projection and attention layers remain frozen.This localizes the immediate watermark bias to positions routed through targeted experts.
  • 4.2. Watermark Injection: The injection loss maximizes green-list probability at triggered positions and normalizes by all valid prediction positions to preserve activation-frequency effects across architectures.Increasing δ strengthens detectability while worsening text quality and perplexity; δ = 0 reduces to standard language-model fine-tuning.

5. Implementation and Evaluation Setup

The evaluation spans eight diverse open-source MoE models and uses C4 and ELI5 to cover high-entropy generation and long-form question answering. It profiles routing separately, selects target layers deterministically, and evaluates detectability, perplexity, and benchmark accuracy.

  • 5.1. Models: Eight recent open-source MoE models span varied experts, layers, parameter counts, developers, and top-k activation settings.The targeted models include DeepSeek-V2-Lite-Chat, GPT-OSS-20B, Hunyuan-A13B-Instruct, Mixtral-8x22B-Instruct-v0.1, Nemotron-3-Nano-30B-A3B, OLMoE-1B-7B-0125-Instruct, Phi-3.5-MoE-Instruct, and Qwen3.6-35B-A3B.
  • 5.2. Datasets: C4 supplies open-ended high-entropy generation, while ELI5 supplies long-form question answering requiring complex factual reasoning.For C4, the first 50 tokens prompt generation; for ELI5, the raw question prompts generation and the answer serves as human-written text.
  • 5.2. Datasets: WikiText-2-Raw-v1 profiles expert activation on 4,000 random samples, with profiling and evaluation datasets strictly separated to prevent leakage.The sample size was determined by monitoring activation distributions until no significant shifts were observed.
  • 5.3. Configuration: Target layers are selected deterministically at the first, second, and third quartiles of the MoE routing depth.The rule is Ltarget = {⌊L/4⌋, ⌊L/2⌋, ⌊3L/4⌋}.
  • 5.4. Evaluation Metrics: Detectability is reported as TPR at a strictly bounded 1% FPR using an unwatermarked architecture-specific null distribution and its 99th-percentile threshold.This setup uses the unwatermarked target architecture as a conservative null distribution.
  • 5.4. Evaluation Metrics: Utility evaluation combines oracle perplexity with accuracy on MMLU, ARC-Challenge, WinoGrande, and HumanEval.Oracle perplexity uses a fixed Qwen3-30B-A3B evaluator and excludes prompt and padding tokens.

6. Experimental Results

Across eight MoE models, WoE generally improves watermark detectability as watermark strength increases, while preserving benchmark utility better than perplexity alone suggests. Detection remains effective with moderate text lengths, exceeds 80% TPR for most models by 250 tokens, and persists under baseline comparisons, fine-tuning, extraction, and layer-selection studies.

  • Watermark Strength: Increasing δ improves TPR but raises perplexity, while benchmark accuracy changes modestly across architectures.The differing responses show that perplexity alone does not adequately represent practical model utility.
  • Watermark Strength: The selected operating point generally lies near a Pareto knee, before further δ increases cause diminishing detection gains and sharp accuracy declines.For Phi-3.5-MoE-Instruct, δ = 7 achieves 93.6% TPR with a 39.5% relative perplexity increase.
  • Task-Level Utility: At the selected operating points, WoE reduces average benchmark accuracy by 1.7 percentage points across eight models.The average decline ranges from 1.3 to 2.3 pp; MMLU declines by 0.6 pp on average, while HumanEval declines by 2.8 pp.
  • Sequence Length: Detection exceeds 80% TPR for most models by 250 tokens, after near-chance performance at T ≤50 and flattening beyond 300 tokens.Signal accumulation depends on how often routing activates targeted experts; denser routing produces faster detection than massively sparse routing.
  • Weight-Embedded Baselines: WoE achieves 89.0% average TPR versus 83.1% for KGW-D, a 5.9-point improvement at comparable average utility cost.WoE’s average perplexity increase is 35.6% versus 34.8% for KGW-D, while benchmark accuracy declines by 1.5 pp versus 1.8 pp.
  • Adversarial Robustness: WoE retains substantially higher detection rates than GaussMarkD throughout broad-domain SFT, while expert-localized injection preserves utility better than globally active perturbation.The watermark’s confinement to conditionally activated experts can reduce how uniformly fine-tuning updates watermark-bearing parameters.
  • Model Extraction: After distillation, the extracted LLaMA-3-8B student retains 81.2% TPR at 1% FPR versus 94.9% for the Mixtral teacher.The dense student partially dilutes the sparse routed signal but still transfers a substantial attribution signal.
  • Layer Selection: Distributing target layers across the first three quartiles provides the strongest detectability–utility balance, while late-layer targeting approaches perfect TPR but sharply reduces accuracy.Randomly selecting one layer from each of the first three quartiles performs comparably to the baseline distributed configuration.

7. Discussion

WoE is designed for practical, repeated provenance analysis while addressing deployment-stage and security trade-offs. Its main constraints are sufficient text length, sparse expert activation, and unresolved targeted attacks.

  • Practical Deployability and Detection Overhead: WoE learns green-list bias directly in selected expert pathways without generating a synthetic corpus.Detection uses a standard forward pass to reconstruct routing, followed by constant-time green-list membership tests.
  • Practical Deployability and Detection Overhead: Detection eliminates backward-pass computation during auditing, supporting repeated, large-scale provenance analysis.
  • Security of WoE: Randomized target selection prevents deriving the exact configuration from the public algorithm alone, although white-box inspection may reveal candidate pathways.
  • Security of WoE: Targeted identification and partial pathway compromise remain important adaptive attacks for future evaluation.
  • Limitations: Sequences shorter than approximately 50 tokens contain too few target-expert activations for reliable detection.Evidence can be aggregated across outputs, while targeting additional experts increases trigger frequency at a utility cost.

8. Related Work

Prior watermarking methods mainly modify inference-time sampling or model weights, but existing weight-based approaches were designed for dense architectures. WoE instead exploits sparse MoE routing to localize the watermark in expert weights.

  • Inference-Time LLM Watermarking: Inference-time watermarking injects a detectable signal by altering the sampling procedure during generation.
  • Weight-Perturbation-Based Watermarking: Weight-based watermarking embeds signals directly into model weights, including through quantization or perturbations to feed-forward layers.
  • Weight-Perturbation-Based Watermarking: Existing weight-based methods target dense architectures, where continuous parameter activation alters the global token distribution and can facilitate scrubbing.

9. Conclusion

WoE uses sparse MoE routing to embed and detect model-specific watermarks without access to an adversary’s deployment or weights. Across eight architectures, it preserves substantial utility while retaining detectability under several attacks and transformations.

  • WoE creates a green list and sparse target configuration, injects localized expert-weight bias via LoRA, and detects routing-conditioned overrepresentation against a calibrated threshold.
  • WoE enables model-specific attribution without access to the adversary’s deployment or model weights.
  • 90.1% average TPR at 1% FPR was achieved across eight MoE architectures, reaching 94.9% on 300-token generations.
  • Average benchmark accuracy declined by 1.7 percentage points while the watermark retained substantial detectability under sustained SFT, model extraction, and text-level perturbations.
  • Weakening the watermark requires additional model adaptation, output transformation, or compromises in output utility.

Appendix A. Per-Benchmark Utility Degradation

WoE’s utility impact varies across benchmarks and architectures. MMLU is least affected, whereas HumanEval shows larger declines associated with precise, low-entropy code-generation decisions.

  • 0.6 pp average decline makes MMLU the least affected benchmark across the evaluated architectures.The larger Mixtral-8x22B-Instruct-v0.1 and Nemotron-3-Nano-30B-A3B models show negligible MMLU declines, while OLMoE-1B declines by 1.2 pp.
  • 2.8 pp average decline makes HumanEval the most affected benchmark.
  • 4.0 pp was the largest reported code-generation decline, for Phi-3.5-MoE-Instruct, followed by 3.8 pp for Mixtral-8x22B-Instruct-v0.1.
  • Biasing selected expert outputs can interfere with valid code generation because code requires precise syntactic token choices.

Appendix B. Robustness Against Text Perturbations

WoE retains detectable attribution under random deletion and round-trip translation, although translation substantially weakens the signal. These perturbations expose a trade-off between evasion and preserving attribution performance.

  • Random Token Deletion: Random word deletion causes WoE detectability to decay continuously as fewer targeted-expert token events remain available.Deletion can also alter routing trajectories, but some detectability remains at minor to moderate deletion rates.
  • Round-Trip Translation: 37.6 percentage points: average TPR falls from 90.5% to 52.9% after English–French–English round-trip translation across four models.Post-translation TPR ranges from 41.0% for Nemotron to 64.7% for Mixtral at 1% FPR.
  • Round-Trip Translation: Round-trip translation weakens WoE because it changes token identities and causal contexts, disrupting green-list membership and reconstructed routing trajectories.This degradation is described as an inherent limitation of token-level and structural watermarks.
  • Round-Trip Translation: 41.0% to 64.7%: translated outputs remain detectable at a strict 1% FPR despite weakened attribution signals.The evaluated translation attack does not uniformly eliminate detection.
  • Adversarial Trade-off: Automated rewriting can remove the watermark only with a trade-off involving a different rewriting model or other output-level adaptation.The reported trade-off concerns weakening attribution while modifying text generated from stolen weights or the defender’s API.

Appendix C. Extended Ablation Studies

Extended ablations identify how layer depth and expert-selection frequency shape WoE’s detectability–utility trade-offs across representative MoE architectures. The results support mid-frequency expert selection and show that targeting later layers improves detection at an accuracy cost.

  • Cross-Architecture Findings: The reported layer-depth and expert-frequency trade-offs hold across varying expert counts and top-k routing configurations.The consistency is reported for four representative MoE architectures, including OLMoE-1B-7B, Mixtral-8x22B, DeepSeek-V2-Lite, and Nemotron-3-Nano.
  • Layer Depth: Deeper routing networks experience greater signal dilution when injection is restricted to early layers, producing lower TPR.Table 9 identifies Mixtral’s 56 layers as an example of this depth-related dilution.
  • Layer Depth: Late-stage layer targeting maximizes detectability but degrades benchmark accuracy by overriding the final semantic projection.The injection volume is held constant at three layers in the reported comparison.
  • Expert Frequency: 75th-percentile expert targeting provides viable mid-frequency choices that help prevent targeted adversarial scrubbing.Core experts degrade utility, whereas rare experts provide insufficient detectability.
  • Layer Volume: Increasing the absolute number of targeted layers yields only small detectability returns while degrading downstream baseline utility.This establishes a layer-volume trade-off distinct from the layer-depth effects reported in Table 9.
  • Expert Volume: Targeting multiple experts per layer introduces statistical redundancy, causing accuracy drops with negligible TPR gains.The ablation compares increasing expert volume against detectability and downstream utility.
Loading 2608.29151v1…