PaperlayerView exact revision on arXiv

Source-linked AI summary

Stability-Aware Feature Design for Robust Watermark Detection in Machine-Generated Text

Sina Mansouri, Mohit Marvania, Abolfazl Safikhani

arXiv:2608.18102v1cs.CLcs.LGstat.ML

TL;DR

Watermark detectors can lose effectiveness after repeated paraphrasing and on shorter texts. This paper introduces Pattern Stability Score, combining local watermark statistics with cross-paraphrase stability, and reports stronger robustness, including 90.0% average AUC when all tested components differ simultaneously.

  • Problem

    Existing watermark detectors can deteriorate under repeated paraphrasing and shorter text lengths, limiting reliable identification of machine-generated content.

  • Method

    Pattern Stability Score combines local rolling-window statistics with watermark-signal stability across paraphrase depths.

  • Results

    Across varied LLMs, paraphrasers, domains, and paraphrase depths, one universal classifier generalizes without retraining, achieving 90.0% average AUC when all components change simultaneously.

  • Takeaways & Limitations

    The detector’s simple implementation and low inference overhead support practical real-world attribution settings.

  • Takeaways & Limitations

    The threat model assumes black-box paraphrasers without watermark-key access, while fully PSS-adaptive attacks remain unevaluated.

Abstract

from arXiv · show

The widespread adoption of large language models (LLMs) has intensified the demand for principled methods to distinguish human from machine-generated text. Watermarking provides a promising avenue, yet existing detectors exhibit sharp performance deterioration under multiple paraphrasing and when applied to shorter texts. We introduce Pattern Stability Score (PSS), a novel detection framework that leverages local statistical features and stability dynamics across paraphrased variants. Specifically, the proposed method combines global and local z-score features with higher-order statistics of run-length patterns, enriched by autocorrelation signals and stability scores computed over paraphrase depth. Numerical evaluations are performed on three benchmark datasets (PG-19, CNN/DailyMail, and WikiText) using multiple LLMs (Llama-3-8B, Qwen2-7B) and paraphrasers (Mistral-7B, Qwen2-7B, Gemma-7B), systematically stress-testing robustness under up to eight rounds of paraphrasing. Compared to prior z-score thresholding baselines and some state-of-the-art deep learning methods, our approach improves detection AUC (area under the receiver operating characteristic curve) by over 10-15 percentage points across different token lengths. Additionally, extensive cross-domain experiments demonstrate that a single universal classifier generalizes across different LLMs, paraphrasers, and text domains without retraining, maintaining above 87.8% AUC even when all components differ from training.

1. Introduction

The paper introduces Pattern Stability Score (PSS), which combines local watermark evidence with stability across paraphrase depth to address robustness under multi-step paraphrasing and short texts. Evaluations across multiple datasets, models, and paraphrasers show strong performance, including 91.2% AUC at 1,500 tokens after eight paraphrasing rounds.

  • Motivation: PSS addresses two open watermark-detection gaps: robustness to multi-step paraphrasing and stability on short texts.The study models a black-box adversary that paraphrases text up to K steps while preserving semantics and approximate length.
  • Method: PSS extracts local watermark evidence with rolling windows and quantifies its stability across paraphrase depth.The method expands windows minimally when they split runs, then computes local statistics and depth-wise variability.
  • Method: The hybrid feature design combines local z-score moments, short-range autocorrelations, longest-run statistics, and a stability functional with 20 static features.Local features preserve spatial structure, while PSS captures central tendency and variability across paraphrase depths.
  • Evaluation: The evaluation uses PG-19, CNN/DailyMail, and WikiText with watermarked passages generated by Llama-3-8B and Qwen2-7B.The study compares performance across benchmark corpora and multiple LLM configurations using greenlist watermarking.
  • Evaluation: 91.2% AUC is achieved by PSS + Static at 1,500 tokens and depth D8, while deep learning baselines collapse to 41–44%.The comparison includes DeepTextMark, Binoculars, and RADAR, alongside the traditional global z-score baseline.

2. Related Work

Prior watermark detectors commonly reduce evidence to a global green-token z-score, but this aggregation is fragile for short or paraphrased text. Related work therefore explores local statistics, deep-learning and semantic methods, adaptive schemes, theoretical limits, and direct paraphrase attacks.

  • Watermarking for LLMs: Greenlist watermarking biases sampling toward keyed-hash vocabulary partitions, then tests whether the observed green-token proportion is unusually high.The standard detector compresses this evidence into a global z-score, often using z-score > 4 as a fixed threshold.
  • Watermarking for LLMs: Global aggregation is fragile when text is short or paraphrased, because paraphrasing disperses green runs, alters token dependencies, and reduces the global statistic.Follow-up work also studies trade-offs among watermark bias, quality, false positives, channel constraints, and adversarial distortion.
  • Deep Learning-Based Detection Methods: DeepTextMark, Binoculars, and RADAR represent deep-learning alternatives using neural features, likelihood ratios, and adversarially robust feature extraction.The passage also mentions commercial classifiers, but its text is truncated before naming them.
  • Local vs. global statistics for detection: Local detection preserves spatial structure through rolling windows, run-length distributions, and short-range autocorrelations; WinMax uses the maximum local z-score.These structures are described as costlier for paraphrasers to randomize without semantic drift.
  • Local vs. global statistics for detection: 65% power improvement over baselines is reported for likelihood-based detection, while theoretical work derives asymptotic error rates, optimal rules, and minimum Type-II error limits.These methods estimate null token probabilities or characterize optimality across watermarking schemes.
  • Paraphrasing Detection and Inversion: Over 96% evasion rate against surveyed watermarks is achieved by adaptive Direct Preference Optimization attacks, while cross-lingual attacks expose further weaknesses.Separate methods detect machine paraphrase patterns directly or attempt to invert paraphrases.

3. Proposed Methodology

The methodology combines local watermark statistics with Pattern Stability Scores (PSS) that measure consistency across paraphrase depths. A hybrid feature vector integrates stability profiles with static z-score, autocorrelation, run-length, and frequency features for classification.

  • Local Statistical Features: Local rolling-window analysis computes z-scores across token indicators, preserving where watermark evidence concentrates rather than relying only on global aggregation.Windows slide across the token sequence and its greenlist-indicator sequence.
  • Local Statistical Features: The 20-dimensional local feature set combines z-score summaries, autocorrelations, run-length statistics, and run-frequency statistics to capture spatial concentration and short-range dependencies.It contains 8 z-score features, 6 run-length features, and 6 run-frequency features.
  • Pattern Stability Score: Watermarked text is expected to retain more stable local patterns across paraphrasing, whereas human text exhibits higher local z-score variability.The distinction motivates using cross-depth stability as a detection signal.
  • Pattern Stability Score: PSS measures variability in aligned local z-score trajectories across paraphrase depths, producing one stability score for each window position.For evaluation at depth Dj, PSS uses depths Dj through DK and aligns sequences to the minimum window count.
  • Classifier Features: The classifier receives a hybrid feature vector containing PSS values across window positions and static features from the current text.Static features include z-score summaries, autocorrelations, run-length patterns, and frequency statistics.

4. Experiments and Results

Experiments across three corpora and multiple LLMs evaluate PSS under varying token lengths, paraphrase depths, adaptive attacks, and deployment-oriented metrics. PSS + Static consistently improves robustness over global and deep-learning baselines and transfers across domains and configurations.

  • Experimental setup: Experiments use PG-19, CNN/DailyMail, and WikiText, with Llama-3-8B and Qwen2-7B watermarked passages evaluated at 300, 500, 1,000, and 1,500 tokens.Standard greenlist watermarking uses γ=0.25, δ=1.5, and a fixed hash key.
  • Paraphrase robustness: PSS + Static remains comparatively flat through mid and late paraphrase depths, whereas the global z-score declines steadily and local features provide intermediate robustness.Figure 2 evaluates AUC across depths D1–D8 at 1,500 tokens using 30 runs with random 70/30 splits.
  • Comparison with prior detectors: 14–19 percentage points separate PSS + Static from WinMax across D1–D8 on PG-19 at 1,500 tokens, while deep-learning detectors trail by 30–50 percentage points.WinMax reaches 82.1% at D1, but reducing local information to a single maximum leaves the reported gap to PSS + Static.
  • Adaptive attack: >80% AUC is maintained by PSS + Static against the DPO-optimized adaptive attack, compared with near-random performance for the global z-score.The evaluation is attributed to Diaa et al. (2025), with full details deferred to Appendix A.6.
  • Deployment metrics: 2× improvement at FPR=1% is achieved by PSS + Static over the global z-score, including 84% vs. 42% at D1 and 64% vs. 24% at D8.These are TPR values reported across paraphrase depths, confirming transfer of AUC findings to threshold-based deployment metrics.
  • Generalization: 84–89% AUC is retained when training on PG-19 and testing on CNN/DailyMail, while deep-learning methods fall to 29–46%; a universal classifier reaches 91.6% when the LLM changes to Qwen2.The universal classifier is trained on Llama-3 + Mistral + PG-19 + D1-D3; changing the paraphraser to Gemma yields 83.1%.

5. Concluding Remarks

The paper concludes that PSS is a stability-aware, keyed detector combining local rolling-window statistics with paraphrase-depth stability, while noting limitations in its black-box threat model and adaptive-attack evaluation. Experiments support universal cross-configuration generalization and improved performance over traditional and deep-learning baselines.

  • Concluding Remarks: PSS fuses local rolling-window statistics with a stability score computed across paraphrase depth for detecting watermarked LLM text.The detector is keyed and designed to preserve spatial structure.
  • Concluding Remarks: Comprehensive experiments across datasets, LLMs, and paraphrasers show PSS outperforming traditional baselines and state-of-the-art deep learning methods.A single universal classifier generalizes across all evaluated configurations.
  • Limitations: The threat model assumes a black-box paraphraser without access to the watermark key, preserving watermark confidentiality.The evaluated adaptive attack is optimized against global z-score rather than PSS-specific signals.
  • Limitations: A fully PSS-adaptive attack would require white-box access to the feature design and joint optimization across windows and depths.The passage characterizes this requirement as substantially stronger than the evaluated attack setting.

Impact Statement

The work aims to improve watermark-based identification of machine-generated text, supporting academic integrity, journalism verification, and content authenticity. Its detector-side techniques use existing watermarking infrastructure without modifying generation systems, while acknowledging potential adversarial misuse.

  • The paper advances machine learning by targeting improved detection of machine-generated text through watermark detection.
  • Improved identification of AI-generated content could support academic integrity, journalism verification, and content authenticity.
  • The techniques are detector-side improvements compatible with existing watermarking infrastructure and require no changes to generation systems.
  • Although adversaries could theoretically study the methods to evade detection, the paper states that improved detection benefits outweigh these risks.

A. Appendix

The appendix evaluates four detector families, including global and local z-score features, static statistical summaries, and PSS-enhanced static features. Classifier-based models use XGBoost unless otherwise stated.

  • Setup recap: The evaluation compares four detector families: Global z threshold, Local z (20), Static features, and PSS + static.Local z (20) uses the first 20 rolling-window local z’s as features.
  • Setup recap: Static features combine windowed moments, short-range autocorrelations, run-length summaries, and run-frequency summaries.These are evaluated as a standalone detector family and as the basis for PSS + static.
  • Setup recap: PSS + static augments static features with Pattern Stability Scores computed from standard deviation of aligned local z-score trajectories across depths.The PSS features capture stability dynamics across paraphrase depths.
  • Setup recap: All classifier-based detector lines use XGBoost unless otherwise stated.The setup also specifies window usage with w=, although the supplied passage leaves the value incomplete.

A.1. Sensitivity to Paraphraser Choice

Across Gemma-7B-IT and Qwen2-7B-Instruct paraphrasing, method rankings remain consistent: PSS + static degrades most slowly and achieves the highest mid- and late-depth accuracies. These results support paraphraser-agnostic detection through local concentration and cross-depth persistence.

  • Evaluation setup: The evaluation reruns the full pipeline with Gemma-7B-IT and Qwen2-7B-Instruct, generating D1–D8 paraphrase chains under unchanged prompts and decoding settings.This tests whether cross-depth local watermark evidence persists across rewriting models.
  • Results: Global thresholding drops fastest with depth, locality slows degradation, and PSS + static produces the flattest curves and highest mid- and late-depth accuracies.The ordering is consistent across both paraphrasers, with full AUC values reported in Tables 4 and 5.
  • Results: The stability signal adds to locality, preserving method margins when token distributions shift under different rewriting policies.This indicates that stability-aware evidence remains useful beyond the paraphraser used in the main text.
  • Takeaway: Consistent rankings support paraphraser-agnostic detection based on local concentration and cross-depth persistence, which are difficult to erase simultaneously without semantic drift or length distortion.The takeaway identifies these properties as the detector’s relevant invariants.

A.2. Additional Sensitivities

Additional sensitivity analyses show that PSS + Static remains robust across window/stride settings, classifier choices, watermarking schemes, shorter texts, and alternating paraphrasers. The method’s stability cue appears broadly applicable, with performance driven primarily by feature design rather than model complexity.

  • Window and stride: 99.0% mean AUC is achieved by the best window/stride setting (50, 15), versus 95.95% for the lowest setting (50, 10), a 3.05-point spread.The configuration used throughout, (50, 10), lies within the observed performance plateau.
  • Classifier choice: XGBoost is typically best among LR, RF, XGB, SVM, and kNN, while gains primarily trace to feature design rather than model complexity.The same ordering holds across the evaluated classifiers for local/static features.
  • Watermarking scheme compatibility: 97.6% AUC at D1 and 94.2% at D8 are obtained with γ=0.5, compared with 96.1% at D1 and 91.2% at D8 for γ=0.25.These results support compatibility with different greenlist ratios.
  • Shorter texts: PSS + Static retains the largest margins on shorter texts, particularly beyond D3, while static features consistently outperform global baselines across depths.All methods degrade as sequences shorten, but locality and stability mitigate the drop.
  • Paraphraser independence: PSS + Static maintains the leading curve and degrades more slowly under an alternating Mistral-7B-Instruct and Qwen2-7B-Instruct paraphraser schedule through D8.This suggests the stability cue is not tied to one paraphraser’s idiosyncrasies.

A.3. Practical Deployment Metrics

PSS is evaluated using deployment-oriented detection rates at fixed low false-positive thresholds and remains effective across paraphrase depths and realistic attacks. It achieves strong detection under severe paraphrasing, manual edits, and mixed-model paraphrasing while preserving practical robustness.

  • Fixed-FPR deployment metrics: PSS is assessed by True Positive Rate at fixed False Positive Rate thresholds of 1% and 5% across paraphrase depths.These thresholds target deployment settings where minimizing false accusations of human-written text is paramount.
  • Fixed-FPR deployment metrics: 92% TPR at D1 is maintained at 75% TPR at D8, demonstrating robustness across eight paraphrasing rounds.The results are reported under minimal false positive rates relevant to high-stakes applications.
  • Semantic preservation: At D1-D3, where semantic similarity is ≥0.85, PSS + Static achieves 93.9–96.1% AUC versus 68–74% for global z-score and 47–58% for DeepTextMark.At D8, semantic similarity drops to 0.68 while PSS maintains 91.2% AUC.
  • Realistic attack scenarios: 95.4% AUC under typical 1-2 paraphrase attacks declines to 91.8% with 20% manual edits, 88.9% with 30% edits, and 90.6% under mixed-model chain paraphrasing.The reported graceful degradation is attributed to multi-window analysis, in which unmodified windows provide stability signals.

A.4. Compute, Implementation, and Qualitative Examples

PSS detection is computationally lightweight, requiring seconds and modest memory while using linear-time inference. The implementation uses rolling-window features, stability scores, and alignment of local z-score sequences across paraphrase depths.

  • Computational efficiency: 0.8–3.2 seconds are required to detect 300–1500-token passages, with rolling-window extraction and stability computation dominating runtime.Rolling-window extraction accounts for 44–45% of detection time, stability computation for 35–37%, and XGBoost classification for ≤6%.
  • Computational efficiency: 200MB memory is required by PSS detection, compared with 8–16GB for transformer-based approaches.This memory profile enables deployment on resource-constrained systems and processing over 10,000 documents per GPU day.
  • Implementation: O(n) inference computes green indicators, window features, and PSS, with memory linear in text length.Feature aggregation costs O(n/w), using default window size w=50 and stride 10; classifier inference costs O(d), with d=20 for local-only features.
  • Implementation: PSS aligns local z sequences across paraphrase depths to the minimum window count before computing per-position standard deviation.Local windows use a non-fragmenting rule that minimally expands windows to avoid cutting consecutive green runs and shrinks the final tail window when needed.

Pseudocode: PSS + Static (Dataset-Level, Matches Implementation) · A.5. Failure Cases and Limits of Detectability · A.6. Adaptive Attack Evaluation

The section specifies dataset-level PSS + Static processing and deployment tiers, then identifies short-text and adaptive-attack limits while showing PSS retains substantial robustness against global-z-score-targeting attacks. It recommends configurations that trade latency and paraphrasing cost against detection accuracy.

  • Pseudocode: PSS + Static (Dataset-Level, Matches Implementation): The implementation aligns rolling-window z-score features across selected paraphrase depths, truncates them to a common window count, and stacks depth-wise tensors for PSS computation.Inputs include CSVs with identifiers, labels, one z-score per window, and static features; experiments use specified depth sequences, split ratios, seeds, and XGBoost hyperparameters.
  • Pseudocode: PSS + Static (Dataset-Level, Matches Implementation): 78–88% AUC is achieved by static-only 20-D features across D1–D8 at 1,500 tokens, exceeding global z-score by 12–14 points and WinMax by 6–7 points.This tier requires no paraphrasing and runs in sub-second time with the same overhead as the global z-score detector.
  • Pseudocode: PSS + Static (Dataset-Level, Matches Implementation): 91–96% AUC is delivered by full PSS + Static, which requires generating one paraphrase and incurs approximately 15–30 seconds per text on an A100.For short texts, PSS + Static retains the leading curve down to 300 tokens, although absolute performance decreases.
  • Pseudocode: PSS + Static (Dataset-Level, Matches Implementation): 83–85% AUC at D8 is recommended for high-throughput deployment using static features only, compared with 40–66% AUC for all baselines.A balanced configuration generates one paraphrase and achieves >90% AUC with two depths, while full chains suit high-stakes verification.
  • A.5. Failure Cases and Limits of Detectability: At 300 tokens and depths ≥D7, all methods degrade substantially because rolling windows cover more of the text and provide less spatial diversity, although PSS + Static remains best.These failures reflect weak underlying watermark signals in very short texts rather than a detector-specific methodological weakness.
  • A.5. Failure Cases and Limits of Detectability: 83.2% AUC is retained by PSS + Static under the DPO-optimized adaptive attack at naive D1, down from 96.1%, because aggressive token substitution eventually damages local statistics.The attack targets the global z-score, but sufficiently suppressing green tokens also removes the local signal PSS exploits.
  • A.6. Adaptive Attack Evaluation: 83.2% and 80.6% AUC are maintained by PSS + Static at adaptive D1 and D2, versus global-z-score values of 52.1% and 53.6%, respectively.This represents a 30+ percentage point advantage against an attack fine-tuned to minimize the global z-score.
Loading 2608.18102v1…