PaperlayerView exact revision on arXiv

Source-linked AI summary

SAC-Copula: Quality-Preserving Watermarking for Diffusion Language Models via Smooth Correlated Gumbel Fields

Baixin Li, Haiyun He

arXiv:2608.20839v1cs.CLcs.CRcs.LG

TL;DR

DLM watermarking needs methods compatible with iterative parallel masked denoising, where autoregressive synchronization and position-wise i.i.d. perturbations may misalign with decoding dynamics. SAC-Copula uses smooth locally correlated Gumbel fields and a SAC-aware detector, achieving a strong quality–detectability trade-off with improved PPL tail stability and strong low-FPR detectability across evaluated settings.

  • Problem

    DLMs generate through iterative masked denoising with parallel updates and token revisions, creating a watermarking setting unlike sequential autoregressive decoding.

  • Method

    SAC-Copula constructs locally correlated Gumbel perturbation fields through a Gaussian copula and detects them using covariance-aware filtering with native-sample calibration.

  • Results

    SAC-Copula achieves a strong quality–detectability trade-off, with improved PPL upper-tail stability over matched i.i.d. Gumbel while retaining strong low-FPR detectability across models and datasets.

  • Takeaways & Limitations

    The results support treating the joint geometry of watermark perturbations as a DLM-specific design dimension for balancing generation quality and detectability.

  • Takeaways & Limitations

    Detection requires private-tape access, representative calibration data, and calibrated H0 samples; cumulative drift and paraphrastic rewriting remain challenging.

Abstract

from arXiv · show

Watermarking diffusion language models (DLMs) requires mechanisms compatible with iterative parallel unmasking rather than autoregressive decoding. Existing sampling-based watermarking methods typically inject position-wise i.i.d. perturbations, which can be poorly aligned with DLM decoding dynamics and degrade generation quality. We propose SAC-Copula, a quality-preserving watermarking method for DLMs based on smooth, locally correlated Gumbel perturbation fields constructed via a Gaussian copula. We further develop a SAC-aware detector using covariance-aware filtering and native-sample calibration. Mechanism-level analysis shows that local correlation reduces latent perturbation roughness and better matches iterative refinement dynamics. Experiments on LLaDA show that SAC-Copula achieves a favorable quality-detectability trade-off compared with existing baselines. In particular, further evaluations on Dream-7B and additional datasets show that SAC-Copula substantially improves PPL tail stability over the i.i.d. Gumbel baseline, while maintaining strong low-FPR detectability and competitive overall generation quality. Additional token-edit stress tests further assess watermark robustness under controlled synchronization drift.

1 Introduction

DLM watermarking must accommodate iterative, parallel masked denoising rather than autoregressive decoding. SAC-Copula addresses this mismatch with locally correlated Gumbel fields and SAC-aware detection, aiming to preserve quality while retaining detectability.

  • Motivation: Position-indexed i.i.d. Gumbel perturbations create independent high-frequency noise across neighboring positions, potentially conflicting with DLM refinement dynamics.The paper motivates smoother local dependence because nearby masked positions are refined together and benefit from more coherent perturbation preferences.
  • Motivation: DLMs update multiple positions in parallel and can revise generated tokens, making watermark synchronization fundamentally different from autoregressive decoding.Autoregressive methods synchronize signals with left context and sequential generation, whereas DLMs use iterative masked denoising.
  • Method: SAC-Copula constructs smooth, locally correlated Gumbel perturbation fields through a Gaussian copula while preserving one-coordinate Gumbel marginals under stated assumptions.The method treats the joint geometry of the perturbation field as a key design dimension for DLM decoding.
  • Mechanism: Mechanism analysis shows that positive low-lag dependence reduces latent perturbation roughness, lowering local-difference energy from D_i.i.d. = 2 to D_SAC = 2(1 − ρ^2 r_1).This provides a mechanistic rationale for the hypothesis that smoother fields better align with DLM refinement.
  • Method: The method adds a SAC-aware detector using covariance-aware filtering and native-sample calibration, with a global-offset scan for controlled insertion/deletion diagnostics.The global-offset scan provides coarse synchronization recovery under token–signal misalignment.
  • Results: Experiments on LLaDA report strong clean detectability and a balanced quality profile, with the clearest gain being substantially improved upper-tail stability over matched i.i.d. Gumbel.The broader evaluation covers semantic, distributional, repetition, diversity, and token-drift dimensions.

2 Watermark Injection and Detection

SAC-Copula injects a smooth, locally correlated Gumbel field into masked positions and detects its structured evidence with covariance-aware filtering and native-sample calibration.

  • Watermark Injection: SAC-Copula replaces position-indexed i.i.d. Gumbel perturbations with a smooth, locally correlated field for masked-position DLM decoding.The field is constructed from keyseeded Gaussian tapes, spatial smoothing, and a Gaussian-copula transform that preserves one-coordinate Gumbel marginals under stated assumptions.
  • Watermark Injection: The matched i.i.d. baseline is the ρ = 0 endpoint, with α = 1.0, temperature 1.0, and the same generation and decoding pipeline.Only the cross-position dependence structure varies through W, σK, and ρ in the main comparison.
  • SAC-Aware Detection: The detector reconstructs keyed token-level evidence without a second model evaluation, then filters and centers it using native-sample statistics.FFR uses a filter motivated by the SAC kernel autocorrelation envelope and covariance estimates from held-out native samples.
  • SAC-Aware Detection: FFR calibrates thresholds from held-out native samples and reports ROC-based AUC and TPR at target FPR levels under the corresponding native-score distribution.This provides a matched linear readout for the structured evidence geometry induced by SAC-Copula.
  • Robust Detection: GO-FFR extends FFR with a global-offset scan that partially recovers mismatched token–signal alignment under controlled insertion and deletion attacks.It reuses the FFR scoring direction and calibrates its threshold under the same offset-scan rule.

3 Why Correlated Perturbations Help DLMs

DLMs revise multiple masked positions through iterative denoising, making neighboring perturbation geometry relevant to decoding. SAC-Copula targets this dynamic by replacing sharp i.i.d. variation with locally correlated perturbations that reduce latent roughness.

  • DLM Decoding Dynamics: DLMs predict multiple masked positions in parallel, accept or remask tokens by confidence, and revise partially generated sequences across denoising steps.Neighboring positions therefore participate in iterative refinement rather than isolated one-step decisions.
  • Why Correlation Helps: Position-wise i.i.d. Gumbel perturbations can vary sharply across neighboring masked positions refined together, potentially mismatching local consistency preferences.This motivates treating perturbation-field geometry as a design dimension beyond marginal strength.
  • Mechanism: SAC-Copula preserves each coordinate's standard Gumbel marginal while changing the perturbation field's joint structure across positions.The latent difference proxy concerns dependence before the nonlinear copula transform, not invariance of the full generated-text distribution.
  • Empirical Prediction: Figure 3 shows positive low-lag dependence alongside a corresponding reduction in the latent roughness proxy.These diagnostics support the hypothesis that correlated perturbations better align with DLM refinement dynamics and quality.

4 Experiments

Experiments evaluate SAC-Copula across detection, generation quality, correlation strength, calibration, transfer, and controlled token-edit robustness. The results show improved PPL-tail stability and strong low-FPR detection, with balanced performance at moderate correlation and partial recovery under token edits.

  • Correlation Strength: ρ = 0.6 yields FFR AUC/TPR@1%FPR of 0.9900/0.980, compared with 0.9683/0.820 at ρ = 0, while ρ = 1 degrades broader quality.P99 decreases from 4256.63 at ρ = 0 to 99.65 at ρ = 0.6 and 27.55 at ρ = 1.
  • Text Quality and Stability: P99 decreases from 4256.634 to 99.653 and PPL > 100 from 3.5% to 1.0% versus matched i.i.d. Gumbel, while median PPL remains close.Median PPL is 8.446 for SAC-Copula versus 8.260 for matched i.i.d. Gumbel; the clearest gain is reduced severe upper-tail failures.
  • Clean Detection: SAC-Copula reaches AUC/TPR@5%FPR of 0.987/0.950 under the legacy readout, while FFR raises TPR@1%FPR/TPR@5%FPR to 0.980/0.985.The legacy readout already distinguishes i.i.d. and SAC signals, while FFR provides an additional covariance-aware readout gain.
  • Calibration: At N = 500 native calibration, AUC/TPR@1%FPR/TPR@5%FPR reaches 0.9900/0.980/0.985, improving most at the strict 1% operating point.Across alternative H0 sources at N = 500, AUC is 0.9861–0.9878 and TPR@1%FPR is 0.950–0.955.
  • Calibration: Threshold transfer to 2,000 held-out C4 records produces realized FPR of 0.95%/5.50% at nominal 1%/5%, testing detector-only operational transfer.The transferred thresholds are from the original LLaDA–ELI5 model-native-H0 calibration and do not involve C4 generation.
  • Targeted Model/Task Transfer: Across Dream-7B/ELI5 and LLaDA/C4-en, SAC-Copula maintains or improves detection while reducing P99 and tail-failure rates with nearly unchanged median PPL on C4-en.On LLaDA/C4-en, median PPL is 13.71 versus 13.73; Table 2 defines tail failure as Composite collapse for Dream-7B/ELI5 and PPL >100 for LLaDA/C4-en.
  • Residual Evidence under Controlled Token Edits: At 20% edits, GO-FFR raises TPR@1%FPR from 0.150 to 0.460 under deletion and from 0.195 to 0.505 under insertion, while low-FPR detection remains degraded.For substitution, SAC+FFR reaches TPR@1%FPR of 0.890 versus 0.755 for i.i.d.+Old.

5 Discussion and Limitations

SAC-Copula targets a strong quality–detectability trade-off through smooth, locally correlated DLM perturbations, but its latent-field rationale does not establish invariance or downstream text-quality guarantees. Detection and deployment also depend on calibration data and private-tape access.

  • Discussion: SAC-Copula achieves a strong quality–detectability trade-off, with improved upper-tail stability relative to matched i.i.d. Gumbel and a competitive broader profile.The latent-field analysis is a mechanism-level rationale rather than proof of invariance of the complete generated-text distribution or downstream quality.
  • Detection: FFR uses covariance-matched evidence, while GO-FFR adds coarse global-offset recovery for insertion/deletion but leaves cumulative local drift unresolved.The detector covariance is estimated from extracted Native evidence rather than taken directly from the latent tape.
  • Deployment: Detection relies on calibrated H0 samples and development data, and deployment requires private-tape access plus representative calibration data.The evaluation includes calibration-sensitivity and frozen-threshold transfer controls.
  • Scope: The smooth-field claims concern the latent watermark tape, not invariance of the complete output-text law.Boundary deviations under zero padding are treated as implementation checks rather than part of the exact marginal claim; numerical clamping also qualifies ideal Gumbel statements.

A.3 Finite-Window Latent Autocorrelation

The appendix derives finite-window autocorrelation for SAC-Copula and uses it to motivate covariance-aware detector filtering, while distinguishing latent-process diagnostics from output-text distributional claims.

  • Autocorrelation: The finite-window autocorrelation is zero beyond the kernel support, symmetric across lags, and normalized so r0 = 1.The white stream contributes no cross-position covariance.
  • Detector rationale: Positive local dependence increases null variance and reduces effective sample size, motivating HAC or covariance-aware calibration.Calibration alone cannot fix a useful watermark shift that is misaligned with the equal-weight direction 1.
  • FFR: FFR combines SAC-motivated filtering with Native H0 observed-evidence covariance whitening.The filter aggregates evidence over the local low-lag structure introduced by the correlated perturbation field.
  • FFR: FFR reduces to the original equal-weight Gumbel detector when its direction is proportional to 1.Thus FFR generalizes the original detector by accounting for covariance and direction as well as SAC structure.
  • GO-FFR: GO-FFR scans a finite global-offset family and calibrates its threshold on Native H0 samples under the same scan rule.This recovers coarse global synchronization within the scanned family, while cumulative local drift and semantic rewriting remain outside it.

B.6 Scope and Limitations

The evaluated alignment extension covers mild insertion/deletion through global offsets, but semantic rewriting and cumulative edit-path drift remain outside its analyzed scope.

  • Scope: FFR adapts detection to correlated evidence, while GO-FFR provides coarse synchronization recovery for mild insertion/deletion.The method uses detector-specific Native H0 calibration and estimates its direction without evaluation leakage.
  • Scope: Semantic rewriting and cumulative edit-path drift remain outside the global-offset family analyzed here.The limitation follows from restricting alignment to global offsets rather than modeling arbitrary drift or rewriting.

C Additional Experiment Evidence

The experiment appendix indexes supporting evidence for clean comparisons, detector checks, parameter sweeps, edit stress tests, and tail-stability analyses without introducing additional experimental values.

  • Evidence organization: The appendix records supporting clean-setting summaries, frozen-output re-analysis, calibration controls, targeted-transfer controls, and frozen attack-summary evidence.Clean quality and detection claims rely on frozen clean multi-baseline sweep summaries rather than attack-summary evidence.
  • Evidence organization: The evidence index covers detector comparisons, SAC parameter sensitivity, token-edit sweeps, failure analyses, tail-stability tables, and a six-point ρ sweep.These materials support the main experimental sections and selected operating-point analysis.
  • Evidence organization: Table 3 is organizational only: each row points to compact appendix evidence supporting Section 4 and introduces no additional experimental claims.The caption identifies the table as an experiment-appendix evidence index.

C.2 Full Experimental Configuration

The appendix documents the clean-setting configurations, quality and detection metrics, SAC operating-point selection, detector readouts, and controlled edit diagnostics. It emphasizes raw metrics and distinguishes generation-side effects from detector-side interpretation.

  • Evaluation setup: The clean evaluation records generation, watermark, baseline, detector, calibration, and evaluator configurations from frozen scripts and run metadata.
  • Evaluation setup: Quality assessment combines fluency, stability, semantic preservation, distributional fidelity, token-distribution drift, repetition, diversity, and PPL-tail behavior.
  • SAC operating point: The selected SAC setting W = 39, σK = 15.0, ρ = 0.6 retains strong low-FPR detection with favorable MAUVE, repetition/diversity, and token-drift behavior.
  • Detector readouts: The old detector provides the i.i.d. equal-weight baseline readout, whereas FFR filters SAC evidence using its local low-lag geometry; GO-FFR extends the readout to edit diagnostics.
  • Edit diagnostics: Controlled deletion, insertion, and substitution tests retain partial residual evidence, but low-FPR TPR is substantially harder than AUC under deletion and insertion.
  • Quality reporting: The normalized quality score is a direction-normalized visualization summary, while raw quality metrics remain the primary evidence.

C.8 Failure Analysis under Edit Drift

The edit-drift analysis treats deletion, insertion, and substitution as controlled diagnostics rather than adversarial-robustness claims. SAC-aware detection retains partial evidence, while coarse offset recovery does not resolve cumulative drift.

  • Synchronization drift: At 50% deletion or insertion, GO-FFR partially improves coarse synchronization, but cumulative local drift leaves edit-path alignment unresolved.
  • PPL-tail analysis: The canonical PPL analysis re-evaluates 200 frozen LLaDA-8B-Instruct ELI5 outputs paired by Prompt_ID and Gen_Len.
  • Correlation sweep: The ρ sweep fixes W = 39 and σK = 15 while varying ρ from 0 to 1, with ρ = 0 representing the matched i.i.d. endpoint.
  • Correlation sweep: Detection improves toward moderate correlation and then declines, supporting ρ = 0.6 as a balanced interior operating point rather than a universal optimum.

C.10 Calibration and Threshold-Transfer Controls

Calibration and threshold-transfer controls show that strong score discrimination does not guarantee portable realized false-positive rates. The reported transfer tests distinguish frozen operational thresholds from recalibrated controls.

  • Calibration controls: Calibration-size controls use LLaDA–ELI5, SAC W = 39, σK = 15, ρ = 0.6, model-native H0, and calibration sizes of 50/100/200/500.
  • Calibration controls: The four calibration-source conditions hold N = 500 and development/evaluation identities fixed while changing only the H0 source.
  • Threshold portability: The prompt-shift control distinguishes changes in question/content samples, prompt template, and their combination, without supporting a broader prompt-invariance claim.
  • Threshold portability: Wikipedia H0 thresholds transferred to ELI5 produced realized FPR 6.5% and 21.0% at nominal 1% and 5%, despite TPR values of 97.5% and 98.5%.
  • C4 transfer: Frozen LLaDA–ELI5 model-native-H0 thresholds transferred to 2,000 held-out non-watermarked C4 realnewslike records yielded 0.95%/5.50% realized FPR at nominal 1%/5%.
  • C4 transfer: Rebuilding H0 on C4 yielded realized FPR 2.95% at nominal 1% and 7.55% at nominal 5%, without establishing that target-domain recalibration is generally harmful.

C.11 Targeted Backbone and Task Transfer

Targeted evaluations extend SAC-Copula to Dream-7B/ELI5 and LLaDA/C4-en. Across both settings, detection remains strong while SAC reduces upper-tail or collapse-related quality failures, with nearly unchanged median PPL in the C4-en comparison.

  • Dream-7B/ELI5: The Dream evaluation uses separate human-answer calibration and negative sets alongside 200 method-specific Dream generations, with exact-tape detection for i.i.d. and FFR for SAC.
  • Dream-7B/ELI5: On Dream-7B/ELI5, both methods retain near-perfect detection, while SAC reduces the i.i.d. upper-tail and composite-collapse profile.
  • LLaDA/C4-en: On LLaDA/C4-en, SAC uses W = 39, σK = 15, ρ = 0.6 without C4-specific watermark retuning, while the method-matched pipelines use Old for i.i.d. and FFR for SAC.
  • LLaDA/C4-en: The method-matched C4-en pipelines have nearly identical median PPL, while SAC+FFR has a substantially less severe quality tail and stronger detection.
  • Common-detector audit: The separate C4 common-detector audit reports Rep3 0.7298 → 0.6954, Distinct3 0.2702 → 0.3046, and Ent3 1.7293 → 1.9986 from i.i.d. to SAC.
Loading 2608.20839v1…