PaperlayerView exact revision on arXiv

Source-linked AI summary

A Robust Semantics-based Watermark for Large Language Model against Paraphrasing

Jie Ren, Han Xu, Yiding Liu, Yingqian Cui, Shuaiqiang Wang, Dawei Yin, Jiliang Tang

arXiv:2311.08721v2cs.CR

TL;DR

Existing token-hash watermarks can lose detection effectiveness after paraphrasing, motivating a more robust approach. SemaMark uses discretized semantic embeddings, contrastive-learning training, and offset detection to stabilize watermarking. Experiments report stronger robustness than baselines across paraphrase settings, while the method remains inapplicable to black-box LLMs and requires model-specific training.

  • Problem

    Token-hash vocabulary-partition watermarks are vulnerable to paraphrasing, which can compromise detection of LLM-generated text.

    Source: §1 · pp. 1–2

  • Method

    SemaMark discretizes semantic embeddings on a normalized embedding ring and combines contrastive-learning training with Q-offset detection.

    Source: §1 · p. 2, §3.1 · p. 4, §5 · p. 9

  • Results

    SemaMark has the best detection performance most of the time across backbone models and paraphrase methods, including ROC-AUC values of 0.9692 and 0.9308 under round-trip translation.

    Source: §4.2 · pp. 6–7

  • Takeaways & Limitations

    Using semantics as the partition-function seed improves watermark robustness under paraphrase.

    Source: §4.2 · p. 7, §5 · p. 9

    SemaMark is not applicable to black-box LLMs and requires a specialized NE-Ring trained for each LLM.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have show great ability in various natural language tasks. However, there are concerns that LLMs are possible to be used improperly or even illegally. To prevent the malicious usage of LLMs, detecting LLM-generated text becomes crucial in the deployment of LLM applications. Watermarking is an effective strategy to detect the LLM-generated content by encoding a pre-defined secret watermark to facilitate the detection process. However, the majority of existing watermark methods leverage the simple hashes of precedent tokens to partition vocabulary. Such watermark can be easily eliminated by paraphrase and correspondingly the detection effectiveness will be greatly compromised. Thus, to enhance the robustness against paraphrase, we propose a semantics-based watermark framework SemaMark. It leverages the semantics as an alternative to simple hashes of tokens since the paraphrase will likely preserve the semantic meaning of the sentences. Comprehensive experiments are conducted to demonstrate the effectiveness and robustness of SemaMark under different paraphrases.

1 Introduction

LLM-generated text detection matters because generated content can support harmful deception, while token-hash watermarks are vulnerable to paraphrasing. SemaMark instead uses discretized semantic representations to preserve watermark detection under paraphrase.

  • Detecting LLM-generated text is crucial because generated fake news and reviews can contribute to panic and unethical competition.The paper frames watermarking as a way to encode a secret signal that supports detection.Source: §1 · p. 1
  • Token-hash watermarks can be disrupted by paraphrasing because changed tokens alter the partition seed and break matching with the green list.This can dramatically compromise detection effectiveness even when the paraphrase retains the text’s general meaning.Source: §1 · pp. 1–2
  • SemaMark seeds vocabulary partitioning with the semantic meaning of precedent token sequences rather than simple token hashes.The approach relies on semantics being more stable under paraphrase than precedent-token identities.Source: §1 · p. 2
  • SemaMark discretizes embeddings by projecting them into a normalized two-dimensional NE-Ring and dividing the ring into semantic sections.The coarser discrete representation is intended to tolerate minor embedding changes caused by paraphrase.Source: §1 · p. 2
  • Contrastive-learning uniformity reduces concentration in particular semantic sections, while offset detection improves robustness near discrete-section boundaries.These strategies address watermark concealment and boundary-sensitive semantic-value changes, respectively.Source: §1 · p. 2

2 Related works

Related work covers learning-based and property-based detectors alongside proactive watermarking methods. Existing vocabulary-partition watermarks use token hashes and statistical matching, while related semantic watermarking introduces additional encoding models.

  • Watermarking proactively changes generation with a predefined rule, creating a concealed signal that distinguishes watermarked from non-watermarked text.Source: §2 · pp. 2–3
  • Vocabulary-partition watermarks hash previous tokens, select a green list, and increase the likelihood of green-list tokens during generation.Source: §2 · p. 3
  • Detection compares the observed number of seed–green-list matches with the expected fraction γ using a z-test.A non-watermarked sentence is expected to contain approximately γL matching tokens, whereas a watermarked sentence is expected to contain more.Source: §2 · p. 3
  • Yoo et al. (2023) extend green–red-list watermarking to multiple vocabulary lists for multi-bit encoding.Source: §2 · p. 3
  • Liu et al. (2023a) propose semantic-invariant watermarking but require two additional models, which can make text encoding time-consuming.Source: §2 · p. 3

3 Method

SemaMark replaces token-hash watermark seeds with discretized semantic representations to preserve vocabulary partitioning under paraphrase. It further uses contrastive training and Q-offset detection to improve concealment and tolerate semantic-boundary mismatches.

3 Method

  • The method section organizes SemaMark around semantic seeding, contrastive MLP training, and smoothed offset-based detection.These components respectively target paraphrase-stable partitioning, watermark concealment, and boundary-induced detection errors.Source: §3 · p. 3, §3.1 · pp. 3–4, §3.3 · p. 5

3.1 The framework of SemaMark

  • SemaMark reduces embedding dimensions to a 2D Normalized Embedding Ring, discretizes semantic values, and uses them to partition vocabulary for watermarking.The green-list logits are shifted so generated tokens are more likely to match the semantic seed.Source: §3.1 · pp. 3–4
  • Weighted pooling aggregates previous m token embeddings with higher weights for closer tokens because distant-token relationships may be less robust after paraphrase.The pooled representation provides a semantic seed for the current token and uses more semantic information than a single preceding token.Source: §3.1 · p. 4
  • NE-Ring discretization maps polar angles into K equal sections, allowing minor paraphrase changes to remain within the same discrete semantic value.Values near section boundaries can still change, motivating the offset-detection method.Source: §3.1 · p. 4

3.2 Training gθ by Contrastive Learning

  • Contrastive Learning trains the MLP to distribute normalized embeddings across the NE-Ring so semantic values cover more sections and are less monotonous.Soft augmentation uses Gaussian noise to form positive pairs from similar embeddings and negative pairs from different sequences.Source: §3.2 · pp. 4–5

3.3 Q-offset detection

  • Existing one-to-one detection can lose accuracy after paraphrase because altered boundary semantic values disrupt matching with consequent tokens.The window slides token by token during autoregressive generation, allowing semantic changes to accumulate.Source: §3.3 · p. 5
  • Q-offset detection tests semantic-token matching after shifting the discrete seed by offsets from −Q to Q and uses the maximum z-statistic as the score.This addresses mismatches caused by sliding windows, accumulated semantic changes, and values near discretization boundaries.Source: §3.3 · p. 5, Figure 2 · p. 5
  • Q-offset detection also raises the green-word fraction for non-watermarked text, so detection treats γ as a validation-set-selected hyper-parameter while generation uses a fixed value.The method therefore improves robustness while requiring calibration of the detection threshold parameter.Source: §3.3 · p. 5

4 Experiment

Experiments show that SemaMark is more robust to paraphrasing than token-hash baselines while largely preserving text quality. Ablations and distribution analysis support semantic discretization, Q-offset detection, and contrastive learning as key design choices.

4.2 Main Results

  • SemaMark has the best detection performance most of the time across two backbone models and three paraphrase methods.Under round-trip translation, its ROC-AUC reaches 0.9692 on OPT-2.7B and 0.9308 on OPT-6.7B, at least 0.05 above every baseline.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Text Quality

  • SemaMark preserves generated-text quality, with perplexity near non-watermarked text on OPT-2.7B and only about one point higher on OPT-6.7B.EXP-Edit and EXP-Exit produce substantially higher perplexity, whereas SemaMark, LeftHash, and SelfHash remain close to the non-watermarked reference.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4 Ablation Study

  • Using m = 15 previous tokens gives the strongest stable robustness trend before longer sequences make ROC-AUC fluctuate.The authors attribute degradation at larger m to paraphrase changes in distant tokens and unstable early-generation predictions.Source: §4.4 · p. 7, Figure 4 · p. 7
  • Q-offset detection improves ROC-AUC up to approximately Q = 15, after which correction is limited and computation increases.The offset helps correct semantic values near partition boundaries, so the experiments use Q = 15.Source: §4.4 · pp. 7–8
  • NE-Ring discretization changes green-list overlap from 24.99% after a 1e-5 embedding perturbation to 100%, increasing tolerance to small semantic changes.The result supports discretization as a mechanism for robustness, while Q-offset further improves detection.Source: §4.4 · p. 8
  • Across larger LLaMA-7B and LLaMA2-7B models, SemaMark consistently achieves the highest robustness against paraphrasing.Increases over baselines are 0.06 and 0.03 ROC-AUC for LLaMA2-7B, and 0.027 and 0.009 for LLaMA-7B.Source: §4.4 · p. 8, Table 2 · p. 8

4.5 Distribution on NE-Ring based on CL

  • Contrastive learning produces an approximately uniform NE-Ring distribution, with density between 0.052 and 0.054 across polar angles.This distributes semantics across discrete values and helps prevent monotonous vocabulary partitions that could be attacked through token-frequency counting.Source: §4.5 · pp. 8–9, Figure 6 · p. 8

5 Conclusion

SemaMark uses semantic information rather than precedent-token hashes to improve watermark robustness against paraphrase, combining NE-Ring discretization, contrastive-learning training, and Q-offset detection.

  • SemaMark maps semantic information to vocabulary partitions, aiming to preserve watermark robustness when paraphrasing changes precedent tokens.The method discretizes embedding space on an NE-Ring and uses contrastive-learning training plus Q-offset detection.Source: §5 · p. 9

6 Limitations

The method has important deployment and generality limitations: it cannot currently support black-box LLMs and requires LLM-specific semantic-ring training, while other attacks may still reduce detection rates.

  • SemaMark is not applicable to black-box LLMs because it requires embedding access and permission to modify generation logits.Source: §6 · p. 9
  • Each LLM requires a specialized NE-Ring, limiting flexibility for universal models or fine-tuning across different LLMs.The authors identify extending the method to black-box LLMs and universal models as future work.Source: §6 · p. 9
  • Although robust under paraphrase, SemaMark may still face reduced detection rates from adaptive attacks targeting green-list watermarking from other perspectives.Source: §6 · p. 9

A More details on experimental settings

The experiments use open-source models, baselines, datasets, and paraphrase systems, with implementation based on PyTorch and supporting open-source packages.

B Examples of paraphrases

The paper includes examples illustrating three paraphrases, presented in Table 3.

Loading 2311.08721v2…