PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Multi-bit Natural Language Watermarking through Invariant Features

KiYoon Yoo, Wonhyuk Ahn, Jiho Jang, Nojun Kwak

arXiv:2305.01904v2cs.CLcs.AI

TL;DR

Natural-language content needs watermarking that protects against piracy and misuse while retaining adequate payload and robust extraction under corruption. The paper anchors watermarks using text features invariant to minor corruption and proposes a corruption-resistant infill model. Its full method improves robustness by +16.8% point on average across four datasets, three corruption types, and two corruption ratios.

  • Problem

    Natural-language watermarking must protect valuable content from piracy and misuse while supporting adequate payload and accurate extraction despite corruption.

    Source: §1 · pp. 1–2

  • Method

    The method anchors watermark positions to semantic or syntactic features invariant to minor text modifications and uses an infill model trained for corruption robustness.

    Source: §1 · p. 2

  • Results

    +16.8% point average robustness improvement over previous work across four datasets, three corruption types, and two corruption ratios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The proposed method outperforms recent neural watermarking methods in robustness and payload while achieving comparable semantic quality.

    Source: §7 · p. 9

    The evaluated corruptions may not reflect imperceptible adversarial changes, and the method was not tested against paraphrasing.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recent years have witnessed a proliferation of valuable original natural language contents found in subscription-based media outlets, web novel platforms, and outputs of large language models. However, these contents are susceptible to illegal piracy and potential misuse without proper security measures. This calls for a secure watermarking system to guarantee copyright protection through leakage tracing or ownership identification. To effectively combat piracy and protect copyrights, a multi-bit watermarking framework should be able to embed adequate bits of information and extract the watermarks in a robust manner despite possible corruption. In this work, we explore ways to advance both payload and robustness by following a well-known proposition from image watermarking and identify features in natural language that are invariant to minor corruption. Through a systematic analysis of the possible sources of errors, we further propose a corruption-resistant infill model. Our full method improves upon the previous work on robustness by +16.8% point on average on four datasets, three corruption types, and two corruption ratios. Code available at https://github.com/bangawayoo/nlp-watermarking.

1 Introduction

The paper addresses the challenge of protecting natural-language content from piracy and misuse while preserving payload and extraction robustness under corruption. It uses corruption-invariant text features and a corruption-resistant infill model, improving robustness by +16.8% point on average across four datasets, three corruption types, and two corruption ratios.

1 Introduction

  • Natural-language watermarking must protect valuable content from piracy and misuse while embedding adequate bits and accurately extracting them despite corruption.Earlier neural approaches have limitations: AWT can err during extraction at higher payloads, while ContextLS has a fixed payload upper bound and does not address extraction under corruption.Source: §1 · pp. 1–2
  • The method uses semantically or syntactically fundamental features as watermark-position anchors and trains an infill model to withstand possible corruption.The authors identify features invariant to minor text modifications rather than embedding directly on the most significant component, which would be too perceptible in text.Source: §1 · p. 2
  • +16.8% point average robustness improvement over previous work across four datasets, three corruption types, and two corruption ratios.The result concerns the full method.Source: §1 · p. 2

2 Preliminaries

The preliminaries formulate blind watermarking as embedding a message in a cover text and extracting it after possible corruption, while preserving semantic similarity and message accuracy. They specify the corruption setting and introduce infill language models as a way to generate candidate text for watermarking.

2.1 Problem Formulation of Watermarking

  • Blind watermarking embeds a message in a cover text and extracts it from a possibly corrupted watermarked text without access to the original cover.The sender aims to preserve semantic similarity to the cover and the receiver aims for accurate message extraction.Source: §2.1 · p. 2

2.2 Corruptions on Xwm

  • Corruptions include word insertion, deletion, and substitution at 2.5% to 5.0% of each sentence’s word count, with low-similarity outputs filtered out.A pretrained sentence transformer filters corrupted texts whose cosine similarity to the original is below 0.98.Source: §2.2 · pp. 2–3

2.3 Infill Model

  • A pretrained infill language model predicts replacements for masked positions and generates candidate watermarked text.The model predicts appropriate words from a sequence in which one or more positions are masked.Source: §2.3 · p. 3

3 Framework for Robust Natural Language Watermarking

The framework selects text features to determine mask positions, then uses infilling to generate valid watermarked texts. It combines invariant features with a corruption-resistant infill model to improve robustness.

3 Framework for Robust Natural Language Watermarking

  • Phase 1 derives a state from the text to guide embedding, and Phase 2 uses that state with the text to generate valid watermarked texts.For this method, the state is the set of mask positions, and an infill model generates the watermarked text according to the bit message.Source: §3 · p. 3

3.1 Phase 1: Mask Position Selection

  • The method anchors masks near semantically essential keywords or selects syntactic dependencies using an ordering based on entailment scores.It uses named-entity recognition and YAKE for keywords, and a parser and NLI model for dependency ordering.Source: §3.1 · p. 4
  • Keywords and syntactic components show considerably high mask-matching robustness across three corruption types, while ContextLS has a drastically lower Rg1.The comparison is reported for IMDB at a 5% corruption rate.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Phase 2: Watermark Encoding

  • Phase 2 filters infill candidates, forms valid watermark sets, and assigns messages only to texts that retain the original state.Even when the state is recovered, local-context dependence can prevent recovery of the same watermark set; Rg2 is substantially lower than Rg1 in all reported cases. A larger Rg2 does not necessarily mean a lower bit error rate because the metric does not measure the extent of set discrepancy.Source: §3.2 · pp. 4–5, §3.3 · p. 5

3.3 Robust Infill Model

  • The robust infill model trains on clean and corrupted masked inputs to produce consistent word distributions, using a sparse target over the top-k1 tokens.Training uses the masking strategy from Phase 1 to align masked words between training and testing.Source: §3.3 · p. 5
  • Table 2 reports a noticeable Rg2 improvement and negligible Rg1 change when the robust infill model is applied to the syntactic component.The table compares with-versus-without effects across four datasets, averaged over three corruption types.Source: Table 2 · p. 5, §3.3 · p. 6

4 Experiment

Experiments across four writing styles evaluate payload, extraction robustness under corruption, and watermarked-text quality. The full method outperforms ContextLS on robustness, while semantic-score and human evaluations show quality trade-offs.

4 Experiment

  • The evaluation uses IMDB, WikiText-2, Dracula, and Wuthering Heights, measuring payload with BPW, robustness with BER, and text quality with entailment, semantic similarity, and human evaluation.IMDB and WikiText-2 represent modern review and informative styles; the novels have distinct styles compared with modern English.Source: §4 · p. 6

4.1 Main Experiments

  • The final model outperforms ContextLS across all four datasets and corruption rates, with Syntactic-component BER improvements of 8.2–14.4 points at CR=2.5% and 10.0–11.7 points at CR=5%.The stated ranges summarize the four dataset-specific improvements reported for the Syntactic component.Source: Table 3 · p. 6, §4.1 · p. 7

4.2 Semantic Scores of Watermark

  • Our method trails ContextLS on semantic scores but exceeds the other methods, while ContextLS's extracted-message accuracy deteriorates under corruption.ContextLS filters candidate tokens with an NLI model to maintain semantic similarity.Source: §4.2 · p. 7, Table 4 · p. 7
  • Human evaluation finds that our method and ContextLS have nearly the same fluency change relative to the original cover text, alongside a distinct fluency gap between ours and AWT.The study compares fluency change and semantic similarity using 20 samples and five annotators.Source: §4.2 · p. 7, Table 5 · p. 7

5 Discussion

The discussion compares payload and runtime with ContextLS, identifies weaknesses in automatic semantic metrics, and reports ablation and corruption stress-test findings.

5.1 Comparison with ContextLS

  • On IMDB, embedding 1,000 sentences takes 81 minutes for ContextLS versus 4 minutes for ours; increasing top-k2 can raise our payload but counterintuitively lowers ContextLS's.The runtime comparison reflects ContextLS's exhaustive search and multiple neural-network forward passes.Source: §5.1 · pp. 7–8, Table 6 · p. 8

5.2 Pitfalls of Automatic Semantic Metrics

  • Automatic metrics can miss human-perceptible semantic shifts from coordination substitution and give abnormally high entailment scores when named entities are masked; ours excludes named entities as mask candidates.The paper suggests ranking coordination dependencies last or discarding them; it notes comparable payload and robustness for that change.Source: §5.2 · p. 8

5.3 Ablations and Other Results

  • Our masking strategy improves BPW and robustness over random masking, while KL direction barely affects final robustness; at CR=0.1, deletion weakens the syntactic component despite robustness above 0.9 for insertion and substitution.The stress test identifies deletion as the syntactic component's most fragile corruption type.Source: §5.3 · p. 8

6 Related Works

Related work spans rule-based and neural text watermarking, image watermarking, steganography, and zero-bit watermarking of language-model outputs. The paper distinguishes its multi-bit objective from approaches that identify generated text or conceal data in arbitrary cover text.

6 Related Works

  • Early text-watermarking methods use synonym substitution, syntactic restructuring, or paraphrasing, while predefined rules can constrain bit capacity and contextual quality (Topkara et al., 2006b; Atallah et al., 2001, 2003).Neural methods include AWT (Abdelnabi and Fritz, 2021) and ContextLS (Yang et al., 2022), which use learned embedding or neural infilling for lexical substitution.Source: §6 · p. 9
  • Image watermarking studies corruption-aware robustness and frequency-domain transforms; this work analogizes keywords and dependencies to semantic and syntactic domains (Zhu et al., 2018).The cited image methods explicitly account for possible corruptions to improve robustness (Zhu et al.; Luo et al.).Source: §6 · p. 9
  • Steganography prioritizes covertness and may generate arbitrary cover text, whereas language-model watermarking cited here targets zero-bit identification rather than embedding arbitrary multi-bit information (Tina Fang et al., 2017; He et al., 2022a,b; Kirchenbauer et al., 2023).The paper says methodologies for language-model watermarking can be adapted to watermark text directly.Source: §6 · p. 9

7 Conclusion

The paper proposes natural-language invariant features for robust watermarking and empirically validates two candidate components. Its method outperforms recent neural watermarking in robustness and payload while maintaining comparable semantic quality.

7 Conclusion

  • The method uses invariant features of natural language to embed watermarks robust to corruption.The authors empirically validate two components that off-the-shelf models can discover.Source: §7 · p. 9
  • The proposed method outperforms recent neural watermarking in robustness and payload while achieving comparable semantic quality.The authors do not claim the studied invariant features are optimal, and point to exploring other domains and solutions as future work.Source: §7 · p. 9

Limitations

The method has lower automatic semantic-metric results than the most recent work, despite its robustness. The corruption evaluation may not match true adversarial evasion, and paraphrasing was not tested.

Limitations

  • The method has subpar automatic semantic-metric results compared with the most recent work, despite its robustness.The authors suggest this may reflect a perceptibility–robustness trade-off; technical tricks include a neural infill model and NLI-sorted ordering.Source: Limitations · p. 10
  • The tested corruptions may only roughly estimate true adversarial evasion, and paraphrasing—which may substantially change syntax—was not evaluated.The authors cite paraphrase attacks' lack of controllability as a reason they did not experiment with them.Source: Limitations · p. 10

A.1 Implementation Details

The experiments use shared dataset subsets and specified train/test splits, with robustness evaluated on corrupted test sentences. The appendix also reports the higher-corruption-rate scope of Figure 3 and infill-model training details.

A.1 Implementation Details

  • The study subsampled the first 5,000 sentences and used the same subset across methods; the reported standard error of mean BPW across three trials was 0.002.Insertion and substitution BER standard errors were both 0.004, described as marginal compared with the performance gap.Source: §A.1 · p. 12
  • For novels, the first 40% of the text served as training data and the remainder as test data; IMDB and Wikitext-2 used their original training splits.The same splits were used to train AWT.Source: §A.1 · p. 12
  • Figure 3 presents g1 robustness at a higher corruption rate.Source: Figure 3 · p. 13
  • The robustness experiment used the first 1,000 sentences of each 5,000-sentence test set; insertion and substitution results average five corrupted samples per sample.Deletion was deterministic and run once; embedding was run once with a fixed random seed, while insertion and substitution could be non-deterministic.Source: §A.1 · p. 13
  • Infill models were trained for 100 epochs with AdamW at a 5e-5 learning rate; Wikitext-2 training took approximately 22 GPU-hours.The models used linear warmup over 0.1 of total training steps and masked 15% of words with whole-word masking.Source: §A.1 · p. 13

A.2 AWT Implementation Details

The AWT comparison follows its official implementation and largely adopts its hyperparameters, with changes for model choice, higher payload, and the novel datasets. Robustness comparisons use matched corruption procedures and exclude certain unknown tokens.

A.2 AWT Implementation Details

  • The implementation replaced AWT's AWD-LSTM with GPT-2 and doubled the reconstruction-loss weight at the second training stage for BPW=0.1.The latter change was made to help the model converge.Source: §A.2 · p. 13
  • Dracula and Wuthering Heights data were combined to train and evaluate AWT because separate training was not possible due to limited data.Source: §A.2 · p. 13
  • For robustness comparisons, watermarked segments were concatenated, split into sentences, and corrupted per sentence; unknown insertion and substitution tokens were excluded for AWT.AWT builds its token dictionary from the corpus before watermark embedding.Source: §A.2 · p. 13

A.3 AWT on IMDB dataset

The appendix documents AWT’s IMDB reconstruction failure, additional corruption tests and BER conventions, human-evaluation scope, and qualitative examples.

A.3 AWT on IMDB dataset

  • AWT’s reconstruction loss failed to converge on IMDB, severely reducing watermarked-sentence quality despite testing robustness under corruption.At CR=0.05, BER across the three corruption types was 0.283, 0.278, and 0.299.Source: §A.3 · p. 13, Table 13 · p. 16

A.4 More Results

  • On IMDB character swaps, Robust Infill lowers BER from 0.079/0.167 to 0.063/0.142 at CR=2.5%/5%, despite no explicit training.Coordination substitution can produce semantic drift undetected by automatic metrics; random-mask infill does not always help ContextLS, with only marginal improvement relative to ours.Source: §A.4 · pp. 13–14, Table 16 · p. 16

A.5 More Discussions

  • BER computation counts unpredicted bits as errors and truncates over-extracted bits, assuming the true bit count is unknown during extraction.Source: §A.5 · p. 14

A.6 Human Evaluation

  • Human evaluation used Dracula and Wuthering Heights samples, excluded Wikitext-2 and IMDB, and retained five of seven responses after thresholding.Fluency change was computed against the original cover text; semantic preservation was rated against that reference.Source: §A.6 · p. 14

A.7 Watermarked Examples

Loading 2305.01904v2…