PaperlayerView exact revision on arXiv

Source-linked AI summary

Protecting Intellectual Property of Language Generation APIs with Lexical Watermark

Xuanli He, Qiongkai Xu, Lingjuan Lyu, Fangzhao Wu, Chenguang Wang

arXiv:2112.02701v1cs.CRcs.CL

TL;DR

Model-extraction attacks can imitate NLG APIs, while existing watermarking methods are not directly suited to protecting these services. The paper applies lexical changes to API outputs and tests for watermarks in suspicious models; it reports stronger identification and less semantic damage than bit-level baselines.

  • Problem

    Model-extraction attacks can imitate victim APIs and infringe their intellectual property, while existing watermarking techniques are not directly suited to protecting NLG APIs.

    Source: §1 · p. 1

  • Method

    The approach inserts interchangeable lexical substitutions into text-generation API outputs and uses a null-hypothesis test for post-hoc ownership verification.

    Source: §3.2 · p. 3, §3.3 · p. 4

  • Results

    Across generation tasks, lexical watermarks outperform unwatermarked and bit-level baselines by up to 12 orders of magnitude in p-value while causing minimal generation-quality damage relative to bit-watermarking baselines.

    Source: §5 · pp. 5–6

  • Takeaways & Limitations

    Because the watermark changes lexicons, its distinction from an innocent system is tangible to people when anchor words are provided, supporting understandable ownership evidence.

    Source: §5 · p. 6

    The evaluation assumes victim and imitator use the same datasets and that the extracted model is trained only on API queries and watermarked outputs.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Nowadays, due to the breakthrough in natural language generation (NLG), including machine translation, document summarization, image captioning, etc NLG models have been encapsulated in cloud APIs to serve over half a billion people worldwide and process over one hundred billion word generations per day. Thus, NLG APIs have already become essential profitable services in many commercial companies. Due to the substantial financial and intellectual investments, service providers adopt a pay-as-you-use policy to promote sustainable market growth. However, recent works have shown that cloud platforms suffer from financial losses imposed by model extraction attacks, which aim to imitate the functionality and utility of the victim services, thus violating the intellectual property (IP) of cloud APIs. This work targets at protecting IP of NLG APIs by identifying the attackers who have utilized watermarked responses from the victim NLG APIs. However, most existing watermarking techniques are not directly amenable for IP protection of NLG APIs. To bridge this gap, we first present a novel watermarking method for text generation APIs by conducting lexical modification to the original outputs. Compared with the competitive baselines, our watermark approach achieves better identifiable performance in terms of p-value, with fewer semantic losses. In addition, our watermarks are more understandable and intuitive to humans than the baselines. Finally, the empirical studies show our approach is also applicable to queries from different domains, and is effective on the attacker trained on a mixture of the corpus which includes less than 10\% watermarked samples.

1 Introduction

NLG services are commercially valuable, but model-extraction attacks can imitate victim APIs and infringe their intellectual property. Existing watermarking leaves a gap for identifying extraction of NLG models through their outputs; this work proposes lexical watermarking and post-hoc ownership verification.

1 Introduction

  • Companies invest substantial workforce and computational resources in NLG models deployed as pay-as-you-use cloud services, making their intellectual property valuable.The source passage also notes these services are expected to support substantial business.Source: §1 · p. 1
  • Model-extraction attacks can steal victim API functionality through carefully designed queries, and have affected commercial APIs as well as laboratory models.Such extraction can infringe intellectual property.Source: §1 · p. 1
  • Prior watermarking work largely targets digital media or discriminative models, leaving limited adaptation for identifying extraction of NLG APIs through their responses.The proposed approach modifies API responses rather than the extracted model's neurons and uses a null-hypothesis test for post-hoc ownership verification.Source: §1 · p. 1

1. We make the first exploitation of IP infringement identi-

The paper introduces lexical watermarking for text-generation APIs and uses statistical testing to verify ownership of suspicious NLG models. Experiments cover several generation tasks and restricted attack settings.

1. We make the first exploitation of IP infringement identi-

  • The paper presents its first effort to use watermarking to identify IP infringement through model extraction in NLG.The owners can manipulate API responses, rather than neurons of the extracted model.Source: §1 · p. 2
  • Interchangeable lexical substitutions watermark text-generation API outputs while preserving semantics, and a null-hypothesis test supports post-hoc ownership verification.The verification is applied to suspicious NLG models.Source: §1 · p. 2
  • Experiments on machine translation, text summarization, and image captioning assess detection under cross-domain queries and mixtures of watermarked and non-watermarked data.Source: §1 · p. 2

2 Preliminary and Related Work

The related work describes model extraction through API queries, reviews watermarking approaches and their limits, and situates the paper's focus on text generation. Existing API backdoor strategies impose storage and verification burdens, while prior text-generation watermarking addressed a different goal.

2.1 Model Extraction Attack

  • Attackers can query a black-box victim API, pair its predictions with queries, and train a surrogate model that imitates the victim's functionality.The resulting surrogate can let users avoid service charges or be offered as a lower-priced cloud service.Source: §2.1 · p. 2

2.2 Watermarking

  • Earlier DNN watermarking superimposes secret noise on protected models and supports post-hoc ownership checks, but assumes white-box access.These methods protect model parameters rather than API outputs.Source: §2.2 · p. 2
  • API backdoor watermarking stores randomly altered queries and outcomes, but requires substantial storage and computation to track users and verify theft.The passage also notes that querying suspicious APIs can cost money under pay-as-you-use services.Source: §2.2 · p. 2

2.3 Text Generation and Watermarking

  • Text generation covers tasks such as translation, summarization, and image captioning, with target sentences conditioned on linguistic or non-linguistic inputs.Source: §2.3 · p. 2
  • Prior statistical machine-translation watermarking encoded bit sequences for data filtering, not model-extraction defense, and its evidence may be difficult for people to understand.Figure 1 depicts watermarking API outputs before they are returned and then examining suspicious-model outputs for an ownership claim.Source: §2.3 · p. 2, Figure 1 · p. 3

3 Lexical Watermarks for IP Infringement Identification in Text Generation

The method embeds lexical watermarks in selected API outputs, then checks suspicious models for an elevated watermark frequency and statistically tests whether that pattern is consistent with random word selection.

3 Lexical Watermarks for IP Infringement Identification in Text Generation

  • The method substitutes lexical watermarks into API responses and later verifies suspicious models using watermark frequency and a null-hypothesis test.The owner queries the suspicious model with reserved queries, then evaluates whether its outputs show a watermark preference.Source: §3 · p. 3, §3.3 · p. 4

3.1 Watermarking Generative Model

  • Seq2seq generation lets service providers inject confidential prior knowledge into outputs, which can carry watermarks into an imitator trained on those outputs.The model conditions target sentences on source inputs and decodes them sequentially from hidden states.Source: §3.1 · p. 3

3.2 Lexical Replacement as Watermarking

  • For synonym watermarks, frequent adjectives are paired with less-used synonyms, and a hash selects the original or substitute while preserving sentence meaning.The paper also describes ranking adjectives by training-set frequency and skipping words with fewer than M synonyms.Source: §3.2 · pp. 3–4
  • Spelling watermarks use confidential US/UK spelling pairs; a hash fixes each candidate’s choice, with approximately equal selection probabilities over a large corpus.This setting uses M = 1 because it considers two spelling systems.Source: §3.2 · p. 4

3.3 IP Infringement Identification

  • A suspicious model is flagged when its hit ratio—the share of candidate or substitute tokens that are watermarked—exceeds a predefined threshold.The owner queries the model with reserved queries; an elevated frequency of watermarked words prompts further evaluation.Source: §3.3 · p. 4
  • The evaluation tests whether outputs favor watermarks over random candidate-word selection, using a binomial model with p approximately 1/(M + 1) and a p-value to assess the null.A lower p-value indicates the tested model is less likely to be innocent.Source: §3.3 · p. 4

4 Experimental Settings

Experiments cover machine translation, document summarization, and image captioning, using task-specific datasets and quality metrics. The setup specifies model architectures, data assumptions, and a statistical machine-translation watermark baseline.

4.1 Natural Language Generation Tasks

  • Experiments use WMT14 German–English translation with BLEU, CNN/DM summarization with Rouge-L, and MSCOCO captioning with SPICE.The test set for each task contains 200 random sentence pairs; WMT14 and CNN/DM use 32K BPE, while MSCOCO uses 10K subword units.Source: §4.1 · pp. 4–5

4.2 Models

  • Victim and extracted models use Transformer-base; summarization and translation also use BART and mBART, respectively.MSCOCO uses precomputed visual features as Transformer encoder inputs.Source: §4.2 · p. 5
  • The setup assumes shared training datasets and an extracted model trained only on queries and watermarked victim outputs; comparison includes Venugopal et al. (2011)’s bit-sequence watermarking.That baseline watermarks generated sentences at n-gram and sentence levels.Source: §4.2 · p. 5

5 Results and Discussion

The experiments assess detection reliability and robustness to domain shift and partial watermark use. Across the tested settings, lexical watermarking supports ownership claims while retaining generation quality.

Mixture of human- and machine-labeled data

  • 10% watermarked WMT14 data suffices for ownership claims, whereas bit-level marks require more than 20%; lexical watermark BLEU is higher.At 5% watermarked data, translation quality is reported to exceed clean-data training, which the authors attribute to noise-injection regularization.Source: Figure 2 · p. 6, §5 · p. 7

Overall results

  • Across generation tasks, p-values outperform unwatermarked and bit-level models by up to 12 orders of magnitude, with better BLEU, ROUGE-L, SPICE and minimal BERTScore damage.The authors describe the approach as providing stronger confidence for ownership claims while causing minimal generation-quality damage.Source: §5 · p. 6

Pretrained models

  • Table 3 supports transfer of injected watermarks to pretrained models fine-tuned on watermarked text.The table reports pretrained-model performance on WMT14 with mBART and CNN/DM with BART, with and without synonym replacement.Source: Table 3 · p. 6, §5 · p. 6

Bit-level watermark comparison

  • For bit-level watermarks, increasing n-gram order makes imitation harder, while p-value approaches the non-watermarking baseline.The authors attribute this trend to the difficulty of imitating higher-order n-gram and sentence patterns.Source: §5 · p. 6

IP identification on cross-domain model extraction

  • The WMT14-trained victim's watermark remains effective on IWSLT14 and OPUS queries, supporting applicability across the tested data and domains.The cross-domain tests use 250K IWSLT14 and 2.1M OPUS (Law) examples.Source: Table 4 · p. 6, §5 · pp. 6–7

Understandable watermarking

  • Once anchor words are supplied, lexical-watermark differences are tangible, unlike bit-watermarked outputs that require a professional toolkit to distinguish.Table 5 compares lexical-watermarked, bit-watermarked, and non-watermarked outputs from corresponding extracted models.Source: §5 · p. 6, Table 5 · p. 7

Influence of synonym set size

  • Increasing synonym candidate-set size lowers p-value; M=2 is the primary setting because it outperforms M=1 on that measure.The authors study M values from 1 to 5, reflecting the average synonym-set size for the selected adjectives.Source: §5 · pp. 6–7, Table 6 · p. 7

6 Conclusion and Future Work

The paper reports that lexical watermarking outperforms baselines and works across domain shifts and mixtures of watermarked and non-watermarked data. It proposes future testing against fine-tuning and pruning.

6 Conclusion and Future Work

  • The approach is reported to outperform baselines and work under domain shift and mixtures of watermarked and non-watermarked data.The conclusion describes the approach as applicable in these settings.Source: §6 · pp. 7–8
  • The authors state that lexical watermarking can help API owners protect intellectual property from illegal copying, redistribution, or abuse.Source: §6 · p. 8
  • Future work will test whether the watermark survives attacker use of model fine-tuning and pruning.Source: §6 · p. 8

A Bit Watermarks

The bit-watermarking baseline hashes candidate generations into bit sequences and chooses the candidate with the most ones. Ownership verification uses a p-value test based on whether ones outnumber zeros.

A Bit Watermarks

  • For each source input, the method hashes complete candidate sentences or n-grams into bit sequences, then selects the candidate with the most ones.Source: A Bit Watermarks · p. 10
  • Ownership is evaluated using the p-value β calculated via Equations 3–5.Source: A Bit Watermarks · p. 10
  • A generation counts as a match when ones exceed zeros; the unwatermarked corpus assumes p = 0.5 because its bits are evenly distributed.Source: A Bit Watermarks · p. 10

B Hyper-parameters

The experiments use fairseq Transformer-base settings across machine translation, summarization, and image captioning. Training duration varies by task and dataset, with Adam and a single RTX 6000 GPU used throughout.

B Hyper-parameters

  • The Transformer-base uses 6 encoder and decoder layers, 512-unit embeddings, 2048-unit feed-forward layers, and 8 attention heads.Source: B Hyper-parameters · p. 10
  • Machine-translation models train for 40 epochs on WMT14 and OPUS and 60 epochs on IWSLT14; summarization and image-captioning models train for 30 epochs.Source: B Hyper-parameters · p. 10
  • All experiments use Adam with 4000 warm-up updates, inverse-square-root learning-rate decay, and one RTX 6000 GPU.Source: B Hyper-parameters · p. 10

C Examples

The examples section presents comparisons of lexical watermarking, bit watermarking, and non-watermarked outputs from extracted models. It includes translation and summarization examples.

C Examples

  • The section provides further examples of the different watermarking approaches in Tables 7 and 8.Source: C Examples · p. 10
  • Table 7 compares lexical watermarking, bit watermarking, and non-watermarked translation from extracted models.Blue marks the selected word and red the watermarked word; parenthetical counts give ones versus zeros in the bit representation.Source: Table 7 · p. 11
  • Table 8 compares lexical watermarking, bit watermarking, and non-watermarked summarization from extracted models.Blue marks the selected word and red the watermarked word; parenthetical counts give ones versus zeros in the bit representation.Source: Table 8 · p. 12
Loading 2112.02701v1…