PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepTextMark: A Deep Learning-Driven Text Watermarking Approach for Identifying Large Language Model Generated Text

Travis Munyer, Abdullah Tanvir, Arjon Das, Xin Zhong

arXiv:2305.05773v2cs.MMcs.LG

TL;DR

Distinguishing LLM-generated text from human writing remains difficult, and classifier efficacy is constrained. DeepTextMark adds a blind watermarking-and-detection pipeline, reporting near-perfect accuracy as text length increases.

  • Problem

    Distinguishing LLM-generated text from human-written text remains difficult, while classifier efficacy is constrained and detection reliability is a concern.

    Source: §1 · p. 1

  • Method

    DeepTextMark inserts watermarks through semantically congruent word substitution using Word2Vec and Universal Sentence Encoder embeddings, then detects them with a transformer classifier.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Near-perfect accuracy is reported as text length increases; validation accuracy also improves with more training sentences and is near-perfect with a small collection.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The method is presented as an add-on to LLM systems that needs no access to the generation phase and classifies using only watermarked text.

    Source: §1 · p. 2

    Detection is limited without initial watermarking of generated text.

    Source: §5 · p. 15

Abstract

from arXiv · show

The rapid advancement of Large Language Models (LLMs) has significantly enhanced the capabilities of text generators. With the potential for misuse escalating, the importance of discerning whether texts are human-authored or generated by LLMs has become paramount. Several preceding studies have ventured to address this challenge by employing binary classifiers to differentiate between human-written and LLM-generated text. Nevertheless, the reliability of these classifiers has been subject to question. Given that consequential decisions may hinge on the outcome of such classification, it is imperative that text source detection is of high caliber. In light of this, the present paper introduces DeepTextMark, a deep learning-driven text watermarking methodology devised for text source identification. By leveraging Word2Vec and Sentence Encoding for watermark insertion, alongside a transformer-based classifier for watermark detection, DeepTextMark epitomizes a blend of blindness, robustness, imperceptibility, and reliability. As elaborated within the paper, these attributes are crucial for universal text source detection, with a particular emphasis in this paper on text produced by LLMs. DeepTextMark offers a viable "add-on" solution to prevailing text generation frameworks, requiring no direct access or alterations to the underlying text generation mechanism. Experimental evaluations underscore the high imperceptibility, elevated detection accuracy, augmented robustness, reliability, and swift execution of DeepTextMark.

1 Introduction

The introduction frames reliable LLM-text identification as difficult because classifiers have constrained efficacy and some watermarking approaches require access to the generation phase. It presents DeepTextMark as an automatic, blind watermarking system and positions it against LLM-specific detection and classical text watermarking.

1 Introduction

  • LLM-generated text is difficult to distinguish from human writing, while classifier efficacy is constrained and text augmentation can modify texts.The paper argues that source detection should be reliable and difficult to bypass.Source: §1 · p. 1
  • DeepTextMark selects semantically congruent replacement words using Word2Vec and Universal Sentence Encoder embeddings, then detects watermarked text with a transformer classifier.The authors describe the method as automatic, blind, robust, and imperceptible, with an extension to multiple sentences.Source: §1 · p. 2

2 Related Work

  • The related-work section is organized around LLM-generated-text detection and classical watermarking, alongside a summary of the paper’s detection and insertion contributions.The paper identifies these as the two areas reviewed in the following sections.Source: §2 · pp. 2–3

2.1 Text Source Detection for Large Language Models

  • An earlier LLM-watermarking method requires access to the generation phase, whereas DeepTextMark is described as model-independent, with about 50 million parameters versus billions for.The paper presents this as enabling use without modifying the LLM.Source: §1 · p. 2, §2.1 · p. 3
  • DeepTextMark targets LLM output; the authors note that a human rewrite of generated text may no longer be subject to plagiarism detection in their scenario.The paper says removing its watermark requires rewriting text because watermark locations are undisclosed.Source: §2.1 · p. 3

2.2 Traditional Text Watermarking

  • Classical watermarking includes whitespace, syntactic, semantic, linguistic, and structural approaches, with cited examples constrained by removability, automation, blindness, or visibility concerns.For example, classical semantic watermarking requires the original text for detection, while structural marks may be identified through visually unusual characters.Source: §2.2 · pp. 3–4
  • DeepTextMark instead uses Word2Vec and Universal Sentence Encoder for insertion and a transformer for detection, an approach the authors say supports blindness, imperceptibility, robustness, and reliability.The paper distinguishes this deep-learning-based approach from traditional watermarking techniques.Source: §2.2 · p. 4

3 The Proposed DeepTextMark

DeepTextMark is presented as an automatic, blind watermarking approach that inserts sentence-level marks and detects them, then aggregates sentence predictions for multi-sentence texts.

3.1 Watermark Insertion

  • Word2Vec nearest neighbors supply replacement words to create sentence proposals, while punctuation, stopwords, and whitespace are preserved.The method considers single-word, multiple-word, and flexible synonym-substitution variants.Source: §3.1 · p. 4
  • The Universal Sentence Encoder ranks proposals by cosine similarity to the original, while grammatical adjustment preserves sentence structure and punctuation.Number- and part-of-speech-matched synonyms help avoid grammatical mismatches.Source: §3.1 · p. 5

3.2 Watermark Detection

  • The BERT detector classifies watermarked versus unmarked text by minimizing binary cross-entropy; staged freezing and unfreezing trains it, and prediction needs no original data.Training begins with frozen BERT parameters and later unfreezes them at a reduced Adam learning rate.Source: §3.2 · pp. 6–7

3.3 Watermark Detection for Multiple Sentences

  • With per-sentence accuracy reasonably high (>85%), majority classification of a sentence collection approaches near-perfect accuracy under the binomial analysis.A higher single-sentence accuracy requires fewer sentences to reach near-perfect collection accuracy.Source: §3.3 · p. 7

4 Experiments

Experiments assess DeepTextMark’s detection reliability, imperceptibility, robustness, comparison with WLP, and runtime. Across these evaluations, the paper reports strong detection and near-zero attack impact under tested conditions.

4.1 Dataset

  • Training used 34,489 Dolly sentences, half watermarked, with a 75/25 training-validation split; testing used 8,800 C4 datasets.Source: §4.1 · pp. 7–8

4.2 Watermark Detection Accuracy

  • Validation accuracy ranged from 76.30% for C4 with single synonyms to 95.72% for C4 with multiple synonyms; predicted collection-level reliability rose with sentence count.Source: §4.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8

4.3 Ablation Study

  • Full configuration D achieved the highest accuracy among the four ablation configurations, which the authors attribute to its additional enhancements.Source: §4.3 · p. 9, Table 5 · p. 9

4.4 Imperceptibility of Watermark Insertion

  • 0.9765 mSMS for single synonyms and 0.9892 for multiple synonyms, compared with 0.9794 for traditional watermarking.Source: §4.4 · p. 9

4.5 Robustness

  • The proposed mIOA measures mean accuracy change after attacks; greater distance from zero indicates greater impact, and tests model common text interactions.Source: §4.5 · pp. 10–11
  • For removal and replacement attacks, mIOA increased with severity but stayed close to zero; greater impact was expected as modified text became more dissimilar.Source: §4.5 · pp. 11–12

4.6 Comparative Analysis

4.7 Emperical Running Speed

  • On a single Intel i9-13900K CPU core, insertion and detection ran quickly, with insertion taking longer; the authors say parallelization could reduce its overhead.Source: §4.7 · pp. 13–14, Table 14 · p. 14

5 Conclusion

DeepTextMark embeds watermarks sentence by sentence to identify LLM-generated text, reporting near-perfect detection accuracy with a small set of sentences. Its applicability depends on prior watermarking and representative training data, while support for shorter and more diverse text remains future work.

5 Conclusion

  • Near-perfect detection accuracy is reported with merely a small set of sentences, while direct classifiers require substantial characters for accurate prediction.The authors state that watermark insertion and detection operate at sentence level.Source: §5 · p. 14
  • DeepTextMark is an add-on watermarking method requiring no access to LLM generation, with automatic and imperceptible insertion and robust, high-accuracy detection.Source: §5 · p. 15
  • Effectiveness depends on representative training data and prior watermarking; current results cover standard-length texts, with very short or stylistically diverse texts still needing adaptation.Source: §5 · p. 15
  • Future directions include stronger resistance to AI rewriting, scaling to diverse datasets, broader LLM compatibility, real-time plugins, and privacy, security, ethical, and legal considerations.Source: §5 · p. 15

SUPPLEMENTARY DOCUMENTS SENTENCE LEVEL ATTACKS

The section examines sentence-level token attacks and a pre-detection option that selectively leaves sentences unchanged to improve imperceptibility. It reports that pre-detection preserved detection performance in the example while 87% of its sentences remained detectable.

SUPPLEMENTARY DOCUMENTS SENTENCE LEVEL ATTACKS

  • The supplementary tests remove random tokens, add random words, or replace random words in single sentences to challenge sentence-level robustness.The passages identify removal as potentially removing the watermark and replacement as combining removal with added noise.Source: SUPPLEMENTARY DOCUMENTS SENTENCE LEVEL… · p. 17
  • The token-removal, token-addition, and token-replacement tables report mIOA for watermarked and unmarked datasets across selected attack sizes.The supplied passages describe the table scopes but do not provide their numerical results.Source: Table 15 · p. 17, Table 16 · p. 17, Table 17 · p. 17
  • Pre-detection checks each sentence before watermarking and uses the original sentence when detection would fail, with a 51% watermarkability threshold.The method checks whether at least 51% of the text is watermarkable; otherwise, it determines that the text is not watermarkable.Source: SUPPLEMENTARY DOCUMENTS SENTENCE LEVEL… · p. 17
  • 87% of sentences remained detectable with pre-detection, which improved watermark imperceptibility without impacting detection performance.The two sentences that were not detectable did not need modification.Source: SUPPLEMENTARY DOCUMENTS SENTENCE LEVEL… · p. 19

Detecting Watermarked Percentage

DeepTextMark can estimate the proportion of a text that is watermarked by counting sentence-level detections. In the example, it reports 67%, which the passage identifies as correct.

Detecting Watermarked Percentage

ORIGINAL VS TRADITIONAL VS DEEPTEXTMARK

The examples compare original text with traditional watermarking and DeepTextMark variants using single or multiple synonym substitutions. The displayed outputs range from one changed word to many altered word choices.

ORIGINAL VS TRADITIONAL VS DEEPTEXTMARK

3. Which is a mintage of fish tope or rope

The examples show a camel-related sentence in an original form and several increasingly altered variants. The variants change grammatical form and substitute words or phrases.

3. Which is a mintage of fish tope or rope

  • The original example says camels use fat in their humps to stay supplied with energy and hydration for long periods.Source: §4 · p. 20
  • One variant changes “Camels” to “Camel,” while keeping the rest of the sentence wording intact.Source: §4 · p. 20
  • A further variant replaces the original phrasing with “utilization” and makes additional word substitutions.Source: §4 · p. 20

1. Alice’s parents have three daughters amy jessy and whats the name of the third daughter

The passages identify Tomoaki Komorida’s birth date and trace his football career; they also describe Lollapalooza and explain chess stalemate as a draw.

4. When was tomoaki komorida born

  • Komorida was born in Kumamoto Prefecture on July 10, 1981, and played for Japanese clubs before retiring in 2012.His career included a 2010 move to Indonesia and a return to Japan that July.Source: §4 · p. 20
  • Stalemate is a draw when the player to move is not in check and has no legal move.The passage says the outcome was standardized as a draw in the 19th century.Source: §4 · pp. 21–22
  • Lollapalooza is an annual four-day music festival held in Grant Park, Chicago, created in 1991 as a farewell tour by Perry Farrell.The passage estimates that it hosts 400,000 people each July and sells out annually.Source: §4 · p. 22

3. Who was lav moses browning

This section covers John Moses Browning’s firearms designs and a set of prompts and responses presenting profiles of Dar es Salaam.

Who was lav moses toasting

  • Browning developed firearms from age 13 in his father’s shop and received his first patent at age 24.He was awarded 128 firearm patents in total.Source: §4 · pp. 23–24
  • His innovations included the first reliable compact autoloading pistol and the first gas-operated firearm, alongside improvements to several rifle and shotgun actions.His telescoping-bolt design became the basis for the pistol slide and is used in nearly every modern semiautomatic pistol.Source: §4 · pp. 23–24
  • The M1911, Browning Automatic Rifle, and Auto-5 are among Browning’s best-known designs and remain in production, sometimes with minor changes.The M1911 and Hi-Power designs are among the world’s most copied or reproduced firearms.Source: §4 · pp. 23–25

4. Who was lav moses toasting

  • The prompts ask for areas associated with Dar es Salaam from a transit list, requesting comma-separated results.A subsequent prompt variant alters wording, including “transit lean” and “comma separated formatting.”Source: §4 · p. 30
  • The responses describe Dar es Salaam as Tanzania’s largest city and financial hub, a major economic centre, and a prominent centre for arts, fashion, media, film, television, and finance.They also recount its founding in 1865 or 1866 and the later move of Tanzania’s capital to Dodoma, completed in 1996.Source: §4 · p. 30
  • The profile identifies five districts in Dar es Salaam: Kinondoni, Ilala, Ubungo, Temeke, and Kigamboni.It places them across the city’s north, centre, south, and east, respectively, with Ubungo and Temeke both in the south.Source: §4 · p. 30

1. Arts fashion media film television finance

Process mining uses event logs to analyze operational processes and turn event data into insights and actions; another passage defines polygons geometrically.

1. Arts fashion media film television finance

  • A polygon is described as a closed chain of connected line segments whose meeting points are vertices; circles are given as nonpolygons.Source: §1 · p. 31
  • Process mining analyzes operational processes from event logs, using event data to identify execution paths and address performance and compliance problems.Each event should include a case ID, activity description, and timestamp; additional attributes such as resources or costs are optional.Source: §1 · p. 32
  • Process mining brings together data mining, data science, and process management to find process efficiencies.Source: §1 · p. 32

1. What are some unique curtain tie backs that you can make yourself

The suggestions include simple materials, repurposed accessories, and decorative objects as possible curtain tiebacks.

1. What are some unique curtain tie backs that you can make yourself

  • Jute rope, wooden beaded rope, and chains are among the suggested materials for curtain tiebacks.Source: §1 · p. 33
  • A circle and pin, a door knob, or a leather belt can be repurposed as a curtain tieback.Source: §1 · p. 33
  • Necklaces or bracelets, macrame rope, and artificial flowers are also offered as tieback options.Source: §1 · p. 33
Loading 2305.05773v2…