PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Watermarks Survive Translation? On the Cross-lingual Consistency of Text Watermark for Large Language Models

Zhiwei He, Binglin Zhou, Hongkun Hao, Aiwei Liu, Xing Wang, Zhaopeng Tu, Zhuosheng Zhang, Rui Wang

arXiv:2402.14007v2cs.CLcs.AI

TL;DR

The paper asks whether LLM text watermarks remain effective after translation, finding current methods inconsistent across languages. It evaluates this gap, introduces CWRA and proposes X-SIR; CWRA lowers AUCs to random-guessing level without quality loss, while X-SIR improves cross-lingual detection.

  • Problem

    The paper asks whether current text watermarking methods maintain their strength across languages, a concern because LLM-generated text can be misused.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Method

    The authors evaluate cross-lingual consistency, propose CWRA using pivot-language generation followed by translation, and develop X-SIR around two cross-lingual semantic factors.

    Source: §1 · p. 2

  • Results

    CWRA decreases AUCs to a random-guessing level without degrading text quality, while X-SIR greatly improves watermark detection under cross-lingual conditions.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Cross-lingual consistency remains unresolved: X-SIR improves detection but does not solve the issue.

    Source: §8 · p. 9, §7 · p. 9

    X-SIR's external dictionary contains only whole words, leaving word units unclustered and potentially compromising performance with finer-grained tokenization; it does not solve cross-lingual consistency.

    Source: §8 · p. 9

Abstract

from arXiv · show

Text watermarking technology aims to tag and identify content produced by large language models (LLMs) to prevent misuse. In this study, we introduce the concept of cross-lingual consistency in text watermarking, which assesses the ability of text watermarks to maintain their effectiveness after being translated into other languages. Preliminary empirical results from two LLMs and three watermarking methods reveal that current text watermarking technologies lack consistency when texts are translated into various languages. Based on this observation, we propose a Cross-lingual Watermark Removal Attack (CWRA) to bypass watermarking by first obtaining a response from an LLM in a pivot language, which is then translated into the target language. CWRA can effectively remove watermarks, decreasing the AUCs to a random-guessing level without performance loss. Furthermore, we analyze two key factors that contribute to the cross-lingual consistency in text watermarking and propose X-SIR as a defense method against CWRA. Code: https://github.com/zwhe99/X-SIR.

1 Introduction

Text watermarks are intended to identify LLM-generated content, but their robustness after translation is uncertain. The paper examines this gap, proposes a translation-based attack, and introduces X-SIR as a defense.

1 Introduction

  • Text watermarking aims to tag and identify LLM-generated content, in response to concerns about misuse such as misleading information, impersonation, and compromised academic integrity.The paper notes that prior robustness evaluations focused on rewriting and copy-paste attacks in monolingual contexts.Source: §1 · p. 1
  • Preliminary results across 2 LLMs × 3 watermarks show that current text watermarking technologies lack consistency across languages.The work frames cross-lingual consistency as the ability of watermarks to preserve their strength across languages.Source: §1 · p. 2
  • CWRA generates a response in a pivot language and translates it back, reducing AUCs to random-guessing level while achieving the highest text quality.The paper reports that CWRA outperforms re-translation and paraphrasing attacks.Source: §1 · p. 2
  • X-SIR defends against CWRA by clustering semantically equivalent tokens across languages and making vocabulary partitions similar for semantically similar contexts.The paper reports that X-SIR substantially elevates AUCs under CWRA.Source: §1 · p. 2

2 Background

The background introduces autoregressive language-model generation and describes watermarking methods that modify generation logits and detect watermarked text. It covers KGW, unbiased watermarking, and SIR.

2.1 Language Model

  • A language model generates text autoregressively by predicting one token at a time from the conditional distribution over its vocabulary.The model computes next-token logits from the current sequence and applies softmax to obtain probabilities.Source: §2.1 · p. 2

2.2 Watermarking for LMs

  • The watermarking methods considered modify generation logits to embed a watermark and detect its presence in text.The section considers KGW, unbiased watermarking, and SIR.Source: §2.2 · pp. 2–3
  • KGW hashes the current prefix, seeds a random generator, partitions the vocabulary into green and red lists, and adjusts logits to favor green tokens.Detection recalculates the partition and uses the green-token z-score to assess watermark strength against a threshold.Source: §2.2 · p. 2
  • SIR trains watermark biases to be similar for semantically similar prefixes; this work uses multilingual S-BERT instead of the original English-only C-BERT.SIR is described as an improvement based on KGW.Source: §2.2 · p. 3

3 Cross-lingual Consistency of Text Watermark

The paper defines cross-lingual consistency as watermark strength retention after translation and evaluates it across models, watermark methods, and languages. The results show generally weak consistency, no significant advantage for similar-language pairs, and comparatively stronger—but still reduced—consistency for SIR.

3.1 Definition

  • Cross-lingual consistency measures a watermark’s ability to retain its strength after translation, assessed using PCC and relative error.PCC captures trends between original and translated watermark strengths; relative error captures their magnitude of deviation.Source: §3.1 · p. 3

3.2 Experimental Setup

  • The evaluation translated watermarked responses from 500 English prompts into Chinese, Japanese, French, and German using two LLMs and three watermark methods.Responses were generated using multinomial sampling; the study notes English is more similar to French and German than to Chinese and Japanese.Source: §3.2 · p. 3, Table 1 · p. 4

3.3 Results

  • PCCs are generally below 0.3 and REs predominantly above 80%, while watermark strengths decrease after translation across the three methods.These findings indicate that current watermarking methods struggle to maintain effectiveness across language translations.Source: §3.3 · p. 4, Figure 2 · p. 4
  • None of the three methods shows significantly better consistency between similar languages than between distant languages.Even shared words or similar language structures do not ensure easy watermark transfer.Source: §3.3 · p. 4
  • SIR achieves the best average results across two models and two metrics, but still shows a notable cross-lingual strength reduction.With BAICHUAN-7B, SIR has higher PCCs than the other methods for every target language.Source: §3.3 · p. 4

4 Cross-lingual Watermark Removal Attack

CWRA translates a prompt into a pivot language, obtains an LLM response there, and translates it back to the original language. Across summarization and question answering, it is the most effective of the tested attacks, while text quality is preserved or slightly improved in most cases.

4 Cross-lingual Watermark Removal Attack

  • CWRA translates the prompt into a pivot language, generates a response there, and translates that response back into the original language.The second translation step effectively removes the watermark; the evaluation used summarization and question answering datasets with Chinese as the pivot language.Source: §4 · p. 5, §4.1 · p. 5

4.1 Setup

  • The evaluation compared CWRA with paraphrase and re-translation on 500 samples each from Multi-News and ELI5.The tests used BAICHUAN-7B; paraphrasing rephrased the response, while re-translation translated it to the pivot language and back.Source: §4.1 · p. 5

4.2 Results

  • CWRA most effectively diminishes watermark-detection AUCs and TPRs, reducing original-response n-grams through language switching.Figure 4 presents overall ROC curves for re-translation, paraphrase, and CWRA attacks.Source: §4.2 · p. 5, Figure 4 · p. 6
  • Under CWRA, KGW and UW reach AUCs of 0.61 and 0.54, respectively, while SIR’s AUC is 0.67.The KGW and UW values approach random guessing; SIR is comparatively more robust.Source: §4.2 · p. 5
  • The tested attacks preserve text quality and bring slight improvements in most cases.The authors speculate CWRA’s stronger text quality may reflect BAICHUAN-7B performing better in Chinese than English.Source: §4.2 · p. 5

5 Improving Cross-lingual Consistency

The paper analyzes two factors for cross-lingual consistency and proposes a defense against CWRA. For KGW-based methods, the stated goal is for green tokens to remain recognized as green after translation.

5 Improving Cross-lingual Consistency

  • The section analyzes two factors for cross-lingual consistency and proposes a defense method against CWRA.The supplied passage does not specify the defense method’s design.Source: §5 · p. 6

5.1 Two Key Factors of Cross-lingual Consistency

  • For KGW-based watermarking, the goal is that green tokens remain recognized as green after translation into other languages.The paper introduces a toy case to analyze this goal.Source: §5.1 · p. 6

1. We define two simple languages. Language 1

The toy analysis identifies two vocabulary-partition conditions for retaining a watermark across translation. X-SIR implements semantic clustering and improves detection under direct translation and CWRA, with gains varying by target language.

1. We define two simple languages. Language 1

  • The toy example requires equivalent tokens to share a red-or-green partition and equivalent prefixes to share vocabulary partitions; violating either condition loses consistency.The example assumes a bilingual model and omits prefix tokens for simplicity.Source: §1 · pp. 6–7, Figure 5 · p. 7

5.2 Defense Method against CWRA

  • X-SIR assigns logit biases by semantic cluster, giving semantically similar tokens the same bias and translations the same list.The clusters are formed from connected components using bilingual-dictionary entries.Source: §5.2 · p. 7
  • X-SIR is evaluated on BAICHUAN-7B using a unified external dictionary covering En, Zh, Ja, Fr, and De.The authors state that they present further results on other LLMs in Appendix C.Source: §5.2 · p. 7
  • For Zh and Ja, X-SIR improves PCC and RE over SIR; for De and Fr, it improves PCC but not RE.The figure plots 100%-RE for visualization.Source: §5.2 · p. 7, Figure 6 · p. 8
  • X-SIR increased watermark-detection TPR by 0.448 and AUC by 0.221 on average across direct translation and CWRA.The comparisons cover direct translation and the full CWRA process, with English as the original language and Chinese as the pivot language.Source: Figure 7 · p. 8, §5.2 · p. 8
  • Without attack, X-SIR and SIR perform comparably; under paraphrase attack, X-SIR performs slightly better.The authors describe paraphrasing as a same-language form of translation that X-SIR's intralingual consistency may improve.Source: Figure 8 · p. 8, §5.2 · p. 8

6 Related Work

Prior work covers watermarking for generated text and methods designed to improve robustness, especially against single-language attacks.

6.1 LLM Watermarking

  • This paper focuses on watermarking generated text, which the related-work section describes as more challenging and practically applicable than watermarking existing text.The section distinguishes watermarking existing text from watermarking generated text.Source: §6.1 · p. 8
  • Related methods include KGW, UW, SIR, publicly verifiable watermarking, sentence-level SemStamp, and the WaterBench benchmark.The section characterizes SIR as accounting for semantic information and robustness to rewriting attacks.Source: §6.1 · p. 9

6.2 Watermark Robustness

  • Prior robustness work mainly studies single-language attacks, including paraphrasing, copy-paste, and editing.Examples include a fixed green list, text-redundancy-based XMark, and entropy-based detection for low-entropy scenarios.Source: §6.2 · p. 9

7 Conclusion

The paper evaluates cross-lingual watermark consistency, proposes CWRA, and develops X-SIR as a defense. It reports improved cross-lingual detection while acknowledging limitations.

7 Conclusion

  • The authors characterize current watermarking methods as struggling to maintain watermark strength across languages.They frame the work as evaluating cross-lingual consistency, proposing an attack, and analyzing a defense.Source: §7 · p. 9
  • CWRA removes watermarks without compromising text quality, and X-SIR improves watermark detection in cross-lingual scenarios despite limitations.The conclusion describes the work as covering evaluation, attack, analysis, and defense.Source: §7 · p. 9
  • X-SIR has been integrated into the open-source MarkLLM toolkit for LLM watermarking.The paper attributes the toolkit to Pan et al. (2024).Source: §7 · p. 9

8 Limitations

X-SIR’s cross-lingual coverage depends on model language support and on whether dictionary entries align with the model’s tokenization.

8 Limitations

  • X-SIR supports only languages supported by the model; BAICHUAN-7B shows larger gains for Zh and Ja than De and Fr because its vocabulary better supports Zh and Ja.The authors note that an attacker can choose the original and pivot languages at will in real scenarios.Source: §8 · p. 9
  • Because the external dictionary contains only whole words, some word units remain isolated, so finer-grained tokenization may compromise X-SIR’s performance.The authors state that X-SIR does not solve cross-lingual consistency, but sets the stage for future research.Source: §8 · p. 9

A Details of Watermarking Methods

The appendix describes how KGW, UW, and SIR embed and detect watermarks, then points to full removal-attack results across summarization and question-answering tasks.

A Details of Watermarking Methods

  • Watermarking methods modify LLM generation logits to embed a watermark, then compute a strength score s to assess its presence.A higher s indicates text is more likely to contain a watermark; a lower s indicates it is less likely.Source: A Details of Watermarking Methods · p. 12

A.1 KGW

  • KGW hashes the previous k tokens; this study follows the original settings with minhash, k = 4, γ = 0.25, and δ = 2.0.Source: §A.1 · p. 12

A.3 SIR

  • SIR assigns each token a watermark bias and detects watermarks using the mean bias across the text.Because the bias is unbiased across the vocabulary, the expected score for normal text is 0, while watermarked text should score significantly above 0.Source: §A.3 · pp. 12–13

B Full Result of Watermark Removal Attack

  • Figure 9 reports full watermark-removal attack results separately for text summarization and question answering.The passage says these results are equivalent to Figure 4, but provides no numerical outcomes.Source: B Full Result of Watermark Removal… · p. 13

C Full Result of X-SIR

This section presents full detection-performance plots for X-SIR and baselines across several LLMs and attack settings. The figures show ROC curves and report AUC and TPR at a fixed FPR of 0.1, but these passages do not state comparative outcomes.

C Full Result of X-SIR

Loading 2402.14007v2…