PaperlayerView exact revision on arXiv

Source-linked AI summary

No Tokenizer, Still a Gap: Cross-Lingual Fairness of Homoglyph and Invisible-Character Watermarking

Alexander Nemecek, Yuqiao Xu, Joseph Falkenburg, Erman Ayday

arXiv:2610.36432v1cs.CLcs.CRcs.CY

TL;DR

Generation-time watermarking can weaken in lower-resourced languages, raising the question of whether edit-based watermarking avoids that gap despite not using a tokenizer. Across 40 varieties and 24 scripts, the paper measures capacity, survival, visibility, and detector needs, finding that disparities remain organized by script and widen under copy-paste.

  • Problem

    Generation-time watermarks weaken in lower-resourced languages, and whether edit-based watermarks avoid this gap despite not using tokenizers had not been examined.

    Source: §1 · pp. 1–2

  • Method

    The paper measures character-level capacity, transformation survival, detector text needs, and visibility across language varieties and scripts, using Unicode character sets and defenses.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Capacity varies more than threefold and script accounts for almost all variation; after copy-paste, detection needs about 2 code points for English or Persian but 93 for Lao.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Removing the tokenizer does not remove the inequity: it shifts its source to scripts’ uneven supply of homoglyphs and routine handling that strips invisible characters.

    Source: §1 · p. 2, §8 · p. 8

    The script-level association cannot separate script identity from resource class, and the Latin-only null result at n = 11 bounds an effect rather than ruling one out.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Homoglyph and invisible-character watermarks are edit-based watermarks. Instead of changing how a model generates text, they hide a signal in the text after it is written. They swap some characters for homoglyphs, which look the same, or very similar, but have different Unicode code points, or they insert characters that do not display, all without access to the model that wrote the text. Generation-time watermarks work less well for lower-resourced languages due to how tokenizers split their text. Because they work on individual characters, these watermarks never touch a tokenizer, so we ask whether they escape this gap. Across 40 language varieties in 24 scripts, we measure capacity (the hidden bits each character can carry), how much of it survives routine handling and targeted removal, and how much text a keyed detector needs. Capacity varies 3.3-fold and script accounts for 96\% of that variation. Within Latin script, resource level makes no detectable difference ($\rho=+0.095$, $p=0.78$), but across scripts capacity rises with it ($\rho=+0.54$, 95\% CI $[+0.14,+0.83]$), because lower-resourced languages tend to use scripts with few homoglyphs. Ordinary copy-paste, which strips invisible characters, widens the gap: a detector then needs about 2 code points of English or Persian but 93 of Lao. Removing the tokenizer does not remove the inequity, but instead only changes its source.

1 Introduction

Edit-based watermarks hide signals in finished text using homoglyph substitutions or invisible-character insertions, avoiding access to the generating model and its tokenizer. This section asks whether that avoids cross-lingual disparities and finds that capacity and detector needs still vary across scripts and resource levels.

Approach

  • Edit-based watermarks modify finished text: a secret key selects visually similar characters with different code points or inserts invisible characters, and a keyed detector checks for the pattern.They do not require access to the model that wrote the text.Source: §1 · p. 1

Research gap

  • Earlier work explains generation-time watermark disparities through tokenization, while cross-script capacity and fairness for edit-based watermarks had not been examined.Prior edit-based reports focused on aggregate capacity or single scripts, and watermark fairness work covered statistical schemes.Source: §1 · pp. 1–2

Evaluation design

  • The evaluation separates routine handling from deliberate removal; combining Unicode mapping with compatibility and whitespace folding erases the watermark in every language.Routine transformations happen without an adversary, whereas deliberate transformations use Unicode defenses.Source: §1 · p. 2

Main findings

  • Capacity varies more than threefold, with script accounting for almost all variation; within Latin, resource level makes no detectable difference, while lower-resourced languages tend to use scripts with few homoglyphs.The section links this disparity to the Unicode supply of homoglyphs across scripts.Source: §1 · p. 2
  • After copy-paste strips invisible characters, a detector needs about 2 code points of English or Persian but 93 of Lao.Ordinary handling widens the detector-length gap.Source: §1 · p. 2

Study scope

  • The study measures capacity, survival, and visibility across 40 language varieties and 24 scripts, and introduces a keyed instrument to relate capacity to detector text needs.It also analyzes fairness with scripts as the unit of evidence.Source: §1 · p. 2

2 Related Work

Prior research documents cross-lingual weaknesses in generation-time watermarks and studies edit-based techniques, Unicode defenses, and watermark fairness. The paper identifies missing cross-script capacity comparisons and character-level fairness analysis as gaps.

Generation-time watermarks across languages

  • Generation-time watermarks can fail after translation and weaken in low- and medium-resource languages; tokenization is the usual explanation for the latter disparity.Lower-resourced languages are split into more and smaller pieces, shifting the per-token statistics detectors use.Source: §2 · p. 2

Edit-based watermarks

  • Prior edit-based work covers homoglyph substitution, variation selectors, spaces, and zero-width characters, but reports no cross-script comparison or capacity weighted by languages’ actual character use.Some work reports only a single aggregate capacity figure or studies one script.Source: §2 · pp. 2–3

Homoglyphs as an attack

  • Security research studies homoglyphs and invisible characters as threats, while UTS #39 lists confusables and specifies defenses that the paper uses as removal attacks.The defenses include mixed-script detection and skeleton matching.Source: §2 · p. 3

Fairness in watermark evaluation

  • Existing watermark-fairness benchmarks cover statistical and frequency-domain schemes and measure detection performance, whereas this work measures character-channel capacity and traces its gap to Unicode’s uneven homoglyph supply.Source: §2 · p. 3

3 Measurement Framework

The framework models a keyed watermark as substitutions and insertions in finished text, then measures capacity and its survival under transformations. It defines the character sets, admissibility policies, and capacity calculations used in those measurements.

Measurement Framework

  • A keyed embedder changes characters or gaps in finished text, and a detector with the same key checks the text; using the original text makes this detector cover-informed and an upper bound on one without it.The original is needed because a homoglyph does not reveal which character it replaced.Source: §3 · p. 3

Data and tracks

  • The framework uses FLORES+ devtest sentences for 40 varieties and models separate substitution and insertion tracks, drawing substitutes from cross-script, compatibility, whitespace, and residual families.Invisible characters form the insertion alphabet, not a substitution family; multi-character homoglyph sequences are excluded.Source: §3 · p. 3

Admissibility

  • Three nested admissibility policies allow substitutes from the same script, the language’s usual scripts, or any script; the paper reports the unrestricted policy unless stated otherwise.The policies limit capacity but do not guarantee stealth.Source: §3 · pp. 3–4

Capacity

  • Capacity averages the bits available from each character’s admissible forms using language character frequencies; insertion capacity comes from gaps that can hold no character or one invisible character.The insertion alphabet contains four characters, or two for Arabic and Brahmic scripts where the joiners affect letter joining.Source: §3 · p. 4

Transformations

  • The transformation ladder measures surviving capacity under routine normalization and copy-paste, then Unicode defenses including mixed-script detection and skeleton mapping.Its final combined step adds compatibility and whitespace folding to the full skeleton, removing both tracks.Source: §3 · p. 4, Table 1 · p. 4

4 Capacity

Untransformed capacity varies substantially by language, largely along script lines. The gap reflects differences in available homoglyphs, especially compatibility variants, while invisible-character insertion contributes a more uniform amount.

4 Capacity

  • 7.09 bits per character for Swahili and English versus 2.14 for Khmer yields a 3.31-fold capacity range.Latin-script languages average 6.81 bits per character, compared with 3.21 for Chinese, Japanese and Korean and 2.93 for Brahmic scripts.Source: §4 · p. 4
  • Substitution ranges from under one to 4.78 bits per character for English, while insertion depends on safe invisible characters rather than letters.Insertion is about 2.30 bits per character, or 1.57 in Arabic and scripts that form conjuncts, which exclude two joiners.Source: §4 · p. 4
  • Compatibility variants contribute more than three bits per character for most Latin-script languages but almost nothing for Brahmic scripts.Unicode’s uneven supply of compatibility variants reflects its legacy and styling needs, rather than a property of the measurement method.Source: §4 · pp. 4–5
  • Khmer and Myanmar have the lowest totals, 2.14 and 2.40 bits per character, while Arabic, Persian and Urdu reach 5.5 to 5.8.Arabic-script languages share the insertion penalty but have many compatibility variants.Source: §4 · p. 5
  • Chinese and Japanese characters have few native homoglyphs; characters outside their native sets provide 49 to 65% of their substitution capacity, versus 18% for English.Those characters account for all Korean substitution capacity.Source: §4 · p. 5
  • Restricting substitutions to intra-script options leaves Latin languages 93 to 95% of substitution capacity but Brahmic and South-East Asian languages 54 to 81%.Keeping only UTS #39 Allowed substitutes leaves English with 1.42 of its 4.78 substitution bits.Source: §4 · p. 5

5 Fairness

Script explains nearly all capacity variation, while resource level shows no detectable association within Latin script but rises with capacity across scripts. These patterns cannot separate script from resource level at the between-script level.

5 Fairness

  • ICC = 0.963: about 25 times more capacity variance lies between scripts than within them.None of 10,000 shuffles reached this value (p < 10−4), and the ICC remains high across transformations and policies.Source: §5 · p. 5, Figure 3 · p. 6
  • Arabic, Persian and Urdu span three resource classes but only 0.29 bits per character in capacity; Latin languages span 1.19 bits.The Latin spread is small beside the roughly 5-bit difference between scripts.Source: §5 · p. 5
  • Changing only the characters gives Simplified and Traditional Mandarin a 1.04-fold capacity difference, rising to 1.24-fold after copy-paste removes insertion.Insertion is the same for both forms.Source: §5 · p. 6
  • Within Latin script, capacity and resource class show no detectable association (ρ = +0.095, permutation p = 0.78).The 11 languages span four classes, whose means show no trend.Source: §5 · p. 6
  • Across 24 scripts, median capacity rises with median resource class (ρ = +0.54, 95% interval [+0.14, +0.83]).Chinese, Japanese and Korean are the main exception: they are in classes 4 and 5 but have little capacity.Source: §5 · p. 6, Figure 4 · p. 6
  • The between-script association does not establish that resource level is irrelevant, because each script contributes only one point.The evidence unit is the script, with intervals resampling whole scripts rather than treating languages sharing a script as independent.Source: §5 · pp. 5–6, Table 2 · p. 6

6 Robustness

Effective capacity counts distinct watermark outcomes that survive a transformation. The tracks respond differently to handling and defenses, and copy-paste widens the capacity disparity most.

6 Robustness

  • Copy-paste removes all inserted characters, and total capacity retains 0.51 of its value on average; the combined attack erases the mark in every language.Across transformations, average total capacity retains 0.99 under NFC, 0.93 under mixed-script detection, 0.83 under NFKC and 0.68 under skeleton mapping.Source: §6 · p. 7
  • NFKC removes 96% of compatibility pairs but keeps 93% of cross-script pairs, so languages lose capacity in proportion to their compatibility-variant share (r = −0.84).Highly Restrictive mixed-script detection retains about 0.65 of substitution capacity for Brahmic and South-East Asian scripts, versus 0.94 to 0.98 for Latin, Arabic and CJK.Source: §6 · p. 7
  • Copy-paste widens the capacity range from 3.31-fold to 8.49, while NFKC and skeleton mapping narrow it to 2.63 and 2.59.Removing insertion costs Khmer 74% of capacity but Swahili 33%.Source: §6 · p. 7
  • The cost of staying hidden also follows script: Tamil, Bengali and Arabic face different stealth constraints tied to Unicode assignments, fonts and shaping.The study checks mixed-script detection, identifier status and rendered glyphs.Source: §6 · p. 7

7 Keyed Validation

The keyed validation tests how span length and measured channel capacity relate to detection, including under routine and targeted attacks. Copy-paste sharply widens the language gap, while the combined attack drives detection to chance.

7 Keyed Validation

  • The keyed detector rebuilds a schedule from the key and cover, counts matching slots, and thresholds the count’s z-score at the 99th percentile of null trials.Source: §7 · pp. 7–8
  • Copy-paste expands the nmin range 50×: English and Persian need about 2 code points, while Lao needs 92.96.Source: Figure 7 · p. 8, §7 · p. 8
  • On clean text, nmin spans 1.85 code points (Persian) to 4.95 (Khmer), and nmin ≈ k/κeff with k = 10.02 bits.The rank correlation between nmin and 1/κeff is 0.877 clean, 0.969 under copy-paste, 0.863 under the mixed-script detector, and 0.552 under NFKC.Source: Figure 7 · p. 8, §7 · p. 8
  • The sweep tests spans of 1–256 code points under clean text, copy-paste, mixed-script detection, and NFKC; unusable slots count as zero evidence.nmin is the shortest span reaching 95% true-positive rate at 1% false-positive rate.Source: §7 · p. 8
  • The combined attack in Table 1 leaves detection at chance for all 40 languages, with largest AUC 0.589.Source: §7 · p. 8

8 Conclusion

Across 40 language varieties in 24 scripts, edit-based watermarks avoid tokenizers but retain resource disparities tied to script. Routine copy-paste widens those disparities by stripping invisible characters.

8 Conclusion

  • Script accounts for 96% of capacity variation, while resource level makes no detectable difference within Latin script.Across scripts, lower-resourced languages tend to use scripts with fewer homoglyphs and therefore have less capacity.Source: §8 · p. 8
  • Copy-paste widens the detection-length gap from 3.3-fold to 8.5-fold, requiring about 2 code points for English or Persian but 93 for Lao.Source: §8 · p. 8
  • The authors conclude that removing the tokenizer changes rather than removes inequity, and that narrowing the gap would require more complete confusables data or a script-independent post-hoc channel.Source: §8 · p. 8

Limitations

The findings are bounded by study design, corpus and taxonomy choices, Unicode and renderer versions, and the detector’s channel assumptions. Several perceptual and language-level results therefore have specific scope limits.

Design

  • Twenty of 24 script groups contain one language, and the Latin-only within-script test (n = 11) can bound but not rule out an effect.At the script level, resource class and script identity cannot be separated; the positive association cannot be attributed to either.Source: Limitations · p. 9

Resource taxonomy

  • Seven resource-class matches are judgment calls, and Serbian has a plausible lower-class alternative; different taxonomy or matching may change classes, not script groupings.Source: Limitations · p. 9

Corpus

  • The study uses one parallel corpus in one register, limiting generalization; the FLORES+ dev split changes substitution capacity by 0.012 bits per character on average and at most 0.060.Source: Limitations · p. 9

Snapshot

  • Capacity estimates are specific to Unicode 17.0.0, CLDR 48.2.0, and FLORES+ 4.5; the NFKC attack uses a library with Unicode 14.0 data.That library cannot fold some newer variants, leaving four Cyrillic-script languages more NFKC capacity than Unicode 17.0.0 normalization would.Source: Limitations · p. 9

Renderer dependence

  • Perceptual measurements depend on font stacks and shaping engines, and 178 of 320 language-engine-character combinations were not measured; no human perceptual study was run.For Arabic, invisible-character capacity is 1.57 bits per character with OpenType and CoreText, but 0 with HarfBuzz shaping Apple’s AAT fonts.Source: Limitations · p. 9

Channel scope and detector

  • The detector is cover-informed and thus an upper bound without the original text; 1,207 multi-character-only confusables are excluded, and native invisible characters matter only for Khmer.Source: Limitations · p. 9

Ethical Considerations

The paper documents unequal protection from edit-based watermarks while presenting a measurement framework and validation instrument, not a product or attack tool. Its language-to-script analysis finds that capacity differences cluster strongly by script, while several script-level covariates offer descriptive context rather than confirmation of the mechanism.

Ethical Considerations

  • The paper contributes a measurement framework and validation instrument, not a watermarking product or attack tool; its documented harm is unequal protection across language communities.The transformation ladder, detector models, and per-language numbers can inform normalization and detection defenses and show deployers where coverage is thin.Source: Ethical Considerations · p. 9
  • The study uses public, licensed data without personal data, user content, or human subjects, and releases derived statistics rather than raw corpus sentences.The released artifacts can be rebuilt from fixed data versions, and the paper does not recommend deployment or certify a configuration as safe.Source: Ethical Considerations · p. 10

A Languages and resource classes

  • Seven resource-class matches are judgment calls, and both Mandarin varieties share one class by construction.The 40 language varieties are grouped into 24 scripts; the resource classes are matched by language name.Source: A Languages and resource classes · pp. 11–12
  • Between-script variance is 3.31 versus 0.13 within scripts, a ratio of about 25; the ICC stays high under every transformation.Languages sharing a script share kinds of homoglyphs, so their capacities rise and fall together as transformations remove particular kinds.Source: A Languages and resource classes · p. 12
  • Across 24 scripts, script size correlates with median capacity at +0.35, while the five bicameral scripts exceed others by 3.08 bits per character.Combining-mark density has a −0.06 correlation with substitution capacity; the three covariates are correlated and are descriptive, not confirmation of the mechanism.Source: A Languages and resource classes · p. 12

B Substitution sets and capacity by language

Capacity depends on both the language’s text and its admissible character substitutions, while routine and targeted transformations remove different parts of the watermark. Skeleton mapping can eliminate native-character substitution capacity entirely for half the varieties, and the full skeleton also removes insertions.

B Substitution sets and capacity by language

  • English has 4.78 bits per character of real-text substitution capacity but 2.09 over its alphabet, because frequent characters have many substitutes.The alphabet view averages log2 |Sπ(c)| over native exemplar characters; it describes writing-system richness rather than running-text richness.Source: B Substitution sets and capacity by… · p. 12
  • Insertion capacity is about 2.3 bits per character where all four invisible characters are safe and about 1.57 where the two joiners are dropped.Insertion capacity does not depend on the substitution admissibility policy, while substitution capacity never decreases across the nested policies.Source: B Substitution sets and capacity by… · p. 13
  • Recomputing on the FLORES+ dev split shifts substitution capacity by 0.012 bits per character on average and at most 0.060 for Yoruba.The reported English total-capacity bootstrap 95% interval is [7.082, 7.089].Source: B Substitution sets and capacity by… · p. 13
  • Characters outside the native set provide 65% of Simplified Chinese substitution capacity, 55% of Traditional Chinese, 49% of Japanese, and all of Korean’s.Foreign letters alone provide 27% of Simplified Chinese capacity, 7% of Japanese, and 10% of Korean, versus none for English.Source: B Substitution sets and capacity by… · p. 13
  • Cross-script twins alone provide 1.99 bits per character for English and 2.12 for Russian, but removing them lowers total substitution capacity only to 4.50 and 3.16, respectively.The comparison is under πintra; compatibility variants remain available, and per-family capacities do not add.Source: B Substitution sets and capacity by… · p. 13

C Survival under attack

  • NFKC most reduces substitution survival for Chinese, Japanese, and Arabic-script languages (0.36 to 0.53), while mixed-script detection hits Brahmic and South-East Asian scripts (0.53 to 0.81).Skeleton mapping most reduces survival for Arabic-script languages, Greek, Thai, Armenian, and Hebrew (0.13 to 0.21); Latin-script languages retain 0.44 to 0.49.Source: C Survival under attack · p. 13
  • Under mixed-script detection, characters outside the native set pass untouched; under πintra, mean survival is 0.98 versus 0.84 under πnone.Tamil is the πintra exception, with 0.79 survival.Source: C Survival under attack · p. 13
  • Full skeleton mapping leaves mean survival of 0.19 and widens the between-language range to 17.0×, with Thai lowest.It also deletes default-ignorables, removing the insertion track; without that deletion, every language retains insertion capacity and some capacity from embedded Latin text.Source: C Survival under attack · p. 13

D Stealth

Stealth depends on which check is applied: script labels, identifier restrictions, and font rendering can disagree, and no single stealth score captures them. The keyed detector’s required text length also varies with language and attack; its evaluation uses canonical keying and pinned data and font versions.

D Stealth

  • Skipping token-breaking substitutes keeps every language within 0.001 of unmarked text under πintra, while substituting everywhere flags 0.7% to 80.1% of tokens.Tamil is worst because an Inherited-script homoglyph of its virama is admitted but its Script_Extensions do not list Tamil.Source: D Stealth · p. 14
  • Avoiding the mixed-script detector can increase identifier-filter exposure: under πintra, more bits fall in Restricted compatibility variants.Most of English’s substitution capacity already lies in substitutes marked Restricted by UTS #39.Source: D Stealth · p. 15
  • On a stack modeling a typical reader with recommended-script Noto fonts plus CJK, picking each character’s closest substitute cuts measured ink difference from 0.62 to 0.14 for English and 0.52 to 0.03 for Arabic.Source: D Stealth · p. 15
  • On the typical-reader font stack, eye-invisible substitutes carry 13.2% of English’s cross-script channel and 2.8% of Russian’s, versus 0.2% for Bengali and Arabic.Among three fonts rendering every genuine cross-script twin, 19% to 52% are drawn identically, so visual identity is font-relative.Source: D Stealth · p. 15, Figure 8 · p. 15
  • Font coverage retains up to 0.94 of substitution capacity for the best-covered languages, but 0.75 for English and 0.58 for Bengali.Bengali loses mainly the unsupported sibling script Tirhuta, while Arabic insertion capacity is 1.57 bits per character under OpenType and CoreText but 0 under HarfBuzz’s AAT shaping.Source: D Stealth · pp. 15–16

E Keyed Instrument and Detection Length

  • Canonical keying preserves 98.5% of labels at a cost of 0.36 statistic units, compared with 59.7% preserved by skeleton keying at a cost of 9.9 units.Raw received-character keying preserves 13.9%, drops the detection statistic by 19.8 standard units, and censors required length beyond 256 code points for 25 languages.Source: E Keyed Instrument and Detection Length · p. 16
  • On clean text, median nmin is 2.83 code points, English needs 1.95, and no language is censored; the detection-length span is 2.7×.Source: E Keyed Instrument and Detection Length · p. 16

F Reproducibility

  • The study identifies CLDR 48.2.0 exemplar characters and FLORES+ 4.5, pins Noto and DejaVu fonts by release and file hash, and reports numbers from the 1,012-sentence-per-language devtest split.The 997-sentence dev split is used for the stability check.Source: F Reproducibility · p. 18

Related papers

Loading 2610.36432v1…