SinoGlyphBench: A Diagnostic Benchmark for Chinese Glyph-Level Obfuscation in Language-Model Moderation

• Glyph obfuscation can leave harmful Chinese content understandable to people while making it difficult for automated systems, so benchmark variants need a usable human reference.Chinese glyphs can be manipulated through decomposition, radical-level rewriting, and cross-script homoglyph substitution without changing the intended message.Source: §1 · pp. 1–2
• SinoGlyphBench pairs original and obfuscated inputs under a shared four-label schema, annotates semantic anchors, varies perturbation scope, and evaluates text and rendered-image inputs.The benchmark perturbs anchors, surrounding context, or both to distinguish label-critical evidence from general surface variation.Source: §1 · p. 2
• Four-way Acc measures diagnostic agreement, while FNRH, FPRH, and CondAcc separate label-policy disagreement from obfuscation-induced loss of original-correct decisions.Output-based localization identifies failures in visible-form reading, intended-message recovery, and final safety judgment.Source: §1 · p. 2
• 5.0 points: four-way accuracy falls; harmful false negatives rise 6.1 points and harmful false positives 4.7 points; 75.7% of original-correct decisions are retained.Source: §1 · p. 2
• Full-scope perturbations are most damaging; anchor-only perturbations hurt more than background-only, and cross-script substitution is especially difficult in text.Source: §1 · p. 2

Harmful content detection and robustness evaluation

• Functional tests cover hate-speech phenomena and languages, while later resources study implicit hate and known, emerging, or extremist coded expressions.These include HATECHECK (Röttger et al., 2021), MULTILINGUAL HATECHECK (Röttger et al., 2022), LATENT HATRED (ElSherief et al., 2021), SILENT SIGNALS (Kruk et al., 2024), FETCH! (Sasse et al., 2025), and IYKYK (de Kock et al., 2026).Source: §2 · p. 2

Chinese benchmarks and glyph-level focus

• SinoGlyphBench holds underlying content fixed while varying visible character forms across semantic regions, including label-critical words and surrounding text.This distinguishes controlled glyph obfuscation from benchmarks targeting implicit, coded, or phonetic expression.Source: §2 · p. 3
• Chinese resources range from COLD’s offensive-language benchmark and TOXICN’s toxicity taxonomy to span-level extraction, homophonic concealment, and broader harmful-content coverage.These include STATE-TOXICN (Bai et al., 2025b), PCR-TOXICN (Guo et al., 2025), and CHINESEHARM-BENCH (Liu et al., 2025).Source: §2 · p. 3

Obfuscation, glyph variation, and multimodal safety

• Studies of Chinese obfuscation span homophonic and emoji cloaking, naturally occurring phonetic replacement, and broader glyph-, phonetic-, and semantic-form transformations.CNTP, released through TOXIBENCHCN, has a broader transformation taxonomy, whereas SinoGlyphBench focuses on glyph-level changes (Yang et al., 2025; Xiao et al., 2024; Guo et al., 2025).Source: §2 · p. 3
• Non-standard characters can widen human–model reading gaps, and rendered-image evaluations address harmful content that text-only safety tests may miss.Related work includes glyph and adversarial-input studies (Boucher et al., 2022; Davis and Suignard, 2025; Wang et al., 2024a) and multimodal datasets MM-SAFETYBENCH (Liu et al., 2024) and FIGSTEP.Source: §2 · p. 3

3.1 Source Items, Labels, and Panels

• Sources from STATE-TOXICN, CNTP, and PCR-TOXICN are normalized into hostile, abusive, benign, and context-dependent labels, with the first two forming the harmful set.Benign and context-dependent items form the operational non-actionable set; context-dependent cases are retained to test overblocking under uncertainty.Source: §3.1 · pp. 3–4

3.2 Character Obfuscation

• The benchmark uses decomposition into visible components and cross-script substitution with visually similar symbols, retaining only manually curated forms where the source character remains recoverable.These interventions test model-specific mismatches in visual-form handling rather than assuming cross-script variants are universally harder for humans.Source: §3.2 · p. 4

3.3 Semantic Anchors and Scoped Variants

• Semantic anchors mark spans whose recovery matters to the final label, and matched original, anchor-only, background-only, and full variants isolate perturbations inside or outside those spans.Obfuscation density records the fraction of source-character positions changed and is a severity descriptor, not an independent attack family.Source: §3.3 · p. 4

3.4 Text and Image Realizations

• Text probes test non-canonical Unicode and mixed-script strings, while rendered images test visually grounded reading and may introduce OCR-like recognition errors.Each obfuscated condition is compared with a matched original holding applicable factors fixed; Noto Sans is the default image font, with Noto Serif as the font ablation.Source: §3.4 · pp. 4–5

3.5 Dataset and Annotation Characteristics

• 980 broad-panel items and 157 strict-panel items support four perturbation scopes in text or image, with the strict panel balanced between anchor and background regions.After deduplication, the strict panel had 523 unique perturbation records in each region; the broad panel had 3,395 anchor and 3,501 background records.Source: §3.5 · p. 5
• 99.13% of 572 unique perturbed forms were recovered exactly by annotators, including 100.00% of cross-script forms and 99.08% of decomposed forms.Across 1,144 recovery responses, agreement on recovered characters was 98.25%, and every form was recovered correctly by at least one annotator.Source: §3.5 · p. 5
• 88.1% of directional annotator preferences favored cross-script substitution over decomposition across 200 paired comparisons.Cross-script substitution was preferred in 148 comparisons, decomposition in 20, and outcomes were neutral or unresolved in 32; collapsed-response agreement was κ = 0.414.Source: §3.5 · p. 5
• κ = 0.333 was the agreement on audits of all 980 examples; the independent 100-item subset had label agreement κ = 0.462 and anchor-overlap F1 = 0.884.Label disagreements and anchor-boundary discrepancies were reviewed and reconciled under written guidelines.Source: §3.5 · p. 5

4.1 Research Questions

• The three research questions examine matched moderation changes, effects of perturbation scope and type, and intervention impacts and error locations.The error analysis distinguishes reading, recovery, and judgment.Source: §4.1 · p. 5

4.2 Models

• The evaluation covers 12 LLMs and MLLMs across six model families, testing each only on supported modalities and comparing text with images on matched subsets.The model set spans GPT, DeepSeek, Intern, Qwen, Gemini, and Claude.Source: §4.2 · pp. 5–6

4.3 Prompts, Conditions, and Outputs

• Conditions vary model, panel, prompt, modality, rendering, obfuscation type, and scope; each obfuscated input is paired with an original holding other factors fixed.Prompts range from generic to obfuscation-aware and glyph-aware; decoding temperature is set to 0 where available, and structured outputs record reading, recovery, interpretation, and judgment.Source: §4.3 · p. 6

4.4 Metrics

• Metrics include four-way accuracy, harmful false-negative and false-positive rates, paired changes under obfuscation, and conditional accuracy on originally correct decisions.Invalid outputs count as incorrect for accuracy and as false negatives for harmful items, but not as false positives; positive paired changes indicate degradation.Source: §4.4 · p. 6

4.5 Output-Based Failure Localization

• Failure localization assigns each response its first observable category—invalid, reading error, recovery error, judgment error, or correct—using output comparisons rather than inferred internal reasoning.Reading and recovery use edit-similarity thresholds of 0.85 and 0.70, respectively; threshold sensitivity is checked in Table 12.Source: §4.5 · p. 6

5.1 RQ1: Glyph Obfuscation Degrades Moderation

• 24.3% of original-correct decisions are lost under obfuscation (CondAcc 75.7%); four-way accuracy drops 5.0 points, harmful false negatives rise 6.1 points, and harmful false positives rise 4.7 points.These results aggregate 176,916 paired evaluations; item-cluster intervals for all three degradation deltas remain above zero.Source: §5.1 · pp. 6–7, Table 1 · p. 7
• Full-scope obfuscation causes a 31.7% loss in original-correct decisions and a 12.5-point increase in FNRH; anchor-only perturbations hurt accuracy, harmful misses, and CondAcc more than background-only perturbations.Anchor-only and background-only overblocking increases are similar: ∆FPRH of 5.1 and 4.8 points, respectively; the scope ordering recurs across controlled slices, especially under cross-script substitution.Source: §5.1 · p. 7, Figure 3 · p. 7

5.2 RQ2: Cross-Script Difficulty Is Modality-Dependent

• Text CondAcc is 83.6% versus 76.2% for decomposition versus cross-script substitution under generic prompting, and 84.6% versus 77.6% under obfuscation-aware prompting.In generic image settings, CondAcc instead is 78.1% for decomposition and 80.7% for cross-script substitution; image input often improves cross-script retention but can weaken decomposition retention, and the image comparison covers fewer models.Source: §5.2 · pp. 7–8, Table 2 · p. 8, §5.3 · p. 8

5.3 RQ3: Mitigation Is Limited and Setting-Dependent

• Obfuscation-aware prompting raises text CondAcc from 83.6% to 84.6% for decomposition and from 76.2% to 77.6% for cross-script substitution; image-setting gains are also modest.Image CondAcc changes from 78.1% to 79.6% for decomposition and from 80.7% to 81.7% for cross-script substitution, but neither image input nor an explicit warning reliably restores original-correct decisions.Source: §5.3 · p. 8
• Across matched slices, intervention effects cluster near zero with both gains and losses; obfuscation-aware prompting is most consistently positive, while image input can introduce recognition or recovery failures.Glyph-aware examples and serif rendering are exploratory Intern-family ablations, not general repairs; interventions may shift the observable mismatch to another stage.Source: §5.3 · p. 8
• 16.7%–33.8% of outputs show reading mismatches, 9.6%–14.6% recovery mismatches, and 26.5%–38.9% judgment mismatches, with no single stage accounting for all mismatches.These threshold-based categories characterize structured outputs rather than model-internal causes; effective moderation requires reliable reading, recovery, and calibrated judgment.Source: Figure 5 · p. 8, §5.3 · p. 8

• The authors introduce SINOGLYPHBENCH, a matched diagnostic benchmark for Chinese glyph-level obfuscation in moderation.Source: §6 · p. 9
• Across 176,916 paired evaluations, glyph obfuscation causes a 24.3% loss of original-correct decisions and a 5.0-point drop in accuracy, while increasing harmful false negatives and false positives.Source: §6 · p. 9
• Full-scope obfuscation causes the largest degradation; anchor-only obfuscation is more damaging than background-only obfuscation, and cross-script substitution is especially difficult in text settings.Source: §6 · p. 9
• Effective Chinese moderation must read non-canonical glyphs, recover visually supported messages, and maintain calibrated safety judgments under visual variation.Source: §6 · p. 9

Limitations

• The benchmark excludes multi-turn context, screenshots, emojis, handwriting, low-resolution images, layout manipulations, and platform-specific conventions, so it does not represent all real-world evasion strategies.It holds source items fixed while varying perturbation scope, transformation type, modality, and rendering to diagnose robustness to recoverable glyph-level variation.Source: Limitations · p. 9
• SinoGlyphBench inherits its source datasets’ coverage and policy assumptions, while moderation standards vary across platforms, application contexts, dialects, and emerging abuse strategies.Its results should be treated as one component of a broader safety evaluation.Source: Limitations · p. 9
• Model comparisons depend on interface availability, modality support, prompting behavior, and provider safety policies, which can change over time despite controls that reduce confounding.Matched comparisons, deterministic decoding when possible, and paired degradation metrics reduce—but do not eliminate—temporal confounding within the reported evaluation snapshot.Source: Limitations · p. 9
• SinoGlyphBench diagnoses instability under glyph-level obfuscation rather than providing a definitive model leaderboard; future work can extend it to broader languages, policies, and deployment conditions.Source: Limitations · p. 9

Intellectual Property

• The authors will release benchmark materials only as source licenses permit, using derived metadata or construction scripts when source text cannot be redistributed.Users would obtain restricted original data from official sources.Source: Ethical Considerations · p. 9

Annotator Treatment

• Undergraduate annotators received task instructions and content warnings, were paid at agreed workload rates, and were told to treat upsetting examples only as evaluation data.They conducted perturbation-quality studies and annotation audits.Source: Ethical Considerations · p. 9

Privacy and Sensitive Content

• The benchmark collects no new private user information, but its sensitive content is accompanied by warnings and is not reproduced unnecessarily.It derives from existing research datasets and applies controlled glyph-level transformations.Source: Ethical Considerations · p. 9

Controlling Potential Risks

• Glyph obfuscation is dual-use, so the benchmark is research-only and should not enable evasion or treat visual irregularity as evidence of harmfulness.The authors recommend robust reading and calibrated judgment rather than blanket-blocking non-canonical writing, mixed scripts, dialects, creative typography, or noisy text.Source: Ethical Considerations · pp. 9–10

AI Usage

• AI-assisted tools were limited to language polishing and formatting; the authors made and verified dataset construction, annotations, experimental design, analysis, and scientific claims.Source: Ethical Considerations · p. 10

A Benchmark Comparison

• COLD and TOXICN evaluate Chinese offensive or toxic language without systematic obfuscation (Deng et al., 2022; Lu et al., 2023).Source: A Benchmark Comparison · p. 12
• TOXICLOAKCN covers homophonic and emoji-based cloaking (Xiao et al., 2024), while CNTP covers broader visual, phonetic, and semantic text perturbations (Yang et al., 2025).Source: A Benchmark Comparison · p. 12
• SinoGlyphBench focuses on recoverable glyph-level variation and adds semantic-region controls and structured-output failure localization.Table 3 compares its design dimensions with related Chinese harmful-content resources.Source: Table 3 · p. 12, A Benchmark Comparison · p. 12

B Annotation Protocol and Agreement

• Annotators normalized each source label and selected minimal, verbatim semantic anchors, recovering intended Chinese only when visible text and local context supported it.Each record retained the original text; anchors captured one to three short spans that supplied label-critical evidence.Source: B Annotation Protocol and Agreement · pp. 12–13
• 88.1% was the directional cross-script win rate among 168 preference outcomes, with inter-annotator agreement of κ = 0.414.The study compared 200 pairs; 148 were cross-script preferences, 20 decomposition preferences, and 32 neutral or unresolved outcomes.Source: B Annotation Protocol and Agreement · p. 14
• 99.13% exact character recovery across 1,144 responses showed that perturbed forms retained recoverable evidence.Recovery was 100.00% for cross-script forms and 99.08% for decomposed forms; annotators agreed on 98.25% of forms, and at least one annotator recovered every form correctly.Source: B Annotation Protocol and Agreement · p. 14
• Across 980 examples, four-label audit agreement was κ = 0.333; on 100 fully audited items, label agreement was κ = 0.462 and anchor-overlap F1 = 0.884.Final labels and anchors were established through corpus-wide review and reconciliation.Source: B Annotation Protocol and Agreement · p. 14

C.1 Metrics

• The metrics distinguish four-way label accuracy from harmful false-negative and false-positive rates, and measure degradation against matched original inputs.Positive matched deltas indicate degradation; conditioned accuracy tracks decisions correct on the original inputs.Source: §C.1 · pp. 14–15

C.2 Prompts and Output Format

• Generic, obfuscation-aware, and glyph-aware prompts pair with text or image inputs and require JSON fields for visible text, recovered text, interpretation, and judgment.The instructions require literal transcription and recovery only when supported, and direct models to consider visual, phonetic, orthographic, semantic, and layout clues.Source: §C.2 · pp. 15–16
• Evaluators combine visual, phonetic, orthographic, semantic, and layout clues to read diverse obfuscations, recovering only supported text and treating obfuscation itself as non-violative.Possible forms include mixed scripts, homoglyphs, Unicode confusables, split or merged strokes, inserted spacing or symbols, and OCR-like confusions.Source: §C.2 · p. 17
• The output is exactly one JSON object with literal text, recovered text, concise interpretation, and one label, without markdown or extra text.Source: §C.2 · pp. 17–18
• Labels distinguish identity-targeted hostility, other abuse, benign or non-endorsing content, and context-dependent cases only when ambiguity could change the label.Source: §C.2 · p. 17
• Tie-breakers prioritize identity-group hostility over personal abuse, classify clear non-identity abuse as abusive, and reserve context_dependent for ambiguity that affects the label.Quoted or criticized offensive language without endorsement, or content with no recoverable harmful meaning and no clear target or stance, is benign; imperfect recovery alone does not justify context_dependent.Source: §C.2 · pp. 17–18
• For text, evaluators transcribe the exact string inside the INPUT_TEXT envelope; for images, they read rendered text in normal order while preserving meaningful spacing and line order.Source: §C.2 · p. 18

C.3 Evaluation Setup

• Runs use TOML configurations with fixed evaluation settings and deterministic decoding at temperature 0 when the model interface supports it.The configuration specifies the corpus, output and cache directories, evaluation limit, parallelism, endpoint variables, timeout, retries, maximum output length, and temperature.Source: §C.3 · p. 18
• All prompt settings share four JSON fields, ending in a judgment of hostile, abusive, benign, or context_dependent.The fields cover visible-form reading, intended-message recovery, moderation interpretation, and final judgment, keeping outputs comparable across prompt conditions.Source: §C.3 · p. 18
• Rendered images use fixed fonts and layout settings, with wrapping that preserves multi-character obfuscated units.The main-character fonts are Noto Sans CJK SC and Noto Sans, replaced by Noto Serif CJK SC and Noto Serif in the font ablation; symbol and emoji fallback fonts stay fixed. Images use 64 px text, 48 px padding, 300 dpi, black text on white, centered alignment, and 32-character line wrapping.Source: §C.3 · p. 18
• The task grid crosses text and image modalities with decomposition and cross-script substitution under original, anchor-only, background-only, and full variants.Table 5 reports available parameter-count information for the 12 evaluated models; hosted sizes may be unverified and some public counts undisclosed.Source: §C.3 · p. 18, Table 5 · p. 18
• Three prompt settings vary recovery guidance while holding the schema, rendering, wrappers, decoding, and task grid fixed; no hyperparameter search is performed.The generic prompt does not explicitly identify deliberate obfuscation; obfuscation-aware guidance adds visual, phonetic, orthographic, semantic, and layout clues, while glyph-aware guidance adds examples of split-radical and component-based substitutions. Prompt and font variants are diagnostic ablations, not validation-selected settings.Source: §C.3 · pp. 18–19

D Additional Results and Robustness

• Appendix D checks matched-degradation robustness, examines scope, transformation, and model variation, evaluates interventions and output-level failures, and collects per-model results for reference.Source: D Additional Results and Robustness · p. 19

D.1 RQ1: Matched-Degradation Robustness

• All three degradation deltas remain positive across item- and model-clustered intervals; leave-one-model-out condition-row ∆Acc ranges from 5.6 to 6.9 points, versus 6.1 with all models.The 12-model intervals are wider and are described as a coarse sensitivity check, not a population-level estimate; the pooled Table 1 estimate is 5.0 points.Source: Table 7 · p. 19, §D.1 · p. 19
• Harmful misses rose 5.6 points for hostile and 7.5 for abusive items; harmful false positives rose 5.6 points for benign and 3.2 for context-dependent items.The passage cautions that low original accuracy for context-dependent items warrants reporting safety deltas and CondAcc alongside raw accuracy.Source: Table 8 · p. 19, §D.1 · pp. 19–20

D.2 RQ2: Scope, Transformation, and Model Variation

• At full scope, cross-script substitution versus decomposition yields lower text CondAcc (66.7 vs 77.8 generic; 70.8 vs 81.2 obfuscation-aware) but higher image CondAcc (74.1 vs 70.2; 76.3 vs 73.4).Full scope usually has the lowest CondAcc across the prompt and modality conditions.Source: Table 9 · p. 20, §D.2 · p. 20
• After density adjustment, the anchor-minus-background ∆Acc gap shifts only from 2.0 [1.6, 2.5] to 1.8 [1.1, 2.4] points.The mean perturbation-density gap is −0.1 points across 124 matched pairs. Models also vary in scope spans and transformation-type separation, motivating matched diagnostic slices rather than a universal ranking.Source: §D.2 · p. 20, Figure 6 · p. 20

D.3 RQ3: Interventions and Failure Analysis

• Obfuscation-aware prompting raises full-scope text CondAcc from 67.0 to 78.0 for decomposition and 54.8 to 63.0 for cross-script substitution, exceeding glyph-aware prompts by 5.6 and 3.0 points, respectively.Relative to generic prompts, obfuscation-aware gains are larger at full than anchor-only scope: 11.0 versus 3.3 points for decomposition and 8.2 versus 4.8 for cross-script substitution; full scope still has the lowest CondAcc in every text row.Source: §D.3 · pp. 20–21, Table 10 · p. 21
• In images, Serif raises full-scope decomposition CondAcc from 60.9 to 68.7 with a glyph-aware prompt but lowers cross-script CondAcc from 74.3 to 70.4.Font effects depend on transformation and prompt: obfuscation-aware prompting with Serif lowers full-scope cross-script CondAcc from 76.5 to 63.3. Qualitative cases include a successful recovery and failures in visible-form reading, intended-message recovery, or final safety judgment; all four source items were correct on matched originals.Source: Table 10 · p. 21, §D.3 · p. 21, Table 11 · p. 22
• As reading and recovery thresholds tighten, the reading share rises from 16.7% to 33.8%, while judgment share falls from 38.9% to 26.5% and correct share from 34.8% to 25.1%.These categories are diagnostic partitions of model outputs, not human-validated explanations of internal failure causes.Source: §D.3 · p. 21, Table 12 · p. 22

D.4 Results by Model

• Across model-level results, full scope usually has the lowest CondAcc, but losses vary substantially; GPT-5.5 obfuscation-aware prompting raises full-scope text CondAcc from 83.0 to 85.6 for decomposition and 66.7 to 76.8 for cross-script.Image tables qualify the aggregate interaction: cross-script substitution yields higher values than decomposition for some models and scopes; image results cover 10 models and text results 12, making cross-modality aggregates diagnostic.Source: §D.4 · p. 22, Table 14 · p. 23, Table 13 · p. 23