PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying and Mitigating Korean Jamo-Level Typographical Vulnerabilities in Large Language Models

Seojin Lee, Hwanhee Lee

arXiv:2608.30229v1cs.CL

TL;DR

Korean jamo-level typos expose a robustness gap because they alter syllable-internal structure in ways that ordinary character-level benchmarks and GEC systems do not reliably handle. The paper benchmarks five perturbation types on KMMLU, analyzes their representational signatures, and introduces TACoT, which selectively applies CoT; performance declines with intensity, while typo-aware routing recovers CoT benefits at lower cost.

  • Problem

    Existing robustness benchmarks and correction systems focus mainly on surface character, grammatical, spacing, or spelling errors, leaving Korean intra-syllabic jamo perturbations insufficiently addressed.

  • Method

    The paper applies five keyboard-derived typo types to KMMLU, evaluates four language models, probes hidden representations, and uses a lightweight probe to route likely typos to CoT inference.

  • Results

    Accuracy declines monotonically with perturbation intensity regardless of model scale or specialization; typo inputs also produce a distinct representational shift that supports detection across unseen error types.

  • Takeaways & Limitations

    TACoT provides cost-aware mitigation by selectively invoking CoT for detected typographical anomalies rather than every input.

  • Takeaways & Limitations

    The study relies on automatically generated perturbations, limited Korean robustness resources, and internal hidden states, restricting TACoT to locally served open-weight models rather than API-based deployments.

Abstract

from arXiv · show

Korean introduces an additional typographical perturbation level not captured by ordinary character-level edit models: because syllable blocks are internally composed of sub-character units called jamo, keyboard-level errors can occur within a syllable, either producing a valid but semantically altered character or exposing raw jamo on the surface. Both outcomes disrupt sub-word tokenization and are not reliably corrected by existing grammatical error correction pipelines, leaving LLMs directly exposed to corrupted inputs. To quantify this vulnerability, we apply five jamo-level perturbation types to the KMMLU benchmark and evaluate four language models, finding that accuracy declines monotonically with perturbation intensity and that parameter scaling does not confer robustness against intra-syllabic noise. We further show that typo-corrupted inputs induce a distinct shift in internal representations that is not reducible to ordinary answer incorrectness, and that a simple linear probe trained on these representations detects unseen perturbation types with high AUROC. Motivated by this signal, we propose Typo-Aware Chain-of-Thought (TACoT), which routes inputs to chain-of-thought inference only when the probe detects a likely typo, recovering a substantial portion of the CoT accuracy gain at a fraction of the inference cost.

1 Introduction

Korean typos introduce intra-syllabic jamo perturbations that existing character-level robustness benchmarks and correction pipelines do not capture reliably. The paper benchmarks this vulnerability, identifies a distinct typo signal in LLM representations, and uses it to route typo inputs to CoT inference.

  • Motivation: Korean keyboard errors can occur within syllable blocks because Hangul syllables are assembled from individually typed jamo.This creates a perturbation level beyond visible character edits.
  • Motivation: Intra-syllabic perturbations can create valid but ambiguous syllables or expose standalone jamo, disrupting tokenization and risking semantic alteration during correction.Existing grammatical error correction pipelines are not designed for these perturbations.
  • Approach: Five keyboard-derived perturbation types applied to KMMLU expose LLM vulnerability across four language models.The benchmark targets realistic Korean jamo-level noise.
  • Findings: Accuracy declines monotonically with perturbation intensity, and parameter scaling does not alleviate intra-syllabic brittleness.The controlled benchmark establishes this pattern across diverse models.
  • Mitigation: A linear probe detects a typo-induced representational shift across unseen error types, enabling TACoT to invoke CoT selectively and reduce inference cost on clean inputs.The signal is distinct from ordinary answer incorrectness and supports low-cost routing.

2 Related Work

Prior robustness and Korean correction research largely addresses character-level, grammatical, spacing, or surface spelling errors. The paper positions Korean jamo-level typo handling as a distinct challenge for existing approaches.

  • LLM robustness: Earlier robustness benchmarks model typos primarily as character or prompt-surface edits, with recent work also examining adversarial typos in LLM reasoning.These approaches provide context for extending typo evaluation beyond visible characters.
  • Korean GEC: Korean GEC research covers grammatical, spacing, punctuation, and spelling errors using datasets, taxonomies, and neural correction models.KoGEC formulates Korean correction as a translation-style task.
  • Korean GEC: KoGEC outputs on perturbed KMMLU questions illustrate repair of exposed jamo, unchanged valid-form typos, and fluent but semantically altered rewrites.These representative cases motivate evaluating jamo typos separately from conventional GEC.

3 Korean Typographical Errors

Korean typography combines visible-character errors with an additional intra-syllabic jamo error level. Because standard GEC targets surface forms, it does not reliably correct these structurally different corruptions.

  • Distinct error level: Conventional spelling correction treats errors as visible-character edits, while Korean also permits perturbations inside syllable blocks.Korean syllables contain onset, nucleus, and optional coda jamo.
  • Failure modes: Valid-form perturbations produce well-formed but potentially ambiguous syllables, whereas exposed-jamo perturbations break syllable structure.The two outcomes create different detection and correction challenges.
  • GEC mismatch: Standard Korean GEC is designed for grammaticality, spacing, punctuation, and surface spelling rather than internal jamo composition.Thus, its editing assumptions do not align with these perturbations.
  • GEC mismatch: KoGEC repairs simple exposed-jamo errors but may leave valid-form typos unchanged or rewrite severe corruption into semantically altered text.The results show that pre-correction is not a stable solution for Korean jamo typos.

4 Korean Typographical Perturbation Framework

The framework defines five keyboard-grounded Korean typo types and applies them independently at controlled rates to KMMLU questions. This produces a large benchmark isolating perturbation type and intensity.

  • Benchmark construction: KMMLU supplies 35,030 expert-level four-choice Korean questions, with perturbations applied only to questions while answer choices remain unchanged.This isolates input comprehension effects from output matching artifacts.
  • Typo taxonomy: The taxonomy covers five realistic perturbations: jamo substitution, jongseong deletion, jamo repetition, space deletion, and jamo transposition.Each targets a different aspect of jamo–syllable structure, except space deletion, which changes spacing between eojeol.
  • Typo taxonomy: Jamo substitution can yield another valid syllable, jongseong deletion always yields a well-formed syllable, and repetition or transposition can expose raw jamo.These mechanisms create distinct structural and detectability properties.
  • Typo taxonomy: The taxonomy is grounded in Korean keyboard mechanics and covers the large majority of real Korean input errors observed in practice.The paper verifies this coverage in Appendix C.
  • Perturbation intensity: Five intensity levels perturb 5% through 25% of Korean syllables, with each typo type applied independently from the clean input.Affected syllables are selected randomly, yielding 875,750 perturbed instances.

5 Impact of Korean Typographical Errors

Across four language models, Korean typo intensity consistently reduces KMMLU accuracy, while space deletion remains comparatively benign. Model scale and Korean specialization do not reliably provide robustness to intra-syllabic perturbations.

  • Results: Accuracy declines monotonically across five typo intensity levels for all four evaluated models.The evaluation uses KMMLU in a zero-shot setting across models spanning different sizes and Korean-language orientations.
  • Results: At the highest intensity, accuracy drops by up to 10.0%p for EXAONE-7.8B and 7.0%p for Qwen3-4B relative to clean baselines.These are reported for jamo-level perturbations.
  • Results: EXAONE-7.8B shows a larger Jamo Transposition drop than EXAONE-2.4B, indicating that scale alone does not confer robustness.The comparison directly challenges robustness gains from parameter scaling under intra-syllabic noise.
  • Results: Space Deletion remains close to the clean baseline across models even at the highest intensity.The passage attributes this behavior to Korean subword tokenizers partially recovering eojeol boundaries during segmentation.
  • Implications: The observed vulnerability persists regardless of model scale or Korean specialization, supporting a structural characterization of the problem.The evidence contrasts the robustness behavior across the evaluated models and identifies intra-syllabic brittleness as persistent.

6 Probing Internal Representations of Korean Typos

The analysis tests whether Korean typos create internal states distinct from ordinary answer failures and whether those states can support typo detection. Hidden-state projections show a distinct typo displacement, and a linear probe generalizes to unseen jamo-level perturbation types.

  • 6.2 Typo Shift Is Distinct from Answer Incorrectness: The study distinguishes typo-induced representation shifts from hidden-state patterns associated with clean questions answered incorrectly.This tests whether typo effects are reducible to ordinary answer correctness.
  • 6.1 Locating Typo-Sensitive Layers: A Fisher-style score selects layers where clean and typo representations are far apart while remaining internally compact.The analysis uses last-token hidden states from clean and typo-perturbed HAERAE-GK inputs across transformer layers.
  • 6.2 Typo Shift Is Distinct from Answer Incorrectness: Typos shift hidden states along a typo direction regardless of whether the final answer is correct or wrong.The projection compares clean-correct, clean-wrong, typo-correct, and typo-wrong examples at the Fisher-selected layer.
  • 6.3 Hidden-State Typo Detection: A logistic-regression probe is trained on Fisher-selected hidden states using clean and typo inputs, with four jamo-level types as the detection focus.Space Deletion is excluded because it produces substantially weaker representational separation.
  • 6.3 Hidden-State Typo Detection: Mean held-out-type AUROC ranges from 0.905 to 0.943 across models, showing generalization to unseen jamo-level perturbation patterns.Jamo Transposition is easiest to detect and Jongseong Deletion is hardest, mirroring their relative Fisher separation scores.

7 Mitigating Korean Typos via Internal Representations

The paper uses hidden-state typo detection to route likely corrupted inputs to chain-of-thought inference, aiming to retain CoT’s accuracy benefits at lower cost. TACoT outperforms simpler correction or warning strategies, generalizes beyond KMMLU, and reduces output length while preserving most CoT gains.

  • 7.1 TACoT: Typo-Aware CoT: TACoT extracts a hidden state, estimates typo probability with a probe, and routes above-threshold inputs to CoT while using standard inference otherwise.The probe is trained on clean and typo-perturbed HAERAE-GK inputs, with thresholds selected using Youden’s J statistic.
  • 7.3 Results: GEC pre-correction and Meta-Cognition provide little reliable improvement over Standard inference and can reduce accuracy.GEC lowers Jamo Substitution performance for all four models, while Meta-Cognition degrades Qwen3-4B across all typo types.
  • 7.3 Results: CoT is the most consistently effective mitigation, improving clean and typo-corrupted accuracy across models.For A.X-Light, CoT raises clean accuracy from 49.5% to 57.1% and increases every typo-type accuracy by more than 5.9 percentage points.
  • 7.3 Results: Approximately 37%: TACoT reduces average output length relative to CoT while preserving most of CoT’s accuracy gains across models and typo types.The reduction ranges from 26% for EXAONE-2.4B to 49% for A.X-Light.
  • 7.3 Results: TACoT is strongest on Jamo Repetition and Jamo Transposition, whereas weak representational separation makes Space Deletion less likely to trigger CoT routing.This links the usefulness of routing to the strength of the internal typo signal.
  • 7.4 Robustness of TACoT: On HRM8K, typos still degrade accuracy, CoT remains the strongest mitigation, and TACoT recovers most of its gain beyond multiple-choice KMMLU.HRM8K uses free-form mathematical solutions and a different domain from KMMLU.
  • 7.4 Robustness of TACoT: At the highest intensity, TACoT exceeds a rate-matched random router by up to 2.3 points, supporting probe-based routing rather than merely invoking CoT more often.The comparison uses 7,311 questions all four models answer correctly when clean.

8 Conclusion

The paper establishes Korean typo robustness as a distinct LLM vulnerability arising from intra-syllabic jamo corruption. It identifies a separable hidden-state signal and uses it for TACoT, a cost-aware mitigation that selectively invokes chain-of-thought inference.

  • 8 Conclusion: Korean typos occur within intra-syllabic jamo composition and degrade LLM performance regardless of model scale or language specialization.The paper presents this vulnerability as a linguistic bottleneck and a testbed for input corruption in hidden states.
  • 8 Conclusion: Typo-corrupted inputs induce a distinct internal-representation shift separate from ordinary answer incorrectness, and a linear probe generalizes across unseen error types.TACoT leverages this signal to route selected inputs to chain-of-thought inference.
  • 8 Conclusion: TACoT selectively invokes chain-of-thought inference to provide cost-aware mitigation without sacrificing most of full CoT’s accuracy gains.The conclusion frames Korean typo robustness as a broader setting for studying corrupted inputs through LLM hidden states.

Limitations

The study’s evidence is constrained by synthetic perturbations, limited Korean robustness resources, hidden-state access requirements, and evaluation primarily on four small-to-mid-scale open-weight models.

  • Limitations: The automatically generated perturbations may not fully match the distribution of real user typos despite being grounded in Korean keyboard input and jamo structure.Broader evaluation with naturally occurring typo distributions would strengthen the analysis.
  • Limitations: Korean LLM robustness resources remain relatively limited compared with English, restricting coverage across domains, task formats, and naturally occurring errors.The paper identifies broader benchmark coverage as a future need.
  • Limitations: Hidden-state access limits probing and TACoT to locally served open-weight models and is incompatible with API-based LLM deployment.This constraint prevents direct application of the main probing and mitigation experiments to the most common real-world deployment scenario.
  • Limitations: Probe and TACoT validation covers four 2.4B–7.8B models, while larger Appendix E models are evaluated only for accuracy degradation.The larger models lack corresponding probe or TACoT validation.

Ethics Statement

The paper evaluates synthetic Korean typo perturbations using public benchmark data, reports aggregate model behavior, and proposes a defensive robustness method. Its benchmark construction is based on Korean keyboard mechanics and jamo structure, with additional checks against real error data.

  • Ethics Statement: The study uses publicly available benchmark data and does not involve personal or sensitive information.Results are reported as aggregate model behavior.
  • Ethics Statement: The perturbations are synthetic but designed to evaluate robustness under realistic Korean input noise.The paper states that the goal is to characterize and mitigate, rather than exploit, typo vulnerabilities.
  • Benchmark and Perturbation Design: The benchmark applies five keyboard-derived perturbation types to KMMLU and keeps answer choices unchanged so accuracy loss reflects input comprehension failure.KMMLU contains 35,030 original-Korean four-choice questions across 45 subjects.
  • Reproducibility: The implementation releases adjacency maps and full generation details so the benchmark can be regenerated exactly.Experiments use fixed prompting and decoding configurations, including greedy decoding and bounded token budgets.
  • Benchmark and Perturbation Design: The perturbation procedure selects eligible sites, applies type-specific jamo edits, and recomposes or alters syllables according to the perturbation type and intensity.Edit counts use levels of 5%, 10%, 15%, 20%, and 25%.
  • Naturalness Validation: The taxonomy covers 96.2% of annotated real errors, with jamo-fragmenting Repetition and Transposition comprising 16.8% of keyboard typos.The analysis uses 729,755 annotated error spans from the AI-Hub corpus.

E Additional Results for Typo Robustness

Additional evaluations show that Korean jamo perturbations remain more damaging than ordinary space deletion across API-scale models, with results stable across perturbation seeds. Representation analyses further identify localized typo-sensitive layers and stronger Fisher separation for jamo-level errors than space deletion.

  • Additional robustness results: Jamo-level perturbations consistently cause larger accuracy drops than Space Deletion across Gemini-3.1-Flash-Lite and Qwen3-235B-A22B-2507.The degradation pattern persists across both API-scale models.
  • Seed robustness: The mean accuracy over all 25 typo conditions moves by at most 0.07 points across three perturbation seeds.The largest single-condition spread is 0.57 points for Jongseong Deletion at level 3 on EXAONE-7.8B.
  • Representation analysis: Each model has a localized typo-sensitive layer where clean and typo representations are most clearly separated.Selected layers are 10 for EXAONE-2.4B, 9 for EXAONE-7.8B, 9 for A.X-3.1-Light, and 18 for Qwen3-4B.
  • Representation analysis: Space Deletion has an average Fisher separation score of 0.009, versus 0.052–0.164 for jamo-level types.Jamo Transposition produces the largest separation, while Jongseong Deletion produces the smallest among jamo-level types.

F.2 Detection Robustness Across Intensity

Held-out probes detect unseen Korean jamo perturbations strongly even at the weakest intensity, and detection improves monotonically with intensity. Hidden-state projections show that typo examples shift along a distinct direction orthogonal to ordinary clean incorrectness, supporting typo-aware routing analyses and recovery cases where CoT corrects standard-inference errors.

  • Detection robustness across intensity: AUROC is 0.79–0.85 at the weakest intensity l1 and rises monotonically with perturbation intensity.The result is averaged over four held-out jamo-level typo types and remains strong across all four models.
  • Hidden-state geometry: Typo-correct and typo-wrong examples both shift along an orthogonal typo direction rather than collapsing onto the clean incorrectness axis.This indicates a representational displacement distinct from ordinary answer failure.
  • Probe method: The held-out probe uses balanced clean and typo inputs, training logistic regression on last-token hidden states at the Fisher-selected layer.Training uses non-held-out typo types from HAERAE-GK.
  • Probe method: TACoT selects its routing threshold by maximizing Youden’s J statistic on an 80/20 calibration split without using KMMLU examples.The probe uses standardized hidden states and logistic regression.
  • Case studies: CoT recovers correct answers in representative cases spanning Jamo Substitution, Jongseong Deletion, Jamo Repetition, and Jamo Transposition.Examples include amplifier voltage calculation, unit conversion, micrometer identification, and output-device identification.
Loading 2608.30229v1…