PaperlayerView exact revision on arXiv

Source-linked AI summary

Phonemizing User-Generated Text: A Benchmark, Taxonomy, and Compositional Approach

MinJu Jeon, Younghan Park, Han Sung Park, Jong-Hwan Kim, Dong-Jin Kim, Hoyeon Lee

arXiv:2609.27205v1cs.CLcs.AI

TL;DR

User-generated text contains non-canonical forms whose pronunciations require inference beyond their surface spelling, a setting not covered by existing G2P robustness research. The paper introduces a multilingual benchmark and taxonomy, and evaluates a compositional approach that explicitly models canonical forms. Non-canonical performance gaps reach 66.8 PER points, while the approach reduces errors against matched backbones across three languages.

  • Problem

    G2P robustness for user-generated text remains unexplored, although such text includes non-canonical forms whose intended pronunciations must be inferred from canonical expansions.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces UGTPHON, a three-language benchmark with an inference-grounded taxonomy, and a compositional G2P baseline using canonical-form hints and staged decoding.

    Source: §1 · p. 2

  • Results

    The canonical–non-canonical gap reaches 66.8 PER points, while the compositional baseline reduces non-canonical errors against matched ByT5 and Qwen2.5-0.5B backbones across three languages.

    Source: §1 · p. 2

  • Takeaways & Limitations

    UGTPHON provides a multilingual benchmark and diagnostic framework for studying canonical-form inference in UGT phonemization.

    Source: §5 · p. 9

    UGTPHON covers English, Vietnamese, and Korean but not logographic, abjad, or abugida writing systems, so transfer to those scripts remains open.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Text-to-speech systems increasingly process user-generated text (UGT) such as ppl and imo, whose pronunciation must be inferred from the canonical rather than surface form. We introduce UGTPhon, the first grapheme-to-phoneme (G2P) benchmark for UGT in English, Vietnamese, and Korean, together with an inference-grounded taxonomy for fine-grained diagnosis. Existing G2P models and frontier LLMs exhibit a systematic canonical-to-non-canonical performance gap, reaching up to 66.8 PER points. As a benchmark baseline, we propose a simple compositional G2P approach that incorporates canonical-form evidence through exact-match lookup and staged decoding. Across matched ByT5 and Qwen2.5-0.5B backbones, explicit canonical-form modeling consistently reduces non-canonical G2P errors. The 0.5B variant also performs competitively with much larger few-shot frontier LLMs, highlighting the benefit of explicitly modeling canonical-form inference for UGT phonemization.

1 Introduction

User-generated text includes non-canonical forms whose intended pronunciation depends on recovering a canonical form, a challenge that existing G2P research had not addressed. The paper introduces a multilingual benchmark and inference-based taxonomy, then presents a compositional baseline that reduces non-canonical errors.

1 Introduction

  • Non-canonical forms include abbreviations, orthographic lengthening, phonetic substitutions, and slang, and their pronunciation must be inferred from canonical expansions rather than surface spelling.Examples include “ppl” for “people” and “imo” for “in my opinion.”Source: §1 · p. 1
  • UGTPHON is the first G2P benchmark for UGT across English, Vietnamese, and Korean, with intended canonical forms and gold phoneme sequences for non-canonical words.Its inference-grounded taxonomy supports fine-grained diagnosis.Source: §1 · p. 2
  • The taxonomy groups non-canonical words by required inference: Reconstruct, Repetition, and Pass-through.These categories distinguish recovering dropped characters, collapsing repetitions, and retaining surfaces that already encode pronunciation.Source: §1 · p. 2
  • The compositional baseline uses contextualized input encoding, exact-match retrieval of a canonical-form hint, and staged decoding that emits the canonical form before phonemes.It makes canonical-form inference explicit rather than collapsing it into end-to-end G2P.Source: §1 · p. 2
  • The compositional baseline reduces non-canonical G2P errors relative to matched ByT5 and Qwen2.5-0.5B backbones across English, Vietnamese, and Korean.The 0.5B variant is also competitive with much larger few-shot frontier LLMs.Source: §1 · p. 2
  • The canonical–non-canonical performance gap reaches up to 66.8 PER points, and further UGTPHON training lowers English and Vietnamese non-canonical errors but leaves a large gap.All baselines degrade on non-canonical words.Source: §1 · p. 2

2 UGTPHON

UGTPHON is a three-language benchmark for contextual G2P on user-generated text, with canonical forms and gold phonemes annotated for non-canonical targets. It organizes these targets by the inference phonemization requires and supports aggregate evaluation and category-level diagnosis.

2 UGTPHON

  • UGTPHON contains 18,373 sentences across English (5,029), Vietnamese (10,847), and Korean (2,497), with target-level annotations supporting aggregate evaluation and category-level diagnosis.Each target is represented with its sentence context, surface form, canonical form, and gold IPA phoneme sequence.Source: §2 · p. 2, §2.1 · p. 2

2.1 Task Formulation

  • Given a sentence and non-canonical target, the task is to predict phonemes derived from its intended canonical form, which remains latent at evaluation.Targets are evaluated individually while conditioning on the full sentence; canonical forms may map one-to-one, one-to-many, or many-to-one.Source: §2.1 · pp. 2–3

2.2 Diagnostic Taxonomy of Non-Canonical Words

  • The taxonomy groups non-canonical forms by required inference—Reconstruct, Repetition, and Pass-through—and subdivides them into nine fine-grained categories.Reconstruct recovers omitted content, Repetition collapses repeated units, and Pass-through reads pronunciation-preserving surfaces directly; Korean Reconstruct consists almost entirely of initial-consonant abbreviations.Source: §2.2 · p. 3, Table 1 · p. 3

2.3 Data Construction

  • UGTPHON construction combines source-corpus selection, phoneme annotation, and manual category labeling with second-pass verification and reconciliation of disagreements.Target phonemes are derived from IPA-Dict and verified by native-speaker experts; reported Cohen’s κ is 0.86 for English, 0.87 for Vietnamese, and 0.75 for Korean.Source: §2.3 · pp. 3–4

3 Method

The method makes canonical-form inference explicit within one G2P architecture, combining sentence-context encoding, exact-match retrieval hints, and staged decoding. It supervises normalization and phonemization while preserving the direct phonemization path for canonical words.

3 Method

  • The compositional baseline jointly supervises normalization and phonemization, avoiding the overcorrection of canonical forms associated with a separate normalize-then-G2P pipeline.It combines contextualized input, retrieval-augmented canonical-form prompting, and staged decoding.Source: §3 · p. 4, §3.3 · p. 5

3.1 Contextualized Input Encoding

  • The input marks each target span within its sentence, and each target is processed independently so multiple non-canonical words do not accumulate cross-word decoding errors.The same format applies across training and inference and across ByT5-small and Qwen 2.5-0.5B.Source: §3.1 · pp. 4–5

3.2 Retrieval-Augmented Canonical-Form Inference

  • An exact-match datastore lookup supplies a canonical-form hint when the target is present; otherwise, a <Miss> marker signals retrieval failure for context-based normalization.Optional datastore expansion generates additional candidates from training-split seed pairs and filters out candidates matching standard words in train or dev.Source: §3.2 · p. 5

3.3 Staged Decoding

  • For non-canonical words, staged decoding emits the canonical form before its phonemes; canonical words bypass normalization and emit phonemes directly.The full target sequence is trained end-to-end with cross-entropy, without auxiliary objectives or additional loss terms.Source: §3.3 · p. 5

4 Experiments

The experiments measure canonical and non-canonical G2P performance across three languages, then evaluate whether explicit canonical-form modeling reduces non-canonical errors. Analyses test the model’s components, category-specific behavior, and comparisons with pipelines and frontier LLMs.

4.1 Experimental Setup

  • The evaluation predicts a marked target word from its full sentence and reports PERc, PERnc, and ∆ on the UGTPHON test split, alongside broad G2P, normalization, and LLM baselines.Lexicon-only supervised models also receive PERipa on IPA-Dict; the compared baselines include rule-based, supervised, task-specific, and frontier-LLM systems.Source: §4.1 · p. 6

4.2 Main Results

  • Baseline canonical-to-non-canonical gaps ∆ range from +27.4 to +36.1 on English, +40.4 to +66.8 on Vietnamese, and +13.2 to +45.3 on Korean.The pattern spans rule-based, transformer, and byte-level models; supervised baselines average ∆≈+64 on Vietnamese despite PERc below 8%.Source: §4.2 · p. 6
  • UGTPHON fine-tuning lowers PERnc on English (46.6 → 29.8) and Vietnamese (72.7 → 43.2), but ByT5 retains ∆= +24.6 and +35.5, while Korean PERnc worsens (46.2 → 47.5).Further training alone does not eliminate the gap; the Korean change is associated with its Pass-through-dominated composition.Source: §4.2 · pp. 6–7

4.3 Analysis

  • Retrieval plus staged decoding gives the lowest English and Vietnamese PERnc (22.5 and 26.7) while keeping PERc at or below the no-retrieval level; Korean gains no further improvement.Retrieval alone lowers Vietnamese PERnc from 41.9 to 29.1 and English from 25.5 to 22.5, whereas staged decoding alone worsens English and Korean; without retrieval, canonical reconstructions can be hallucinated.Source: Table 4 · p. 7, §4.3 · p. 7

4.2 Main Results

  • Matched-backbone PERnc falls from 29.8 to 22.5 (ByT5, English), 43.2 to 26.7 (ByT5, Vietnamese), and 47.5 to 32.0 (ByT5, Korean); Qwen2.5-0.5B falls from 14.8 to 14.3, 23.1 to 20.0, and 77.3 to 32.4, respectively.Paired bootstrap confidence intervals exclude zero for all six comparisons; Korean results have higher variance.Source: §4.2 · p. 7

4.3 Analysis

  • PERnc gains concentrate on Reconstruct: Qwen2.5-0.5B A-average falls 33.4 points on English (54.4 → 21.0) and 15.2 on Vietnamese (38.3 → 23.1), with smaller margins on Repetition and Pass-through.The model reaches English A3 PERnc of 11.5 versus 66.3 for the word-level baseline and 36.8 for GPT-4o; Pass-through benefit depends on residual phonemization difficulty, with no gain on near-zero Korean C4.Source: Figure 3 · p. 7, §4.3 · pp. 7–8
  • The model is competitive with or better than evaluated frontier LLMs across all three languages; Qwen2.5-0.5B reaches English PERnc 14.3 versus Gemini-3-flash’s 17.4 at K = 27.On Vietnamese, Qwen2.5-0.5B achieves the lowest overall PERnc of 20.0; on Korean, its 32.4 is below Claude-Sonnet-4.6’s 42.6 at K = 27.Source: §4.3 · p. 8, Figure 4 · p. 8
  • The training-set datastore reaches PERnc 22.7 on English and 26.7 on Vietnamese, within 2.1 and 0.8 points of oracle retrieval, respectively, without test-time canonical-form access.LLM-generated datastore augmentation reduces English PERnc by only 0.2 points and leaves Vietnamese unchanged; downstream TTS-ASR also records the lowest observed WER among predicted-input systems for ByT5 on English (0.359) and Qwen2.5-0.5B on Vietnamese (0.640).Source: §4.3 · p. 8, Table 6 · p. 9

5 Conclusion

The conclusion frames UGT phonemization as requiring surface-to-canonical inference alongside phonemization. It presents UGTPHON as a multilingual diagnostic benchmark and the compositional baseline as a way to make canonical-form inference explicit.

5 Conclusion

  • UGT G2P must compose surface-to-canonical inference with phonemization.The conclusion presents this as the setting UGTPHON addresses.Source: §5 · p. 9
  • UGTPHON is the first G2P benchmark for UGT across English, Vietnamese, and Korean, with an inference-grounded taxonomy.The taxonomy is described as a diagnostic framework for studying canonical-form inference.Source: §5 · p. 9
  • The compositional baseline uses exact-match hints and staged decoding, reducing non-canonical PER against matched ByT5 and Qwen2.5-0.5B backbones in all three languages.Gains are largest on Reconstruct cases, and the 0.5B variant remains competitive with evaluated frontier LLMs.Source: §5 · p. 9

Limitations

The paper identifies limits in TTS-quality evaluation, script coverage, and retrieval maintenance, while documenting UGTPHON’s construction, annotation, and category procedures.

Limitations

  • UGTPHON does not report subjective MOS ratings or evaluate multiple TTS systems, and its coverage excludes logographic, abjad, and abugida scripts.The relationship between PER and TTS quality on UGT and transfer to those scripts remain open questions; emerging vocabulary also requires periodic datastore refresh.Source: Limitations · p. 9

A Creating UGTPHON

  • UGTPHON construction comprises source-corpus selection, phoneme annotation, and category labeling, while preserving raw UGT sentences in the released text field.Punctuation and numbers may be dropped or verbalized during phoneme construction, and unphonemizable items may remain as placeholders.Source: A Creating UGTPHON · p. 12, §A.1 · p. 12

A.1 Source Corpora and Preprocessing

  • Preprocessing preserves sentence boundaries, casing, and context around each target token, with non-lexical elements handled during phoneme construction rather than uniformly removed.Punctuation may be dropped, numbers verbalized, and elements that cannot be phonemized retained as placeholders.Source: §A.1 · p. 12
  • English and Vietnamese source corpora provide canonical forms, whereas Korean canonical forms are annotated from scratch by two native-speaker experts.The English and Vietnamese subsets also include 175 and 385 human-curated augmented UGT samples, respectively.Source: §A.2 · p. 12, §A.1 · p. 12

A.2 Canonical Form Annotation

  • Korean category-specific annotation expands consonant abbreviations, collapses lengthening, treats repetition according to unit type, and retains pass-through surfaces.Consonant-only repetition is rendered in Hangul while preserving intended count; syllable- or word-level repetition retains the surface form.Source: §A.2 · p. 12

A.3 Category Annotation

  • The nine fine-grained categories are grouped into Reconstruct, Repetition, and Pass-through, with unclear category assignments defaulting to the more specific category.Annotators flag ambiguous canonical forms or category assignments; canonical-form ambiguity is resolved using sentence context.Source: §A.3 · pp. 12–13, Table 7 · p. 13

A.4 Phoneme Annotation

Phoneme annotation derives target sequences from canonical forms, with human selection for context-dependent English heteronyms; agreement is substantial across the three languages.

A.4 Phoneme Annotation

  • IPA-Dict supplies phonemes from canonical forms, but an expert annotator selects English heteronym pronunciations using sentence context.IPA-Dict alone is insufficient for context-dependent cases such as “lead.”Source: §A.4 · p. 13
  • Agreement is lower for selected categories, including English A3 and C4 and Vietnamese B2 and C4; Vietnamese C4 zero agreement primarily reflects genuine acronym-rendering ambiguity.Annotator conflicts are resolved through discussion to produce one adjudicated released version.Source: §A.4 · pp. 13–14
  • Cohen’s κ is 0.86 for English, 0.87 for Vietnamese, and 0.75 for Korean phoneme sequences.The lower Korean agreement mainly reflects differing IPA conventions, particularly consonant allophone symbols.Source: §A.4 · p. 13, Table 8 · p. 14

B Content Moderation Analysis

The content-moderation analysis characterizes UGTPHON’s toxicity profile and examines whether canonical-form normalization changes moderation signals.

B Content Moderation Analysis

  • The analysis applies OpenAI’s omni-moderation-2024-09-26 model to every UGTPHON sentence.It scores both raw UGT forms and their canonical-form normalizations across 13 moderation categories.Source: B Content Moderation Analysis · p. 14
  • The analysis compares moderation scores for raw UGT sentences with scores after canonical-form normalization.Both forms are evaluated across the same 13 moderation categories.Source: B Content Moderation Analysis · p. 14
  • The moderation analysis is designed both to characterize corpus toxicity and to measure normalization’s effect on moderation signals.Source: B Content Moderation Analysis · p. 14

B.1 Flagged Rates

Sentence-level flagged rates differ across the three language subsets, and canonical-form normalization changes the aggregate rate only modestly.

B.1 Flagged Rates

  • 12.9% of Korean sentences, 11.3% of English sentences, and 3.7% of Vietnamese sentences are flagged; harassment dominates in English and Korean, while violence dominates in Vietnamese.The reported category pattern reflects source-corpus content rather than language-intrinsic properties.Source: §B.1 · p. 14

B.2 Effect of Normalization

  • 385 of 18,373 sentence flag states change after normalization, while the aggregate flagged rate rises 0.60 percentage points, from 7.02% to 7.63%.Most sentences retain the same moderation outcome under normalization.Source: §B.2 · p. 14
  • Vietnamese flagged rates increase from 3.70% to 4.36%, compared with 11.29% to 12.05% for English and 12.86% to 12.90% for Korean.Source: §B.2 · p. 14

C LLM-Based Datastore Expansion

The datastore is expanded with LLM-generated canonical mappings to extend retrieval coverage for new UGT forms. Generated candidates are filtered to avoid identity pairs, canonical-token conflicts, and test leakage.

C.1 Prompt Template

  • GPT-5.4-mini generates additional plausible canonical-form pairs from training-set seed examples, with augmentation limited to UGT types A1, A2, and A3.The prompt supplies seed pairs and asks for additional examples similar in style but not identical; the other listed categories use training entries only.Source: §C.1 · p. 14

C.2 Filtering Criteria

  • Candidates are discarded when they are identity mappings or when a non-canonical form conflicts with a canonical training or development token.Test data is never inspected during generation or filtering; a candidate matching a test form is retained only if that form already occurs consistently in training.Source: §C.2 · p. 15

C.3 Coverage Statistics

  • The expanded datastore adds 491/220/246 entries for English, Vietnamese, and Korean, respectively, beyond training-only inventories of 1,413/3,674/1,607 keys.These training-only and expanded inventories form the datastore used in the main results.Source: §C.3 · p. 15
  • Training-only datastore coverage is 70.9% (941/1,327) for English, 82.9% (1,785/2,152) for Vietnamese, and 25.3% (183/722) for Korean test targets.Coverage counts target-token occurrences whose non-canonical surface form has an exact-match datastore key.Source: §C.3 · p. 15

D Training and Implementation Details

The experiments train matched ByT5-small and Qwen2.5-0.5B backbones and compare the proposed variants with task-relevant and frontier-LLM baselines. Frontier-LLM few-shot prompts use sentence-level examples and numbered token-to-IPA lines.

D Training and Implementation Details

  • ByT5-small (300M parameters) and Qwen2.5-0.5B (500M parameters) are trained with AdamW for 10 epochs, selecting checkpoints by development PERnc.Training uses learning rate 1×10−4 and batch size 32; inference decodes greedily, producing a canonical form before phonemes for non-canonical tokens.Source: D Training and Implementation Details · p. 15
  • Baselines include rule-based, supervised neural, task-relevant, and frontier-LLM systems, evaluated on the same UGTPHON splits and PER pipeline.The frontier models are queried at 0, 9, and 27 shots; task-relevant baselines include ICKR, R-G2P, MoNoise, and PolyNorm.Source: D Training and Implementation Details · pp. 15–16
  • Three-seed results are stable for English and Vietnamese, while Korean varies more, consistent with its smaller test split and larger unseen-form rate.Paired bootstrap tests find all PERnc intervals exclude zero, though the English Qwen effect is small.Source: D Training and Implementation Details · p. 16, Table 12 · p. 17

E Frontier LLM Prompt Template

  • Few-shot prompts present sentence-level examples in numbered one-token-per-line format, with each model returning IPA on the corresponding numbered line.Nominal K=9 and K=27 correspond to 9/27 examples for English and Vietnamese and 7/21 for Korean, which lacks categories A2 and A3.Source: E Frontier LLM Prompt Template · pp. 16–17

F Downstream TTS-ASR Evaluation

The downstream evaluation tests whether token-level phonemization gains carry over to synthesized speech, using TTS-ASR evaluations in English and Vietnamese. English results favor the ByT5 variant overall, while category-level outcomes vary with the type of UGT.

F Downstream TTS-ASR Evaluation

  • The English evaluation samples 100 stratified test sentences and synthesizes predicted IPA with eSpeak NG before transcription by Whisper large-v3.The setup includes canonical gold-IPA, Word ByT5, GPT-4o 9-shot, and the proposed ByT5 and Qwen backbones.Source: F Downstream TTS-ASR Evaluation · p. 17
  • English WER differences below roughly ±0.02 are unreliable without a bootstrap interval, which this 100-sentence evaluation does not compute.Canonical is not a strict upper bound because eSpeak NG does not always realize length and secondary-stress marks in the gold IPA.Source: F Downstream TTS-ASR Evaluation · p. 17
  • 0.359 WER for Ours (ByT5) is lower than Word ByT5 (0.368, −0.009) and GPT-4o (0.382, −0.023) in the 100-sentence English evaluation.The downstream ranking matches token-level PER across all four trained systems; Ours (Qwen) has WER 0.436 and token-level PER 0.140.Source: F Downstream TTS-ASR Evaluation · p. 17, Table 14 · p. 18
  • Ours (ByT5) wins on 5/9 English UGT categories, with the largest gains on A2 (−0.087 vs. Word ByT5) and A1 (−0.015).The approach does not win on A3, B-type repetition, or C4; the cited passage describes distinct failure-mode explanations for those categories.Source: F Downstream TTS-ASR Evaluation · pp. 17–18, Table 15 · p. 18
  • Qwen2.5-0.5B has the lowest observed WER among Vietnamese predicted-input systems (0.640), followed by ByT5 (0.660) on 297 examples.The Vietnamese results are reported with 95% bootstrap confidence intervals from 10,000 resamples.Source: F Downstream TTS-ASR Evaluation · p. 17, Table 16 · p. 18
Loading 2609.27205v1…