Source-linked AI summary
THA: Weighted Finite-State Text Normalization and Inverse Text Normalization for Khmer
Seanghay Yath
TL;DR
Khmer lacks a maintained open-source toolkit for text normalization and inverse text normalization, and its unsplit words and embedded number words complicate both tasks. Tha uses WFSTs to tag whole lines and filter syllable-internal boundaries; on 2,906 real TTS prompts, 153 of the 158 sentences it rewrites are correct.
Problem
Khmer lacks a maintained open-source tool for both normalization directions, while words are not space-separated and number words occur inside ordinary words.
Source: §1 · p. 1Method
Tha uses WFSTs to tag and classify a whole line in one shortest-path search, then filters token boundaries that fall inside Khmer syllables.
Source: §1 · p. 1, §4.1 · p. 2Results
153 of the 158 sentences Tha rewrote among 2,906 real TTS prompts are correct (96.8%).
Source: §5.2 · p. 5Takeaways & Limitations
Tha provides an open-source toolkit for Khmer TN and ITN, with whole-line tagging and a boundary filter for raw text.
Source: §1 · p. 1, §7 · p. 7The evaluation lacks a large, independently annotated gold standard of written Khmer text and does not compare with neural or LLM-based normalizers.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Text-to-speech needs written text in spoken form, and speech recognition output needs the reverse. For Khmer, neither direction has a maintained open-source tool, and the script makes both harder: words are not separated by spaces, and number words occur inside ordinary words. We present Tha, a Khmer text normalization and inverse text normalization toolkit built from weighted finite-state transducers. It segments and classifies a whole line in one shortest-path search, and a second transducer rejects token boundaries inside a Khmer syllable. On Google's Khmer test suite, Tha agrees with the reference on all 274 cardinals up to one spelling variant, and on 2,906 real TTS prompts, 153 of the 158 sentences it rewrites are correct. Tha is open source under the Apache 2.0 license.
1 Introduction
Khmer TN and ITN lack a maintained open-source tool, while Khmer’s unsplit words and number words embedded inside ordinary words challenge token-by-token normalization. Tha addresses both directions with whole-line WFST tagging and a syllable-boundary filter.
1 Introduction
- Khmer’s lack of word spaces and number words embedded inside ordinary words make token-by-token normalization unsuitable.Existing released Khmer grammars do TN only, expect pre-segmented input, and have been archived since 2021.Source: §1 · p. 1
- Tha handles both TN and ITN as an open-source toolkit, using one shortest-path search to tag a whole line without a word segmenter.A second transducer rejects tokens that start or end inside a Khmer syllable.Source: §1 · p. 1
- Its evaluation includes Google’s test suite, spoken-form TTS prompts, round trips, and an ablation.There is no public gold standard for Khmer text normalization, so the authors use four smaller evaluations that can be rerun from public data.Source: §1 · p. 1
2 Khmer numbers and script
Khmer’s writing system and number vocabulary create ambiguities for normalization: spaces mark phrase boundaries, Unicode permits equivalent character orders, and number words can occur in ordinary words. Number forms and written conventions also vary by context.
2 Khmer numbers and script
- A Khmer syllable consists of a base consonant, optional subscript consonants, then dependent vowel signs and diacritics.Unicode permits multiple character orders for the same visible syllable, although SIL defines a canonical order.Source: §2 · p. 1
- Spaces mark phrase boundaries rather than words, and zero-width spaces are used inconsistently to mark word boundaries.The repetition mark repeats the preceding word, so interpreting it requires knowing where that word begins.Source: §2 · p. 1
- Khmer number words can have competing readings: បាំបួន can mean nine or “five, four,” and short tens omit សិប in speech.The short spoken forms are used before a digit.Source: §2 · p. 1
- Scale words include ពាន់ (10^3), មុឺន (10^4), ែសន (10^5), and លាន (10^6); above a million, ប៊ីលាន and ទីលាន are borrowed, and ពាន់លាន (“thousand million”) is common for 10^9.Some speakers count thousands past ten with ដប់ពាន់ or មួយរយពាន់ instead of using មុឺន and ែសន.Source: §2 · p. 1
- Several number words also occur as ordinary words or names, including ែសន (“very”), លាន (“yard”), and មួយ (“indefinite article”).Source: §2 · p. 1
- Digits, separators, decimals, and dates have variable written conventions; a hyphen can mark a range or a sports score.A hyphen is read ដល់ (“to”) in a range but ទល់ (“against”) in a sports score, requiring context beyond the token.Source: §2 · pp. 1–2
3 Related work
Earlier normalization systems commonly use hand-written WFST grammars, while neural and hybrid alternatives address occasional unacceptable errors. Tha follows established tagger-verbalizer interfaces but addresses Khmer’s segmentation challenges.
3 Related work
- Kestrel and Sparrowhawk use a tokenizer/classifier followed by a verbalizer, with grammars compiled to WFSTs and shortest-path search.NeMo and WeTextProcessing bring this design to many languages; Tha follows NeMo’s tagger and verbalizer interfaces.Source: §3 · p. 2
- Neural sequence-to-sequence normalizers can make occasional unacceptable errors, including reading the wrong number; covering grammars and WFST shallow fusion constrain them.Large language models have also been prompted for text normalization and spoken-to-written conversion.Source: §3 · p. 2
- Google’s Khmer Thrax grammars require CRF-segmented input, while Burmese has FST normalization grammars despite also being written without spaces.Nisaba provides finite-state script normalization for Brahmic scripts.Source: §3 · p. 2
- Prior induced Khmer number grammars cover cardinals only and were not released; Tha uses khmercut only to expand the repetition mark.Tha’s ITN grammars could replace the rule-based converter for tagged spans from KhmerTagger.Source: §3 · p. 2
4 Design
Tha uses weighted finite-state transducers to normalize and inverse-normalize Khmer without first segmenting the line into words. Its grammars handle Khmer-specific boundaries and spoken forms across both directions.
4 Design
- Tha’s classifier, filter, and verbalizer form two whole-line pipelines, with the ITN class grammars composed with a flexible-join transducer before classification.The tagged output is parsed in NeMo’s format, and spoken-token parts are joined with a configurable separator.Source: §4 · p. 2, Figure 1 · p. 3
4.1 Segmenting while classifying
- Semiotic tokens have a flat class weight below 1 while plain text costs 1 per character, favoring longer token matches and lower-weight classes for the same span.Alternating name and semiotic tokens also prevents adjacent semiotic tokens; TN excludes digits from name tokens so every digit is verbalized.Source: §4.1 · p. 2
4.2 Boundary filter
- The boundary filter rejects token edges inside Khmer syllables and constrains class matches against neighboring Khmer, Latin, and numeric text.Its TN rules also guard against dot-thousands readings before scale words; applying constraints to the tagged lattice keeps grammars independent of context.Source: §4.2 · pp. 2–3
4.3 Normalization grammars
- TN cardinals use scale-based grammars for numbers up to 15 digits, while longer numbers and numbers with leading zeros are read digit by digit.Tha covers 17 TN classes, including money, measures, dates, codes, and URLs; cleanup canonicalizes syllable character order before either direction.Source: §4.3 · p. 3
4.4 The repetition mark
- For ៗ, Tha segments up to 120 preceding characters with khmercut and uses rules on the last three words to choose the repetition span.A preceding space ends the phrase; ITN post-processing restores the repetition mark.Source: §4.4 · p. 4, Table 3 · p. 4
4.5 Inverse normalization grammars
- ITN cardinal grammars accept standard and colloquial spoken structures by assembling zero-padded 15-digit blocks and removing leading zeros.ITN covers 13 classes; flexible joins allow spaces or zero-width spaces between characters, but a space marks a phrase boundary except within dates.Source: §4.5 · p. 4
4.6 Implementation
- The grammars compile on first use in 4.7 s for TN and 1.5 s for ITN on a laptop, then are cached for the process.The compiled classifiers have 425,482 TN states and 246,685 ITN states.Source: §4.6 · p. 5, Figure 4 · p. 5
5 Evaluation
Across four rerunnable evaluations, Tha was compared with Google’s Khmer grammar, tested on real TTS prompts, checked through round trips, and examined by ablation and timing. Results show strong agreement and prompt accuracy, with errors tied to spelling, ambiguity, and unsupported notation.
5 Evaluation
- The evaluation used rerunnable public-data tests because no public gold standard for Khmer text normalization exists.The tests covered reference-grammar agreement, spoken-form ITN, round trips, and an ablation, including 274 cardinals and 2,906 TTS prompts.Source: §5 · p. 5
5.1 Agreement with Google’s grammar
- With the spelling បាំពិល mapped, Tha matched all 274 reference cardinal verbalizations; its ITN read all cardinals and times correctly after that spelling was added.Four decimal errors remained for scientific notation, which Tha does not support.Source: Table 4 · p. 5, §5.1 · p. 5
5.2 Inverse normalization of spoken text
- 153 of the 158 sentences Tha rewrote among 2,906 TTS prompts were judged correct; five errors involved misspelled or ambiguous units.The unchanged prompts had no number or only single-digit words; a search found one missed number caused by a typo.Source: §5.2 · p. 5, Table 5 · p. 6
5.4 Ablation
- With every rule enabled, Tha changed 36 of 62,175 lexicon entries, with four flagged as inside a word; removing the boundary filter yielded nine inside-syllable splits.Allowing single digits changed 186 entries, 61 flagged inside a word, and 438 prompt sentences rather than 158; the rule remains on to avoid broken words.Source: §5.4 · p. 6
5.3 Round trips
- Eight of ten classes round-tripped exactly in every tested joining style, and cardinals from 2 to 15 digits reached 100.0% accuracy in every style.Decimals round-tripped in 91.0% of cases; dot-thousands interpretation accounts for the described failures, while telephone forms can be ambiguous.Source: Figure 6 · p. 6, §5.3 · p. 6, Figure 5 · p. 6
5.5 Speed
- Median processing time was 8.6 ms per prompt for ITN and 3.4 ms for TN on one laptop core; time grew linearly with sentence length.The respective 95th percentiles were 17.8 and 6.8 ms.Source: §5.5 · p. 6
6 Discussion and limitations
The discussion identifies ambiguity, spelling variation, boundary cases, style preferences, coverage gaps, and limits in the evaluation. It also describes ablation and timing figures without reporting their outcomes in the supplied passages.
Ambiguity
- Token-level information cannot resolve every ambiguity: Tha uses sentence cues for “ដក” before counts and weights for telephone chunking and currency or weight readings.Language-model rescoring or a neural span tagger could supply missing context while the grammars keep conversion exact.Source: §6 · p. 6
Orthographic variation
- Remaining evaluation errors stem from unrecognized pronunciation spellings, misspelled units, and typos; a large corpus of attested variants would help identify additions to the data-file word lists.Adding variants does not require grammar changes, but the variants must first be found.Source: §6 · p. 6
Evaluation
- Figure 7 tracks changed ITN lexicon entries by number-span position relative to khmercut word boundaries and counts changed TTS prompt sentences, but the supplied caption reports no outcomes.Source: Figure 7 · p. 7
- Figure 8 plots time per sentence against sentence length, with medians computed over 20-character bins; the supplied caption gives no timing values.Source: Figure 8 · p. 7
Orthographic variation
- A number word that forms a whole syllable at another word’s end, such as បី in កបី, cannot be distinguished by the boundary filter alone; the lexicon must resolve it.Source: §6 · p. 7
Style choices
- Output conventions include digit scripts, symbols versus unit words, date formats, and single-digit rendering; Tha exposes options, but downstream users may prefer different choices.Source: §6 · p. 7
Coverage
- ITN omits Roman numerals, letter codes, and whitelist abbreviations, while TN expands abbreviations only from a short whitelist.Source: §6 · p. 7
Evaluation
- The evaluation lacks a large independently annotated gold standard, a measurement of repetition rules, and comparison with neural or LLM normalizers.The authors identify building such a test set as the most useful next step.Source: §6 · p. 7
7 Conclusion
Tha applies whole-line tagging and syllable-boundary filtering to Khmer normalization in both directions. Its main open problems are spelling variation and a proper annotated test set.
7 Conclusion
- Tha brings the tagger and verbalizer design of Sparrowhawk and NeMo to Khmer.Source: §7 · p. 7
- Whole-line tagging removes the need for a word segmenter, while the boundary filter lets ITN run on raw text without breaking ordinary words.Source: §7 · p. 7
- The conclusion identifies spelling variation and a proper annotated test set as the main open problems, and says code, grammars, experiment scripts, and results are available.Source: §7 · p. 7