RNN Approaches to Text Normalization: A Challenge

• Text normalization converts written text into how it should be spoken, supporting TTS and ASR applications where errors can substantially degrade perceived quality.Source: §1 · p. 2
• The paper presents a dataset and experiments with several neural architectures to test whether RNNs can learn this mapping directly.Source: §1 · p. 2
• Some RNNs achieve very good overall accuracy, but their occasional normalizations can convey completely wrong messages in real speech applications.Source: §1 · p. 2
• A simple FST-based filter used with the RNN can ameliorate these risky errors.Source: §1 · p. 2
• Pure RNN normalization has not yet avoided the risky errors, leaving open whether an RNN-only solution can do so.Source: §1 · p. 2

• Normalization maps non-standard tokens such as 6ft and 150lb into ordinary spoken words across semiotic classes including measures, currencies, dates, and times.Source: §2 · p. 2
• Training data must be created because normalized written text has no natural economic source, unlike many translation or speech-recognition datasets.Source: §2 · p. 3
• Interesting normalization cases are sparse, and evaluation requires exact distinctions because plausible alternatives can still convey the wrong number, date, or unit.Source: §2 · p. 3
• Unlike machine translation, text normalization receives credit mainly for sparse interesting cases rather than for tokens that map to themselves.Source: §2 · p. 3
• RNNs can score well by leaving most tokens unchanged yet remain unusable when they make conspicuous substitutions such as reading £900 as nine hundred Euros.Source: §2 · p. 3
• The experiments include a source-channel character-to-word model with an LSTM language model and a sequence-to-sequence LSTM architecture.Source: §2 · pp. 3–4

• WFSTs provided a unifying model for many text-normalization problems, while later work treated normalization as language modeling and applied machine learning to TTS.Source: §3 · p. 4
• Social-media normalization often targets unusual spellings, whereas TTS normalization emphasizes expanding digits, measures, dates, and related expressions.Source: §3 · p. 4
• A prior neural social-media system achieved an F1 score of 81.75%, but comparable or better TTS accuracies are still argued insufficient for the intended application.Source: §3 · p. 4

• The corpus contains 1.1 billion English words and 290 million Russian words from Wikipedia.Source: §4 · p. 4
• Sentences were processed by Kestrel to produce verbalizations, using tokenization, semiotic-class classification, and mostly hand-built Thrax finite-state rules.Source: §4 · p. 4
• The data were split into 90 training files, 5 development files, and 5 test files, with roughly 92K English and 93K Russian test tokens evaluated.Source: §4 · p. 4
• Manual analysis estimated error rates of approximately 0.1% for English and 2.1% for Russian, with Russian errors concentrated in years read as cardinals instead of ordinals.Source: §4 · p. 4
• Because 99.5% of English and 98.9% of Russian test tokens appeared in training, the authors warn that models may memorize rather than generalize.Source: §4 · pp. 4–5

5 Experiment 1: Text normalization using LSTMs

• The channel maps character sequences to word outputs, while the language model selects contextually appropriate verbalizations from those channel hypotheses.The decomposition separates enumerating possible token normalizations from choosing the appropriate one in context.Source: §5 · p. 5, §5.1 · p. 5, Table 1 · p. 5

5.1 LSTM architecture

• The channel uses bidirectional shallow or deep LSTMs with CTC, while the language model uses a standard recurrent architecture with projection, feedback, and hierarchical softmax.The shallow and deep channel models use two or three forward and backward hidden layers, respectively, with 256 nodes per layer.Source: §5.1 · pp. 5–6, Figure 2 · p. 6, Figure 3 · p. 6

5.2 Decoding

• Decoding prunes channel hypotheses, retains the five best at each position, and combines channel and language-model probabilities with an LM weighting factor of 1.The resulting representation approximates channel–language-model WFST composition using a confusion-network-like structure.Source: §5.2 · pp. 6–7

5.3 Results

• About 99% English and 98% Russian overall accuracy is driven largely by correctly leaving ordinary tokens unchanged or mapping punctuation to silence, while performance breaks down on sparse semiotic classes such as TIME.Deep models are generally better than shallow models, although this is not universal, such as for MONEY in English.Source: §5.3 · p. 7, Table 3 · p. 8
• Many serious errors arise from the channel offering plausible but incorrect verbalizations, including wrong numbers, units, or silence, which the language model cannot reliably disambiguate.For the English deep model, oracle channel accuracy is 0.998 versus overall accuracy of 0.993; the authors attribute about 29% of errors to the channel not offering the correct choice.Source: §5.3 · p. 7
• Unseen-token accuracy remains 82.9% in English and 83.8% in Russian, below seen-token accuracy but indicating generalization beyond memorization.The models also correctly handle complicated examples, including dates in both languages.Source: §5.3 · pp. 7, 9

6 Experiment 2: Attention-based RNN sequence-to-sequence models

• The model uses a character-reading bidirectional LSTM with attention and a word-sequence decoder, trained on token-centered context rather than whole sentences.The architecture includes four bidirectional reader layers, 256 attention units, and a two-layer decoder.Source: §6 · p. 9

6.1 Results

• The attention-based sequence-to-sequence model generally outperformed the earlier LSTM model, supporting pure sequence-to-sequence transduction as a viable alternative.The comparison is reported in Table 6, whose classes follow the paper’s earlier evaluation setup.Source: §6.1 · p. 9, Table 6 · p. 10
• For English unseen cases, 90.6% were correctly produced, compared with 99.8% of seen cases; for Russian, 86.7% of unseen cases were correct versus 99.4% of seen cases.Complicated previously unseen examples included Russian dates such as 9 июня 1966 г. and 17.04.1750, which were correctly read aloud.Source: §6.1 · p. 10

6.2 Results on “reasonable”-sized data sets.

• The authors present about 10 million words of Wikipedia-style text as a plausible hand-labeling scale for developing a new-language system with a small native-speaker team over a few weeks.Source: §6.2 · p. 10
• With smaller training sets, overall performance remained close to that of the larger-data system, while non-overlapped tokens reached 95.0% accuracy in English and 93.5% in Russian.The test data overlapped the training data in 96.9% of English tokens and 95.5% of Russian tokens.Source: §6.2 · p. 10, Table 8 · p. 11
• Despite strong accuracy with reasonable-sized data, the attention-based system still produced “silly” errors and was insufficient by itself for a TTS text-normalization component.Reported errors included incorrect readings of numeric dates, although some complex unseen measures, dates, and money amounts were handled correctly.Source: §6.2 · p. 11, Table 7 · p. 11

7 Finite-state filters

• A roughly 150-line Thrax grammar covers MEASURE and MONEY expressions using language-independent currency rules, language-specific lexical lists, and a learned number FST.The authors characterize the resources as requiring much less effort than a complete hand-built normalization grammar.Source: §7 · pp. 11–12
• The filter composes an input token with a grammar, projects allowable verbalizations into an output lattice, and uses completion rewards to guide RNN decoding.If composition fails, decoding proceeds with the RNN alone; if it succeeds, the lattice restricts possible outputs.Source: §7 · p. 12
• The FST filter substantially improves MEASURE and MONEY accuracy while leaving other categories unaffected in the main English evaluation.The authors retain unaffected-category scores to show that the filters do not overapply.Source: §7 · p. 12, Table 9 · p. 13
• On a MEASURE-MONEY-rich English set, filtering again improved MONEY and MEASURE accuracy without changing other categories, with failures concentrated where the grammar did not match.The filter did not help examples involving unsupported units, number forms, or numberless measure expressions.Source: §7 · p. 13, Table 10 · p. 13
• For previously unseen English expressions, filtering produced no MONEY improvement but substantially improved MEASURE performance.Most failed MONEY improvements occurred when the filter did not match the input.Source: §7 · p. 13, Table 12 · p. 14
• For Russian, apparent regressions often reflected ill-formed source data, while 67 cases were unequivocally improved by correcting RNN category errors.The filter preserved written grammatical errors rather than forcing corrections, which the authors regard as appropriate for TTS.Source: §7 · pp. 13–15, Table 13 · p. 14

8 Discussion and the challenge

• The evidence suggests that aligned text alone and a general neural model may be insufficient, while combining an RNN with an FST is probably viable but remains unproven for RNNs alone.The authors make the data publicly available so others can test alternative models and objectives.Source: §8 · p. 15
• RNNs learn complex mappings and often perform well, but can produce random outputs such as reading the wrong number or substituting hours for gigabytes.The FST filter works by disallowing these random mappings.Source: §8 · p. 15
• Overall accuracies above 99% can conceal serious failures on interesting categories, where substitutions such as reading 2012 as “two twelve” are unacceptable.Text normalization faces a higher accuracy bar than applications where occasional failures on unusual constructions may be tolerable.Source: §8 · p. 15
• Traditional normalization prioritizes semiotic classes such as dates because TTS systems must reliably read forms like Sep 12, 2014.This motivates evaluating difficult categories rather than relying primarily on overall scores.Source: §8 · p. 15
• The proposed challenge asks researchers to train pure deep-learning systems for English and Russian that outperform the reported systems on difficult categories, not merely overall accuracy.A successful system should avoid errors such as reading £ as dollars or euros.Source: §8 · p. 16