Source-linked AI summary
Deep Transformer based Data Augmentation with Subword Units for Morphologically Rich Online ASR
Balázs Tarján, György Szaszák, Tibor Fegyó, Péter Mihajlik
TL;DR
Neural LMs are costly to use in a single online ASR pass, while morphologically rich Hungarian creates large, sparse vocabularies. The paper retokenizes Transformer-generated augmentation text into subwords for BNLM training, reporting improved WER and OOV recognition alongside lower vocabulary and memory requirements.
Problem
Neural LMs are computationally heavy for single-pass ASR, and agglutination in morphologically rich languages creates large, sparse vocabularies that challenge language modeling.
Source: §1 · p. 1Method
The authors pretrain GPT-2 on general text, fine-tune it on Hungarian call-center data, generate augmentation text, and retokenize it into statistical subwords for BNLM training.
Source: §1 · p. 2, §3.3 · p. 3Results
The subword-based approach improved Hungarian call-center ASR WER by more than 10% relative, from 21.9% to 19.6%, and outperformed word-based augmentation in WER and OOV recognition.
Source: §5 · p. 5Takeaways & Limitations
Subword-based augmentation can improve recognition while keeping vocabulary size and memory requirements low in the evaluated Hungarian ASR setting.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Recently Deep Transformer models have proven to be particularly powerful in language modeling tasks for ASR. Their high complexity, however, makes them very difficult to apply in the first (single) pass of an online system. Recent studies showed that a considerable part of the knowledge of neural network Language Models (LM) can be transferred to traditional n-grams by using neural text generation based data augmentation. In our paper, we pre-train a GPT-2 Transformer LM on a general text corpus and fine-tune it on our Hungarian conversational call center ASR task. We show that although data augmentation with Transformer-generated text works well for isolating languages, it causes a vocabulary explosion in a morphologically rich language. Therefore, we propose a new method called subword-based neural text augmentation, where we retokenize the generated text into statistically derived subwords. We compare Morfessor and BPE statistical subword tokenizers and show that both methods can significantly improve the WER while greatly reducing vocabulary size and memory requirements. Finally, we also demonstrate that subword-based neural text augmentation outperforms the word-based approach not only in terms of overall WER but also in recognition of OOV words.
1. INTRODUCTION AND PROBLEM STATEMENT
Neural language models capture information that back-off n-grams miss, but their decoding cost makes them difficult to use in a single online ASR pass. The paper studies Transformer-generated augmentation and subword retokenization for Hungarian, where agglutination creates large, sparse vocabularies.
1. INTRODUCTION AND PROBLEM STATEMENT
- Neural LMs are computationally heavy in decoding, so they are generally used to rescore lattices after a BNLM first pass, adding latency that hampers strict online use.The first-pass network is pruned using short-context syntax, losing longer-context syntactic and nearly all semantic knowledge.Source: §1 · p. 1
- Prior work showed that neural-generated text can train improved single-pass BNLMs, and general-domain pretraining followed by in-domain Transformer fine-tuning improved augmentation effectiveness.The paper contrasts its subword retokenization with Suzuki et al., who compose subwords back into words for the final LM.Source: §1 · pp. 1–2
- Agglutination enlarges vocabularies and makes word forms less frequent, while flexible word order further impairs BNLM estimation and may weaken embedding consistency.The paper notes that corpus size must increase to maintain predictive power as individual forms become less frequent.Source: §1 · p. 1
- The paper proposes retokenizing Transformer-generated text into subwords for Hungarian call-center ASR, reporting better WER and OOV recognition with lower vocabulary and memory requirements.The authors also hypothesize transferability to other morphologically rich languages.Source: §1 · p. 2
2. DATASETS AND MODELS
The study uses Hungarian call-center transcripts and parliamentary text for Transformer pretraining, with 4-gram back-off models and a GPT-2 medium language model. It evaluates on held-out conversations and describes the model’s training configuration.
2.1. Datasets
- 290 hours of recordings were selected for acoustic-model training, while the in-domain LM transcripts contain 3.4M tokens and 100k unique word forms.The general pretraining corpus consists of Hungarian National Assembly parliamentary transcripts.Source: §2.1 · p. 2
- The evaluation set contains about 12 hours of transcribed conversations, with a separate validation set of about 7.5 hours for tokenizer and LM hyperparameter optimization.The 20 hours selected for testing are split into these disjoint sets.Source: §2.1 · p. 2
2.2. Back-off n-gram language models
- The authors use 4-gram BNLMs, selected as optimal for both word and subword models after development-set experiments.Models are trained and interpolated with SRI tools using modified Kneser-Ney discounting.Source: §2.2 · p. 2
2.3. Transformer language model
- The study uses GPT-2 medium with 345M parameters, pretrained for 15 epochs on the general training corpus and fine-tuned for 4 epochs on the in-domain training set using the same hyperparameters.It has 24 decoder-only Transformer blocks, each with 16 attention heads and 1024-dimensional embedding and bottleneck layers. Tokenization used byte-level BPE with 30k vocabulary items, trained on the in-domain training set.Source: §2.3 · p. 2
3. DATA AUGMENTATION
The paper generates large text corpora with pretrained and fine-tuned or in-domain-only Transformers, then uses generated text to train BNLMs. Its proposed extension segments generated and in-domain text into subwords before training subword BNLMs.
3.1. Neural text generation
- Two one-billion-word corpora are generated using a pretrained-and-fine-tuned Transformer and a Transformer trained directly on in-domain text.Generation uses sampled prompts of 1–7 words and randomly varied temperature from 1.0 to 1.5.Source: §3.1 · p. 3
3.2. Word-based data augmentation
- In word-based augmentation, a Transformer generates text for training a BNLM that approximates its short-term dependencies, optionally interpolated with an in-domain BNLM.The process follows the general-domain pretraining and in-domain fine-tuning approach described for Transformer augmentation.Source: §3.2 · p. 3
3.3. The proposed subword-based data augmentation
- The proposed method trains Morfessor or character-level BPE on in-domain text, then segments generated and in-domain corpora before training subword BNLMs.Non-initial subwords receive a ‘+’ tag to preserve word boundaries during ASR decoding; the models can be interpolated.Source: §3.3 · pp. 3–4
4. RESULTS AND DISCUSSION
Subword-based augmentation improved WER while sharply reducing vocabulary and memory requirements, and it recognized more OOV words than word-based augmentation. Word-based Transformer augmentation also improved WER, but its strongest results required very large vocabularies and higher memory use.
4.1. ASR setup
- The online ASR setup used a TDNN-F acoustic model and had typical decoding latency around 250 ms; the first language-model comparison used 100k words and a 1 GB runtime-memory limit.Source: §4.1 · p. 4, §4.2.1 · p. 4
4.2.1. Comparing language modeling approaches
- The pretrained Transformer BNLM reduced WER by 6% relative with interpolation; combining recurrent and Transformer augmentation added another 1% WERR.Source: §4.2.1 · p. 4
4.2.2. Extended word-based augmentation
- Increasing the word-based vocabulary from 100k to 300k reduced WER from 20.6% to 20.2%, and 1M words reduced it to 20.0%; 4 GB enabled 19.7% with 3M words.The authors report that exploiting the full benefit of neural text generation in this setting requires large vocabularies and high memory use.Source: §4.2.2 · p. 4, Figure 3 · p. 4
4.3. Results with subword-based augmentation
- With 40k vocabulary items, subword augmented models covered the test set (0% OOV), whereas the word-based case still had around 1% OOV with 3M words.Source: §4.3 · p. 5
- Subword BNLM + TR-BNLM cuts WER by 5% (20.6% to 19.6%) versus the 100k-word model and 2% relative (20.0% to 19.6%) versus the 3M-word model; 19.6% at 1 GB is roughly the same as 19.7% at 4 GB.Morfessor slightly outperformed BPE, but the difference was not statistically significant.Source: §4.3 · p. 5, Table 3 · p. 5
4.4. OOV recognition analysis
- The subword system recognized about 25% of OOV words with 40k subwords, while the 3M-word augmented model recognized almost 22%.All examined models recognized OOVs with high precision; recall distinguished their performance.Source: §4.4 · p. 5, Figure 4 · p. 5
5. CONCLUSIONS
Subword-based neural text augmentation improved Hungarian call-center ASR while outperforming word-based augmentation on WER and OOV recognition, with low vocabulary size and memory requirements.
5. CONCLUSIONS
- More than 10% relative WER improvement, from 21.9% to 19.6%, was achieved on Hungarian call-center conversations.The approach also outperformed word-based augmentation in WER and OOV recognition while keeping vocabulary size and memory requirements quite low.Source: §5 · p. 5
- Subword-based neural text augmentation outperformed the original word-based technique in WER and OOV recognition while keeping vocabulary size and memory requirements quite low.Source: §5 · p. 5
- The paper reports, to the authors’ knowledge, the first use of GPT-2 Transformer to generate augmentation data for an ASR language model.Source: §5 · p. 5