Source-linked AI summary
CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition
Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, Gerhard Rigoll
TL;DR
Large end-to-end ASR models need more training data, and extracting aligned utterances from unsegmented or unlabeled audio is a data-preparation challenge. The paper proposes CTC-segmentation to obtain utterance alignments and trains a hybrid CTC/attention model; its best model achieves 12.8% WER on Tuda-DE, 1.6% WER absolute better than conventional hybrid DNN/HMM ASR.
Problem
End-to-end ASR models require more training data, while large audio files can contain extra, missing, or altered speech that complicates utterance alignment.
Source: §1 · p. 2, §5 · p. 10Method
CTC-segmentation uses a CTC-based end-to-end network to extract utterance segments with time-wise alignments from speech audio.
Source: §3.1 · p. 4, §5 · p. 11Results
12.8% WER on the Tuda-DE test set, an absolute improvement of 1.6% WER over conventional hybrid DNN/HMM ASR.
Source: §5 · p. 11Takeaways & Limitations
CTC-segmentation alignments were more accurate than Viterbi- or DTW-based approaches on hand-labeled TEDlium v2 development and test data.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Recent end-to-end Automatic Speech Recognition (ASR) systems demonstrated the ability to outperform conventional hybrid DNN/ HMM ASR. Aside from architectural improvements in those systems, those models grew in terms of depth, parameters and model capacity. However, these models also require more training data to achieve comparable performance. In this work, we combine freely available corpora for German speech recognition, including yet unlabeled speech data, to a big dataset of over $1700$h of speech data. For data preparation, we propose a two-stage approach that uses an ASR model pre-trained with Connectionist Temporal Classification (CTC) to boot-strap more training data from unsegmented or unlabeled training data. Utterances are then extracted from label probabilities obtained from the network trained with CTC to determine segment alignments. With this training data, we trained a hybrid CTC/attention Transformer model that achieves $12.8\%$ WER on the Tuda-DE test set, surpassing the previous baseline of $14.4\%$ of conventional hybrid DNN/HMM ASR.
1 Introduction
End-to-end ASR simplifies training and decoding but requires more training data, while large audio files still need utterance-level audio–text alignment. The paper proposes CTC-segmentation to extract aligned utterances from large recordings and reports alignments closer to manual segments than other automated tools.
1 Introduction
- End-to-end ASR directly infers token sequences from speech and can use raw utterance text for training; hybrid CTC/attention systems combine CTC and attention-based approaches.CTC is described as a loss function for training end-to-end ASR architectures.Source: §1 · p. 2
- Large-file segmentation is difficult when transcripts mismatch recordings, while forced-alignment tools require strategies for out-of-vocabulary words and scaling and can require tedious manual checks.Public-domain recordings may include extra speech or omit or add sentences relative to the transcript.Source: §1 · p. 2
- The proposed method uses a CTC-based end-to-end ASR network to extract correctly aligned utterances from large audio files.The introduction also notes additional German orthography cleanup steps.Source: §1 · p. 2
- CTC-segmentation alignments were observed to correspond more closely to manually segmented utterances than alignments from other automated segmentation tools.The method is proposed as a scalable way to extract utterance segments from speech corpora.Source: §1 · p. 2
- The work extends a German open-source corpus collection with Librivox and CommonVoice and ports it to ESPnet.The introduction also identifies a refinement of the existing Kaldi recipe as a contribution.Source: §1 · p. 3
2 Related Work
Prior German ASR work combined freely available corpora and used both conventional DNN/HMM and hybrid CTC/attention approaches. Existing segmentation tools use HMM-based alignment or dynamic time warping; this work proposes CTC-based segmentation.
2.1 Speech Recognition for German
- The German corpora discussed include Tuda-DE, SWC, M-AILABS, CommonVoice, and Librivox, with differing recording and alignment sources.SWC includes Wikipedia article recordings, while Librivox supplies volunteer readings of public-domain books.Source: §2.1 · p. 3
- Milde et al. combined freely available German speech corpora, while this work uses an ESPnet hybrid CTC/attention Transformer architecture.The architecture differs from the conventional DNN/HMM approach mainly used by Milde et al..Source: §2.1 · p. 3
2.2 Alignment and Segmentation Methods
- MAUS and Gentle estimate phone-wise alignments using HMMs, whereas Aeneas aligns synthetic and actual audio using dynamic time warping.These approaches estimate utterance start and end times within audio files.Source: §2.2 · p. 4
- The authors propose CTC-based segmentation, noting that CTC was used as a loss and previously proposed for segmentation, but that no CTC-based speech-audio segmentation tool was known to them.The proposal targets utterance alignment in recordings with unknown extra speech at the beginning or end.Source: §2.2 · p. 4
3 Methodology
CTC-segmentation aligns utterance text to audio using frame-level CTC probabilities, then scores segments for mismatches. Text preparation normalizes corpus transcripts to Tuda-DE conventions and addresses older German spelling.
3.1 CTC-Segmentation of Utterances
- A pretrained CTC network supplies frame-level character probabilities, which dynamic programming uses to align transcript characters with audio frames.The network is trained beforehand on already aligned data.Source: §3.1 · p. 4
- The alignment recurrence chooses between a blank transition and consuming the next character, while allowing the transcription to start at an arbitrary audio position.The transition cost for staying at the first character is set to zero.Source: §3.1 · p. 4
- Backtracking from the most probable position of the final transcript character determines each audio frame’s character alignment.The highest-probability transitions define the alignment.Source: §3.1 · pp. 4–5
- The segment confidence score is the lowest mean frame probability across length-L parts, penalizing mismatches such as a missing word.Frames aligned to an utterance are split into parts before their mean probabilities are calculated.Source: §3.1 · p. 5
- A near-constant audio-to-text position ratio restricts each character’s search to a window, reducing alignment complexity from O(M·N) to O(M).The window is centered at the audio position proportional to the character position.Source: §3.1 · p. 5
3.2 Data cleaning and text preparation
- Transcript cleaning removes punctuation and unsupported letters, expands abbreviations, units, and numbers into spoken forms, and matches Tuda-DE orthography.Number expansion accounts for context-dependent spoken forms and uses neighboring words to determine case.Source: §3.2 · p. 5
- For older German spellings in Librivox text, a self-updating lookup table replaces letters using known correctly spelled German words.The old spelling is associated with Librivox texts that are at least 70 years old.Source: §3.2 · pp. 5–6
4 Evaluation and Results
The evaluation finds that CTC-segmentation produces accurate utterance alignments, including when recordings contain extra speech at their edges. On Tuda-DE, the best reported test WER is 12.8% using a corpus selection that includes custom-aligned SWC and Librivox.
4.1 Alignment evaluation
- Against Gentle, CTC-segmentation was more accurate at segment starts and slightly more accurate at ends; most predictions from both tools deviated by less than one second.The figure plots deviation density in 60 bins.Source: §4.1 · p. 6, Figure 1 · p. 7
- With extra speech added to recording starts and ends, MAUS and Aeneas accuracy dropped sharply, while Gentle and CTC-segmentation retained their alignment abilities.The added speech segments were randomly sampled from 10 to 30 seconds.Source: §4.1 · p. 8, Table 2 · p. 8
- CTC-segmentation alignments were closer to ground truth than the tested DTW- and HMM-based methods across architectures and training sets.The evaluation used manually labeled TEDlium v2 dev and test data and compared mean deviation, standard deviation, and the share within 0.5 seconds.Source: §4.1 · pp. 6, 8, Table 1 · p. 7
4.2 Composition of German Corpora for Training
- The training-corpus selections contained 649h, 968h, and 1460h; the largest combined Tuda-DE and CommonVoice with CTC-segmented SWC and Librivox.The corpora collection spans up to 1772h; SWC was realigned and cleaned, and Common Voice text numbers were normalized to spoken forms.Source: §4.2 · pp. 8–9
4.3 ASR configuration
- All ASR experiments used a 12-layer encoder and 6-layer decoder Transformer hybrid CTC/attention model, trained for 23 epochs without data augmentation.Decoding used beam search of size 16 and RNNLM language models.Source: §4.3 · p. 9
4.4 Discussion of Results
- 12.8% test WER followed replacing SWC and M-AILABS with custom-aligned SWC and Librivox.The comparison is between dataset combinations evaluated on the Tuda-DE test set.Source: §4.4 · p. 10, Table 4 · p. 10
- The larger RNNLM was associated with a 2% WER improvement and better recognition of German compounds.The authors attribute the improvement to better detection and prediction of German words and grammar forms.Source: §4.4 · p. 10
- 4.3% WER was the biggest improvement, with the 1460h corpus selection; increased training data and better CTC-segmentation alignments contributed.The smaller CommonVoice addition of 319h improved test WER by an absolute 0.1%.Source: §4.4 · p. 10
5 Conclusion
The paper proposes CTC-segmentation to extract time-aligned utterances from unlabeled audio using a CTC-based network. It reports more accurate alignments than Viterbi- or DTW-based approaches and a 12.8% WER on Tuda-DE, 1.6% absolute better than the conventional hybrid DNN/HMM system.
5 Conclusion
- End-to-end ASR models require more training data as their depth, parameter count, and capacity grow.Source: §5 · p. 10
- CTC-segmentation uses a CTC-based end-to-end network to extract utterance segments with exact time-wise alignments.Source: §5 · p. 11
- 12.8% WER on Tuda-DE test was 1.6% absolute better than the conventional hybrid DNN/HMM system; alignments were more accurate than Viterbi- or DTW-based approaches.Source: §5 · p. 11