Source-linked AI summary
Code-Switching Detection with Data-Augmented Acoustic and Language Models
Emre Yılmaz, Henk van den Heuvel, David A. van Leeuwen
TL;DR
Frisian-Dutch code-switching ASR faces scarce training speech and text, limiting reliable recognition of words and their languages. This paper evaluates switch detection using augmented acoustic and language models and analyzes the resulting errors. The augmented systems improve detection over the baseline, while the best-performing system still overestimates switches through very short erroneous segments.
Problem
Code-switching ASR lacks training speech and text needed to reliably recognize words and their languages, while code-switching text is scarce in written resources.
Source: §1 · p. 1, §3.2 · p. 2Method
The study evaluates code-switch detection using acoustically augmented ASR and a language model enriched with generated, transcribed, and translated code-switching text.
Source: §1 · p. 1Results
Data-augmented ASR improves code-switch detection over the baseline, and adding generated code-switching text yields the best EER, 8.6% (5.5%).
Source: §5.2 · p. 4, §6 · p. 4Takeaways & Limitations
The best-performing system overestimates switches, mostly through numerous very short erroneous switches.
Source: §6 · p. 4, §5.3 · p. 4The best-performing system contains more segments shorter than 10 seconds than forced alignment, especially segments shorter than 2 seconds, indicating false alarms from very short erroneous switches.
Source: §5.3 · p. 4
Abstract
from arXiv · showhide
In this paper, we investigate the code-switching detection performance of a code-switching (CS) automatic speech recognition (ASR) system with data-augmented acoustic and language models. We focus on the recognition of Frisian-Dutch radio broadcasts where one of the mixed languages, namely Frisian, is under-resourced. Recently, we have explored how the acoustic modeling (AM) can benefit from monolingual speech data belonging to the high-resourced mixed language. For this purpose, we have trained state-of-the-art AMs on a significantly increased amount of CS speech by applying automatic transcription and monolingual Dutch speech. Moreover, we have improved the language model (LM) by creating CS text in various ways including text generation using recurrent LMs trained on existing CS text. Motivated by the significantly improved CS ASR performance, we delve into the CS detection performance of the same ASR system in this work by reporting CS detection accuracies together with a detailed detection error analysis.
1. Introduction
Code-switching detection is relevant to multilingual ASR, but reliable Frisian-Dutch recognition is constrained by scarce training speech and text. This work examines detection performance using augmented acoustic and language models and analyzes the errors in hypothesized switches.
1. Introduction
- Detecting language switches in speech is a relevant application when designing ASR systems for multilingual societies.The paper situates its work in the FAME! project’s Frisian-Dutch bilingual ASR system.Source: §1 · p. 1
- Reliable code-switching ASR is challenged by limited training speech and text, especially for recognizing both the words and their languages.Frisian and Dutch share orthographic similarities and vocabulary.Source: §1 · p. 1
- Prior work expanded acoustic training with automatically transcribed code-switching speech and explored adding Dutch speech to the increased training set.The acoustic models included a time-delay and recurrent-layer architecture.Source: §1 · p. 1
- The language model was enriched with generated code-switching text, translated Dutch text retaining some Dutch words, and transcriptions from automatic speech transcription.Recurrent language models trained on limited code-switching text generated larger amounts of text.Source: §1 · p. 1
- The study evaluates code-switching detection with augmented acoustic and language models and analyzes switch counts, segment durations, language-tag accuracy, and common word-tag confusions.The focus is Frisian-Dutch speech, where Frisian is under-resourced.Source: §1 · p. 1
2. Frisian-Dutch Radio Broadcast Database
The FAME! database contains Frisian-Dutch radio broadcasts, and a representative subset was selected to include switching cases and speaker diversity across a broad time span.
2. Frisian-Dutch Radio Broadcast Database
- The FAME! database contains bilingual radio broadcasts in Frisian and Dutch.The project uses a representative subset of archive recordings that includes language switching and speaker diversity, spanning 1966–2015.Source: §2 · p. 1
- The selected archive recordings include inter-sentential, intra-sentential, and intra-word code-switching alongside Frisian-only and Dutch-only utterances.The database supports a spoken-document retrieval system for Omrop Fryslân’s bilingual archive.Source: §2 · p. 1
3. Data Augmentation
The data-augmentation section addresses scarcity in both acoustic and language-model training. It describes using high-resource Dutch and generated or collected code-switching text to expand those training resources.
3.1. Acoustic Modeling
- Earlier automatic annotation of raw bilingual broadcasts enabled more acoustic training data and improved code-switching recognition.The data was used for acoustic training together with manually annotated reference data.Source: §3.1 · p. 2
- Adding monolingual Dutch speech is explored as an acoustic-modeling resource after earlier additions of Dutch data had harmed recognition in the low-resource mixed language.Flemish speech is also included to investigate its contribution to accent robustness for West Frisian-accented Dutch.Source: §3.1 · p. 2
3.2. Language Modeling
- Code-switching language modeling is limited by sparse written training material, with training-speech transcripts providing only a small share of the bilingual text.The section motivates enriching the code-switching language model with additional text.Source: §3.2 · p. 2
- The code-switching language model is enriched with recurrent-LM-generated text, automatic speech transcriptions, and Dutch text translated into code-switching text.The perplexity improvements from each component are reported in.Source: §3.2 · p. 2
4. Experimental Setup
The experiments use FAME! and other speech and text resources, implement acoustic and language models with defined training configurations, and evaluate recognition and duration-based code-switching detection.
4.1.1. Spoken Material
- FAME! supplies 8.5 hours of Frisian and 3 hours of Dutch training speech; its development and test sets each contain 1 hour of Frisian and 20 minutes of Dutch speech.Automatically annotated Frisian Broadcast speech totals 125.5 hours after very short segments are removed; the Dutch and Flemish CGN components contain 442.5 and 307.5 hours, respectively.Source: §4.1.1 · p. 2
4.1.2. Written Material
- Baseline language models use 37M Frisian and 8.8M Dutch words, while the final text corpus after augmentation contains 107M words.The baseline code-switching text comes from FAME! training transcripts and contains 140k words; augmentation includes generated, translated, and automatic-transcription text.Source: §4.1.2 · p. 2
4.2. Implementation Details
- The acoustic setup uses an LF-MMI TDNN-LSTM model with MFCC features and speaker-adaptation i-vectors, while baseline recognition and lattice rescoring use a bilingual 3-gram LM and an RNN-LM.Table 2 reports WER on FAME! development and test sets by acoustic-model training data, identified using Table 1.Source: §4.2 · p. 2, Table 2 · p. 3
- The bilingual lexicon contains 110k Frisian and Dutch words, with additional pronunciations learned for training-data words missing from the initial lexicons using language-specific G2P models.Phonetisaurus is used for grapheme-to-phoneme transcription creation.Source: §4.2 · pp. 2–3
4.3. Recognition and CS Detection Experiments
- Recognition is evaluated with WER across Frisian-only, Dutch-only, and mixed segments, with and without language tags; detection uses interpolated monolingual LMs to vary language priors and a time-based accuracy metric summarized by EER.Figure 1 presents code-switching detection performance on the FAME! development and test sets.Source: §4.3 · p. 3, Figure 1 · p. 3
5. Results and Discussion
Data augmentation improves ASR and code-switching detection, with generated CS text yielding the best reported detection accuracy. The analysis also finds overestimated switches, especially very short ones, and recurring word–language-tag confusions.
5.1. ASR Results
- Data augmentation consistently improves ASR performance, but language-tagged results retain a 2%-2.5% absolute WER gap versus results without tags.The baseline total WER is 37.8% without language tags and 40.3% with them.Source: §5.1 · p. 3
- Dutch-only and mixed segments lose more recognition accuracy from language-tag errors than Frisian segments, consistent with the CS language model assigning Frisian tags more often to similar words.The CS language model used in the experiments yielded the lowest perplexity on development text.Source: §5.1 · p. 3
5.2. CS Detection Results
- 8.6% (5.5%) EER is the best reported CS detection result, achieved by enriching the CS language model with generated text.The baseline EER is 14.4% (8.7%); adding automatically annotated in-domain and monolingual Dutch speech reduces it to 9.8% (6.3%).Source: §5.2 · pp. 3–4
5.3. Discussion
- All systems overestimate language switches, while the system with augmented acoustic and language models hypothesizes more switches than other systems on both datasets.Enriching the CS language model with generated CS text consistently increases hypothesized switch counts.Source: §5.3 · p. 4, Figure 2 · p. 4
- The best-performing system has more monolingual segments under 10 seconds than forced alignment, especially segments under 2 seconds, indicating short-switch false alarms despite its duration-based detection performance.Smoothing language-tag switches to reduce these false alarms remains future work.Source: §5.3 · p. 4, Figure 3 · p. 4
- Frequent word–language-tag confusions are mostly short fillers that are orthographically identical or similar and share meaning in Frisian and Dutch.These errors induce the 2%-2.5% absolute WER difference between results with and without language tags, but are less concerning for spoken document retrieval.Source: §5.3 · p. 4, Table 3 · p. 4
6. Conclusions
The data-augmented Frisian-Dutch ASR system outperforms the baseline in code-switching detection, while analysis identifies short erroneous switches and orthographically identical fillers as key error sources.
6. Conclusions
- The data-augmented system achieves superior code-switching detection, but overestimates switches through short errors, with orthographically identical fillers the main source of detection errors.Without language tags, those filler-word errors are invisible.Source: §6 · p. 4