Source-linked AI summary
Streaming End-to-End Bilingual ASR Systems with Joint Language Identification
Surabhi Punjabi, Harish Arsikere, Zeynab Raeesy, Chander Chandak, Nikhil Bhave, Ankish Bansal, Markus Müller, Sergio Murillo, Ariya Rastrow, Sri Garimella, Roland Maas, Mat Hans, Athanasios Mouchtaris, Siegfried Kunzmann
TL;DR
Multilingual ASR needs language information at runtime, yet users may switch languages without specifying one in advance. The paper uses acoustic LID embeddings and joint language targets in streaming RNN-T systems. English-Spanish performance is comparable to monolingual baselines, while English-Hindi results vary by language.
Problem
Multilingual ASR often assumes language is known beforehand, although users may switch languages and need not want to specify it before each interaction.
Source: §1 · p. 1Method
The paper uses streaming RNN-T models that take pretrained acoustic LID embeddings and jointly predict spoken content and language.
Source: §1 · p. 1Results
English-Spanish performance is comparable to monolingual ASR and acoustic LID baselines; for Hindi, ASR surpasses baseline performance while LID almost matches it.
Source: §5 · p. 4Takeaways & Limitations
For dynamic language switching, joint ASR-LID offers a promising simplification with compute savings over running multiple systems.
Source: §1 · p. 1, §5 · p. 4English ASR and LID metrics degrade for English-Hindi, which the authors attribute to code switching and dual Latin and Devanagari word representations.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Multilingual ASR technology simplifies model training and deployment, but its accuracy is known to depend on the availability of language information at runtime. Since language identity is seldom known beforehand in real-world scenarios, it must be inferred on-the-fly with minimum latency. Furthermore, in voice-activated smart assistant systems, language identity is also required for downstream processing of ASR output. In this paper, we introduce streaming, end-to-end, bilingual systems that perform both ASR and language identification (LID) using the recurrent neural network transducer (RNN-T) architecture. On the input side, embeddings from pretrained acoustic-only LID classifiers are used to guide RNN-T training and inference, while on the output side, language targets are jointly modeled with ASR targets. The proposed method is applied to two language pairs: English-Spanish as spoken in the United States, and English-Hindi as spoken in India. Experiments show that for English-Spanish, the bilingual joint ASR-LID architecture matches monolingual ASR and acoustic-only LID accuracies. For the more challenging (owing to within-utterance code switching) case of English-Hindi, English ASR and LID metrics show degradation. Overall, in scenarios where users switch dynamically between languages, the proposed architecture offers a promising simplification over running multiple monolingual ASR models and an LID classifier in parallel.
1. Introduction
The introduction motivates multilingual ASR for seamless use across languages, while noting that runtime language information is often unavailable when users switch languages. It proposes a streaming RNN-T that combines acoustic LID guidance with joint speech and language prediction, and studies English-Spanish and English-Hindi.
1. Introduction
- Multilingual ASR supports cross-lingual data sharing for under-resourced languages and enables one model across languages, simplifying multilingual deployment.The introduction also frames the goal as a seamless experience without users specifying a language before each interaction.Source: §1 · p. 1
- Prior multilingual end-to-end studies often assume language is known, but preselecting it can frustrate users who switch languages dynamically.A common alternative runs parallel ASRs with a language detector; multilingual ASR could simplify that architecture.Source: §1 · p. 1
- The proposed streaming RNN-T uses acoustic LID embeddings and predicts both spoken content and language.This targets seamless multilingual use without requiring users to specify a language before each interaction.Source: §1 · p. 1
- Joint ASR-LID training appends language tags to ground-truth transcripts and includes those tags in the model vocabulary.The authors note that this approach had been explored for LAS but, to their knowledge, not RNN-T.Source: §1 · p. 1
- The study feeds frame-level acoustic LID embeddings into RNN-T training and inference, tests joint-network biasing, and examines English-Hindi and English-Spanish.The language pairs were chosen as unrelated languages motivated by real-world bilingual-market usage patterns.Source: §1 · p. 1
2. Methods
The methods adapt RNN-T for multilingual ASR by supplying language information at different network stages and by jointly predicting language and transcription targets. Experiments compare against monolingual, pooled multilingual, and acoustic-only LID baselines.
2. Methods
- The RNN-T encoder maps acoustic features to hidden representations, and the joint network combines encoder and decoder outputs to produce logits for the next output symbol.The next symbol is either blank or an ASR target.Source: §2 · p. 2
- Language information can supplement acoustic input at the encoder, and the study also tests supplying it to the joint network where higher-level acoustic and lexical information is combined.The model extends the output symbol space with language targets for joint ASR-LID training.Source: §2 · p. 2
2.1. Baselines
- ASR is compared with monolingual RNN-T and pooled multilingual RNN-T baselines, while LID is compared with an acoustic-only classifier.The pooled baseline uses training data from the languages of interest without language adaptation.Source: §2.1 · p. 2
2.2. Multilingual ASR using oracle language identity
- Oracle language identity is supplied as a one-hot vector to the encoder, joint network, or both, with the three configurations evaluated separately.Because runtime language identity is unknown in dynamic multilingual settings, one-hot language encoding is treated as an oracle approach.Source: §2.2 · p. 2
2.3. Multilingual ASR using inferred language identity
- Inferred language identity comes from frame-wise embeddings of a pretrained acoustic LID classifier, supplied to the encoder, joint network, or both.The paper notes that early-utterance guidance may be less valuable because only a few audio frames may be available to detect language.Source: §2.3 · p. 2
2.4. Multilingual joint ASR-LID modeling
- The joint ASR-LID model adds language tags to the RNN-T output space and appends them to transcripts so language predictions follow accumulated acoustic and textual evidence.Joint models may use input-side language embeddings; during decoding, an emission penalty controls when language tags are valid beam candidates, while language is predicted from unscaled posteriors after the last audio frame.Source: §2.4 · pp. 2–3
3. Experimental setup
The experiments cover English-Spanish in the United States and English-Hindi in India. English-Hindi is especially challenging because of within-utterance code switching and script variation.
Experimental languages
- Experiments use English-Spanish as spoken in the United States and English-Hindi as spoken in India.The language pairs are denoted en-us and esp-us, and en-in and hi-in, respectively.Source: §3 · p. 3
English-Hindi characteristics
- English-Hindi has more within-utterance code switching, including common English words in colloquial Hindi and named entities from other Indian languages.These properties make English-Hindi a challenging language pair.Source: §3 · p. 3
- English-India utterances are transcribed in Latin script, while Hindi-India utterances use both Latin and Devanagari scripts.Hindi words use Devanagari, while English and other-language words use Latin; shared named entities can be output in either script.Source: §3 · p. 3
4. Results and discussion
Results vary by language pair and method: joint ASR-LID is closest to baseline performance for English-Spanish, while English-Hindi remains more challenging. Combining joint training with inferred LID embeddings narrows some gaps, and penalizing language-tag emissions improves ASR metrics without significantly affecting LID accuracy.
4. Results and discussion
- The experiments report ASR WERR relative to monolingual ASR systems and LID accuracy relative to acoustic LID classifiers.Table 3 summarizes the results across the experiments.Source: §4 · p. 3, Table 3 · p. 4
4.1. Results using data pooling and oracle language inputs
- Oracle language inputs yielded healthy WERRs, especially for hi-in and esp-us, while simple pooled training generally degraded performance, particularly for English-Hindi.The authors identify potential script-based confusions between common words represented in Latin and Devanagari scripts.Source: §4.1 · p. 3
4.2. Results using inferred language embeddings
- With inferred LID embeddings, A3J was almost on par with baseline performance for English-Spanish, but English-Hindi showed a significant drop.The authors suggest Indic words in English utterances and Hindi code switching may reduce embedding discrimination; frame-level embeddings may also be susceptible to within-utterance language fluctuations.Source: §4.2 · pp. 3–4
4.3. Results for joint ASR-LID training
- Vanilla joint training increased WERs across languages, with deletions 2–6 times higher than in monolingual counterparts; penalizing language-tag emissions improved ASR metrics.The extreme penalty, β = 1, performed best in WERR, without significantly affecting LID accuracy because LID uses utterance-final posteriors.Source: §4.3 · p. 4
- A5J was comparable to or slightly worse than baseline systems for English-Spanish; for English-Hindi, hi-in had 4.9% WERR with slightly worse LID accuracy, while en-in retained larger baseline gaps.A5J performed best in general; the en-in gaps were much smaller than with A3J and A4.Source: §4.3 · p. 4, Table 3 · p. 4
- Replacing projected embeddings with acoustic-LID posteriors yielded marginal gains, while filtering wake-word-only utterances produced useful improvements in general.The wake-word filtering addresses possible confusion between accent and language.Source: §4.3 · p. 4
5. Conclusions
The paper presents streaming bilingual RNN-T systems that jointly perform speech recognition and language identification, using acoustic LID information and language targets. English-Spanish results are comparable to monolingual ASR and acoustic LID baselines, while English-Hindi results vary by language; the authors see joint training as promising given its modeling simplification and compute savings.
5. Conclusions
- The proposed streaming, end-to-end bilingual RNN-T systems jointly perform speech recognition and language identification, using acoustic LID embeddings and jointly modeled language targets.The authors state that language information to the joint network performs best and that ASR can improve without significantly affecting LID accuracy by penalizing language-tag emissions and using utterance-final posteriors.Source: §5 · p. 4
- English-Spanish performance was comparable to monolingual ASR and acoustic LID systems; for English-Hindi, English metrics degraded while Hindi ASR surpassed its baseline and LID nearly matched it.The authors attribute the English degradation to code switching and dual Latin and Devanagari word representations.Source: §5 · p. 4
- The authors consider joint ASR-LID training promising given its modeling simplification and compute savings, and describe the techniques as language agnostic and scalable to multiple languages.Source: §5 · p. 4