Source-linked AI summary
Universal Phone Recognition with a Multilingual Allophone System
Xinjian Li, Siddharth Dalmia, Juncheng Li, Matthew Lee, Patrick Littell, Jiali Yao, Antonios Anastasopoulos, David R. Mortensen, Graham Neubig, Alan W Black, Florian Metze
TL;DR
Multilingual acoustic models often conflate language-dependent phonemes with their language-independent phone realizations, limiting recognition across languages. Allosaurus jointly models phones and phonemes through a shared phone predictor and language-specific allophone layers, improving low-resource multilingual ASR and unseen-language phone recognition. The model supports phone-inventory-based customization for more than 2000 languages, with reported gains on Inuktitut and Tusom.
Problem
Multilingual acoustic models generally ignore phoneme–phone differences, although identical phoneme labels can correspond to different phonetic realizations across languages.
Method
Allosaurus predicts language-independent phones and maps them into language-dependent phoneme distributions through trainable allophone layers.
Results
2.0% phoneme error rate improvement was achieved in low-resource multilingual acoustic modeling, while unseen-language phone recognition improved by 17%.
Takeaways & Limitations
Combining the recognizer with PHOIBLE enables customization for over 2000 languages, including unseen languages such as Inuktitut and Tusom.
Takeaways & Limitations
Customization assumes that a target language’s phone inventory is sufficiently covered by the universal inventory, expressed as Pi ≈ Pi ∩ Puni.
Abstract
from arXiv · showhide
Multilingual models can improve language processing, particularly for low resource situations, by sharing parameters across languages. Multilingual acoustic models, however, generally ignore the difference between phonemes (sounds that can support lexical contrasts in a particular language) and their corresponding phones (the sounds that are actually spoken, which are language independent). This can lead to performance degradation when combining a variety of training languages, as identically annotated phonemes can actually correspond to several different underlying phonetic realizations. In this work, we propose a joint model of both language-independent phone and language-dependent phoneme distributions. In multilingual ASR experiments over 11 languages, we find that this model improves testing performance by 2% phoneme error rate absolute in low-resource conditions. Additionally, because we are explicitly modeling language-independent phones, we can build a (nearly-)universal phone recognizer that, when combined with the PHOIBLE large, manually curated database of phone inventories, can be customized into 2,000 language dependent recognizers. Experiments on two low-resourced indigenous languages, Inuktitut and Tusom, show that our recognizer achieves phone accuracy improvements of more than 17%, moving a step closer to speech recognition for all languages in the world.
1. INTRODUCTION
The paper addresses phoneme–phone mismatches in multilingual acoustic models by introducing Allosaurus, which predicts shared phones before language-specific phonemes. This design also supports universal phone recognition and improves low-resource and unseen-language performance.
- Motivation: Multilingual acoustic models often merge phoneme inventories despite language-specific phone realizations, which can confuse distinctions such as English [p]/[ph] and Mandarin /p//ph/.Phonemes are language-dependent perceptual units, whereas phones are the actual spoken sounds; allophones link phones to particular phonemes.
- Approach: Allosaurus predicts a shared phone distribution and maps it to each language’s phoneme distribution through an allophone layer.The architecture is described as a shared phone inventory followed by language-specific mapping.
- Results: 2.0% phoneme error rate improvement was achieved in low-resource multilingual acoustic modeling.The model incorporates phonology through a trainable allophone layer initialized from an allophone list and optimized end-to-end.
- Universal phone recognition: Allosaurus enables universal phone recognition because it directly models language-independent phones rather than only language-specific phonemes.This allows the recognizer to preserve fine-grained phone distinctions that phoneme-only systems may collapse.
- Universal phone recognition: The phone recognizer can be customized for over 2000 languages using the PHOIBLE phone-inventory database, without training data in those languages.The reported evaluation compares performance on completely unseen testing languages.
2. RELATED WORK
Prior multilingual ASR work mainly uses shared phoneme inventories or separate language-specific phoneme models. Earlier phone-recognition applications also depend heavily on language-specific training data and phonemic transcriptions.
- Multilingual acoustic models: Shared phoneme models combine phonemes from all training languages, whereas private phoneme models classify each language’s phonemes separately using a shared encoder.Both approaches handle phonemes directly and lack an explicit representation of underlying phones.
- Phone recognition: Existing phone-recognition applications have supported alignment, language identification, documentation, and linguistic annotation, but depend heavily on data and phonemic transcriptions for the target language.The paper contrasts these language-dependent applications with an approach that abstracts away from phonemes through allophone transformations.
3. APPROACH
Allosaurus separates universal phone recognition from language-specific phoneme prediction by mapping phone distributions through trainable allophone layers. It regularizes these mappings using linguist annotations and supports universal phone decoding with language inventory constraints.
- Phone-Phoneme Annotation: Each phoneme–phone association is encoded in a binary signature matrix whose entries indicate whether a universal phone realizes a language-specific phoneme.A matrix cell is 1 when the phone is an allophone of the phoneme and 0 otherwise.
- Allophone Layer: Traditional multilingual models either share one phoneme inventory or use separate language-specific phoneme classifiers, without representing underlying phones.The shared approach forms a union of phoneme inventories, whereas the private approach uses separate phoneme distributions for each language.
- Allophone Layer: Allosaurus uses a language-independent encoder and phone predictor followed by a language-dependent allophone layer that produces each language’s phoneme distribution.The encoder predicts over the universal phone inventory Puni, and each allophone layer maps that distribution into the corresponding Qi.
- Allophone Layer: The trainable allophone matrix is initialized from the annotated signature matrix and regularized to discourage divergence during training.The model optimizes the matrix while applying an L2 penalty relative to the original signature matrix.
- Universal Phone Recognition: The model can decode universal phone sequences directly and restrict outputs to phones shared by its universal inventory and a target language’s PHOIBLE inventory.PHOIBLE provides phone inventories for more than 2000 languages and dialects, enabling inventory-based customization without language-specific training data.
4. EXPERIMENTS
Experiments compare Allosaurus with shared and private phoneme baselines on 11 training languages and two unseen indigenous languages. The model improves low-resource multilingual recognition, phone coverage, and unseen-language phone recognition, while qualitative examples show both fine-grained distinctions and remaining errors.
- 4.1. Settings: 11 training languages were evaluated with shared phoneme, private phoneme, and Allosaurus models using phoneme error rate.The experiments also included two unseen languages, Inuktitut and Tusom, evaluated with phone error rate.
- 4.2. Main Results: 2.0% lower error was achieved by Allosaurus than the private phoneme model when training data was limited.With full training data, the private phoneme and Allosaurus models achieved similar results; limiting data reduced accuracy across models.
- 4.3. Universal Phone Recognition Results: 82% mean phone coverage with 12.8% standard deviation was obtained across PHOIBLE languages using the shared 187-phone inventory.The model improved phone coverage by 30% over the baseline formed by merging training-language phoneme inventories as-is.
- 4.3. Universal Phone Recognition Results: Allosaurus distinguished [ph] and [p] for English phoneme /p/ by leveraging knowledge from other languages, although it missed the second /p/ in “people.”The example illustrates both cross-language learning of allophonic distinctions and an observed recognition error.
- 4.3. Universal Phone Recognition Results: 17% average improvement over the shared phoneme baseline was achieved on unseen Inuktitut and Tusom when Allosaurus was combined with PHOIBLE.Phone error rates were 73.1% for Inuktitut and 64.2% for Tusom, compared with more than 90.0% for the shared phoneme baseline.
5. CONCLUSION
Allosaurus models the relationship between phones and phonemes in multilingual acoustic modeling. It improves phone recognition accuracy on unseen languages by 17%.
- Allosaurus models the relationship between phones and phonemes in multilingual acoustic modeling.
- 17% improvement in phone recognition accuracy was achieved on unseen languages.