Source-linked AI summary
Improved Language Identification Through Cross-Lingual Self-Supervised Learning
Andros Tjandra, Diptanu Gon Choudhury, Frank Zhang, Kritika Singh, Alexis Conneau, Alexei Baevski, Assaf Sela, Yatharth Saraf, Michael Auli
TL;DR
Language identification helps route speech to language-specific ASR systems, but prior LID work has often relied on labeled data and wav2vec 2.0 pre-trained only on English. This paper uses multilingual self-supervised pre-training and reports over 89.2% accuracy with just 10 minutes of labeled data per language in a 26-language setup.
Problem
Language identification is needed to route utterances to language-specific ASR systems, while most LID approaches rely on labeled data and prior wav2vec 2.0 work used English-only pre-training.
Source: §1 · p. 1Method
The paper pre-trains speech representations on unlabeled multilingual audio, then fine-tunes them for LID with little labeled data per language.
Source: §1 · p. 1Results
Takeaways & Limitations
Cross-lingually pre-trained LID with little labeled data is a supported route toward speech technology for more languages.
Source: §7 · p. 4
Abstract
from arXiv · showhide
Language identification greatly impacts the success of downstream tasks such as automatic speech recognition. Recently, self-supervised speech representations learned by wav2vec 2.0 have been shown to be very effective for a range of speech tasks. We extend previous self-supervised work on language identification by experimenting with pre-trained models which were learned on real-world unconstrained speech in multiple languages and not just on English. We show that models pre-trained on many languages perform better and enable language identification systems that require very little labeled data to perform well. Results on a 26 languages setup show that with only 10 minutes of labeled data per language, a cross-lingually pre-trained model can achieve over 89.2% accuracy.
1. INTRODUCTION
Language identification routes utterances to language-specific ASR systems, and its accuracy is crucial to avoiding poor downstream recognition. This paper extends prior LID work with multilingual self-supervised pre-training and reports strong performance with little labeled data.
1. INTRODUCTION
- LID identifies an utterance’s language so it can be routed to the corresponding ASR system, and its accuracy is crucial to preventing poor recognition performance.Although multilingual speech recognition has been studied, most systems are trained for one language.Source: §1 · p. 1
- LID approaches have used conventional methods and neural networks, while self-supervised learning uses unlabeled data to learn representations for downstream fine-tuning.Unlabeled data is typically easier to collect than labeled data.Source: §1 · p. 1
- The study extends prior LID work using wav2vec 2.0 pre-trained only on English [22] by considering cross-lingually trained self-supervised models [23].The models are pre-trained on unlabeled speech from many languages and fine-tuned with as little as 10 minutes of labeled data per language.Source: §1 · p. 1
- Multilingual pre-trained models achieve better performance than single-language-pre-trained models, while wav2vec 2.0 with log-mel spectrogram input shows strong LID-classification performance using as little as ten minutes of labeled data per language.The authors compare this performance with models trained on much larger amounts of labeled data.Source: §1 · p. 1
2. LOG-MEL WAV2VEC ARCHITECTURE
The architecture replaces raw waveform input with log-mel spectrograms and processes them through feature and context encoders. It also uses a quantization block to produce representations for the contrastive task.
2. LOG-MEL WAV2VEC ARCHITECTURE
- Log-mel spectrograms replace raw waveforms as input, improving memory and computational efficiency.The input is represented as frames, with 80-dimensional log-mel features given as an example.Source: §2 · p. 1
2.1. Feature encoder
- The feature encoder replaces seven convolutional layers with time-stacking and a linear layer, reducing sequence length to improve Transformer training and inference efficiency.Time-stacking concatenates R consecutive frames into one, reducing sequence length by a factor of 1/R; Transformer time and memory costs are quadratic in sequence length.Source: §2.1 · p. 1
2.2. Context encoder
- The context encoder maps latent speech representations Z to context representations C using linear, normalization, convolutional, and Transformer layers.The convolution layer encodes relative position information.Source: §2.2 · p. 1, Figure 1 · p. 2
2.3. Quantization block
- The quantization block maps feature-encoder output Z to quantized representations Q using product quantization and differentiable Gumbel softmax selection.It applies a linear projection, product quantization across G groups and V codebook entries, concatenation, and another linear layer.Source: §2.3 · p. 2
3. CROSS-LINGUAL PRE-TRAINING
The authors pre-train cross-lingual speech representations on pooled, unlabeled speech from multiple languages and train them using a contrastive objective with a diversity term. The data comes from unconstrained public social-media videos, and language IDs are unavailable for balancing language sampling.
3. CROSS-LINGUAL PRE-TRAINING
- Multilingual pre-training combines unlabeled speech from multiple languages sampled from public social-media videos, with varied styles, noise, disfluencies, accents, and language switching.Because language IDs are unavailable, the authors do not upsample or downsample languages during training.Source: §3 · p. 2
- The contrastive task identifies the true quantized latent among K +1 candidates, with false latents sampled uniformly from masked time-steps.The loss compares context representations with quantized latent speech representations using cosine similarity.Source: §3 · p. 2
- A diversity loss encourages equal use of codebook entries by maximizing entropy, and the final pre-training loss adds it to the masked contrastive loss with weight λ.The final objective is L = Lm + λLd.Source: §3 · p. 2
4. LID FINETUNING
The LID classifier initializes its encoder from a pre-trained log-mel wav2vec model, pools context representations into one vector, and projects that vector to language probabilities.
4. LID FINETUNING
- The classifier initializes its encoder from the latest checkpoint of a pre-trained log-mel wav2vec model and pools context representations into one vector.Several pooling operations are explored.Source: §4 · p. 2
- Self-attention pooling and CLS-token pooling are among the aggregation strategies considered for utterance-level classification.CLS-token pooling prepends a special token and uses its context representation.Source: §4 · pp. 2–3
- A linear layer with L output dimensions maps the pooled representation to language scores, which are normalized with softmax and trained using cross-entropy.The loss uses the predicted probability of the utterance's target language.Source: §4 · p. 3
5. EXPERIMENTAL SETUP
The experiments use in-house public-video audio, compare English-only and multilingual self-supervised pre-training, and fine-tune LID classifiers across labeled-data amounts and 26 languages.
5.1. Dataset
- The in-house dataset uses de-identified public-video audio; XLSR pre-training uses up to 6.3 million unlabeled hours, while the English-only corpus is selected with an in-house LID model.Input features are 80-dimensional log-mel spectrograms, normalized using means and standard deviations from a random pre-training subset.Source: §5.1 · p. 3
5.2. Pre-training setup
- Two same-architecture models are pre-trained: wav2vec 2.0 En on English-only audio and XLSR on all unlabeled audio.Both are trained up to 300,000 updates.Source: §5.2 · p. 3
5.3. Fine-tuning setup
- Fine-tuning covers 26 languages with 10 minutes, 1, 10, or 100 hours of labeled data per language and compares scratch, English-pretrained, and XLSR models.Training uses randomly cropped 6-second chunks; evaluation averages predictions from 6-second windows stepped every 3 seconds.Source: §5.3 · p. 3, Table 1 · p. 4
6. RESULTS
In the 26-language evaluation, cross-lingual pre-training performs strongly with little labeled data. Ablations find useful representations in middle and upper Transformer blocks and show mean pooling performs well against alternatives.
6.1. Language identification for 26 languages
- Over 89.2% accuracy is achieved by XLSR with 10 minutes of labeled data per language in the 26-language setup.The test set contains 3,700 hours; more-language pre-training performs better with little labeled data, while labeled data provides sufficient signal in the high-resource regime.Source: §6.1 · p. 4
6.2. Ablations
- Outputs from Transformer layers 8–24 perform significantly better than layers 2–6 across the tested fine-tuning data amounts.Keeping eight blocks reduces the model from 300 million to 100 million parameters and allows pruning up to two-thirds of the context encoder blocks.Source: Figure 3 · p. 4, §6.2 · p. 4
- Mean pooling works very well compared with the tested alternatives for aggregating XLSR context representations.The comparison includes max, concatenated statistical, CLS-token, and self-attention pooling strategies.Source: Table 2 · p. 4, §6.2 · p. 4
7. CONCLUSION
Self-supervised pre-training improves language identification, especially for low-resource setups, and cross-lingually pre-trained models can perform well with little labeled data. The conclusion also reports model pruning results and identifies inference efficiency as future work.
7. CONCLUSION
- Over 89.2% accuracy is achieved on a 26-language setup with only 10 minutes of labeled data per language using cross-lingually pre-trained LID.The conclusion also reports benefits of cross-lingual pre-training with higher amounts of labeled data.Source: §7 · p. 4
- Up to two-thirds of the pre-trained model can be pruned while achieving the same accuracy.Source: §7 · p. 4
- Self-supervised pre-training is more effective than training LID models solely from labeled data, and cross-lingual representations are particularly effective when labeled data is scarce.The conclusion presents this as important to enabling speech technology for more languages.Source: §7 · p. 4
- Future work may explore improving inference efficiency because pre-trained models remain very large.Source: §7 · p. 4