Source-linked AI summary
VoxLingua107: a Dataset for Spoken Language Recognition
Jörgen Valk, Tanel Alumäe
TL;DR
Spoken language recognition needs large, language-labeled speech corpora, but established resources are costly and do not prioritize broad language coverage. This paper constructs VoxLingua107 from Wikipedia-guided YouTube collection and filtering, then evaluates models trained on it across spoken language identification tasks. The experiments find that the automatically retrieved data supports competitive recognition, with accuracy on NIST LRE07 not far from models trained on large in-domain datasets.
Problem
Spoken language recognition requires large language-labeled speech corpora, while established telephone-speech resources are costly and benchmarks provide limited broad language coverage.
Method
The paper uses TF-IDF phrases from language-specific Wikipedia data to retrieve YouTube videos, then validates and filters the resulting speech data for VoxLingua107.
Results
VoxLingua107 supports spoken language recognition on wild data, and its LRE07 classifier is not far in accuracy from one trained on large amounts of in-domain data.
Takeaways & Limitations
Automatically collected web audio can provide a publicly available resource for training spoken language recognition models across broad language coverage.
Takeaways & Limitations
False positives remain because a video's speech language may not match the language of its title and description, and YouTube terms of service restrict downloading, storing, and distributing videos.
Abstract
from arXiv · showhide
This paper investigates the use of automatically collected web audio data for the task of spoken language recognition. We generate semi-random search phrases from language-specific Wikipedia data that are then used to retrieve videos from YouTube for 107 languages. Speech activity detection and speaker diarization are used to extract segments from the videos that contain speech. Post-filtering is used to remove segments from the database that are likely not in the given language, increasing the proportion of correctly labeled segments to 98%, based on crowd-sourced verification. The size of the resulting training set (VoxLingua107) is 6628 hours (62 hours per language on the average) and it is accompanied by an evaluation set of 1609 verified utterances. We use the data to build language recognition models for several spoken language identification tasks. Experiments show that using the automatically retrieved training data gives competitive results to using hand-labeled proprietary datasets. The dataset is publicly available.
1. INTRODUCTION
Spoken language recognition classifies speech by language and supports applications ranging from call routing to multilingual transcription. Existing benchmarks rely on costly telephone-speech resources and provide limited broad language coverage, motivating automatically collected web audio.
- Spoken language recognition automatically classifies an utterance according to its spoken language.It is used in call routing, multilingual translation and communication, speech transcription, spoken document retrieval, and intelligence and security.
- NIST LRE benchmarks rely mostly on narrowband conversational telephone speech and large amounts of language-specific training data.These resources are typically distributed by the Linguistic Data Consortium.
- $15400 is the stated non-member cost of the 18 LDC datasets used by the standard Kaldi LRE07 recipe.The passage presents this cost as an obstacle for new research groups entering spoken language recognition.
- NIST LREs emphasize challenging fine-grained classification tasks rather than very large language coverage.The passage gives Arabic and Spanish dialect distinctions as examples of such tasks.
- The work investigates whether automatically scraped and labeled YouTube speech can support SLR across wide-band conditions and many languages.Language-specific search phrases are used to retrieve videos, providing a relatively cheap source of somewhat noisy data.
2. SCRAPING FOR SPOKEN LANGUAGE DATA
The paper builds a large spoken-language corpus by combining Wikipedia-derived search phrases with YouTube retrieval, speech segmentation, crowd validation, and data-driven filtering. The resulting process covers 107 languages, yields millions of utterances, and reduces labeling errors while retaining most correctly labeled data.
- Generation of search phrases: Wikipedia-derived three-word phrases are language-filtered with TF-IDF and text-based language identification before YouTube retrieval.Three-word phrases balance vague false positives against too few search matches; Polyglot removes phrases with unknown or mismatched languages.
- Retrieving audio data: YouTube videos matching the generated phrases are further filtered because titles can match the expected language while the video content does not.The pipeline retrieves videos one by one and applies additional text-based filtering to reduce false positives.
- Segmenting audio: 2–20-second utterance-like segments are extracted with LIUM SpkDiarization, removing segments dominated by noise, music, or silence.Most resulting speech segments last 4–10 seconds.
- Scraping results: 107 languages and 14,044 hours of pre-segmentation audio were collected from nearly 78,000 videos, producing about 3.5 million utterances after processing.The majority of videos are 1–10 minutes long, increasing speaker, acoustic-condition, and topic variety.
- Validation using crowd-sourcing: Crowd validation found 85.3% of segments in the expected language, with 97.0% average inter-annotator agreement when indefinite answers were excluded.Among validated segments, 5.8% were in another language, 7.5% contained non-speech, and 1.4% had no definite answer.
3. LANGUAGE IDENTIFICATION EXPERIMENTS
The experiments evaluate x-vector language embedding systems trained on automatically collected data across in-domain, KALAKA-3, and NIST LRE07 conditions. Filtered data helps in some out-of-domain settings, while in-domain cleaning does not improve classification performance.
- Experimental setup: x-vector systems use AugMix augmentation, ResNet34 frame-level features, and an LDA/PLDA backend classifier.The backend extracts utterance embeddings, while the frontend generative classifier is trained on those x-vectors.
- 3.1. In-domain data, 107-language classification: In-domain cleaning does not improve classification performance, and most errors occur between closely related languages.The dataset contains utterances up to 20 seconds; errors between Urdu and Hindi are noted, while English–Welsh substitutions are linked to poor Welsh data quality.
- 3.2. KALAKA-3: Filtered training data improves KALAKA-3 results, especially when the final classifier is trained with the development data for the 6-language tasks.The authors suspect noisy scraped data for several Spanish languages explains why even small amounts of hand-labeled development data help.
- 3.3. LRE07: The LRE07 evaluation uses 14 languages and contrasts scraped-data training with telephone-speech systems under substantially different acoustic conditions.The scraped-data models are trained on resampled 8 kHz audio with centered, length-normalized embeddings and logistic regression classification.
- 3.3. LRE07: On LRE07, out-of-domain ResNet models achieve better average results than older in-domain GMM, phonotactic, and i-vector systems, but modern in-domain systems perform better.The models perform particularly well on short utterances, and cleaned data generally helps except for the 30-second subset.
4. ANALYSIS OF LANGUAGE EMBEDDINGS
The paper constructs language embeddings by averaging language-specific utterance embeddings after dimensionality reduction and normalization. A T-SNE visualization shows that these embeddings largely reflect language-family structure, with deviations attributed to cultural and geographical influences.
- Embedding construction: Language embeddings are formed by averaging language-specific utterance embeddings, reducing them to 250 dimensions with LDA, and applying length normalization.The process starts from x-vector embeddings extracted from all training utterances.
- Embedding analysis: The T-SNE plot shows that learned language embeddings represent language-family structure remarkably well.The visualization also contains deviations from the linguistic hierarchy, which the authors associate with cultural and geographical influences.
5. ALTERNATIVE APPROACHES
The paper situates YouTube scraping among alternative multilingual speech-data sources. Common Voice offers validated volunteer speech but only read speech and lower language coverage, while podcasts offer spontaneous speech but are difficult to browse by language.
- Mozilla Common Voice: Common Voice contains 5671 validated hours in 54 languages, with speaker, prompt, language, and often demographic metadata.Its speech is donated and prompted, providing structured metadata for multilingual speech data.
- Mozilla Common Voice: Common Voice’s main limitation for ‘in the wild’ language recognition is that it contains only read speech.The passage also notes uneven data quantities across languages and lower language coverage than this dataset.
- Podcasts: Podcasts generally provide spontaneous speech, but large podcast directories do not allow browsing by language.Existing language-specific podcast listings are described as domain-specific and limited.
6. AVAILABILITY
VoxLingua107 is publicly available under the Creative Commons Attribution 4.0 International License, while video copyright remains with the original owners. The authors also warn that the dataset’s demographic and linguistic distributions are not globally representative and may introduce unintended biases.
- Availability and licensing: VoxLingua107 is publicly available under the Creative Commons Attribution 4.0 International License.The distributed material is segmented audio-only content, and copyright remains with the original video owners.
- Copyright considerations: The authors characterize use of the audio for language-identification training as transformative fair use, while YouTube’s terms forbid downloading, storing, and distributing videos.The dataset’s segmented audio-only format is described as making monetization of the distributed content extremely difficult.
- Scope and bias: The dataset’s language, accent, dialect, gender, race, and societal-factor distributions are not representative of the global population.The authors warn that training and deployment may therefore introduce unintended biases.
7. CONCLUSION
VoxLingua107 is a YouTube-derived speech dataset suitable for training spoken language recognition models on data from the wild. It also performs competitively with large in-domain training on NIST LRE07, while future work considers combining it with in-domain data and multilingual bottleneck features.
- VoxLingua107 supports training spoken language recognition models for classifying data “from the wild”.
- Experiments with KALAKA-3 showed VoxLingua107 can train both backend feature extractors and frontend classifiers, or only the backend with a frontend trained on limited hand-labeled data.
- On NIST LRE07 evaluation data, a classifier trained with VoxLingua107 was not far in accuracy from a model trained on large amounts of in-domain data.
- Future work will combine VoxLingua107 with in-domain training data and augment filterbank features with multilingual bottleneck features.