PaperlayerView exact revision on arXiv

Source-linked AI summary

ELARSpeech: A 30K-Hour Dataset Across 675 Underrepresented Languages for Long-Tail Multilingual Speech Technologies

Sangmin Lee, Yejin Jeon, David Ifeoluwa Adelani

arXiv:2610.19388v1cs.CLcs.AIcs.SD

TL;DR

Long-tail multilingual speech remains underserved by resources and evaluations with broad language coverage, public accessibility, and diverse recordings. ELARSpeech brings together curated labeled and unlabeled archival speech and evaluates ASR, TTS, pseudo-labeling, and representation transfer. The experiments show utility for underrepresented-language ASR and TTS while identifying transfer limitations in existing multilingual models.

  • Problem

    Existing resources and evaluations underrepresent the linguistic long tail, and some broad-coverage datasets are inaccessible or restrictive, while evaluations often focus on higher-resource languages.

    Source: §1 · p. 1

  • Method

    ELARSpeech curates 30,202 hours of labeled and unlabeled speech from the Endangered Languages Archive across 675 underrepresented languages.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    ELARSpeech supports ASR and TTS adaptation across diverse underrepresented languages, pseudo-labeling further improves performance, and conventional multilingual advances do not necessarily transfer to the linguistic long tail.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ELARSpeech serves as a resource for advancing speech technologies in underrepresented languages and as a foundation for evaluating models across the linguistic long tail.

    Source: §6 · p. 8

    Archival language and recording metadata may retain inaccuracies; MOS evaluation relies primarily on non-native listeners and therefore emphasizes general perceptual qualities rather than linguistic accuracy or native-like pronunciation and prosody.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Massively multilingual speech modeling has made remarkable progress in recent years, yet existing training resources and benchmarks still underrepresent much of the linguistic long tail. To address this gap, we introduce ELARSpeech, a large-scale multilingual speech resource comprising 30,202 hours of speech across 675 underrepresented languages, curated from the Endangered Languages Archive. Combining human-annotated and unlabeled recordings, the corpus spans diverse recording conditions, domains, and language families. Beyond corpus construction, our experiments show how ELARSpeech supports extending ASR and TTS capabilities to diverse underrepresented languages, with further benefits from pseudo-labeling unlabeled data. At the same time, our speech representation analysis reveals remaining limitations in transferring existing multilingual models to the linguistic long tail. Together, ELARSpeech provides both a resource and a testbed for extending multilingual speech technology further into the linguistic long tail. We publicly release the dataset, all trained ASR and TTS models, and the complete data construction pipeline to facilitate research on long-tail multilingual speech technologies.

1 Introduction

The linguistic long tail remains underserved by resources that combine broad language coverage, public accessibility, diverse recordings, and suitable evaluation. ELARSpeech addresses these gaps with a large corpus and experiments spanning ASR, TTS, pseudo-labeling, and representation analysis.

1 Introduction

  • Public access remains a challenge: datasets spanning thousands of languages may have restrictive licenses or be unavailable, leaving speech data for many long-tail languages difficult to access.This limits reproducible research further into the linguistic long tail.Source: §1 · p. 1
  • Existing multilingual corpora often draw heavily on a single source, leaving spontaneous and conversational speech from diverse real-world settings less represented.Examples include parliamentary proceedings and Bible readings.Source: §1 · p. 1
  • Long-tail evaluation remains insufficient: existing multilingual evaluations tend to focus on higher-resource languages, making it difficult to assess representation gains for underrepresented languages.The paper identifies a lack of suitable evaluation resources for the linguistic long tail.Source: §1 · pp. 1–2
  • ELARSpeech supports ASR and TTS adaptation across diverse underrepresented languages, pseudo-labeling further improves performance, and representation analysis finds that conventional multilingual advances do not necessarily transfer to the long tail.The experiments examine adaptation, pseudo-labeled scaling, and speech representation.Source: §1 · p. 2
  • 30,202 hours of speech across 675 underrepresented languages make ELARSpeech a large corpus combining human-labeled and unlabeled recordings.It extends long-tail resources while preserving linguistic, acoustic, and domain diversity in real-world field recordings.Source: §1 · p. 2

2 Related Work

Related work describes growing labeled and unlabeled multilingual speech resources, while noting that broad coverage does not eliminate gaps in recording diversity.

2.1 Labeled Multilingual Speech Corpora

  • Labeled multilingual corpora range from crowdsourced or controlled read speech to over 65K hours of transcribed speech, while low-resource datasets remain scarce when broad coverage must be combined with diverse environments.Examples include CommonVoice, FLEURS, WorldSpeech, Omnilingual ASR, and OpenBibleTTS.Source: §2.1 · p. 2

2.2 Unlabeled Multilingual Speech Corpora

  • Unlabeled resources such as MMS and XEUS offer broad language coverage, but further diversification of recording domains and conditions with real-world speech remains possible.MMS's underlying corpus is not publicly distributed; XEUS reconstructs it and adds crawled resources.Source: §2.2 · p. 2

3 ELARSpeech

ELARSpeech is curated from archival language-documentation recordings to expand long-tail language coverage and recording diversity. Separate pipelines construct its labeled and unlabeled subsets, followed by common normalization and filtering.

3.1 Corpus Overview

  • 30,202 hours across 675 languages span 78 language families, diverse continents, communicative domains, and recording conditions; 259 languages are unsupported by existing multilingual models and 53 are absent from ISO 639-3.The corpus has 2,791 labeled hours across 443 languages and 27,412 unlabeled hours across 665 languages.Source: §3.1 · pp. 3–4, Table 2 · p. 4

3.2 Constructing Labeled Subset

  • For timestamped recordings, the labeled-subset pipeline matches audio with annotations, selects valid transcription tiers, and segments recordings into time-aligned speech-text pairs.Tier selection uses confidence levels: high-confidence tiers are accepted, medium-confidence tiers manually verified, and low-confidence tiers discarded; out-of-bounds timestamps lead to discarding affected recordings.Source: §3.2 · p. 4

3.3 Constructing Unlabeled Subset

  • The unlabeled-subset pipeline excludes recordings already used in the labeled subset, filters severely degraded audio, then diarizes and segments recordings.It retains original audio without enhancement and keeps overlaps shorter than 0.5 seconds; segments over 30 seconds are recursively split at the longest silence.Source: §3.3 · pp. 4–5

3.4 Normalization and Filtering

  • Both subsets are resampled to 24 kHz, loudness-normalized with constrained gain and a peak ceiling, assigned standardized language metadata, and filtered to remove sign-language or speech-free samples.Language aliases are mapped to canonical identifiers using the Ethnologue catalog.Source: §3.4 · p. 5

4 Experiments

The experiments test whether ELARSpeech supports ASR and TTS adaptation across long-tail languages and assess how multilingual speech representations transfer to them. They compare monolingual and multilingual training, varying supervision and evaluating several speech tasks.

4.1 Long-Tail Speech Modeling

  • ASR and TTS adaptation are evaluated monolingually on 30 pivot languages and multilingually on all 233 languages in the train, development, and test splits.For scaling experiments, models are progressively augmented with pseudo-labeled ELARSpeech-U; ASR uses Whisper-large-v3 and OmniASR-CTC-v2, while TTS uses OmniVoice.Source: §4.1 · p. 5
  • TTS training retains utterances with both SNR and C50 above 20 dB, while ASR is evaluated by CER and TTS by CER, MOS, and UTMOS across languages.ASR CER is measured against human transcriptions; UTMOS serves as an automatic proxy for evaluation across all 233 languages.Source: §4.1 · p. 5

4.2 Long-Tail Representation Analysis

  • Representation analysis benchmarks five multilingual speech models and mel-filterbank features on monolingual ASR, multilingual ASR, language identification, and joint ASR with LID.Evaluation follows ML-SUPERB under 10-minute and 1-hour supervision, reporting macro-averaged CER for ASR and accuracy for LID.Source: §4.2 · pp. 5–6
  • Table 3 reports CER for monolingual and multilingual ASR across 30 pivot languages, alongside macro CER and per-language win rate.Languages are sorted by data duration within each category.Source: Table 3 · p. 6

5 Results

Results show ELARSpeech supports ASR and TTS adaptation across underrepresented languages, while pseudo-labeling benefits ASR more consistently than TTS. Representation analysis finds that pretrained models retain benefits over a baseline but transfer less effectively to ELARSpeech than to ML-SUPERB.

5.1 Automatic Speech Recognition

  • Up to a 6.1% relative reduction in CER follows ASR training-data augmentation with pseudo-labeled ELARSpeech-U, with moderate confidence filtering outperforming retention of the entire corpus.Figure 2 reports ASR performance as the amount of pseudo-labeled data increases; confidence is averaged over predicted tokens and language-specific quartile cutoffs vary retained data.Source: Figure 2 · p. 6, §5.1 · p. 6
  • 22.48 macro CER for monolingual Whisper versus 30.71 for multilingual adaptation; the multilingual model outperforms the monolingual counterpart on only one of 30 pivot languages.Whisper achieves the lowest CER for 18 of 30 pivot languages; the authors describe the multilingual result as evidence of possible linguistic interference under limited, distributed supervision.Source: §5.1 · p. 6
  • ELARSpeech augmentation improves external-domain ASR results in three of four languages, while external-only models perform notably worse on ELARSpeech.The cross-domain evaluation covers ELARSpeech and external test sets CV26 and INEL.Source: Table 4 · p. 7, §5.1 · p. 7

5.2 Text-to-Speech

  • TTS adaptation yields intelligible speech across 14 pivot languages with at least five training hours, including languages with only a few hours of data.Some synthesized speech has lower CER than ground-truth recordings, which may partly reflect its more standardized acoustic characteristics.Source: §5.2 · p. 7
  • As more pseudo-labeled speech is added for TTS, CER rises slightly, while UTMOS peaks at an intermediate retention level and declines when all pseudo-labels are retained.Figure 3 presents TTS performance as the amount of pseudo-labeled ELARSpeech-U increases.Source: Figure 3 · p. 7, §5.2 · p. 7
  • Multilingual TTS has slightly higher CER overall but improves MOS for most evaluated languages, indicating an intelligibility–perceptual-quality trade-off.The MOS ratings for Table 5 come from non-native listeners.Source: §5.2 · p. 7, Table 5 · p. 7

5.3 Representation Analysis

  • Pretrained representations consistently outperform mel-filterbanks across evaluated tasks and both supervision budgets, but their relative advantages shrink on ELARSpeech.For 10-minute monolingual ASR, CER ranges from 30.3 to 41.0 on ML-SUPERB versus 51.9 to 53.8 on ELARSpeech.Source: §5.3 · p. 8, Table 6 · p. 8
  • MMS has the strongest overall performance despite 23.7% overlap with ELARSpeech languages, while XEUS has greater absolute overlap but only 8.4% of its broader inventory overlaps.The authors suggest long-tail transfer may depend less on pretraining language count alone and more on balanced exposure to low-resource languages.Source: §5.3 · p. 8

6 Conclusion

ELARSpeech extends multilingual speech resources into the linguistic long tail, with 30,202 hours across 675 underrepresented languages. Experiments show ASR and TTS adaptation and pseudo-labeling benefits, while representation analysis points to the importance of balanced low-resource language exposure.

6 Conclusion

  • ELARSpeech comprises 30,202 hours of speech across 675 underrepresented languages.The corpus extends multilingual speech resources into the linguistic long tail while preserving the acoustic and domain diversity of real-world field recordings.Source: §6 · p. 8
  • Experiments show that ELARSpeech supports ASR and TTS adaptation across diverse underrepresented languages, with pseudo-labeled data offering further scaling opportunities beyond limited human annotations.Source: §6 · p. 8
  • Representation analysis indicates that pretraining scale and language coverage alone are insufficient predictors of long-tail transfer, highlighting balanced exposure to low-resource languages.Source: §6 · p. 8

Limitations

The limitations concern inherited archival metadata uncertainty and the scope of human MOS evaluation, which relies mainly on non-native listeners.

Limitations

  • Archival language and recording metadata may retain inconsistencies or uncertainties from original deposits despite pipeline standardization and filtering.The pipeline cannot fully resolve inaccuracies in archival metadata.Source: Limitations · p. 9
  • MOS evaluation relies primarily on non-native listeners because recruiting native speakers across hundreds of low-resource languages is difficult.Source: Limitations · p. 9
  • The listening evaluation addresses general perceptual qualities rather than linguistic accuracy or native-like pronunciation and prosody.The authors identify this as an important direction for future work and complement human judgments with automatic measures.Source: Limitations · p. 9

Ethical Statement

This material documents ELARSpeech’s access and attribution conditions, its approach to preserving source-archive language distinctions, and procedures for selecting annotation tiers. It also reports corpus statistics and validation results for tier selection.

A License and Terms of Use

  • ELARSpeech includes only Open or User Access materials, preserves collection provenance and conditions, and requires downstream use to follow applicable source terms.The derivatives are released under CC BY-NC-SA 4.0, which does not supersede the ownership, attribution, or access conditions of original materials.Source: A License and Terms of Use · p. 12

B Dataset Design Principles

  • The dataset preserves source-collection language names, variety distinctions, and orthographies rather than normalizing or merging entries to impose uniformity.It retains varieties without ISO 639-3 codes and defers language-versus-dialect distinctions to the source collections.Source: B Dataset Design Principles · p. 12
  • Reported language counts reflect documented archive identities, not an independently verified linguistic taxonomy, because archival names, boundaries, and orthographies can be ambiguous.The stated limitations include variation across collections and sociolinguistically complex language–dialect boundaries.Source: B Dataset Design Principles · p. 12

C Detailed Statistics of ELARSpeech

  • Figure 4 presents language counts and total speech hours across 78 language families; Table 7 details split durations for 606 canonical languages.The multilingual subset is excluded from Table 7 because its samples cannot be unambiguously assigned to a single language.Source: C Detailed Statistics of ELARSpeech · p. 12, Figure 4 · p. 13

D Automatic Annotation Tier Selection

  • For each ELAN file, the highest-scoring annotation tier is accepted at 70 or above, manually reviewed from 50 to below 70, and rejected below 50.Manual validation found 993 of 1,000 sampled automatically accepted tiers were valid, and 999 of 1,000 sampled rejected tiers were invalid.Source: D Automatic Annotation Tier Selection · p. 14
  • 99.3% precision was estimated for automatically accepted transcription tiers, while 99.9% of automatically rejected tiers were confirmed invalid.Of 39,338 tiers, 7,144 scored at least 70, 3,936 scored between 50 and 70, and 28,258 scored below 50; review identified 1,035 valid tiers among the intermediate group (26.3%).Source: D Automatic Annotation Tier Selection · p. 14

E Unlabeled Data Segmentation

Unlabeled recordings are converted into training-ready clips through duration filtering and recursive segmentation. Evaluation pivots are selected to balance resource levels, language-family diversity, and coverage of languages without ISO codes.

E Unlabeled Data Segmentation

  • The segmentation pipeline takes speaker-homogeneous segments after diarization postprocessing and produces clips for ELARSpeech-U.The procedure is described as clip extraction with recursive splitting.Source: E Unlabeled Data Segmentation · p. 14
  • Segments of 0.3 seconds or less are discarded, and longer-than-30-second segments are recursively split until clips are at most 30 seconds.Splits prioritize the longest silence leaving at least 0.3 seconds on either side, then use the nearest valid VAD boundary or the midpoint; overlap information is retained as metadata.Source: E Unlabeled Data Segmentation · p. 14

F Pivot Language Selection

  • The 30 pivot languages span resource levels and language families, with 15 selected from languages lacking an ISO 639-3 code.Selection balances high-, mid-, and low-resource bins and replaces overlapping family selections where possible with candidates having comparable training data.Source: F Pivot Language Selection · p. 14

G MOS Evaluation Protocol

The TTS listening evaluation measures acoustic naturalness and listening quality for 14 extremely low-resource languages. Its protocol uses multiple listeners per utterance and controlled headphone or earphone playback.

Participants

  • 36 recruited listeners rated acoustic naturalness and listening quality on a 5-point scale for 14 extremely low-resource languages.They were instructed not to assess intelligibility, pronunciation, intonation, or language familiarity, which were evaluated separately using automatic metrics.Source: G MOS Evaluation Protocol · p. 15

Evaluation Procedure

  • Each of 210 utterances, sampled from five clips per each of 42 system-language pairs, was rated by 9 listeners.Listeners used headphones or earphones in a quiet environment; loudspeakers were not allowed.Source: G MOS Evaluation Protocol · p. 15

H Training Details

The paper describes fine-tuning and language adaptation for ASR and TTS, alongside the optimization schedule for its models. Full evaluation results are reported across languages represented in all three data splits.

H Training Details

  • Whisper and OmniASR-CTC-v2 use different vocabulary and language-token adaptations for languages unsupported during pretraining.Whisper adds a language token initialized from <en> when needed without expanding its BPE vocabulary; OmniASR-CTC-v2 expands its fixed output vocabulary for missing tokens.Source: H Training Details · p. 15
  • OmniVoice is fine-tuned for 50 epochs monolingually and 100K steps multilingually, with new language tokens initialized from <en> for unsupported languages.Monolingual training uses a total batch duration of 2.5 minutes.Source: H Training Details · p. 15
  • Models use AdamW with a tri-stage learning-rate schedule, and multilingual sampling probabilities follow p_l ∝ (n_l/N)^βL with βL = 0.5.The warmup, hold, and decay phases use ratios of 0.1, 0.5, and 0.4, respectively.Source: H Training Details · p. 15

I Full Evaluation Results

  • Tables 8 and 9 report full ASR and TTS evaluation across 233 languages present in train, development, and test splits.Table 9 includes CER, UTMOS, and speaker similarity; UTMOS and OmniVoice results are unavailable for Mezquital Otomi and Totonac, Zongozotla after filtering left no test samples.Source: I Full Evaluation Results · p. 15, Table 8 · p. 20, Table 9 · p. 22

Related papers

Loading 2610.19388v1…