Source-linked AI summary
The Multilingual TEDx Corpus for Speech Recognition and Translation
Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi, Douglas W. Oard, Matt Post
TL;DR
Speech translation lacks broadly available, multilingual, speech-aligned resources, especially beyond English and in varied domains. The paper introduces the Multilingual TEDx corpus with sentence-aligned transcripts, translations, and audio across 8 source languages, and finds that multilingual and pretrained resources improve low-resource translation performance.
Problem
Speech-translation resources have limited language and domain coverage, with many datasets constrained by licensing, size, read speech, or inconsistent multilingual segmentation.
Method
The paper builds the Multilingual TEDx corpus by segmenting transcripts into sentences and aligning them to source audio and target-language translations, with open code for extension.
Results
The corpus supports ASR and ST research across 8 source languages, while multilingual transfer improves translation by up to 24 BLEU in some low-resource language pairs and pretrained multilingual MT further improves ST performance.
Takeaways & Limitations
The publicly released corpus and methodology enable reproducible multilingual ASR and ST evaluation, including comparisons of transfer and end-to-end approaches.
Takeaways & Limitations
The corpus contains noisy punctuation and non-speech annotations, and some speech segments fail to align or lack complete transcript and translation metadata.
Abstract
from arXiv · showhide
We present the Multilingual TEDx corpus, built to support speech recognition (ASR) and speech translation (ST) research across many non-English source languages. The corpus is a collection of audio recordings from TEDx talks in 8 source languages. We segment transcripts into sentences and align them to the source-language audio and target-language translations. The corpus is released along with open-sourced code enabling extension to new talks and languages as they become available. Our corpus creation methodology can be applied to more languages than previous work, and creates multi-way parallel evaluation sets. We provide baselines in multiple ASR and ST settings, including multilingual models to improve translation performance for low-resource language pairs.
1. Introduction
The Multilingual TEDx corpus addresses limited language and domain coverage in speech-translation resources by providing multilingual TEDx speech, transcripts, and translations. It also supports reproducible extension and experiments for ASR and ST, including low-resource multilingual modeling.
- Existing speech-translation resources have limited language coverage, restricted licensing, small sizes, narrow domains, read-speech bias, or inconsistent multilingual segmentation.These constraints hinder broad evaluation and analysis of multilingual models.
- mTEDx provides TEDx speech in 8 source languages with manual transcriptions and translations in 6 target languages.
- The released code supports reproducibility and extension to new languages or additional data as they become available.
- The corpus supports ASR and ST experiments, including multilingual models intended to improve performance in low-resource language settings.
2. Corpus Creation
The corpus is constructed from multilingual TEDx recordings by selecting eligible speech, creating sentence-level text alignments, and aligning sentences to audio. Standardized splits and reported alignment failures support reproducible ASR, MT, and ST evaluation.
- 2.1. Data Collection: TEDx supplies short, prepared multilingual talks with volunteer transcripts and translations, while the corpus targets non-English languages with segmental orthography and at least 15 hours of transcribed speech.Some talks lack transcripts, translations, or metadata, and transcript punctuation and non-speech annotations can be noisy.
- 2.2. Sentence Alignment for Text: Sentence-level source segmentation is generated with Punkt, while Vecalign uses LASER sentence embeddings and overlapping target-subtitle windows to produce multi-way alignments.The source segmentation is held constant across available translations.
- 2.2. Sentence Alignment for Text: Vecalign alignments improve downstream MT performance across all language pairs relative to Gargantua, with gains ranging from slight to substantial.The comparison is evaluated using SACREBLEU after independently aligning both systems to fixed reference sentences.
- 2.3. Sentence Alignment for Speech: Speaker-adapted HMM-GMM acoustic models trained with MFCC features and phonemic lexicons force-align source transcripts to audio and yield sentence time boundaries.CTM word timings are aligned back to source sentences to derive the boundaries.
- 2.3. Sentence Alignment for Speech: A modified Viterbi alignment handles extra speech and missing audio by inserting sentence-boundary tokens, while failed alignments are reported for optional exclusion.Only about 0.1% of segments failed to align, and two talks with majority failures were excluded.
- 2.3. Sentence Alignment for Speech: Training, validation, and evaluation splits hold out complete talks and use the same talks for ASR and MT evaluation to prevent cross-task training contamination.
3. Experiments
The experiments establish ASR, MT, and ST baselines on Multilingual TEDx, showing benefits from multilingual transfer, pretrained text models, and encoder pretraining, especially for lower-resource settings.
- 3.1. Automatic Speech Recognition (ASR): Hybrid LF-MMI ASR models were more robust in lower-resource conditions, while performance generally scaled with data size but had exceptions including Italian and Arabic.
- 3.2. Machine Translation (MT): Translation baselines were strongest for more related language pairs and pairs with more than 10k sentences, consistent with more monotonic token alignment.
- 3.2. Machine Translation (MT): Up to 24 BLEU improvement came from multilingual transfer for language pairs with limited parallel data but shared source or target languages.
- 3.2.1. Multilingual Models: M2M 100 produced further MT improvements and could combine with smaller speech-aligned datasets to improve speech translation despite lacking aligned speech.
- 3.3.1. Cascaded Models: Multilingual MT cascades improved speech translation for all but es-pt, with substantial BLEU gains for the smallest Romance languages.
- 3.3.2. End-to-End Models: Multilingual end-to-end ST improved over bilingual end-to-end models and the lowest-resource bilingual cascades, but remained behind multilingual cascades.
4. Conclusion
The paper releases Multilingual TEDx to broaden speech recognition and translation research across languages and support comparisons among diverse modeling approaches.
- The released corpus supports multilingual ASR and ST research and comparison of cascaded, end-to-end, multilingual, and transfer-learning methods.