PaperlayerView exact revision on arXiv

Source-linked AI summary

CoVoST 2 and Massively Multilingual Speech-to-Text Translation

Changhan Wang, Anne Wu, Juan Pino

arXiv:2007.10310v3cs.CLcs.SDeess.AS

TL;DR

Existing speech-translation datasets cover a limited number of languages, including low-resource pairs. CoVoST 2 expands multilingual speech-translation data and provides baseline systems; the authors describe it as the largest open dataset by total volume and language coverage.

  • Problem

    Existing speech-translation datasets cover a limited number of languages, while research also seeks to address low-resource language pairs.

    Source: §1 · p. 1

  • Method

    The authors extend CoVoST with multilingual translation data, apply translation sanity checks, and provide ASR, MT, and speech-translation baselines.

    Source: §1 · p. 1, §2.1 · p. 1, §4 · p. 3

  • Results

    CoVoST 2 is described as the largest open speech-translation dataset by total volume and language coverage, spanning 21 languages into English and English into 15 languages.

    Source: §1 · p. 1, §5 · p. 5

  • Takeaways & Limitations

    The CC0 corpus and open-source baselines support research on massive multilingual and low-resource speech translation.

    Source: §4 · p. 3, §5 · p. 5

    Many sentences that cannot be translated properly are contextless article extracts lacking clarity for translation.

    Source: §2.2 · p. 2, §2.1 · p. 2

Abstract

from arXiv · show

Speech translation has recently become an increasingly popular topic of research, partly due to the development of benchmark datasets. Nevertheless, current datasets cover a limited number of languages. With the aim to foster research in massive multilingual speech translation and speech translation for low resource language pairs, we release CoVoST 2, a large-scale multilingual speech translation corpus covering translations from 21 languages into English and from English into 15 languages. This represents the largest open dataset available to date from total volume and language coverage perspective. Data sanity checks provide evidence about the quality of the data, which is released under CC0 license. We also provide extensive speech recognition, bilingual and multilingual machine translation and speech translation baselines with open-source implementation.

1 Introduction

Benchmark datasets helped make speech-to-text translation a growing research topic, but the corpora described cover limited language ranges and generally emphasize high-resource European languages. CoVoST 2 extends this landscape with broader multilingual coverage and substantially more speech data.

  • Earlier benchmarks supported end-to-end and multilingual speech translation, but their language coverage remained limited and generally centered on high-resource European languages.MuST-C covered English into eight European languages, while Europarl-ST covered six European languages across 30 directions.Source: §1 · p. 1
  • The original CoVoST brought low-resource languages such as Mongolian into a corpus spanning 11 languages into English and enabled many-to-one speech translation research.Source: §1 · p. 1
  • CoVoST 2 expands coverage to 21 languages into English and English into 15 languages, increasing speech duration from 700 to 2,880 hours and speakers from 11K to 78K.The corpus is released under CC0.Source: §1 · p. 1

2 Dataset Creation

CoVoST 2 combines translation quality checks with expanded speech splits, then reports broad speaker and language statistics. Its dataset procedures also mark sentences that cannot be translated properly and identify a repeated-sentence caveat for one evaluation split.

2.1 Data Collection and Quality Control

  • Professional translations are checked using language-model perplexity, LASER scores, and length ratios; low-perplexity examples are manually examined and may be sent for retranslation.Language-model checks use 20M sampled lines per language, with pre-trained models used for English and Russian.Source: §2.1 · pp. 1–2
  • Sentences that cannot be translated properly are marked in the release, and many are contextless article extracts lacking clarity for translation.Source: §2.1 · p. 2, §2.2 · p. 2

2.2 Dataset Splitting

  • CoVoST splits add some discarded Common Voice samples while keeping speakers disjoint and sentence assignments consistent across splits, raising speech-data utilization from 44.2% (1,273 hours) to 78.8% (2,270 hours).Source: §2.2 · p. 2
  • The default setup trains on the CoVoST train split and evaluates on Common Voice dev/test splits; the complementary CoVoST dev/test split supports multi-speaker evaluation but repeated sentences may skew aggregate scores.Source: §2.2 · pp. 2–3

2.3 Statistics

  • CoVoST 2 statistics cover speech duration, speaker counts, and transcript and translation tokens; speakers span 66 accent groups, 8 age groups, and 3 gender groups, including in some low-resource languages.Table 1 compares original Common Voice and extended CoVoST speech splits; Chinese and Japanese token counts are character-based.Source: Table 1 · p. 2, §2.3 · p. 3

3 Models

The paper describes Transformer-based speech recognition and speech translation models, plus a Transformer-based machine translation setup. Multilingual speech models use a target-language token to direct decoding.

  • ASR and speech-translation models share a Transformer encoder-decoder with 12 encoder layers and 6 decoder layers, preceded by a convolutional downsampler that reduces speech-input length by 3.Source: §3 · p. 3
  • For multilingual En→All and All→All speech models, decoding is directed to a specified language with a target-language ID token as the first decoding token.Source: §3 · p. 3
  • Machine translation uses a Transformer base with shared encoder/decoder embeddings, while multilingual models share encoders and decoders based on preliminary competitive results.The passage specifies 0.3 dropout and denotes encoder and decoder layer counts as le and ld.Source: §3 · p. 3

4 Experiments

The experiments provide ASR, MT, cascaded ST, and end-to-end ST baselines in monolingual, bilingual, and multilingual settings. The reported results identify multilingual modeling as a promising direction for improving low-resource speech translation.

4 Experiments

  • The paper provides monolingual and multilingual ASR baselines and bilingual and multilingual MT, cascaded ST, and end-to-end ST baselines.The multilingual ST settings include All→En, En→All, and All→All.Source: §4 · p. 3

4.1 Experimental Settings

  • Settings compare bilingual and multilingual training, including reverse-direction MT data (+Rev), and report BLEU for MT/ST and WER or CER for ASR.Source: §4.1 · pp. 4–5

4.2 Monolingual and Bilingual Baselines

  • For low-resource language pairs, opposite-direction training data can improve MT and cascaded ST results, while transcription and translation quality depend strongly on training-data volume.Source: §4.2 · p. 5

4.3 Multilingual Baselines

  • Multilingual modeling is reported as a promising direction for improving low-resource speech translation.The multilingual results cover X→English and English→X tasks in Tables 3 and 4.Source: Table 3 · p. 4, Table 4 · p. 4, §4.3 · p. 5

5 Conclusion

CoVoST 2 is presented as the largest speech-to-text translation corpus to date by language coverage and total volume. The paper also provides extensive ASR, MT, and ST baselines and identifies research directions supported by the corpus.

  • CoVoST 2 covers 21 languages into English and English into 15 languages, and is described as the largest speech-to-text translation corpus to date by language coverage and total volume.Source: §5 · p. 5
  • The corpus is released under CC0 with extensive ASR, MT, and ST baselines, supporting research on multilingual and low-resource speech translation, self-supervision, and semi-supervised modeling.Source: §5 · p. 5
Loading 2007.10310v3…