FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech

Alexis Conneau; Min Ma; Simran Khanuja; Yu Zhang; Vera Axelrod; Siddharth Dalmia; Jason Riesa; Clara Rivera; Ankur Bapna

Source-linked AI summary

FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech

Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, Ankur Bapna

arXiv:2205.12446v1May 25, 2022cs.CLcs.LGcs.SDeess.AS

TL;DR

FLEURS addresses the limited availability of broad multilingual speech evaluation by introducing an n-way parallel dataset in 102 languages. It constructs the resource from FLoRes-101 and evaluates multilingual pre-trained representations across recognition, identification, translation, and retrieval tasks. The paper concludes that FLEURS is suited to data-efficient multilingual evaluation and reports baseline performance across its tasks.

Problem
Existing multilingual speech datasets cover fewer languages or narrower task settings, limiting evaluation across more than 100 languages.
Source: §1 · p. 1
Method
FLEURS builds n-way parallel speech and text from FLoRes-101 and fine-tunes multilingual pre-trained models for several speech tasks.
Source: §2.1 · p. 1, §1 · p. 1, §3.1 · p. 3
Results
The paper provides baseline ASR, Speech LangID, and retrieval systems and reports 73.3% macro-average accuracy for mSLAM on FLEURS LangID.
Source: §4.2 · p. 4, §5 · p. 5
Takeaways & Limitations
FLEURS is presented as a benchmark for evaluating data-efficient multilingual speech and text representations across 102 languages.
Source: §1 · p. 1, §5 · p. 5
FLEURS-LangID uses different speakers across train and dev/test, making avoidance of speaker-ID overfitting essential for good performance.
Source: §4.2 · p. 4

Abstract

from arXiv · show

We introduce FLEURS, the Few-shot Learning Evaluation of Universal Representations of Speech benchmark. FLEURS is an n-way parallel speech dataset in 102 languages built on top of the machine translation FLoRes-101 benchmark, with approximately 12 hours of speech supervision per language. FLEURS can be used for a variety of speech tasks, including Automatic Speech Recognition (ASR), Speech Language Identification (Speech LangID), Translation and Retrieval. In this paper, we provide baselines for the tasks based on multilingual pre-trained models like mSLAM. The goal of FLEURS is to enable speech technology in more languages and catalyze research in low-resource speech understanding.

1. Introduction

FLEURS addresses the need for massively multilingual speech evaluation by providing n-way parallel speech and text in 102 languages. It supports diverse speech tasks and includes baselines based on multilingual pre-trained models.

Unlike other cited parallel speech datasets, FLEURS spans over 100 languages and supports diverse language research.Source: §1 · p. 1
FLEURS is an n-way parallel speech and text dataset covering 102 languages.Source: §1 · p. 1
The dataset provides natural human speech, high-quality transcripts, and strong quality control.Source: §1 · p. 1
FLEURS uses bottom-up collection of spoken utterances for aligned segments rather than primarily document-level automatic alignment.Source: §1 · p. 1
FLEURS supports ASR, speech translation, Speech LangID, and multilingual speech-text retrieval.Source: §1 · p. 1
The paper provides Speech LangID, ASR, and speech-text retrieval baselines by fine-tuning mSLAM and multilingual w2v-BERT.Source: §1 · p. 1

2. Dataset

FLEURS builds a multilingual speech dataset from parallel FLoRes-101 sentences, with recordings from native speakers and transcripts inherited from human translations. The dataset preserves parallel indexing, linguistic diversity, and reproducibility-oriented preprocessing.

Speech Data Collection

FLEURS is constructed from 2009 FLoRes-101 dev and devtest sentences across 102 languages.The splits contain 1509 training, 150 development, and 350 test sentences.Source: §2.1 · p. 1
Each sentence initially receives three recordings from three native speakers, with an attempted sex-ratio balance of at least 30/70%.Source: §2.1 · p. 1
About 21.5% of sentences are missing in the first dataset version because none of their recordings were validated.The authors plan to fill these gaps in future versions.Source: §2.1 · p. 2

Transcripts and Representation

FLEURS reuses human-translated source transcripts and retains English translations for multimodal speech-translation evaluation.Source: §2.2 · p. 2
The dataset adopts a universal character vocabulary as both its modeling unit and character-level evaluation unit.Source: §2.2 · p. 2

Taxonomy and Statistics

FLEURS covers 17 language families and 27 writing systems, with sentence indices enabling recovery of n-way parallelism.Source: §2.3 · p. 2

3. Task Baselines

The baselines use pre-training followed by fine-tuning because training a giant model directly on FLEURS would overfit. The setup initializes multilingual fine-tuning from a large wav2vec-BERT model trained on unlabeled speech.

Experimental Setup

The baseline methodology fine-tunes multilingual pre-trained models rather than training a giant model from scratch on FLEURS.The paper states that direct training would easily overfit.Source: §3.1 · p. 3
The main initialization is a 600M-parameter wav2vec-BERT model pre-trained on 429k hours of unlabeled speech in 51 languages.Source: §3.1 · p. 3
The baselines cover speech recognition, Speech LangID, and speech-text retrieval tasks.Source: §3.1 · p. 3

Language Splits

The experiments distinguish 54 seen languages, whose speech data appeared in pre-training, from 48 unseen languages without such speech data.Source: §3.2 · p. 3

4. Downstream Tasks

FLEURS is evaluated across multilingual ASR, Speech LangID, and cross-modal speech-text retrieval using pretrained-model baselines. Performance varies substantially by language group and pretraining coverage, with consistent challenges for underrepresented or unseen languages.

Speech recognition

10.7 and 10.6 average CER were reported for Western European languages, compared with 15.6 and 16.4 for Sub-Saharan African languages and 17.4 and 19.2 for South Asian languages.The paper partly attributes these gaps to larger amounts of unlabeled Western European pretraining data.Source: §4.1.1 · pp. 3–4
Speech-only pretraining produced 0.5% lower CER than speech-text pretraining across the multilingual ASR baselines.The regression with speech-text initialization was observed especially in South Asian, Sub-Saharan African, and CJK groups.Source: §4.1.1 · p. 4
Languages unseen during pretraining often had worse-than-average test CER because fine-tuning used very limited supervised data.Some unseen languages nevertheless achieved below-average CER, including languages using Latin, Cyrillic, Malayalam, Kannada, Gujarati, Devanagari, Burmese, and Arabic scripts.Source: §4.1.2 · p. 4

Speech Language Identification

73.3% macro-average accuracy was obtained by mSLAM on FLEURS Speech LangID, versus 71.4% for multilingual w2v-bert-51.Group accuracy decreased in the order CJK, Western European, Eastern European, Central-Asia/Middle-East/North-Africa, South-East Asia, Sub-Saharan Africa, and South Asia.Source: §4.2 · p. 4

Cross-modal Speech-Text Retrieval

76.9% P@1 was achieved for speech-to-text retrieval and 74.4% P@1 for text-to-speech retrieval.Seen languages generally outperformed unseen languages, with especially steep degradation for CJK languages.Source: §4.3 · p. 5
Retrieval performance exposed modality- and script-specific weaknesses: Odia lacked tokenizer representation, while Urdu performed considerably worse for text-to-speech retrieval than speech-to-text retrieval.The paper associates Urdu’s difficulty with phonetic similarity to other South Asian languages and missing speech-modality pretraining.Source: §4.3 · p. 5

5. Conclusion

FLEURS is a 102-language n-way parallel speech dataset for evaluating speech recognition, translation, classification, and retrieval methods, especially data-efficient multilingual pre-trained representations.

FLEURS is an n-way parallel speech dataset covering 102 languages.Source: §5 · p. 5
FLEURS supports evaluation of speech recognition, translation, classification, and retrieval methods.Source: §5 · p. 5
Baseline ASR, language identification, and retrieval systems show FLEURS is suited to evaluating data-efficient multilingual pre-trained speech and text representations.Source: §5 · p. 5
The dataset is intended to catalyze few-shot understanding research across many languages.Source: §5 · p. 5