Source-linked AI summary
Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, Zhizheng Wu
TL;DR
Speech-generation models still lack large, diverse, spontaneous speech data for producing highly spontaneous, human-like speech. The paper introduces Emilia, a large multilingual dataset, and Emilia-Pipe, an open-source pipeline for converting in-the-wild speech into annotated training data. Evaluations demonstrate Emilia’s effectiveness for high-quality spontaneous and human-like speech generation, while Emilia-Pipe improves processed-data quality and scales efficiently.
Problem
Existing speech-generation datasets are rooted in formal audiobook speech, while real-world speech contains more diverse and spontaneous speaking styles.
Method
The paper constructs Emilia from in-the-wild speech and develops Emilia-Pipe to preprocess, annotate, and scale such data for speech generation.
Results
Evaluations demonstrate that Emilia effectively trains models for high-quality, spontaneous, human-like speech, with promising performance in multilingual TTS.
Takeaways & Limitations
Emilia and Emilia-Pipe provide publicly available resources for large-scale multilingual speech generation using diverse in-the-wild speech.
Abstract
from arXiv · showhide
Recent advancements in speech generation models have been significantly driven by the use of large-scale training data. However, producing highly spontaneous, human-like speech remains a challenge due to the scarcity of large, diverse, and spontaneous speech datasets. In response, we introduce Emilia, the first large-scale, multilingual, and diverse speech generation dataset. Emilia starts with over 101k hours of speech across six languages, covering a wide range of speaking styles to enable more natural and spontaneous speech generation. To facilitate the scale-up of Emilia, we also present Emilia-Pipe, the first open-source preprocessing pipeline designed to efficiently transform raw, in-the-wild speech data into high-quality training data with speech annotations. Experimental results demonstrate the effectiveness of both Emilia and Emilia-Pipe. Demos are available at: https://emilia-dataset.github.io/Emilia-Demo-Page/.
1. INTRODUCTION
Emilia addresses the lack of large, diverse, spontaneous speech data by combining a multilingual dataset with an open-source pipeline for processing in-the-wild speech. The dataset contains over 101k hours across six languages and supports high-quality, spontaneous speech generation.
- Motivation: Existing speech-generation datasets largely derive from audiobooks and formal reading, whereas real-world speech includes more diverse and spontaneous styles.Real-world speech may include breathing, pauses, repetitions, changing speed, and varying emotions.
- Motivation: In-the-wild speech requires preprocessing because it varies in quality and length, contains noise and multiple speakers, and often lacks transcriptions.These issues make direct use of raw speech data infeasible for training.
- Contributions: Emilia-Pipe is an open-source pipeline that converts in-the-wild speech into high-quality, annotated training data while supporting multiple languages and efficient processing.It processes one hour of raw speech in a few minutes and is intended to support broader collaboration.
- Contributions: Over 101k hours of 24 kHz speech across six languages make Emilia an extensive multilingual dataset for speech generation.The languages are English, Chinese, German, French, Japanese, and Korean.
- Contributions: Emilia consists mostly of spontaneous speech covering diverse speaking styles for training models to generate more spontaneous and human-like speech.The dataset’s diversity is presented as a key advantage for speech generation.
- Contributions: The dataset is dynamic because Emilia-Pipe can incorporate readily available raw in-the-wild or user-specified speech sources for scaling.Both Emilia and Emilia-Pipe are publicly available.
- Validation: Evaluations find Emilia effective for high-quality, spontaneous, human-like speech, while full Emilia also shows promising multilingual TTS performance.The comparison trains two TTS models on Emilia’s English subset and compares them with models trained on MLS audiobooks.
2. THE EMILIA-PIPE PREPROCESSING PIPELINE
Emilia-Pipe transforms raw in-the-wild speech into standardized, single-speaker, transcribed, filtered training data through sequential preprocessing stages. On approximately 600 hours of raw speech, it retained 29.43% of the data while improving average DNSMOS P.835 OVRL from 2.50 to 3.26 and processing about 2.50 hours per minute.
- Pipeline Overview: Emilia-Pipe comprises Standardization, Source Separation, Speaker Diarization, Fine-grained Segmentation by VAD, ASR, and Filtering.These six stages are designed to convert raw speech into training data with speech annotations.
- Standardization: Standardization converts samples to mono 24 kHz WAV audio with 16-bit width, target level -20 dBFS, bounded volume adjustment, and normalized amplitudes.Volume adjustment is constrained between -3 and 3 dB, and waveform values are normalized to the range -1 to 1.
- Source Separation: Source separation extracts clean human vocals from raw speech containing background music or noise.The pipeline uses Ultimate Vocal Remover with the UVR-MDX-Net Inst 33 pretrained model.
- Speaker Diarization: Speaker diarization partitions long-form speech into single-speaker segments using speaker segmentation, embeddings, and clustering.The output is a list of temporal annotations marking the start and end of single-speaker segments.
- ASR: ASR transcribes segmented speech with Whisper-Medium, while WhisperX, faster-whisper, CTranslate2, and batched inference improve processing efficiency.WhisperX is reported as up to four times faster than the official Whisper implementation while maintaining nearly the same accuracy.
- Performance Evaluation: After processing approximately 600 hours, Emilia-Pipe retained 29.43% of raw speech, raised average DNSMOS P.835 OVRL from 2.50 to 3.26, and processed about 2.50 hours per minute.The evaluation used eight NVIDIA RTX 4090 GPUs and took about 3.99 hours.
3. THE EMILIA DATASET
Emilia is a 101,654-hour, six-language speech dataset built from diverse online sources and processed for quality. Its speech exhibits broader acoustic and semantic diversity than the audiobook-derived MLS dataset, while achieving comparable speech quality after preprocessing.
- Overview: 101,654 hours of speech across six languages form the initial Emilia dataset, sourced from video platforms and podcasts spanning varied content categories.Sources include talk shows, interviews, debates, sports commentary, and audiobooks.
- Overview: Emilia is designed around mostly spontaneous speech and a wide range of speaking styles.The dataset targets real human speech rather than standardized reading patterns.
- Quality: 3.26 DNSMOS P.835 OVRL places Emilia third among the compared datasets and above all existing in-the-wild speech datasets.The comparison uses a randomly sampled 600-hour subset for Emilia and MLS.
- Diversity: Emilia’s acoustic representations show broader dispersion than MLS, indicating richer coverage of speaker, emotion, and prosody characteristics.The analysis samples 5,000 English examples from each dataset and reduces WavLM representations to two dimensions with PCA.
- Diversity: Emilia’s textual representations scatter across a wider range of content, supporting greater semantic diversity than audiobook-derived MLS.Sentence-BERT transcripts are reduced from 768 dimensions to two for visualization.
4. EXPERIMENTS
The experiments compare TTS models trained on English Emilia and MLS, then evaluate models trained on full Emilia across six languages. Results show comparable core quality, stronger AR performance on spontaneous prompts, and strong zero-shot multilingual TTS.
- Evaluation Metrics: The evaluation measures intelligibility with WER, speaker similarity with SIM-O and SMOS, and naturalness with FSD and CMOS.Objective and subjective evaluations are conducted on LibriSpeech-Test and Emilia-Test.
- English TTS: English experiments compare AR+SoundStorm and VoiceBox models trained on similarly sized Emilia and MLS datasets.English Emilia contains 46k hours, while MLS contains 44.5k hours.
- English TTS: Models trained on Emilia and MLS achieve similar speaker similarity and intelligibility, while NAR models show similar naturalness.This result holds across the LibriSpeech-Test and Emilia-Test evaluations.
- English TTS: On Emilia-Test’s diverse spontaneous prompts, the AR model shows significant improvement in FSD and CMOS when trained on Emilia rather than MLS.The results may indicate that AR TTS models benefit more from diverse speaking styles than NAR models.
- Multilingual TTS: Models trained on full Emilia demonstrate strong zero-shot multilingual TTS performance across English, Chinese, German, French, Japanese, and Korean.The multilingual evaluation uses at least 500 reference samples per test set and reports objective results.
5. CONCLUSIONS
The paper introduces Emilia and Emilia-Pipe as an extensive, multilingual, diverse dataset and open-source preprocessing pipeline for speech generation. Evaluations support their effectiveness for producing more natural, spontaneous, and human-like speech.
- Contributions: Emilia contains over 101k hours of diverse in-the-wild speech in six languages, while Emilia-Pipe converts raw speech into high-quality training data.The pipeline is open source and designed to process speech efficiently.
- Contributions: Objective and subjective evaluations demonstrate that training on Emilia can advance speech generation toward more natural, spontaneous, and human-like speech.The paper also open-sources the dataset and pipeline for community collaboration.