PaperlayerView exact revision on arXiv

Source-linked AI summary

LEMAS: Large A 150K-Hour Large-scale Extensible Multilingual Audio Suite with Generative Speech Models

Zhiyuan Zhao, Lijian Lin, Ye Zhu, Kai Xie, Yunfei Liu, Yu Li

arXiv:2601.04233v1cs.SDeess.AS

TL;DR

Large-scale multilingual speech generation remains limited by gaps in corpus scale, coverage, and annotation quality. LEMAS introduces a 150,000-hour, word-timestamped corpus and trains complementary synthesis and editing models on it. The reported models achieve robust zero-shot cross-lingual synthesis and smooth-boundary multilingual editing.

  • Problem

    Open multilingual speech datasets and foundation models remain limited in scale, language coverage, and annotation quality.

    Source: §1 · p. 1

  • Method

    LEMAS releases a large multilingual corpus with word-level timestamps and trains flow-matching TTS and autoregressive speech-editing models on it.

    Source: §1 · p. 3

  • Results

    LEMAS-TTS achieves robust zero-shot cross-lingual synthesis, while LEMAS-Edit enables seamless, smooth-boundary multilingual editing.

    Source: §1 · p. 3

  • Takeaways & Limitations

    LEMAS-Dataset provides a foundation for training and evaluating next-generation speech generative models.

    Source: §7 · p. 13

    No subjective evaluation was conducted for languages other than Chinese and English.

    Source: §6.1 · p. 11

Abstract

from arXiv · show

We present the LEMAS-Dataset, which, to our knowledge, is currently the largest open-source multilingual speech corpus with word-level timestamps. Covering over 150,000 hours across 10 major languages, LEMAS-Dataset is constructed via a efficient data processing pipeline that ensures high-quality data and annotations. To validate the effectiveness of LEMAS-Dataset across diverse generative paradigms, we train two benchmark models with distinct architectures and task specializations on this dataset. LEMAS-TTS, built upon a non-autoregressive flow-matching framework, leverages the dataset's massive scale and linguistic diversity to achieve robust zero-shot multilingual synthesis. Our proposed accent-adversarial training and CTC loss mitigate cross-lingual accent issues, enhancing synthesis stability. Complementarily, LEMAS-Edit employs an autoregressive decoder-only architecture that formulates speech editing as a masked token infilling task. By exploiting precise word-level alignments to construct training masks and adopting adaptive decoding strategies, it achieves seamless, smooth-boundary speech editing with natural transitions. Experimental results demonstrate that models trained on LEMAS-Dataset deliver high-quality synthesis and editing performance, confirming the dataset's quality. We envision that this richly timestamp-annotated, fine-grained multilingual corpus will drive future advances in prompt-based speech generation systems.

1. Introduction

Large-scale multilingual speech generation is constrained by gaps in corpus scale, language coverage, and annotation quality. LEMAS-Dataset addresses these gaps with more than 150,000 hours of word-aligned speech across 10 languages and supports two generative models for synthesis and editing.

1. Introduction

  • Existing resources are either largely monolingual, limited in scale or domain, or variable in quality and fine-grained annotation; timestamped synthesis data such as WenetSpeech4TTS remains single-language.LEMAS aims to extend rigorous temporal supervision to a multilingual setting.Source: §1 · pp. 1–2
  • LEMAS-Dataset provides more than 150,000 hours of open-source speech across 10 major languages with rigorous word-level timestamps.Its duration and language coverage are extensive among diverse-source datasets with utterance-level timestamps.Source: §1 · pp. 2–3, Table 1 · p. 3
  • A multi-stage pipeline uses the MMS aligner to assign word-level timestamps and confidence scores, supporting reliability-aware filtering that trades data scale against alignment precision.This addresses challenges aligning noisy, unsegmented in-the-wild audio.Source: §1 · pp. 2–3
  • LEMAS-TTS combines a shared phonetic space with CTC alignment regularization and accent-adversarial objectives for robust zero-shot cross-lingual synthesis.The objectives target intelligibility and cross-lingual accent interference.Source: §1 · p. 3
  • LEMAS-Edit uses an autoregressive decoder-only system with stabilized inference and a pipeline for denoising, re-alignment, and verification to support smooth-boundary multilingual editing.The contribution also reports generalization to in-the-wild noisy audio.Source: §1 · p. 3

2. Related Work

Speech corpora have expanded from curated recordings to large web-mined and multilingual collections, while generative systems now span discrete, continuous, and hybrid approaches. Related work also covers multilingual synthesis and text-based speech editing.

2.1 Large-Scale Speech Corpora

  • Early curated and crowdsourced resources offered high-quality recordings and speaker diversity, while read-speech corpora became reproducible ASR benchmarks but remained limited in scale or domain.Examples include GlobalPhone, IARPA BABEL, Common Voice, LibriSpeech, and MLS.Source: §2.1 · p. 4
  • Web-mined corpora broadened scale and recording conditions, while multilingual collections expanded language coverage but often traded label quality for scale.GigaSpeech and WenetSpeech represent large multi-domain English and Mandarin resources; VoxPopuli and YODAS provide multilingual unlabeled or weakly supervised speech.Source: §2.1 · p. 4

2.2 Generative Speech Models

  • Generative speech models use discrete codec tokens, continuous noise-to-audio generation, or hybrid combinations of semantic modeling and acoustic refinement.The section surveys examples including VALL-E, VoiceBox, F5-TTS, CosyVoice, and FireRed-TTS.Source: §2.2 · p. 4
  • Multilingual TTS models have grown in language coverage and training scale, including systems trained jointly across many languages and large multilingual corpora.The cited examples include XTTS-v2, OpenAudio-S1-mini, and CosyVoice 3.Source: §2.2 · pp. 4–5
  • Text-based speech editing is treated as a TTS subtask, with prior systems using in-context generation, masked prediction, joint TTS-editing, or autoregressive sequence editing.Examples include VoiceBox, MaskGCT, F5-TTS, and VoiceCraft.Source: §2.2 · p. 5

3. LEMAS-Dataset

LEMAS-Dataset provides more than 150,000 hours of speech across 10 languages with word-level alignments and confidence scores. Its processing pipeline combines multilingual alignment with quality control, and its statistics describe both a large training corpus and a balanced evaluation set.

3.1 Dataset Processing Pipeline

  • The pipeline unifies heterogeneous sources, performs multilingual alignment, and applies scalable quality control to retain reliably grounded temporal structure.Its stated design separates language diversity from annotation fragility.Source: §3.1 · p. 5
  • Romanizing normalized transcripts lets the MMS aligner produce word boundaries and token confidence scores without language-specific pronunciation dictionaries.The approach is intended to handle rare words, named entities, and code-switching.Source: §3.1 · p. 5
  • Alignment-confidence and temporal-consistency filters retain reliable samples, using dataset-specific confidence thresholds from 0.2 to 0.5.Utterances shorter than 0.5 seconds or longer than 30 seconds, and samples with silence intervals exceeding 4 seconds, are removed.Source: §3.1 · p. 6
  • Language-specific speech-rate limits screen out abnormally fast or slow speech, while preprocessing retains in-the-wild acoustic conditions without explicit enhancement or noise removal.The thresholds are selected empirically to balance data scale and alignment precision.Source: §3.1 · p. 6

3.2 Dataset Statistics

  • Approximately 150,144 hours across 10 languages, with word-level alignments and confidence scores, make up the LEMAS-Dataset.The lower-resource Italian and Vietnamese subsets each exceed 6,000 hours.Source: §3 · p. 5, §3.2 · pp. 6–7
  • LEMAS-Dataset contains approximately 150,144 audio hours across 10 major languages; Figure 2 reports language-wise duration and average sentence duration, with substantial variation in both measures across languages.Chinese and English remain slightly larger, while Italian and Vietnamese each exceed 6,000 hours.Source: §3.2 · pp. 6–7, Figure 2 · p. 6
  • Chinese and English have longer average utterances than other languages, reflecting their longer-form source corpora compared with the shorter, pre-segmented sources used elsewhere.Table 2 summarizes training-set statistics, including average duration and average words per utterance.Source: §3.2 · p. 7, Table 2 · p. 7
  • The evaluation set selects 500 utterances per language using alignment, duration, word-count, and speech-rate criteria, with about 20% manually reviewed.Selection aims to support comparable cross-language evaluation without bias from data-volume or quality differences.Source: §3.2 · p. 7

4. LEMAS-TTS

LEMAS-TTS extends F5-TTS’s flow-matching and DiT framework with multilingual representations, alignment and accent stabilization, and prosody modeling. Its inference controls reshape guidance and sampling over time, with reported gains in perceptual stability for long multilingual utterances.

4. LEMAS-TTS

  • LEMAS-TTS extends F5-TTS’s non-autoregressive flow-matching and DiT framework with multilingual linguistic representations and stabilization mechanisms.The approach targets alignment stability and cross-language prosody transfer.Source: §4 · p. 8

4.1 Multilingual Training Framework

  • Its unified phonetic front end uses tonal Pinyin for Chinese and IPA for other languages, adding language identifiers and timestamp-derived pause tags.The pause tags represent short, medium, long, and abnormal pauses.Source: §4.1 · p. 8
  • Auxiliary CTC alignment and accent-adversarial losses target duration accuracy and cross-lingual accent leakage, while prosody embeddings provide explicit cross-lingual prosodic supervision.The prosody encoder’s embeddings are aligned with mel frames and text tokens and injected into the DiT.Source: §4.1 · p. 8
  • The final architecture excludes speaker InfoNCE and clip-and-shuffle augmentation after the authors observed instability and disrupted temporal coherence, respectively.The authors favor alignment and prosody objectives instead.Source: §4.1 · p. 8

4.2 Inference Pipeline and Sampling Dynamics

  • A decaying CFG schedule and power-law Sway Sampling reshape guidance and time-step allocation to balance pronunciation accuracy with speaker similarity and audio quality.CFG is stronger early and relaxes later; Figure 3 illustrates the guidance schedules and time warping.Source: §4.2 · p. 9, Figure 3 · p. 9
  • Redistributing sampling effort improves signal recovery and perceptual stability for long multilingual utterances.The authors report this effect for the power-law time-step warping controlled by γ.Source: §4.2 · p. 9
  • The inference interface supports reference-based prosody transfer and reference-free text-driven synthesis, alongside language-specific text normalization and explicit pause controls.The inference framework is designed to align with the training paradigm and minimize distribution shift.Source: §4.2 · pp. 8–10

5. LEMAS-Edit

LEMAS-Edit adapts a decoder-only speech-editing model into a multilingual framework through expanded training data, re-engineered training, and adaptive decoding. Its pipeline combines language-aware training with alignment-based editing controls and multilingual audio-processing features.

5. LEMAS-Edit

  • LEMAS-Edit extends VoiceCraft into a multilingual speech-editing framework with expanded training data, re-engineered distributed training, and adaptive decoding strategies.The stated aim of the decoding strategies is generation stability.Source: §5 · p. 10

5.1 Multilingual Adaptation and Training

  • Dynamic language tags let the autoregressive model switch linguistic context within a sentence for mixed-lingual editing.Source: §5.1 · p. 10
  • The distributed training framework shards data across heterogeneous datasets and GPU ranks, and replaces gradient accumulation with single-step full-batch updates.The authors report that this simplifies training and improves stability and robustness to memory fragmentation on large GPU clusters.Source: §5.1 · p. 10

5.2 Inference and Decoding Strategy

  • History-aware repetition penalties modify previously generated token logits, increasing the cost of repeats as sequences grow.Positive logits are divided by the penalty and negative logits multiplied by it, targeting silence loops and content stuttering.Source: §5.2 · p. 10
  • Adaptive decoding estimates a reference speaking rate, constrains maximum generation rate, and retries flagged or pathologically short outputs with expanded masks and stronger repetition penalties.Retries continue until the output matches the expected prosodic structure or the maximum retry limit is reached.Source: §5.2 · pp. 10–11

5.3 Multilingual Editing Front-End

  • The multilingual front end uses Whisper transcription and MMS word alignment to construct precise editing masks that confine changes to intended regions.Users can also choose between two denoising backends, process long recordings in cross-faded chunks, and use a shared interface for TTS and editing.Source: §5.3 · p. 11

6. Experiment

The experiments evaluate multilingual synthesis and speech editing with objective and subjective measures. LEMAS-TTS outperforms its baseline on objective metrics, while listener judgments for editing are balanced overall.

6. Experiment

  • The unified benchmark assesses TTS pronunciation and speaker similarity, plus editing naturalness after localized word-level replacements.TTS uses 10,000 samples spanning intra- and cross-lingual settings; editing cases use aligned utterances across seven languages.Source: §6 · p. 11

6.1 Evaluation of LEMAS-TTS

  • TTS evaluation compares LEMAS-TTS with OpenAudio-S1-mini using WER and speaker similarity, without subjective evaluation beyond Chinese and English.WER uses FunASR Paraformer-zh for Chinese and Whisper-large-v3 for other languages; similarity uses WavLM-large.Source: §6.1 · p. 11
  • LEMAS-TTS consistently achieves lower WER and higher speaker similarity than OpenAudio-S1-mini across languages.The prosody-aware variant lowers WER but slightly reduces speaker similarity, trading expressive richness for articulation stability; Vietnamese baseline WER is excluded from average WER.Source: Table 4 · p. 12, §6.1 · p. 12

6.2 Evaluation of LEMAS-Edit

  • LEMAS-Edit is warm-started from a 330M-parameter VoiceCraft checkpoint and trained on curated data spanning seven languages.Its combined training corpus draws on LEMAS-Dataset, WenetSpeech4TTS, GigaSpeech, and MLS, extending VoiceCraft to multilingual editing without changing the architecture.Source: §6.2 · pp. 12–13
  • Listener judgments of edited-audio naturalness are balanced overall, despite minor language-specific preferences.Six listeners compared outputs for 20 utterances across seven languages in an A/B preference test.Source: §6.2 · p. 13, Figure 4 · p. 13

7. Conclusion

The conclusion presents LEMAS as a multilingual research suite combining a large, fine-grained speech dataset with generative models. Its TTS and editing results underscore the role of data scale, diversity, and temporal granularity alongside model architecture.

7. Conclusion

  • LEMAS combines over 150,000 hours across 10 languages with word-level timestamps and confidence scores for training and evaluating speech-generation models.The paper presents the suite as a unified and extensible foundation for multilingual generative speech research.Source: §7 · p. 13
  • LEMAS-TTS results support alignment and disentanglement objectives for stabilizing large-scale multilingual flow-based models.The conclusion frames these objectives as essential to stabilization.Source: §7 · p. 13
  • LEMAS-Edit results support precise temporal supervision as a key enabler for robust zero-shot speech editing.The paper argues that progress depends on data scale, diversity, and granularity as well as model architecture.Source: §7 · p. 13
Loading 2601.04233v1…