PaperlayerView exact revision on arXiv

Source-linked AI summary

Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition

Ye Bai, Jingping Chen, Jitong Chen, Wei Chen, Zhuo Chen, Chuang Ding, Linhao Dong, Qianqian Dong, Yujiao Du, Kepan Gao, Lu Gao, Yi Guo, Minglun Han, Ting Han, Wenchao Hu, Xinying Hu, Yuxiang Hu, Deyu Hua, Lu Huang, Mingkun Huang, Youjia Huang, Jishuo Jin, Fanliu Kong, Zongwei Lan, Tianyu Li, Xiaoyang Li, Zeyang Li, Zehua Lin, Rui Liu, Shouda Liu, Lu Lu, Yizhou Lu, Jingting Ma, Shengtao Ma, Yulin Pei, Chen Shen, Tian Tan, Xiaogang Tian, Ming Tu, Bo Wang, Hao Wang, Yuping Wang, Yuxuan Wang, Hanzhang Xia, Rui Xia, Shuangyi Xie, Hongmin Xu, Meng Yang, Bihong Zhang, Jun Zhang, Wanyi Zhang, Yang Zhang, Yawei Zhang, Yijie Zheng, Ming Zou

arXiv:2407.04675v2eess.AScs.SD

TL;DR

ASR systems must recognize diverse speech while using application-specific context, but conventional end-to-end systems increasingly depend on extra language-model fusion. Seed-ASR addresses this with an audio-conditioned LLM trained through large-scale, stage-wise learning and context-aware training. It reports stronger performance across diverse evaluations and 10%-40% lower word or character error rates than recently released large ASR models on Chinese and English public tests.

  • Problem

    ASR systems need to transcribe diverse speech with application-specific context, while conventional end-to-end models are limited in contextual reasoning and rely on extra language-model fusion.

  • Method

    Seed-ASR uses an audio-conditioned LLM that combines continuous speech representations with instructions and context, trained through large-scale audio pretraining, SFT, context SFT, and RL.

  • Results

    Seed-ASR demonstrates strong performance across multi-domain, multi-language, multi-dialect, multi-accent, and context-aware evaluations, with improvements over strong end-to-end baselines.

  • Takeaways & Limitations

    Seed-ASR provides a unified model structure for diverse speech-recognition and customized context-aware application scenarios without requiring extra language models.

Abstract

from arXiv · show

Modern automatic speech recognition (ASR) model is required to accurately transcribe diverse speech signals (from different domains, languages, accents, etc) given the specific contextual information in various application scenarios. Classic end-to-end models fused with extra language models perform well, but mainly in data matching scenarios and are gradually approaching a bottleneck. In this work, we introduce Seed-ASR, a large language model (LLM) based speech recognition model. Seed-ASR is developed based on the framework of audio conditioned LLM (AcLLM), leveraging the capabilities of LLMs by inputting continuous speech representations together with contextual information into the LLM. Through stage-wise large-scale training and the elicitation of context-aware capabilities in LLM, Seed-ASR demonstrates significant improvement over end-to-end models on comprehensive evaluation sets, including multiple domains, accents/dialects and languages. Additionally, Seed-ASR can be further deployed to support specific needs in various scenarios without requiring extra language models. Compared to recently released large ASR models, Seed-ASR achieves 10%-40% reduction in word (or character, for Chinese) error rates on Chinese and English public test sets, further demonstrating its powerful performance.

1 Introduction

Seed-ASR is an LLM-based ASR model designed to improve recognition across diverse speech, languages, domains, and contexts. It combines large-scale training with context-aware modeling and reports strong performance across broad evaluations.

  • Seed-ASR uses an audio-conditioned LLM that inputs continuous speech representations together with instructions and contextual information.The model is developed under the AcLLM framework to leverage LLM capabilities for speech recognition.
  • More than 20 million hours of speech data and nearly 900 thousand hours of paired ASR data support Seed-ASR’s Chinese multi-dialect and multilingual models.The models use large audio and language-model capacity for evaluation on public and in-house datasets.
  • Training uses a stage-wise recipe of self-supervised audio-encoder learning, supervised fine-tuning, context SFT, and reinforcement learning.Each stage is presented as having a distinct role in improving model performance.
  • Figure 1 compares Seed-ASR with strong released models on internal multi-domain and public Mandarin and English evaluations.The MLS en-US Whisper Large-v3 result was obtained by local decoding because no published WER was available.
  • Seed-ASR targets broad ASR improvements across Chinese and English, varied speech types, and application scenarios requiring contextual customization.Its evaluation sets cover domains, accents or dialects, languages, speech duration, and context-dependent keyword accuracy and consistency.

2 Motivation

Existing end-to-end ASR systems improve recognition accuracy but remain limited in contextual reasoning and efficient use of broad knowledge. The motivation is to draw on LLM advances while addressing the challenge of making LLMs understand speech.

  • The field has progressed from hybrid ASR frameworks to end-to-end models that directly generate transcriptions, with WER as a central accuracy measure.The passage frames this progression as the background for considering a new model framework.
  • Current end-to-end ASR models remain limited by model capacity and from-scratch training, hindering contextual reasoning and use of common-sense knowledge.They consequently rely on complicated fusion strategies with extra language models.
  • LLMs offer a potential ASR framework through unified text understanding and generation, scaling-law benefits, and comprehensive training pipelines.The motivation identifies unified decoder-only modeling, large-scale capacity, and staged training as relevant advances.
  • ASR’s speech-to-text objective aligns with LLM text generation, allowing LLM knowledge and contextual reasoning to provide semantic guidance.The remaining challenge is enabling LLMs to understand speech as a modality distinct from text.

3 Methods

Seed-ASR uses an audio-conditioned LLM framework that maps continuous speech representations, instructions, and context into an LLM for transcription. Its method combines large-scale audio pretraining, staged adaptation, context-aware decoding, and reinforcement learning to improve recognition across diverse conditions.

  • Seed-ASR inputs encoded continuous speech representations with task instructions and relevant context into a pretrained LLM for transcription.
  • SSL of Audio Encoder: Large-scale self-supervised learning trains the nearly 2-billion-parameter LUISE audio encoder on speech data, producing rich speech and semantic representations for downstream fine-tuning.LUISE uses masked language prediction with iterative discrete labels and removes its softmax layer before supervised fine-tuning.
  • Framework and Training Recipe: The stage-wise recipe proceeds through audio-encoder SSL, speech-text SFT, context SFT, and reinforcement learning, with each stage assigned a distinct role.The converter downsamples and projects audio representations into the LLM semantic space while keeping the LLM fixed and learning the encoder and converter.
  • Context SFT: Context-aware training addresses ambiguous speech by teaching the model to use generated contextual clues, while joint beam search balances speech and context scores.Pruning first removes acoustically implausible tokens using context-independent scores to reduce hallucination.
  • Reinforcement Learning: Reinforcement learning aligns training with inference metrics such as WER, correcting the mismatch between cross-entropy training and ASR evaluation.The RL stage uses an ASR-metric-based reward after SFT and context SFT.
  • Long-form Ability: Long-form training and testing without segmentation reduce relative WER by nearly 8.8% compared with short-form training using VAD-based segmentation.The long-form test sets comprise five datasets, with maximum duration of five minutes.

4 Model and Evaluation

Seed-ASR comprises Chinese multi-dialect and multilingual models built with the same structure and training recipe, targeting diverse speech recognition scenarios. Seed-ASR (CN) covers Mandarin and 13 Chinese dialects, while Seed-ASR (ML) targets multilingual speech excluding Chinese.

  • Seed-ASR (CN) and Seed-ASR (ML) share the same model structure and training recipe while focusing on Chinese and multilingual speech, respectively.
  • Seed-ASR (CN) transcribes Mandarin and 13 Chinese dialects with one model and improves performance across multidomain, multidialect, multiaccent, public, and context-aware evaluations.Seed-ASR (ML) achieves competitive results on multilingual public and multidomain sets and is being extended to more than 40 languages.
  • WER is the main objective metric, calculated as character error for Chinese, Japanese, and Korean and as word error for English and other languages.

4.1 Seed-ASR (CN)

Seed-ASR (CN) combines large-scale encoder and LLM training with broad evaluation across public, domain, dialect, accent, hardcase, context, and intelligibility settings. It delivers strong gains over released large models and end-to-end baselines, while staged training improves complementary capabilities.

  • 4.1.1 Evaluation on Public Set: Seed-ASR (CN) achieves state-of-the-art public-benchmark performance, reducing average WER by more than 24%-40% versus published models across six Chinese test sets.The comparison includes Paraformer-Large, Qwen-Audio, and Hubert+Baichuan2.
  • 4.1.2 Evaluation on Multi-domain and Multi-source Video Set: Seed-ASR (CN) reduces WER by more than 47% relative to a strong end-to-end model on weighted multidomain evaluation, covering video, live, voice search, meetings, and intelligent assistants.It also improves performance on seven video subsets.
  • 4.1.2 Evaluation on Multi-domain and Multi-source Video Set: Seed-ASR (CN) improves proper-noun keyword F1 by 3.3 percentage points over the end-to-end baseline on ten hardcase sets.The hardcases include book titles, car names, idioms, drug names, films, ancient poems, products, and music.
  • 4.1.3 Evaluation on Multi-dialect and Multi-accent Set: The single model recognizes Mandarin and 13 Chinese dialects, with dedicated evaluations spanning 13 dialects and 11 Chinese accents.Seed-ASR (CN) improves accent performance over a strong end-to-end model, and remains strong after removing accent SFT data.
  • 4.1.4 Evaluation on Dialogue Context Set: Context information increases keyword recall by more than 15% over context-free inference, and Seed-ASR (CN) outperforms a context-FST end-to-end model on dialogue evaluation.Useful context includes conversation history, bot information, subtitle history, caption edits, and meeting participants.
  • 4.1.6 Summary: The SFT, context SFT, and RL stages improve different evaluation capabilities, with RL helping most sets and context SFT notably improving strict-context recall.The accent set shows a slight degradation after RL, attributed to the training-data ratio.

4.2 Seed-ASR (ML)

Seed-ASR (ML) extends the Seed-ASR methodology to multilingual speech using evaluations across English accents, hardcases, multidomain settings, and public datasets. It reaches top performance on most public sets, with 10%-40% improvements and generalization to unseen domains.

  • 4.2 Seed-ASR (ML): Seed-ASR (ML) is evaluated across English accents, hardcases, multidomain speech, and multilingual multidomain settings against Google USM, Whisper Large-v3, and Universal-1.These evaluations test speech variation across accents, professional vocabulary, acoustic environments, and semantic domains.
  • 4.2.2 Evaluation on Public Sets: Seed-ASR (ML) reaches top performance on most English and multilingual public test sets, with improvements ranging from 10% to 40%.The public benchmarks include LibriSpeech, MLS, TED-LIUM 3, Callhome, Switchboard, AMI, and Fleurs.
  • 4.2 Seed-ASR (ML): Seed-ASR (ML) performs strongly across diverse acoustic environments, semantic contexts, accents, and languages, including domains unseen during training.

5 Conclusion

Seed-ASR uses stage-wise SFT, context SFT, and RL to support strong recognition across domains, accents, dialects, languages, and long speech. Its LUISE-to-LLM integration supports diverse speech understanding, while context SFT improves context-aware recognition.

  • Seed-ASR delivers superior capabilities across acoustic and semantic domains, accents, dialects, languages, and long-range speech compared with strong end-to-end models.
  • Large-scale LUISE pretraining and SFT connecting LUISE with an LLM provide capacity to understand diverse speech content.
  • Context SFT improves Seed-ASR’s ability to use contextual information for recognition.

6 Authors (alphabetical order)

This section lists the paper’s authors by name.

  • The author list contains the paper’s contributors.

A.1 Detailed Results of Seed-ASR (ML)

Seed-ASR (ML) outperforms Google USM and Whisper Large-v3 across every evaluated non-English language.

  • 26%-47% WER reduction is achieved across every non-English language compared with Google USM and Whisper Large-v3.For Arabic and Indonesian, relative WER reduction exceeds 45%.

A.2 Details of English and Multilingual public test sets used in Seed-ASR (ML) evaluation

The evaluation uses English and multilingual public test sets spanning read speech, talks, conversations, meetings, and multiple languages.

  • The English evaluation includes LibriSpeech, Tedlium 3, CallHome, Switchboard, AMI IHM, and the English portion of MLS.
  • Fleurs evaluation covers eight languages after inverse text normalization of its annotated transcripts.

A.3 Training Dataset Statistics

Seed-ASR training combines large-scale speech-only self-supervised data with general ASR data for Chinese and multilingual models.

  • The statistics cover speech-only data for LUISE and general ASR data for Seed-ASR (CN) and Seed-ASR (ML).
  • Self-supervised learning uses 7.7 million hours for Chinese and 12.4 million hours for multilingual training.Mandarin comprises about 74% of Chinese data, while English comprises 51.52% of multilingual data.
  • Supervised fine-tuning uses 562k hours for Chinese and 314k hours for multilingual training.Mandarin accounts for 73.94% of Chinese fine-tuning data, while English accounts for 39.84% of multilingual data.
Loading 2407.04675v2…