PaperlayerView exact revision on arXiv

Source-linked AI summary

Spark-Audio 1.0 Technical Report

Chenyue Zhang, Cong Liu, Dingshu Tian, Dong Sun, Genshun Wan, Guanming Meng, Guolong Zhong, Hang Chen, Hankun Yang, Haoming Wang, Haoyang Wang, Hengquan Mei, Huaxin Wu, Jiahao Li, Jiahuan Fan, Jiamin Ma, Jianqing Gao, Jigang Tang, Jin Liu, Jingchen Li, Jinjie Fu, Kaixuan Guo, Kun Song, Li Xu, Li Zhang, Lijuan Liu, Liyan Luo, Ming Sheng, Mingen Zhao, Mingting Hu, Ningqian Wu, Pan Xu, Pei Xu, Qiangyu Tan, Rui Wang, Ruzi Li, Saier Jin, Shibo Xu, Shifu Xiong, Shihao Jiang, Tan Liu, Wei Wu, Wei Xu, Wenyu Wang, Xiangyu Jia, Xiao Li, Xiaowen Duan, Xiaoxue Guo, Xin Chen, Xinyi Liang, Xu Chen, Yanan Song, Yang Xu, Yexia Huang, Yibo Li, Yifan Xu, Yikang Liu, Yikun Li, Yiting Lin, Yi Fang, Yuxin Wang, Zan Feng, Zhenchuan Li, Zhuo Chen, Zichao Zhang, Ziwen Wang

arXiv:2610.31292v1eess.AS

TL;DR

Conventional ASR focuses on verbatim transcription, while speech foundation models seek to unify acoustic perception and linguistic reasoning. Spark-Audio 1.0 combines a general-purpose speech model with domestic training and reports comparable or better results than Qwen3.5-Omni-Flash on several tasks despite using much less audio data.

  • Problem

    Conventional ASR is optimized for verbatim transcription, while holistic speech understanding requires acoustic perception alongside linguistic reasoning.

    Source: §1 · p. 1

  • Method

    Spark-Audio 1.0 pairs a dense audio encoder with an MoE language model and is trained on domestic infrastructure using representation-enhanced BEST-RQ and staged training.

    Source: §1 · p. 1, §6 · pp. 14–15

  • Results

    Despite using one-tenth the training data of Qwen3.5-Omni-Flash, Spark-Audio 1.0 reports comparable or superior audio-task results, advantages on realistic tasks, and comparable text-task results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results demonstrate the feasibility of training a competitive speech foundation model on domestic infrastructure.

    Source: §1 · p. 1, §6 · p. 14

    Spark-Audio 1.0 still has room to improve in instruction following and complex audio reasoning.

    Source: §1 · p. 2

Abstract

from arXiv · show

This paper introduces Spark-Audio 1.0, a speech foundation model trained on domestically produced computing infrastructure. The model accepts text and speech inputs and generates text outputs. It supports multilingual (99 languages) and multi-dialect (202 dialects) speech recognition, speech translation, audio perception and understanding, and both text- and speech-based dialogue, marking a shift from merely ``hearing clearly'' to genuinely ``understanding'' speech. Architecturally, Spark-Audio 1.0 couples a 0.65B-parameter dense audio encoder with a 30B-A3B Mixture-of-Experts (MoE) large language model (LLM). The audio encoder is pretrained with representation-enhanced BEST-RQ and then refined through multi-task supervised training. The audio--LLM is trained through staged pretraining, followed by post-training that combines supervised fine-tuning with on-policy distillation. In total, the model is trained on 13 million hours of audio data together with large-scale text data. We evaluate Spark-Audio 1.0 on mainstream audio and text benchmarks, as well as on in-house test sets drawn from real-world application scenarios. Despite using substantially less audio training data than Qwen3.5-Omni-Flash, Spark-Audio 1.0 achieves comparable performance on most audio and text tasks, outperforms it in multilingual speech recognition, spoken dialogue, and real-world application scenarios, and approaches the performance of larger closed-source models. Nevertheless, there remains room for improvement in instruction following and complex audio reasoning. Finally, the successful training of Spark-Audio 1.0 on domestic hardware, including Huawei Ascend 910B and Hygon BW1000 chips, demonstrates the feasibility of a domestic technical pathway for building speech foundation models.

1 Introduction

Conventional ASR prioritizes verbatim transcription, while speech foundation models aim to unify acoustic perception with linguistic reasoning. Spark-Audio 1.0 combines that broader approach with domestic training and reports competitive results despite using substantially less audio data than a similarly sized counterpart.

1 Introduction

  • The model addresses the shift from transcription accuracy alone toward holistic speech understanding, including linguistic reasoning and paralinguistic and ambient cues.The cited motivation names emotion, speaker identity, age, gender, prosody, and ambient sound events as information conventional ASR can discard.Source: §1 · p. 1
  • Spark-Audio 1.0 pairs a 0.65B-parameter dense audio encoder with a 30B-A3B MoE LLM, using semantic-target encoder pretraining and staged audio–LLM training with SFT and OPD.The encoder is further supervised on multilingual recognition and multiple perception tasks; OPD uses a frozen text-only teacher.Source: §1 · p. 2
  • Spark-Audio 1.0 is trained end to end on domestic accelerators, including Huawei Ascend 910B and Hygon BW1000 chips.Source: §1 · p. 2
  • Using one-tenth the training data of Qwen3.5-Omni-Flash, Spark-Audio 1.0 achieves comparable or superior results on audio tasks, with advantages on realistic tasks, and comparable results on text tasks.The evaluation includes public benchmarks and in-house tests drawn from real-world application scenarios; the model still has room to improve in instruction following and complex audio reasoning.Source: §1 · p. 2
  • The model supports speech recognition across 99 languages and 202 dialects.Source: §1 · p. 2
  • The paper reports that representation-enhanced BEST-RQ encoder pretraining and its staged recipe enable a similarly sized model to match or exceed a counterpart with approximately one-tenth the data.Source: §1 · p. 2

2 Audio Encoder

The audio encoder combines shared streaming and non-streaming computation with a three-stage curriculum, progressing from teacher-informed masked prediction to multilingual ASR and broader textual supervision. Its representation-enhanced BEST-RQ targets use contextual representations from a frozen multilingual ASR teacher.

2.2 Representation-Enhanced BEST-RQ Pretraining

  • Representation-enhanced BEST-RQ quantizes contextual representations from a frozen multilingual ASR teacher, rather than low-level acoustic features, into semantic-informed discrete targets.The teacher-derived targets incorporate multilingual ASR supervision without requiring transcripts for the student pretraining corpus.Source: §2.2 · pp. 3–5

2.3 Training Curriculum and Data Scaling

  • The three-stage curriculum moves from unlabeled semantic-informed masked prediction to supervised multilingual ASR adaptation, then to multi-task training with textual targets.The largest unlabeled corpus contains approximately 13 million hours of audio; Stage II uses approximately 1.3 million hours of transcribed multilingual speech.Source: §2.3 · p. 5
  • Stage III adds speaker attributes, emotion, and sound-event recognition to ASR, expressing all five tasks through textual target sequences.The additional targets describe relevant speaker attributes, expressed emotion, or sound events in natural language.Source: §2.3 · p. 6

2.1 Overview

  • The shared streaming/non-streaming encoder is the acoustic front end, trained to capture linguistic content alongside speaker, emotional, and non-speech acoustic cues.The curriculum is designed to support downstream speech-language modeling.Source: §2.1 · p. 3, §2.3 · p. 6

3 Audio–LLM Pretraining and Post-Training

Spark-Audio’s audio–LLM uses a staged pretraining curriculum followed by supervised fine-tuning and dual-trajectory on-policy distillation. The recipe combines multilingual audio and text training with long-context tasks and paired spoken/text supervision.

3.1 Pretraining

  • The pretrained audio encoder feeds a 30B-A3B MoE language backbone through an adapter that downsamples representations from 50 Hz to 12.5 Hz.Pretraining uses adapter alignment, full-parameter joint training, and long-context extension.Source: §3.1 · p. 6

3.1.1 Pretraining Data

  • Training data span speech recognition, translation, audio understanding, and text tasks across languages, dialects, domains, and acoustic conditions.Task prompts specify transcription language, translation language pair, or dialect-ASR output format; data mixtures vary across training stages.Source: §3.1.1 · pp. 6–7

3.1.2 Multi-Stage Training

  • Pretraining progresses from adapter-only alignment to joint optimization of all components, then extends sequence length from 4K to 32K tokens.The final stage adds long-form tasks and increases maximum audio chunk duration from 30 s to 45 s.Source: §3.1.2 · pp. 7–8

3.1.3 Batch Construction and Long-Context Computation

  • Batch construction preserves source proportions, and the loss is normalized by valid target tokens across the optimizer update.Long-context training combines expert and context parallelism, distributing audio chunks before partitioning the multimodal sequence.Source: §3.1.3 · p. 8

3.2.1 Supervised Fine-Tuning

  • Supervised fine-tuning combines fundamental audio tasks with reasoning, mathematics, knowledge, and interaction data in text and speech input formats.Speech versions of cognitive-task instructions aim to narrow instruction-following and reasoning gaps between speech and text inputs.Source: §3.2.1 · p. 8

3.2.2 On-Policy Distillation

  • Dual-trajectory on-policy distillation uses a frozen text-only teacher to supervise student responses generated under audio and text conditioning.The training scope includes tens of thousands of paired spoken and textual reasoning and instruction-following inputs, without added ASR or general audio-perception targets; modest gains on some such evaluations may reflect better linguistic-information use, task interpretation, and response generation.Source: §3.2.2 · pp. 8, 10

4 Evaluation

Spark-Audio 1.0 is evaluated on public audio and text benchmarks and in-house tests drawn from real-world scenarios. Results show strong recognition on several evaluated sets, with performance varying across translation, audio understanding, spoken interaction, and text tasks.

4.1 Evaluation Setup

  • Evaluation spans public speech recognition, translation, audio understanding, spoken interaction, and text-only tasks, alongside in-house real-world tests including 202-dialect recognition and challenging acoustic conditions.ASR uses WER or CER, translation uses BLEU, and in-house spoken interaction uses full-score rate.Source: §4 · p. 10, §4.1 · p. 10

4.2 Speech Recognition

  • Spark-Audio 1.0 has the lowest reported FLEURS Chinese and English errors (1.88% CER; 2.25% WER) and 14- and 20-language averages (2.81%; 6.30%), but its 64-language average (31.89%) exceeds Qwen3.5-Omni-Plus (28.66%) and Gemini 3.1 Pro (20.09%).Source: §4.2 · pp. 10–11
  • On every in-house ASR evaluation, Spark-Audio 1.0 has the lowest reported error rate, including a 14.29% CER on the 202-dialect corpus versus 19.79% for the best comparison system.For high-noise and low-volume Chinese speech, its CERs are 37.76% and 17.17%, versus 38.66% and 23.62% for Qwen3.5-Omni-Flash; high-noise errors remain substantial.Source: §4.2 · pp. 11–12, Table 5 · p. 12, Table 6 · p. 12

4.3 Speech-to-Text Translation

  • In-house translation favors Spark-Audio 1.0, with 44.76 BLEU Chinese-to-English and 51.40 BLEU English-to-Chinese, while Qwen3.5-Omni-Plus leads on both FLEURS directions.On FLEURS, Spark-Audio 1.0 scores 28.40 and 44.54 BLEU, compared with 32.20 and 47.80 BLEU for Qwen3.5-Omni-Plus.Source: §4.3 · p. 12

4.4 Audio Understanding and Spoken Interaction

  • Audio-understanding results are competitive on AIR-Bench and MMSU, but Spark-Audio 1.0 trails all three comparison systems on MMAR, indicating a remaining complex-reasoning gap.For spoken interaction, it scores 90.35 on VoiceBench, above Qwen3.5-Omni-Flash and Gemini 3.1 Pro but below Qwen3.5-Omni-Plus; in-house instruction-following FSR also trails comparison systems.Source: §4.4 · p. 12

4.5 Text-Only Capabilities

  • On text-only benchmarks, Spark-Audio 1.0 is within 0.6 percentage points of Qwen3.5-Omni-Flash on three knowledge tests, scores higher on GSM8K and HumanEval, and lower on MATH-500.Instruction following remains weaker: its IFEval and IFBench scores are 82.19 and 36.33, versus 85.20 and 38.40 for Qwen3.5-Omni-Flash.Source: §4.5 · p. 13, Table 7 · p. 13

5 Representation Quality of the Audio Encoder

The Spark-Audio encoder is evaluated as a front end for speech recognition and audio understanding using a shared Qwen3-4B backbone. It performs strongly in multilingual recognition and sound, scene, and age prediction, with less uniform gains on emotion and gender tasks.

Experimental Setup

  • Encoder comparisons use the same Qwen3-4B backbone, with adapter-only training under shared data and settings except for starred multilingual AuT results.Those AuT FLEURS results use two-stage alignment, including joint encoder-adapter training and LoRA fine-tuning of the language model.Source: §5 · pp. 13–14

Speech Recognition

  • 8.93% and 6.17% average error rates on Chinese and English test sets, respectively, compare with AuT’s 10.47% and 7.35%.Spark-Audio also outperforms Whisper large-v3 on all 20 Chinese and English test sets and AuT on 19 of 20.Source: §5 · p. 13, Table 8 · p. 14
  • 26.24% macro-averaged error across 37 FLEURS languages is 19.41 points below Whisper large-v3 and 12.22 points below AuT with two-stage adaptation.Spark-Audio has lower error than Whisper on 36 of 37 languages and AuT on 34 of 37.Source: §5 · pp. 13–14, Table 8 · p. 14

Audio Understanding

  • 61.49% average accuracy across eight audio-understanding sets exceeds Whisper large-v3 by 28.00 points and AuT by 6.95 points.Spark-Audio achieves the highest or joint-highest accuracy on six of the eight sets.Source: §5 · pp. 13–14, Table 9 · p. 14
  • 16.72 percentage points of average accuracy separate Spark-Audio from AuT across four sound and acoustic-scene benchmarks, while emotion and gender results are less uniform.Spark-Audio also exceeds AuT by 13.50 points on age prediction, matches it at 30.77% on the in-house emotion set, and trails on CASIA emotion and gender prediction.Source: §5 · p. 14

Overall

  • The results support the encoder as a general audio-language-model front end whose representations capture semantic and acoustic information.The paper highlights multilingual recognition and gains on sound, scene, and age prediction.Source: §5 · p. 14

6 Conclusion

Spark-Audio 1.0 combines a dense audio encoder with a MoE language model and supports multilingual, multi-dialect, translation, understanding, and dialogue tasks. It is competitive with Qwen3.5-Omni-Flash on several tasks, outperforms it on multilingual recognition and spoken interaction, and retains room to improve in instruction following and complex audio reasoning.

Conclusion

  • Spark-Audio 1.0 supports speech recognition in 99 languages and 202 dialects, alongside translation, audio understanding, and text- and speech-based dialogue.It couples a 0.65B-parameter dense audio encoder with a 30B-A3B MoE language model, trained on 13 million hours of audio data with extensive text data.Source: §6 · pp. 14–15
  • Spark-Audio is competitive with Qwen3.5-Omni-Flash on Chinese–English speech-to-text translation, audio perception, and text-based dialogue, and outperforms it on multilingual recognition and spoken interaction.The conclusion reports particularly clear advantages on realistic, application-oriented tasks.Source: §6 · p. 15
  • Instruction following and complex audio reasoning remain areas for improvement.Future work also targets broader acoustic-condition and application-domain coverage, hour-long and multi-speaker recognition, and spatially aware audio understanding.Source: §6 · p. 15

7 Authors

The author list marks some individuals as having departed from the team.

Authors

  • Names marked with # denote individuals who have departed from the team.The paper expresses appreciation for their contributions.Source: §7 · p. 15

Related papers

Loading 2610.31292v1…