PaperlayerView exact revision on arXiv

Source-linked AI summary

Xiaomi-CocktailASR-1 Technical Report

Yiru Zhang, Hang Su, Lichun Fan, Ying Zeng, Chang Liu, Yifeng Wang, Yuquan Liang, Tao Li, Lian Li, Wenhao Yang, Jian Luan, Cong Zou, Heng Qu

arXiv:2609.11274v1cs.SDcs.CLeess.AS

TL;DR

The paper addresses the cocktail party problem, where ASR systems struggle to select a target speaker amid overlapping speech. Xiaomi-CocktailASR-1 uses reference speech as a voiceprint prompt in an end-to-end LLM-based TS-ASR architecture, achieving state-of-the-art multi-speaker benchmark performance while retaining single-speaker recognition, rejection, and CoT capabilities.

  • Problem

    Existing ASR and TS-ASR systems struggle with multi-speaker speech, degraded single-speaker performance, and absent-target rejection.

  • Method

    Xiaomi-CocktailASR-1 directly transcribes target speech from reference and mixed speech using an end-to-end LLM-based TS-ASR architecture with standard and CoT modes.

  • Results

    Xiaomi-CocktailASR-1 achieves state-of-the-art performance across synthetic and real-world multi-speaker benchmarks while maintaining competitive single-speaker recognition and rejection capability.

  • Takeaways & Limitations

    A unified architecture can combine target-speaker recognition, single-speaker recognition, negative-sample rejection, and interpretable CoT reasoning.

Abstract

from arXiv · show

Recently, large language model (LLM) based ASR models have achieved significant progress, yet they generally lack support for multi-speaker scenarios, where the cocktail party problem remains a critical bottleneck for further advancing ASR. Existing TS-ASR methods, including end-to-end architectures with speaker embeddings and latest LLM-based explorations suffer from degraded single-speaker performance and the inability to reject when the target speaker is absent. In this paper, we propose Xiaomi-CocktailASR-1, an LLM-based end-to-end TS-ASR architecture. By utilizing reference speech as voiceprint prompts, it directly transcribes the target speaker's speech without requiring speech separation. Xiaomi-CocktailASR-1 maintains competitive performance in single-speaker scenarios, comparable to mainstream ASR models. It also features a negative sample rejection capability, outputting empty text when the target speaker is absent from the mixed speech. Additionally, Xiaomi-CocktailASR-1 supports a Chain-of-Thought (CoT) reasoning mode to provide explicit reasoning steps. Extensive experiments on various synthetic and real-world multispeaker benchmarks demonstrate that Xiaomi-CocktailASR-1 achieves state-of-the-art performance, effectively addressing the cocktail party problem through a unified architecture that balances multispeaker and single-speaker recognition accuracy, along with rejection capability.

1 Introduction

Xiaomi-CocktailASR-1 addresses the cocktail party problem with an end-to-end LLM-based TS-ASR system designed to preserve single-speaker accuracy, reject absent targets, and support explicit reasoning.

  • Background: The cocktail party problem requires selectively focusing on one speaker in complex multi-speaker speech, which TS-ASR addresses using target-speaker reference information.Unlike MT-ASR, TS-ASR is intended to associate transcription with a specified speaker identity.
  • Motivation: Existing TS-ASR systems degrade in single-speaker cases and often transcribe irrelevant speech when the target speaker is absent.These limitations motivate a system that handles varying speaker counts and outputs empty text for target absence.
  • Proposed approach: Xiaomi-CocktailASR-1 uses reference speech as a voiceprint prompt to directly transcribe target speech without explicit separation.Its unified architecture handles single-speaker recognition, multi-speaker target-speaker ASR, and negative-sample rejection.
  • Contributions: The model provides CoT reasoning that generates explicit intermediate steps for the recognition process.The introduction presents CoT reasoning alongside rejection and unified recognition as core capabilities.
  • Contributions: Xiaomi-CocktailASR-1 achieves competitive single-speaker accuracy, rejection capability, CoT reasoning, and state-of-the-art results across simulated and real-world multi-speaker benchmarks.These capabilities are presented as the paper’s main contributions.

2 Architecture

The architecture concatenates reference and mixed speech, encodes them jointly, aligns audio features with text, and uses an LLM to generate the target-speaker transcription.

  • Architecture: Reference and mixed speech are concatenated and passed through an Audio Encoder, Adapter, and LLM to generate target-speaker transcription.The encoder produces frame-level embeddings containing speech and speaker information before cross-modal alignment and language-model decoding.
  • Input construction: A one-second silence segment separates one-to-four-second reference speech from the mixed signal and anchors voiceprint features.The reference duration is chosen to provide voiceprint information without significant computational cost.
  • Audio encoding: The Audio Encoder uses FBank inputs and SSL-derived embeddings to combine semantic and speaker information while avoiding a separate speaker encoder.Reference audio conditions feature extraction so the model focuses on the target and suppresses irrelevant speakers.
  • Cross-modal alignment: A lightweight Adapter maps audio embeddings into the LLM hidden space, where aligned audio and text embeddings are jointly processed.The LLM backbone is based on Qwen3-8B.

3 Training

Training uses multi-speaker, single-speaker, negative, and CoT data in a staged pipeline to balance target extraction, ordinary recognition, rejection, and interpretability.

  • 3.1.1 Multi-speaker Recognition: Multi-speaker training mixes real and synthetic overlaps with probabilistic environmental and human-speech noise to improve robustness.The augmentation produces random overlap patterns and diverse acoustic interference.
  • 3.1.2 Single-speaker Recognition: Single-speaker data from different utterances of the same speaker is balanced with multi-speaker data to reduce over-suppression.This targets the trade-off between preserving clean target speech and suppressing interference.
  • 3.1.3 Negative Sample Rejection: Negative-sample training uses mismatched reference speech and progressively increases its sampling proportion so absent targets produce empty text without harming normal recognition.The rejection capability is internalized in model weights and requires no additional inference threshold.
  • CoT data: CoT templates encode speaker count, gender, and reference-similarity levels for single-speaker, multi-speaker, and negative examples.Similarity is discretized into five levels, while a threshold of 3 separates likely target presence from absence.
  • 3.2 Training pipeline: The training pipeline progressively develops standard ASR, target-speaker ASR, and CoT reasoning through three stages.The stages first establish speech-to-text ability, then target-speaker recognition, and finally explicit reasoning through joint training.

4.1 Benchmarks

The evaluation covers synthetic mixtures, real-world far-field recordings, single-speaker speech, and negative samples to assess recognition, robustness, preservation of clean-speech accuracy, and rejection.

  • Evaluation protocol: Each speaker in a synthetic mixture is alternately treated as the target during evaluation to support objective and fair testing.This protocol tests target selection across speakers rather than privileging one mixture position.
  • Synthetic datasets: Synthetic multi-speaker benchmarks randomize speaker energy and start times to prevent loudness or ordering shortcuts.LibriMix uses 0–15 dB random SNRs, while LibriSpeechMix randomizes speaker starting times.
  • Real-world datasets: Real-world benchmarks use far-field recordings with overlapping speech, reverberation, noise, and unpredictable interference conditions.AMI-SDM and AliMeeting-Far provide English and Chinese meeting scenarios with challenging acoustics.
  • Single-speaker datasets: Single-speaker datasets test whether the model preserves recognition accuracy without deletion errors from over-suppression.They include clean English speech and near- or headset-microphone meeting recordings.
  • Negative datasets: Negative test sets contain only interfering speakers and evaluate whether the model outputs empty text instead of irrelevant transcriptions or hallucinations.The sets cover English, Chinese, and real-world Chinese scenarios.

4.2 Evaluation Metrics

The evaluation uses target-speaker error rates, rejection accuracy, false rejection, and non-empty WER to measure recognition and rejection behavior.

  • TS-WER and TS-CER measure target-speaker transcription errors on English and Chinese test sets, respectively.
  • Rejection Rate measures correctly rejected negative samples, with higher values indicating better non-target rejection.
  • False Rejection Rate measures incorrectly rejected positive samples, so lower values indicate less degradation when the target is speaking.
  • Non-empty WER excludes empty-output samples to isolate transcription accuracy when the model produces text.

4.3 Results

Xiaomi-CocktailASR-1 achieves state-of-the-art performance across synthetic and real-world multispeaker benchmarks, while retaining strong single-speaker accuracy, rejection capability, and CoT benefits.

  • 4.3.1 Target Speaker ASR in multi-talker senorial: Xiaomi-CocktailASR-1 achieves state-of-the-art performance across the evaluated synthetic and real-world multispeaker benchmarks.The comparison is reported in Tables 1 and 2.
  • 4.3.1 Target Speaker ASR in multi-talker senorial: On AMI-SDM, Xiaomi-CocktailASR-1 reaches 21.81% TS-WER versus 22.0% for SQ-Whisper, while on AliMeeting-Far it reaches 20.63% versus 27.5% for MC-TS-ASR.
  • 4.3.2 Single Speaker ASR: Xiaomi-CocktailASR-1 remains competitive on single-speaker data, achieving 8.89% WER on AMI-IHM and 1.73% on LibriSpeech.It also reports 6.57% and 5.81% WER on AliMeeting-Near and WenetSpeech(meeting), respectively.
  • 4.3.2 Single Speaker ASR: The model balances rejection with target-present recognition, attaining a 0.36% FRR on LibriSpeech while Gemini-2.5-pro reaches 21.31%.
  • 4.3.3 Negative Rejection: Its negative-sample rejection rates are 79.59% on LibriSpeech Neg, 75.35% on Aishell Neg, and 68.54% on a real-world Chinese in-house set.Mainstream ASR baselines Qwen3-ASR-1.7b and StepAudio2 report 0% rejection rates.
  • 4.3.4 Chain-of-Thought: CoT reasoning modestly improves overall WER, including a 0.24% reduction on LibriMix 2mix.The generated reasoning also provides estimated speaker counts and temporal activity patterns.

5 Conclusion

Xiaomi-CocktailASR-1 is an LLM-based end-to-end TS-ASR architecture that uses reference audio as a voiceprint prompt to transcribe target speech without explicit separation. It achieves state-of-the-art performance on synthetic and real-world multispeaker benchmarks while supporting rejection, CoT reasoning, and competitive single-speaker recognition.

  • Xiaomi-CocktailASR-1 achieves state-of-the-art performance on various synthetic and real-world multi-speaker benchmarks.
  • The model directly transcribes target speech from reference audio used as a voiceprint prompt, without explicit speech separation.
  • It combines negative sample rejection and CoT reasoning capabilities with competitive recognition performance in single-speaker scenarios.
Loading 2609.11274v1…