SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models

1 Introduction

• Cascaded SDR pipelines can pass diarization errors into transcription, struggle with overlapping speech, and lack joint SD/ASR optimization.The paper identifies these constraints as limiting overall SDR performance.Source: §1 · p. 1
• Earlier approaches jointly trained SD and ASR or used LLMs for post-processing, but the latter remained limited by the initial module outputs and their misalignment.The passage also notes that SA-ASR systems require pre-extracted speaker embeddings.Source: §1 · p. 1
• SpeakerLM adds flexible registration for unknown, known, and redundant speaker identities, with prior speaker embeddings projected and concatenated with audio and text tokens.This is intended to adapt SDR to differing levels of speaker information availability.Source: §1 · p. 2
• SpeakerLM is presented as the first multimodal LLM designed for end-to-end SDR, using an audio encoder, two projectors, and an LLM.Its encoder-projector-LLM architecture is tailored for SDR.Source: §1 · p. 2

2.1 Speaker Diarization

• Traditional diarization combines speech-activity detection, speaker embeddings, and clustering, while conventional systems struggle with overlapping speech under single-speaker VAD assumptions.Neural speaker-activity detection is one proposed response; Pyannote and Diarizen are examples, with Diarizen adding pretrained WavLM representations.Source: §2.1 · p. 2

2.2 Speaker Diarization and Recognition

• Cascaded SD+ASR pipelines lack joint optimization and risk error propagation, while end-to-end systems jointly model the tasks.Speaker-Attributed ASR uses pre-extracted speaker embeddings and assumes they match ground-truth speakers, limiting mismatched registration settings.Source: §2.2 · p. 2

2.3 Multimodal Large Language Model

• SpeakerLM is described as the first audio-text MLLM for SDR, explicitly combining speech-content understanding with speaker-aware modeling.Unlike basic ASR or change-point detection, it explores speaker assignment in multi-speaker transcripts.Source: §2.3 · p. 2
• The comparison distinguishes anonymous speaker IDs without registration from actual speaker names when registration is enabled.The registration overview separates no-registration, exact-match, and over-registration conditions.Source: Figure 2 · p. 3, Figure 3 · p. 3

3.1 Model Architecture

• SpeakerLM projects encoded multi-speaker audio into a pretrained text LLM and projects registered-speaker embeddings into its backbone.The registered speaker’s name and special markers are tokenized; the architecture also uses a frozen speaker-embedding extractor.Source: §3.1 · p. 3

3.2 Flexible Speaker Registration Mechanism

• No-Regist outputs anonymous IDs, Match-Regist assigns registered names to all actual speakers, and Over-Regist must reject registered identities absent from the audio.The framework is intended to span anonymous through personalized speaker transcription.Source: §3.2 · pp. 3–4

3.3 Multi-Stage Training Strategy

• Training progresses from ASR-only learning to simulated SDR projector alignment, real-data acoustic adaptation, and joint fine-tuning of all modules.In the final stage, LoRA is applied to the LLM while all modules are jointly fine-tuned.Source: §3.3 · p. 4

4.1 Real Data Collection

• 238.55 hours of public audio and 7456.99 hours of in-house data support training and evaluation; AISHELL5-Eval is used to assess out-of-domain SDR generalizability.AISHELL5-Eval comes from far-field audio in a hybrid electric car with external and internal noise sources.Source: §4.1 · p. 4

4.2 Simulated Data Creation

• Simulated SDR data total 5,000 training hours and 5.6 evaluation hours, using 50-second mixtures with 2 to 4 speakers and added noise and reverberation.The source near-field samples are randomly mixed, with SNRs uniformly sampled from 10dB to 20dB.Source: §4.2 · pp. 4–5

5.1 Implementation Details

• Training used 40–50-second clips, averaged embeddings from 2–10-second registered-speaker clips, and Over-Regist training with 1–50 redundant speakers.Models trained for 1M steps at each stage on four NVIDIA A800 GPUs; audio was resampled to 16 kHz when needed.Source: §5.1 · p. 5

5.2 Evaluation Setups

• Evaluation covered in-domain AliMeeting-Eval and AISHELL4-Eval and out-of-domain AISHELL5-Eval, using cpCER and saCER as primary metrics for joint diarization and recognition performance.For no registration, cpCER uses speaker-label permutation alignment; with registered speakers, saCER aligns transcripts by speaker names without permutation search.Source: §5.2 · p. 5

5.3 Performance without Speaker Registration

• Zero-shot Qwen2.5-7B-Instruct and ChatGPT-4.5 correction degraded overall SDR performance, while fine-tuning Qwen2.5-7B-Instruct mitigated hallucinations and slightly improved performance over Diarizen-large+Para.The reported hallucinations included changes to utterance content despite instructions to modify only speaker labels.Source: §5.3 · pp. 5–6
• 6.60%, 6.56%, and 13.82% absolute cpCER improvements over the strongest cascaded system on AliMeeting-Eval, AISHELL-4-Eval, and AISHELL-5-Eval, respectively, were achieved by SpeakerLM trained with 7,638.95 hours of SDR data.On out-of-domain AISHELL5-Eval, its Δcp was 0.57.Source: §5.3 · p. 6, Table 2 · p. 6
• SpeakerLM’s performance improved progressively across four training stages on AliMeeting-Eval and Simulation-Test, while Stage 2 CER rose on AISHELL4-Eval and AISHELL5-Eval because their audio was absent from the simulated training data.The authors report that later stages using more realistic, diverse meeting-style data mitigate this domain mismatch.Source: Figure 4 · p. 6, §5.3 · p. 7

5.4 Performance with Speaker Registration

• SpeakerLM outperformed SA-Transformer by a 25.98% absolute improvement in saCER on AliMeeting-Eval; redundant registered speakers left CER and cpCER comparable but worsened Δsa relative to Match-Regist.Across increasing numbers of over-registered speakers, the authors report no significant performance degradation.Source: §5.4 · p. 7, Table 3 · p. 7

5.5 Impact of Embedding Extractors

• SpeakerLM using ERes2NetV2 embeddings outperformed its CAM++ variant on AliMeeting-Eval, indicating that improvements in the embedding model can further enhance overall performance.ERes2NetV2 was the embedding extractor used in the preceding experiments.Source: §5.5 · p. 7, Table 4 · p. 7

• SpeakerLM outperforms strong cascaded baselines built from state-of-the-art SD and ASR models on both in-domain and out-of-domain test sets.Source: §6 · p. 7
• SpeakerLM introduces a flexible registration mechanism that supports SDR with different numbers of registered speakers.Source: §6 · p. 7
• The work demonstrates the potential of MLLMs to model SDR in a scalable, data-driven, and autoregressive manner.Source: §6 · p. 7