PaperlayerView exact revision on arXiv

Source-linked AI summary

Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio

Mohan Shi, Xiong Xiao, Ruchao Fan, Shaoshi Ling, Jinyu Li

arXiv:2511.16046v1eess.AS

TL;DR

Joint ASR and diarization targets who spoke what, a central need for understanding multi-speaker conversations. JEDIS-LLM combines short-audio training with cache-based streaming over long recordings; it outperforms strong local and long-form baselines while remaining end-to-end.

  • Problem

    Joint ASR and diarization must identify who spoke what, which is crucial for understanding multi-speaker conversations; prior Speech-LLMs addressed multi-talker ASR without diarization, and chunking long audio can cause inconsistent speaker labels.

    Source: §1 · p. 1, §2.2.1 · p. 2

  • Method

    JEDIS-LLM is an end-to-end Speech-LLM trained on short audio, using an on-the-fly Speaker Prompt Cache for chunk-wise long-form streaming and word-level speaker supervision in the speech encoder.

    Source: §1 · p. 1, §2.2.1 · p. 2, §2.1.2 · p. 2

  • Results

    JEDIS-LLM outperforms Sortformer [15] and Meta-Cat in local settings and DiarizationLM on long-form audio while remaining end-to-end and streamable.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The paper reports the first zero-shot streamable joint ASR and diarization on arbitrary long-form audio using a Speech-LLM trained only on short clips.

    Source: §1 · p. 1

    When the reasoning chain supplied the actual speaker count, cpWER did not improve and WDER improvement was limited, suggesting conversational dynamics remained uncaptured.

    Source: §7.3 · p. 6

Abstract

from arXiv · show

Joint automatic speech recognition (ASR) and speaker diarization aim to answer the question "who spoke what" in multi-speaker scenarios. In this paper, we present an end-to-end speech large language model (Speech-LLM) for Joint strEamable DIarization and aSr (JEDIS-LLM). The model is trained only on short audio under 20s but is capable of streamable inference on long-form audio without additional training. This is achieved by introducing a Speaker Prompt Cache (SPC) with an on-the-fly update mechanism during chunk-wise streaming inference, inspired by the autoregressive nature of LLMs. The SPC also allows the seamless use of pre-enrolled speaker profiles which is common in many scenarios like meeting transcription. To further enhance diarization capability, we incorporate word-level speaker supervision into the speech encoder during training. Experimental results demonstrate that our system outperforms strong baselines, including Sortformer and Meta-Cat in the local setting on audio up to 20s, and DiarizationLM on long-form audio, despite being fully end-to-end and streamable while DiarizationLM follows a cascaded offline pipeline. To the best of our knowledge, this is the first work enabling zero-shot streamable joint ASR and diarization on long audio using a Speech-LLM trained only on short audio, achieving state-of-the-art performance.

1. INTRODUCTION

Joint ASR and diarization must identify who spoke what, while existing approaches leave gaps in end-to-end, long-form streaming. JEDIS-LLM uses a speaker prompt cache to stream long audio after training only on short clips, and outperforms strong baselines in local and long-form settings.

1. INTRODUCTION

  • Sortformer [15] and Meta-Cat integrate speaker IDs but rely on separate encoders, while DiarizationLM post-processes independent systems and SpeakerLM [24] is limited to short local audio.Streaming Sortformer [17] adds a cache for global diarization but does not incorporate ASR.Source: §1 · p. 1
  • The Speaker Prompt Cache (SPC) stores a representative utterance per speaker and updates on the fly to preserve speaker consistency across streaming chunks without retraining or explicit permutation resolution.SPC also supports pre-enrolled speaker profiles.Source: §1 · p. 1
  • JEDIS-LLM is an end-to-end Speech-LLM for joint streamable diarization and ASR, trained only on short audio but designed for chunk-wise inference on arbitrary long recordings without additional training.The method also introduces word-level speaker supervision to enhance diarization capability.Source: §1 · p. 1
  • JEDIS-LLM outperforms Sortformer [15] and Meta-Cat locally and DiarizationLM on long-form audio while remaining end-to-end and streamable.The compared local setting covers audio up to 20 seconds; the model is trained only on short audio.Source: §1 · p. 1

2. METHOD

JEDIS-LLM combines a Speech-LLM trained on speaker-attributed transcriptions with word-level speaker supervision, then uses a Speaker Prompt Cache to maintain speaker consistency across streamed long-audio chunks. Fixed speaker profiles can replace the cache during inference.

2.1. Training on Short Audio

  • For long-form audio, SPC supports chunk-wise streaming inference; the figure identifies a Word-Timing Model for word timestamp alignment and notes that SPC is unnecessary for offline inference on short audio.The overall training pipeline and streaming inference with SPC are shown in Figure 1(a) and Figure 1(b), respectively.Source: Figure 1 · p. 2

2.1.1. Speech-LLM for joint ASR and Diarization

  • JEDIS-LLM uses speaker-attributed transcription as its Speech-LLM objective, arranging words in temporal order and inserting a speaker ID only when the speaker changes.The segment-level objective is adopted instead of inserting a speaker ID before every word.Source: §2.1.1 · p. 2

2.1.2. Word-level Speaker Supervision for Speech Encoder

  • Word-level Speaker Supervision replaces each transcription word with its speaker ID and trains a transformer-based Spk-Decoder to predict the resulting sequence from encoder outputs.The auxiliary Spk-Decoder is used only during training and discarded at inference; its loss is combined with the LLM token-prediction loss using a weight µ.Source: §2.1.2 · p. 2

2.2.1. SPC for Streaming Inference on Long-Form Audio

  • SPC stores an audio clip and transcription for each observed speaker, then prepends cached clips and speaker-attributed transcriptions to each chunk to preserve a consistent speaker permutation.The cache is maintained during inference; this approach avoids explicit permutation resolution.Source: §2.2.1 · p. 2

2.2.2. Seamless Integration with Speaker Profiles

  • Fixed, manually segmented speaker clips and transcriptions can replace SPC as speaker profiles, enabling retrieval of speaker names through a profile mapping and avoiding cache updates.The paper describes the fixed profiles as high-quality clips and transcriptions.Source: §2.2.2 · p. 3

3. EXPERIMENTAL SETTINGS

The experiments use five data sources totaling about 10k training hours and evaluate local short-audio and global long-form settings. Streaming evaluation uses chunk-wise inference with a Speaker Prompt Cache and reports diarization and speaker-attributed transcription metrics.

3.1. Training Datasets

  • JEDIS-LLM is trained on about 10k hours from AMI, ICSI, Fisher, internally collected data, and simulated VoxCeleb conversations.The simulated conversations mix five speakers with mild overlap.Source: §3.1 · p. 3

3.2. Training Setting

  • Training initializes the speech encoder, projector, and LLM from Phi-4-Multimodal and uses randomly segmented 15–20s clips.The passage also specifies the LoRA, Spk-Decoder, optimizer, and training schedule.Source: §3.2 · p. 3

3.3. Evaluation Setting

  • The evaluation covers local AMI and CH109 audio and an internal test set, plus global CH109 and Fisher long-form audio.Global streaming uses Oracle or VAD chunks up to 10 seconds; profile integration is tested on CH109 Test.Source: §3.3 · p. 3
  • Streaming inference processes each audio chunk, generates speaker-attributed output, and uses word timing to select non-overlapping speaker audio for cache prompts.The algorithm initializes an empty cache and sets profile-audio, text-length, and dvector-similarity thresholds; existing prompts are considered for updating when text is short or lacks punctuation, or when a longer audio segment is found.Source: §3.3 · p. 3
  • WDER and cpWER are the primary metrics, with SA-WER also reported for speaker profiles; SA-WER matches predicted IDs directly to reference IDs.Pure WER is excluded as ambiguous for multi-speaker scenarios, and SA-WER is stricter than cpWER.Source: §3.3 · pp. 3–4

4. EXPERIMENTAL RESULTS

In local evaluation, JEDIS-LLM improves cpWER over Sortformer and Meta-Cat, while in long-form evaluation its streaming system outperforms DiarizationLM. Ablations support combining segment-level LLM transcription with word-level encoder supervision, and speaker profiles narrow the gap between cpWER and SA-WER.

4.1. Evaluation on Local Setting for Short Audio

  • Combining segment-level LLM transcription with word-level speaker supervision for the encoder yields the best performance in the local-setting ablation.Removing encoder speaker supervision raises WDER; frame-level supervision improves diarization over no supervision but degrades cpWER, while word-level speaker-attributed transcription has worse cpWER than segment-level transcription.Source: §4.1 · p. 4

4.2. Evaluation on Global Setting for Long-form Audio

  • Global evaluation compares offline chunk inference plus global clustering with chunk-wise streaming that uses the Speaker Prompt Cache to maintain speaker consistency.The SPC update mechanism improves performance by refreshing the cache with higher-quality speaker prompts.Source: §4.2 · p. 4
  • JEDIS-LLM achieves better cpWER than Sortformer and Meta-Cat locally and substantially outperforms DiarizationLM with either Oracle or VAD chunks globally.Streaming also surpasses offline chunk inference with global clustering on CH109 and achieves the best Fisher WDER with Oracle chunks.Source: §4.2 · p. 4, §4.1 · p. 4

4.3. Performance of Speaker Profiles Integration

  • On CH109 long-form streaming, speaker profiles narrow the gap between cpWER and SA-WER compared with the automatically updated SPC.Profile-based IDs also map directly to real speaker names rather than index labels.Source: §4.3 · p. 4

5. CONCLUSION

JEDIS-LLM is an end-to-end Speech-LLM trained on short audio segments under 20 seconds that performs chunk-wise streaming inference on long-form audio without additional training. Its SPC update mechanism maintains speaker consistency across chunks, and the system outperforms the stated local and long-form baselines while remaining end-to-end and streamable.

5. CONCLUSION

  • JEDIS-LLM performs chunk-wise streaming inference on long-form audio without additional training despite training only on audio segments under 20 seconds.Its on-the-fly Speaker Prompt Cache update mechanism preserves speaker consistency across chunks, while word-level encoder supervision enhances diarization capability.Source: §5 · p. 4

7. APPENDIX

The appendix presents training-data statistics and examines streaming chunk and Speaker Prompt Cache lengths. It also explores Chain-of-Thought reasoning on the AMI test set.

7.1. Detail of Training Data

  • Table 4 presents statistics for the training data sets.The supplied passage does not specify the statistics or dataset values.Source: Table 4 · p. 6, §7.1 · p. 6

7.2. Different Chunk and SPC Lengths for Streaming Inference

  • Reducing SPC length to 3 seconds benefits ASR, while reducing chunk length degrades overall performance.The paper attributes the SPC result to shorter hypotheses containing fewer errors and the chunk result to fewer high-quality segments being available for on-the-fly SPC updates.Source: §7.2 · p. 6, Table 5 · p. 6

7.3. Chain-of-Thought Exploration

  • The CoT exploration prepends a reasoning chain that estimates the number of speakers before the speaker-attributed transcription.The token prediction loss is computed jointly over the reasoning chain and transcription.Source: §7.3 · p. 6
  • Even with accurate speaker counts, cpWER does not improve and WDER gains remain limited, suggesting CoT does not capture conversational dynamics.Source: §7.3 · p. 6
  • On AMI, CoT slightly improves WDER and speaker-counting accuracy, especially for utterances with four speakers; ground-truth speaker counts further reduce WDER and bring counting accuracy close to 100%.Table 6 compares JEDIS-LLM with and without CoT on AMI using WDER, cpWER, and speaker-counting accuracy.Source: Table 6 · p. 6, §7.3 · p. 6
Loading 2511.16046v1…