Source-linked AI summary
HINTT Submission to the 2nd MLC-SLM Challenge: Comparing Cascaded and Unified Approaches to Diarization and ASR
Takanori Ashihara, Kohei Matsuura, Masato Mimura
TL;DR
Speaker-attributed ASR must establish who spoke when and what was said, and the paper compares cascaded and unified approaches for multilingual conversational speech. Its HINTT submission combines fine-tuned diarization and ASR with LLM-based correction; under the challenge conditions, the cascade was more reliable than the chunk-fine-tuned unified system.
Problem
Multilingual conversational ASR must jointly identify speaker turns, assign utterances to speakers, and transcribe speech, including without oracle boundaries or speaker labels in MLC-SLM Task 1.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Modular diarization–ASR pipelines remain practical solutions under current challenge conditions, especially when unified models cannot be fine-tuned on full-session audio.
Source: §4 · p. 4Full-session fine-tuning was computationally infeasible, so the unified model was fine-tuned chunk-wise and independently decoded chunks often had inconsistent speaker labels.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
This paper presents the HINTT system submitted to the 2nd Challenge and Workshop on Multilingual Conversational Speech Language Model (MLC-SLM). We address multilingual speaker-attributed ASR, where systems must determine who spoke when and what was spoken. We investigate two modeling strategies for this problem: a cascaded pipeline that combines speaker diarization with speech-LLM-based ASR, and a unified speech LLM that directly generates speaker labels, timestamps, and transcriptions. Our final submission is based on the cascaded pipeline, consisting of a fine-tuned DiariZen diarization model, a fine-tuned Qwen3-ASR model, and LLM-based generative error correction. For comparison, we also fine-tune VibeVoice-ASR as a unified model using the same official training data. All task-specific fine-tuning and model selection are performed using only the official MLC-SLM data, without external data or pseudo-labels. Experimental results demonstrate that the cascaded system remains more reliable under the MLC-SLM Task 1 conditions, while unified speech LLMs offer a promising direction for future speaker-attributed ASR.
1. Introduction
Speaker-attributed ASR must determine who spoke when and what was said, a capability needed for several conversational-speech applications. The paper compares cascaded and unified approaches because it remains unclear whether unified models can match carefully tuned cascades in MLC-SLM Task 1.
1. Introduction
- Speaker-attributed transcription jointly identifies speaker turns, assigns utterances to speakers, and transcribes speech, supporting meeting transcription, conversation analysis, retrieval, and downstream understanding.The paper situates this joint problem in prior benchmarks and challenges, including CHiME.Source: §1 · p. 1
- MLC-SLM Task 1 requires multilingual diarization and recognition without oracle utterance boundaries or speaker labels, and ranks systems by average tcpWER/tcpCER across languages.The setting includes multilingual and accented speech in natural two-speaker conversations.Source: §1 · p. 1
- Cascaded systems use separate diarization and ASR modules that can be optimized, replaced, and analyzed independently, and have remained strong practical baselines in prior MLC-SLM work.The passage also notes their use in CHiME challenge submissions.Source: §1 · p. 1
- The paper compares its cascaded system with unified speech LLMs such as VibeVoice-ASR [14], whose ability to match carefully tuned cascades in MLC-SLM Task 1 remains underexplored.Unified models generate speaker labels, timestamps, and transcriptions together [13] [14].Source: §1 · p. 1
- The HINTT submission uses fine-tuned DiariZen for diarization, fine-tuned speech-LLM ASR for segment transcription, and LLM-based generative error correction; its cascade was more reliable under challenge conditions.The comparison also fine-tunes VibeVoice-ASR as a unified counterpart; it was trained only on about 60-second chunks due to computational constraints.Source: §1 · p. 1
2. System overview
The system overview contrasts separate diarization and ASR modules with a unified speech LLM that generates speaker-attributed transcriptions. The submitted cascade combines fine-tuned DiariZen and Qwen3-ASR with LLM-based correction, while the unified comparison uses chunked, prefix-conditioned VibeVoice-ASR.
2. System overview
- The overview compares a cascade of separate diarization and ASR models with a unified speech LLM that predicts speaker labels, timestamps, and transcriptions.Training and validation use only official challenge splits; silence masking replaces audio outside annotated speech intervals to reduce dataset mismatch.Source: §2 · p. 2
2.1. Submitted cascaded system
- The cascade uses fine-tuned DiariZen to estimate speaker-labeled segments, then fine-tuned Qwen3-ASR to transcribe each segment, separating “who spoke when” from “what was spoken.”This modular setup permits independent optimization and segment-level ASR without full-session training; speaker clustering fixes the count at two for these recordings.Source: §2.1 · p. 2
- Qwen3-ASR decoding retains five hypotheses for GEC, and detected repetition loops trigger a retry with progressively higher sampling temperatures.Detection checks repeated character substrings or word n-grams; the recovery path was triggered for only a few evaluation utterances.Source: §2.1 · p. 2
- LLM-based GEC receives session-level 5-best Qwen3-ASR hypotheses with normalized beam scores and selects alternatives only when conversational context supports them.It makes minimal corrections for shared phonetic errors and leaves the top hypothesis unchanged when uncertain.Source: §2.1 · pp. 2–3
2.2. Unified comparison approach
- Full-session VibeVoice-ASR fine-tuning was computationally infeasible, and independently decoded overlapping chunks often assigned inconsistent speaker labels.The model was fine-tuned on approximately 60-second chunks based on oracle utterance boundaries.Source: §2.2 · p. 3
- The unified system conditions each chunk on prior VibeVoice-ASR JSON segments, carrying forward timestamps, speaker labels, and transcriptions to continue from established context.Its inference uses overlapping chunks and discards each chunk’s final segment before committing outputs because that segment may be truncated at the boundary.Source: §2.2 · p. 3
- VibeVoice-ASR repetition loops are handled first by changing the random seed without sampling, then by enabling temperature-based sampling if the loop persists.The seed retry can alter generation because the model applies random perturbation before its acoustic representations reach the LLM.Source: §2.2 · p. 3
3. Experimental results
The cascaded system delivered the strongest reported average results, while diarization fine-tuning reduced segmentation errors and speaker confusion remained dominant.
Diarization results
- DiariZen-FT substantially reduced average DER versus the baseline and original DiariZen, and D4 further improved average DER.VibeVoice-ASR-FT had higher average DER than D4 despite leading for several languages or locales.Source: §3 · p. 3, Table 1 · p. 4
Overall results
- 15.28% was the best development-set score, achieved by final system S4 after SpecAugment and GEC post-processing.Qwen3-ASR-FT improved on the official baseline, and SpecAugment further improved the score.Source: §3 · p. 4, Table 2 · p. 4
Diarization results
- DiariZen-FT reduced missed speech and false alarms versus DiariZen, while D4 further reduced speaker confusion, which remained dominant.Table 3 breaks DER down into missed speech, false alarms, and speaker confusion.Source: §3 · pp. 3–4, Table 3 · p. 4
Overall results
- 16.70% fell to 16.39% with SpecAugment and to 16.27% with GEC in the evaluation-set leaderboard results.Both additions consistently contributed to final system performance.Source: §3 · p. 4
- Unified VibeVoice-ASR-FT performed better in some languages or locales but had poorer, less stable average performance than the cascade.Fine-tuning used only 60-second chunks, so the authors identify full-session fine-tuning as future work.Source: §3 · p. 4
4. Conclusions
The paper compares cascaded and unified approaches for multilingual speaker-attributed ASR and submits a cascaded system. Under its computationally constrained fine-tuning setup, that system was more reliable, while unified speech LLMs remain promising for full-session fine-tuning.
- The submitted HINTT system combines fine-tuned DiariZen diarization, fine-tuned Qwen3-ASR, and LLM-based generative error correction.Source: §4 · p. 4
- Task-specific fine-tuning and model selection used only official MLC-SLM data, without external data or pseudo-labels.Source: §4 · p. 4
- Under the computationally constrained fine-tuning setup, the cascaded system was more reliable than the unified system.Source: §4 · p. 4
- Unified speech LLMs remain promising if reliable long-form speaker-attributed labels and sufficient resources permit full-session fine-tuning.Source: §4 · p. 4
5. Generative AI Use Disclosure
The disclosure limits the LLM's manuscript role to language editing and confirms author responsibility for the submission.
- An LLM was used only for limited language editing focused on grammar and readability.Source: §5 · p. 5
- The LLM did not generate scientific content or produce a substantial part of the manuscript.Source: §5 · p. 5
- All authors accept responsibility and accountability for the paper and consent to its submission.Source: §5 · p. 5