Source-linked AI summary
The Second MLC-SLM Challenge: Multilingual Conversational Speech Diarization, Recognition, and Understanding
Bingshen Mu, Mingchen Shao, Zhennan Lin, Liumeng Xue, Hexin Liu, Lei Xie, Eng Siong Chng, Longshuai Xiao, Qiangze Feng, Daliang Wang
TL;DR
Robust multilingual conversational speech models require real-world data that reflects natural communication, while transcription alone does not measure whole-conversation understanding. The second MLC-SLM Challenge expands the task and dataset scope and summarizes participating systems; its results show effectiveness for end-to-end and cascaded recognition systems, while identifying audio-dependent supervision and task-specific optimization as important for understanding.
Problem
Robust multilingual conversational speech models need real-world corpora reflecting natural communication, and transcription accuracy alone does not assess acoustic events, speaker relationships, and conversation-level meaning.
Source: §1 · p. 1Method
The paper presents the challenge’s multilingual conversational dataset, two task settings, evaluation protocols, baselines, and representative participating systems.
Source: §1 · p. 1, §5 · p. 2, §5.3 · p. 3Results
The challenge results show both end-to-end and cascaded systems are effective for diarization and recognition, while audio-dependent supervision and task-specific optimization are important for understanding.
Source: §7 · p. 4Takeaways & Limitations
The challenge resources and findings are intended to facilitate further research on robust multilingual conversational SLLMs.
Source: §7 · p. 4Evaluation rules prohibit using the Eval set for model development and prohibit multi-system output fusion; commercial APIs cannot generate Eval predictions.
Source: §2 · p. 2
Abstract
from arXiv · showhide
This paper summarizes the Interspeech2026 second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge, which aims to advance the development of effective multilingual conversational speech language models. We describe the two challenge tasks: multilingual conversational speech diarization and recognition, and multilingual conversational speech understanding, together with the released real-world conversational speech dataset, evaluation protocols, and baseline systems. The challenge attracted 91 teams worldwide, with 704 valid leaderboard results and 14 technical reports across the two tasks. Based on the participating systems, we summarize representative approaches and distill practical insights into multilingual conversational speech recognition and understanding to support future research in the community.
1. INTRODUCTION
The paper argues that robust multilingual conversational speech models need real-world data reflecting the complexity of natural communication. The second MLC-SLM Challenge broadens the earlier challenge from transcription to speaker-aware recognition and holistic spoken-language understanding.
1. INTRODUCTION
- Real-world multilingual conversational corpora remain scarce despite the complexity of natural communication, including turns, overlap, accents, acoustic variation, and long-range context.These challenges motivate the need for data that reflects natural conversations.Source: §1 · p. 1
- The first MLC-SLM Challenge advanced multilingual conversational recognition, but determining who speaks when remained difficult and transcription alone did not assess whole-conversation understanding.The identified gaps include acoustic events, speaker relationships, and semantic content.Source: §1 · p. 1
- The second challenge expands linguistic coverage and task scope to speaker-aware recognition and holistic spoken-language understanding.It adds Tagalog, Urdu, and Turkish, and extends regional variety coverage.Source: §1 · p. 1
- The challenge releases around 2,100 hours of two-speaker conversations across 14 languages and 21 language and accent variants, and draws 91 teams with 704 valid results and 14 technical reports.The paper describes the dataset, task settings, evaluation protocols, baselines, and representative approaches.Source: §1 · p. 1
2. TASK SETTINGS AND RULES
The challenge defines two tasks using complete multilingual conversations: joint speaker-attributed transcription and multiple-choice conversational understanding. It permits both cascaded and end-to-end systems, subject to rules against evaluation leakage and multi-system output fusion.
2. TASK SETTINGS AND RULES
- Task 1 requires systems to jointly identify speakers, assign time boundaries, and transcribe speech from unsegmented recordings without oracle speaker labels or utterance boundaries.Systems are evaluated using DER and tcpWER or tcpCER, with tcpMER determining final ranking and lower values indicating better performance.Source: §2 · pp. 1–2
- Task 2 measures understanding of acoustic, semantic, and joint acoustic-semantic information through multiple-choice questions about complete conversations.Evaluation provides no oracle segmentation, timestamps, or speaker labels; performance is answer accuracy, and Phase 2 determines final ranking.Source: §2 · p. 2
- Both tasks require LLM-, SLLM-, or multimodal-LLM-based systems; public external resources and a single multi-component pipeline are allowed, but evaluation-set use and multi-system output fusion are prohibited.Commercial APIs may be used only to construct Task 2 training data, not to generate evaluation predictions.Source: §2 · p. 2
3. RELEASED DATASET
The released corpus contains approximately 2,100 hours of natural two-speaker conversations across 14 languages and 21 language and accent variants. Train and Dev materials include transcription and speaker-timing annotations, while evaluation recordings omit oracle labels.
3. RELEASED DATASET
- The corpus contains 7,080 natural two-speaker conversations recorded at 16 kHz, mainly in quiet indoor settings using mobile devices.Conversations focus on randomly assigned topics and include speakers of diverse ages and genders.Source: §3 · p. 2
- Coverage spans 14 languages and 21 language and accent variants, adding Tagalog, Urdu, Turkish, Canadian French, Brazilian Portuguese, and Mexican Spanish relative to the first challenge.English varieties include American, British, Filipino, Australian, and Indian English.Source: §3 · p. 2
- Train and Dev sets provide transcriptions, oracle segmentation, timestamps, and speaker labels, while evaluation recordings omit these annotations and reference transcriptions.Task 2 uses the same conversational recordings but has no dedicated multiple-choice Train set.Source: §3 · p. 2
- Task 2 Dev contains 150 long-form conversations and 4,500 questions, and its Phase 2 evaluation contains 9,470 questions reviewed for clarity and a unique correct answer.Questions assess acoustic, semantic, and joint acoustic-semantic understanding.Source: §3 · p. 2
4. BASELINES
The paper establishes separate baselines for diarization and recognition and conversational understanding. The speech baseline fine-tunes VibeVoice-ASR, while the understanding baseline fine-tunes Qwen2.5-Omni-7B and obtains 35.33% accuracy on a development subset.
4. BASELINES
- 79.15% average tcpMER is achieved by the VibeVoice-ASR Task 1 baseline on Dev.It uses a frozen speech encoder and LoRA on the Qwen2.5-7B language model; Japanese, Korean, and Thai use tcpCER, and other languages use tcpWER.Source: §4 · p. 2
- 35.33% overall accuracy is achieved by the Qwen2.5-Omni-7B Task 2 baseline on a 300-question development subset.The baseline directly processes the conversation, question, and candidate answers to generate the selected option.Source: §4 · p. 2
- 47.75% acoustic, 33.67% joint acoustic-semantic, and 21.98% semantic accuracy are reported for the Task 2 baseline.The authors note that conversation-level semantic reasoning remained particularly challenging.Source: §4 · p. 2
- The Task 2 baseline uses Gemini2.5-Pro-generated questions and LoRA fine-tuning with the ms-swift framework and Megatron-LM backend.A retry mechanism and sampling strategy are used to improve answer extraction.Source: §4 · p. 2
5. METHODS OF TASK 1
Task 1 systems span end-to-end, cascaded, and intermediate designs; both end-to-end and cascaded systems achieve the best evaluation results. The reported approaches address speaker attribution, training, data quality, and long-form inference.
5.1. System Architecture
- End-to-end systems generate timestamps, speaker labels, and transcriptions jointly, while cascaded systems connect diarization and ASR modules; both approaches remain competitive.Intermediate systems pass explicit speaker and temporal information to the ASR model.Source: §5.1 · p. 2
5. METHODS OF TASK 1
- Both end-to-end and cascaded systems achieve the best evaluation results, indicating that performance also depends on speaker modeling, temporal alignment, training-data quality, and long-form inference.Table 1 summarizes results and techniques for the top-ranked Task 1 systems with valid technical reports.Source: §5 · p. 2, Table 1 · p. 3
5.2. Speaker Diarization and Attribution
- In cascaded systems, in-domain diarization adaptation and speaker-embedding refinement substantially reduce final recognition error.Systems use local-to-global diarization, then assign recognized tokens to speaker segments by temporal overlap or nearest segment.Source: §5.2 · p. 3
5.3. Training and Model Adaptation
- Systems use full fine-tuning, LoRA, or multistage training; SQZ’s ablations attribute most improvement to supervised fine-tuning, with smaller additional gains from synthetic-data adaptation and reinforcement learning.SQZ also uses GRPO with recognition- and hallucination-aware rewards.Source: §5.3 · p. 3
5.4. Data Augmentation and Data Quality
- Teams augment speech and address annotation mismatches by recovering reliable missing transcripts, silencing uncertain regions, and masking audio outside annotated intervals.These approaches reduce audio-supervision mismatch and help prevent deletion errors or spurious transcriptions.Source: §5.4 · p. 3
5.5. Long-form Inference and Post-processing
- Robust long-form decoding and accurate speaker–time–text alignment are more decisive than choosing end-to-end versus cascaded architecture alone.Reported strategies include periodic time markers, overlapping chunks with speaker-identity voting, timestamped recognition, conservative correction, and repetition handling.Source: §5.5 · p. 3
6. METHODS OF TASK 2
Task 2 systems differ in how they represent conversational evidence, construct supervision, adapt models, and control answers. Reported findings highlight question-conditioned evidence, audio-dependent supervision, and answer extraction.
6.1. Evidence Representation and System Architecture
- Systems represent conversational evidence using audio-native models, speaker-attributed transcripts, question-dependent audio, or training-free in-context learning.Table 2 summarizes five Task 2 systems with available technical reports.Source: §6 · p. 3, Table 2 · p. 4
- Xdynamics’ development accuracy rises from 92.69% to 94.22% when localized audio is added to the transcript backbone, and its final hybrid router achieves 94.84%.Eloquence’s frozen Voxtral-24B model reaches 81.0% Phase 2 accuracy with six multimodal demonstrations, versus 74.3% without demonstrations.Source: §6.1 · p. 4
6.2. Task-specific Supervision Construction
- Teams construct task-specific supervision from released recordings; Roysun2006’s cumulative results improve from 81.0% before audio-dependency filtering to 83.0% after filtering and 85.0% after distribution-matched augmentation.The results indicate that synthetic supervision’s relevance and acoustic dependence matter more than its raw size.Source: §6.2 · p. 4
6.3. Model Adaptation and Learning Objectives
- ClaudeAKE’s routing-based training achieves 90.92%, compared with 88.24% for a control system trained without separating weak and strong examples.The two highest-ranked systems combine supervised fine-tuning with task-specific reinforcement learning.Source: §6.3 · p. 4
6.4. Inference and Answer Control
- Inference varies between complete recordings and question-dependent evidence, while strict answer formatting and candidate-set checks support reliable answer extraction.The reported findings identify audio-dependent supervision, question-conditioned evidence selection, and answer extraction as central to conversational speech understanding.Source: §6.4 · p. 4
7. CONCLUSION
The paper summarizes the second MLC-SLM Challenge and its dataset, tasks, baselines, and representative submissions. It reports results for both recognition and understanding, while noting limits on cross-system comparisons.
7. CONCLUSION
- Challenge results demonstrate the effectiveness of both end-to-end and cascaded systems for multilingual conversational speech diarization and recognition.For conversational speech understanding, the paper identifies audio-dependent supervision and task-specific optimization as important.Source: §7 · p. 4
- Cross-system comparisons are observational because submitted systems differ in model scale, external data, synthetic supervision, and computational resources.The paper recommends controlled-data tracks and reporting by language, overlap condition, and question category for finer-grained, more reproducible comparisons.Source: §7 · p. 4
- The paper presents the challenge resources and findings as support for further research on robust multilingual conversational SLLMs.Source: §7 · p. 4