Source-linked AI summary
Summary on The Multilingual Conversational Speech Language Model Challenge: Datasets, Tasks, Baselines, and Methods
Bingshen Mu, Pengcheng Guo, Zhaokai Sun, Shuai Wang, Hexin Liu, Mingchen Shao, Lei Xie, Eng Siong Chng, Longshuai Xiao, Qiangze Feng, Daliang Wang
TL;DR
The paper addresses the scarcity of realistic multilingual conversational speech data for robust spoken dialogue systems. It summarizes the MLC-SLM Challenge, its dataset, tasks, baselines, and submissions, finding diverse effective solutions alongside limited benefits from straightforward multilingual adaptation, context use, and speaker-temporal processing.
Problem
Real-world multilingual conversational speech data is scarce, limiting development of robust multilingual spoken dialogue systems.
Method
The paper describes the challenge's multilingual dataset, task settings, baseline systems, and participant methods across recognition and diarization-recognition tasks.
Results
The challenge produced diverse effective solutions in architecture, training, and augmentation, while straightforward multilingual adaptation, context use, and speaker-temporal processing did not yield expected benefits.
Takeaways & Limitations
Further exploration is needed for multilingual adaptation, conversation context utilization, and speaker-temporal processing in speech diarization.
Takeaways & Limitations
Evaluation subsets could not be used for model training or fine-tuning, and submissions had to come from a single system without fusion.
Abstract
from arXiv · showhide
This paper summarizes the Interspeech2025 Multilingual Conversational Speech Language Model (MLC-SLM) challenge, which aims to advance the exploration of building effective multilingual conversational speech LLMs (SLLMs). We provide a detailed description of the task settings for the MLC-SLM challenge, the released real-world multilingual conversational speech dataset totaling approximately 1,604 hours, and the baseline systems for participants. The MLC-SLM challenge attracts 78 teams from 13 countries to participate, with 489 valid leaderboard results and 14 technical reports for the two tasks. We distill valuable insights on building multilingual conversational SLLMs based on submissions from participants, aiming to contribute to the advancement of the community.
1. INTRODUCTION
The challenge addresses the scarcity of real-world multilingual conversational speech data by releasing a large dataset and organizing two tasks for multilingual conversational SLLMs. It also consolidates participant results and methods to provide development insights.
- The MLC-SLM Challenge releases real-world multilingual conversational speech data to advance effective multilingual conversational SLLMs.It targets the data scarcity that limits robust spoken dialogue systems, especially in multilingual settings.
- Task 1 evaluates multilingual conversational speech recognition, while Task 2 evaluates multilingual conversational speech diarization and recognition.
- The challenge attracted 78 teams from 13 countries, producing 489 valid leaderboard results and 14 technical reports.
- The paper describes the dataset, task setups, baseline systems, and participant methods to distill insights for building multilingual conversational SLLMs.
2. TASK SETTINGS AND RULES
The challenge evaluates SLLM-based systems on multilingual conversational recognition, with separate settings for oracle-assisted recognition and unconstrained diarization plus recognition. It permits external resources and augmentation but prohibits evaluation-set training and system fusion.
- Task 1 measures multilingual conversational speech recognition using oracle segmentation and speaker labels, with WER or CER selected by language.CER is used for Japanese, Korean, and Thai; WER is used for the other languages.
- Task 2 requires joint speech diarization and recognition without pre-segmented utterances or speaker labels, evaluated with tcpWER or tcpCER.
- Participants may use public external datasets, pretrained models, and training-subset augmentation such as noise, reverberation, speed perturbation, or tone modification.
- Evaluation subsets cannot be used for model training or fine-tuning, and submitted results must come from a single unfused system.
3. RELEASED DATASET
The released MLC-SLM dataset contains approximately 1,604 hours of multilingual conversational speech across training, development, and two evaluation subsets. It covers 11 languages and provides progressively different levels of segmentation, speaker-label, and transcription access.
- The dataset contains approximately 1,604 hours: 1,507 training hours, 32 development hours, and 32 hours for each evaluation subset.Train and Dev are shared by both tasks.
- Train and Dev provide oracle segmentation, speaker labels, and transcriptions; Eval-1 provides segmentation and speaker labels, whereas Eval-2 provides neither.
- All subsets cover 11 languages, while English includes American, Australian, British, Filipino, and Indian accents.
- Each recording is a roughly 20-minute, two-speaker, multi-turn conversation on a randomly assigned topic, with diverse speaker ages and genders.
4. BASELINES
The baseline systems use a standard speech-encoder/projector/LLM architecture for Task 1 and a diarization-recognition cascade for Task 2. Baseline results reveal that vanilla Whisper outperforms the tested SLLMs, while Task 2 suffers from short-segment fragmentation.
- The Task 1 baseline combines a Whisper-large-v3 encoder, a convolutional and linear projector, and Qwen2.5-7B or Llama3.1-8B decoding.It uses a two-stage training strategy.
- The Task 2 baseline cascades 3D-Speaker diarization with the Task 1 pretrained SLLM for recognition.Its three-stage strategy fine-tunes overlap detection, infers diarization, and then performs recognition.
- For Task 2, diarization without overlap detection performs better because dataset overlaps are too brief, while short baseline segments produce high tcpMER.
5. METHODS OF TASK 1
Task 1 submissions use diverse SLLM architectures, training schedules, multilingual adaptation methods, augmentation strategies, and conversation-context techniques to improve multilingual conversational ASR.
- 5.1. SLLM Architecture: Most teams use a speech encoder, projector, and LLM decoder, with Whisper-large-v3 the most common encoder and different LLM decoders showing no significant performance gap.Projectors typically downsample speech features and map them into the LLM text-embedding space; dual encoders and Q-formers are alternative designs.
- 5.2. Training Strategy: Training strategies range from one-stage to multi-stage procedures, including joint projector–LLM training, staged encoder preparation, and monolingual-to-multilingual training.LoRA may be used for parameter-efficient fine-tuning in one-stage training.
- 5.3. Multilingual Adaptation: Language-specific prompts, trainable special tokens, and language-specific projector or LLM LoRA parameters are used to reduce language confusion in multilingual outputs.More advanced approaches include language-conditioned feature fusion with MoE LoRA and dual encoders.
- 5.4. Data Augmentation: Because the released dataset is relatively small, teams augment training with external speech data, filtering low-quality resources and balancing languages with unequal data availability.The supplied passage identifies external datasets and data balancing as key augmentation strategies but does not quantify their effects.
- 5.5. Conversation Context: Using adjacent historical or future utterance hypotheses as conversation context can outperform SLLMs trained with substantially more additional data.Ground-truth transcriptions of neighboring utterances provide another way to investigate context utilization.
6. METHODS OF TASK 2
Task 2 methods combine diarization and recognition through cascade, semi-integrated, or end-to-end designs, with the cascade pipeline the most common approach.
- 6.1. Cascade Pipeline: The Task 2 analysis summarizes how multilingual conversational SLLMs combine speech diarization and ASR across 10 Eval-2 submissions and six reported top-ranking teams.The comparison is presented in Table 4, but the supplied passages do not include its cell values.
- 6.1. Cascade Pipeline: The cascade pipeline combines VAD, speaker embeddings, and speaker clustering before applying Task 1 SLLMs to diarized speech segments for transcription.Common refinements include maximum segment durations and concatenating adjacent segments from the same speaker.
- 6.1. Cascade Pipeline: Most teams use an FSMN-based VAD, while speaker representations come from models such as CAM++, ERes2Net-large, and ResNet.One team instead uses Sequence-to-Sequence Neural Diarization as its VAD module.
- 6.2. Semi-integrated Methods: Semi-integrated systems connect diarization and recognition more deeply by supplying speaker-enrollment triplets or using a diarization model before generating segment transcriptions.These designs aim to reduce the complexity of strictly cascaded processing.
- 6.3. End-to-end Method: Only one team explores an end-to-end SLLM that jointly predicts speaker labels, segment boundaries, and transcriptions from raw conversational speech.Its design uses local sliding non-overlapping windows and prompt-based context from previous speaker turns.
7. CONCLUSION
The challenge reveals diverse and effective multilingual conversational SLLM solutions, while multilingual adaptation, context use, and speaker-temporal processing remain relatively straightforward.
- 7. CONCLUSION: Submissions demonstrate diverse and effective solutions in model architecture, training strategies, and data augmentation for multilingual conversational SLLMs.The conclusion is based on the paper’s distillation of all challenge submissions.
- 7. CONCLUSION: Multilingual adaptation, conversation-context utilization, and speaker and temporal information processing for diarization have not yet yielded the expected benefits.The paper identifies these areas as requiring further exploration.