Source-linked AI summary
Leading-Silence Augmentation and Multi-Stage Synthetic Supervision for the Second MLC-SLM Challenge
Kexin Shi, Renhe Sun, Yuge Huang, Ximeng Wang, Jiayi Zhou, Jian Liu, Malu Zhang
TL;DR
The challenge requires diarization, recognition, and conversational understanding from complete multilingual conversations without oracle boundaries or Task 2 question-answer training data. The paper adapts separate speech and multimodal systems with augmentation, synthetic supervision, and training strategies, reducing Task 1 tcpMER to 16.73% and raising Task 2 accuracy to 86.0%.
Problem
The MLC-SLM Challenge evaluates complete, unsegmented multilingual conversations without oracle boundaries or speaker labels, while Task 2 lacks question-answer training data.
Method
The paper adapts VibeVoice-ASR-7B for Task 1 and Qwen3-Omni for Task 2 using temporal augmentation, synthetic audio-question supervision, filtering, distribution matching, and tagged answering.
Results
Both task-specific systems improve evaluation performance, with Task 1 tcpMER reaching 16.73% after cumulative cropping and EMA.
Takeaways & Limitations
The final configurations provide stronger single-model performance across both challenge tasks, reaching 86.0% accuracy for Task 2.
Takeaways & Limitations
The leading-silence augmentation ablation does not directly establish robustness to turn-boundary errors.
Abstract
from arXiv · showhide
The second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge evaluates two tasks over complete, unsegmented multilingual conversations: speaker diarization and recognition (Task 1) and conversational speech understanding (Task 2). Neither task provides oracle utterance boundaries or speaker labels at evaluation, and Task 2 provides no question-answer training set. For Task 1, we fine-tune VibeVoice-ASR-7B with random leading-silence cropping, consistent timestamp correction, and an exponential moving average (EMA) training strategy. For Task 2, we construct synthetic question-answer pairs through multimodal candidate generation, silent-audio filtering, and distribution-matched augmentation, and fine-tune Qwen3-Omni-30B-A3B-Instruct for tagged direct answering. On the Task 1 evaluation set, cropping reduces tcpMER from 18.30% to 17.27%, and EMA further reduces it to 16.73%. On the Task 2 evaluation set, jointly applying distribution-matched augmentation and tagged direct answering raises accuracy from 83.0% to 86.0%.
1. Introduction
The second MLC-SLM Challenge evaluates diarization and recognition plus conversational speech understanding on complete, unsegmented multilingual conversations without oracle boundaries or speaker labels. The paper addresses these constraints with fine-tuning improvements for Task 1 and synthetic, filtered, distribution-matched supervision for Task 2, achieving gains on both evaluations.
- Challenge setup: The challenge evaluates Task 1 speaker diarization and recognition and Task 2 acoustic, semantic, and joint acoustic–semantic understanding on complete, unsegmented multilingual conversations.Neither task provides oracle utterance boundaries or speaker labels at evaluation, and Task 2 provides only development examples rather than a question–answer training set.
- Task 1: Task 1 fine-tunes VibeVoice-ASR-7B with random leading-silence cropping, consistent timestamp correction, and exponential moving average training.The approach retains the single-pass backbone while modifying fine-tuning for long recordings without oracle utterance boundaries.
- Task 2: Task 2 generates approximately 127k synthetic question–answer examples using multimodal candidate generation, silent-audio filtering, and distribution-matched augmentation.The resulting supervision supports fine-tuning Qwen3-Omni-30B-A3B-Instruct for tagged direct answering.
- Task 1 results: 16.73% tcpMER is obtained after adding EMA to leading-silence cropping with timestamp correction, improving over 18.30% for the LoRA baseline.The cumulative ablation reports 17.27% tcpMER after cropping with timestamp correction and 16.73% after adding EMA.
- Task 2 results: 86.0% evaluation accuracy is achieved with distribution-matched augmentation and tagged direct answering, up from 83.0%.The system uses one submitted prediction model per task without model-, score-, or output-level fusion.
2. System Description · 2.1. Task 1: Multilingual Conversational Speech Diarization and Recognition
The Task 1 system fine-tunes VibeVoice-ASR-7B on complete multilingual recordings using its original single-pass architecture and output format. It adds random leading-silence cropping with timestamp correction and EMA training while preserving speech annotations and speaker-turn structure.
- 2.1.1. Backbone and Task Formulation: VibeVoice-ASR-7B serves as the backbone for complete recordings, supporting up to 60 minutes, more than 50 languages, and code-switching.Predictions are serialized directly in the official output format.
- 2.1.1. Backbone and Task Formulation: The original architecture and single-pass formulation remain unchanged: each recording yields ordered transcripts, speaker identities, and timestamps.Only fine-tuning changes; decoder architecture and output format are preserved.
- 2.1.2. Random Leading-Silence Cropping: Random leading-silence cropping removes part of the variable non-speech region before the first annotated utterance while preserving the annotated speech.Unlike waveform-speed or time–frequency masking augmentation, it changes the temporal origin and leading context.
- 2.1.2. Random Leading-Silence Cropping: A common timestamp shift after cropping preserves transcripts, speaker labels, turn order, segment durations, and relative timing.The constraint δ ≤s1 ensures that no annotated speech is removed; only leading context, absolute time origin, and total duration change.
- 2.1.2. Random Leading-Silence Cropping: Leading-silence augmentation changes an existing example’s temporal presentation without synthesizing conversations or perturbing internal speaker-turn boundaries.Thus, its aggregate ablation measures leading-silence augmentation but does not directly establish robustness to turnboundary errors.
- 2.1.3. EMA Training Strategy: An exponential moving average of trainable parameters is updated after every optimizer step throughout fine-tuning with decay coefficient λ = 0.99.EMA is applied after random leading-silence cropping and does not alter decoding or output serialization.
- 2.1.4. Training Configuration: Fine-tuning uses LoRA with rank 32, scaling factor 128, dropout 0.05, five epochs, and effective global batch size 64.Optimization uses learning rate 1 × 10−4, warmup ratio 0.03, weight decay 0.01, gradient clipping at 1.0, checkpointing, bfloat16, and ZeRO-2.
2.2. Task 2: Multilingual Conversational Speech Understanding
Task 2 uses Qwen3-Omni-30B-A3B-Instruct to answer questions directly from complete conversation audio, without externally generated transcripts. Synthetic supervision combines large-scale candidate generation, silent-audio filtering, distribution-matched augmentation, and tagged answer serialization.
- Model and input format: Qwen3-Omni-30B-A3B-Instruct predicts answers directly from complete conversation audio, retaining lexical and non-lexical acoustic information without an external transcript.The model receives a user message containing the audio, one question, two to four options, and an answer-tag instruction.
- Synthetic supervision: Gemini 2.5 Pro generates approximately 210k candidate question–answer pairs from the released training audio using development-matched prompts.The prompts match the released examples in question type, language, timestamp style, and two-to-four-option single-answer structure.
- Synthetic supervision: Distribution-matched augmentation adds approximately 60k instances by translating selected questions and synthesizing long-option examples based on label-free evaluation-input properties.Evaluation inputs have longer options on average, 77.9 versus 38.0 characters, and pair some non-English conversations with English questions and options.
- Serialization and inference: All examples use a unified chat schema in which training assistants output only the gold answer inside <answer></answer> tags, while evaluation supplies only the user message.Tagged outputs provide consistent supervision and uniform prediction extraction.
- Submission setup: Each submitted Task 2 prediction is generated solely by a single Qwen3-Omni model, without model-, score-, or output-level fusion.Gemini 2.5 Pro and Qwen2.5-Omni are used only offline for data construction.
3. Experiments and Results
The experiments evaluate Task 1 with official diarization and recognition metrics and Task 2 with multiple-choice accuracy. Cumulative ablations show that leading-silence cropping and EMA improve Task 1, while synthetic-data construction and response formatting improve Task 2.
- Task 1 Evaluation: Task 1 reports across-language tcpMER after speaker mapping, using tcpCER for Japanese, Korean, and Thai and tcpWER for other languages.The official 5-s collar and MeetEval are used; lower tcpMER is better.
- Task 1 Ablation: 18.30% tcpMER is obtained by the LoRA baseline, which retains VibeVoice-ASR-7B’s original architecture, single-pass decoding, and serialized output format.This is the reference configuration for the cumulative ablation.
- Task 1 Ablation: 17.27% tcpMER follows random leading-silence cropping with timestamp correction, a 1.03-point absolute and 5.6% relative reduction from the LoRA baseline.The timestamp correction shifts every annotation by the same offset to preserve alignment.
- Task 1 Ablation: 16.73% tcpMER follows EMA training after cropping, a further 0.54-point absolute and 3.1% relative reduction from the preceding configuration.EMA updates the trainable-parameter average after every optimizer step without modifying decoding.
- Task 2 Ablation: 86.0% accuracy is reached in Task 2 from a 78.0% direct-answering baseline after progressively adding candidate generation, silent-audio filtering, distribution-matched augmentation, and tagged direct answering.Candidate generation raises accuracy from 78.0% to 81.0%, and silent-audio filtering further raises it from 81.0% to 83.0%.
4. Conclusion
The paper presents independently adapted single-model systems for both MLC-SLM Challenge tasks, combining targeted Task 1 training strategies with synthetic supervision and tagged answering for Task 2. Future work targets richer conversational modeling, stronger multilingual adaptation, and more reliable long-context audio understanding.
- Task 1: Task 1’s random leading-silence cropping, consistent timestamp correction, and EMA training reduce tcpMER from 18.30% to 16.73%.These strategies are applied in an independently adapted single-model system.
- Task 2: Task 2 uses approximately 127k synthetic examples from multimodal candidate generation, silent-audio filtering, and distribution-matched augmentation.The resulting system fine-tunes Qwen3-Omni for tagged direct answering.
- Future work: Future Task 1 work will explore richer speaker–temporal modeling and better use of conversational context.These directions address modeling of speakers, time, and conversation context.
- Future work: Future Task 2 work will investigate stronger multilingual adaptation and more reliable audio-grounded long-context understanding.The proposed directions focus on multilingual capability and long-context audio grounding.