Source-linked AI summary
The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challenge
Jordi Luque, Lorenzo Concina, Marco Matassoni, Alessio Brutti, Filippo Vella
TL;DR
The paper tackles multilingual multiple-choice understanding of raw multi-speaker conversations in the MLC-SLM challenge. It compares LoRA fine-tuning, multimodal in-context learning, and training-free voice-anchored retrieval. In-context learning achieves the best reported Phase 2 result at 0.81, while all three systems outperform the official baseline.
Problem
Task 2 requires multilingual multiple-choice question answering about acoustic and semantic properties of entire multi-speaker conversations without oracle segmentation, speaker labels, or diarization.
Method
The team evaluates LoRA fine-tuning of Voxtral-Mini-3B, multimodal in-context learning on frozen Voxtral-24B, and training-free retrieval using voice-anchored memory.
Results
0.81 on Phase 2 is the best reported result, compared with 0.72 for fine-tuning and 0.68 for training-free retrieval.
Takeaways & Limitations
In-context learning corrects strong label bias without parameter updates, while voice-anchored retrieval remains a training-free alternative over frozen models.
Abstract
from arXiv · showhide
This paper details the Eloquence team's approach to Task 2 of the 2nd MLC-SLM challenge at Interspeech 2026, which involves multilingual Multiple-Choice Question Answering (MCQA) across 21 languages. Three approaches are explored. First, we fine-tune Voxtral-Mini-3B via LoRA with cross-lingual data augmentation, ASR transcript augmentation and timestamp-aware audio cropping, achieving 0.72 macro-accuracy on evaluation Phase 2. Second, we apply multimodal in-context learning (ICL) to the frozen Voxtral-24B model to correct a strong label bias, reaching 0.81, our best result. Third, a training-free retrieval system based on a three-layer voice-anchored memory combining acoustic identity, semantic content, and a knowledge graph achieves 0.68. All three systems substantially outperform the official baseline.
1. Introduction
Task 2 requires multilingual systems to answer questions about both acoustic and semantic properties of raw, multi-speaker conversations. Eloquence addresses this with fine-tuning, in-context learning, and retrieval-based systems.
- Task 2 asks systems to answer multiple-choice questions about acoustic properties and the semantic content of entire multi-speaker conversations.
- Evaluation provides no pre-segmented utterances, speaker labels, or ground-truth diarization, requiring systems to process raw recordings directly.
- Eloquence submits three approaches: LoRA fine-tuning, multimodal in-context learning on a frozen model, and retrieval with persistent voice-anchored memory.
2. Challenge and baseline description
The challenge targets multilingual conversational speech understanding in realistic, context-rich dialogue. Its second edition broadens language coverage and includes interruptions, pauses, and speaker overlaps.
- The dataset captures multilingual conversational complexity, including pauses, interruptions, speaker overlaps, and diverse conversational styles.
- The challenge aims to support speech models that respond naturally in multilingual, dynamic, and context-rich environments.
- The second challenge expands language coverage with Tagalog, Urdu, Turkish, and regional varieties of Canadian French, Mexican Spanish, and Brazilian Portuguese.
3. Fine-Tuning Speech-LLM
The fine-tuning system adapts Voxtral-Mini-3B with LoRA, cross-lingual augmentation, ASR transcripts, and timestamp-aware cropping. Cropping produced strong development performance, but the final evaluation score fell to 0.72, suggesting limited generalization.
- LoRA fine-tunes Voxtral-Mini-3B while restricting training to 0.25× an epoch to limit memorization of challenge audio sessions.
- Translated question-and-option copies address an estimated 52% cross-lingual evaluation mismatch absent from the original training data.
- ASR transcript augmentation and ±30s timestamp-centered cropping provide additional context and focus the encoder on relevant audio segments.
4. In-Context Learning with Speech-LLM
Multimodal in-context examples correct a strong class-label bias in the frozen Voxtral-24B model without parameter updates. Six multimodal shots reach the best Phase 2 result, while prior calibration adds no further gain.
- 4. In-Context Learning: The 24B baseline predicts class A for 63% of questions and class D for only 2%, revealing a strong systematic label bias.
- 4.1. In-Context Learning: 0.8095 Phase 2 accuracy is achieved with six multimodal shots, the best reported result for the 24B model.
- 4.1. In-Context Learning: Two text-only shots shift class-A predictions from 63% to 44% and improve accuracy by +5.1 pp, while six multimodal shots add +1.5 pp.
- 4.2. Inference-Time Calibration: Prior calibration reaches approximately 0.795 with ICL-2, matching ICL-2 and indicating no further gain over in-context learning alone.
5. Training-Free Retrieval Approach
The training-free system treats Task 2 as retrieval, using persistent voice-anchored memory that combines acoustic identity, semantic content, and a knowledge graph before a frozen answer LLM responds.
- A voice-anchored memory layer: A three-layer memory stores acoustic identity, semantic content, and graph relationships under shared utterance identifiers for cross-layer retrieval.Audio utterances receive UUIDs propagated across Lac, Lsem, and Lkg; chat turns skip Lac but retain shared identifiers across applicable layers.
- A voice-anchored memory layer: The acoustic layer identifies enrolled speakers and resolves same-session probes using Titanet embeddings and cosine retrieval.Records distinguish durable enrollment profiles from transient query-session probes.
- A voice-anchored memory layer: The semantic layer supports speaker-conditioned search over transcribed or verbatim utterances, with optional emotion, time, and dialogue filters.Utterance text is embedded with all-MiniLM-L6-v2 and stored under the same UUIDs as the acoustic layer.
- A voice-anchored memory layer: The knowledge graph represents speakers, utterances, entities, and dialogues, preserving contradictory facts through validity intervals and provenance.Structural edges are added at ingestion, while an LLM extractor supplies mentions and typed entity facts.
- Application to the MLC-SLM Task 2: At query time, retrieval combines speaker identity, relevant utterances, graph context, and independent emotion labels into a fact sheet for the downstream LLM.A question classifier selects profile, dialogue, or entity graph retrieval, while prompts constrain the answer model to fact-sheet content.
- Application to the MLC-SLM Task 2: For Task 2, diarization, speaker embedding, transcription, and attribution populate all three layers before speaker-conditioned retrieval supplies the MCQ context.The frozen Qwen answer model emits one option letter and does not perform speaker verification or long-context recall itself.
6. Results
The three systems show different generalization profiles: multimodal in-context learning achieves the strongest Phase-2 result, while fine-tuning and retrieval reach lower evaluation accuracy despite useful development performance.
- Fine-tuned Voxtral-3B: 0.72 evaluation accuracy follows fine-tuned Voxtral-3B’s 0.85 development score, indicating poor generalization from the fine-tuning setup.The authors attribute the drop likely to overfitting or language-specific shortcuts learned from limited data.
- Voxtral-24B with ICL: 0.81 Phase-2 accuracy from Voxtral-24B with six multimodal in-context shots is the team’s best result overall.The method corrects strong label bias without parameter updates or training data.
- Training-free retrieval: 0.68 Phase-2 accuracy is obtained by the training-free retrieval system, after its unified Voxtral Mini 3B frontend reached 0.83 versus 0.78 for the multi-model development variant.The unified frontend simplified the pipeline while outperforming the alternative on development data.
7. Conclusions
The Eloquence team evaluated three complementary Task 2 systems under fully blind inference, with multimodal in-context learning achieving the best Phase 2 result and retrieval offering a lower-cost alternative.
- 0.81 Phase 2 accuracy was achieved by in-context learning with frozen Voxtral-24B, the team's best result.This approach mitigated strong pretrained label bias without parameter updates.
- 0.68 Phase 2 accuracy was achieved by the training-free retrieval system, which used a voice-anchored memory layer over frozen models.Its 0.83 development result indicates a substantial evaluation-set generalization gap.
- All three systems operated without oracle segmentation, speaker labels, or diarization during inference and together placed the team fifth overall.The approaches trade off accuracy, training cost, and deployment simplicity.