Source-linked AI summary
SEAR: Segment-Evidence-Aware Routing for Weak-to-Strong Multilingual Speech MCQ
Huy Hoang Le, Long-Bao Nguyen, Minh Tri Dao
TL;DR
The paper addresses audio MCQ systems that can exploit text without genuinely listening. It introduces segment-evidence-aware weak-to-strong post-training and reaches 90.92% accuracy on the final evaluation set, ranking second.
Problem
Audio LLMs can answer from textual cues while ignoring audio, inflating accuracy and weakening training for acoustically grounded questions.
Method
The system localizes event evidence, generates complementary semantic and acoustic MCQs, verifies them, then routes weak items to SFT and audio-dependent items to GSPO reinforcement learning.
Results
90.92% accuracy on the final evaluation set ranked the system 2nd on the official leaderboard.
Takeaways & Limitations
Separating text-answerable from audio-dependent supervision supports a weak-to-strong post-training recipe for multilingual conversational Audio MCQ.
Takeaways & Limitations
The method assumes LLM-based event and quality judgments are sufficiently reliable, although correlated verifier errors may occur.
Abstract
from arXiv · showhide
This paper describes our system for Task~2 of the second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge. We adapt Qwen3-Omni-30B-A3B-Instruct with a segment-evidence-aware data and post-training pipeline. A language model converts timestamped ASR into coherent event spans, which are expanded by a boundary margin and cropped from the original recording. We then synthesize complementary semantic MCQs with Qwen3.6-27B and acoustic MCQs with Gemini~3.1 Flash-Lite, followed by structural, grounding, answer-consistency, and target-model trainability checks, yielding 359,825 verified MCQs across 21 language and accent variants. A text-only probe partitions the data into weak, text-answerable items used for supervised fine-tuning and strong, audio-dependent items used for reinforcement learning with Group Sequence Policy Optimization (GSPO), stabilized by debiased advantages, sequence-level importance correction, and dynamic filtering. Our system obtains 90.92% accuracy on the final official evaluation set.
1. Introduction
Task 2 requires multilingual conversational speech systems to answer questions about both linguistic content and acoustic delivery, while avoiding answers based solely on textual cues. The paper addresses this with segment-aware data construction, dual semantic/acoustic question generation, verification, and weak-to-strong post-training.
- Motivation: Text priors can produce correct answers without listening, motivating training that isolates questions requiring genuine audio use.The paper identifies this as a zero-audio-contribution effect that dilutes supervision for acoustically grounded questions.
- Data construction: The pipeline converts timestamped ASR into event spans, adds boundary context, and crops corresponding waveform segments for training.Model-facing timestamps are rebased to each crop.
- Question synthesis: Qwen3.6-27B generates semantic questions from transcript evidence, whereas Gemini 3.1 Flash-Lite generates acoustic questions directly from waveform segments.This dual-branch design separates questions about linguistic meaning from questions about audible properties.
- Routing and training: Verification rejects ungrounded, ambiguous, malformed, or trivially leaked questions before SFT uses weak items and GSPO trains on audio-dependent items.The verifier also tests whether the target model produces informative mixed-reward rollouts.
- Main result: 90.92% accuracy was achieved on the final official evaluation set with the proposed weak-to-strong system.The system combines supervised fine-tuning with GSPO-based reinforcement learning.
2. Related Work
Prior work constructs and filters audio MCQs and separates weak from audio-dependent supervision, while related reinforcement-learning methods address group-relative normalization and sequence-level stability. Figure 1 summarizes the paper’s segment-level curation and routing pipeline.
- Audio MCQ construction: AudioMCQ establishes candidate filtering and audio-contribution-aware weak-to-strong training, which this paper extends to long multilingual conversations.The extension uses timestamped transcripts and separate text and audio generators.
- Audio-contribution-aware post-training: This paper adds target-model trainability testing because audio dependence alone does not guarantee useful group-relative reward variance.The added test distinguishes audio-dependent items that can support reinforcement learning from those that cannot yet do so.
- Sequence-level reinforcement learning: GSPO uses a length-normalized sequence likelihood ratio and whole-response clipping, while Dr.GRPO and TIS address normalization bias and numerical importance mismatch.These methods motivate the paper’s stabilized group-relative reinforcement-learning design.
- Pipeline overview: Figure 1 organizes the proposed pipeline into event localization, dual-branch synthesis and verification, and audio-contribution routing.The final routing sends weak items to SFT and strong items toward GSPO after dynamic filtering.
3. Method
The method routes localized multilingual speech questions by evidence type, using weak items for SFT and audio-dependent items for GSPO. It combines event-preserving segmentation, dual semantic/acoustic MCQ synthesis, verification, and stabilized sequence-level reinforcement learning.
- Data construction: 359,825 verified segment-level MCQs span 21 language and accent variants, with balanced coverage across variants.The corpus contains 11.5K–21.9K items per variant.
- Segmentation: Timestamped ASR is converted into coherent event spans, expanded with varied context margins, cropped, merged when overlapping, and rebased to local timestamps.This preserves complete turns and nearby prosodic context while avoiding unrelated conversation segments.
- MCQ synthesis: Qwen3.6-27B generates transcript-based semantic questions, while Gemini 3.1 Flash-Lite generates waveform-based acoustic questions with provenance and timestamp metadata.Correct-answer positions are permuted to reduce option-position bias.
- Verification and routing: Verification checks grounding, answer uniqueness, distractor plausibility, timestamp validity, fluency, leakage, and target-model trainability before routing examples.The target-model probe uses G = 8 responses; learnable items form the primary GSPO pool, while weak items remain useful for SFT.
- Post-training: The training recipe uses weak items for SFT and strong items for GSPO with binary rewards, debiased group-relative advantages, sequence-level clipping, truncated importance correction, and dynamic filtering.All-correct and all-wrong groups are discarded because they provide no group-relative gradient.
4. Experiments
Experiments evaluate SEAR on official development and evaluation sets, with weak items routed to SFT and strong items to GSPO. The final system reaches 90.92% accuracy, while an all-data control reaches 88.24 Eval.
- Experimental setup: The corpus contains 359,825 balanced items across 21 language and accent variants, partitioned into weak items for SFT and strong items for GSPO.The text-only probe identifies 197,231 weak and 162,594 strong items.
- Experimental setup: Evaluation uses the official MLC-SLM Task 2 development and evaluation sets with accuracy as the official metric.
- Training setup: GSPO training applies group advantages, sequence-level importance correction, dynamic filtering, and monitoring of reward and optimization diagnostics.The configuration uses G = 8 and filters groups with near-zero reward variance.
- Main results: 90.92% accuracy is achieved on the Phase-2 final evaluation set, ranking the system 2nd on the official leaderboard.
- Ablation study: 88.24 Eval for the all-data control is 2.68 points below SEAR’s 90.92, supporting weak/strong classification as an important design decision.The control omits weak/strong classification while using the same data.
5. Conclusions
SEAR combines event-localized evidence, complementary semantic and acoustic question generation, and trainability-aware routing for multilingual conversational Audio MCQ. It reaches 90.92% accuracy on the final evaluation set and ranks 2nd, while acknowledging correlated errors in LLM-based judgments.
- The final system reaches 90.92% accuracy on the final evaluation set and ranks 2nd on the official leaderboard.
- SEAR routes verified weak items to SFT and audio-dependent items to debiased, importance-corrected, dynamically filtered GSPO.
- LLM-based event and quality judgments may introduce correlated errors, motivating independent verifiers and refreshed trainability labels as the policy improves.