Source-linked AI summary
X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs
Di Cao, Dongjie Fu, Hai Yu, Siqi Zheng, Xu Tan, Tao Jin
TL;DR
Speech LLMs often lag behind text-based counterparts because paired speech-reasoning data are scarce and acoustic representations are misaligned with text reasoning. X-OPD uses on-policy multimodal student rollouts and token-level feedback from a text teacher to perform cross-modal distillation. Across benchmarks, it narrows the gap while preserving capabilities, including with a modest 27k-sample dataset.
Problem
Speech LLMs exhibit capability degradation relative to text-based counterparts, while standard SFT and RL training do not fully transfer text-model capabilities across modalities.
Method
X-OPD aligns speech and text through student on-policy rollouts, synchronized textual inputs, and KL-divergence-based token-level feedback from a text-based teacher.
Results
X-OPD significantly narrows the speech–text performance gap and mitigates catastrophic forgetting across evaluated benchmarks.
Takeaways & Limitations
X-OPD provides a robust, data-efficient, and annotation-free pathway for aligning multimodal agents while preserving pretrained capabilities.
Abstract
from arXiv · showhide
While the shift from cascaded dialogue systems to end-to-end (E2E) speech Large Language Models (LLMs) improves latency and paralinguistic modeling, E2E models often exhibit a significant performance degradation compared to their text-based counterparts. The standard Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) training methods fail to close this gap. To address this, we propose X-OPD, a novel Cross-Modal On-Policy Distillation framework designed to systematically align the capabilities of Speech LLMs to their text-based counterparts. X-OPD enables the Speech LLM to explore its own distribution via on-policy rollouts, where a text-based teacher model evaluates these trajectories and provides token-level feedback, effectively distilling teacher's capabilities into student's multi-modal representations. Extensive experiments across multiple benchmarks demonstrate that X-OPD significantly narrows the gap in complex tasks while preserving the model's inherent capabilities.
1. Introduction
End-to-end speech LLMs offer lower latency and richer paralinguistic modeling but retain a substantial capability gap relative to text-based models. X-OPD addresses this gap through on-policy cross-modal distillation designed to preserve general proficiency.
- E2E speech LLMs reduce latency and capture paralinguistic information, including intonation, emotion, and environmental context.
- Speech LLMs frequently underperform text-based counterparts on complex instruction following, logical reasoning, and knowledge-intensive queries.
- Limited paired speech-reasoning data and misalignment between acoustic and discrete logical representations constrain transfer from text-LLM SFT and RL training.
- Offline distillation can suffer exposure bias because inference trajectories diverge from training distributions, while cascaded-pipeline errors accumulate.
- X-OPD uses student rollouts in speech and text, synchronized textual inputs, and KL-divergence-based teacher feedback to align multimodal representations.
- X-OPD is presented as an optimization strategy that narrows the modality gap while preserving the pretrained model’s general proficiency.
2. Related Work
Prior speech-LLM distillation methods largely rely on offline teacher trajectories or fixed targets, leaving them vulnerable to exposure bias. X-OPD instead uses on-policy student sampling with teacher guidance for direct cross-modal alignment.
- Prior methods use audio-caption targets, active data selection, or logit-level supervision to transfer capabilities from text or cascaded teachers.
- These approaches are fundamentally off-policy because they depend on static teacher trajectories or fixed targets rather than the student’s inference rollouts.
- Qwen3-Omni shows performance drops against its text counterpart on Big Bench Audio and Audio Multi-Challenge, indicating an unresolved cross-modality gap.
- On-policy methods such as GKD and MiniLLM train on student-generated sequences with teacher feedback to address distribution mismatch and exposure bias.
3. Cross-Modal On-Policy Distillation
X-OPD aligns Speech LLMs with text-based teachers by combining on-policy student rollouts, token-level cross-modal feedback, and dual in-modal/cross-modal policy-gradient losses. Multiple trajectories reduce gradient variance, while the weighted objective balances foundational textual proficiency with cross-modal alignment.
- X-OPD uses student on-policy sampling across speech and text modalities, guided by token-level scoring from a text-based teacher and optimized with policy gradients.The framework aligns the student’s own inference distribution with the teacher’s capabilities rather than relying only on fixed trajectories.
- The distillation dataset contains paired speech and text prompts whose acoustic and textual inputs must preserve the same logical intent.Pairs may be created by synthesizing speech from text or transcribing existing audio prompts.
- Multiple independent candidate trajectories reduce gradient-estimation variance and broaden coverage of the student policy space.The method marginalizes gradients across sampled paths to attenuate volatility in on-policy updates.
- The dual-advantage mechanism uses an in-modal advantage to stabilize textual proficiency and a cross-modal advantage to connect textual logic with speech-conditioned outputs.Together, the advantages provide the reward signal for aligning the student with the teacher’s established proficiency.
- The in-modal advantage measures the teacher–student log-probability discrepancy for each sampled token under the text prompt.The trajectory is sampled from the student policy, and the advantage compares teacher and student token probabilities.
- The optimization objective combines in-modal and cross-modal policy-gradient losses as a weighted sum.The formulation uses multiple sampled trajectories and probability ratios between the current and sampling policies.
4. Experiments
The experiments evaluate X-OPD across speech benchmarks, compare it with standard distillation and fine-tuning baselines, and assess modality alignment, ablations, and catastrophic forgetting. X-OPD reduces performance degradation while preserving foundational capabilities.
- Evaluation Setup: The evaluation covers BIG Bench Audio, Audio Multi-Challenge, and VoiceBench, measuring accuracy, Average Pass Rate, and mean subset score respectively.BIG Bench Audio contains 1,000 samples; Audio Multi-Challenge contains 1,712 rubrics across 452 conversations; VoiceBench contains 5,783 samples.
- Training Setup: X-OPD uses pure on-policy rollouts from the current speech policy, with full-parameter language-backbone training while freezing the audio tower and modal adapter.The implementation uses the verl framework, Adam optimization, a learning rate of 2 × 10−6, and batch size 256.
- Main Results: Compared with SFT, Offline KD, and GKD, X-OPD reduces Qwen3-Omni-A3B-Instruct’s average performance drop from 11.29% to 3.43% for Speech and from 5.51% to 0.97% for Text.The recovery is strongest on BIG Bench Audio and Audio Multi-Challenge, while VoiceBench reaches parity with the base model.
- Ablation Study: The ablation study finds that the A3B teacher outperforms the stronger A22B teacher, while the balanced λ = 0.5 setting produces the best overall results.Textual distillation improves speech performance, whereas speech-focused distillation improves text-only scores, indicating complementary objectives.
- Catastrophic Forgetting: On MMAR, traditional methods reduce accuracy from 71.3% to 59.9%, whereas every X-OPD variant remains above 69%.The λ = 1 setting yields the highest retention at 70.7%.
5. Conclusion
The conclusion presents X-OPD as a framework for aligning Speech LLMs with text-based counterparts while preserving general capabilities. It reports effective gap reduction, mitigation of catastrophic forgetting, and strong sample efficiency.
- Conclusion: X-OPD aligns Speech LLMs with text-based counterparts, narrows the speech–text performance gap, and mitigates catastrophic forgetting using only 27k samples.The paper characterizes this as a robust, data-efficient, and annotation-free pathway for foundational alignment in multimodal agents.
7. Generative AI Use Disclosure
The disclosure states that all authors are responsible for the work and that Gemini 3 was used only for linguistic refinement.
- AI Use Disclosure: Gemini 3 was used exclusively to improve manuscript clarity and flow, not to generate a significant portion of the research.The authors state that they remain fully responsible and accountable for the presented work and content.