Source-linked AI summary
Closing the Modality Reasoning Gap for Speech Large Language Models
Chaoren Wang, Heng Lu, Xueyao Zhang, Shujie Liu, Yan Lu, Jinyu Li, Zhizheng Wu
TL;DR
Speech LLMs still reason less effectively from speech than from text, with the gap associated with representational drift and behavioral deviations. TARS uses asymmetric reinforcement learning to align speech- and text-conditioned trajectories through representation and behavior rewards, narrowing the gap and achieving state-of-the-art performance among 7B-scale Speech LLMs.
Problem
Speech LLMs exhibit a substantial modality reasoning gap, with performance on speech inputs lagging behind text capabilities.
Method
TARS uses on-policy reinforcement learning with asymmetric dense rewards for layer-wise representation alignment and semantic behavior alignment.
Results
TARS substantially narrows the modality reasoning gap and achieves state-of-the-art performance on MMSU and OBQA among 7B-scale Speech LLMs.
Takeaways & Limitations
The alignment framework improves speech reasoning without architectural modifications and can preserve or improve text reasoning performance.
Takeaways & Limitations
Evaluation is limited to 7B-scale models and single-turn reasoning, while text-only references may miss paralinguistic cues without explicit textual counterparts.
Abstract
from arXiv · showhide
Although Speech Large Language Models have achieved notable progress, a substantial modality reasoning gap remains: their reasoning performance on speech inputs is markedly weaker than on text. This gap could be associated with representational drift across Transformer layers and behavior deviations in long-chain reasoning. To address this issue, we introduce TARS, a reinforcement-learning framework that aligns text-conditioned and speech-conditioned trajectories through an asymmetric reward design. The framework employs two dense and complementary signals: representation alignment, which measures layer-wise hidden-state similarity between speech- and text-conditioned trajectories, and behavior alignment, which evaluates semantic consistency between generated outputs and reference text completions. Experiments on challenging reasoning benchmarks, including MMSU and OBQA, show that our approach significantly narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs.
1 Introduction
Speech LLMs face a persistent modality reasoning gap: reasoning on speech inputs is weaker than on text. TARS addresses this gap with on-policy trajectory alignment using complementary representation and behavior rewards without architectural modifications.
- Motivation: Speech LLM reasoning performance substantially declines on speech inputs compared with text, creating a persistent modality reasoning gap.The gap is supported by empirical analyses on VoiceBench and SpeechMMLU.
- Training objective: An asymmetric reward remains informative when speech-conditioned samples receive uniformly zero task-accuracy reward.This enables trajectory alignment in difficult speech reasoning settings where task rewards are sparse or absent.
- Approach: TARS aligns speech-conditioned reasoning trajectories with text-conditioned counterparts through on-policy reinforcement learning.The framework is designed to close the gap without changing the model architecture.
- Approach: Representation alignment reduces layer-wise hidden-state drift, while behavior alignment enforces semantic consistency in generated outputs.The two signals provide complementary feedback at representation and output levels.
- Results: TARS achieves state-of-the-art performance on MMSU and OBQA among 7B-scale Speech LLMs.The method and accompanying datasets and code are presented to support reproducibility.
2 Related Works
Speech LLM research has progressed toward unified end-to-end systems, but speech-conditioned reasoning still diverges from text-conditioned reasoning. Existing approaches mainly modify input representations or supervise outputs with static targets, while reinforcement learning for speech remains comparatively limited.
- Speech LLMs: Speech LLMs commonly combine a pretrained speech encoder, lightweight projectors, and a decoder-only text LLM.This architecture maps acoustic features into the text-embedding space while retaining text-pretrained reasoning capabilities.
- Modality Alignment and Reasoning Gap: Speech-conditioned hidden states often drift from text counterparts, contributing to degraded reasoning despite unified architectures.The literature identifies this disparity as the modality reasoning gap.
- Architectural Adaptations: Frozen-backbone architectural methods refine input projectors but may produce only surface-level alignment and divergent reasoning trajectories.The cited related work contrasts input projection alignment with adaptation to speech-specific dynamics.
- Supervised Alignment Strategies: Supervised methods use static matching, prompt switching, distillation, or data selection to align outputs with text-based behavior.These approaches include output-side supervision and combined representation-behavior alignment.
- Reinforcement Learning for Reasoning: Reinforcement learning has improved reasoning in text LLMs, but its use for speech remains nascent and focused mainly on paralinguistic attributes or general helpfulness.This leaves speech reasoning trajectory alignment comparatively underexplored.
3 Method
TARS formulates speech reasoning alignment as reinforcement learning over speech- and text-conditioned trajectories. It combines task rewards with representation and behavior alignment, then uses modality-specific normalization to prevent the stronger text branch from suppressing speech learning.
- Problem Formulation: MRR measures how much speech-conditioned reasoning retains the base model’s text-conditioned capability, with the objective MRR ≥100%.The metric compares a current model’s speech completion against the base model’s text completion under a reasoning metric such as QA accuracy.
- Trajectory Alignment: The model generates speech and text completions from the same query, with text-conditioned outputs serving as moving references during joint optimization.The text branch continues improving under the base reward while speech trajectories align with its evolving reasoning capability.
- Reward Modeling: The asymmetric reward combines base task accuracy and format rewards with representation alignment and behavior alignment for speech completions.Representation alignment compares layer-wise hidden-state geometry, while behavior alignment measures semantic consistency between generated speech outputs and text references.
- Reward Modeling: Representation alignment uses mean-pooled reasoning-token states and averages cosine similarity across layers against a correct text completion.The fallback sets the representation reward to zero when no correct text completion exists; this occurs in 5.9%–10.7% of cases.
- Reinforcement Learning Framework: GRPO supplies continuous alignment guidance even when speech task accuracy is uniformly zero, while modality-specific normalization computes advantages separately for speech and text.Separate baselines prevent higher text rewards from giving speech completions consistently negative advantages and suppressing learning.
4 Experiments
Experiments evaluate TARS against shared-backbone baselines on speech-text reasoning, showing that asymmetric dense alignment rewards deliver strong modality recovery while preserving text and ASR capabilities.
- Main Results: 76.84% audio accuracy and MRR = 98.89% are achieved on Qwen2.5-Omni, while Phi-4-MM reaches 79.80% accuracy and MRR = 100.45%.These results outperform cited end-to-end alignment and knowledge-distillation baselines.
- Main Results: Text accuracy rises from 76.17% to 78.56% for Qwen2.5-Omni and from 78.39% to 83.82% for Phi-4-MM.The reported improvements indicate that speech gains are not obtained at the expense of text reasoning.
- Main Results: The end-to-end models outperform ASR-plus-LLM pipelines, reaching 76.84% versus 75.55% and 79.80% versus 73.40%, respectively.The authors suggest direct speech processing can avoid certain ASR-induced errors.
- Reward Components: Adding representation alignment raises MRR from 92.21% to 95.56%, while behavior alignment reaches MRR = 99.22%.The ablation supports complementary contributions from layer-wise representation and output semantic alignment.
5 Discussion and Analysis
Analysis links TARS’s gains to improved layer-wise trajectory similarity, identifies middle layers as the most effective reward location, and tests robustness, generalization, and efficiency.
- Layer-wise Trajectory Alignment: Adding Rrep consistently lifts cosine similarity between correct text and speech trajectories across layers, aligning with external MRR improvements.The analysis evaluates mean hidden-state similarity across corresponding response tokens and test samples.
- Impact of Alignment Stage on Reward Design: Middle layers 11–20 achieve 75.48% accuracy, outperforming reward placement limited to shallow or final layers.The sensitivity analysis suggests representation drift is concentrated in mid-to-late reasoning stages.
- Impact of Alignment Stage on Reward Design: On Qwen2.5-Omni, strict representation alignment overregularizes the model, whereas behavior-only alignment achieves the highest average audio accuracy of 77.71%.Output-level guidance preserves flexibility for modality-specific internal processing.
- Preserving Text Reasoning: Text-only GRPO reaches 83.76% text accuracy versus 83.56% for joint alignment, while joint alignment improves speech accuracy to 79.57% versus 76.86%.The ablation indicates that speech alignment does not interfere with text capability.
- Generalization and Error Analysis: TARS consistently improves performance across 12 MMSU domains and generalizes to real-speech QA and paralinguistic perception benchmarks.Lower-performing domains still show errors involving semantic traps or chronological retrieval failures.
- Computational Efficiency: Online RL introduces only a marginal 4.4% overhead compared with Standard GRPO while achieving better performance.This result summarizes the reported computational-efficiency analysis.
6 Conclusion
The paper concludes that on-policy trajectory alignment combines dense representation and behavior rewards to mitigate representational drift and improve speech reasoning.
- Conclusion: TARS substantially narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs on MMSU and OBQA.The conclusion attributes these results to an asymmetric RL objective combining representation and behavior alignment.
- Conclusion: The framework mitigates representational drift and improves semantic consistency in speech reasoning through on-policy trajectory alignment.These are the two central outcomes stated in the conclusion.
Limitations
The framework is evaluated only at the 7B scale and on single-turn reasoning, leaving its behavior in other model sizes and interactive settings unclear. Its text-only reference completions may also omit paralinguistic information without explicit textual counterparts.
- Evaluation is limited to 7B-scale models, so behavior for smaller or larger models remains unclear.
- The method focuses on single-turn reasoning rather than multiturn, interactive, or dialogue-driven speech reasoning.
- Text-only reference completions may not fully capture emotion, prosody, or intent without explicit textual counterparts.
E Layer-wise Analysis Methodology
The layer-wise analysis uses teacher forcing to compare hidden states for identical response tokens across text- and speech-conditioned branches. Similarities are computed per layer and averaged over valid tokens and test samples for MMSU and OBQA.
- Teacher forcing supplies identical text-conditioned chain-of-thought response tokens to both branches for aligned comparison.
- Layer-wise cosine similarities are averaged over corresponding response tokens, valid tokens, and test samples with 95% confidence intervals.
F Per-domain Breakdown Analysis
The MMSU per-domain analysis reports that trajectory alignment improves performance consistently across all 12 categories.
- Trajectory alignment consistently improves MMSU performance across all 12 evaluated categories.
G Training Efficiency
The efficiency analysis compares online trajectory alignment with offline training strategies and Standard GRPO. Offline methods incur substantial data-generation cost, while the proposed method preserves the GRPO peak-memory footprint.
- The proposed method maintains an identical 90% peak-memory footprint relative to the GRPO baseline.
H Qualitative Analysis
The qualitative analysis shows that training improves reasoning capabilities and reward signals, while persistent failures remain in lower-performing domains due to semantic, logical, and temporal difficulties.
- Free-Generation Alignment Analysis: Upward trends in Rbase, Rrep, and Rbeh indicate improvements in task accuracy, representation similarity, and behavioral alignment during free-generation training.The reward curves are analyzed from autoregressive online RL rollouts without teacher forcing.
- Training Improvements: Training improves math calculation, logical reasoning, and domain knowledge while correcting hallucinations and reasoning failures.These qualitative improvements are summarized in Table 9.
- Persistent Errors: Persistent errors remain in lower-performing domains, often involving subtle semantic traps, conflicting domain logic, or chronological retrieval failures.Table 10 analyzes cases that still fail to reach the ground truth after training.