Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text

Hyeonyu Kim; Hwayeon Kim; Youngwon Choi; Myeongkyun Cho; Huu-Kim Nguyen

Source-linked AI summary

Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text

Hyeonyu Kim, Hwayeon Kim, Youngwon Choi, Myeongkyun Cho, Huu-Kim Nguyen

arXiv:2608.22908v1Aug 24, 2026cs.CLcs.AI

TL;DR

Current SLMs can perform well while their internal speech representations remain weakly aligned and structurally different from text, alongside instruction-following and generalization limitations. The paper separates speech-feature length matching from semantic alignment and adds token-level internal alignment, achieving competitive performance across multiple benchmarks while showing that alignment strength need not alone determine downstream performance.

Problem
Current SLMs leave internal speech-text alignment largely implicit and still show instruction-following and diverse-task generalization limitations compared with text-only models.
Method
The framework dynamically matches mapped speech-feature length to text embeddings and adds token-level internal alignment alongside behavior alignment.
Results
The approach improves representational alignment and achieves competitive performance against strong baselines across multiple benchmarks.
Takeaways & Limitations
Explicitly addressing structural differences between speech and text is important for more effective SLM training, but stronger alignment does not necessarily improve performance.
Takeaways & Limitations
The appropriate strength of internal alignment may depend on the target task, model architecture, and training data, so alignment alone need not improve downstream performance.

Abstract

from arXiv · show

Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.

1 Introduction

This work finds that current SLMs can perform well while their speech and text representations remain weakly aligned and structurally different. It proposes separating length matching from semantic alignment and adding token-level internal alignment, achieving competitive benchmark performance.

Prior limitations: Behavior-alignment methods encourage matching responses but leave internal speech-text alignment implicit.Current SLMs consequently continue to show instruction-following and task-generalization limitations relative to text-only models.
Structural gap: Speech is continuous and time-varying, producing longer, structurally distinct representations than discrete text.Prior work has therefore focused largely on reducing the length of mapped speech features.
Approach: The proposed framework dynamically matches mapped speech-feature length to text embeddings, decoupling length matching from semantic alignment.It further incorporates token-level internal alignment alongside behavior alignment.
Motivation: Current SLMs remain weakly aligned internally even when they perform well on downstream tasks.Their speech representations remain structurally different from text embeddings despite the semantic correspondence between speech and transcription.
Results: Experiments across multiple benchmarks show competitive performance against strong baselines, including closed-source models.The authors present this result as highlighting the importance of explicitly addressing speech-text structural differences during SLM training.

2 Related Work

Related work addresses SLM modality and training diversity, but a central challenge remains the mismatch between encoded speech and the LLM input space. Existing approaches use temporal compression, sequence-length matching, or behavior alignment, while similarity maps visualize structural correspondence.

Scope: This work focuses on SLMs that generate textual responses from speech inputs.The broader SLM literature includes varied input/output modality setups and training methods.
Training paradigms: Early SLMs often used single-task ASR or AST training, while later studies used multiple speech tasks to broaden spoken-language understanding.These systems commonly integrate a pretrained language model with speech-processing components.
Modality mismatch: Encoded speech differs from the LLM input space in semantics and substantially longer sequence length.Modality adapters often downsample speech features, but temporal compression may not explicitly preserve higher-level linguistic structure.
Representation analysis: Figure 1 compares token-wise similarity maps between mapped speech and text representations for existing SLMs and the proposed models.Weak or inconsistent diagonal patterns indicate less consistent token-level correspondence than the clear diagonal trend reported for the proposed model.
Behavior alignment: Behavior-alignment methods train on synthetic instruction-response pairs but leave internal speech-text representation alignment implicit.They mainly encourage producing the same responses from speech inputs.

3 Method

The method addresses speech–text structural mismatch by separating sequence-length matching from semantic alignment, then combining behavior alignment with token-level internal alignment. It evaluates this design across benchmarks and controlled analyses of alignment strength and instruction diversity.

3.1 Do SLMs Hear Speech as They Read Text?: ASR-trained SLMs can ignore textual instructions, while behavior alignment leaves internal speech–text representation alignment implicit.Existing models also show low CKA and weak or inconsistent token-wise similarity patterns between mapped speech features and text embeddings.
3.2 Structural Differences Between Speech and Text: Speech and text differ in both semantic structure and sequence length, motivating explicit treatment of their structural mismatch.Speech encodings are continuous, temporally varying, and substantially longer than discrete textual representations; prior methods mainly reduce length through compression or downsampling.
3.3 Our Approach: The framework dynamically allocates queries to match mapped speech-feature length to target text length during training, decoupling length alignment from semantic alignment.At inference, a speech-rate predictor estimates target token length for query allocation.
3.3 Our Approach: The approach achieves competitive performance across AIR-Bench Chat, SpeechR, MMSU, and Speech-IFEval, including comparisons with closed-source models.The benchmark table reports multiple task-specific metrics, while the ablation table reports performance and relative changes against the full model.
3.3 Our Approach: The model combines behavior alignment with a cosine-based token-level internal alignment loss over corresponding speech and text positions.The internal loss is computed at the input embedding layer and four evenly spaced hidden layers, then averaged across layers; λ controls its relative strength.

4 Experiments

Experiments evaluate structural alignment, benchmark performance, ablations, and instruction diversity. The proposed model achieves stronger representation similarity and competitive results while showing that alignment components and their strength must be balanced.

4.3 Main Results: The model shows stronger structural similarity between speech and text representations, with higher CKA scores and more consistent token-level similarity patterns.
4.3 Main Results: The model achieves competitive performance on most benchmarks, including comparisons with strong closed-source models.
4.3 Main Results: The model outperforms SALMONN on most benchmarks, with a particularly large improvement in forgetting rate on Speech-IFEval.
4.3 Main Results: Despite substantially less training data, the model achieves competitive performance and surpasses Qwen2-Audio-Instruct on several SpeechR and Speech-IFEval subsets.
4.4 Ablation Studies: Removing either alignment term causes over 10% performance degradation, with a larger drop when behavior alignment is excluded.
4.4 Ablation Studies: Increasing internal-alignment strength improves CKA, but excessively large values degrade performance, indicating that alignment is beneficial only at an appropriate strength.
4.4 Ablation Studies: Performance improves with instruction diversity, particularly when more complex instructions such as Keyword Extraction are introduced.
4.4 Ablation Studies: Alternative MSE and InfoNCE alignment losses raise CKA but degrade performance by approximately 15%, showing that representational similarity does not guarantee downstream improvement.

4.5 How SLMs Understand Speech-Specific Information

The analysis tests whether internal alignment preserves speech-specific information by comparing attention to emotional speech, neutralized speech, and text. Results indicate that the model retains sensitivity to emotional expression despite token-level alignment.

4.5 How SLMs Understand Speech-Specific Information: The analysis compares attention patterns for real emotional speech, neutralized cloned speech, and text during emotion classification.
4.5 How SLMs Understand Speech-Specific Information: Real speech produces strong activations around emotional-expression regions, while neutralized speech weakens those activations.
4.5 How SLMs Understand Speech-Specific Information: Text inputs show the strongest activations on punctuation tokens, contrasting with the emotion-related regions highlighted in real speech.
4.5 How SLMs Understand Speech-Specific Information: The findings suggest that token-level internal alignment does not eliminate the model’s ability to capture speech-specific cues.
4.5 How SLMs Understand Speech-Specific Information: Speech-text alignment should be treated as a balanced objective rather than maximized, because overly strong alignment can compromise overall performance.
4.5 How SLMs Understand Speech-Specific Information: Understanding how linguistic content and speech-specific information are encoded and balanced remains an important direction for future work.

5 Conclusion

The paper finds that current SLM speech representations remain weakly aligned with text because structural differences between the modalities persist. Its framework decouples length matching from semantic alignment and achieves stronger alignment with competitive benchmark performance.

Current SLM speech representations remain weakly aligned with text despite strong downstream performance, reflecting persistent structural differences between the modalities.
The proposed framework decouples length matching from semantic alignment while encouraging closer speech-text correspondence.
Across multiple benchmarks, the approach improves representational alignment while achieving competitive performance against strong baselines.
The findings highlight the importance of addressing structural differences between speech and text in SLM training.

6 Limitations

The limitations concern how broadly the alignment findings generalize, how internal information is characterized, and whether the approach extends beyond speech-text pairs.

The appropriate strength of internal alignment may depend on the target task, model architecture, and training data.
Improving internal speech-text alignment alone does not necessarily guarantee better downstream performance.
The work does not fully characterize how linguistic content and speech-specific information are jointly encoded and balanced inside the model.
The study focuses on speech-text correspondence, while broader audio settings often lack a direct textual counterpart for alignment.

A.1 Further Details on Dataset and Implementation

The training setup combines speech transcriptions with speech-specific annotations, filters generated responses, and uses a parameter-efficient adaptation strategy. Implementation details include the training hardware, optimization settings, and evaluation of a speech rate predictor.

Training datasets pair speech transcriptions with emotion, intent, and gender annotations for response generation.
The instruction set is based largely on prior work, excluding Speech Translation because it yields too many language pairs.
Only instruction–response pairs receiving a score of 3 or higher on a 5-point LLM-as-a-judge scale are used for training.
The modality adapter uses a 512-query Q-former with 2 transformer layers and 4 attention heads, while freezing the speech encoder and applying LoRA with r = 2 and α = 2 to the LLM.The model has approximately 350M trainable parameters.
Training uses 8 NVIDIA H100 GPUs, per-device batch size 10, 30 gradient accumulation steps, λ = 0.1, 8K steps, and a learning rate of 5 × 10−5.Optimization uses AdamW with β1 = 0.9, β2 = 0.99, and ϵ = 1e−08; the speech-rate predictor is evaluated with L1 distance and Pearson correlation against ground-truth token lengths.

A.2 Additional Details for Speech-Specific Information Analysis

The speech-specific information analysis examines how speech and transcription inputs attend to emotion labels. It constructs controlled prompts, averages attention across heads and label tokens, and visualizes the resulting maps.

The emotion instruction requires selecting exactly one label from nine specified emotion categories.
Prompts include a system prompt and emotion instruction, with the ground-truth label appended to analyze attention toward label tokens.
For each layer, attention weights are averaged across grouped-query attention heads and visualized for speech inputs and text transcriptions.
When emotion labels split into multiple tokens, their attention weights are averaged and displayed as one label; label-position weights are zeroed in Figure 5 for clarity.

A.3 Artifact Licenses

The paper uses publicly available datasets, benchmarks, and pretrained models under their respective licenses and terms of use. It reports ablations covering λ and instruction count, including representation similarity and benchmark performance.

The paper does not redistribute original datasets, benchmark data, or model checkpoints, and follows the usage conditions of pretrained models and codebases.
The evaluated benchmarks include Air-Bench, SpeechR, MMSU, and Speech-IFeval, used solely for research evaluation.
The λ ablation reports performance across multiple benchmarks together with representation similarity measured by CKA.
All metrics improve consistently as the number of instructions increases, with the largest gains at instructions = 6.Higher is better for all metrics, while Speech-IFeval is better when closer to zero.
Tables 9 and 10 summarize the datasets and instruction set used in the study.