Interleaved Speech Language Models Latently Work In Text

Talia Sternberg; Gallil Maimon; Yossi Adi

Source-linked AI summary

Interleaved Speech Language Models Latently Work In Text

Talia Sternberg, Gallil Maimon, Yossi Adi

arXiv:2606.22473v1Jun 21, 2026cs.CLcs.LGcs.SDeess.AS

TL;DR

How speech and text interact inside interleaved speech language models remains unclear. Using the logit lens, this paper finds that models implicitly transcribe speech into text, predict in text space, and then return to speech tokens.

Problem
The internal interplay between speech and text in interleaved speech language models remains unclear despite improved speech-to-speech capabilities.
Method
The paper applies the logit lens to project intermediate hidden states into vocabulary logits and trace layer-wise latent dynamics.
Results
Interleaved speech language models implicitly transcribe speech, form next-word hypotheses in text space, and project representations back into speech-token space.
Takeaways & Limitations
Implicit transcription positively correlates with factual knowledge retrieval from speech, while emerging most clearly from combining interleaved data with a pretrained textual prior.
Takeaways & Limitations
The precise mechanism remains unidentified, and the correlation with factual knowledge reaches only ρ = 0.70 without causal testing.

Abstract

from arXiv · show

Speech language models (SLMs) have been extensively studied, with the common paradigm incorporating text data and pre-trained text LMs. A leading approach is speech-text interleaving in which models are trained over sequences containing both speech and text tokens, aiming to boost even speech-only capabilities. Yet the way these two modalities interact in the model latent space remains unclear. In this work, we analyze interleaved speech-text LMs from different model families and sizes through the scope of the logit lens to provide such insight. We reveal that these models go through an implicit transcription phase in which the text token of the spoken word becomes decodable in intermediate layers, despite not being trained for speech recognition. The transcription of the word appears as one of the top candidate words for as much as 77\% of the data. Following this stage, the models proceed to predict the next word in the text space before transforming back to the speech domain. We finally analyze the role of interleaving data, and initializing from text LMs in eliciting this behavior, as well as seeing how this correlates with spoken knowledge abilities. Our analysis sheds light on the internal mechanisms underlying the relationship between speech and text modalities and could shape SLM optimization.

1 Introduction

The paper investigates why speech-text interleaved language models exhibit stronger semantic capabilities by analyzing their latent representations with the logit lens. It finds that these models implicitly transcribe speech and reason in text, with this behavior requiring both text-LM initialization and interleaving data and relating positively—but incompletely—to spoken factual knowledge.

Motivation: Speech-only SLMs show limited semantic and knowledge capabilities, motivating approaches that integrate text data and pretrained text language models.Speech-text interleaving trains models on streams containing both speech and text tokens and improves semantic scaling relative to speech-only training.
Method and central finding: Logit-lens analysis shows that interleaved SLMs develop a meaningful textual latent space in which speech representations become decodable as transcriptions and later text hypotheses.The analysis projects intermediate hidden states into the token vocabulary to examine cross-modal latent dynamics.
Conditions for emergence: Both initialization from a trained text LM and interleaving training data are necessary for implicit latent transcription to emerge.When both conditions hold, the phenomenon appears across model families, model sizes, and training compute budgets.
Relation to spoken knowledge: Implicit transcription is positively associated with factual knowledge retrieval from spoken input but does not fully explain spoken knowledge abilities.Transcriptions can build gradually over a spoken word and may include acoustic errors.

2 Background

Interleaved speech-text SLMs jointly model discrete speech units and text tokens in a single sequence. They are trained to generate cross-modal continuations, assuming semantic information transfers between modalities.

2 Background: Interleaved speech-text SLMs mix speech units and text tokens within the same sequence.They model both modalities in a single token stream.
2 Background: A typical speech-text LM pipeline contains speech-to-unit, joint language-model, and unit-to-speech components.The speech-to-unit module converts raw audio into discrete units, often using self-supervised representations such as HuBERT followed by quantization.
2 Background: Training assigns each word in time-aligned speech either the speech or text modality, grouping consecutive same-modality words into spans.Text spans are tokenized as text, while speech spans are replaced by discrete units.
2 Background: This setup trains SLMs to generate cross-modal continuations within sentences under the assumption that semantic information transfers across modalities.The approach uses one stream of tokens while combining speech and text spans.

3 Approach

The approach uses the logit lens to track information in speech-LM latent representations across layers, adapting analysis to variable-length speech spans through word alignment and max aggregation. It also evaluates spoken factual knowledge comparatively and studies how initialization and training-data composition affect implicit transcription.

Logit-lens analysis: The logit lens projects intermediate hidden states through the learned output projection to obtain layer-wise vocabulary distributions and identify linearly decodable tokens.This tracks how predictions develop across transformer layers rather than only at the final layer.
Speech-latent interpretation: Because spoken words span variable-length speech-token sequences, the method aligns speech positions with transcriptions and aggregates hidden-state scores at the word level.For modality preference, it takes the maximum speech- and text-token mass across positions; for top-k detection, it checks all positions.
Speech-latent interpretation: Max aggregation targets localized transcription-like signals that may appear strongly in only one or two positions within a speech span.The procedure is motivated by detecting informative positions rather than averaging less-informative tokens.
Knowledge evaluation: The study evaluates spoken factual knowledge with matched true and counterfactual facts, scoring a model correct when log p(fact) > log p(counterfactual fact).The manually constructed prompts cover elementary factual categories and are synthesized into speech with time-aligned transcriptions.
Controlled training experiments: Controlled Llama 3.2-3B experiments vary text-LM versus random initialization and speech-only, speech+text, versus speech–text interleaved training.All variants share the same architecture and optimization setup, enabling comparisons across initialization and data composition.

4 Results

The results show that interleaved speech language models route speech through a text-like latent workspace: intermediate layers encode current-word transcriptions and textual continuations before returning to speech-token predictions. This implicit transcription is strongest with text pretraining and moderate interleaved speech-text supervision, and correlates positively with speech-based knowledge retrieval.

Latent speech-to-text-to-speech processing: Speech prompts follow a three-stage pattern: speech-token predictions in layers 0–2, text-token predictions in layers 2–25, and speech-token predictions again in layers 26–28.Text inputs remain concentrated on text tokens throughout, and the pattern persists after restricting analysis to the top-200 tokens.
Implicit latent transcription: Nearly 40% Recall@1 and nearly 80% Recall@50 are reached for current-word transcription in SIMS-Llama3.2-PI-1/3 by layer 23.Correct transcriptions emerge across three interleaved speech language models, while the top-50 random-overlap baseline remains at most 0.01.
Textual continuations: Nearly 40% cumulative Recall@50 is achieved for next-word prediction by fine-tuned Llama variants, compared with approximately 30% for the Qwen-based model.All models exceed 40% cumulative Recall@50 when evaluated against expected textual answers, with SIMS-Llama-3.2-PI-1/3 recovering nearly 60% of prompts at some layer; intermediate next-word targets can be ambiguous.
Ablations: Strongest transcription occurs in text-pretrained models with lower or intermediate interleaved-token fractions, whereas the 5/6 model shows a substantially weaker signal.Models without interleaved data or with random initialization show weaker transcription, indicating that sufficient interleaved supervision aligns spoken units with textual representations rather than merely exposing models to text tokens.
Knowledge ability: Current-word and next-word transcription correlate positively with knowledge ability, with Spearman ρ = 0.70, p = 0.00526 and ρ = 0.65, p = 0.0119, respectively.The relationship is incomplete because the transcription score is coarse and knowledge likelihood may also depend on model-specific factors.
Gradual transcription: Implicit transcription can emerge incrementally, with partial acoustic evidence decoded into approximations before converging to the full spoken word.Examples include “white” becoming “why,” “lime” becoming “lie,” “kingdom” becoming “king,” and “Pakistan” becoming “pack.”

5 Related Work

Related work interprets latent computation across languages and modalities through shared intermediate representations, while speech-text LM studies examine modality gaps and transcript-like processing. Mechanistic ASR analyses trace transcript-token emergence, but this work targets generative speech-text LMs that both consume and produce speech.

Interpreting Latent Computation in LMs: Multilingual LLMs may share information through English-centric latent representations, with related English-mediated processing and steering effects reported across studies.Related multimodal work similarly proposes that representations from different modalities align in intermediate layers.
Analyzing Speech-Text LMs: Speech-text LM research analyzes modality gaps arising from mismatched modality structures, including redundant speech tokenization that can produce diffuse attention.Other studies find audio LMs rely on transcript-like information and resemble ASR→LLM cascades on text-sufficient tasks.
Analyzing Speech-Text LMs: Mechanistic ASR studies trace transcript-token emergence across layers using logit-lens analyses, probing, and activation patching to follow acoustic and semantic information.These studies focus on audio-to-text models or static speech-text alignment, whereas this work studies a generative speech-text LM that consumes and produces speech.

6 Discussion & Conclusion

The study concludes that speech LMs implicitly operate in a text latent through transcription and next-word prediction before returning to speech tokens, and that this behavior correlates positively with spoken factual knowledge. It also identifies unresolved questions about mechanism, causality, acoustic abilities, and the modality gap.

Discussion & Conclusion: Speech LMs implicitly transcribe speech, predict the next word in text, and project the result back to speech tokens.This latent text pathway positively correlates with spoken fact knowledge abilities.
Discussion & Conclusion: Future work should examine whether optimizing implicit transcription improves performance without harming acoustic abilities.The effect on acoustic abilities remains unresolved because avoiding explicit transcription is a key motivation for directly modeling speech.
Discussion & Conclusion: The persistent modality gap despite latent text processing may reflect transcription errors, computation spent on transcription, or insufficient temporal compression.These possibilities could make language reasoning more challenging and limit spoken abilities relative to text.
Discussion & Conclusion: The precise heads, layers, and pathways computing implicit transcription remain unidentified.The study observes transcription-like signals in intermediate layers but does not determine their underlying mechanism.
Discussion & Conclusion: ρ = 0.70 is the maximum reported correlation between transcription-like signals and factual knowledge retrieval, so it does not explain all model variation.The relationship is positive but not sufficient to account for every difference across models.
Discussion & Conclusion: The study does not causally test whether increasing implicit transcription improves knowledge retrieval.Training or intervening on models to increase implicit transcription is left for future work.

A Appendix · A.1 Common Sense Dataset

The appendix’s Common Sense Dataset section summarizes evaluation-subset statistics and provides one representative example for each subset.

A.1 Common Sense Dataset: Table 2 reports statistics for each evaluation-dataset subset alongside one representative example.The table covers the different subsets used in the evaluation dataset.

A.2 Experimental Setup

The setup trains typically text-pretrained language models on mixed speech-text sequences with a standard next-token objective. Speech spans are sampled and combined with a broad English speech mixture, while text-only data comes from filtered RedPajama.

Interleaving: Speech segments are sampled from a Poisson distribution with λ = 10 until they cover η = 0.3 of the words, producing mixed-modality sequences.The sequences are used to train a typically text-pretrained LM with standard next-token prediction.
Training Data: Training uses an English speech mixture spanning LibriSpeech, LibriLight, Vox-Populi, TED-LIUM, People’s Speech, SWC, and synthetic sTinyStories.Text-only data is taken from RedPajama and filtered with Gopher rules.

A.3 Additional Results … A.4 AI Tools Usage

Additional analyses confirm that implicit transcription and textual continuation recur across models, while their strength varies and can disappear in randomly initialized systems. The appendix also reports recall and likelihood evaluations, illustrative logit-lens cases, and the authors’ limited use of AI tools for language and code assistance.

A.3 Additional Results: Figures 7–20 show that several models exhibit implicit transcription and next-word prediction, although the strength and extent vary across models.These appendix analyses follow trends reported in the main paper.
A.3.1 Transcription Examples: Single-word examples show gradual transcription, occasional errors, and a complete failure to transcribe for the randomly initialized model.The non-transcribing case is consistent with that model’s low transcription capacity in Table 1 and Section 4.
A.3.1 Transcription Examples: The logit lens decodes transcriptions as the most probable textual token, with dark blue indicating high textual-token probability and light yellow indicating zero probability.This visualization convention applies across the appendix figures.
A.3.2 Transcription Recall For Different Top Ks Across Different Models: Recall@k analyses for Llama3.2-3B variants use k ∈{1, 5, 30}, with consistent trends reported in Tables 3–5 and cumulative patterns shown in Figure 6.The analysis covers different top-k values and models.
A.3.3 Model Evaluation: Randomly initialized models and models without interleaving underperform models with interleaving on standard likelihood speech metrics.The authors also report that their SIMS version performs comparably to the official version.
A.4 AI Tools Usage: AI tools assisted with grammar correction, sentence paraphrasing, and parts of code implementation, while the authors reviewed all content and used them only as supportive aids.The authors describe this use as responsible and limited.
A.3 Additional Results: Examples include partial-to-complete word convergence, such as “tea” to “teach” to “teacher,” and factual prompts evaluated across Llama and Qwen model variants.The figures also include transcription errors such as “line” for “lime” and a randomly initialized RST case.