Source-linked AI summary
Do Audio Language Models Hear and Read Distinctive Features Alike?
Yuanhao Chen, Peter Chin
TL;DR
Audio language models route speech and text through one decoder, but whether it represents distinctive features alike across those streams is unclear. The paper compares feature directions from heard and read minimal pairs against random-pair references, finding corrected cross-stream agreement only for voicing in two Qwen2.5-Omni models. Across-language audio agreement is broader, and the model family rather than size predicts which stream represents a feature.
Problem
It is unclear whether a shared decoder represents a distinctive feature in the same direction when its phoneme is heard and when it is read.
Source: §1 · p. 1Method
The study compares directions from minimal-pair offsets in audio and text, evaluating their cosine against references built from random pairings.
Source: §1 · p. 1, §2.3 · p. 2Results
Only voicing in the two Qwen2.5-Omni models exceeds the reference after correction; their c values are +0.40 and +0.45 (q = 0.011), with all 14 languages positive in both.
Source: §3.1 · p. 3, §4 · p. 4Takeaways & Limitations
Model family, rather than model size, predicts which stream represents a feature.
Source: §1 · p. 1, §3.2 · p. 3With only two model families, the study cannot separate encoder architecture’s effect from whether the encoder was trained.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Audio language models pass speech and text through a single decoder. We ask whether that decoder represents a distinctive feature in the same direction when a phoneme is heard and when it is read. For minimal pairs of phonemes differing in one feature, we take the offset between the two members' mean representations. Averaging those offsets gives a direction for each stream, and we measure the cosine between the two. Because the two streams already agree about arbitrary phoneme pairs, we compare every measure against a reference built from random pairings rather than against zero. We apply this to 6 models, 7 features and 15 languages from 11 families. Only voicing in the two Qwen2.5-Omni models exceeds that reference after correction for multiple testing, and the reference varies by a factor of seven between models. In three of the six models, voicing has one direction in audio across the 14 languages with enough minimal pairs to measure it, and every language pair agrees in two of them. The model family, not the model size, predicts which stream represents a feature.
1. INTRODUCTION
Audio language models process speech and text through separate input routes but a shared decoder, raising the question of whether distinctive features have the same representation in both streams. The study compares feature directions across heard and read minimal pairs, using random pairings as a reference; only voicing in the two Qwen2.5-Omni models exceeds that reference after multiple-testing correction.
1. INTRODUCTION
- The study asks whether a shared decoder represents a distinctive feature in the same direction when its phoneme is heard and when it is read.Speech enters through an audio encoder and text through a tokeniser, but both streams pass through one decoder.Source: §1 · p. 1
- Earlier ALAS work measures temporal binding between audio frames and transcript tokens, rather than the geometry of an individual feature.Prior work also reports feature additivity, depth profiles, and multilingual latent units in self-supervised speech models; ALAS is.Source: §1 · p. 1
- Only voicing in the two Qwen2.5-Omni models exceeds the random-pair reference after correction for multiple testing.The reference varies more between models than any feature effect; three of six models also show one audio voicing direction across measurable languages.Source: §1 · p. 1
- Feature directions are compared using minimal pairs across 15 languages from 11 families.Each pair differs in one distinctive feature, and its offset is compared between heard and read streams.Source: §1 · p. 1
2. METHOD
The method samples eligible languages and models, locates phoneme occurrences in audio and text, and compares feature directions across streams. It uses random-pair references and language-level tests to assess whether cross-stream and cross-language agreement exceeds those references.
2.1. Corpus, languages and models
- Forty of 47 corpus languages qualify, and the first 15 ranked by feature coverage and family representation are selected across 11 families.Eligibility requires distributed audio, linked segments, and sufficient minimal pairs; two pairs suffice to measure a feature.Source: §2.1 · p. 1
2.2. Locating a phoneme in audio and in text
- Audio occurrences are located by mapping decoder positions to time, while text occurrences are located by overlapping phoneme characters with token spans.Audio positions are averaged over the occurrence span; text averages every token whose span overlaps the IPA characters.Source: §2.2 · p. 2
2.3. Feature directions from minimal pairs
- For each feature, the method forms directions from minimal-pair offsets in each stream and measures their same-layer cosine in the shared decoder space.Subtraction removes what pair members share, including the large mean of anisotropic representations.Source: §2.3 · p. 2
2.4. What each measure is compared against
- Because arbitrary-pair measures are not centered on zero, each observed measure is compared with references from 2,000 random pairings.The same pairing is used for both streams within each repetition; cross-feature comparison separately tests whether agreement is specific to the feature.Source: §2.4 · p. 2
2.5. Testing across languages
- Across languages, the method takes the largest median cosine over layers and tests each model-feature combination against references, reporting Benjamini–Hochberg q-values across 42 tests.It also measures agreement between languages’ directions by shuffling which feature each language’s direction belongs to; Table 2 reports median c and reference and language counts.Source: §2.5 · pp. 2–3, Table 2 · p. 3
3. RESULTS
Only voicing in the two Qwen2.5-Omni models exceeds the random-pair reference after multiple-testing correction, while the reference itself varies substantially between models. Across models, features can be represented differently in audio and text, and agreement across languages does not always track representation within a language.
3.1. Features with matching directions in audio and in text
- +0.40 and +0.45 are the matching-direction voicing values for Qwen2.5-Omni, both with q = 0.011; no other model-feature combination reaches q ≤0.05.All 14 languages are positive in both models, and each exceeds its cross-feature value.Source: §3.1 · p. 3
- Qwen2-Audio’s +0.54 voicing median is below its +0.75 reference, and reference values vary from +0.11 in Gemma 4 E4B to +0.75 in Qwen2-Audio.No median exceeds its own reference by more than 0.10.Source: §3.1 · p. 3
- Qwen2.5-Omni-7B’s +0.27 height value is below its +0.36 reference despite all 11 languages being positive and exceeding the cross-feature value.The reference tests against arbitrary phoneme pairs; the cross-feature value tests whether agreement is specific to that feature.Source: §3.1 · p. 3
3.2. Which stream represents the features better
- Qwen2.5-Omni represents voicing in audio at 0.84 and 0.85 of layers, while Gemma represents height and length more in text than audio.For Gemma, the reading-stream fractions are 0.39 to 1.00 of layers versus 0.00 to 0.08 when hearing; the family, not model size, predicts the pattern.Source: Table 3 · p. 3, §3.2 · p. 3
- Model size does not predict median c: the smaller Qwen2.5-Omni and Gemma models have larger voicing medians, while the larger Qwen2.5-Omni model has larger height and length medians.For voicing, the smaller-versus-larger medians are +0.45 versus +0.40 in Qwen2.5-Omni and +0.11 versus +0.07 in Gemma.Source: §3.2 · p. 3
- Gemma 4 E2B represents length in heard speech at only 0.01 of layers, yet languages agree on one length direction at 0.83 of layers.A language’s direction is the mean of several offsets, so it can be shared across languages even when the offsets within a language disagree.Source: §3.2 · p. 4
3.3. One direction across languages, or one per language
- Three models show above-reference across-language agreement for audio voicing at every layer, and the two Qwen2.5-Omni models agree across all 91 language pairs.Within-language representation and across-language agreement correlate at +0.67 in audio and +0.73 in text; Audio-Flamingo 3 has across-language agreement at every layer despite no within-language voicing layer exceeding reference for the median language.Source: §3.3 · p. 4
- Text length has the highest across-language agreement, +0.88 or higher in every model, peaking at the first or second layer in all six.Length is marked by adding “:” rather than changing a symbol, so the shared direction is present in token embeddings and may be trivial rather than phonological.Source: §3.3 · p. 4
4. DISCUSSION
After correction, cross-stream feature-direction agreement is limited to voicing in the two Qwen2.5-Omni models, while RSA finds above-reference phoneme-similarity alignment in every model. The discussion qualifies these results with occurrence-context confounds and a same-layer comparison that may miss features encoded at different depths.
4. DISCUSSION
- Because offsets include all occurrences, systematic differences in neighboring-phoneme distributions remain a confound; their correlation with c is −0.01 for voicing, +0.05 for height, and +0.06 for length.Source: §4 · p. 4
- At the embedding layer, c never exceeds +0.10, so Qwen2.5-Omni’s voicing agreement arises in the decoder rather than being inherited from the tokeniser.Source: §4 · p. 4
- Every model has a layer where all 15 languages exceed the shuffled-correspondence reference in RSA, although a near-zero feature-direction cosine does not rule out shared structure.RSA compares phoneme-similarity matrices without requiring minimal pairs.Source: §4 · p. 4
- The comparison takes both directions at the same layer, so a feature encoded early in audio and late in text could appear absent; testing that requires searching layer pairs.In 12 of 18 combinations, no layer has a majority of languages exceeding the reference in both streams.Source: §4 · p. 4
- After correction, cross-stream agreement exceeds the random-pair reference only for voicing in the two Qwen2.5-Omni models; in three models, languages share one voicing direction in audio.Source: §4 · p. 4