PaperlayerView exact revision on arXiv

Source-linked AI summary

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh, Bhiksha Raj

arXiv:2609.00727v1cs.SDcs.AI

TL;DR

Audio-language models are expected to understand speech beyond its literal words, but it is unclear whether they use speaking style. The paper traces style information through four models using complementary representation, probing, tone-prediction, and leakage analyses. It finds that models strongly encode style in late audio-encoder layers but often underuse it in final predictions, with behavior shaped by decoder architecture and training objective.

  • Problem

    It is unclear whether audio-language models capture and use speaking style beyond spoken content, despite speech carrying paralinguistic information essential to communication.

  • Method

    The study traces speaking-style information through four audio-language models using CKA, linear probing, open-ended tone prediction, leakage analysis, and controlled Expresso style pairs.

  • Results

    All four models encode speaking style strongly in the late audio encoder, but later stages differ in how much style information reaches model outputs.

  • Takeaways & Limitations

    The results identify a gap between paralinguistic information that audio-language models encode and the information they use in predictions.

  • Takeaways & Limitations

    The study uses four Expresso speakers, so its findings may not generalize to larger populations; leakage ratios are also biased by content-style entropy asymmetry.

Abstract

from arXiv · show

Audio language models are designed to understand speech, yet it remains unclear whether they capture how something is said beyond what is said. We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B, using the Expresso dataset with controlled speaking styles. We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output. All models strongly encode speaking style in the late encoder, that is, the top third of the audio encoder's layers, but this information is consistently degraded before reaching the output. The projector reshapes representation geometry without removing information, while decoders differ in how much style they preserve depending on architecture and training objective. At the output level, models fall into two behaviors. Some are content driven, where predictions depend mainly on text. Others are acoustic driven, where predictions vary with speaking style. The leakage metric quantifies this difference, and qualitative results confirm it. Overall, we identify a gap between what models encode and what they use, highlighting a key limitation in current audio language models.

1 Introduction

The paper asks whether audio-language models use paralinguistic information such as speaking style, rather than relying mainly on literal speech content. It studies this question across four models and identifies a gap between information encoded internally and information used in predictions.

  • Motivation: Modern audio-language models can process speech, audio, and visual inputs, but often lag behind text-only models when understanding information beyond literal content.
  • Motivation: Paralinguistic information includes prosody, emotion, speaking rate, voice quality, and other meaning conveyed by how speech sounds.Speech can also encode biometric signals such as identity, age, gender, accent, and emotional state.
  • Research Aim: The study analyzes how speaking-style information is encoded and lost across the internal stages of four audio-language models.Its scope is speaking style, specifically prosody and tone of delivery; speaker identity and accent are excluded.
  • Approach: The analysis combines CKA, linear probing, open-ended tone evaluation, and content-prosody leakage across the models’ processing stages.
  • Central Finding: All models strongly encode paralinguistic information in the audio encoder, but later processing determines how much of it is used.The paper characterizes this as models hearing paralinguistic information without consistently using it.

2 Related Work

Prior work shows that speech encoders capture rich information beyond content, while interpretability tools reveal how information is structured inside networks. This paper extends those directions by tracing paralinguistic information through the full audio-language-model pipeline.

  • Existing Speech Studies: Speech-representation research has examined emotion recognition, speaker traits, and prosody, with benchmarks showing that speech encoders capture rich paralinguistic information.
  • Analysis Tools: CKA and linear probing have been used to compare representations across layers and test whether specific information can be decoded from them.
  • Research Gap: Analyses of how information moves through complete audio-language-model pipelines remain limited, especially from audio input to final output.
  • This Work: The paper traces paralinguistic information from the audio encoder through the projector to the final prediction, locating preservation, transformation, and eventual disregard.

3 Paralinguistic Analysis Framework

The framework examines representation similarity, information accessibility, and output behavior to determine whether models encode speaking style and whether their predictions depend on content or acoustic style.

  • Framework Overview: The analysis has three components: representation similarity, information accessibility, and output-level behavior.
  • Representation Accessibility: Mean-pooled hidden states from every layer provide fixed-size representations for controlled comparison across all four models.The same pooling procedure is used across models, although it discards temporal dynamics that may matter for styles such as laughing.
  • Representation Similarity: CKA compares layer-wise representation geometry through centered Gram matrices, with high CKA indicating similar mappings and low CKA indicating stronger separation.
  • Information Accessibility: Linear probes test whether speaking style can be decoded from each layer, while open-ended tone predictions are mapped to predefined tone categories.
  • Output-Level Behavior: The leakage ratio compares normalized mutual information between predicted tone and text content against predicted tone and speaking style.Values near 1 indicate mostly content dependence, while values near 0 indicate greater reliance on acoustic style.
  • Output-Level Behavior: Leakage ratios should be interpreted comparatively across models because content has higher entropy than the seven-category style variable.

4 Experimental Setup

The experiments compare four audio-language models spanning different encoder, decoder, and training-objective choices. Controlled same-content, same-speaker clips spoken in seven styles enable evaluation of style-specific information.

  • Model Selection: The study compares Whisper-large-v2, Qwen2-Audio-7B-Instruct, Qwen2.5-Omni-7B, and Chroma across major audio-language-model design axes.Qwen2-Audio and Qwen2.5-Omni share similar encoder and language-model backbones but differ in training objective.
  • Terminology: The late encoder denotes the top third of an audio encoder’s layers, such as layers 20–31 in Whisper-large-v2’s 32-layer encoder.
  • Dataset: Expresso contains identical utterances spoken in different styles, allowing style to vary while textual content remains fixed.The analysis uses seven labels: confused, default, enunciated, happy, laughing, sad, and whisper.
  • Controlled Evaluation: Cross-style pairs hold text and speaker constant while varying paralinguistic delivery.
  • Evaluation: Linear probes use leave-one-speaker-out cross-validation to reduce speaker leakage during evaluation.

5 Results

All four models encode speaking style strongly in late audio-encoder layers, but style information changes or degrades through later stages and is unevenly reflected in tone outputs. The projector mainly reshapes representation geometry, while decoder behavior and training objectives distinguish content-driven from acoustically sensitive predictions.

  • Whisper: A Style-Encoding ASR Baseline: Whisper’s style-pair CKA remains 0.92–1.00 across 32 encoder layers, while probing rises from ∼51% to ∼83%, showing style encoding despite geometric similarity.This establishes that high CKA does not imply absent information.
  • Encoder Encoding: All Models Peak at Late Encoder: All models peak at ∼82–85% probing accuracy in the late encoder, far above the 14.3% chance level.Qwen2-Audio peaks near encoder layer 20, Omni near layer 22, and Chroma around layers 18–20.
  • The Projector: Geometric Disruption Without Information Loss: The projector causes sharp CKA discontinuities, yet probing remains ∼78–81%, indicating geometric reorganization without removing linearly accessible style information.For Qwen2-Audio, a 3× dimensional expansion changes geometry while preserving linear decodability.
  • Decoder Divergence: Three Distinct Patterns: Decoder trajectories diverge: Qwen2-Audio stays near ∼78–80%, Omni declines from ∼81% to ∼74%, and Chroma follows a drop–recovery–drop pattern.Chroma falls from ∼85% to ∼58%, recovers to ∼80%, then ends near ∼60%.
  • Output-Level Collapse: Two Categories: Encoder probing reaches ∼82–85%, but output tone accuracy is only 19.7%–53.7%, revealing a substantial encoding-to-use gap.Omni leads acoustically extreme styles, while Qwen2-Audio performs best on clarity styles and Chroma fails on subtle styles.
  • Content-Prosody Leakage: Leakage is 97.7% content-driven for Qwen2-Audio, 84.4% for Chroma, and 27.2% for Omni; qualitative outputs likewise show Omni varying with speaking style.Explicit emotion words content-lock predictions across styles, validating the metric’s interpretation.

6 Discussion

Across models, strong encoder-level style information is consistently degraded or underused at the output, with decoder behavior shaped mainly by training objective and representation geometry.

  • 6.1 The Encoder-Output Gap: ∼82–85% encoder probing accuracy versus 19.7%–53.7% output tone perception reveals a reproducible encoder–output gap.The gap appears across methods, models, and speaking styles, indicating a structural pipeline property.
  • 6.2 Projector Effects: Projectors can reshape representation geometry without removing style information, although their effects differ across architectures.Qwen2-Audio maintains ∼78–81% probing accuracy at the projector, while Omni shows gradual information erosion.
  • 6.3 Training Objectives: Training objectives distinguish acoustic-driven from content-driven behavior: speech generation preserves acoustic style, whereas text prediction provides no such incentive.Chroma preserves style for synthesis but sharply collapses it at the audio-to-language boundary for text prediction.
  • 6.3 Component Boundaries: Chroma’s style information collapses at the thinker audio-to-LM boundary, although later probing suggests the backbone partially reconstructs style from context.The collapse explains Chroma’s high leakage ratio and flat CKA at that boundary.
  • 6.4 Closing the Encoder-Output Gap: GRL training reduced content NMI from 0.234 to 0.203 but also reduced style NMI from 0.006 to 0.004, leaving leakage nearly unchanged at 0.977 to 0.979.The result supports linear entanglement of style and content and motivates nonlinear projectors or encoder-level disentanglement.
  • 6.5 Future Directions: The paper motivates auxiliary style objectives and nonlinear or contrastive training on same-content, different-style utterances.These directions aim to reduce content dependence while increasing acoustic sensitivity.
  • 6.6 Limitations: The study is limited to four Expresso speakers, speaking style, and text outputs, so broader population and speech-generation generalization remain open.LOSO prevents speaker leakage, but generalization to larger populations is not guaranteed.

7 Conclusion

The paper traces speaking-style information through four audio-language models and finds strong late-encoder encoding but systematic output underuse. Training objective primarily determines whether decoders preserve or collapse acoustic style.

  • 7 Conclusion: Four complementary methods trace speaking style from late audio-encoder representations through projectors and decoders to final outputs.The analysis covers four audio-language models.
  • 7 Conclusion: ∼82–85% late-encoder probe accuracy contrasts with systematic output underuse, while projectors transform geometry rather than acting as information bottlenecks.Decoder behavior ranges from preservation to erosion or collapse across models.
  • 7 Conclusion: Training objective is the primary determinant of content-driven versus acoustic-driven output behavior.The conclusion motivates paralinguistic-aware objectives and architectures that preserve expressive information.

LLM Usage Disclosure

The authors used Claude for research discussion and writing assistance, while retaining authorship of the experiments, code, results, and scientific conclusions. The section also specifies implementation details for CKA computations and pair aggregation.

  • Claude was used for research discussion and writing assistance, while the authors conducted the experiments and produced the code, results, and final scientific conclusions.
  • CKA uses mean-centered linear kernels on mean-pooled hidden states from paired clips.For each pair, hidden states are pooled over T before aggregation into matrices X and Y.
  • All 31,896 style-paired clips are included in aggregation, with pairs below a 10−10 denominator threshold excluded as degenerate.

B Probe Training Details

The appendix specifies probe training, nonlinear comparisons, tone mapping, layer coverage, and CKA analysis details used to evaluate paralinguistic information.

  • B Probe Training Details: Logistic-regression probes use L2 regularization, balanced classes, four LOSO speaker folds, and per-fold feature standardization.Results are reported as mean ± standard deviation across folds.
  • C Nonlinear Probing: The nonlinear probe is a single-hidden-layer MLP evaluated at each model’s key boundary layers under the same LOSO protocol.Table 4 compares linear and MLP accuracy across 19 selected layers.
  • C Nonlinear Probing: −4.1 to +0.4 points is the full MLP-minus-linear range across 19 boundary layers, with a mean difference of −1.0 points.No layer shows substantively higher nonlinear accuracy, supporting linear probes as near the decodability ceiling.
  • Probe and Tone Evaluation: Tone evaluation prompts models to describe speech content and tone in English, then maps responses to predefined categories using first-match keyword rules.The categories include angry, happy, sad, neutral, whisper, laughing, confused, disgusted, surprised, and sleepy.
  • Layer Coverage: Chroma’s full architecture has 88 layers, but the excluded thinker LM leaves 52 probed layers after sequential renumbering.The exclusion follows complete style collapse at the thinker audio-to-LM boundary.

G Content-Prosody Leakage: Additional Details

The leakage analysis accounts for label-entropy differences and tests whether content granularity biases model comparisons. Content-locked cases show when explicit emotion words make predictions content-driven.

  • Normalized mutual information (NMI) adjusts leakage ratios for differences in label entropy, while chi-squared tests use the full predicted-tone-by-style contingency table.
  • Higher entropy in raw utterance content biases absolute leakage ratios toward 1.0, so content was also mapped to seven TF-IDF/k-means semantic clusters.The clustered version matches the seven-class style label, enabling a direct bias check.
  • Leakage-ratio model ordering is unchanged when raw per-utterance text is replaced by seven semantic content buckets.
  • Seven content-locked utterances shared perfect agreement across all three models and styles, and all contained explicit emotion words.These examples predictably collapse model behavior toward content-driven tone prediction.

H GRL Intervention: Training Details

The intervention trains only Qwen2-Audio’s projector with an adversarial GRL objective to preserve style information while suppressing content information. Direct supervision produced near-perfect style classification, partial content suppression, and proportionally suppressed generation outputs.

  • Intervention design: Qwen2-Audio’s multimodal projector was trained with a Gradient Reversal Layer objective while all other model weights remained frozen.
  • Architecture and labels: The projector used separate style and content heads, with content labels formed from 20 TF-IDF-plus-KMeans semantic clusters.The heads operated on the mean-pooled projector output.
  • Loss and schedule: The adversarial coefficient λ ramped from 0 to 1 during training, and no language-modeling loss was used.The authors omitted language-modeling loss to avoid generation collapse observed in a preliminary run.
  • Training setup: Training used five epochs of AdamW with a cosine schedule, 5% warmup, batch size 8, and 8 V100-32GB GPUs on stratified Expresso splits.The dataset contained 8,511 training clips and 2,123 validation clips under an 80/20 split.
  • Results: 93.3% style-head validation accuracy confirmed near-perfect style classification, while content accuracy stabilized at 14–22% versus 5% chance.The epoch 2 checkpoint, with 14.2% content accuracy, was selected for generation evaluation, whose blind-prompt outputs showed proportional suppression.
Loading 2609.00727v1…