Do Audio LLMs HEAR Pain? Tracing Acoustic Evidence from Speech to Language Reasoning

1 Introduction

• Strong performance on word-informative speech does not establish whether a model uses acoustic properties or mainly transcribes and reasons over lexical content.Pain can change voice characteristics such as loudness and pitch even when the words do not mention pain.Source: §1 · p. 1
• TAME-Pain contrasts pain-neutral Harvard sentences, where pain must be inferred from the voice, with spoken ratings, where pain is explicit in both words and acoustics.Comparing the conditions separates acoustic from lexical evidence without editing speech.Source: Figure 1 · p. 1, §1 · p. 2
• HEAR traces pain-related information through four stages: behavior, encoding, perception of acoustic cues, and use of cues provided as text.The framework asks whether pain is recognized, represented, perceptually distinguished, and reasoned over when cues are explicit.Source: §1 · p. 2, Figure 2 · p. 2
• Across seven LALMs, pain prediction is near chance when inferred from neutral speech but approaches ceiling when pain is stated aloud.Same-speaker references and 20-shot examples provide little improvement; probing, perception tests, and cue reinjection further examine where the acoustic signal is lost or unused.Source: §1 · pp. 2–3
• Pain-related information is decodable in early audio representations but becomes less accessible with depth; adding acoustic cue values as text improves pairwise prediction by up to 32 points.In Qwen2.5-Omni-7B, probes reach 74.1% balanced accuracy for binary classification and 45.0% for three-class classification; stronger models can perceive most cue changes but still fail to infer pain.Source: §1 · pp. 2–3

Speech-based pain assessment

• Speech-based pain studies use vocal and prosodic features to predict pain across chronic and experimentally induced conditions, while TAME-Pain adds controlled cold-pressor recordings paired with self-reported pain levels.The paper uses pain as a controlled setting to examine general-purpose LALMs rather than developing another supervised detector.Source: §2 · p. 3

Paralinguistic understanding in audio language models

• Prior work finds that strong speech recognition need not imply strong paralinguistic understanding, documenting lexical reliance and gaps across emotion, speaker, expressive, and acoustic attributes.Related studies also examine where paralinguistic information becomes less decodable and whether speaking-style information degrades before reaching model outputs.Source: §2 · p. 3

Interpreting audio language models

• Interpretability studies use sparse autoencoders, logit-lens analysis, and causal interventions; this work instead traces pain-related acoustic cues to representations and downstream behavior.Its analysis is complementary to broad searches for interpretable audio concepts.Source: §2 · p. 3

Analysis framework

• HEAR compares the same speakers and pain conditions across neutral Harvard sentences and spoken pain ratings, isolating whether evidence comes from words or acoustics.The four stages are behavior, encoding, perception of pain-associated cues, and use of those cues when supplied as text.Source: §3 · p. 3

Dataset and preprocessing

• TAME-Pain contributes 4,658 retained recordings from 51 speakers, including neutral sentence readings and spoken ratings with revised self-reported pain levels from 1–10.Cold- and warm-water recordings are pooled, noisy samples are dropped, and 88 eGeMAPSv02 features are extracted for acoustic analysis.Source: §3 · p. 3

Task settings

• Tasks use audio, transcripts, or both, with binary labels of no pain (1–3) versus pain (4–10) and three severity classes: mild (1–3), moderate (4–6), and severe (7–10).Settings include neutral Harvard sentences, spoken ratings, same-speaker pairwise comparisons, and same-speaker 20-shot examples.Source: §3 · p. 4, Figure 3 · p. 4

Models

• Seven LALMs are evaluated: four closed-source models and three open-weight models.The set includes the gpt-realtime and Gemini families, Qwen2.5-Omni-7B, Qwen3-Omni-30B-A3B-Instruct, and Kimi-Audio-7B-Instruct.Source: §3 · p. 4

Supervised probing and evaluation

• Frozen hidden states from open-weight LALMs are probed with linear, MLP, and CRNN heads, selecting layers and heads through speaker-grouped cross-validation.The speaker-independent split uses 41 training and 10 held-out test speakers; balanced accuracy is the primary metric because pain labels are imbalanced.Source: §3 · p. 4

4 RQ1: Do LALMs Recognize Pain from the Voice?

• All LALMs perform at chance on pain-neutral Harvard sentences, while the same models are near ceiling from audio when pain is verbalized.Neither same-speaker references nor 20-shot examples help detect pain from neutral sentences.Source: §4 · p. 4, Table 1 · p. 5
• For gpt-realtime, audio of neutral sentences prompts “Pain” on 85.0% of no-pain clips, compared with 6.3% from transcripts alone.The comparison indicates that audio can induce false pain responses even when the transcript is neutral.Source: §4 · p. 5
• Model errors reflect fixed response preferences rather than access to pain signals in the audio.Most models answer “No pain” on neutral audio; in pairwise comparisons most pick Clip 2 regardless of content, and 20 examples can raise “Pain” answers without distinguishing pain from no-pain clips.Source: §4 · p. 5

• 34 of 88 acoustic features were significantly associated with pain, with the strongest associations involving loudness, spectral flux, and pitch.Features were grouped into nine categories after within-speaker standardization and Benjamini–Hochberg correction.Source: §5 · p. 5, Figure 4 · p. 5
• 74.1% balanced accuracy for binary detection and 45.0% for three-class severity were achieved by probes on Qwen2.5-Omni-7B hidden states, despite near-chance model answers.The probes used frozen representations under the neutral Harvard sentence setting.Source: Table 2 · p. 6, §5 · p. 6
• Pain-probe accuracy peaks in early encoder blocks and declines through the encoder and language model for neutral Harvard sentences, but rises near ceiling for spoken ratings.Kimi-Audio shows the same pattern.Source: §5 · p. 6, Figure 5 · p. 6
• Most pain-associated acoustic feature groups are well recovered in shallow encoder layers but become less decodable with depth, including loudness and pitch within language-model layers.The analysis uses ridge regression to recover features from layer representations.Source: §5 · pp. 6–7, Figure 5 · p. 6
• Voice quality and pitch variability are pain-associated features that remain poorly recovered at every layer.Both encoder families are initialized from Whisper-largev3; the authors suggest transcription-focused training may trade acoustic cues for lexical content.Source: §5 · p. 7
• Pain-related acoustic information weakens toward the language model, while pain stated in words becomes stronger.Source: §5 · pp. 6–7

• Strong LALMs reach 72–100% accuracy on most exaggerated acoustic cue contrasts yet still fail to predict pain; weaker models stay near chance on almost every cue.Human accuracy on the test was 98.7%.Source: §6 · p. 7, Figure 6 · p. 7
• The test pairs low and high versions of the same recording, each altered in opposite directions along one acoustic feature, and asks which clip is higher on that feature.Chance performance is 50%; significance is assessed with a two-sided binomial test against chance.Source: Figure 6 · p. 7, §6 · p. 7
• Qwen2.5-Omni’s final language-model layers encode pitch level (R2 ≈0.5), although the model cannot identify which exaggerated clip has higher pitch.This illustrates that decodable information need not be available in the model’s response.Source: §6 · p. 7

• 32.2 points: adding explicit acoustic cues improves gemini-3.1-pro-preview’s binary pairwise balanced accuracy by up to this amount.The cues are attached to the prompt with raw values and explanations.Source: §7 · p. 7, Table 3 · p. 8
• Cue-only input mostly outperforms cues combined with audio, suggesting the gain may come from verbalized features rather than improved attention to raw audio.Source: §7 · pp. 7–8
• In the spoken pain-rating acoustic-feature-only setting, all four evaluated models drop close to random performance, consistent with their relying on lexical pain information for the high spoken-rating scores.Source: §7 · p. 8
• The authors report a similar pairwise improvement trend on LISTEN, especially for Qwen3-Omni.Source: §7 · p. 8
• The authors conclude that LALMs can compare pain within a speaker when acoustic cues are verbalized, but do not ground pain judgments in the audio itself.Source: §7 · pp. 7–8

• Pain-related acoustic cues are most decodable in early audio representations and fade with depth, while lexical pain information grows stronger; depth-wise loss alone does not explain the failure to infer pain.Strong models perceive most cues and some cues persist into final layers, but are still not used to infer pain.Source: §8 · p. 8
• Cue-reinjection gains are largest in within-speaker pairwise comparisons and vanish for single recordings; adding audio back often lowers accuracy.Source: §8 · p. 8
• Strong speech recognition should not be treated as evidence of strong speech understanding for tasks that depend on paralinguistic information.Pain is presented as one controlled case study of this broader issue.Source: §8 · p. 8

9 Conclusion

• The study focuses on whether models use pain-related information carried by the voice rather than relying on lexical content alone.Source: §9 · p. 8
• Pain-related information and several associated acoustic cues are most decodable in early audio encoder representations but become less accessible with depth.The authors conclude that current LALMs encode paralinguistic evidence without making it available to language-level reasoning.Source: §9 · p. 8
• Explicitly providing recovered acoustic cues can improve pain prediction in same-speaker comparison settings.The conclusion frames this as a result of cue reinjection, not evidence that the cues alone suffice for real-world pain assessment.Source: §9 · p. 8

Limitations

• TAME-Pain covers experimentally induced acute pain from 51 speakers, so findings may not generalize to chronic pain, clinical settings, or naturally occurring speech.Source: Limitations · p. 9
• The dataset is relatively small, and cold-pressor and warm-water recordings are pooled to increase sample size.Pain labels are self-reported, and acoustic correlates can vary substantially across speakers.Source: Limitations · p. 9
• Layer-wise analyses focus primarily on Qwen2.5-Omni-7B and Kimi-Audio, while acoustic perturbations and textualized-cue interventions simplify naturally occurring vocal changes.The authors describe these interventions as diagnostic analyses, not evidence that the cues alone suffice for real-world pain assessment.Source: Limitations · p. 9

Ethics Statement

• TAME-Pain was used under the PhysioNet Restricted Health Data License 1.5.0, and data users signed its data use agreement.Source: Ethics Statement · p. 9
• The authors made no attempt to re-identify participants and do not redistribute any audio.Source: Ethics Statement · p. 9
• Automated pain assessment without clinical validation could harm patients, and the authors do not support relying on current LALMs for it.Source: Ethics Statement · p. 9

A.1 Probing Experiment Setting

• Probing holds out 10 test speakers from training, normalization, and model selection; audio-token representations are tested with linear, MLP, and CRNN heads.Features come from the audio encoder, projector, and LM layers; tuning uses speaker-grouped five-fold cross-validation on training speakers.Source: §A.1 · p. 12

A.2 Task Description

• Tasks are specified by their label scheme, setting, and input modality, with each task's prompt listed in Appendix H.Source: §A.2 · p. 12

A.3 Models and Data Processing

• The appendix lists model versions and decoding settings; GPT audio is resampled to 24 kHz, and unparseable responses count as incorrect.Parsing success rates are reported for the main results and cue-augmented prompts.Source: §A.3 · p. 12, Table 7 · p. 13

B Probing on Kimi-Audio

• Kimi-Audio probing repeats the layer-wise analysis with a linear head because the three Qwen2.5-Omni-7B heads showed the same trend.Figure 7 depicts probe balanced accuracy across Whisper encoder, LM text-path, and MIMO audio-branch layers for binary detection and three-class severity.Source: B Probing on Kimi-Audio · p. 12, Figure 7 · p. 14

D.1 Feature–Pain Association

• Pain-feature associations are estimated from 88 eGeMAPSv02 features using speaker-normalized pain scores and separate mixed-effects models with speaker random intercepts.The analysis uses recordings from the 41 training speakers.Source: §D.1 · p. 13

D.2 Feature Groups

• The 34 significant features are grouped into nine cue groups according to the acoustic property they describe.Source: §D.2 · pp. 13–14

C Additional Main Results

• At least 95.7% of outputs in every condition and 99.0% overall parse successfully, so formatting failures do not explain low neutral-sentence scores.Source: C Additional Main Results · p. 12, Table 8 · p. 14

F Acoustic Perception Dataset Creation

• The perception test used low/high versions of 100 neutral recordings for five cues, excluding voice quality; listeners’ majority vote was correct on all 100 pairs (Fleiss’ κ = 0.95).Cue magnitudes were set far beyond natural within-speaker variation to make contrasts unambiguous to human listeners.Source: F Acoustic Perception Dataset Creation · pp. 16–17, Table 14 · p. 19

G Prompt with Cues Additional Results

• On LISTEN, adding raw acoustic cues improved accuracy across models in most settings, particularly Text+Audio.Chen et al. (2026a) report 34.0% accuracy for Qwen2.5 in single-utterance audio-only, versus 28.18% in this reference-conditioned audio setting; LISTEN uses 5–8 candidate labels.Source: G Prompt with Cues Additional Results · p. 17

H Prompt Templates

• The prompts use multiple-choice answers with shuffled options; paired Text+Audio prompts use the same items as Audio prompts with transcriptions added.The prompts describe the pain scale as 0–10 because some speakers report 0, despite being asked to rate on a 1–10 scale.Source: H Prompt Templates · pp. 17–18

H.1 Zero-shot Prompts

• Zero-shot prompts present the target recording, its transcription, or both, and ask for binary pain or three-class severity judgments.The severity categories are Mild (ratings 0–3), Moderate (4–6), and Severe (7–10).Source: §H.1 · pp. 18, 20

E Perturbation Analysis of the Probing Head

• Spectral balance and pauses had the largest credible effects on the probe’s pain logit: more high-frequency energy pushed toward pain, while longer pauses pushed away.Loudness and mean pitch had smaller credible effects despite loudness showing the strongest association with pain; speech-rate effects were attributed to resynthesis artifacts.Source: E Perturbation Analysis of the Probing… · pp. 14–15, Table 12 · p. 19

G Prompt with Cues Additional Results

• Cue augmentation yielded no meaningful gain on neutral sentences, and acoustic-features-only prompts stayed at chance; on spoken ratings, it lowered three-class accuracy by up to 5.3 points for gpt-realtime-mini and Qwen2.5-Omni.gemini-3.5-flash had lower parsing success in 20-shot settings (83.0–92.7%), partly explaining its low 20-shot scores.Source: G Prompt with Cues Additional Results · p. 17, Table 16 · p. 20, Table 15 · p. 20

H.2 Pairwise Prompts

• Pairwise prompts compare recordings from the same speaker reading the same sentence; binary pairs contain one pain and one no-pain clip and are shown in both orders.For three-class severity, a known Mild recording is the reference and the model classifies the second recording; their order is randomized.Source: §H.2 · p. 21
• Pairwise prompts compare same-speaker recordings for binary pain or three-level severity, using either audio alone or audio with transcripts and, for severity, a labeled mild reference.Binary labels are ratings 0–3 versus 4–10; severity labels are 0–3 mild, 4–6 moderate, and 7–10 severe.Source: §H.2 · pp. 22–23

H.3 20-shot Prompts

• 20-shot prompts provide 20 labeled examples from the target speaker to support binary or three-class classification using audio, with optional transcripts.The examples are intended to show how that speaker’s vocal, and when included textual, expression varies with pain; only the target is classified.Source: §H.3 · pp. 23–24

H.4 Acoustic Contrast Perception Test Prompt

• The acoustic contrast test presents low- and high-magnitude versions of the same recording as separate clips and asks which has more of a manipulated cue.The high-magnitude clip is always correct, its A/B position is randomized, and unparseable responses count as incorrect.Source: §H.4 · pp. 24–25

H.5 Prompt Variants with Explicit Acoustic Cue

• Acoustic-measurement prompt variants add feature definitions and recording-specific measurements after each audio input, then ask judgments based on vocal expression and textual acoustic measurements.Defined features include pitch and loudness measures, loudness dynamics, voice quality, and spectral flux.Source: §H.5 · p. 25
• The pain variant presents acoustic measurements with the target audio and asks whether the speaker is in pain, defining pain as self-reported ratings 4–10 versus no pain at 0–3.Its measurement block lists pitch level and variability, loudness level and dynamics, voice quality, spectral flux, MFCC1, and spectral tilt.Source: §H.5 · p. 25

H.6 Pairwise Prompts with Raw Acoustic Measurements for LISTEN Dataset

• The LISTEN pairwise variant adds eight eGeMAPSv02 acoustic descriptors as raw values, with undefined measures shown as unavailable, and places transcripts after measurements in audio-plus-text prompts.The prompt provides a neutral reference for classifying the target; the descriptors cover pitch, loudness, harmonics-to-noise ratio, shimmer, jitter, and spectral tilt.Source: §H.6 · pp. 25–26