PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio System One Model: Typed Probabilistic Perception of Emotion in Speech

Jaime Bellver-Soler, Iván Martín-Fernández, Anmol Guragain, Luis Fernando D'Haro

arXiv:2610.22401v1cs.SD

TL;DR

Speech classifiers are fast but fixed-label, while audio language models accept text questions but generate one answer per question with poorly calibrated probabilities. ASOM scores typed text criteria against speech in one pass and returns distributions; on MSP-Podcast, it exceeds zero-shot and fine-tuned audio LLM baselines in F1.

  • Problem

    Speech classifiers are fast and calibrated but limited to fixed labels, while audio language models accept text questions but generate per-question answers with poorly calibrated probabilities.

    Source: §1 · p. 1

  • Method

    ASOM answers request-time choice, score, and yes/no questions by scoring text criteria against audio in one pass, using a 6.2M-parameter reader trained on listener vote distributions.

    Source: §1 · p. 1

  • Results

    0.072 higher F1 than the best zero-shot ALM and 0.026 higher than an ALM fine-tuned on the same 169k clips on MSP-Podcast.

    Source: §5.1 · p. 3

  • Takeaways & Limitations

    The same weights handle reworded or reduced label sets, new emotion words, scales, and attributes such as gender without retraining.

    Source: §1 · p. 1

    The tests cover affect, and ASOM answers only what its three listener-vote corpora cover; labels outside them are ranked well but seldom chosen.

    Source: §6 · p. 4

Abstract

from arXiv · show

Speech applications ask many questions about the same voice: which emotion a listener perceives, how aroused the speaker sounds, whether they sound frustrated. Classifiers answer fast, over one fixed label set; audio language models (ALMs) take any question in text, but answer by generation, one pass per question, with poorly calibrated probabilities. We present ASOM (Audio System One Model), a System One model for speech: choice, score and yes/no questions written at request time are answered in one pass, without generation, each with a probability distribution. Frozen speech and text encoders feed a 6.2M-parameter reader with no per-class parameters, trained on listener vote distributions. On MSP-Podcast, ASOM matches a classifier on the same features and data, leads the best of 6 zero-shot ALMs by 0.072 and an ALM fine-tuned on the same data by 0.026, and answers 20 questions in 13.1 ms, 17-41 times faster than the ALMs. It also answers reworded, reduced and new label sets, scales and gender, and on 21 unseen corpora in 13 languages it leads the best zero-shot ALM by 0.138 where the words carry no emotion; where they do, the ALMs mostly read.

1. INTRODUCTION

Speech classifiers are fast and calibrated but fixed to predefined labels, while audio language models accept text questions but generate answers question by question with poorly calibrated probabilities. ASOM scores request-time text criteria against speech in one pass, supporting typed questions and label or attribute changes with shared weights.

1. INTRODUCTION

  • Classifiers provide fast, calibrated answers over fixed labels, whereas audio language models accept text questions but require a generation pass per question and have poorly calibrated probabilities.New emotion words, coarser label sets, or attributes can require a new classifier.Source: §1 · p. 1
  • ASOM returns a probability distribution for each request-time choice, score, or yes/no question, without using a fixed label head or autoregressive generation.Each option becomes a text criterion scored against one audio vector.Source: §1 · p. 1
  • A shared model can handle reworded or reduced label sets, new emotion words, scales, and attributes such as gender without retraining.The criteria are independent of audio and can be cached; fixed-question deployments pay only for the speech encoder.Source: §1 · p. 1
  • The paper introduces a 6.2M-parameter reader without per-class parameters, comparisons on MSP-Podcast, and tests of label changes, scales, gender, and 21 unseen corpora.The transfer tests include a transcript-only control to separate hearing from reading.Source: §1 · p. 1

2. RELATED WORK

Related work spans fixed-head speech emotion classifiers, text–audio scoring models, audio LLMs, and label-as-text approaches. ASOM is positioned among these approaches through its use of listener-vote supervision and typed, open-label questions.

2. RELATED WORK

  • Speech emotion classifiers use fixed-class heads on pretrained speech encoders, with stronger systems adding text streams over ASR transcripts.Training on listener vote distributions captures disagreement, while ASOM keeps that supervision but represents each class as text.Source: §2 · p. 2
  • Contrastive audio–text models score clips against text, including speech-emotion systems, but learn similarities from captions or consensus labels rather than calibrated answers from listener votes.WavLink scores question–option texts against a Whisper token.Source: §2 · p. 2
  • Audio LLMs accept arbitrary questions but trail specialized emotion models [4]; fine-tuning toward annotator distributions retains a fixed label set and a billion-parameter decoder.The comparison identifies both label flexibility and decoder scale as distinctions from the fine-tuned approach.Source: §2 · p. 2
  • Label-as-text scoring predates the System One name, appearing in dataless and entailment classification, option scoring, and image and audio dual encoders.TypeSafe named System One models and introduced Jev; two concurrent audio prototypes use neither listener votes nor shared benchmarks.Source: §2 · p. 2

3. AUDIO SYSTEM ONE MODEL

ASOM converts typed questions into text criteria, compares their encodings with a clip representation, and normalizes scores into per-question probabilities. Its reader is trained on listener-answer distributions, with cross-fitted predictions used to smooth noisy targets.

3. AUDIO SYSTEM ONE MODEL

  • A request contains a clip and typed choice, ordinal-score, or yes/no questions; each option becomes a criterion made from the question, option name, and short description.A choice uses one criterion per option, a score one per level, and yes/no two criteria.Source: §3 · p. 2
  • Frozen Qwen2.5-1.5B encodes criteria, while frozen Whisper-large-v3 encodes the clip using pooled information from four transformer blocks.Criterion vectors are audio-independent and cached; the audio representation combines block-level features and attended time segments.Source: §3 · p. 2
  • The reader projects criterion and audio vectors to unit length, scores them, and applies a softmax over each question’s options to produce probabilities.A score question also returns its expected level, and a yes/no question returns the affirmative probability; all criteria are scored in one matrix product.Source: §3 · p. 2
  • Training targets are distributions of listener answers, optimized with cross-entropy so the reader learns disagreement rather than only the majority response.Cross-fitted auxiliary readers predict held-out clips to form smoother targets; the final system remains one reader, and removing this step worsens log loss.Source: §3 · p. 2

4. EXPERIMENTAL SETUP

The experiments train on three listener-vote corpora and evaluate on MSP-Podcast plus unseen corpora, with classifiers, audio LLMs, and CLAP models as baselines. Evaluation covers emotion, other attributes, transfer, calibration, and latency.

4. EXPERIMENTAL SETUP

  • Training uses MSP-Podcast 2.0, CREMA-D, and Emozionalmente, which release individual listener answers; most other emotion corpora are reserved for testing.MSP-Podcast contributes 169k training clips with emotion and 7-point arousal, valence, and dominance ratings.Source: §4 · p. 3
  • Training asks each clip about primary emotion, arousal, valence, dominance, gender, and yes/no emotion statements, with half the steps paraphrasing questions and options.Cross-fitting uses 32 auxiliary readers, and results average three seeds.Source: §4 · p. 3
  • Evaluation uses 3,113 speaker-stratified MSP-Podcast clips and up to 1,200 clips from each of 21 unseen corpora spanning 13 languages.Ten unseen corpora include gender labels; EmoNet-Voice Bench and EARS add emotions outside all training label sets.Source: §4 · p. 3
  • Baselines include supervised speech classifiers, six zero-shot audio LLMs, two LoRA-fine-tuned Omni-3B models, and three CLAP systems.MSP-Podcast uses macro-F1, LL8, CCC, and ROC-AUC; transfer uses macro-F1 and gender accuracy, with latency measured for 20 questions on a 5.9-second clip.Source: §4 · p. 3

5. RESULTS

On MSP-Podcast, ASOM outperforms most comparators and answers many questions with nearly constant cost. It also handles changed and new text-defined questions, transfers to unseen corpora, and shows trade-offs between calibrated answers and transfer in ablations.

5.1. Emotion recognition on MSP-Podcast

  • ASOM reaches F1 0.383, ahead of every audio-only classifier except DeepSER (+0.017) and ahead of the best zero-shot ALM by 0.072 and fine-tuned Omni-3B by 0.026.Every zero-shot ALM has log loss above the class prior (2.097 or more vs. 1.863), while ASOM reaches 1.507.Source: §5.1 · p. 3
  • 20 questions take ASOM 13.1 ms, compared with 230–545 ms for ALMs, making it 17–41 times faster.The reader scores all criteria in one call; criteria are cached, while a new question requires one text-encoder pass.Source: §5.1 · p. 3

5.2. Beyond a fixed label set

  • ASOM scores F1 0.360 on synonyms and paraphrased descriptions and 0.363 on a new question, best in each column.These are -0.038 and -0.035 versus trained wording; the fine-tuned ALM reaches 0.349 on Syn., and no zero-shot ALM exceeds 0.312 in any column.Source: §5.2 · p. 3, Table 2 · p. 4
  • For unseen emotion words, ASOM reaches AUC 0.714; its arousal, valence, and dominance CCCs are 0.674, 0.735, and 0.584, and gender accuracy is 0.873.The best other systems reach CCCs of 0.534, 0.701, and 0.311, while Omni-7B reaches gender accuracy 0.927.Source: §5.2 · pp. 3–4

5.3. Unseen corpora: hearing or reading

  • Across 21 unseen corpora, ASOM reaches mean macro-F1 0.433 versus 0.313 for the best zero-shot ALM and 0.402 for the fine-tuned ALM.Emotion2vec+ reaches 0.667 after fine-tuning on EmoBox, which contains 17 of those corpora; the two systems are close on the other four.Source: §5.3 · p. 4
  • On 17 unseen corpora where transcripts are at chance, ASOM reaches F1 0.419 versus 0.281 for the best ALM; on four where wording reveals emotion, the ALM leads 0.550 to 0.494.The transcript-only control retains 84% of Omni-7B's MSP-Podcast F1, supporting the authors' conclusion that ALMs mostly read while ASOM hears.Source: §5.3 · p. 4

5.4. Ablations

  • Replacing text criteria with a learned class table preserves MSP-Podcast F1 (0.379 vs. 0.382) and LL8 (1.502 vs. 1.506), but lowers unseen mean from 0.433 to 0.417.Names alone lower F1 by -0.063 to 0.335. Removing distillation worsens LL8 by +0.019, paraphrases worsen held-out-wording log loss by +0.078, and omitting CREMA-D and Emozionalmente lowers unseen mean from 0.440 to 0.362.Source: §5.4 · p. 4
  • Using CLIP’s caption-based contrastive loss instead of listener votes changes F1 by -0.053, LL8 by +0.369, and yes/no AUC from 0.763 to 0.596, but raises unseen mean by +0.067.The ablation reports a trade-off between calibrated typed answers and transfer under a prior-free objective.Source: §5.4 · p. 4

6. CONCLUSION

ASOM answers typed speech questions quickly and accurately, matching a same-feature classifier while outperforming tested ALMs. It also handles new wording and labels across unseen corpora, but its evaluation and training remain bounded to affect data.

6. CONCLUSION

  • 13.1 ms for 20 typed questions; ASOM matches a classifier on the same features and data, is 17–41 times faster than six zero-shot ALMs, and beats a fine-tuned ALM in accuracy.The fine-tuned ALM used the same 169k clips.Source: §6 · p. 4
  • Across 21 unseen corpora in 13 languages, ASOM is the best open-label system, almost 50% above the best zero-shot ALM where words carry no emotion.Where words carry emotion, a transcript still matters.Source: §6 · p. 4
  • Synonyms, paraphrases, or a new question cost under 0.04 F1; eight emotion words unseen in training reach an AUC of 0.714.These tests show flexibility with the same model weights.Source: §6 · p. 4
  • Evaluation is limited to affect; training uses three listener-vote corpora, gender trails Omni-7B, and labels outside those corpora are seldom chosen despite ranking well.The authors identify data, rather than the interface, as the bottleneck.Source: §6 · p. 4

7. COMPLIANCE WITH ETHICAL STANDARDS

The study uses public corpora under their licenses and collects no new data; its estimates concern listeners’ judgments of voices, not speakers’ internal states, and should not guide high-stakes decisions.

7. COMPLIANCE WITH ETHICAL STANDARDS

  • The study uses public corpora under their licenses and collects no new data; its estimates describe listeners’ judgments of voices, not speakers’ internal states, and should not drive high-stakes decisions.Source: §7 · p. 4
Loading 2610.22401v1…