PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio emotion recognition for atypical hearing

Ulysse Roussel

arXiv:2609.31168v1cs.LG

TL;DR

The work asks whether affective responses can be generalized from few examples, against the background of auditory hypersensitivity in ASD and the absence of atypical-listener affective audio data. It fine-tunes CLAP using neurotypical ratings; Full-Pipeline showed the largest reported cross-domain gain on EmoSoundscapes, while the KL variant led several geometry metrics.

  • Problem

    No affective audio dataset annotated by atypical listeners is available, while the work asks whether affective responses can be generalized from few evaluated examples.

    Source: A. Context · p. 1, §I · p. 1

  • Method

    The method adapts a pretrained audio–text foundation model to valence and arousal with low-rank fine-tuning, training on AAD and testing externally on EmoSoundscapes.

    Source: C. Machine learning-based prediction… · p. 2, B. Dataset and split strategy · p. 3

  • Results

    Sp(V)=0.705 for Full-Pipeline on EmoSoundscapes marked its largest cross-domain gain, while Full-Pipeline + KL led RSA (0.353), Trustworthiness (0.674), and Sp(A) across both datasets.

    Source: F. Results · p. 4

  • Takeaways & Limitations

    This first step validates generalization from limited data on neurotypical datasets rather than modeling atypical listening directly.

    Source: A. Context · p. 1

    The datasets differ in sound duration, loudness normalization, categories, and evaluation methods, constraining their comparability.

    Source: B. Advantages and limitation of… · p. 2

Abstract

from arXiv · show

My doctoral work aims to explore Audio Emotion Recognition (AER) in the context of atypical listening. This research focuses on auditory hypersensitivity in people with autism, a phenomenon that is often difficult to evaluate and unique to each individual. Our core idea is to leverage our understanding of affect from acoustic traits, relying on the possibility of generalizing affective responses from a small amount of annotated data. As a first step, we fine-tune a large foundation model, Contrastive Language-Audio Pretraining (CLAP) using low-rank adaptation (LoRA), trained on a valence and arousal dataset of neurotypical listeners.

I. INTRODUCTION (KEY RESEARCH QUESTIONS)

Auditory hypersensitivity in autism can involve intense emotional reactions to sounds, while auditory processing in people with ASD remains insufficiently understood. This work first tests whether affective responses can be generalized from few examples using neurotypical data, because no affective audio dataset annotated by atypical listeners is available.

I. INTRODUCTION (KEY RESEARCH QUESTIONS)

  • Auditory hypersensitivity can make sounds that others barely notice consume energy and provoke intense emotional reactions in some people with ASD.Hypersensitivity is included in ASD diagnostic criteria.Source: §I · p. 1
  • The thesis aims to characterize atypical perception through acoustic metrics, as auditory processing in individuals with ASD remains insufficiently understood despite substantial research.The goal is to understand what may be disturbing.Source: §I · p. 1
  • The research asks how to generalize affective responses from few evaluated examples, collect ecological data suited to hypersensitive listeners, and model individual listening experiences.It focuses primarily on the first question while keeping the other two in view.Source: §I · p. 1

A. Context

  • Russell’s circumplex represents 36 subjective feelings along valence and arousal, dimensions used here to describe affective responses.Valence spans pleasure–displeasure; arousal spans activated–sleepy.Source: A. Context · p. 1, Figure 1 · p. 1
  • Because no affective audio dataset annotated by atypical listeners is available, the initial methodology is validated on neurotypical datasets rather than modeling atypical listening directly.The immediate goal is to test generalization from limited data.Source: A. Context · p. 1

B. Advantages and limitation of existing datasets

The datasets differ in sound content, normalization, duration, annotation methods, and rating density, so their suitability for affective audio modeling varies. AAD offers continuous ratings of normalized, non-musical sounds, while the other datasets have distinct constraints.

B. Advantages and limitation of existing datasets

  • AAD contains 780 normalized, six-second, exclusively non-musical and nonanthropomorphic mono sounds, with continuous ratings from 40 of 867 total raters per sound.The stimuli come from the BBC Sound Effects library and were normalized to −23 LUFS.Source: B. Advantages and limitation of… · p. 2
  • EmoSoundscapes has 1,213 clips annotated by pairwise ranking, but mixed sources complicate stimulus-level affect labels and splits must prevent leakage.Without loudness normalization, arousal prediction may be driven by raw amplitude.Source: B. Advantages and limitation of… · p. 2
  • IADS-E has 935 sounds rated on a discrete 1–5 SAM scale, while IADS-2 has 167 non-normalized sounds with mixed content despite dense annotation.IADS-E ratings may have reduced reliability and reproducibility; IADS-2’s small size limits its suitability for soundscape modeling.Source: B. Advantages and limitation of… · p. 2
  • The datasets differ substantially in duration, loudness normalization, sound categories, and evaluation methods, complicating direct comparison across them.The passage identifies several relevant databases but notes that their properties vary.Source: B. Advantages and limitation of… · p. 2

C. Machine learning-based prediction methodology

Affective audio prediction must generalize from limited labeled examples, but common modeling and evaluation choices raise concerns about transfer and whether metrics capture affective-space structure. The proposed methodology adapts a pretrained audio–text model and tests it across datasets.

C. Machine learning-based prediction methodology

  • Viable data augmentation strategies are scarce because spectral or temporal audio transformations can alter perceived valence and arousal.Reliable augmentation would require a more systematic mapping between transformations and affect.Source: C. Machine learning-based prediction… · p. 2
  • Hand-crafted-feature methods tend to transfer poorly across datasets, while deep spectrogram models typically require more labeled data than affective audio corpora provide.Prior deep approaches include convolutional and recurrent architectures.Source: C. Machine learning-based prediction… · p. 2
  • Overlapping one-second chunks from six-second EmoSoundscapes clips contaminate test data, while non-overlapping chunks assume affective uniformity not validated at that scale.The authors suspect chunk-level scores overfit the dataset rather than reflect robust generalization.Source: C. Machine learning-based prediction… · p. 2
  • The approach must generalize from few evaluated examples, and it is unclear whether R2 and RMSE capture the structure of affective space..Source: C. Machine learning-based prediction… · p. 2
  • The methodology adapts a pretrained audio–text foundation model to Russell’s circumplex using low-rank fine-tuning, with AAD training and EmoSoundscapes held out for cross-domain testing.It also introduces metrics intended to assess valence–arousal space structure.Source: C. Machine learning-based prediction… · p. 2

III. METHODOLOGY AND PROPOSED SOLUTION

The proposed solution uses CLAP’s audio–text embeddings to represent affect through textual anchors rather than a direct regression head. LoRA adapts the encoders in two phases, and AAD splits are stratified by median valence.

A. Bimodal text-audio foundation model

  • CLAP’s audio–text foundation model leverages language–affect links and pretrained acoustic and semantic structure, which are useful when labeled affective data are scarce.The passage notes that contrastive text–audio training can improve generalization over unimodal approaches.Source: A. Bimodal text-audio foundation model · p. 2
  • The model uses 36 text anchors in CLAP’s embedding space and scores audio by cosine similarity rather than mapping embeddings directly to valence–arousal coordinates.The anchors comprise eight axis labels and 28 affective descriptors.Source: A. Bimodal text-audio foundation model · p. 2
  • LoRA adapts trainable parameters in two phases: aligning the text encoder to 36 feeling anchors, then fine-tuning the audio encoder toward its nearest text anchors.The adaptation uses rank r=64 to reduce overfitting on the small AAD training set.Source: A. Bimodal text-audio foundation model · pp. 2–3
  • AAD train, validation, and test splits are stratified by median valence, and EmoSoundscapes is reserved as a held-out external test set.Figure 2 depicts AAD sounds in valence–arousal space across the three splits, stratified by valence median.Source: Figure 2 · p. 3

B. Dataset and split strategy

The proposed AAD split separates train, validation, and test data, while the anchor lexicon is built by matching text prompts to affective reference sounds.

B. Dataset and split strategy

  • AAD was split into 624/78/78 train/validation/test samples, stratified by median valence (V =−0.205; seed = 42).EmoSoundscapes (n=1,213) served as a held-out external test set.Source: B. Dataset and split strategy · p. 3

C. Anchor lexicon construction

  • For each of 36 circumplex anchors, the closest-labeled AAD sound supplied a reference audio embedding for prompt selection.Prompts were scored by cosine similarity against that embedding, with axis anchors assigned first and duplicate prompts disallowed.Source: C. Anchor lexicon construction · p. 3
  • The selected prompts achieved 94.4% Voronoi accuracy on Russell’s circumplex, measuring anchors assigned to their correct affective region.The candidate pool contained 728 prompts filtered from 1,005 entries,.Source: C. Anchor lexicon construction · p. 3

D. Fine-tuning strategy

Fine-tuning proceeds in two phases: text anchors are aligned to Russell’s circumplex, then the audio encoder is tuned using complementary affective losses.

D. Fine-tuning strategy

  • Phase 1 fine-tuned the text encoder with the audio encoder frozen, aligning 36 anchor embeddings to Russell’s circumplex using MKA [30], a manifold-aware variant of kernel alignment.A lightweight TextMLP further improved anchor placement.Source: D. Fine-tuning strategy · p. 3
  • Phase 2 froze the Phase 1 text encoder, appended an AudioMLP, and explored local label, soft-target KL, and global audio-geometry alignment losses.The KL objective follows prior CLAP affective-prediction work [32]; MKA aligns audio-embedding geometry to the (V, A) structure without per-sample supervision [30].Source: D. Fine-tuning strategy · p. 3
  • Ablations varied loss weights (λ1, λ2, λ3 ∈{0, 0.25, 0.5, 1}) and compared Audio-only, Audio-Text, Full-Pipeline, and Full-Pipeline + KL configurations.All models used batch size n=32.Source: D. Fine-tuning strategy · p. 3

E. Evaluation

Evaluation combines standard regression metrics with measures of affective ordering, embedding geometry, neighborhood structure, and global shape preservation.

E. Evaluation

  • Spearman rank correlation measures whether predicted sound ordering matches true ordering along valence or arousal, regardless of absolute scale.It was the primary selection criterion on out-of-distribution EmoSoundscapes.Source: E. Evaluation · p. 3
  • RSA measures Spearman correlation between pairwise distances in the 512-dimensional audio embedding space and the 2D (V, A) label space.Trustworthiness checks whether embedding neighbors are also neighbors in affective space.Source: E. Evaluation · p. 3
  • Voronoi accuracy for text-anchor embeddings increased from 94.4% to 97.2% after text-encoder fine-tuning.Fig. 3 presents a PCA of the embeddings before and after fine-tuning.Source: Figure 3 · p. 4
  • Procrustes disparity measures residual distance after optimally rotating, scaling, and translating predicted affect points to true labels, capturing global shape beyond per-axis calibration.This complements R2.Source: E. Evaluation · p. 4

F. Results

Re-tuning the text encoder after audio alignment produced the largest cross-domain EmoSoundscapes gain and best Procrustes disparity, while adding KL maximized representation metrics.

F. Results

  • Full-Pipeline achieved the largest cross-domain EmoSoundscapes gain (Sp(V)=0.705) and best Procrustes disparity (0.514).These results indicate better preservation of affective geometry.Source: F. Results · p. 4
  • Full-Pipeline + KL reached the highest RSA (0.353), Trustworthiness (0.674), and Sp(A) across both datasets.Across all fine-tuned models, RSA significance was confirmed by a Mantel permutation test (p<0.001).Source: F. Results · p. 4
  • Audio-only fine-tuning raised AAD Sp(V) from −0.09 to 0.71, showing that audio-encoder fine-tuning alone generalized.Adding Phase 1 Audio-Text yielded a consistent but modest further improvement across datasets.Source: F. Results · p. 4

IV. FUTURE WORK AND EXPECTED CONTRIBUTIONS TO AFFECTIVE COMPUTING

The future work identifies modeling choices to explore, asks whether CLAP’s pretrained affective prior transfers to atypical listening, and notes scope limits in the current data. It also proposes investigating individual response variability.

A. Open Issues/discussion

  • Only a limited portion of the fine-tuning space has been explored; alternative contrastive losses and adaptation strategies remain to be evaluated.The work has begun investigating attention layers in the HTS-AT audio encoder, which control attention to spectrogram regions and could emphasize regions more relevant to affective perception.Source: A. Open Issues/discussion · p. 4
  • Whether CLAP can reorganize its affective representation around atypical listeners’ labels or its pretrained prior will dominate remains an open question.The text frames personalization against the useful pretrained prior as a central problem for the thesis.Source: A. Open Issues/discussion · p. 4
  • The study excludes spatialization by focusing on monophonic signals, while aggregated AAD labels provide no measure of interrater variability.The text identifies disparities between individual evaluations as potentially informative for characterizing heterogeneity in atypical auditory processing.Source: A. Open Issues/discussion · p. 4

B. Probing the models

The work proposes probing affective predictions and acoustic transformations, while framing the study as a resource-efficient step toward modeling affect for autistic listeners. It currently relies on public datasets collected from neurotypical participants.

B. Probing the models

  • Reverse-correlation and bubble-based methods could probe which spectro-temporal cues underlie predicted valence and arousal.Controlled acoustic transformations could also inform label-aware augmentation strategies, addressing the augmentation problem raised in Section II.Source: B. Probing the models · p. 4

ETHICAL IMPACT STATEMENT

  • The framework aims to generalize from few labeled examples to reduce annotation workload and support lightweight, low-footprint models.The present study uses publicly available datasets collected from neurotypical participants; no human data were collected here.Source: ETHICAL IMPACT STATEMENT · p. 5
  • Modeling affective responses for autistic listeners with auditory hypersensitivity is a longer-term objective, pursued through a collaboration with CRAIF.The text states that human data collection will require approval by a French ethic committee.Source: ETHICAL IMPACT STATEMENT · p. 5

Related papers

Loading 2609.31168v1…