Source-linked AI summary
SEER: Source-Conditioned Emotion Enhancement via Retrieval for Cochlear-Implant Speech
Hsing-Hang Chou, Yun-Shao Lin, Ching-Chin Sung, Chi-Chun Lee
TL;DR
Cochlear implants weaken speech cues needed for emotion recognition, and prior CI-oriented enhancement depends on paired recordings and intensity labels. SEER learns source-specific same-emotion reference selection from sampled CI-processed conversions, improving emotion recognition across objective and listener evaluations.
Problem
Cochlear implants inadequately transmit cues important for emotion perception, while scalable reference selection is needed because reference expression may not transfer equally across sources and exhaustive pair evaluation is costly.
Source: §1 · p. 1Method
SEER learns CI-aware source–reference utility from sampled emotional voice conversions and uses uncertainty-guided exploration to train source-conditioned retrieval without parallel recordings or intensity labels.
Source: §1 · p. 1, §2 · p. 2, §2.3.1 · p. 2Results
SEER raises Source macro-F1 at the N8 target by 7.30 points on RAVDESS and 11.66 points on ESD, with significant listener gains across RAVDESS N4, N8, and N16.
Source: §4.1 · p. 4, §4.2 · p. 4Takeaways & Limitations
Strong expression helps overall, but matched gender, content, and intensity alone do not determine reference utility.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Cochlear implants (CIs) restore speech access but weaken cues needed for vocal emotion recognition. Prior CI-oriented enhancement requires parallel normal/strong recordings and intensity labels. We propose SEER, a retrieval-based framework that learns which same-emotion reference helps each source remain recognizable after CI processing. A source-conditioned retriever learns CI-aware utility from sampled emotional voice conversion outcomes, while uncertainty-guided exploration avoids exhaustive pair evaluation; neither parallel recordings nor intensity labels are required. SEER improves Source macro-F1 at N8 by 7.30 points on RAVDESS and 11.66 points on ESD, with significant ESD gains across N4/N8/N16. Sixteen-listener RAVDESS gains are significant across all conditions. Exhaustive analysis finds an aggregate benefit from stronger references but little effect from matching gender or content.
1. INTRODUCTION
Cochlear-implant processing weakens cues important for vocal emotion recognition, while earlier enhancement methods rely on paired normal/strong recordings and intensity labels. SEER instead learns to select a useful same-emotion reference for each source from sampled conversion outcomes.
1. INTRODUCTION
- Cochlear implants inadequately transmit fundamental-frequency cues that contribute strongly to emotion perception, and emotion recognition is reduced for CI users and CI-vocoded speech.Conventional CI outcome assessment has centered on spoken-word and sentence recognition, while emotion perception has received comparatively less attention.Source: §1 · p. 1
- Prior JD-NMF enhancement uses paired same-speaker, same-sentence normal/strong utterances and explicit intensity distinctions, limiting use with broader emotion datasets.Recorded strong speech was more recognizable overall, while converted-speech gains were limited and emotion-dependent.Source: §1 · p. 1
- SEER learns source-specific reference selection from sampled post-conversion outcomes, avoiding exhaustive source–reference evaluation.Its retriever estimates conversion utility and uses acquisition-based exploration to balance predicted utility with uncertainty.Source: §1 · p. 1
- Reference-conditioned emotional voice conversion removes the paired-data requirement, but the reference cues that survive CI processing may not transfer effectively to every source.An ordinary categorical emotion corpus can therefore serve as a reference bank, while source-specific selection remains necessary.Source: §1 · p. 1
2. METHODOLOGY
SEER converts each source toward a same-emotion reference and learns which references are useful from sampled CI-processed outcomes. Its retriever combines source–reference utility estimates with uncertainty-guided exploration.
2. METHODOLOGY
- SEER samples source–reference pairs because exhaustive conversion does not scale, then uses their CI-aware outcomes to train the retriever.It retrieves only references with positive predicted utility and otherwise retains the source.Source: §2 · p. 2, §2.2 · p. 2
2.1. Reference-Conditioned Emotion Enhancement
- Maestro-EVC shifts a source toward an independent same-emotion reference while preserving linguistic content and speaker identity.A reference enters the bank only when the SER model predicts its intended emotion both before and after CI simulation.Source: §2.1 · p. 2
2.2. Source-Conditioned Reference Retriever
- SEER learns joint source and reference representations, then scores their absolute difference and element-wise product to predict enhancement utility.A source-side emotion classifier helps retain emotion information; the retriever selects the highest positive score or retains the source.Source: §2.2 · p. 2
2.3.1. CI-Aware Guiding Reward
- The CI-aware utility target compares the converted utterance with its original source using emotion retention and post-CI confidence.The retention difference is +1 for repaired retention, −1 for harm, and zero otherwise; confidence adds finer guidance when the clean prediction is correct.Source: §2.3.1 · pp. 2–3
2.3.2. Reference Exploration
- Uncertainty-guided acquisition favors score representations less covered by history, while evaluating the four highest-scoring references for each source.The acquisition score is used for reward evaluation; final retrieval uses predicted utility.Source: §2.3.2 · p. 3
3. EXPERIMENTAL SETUP
The experiments use RAVDESS and English ESD with actor-disjoint splits and training-only reference banks. SEER is evaluated across CI conditions using objective metrics and a 16-listener subjective study.
3.1. Datasets
- RAVDESS uses 24 actors and English ESD uses ten speakers; both are restricted to Angry, Happy, and Sad, with actor-disjoint splits.Training, validation, and test sets contain 384/96/96 RAVDESS and 9,000/600/900 ESD utterances; training-only banks contain 281 and 6,630 recordings, respectively.Source: §3.1 · p. 3
3.2. Training and Evaluation Setup
- SEER is optimized for N8 by selecting eight maxima from a 22-channel CI24-style PSHC vocoder; N4 and N16 test sparser and denser spectral information.A separate Maestro-EVC backbone is trained for each dataset; SEER retrieves only target emotions while the backbone trains on targets plus Neutral and Surprise.Source: §3.2 · p. 3
- Emotion is evaluated with Emotion2Vec+ macro-F1, treating Neutral as an error, and intelligibility with Whisper large-v3-turbo WER.Validation N8 macro-F1 selects the checkpoint; subjective evaluation uses 16 listeners labeling 279 trials across systems and N4/N8/N16.Source: §3.2 · p. 3
- Comparisons include Source, RAVDESS Strong, JD-NMF, Fixed reference, and SEER; RAVDESS-trained JD-NMF transfers unchanged to ESD.Fixed reference selects one template per emotion using its N8 retention lower bound, then applies it to every source of that emotion.Source: §3.2 · p. 3
4. EXPERIMENTAL RESULTS AND ANALYSIS
SEER improves post-CI emotion recognition over Source across datasets and listener evaluations, while source-conditioned retrieval limits the intelligibility loss of a fixed reference. Reference strength helps in aggregate, but gender and content matching show little effect.
4.1. Objective Evaluation
- 7.30 points on RAVDESS and 11.66 points on ESD: SEER raises Source macro-F1 at the N8 target and exceeds JD-NMF by 8.96–22.98 points across six dataset–CI comparisons.ESD gains are significant at N4, N8, and N16; on RAVDESS, the 15.48-point N16 gain is significant, but N4 does not improve.Source: §4.1 · p. 4
- At N4, RAVDESS Strong raises WER from 1.39% to 3.13%, while Fixed reference raises ESD WER from 10.99% to 22.41%.Compared with Fixed reference, SEER lowers WER in every condition, including 2.43% to 1.39% on RAVDESS N8 and 14.19% to 10.00% on ESD N8.Source: §4.1 · p. 4
4.2. Subjective Evaluation
- 21.94, 17.48, and 8.68 macro-F1 points: listeners rate SEER above Source at N4, N8, and N16, with every gain significant after Holm correction.Strong remains highest at 77.91–83.11%.Source: §4.2 · p. 4
- SEER raises Angry recognition from 31.3% to 82.0% and lowers Neutral responses from 42.2% to 5.5%, but Happy remains confused with Angry and Sad with Neutral.For Sad, recognition remains near Source at 63.3% versus 60.9%.Source: §4.2 · p. 4
4.3. Successful-Reference Structure
- Same- versus different-gender references differ by 0.7–2.1 points, and same- versus different-content references by 0.2–1.8 points; neither difference is significant.Strong exceeds Normal at N4 (65.1% versus 56.0%), N8 (63.1% versus 59.4%), and N16 (75.1% versus 64.5%), but only N4 remains significant after Holm correction.Source: §4.3 · p. 4, Table 3 · p. 4
5. CONCLUSION
SEER learns source–reference utility from sampled conversions without exhaustive pairing, supporting source-conditioned enhancement without parallel recordings or intensity labels.
5. CONCLUSION
- SEER learns CI-aware source–reference utility from sampled conversions without exhaustive pairing.The approach enables source-conditioned enhancement from categorical corpora without parallel recordings or intensity labels.Source: §5 · p. 4
- Objective and listener results show improved post-CI emotion recognition with less intelligibility loss than forced-reference conversion.Source: §5 · p. 4
- Strong expression helps overall, but matched gender, content, and intensity alone do not determine utility.Source: §5 · p. 4
- Future work will expand emotion and simulator coverage and add CI-listener rewards.Source: §5 · p. 4