Source-linked AI summary
VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models
Models Luc Debaupte, Tyler Baumgartner, Brandon Tai, Candice Fan, Bill Wang, Yi Zhong
TL;DR
Voice-agent evaluation needs to capture affective information carried by vocal delivery rather than only transcript words. VocalAffectBench addresses this with a fixed, auditable raw-audio benchmark, finding partial but brittle seven-way recognition and substantial neutral bias. The benchmark supports more actionable evaluation of class-specific misses and product safeguards, while its acted, English-only scope limits claims about spontaneous conversations or inner states.
Problem
Transcript-based evaluation can miss affective cues in vocal delivery, while text-only sentiment analysis measures lexical sentiment rather than expressed vocal emotion.
Method
VocalAffectBench evaluates audio models on a fixed, test-only English benchmark of expressed vocal emotion using raw audio without transcripts or contextual metadata.
Results
The baseline results show partial but brittle affect recognition, with neutral predictions overused and several non-neutral classes frequently missed.
Takeaways & Limitations
Reporting aggregate accuracy with per-class metrics, confusions, and valence skew gives teams a more actionable view of quality and where safeguards are needed.
Takeaways & Limitations
Acted emotional speech limits claims about spontaneous real-world conversations, and the English General American scope does not establish multilingual or cross-accent performance.
Abstract
from arXiv · showhide
Voice products increasingly need affective cues that are present in speech but absent from transcripts. We introduce VocalAffectBench, a public, test-only benchmark for evaluating whether AI audio models can identify expressed vocal emotion from raw audio. The benchmark contains 273 human-recorded English WAV clips from 51 speaker accounts totaling 1.95 hours across seven labels: angry, disgusted, fearful, happy, neutral, sad, and surprised, with 39 clips per class. All baselines are evaluated from audio alone, without transcripts or contextual metadata. Across six released baselines, average accuracy is 35.5%. The strongest baseline, gemini_3_5_flash, reaches 46.5% on the seven-way task, above the 14.3% random baseline but far from robust emotion recognition. A secondary valence-bucket analysis maps labels into positive, neutral, and negative classes, excluding surprised because its valence is ambiguous. Aggregate accuracy under this coarser view is 50.9%. Performance is highly uneven across classes. By recall, neutral is identified most reliably at 75.6% averaged across baselines, while surprised and fearful reach only 10.7% and 15.4%, respectively. These results show that the evaluated baselines can extract some affective signal from speech, but discrete expressed-emotion recognition remains fragile, especially for non-neutral emotions that are often most important in voice agent workflows.
Introduction
VocalAffectBench targets expressed vocal emotion that transcripts and text-only sentiment analysis cannot fully capture. It provides a fixed, auditable raw-audio evaluation protocol for comparing models.
- Motivation: Voice delivery carries affective cues that transcripts omit, including tone, pacing, pauses, and intensity.Identical words can convey different meanings depending on delivery.
- Problem: Text-only sentiment analysis measures lexical sentiment rather than expressed vocal emotion.Applying affect analysis only after transcription may answer a different question from speech emotion recognition.
- Contribution: VocalAffectBench is a small, auditable, test-only benchmark for single-label expressed vocal emotion in English speech.The released benchmark uses equal class counts and a fixed label mapping under the same input protocol.
- Benchmark Design: The benchmark uses human speakers performing scripts under assigned emotions, with human review of delivery quality and label fit.The script was designed so words could plausibly be delivered in multiple emotions, preventing lexical content from forcing one label.
- Evaluation Protocol: Models receive raw audio and a closed seven-label target without transcripts or contextual metadata.The released audio is English 16 kHz mono WAV, and provider outputs are mapped to the benchmark label set before scoring.
- Metrics: The evaluation reports accuracy alongside per-class precision, recall, and descriptive confidence intervals.Per-class metrics expose class-specific failure modes, while overlapping intervals are not treated as statistically significant ordering.
- Secondary Analysis: A secondary valence analysis groups emotions into negative, positive, and neutral buckets while excluding surprised because its valence is ambiguous.This view asks whether models capture coarse affective direction despite missing the exact emotion label.
Baseline Models
Across six released baselines, performance on the fixed seven-way benchmark is limited, with substantial variation between the strongest and weakest systems.
- Aggregate Performance: 35.1% average accuracy was achieved across all six released baselines.The systems made 590 correct decisions overall.
- Aggregate Performance: 44.3% accuracy made gemini_3_5_flash the strongest released baseline.It produced 124 correct predictions out of 280, with a 95% confidence interval of 38.6–50.1.
- Aggregate Performance: 27.9% accuracy made openai_realtime the lowest released baseline.It produced 78 correct predictions, with a 95% confidence interval of 22.9–33.4.
Results
The results show brittle seven-way emotion recognition, with strong neutral bias and uneven class performance. Coarser valence classification performs better descriptively but remains incomplete and output-skewed.
- Class Difficulty: 76.2% neutral recall was paired with only 24.0% precision, indicating extensive neutral over-prediction.Baselines predicted neutral 761 times across 1,679 decisions.
- Class Difficulty: 15.0% fearful recall contrasted with 78.3% precision, showing that fearful was rarely predicted but often correct when selected.This illustrates why precision and recall reveal different forms of label bias.
- Neutral Bias: 578 of 1,089 incorrect predictions were mapped to neutral.All six true non-neutral classes were often collapsed to neutral, with fearful-to-neutral the largest confusion.
- Valence as a Coarser Target: 49.2% aggregate accuracy was obtained under the coarse valence mapping.The mapping assigns four emotions to negative, happy to positive, and neutral to neutral, excluding surprised.
- Valence as a Coarser Target: 66.2% valence accuracy made gemini_3_5_flash the best model on the coarse view.The result suggests valence is more tractable than discrete emotion for current models, but not solved.
- Valence as a Coarser Target: Three baselines were negative-skewed and three neutral-skewed, while none was positive-skewed.Output valence skew is descriptive under this benchmark, not an intrinsic model property.
Discussion
The baselines detect some vocal affect, but seven-way recognition remains brittle and uneven. Neutral bias and missed high-salience emotions limit the reliability of affect outputs for production decisions.
- 35.1% average accuracy across models shows partial but brittle affect recognition, with the leading model still missing more than half of clips.
- Neutral performs best overall, while surprised, fearful, and disgusted remain difficult for most baselines.
- Neutral predictions can reflect missed salient affect, model uncertainty, or calibration error rather than facts about users.
- Emotion recognition should be a probabilistic auxiliary signal after product-specific validation, not the sole controller of action.
Intended Use
VocalAffectBench is intended for diagnostic evaluation rather than deployment as a general-purpose emotional-intelligence measure. Its uses include provider comparison, error analysis, regression tracking, and class-specific validation, supported by inspectable benchmark outputs.
- VocalAffectBench is intended for diagnostic evaluation of audio emotion recognition models.
- All baselines scored all 280 clips in the seven-class baseline results.
- Aggregate per-class precision and recall are reported across six baselines.
- The benchmark reports the most frequent aggregate confusions across all baselines.
- Provider comparison, error analysis, regression tracking, and class-specific product validation are appropriate uses.
- The benchmark is not intended as a training corpus, hidden leaderboard, universal emotional-intelligence measure, or biometric dataset.
Limitations
The benchmark has deliberate scope and measurement boundaries. It is English-only, uses a seven-label taxonomy and balanced classes, relies on performed speech, and uses labels verified by one reviewer.
- English-only clips in a General American accent limit direct extension to multilingual and cross-accent evaluation.Additional collection and label validation are required rather than direct translation.
- The seven-label taxonomy supports reproducible benchmarking but cannot capture every affective state or replace richer affect modeling.
- Equal class counts enable fair comparison and per-class analysis but do not estimate emotion prevalence in production conversations.Real workloads should be evaluated against their own traffic distribution.
- Acted or performed emotional speech limits claims about spontaneous conversations and what speakers truly feel.
- Labels verified by one reviewer provide a practical benchmark signal without population-level perception or inter-rater reliability analysis.
Ethical and Privacy Considerations
The benchmark’s human voice recordings create privacy and misuse boundaries beyond ordinary evaluation concerns. Reporting should preserve that labels describe expressed performance, not a speaker’s inner state or personal risk.
- Privacy boundaries: Publicly released speech can contain acoustic characteristics that identify or profile speakers, even when the text is not sensitive.Contributors consented to public research and benchmarking release, but the dataset is not intended for biometric or high-stakes use.
- Interpretation boundary: VocalAffectBench labels expressed performance rather than inner state.Downstream reports should not claim that voice-only predictions determine a person’s inner state or personal risk level.
- Release scope: The release includes audio, metadata, predictions, aggregate results, and documentation, but excludes source materials and demographic metadata.The main files include data/metadata.jsonl, data/predictions.csv, and aggregate result CSVs.
- Deployment implications: The benchmark connects emotion-recognition performance to product risk through transcript-free evaluation, released predictions, and per-class error analysis.These analyses help identify models that miss high-salience emotions and where additional validation or safeguards may be needed before affect predictions influence user-facing behavior.
Appendix: Released File Schema
The released schema organizes clip metadata, model predictions, normalization, scoring, and prompted-model outputs for a fixed benchmark protocol. A secondary valence analysis is documented separately and excludes surprised.
- Metadata schema: Each metadata record stores an audio identifier, WAV filename, target emotion, duration, sample rate, and channel count.The example record uses neutral, 12.4 seconds, 16 kHz, and mono audio.
- Prediction schema: The prediction CSV records identifiers, target and model labels, provider model, confidence, correctness, and error status.The columns include required_emotion, predicted_label, mapped_label, confidence, correct, and error.
- Valence analysis: The secondary valence analysis maps four negative emotions, happy to positive, and neutral to neutral while excluding surprised.The analysis asks whether models capture coarse affective direction despite missing the exact emotion label.
- Prompted baseline output: Prompted audio-model baselines return one primary expressed emotion, confidence, and a brief explanation in JSON.The prompt requires exactly one label from the allowed seven-label set, verbatim, and prohibits inferring private internal state.
- Prompt constraints: The prompting protocol directs models to base predictions on expressed vocal tone, prosody, pace, intensity, pauses, and wording.Outputs must be valid JSON.
Appendix: Label Mapping
Provider-native affect labels and prompted-model synonyms are normalized into the seven-label benchmark set before scoring. The appendix lists both a Hume-specific mapping and a general alias mapping.
- Hume mapping: Hume-specific labels are mapped to benchmark emotions such as amusement to happy, anxiety to fearful, calmness to neutral, and disgust to disgusted.The listed mapping also assigns provider terms including contemplation to sad and surprise to surprised.
- General alias mapping: The general alias mapping converts anger-related terms to angry and anxiety, fear, or scared to fearful.It also groups calmness-related terms under neutral.
- General alias mapping: The general mapping assigns sadness-related terms to sad and confusion, interest, realization, or surprise-related terms to surprised.These aliases normalize varied provider outputs into the benchmark vocabulary.
- General alias mapping: Determination and frustration map to angry, disgust maps to disgusted, and excitement, happiness, joy, pride, or satisfaction map to happy.The mapping is designed to normalize provider-specific output vocabulary before evaluation.
Appendix: Citation
The appendix provides a BibTeX citation for VocalAffectBench as a 2026 benchmark dataset under the MIT License, with its Hugging Face repository URL.
- Bibliographic entry: The citation names the work “VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models.”The entry identifies it as a benchmark dataset published in 2026.
- Bibliographic entry: The BibTeX entry lists the authors Debaupte, Baumgartner, Tai, Fan, Wang, and Zhong.It also includes the Hugging Face dataset URL.
- License and repository: The dataset citation specifies the MIT License.The entry identifies the repository as the VocalAffectBench Hugging Face dataset.