Revisiting Modeling and Evaluation Approaches in Speech Emotion Recognition: Considering Subjectivity of Annotators and Ambiguity of Emotions

1.1 Motivation

• Because emotion perception is subjective and overlapping, the dissertation asks whether to retain minority ratings, learn from multiple annotators, and predict multiple emotions per sample.It frames these questions around emotion subjectivity and ambiguity, including possible human-perception biases related to gender, culture, and age.Source: §1.1 · p. 23

1.2.1 Emotion Representations

• The dissertation focuses on categorical emotions, contrasting them with dimensional attributes such as arousal and valence.It states that categorical emotions are better perceived across cultures than dimensional attributions.Source: §1.2.1 · p. 24

1.2.2 Emotion Recognition Systems using Multi-/Uni-modality

• Although emotion recognition spans modalities including faces, bodies, speech, and text, this dissertation concerns speech-based emotion recognition systems.The introduction notes that people can recognize different numbers of emotions across modalities, including at least 24 from vocal expression.Source: §1.2.2 · p. 24

1.2.3 Evaluation of SER Systems

• Majority or plurality consensus labels discard minority opinions and samples without consensus, while single-label SER also neglects co-occurring emotions.Prior soft-label training work still excluded tied samples from testing, and the introduction argues that this limits how accurately evaluations reflect system performance.Source: §1.2.3 · p. 25
• The dissertation proposes evaluating SER with all data and emotional ratings, using soft-label ground truth and metrics for distribution similarity.The soft labels can also be converted to binary vectors to report traditional accuracy.Source: §1.2.3 · p. 26

1.2.4 Label Prepossessing for Training SER Systems

• For training, the dissertation aims to retain all data and emotional ratings rather than rely solely on majority or plurality aggregation.These are introduced as the most common approaches for obtaining consensus labels.Source: §1.2.4 · p. 26
• The authors propose modeling individual raters’ SER systems and an “all-inclusive” aggregation rule that incorporates all emotional ratings.Source: §1.2.4 · p. 27

1.2.5 Co-occurrence of Emotions

• The work models relationships among co-occurring emotion categories from their frequencies, allowing mixed states such as happiness and anger.Source: §1.2.5 · p. 27
• A normalized co-occurrence-frequency penalty matrix can be integrated into training losses to penalize rare emotional combinations, including concurrent positive and negative emotions.Source: §1.2.5 · p. 28

1.2.6 Disagreement between Raters on the Emotion Datasets

• Emotion recognition lacks clear gold-standard labels, and annotator disagreements reflect subjective interpretation and ambiguity, not merely annotation noise.Source: §1.2.6 · p. 28
• Majority and plurality aggregation can overlook nuanced subjective perception, motivating a more comprehensive and inclusive alternative for SER labels.Source: §1.2.6 · p. 29

tators

• Combining embeddings from standard and five annotator-specific SER systems improves performance on majority-vote test sets.The framework uses ratings from individual annotators alongside standard SER systems.Source: tators · p. 30

1.3.2 Novel Evaluation Method by an All-Inclusive Aggregation Rule

• The all-inclusive rule retains all annotated test ratings and represents ground truth as per-emotion vote-frequency distributions.The approach aims to ensure no test data is overlooked.Source: §1.3.2 · p. 30
• Distributional similarity metrics are argued to better match subjective SER than macro-F1, while converting distributions to binary vectors for accuracy may discard ratings.Source: §1.3.2 · p. 30

1.3.3 Training Loss Using Co-occurrence Frequency of Emotions

• The co-occurrence penalization method improves SER performance on both single-label and multi-label test conditions.Source: §1.3.3 · p. 31

1.4 Outline of the Dissertation

• The dissertation presents public emotion databases before chapters on annotator subjectivity, evaluation, and co-occurrence-aware training loss.Source: §1.4 · p. 31

Emotion Databases

• The dissertation uses four public emotion databases, with one or more databases used in different chapters.Source: Emotion Databases · p. 32

2.1 IEMOCAP

• IEMOCAP contains 10,039 utterances from scripted and spontaneous dialogues, with raters choosing from ten emotion categories.The dissertation uses IEMOCAP in Chapters 3 and 4 and documents dataset splits to support reproducibility.Source: §2.1 · pp. 32–33

2.2 IMPROV

• IMPROV contains 8,438 clips, each assessed by at least five crowdsourced annotators, and is used in Chapter 4.Its labeling procedures cover primary and secondary emotions.Source: §2.2 · p. 33

2.3 CREMA-D

• CREMA-D includes 7,442 segments rated by more than 2,000 crowdsourcing raters, while this dissertation uses voice-only labels in Chapter 4.Each segment was assessed by at least six annotators.Source: §2.3 · pp. 33–34

2.4 MSP-PODCAST

• The labeling protocol uses crowdsourcing and covers primary and secondary emotions, with nine primary categories and eight additional secondary categories, totaling 17 emotional categories.Source: §2.4 · p. 34

2.5 Standard Partition

• Standard partitions for four prominent public emotion databases are established to improve SER transparency and reproducibility.The passage reports that 80.77% of SER papers using IEMOCAP produced irreproducible results, citing the absence of standardized splits as the primary obstacle.Source: §2.5 · p. 35

2.5.1 Standard Partition of the IEMOCAP

• IEMOCAP is divided into five speaker-independent session splits and evaluated with five-fold cross-validation.Each session involves two speakers in interactive dialogues.Source: §2.5.1 · p. 35

2.5.2 Standard Partition of the MSP-IMRPOV

• IMPROV uses six folds with varied speaker-pair interactions and tests on entirely new speaker pairings.The split strategy evaluates generalization across dyadic exchanges.Source: §2.5.2 · p. 36

2.5.3 Standard Partition of the CREMA-D

• CREMA-D is divided into five speaker-ID subsets for speaker-independent evaluation, with subsets specifying male and female speakers.Its partitioning follows the IEMOCAP methodology described in the dissertation.Source: §2.5.3 · p. 36

Subjectivity of Annotators

• Jointly modeling crowd-level emotion labels and individual annotator perspectives yields the highest emotion-recognition accuracy.The dissertation combines hard and soft emotion labels with individual and crowd annotator models.Source: Subjectivity of Annotators · p. 37

3.1 Motivation

• Traditional SER systems use plurality or majority voting to create a single hard-label target, overlooking diverse and subjective annotations.The cited motivation notes that emotion perception can vary with factors including gender, culture, and age.Source: §3.1 · p. 37

3.2.1 Subjectivity of Emotion Perception

• The method models individual raters’ emotion perceptions directly, beyond approaches that model ratings collectively at a given time.The dissertation frames this as preserving more of the subjectivity of emotion perception.Source: §3.2.1 · p. 38

3.2.2 Mixture of Annotators

• Soft labels represent emotion perceptions as distributions, while this dissertation retains the original labels rather than reassigning categories.Earlier work described in the passage used annotator mixtures or disagreement modeling in other tasks, including medical image tagging and continuous emotion tracking.Source: §3.1 · p. 38, §3.2.1 · p. 38, §3.2.3 · p. 39

3.3 Resource and Task Formulation

• The task uses 5,531 IEMOCAP samples for four-class evaluation, consolidating happiness and excitement into one category.The four classes are sad, neutral, happy, and angry; evaluation data use majority-vote labels from more than three annotators.Source: §3.3 · pp. 39–40

3.4.1 Input Features

• The study extracts 45-dimensional openSMILE Emobase features in 60 ms frames with 10 ms steps, then normalizes per speaker and averages five consecutive frames.Features include MFCCs, voice probability, zero-crossing rates, F0, loudness, and derivatives.Source: §3.4.1 · p. 40

3.4.2 Model Structure

• All models use a BLSTM-FC architecture with attention-weighted pooling over frame-level features.The architecture includes an input layer, a bidirectional LSTM, a fully connected layer, and an output layer.Source: §3.4.2 · p. 41

3.4.3 Training Labels

• Crowd hard-label models learn a single consensus emotion, whereas crowd soft-label models use vote-based distributions to represent blended perceptions.The study sets α to 0.75 and uses four emotion classes for its soft-label formulation.Source: §3.4.3 · pp. 41–42

3.5.1 Rater-Modeling

• Rater modeling trains crowd models on hard or soft labels and individual E models on each annotator’s available soft labels.The amount of annotated data varies across individual raters; the cited table captions describe training-sample totals and emotion-class label distributions.Source: §3.5.1 · p. 42

3.5.2 Final Concatenation Layer

• The fusion stage freezes the crowd and five E-model weights, concatenates their pre-softmax representations, and adds a final softmax layer.Figure 3.1 illustrates the overall proposed model.Source: §3.5.2 · p. 43, Figure 3.1 · p. 43

3.6.2 All Model Comparison

• The comparison includes hard-label and soft-label crowd baselines, their fusion, individual E models, fused E models, and the proposed model.The proposed model incorporates all available crowd and E-model information.Source: §3.6.2 · p. 44
• The proposed model combines crowd hard- and soft-label representations with five individual-rater models before a final softmax prediction.The individual-rater models are trained on soft labels from specific annotators; the final model uses representations from both crowd models and the rater models.Source: §3.6.2 · p. 45

3.7 Other Training Details

• Training uses leave-one-session-out cross-validation and UAR evaluation, with batch size 32, learning rate 0.0001, and 200 epochs.Early stopping minimizes validation loss, and ADAMMAX is the optimizer.Source: §3.7 · p. 45

3.8 Experimental Results and Analyses

• 61.48% UAR exceeds CrowdH by 3.18% and CrowdS by 4.36% in absolute terms.The reported comparison is for the proposed framework on IEMOCAP.Source: Table 3.4 · p. 46, §3.8 · p. 46
• CrowdS demonstrates better performance on happy emotions than CrowdH, whereas CrowdH performs better on neutral and sad emotions, showing complementary strengths across these emotion classes.Source: §3.8 · p. 46

3.9 Summary

• The proposed framework combines dominant sentiment annotations with individual subjectivity models for four-class emotion categorization.The dissertation’s summary reports a 61.48% score for this task.Source: §3.9 · p. 47

Chapter 4

• Chapter 4 introduces a novel SER evaluation method centered on comprehensive emotional data rather than consensus-filtered test sets.The method examines training annotation aggregation and label-learning strategies for evaluation on the entire dataset.Source: Chapter 4 · p. 48, §4.1 · pp. 50–51

All-Inclusive Aggregation Rule

• Models trained with the all-inclusive rule achieve better macro-F1 scores when evaluated on the whole test set, including ambiguous non-consensus samples.Traditional majority or plurality aggregation can exclude data when there is no consensus, omitting ambiguous cases from evaluation.Source: All-Inclusive Aggregation Rule · p. 48

4.1 Motivation and Background

• Majority and plurality evaluation can omit ambiguous samples, even though co-existing emotions make a single label incomplete for some perceptions.The dissertation notes that majority-rule samples are dismissed when no class receives more than 50% of votes.Source: §4.1 · p. 49
• The all-inclusive rule retains every sample and annotation in training and testing, allows co-occurring emotions, and evaluates on the whole test set.The figure description contrasts this with majority and plurality methods, noting that AR always includes the entire available test set.Source: §4.1 · pp. 49–50

4.2.1 Evaluation of SER Systems

• Consensus-based single-label evaluation can omit co-occurring emotions and non-consensus samples, limiting how well predictions reflect complex emotional behavior.Emotions such as sadness and anger can co-occur.Source: §4.2.1 · p. 51
• The dissertation retains original emotional classes, unlike prior work that relabeled some emotions, and tests different label-learning methods and aggregation-based test sets.The prior study's relabeling included mapping excited to happy and surprised to “other.”Source: §4.2.1 · p. 52

4.2.2 Curriculum Learning for Emotion Recognition

• Lotfian and Busso used annotator disagreement to enhance SER valence/arousal prediction; Yang et al. categorized conversational emotion samples by emotion shift, trained from easy to challenging, and reported curriculum-learning gains.Source: §4.2.2 · p. 52

4.3.1 Proposed All-inclusive Rule

• The all-inclusive rule (AR) retains all annotations regardless of vote frequency and supports one-hot or vote-distribution targets for training.With hard labels, it selects a top-voted class, randomly choosing among top classes when there is no clear majority.Source: §4.3.1 · p. 53
• AR uses the distributional ground truth for testing regardless of the label-learning strategy, retaining every annotated sample and its annotations.The authors say this better captures ambiguous emotional nuances in the test set.Source: §4.3.1 · p. 53

4.3.2 Employing the All-Inclusive Rule for Test Set Construction

• The authors use AR to include all samples and available emotions, enabling evaluation on non-consensus cases and secondary emotions rather than only selected classes.They describe this as the first evaluation of SER models using such non-consensus data.Source: §4.3.2 · p. 54
• In MSP-Podcast, classifying secondary emotions produces the most significant data loss: MR discards up to 92.01% and PR up to 33.72% of the data, respectively.Source: §4.3.2 · p. 54

4.4.1 Resource

• Four publicly available emotion databases are used; excluding the “Test2” set reduces corpus version 1.10 from 104,267 to 90,978 annotated utterances.The passage refers to the corpus version without naming it.Source: §4.4.1 · p. 55

4.4.2 Speech Emotion Classifier

• Experiments implement wav2vec2-large-robust to process raw audio directly, sampling at 16 kHz to match pretrained-model data, and optimize the model by removing its top layers.Source: §4.4.2 · p. 55
• PODCAST uses predefined train, development, and test sets, while IEMOCAP, IMPROV, and CREMA-D use speaker-independent session-based cross-validation.PODCAST has 63,076 training, 10,999 development, and 16,903 test samples; the other corpora use five, six, and five sessions, respectively, with one session each for testing and development.Source: §4.4.2 · p. 56
• Data-loss trends were relatively consistent across all four datasets, with Table 4.3 illustrating the PODCAST distributions.The ratios cover training, development, and test partitions; similar trends are reported in Table 4.1.Source: §4.4.2 · p. 56, Table 4.3 · p. 56

4.4.3 Train/Test Set Defined by Aggregation Rules

• Training and evaluation combine matching and mismatching MR, PR, and AR aggregation rules, with PR-MR and AR-PR test conditions for non-consensus samples.PR-MR includes samples accepted by PR but not MR; AR-PR includes samples accepted by AR but not PR and is identified as the most ambiguous set. This study is described as the first to evaluate SER models on non-consensus annotations.Source: §4.4.3 · p. 57

4.4.4 Label Learning for SER

• Hard-, soft-, and distribution-label learning are compared, with soft and distribution targets representing annotator vote proportions.Hard labels use a one-hot majority class, randomly selecting among top-voted classes under AR when there is no clear consensus; soft labels use cross-entropy, while distribution labels use Kullback–Leibler divergence.Source: §4.4.4 · pp. 57–58

Hard-decision-based assessment

• Macro-F1 assesses single-label MR, PR, and PR-MR sets using the most-voted target, while AR and AR-PR allow multiple emotions when proportions exceed 1/C.Predictions are binarized for multi-emotion assessment; the threshold is 1/C, where C is the number of emotion classes.Source: Hard-decision-based assessment · p. 58

Distribution-based Assessment

• KLD compares predicted emotion distributions with subjective annotations across all test sets, with lower values indicating better alignment.This distribution-based assessment follows the entropy-based approach proposed by Steidl et al. and retains model probabilities rather than binarizing outputs.Source: Distribution-based Assessment · p. 59

Distribution-based Assessment Versus Hard-decision-based Assessment

• Both KLD and traditional accuracy metrics—macro-F1, micro-F1, and weighted-F1—are reported to capture complementary assessment perspectives.The paper notes that each assessment has advantages and disadvantages.Source: Distribution-based Assessment Versus… · p. 59, Table 4.4 · p. 60

Statistical Significance

• For cross-validation, predictions across folds are concatenated so each sample appears once in a test fold.For PODCAST, predictions from predefined test sets are segmented into 40 folds to average macro-F1.Source: Statistical Significance · pp. 59–60
• A two-tailed t-test determines significance using a p-value threshold below 0.05.The significance assessment is conducted for each aggregation method.Source: Statistical Significance · pp. 59–60
• The ∗, †, and ⋆ symbols mark significant superiority over models trained using MRTrain, PRTrain, and ARTrain, respectively.Source: Statistical Significance · p. 60

4.5.1 Comparison of Results with Prior SOTA Methods

• ARTrain surpassed all examined SOTA results on IMPROV(P), CREMA-D, IEMOCAP, and PODCAST(P).The comparison reports macro-F1 under majority-rule or plurality-rule test paradigms.Source: Table 4.5 · p. 61, §4.5.1 · pp. 61–62

Assessment of the Incomplete Test Sets (MR & PR)

• Testing with PR yielded lower performance than MR, while PR-MR testing was generally worse than PR, indicating that ambiguous samples can degrade measured SER performance.The passages warn that MR or PR alone may not reflect real-world recognition, where every sentence must be recognized.Source: Assessment of the Incomplete Test Sets… · p. 64

Assessment on the Complete Test Set (AR)

• On complete AR test sets, ARTrain achieved significantly higher macro-F1 than MRTrain and PRTrain across 18 conditions, ranking highest in 14 of 18 experiments.The comparison spans multiple databases and label-learning strategies.Source: Assessment on the Complete Test Set (AR) · p. 64, Figure 4.2 · p. 65

Assessment of the Incomplete Test Sets (MR & PR)

• ARTrain yielded the best performance in 11 of 18 PR-test cases and 14 of 18 AR-test cases.The reported results cover 18 experiments across databases and label-learning strategies.Source: Assessment of the Incomplete Test Sets… · p. 65, Figure 4.2 · p. 65
• On MR tests, ARTrain outperformed other training sets in only 7 of 18 experiments, trading accuracy on straightforward samples for resilience across mixed real-world samples.The authors suggest this trade-off may make ARTrain more useful in practical settings containing both ambiguous and unambiguous samples.Source: Assessment of the Incomplete Test Sets… · pp. 65–66

Assessment in Cross-Corpus Scenarios

• Cross-corpus evaluation trained on PODCAST(P) and tested directly on IMPROV(P), thresholding predictions for anger, sadness, happiness, and neutral into binary labels for macro-F1.The corpora share those four emotions, enabling this evaluation.Source: Assessment in Cross-Corpus Scenarios · p. 66
• ARTrain training yielded superior performance on the MR, PR, AR, and AR-PR test sets in cross-corpus testing.Source: Table 4.9 · p. 67, Assessment in Cross-Corpus Scenarios · p. 67

4.5.3 Evaluation on the Ambiguous Set

• Across 18 evaluations, models trained with MRTrain did not yield the best performance in 17 cases, while PRTrain models demonstrated significantly better performance in 10 experiments.Source: Performance on the AR −PR Condition · p. 68
• ARTrain achieved significantly higher averaged macro-F1 scores than MRTrain and PRTrain on both AR−PR and PR−MR test conditions.Source: Figure 4.3 · p. 67, Performance on the AR −PR Condition · p. 68

Analysis of the Feature Embeddings

• T-SNE plots showed high-agreement emotions separating distinctly, while two-emotion samples often lay between pure-emotion clusters.Source: Analysis of the Feature Embeddings · pp. 69–70, Figure 4.5 · p. 70
• ARTrain had the highest silhouette scores for three emotion pairs, while PR−MRTrain scored highest for anger-sadness; ambiguous-data models generally clustered complex emotions better than MRTrain.Source: Analysis of the Feature Embeddings · p. 71

Impact of Extra Data Incorporated by the AR Approach

• ARTrain consistently outperformed synthetic-data alternatives and delivered superior macro-F1 after undersampling, indicating its advantage was not solely due to training-set size.Source: Table 4.11 · p. 72, Impact of Extra Data Incorporated by… · p. 72
• Adding samples was consistently beneficial for macro-F1: the 32,831-sample set, formed by randomly adding 12,831 samples regardless of consensus to the 20,000-sample set, improved results, and ARTrain frequently achieved the highest scores in both conditions.Source: Impact of Extra Data Incorporated by… · p. 73, Table 4.12 · p. 73

4.5.4 What is the most effective label learning method for SER?

• Soft-label learning surpassed hard-label learning in 17 of 18 cases (around 94%), while cross-entropy outperformed KLD and KLD performed worst.Results were evaluated on the entire test set using the AR method.Source: §4.5.4 · p. 73, Figure 4.6 · p. 74
• On the ambiguous AR−PR test set, soft-label learning was the most appropriate evaluated approach for recognizing mixed emotions.This finding held across models trained with MR, PL, and AR.Source: §4.5.4 · pp. 74–75, Figure 4.7 · p. 75

• Majority or plurality labeling excluded notable portions of annotated test samples, limiting how well evaluation represented expected SER performance in realistic scenarios.The study examined speaker-independent categorical SER systems and notes that real-world classifiers must recognize emotions across sentences.Source: §4.6 · pp. 75–76
• The all-inclusive rule produced more reliable SER performance by including low-agreement speech samples, while performance declined as more ambiguous samples entered the test set.Training only on high-agreement data was insufficient for predicting ambiguous emotions.Source: §4.6 · p. 76
• On the entire test set, soft-label learning led the label-learning strategies, while all-inclusive training data consistently surpassed majority- or plurality-selected data on incomplete and comprehensive tests.The all-inclusive rule retains all annotated data and emotional ratings.Source: §4.6 · pp. 75–76
• The authors propose exploring the all-inclusive rule in text-to-speech and textless speech-to-speech translation, where it may support more varied emotional expression and realism.These are proposed future applications, not results reported in this section.Source: §4.6 · pp. 76–77

Training Loss by Using Co-occurrence

• Because emotions can overlap, multi-category approaches may still neglect relationships among emotions by treating categories independently; this study investigates those relationships in SER training.Source: Frequency of Emotions · p. 78
• The matrix uses emotion frequencies from perceptual evaluations to encode class dependencies, applying harsher penalties to errors between more dissimilar emotions.It combines three established label-learning techniques with a modified loss function.Source: Frequency of Emotions · p. 78
• On PODCAST, the penalization matrix increased macro F1 by 17.12% for hard-label, 12.79% for multi-label, and 25.8% for distribution-label learning.Source: Frequency of Emotions · p. 79

5.1 Motivation

• Conventional consensus labeling treats annotation variability as noise and can discard secondary emotions reported by raters outside the majority.The section frames emotion perception as subjective and notes that speech may convey co-occurring emotions.Source: §5.1 · p. 79
• The proposed approach uses emotion co-occurrence frequencies from perceptual evaluations to build a penalization matrix that raises loss for rarely co-occurring predictions.The matrix is incorporated into existing cost functions as a penalty loss.Source: §5.1 · p. 80

5.2.1 Contrastive Learning in Emotion Recognition

• Unlike contrastive approaches discussed here, this work models relationships between emotion classes using their co-occurrence frequencies in the training set.The contrast is with prior approaches described as separating classes or using local neighborhood information.Source: §5.2.1 · p. 80

5.2.2 Label Learning in Emotion Recognition

• The section introduces hard-label, multi-label, and distribution-label learning as distinct ways to represent emotional ratings for SER.Its illustrative example uses five annotators independently assigning one label each to a sentence.Source: §5.2.2 · p. 81

Emotion Recognition using Hard-label Learning

• Hard-label learning collapses ratings to one consensus class, excluding secondary emotions noted by raters who do not align with the majority.The section also notes prior soft-label and individual-annotator approaches that retain multiple perceptions.Source: Emotion Recognition using Hard-label… · p. 81

Emotion Recognition using Multi-label Learning

• Multi-label learning represents several emotions per sample but assumes they manifest independently and does not indicate their relative importance.The section contrasts this with distribution-label learning, which assigns probabilities across categories summing to one.Source: Emotion Recognition using Multi-label… · p. 82, Emotion Recognition using… · p. 82

Emotion Recognition using Distribution-label Learning

• Distribution-label learning represents category probabilities that sum to one and trains models to minimize the distance between actual and predicted distributions.Emotional ratings can be translated into distribution labels for training.Source: Emotion Recognition using… · p. 82

tions

• The method forms an 8 × 8 matrix indicating how frequently emotion pairs co-occur, normalizes it column-wise into an asymmetric co-existing weight matrix, and subtracts it from one to produce a penalization matrix.Source: tions · pp. 83–84

5.3.2 Label Processing to Train SER Systems

• The penalization matrix is applied to hard-label, multi-label, and distribution-label training with cross-entropy, binary cross-entropy, and KLD losses, respectively.Label smoothing is also applied to hard-label and distribution-label learning.Source: §5.3.2 · p. 84

trix

• The penalty loss uses the matrix to weight per-sample, per-category losses, and the training objective combines it with the primary loss using α and β weights.The formulation requires prediction and target matrices; the target rows vary with the label-learning approach.Source: trix · pp. 84–85

5.4.1 Resource

• On the Podcast corpus, the training, development, and test sets contain 55,283, 9,546, and 16,570 utterances, respectively.The evaluation covers eight emotion classes; hard-label training retains only turns with consensus labels.Source: §5.4.1 · p. 86

5.4.2 Acoustic Features

• The models use 512-dimensional wav2vec features standardized with training-set statistics.The selected feature set is described as one of the more robust extraction techniques in a cited study.Source: §5.4.2 · p. 86

5.4.3 SER Models and Other Details

• Chunk-level models use an LSTM feature encoder and RNN-AttenVec attention, with sentences converted into uniformly sized overlapping chunks.Models were trained for 25 epochs with Adam, learning rate 0.0001, and batch size 128.Source: §5.4.3 · pp. 86–87

5.4.4 Evaluation Metrics

• Hard-label learning is assessed with F1, unweighted average recall, and precision; multi-label and distribution-label learning also use ranking and Hamming loss and coverage error.Predictions are binarized for multi-label and distribution-label evaluation using method-specific thresholds.Source: §5.4.4 · p. 87

5.4.5 Statistical Significance

• Statistical significance is tested with two-tailed t-tests after randomly splitting each test set into 30 roughly equal subsets.The test compares the proposed approach against baseline models.Source: §5.4.5 · p. 88

tems?

• On single-emotion utterances, LP+loss with α = 0.5 gains 16.22% relative maF1 over baseline; in multi-label classification, α = 1.0 gains 7.31%.These configurations lead four of five and four of six evaluation metrics, respectively.Source: tems? · pp. 88–89
• LP+loss typically gives the best outcomes across label-learning strategies, with 25.8% relative enhancement over baseline and results surpassing ’s 31.6% maF1.Table 5.1 covers distributional-label, multi-hard-label, and single-label primary-emotion tasks; its asterisk marks statistically significant differences from the baseline.Source: tems? · pp. 88–89, Table 5.1 · p. 89

5.5.2 Effect of Co-occurrence Matrix

• With LP+loss, co-occurrence-matrix distance falls from 4.27 to 4.00 for multi-label and from 4.13 to 3.39 for distribution-label predictions.The distance is measured with the Frobenius norm against the target matrix.Source: §5.5.2 · pp. 89–90

5.6 Summary

• The co-occurrence-based penalty loss generally improves most evaluation metrics in eight-class primary emotion classification tasks.It weights errors more heavily between more distinct emotions and was assessed with distribution-based, hard-label, and multi-label learning.Source: §5.6 · p. 90

Conclusion

• Neglecting minority ratings reduces standard approaches’ ability to cluster dual-emotion samples, while retaining those ratings can improve recognition of ambiguous utterances.The dissertation argues that retaining minority ratings accounts for subjectivity in emotion perception.Source: Conclusion · p. 91
• Learning from a larger group can enhance recognition through the all-inclusive rule or individual-rater modeling.The dissertation also links broader rater coverage to performance improvement.Source: Conclusion · pp. 91–92
• Multi-label SER systems achieve superior performance on test sets with single consensus labels from either the plurality rule or the all-inclusive rule.The dissertation recommends multi-label prediction rather than focusing on a single emotion.Source: Conclusion · p. 92

6.1 Discussion and Limitation

• The impact of dataset size on SER performance was not examined, and extensive cross-domain experiments were not undertaken; only one such experiment was performed.The proposed methods target traditional categorical emotion databases, while intensity-score databases may require optimization for training and evaluation.Source: §6.1 · pp. 92–93

6.2 Future Works

• Future work proposes a benchmark using the dissertation’s partitions to standardize evaluation and facilitate performance comparisons across SER models.The proposed analyses include performance disparities associated with natural biases in emotional perception.Source: §6.2 · p. 93