PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition

Muhammad Usama, Yeoreum Jung

arXiv:2610.13013v1cs.LG

TL;DR

Softmax-based multimodal emotion recognition cannot represent ignorance when inputs are weak or conflicting. EMT-DS combines evidential modality encoders with Dempster-Shafer fusion, and reports accuracy gains across four benchmarks. Its conflict measure also flags samples for human review, while its theoretical analysis assumes conditional independence between modalities.

  • Problem

    Softmax-based fusion forces a deterministic distribution even when evidence is weak or conflicting, limiting representation of sensor failure and semantic conflict.

    Source: A. Problem Definition · p. 1

  • Method

    EMT-DS uses Subjective Logic opinions and Dempster-Shafer fusion, with conflict serving as an explicit signal for complex emotional states.

    Source: 3) Total Loss · p. 6, §VI · p. 17

  • Results

    Across four benchmarks, EMT-DS improves accuracy over previous best methods by 2.6, 4.3, and 2.3 points on IEMOCAP, DEAP, and KEMDy20, and by 2.4 points on CMU-MOSEI over the strongest evidential baseline.

    Source: §VI · p. 17

  • Takeaways & Limitations

    The conflict term flags high-conflict samples for human review, and conflict rises under cross-dataset shift as accuracy degrades.

    Source: §V · p. 16, §VI · p. 17

    Theorem III.2 assumes conditional independence between modalities; explicitly modeling residual dependence is left to future work.

    Source: §V · p. 17

Abstract

from arXiv · show

Softmax-based multimodal emotion recognition methods often assign high confidence even when sensor data are corrupted or contradictory, for example when facial expressions suggest positive valence while vocal prosody indicates negative arousal. We present an Evidential Multimodal Transformer with Dempster-Shafer Fusion (EMT-DS) that addresses this problem through formal uncertainty quantification. EMT-DS models predictions as Subjective Opinions (Dirichlet distributions), separating epistemic uncertainty (lack of evidence) from aleatoric uncertainty (sensor noise). Modality-specific transformer encoders produce belief mass distributions that are fused with Dempster's Rule of Combination, which normalizes the joint belief mass by a conflict coefficient 1/(1-c), so that contradictory evidence is explicitly quantified as conflict and the fused prediction becomes diffuse rather than being forced into an unjustified consensus. Under subject-independent and speaker-independent evaluation on four benchmarks, EMT-DS achieves 84.3% accuracy on IEMOCAP versus 81.7% for the best baseline (p < 0.01), 82.4% on DEAP under leave-one-subject-out evaluation versus 78.1%, and 96.8% on the highly imbalanced KEMDy20, and it improves accuracy on CMU-MOSEI by 2.4 points over the strongest evidential baseline. In controlled ablation, it reduces Expected Calibration Error from 0.153 (softmax transformer) to 0.042. It maintains 71.2% accuracy under severe Gaussian noise on audio features (sigma = 1.0), where attention baselines drop to 45.3%, and degrades gracefully when entire modalities are absent. The fusion step has O(MK^2) complexity, compared with O(M^2 d^2) for cross-modal attention. The learned conflict metric also flags likely errors: reviewing 20% of samples captures 80% of misclassifications, which supports selective prediction in clinical and safety-critical systems.

I. INTRODUCTION

Multimodal emotion recognition combines complementary signals, but softmax-based fusion cannot represent ignorance when evidence is weak, conflicting, or missing. EMT-DS introduces evidential modeling and conflict-aware fusion to address this gap.

I. INTRODUCTION

  • Emotion recognition draws on complementary signals: facial expressions convey valence, vocal prosody arousal, physiological signals autonomic responses, and language semantic context.Multimodal systems integrating these streams outperform unimodal approaches.Source: §I · p. 1

A. Problem Definition

  • Softmax fusion forces a deterministic distribution even when evidence is weak or conflicting, limiting its ability to represent sensor failure, semantic conflict, and missing modalities.EMT-DS instead formulates predictions as Dirichlet distributions and separates belief from uncertainty using Subjective Logic.Source: A. Problem Definition · p. 1, A. Notation and Problem Formulation · p. 3
  • EMT-DS improves accuracy by 2.3 to 4.3 percentage points on subject-independent and speaker-independent benchmarks, and by 2.4 points over the strongest evidential baseline on CMU-MOSEI.It also reduces Expected Calibration Error by up to a factor of three.Source: A. Problem Definition · p. 2

A. Multimodal Emotion Recognition

  • Prior multimodal approaches include concatenation, late and hybrid fusion, attention, transformers, graph-based interactions, and mixture-of-experts, but softmax outputs lack formal uncertainty quantification.EMT-DS is presented as complementary to interaction-structuring methods, adding belief and conflict quantities with fixed probabilistic meaning.Source: A. Multimodal Emotion Recognition · p. 2

B. Physiological-Based Emotion Recognition

  • Physiological methods use signals such as EEG, EDA, and HRV, while existing credibility measures for EEG systems are described as heuristic rather than formally grounded.The cited DEAP work includes spatiotemporal attention with graph-based smoothing and EEG–electrooculography complementarity.Source: B. Physiological-Based Emotion… · p. 2

C. Evidential Deep Learning and Subjective Logic

  • Evidential deep learning parameterizes predictions with Dirichlet distributions, while Subjective Logic represents opinions using belief, uncertainty, and base rates.Prior evidential multimodal fusion work targets generic multi-view inputs, and cited methods in other domains focus on static images rather than temporal emotion dynamics.Source: C. Evidential Deep Learning and… · pp. 2–3

D. Uncertainty Quantification in Deep Learning

  • Bayesian neural networks and ensembles can estimate uncertainty but require multiple forward passes, whereas evidential methods offer single-pass uncertainty through predictive-distribution parameterization.EMT-DS combines evidential encoding with Dempster-Shafer fusion for multimodal emotion recognition.Source: D. Uncertainty Quantification in Deep… · p. 3

E. Dempster-Shafer Theory in Emotion Recognition

  • Unlike prior emotion-recognition methods applying Dempster-Shafer post hoc, EMT-DS learns Dirichlet evidence end to end and backpropagates through conflict computation.Conflict also serves as an explicit signal for complex emotional states rather than being normalized away.Source: E. Dempster-Shafer Theory in Emotion… · p. 3

B. Modality-Specific Evidential Encoding

EMT-DS encodes each modality with a specialized feature extractor and evidential head, converting bounded evidence into a Subjective Opinion that distinguishes belief from uncertainty.

1) Transformer-Based Feature Extraction:

2) Evidential Output Layer:

  • The evidential head replaces softmax with Dirichlet parameters, which are mapped to Subjective Logic belief masses and uncertainty.Softplus ensures nonnegative evidence, and the evidence vector is projected onto an L1 ball with radius Emax.Source: 2) Evidential Output Layer · p. 4
  • The evidence cap preserves the low-evidence regime while ensuring a finite evidence bound; with zero evidence, the opinion remains vacuous with uncertainty u = 1.The cap is set above the empirically observed evidence norm.Source: 2) Evidential Output Layer · p. 4
  • Epistemic uncertainty represents lack of evidence, while aleatoric uncertainty is captured by belief-distribution entropy; belief mass is not softmax confidence and uncertainty is not entropy.For four classes, evidence [8,0,0,0] gives belief b1 = 0.667, uncertainty u = 0.333, and projected probability p1 = 0.75; zero evidence gives u = 1.Source: 2) Evidential Output Layer · p. 5

C. Dempster-Shafer Fusion

EMT-DS fuses modality-specific Subjective Opinions with Dempster’s Rule, explicitly measuring disagreement so conflicting evidence is not simply averaged away.

C. Dempster-Shafer Fusion

  • Dempster-Shafer fusion explicitly computes conflict and adjusts fused belief, unlike weighted averaging or concatenation, which lack a mechanism for handling contradictory evidence.The method favors conflict-normalized combination over Yager’s redistribution of conflict or Bayesian averaging that ignores conflict.Source: C. Dempster-Shafer Fusion · p. 5

1) Pairwise Belief Fusion:

  • Pairwise conflict measures belief assigned to mutually exclusive hypotheses, ranging from agreement at zero conflict toward complete disagreement as conflict approaches one.The conflict-based normalization factor is n_ij = 1/(1-c_ij).Source: 1) Pairwise Belief Fusion · p. 5
  • Fused belief combines agreement on a class with each modality’s support when the other is uncertain, while fused uncertainty depends on the modalities’ uncertainties.The combined uncertainty decreases exponentially as informative modalities are added.Source: 1) Pairwise Belief Fusion · p. 5

2) Multi-Modal Fusion Algorithm:

  • For more than two modalities, EMT-DS iteratively applies pairwise fusion and summarizes disagreement using mean pairwise conflict from the original opinions, which is permutation invariant.Near-total conflict is clipped at 0.999 to cap the normalization factor at 1000 for numerical stability.Source: 2) Multi-Modal Fusion Algorithm · p. 5

D. Training Procedure and Loss Functions

The training procedure uses a Dirichlet-aware Bayes risk objective because evidential outputs are distributions over probability vectors, not fixed probability vectors.

D. Training Procedure and Loss Functions

1) Bayes Risk Loss:

  • The Bayes risk loss combines squared error between the ground-truth label and expected class probability with a variance penalty for high uncertainty.The expected probability and variance are defined under the Dirichlet distribution.Source: 1) Bayes Risk Loss · p. 6

2) KL Divergence Regularization:

The section regularizes evidential predictions toward maximum uncertainty when evidence is weak and develops a conflict-aware total loss. It also states a bounded-evidence stability guarantee for Dempster-Shafer fusion.

2) KL Divergence Regularization:

  • KL regularization toward Dir(1) favors maximum uncertainty when inputs lack discriminative features; λt ramps from 0.01 to 1.0 over the first 10 epochs.The annealing coefficient controls the regularization schedule.Source: 2) KL Divergence Regularization · p. 6

3) Total Loss:

  • The total objective aggregates risk and KL terms across modalities and the fused opinion, while a thresholded conflict penalty targets incorrect low-conflict predictions.The threshold is τ = 0.5; a detached argmax mask gates the penalty, so gradients flow through conflict magnitude rather than the argmax.Source: 3) Total Loss · pp. 6–7

E. Theoretical Analysis

  • The theoretical analysis reports Bayes consistency under conditional independence, unique KL minimization by the vacuous opinion, and empirical corroboration of out-of-distribution behavior.It also identifies removal of the fusion divergence associated with Zadeh’s Paradox through an architectural evidence cap.Source: E. Theoretical Analysis · p. 7

1) Unconditional Stability via an Architectural Evidence Cap:

  • With ||e_m||1 ≤ Emax, fusion has a finite global Lipschitz bound, although the stated worst-case L ≈ 4 × 10^8 far exceeds observed operating behavior.The evidence cap guarantees stability for every input; the projection remains inactive on observed samples.Source: 1) Unconditional Stability via an… · p. 7

2) Bayesian Consistency of Evidential Fusion:

The section characterizes when evidential Dempster-Shafer fusion yields calibrated Bayesian predictions and how its regularizer represents absent evidence. It also describes fusion behavior under disagreement and uninformative modalities.

2) Bayesian Consistency of Evidential Fusion:

  • When one modality is vacuous, fusion returns the other modality’s opinion; confident contradictory opinions can instead yield a diffuse prediction.The section gives [0.9,0.1] and [0.1,0.9] as an example that fuses to [0.5,0.5].Source: 2) Bayesian Consistency of Evidential… · pp. 7–8
  • Under conditional independence and a uniform class prior, calibrated modality predictions yield the Bayes-optimal posterior in the confident limit, which is itself calibrated.For nonzero uncertainty, the fused prediction interpolates between the Bayesian product and deferral to the more certain modality.Source: 2) Bayesian Consistency of Evidential… · p. 7

3) Out-of-Distribution Uncertainty via the Evidential Regularizer:

  • The KL regularizer is uniquely minimized by the vacuous opinion, encoding maximum epistemic uncertainty when discriminative evidence is absent.The stated minimizer is α = 1, equivalently e = 0 and u = 1.Source: 3) Out-of-Distribution Uncertainty via… · p. 8
  • The regularizer is not globally convex; the section relies on its unconditional minimizer property rather than a global convexity claim.At α = (1,10), the exact Hessian is indefinite.Source: 3) Out-of-Distribution Uncertainty via… · p. 8
  • When a modality is uninformative, fusion becomes invariant to it and is determined by the remaining informative modalities.The section also reports empirically that predicted uncertainty rises as corruption increases.Source: 3) Out-of-Distribution Uncertainty via… · p. 8

IV. EXPERIMENTS AND RESULTS

The experiments evaluate EMT-DS across four benchmarks using independent splits and specified training and evaluation protocols. The reported comparisons find EMT-DS best on every dataset and metric.

A. Benchmark Datasets

  • IEMOCAP uses session-independent folds, DEAP leave-one-subject-out, KEMDy20 speaker-independent folds, and CMU-MOSEI official splits, with no subjects, speakers, or sessions shared across splits.Preprocessing statistics are estimated on training folds only and applied unchanged to validation and test.Source: A. Benchmark Datasets · p. 9

Characteristic IEMOCAP DEAP KEMDy20 CMU-MOSEI

1) Hardware and Software Configuration:

  • Experiments use an NVIDIA GeForce RTX 5090 with 32GB VRAM, PyTorch 2.1, and mixed-precision training.The manuscript specifies preprocessing, model-selection, cross-validation, and tuning details for reproducibility.Source: 1) Hardware and Software Configuration · p. 9

2) Modality-Specific Feature Extraction:

3) Training Protocol:

  • Training uses batch sizes of 32 for IEMOCAP and KEMDy20 and 16 for DEAP, with five seeds; KEMDy20 adds focal loss and class-balanced sampling.Model selection uses the best validation unweighted accuracy, with early stopping on the same metric.Source: 3) Training Protocol · pp. 9–10

4) Evaluation Metrics:

  • IEMOCAP and KEMDy20 report weighted accuracy, unweighted accuracy, and macro F1; DEAP reports unweighted accuracy, alongside calibration and selective-prediction metrics.Additional metrics include ECE, MCE, NLL, Brier score, and AURC.Source: 4) Evaluation Metrics · p. 10

C. Comparison with Existing Methods

  • 84.3% accuracy (UA) and 82.7% F1 on IEMOCAP surpass DCT by 2.6 accuracy points; DEAP reaches 82.4% accuracy versus TMC’s 78.1%.KEMDy20 reaches 96.8% accuracy and 94.9% F1, exceeding CAG-MoE by 2.3 and 2.7 points, respectively; improvements over strongest baselines are significant on all three datasets (p < 0.01).Source: C. Comparison with Existing Methods · p. 10

D. Large-Scale Generalization on CMU-MOSEI

EMT-DS exceeds the strongest reported CMU-MOSEI baselines, with better calibration than the best-calibrated baseline. The section also examines class-level performance, ablation results, uncertainty and conflict, and modality-specific contributions.

D. Large-Scale Generalization on CMU-MOSEI

  • 54.8% accuracy on CMU-MOSEI exceeds the strongest baseline, Atsuko, by 1.7 points and the strongest evidential baseline, TMC, by 2.4 points.EMT-DS also reports ECE 0.052 versus 0.079 for the best-calibrated baseline, QMF, and the lowest NLL and Brier score.Source: D. Large-Scale Generalization on… · p. 10, Table IV · p. 11

E. Confusion Matrix Analysis

  • On KEMDy20, Sad, Angry, and Surprise each exceed 90% recall despite severe class imbalance; IEMOCAP’s largest confusion is Sad–Neutral at 6.1%.On DEAP, diagonal elements average 0.82 under leave-one-subject-out evaluation; Figure 2 shows confusion matrices with correct classification rates on the diagonal.Source: E. Confusion Matrix Analysis · p. 11, Figure 2 · p. 12

F. Ablation Study

  • Evidential encoding reduces ECE from > 0.14 to < 0.09 compared with softmax encoding across both concatenation and attention configurations.The ablation compares five configurations crossing softmax or evidential outputs with concatenation, attention, or Dempster-Shafer fusion.Source: F. Ablation Study · p. 11
  • EMT-DS (Row 5) beats the computationally expensive attention-based evidential model (Row 4) by +2.2 accuracy points, while its 8.7 GB memory footprint is comparable to concatenation’s 8.2 GB.Dempster-Shafer fusion operates on low-dimensional belief vectors with O(MK2), rather than high-dimensional feature maps.Source: F. Ablation Study · p. 11
  • 84.3% accuracy for full EMT-DS follows component gains: evidential encoding +2.8 points, D-S fusion +2.3, KL annealing +1.3 to 82.8%, then conflict loss +1.5.Average conflict falls from 0.198 to 0.143 with optimized KL annealing and to 0.126 with conflict regularization.Source: F. Ablation Study · pp. 11–12

G. Comparison with Uncertainty Quantification Methods

  • EMT-DS achieves ECE 0.042 and AURC 0.071 at one forward pass; deep ensembles match its calibration at five times the inference cost, and MC dropout at thirty times.The comparison includes temperature scaling, MC dropout, a five-model deep ensemble, and an evidential model without Dempster-Shafer fusion.Source: G. Comparison with Uncertainty… · p. 12

H. Uncertainty and Conflict Analysis

  • On KEMDy20, fused uncertainty is 0.086 ± 0.071, versus 0.176 ± 0.083 for text and 0.263 ± 0.112 for physiological signals; high conflict correlates with lower accuracy.The figure compares average uncertainty by modality on KEMDy20 and conflict with prediction accuracy on IEMOCAP; samples with ctotal > 0.4 have substantially lower accuracy, while those below 0.2 consistently achieve high accuracy.Source: H. Uncertainty and Conflict Analysis · p. 12

I. Modality Contribution Analysis

  • Text contributes the highest belief mass for Neutral (0.35), while audio leads for Happy and Sad (0.34 each) and Angry (0.36).The reported pattern distinguishes text’s semantic context from audio’s paralinguistic affective cues.Source: §I · p. 12

J. Extended Ablation Studies

Extended ablations test fusion choices, backbones, modality combinations, robustness, and calibration. Results favor evidential Dempster-Shafer fusion and show graceful performance under corruption and missing inputs.

1) Fusion Strategy Comparison:

3) Backbone-Agnostic Ablation:

  • Replacing transformer encoders with CNN-LSTM audio and BiLSTM text lowers ECE from 0.178 to 0.054 and raises σ = 1.0 accuracy from 38.4% to 66.8%.The evidential head and Dempster-Shafer fusion operate on modality feature vectors, independent of encoder architecture.Source: 3) Backbone-Agnostic Ablation · p. 13

4) Modality Combination Analysis:

  • 96.8% accuracy: the four-modality KEMDy20 configuration exceeds audio-plus-text at 88.4% and audio-plus-text-plus-physiology at 92.7%.Text alone achieves 71.2%; adding video to the three-modality configuration contributes an additional 4.1%.Source: 4) Modality Combination Analysis · p. 13, Table VII · p. 13

5) Hyperparameter and Loss Sensitivity Analysis:

  • Accuracy peaks at λt = 0.8; λc = 0.10 is stable, while removing conflict loss lowers accuracy to 82.8% and raises ECE to 0.058.The learning-rate plateau spans 1×10^-4 to 5×10^-4; the clipping constant leaves accuracy and ECE invariant from 0.99 to 0.9999.Source: 5) Hyperparameter and Loss Sensitivity… · p. 13

6) Cross-Dataset Generalization:

  • Under zero-shot transfer, IEMOCAP → KEMDy20 reaches 62.3%; rejecting the least-confident half restores over 90% accuracy on the remainder.Mean conflict rises to 0.342 and 0.468 for the two transfers from IEMOCAP, versus 0.126 in-domain.Source: 6) Cross-Dataset Generalization · p. 14

K. Robustness to Modality Corruption

1) Complete Modality Missingness:

  • Dropping audio on IEMOCAP retains 71.2% accuracy versus 56.4% for attention fusion; dropping audio and video on KEMDy20 retains 82.6% versus 54.2%.When a modality is missing, its evidence is set to zero and uncertainty rises, so fusion defers to remaining modalities.Source: 1) Complete Modality Missingness · p. 14

L. Uncertainty Calibration Analysis

  • ECE falls from 0.153 for standard Softmax to 0.042 for EMT-DS, whose reliability curve closely tracks ideal calibration.The softmax baseline is systematically overconfident, frequently assigning > 0.9 probability to incorrect predictions.Source: L. Uncertainty Calibration Analysis · p. 14, Figure 6 · p. 15

M. Conflict-Triggered Selective Review

The section evaluates conflict as an error-review signal, then examines EMT-DS’s computational cost, memory use, broader benchmark behavior, and limits. It concludes with the method’s cross-benchmark gains and describes the supplementary material.

M. Conflict-Triggered Selective Review

  • Conflict correlates with human-rated cross-modal incongruence on 300 IEMOCAP utterances (Spearman ρ = 0.57, p < 0.001), more than the error head (ρ = 0.34).The labels were assigned independently of model outputs.Source: M. Conflict-Triggered Selective Review · p. 15
  • 80.2% of errors are detected within a 20% review budget using conflict loss, versus 71.2% for intrinsic conflict without that loss.Intrinsic conflict also exceeds predictive entropy (58.3%) and random review (20.1%); a supervised error head reaches 80.9% detection but has worse risk-coverage (0.074 versus 0.071).Source: Figure 7 · p. 15, Table IX · p. 15, M. Conflict-Triggered Selective Review · p. 15

1) Theoretical Complexity:

  • O(MK^2) Dempster-Shafer fusion is more efficient than O(M^2d^2) cross-modal attention when K ≪ d.For the reported typical settings, transformer encoders—not fusion—dominate EMT-DS’s asymptotic cost.Source: 1) Theoretical Complexity · p. 16

2) Empirical Runtime Analysis:

  • EMT-DS runs at 87.0 samples/second in training and 142.9 in inference, with 11% and 8% overhead over concatenation, respectively.It is 3.76× faster during training than CAG-MoE and converges in 1.4 hours versus CAG-MoE’s 4.5 hours.Source: 2) Empirical Runtime Analysis · p. 16

3) Memory Footprint:

  • 8.7 GB peak memory at batch size 32 is comparable to concatenation, while attention-based fusion requires 12.3 GB.D-S fusion uses compact belief representations rather than full feature tensors.Source: 3) Memory Footprint · p. 16

V. DISCUSSION

  • Across the discussion, EMT-DS improves on all four benchmarks, uses adaptive belief allocation on imbalanced KEMDy20, and presents conflict and uncertainty as interpretable review signals.Its conflict is negatively correlated with accuracy (Pearson r = −0.68); the authors also report robustness across tested hyperparameter ranges and rising conflict under zero-shot cross-dataset transfer. They note that Theorem III.2 assumes conditional independence, while measured residual dependence is modest on average (|ρ| = 0.16±0.09).Source: §V · pp. 16–17

VI. CONCLUSION

  • The conclusion reports accuracy gains of 2.6, 4.3, and 2.3 points on IEMOCAP, DEAP, and KEMDy20, and 2.4 points over the strongest evidential baseline on CMU-MOSEI.It also reports reduced calibration error and modest overhead, and identifies continuous emotion spaces, temporal dynamics, and adaptive handling of sensor failures as future work.Source: §VI · p. 17

Supplementary Material for “Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition”

  • The supplementary material collects full theoretical proofs, implementation and reproducibility details, and extended supporting experiments.It distinguishes main-manuscript references from supplement-prefixed equation, table, and figure numbers.Source: Supplementary Material for “Evidential… · p. 19

I. PROOFS OF THE THEORETICAL RESULTS

The proofs establish bounded, stable Dempster-Shafer fusion and relate its confident limit to Bayesian combination under stated assumptions. They also show how fused predictions behave when one modality is uninformative.

A. Proof of Theorem III.1 (Unconditional Stability)

  • The conflict normalization factor remains finite because the non-conflicting mass is bounded away from zero by positive modality uncertainties.The evidence cap bounds each modality’s strength, supporting the bounded normalizer used in the stability argument.Source: A. Proof of Theorem III.1… · p. 19
  • Induction over modalities and a bounded fusion Jacobian establish that the fusion operator is Lipschitz continuous.The Jacobian depends on polynomial products scaled by bounded normalizers.Source: A. Proof of Theorem III.1… · p. 19

B. Proof of Proposition III.1 and Theorem III.2 (Bayesian Consistency)

  • As evidence and conflict approach zero for one modality, fusion reduces to the projected probability of the other modality.This follows from the confident-limit fusion expression and its normalization tending to one.Source: B. Proof of Proposition III.1 and… · p. 19
  • Under conditional independence given the label and calibrated modality predictions, the joint likelihood factorizes and the confident-limit fused probability equals the Bayesian posterior.For nonzero uncertainty, the fused probability smoothly interpolates between that posterior and the more certain modality’s projected probability.Source: B. Proof of Proposition III.1 and… · p. 19

C. Proof of Theorem III.3 (Vacuous Opinion Minimizes the Regularizer)

The proofs show that the KL evidential regularizer uniquely favors the vacuous opinion, while fusion with a maximally uncertain modality leaves an informative modality’s opinion unchanged in the limit.

C. Proof of Theorem III.3 (Vacuous Opinion Minimizes the Regularizer)

  • The KL regularizer is nonnegative and uniquely minimized at α = 1, equivalent to zero evidence and maximum uncertainty.With the Softplus floor, this minimum is approached as evidence tends to zero.Source: C. Proof of Theorem III.3 (Vacuous… · pp. 19–20
  • The regularizer is strictly convex near the vacuous opinion but not globally convex; its Hessian is indefinite at α = (1, 10).The unique-minimizer result relies on Gibbs’ inequality, not global convexity.Source: C. Proof of Theorem III.3 (Vacuous… · p. 20

D. Proof of Lemma III.1 (Modality Noise Immunity)

  • When one modality becomes maximally uncertain, fused belief and uncertainty converge to those of the informative modality.The limiting conflict is zero and the fusion normalization approaches one.Source: D. Proof of Lemma III.1 (Modality… · p. 20

E. Proof of Lemma III.2 (OOD Gradient Vanishing)

The supplement examines why evidential learning resists uninformative inputs and tests calibration, robustness, and conflict-design choices. Its analyses also assess backbone independence and zero-shot transfer.

E. Proof of Lemma III.2 (OOD Gradient Vanishing)

  • At vacuous evidence e = 0, the KL evidence gradient vanishes, so the parameter gradient is zero and no update fits uninformative data.Under increasing input corruption, predicted evidence approaches this configuration and the gradient norm collapses.Source: E. Proof of Lemma III.2 (OOD Gradient… · p. 20

F. Proof of Lemma III.3 (Non-Triviality of Evidential Learning)

II. DETAILED IMPLEMENTATION AND REPRODUCIBILITY

  • The supplement collects extended ablations and analyses; preprocessing, alignment, encoder, and training details remain in the main manuscript.Source: §II · p. 20

A. Comparison with Uncertainty Quantification Methods

  • 0.042 Expected Calibration Error and 0.071 AURC at a single forward pass are EMT-DS’s best results among the compared uncertainty methods.It is also the most accurate in the comparison; deep ensembles require five training and inference cycles, while MC dropout incurs thirtyfold inference overhead.Source: A. Comparison with Uncertainty… · p. 20

B. Backbone-Agnostic Ablation

  • 0.054 ECE and 66.8% accuracy at σ = 1.0 noise with CNN-LSTM/BiLSTM backbones show calibration and noise-robustness gains beyond transformer encoders.The softmax attention comparison records ECE 0.178 and noise-corrupted accuracy 38.4% at σ = 1.0.Source: Table II · p. 21, B. Backbone-Agnostic Ablation · p. 21

C. Conflict-Loss and Numerical Sensitivity

  • At τ = 0.5, accuracy and selective-review error detection peak; removing conflict loss (λc = 0) lowers accuracy to 82.8% and worsens calibration.Accuracy and ECE are invariant across clipping constants 0.99–0.9999, and clipping activates on at most 0.02% of fusion operations.Source: C. Conflict-Loss and Numerical… · p. 21

A. Cross-Dataset Generalization

  • Under IEMOCAP-to-KEMDy20 transfer, accuracy rises above 90% when retaining the most confident 50% of samples, while conflict increases under dataset shift.The reported conflict values are 0.342 and 0.468 for the two IEMOCAP transfers, versus 0.126 in-domain.Source: A. Cross-Dataset Generalization · p. 21

B. Scalability

The supplement evaluates EMT-DS’s scaling, computational cost, and behavior across classes, fusion strategies, corruption, and modality contributions. Its analyses also show how conflict and uncertainty can inform selective review.

D. Per-Class Calibration Error Correlation

  • Fused ECE remains low across IEMOCAP classes; audio/text calibration-error correlation ranges from ρ = 0.08–0.11 for Neutral and Sad to 0.28 for Angry.The passage interprets this as evidence that residual cross-modal dependence does not materially degrade calibration in practice.Source: D. Per-Class Calibration Error… · pp. 21–22

E. Fusion Strategy Comparison

A. Gradient Vanishing under Corruption

  • As input noise approaches σ = 1, EMT-DS’s KL-loss gradient norm collapses toward zero, while the softmax baseline continues propagating gradients.The passage says the collapse prevents weight updates from corrupted data.Source: A. Gradient Vanishing under Corruption · p. 22

B. Scalability

  • Training time scales approximately linearly with modality count (R² = 0.993), and inference throughput stays above 95 samples per second with six modalities.The comparison contrasts this with quadratic scaling for pairwise attention.Source: B. Scalability · p. 21, Figure 2 · p. 22

C. End-to-End Cost Profiling

  • EMT-DS converges in 26 epochs and 1.4 hours, versus 1.2 hours for concatenation and 4.5 hours for CAG-MoE, with 8.7 GB peak memory.Frozen-encoder features are cached, amortizing feature extraction during training.Source: C. End-to-End Cost Profiling · p. 21, Table VII · p. 22

B. Modality Contribution Analysis

  • Text contributes most belief mass for Neutral (0.35), while audio contributes most for Happy (0.34), Sad (0.34), and Angry (0.36).The passage associates these patterns with descriptive language and paralinguistic cues, respectively.Source: B. Modality Contribution Analysis · p. 22

C. Qualitative Failure Cases

  • The reported conflict-by-correctness test cases illustrate that conflict and fused uncertainty have interpretable, actionable meaning.Under zero-shot domain shift, rejecting high-uncertainty samples improves retained-set accuracy.Source: C. Qualitative Failure Cases · p. 22, Figure 1 · p. 22

k bm

  • The D-S rule normalizes away conflicting mass, combines joint agreement, and yields fused uncertainty that rises with conflict; combination is iterative for more than two modalities.Figures 5 and 6 illustrate the evidential output and fusion modules.Source: §VI · p. 22, k bm · p. 23

Related papers

Loading 2610.13013v1…