Oracle Complementarity Is Not Realizable Complementarity in Frozen-Encoder Audio-Visual Emotion Recognition

1. INTRODUCTION

• Oracle headroom counts clips where branches disagree in the right direction, but deployable fusion must identify and act on those clips from inputs alone.The authors therefore characterize the unrealized gap as structural, not simply a consequence of insufficiently clever fusion.Source: §1 · p. 1
• 3.4× more oracle headroom appears on CREMA-D with the clean self-supervised audio encoder than with the contaminated fine-tuned encoder.The authors attribute the increase to audio-branch accuracy; controlled degradation follows the same pattern.Source: §1 · p. 1
• Concatenation converts 0.39 to 0.55 of CREMA-D headroom, but converts an indistinguishable-from-zero fraction on EAV despite 2.5× greater headroom.The EAV comparison is against CREMA-D with the contaminated encoder.Source: §1 · pp. 1–2
• A learned router underperforms plain concatenation in every tested corpus and regime.This includes the clean regime, where headroom is largest.Source: §1 · p. 2

2. EXPERIMENTAL SETUP

• All encoders remain frozen while only a small MLP head is trained, with subject-independent GroupKFold evaluation across five CREMA-D folds and seven EAV folds.The head-level splits have no actor overlap; results are averaged over three seeds, with 95% actor-level cluster-bootstrap intervals.Source: §2 · p. 2
• The evaluation covers CREMA-D with contaminated and clean audio encoders, plus EAV with the contaminated encoder; a clean bimodal corpus was unavailable.Audio is emotion2vec+ base or self-supervised non-plus emotion2vec; XEUS is added as a clean, architecturally distinct provenance check.Source: §2 · p. 2
• CREMA-D may enter the contaminated audio encoder through a two-hop fine-tuning lineage, which head-level cross-validation does not remove.EAV is not an EmoBox member, and visual encoders are unaffected; the clean non-plus encoder is not evaluated on EAV because its scripted content enables near-perfect audio classification.Source: §2 · p. 2
• The reported frozen-encoder accuracies are not comparable to published fine-tuned CREMA-D results because no frozen audio-visual baseline exists for that corpus.The study reports weighted average recall (WAR).Source: §2 · p. 2

3. ORACLE, REALIZED GAIN, AND UNREALIZED GAP

• The oracle O is the accuracy of a selector with perfect knowledge of which branch is correct, and bounds any late-fusion selector.For two branches, the double-fault rate D = 1 − O is the floor on which no branch selector can help.Source: §3 · p. 2
• H is oracle headroom above the stronger branch, Δ is realized gain, U is the unrealized gap, and ρ is the converted fraction.The branch correctness indicators a_i, v_i, and f_i denote audio, visual, and fusion correctness for clip i over N clips.Source: §3 · p. 2
• Embedding-level concatenation is not bounded by O and can be correct even when both branches err, so U can in principle be negative.The tested cells all have positive U; O is constructed over audio and the visual ensemble.Source: §3 · p. 2
• Concat trains a head on concatenated frozen embeddings, while the learned router predicts audio errors from concatenated embeddings and per-branch confidence statistics to defer to the visual head.The oracle-route is included as the infeasible ceiling arm.Source: §3 · p. 2

4.1. CREMA-D under the contaminated audio encoder

• Concatenation raises CREMA-D accuracy from 0.898 to 0.919, while visual accuracy gains from 0.574 to 0.652 leave the oracle near 0.95 and headroom near +0.05.Fusion accuracy was statistically equivalent across the four visual encoders at a ±0.01 margin.Source: §4.1 · pp. 2–3

4.2. CREMA-D under the clean audio encoder

• Replacing emotion2vec+ with the self-supervised non-plus encoder raises CREMA-D headroom from +0.055 to +0.186 (3.4×), while converted fraction rises only from 0.39 to 0.55.The audio-accuracy drop also includes a supervised-versus-self-supervised capability gap that the study cannot separate from contamination.Source: §4.2 · p. 3
• At matched audio accuracy, degraded contaminated audio reproduces clean-encoder headroom, and XEUS lies on the same curve with H = +0.133 at 0.758 WAR.Along the degradation sweep, converted fraction is non-monotone, dipping to 0.36 before rising to 0.53.Source: §4.2 · p. 3
• A learned combiner over frozen penultimate representations reaches about 0.81 under clean audio versus concatenation's 0.787, but still leaves about 0.06 absolute accuracy unrealized.The gain is significant per actor and in all six seeds.Source: §4.2 · p. 3

4.4. Selection does not realize the ceiling

• The learned router loses to concatenation in every tested regime, while Fig. 1 places contaminated, clean, and XEUS encoders on one accuracy-linked degradation curve.Under contaminated CREMA-D audio the router reaches 0.883 versus concatenation's 0.919; under clean audio it reaches 0.693 versus 0.787, and on EAV it reaches 0.632 below concatenation.Source: §4.4 · pp. 3–4, Figure 1 · p. 4

5. DISCUSSION AND LIMITATIONS

• The router predicts audio errors only moderately (AUROC 0.773 on CREMA-D and 0.621 on EAV), and the reported AUROCs are not directly comparable.The routers predict different targets across the two corpora.Source: §5 · p. 4
• Richer router inputs improve discrimination and calibration in all three cells, yet no variant beats concatenation, consistent with hard deferral being the bottleneck.The authors hypothesize that unconditional concatenation avoids per-clip identification by learning a fixed mixture.Source: §5 · p. 4
• The study tests frozen encoders with trained heads on acted or elicited studio recordings, not spontaneous in-the-wild speech.Only three cells were evaluated because no clean bimodal corpus was available to the authors.Source: §5 · p. 4

• +0.055 to +0.186: replacing a fine-tuned audio encoder with a self-supervised one triples oracle headroom on CREMA-D.Source: §6 · p. 4
• 0.39 to 0.55: concatenation’s converted fraction rises, while on EAV it is indistinguishable from zero despite larger headroom.Source: §6 · p. 4
• Explicit selection is beaten by plain concatenation in every tested corpus and regime; oracle ceilings should be diagnostics, not fusion targets.Complementarity claims should report ρ alongside O, compare H with the independence prediction (1 −WAR(a)) WAR(v), and check each encoder’s training lineage.Source: §6 · p. 4

• The work is a secondary analysis of publicly available CREMA-D and EAV corpora.Source: Compliance with Ethical Standards · p. 5
• No new human-subject data were collected.Source: Compliance with Ethical Standards · p. 5
• No additional ethical approval was required for the reported analyses.Source: Compliance with Ethical Standards · p. 5