A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

1. Introduction

• Dimensional emotion recognition represents emotions on continuous valence and arousal scales, but obtaining continuous annotations is harder than labeling discrete emotions and can produce noisy, ambiguous labels.In-the-wild videos add challenges such as poor illumination, pose variation, and background noise.Source: §1 · p. 2
• Concatenation-based audio-visual fusion can miss inter-modal relationships, motivating methods that better exploit complementary facial and vocal information.The modalities can provide cues when the other is degraded or absent, such as voice when facial imagery is blurred or visuals during silent audio.Source: §1 · p. 2
• Prior attention approaches for dimensional emotion recognition often model intra-modal relationships, while existing cross-attention methods do not effectively use audio-visual feature correlation.The paper proposes computing cross-correlation between joint audio-visual features and each individual modality.Source: §1 · p. 2
• The paper introduces joint cross-attention for audio-visual fusion and analyzes its effectiveness through experiments and ablations on Affwild2.The stated contribution uses joint audio-visual representations to compute attention weights from correlations with individual features.Source: §1 · pp. 2–3

2.1. A-V Fusion Based Emotion Recognition

• Earlier audio-visual emotion-recognition systems used concatenated features with recurrent networks, two-stream models, or teacher–student learning, yet the surveyed approaches were described as insufficiently capturing inter-modal relationships and salient task-relevant features.The section also reviews methods using CNN and RNN features, auto-encoders, knowledge distillation, and semi-supervised pseudo-labeling.Source: §2.1 · p. 3

2.2. Attention Models for A-V Fusion:

• Attention-based fusion methods reweight or integrate audio and visual features, but the surveyed transformer cross-attention approaches do not leverage semantic relevance based on cross-correlation.The review also covers attentive fusion and leader-follower fusion for dimensional emotion recognition.Source: §2.2 · p. 3
• The proposed approach uses concatenated audio-visual features to attend to each modality and itself, jointly modeling intra- and inter-modal relationships through cross-correlation.This contrasts with prior cross-attention approaches that operate across individual modalities.Source: §2.2 · p. 3

3.1. Visual Network:

• I3D is paired with a temporal convolutional network to capture short- and long-term visual dynamics; R(2plus1)D and ResNet with GRU are also explored.The paper notes that I3D alone fails to capture long-term dependencies.Source: §3.1 · p. 4

3.2. Audio Network:

• Spectrograms and 2D CNNs provide the audio representations, drawing on spectrograms’ para-lingual information relevant to affective state.ResNet18 is among the architectures used for audio feature extraction.Source: §3.2 · p. 4

3.3. Joint Cross-Attentional AV-Fusion:

• Joint cross-attention uses concatenated A-V features to attend to each modality, capturing intra- and inter-modal relationships before valence or arousal prediction.The approach trains audio and visual models independently before fusion; the joint representation is intended to reduce modality heterogeneity. Joint correlation matrices provide relevance measures within and across modalities.Source: §3.3 · pp. 4–5

4.1. Dataset:

• Affwild2 provides 567 valence-arousal videos, partitioned by subject into 341 training, 71 validation, and 152 test videos.The annotations are continuous in [-1, 1] and average four expert ratings.Source: §4.1 · p. 6

4.2. Implementation Details:

• Visual inputs are cropped, aligned, and resized to 224x224; I3D processes eight non-overlapping clips per 64-frame subsequence.Missing visual frames are represented by black frames; the I3D model is pretrained on ImageNet and inflated to a 3D CNN using Affwild2 videos.Source: §4.2 · p. 6
• Audio is resampled to 44100 Hz and converted into normalized log-power spectrograms that ResNet18 processes as single-channel inputs.Audio segments are synchronized with visual clips; each subsequence yields a 64 x 107 spectrogram, and ResNet18 is trained from scratch.Source: §4.2 · p. 6
• The fusion network uses concatenated A-V features of size 1024, with dropout of 0.5 on attended features and weight decay of 5e-4.The cross-attention weights are Xavier-initialized and optimized with Adam.Source: §4.2 · p. 6
• Decision-level fusion uses a fully connected network to weight predictions for valence and arousal.Source: §4.2 · p. 7

5.1. Ablation Study

• With the fusion model, I3D’s improvement over R3D is greater for valence than arousal; adding TCN to I3D performs even better than vanilla I3D by capturing longer-term temporal cues.The fusion model outperforms other attention strategies, especially for valence; 3D-CNNs perform slightly better than ResNet18 with GRU.Source: §5.1 · p. 7

5.2. Comparison to state-of-the-art

• The state-of-the-art comparison reimplements prior approaches under the authors’ experimental protocol because their original protocols and training strategies differ.The comparison is on the Affwild2 validation set.Source: §5.2 · p. 7, Table 2 · p. 8
• Joint cross-attention improves valence fusion over vanilla cross-attention and prior attention strategies, while arousal remains below some prior methods but exceeds unimodal visual performance.The authors characterize the method as more effective for emotion-range variation (valence) than intensity (arousal).Source: §5.2 · p. 7, Table 2 · p. 8
• On the ABAW challenge test set, the proposed fusion improves over unimodal performance, particularly for valence, without external datasets, multi-backbone features, or advanced loss components.The authors note that predictions are clipped to [-1,1].Source: §5.2 · p. 7
• The authors identify fusion across multiple audio and visual backbones, at feature or decision level, as a way to further improve the model.Source: §5.2 · p. 8

• The model uses joint audio-visual features to compute attention over each modality and itself, then concatenates attended features to predict valence and arousal.The audio and visual backbones are trained individually before joint-feature attention is applied.Source: §6 · p. 8
• Experiments on Affwild2 using different audio and visual backbones support the authors’ conclusion that the fusion approach can outperform state-of-the-art methods.The authors describe the approach as cost-effective.Source: §6 · p. 8