Source-linked AI summary
AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning
Chuan Han Li, Yun Chien Cheng
TL;DR
Emotion recognition from continuous audio-visual streams requires systems to identify utterance boundaries that are often unavailable in advance. AGF-Net combines automatic segmentation with multimodal fusion, and reports competitive performance across clean and noisy conditions.
Problem
Many multimodal emotion-recognition methods rely on manually segmented utterances, while real-world systems receive continuous audio-visual streams without predefined boundaries.
Source: §1 · pp. 2–3Method
AGF-Net detects candidate utterance segments from continuous audio-visual streams, extracts audio and visual features, and fuses them for emotion classification.
Source: §1 · pp. 2, 4Results
64.92% Accuracy and 65.47% F1-score (clean), 62.13% Accuracy and 63.08% F1-score (moderate noise), and 58.19% Accuracy and 59.41% F1-score (higher noise) are reported for AGF-Net.
Source: §4.9 · p. 10Takeaways & Limitations
The framework reduces reliance on manually annotated utterance boundaries and maintains competitive performance on automatically segmented inputs across noise levels.
Source: §1 · p. 4, §4.9 · p. 10
Abstract
from arXiv · showhide
This study develops an audio-visual emotion recognition system that integrates automatic utterance segmentation with multimodal fusion. Unlike conventional methods that rely on manually annotated utterance boundaries, the proposed system automatically generates candidate segments from continuous dialogue and extracts the corresponding audio and visual features for emotion classification. To address the challenges introduced by automatic segmentation, the proposed AGF-Net integrates Audio-Guided Fusion (AGF), Multimodal Negative Learning (MNL), and Predictive Dynamic Fusion (PDF) to improve cross-modal fusion, modality balance, and robustness to varying modality quality. Experimental results show that AGF-Net achieves 64.92% Accuracy and 65.47% F1-score under clean conditions, 62.13% Accuracy and 63.08% F1-score under moderate noise, and 58.19% Accuracy and 59.41% F1-score under higher noise. Overall, the proposed framework reduces reliance on manually annotated utterance boundaries and maintains competitive performance under automatically segmented and noisy audio-visual inputs.
1. Introduction
The paper targets emotion recognition from continuous audio-visual streams, where utterance boundaries are not known in advance and common fusion approaches face segmentation, modality-balance, and alignment challenges. It proposes automatic segmentation and multimodal fusion to address these challenges.
1. Introduction
- The proposed system detects candidate utterance segments from continuous audio-visual streams, extracts modality-specific features, and fuses them for emotion classification.The paper combines voice activity detection with semantic assistance to generate candidate segments and refine their boundaries.Source: §1 · pp. 2–3
- Most existing multimodal emotion-recognition approaches use manually segmented utterances, although real-world systems receive continuous streams without predefined boundaries.Inaccurate segmentation can create incomplete, excessively short, or overly long clips, degrading extracted features and recognition performance.Source: §1 · pp. 2–3
- The paper adopts an anchor-based strategy to address temporal asynchrony and cross-modal misalignment in multimodal data.This approach uses a modality with stronger signal quality or discriminative power to guide alignment and fusion of other modalities.Source: §1 · p. 3
- AGF-Net targets insufficient cross-modal complementarity and modality imbalance through stronger feature fusion and reduced modality dominance.The study focuses on feed-forward and objective-level changes to mitigate modality imbalance.Source: §1 · p. 4
- The proposed framework aims to improve recognition robustness under imperfect segmentation, noise, missing modalities, and temporal misalignment.The paper’s stated contributions include reducing reliance on predefined utterance boundaries and improving robustness under these conditions.Source: §1 · p. 4
2. Related Work
The related-work section describes the speech, vision, and segmentation components used in the system. It covers wav2vec 2.0, ResNet18, Silero VAD, and WhisperX.
2.1. Wav2vec 2.0
- Wav2vec 2.0 learns transferable speech features directly from raw waveforms using self-supervised pretraining and is fine-tuned for downstream tasks.Its pretraining masks time steps and uses a contrastive objective to identify target representations from negative samples.Source: §2.1 · p. 4
2.2. ResNet18
- ResNet18 is a convolutional vision network whose residual connections stabilize optimization and help alleviate performance degradation as depth increases.The paper also characterizes it as a backbone for vision tasks with a balance of accuracy, robustness, and computational cost.Source: §2.2 · p. 4
2.3. Silero VAD
- Silero VAD detects speech and non-speech regions in continuous audio and serves as the foundation for automatic utterance segmentation.It outputs speech probabilities for short streaming chunks, which can be combined with smoothing and threshold rules to detect speech onsets and endpoints.Source: §2.3 · p. 4
2.4. WhisperX
- WhisperX combines speech recognition with voice activity detection and forced alignment to provide transcriptions and more precise word-level timestamps.Its process segments speech, transcribes each region, and aligns recognized text with the original waveform.Source: §2.4 · pp. 4–5
3. Material and Methods
AGF-Net processes automatically segmented audio-visual clips, extracts modality-specific features, and combines AGF, PDF, and MNL. The experiments use IEMOCAP and examine segmentation, fusion, and robustness under modality-quality variation.
3.1. Dataset
- The study uses IEMOCAP’s audio-visual recordings from 10 actors across five sessions, focusing on angry, happy, neutral, and sad emotion categories.The dataset includes scripted and improvised dyadic conversations; actor-ID-based splitting keeps each actor in only one data split.Source: §3.1 · p. 5, §3.3.2 · p. 5
3.2. Experimental Procedure
- The experimental procedure splits data, automatically segments continuous dialogue, extracts audio-visual features, and evaluates segmentation, fusion methods, AGF, MNL warm-up settings, and PDF robustness.The listed evaluations include model performance under noise after adding PDF.Source: §3.2 · p. 5
3.3.1 Image Preprocessing
- Visual preprocessing aligns and crops detected faces using five facial landmarks, while audio is resampled to 16 kHz and z-score normalized.Each aligned face is resized to 224×224 for ResNet18; 16 kHz matches the wav2vec 2.0 input requirement.Source: §3.3.1 · p. 5, §3.3.2 · p. 5
3.4. Automatic Utterance Segmentation
- Automatic segmentation tracks speech probabilities from streaming audio, smooths short-term fluctuations, and applies hysteresis and adaptive endpoint decisions to emit candidate clips.Pre-roll and post-roll preserve boundary information, and overly short segments are removed.Source: §3.4 · pp. 5–6
3.5. Audio-Guided Lag-aware Network
- The lag-aware network predicts a temporal center and span for each audio event, then biases attention toward temporally consistent visual features with a Gaussian mask.The predicted span represents alignment uncertainty or temporal extent.Source: §3.5 · p. 6
3.6. PDF
- PDF estimates modality confidence and dynamically adjusts fusion weights, using relative calibration to downweight more uncertain modalities.Its confidence estimates combine each modality’s class probability with information from other modalities.Source: §3.6 · p. 6
3.7. MNL
- MNL uses a more reliable modality to guide weaker modalities in suppressing non-target classes, aiming to stabilize decisions while preserving modality-specific cues.It identifies strong and weak modalities per sample and is activated after supervised warm-up.Source: §3.7 · p. 6
3.8. Model Overview
- AGF-Net extracts audio and visual features from automatically segmented clips, uses AGF to integrate their cues, PDF to weight branches by reliability, and MNL to stabilize weaker branches.The model combines these components to improve stability and robustness under automatically segmented inputs.Source: §3.8 · p. 6
4. Result
The evaluation reports classification metrics and segmentation overlap, with implementation details describing training, inputs, and noise perturbations.
4.1. Evaluation Methods
- Accuracy, Precision, Recall, and F1-score evaluate classification, while IoU measures temporal overlap between predicted segments and ground-truth utterances.Lower IoU can indicate boundary shifts, over-segmentation, or under-segmentation.Source: §4.1 · p. 7
4.2. Implement Detail
- Training uses 100 epochs with early stopping, AdamW, batch size 8, and learning rates of 1 × 10−4 for the prediction head and 1 × 10−5 for both modality branches.The inputs preserve each automatically segmented audio clip’s original duration rather than truncating or padding clips to a fixed duration.Source: §4.2 · p. 7
- Noise is randomly added to either audio or visual input per sample, simulating degradation of one modality while the other remains relatively reliable.Audio uses additive white Gaussian noise and visual input uses Gaussian noise.Source: §4.2 · p. 7
4.3. Automatic Segmentation
Automatic segmentation produces usable candidate clips, with semantic hold improving segmentation F1 over endpoint strategies. In the fusion comparison, Hybrid Fusion achieves the strongest average performance among the tested fusion strategies.
4.3. Automatic Segmentation
- F1-score rises from 0.4372 with confidence endpoints to 0.4608 with partial-ASR semantic hold, at 564.0 ms average endpoint latency.Adaptive endpoints reach 0.4532 F1-score; the authors attribute the further gain mainly to reducing over-segmentation from conversational pauses.Source: §4.3 · p. 8, Table 1 · p. 8
4.4. Comparison of hybrid fusion
- Audio-only performance exceeds visual-only performance, while Mid Fusion improves on audio-only by 3.75 percentage points in Accuracy and 2.77 in Macro F1-score.Late Fusion reaches 49.85 ± 4.74% Accuracy and 48.52 ± 6.52% Macro F1-score; the authors describe decision-level fusion as less effective with imbalanced modality strengths.Source: §4.3 · p. 8
- Hybrid Fusion leads the compared fusion strategies, reaching 53.74 ± 4.47% Accuracy and 52.61 ± 6.16% Macro F1-score.It exceeds Mid Fusion by 0.80 percentage points on both metrics and Late Fusion by 3.89 points in Accuracy and 4.09 in Macro F1-score.Source: §4.3 · p. 8, Table 2 · p. 8
- Hybrid Fusion combines feature-level fusion with decision information from the audio, visual, and fusion branches for the final prediction.The authors report improved average recognition performance under automatically segmented audio-visual inputs.Source: §4.3 · p. 8
4.5. Ablation study of AGF
Adding Audio-Guided Fusion improves average recognition performance under automatically segmented inputs and slightly reduces variation across sessions.
4.5. Ablation study of AGF
- 54.65 ± 3.98% Accuracy and 53.87 ± 5.53% Macro F1-score are achieved with AGF, versus 53.74 ± 4.47% and 52.61 ± 6.16% without it.The gains are 0.91 percentage points in Accuracy and 1.26 in Macro F1-score.Source: §4.5 · p. 8, Table 3 · p. 8
- AGF reduces standard deviation from 4.47% to 3.98% for Accuracy and from 6.16% to 5.53% for Macro F1-score.The authors interpret this as slightly reduced performance variation across sessions.Source: §4.5 · p. 8
- Audio-guided temporal fusion uses audio events to find temporally relevant visual information when clip boundaries or audio-visual emotion timing are imperfect.The authors report consistent average-performance gains while slightly reducing cross-session variability.Source: §4.5 · p. 8
4.6. Comparison of MNL Warmup epochs
MNL performance depends on its warm-up duration, with 30 epochs best among the tested settings. PDF improves Macro F1 under clean and noisy conditions, with larger gains under noise.
4.6. Comparison of MNL Warmup epochs
- 57.17 ± 5.15% Accuracy and 56.79 ± 5.51% Macro F1-score are the best results among the tested MNL warm-up settings, at 30 epochs.A 10-epoch warm-up gives the lowest results; performance declines at 40 epochs and partially recovers at 50, remaining below the 30-epoch setting.Source: §4.6 · pp. 8–9, Table 4 · p. 9
4.7. Ablation of PDF
- With PDF, clean-condition Macro F1-score rises from 56.79 ± 5.51% to 57.40 ± 5.87%, a 0.61-point improvement.The authors attribute the gain to dynamically adjusting prediction-branch contributions by estimated reliability.Source: §4.7 · p. 9
- Under moderate noise (λ=5), Macro F1-score rises from 53.07 ± 4.44% to 55.21 ± 5.23%; under higher noise (λ=10), it rises from 50.60 ± 3.73% to 52.58 ± 4.52%.The authors report that PDF reduces the influence of less reliable predictions and helps preserve performance as modality quality varies.Source: §4.7 · p. 9, Table 5 · p. 9
4.8. Comparison of other Models
AGF-Net remains competitive across clean and noisy conditions, attaining the highest F1-score in the moderate-noise comparison. The experiment summary also reports gains from Hybrid Fusion, AGF, and PDF, with a 30-epoch MNL warm-up performing best among tested settings.
4.8. Comparison of other Models
- In the clean setting, AGF-Net scores 64.92% Accuracy and 65.47% F1-score, below Social-MAE's 66.72% and 67.89% but above AVF-MAE and VAVL.Under higher noise, its F1-score is 0.20 percentage points below Social-MAE's 59.61%, although its Accuracy is lower than Social-MAE and AVF-MAE.Source: §4.8 · p. 9
- Under moderate noise, AGF-Net's 63.08% F1-score exceeds Social-MAE, AVF-MAE, and VAVL by 0.52, 4.15, and 7.89 percentage points, respectively.Its 62.13% Accuracy is comparable to Social-MAE's 62.17%.Source: §4.8 · p. 9
4.9. Experiment Summary
- The experiment summary reports Hybrid Fusion at 53.74 ± 4.47% Accuracy and 52.61 ± 6.16% Macro F1-score, with AGF raising these to 54.65 ± 3.98% and 53.87 ± 5.53%.It also reports the best tested MNL warm-up at 30 epochs and PDF Macro F1 gains under clean, moderate-noise, and higher-noise conditions.Source: §4.9 · p. 10
4.8. Comparison of other Models
- AGF-Net achieves 64.92% Accuracy and 65.47% F1-score when clean, 62.13% and 63.08% under moderate noise, and 58.19% and 59.41% under higher noise.It has the highest F1-score under moderate noise and remains close to the best-performing method under higher noise.Source: Table 6 · p. 9, §4.8 · p. 9, §4.9 · p. 10
5. Result
AGF-Net reports its overall performance across clean and noisy settings, alongside ablation results for AGF, MNL, and PDF.
5. Result
- 64.92% Accuracy and 65.47% F1-score under the clean setting; at λ=5, 62.13% Accuracy and 63.08% F1-score; at λ=10, 58.19% Accuracy and 59.41% F1-score.These are the reported overall AGF-Net results across clean and noisy conditions.Source: §5 · p. 10
- AGF raises average Accuracy from 53.74 ± 4.47% to 54.65 ± 3.98% and Macro F1-score from 52.61 ± 6.16% to 53.87 ± 5.53% in five-fold LOSO ablations.The comparison is for incorporating the AGF module.Source: §5 · p. 10
- MNL's 30-epoch warm-up achieves the best average test performance, with 57.17 ± 5.15% Accuracy and 56.79 ± 5.51% Macro F1-score.Source: §5 · p. 10
- PDF raises clean-condition Macro F1-score from 56.79 ± 5.51% to 57.40 ± 5.87%, with noisy-condition Macro F1-scores of 55.21 ± 5.23% at λ=5 and 52.58 ± 4.52% at λ=10.Source: §5 · p. 10
6. Future Work
Future work proposes adding auxiliary modalities, using conversational context, and testing generalization on other audio-visual emotion recognition datasets.
6. Future Work
- Additional textual, semantic, or physiological modalities could complement audio and visual information for recognizing complex emotional states.Source: §6 · p. 10
- Neighboring utterances, dialogue history, or short-term memory could help account for emotional changes and reduce the impact of insufficient information or incomplete single-segment automatic segmentation.Source: §6 · p. 10
- Evaluation on other datasets is proposed to further verify generalization across data distributions, speakers, languages, and interaction scenarios.Source: §6 · p. 10