Source-linked AI summary
Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition
Srinivas Parthasarathy, Shiva Sundaram
TL;DR
Audio-visual expression recognition must handle real-world cases where either input modality is missing. This paper evaluates a transformer under modality ablations and trains models with visual cues randomly removed at clip or frame level; the proposed models outperform the baseline under ablated tests, with gains up to 17% for frame-level ablations.
Problem
The robustness of audio-visual expression recognition when an input modality is completely or partially missing remains an open research question relevant to real communication scenarios.
Source: §1 · pp. 1–2Method
Results
Proposed models gain up to 17% in frame-level ablations and outperform the baseline under ablated evaluation while retaining near-state-of-the-art performance under ideal conditions.
Source: §1 · p. 2, §4.2 · p. 4, §5 · p. 4Takeaways & Limitations
The proposed training techniques generalize across varied test conditions involving missing data, including conditions intended to address real-world use cases.
Source: §4.2 · p. 4, §5 · p. 4
Abstract
from arXiv · showhide
Automatic audio-visual expression recognition can play an important role in communication services such as tele-health, VOIP calls and human-machine interaction. Accuracy of audio-visual expression recognition could benefit from the interplay between the two modalities. However, most audio-visual expression recognition systems, trained in ideal conditions, fail to generalize in real world scenarios where either the audio or visual modality could be missing due to a number of reasons such as limited bandwidth, interactors' orientation, caller initiated muting. This paper studies the performance of a state-of-the art transformer when one of the modalities is missing. We conduct ablation studies to evaluate the model in the absence of either modality. Further, we propose a strategy to randomly ablate visual inputs during training at the clip or frame level to mimic real world scenarios. Results conducted on in-the-wild data, indicate significant generalization in proposed models trained on missing cues, with gains up to 17% for frame level ablations, showing that these training strategies cope better with the loss of input modalities.
1 INTRODUCTION
Audio-visual expression recognition combines complementary cues, but real-world communication can lose either modality completely or intermittently. This study evaluates missing-modality robustness and proposes training with visual-input ablations, which improves performance under ablated tests while retaining near-state-of-the-art performance in ideal conditions.
1 INTRODUCTION
- Real communication services can lose a complete modality or individual frames through camera shutoff or connectivity problems, motivating robustness evaluation.Prior work established complementary audio and visual cues, while the robustness of expression recognition without an input modality remains an open question.Source: §1 · p. 1
- The study examines a state-of-the-art transformer under missing-input conditions and proposes clip- or frame-level visual ablation during training.Clip-level training zeroes an entire clip; frame-level training zeroes or repeats frames.Source: §1 · pp. 1–2
- Proposed models significantly outperform the baseline under frame- and clip-level ablated tests while retaining near-state-of-the-art performance under ideal conditions.The study evaluates audio-visual expression recognition on Aff-Wild2.Source: §1 · p. 2
- The paper evaluates audio-visual expression systems across mismatched conditions found in real-world scenarios.It distinguishes its expression-recognition focus from prior missing-modality studies of lipreading and action recognition.Source: §1 · pp. 1–2
2 PROPOSED MODEL
The paper uses a transformer for continuous audio-visual expression prediction, with separate modality encoders and cross-modal attention. It trains additional models by randomly removing visual information at clip or frame level to address missing-input conditions.
2 PROPOSED MODEL
- The task is to predict arousal and valence for each frame in a sequence of audio-visual input.Arousal denotes excitement level and valence denotes positivism level.Source: §2 · p. 2
2.1 Crossmodal Transformer Architecture
- Separate audio and video encoder branches process their inputs before a cross-modal attention layer fuses the modalities.The cross-modal layer uses queries from one modality and keys and values from the other, then adds attention outputs back to the encoder representations with learnable scalar weights.Source: §2.1 · p. 2
2.2 Training with missing data
- Training ablations target visual cues because baseline results show greater performance impact from visual loss than from audio loss.Clip-Zero drops an entire sequence; Frame-Zero and Frame-Repeat respectively zero or repeat individual frames, each with a specified probability.Source: §2.2 · p. 2
3 EXPERIMENTAL SETUP
The experiments evaluate frame-level valence-arousal prediction on Aff-Wild2, using face and audio features as transformer inputs. Training and validation use three-second sequences, and model selection uses validation data because the test set is not publicly released.
3.1 Dataset
- Aff-Wild2 provides 558 in-the-wild videos split into speaker-independent training, validation, and test sets of 350, 70, and 138 videos.This study focuses on frame-level valence-arousal estimation; the dataset also includes expression-classification and action-unit tasks.Source: §3.1 · p. 2
3.2.1 Video.
- Video inputs are frontal-face crops resized to 96x96x3, with features extracted from a pretrained VGG-Face model.The 4096-dimensional representation from its first fully connected layer feeds the transformer video branch.Source: §3.2.1 · p. 2
3.2.2 Audio.
- Audio inputs use 65 low-level descriptors, including MFCC, loudness, and fundamental frequency, synchronized and concatenated into a 3900-dimensional vector.Features use 30 ms windows with a 10 ms step and are downsampled and concatenated over a 2 s window.Source: §3.2.2 · p. 3
3.3 Experimental framework
- Models predict on 100-frame sequences corresponding to three seconds, with clip- and frame-ablation probabilities fixed at 0.5.Because the test set is not publicly released, models are optimized on part of the training set and evaluated on validation data; CCC is the evaluation metric.Source: §3.3 · p. 3
4 RESULTS
Training with missing visual cues improves robustness to video ablations, while audio ablations have little effect on performance. Under ideal conditions, the baseline reaches higher CCC for arousal than valence, and valence appears more dependent on video.
4 RESULTS
- Evaluation tests Clip-Zero at p∈{1.0, 0.7, 0.5, 0.3, 0.0} and Frame-Zero and Frame-Repeat at p∈{1.0, 0.95, 0.90, 0.85, 0.0}.The extremes represent complete absence or complete presence of the modality for Clip-Zero.Source: §4 · p. 3
- At ideal conditions (p=0.0), the baseline achieves CCC values of 0.393 for valence and 0.522 for arousal; proposed models match baseline performance in most cases.Valence suffers more than arousal under these conditions, indicating greater dependence on the visual modality.Source: §4 · p. 3
4.1 Audio Ablations
- Audio ablations cause less than 1% absolute performance drop across models and test conditions, while proposed models cope with baseline performance after half the video data is removed.A video-only model without cross-modal attention reaches CCC values of 0.381 for valence and 0.489 for arousal.Source: §4.1 · p. 3, Figure 3 · p. 4
4.2 Video Ablations
- Clip-Zero training handles video ablations better than the baseline, especially for arousal at high missing-data probabilities.At p=1.0, baseline performance decreases by almost 100%; the trained model performs similarly to an audio-only model without cross-modal attention.Source: §4.2 · p. 3
- At p=0.95, proposed Frame-Zero and Frame-Repeat models significantly outperform the baselines for both valence and arousal.Baseline CCC values are 0.113 for valence and 0.235 for arousal under Frame-Zero, and 0.285 and 0.410, respectively, under Frame-Repeat.Source: §4.2 · p. 3
- Up to 17% gains and performance that quickly saturates at ideal conditions show generalization across video-ablation test conditions.Frame-Repeat gains are less pronounced than Frame-Zero gains; the results suggest the model can learn expressions from a few video frames.Source: §4.2 · p. 4
5 CONCLUSION
The conclusion presents clip- and frame-level visual ablations as strategies for training an audio-visual expression-recognition transformer with missing inputs. The proposed models perform better than the baseline when evaluation data is missing, without significant loss under ideal conditions.
5 CONCLUSION
- The training strategy randomly ablates visual cues by zeroing entire clips or zeroing or repeating frames.The study analyzes a state-of-the-art transformer with attention for in-the-wild audio-visual expression recognition.Source: §5 · p. 4
- Proposed models perform better than the baseline when data is missing during evaluation, without significant performance loss under ideal conditions.The authors characterize this as superior generalization across test cases addressing a real-use challenge.Source: §5 · p. 4, Figure 4 · p. 4
- Future work will explore subclip ablations, alternative frame-level probability distributions, and ablating faces while retaining background scene information.Source: §5 · p. 4