PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting multi-CNN features in CNN-RNN based Dimensional Emotion Recognition on the OMG in-the-wild Dataset

Dimitrios Kollias, Stefanos Zafeiriou

arXiv:1910.01417v2cs.LGcs.CVstat.ML

TL;DR

The paper addresses visual-only valence and arousal estimation on the in-the-wild OMG-Emotion dataset. It pre-trains CNN-RNN systems on Aff-Wild and Aff-Wild2, uses multi-level CNN features with recurrent subnetworks, and evaluates aggregation and fusion strategies. The resulting architectures outperform standard CNN-RNN and multimodal state-of-the-art methods, with especially strong valence performance and improved arousal when low- and high-level features are combined.

  • Problem

    The paper addresses valence and arousal estimation for subtle affective behavior in the in-the-wild OMG-Emotion dataset using visual information.

  • Method

    The approach pre-trains CNN-RNN architectures on Aff-Wild and Aff-Wild2, processes low-, mid-, and high-level CNN features with recurrent subnetworks, and aggregates and fuses their predictions.

  • Results

    CNN plus multi-RNN architectures outperform standard CNN-RNN and multimodal state-of-the-art networks, while combining low- and high-level features improves arousal estimation.

  • Takeaways & Limitations

    Independent recurrent processing of multi-level CNN features and RNN-based fusion provide the paper’s strongest visual-only architecture results for dimensional emotion recognition.

  • Takeaways & Limitations

    Arousal performance is worse than valence performance, and audio cues appear more discriminating for arousal than facial features in correlation coefficient.

Abstract

from arXiv · show

This paper presents a novel CNN-RNN based approach, which exploits multiple CNN features for dimensional emotion recognition in-the-wild, utilizing the One-Minute Gradual-Emotion (OMG-Emotion) dataset. Our approach includes first pre-training with the relevant and large in size, Aff-Wild and Aff-Wild2 emotion databases. Low-, mid- and high-level features are extracted from the trained CNN component and are exploited by RNN subnets in a multi-task framework. Their outputs constitute an intermediate level prediction; final estimates are obtained as the mean or median values of these predictions. Fusion of the networks is also examined for boosting the obtained performance, at Decision-, or at Model-level; in the latter case a RNN was used for the fusion. Our approach, although using only the visual modality, outperformed state-of-the-art methods that utilized audio and visual modalities. Some of our developments have been submitted to the OMG-Emotion Challenge, ranking second among the technologies which used only visual information for valence estimation; ranking third overall. Through extensive experimentation, we further show that arousal estimation is greatly improved when low-level features are combined with high-level ones.

1 INTRODUCTION

The paper targets continuous valence and arousal estimation from in-the-wild facial behavior using visual information and novel CNN plus multi-RNN architectures. It combines multi-level CNN features, task-specific recurrent processing, pre-training, sequence-aware aggregation, and fusion to improve performance.

  • Motivation: Continuous valence and arousal estimation models affective behavior beyond discrete expressions, including subtle emotions in everyday interactions.Valence represents positivity or negativity, while arousal measures emotional activation.
  • Motivation: In-the-wild databases support facial analysis under unconstrained conditions, complementing earlier research based mainly on controlled data and handcrafted features.Examples include OMG-Emotion, Aff-Wild, Aff-Wild2, SEWA, and AffectNet.
  • Approach: The proposed CNN plus multi-RNN architecture extracts low-, mid-, and high-level CNN features and processes them within a multi-task optimization framework.These features are passed to recurrent components because they may provide rich information for valence-arousal prediction.
  • Findings: Independent RNNs processing features from separate CNN layers outperform concatenating those features for a single RNN, and outperform standard CNN-RNN and multimodal state-of-the-art networks.The paper compares concatenation into one RNN with separate recurrent subnetworks.
  • Approach: Utterances are split into sequences, whose mean or median predictions are aggregated to produce utterance-level valence and arousal estimates.This procedure adapts the architecture to utterance-level OMG-Emotion annotations.
  • Findings: Low-level features combined with high-level features improve arousal estimation, while visual-only networks perform better for valence than arousal.The study also evaluates decision-level and model-level fusion, using an RNN for model-level fusion.

2 RELATED WORK

Related work spans CNN, CNN-RNN, multimodal, sequential, and task-specific facial-affect models for valence-arousal estimation. Existing methods include handcrafted and learned features, AU-based representations, and recurrent architectures evaluated across relevant databases.

  • Deep architectures: Early deep architectures compared frame-based CNNs with CNN-RNN models for valence and arousal estimation.The CNN-RNN variants retained convolutional representations before recurrent prediction.
  • Multimodal methods: Multimodal systems fused acoustic, visual, and textual features, including learned visual representations and acoustic descriptors, before recurrent processing.One described approach used a LSTM after combining the modalities.
  • Task-specific representations: FATAUVA-Net sequentially trained core, facial-attribute, action-unit, and valence-arousal layers, using AUs as mid-level representations.It achieved the highest results in the First Affect-in-the-wild Challenge according to the cited work.
  • State of the art: AffWildNet combined ResNet-50 convolutional and pooling layers with a fully connected layer, a 2-layer GRU, and output estimates.The cited work reports it as the best-performing network in the Aff-Wild database.
  • Comparison: Table 1 summarizes state-of-the-art algorithms, their valence-arousal performance, and the databases used.The table is intended to support comparisons across methods and datasets.

3 THE UTILIZED IN-THE-WILD DIMENSIONAL EMOTION DATABASES

The paper uses three in-the-wild dimensional-emotion resources: Aff-Wild and Aff-Wild2 for pre-training, and OMG-Emotion as the target benchmark. These datasets provide unconstrained videos annotated for valence and arousal, with OMG-Emotion emphasizing emotions that emerge over time.

  • Aff-Wild: Aff-Wild is a large in-the-wild database annotated for valence and arousal and contains more than 30 hours of video from 200 subjects.It includes 298 videos and 1,224,100 frames.
  • Aff-Wild2: Aff-Wild2 extends Aff-Wild with 13 hours and 5 minutes of additional YouTube videos and increases the number of subjects under arbitrary recording conditions.The extension was used as a basis for the ABAW Competition.
  • OMG-Emotion: OMG-Emotion contains YouTube monologue videos in which emotional expressions emerge and develop over time, with valence and arousal annotations and many identities.Sample images depict varied people and in-the-wild emotional situations.
  • OMG-Emotion: OMG-Emotion is split into subject-independent training, validation, and test sets for the OMG-Emotion Challenge.Each subject appears in only one split.
  • Annotations: Annotations are assigned at the utterance level from an average of five independent annotations, with annotators receiving preceding video context.The dataset also provides categorical annotations and transcripts.

4 PRE-PROCESSING: FACE DETECTION & ALIGNMENT, IMAGE RESIZING & NORMALIZATION

Pre-processing prepares in-the-wild facial video for feature extraction by detecting faces, locating landmarks, and aligning frames with Procrustes analysis.

  • Face detection: Face bounding boxes are extracted from all video frames using the Deformable Part Model detector ffld2.The detector was selected for face detection in-the-wild.
  • Annotation inspection: Figure 3 displays utterance-level valence-arousal annotations as training, validation, and test histograms and distributions.Training is shown in green, validation in red, and test in blue.
  • Face alignment: Facial landmarks are located with dlib before alignment is performed using five anchor points around the eyes, nose, and mouth.The implementation detects 68 landmarks per frame and uses five corresponding facial points for alignment.
  • Face alignment: Generalized Procrustes Analysis removes in-plane rotation, isotropic scaling, and translation from the selected landmark coordinates.

5 THE DEVELOPED ARCHITECTURES

The developed framework combines CNN and recurrent architectures for dimensional emotion estimation and focuses on multi-task learning because it performed better for both affective dimensions.

  • Architecture framework: The framework includes CNN, standard CNN-plus-RNN, CNN-plus-Multi-RNN architectures, and network ensembles for fusion.
  • Multi-task learning: Multi-task learning provided better valence and arousal estimation than independently trained uni-task models.The authors relate this result to inter-correlations between valence and arousal.

5.1 CNN architectures

The CNN architectures use pre-trained visual backbones, frame sequences, and facial landmarks to estimate valence and arousal for utterance-derived sequence segments.

  • CNN backbones: VGG-Face, ResNet-50, and DenseNet-121 were pre-trained on Aff-Wild or Aff-Wild2 before training on OMG-Emotion.
  • Sequence processing: Each CNN receives a frame sequence with 68 facial landmarks per frame and predicts a valence-arousal pair for every frame.
  • Sequence aggregation: The final sequence-level valence and arousal estimates are computed as either the mean or median of the corresponding per-frame predictions.
  • Feature integration: Landmarks are concatenated with features from the CNN pooling output before the fully connected prediction layer.For VGG-Face, the first fully connected layer has 4096 units; ResNet-50 and DenseNet-121 use averaged pooled features before concatenation.

5.2 Standard CNN plus RNN architectures

The standard CNN-plus-RNN architectures add temporal modeling after CNN feature extraction, while AffWildNet uses a ResNet-50-based CNN, fully connected layer, and two-layer GRU.

  • Standard CNN-plus-RNN: The standard CNN-plus-RNN design feeds the CNN’s last pooling-layer output through a fully connected layer into the RNN layers.
  • AffWildNet: AffWildNet uses ResNet-50 convolutional and pooling layers, a 1500-unit fully connected layer, and a two-layer GRU with 128 units per layer.Facial landmarks are concatenated with averaged pooled ResNet-50 features before the fully connected layer.
  • Sequence aggregation: The CNN-RNN network processes frame sequences, predicts valence-arousal values per frame, and aggregates them by mean or median into the final estimate.
  • Architecture overview: Figure 4 represents the sequence-level V-A output and allows VGG-FACE, ResNet-50, or DenseNet-121 as the CNN component.

5.3 CNN plus Multi-RNN networks

The paper develops CNN plus Multi-RNN architectures that extract low-, mid-, and high-level CNN features, either concatenating them for one RNN or processing them independently with three RNN subnets. The architectures predict valence and arousal from frame sequences, with CNN-1RNN and CNN-3RNN variants differing in how multi-level features are processed.

  • Architectures: CNN-1RNN concatenates features from three CNN layers and passes them to one RNN, whereas CNN-3RNN processes the three feature groups independently through three RNN subnets.The extracted features represent low-, mid-, and high-level CNN information.
  • CNN-3RNN: CNN-3RNN uses three 2-layer GRU networks to predict valence and arousal from low-, mid-, and high-level features.Each GRU layer comprises 128 units, and predictions are made for each frame in the input sequence.
  • CNN-3RNN: The CNN-3RNN-2nd-pool last-pool fc variant routes fully connected, last-pooling, and second-pooling features to RNN1, RNN2, and RNN3, respectively.This design uses separate recurrent processing for the three selected CNN feature levels.
  • CNN-1RNN: CNN-1RNN concatenates low-, mid-, and high-level features before passing them to a 2-layer GRU that predicts valence and arousal.The 68 facial landmarks are concatenated with last-pooling-layer features before the fully connected layer.
  • CNN-1RNN: The CNN-1RNN-2nd-pool last-pool fc variant concatenates second-pooling, last-pooling, and fully connected features before the RNN and provided the best results among these networks.The architecture produces a valence-arousal estimate for each input sequence of consecutive frames.

5.4 Ensemble Methodology

The ensemble methodology fuses CNN-1RNN and CNN-3RNN networks at model or decision level. Model-level fusion combines recurrent outputs through another RNN or a fully connected layer, while decision-level fusion computes performance-weighted averages of network estimates.

  • Fusion strategies: Model-level fusion concatenates high-level features from different networks, while decision-level fusion uses weighted averaging of their predictions.The two approaches are intended to exploit mutual information or combine network outputs, respectively.
  • Model-level Fusion: Model-level fusion feeds concatenated outputs from the component RNNs to either a 128-unit GRU layer or a 128-unit fully connected layer.The resulting network then predicts valence-arousal values for each frame and computes mean or median final estimates.
  • Decision-level Fusion: Decision-level fusion computes final valence or arousal estimates as weighted averages of the corresponding estimates from CNN-1RNN and CNN-3RNN networks.Each network weight is proportional to its validation-set performance.
  • Decision-level Fusion: The decision-level weights use the validation-set Concordance Correlation Coefficient for valence or arousal.The CCC is the OMG-Emotion Challenge evaluation criterion and ranges from −1 to 1.
  • Decision-level Fusion: CCC is computed from label and prediction variances, means, and covariance for each affective dimension.The notation distinguishes valence and arousal through i ∈ {v, a}.

6 NETWORK TRAINING DETAILS

The networks are trained with CCC-based objectives and selected sequence, batch, learning-rate, and dropout settings. Validation-set CCC determines whether median filtering and utterance-level smoothing are retained for test-time post-processing.

  • Training setup: The training-details section covers learning rates, dropout, batch size, sequence length, the loss function, and post-processing of valence and arousal estimates.These settings are described for the developed CNN, CNN-RNN, and CNN Multi-RNN architectures.
  • Regularization: Dropout is applied with probability 0.5 on fully connected layers and probability 0.8 after the first GRU layer.The fully connected layers sit above the convolutional and pooling components.
  • Training setup: 80-frame sequences and batch size 4 are used for CNN plus RNN and CNN plus Multi-RNN training, while CNN training evaluates sequence sizes from 40 to 100 frames.The 80-frame setting provided the best results for CNN training.
  • Optimization: End-to-end CNN plus RNN training uses learning rate 10^-4 or 10^-5, whereas training with fixed CNN parts uses 10^-3.All networks were trained using TensorFlow on a Quadro GV100 Volta GPU, with training taking about a day.
  • Objective function: The total loss is defined as 1 − ρ_a + ρ_v, where ρ_a and ρ_v are the CCC values for arousal and valence.The loss is based on the CCC evaluation criterion used in the OMG-Emotion Challenge.
  • Post-processing: Median filtering of per-frame predictions and smoothing of per-utterance predictions are retained only when they improve validation-set CCC.Retained post-processing configurations are applied unchanged to the test partition.

7 EXPERIMENTAL RESULTS

Experiments show that multi-RNN processing of CNN features, pre-training, landmarks, and model-level fusion improve dimensional emotion recognition, while visual-only arousal remains weaker than valence.

  • Median aggregation produced the best final per-frame valence and arousal estimates across experiments.Averaging multiple trainings and 10-fold cross-validation showed no significant difference.
  • CNN-RNN Component Analysis: 6%–10% average improvement over CNN plus RNN architectures was obtained by CNN-1RNN and CNN-3RNN models.CNN-3RNN further outperformed CNN-1RNN by processing low- and high-level feature time variations independently.
  • Ensemble Architectures: Model-level fusion outperformed decision-level fusion, and RNN-based fusion outperformed fully connected fusion.The fused model exploits richer representations from different networks.
  • Performance Limitations: Visual-only arousal performance was worse than valence performance, while post-processing mainly improved valence estimation.The authors state that audio cues appear more discriminating for arousal than facial features.
  • Aff-Wild2 pre-training boosted all networks compared with training directly on OMG-Emotion.The authors attribute this to Aff-Wild2’s size and emotion diversity.
  • CNN-RNN Component Analysis: Low- and high-level CNN features performed best, while mid-level features alone performed worst in the CNN-3RNN ablation.The optimal combination was CNN-3RNN-2nd-pool last-pool fc, and low-level features significantly affected both valence and arousal prediction.
  • CNN-RNN Component Analysis: Landmarks increased performance by 1.2%–1.9% across the evaluated architectures.Encoding landmarks to match CNN feature size did not significantly improve performance and increased model complexity.
  • Comparison with State-of-the-Art: Model-level Fusion + RNN outperformed all compared methods on valence and arousal, including methods using audio.CNN-3RNN also outperformed the state-of-the-art networks except a multimodal method and two multimodal ensembles.

8 CONCLUSIONS

The paper develops visual CNN-RNN architectures for valence-arousal prediction on OMG-Emotion, using multi-level CNN features, RNN subnets, fusion, and Aff-Wild or Aff-Wild2 pre-training. Model-level fusion through an RNN produced the best results.

  • The proposed architectures predict valence-arousal from OMG-Emotion using visual information and pre-training on Aff-Wild or Aff-Wild2.They process low-, mid-, and high-level CNN features with either a single RNN or RNN subnets.
Loading 1910.01417v2…