PaperlayerView exact revision on arXiv

Source-linked AI summary

Analysing Affective Behavior in the First ABAW 2020 Competition

Dimitrios Kollias, Attila Schulc, Elnar Hajiyev, Stefanos Zafeiriou

arXiv:2001.11409v2cs.LGstat.ML

TL;DR

The paper addresses the need for automatic affective behavior analysis in-the-wild across dimensional, categorical, and facial-action tasks. It presents the ABAW 2020 Competition, its common Aff-Wild2 benchmark, evaluation, baseline systems, and leading team methods. Across the three Challenges, participating teams outperformed the baselines, with different teams leading different metrics.

  • Problem

    Affective behavior analysis in-the-wild is needed for machines and robots to understand human feelings, emotions, and behaviors in real-world interaction.

  • Method

    The paper presents three Competition Challenges using Aff-Wild2 for valence-arousal estimation, seven-expression classification, and eight-action-unit detection, alongside metrics, baselines, and top-team methodologies.

  • Results

    All participating teams outperformed their baselines across the three Challenges, while leading teams differed by task and evaluation metric.

  • Takeaways & Limitations

    ABAW 2020 establishes a common in-the-wild benchmark for evaluating three complementary affective behavior-analysis tasks.

Abstract

from arXiv · show

The Affective Behavior Analysis in-the-wild (ABAW) 2020 Competition is the first Competition aiming at automatic analysis of the three main behavior tasks of valence-arousal estimation, basic expression recognition and action unit detection. It is split into three Challenges, each one addressing a respective behavior task. For the Challenges, we provide a common benchmark database, Aff-Wild2, which is a large scale in-the-wild database and the first one annotated for all these three tasks. In this paper, we describe this Competition, to be held in conjunction with the IEEE Conference on Face and Gesture Recognition, May 2020, in Buenos Aires, Argentina. We present the three Challenges, with the utilized Competition corpora. We outline the evaluation metrics, present both the baseline system and the top-3 performing teams' methodologies per Challenge and finally present their obtained results. More information regarding the Competition, the leaderboard of each Challenge and details for accessing the utilized database, are provided in the Competition site: http://ibug.doc.ic.ac.uk/resources/fg-2020-competition-affective-behavior-analysis.

I. INTRODUCTION

The competition targets affective behavior analysis in-the-wild to support more human-centered interaction between people and machines. It unifies dimensional, categorical, and facial-action analysis across three Challenges using Aff-Wild2.

  • Motivation: Affective behavior analysis in-the-wild is presented as important for human-centered interaction with machines and robots in real-world applications.The intended systems should understand feelings, emotions, and behaviors to interact naturally and assist people.
  • Emotion representations: Human emotions are represented categorically through seven basic emotions, facially through Action Units, and dimensionally through continuous valence and arousal.Valence indicates how positive or negative an emotional state is, while arousal indicates how passive or active it is.
  • Competition design: The Competition introduces three Challenges for valence-arousal estimation, seven basic expression classification, and eight action unit detection.Each Challenge addresses one of the three targeted behavior-analysis tasks.
  • Competition design: All three Challenges use Aff-Wild2, described as the first large in-the-wild audiovisual database annotated for all three behavior tasks.This provides a common benchmark across dimensional, categorical, and facial-action recognition.

II. COMPETITION CORPORA

The Competition relies on Aff-Wild2, a large YouTube-derived in-the-wild database supporting all three behavior tasks. Its valence-arousal annotations use continuous expert ratings and subject-independent data partitions.

  • Database overview: Aff-Wild2 is the first database annotated for valence-arousal estimation, action unit detection, and basic expression classification.These annotations form the three Competition Challenges, and cropped aligned images were provided to participating teams.
  • Database overview: Aff-Wild2 contains 545 videos and 2,786,201 frames, including videos with two annotated subjects.The videos were collected from YouTube, and the database extends the earlier Aff-Wild collection.
  • Valence-arousal annotation: Four experts produced frame-by-frame valence and arousal annotations through joystick-based continuous ratings.Values range from −1 to 1, and final labels are the mean of the four annotations.
  • Valence-arousal annotation: The mean inter-annotation correlation is 0.63 for valence and 0.60 for arousal.All subjects appearing in each video were annotated.
  • Data partitioning: Subject-independent partitions allocate 346 videos to training, 68 to validation, and 131 to test.A person can appear in only one of the three subsets.

B. Aff-Wild2: Seven Basic Expression Annotation

The basic-expression corpus extends Aff-Wild2 with frame-level labels for seven expressions, retaining annotations only where at least five of seven experts agreed. It contains 539 videos and uses subject-independent partitions.

  • Corpus: The seven-expression corpus contains 539 videos, 2,595,572 frames, and 431 subjects.The annotated subjects include 265 males and 166 females; videos with two subjects were fully annotated.
  • Annotation procedure: Seven experts annotated each video frame as one of seven basic expressions or none.The none label accommodates affective states outside the seven basic expressions.
  • Annotation procedure: Only annotations supported by at least five of seven experts were retained because annotators disagreed on some difficult images.The retained labels are summarized in Table I.
  • Distribution: Table I reports the number of annotated images assigned to each of the seven basic expressions.
  • Data partitioning: Subject-independent training, validation, and test partitions contain 253, 71, and 223 videos, respectively.Individuals are restricted to one partition.

C. Aff-Wild2: Eight Action Unit Annotation

The action-unit corpus provides frame-level annotations for eight action units on a smaller Aff-Wild2 subset. Labels were retained only when all three experts agreed, with subject-independent partitions and aligned face preprocessing.

  • Corpus: The action-unit subset contains 56 videos, 63 subjects, and 398,835 frames annotated for eight action units.The subjects comprise 32 males and 31 females, and seven videos contain two annotated subjects.
  • Annotation procedure: Three experts annotated the occurrence of each action unit on a frame-by-frame basis.All subjects present in each video were annotated.
  • Annotation procedure: Only action-unit annotations on which all three experts agreed were retained.The paper notes that annotator agreement was not always 100%.
  • Data partitioning: Subject-independent training, validation, and test partitions contain 37, 7, and 12 videos, respectively.The partitioning prevents a person from appearing in multiple subsets.
  • Preprocessing: Preprocessing detects face bounding boxes, crops images, extracts five landmarks, and applies a similarity transformation for face alignment.The resulting cropped and aligned images were provided to participating teams.

III. EVALUATION METRICS PER CHALLENGE

The Competition evaluates valence-arousal estimation using the Concordance Correlation Coefficient (CCC), which measures agreement between annotation and prediction time series. The mean CCC for valence and arousal is the main criterion.

  • CCC evaluates agreement between annotation and prediction time series while penalizing correlated predictions that are shifted in value.It scales correlation by mean square difference, penalizing deviations from the annotations.
  • CCC ranges from −1 to 1, with +1 indicating perfect concordance and −1 indicating perfect discordance.
  • The mean CCC for valence and arousal estimation is the main evaluation criterion.

B. Seven Basic Expression Classification Challenge

Seven basic expression classification is evaluated with frame-level emotion predictions using F1 score and total accuracy. Their weighted average is the main evaluation criterion.

  • The F1 score combines recall and precision to evaluate per-frame emotion classification.Recall measures finding positive samples, while precision measures avoiding negative samples labeled as positive.
  • Total accuracy is the fraction of all test-sample predictions that the model classified correctly.It is defined over all test samples.
  • The main evaluation criterion is a weighted average of F1 score and total accuracy.

C. Eight Action Unit Detection Challenge

Eight action unit detection is evaluated by combining per-AU F1 scores with total accuracy. The overall score averages the resulting mean AU F1 and total accuracy.

  • F1 is computed independently for each action unit using per-frame presence-or-absence detections.
  • AF1 is the unweighted average of the F1 scores across all 8 action units.
  • The main evaluation criterion averages AF1 and total accuracy.The two components receive equal weight in the overall score.

IV. BASELINE & PARTICIPATING TEAMS’ SYSTEMS AND RESULTS

The baseline systems use reproducible open-source toolkits and standardized cropped, aligned 96×96×3 face images. PatchGAN supports valence-arousal regression, while MobileNetV2 supports expression classification and action unit detection.

  • All baseline systems rely exclusively on existing open-source machine-learning toolkits to support reproducibility.The section presents baseline systems, top-three team algorithms, and their results for each Challenge.
  • Aff-Wild2 faces are cropped, aligned, resized to 96×96×3, and normalized to pixel intensities in [-1,1].No on-the-fly or off-the-fly data augmentation was used.
  • PatchGAN was adapted for valence-arousal regression as a deep convolutional baseline trained from scratch.Its feature-map output is averaged into a single score, and training used a learning rate of 10^-4 for around two days on a Titan X GPU.
  • MobileNetV2 provided the baseline architectures for seven-expression classification and eight-action-unit detection.The lightweight architecture uses depth-wise separable convolutions to reduce complexity, cost, and model size.
  • The MobileNetV2 baselines use batch normalization, Relu6, no final average pooling, and task-specific softmax or sigmoid outputs.The final fully connected layer has 7 units for expressions or 8 units for action units; training was from scratch for around three days.

C. Top-3 Performing Teams per Challenge and their Methodologies

The Competition drew 62 registered teams, with leading approaches spanning multi-task learning, multimodal fusion, attention-based expression recognition, and class balancing across its three Challenges.

  • 62 teams registered, although many could not submit test results before the final deadline because of the coronavirus.
  • NISL2020 ranked first, third, and first in valence-arousal estimation, expression classification, and action unit detection, respectively, using multi-task learning with partial labels.
  • TNT ranked second, first, and second across the three Challenges using multi-task learning with separate image and audio streams.
  • ICT-VIPL-VA ranked third in valence-arousal estimation by fusing visual video features with acoustic audio features.
  • ICT-VIPL-Expression ranked second in expression classification with a Deep Residual Network, attention module, and bidirectional long-short-term memory units.
  • SALT ranked third in action unit detection after applying multi-label class balancing before training a ResNet on augmented data.

D. Results

Across all three Challenges, participating teams outperformed their respective baselines, while different teams led different task-specific metrics.

  • All participating teams outperformed the PatchGAN baseline by a large margin in valence-arousal estimation.NISL2020 achieved the best overall score and arousal CCC, while TNT achieved the best valence CCC.
  • All participating teams outperformed the MobileNetV2 baseline in basic expression classification.TNT led the other teams by a large margin in F1 Score, Total accuracy, and the total evaluation metric.
  • NISL2020 achieved the best overall action unit detection performance, with only a very small difference from TNT.NISL2020 led average F1 Score, whereas TNT led Total Accuracy.

V. CONCLUSION

ABAW 2020 organized valence-arousal estimation, seven basic expression classification, and eight action unit detection as three Challenges using Aff-Wild2.

  • The Competition comprised three Challenges targeting valence-arousal estimation, seven basic expression classification, and eight action unit detection.
  • Aff-Wild2 was the large-scale database used for the Competition and the first database annotated for all three behavior tasks.
  • The paper presented baseline networks and results, plus the top-three team methodologies and results for each Challenge.
Loading 2001.11409v2…