PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition

Guanbin Li, Xin Zhu, Yirui Zeng, Qing Wang, Liang Lin

arXiv:1904.09939v1cs.CV

TL;DR

AU recognition needs to account for dependencies among action units while handling difficult visual conditions. SRERL embeds a structured AU knowledge graph into multiscale CNN feature learning through GGNN propagation, and experiments on BP4D and DISFA report superiority over prior methods.

  • Problem

    Existing AU recognition methods struggle to jointly model complex regional appearance representations and semantic relationships among interdependent AUs.

  • Method

    SRERL constructs an AU knowledge graph and integrates GGNN node propagation into a multiscale CNN framework for end-to-end representation learning.

  • Results

    SRERL outperforms state-of-the-art methods on BP4D and DISFA, including 4% and 2.3% higher F1-score than DSIN, respectively.

  • Takeaways & Limitations

    The learned representation combines facial appearance with AU relationship reasoning and is reported to be more robust to illumination change and partial occlusion.

Abstract

from arXiv · show

Facial action unit (AU) recognition is a crucial task for facial expressions analysis and has attracted extensive attention in the field of artificial intelligence and computer vision. Existing works have either focused on designing or learning complex regional feature representations, or delved into various types of AU relationship modeling. Albeit with varying degrees of progress, it is still arduous for existing methods to handle complex situations. In this paper, we investigate how to integrate the semantic relationship propagation between AUs in a deep neural network framework to enhance the feature representation of facial regions, and propose an AU semantic relationship embedded representation learning (SRERL) framework. Specifically, by analyzing the symbiosis and mutual exclusion of AUs in various facial expressions, we organize the facial AUs in the form of structured knowledge-graph and integrate a Gated Graph Neural Network (GGNN) in a multi-scale CNN framework to propagate node information through the graph for generating enhanced AU representation. As the learned feature involves both the appearance characteristics and the AU relationship reasoning, the proposed model is more robust and can cope with more challenging cases, e.g., illumination change and partial occlusion. Extensive experiments on the two public benchmarks demonstrate that our method outperforms the previous work and achieves state of the art performance.

Introduction

AU recognition matters for facial-expression analysis, but treating AUs independently misses their co-occurrence and anatomical incompatibilities. SRERL addresses this by combining multiscale facial-region features with semantic AU-relationship propagation.

  • Facial action units describe local facial muscle actions and support comprehensive facial-expression analysis.
  • Independent AU classification overlooks expression-linked co-occurrence and anatomical constraints between action units.Happiness commonly activates AU6 and AU12, whereas stretching the mouth and raising the cheek are generally incompatible.
  • SRERL organizes AUs as a structured knowledge graph and integrates GGNN propagation into a multiscale CNN framework.The framework learns regional features together with structured AU relationships in an end-to-end manner.
  • The resulting representation combines facial appearance characteristics with AU relationship reasoning for challenging cases such as illumination change and partial occlusion.

Related Work

AU recognition research has progressed through discriminative feature learning and relationship modeling, but earlier approaches often separated relationship inference from end-to-end feature learning. The related work motivates integrated representations that capture AU dependencies within deep models.

  • AU recognition methods broadly rely on robust feature representations or semantic relationship modeling.
  • Traditional approaches use handcrafted appearance, geometry, or temporal features with conventional classifiers for AU detection.
  • Early relationship models include dynamic Bayesian networks and restricted Boltzmann machines for modeling dependencies among AUs.
  • SRERL is presented as a pipeline combining multiscale feature learning and cropping with GGNN node-information propagation.
  • Some later methods jointly capture feature- and label-level dependencies, but handcrafted features prevent fully end-to-end optimization.

Method

The method section introduces SRERL as a framework that constructs an AU relationship graph and uses it to guide feature learning through a multiscale CNN and GGNN pipeline.

  • SRERL first reviews GGNNs and constructs an AU relationship graph for guided propagation.
  • The framework then presents its complete SRERL structure, whose overall pipeline is illustrated in Figure 2.

Preliminary: Gated Graph Neural Network (Li et al. 2015)

GGNNs propagate information over graph nodes through recurrent updates, using adjacency structure and learned gates to produce final node-level outputs.

  • A GGNN receives a graph with N nodes and initial node representations, then learns propagation similar to an LSTM.
  • Each node starts with its initial feature, while adjacency matrix A encodes the graph structure.
  • At each recurrent step, a node updates its hidden state using its current state and adjacent-node information.
  • After T time steps, the final hidden states are used to compute node-level outputs through a fully connected network.

Preliminary: AU Relationship Graph Construction

The AU relationship graph represents action units as nodes connected by positive or negative correlations derived from training data, facial-expression knowledge, and facial anatomy. These relationships define structured information for AU reasoning.

  • Graph structure: Each graph node represents a specific AU, while edges encode correlations between AUs.
  • Relationship types: Positive correlations indicate AUs likely to cooccur, whereas negative correlations indicate AUs that rarely appear together.
  • Relationship estimation: The method estimates AU relationships from conditional probabilities computed on training samples.
  • Relationship estimation: The adjacency matrix combines positive and negative relationship matrices, including directed relationships and self-connection handling.
  • Prior knowledge: Additional BP4D relationships are added from prior AU research to cover common connections absent from the dataset.

Relationship-Embedded Representation Learning

SRERL combines multiscale CNN feature learning and facial-region cropping with GGNN propagation over the AU knowledge graph. The resulting node representations incorporate localized appearance information and propagated AU relationships.

  • Framework: SRERL uses a multiscale CNN feature-learning and cropping module followed by GGNN-based node information propagation.
  • Multiscale feature learning: VGG19 extracts multiscale appearance features by concatenating outputs from four convolutional groups after resizing them to 14 × 14.
  • Regional representation: AU regions are cropped using facial landmarks and processed through separate regional learning streams.
  • Regional representation: Because of facial symmetry, each AU produces two patch-wise feature maps and receives two independent regional learning branches.

Balanced Loss Function

The paper addresses AU class imbalance with an adaptive loss function within its multi-label training setting. The training formulation accounts for positive and negative sample proportions and operates alongside regional feature and GGNN learning.

  • Motivation: AU recognition faces data imbalance in multi-label training, making conventional undersampling or oversampling difficult under ensemble classification.
  • Adaptive loss: The method introduces an adaptive loss function for imbalanced data training.
  • Loss formulation: The loss uses ground-truth labels, predicted probabilities, AU count C, and batch size N in its formulation.
  • Loss formulation: Positive and negative sampling proportions are represented by r_i^pos and r_i^neg, while M denotes the number of training samples.
  • Integration with recognition: The regional features initialized for GGNN nodes are propagated through graph updates before final label prediction.

Experimental Results

The model is evaluated on BP4D and DISFA using cross-validation and F1/AUC metrics. The reported setup specifies dataset construction, label handling, optimization, and prediction thresholds, but these passages do not provide numerical results.

  • Datasets: Experiments use the spontaneous BP4D and DISFA datasets, which contain 146,847 and 130,815 labeled face images or frames, respectively.BP4D includes 41 subjects and DISFA includes 27 subjects.
  • Evaluation protocol: Both datasets are split into three folds, with alternating training and testing folds used to report average results.
  • Metrics: The evaluation uses F1 score and area under the ROC curve (AUC).
  • Training setup: Training uses Adam with a learning rate of 0.0001, mini-batches of 64, early stopping, and a 0.5 prediction threshold for F1-score.

Ablation Studies

Ablation studies show that balanced loss, multiscale regional features, and AU relationship modeling each improve recognition, with relationship embedding providing additional gains on BP4D and DISFA. The experiments also indicate that balanced loss particularly benefits less frequent AUs.

  • Multiscale CNN: SS RL outperforms VGG BL by 2.4% in average F1-score and 1.7% in AUC on BP4D.The comparison attributes the improvement to independent regional feature representations that reduce interference between AU regions.
  • Multiscale CNN: 0.50% F1-score and 0.40% AUC gains show that MS RL improves over SS RL on BP4D.This supports the effectiveness of multiscale feature learning.
  • Relationship-Embedded Feature Enhancement: 0.90% F1-score and 0.50% AUC gains on BP4D, plus 2.90% F1-score and 0.50% AUC gains on DISFA, show the benefit of SRERL over MS RL.The comparison directly evaluates AU relationship-embedded feature enhancement.
  • Balanced Loss: AU23 occurs in 17% of BP4D samples and has F1-score and AUC values of 29.4% and 58.5%, respectively.This contrasts with the higher occurrence and performance reported for AU12.
  • Balanced Loss: 8.7% and 9.5% F1-score gains for AU2 and AU15 show that VGG BL particularly improves lower-occurrence AUs over original VGG.The corresponding AUC gains are 5.6% and 8.5%, while average F1-score and AUC improve by about 4% and 3%.

Comparison with the State of the Art

SRERL outperforms the compared methods on BP4D and DISFA, including both feature-learning baselines and methods that model AU relationships.

  • 27.6% and 41.9%: SRERL exceeds LSVM on BP4D F1-score and AUC, respectively; on DISFA, the gains are 34.1% and 53.2%.
  • 4.0% and 18.1%: SRERL outperforms the second-best BP4D method in F1-score and AUC, respectively.
  • 2.3% and 28.4%: SRERL outperforms the second-best DISFA method in F1-score and AUC, respectively.
  • SRERL improves F1-score over DRML and ROI by 14.6% and 6.5% on BP4D, and by 29.2% and 7.4% on DISFA.
  • SRERL exceeds DSIN by 4% on BP4D and 2.3% on DISFA in F1-score.

Conclusion

The paper concludes that SRERL combines multiscale regional feature learning with AU knowledge-graph propagation and outperforms state-of-the-art methods on BP4D and DISFA.

  • SRERL combines multiscale feature learning and cropping with GGNN-based node propagation across an AU knowledge-graph.
  • Experiments on BP4D and DISFA demonstrate SRERL’s superiority over state-of-the-art methods.
Loading 1904.09939v1…