Source-linked AI summary
Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition
Cheng Luo, Siyang Song, Weicheng Xie, Linlin Shen, Hatice Gunes
TL;DR
AU recognition requires modeling relationships that vary across AU pairs and facial displays, while prior methods use limited or fixed relationship representations. The paper learns a unique AU relation graph from the full-face representation, combining relationship-aware node features with multi-dimensional edge features. On BP4D and DISFA, the approach improves CNN and transformer backbones and achieves state-of-the-art AU recognition results.
Problem
Existing approaches do not explicitly represent complex, pair-specific AU relationship cues for each facial display, and fixed graph topologies ignore display-specific influences.
Method
The approach learns a unique AU relation graph from the full-face representation, encoding AU activation and associations in node features and multiple task-specific cues in edge features.
Results
Both CNN and transformer-based backbones are largely enhanced, achieving state-of-the-art AU recognition results on BP4D and DISFA.
Takeaways & Limitations
The relationship-modelling approach can be incorporated into standard CNN and transformer backbones for AU recognition and related multi-label tasks.
Takeaways & Limitations
The approach assumes that AU relationship cues are unique for each facial display, so training-set relationship cues may not generalize well at inference.
Abstract
from arXiv · showhide
The activations of Facial Action Units (AUs) mutually influence one another. While the relationship between a pair of AUs can be complex and unique, existing approaches fail to specifically and explicitly represent such cues for each pair of AUs in each facial display. This paper proposes an AU relationship modelling approach that deep learns a unique graph to explicitly describe the relationship between each pair of AUs of the target facial display. Our approach first encodes each AU's activation status and its association with other AUs into a node feature. Then, it learns a pair of multi-dimensional edge features to describe multiple task-specific relationship cues between each pair of AUs. During both node and edge feature learning, our approach also considers the influence of the unique facial display on AUs' relationship by taking the full face representation as an input. Experimental results on BP4D and DISFA datasets show that both node and edge feature learning modules provide large performance improvements for CNN and transformer-based backbones, with our best systems achieving the state-of-the-art AU recognition results. Our approach not only has a strong capability in modelling relationship cues for AU recognition but also can be easily incorporated into various backbones. Our PyTorch code is made available.
1 Introduction
Facial Action Units provide an objective, multi-label representation of facial expressions, but modeling their mutually influential relationships remains an open challenge. Existing approaches inadequately capture pair-specific, multi-dimensional, and facial-display-dependent AU relationships, motivating a learned relation graph.
- Facial Action Units represent facial muscle movements and support a more comprehensive, objective description of expressions than emotion-based categories.
- AU recognition is a multi-label classification problem because multiple AUs can be activated simultaneously.
- Existing models encode all AU descriptors into one representation, failing to individually model relationships between AU pairs.
- Graph-based approaches often represent each AU pair with a binary value or single weight, which may not capture complex underlying relationships.
- The proposed approach learns a facial-display-specific graph with node features encoding AU activation and associations, plus multi-dimensional edge features for pairwise relationship cues.
2 The Proposed Approach
The approach learns a facial-display-specific AU relation graph from full-face features, combining relationship-aware node features with multi-dimensional directed edge features before GatedGCN recognition.
- Graph construction: A unique AU relation graph is learned for each facial display, with AU-specific nodes and directed edges encoding recognition-related relationship cues.The graph is produced from the target face representation and contains N node features and N × N multi-dimensional directed edge features.
- Node feature learning: The AFG generates AU-specific representations, while the jointly trained FGG defines graph connectivity from feature similarity and updates AU activation representations.Each AU-specific extractor projects the full face representation into an AU-specific feature map and applies global average pooling; the FGG selects K nearest neighbours.
- Node feature learning: The FGG uses a trainable similarity-calculating strategy to predict AU occurrence probabilities from graph-updated node representations.A trainable vector for each AU is compared with its graph-updated representation using cosine similarity.
- Edge feature learning: The MEFL module learns a pair of multi-dimensional edge features for every AU pair, including connected and unconnected pairs, to capture task-specific relationship cues.This preserves relationship information even when node similarity is low, where useful recognition cues may otherwise be ignored.
- Edge feature learning: FAM independently locates AU-specific cues from the full face, while ARM extracts cues related jointly to both AUs and produces directional edge vectors.Cross-attention uses AU-specific feature maps as queries and the full face representation as key and value; ARM then processes the resulting AU-specific features in both directions.
- Recognition and training: The learned graph is processed by L gated graph convolution layers, and the final node features are used to predict all target AUs.The graph retains its topology through GatedGCN processing, while inference uses the trained AFG and MEFL to generate the AU relation graph.
- Recognition and training: Training jointly optimizes the relationship-learning modules with the backbone and classifier, and models AU-pair co-occurrence using four activation-state classes.The four classes represent both AUs inactive, only the second active, only the first active, or both active.
3 Experiments
Experiments evaluate the approach on BP4D and DISFA using standard AU-recognition protocols and frame-based F1 scores. Results show consistent gains from relationship modelling across backbones, datasets, and ablations.
- Experimental Setup: The approach is evaluated on BP4D and DISFA, with multiple-AU occurrence labels for each frame.BP4D contains about 140,000 frames from 41 adults, while DISFA contains 130,815 frames from 27 subjects.
- Experimental Setup: Subject-independent three-fold cross-validation and frame-based F1 scores are used for evaluation on both datasets.Faces are detected, aligned, and cropped to 224 × 224 before backbone processing.
- Comparison to State-of-the-art Methods: 0.5% and 1.3% average F1 improvements over the state-of-the-art are achieved on BP4D by the ResNet-50 and Swin-B systems, respectively.Both backbones also reach top-three performance for 9 of 12 AUs.
- Comparison to State-of-the-art Methods: 1.6% and 0.9% average F1 improvements over the state-of-the-art are achieved on DISFA by the two backbones, respectively.The approach achieves state-of-the-art average F1 scores over eight AUs.
- Comparison to State-of-the-art Methods: 1.8% and 7.2% average F1 improvements over SRERL are obtained on BP4D and DISFA, respectively.The system also exceeds UGN-B and HMP-PS by more than 1.3% and 2.1% average F1 on both datasets.
- Ablation Studies: 5.6% and 2.9% F1 improvements over the original backbones result from jointly using FGG, MEFL, and weighted asymmetric loss.These components also improve over the backbone-AFG systems by 1.7% and 0.9%.
- Ablation Studies: The weighted asymmetric loss outperforms weighted binary cross-entropy, while AU co-occurrence supervision slightly improves recognition for both backbones.The ablation results attribute additional recognition-related cues to node and multi-dimensional edge feature learning.
4 Conclusion
The approach learns facial-display-specific AU relation graphs with relationship cues encoded in node and multi-dimensional edge features. It enhances CNN and transformer backbones, achieving state-of-the-art results on two widely used datasets.
- The method deep learns graphs that explicitly represent pairwise AU relationship cues for each facial display.
- Node and multi-dimensional edge features encode task-specific relationship cues among AUs.
- Both CNN and transformer-based backbones are largely enhanced by the proposed node and edge feature learning methods.
- The approach achieves state-of-the-art AU recognition results on two widely used datasets.
- The graph-based approach can be incorporated into standard CNN and transformer backbones.
A Additional Experimental Results
The additional experimental-results section reports AUC results for AU recognition on BP4D and DISFA datasets.
- Table 4 reports AUC results for 12 AUs on the BP4D dataset.
- Table 5 reports AUC results for 8 AUs on the DISFA dataset.