Source-linked AI summary
Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment
Zhiwen Shao, Zhilei Liu, Jianfei Cai, Lizhuang Ma
TL;DR
AU detection and face alignment are correlated tasks, but existing approaches often separate them or use alignment only as preprocessing. JAA-Net jointly learns both tasks end to end with shared multi-scale features, alignment-guided adaptive attention, and integrated local and global features, and it significantly outperforms state-of-the-art methods on benchmark datasets.
Problem
AU detection and face alignment are closely related, but prior approaches commonly treat alignment independently or as preprocessing, and joint deep modeling remains rare.
Method
JAA-Net jointly learns multi-scale shared features, feeds high-level alignment features into AU detection, and adaptively refines landmark-initialized AU attention maps.
Results
The framework significantly outperforms state-of-the-art methods on BP4D and DISFA, particularly for AU detection.
Takeaways & Limitations
Joint learning contributes to both AU detection and face alignment, while adaptive attention localizes AU regions for improved local-feature extraction.
Takeaways & Limitations
Prior landmark-aided methods rely on existing landmark detectors and treat face alignment as an independent task; this paper's reported scope is BP4D and DISFA.
Abstract
from arXiv · showhide
Facial action unit (AU) detection and face alignment are two highly correlated tasks since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. Most existing AU detection works often treat face alignment as a preprocessing and handle the two tasks independently. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared features are learned firstly, and high-level features of face alignment are fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment features and global features for AU detection. Experiments on BP4D and DISFA benchmarks demonstrate that our framework significantly outperforms the state-of-the-art methods for AU detection.
1 Introduction
AU detection and face alignment are closely related, but prior work rarely models them jointly. JAA-Net addresses this gap with end-to-end shared learning, adaptive AU attention, and joint optimization, outperforming state-of-the-art methods on benchmark datasets.
- Motivation: Facial landmarks localize distinctive facial locations and can provide more precise AU locations for extracting features.AUs are defined as basic facial muscle actions at specific locations under FACS.
- Motivation: Jointly modeling AU detection and face alignment is motivated by their coherent relationship, yet such studies are rare.
- Prior work: Prior landmark-aided methods use alignment mainly as preprocessing, with fixed AU regions or handcrafted features limiting joint deep learning.EAC-Net uses fixed-size, fixed-attention ROIs, while an earlier cascade method uses handcrafted features.
- Proposed framework: JAA-Net learns multi-scale shared features, feeds high-level alignment features into AU detection, and adaptively refines each AU's attention map.Attention maps are initially specified by predicted landmarks, and local, alignment, and global features are integrated for AU detection.
- Results: The framework is end-to-end, jointly optimizes all modules, and significantly outperforms state-of-the-art methods on two benchmark datasets.The reported experiments particularly emphasize gains in AU detection.
2 Related Work
Earlier work used landmarks or shared early features to support related face-analysis tasks, but generally did not deeply couple AU detection with face alignment. JAA-Net extends this line with end-to-end joint learning and adaptive attention based on predicted landmarks.
- Landmark-aided AU detection: Existing AU methods use facial landmarks, geometry, patches, or local texture to define or extract AU-related features.Examples include normalized landmark distances, Delaunay-mask angles, landmark-centered patches, and landmark-guided ROI cropping.
- Landmark-aided AU detection: These landmark-aided approaches typically treat face alignment independently and rely on existing landmark detectors.
- Face alignment with multi-task learning: Multi-task face-alignment methods such as TCDCN and HyperFace share early layers with other tasks but do not deeply integrate alignment representations.
- Proposed joint framework: JAA-Net feeds high-level face-alignment representations into AU detection and uses estimated landmarks to initialize adaptive attention learning.
- Joint AU detection and face alignment: Compared with handcrafted joint methods, JAA-Net uses an end-to-end deep framework and adaptively explores AU feature distributions across landmark-specified ROIs.
3 JAA-Net for Facial AU Detection and Face Alignment
JAA-Net jointly learns AU detection and face alignment through shared multi-scale features, alignment-guided adaptive attention, and integrated local and global representations.
- Shared feature learning: JAA-Net uses hierarchical and multi-scale region learning as a shared foundation for the face alignment, global feature, and adaptive attention modules.The region-learning module combines Rhm(l, l, c) and Rhm(l/2, l/2, 2c), followed by max-pooling.
- Face alignment: The face alignment module predicts facial landmark locations, which initialize AU attention maps for subsequent local feature learning.Its convolutional features feed a landmark prediction network, while estimated landmarks define AU centers and predefined regions.
- Joint optimization: The overall objective combines AU detection, face alignment, and attention-refinement consistency losses, while AU detection uses weighted softmax and Dice losses.The weighted losses address AU occurrence imbalance and align training with the F1-score evaluation metric.
- Shared feature learning: The hierarchical and multi-scale region layer uses different weight-sharing patch sizes to capture local AU features at multiple spatial scales.Its second, third, and fourth convolutional layers operate on 8×8, 4×4, and 2×2 patches, whose outputs are concatenated.
- Adaptive attention learning: Adaptive attention learning refines each AU’s whole-face attention map from landmark-initialized regions before extracting local AU features.The attention module has separate AU attention-refinement and local AU feature-learning steps.
- Adaptive attention learning: The initialization assigns attention weights that decay with Manhattan distance from AU centers, while overlapping subregions retain the maximum associated weight.The maximization operation keeps weights within [0, 1], and ξ controls the distance-based decay.
- Adaptive attention learning: Padding removal rescales and crops initial attention maps and pool2 to reduce padding effects during local AU feature learning.The process applies bilinear-interpolation scaling followed by center cropping, producing the feature map named “new pool2.”
4 Experiments
Experiments evaluate JAA-Net for AU detection and face alignment on BP4D and DISFA, using standard benchmark metrics and comparisons with prior methods. Ablations examine multi-scale learning, loss design, joint-task learning, adaptive attention, and AU-loss weighting.
- Datasets and metrics: JAA-Net is evaluated on BP4D and DISFA, which provide AU labels and facial landmark annotations.BP4D evaluates 12 AUs, while DISFA evaluates 8 AUs under subject-exclusive 3-fold cross-validation.
- Datasets and metrics: The experiments report F1-frame and accuracy for AU detection, plus normalized mean error and failure rate for face alignment.F1-frame and accuracy are averaged over AUs; face-alignment failure is defined by mean error above 10%.
- Comparison with state-of-the-art methods: 6.38% and 7.33% relative average F1-frame increments over ROI and EAC-Net, respectively, are obtained by JAA-Net on BP4D.JAA-Net also achieves high accuracy without sacrificing F1-frame, which the authors attribute to combining softmax and Dice coefficient losses.
- Comparison with state-of-the-art methods: 15.46% and 15.01% relative increases in average F1-frame and accuracy over EAC-Net, respectively, are obtained by JAA-Net on DISFA.AU-specific loss weighting is reported to improve balance and detection precision under DISFA’s severe data imbalance.
- Ablation study: Hierarchical multi-scale region learning improves AU detection over the region layer while using fewer parameters.The authors associate the improvement with adapting to multi-scale AUs and obtaining larger receptive fields.
- Ablation study: Ablations report gains from Dice loss, AU-loss weighting, face alignment, adaptive attention refinement, and back-propagation enhancement.The full JAA-Net performs best among adaptive-attention variants, while joint learning also improves face-alignment mean error and failure rate.
5 Conclusions
The paper develops an end-to-end framework that jointly learns AU detection and face alignment, using shared features and landmark-initialized adaptive attention. Experiments demonstrate effectiveness for both tasks, with potential application to broader face-analysis and multitask problems.
- The proposed framework jointly learns AU detection and face alignment in an end-to-end deep learning model.
- Shared features and face-alignment results initialize AU attention maps, supporting interaction between the two tasks.
- The adaptive attention learning module localizes AU regions adaptively to extract better local features.
- Extensive experiments demonstrate the method’s effectiveness for both AU detection and face alignment.
- The framework is promising for other face-analysis tasks and multitask problems.