J$\hat{\text{A}}$A-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention

Zhiwen Shao; Zhilei Liu; Jianfei Cai; Lizhuang Ma

Source-linked AI summary

J$\hat{\text{A}}$A-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention

Zhiwen Shao, Zhilei Liu, Jianfei Cai, Lizhuang Ma

arXiv:2003.08834v3Mar 18, 2020cs.CV

TL;DR

Existing AU detection systems generally separate face alignment from detection and often impose fixed AU regions or attention, despite their strong correlation. JÂA-Net jointly learns alignment and detection with adaptive AU attention, and reports stronger AU detection across four benchmarks, competitive alignment, and robustness to occlusion and pose variation.

Problem
Existing methods usually treat face alignment as preprocessing and use fixed AU regions or attention, while joint deep learning of AU detection and alignment has been little explored.
Method
JÂA-Net jointly learns multi-scale features, feeds face-alignment features into AU detection, and adaptively refines landmark-initialized attention maps end to end.
Results
JÂA-Net significantly outperforms state-of-the-art AU detection methods on BP4D, DISFA, GFT, and BP4D+, while achieving competitive face alignment and working under occlusion and non-frontal poses.
Takeaways & Limitations
Adaptive attention captures irregular AU regions, and the reported ablations indicate that each framework component benefits AU detection.
Takeaways & Limitations
Prior landmark-aided methods rely on existing landmark detectors and treat face alignment as an independent task; the paper’s framework addresses this limitation through joint learning.

Abstract

from arXiv · show

Facial action unit (AU) detection and face alignment are two highly correlated tasks, since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. However, most existing AU detection works handle the two tasks independently by treating face alignment as a preprocessing, and often use landmarks to predefine a fixed region or attention for each AU. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared feature is learned firstly, and high-level feature of face alignment is fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment feature and global feature for AU detection. Extensive experiments demonstrate that our framework (i) significantly outperforms the state-of-the-art AU detection methods on the challenging BP4D, DISFA, GFT and BP4D+ benchmarks, (ii) can adaptively capture the irregular region of each AU, (iii) achieves competitive performance for face alignment, and (iv) also works well under partial occlusions and non-frontal poses. The code for our method is available at https://github.com/ZhiwenShao/PyTorch-JAANet.

1 Introduction

AU detection and face alignment are closely related, yet prior work largely treats alignment as preprocessing and uses fixed AU regions or attention. JÂA-Net jointly learns both tasks end to end and adaptively refines AU attention, achieving broad empirical gains and robustness.

Prior AU methods typically use landmarks for alignment or fixed regions, while joint deep modeling of AU detection and face alignment remains rare.
JÂA-Net learns shared multi-scale features, feeds face-alignment features into AU detection, and optimizes both tasks jointly without post-processing.
Its adaptive attention module refines each AU’s landmark-initialized attention map through an independent branch supervised by a local AU detection loss.
The framework significantly outperforms state-of-the-art AU detection methods on BP4D, DISFA, GFT, and BP4D+ benchmarks.
Experiments show competitive face alignment and good performance under partial occlusions and non-frontal poses.
Compared with the earlier conference version, the new local AU detection loss and removal of the attention-difference constraint make attention learning more general and improve AU detection.

2 Related Work

Prior AU detection methods use facial landmarks to extract local features, but generally treat face alignment independently. Joint approaches exist, yet earlier methods rely on handcrafted features, motivating end-to-end deep joint learning with adaptive attention.

Most AU recognition methods use face detection and alignment as preprocessing before AU detection.
Landmark-aided methods define AU regions around landmarks, often using fixed ROI sizes and attention distributions.
Existing landmark-based approaches generally treat face alignment as an independent task and use existing landmark detectors.
Face alignment and AU detection are correlated, but prior interactions commonly use landmarks one-way to extract AU features.
Earlier joint work combined landmark tracking and AU recognition, while another cascade framework jointly modeled facial shape, AU relationships, and landmark dependencies.
The proposed approach uses an end-to-end deep framework and adaptive attention to explore AU feature distributions in landmark-specified regions.

3 J ˆAA-Net for Facial AU Detection and Face Alignment

JÂA-Net jointly learns facial AU detection and face alignment through shared multi-scale features, landmark-guided adaptive attention, and integrated local and global representations. Its end-to-end design refines AU-specific attention while using alignment features for AU prediction.

Hierarchical and Multi-Scale Region Learning: JÂA-Net extracts multi-scale local-region features through hierarchical region learning before passing them to the alignment, global-feature, and adaptive-attention modules.The module uses two hierarchical and multi-scale region blocks with max pooling, producing an l/4 × l/4 × 8c feature map.
Face Alignment and Global Feature Learning: The face alignment module estimates facial landmarks and produces features containing global facial shape and local landmark information.Its output is processed by fully connected layers for landmark prediction and also supplied to AU detection.
Facial AU Detection: The final AU detector integrates local AU features with face-alignment and global features in an end-to-end jointly trained framework.The overall training combines AU losses with alignment loss, while joint training allows the two tasks to contribute to each other.
Adaptive Attention Learning: The adaptive attention module refines one landmark-initialized attention map per AU, then learns local AU features from the refined maps.It has separate attention-refinement and local-feature-learning steps, with one branch for each AU.
Adaptive Attention Learning: Predefined AU attention maps use two landmark-centered subregions whose weights decay with Manhattan distance from each AU center.The maps initially emphasize only the two subregions, and overlapping regions retain the maximum associated attention weight.
Adaptive Attention Learning: Zero padding is used in the attention-refinement convolutions to avoid harming edge-region refinement and to produce attention maps matching the shared feature resolution.The refined attention map is sized l/4 × l/4 × 1 to match the l/4 × l/4 × 8c multi-scale feature.

4.1 Datasets and Settings

The framework is evaluated on four AU datasets with landmark annotations, using standard preprocessing, optimization, and task-specific metrics. Comparisons cover BP4D, DISFA, GFT, and BP4D+ settings.

Datasets: Four datasets—BP4D, DISFA, GFT, and BP4D+—provide both AU and facial-landmark labels for evaluation.
Datasets: BP4D contains about 140,000 frames from 41 subjects, with 49 landmarks and 12 evaluated AUs under subject-exclusive 3-fold cross-validation.
Datasets: DISFA contains 27 videos with AU intensities on a six-point scale and 66 annotated landmarks, of which 49 are used.
Datasets: GFT includes 96 subjects and moderate out-of-plane poses, while BP4D+ provides 197,875 annotated frames for large-scale testing.
Implementation: Faces are similarity-aligned to 200×200×3, randomly cropped to 176×176, and horizontally flipped during training.
Evaluation Metrics: AU detection is evaluated with F1-frame, accuracy, and their averages, while alignment uses normalized mean error and failure rate.

4.2 Comparison with State-of-the-Art Methods

JˆAA-Net is compared with established single-frame AU detectors across multiple benchmarks. It reports strong performance on DISFA and GFT, including improvements under severe imbalance and non-frontal poses.

Benchmark Comparisons: JˆAA-Net is compared against state-of-the-art single-frame AU detectors under the same evaluation setting.
Loss Design: The integrated cross-entropy and Dice losses yield high accuracy without sacrificing F1-frame performance.
DISFA: On DISFA, JˆAA-Net increases average F1-frame by 4.8 and average accuracy by 0.7 over ARL.
BP4D+: BP4D+ evaluation is reported for 12 AUs using a large-scale testing protocol, with Table 5 comparing methods on that benchmark.
GFT: On GFT, JˆAA-Net achieves average F1-frame 53.7 and average accuracy 90.5, outperforming several recent methods.

4.3 Ablation Study

Ablations show that hierarchical multi-scale regions, imbalance-aware losses, joint alignment features, global features, and adaptive attention each contribute to AU detection. The newer local AU loss avoids restrictions associated with earlier refinement strategies.

Each Component in JˆAA-Net: Hierarchical and multi-scale region learning improves AU detection by accommodating different AU sizes and enlarging receptive fields.
Each Component in JˆAA-Net: Dice loss improves F1-frame and accuracy by emphasizing precision and recall, while reducing non-occurrence bias for rare AUs.
Each Component in JˆAA-Net: Weighting each AU loss raises average F1-frame and accuracy to 57.4 and 76.4, respectively, over the unweighted variant.
Feature Integration: Integrating face-alignment, global, and assembled local features progressively improves AU detection performance.
Adaptive Attention Learning: Fixed predefined attention maps limit adaptation to varying AU sizes, whereas adaptive refinement captures more accurate irregular AU regions.
Adaptive Attention Learning: The refinement constraint worsens results by limiting the solution space, while the local AU loss avoids the enhancement coefficient λe and irrelevant-AU interference.

4.4 J ˆAA-Net for Face Alignment

The joint model also performs strongly on face alignment, and AU detection improves landmark localization. This supports mutual benefit between the two tasks within the joint framework.

Face Alignment Results: JˆAA-Net significantly outperforms ERT, TCDCN, and MCL on both mean error and failure rate, while achieving the lowest mean error among compared methods.
Qualitative Results: Example BP4D results visually compare predicted landmarks from the joint model with the AU-removed baseline and ground-truth locations.
Occlusion Evaluation: Occlusion examples assess the model with the whole face visible as the Full condition and with different partial occlusions.
Mutual Benefits: Adding AU detection reduces mean error by 2.98 and failure rate by 5.90 compared with JˆAA-Net w/o AU.

4.5 J ˆAA-Net for Faces with Partial Occlusions

JˆAA-Net remains competitive and more balanced than comparison methods when BP4D faces are partially occluded. Its robustness is attributed partly to joint alignment and AU detection, with correlated AUs supplying information from occluded regions.

Partial occlusions: JˆAA-Net achieves competitive performance against EAC-Net and ARL on BP4D faces with half-face occlusions.Testing uses lower, upper, right, and left half-face visibility settings.
Partial occlusions: Its average F1-frame results across four half-face occlusions are more balanced than those of EAC-Net and ARL.
Partial occlusions: Joint AU detection and face alignment, with landmarks initializing attention maps, partially supports robustness under occlusion.
Partial occlusions: Predictions can use information from correlated AUs in other facial parts that may be occluded.

4.6 J ˆAA-Net for Faces with Non-Frontal Poses

On FERA 2017's nine-pose benchmark, JˆAA-Net performs strongly across non-frontal head poses. It outperforms other methods overall and uses only a single input frame, unlike the temporal CNN+BLSTM-RNN comparator.

Non-frontal poses: JˆAA-Net outperforms all other methods on overall F1-frame results across FERA 2017's nine poses.FERA 2017 varies pitch at −40, −20, and 0 degrees and yaw at −40, 0, and 40 degrees.
Non-frontal poses: Joint learning with face alignment contributes to AU detection under non-frontal poses because facial shape from landmarks relates to head pose.
Non-frontal poses: JˆAA-Net achieves better performance than the FERA 2017 CNN+BLSTM-RNN state-of-the-art method using only a single input frame.
Non-frontal poses: JˆAA-Net has the best average F1-frame performance for most individual poses: 2, 3, 4, 6, and 9.

5 Conclusion

The paper presents an end-to-end joint framework in which AU detection and face alignment share information and adaptive attention localizes irregular AU regions. Across multiple benchmarks and robustness tests, it reports strong AU detection, competitive alignment, and benefits from both task components.

Conclusion: The end-to-end framework jointly learns facial AU detection and face alignment without post-processing.
Conclusion: Adaptive attention learning localizes irregular AU regions to extract more precise local features.
Conclusion: The approach significantly outperforms state-of-the-art AU detection methods on BP4D, DISFA, GFT, and BP4D+ benchmarks.
Conclusion: Ablation results indicate that each framework component benefits AU detection and that local AU detection loss effectively supervises adaptive attention learning.
Conclusion: AU detection contributes to face alignment, while the joint framework achieves competitive face alignment performance and shows robustness to partial occlusions and non-frontal poses.