Source-linked AI summary
Facial Action Unit Detection Using Attention and Relation Learning
Zhiwen Shao, Zhilei Liu, Jianfei Cai, Yunsheng Wu, Lizhuang Ma
TL;DR
Existing AU attention methods rely on predefined regions or restricted refinements, limiting how they model varied AUs and correlated regions. ARL learns attention and pixel-level relations from AU labels, and the paper reports stronger detection and intensity-estimation results across four benchmarks, including occlusion and large-pose settings.
Problem
Predefined attention or refinement near predefined regions limits modeling of varied AUs, while existing attention methods omit channel-wise attention and integration with pixel-level relations.
Source: §1 · p. 2Method
ARL learns multi-scale features, channel-wise and spatial attention, and pixel-level relations from AU labels in an end-to-end framework.
Source: §1 · p. 2Results
ARL soundly outperforms state-of-the-art methods for AU detection and intensity estimation on BP4D, DISFA, FERA 2015, and BP4D+, and works under occlusion and large poses.
Source: §1 · p. 2, §5 · p. 13Takeaways & Limitations
Learned attentions capture correlated AU regions, and the trained model can detect AUs with only half-faces visible.
Source: §5 · p. 13Learned AU correlations are statistical laws in the training set, although specific AU relationships depend on expression.
Source: §4.4.3 · p. 11
Abstract
from arXiv · showhide
Attention mechanism has recently attracted increasing attentions in the field of facial action unit (AU) detection. By finding the region of interest of each AU with the attention mechanism, AU-related local features can be captured. Most of the existing attention based AU detection works use prior knowledge to predefine fixed attentions or refine the predefined attentions within a small range, which limits their capacity to model various AUs. In this paper, we propose an end-to-end deep learning based attention and relation learning framework for AU detection with only AU labels, which has not been explored before. In particular, multi-scale features shared by each AU are learned firstly, and then both channel-wise and spatial attentions are adaptively learned to select and extract AU-related local features. Moreover, pixel-level relations for AUs are further captured to refine spatial attentions so as to extract more relevant local features. Without changing the network architecture, our framework can be easily extended for AU intensity estimation. Extensive experiments show that our framework (i) soundly outperforms the state-of-the-art methods for both AU detection and AU intensity estimation on the challenging BP4D, DISFA, FERA 2015 and BP4D+ benchmarks, (ii) can adaptively capture the correlated regions of each AU, and (iii) also works well under severe occlusions and large poses.
1 INTRODUCTION
Existing AU attention methods rely on predefined regions or restricted refinements, while AU-level relations do not fully connect relation learning with attention. ARL learns attention and pixel-level relations from AU labels, with reported gains across detection, intensity estimation, occlusion, and pose settings.
1 INTRODUCTION
- Existing attention methods rely on landmark-based predefined regions or refine attention only near those regions, missing correlated AU areas.The paper contrasts these constraints with ARL's learned attention; its example concerns AU 1 and AU 15 in a sad face.Source: §1 · pp. 1–2, Figure 1 · p. 1
- Prior AU-relation methods do not integrate relation learning with attention, and AU-level relations provide only rough locations for subtle facial changes.The paper motivates pixel dependencies to refine pixel-wise attention, including regions with tiny muscle actions.Source: §1 · p. 2
- ARL learns shared multi-scale features, channel-wise and spatial attention, and pixel-level relations from AU labels end-to-end, and extends to intensity estimation without architecture changes.The pixel-level relation module refines spatial attention to capture correlated regions and features.Source: §1 · p. 2
- ARL outperforms state-of-the-art methods for AU detection and intensity estimation, captures correlated regions, and works under severe occlusions and large poses.The introduction reports these outcomes across the paper's experiments.Source: §1 · p. 2
2 RELATED WORK
The related work covers attention-based AU detection and AU-relation modeling. The paper positions ARL against restricted attention learning and AU-level relation methods by combining attention with pixel-level relations.
2.1 Attention Based AU Detection
- Attention-based AU methods use landmark-predefined attentions or refine them within a small range, limiting adaptation to AUs with different sizes and non-rigid transformations.JAA-Net adaptively refines attention, but its refinements remain near the initially predefined regions.Source: §2.1 · p. 2
2.2 Relation Based AU Detection
- BN, DBN, and RBM generative models cannot be jointly optimized with deep networks end-to-end; ARL instead combines a CNN with a discriminative CRF.ARL models pixel-level relations for each AU and integrates them with attention.Source: §2.2 · p. 3, Figure 2 · p. 3
- Prior work models local, global, or combined AU dependencies, but AU-level relations give only rough correlated locations and are not integrated with attention.The paper motivates pixel dependencies to refine spatial attention for subtle facial changes.Source: §2.2 · pp. 2–3
3 ARL FOR FACIAL AU DETECTION
ARL predicts AU occurrence probabilities from shared multi-scale facial features, then learns AU-specific channel and spatial attention and refines spatial attention with pixel-level relations. Its modules are trained end-to-end, and the same architecture extends to intensity estimation.
3.1 Overview
- ARL predicts AU occurrence probabilities from an input face image using shared multi-scale features and AU-specific attention and relation-learning modules.The framework comprises hierarchical and multi-scale region learning, channel-wise and spatial attention learning, and pixel-level relation learning.Source: §3.1 · p. 3
3.2 Hierarchical and Multi-Scale Region Learning
- Hierarchical region layers extract multi-scale features from local patches, supporting subsequent attention and relation learning for each AU.The module concatenates intermediate-layer features and adds them element-wise to the input-layer output.Source: §3.2 · pp. 3–4
3.3 Channel-Wise and Spatial Attention Learning
- Channel-wise attention selects feature channels with AU-related attributes, while spatial attention learns where AU-related features occur.The channel-wise weights multiply corresponding feature-map channels.Source: §3.3 · p. 4
3.4 Pixel-Level Relation Learning
- A fully connected CRF models pixel dependencies to refine spatial attention, removing attention from irrelevant regions to extract more accurate AU-related features.Its pairwise terms model relationships between near and far regions; learned spatial pixel-wise features, rather than RGB values, are used in the relation model.Source: §3.4 · pp. 4–5
3.5 Overall Objective Function
- The network is trained end-to-end, with hierarchical multi-scale learning, channel-wise and spatial attention, and CRF-based pixel-level relation learning optimized simultaneously.For intensity estimation, iARL combines a weighted Euclidean loss with a cosine-similarity loss; this extension does not change the network architecture.Source: §3.5 · p. 5
4.1 Datasets and Settings
The experiments use four FACS-coded benchmarks, with different datasets and protocols for AU detection, intensity estimation, and cross-dataset testing. The section also specifies image preprocessing, training settings, and task-specific evaluation metrics.
4.1.1 Datasets
- Experiments use BP4D, DISFA, FERA 2015, and BP4D+ for frame-level AU detection and intensity estimation; video-level-only datasets are excluded.BP4D+ is used for cross-dataset testing after training on BP4D.Source: §4.1.1 · pp. 5–6
4.1.2 Implementation Details
- Faces are aligned to 200 × 200 × 3, cropped to 176 × 176 × 3, and randomly horizontally flipped before training.Training uses SGD for up to 12 epochs, with separate starting learning rates for detection and intensity estimation.Source: §4.1.2 · p. 6
4.1.3 Evaluation Metrics
- AU detection is evaluated with frame-based F1-score and accuracy, while intensity estimation uses ICC(3,1) and MAE.The reported metrics are also averaged across AUs.Source: §4.1.3 · pp. 6–7
4.2 Comparison with State-of-the-Art Methods
The framework is compared with prior methods across AU detection and intensity estimation, including within-dataset and cross-dataset evaluations. It generally outperforms comparison methods, with performance patterns also reflecting differences in AU occurrence rates.
4.2.1 AU Detection
- On BP4D, ARL outperforms state-of-the-art AU detection methods, including relation-based and attention-based approaches.On DISFA it also outperforms previous methods, while on BP4D+ it exceeds JAA-Net in both F1-frame and accuracy.Source: §4.2.1 · p. 7
- On BP4D+, ARL outperforms JAA-Net on both F1-frame and accuracy despite the domain gap from BP4D training.This comparison is presented as evidence of better generalization.Source: §4.2.1 · p. 7
- Higher-occurrence AUs tend to have higher F1-frame, while low-occurrence AUs often bias toward non-occurrence and lower F1-frame.The reported higher-occurrence groups differ between BP4D and DISFA; other factors, including loss weighting and AU correlations, also influence per-AU performance.Source: §4.2.1 · pp. 7–8
4.2.2 AU Intensity Estimation
- On FERA 2015, iARL achieves higher ICC and lower MAE than state-of-the-art methods.This includes methods using ordinal models, although iARL was designed for AU detection without an ordinal model.Source: §4.2.2 · p. 8, Table 6 · p. 8
- On DISFA, iARL beats OR-CNN, DRML and CCNN-IT on ICC and MAE, and is comparable to ordinal method 2DC.On BP4D+ cross-dataset testing, it has higher average ICC and lower average MAE than JAA-Net.Source: §4.2.2 · p. 8, Table 7 · p. 8
4.3 Ablation Study
The ablations evaluate AU-loss weighting, channel-wise and spatial attention, and pixel-level relation learning. Adding these components improves detection and intensity-estimation results over the corresponding baselines.
4.3.1 AU Detection
- Adding spatial attention raises average F1-frame and accuracy from 58 and 76 to 59.6 and 77.3 in the reported BP4D ablation.Weighting AU losses improves W-Net over B-Net, and channel-wise attention improves WC-Net over W-Net.Source: §4.3.1 · p. 9
- On BP4D, ARL with pixel-level relation learning reaches average F1-frame 61.1 and average accuracy 78.2.The prose attributes its gains over WC-Net to integrating spatial attention with pixel-level relation learning.Source: §4.3.1 · p. 9
4.3.2 AU Intensity Estimation
- On FERA 2015, iWCSP-Net reaches average ICC 0.65 and average MAE 0.67; adding cosine-similarity loss further improves both results.The authors state that intensity and cosine-similarity losses train iARL using distances and correlations between ground-truth and predicted intensities.Source: §4.3.2 · p. 10
4.4 Visual Results
The visualizations examine channel selection, refinement of spatial attention, and the regions attended across AUs, expressions, people, and intensity levels. They show that the learned attention maps vary with the AU and image, while selected feature maps can retain irrelevant content.
4.4.1 Channel-Wise Attention Learning
- For AU 1, selected feature maps highlight the forehead; for AU 15, they highlight the mouth.The selected maps can still include cheek and nose information and background noise, which spatial attention is intended to process further.Source: §4.4.1 · p. 10
4.4.2 Integration of Spatial Attention Learning and Pixel-Level Relation Learning
- For AU 2, pixel-level relation learning reduces noisy attention on facial contours and background while preserving brow attention.The comparison of initial and refined attention weights illustrates the refinement.Source: §4.4.2 · p. 10
4.4.3 Analysis of Spatial Attention Weight
- ARL’s spatial attention details vary for the same AU across different people and expressions, unlike the more similar JAA-Net maps.The examples include two different people with sad expressions and one person with a happy expression.Source: §4.4.3 · p. 11
- Learned attention highlights correlated regions beyond predefined ROIs, with stronger responses in closely related regions than in slightly related ones.The authors describe the observed AU correlations as statistical patterns in the training set, while noting that specific AU relationships depend on expression.Source: §4.4.3 · p. 11
- For intensity estimation, attention spans broader facial regions than for detection; at intensity 0, mouth responses are lower and narrower than at positive intensities.The authors relate the broader regions to using more information for detailed intensity estimates and report adaptive attention across DISFA examples, including occlusion and illumination variation.Source: §4.4.3 · pp. 11–12, Figure 7 · p. 11, Figure 8 · p. 12
4.5 ARL for Partially Occluded Faces
The occlusion evaluation applies trained models to partially masked BP4D faces without additional processing. ARL is comparable to EAC-Net overall, with results varying according to which facial half remains visible.
4.5 ARL for Partially Occluded Faces
- For partially occluded BP4D faces, ARL performs comparably to EAC-Net, outperforming it when only the right or left half is visible.EAC-Net performs better when only the lower or upper half is visible, and uses landmark locations to define AU regions.Source: §4.5 · p. 12
- ARL’s right- and left-half advantage is attributed to capturing pixel-level relations, including bilateral symmetry, so visible-side information can aid prediction on the occluded side.The model is tested without additional processing on partially occluded faces.Source: §4.5 · p. 12
- Performance can worsen even when an AU’s predefined ROI remains visible, because correlated AU information from other facial parts may be occluded.The lower half performs best for both methods, consistent with many AU ROIs being in the lower face.Source: §4.5 · pp. 12–13
4.6 ARL for Faces with Wide Pose Variations
On FERA 2017's nine-pose validation setup, pARL achieved competitive performance against pEAC-Net and was reported robust across poses without landmark information.
4.6 ARL for Faces with Wide Pose Variations
- FERA 2017 uses 41 BP4D subjects for training and BP4D+ subsets of 20 and 30 subjects for validation and testing, respectively.Following pEAC-Net, the paper reports validation-set results.Source: §4.6 · p. 13
- pARL achieves competitive overall F1-frame performance against pEAC-Net across nine poses without additional landmark information.Results are reported on the validation set.Source: §4.6 · p. 13
- Average F1-frame was 54.2 for pose 4 and 55.1 for pose 5, near pose 6's frontal 55.2.The authors report pARL as robust to pose changes, including large poses.Source: §4.6 · p. 13
5 CONCLUSION
ARL outperformed state-of-the-art methods for AU detection and intensity estimation across several benchmarks, while also working well in cross-dataset testing, occlusions, and large poses. Its attention and pixel-level relation learning capture AU-related features and correlated regions.
5 CONCLUSION
- Channel-wise and spatial attention select AU-related local features, while pixel-level relation learning refines spatial attention to capture correlated regions.The conclusion reports each component as beneficial for AU detection and intensity estimation.Source: §5 · p. 13
- ARL extends to AU intensity estimation without changing its network architecture.Source: §5 · p. 13
- ARL soundly outperforms state-of-the-art methods for AU detection and intensity estimation on BP4D, DISFA, FERA 2015, and BP4D+.The authors also report that it works well in cross-dataset testing on large-scale images.Source: §5 · p. 13
- The trained ARL model detects AUs with only half-faces visible and performs well across wide poses, with performance comparable to landmark-assisted EAC-Net.EAC-Net uses ground-truth landmark locations to predefine each AU's region of interest.Source: §5 · pp. 13–14
- Joint attention and pixel-level relation learning is described as promising for facial expression recognition and face recognition.Source: §5 · p. 14