Source-linked AI summary
Action Unit Detection with Region Adaptation, Multi-labeling Learning and Optimal Temporal Fusing
Wei Li, Farnaz Abitahi, Zhigang Zhu
TL;DR
AU detection must handle region-specific facial features, multiple AU labels, and temporal dynamics. The paper addresses these gaps with adaptive ROI Nets, multi-label learning, and LSTM-based temporal fusion, reporting improvements over prior approaches on BP4D and DISFA. Its conclusion emphasizes joint treatment of region adaptation, label relationships, and temporal information.
Problem
AU detection remains challenged by differing facial regions, multiple AUs in one image, and the temporal nature of facial actions.
Method
The framework uses region-specific ROI Nets, multi-label integration of regional outputs, and LSTM layers to fuse static CNN features over time.
Results
The proposed approach outperforms the state of the art on BP4D and DISFA, with reported average improvements of 13% and 25%, respectively.
Takeaways & Limitations
The study supports combining adaptive region learning, multi-label AU modeling, and temporal fusion for AU detection.
Abstract
from arXiv · showhide
Action Unit (AU) detection becomes essential for facial analysis. Many proposed approaches face challenging problems in dealing with the alignments of different face regions, in the effective fusion of temporal information, and in training a model for multiple AU labels. To better address these problems, we propose a deep learning framework for AU detection with region of interest (ROI) adaptation, integrated multi-label learning, and optimal LSTM-based temporal fusing. First, ROI cropping nets (ROI Nets) are designed to make sure specifically interested regions of faces are learned independently; each sub-region has a local convolutional neural network (CNN) - an ROI Net, whose convolutional filters will only be trained for the corresponding region. Second, multi-label learning is employed to integrate the outputs of those individual ROI cropping nets, which learns the inter-relationships of various AUs and acquires global features across sub-regions for AU detection. Finally, the optimal selection of multiple LSTM layers to form the best LSTM Net is carried out to best fuse temporal features, in order to make the AU prediction the most accurate. The proposed approach is evaluated on two popular AU detection datasets, BP4D and DISFA, outperforming the state of the art significantly, with an average improvement of around 13% on BP4D and 25% on DISFA, respectively.
1. Introduction
The paper targets three unresolved AU-detection challenges: region-specific facial features, simultaneous AU labeling, and temporal information fusion. It proposes adaptive ROI Nets, multi-label learning, and LSTM-based temporal fusion as a unified framework.
- AU detection must address differing facial-component features, multiple AUs within one image, and the temporal nature of facial actions.
- The framework combines adaptive region learning, multi-label AU prediction, and temporal fusion in a single proposed network.
- Adaptive ROI Nets learn regional features separately using convolutional filters trained only for corresponding facial regions.
- Multi-label learning integrates individual ROI outputs to capture AU inter-relationships and global features across facial sub-regions.
- An LSTM-based temporal fusion net combines static CNN features so AU predictions improve over static images alone.
2. Related Work
Prior AU-detection work includes landmark-based geometry and texture features, deep CNN representations, region-wise learning, and CNN-LSTM sequence modeling. These approaches motivate region-adaptive processing while exposing alignment dependence and limitations in conventional CNN feature extraction.
- Landmark-based methods use normalized landmark geometry, Delaunay-mask angles, and Gabor-filter texture features for AU-related facial analysis.
- CNNs learn deeper image representations than many manually designed features and have been applied to AU detection.
- Learning 8x8 face blocks separately can model individual facial parts but relies heavily on face alignment and treats all blocks equally.
- CNN-LSTM models combine spatial CNN and temporal features, but conventional CNNs cannot extract local features from specific facial regions.
3. Region of Interest Learning: ROI Nets
The ROI Nets use landmark- and muscle-informed centers to crop facial sub-regions, then learn their features with separate local filters. These regional representations can be paired for single-AU detection or concatenated for multi-label prediction.
- AU centers are selected from related muscle positions and adjusted across faces using detected facial landmarks.
- The method uses 20 landmark points and defines each AU center at a landmark or a specified distance from one.
- VGG layer 12 provides feature maps from which 20 sub-regions are cropped around the selected AU centers to form the ROI Nets.
- After local ROI learning, regional fully connected features are paired symmetrically for single-AU detection or concatenated for multi-label detection.
- Each ROI Net learns separate filters for its corresponding facial region, making feature learning adaptive to different local facial properties.
4. Temporal Fusing: LSTM Net
The LSTM Net fuses sequential CNN features by updating memory and output states across frames, with stacked LSTM layers used to model deeper temporal relationships for AU detection.
- LSTM block operation: At each time step, the LSTM uses the previous output and new input to update its cell state and generate the current output through gates.The described steps include forgetting or retaining prior cell information, updating the cell state, and producing the output.
- LSTM block operation: The forget gate determines what information to retain or discard from the previous cell state using the prior output and current input.The forget vector is defined by equation 1 and its parameters are identified as Wf and bf.
- LSTM block operation: The input gate combines the current input and previous output to update the cell state through an input transformation and tanh activation.The input-gate result and candidate cell state contribute to the updated cell state.
- CNN–LSTM connection: LSTM receives CNN feature vectors sequentially and produces each output from the current frame and the preceding output.Figure 4 describes a 24-frame sequence connection between CNN image features and LSTM processing.
- Stacked temporal fusion: Multiple LSTM layers can be stacked into an LSTM Net to represent temporal relationships more deeply for AU detection.The paper illustrates an LSTM Net with two stacked layers and compares different depths experimentally.
5. Experimental Evaluation
Experiments on BP4D and DISFA evaluate adaptive region learning, multi-label AU prediction, and temporal fusion. ROI Nets, multi-label learning, and a one-layer LSTM each improve performance, with the combined model outperforming prior methods on both datasets.
- Datasets and metrics: BP4D uses 3-fold subject-based cross-validation, while DISFA evaluation extracts features with the BP4D-trained model and performs 3-fold cross-evaluation.F1 scores are computed for 12 AUs on BP4D and 8 AUs on DISFA.
- Adaptive region learning: 12.4% average improvement over conventional FVGG is achieved by ROI Nets on BP4D.The comparison uses static-image models trained under the same general VGG-based setting.
- Single versus multi-label learning: 1.3% average improvement is obtained by multi-label over single-AU detection on BP4D, even with equal positive and negative sample distributions.Multi-label learning performs better for most AUs, while balancing helps under-represented AUs and AU correlation benefits AU6 and AU12.
- Temporal versus static prediction: 9.7% average F1 improvement is obtained by one-layer temporal fusion over ROI Nets using static images.Adding more LSTM layers decreases performance, with one layer reported as sufficient for the ROI features.
- Overall comparison: 12.9% improvement over the state of the art is achieved by the best BP4D model, R-T1, which outperforms prior results on 11 of 12 AUs.CNN+LSTM remains best for AU4.
- Overall comparison: 25% improvement over the state-of-the-art model is reported for R-T1 on DISFA.The DISFA temporal model uses the one-layer LSTM structure selected from BP4D evaluation.
6. Conclusion
The paper addresses region adaptation, temporal fusion, and single- versus multi-label learning for AU detection through a unified ROI-based recurrent approach. It reports substantial gains over the state of the art on BP4D and DISFA.
- The approach uses adaptive ROI cropping nets to learn separate filters for different facial regions, improving AU detection over conventional CNN processing.Each ROI has a local CNN whose filters are trained only for the corresponding region.
- Multi-label AU training outperforms single-AU detection by modeling AU relationships and combining global features across regions.
- LSTM-based temporal fusion substantially improves AU detection compared with static image-based approaches.The study also searches for an optimal LSTM-layer structure connected to the ROI networks.
- 13% average improvement on BP4D and 25% on DISFA were achieved over the state of the art.