EAC-Net: A Region-based Deep Enhancing and Cropping Approach for Facial Action Unit Detection

Wei Li; Farnaz Abtahi; Zhigang Zhu; Lijun Yin

Source-linked AI summary

EAC-Net: A Region-based Deep Enhancing and Cropping Approach for Facial Action Unit Detection

Wei Li, Farnaz Abtahi, Zhigang Zhu, Lijun Yin

arXiv:1702.02925v1Feb 9, 2017cs.CV

TL;DR

AU detection methods may rely on frontal, processed faces or hand-designed texture features that are not well learned. EAC Net adds attention-based enhancement and region cropping to a pretrained CNN. On BP4D and DISFA, it reports improved AU detection performance, including gains over E-Net on both datasets.

Problem
Some AU detection approaches require processed frontal faces or use artificially designed texture features that are not well learned.
Source: §I · p. 1
Method
EAC Net adds attention-based enhancement and landmark-guided cropping layers to a pretrained CNN to learn from AU regions of interest.
Source: §I · pp. 1–2
Results
On BP4D, E-Net improves average F1 scores over FVGG by 8.3% and 10.1%, and EAC adds 3.5% F-score and 0.7% average accuracy; on DISFA, EAC gains 4.1% F1 and 4.9% average accuracy over E-Net.
Source: B. Implementation Details and Results · p. 6, C. Discussions · p. 7
Takeaways & Limitations
The authors report better performance on common AU datasets and robustness to additional facial alignment preprocessing.
Source: §V · pp. 7–8
The interest-region positions are currently located manually, and integrating more temporal information remains future work.
Source: §V · p. 8

Abstract

from arXiv · show

In this paper, we propose a deep learning based approach for facial action unit detection by enhancing and cropping the regions of interest. The approach is implemented by adding two novel nets (layers): the enhancing layers and the cropping layers, to a pretrained CNN model. For the enhancing layers, we designed an attention map based on facial landmark features and applied it to a pretrained neural network to conduct enhanced learning (The E-Net). For the cropping layers, we crop facial regions around the detected landmarks and design convolutional layers to learn deeper features for each facial region (C-Net). We then fuse the E-Net and the C-Net to obtain our Enhancing and Cropping (EAC) Net, which can learn both feature enhancing and region cropping functions. Our approach shows significant improvement in performance compared to the state-of-the-art methods applied to BP4D and DISFA AU datasets.

I. INTRODUCTION

The paper targets AU detection approaches that rely on frontal, processed faces or hand-designed texture features, and proposes EAC Net to focus learning on facial regions of interest. It combines attention-based enhancement with landmark-guided cropping.

The authors motivate region-focused learning by noting that CNNs may not know which image regions deserve attention or shift pixels to compare corresponding regions.Source: §I · p. 1
On BP4D, EAC Net reports up to a 7.6% increase in average F1 score and a 19.2% improvement in accuracy over state-of-the-art approaches.Source: §I · p. 1
E-Net applies attention layers to a pretrained VGG net, while EAC Net adds cropping layers that extract AU regions from high-level feature maps.Source: §I · p. 1
The approach is intended to handle face-position and orientation changes without facial preprocessing, and its landmarks need not be located very accurately.Source: §I · p. 1
The EAC Net combines a fine-tuned pretrained VGG-19, enhancing layers over convolutional groups 3 and 4, and cropping layers for ten interest areas.The cropped sub-features are upscaled and processed by convolutional layers to learn deeper features for each region.Source: §I · p. 2

II. RELATED WORK

Prior AU detection methods used landmark-centered texture and geometry features or CNN-based regional processing. EAC Net instead selects and crops interest regions, with the authors contrasting its landmark tolerance with normalization-based approaches.

Conventional AU methods often designed texture features near facial key points, including Gabor or SIFT representations, and paired them with classifiers or geometry features.Source: §II · p. 2
Related work also studied AU co-occurrence, joint AU and landmark detection, training-set size, and AU intensity estimation.Source: §II · pp. 2–3
Unlike Zhao et al., who divided images into 8x8 blocks, EAC Net crops selected interest regions; the authors say its larger local regions tolerate landmark shifts.The cited comparison also notes that facial normalization may neutralize expressions.Source: §II · p. 3

III. THE EAC NET

The paper describes EAC Net as a three-part model built from VGG fine-tuning, E-Net, and C-Net. Its comparisons use FVGG, E-Net, and the integrated EAC model.

EAC Net is composed of a VGG fine-tuning network, E-Net, and C-Net.Source: §III · p. 3
FVGG is the fine-tuning network of VGG and serves as one of the three implemented comparison networks.Source: §III · p. 3
EAC integrates the Enhancing and Cropping Nets based on the trained E-Net model.Source: §III · p. 3

A. F-VGG: VGG finetuning model

The F-VGG baseline adapts a pretrained VGG-19 network for AU detection. Its training keeps early convolutional groups fixed, changes later layers and fully connected outputs, and uses dropout.

The VGG-19 architecture has 16 convolutional layers, 3 fully connected layers, and 5 pooling layers, with convolutional layers divided into five groups.Source: A. F-VGG: VGG finetuning model · p. 3
F-VGG keeps the first three convolutional groups unchanged, updates the remaining layers, and changes the final fully connected layers to 2048 and 12 nodes.The 12-node output matches the 12 AU targets.Source: A. F-VGG: VGG finetuning model · p. 3
Dropout is applied to both modified fully connected layers to prevent overfitting during training.Source: A. F-VGG: VGG finetuning model · p. 3

B. E-Net: The Enhancing Net

E-Net builds AU-focused attention maps from facial landmarks and applies them alongside VGG convolutional features. The maps emphasize AU-related areas while preserving contextual features through fusion.

B. E-Net: The Enhancing Net

The attention map is applied in parallel with convolutional layers in groups 3 and 4, then fused by element-wise summation.The feature map from group 2 is multiplied by the designed attention map, and the result is fused with group 3 convolutional features; the operation is repeated for group 4.Source: B. E-Net: The Enhancing Net · pp. 3–4
The authors define 20 AU centers for 12 AUs, using paired points for symmetry and allowing multiple AUs to share a center.Centers are manually set based on facial muscles, and landmark-shift distances are scaled using the distance between the eye corners.Source: B. E-Net: The Enhancing Net · p. 3
Images are resized to 100x100 pixels; for each AU center, the nearby 5 pixels belonging to the same area define an 11x11 AU area, with higher weights assigned to closer points.Source: B. E-Net: The Enhancing Net · p. 4
Attention maps weight regions around AU centers, emphasizing active areas that correspond to higher map values.The AU centers are derived from facial landmarks; higher weights are assigned to pixels closer to each center.Source: B. E-Net: The Enhancing Net · p. 4

C. EAC-Net: The Integrated Model with Enhancing and Cropping

C-Net crops AU-related feature regions and learns them with independent convolutional layers, addressing variation in AU-area alignment across images. EAC-Net combines this cropping path with pretrained E-Net features.

C. EAC-Net: The Integrated Model with Enhancing and Cropping

C-Net targets AU-related areas without changing original image texture, using independent convolutions to learn features for each area.The motivation is that E-Net emphasizes AU regions but does not normalize the same regions across images; face normalization may lose context and weaken expressions.Source: C. EAC-Net: The Integrated Model with… · p. 4
Each of 20 AU centers yields a 512x3x3 feature crop, which is upscaled to 512x6x6 before added convolutions.The crop size follows the ratio between an 11x11 AU area and a 100x100 face; the upscaling layer alone improves average AU F1 score by approximately 1%.Source: C. EAC-Net: The Integrated Model with… · p. 4
C-Net is built on pretrained E-Net features, and local response normalization is added before its convolutions because group 4 values otherwise prevent convergence.The reported normalization settings are k=2, α=0.002, and β=0.75.Source: C. EAC-Net: The Integrated Model with… · pp. 4–5
AU detection is treated as multi-label binary classification because multiple AUs can co-occur, and cross entropy with offsets measures loss.The offsets are added to prevent the number from becoming too large.Source: C. EAC-Net: The Integrated Model with… · p. 5

A. Datasets and Evaluation Methods

The evaluation uses BP4D and DISFA AU data, with subject-based folds for BP4D and all DISFA samples reserved for testing. Training selection is adjusted to address AU imbalance, and performance is measured with F1 score and average accuracy.

A. Datasets and Evaluation Methods

DISFA contains many more inactive than active images, while BP4D provides around 140,000 AU-labeled images from varied expression sessions.The datasets differ in collection setup: DISFA records spontaneous expressions during videos, while BP4D captures expressions across eight sessions per subject.Source: A. Datasets and Evaluation Methods · p. 5
BP4D is split into three subject-based folds, with two folds for training and one for testing in each run; all DISFA samples are used for testing.The BP4D data comprise around 140,000 AU-labeled images from 41 young adults; DISFA includes more than 100,000 labeled images from 26 people.Source: A. Datasets and Evaluation Methods · p. 5
Less frequent AUs receive 4 to 7 times higher selection rates during random training-sample selection to compensate for their lower occurrence.The authors retain other samples to maintain data diversity despite imbalance among the less representative AUs.Source: A. Datasets and Evaluation Methods · p. 5
Performance is measured with F1 score and average accuracy; F1 combines precision and recall.Precision is the fraction of true-positive predictions among positive predictions, while recall is the fraction of ground-truth positive samples predicted positive.Source: A. Datasets and Evaluation Methods · p. 5

B. Implementation Details and Results

The paper evaluates FVGG, E-Net, and EAC Net on BP4D, with EAC Net achieving the best reported AU-detection performance. E-Net improves average accuracy and F1 over FVGG, while its feature maps focus on facial regions emphasized by attention.

Training resizes images to 224x224, uses facial landmarks, and sets the learning rate to 0.0001 and momentum to 0.9.The evaluation trains FVGG, then E-Net, then EAC Net using the BP4D dataset.Source: B. Implementation Details and Results · p. 5
FVGG has higher average accuracy than state-of-the-art approaches on BP4D, but its average F1 score does not outperform them.Representative AUs are predicted even without their positions being specified, whereas less representative AUs have less discriminative texture and lower occurrence rates.Source: B. Implementation Details and Results · pp. 5–6
EAC Net improves F-score by 3.5% and average accuracy by 0.7% from E-Net, achieving the best reported AU-detection performance.The reported improvement excludes AU2 from the nearly-all-AU gains.Source: Table III · p. 6, Table IV · p. 6, B. Implementation Details and Results · p. 6
E-Net improves average accuracy and F1 over state-of-the-art approaches and FVGG, with average F-score improvements over FVGG reported as 8.3% and 10.1%, respectively.Its group-4 feature maps concentrate on face regions enhanced by attention, unlike FVGG maps with unfocused hot areas and highlighted edges.Source: B. Implementation Details and Results · p. 6, Figure 4 · p. 6

C. Discussions

On DISFA, EAC Net improves F1 and accuracy over E-Net while using a model trained on BP4D rather than training directly on DISFA. On BP4D, alignment slightly hurts E-Net performance and substantially increases processing time; the discussion also reports gains from balancing data and robustness to non-frontal faces.

EAC Net increases DISFA F1 by 4.1% and average accuracy by 4.9% over E-Net.The authors report that C-Net's improvement is more significant than E-Net's on DISFA.Source: Table VI · p. 7, Table VII · p. 7, C. Discussions · p. 7
On BP4D, adding alignment slightly hurts E-Net performance and increases processing time from 30 ms to 2500 ms.The authors conclude alignment does not contribute to AU-detection performance in this comparison, while costing substantially more time.Source: C. Discussions · p. 7, Table V · p. 7
For DISFA, the authors use a BP4D-trained model rather than training directly on DISFA, then extract features and apply linear regression to predict labels.Three DISFA AUs are absent from BP4D; 27 subjects are split into three folds for independent predictions.Source: C. Discussions · p. 7
The authors report that balancing the training data can improve AU-detection results.They also describe the approach as more robust to non-frontal faces because it needs only approximate landmark positions, unlike approaches requiring normalization.Source: C. Discussions · p. 7

V. CONCLUSIONS

EAC-Net combines ROI-focused enhancing and cropping nets for AU detection. The authors report better performance on common AU datasets and robustness to facial alignment preprocessing, while identifying manual region placement and temporal information as future work.

EAC-Net integrates E-Net attention to focus on AU regions with C-Net learning features from aligned facial areas, supporting robustness to facial shifts and orientations.Source: §V · p. 7
The approach achieves better performance on common AU datasets.Source: §V · pp. 7–8
The authors report robustness to additional facial alignment preprocessing, while noting that region positions are currently manual and that more responsive regions and temporal information remain future work.They identify wild-video AU detection as a future direction.Source: §V · p. 8