Deep Structure Inference Network for Facial Action Unit Recognition

• AUs are localized patterns of muscular activation, while face patches can contribute to specific AUs or groups of AUs.The introduction motivates learning patch combinations and patch-specific features because faces are highly articulated.Source: §1 · p. 1
• Several AUs can be active together, and certain AU combinations are more probable, making recognition a structured multi-label problem.Existing deep approaches address correlations implicitly, while graphical-model approaches are not end-to-end trainable.Source: §1 · p. 1
• The proposed network learns local and holistic representations from patches, then models patch structure and AU correlations with message passing.Its structure-inference topology uses a recurrent neural network to replicate inference in probabilistic graphical models.Source: §1 · pp. 1–2
• 5.3% on BP4D and 8.2% on DISFA are the reported state-of-the-art performance improvements from jointly optimizing the model's components.The approach separates representation, patch, and output-structure learning across network components while training them end-to-end.Source: §1 · p. 2

Patch Learning

• Prior patch-learning approaches use regional convolution, early fusion, patch selection, geometry-based regions, or sparse learning of common and specific patches.Their strategies vary in whether patch locations are learned or predefined and whether facial geometry is used.Source: §2 · p. 2

Multi-label and Structure Learning

• Structure-learning work models AU dependencies with kernels, CRFs, restricted Boltzmann machines, competitions, or combined recognition losses.The cited approaches include methods for local pairwise dependencies as well as higher-order AU interactions.Source: §2 · p. 3

Others

• Other methods focus on temporal decision fusion, easy-to-hard classifier ensembles, or disentangling shared and expression-specific features.These approaches address alternatives to jointly learning deep local representations, patches, and output structure.Source: §2 · p. 3

3. Method

• The model represents multi-label AU recognition with a graph: CNN patch outputs augment AU nodes, AU nodes are fully connected, patch-to-AU edges are selective, and patch nodes have no mutual edges.This structure supports joint learning of local predictions and AU relationships.Source: §3 · pp. 3–4

3.1. Patch Prediction

• Independent CNNs produce AU probabilities from local patches and the whole face, allowing each network to specialize in its region despite possible patch overlap.The patch prediction functions provide AU probabilities through sigmoid outputs.Source: §3.1 · p. 4

3.2. Fusion

• Per-AU fusion combines that AU’s predictions across patches through two fully connected layers with 64 hidden units; higher dimensionality added no performance and quickly overfit.The directed patch-to-AU edges reduce the complexity of marginalizing pairwise relationships.Source: §3.2 · p. 5

3.3. Structure Inference

• Recurrent SIUs iteratively refine each AU from its fused estimate, messages from other AU nodes, and its previous prediction.Parameters are shared across iterations, and gradients from structure inference train CNN features end-to-end.Source: §3 · p. 4, §3.3 · p. 5

4. Experimental Analysis

• The section describes the experimental setting.This is the stated scope of the section.Source: §4 · p. 6
• The section presents detailed results.The passage identifies detailed results as a focus.Source: §4 · p. 6
• The section is an experimental analysis.Its title identifies the section as experimental analysis.Source: §4 · p. 6

4.1.1 Data.

• BP4D includes 328 videos from 41 young adults and about 140k valid face images, while DISFA contains about 130k frames with AU intensities above 3 treated as active.BP4D has higher label density and shorter sequences than DISFA.Source: §4.1.1 · p. 6

4.1.2 Preprocessing.

• Faces are aligned to three mean-landmark anchors, resized to 224 × 224 × 3, and represented by five 56×56×3 patches around detected landmarks.Symmetrical patches are omitted to reduce redundancy.Source: §4.1.2 · p. 6

4.1.3 Training.

• Training incrementally supervises patch prediction, fusion, and structure inference before joint optimization, using class-weighted patch loss and binary cross-entropy for fusion and inference.The patch-loss weights inversely reflect each AU class's positive frequency; correction factors are regularized to encourage sparse message passing.Source: §4.1.3 · p. 7, §4.1.2 · p. 7

4.1.4 Methods and metrics.

• Evaluation uses subject-exclusive 3-fold cross-validation and compares CPM, APL, JPML, DRML, and ROI using F1-frame scores averaged across AUs.On DISFA, BP4D-trained patch CNNs are adapted by retraining fully connected layers while keeping convolutional filters fixed.Source: §4.1.4 · p. 7, §4.1.3 · p. 7

4.2.1 Ablation Study.

• Class balancing improves poorly represented classes but can reduce performance on frequent classes, and the proposed patch-prediction topology outperforms fine-tuned VGG-16.The BP4D classes with fewer than 30% positive samples improve most with balancing.Source: §4.2.1 · p. 7
• Training on groups of AUs or all AUs is usually beneficial, while patch-only predictions can miss AUs outside the visible region and patch performance varies by AU.Individual-AU training is sometimes superior, including for AU01 from the right-eye patch; specific AUs also favor different facial patches.Source: §4.2.1 · pp. 7–8
• AU-wise patch fusion benefits both datasets, especially DISFA, and usually approaches or exceeds the best patch prediction, but struggles with noisy inputs and AU14 mouth predictions on BP4D.On BP4D, fusion improves results over face prediction for almost all AUs.Source: §4.2.1 · p. 8
• Structure inference adds more than 5% improvement over fusion on DISFA and considerably improves AU2 and AU14 on BP4D.The DISFA comparison uses T = 10; results are inferior without correction factors.Source: §4.2.1 · pp. 8–9, Table 4 · p. 9
• Regularizing correction factors brings no significant performance benefit; values above 5 × 10^-2 hurt performance, so the model uses r = 5 × 10^-3.Small regularization values make correction factors diverge faster, while stronger regularization can collapse them and hinder message passing.Source: §4.2.1 · p. 9, Figure 6 · p. 9
• Per-AU threshold tuning gains 2.8% on BP4D and 3.1% on DISFA; on BP4D, thresholds from 0.1 to 0.3 are preferable for most classes.The reported baseline threshold is τ = 0.5.Source: §4.2.1 · p. 9, Figure 7 · p. 9

4.2.2 Comparison with state-of-the-art.

• DSIN improves performance by 5.3% absolute on BP4D and 8.2% absolute on DISFA, achieving the best results on both datasets.It performs best on 5 of 12 BP4D AUs and all but one DISFA AU.Source: Table 5 · p. 10, Table 6 · p. 10, §4.2.2 · p. 10

4.2.3 Qualitative results.

• Qualitative examples show structure inference correcting predictions using positive and negative AU correlations, while DSIN attention for AU24 extends beyond localized patch regions.The examples include correcting AU06 and AU07 predictions and an AU17 false positive.Source: §4.2.3 · p. 10

5. Conclusion

• DSIN integrates patch and multi-label learning with independent deep local and global representations and message passing to capture AU correlations.Its structure-inference stage learns relationships between AU predictions through stacked fully connected layers.Source: §5 · p. 10
• The end-to-end model improves state-of-the-art performance by 5.3% on BP4D and 8.2% on DISFA.Source: §5 · p. 10
• Future work includes learning patch structure at the feature level and increasing the structure-inference module’s capacity for output-structure learning.Source: §5 · p. 10