Source-linked AI summary
Incremental Boosting Convolutional Neural Network for Facial Action Unit Recognition
Shizhong Han, Zibo Meng, Ahmed Shehab Khan, Yan Tong
TL;DR
Limited AU-coded training images make it difficult for CNNs to generalize from spontaneous facial expressions to unseen subjects. IB-CNN incrementally selects and accumulates discriminative neurons and fine-tunes the network using errors from strong and weak classifiers; across four benchmark databases it outperforms CNN-based comparisons, with stronger gains on infrequent AUs.
Problem
CNNs can overfit and generalize poorly to unseen subjects when AU-coded training images are limited, while spontaneous expressions also involve subtle changes, head movements, and occlusions.
Source: §1 · p. 1Method
IB-CNN selects discriminative fully connected features across mini-batches, accumulates them into an incremental strong classifier, and fine-tunes the CNN using strong- and weak-classifier errors.
Source: §1 · p. 2Results
IB-CNN significantly outperforms traditional CNN and state-of-the-art CNN-based methods on four benchmark AU databases, with more impressive gains for infrequent AUs.
Source: §1 · p. 2Takeaways & Limitations
The reported results indicate IB-CNN is more effective for recognizing infrequent AUs with limited training data.
Source: §5 · p. 8Because SEMAINE and BP4D test sets are unavailable, comparisons for those databases use validation sets.
Source: §4.4 · p. 7
Abstract
from arXiv · showhide
Recognizing facial action units (AUs) from spontaneous facial expressions is still a challenging problem. Most recently, CNNs have shown promise on facial AU recognition. However, the learned CNNs are often overfitted and do not generalize well to unseen subjects due to limited AU-coded training images. We proposed a novel Incremental Boosting CNN (IB-CNN) to integrate boosting into the CNN via an incremental boosting layer that selects discriminative neurons from the lower layer and is incrementally updated on successive mini-batches. In addition, a novel loss function that accounts for errors from both the incremental boosted classifier and individual weak classifiers was proposed to fine-tune the IB-CNN. Experimental results on four benchmark AU databases have demonstrated that the IB-CNN yields significant improvement over the traditional CNN and the boosting CNN without incremental learning, as well as outperforming the state-of-the-art CNN-based methods in AU recognition. The improvement is more impressive for the AUs that have the lowest frequencies in the databases.
1 Introduction
Facial AU recognition is difficult because spontaneous expressions, limited AU-coded training data, and mini-batch learning challenge CNN generalization to unseen subjects. IB-CNN combines boosting with incremental learning and a loss accounting for strong- and weak-classifier errors; it outperforms CNN-based comparisons across four benchmark databases, especially on infrequent AUs.
1 Introduction
- Limited AU-coded images, subtle appearance changes, head movements, and occlusions make spontaneous AU recognition challenging and can leave CNNs overfit to training subjects.The introduction gives examples of 48,000 images from 15 subjects in SEMAINE and 130,814 images from 27 subjects in DISFA.Source: §1 · p. 1
- IB-CNN selects discriminative fully connected features per batch and combines current and earlier selections into an incremental strong classifier.Its classifier loss is backpropagated to fine-tune CNN parameters.Source: §1 · p. 2, Figure 1 · p. 2
- The proposed loss combines overall incremental strong-classifier error with individual weak-classifier errors to fine-tune IB-CNN.The paper also integrates feature selection and classification with CNN optimization and updates the boosted classifier across batches.Source: §1 · p. 2
- IB-CNN significantly outperforms traditional CNN and state-of-the-art CNN-based methods on four benchmark AU databases, with larger gains for infrequent AUs.The reported results cover CK, FERA2015 SEMAINE, FERA2015 BP4D, and DISFA.Source: §1 · p. 2
2 Related Work
Related work spans hand-crafted AU features, learned representations, CNN refinements, and boosting or incremental-learning alternatives. The paper distinguishes IB-CNN by selecting discriminative fully connected neurons and accumulating weak learners across batches.
2 Related Work
- AU recognition has used hand-designed features and their spatiotemporal extensions, while feature-learning approaches include sparse coding and deep learning.The listed hand-designed descriptors include Gabor wavelets, LBP, HOG, SIFT, and LPQ.Source: §2 · p. 2
- CNN-based AU methods have added input normalization, multi-size filters, 3D spatiotemporal features, joint temporal appearance and geometry, or bidirectional long-term memory.These approaches modify inputs, features, or temporal modeling around CNNs.Source: §2 · pp. 2–3
- Other CNN strategies replace softmax with an SVM or use dropout, whereas IB-CNN selects discriminative neurons instead of dropping activations randomly [22, 6, 23, 25].The paper characterizes dropout as random with respect to individual neurons’ discriminative power.Source: §2 · p. 3
- Prior incremental learning used multiple separately trained CNNs and weighted voting, while a boosted deep belief network learned weak classifiers from image patches.IB-CNN instead uses one CNN with an incremental strong classifier and selects weak learners from the fully connected layer, using whole-face features.Source: §2 · p. 3
3 Methodology
The method integrates boosting into the CNN decision layer, selecting features batch by batch and combining them into an incremental classifier. A loss covering both strong- and weak-classifier errors is used to fine-tune the network.
3 Methodology
- The IB-CNN integrates neuron selection, incremental classifier construction, and loss-based fine-tuning into CNN optimization.Boosting selects active features from the fully connected layer, combines selections across batches, and forms the decision layer.Source: §3 · p. 3, §3.1 · p. 3
3.1 A Brief Review of CNNs
- CNN decision scores usually use FC-layer inner products; this method replaces that score function with a boosting score function.The score function is used at the decision layer, whose output count matches the target classes.Source: §3.1 · p. 3
3.2 Boosting CNN
- B-CNN fine-tuning minimizes strong-classifier and individual weak-classifier losses, while mini-batch training can discard prior information and overfit.The weak-classifier loss sums individual losses; the strong-classifier loss measures Euclidean distance between prediction and ground-truth label.Source: §3.2 · p. 4
3.3 Incremental Boosting
- IB-CNN combines features selected in the current batch with previously selected features; repeatedly selected weak classifiers receive higher weights.Its incremental strong classifier uses information from all mini-batches, unlike B-CNN testing, which uses the classifier learned in the last iteration.Source: §3.3 · pp. 4–5, Figure 2 · p. 4, Figure 3 · p. 5
3.4 IB-CNN Fine-tuning
- Stochastic-gradient fine-tuning minimizes the IB-CNN loss, with gradients for activation features and thresholds backpropagated through the network.Threshold gradients are calculated only for active nodes, and activation-feature gradients continue through lower FC and convolutional layers.Source: §3.4 · p. 5
4 Experiments
The experiments evaluate IB-CNN on four benchmark AU databases and compare it with CNN baselines and state-of-the-art methods. They also examine preprocessing, implementation choices, and sensitivity to parameters.
4 Experiments
- The evaluation covers CK, SEMAINE, BP4D, and DISFA, with 14, 6, 11, and 12 labeled AUs, respectively.The databases contain 97, 31, 41, and 27 subjects, respectively.Source: §4 · p. 5
4.1 Pre-Processing
- Preprocessing aligns faces using eye and mouth centers, warps them toward a frontal view, and normalizes images within short video sequences.Images are scaled to 128 × 96; time-sequence normalization uses at least 800 continuous frames.Source: §4.1 · p. 6
4.2 CNN Implementation Details
- The IB-CNN uses three convolutional layers, two max-pooling layers, a 128-node fully connected layer, and an incremental boosting layer.Training uses stochastic gradient descent with momentum 0.9 and mini-batches of 100.Source: §4.2 · p. 6
4.3 Experimental Results
- On SEMAINE, IB-CNN reaches average F1 0.416 and 2AFC 0.775, versus CNN at 0.347 and 0.735 and B-CNN at 0.310 and 0.673.For AU28, with around 1.25% positive training samples, F1 rises to 0.490 from 0.280 (CNN) and 0.144 (B-CNN).Source: §4.3 · p. 6, Table 1 · p. 6
- Across four benchmark databases, IB-CNN is comparable to state-of-the-art methods and outperforms the CNN-based methods.The comparison uses metrics common to the respective papers.Source: Table 3 · p. 7, §4.3 · p. 7
- On DISFA, IB-CNN improves average F1 to 0.457 from 0.405 (CNN) and 0.398 (B-CNN), and 2AFC to 0.823 from 0.780 and 0.815, respectively.The evaluation used 9-fold cross-validation, with 24-subject training subsets and 3-subject test subsets.Source: §4.3 · p. 7, Table 2 · p. 7
4.4 Data Analysis
- On SEMAINE, average F1 is robust to η choices corresponding to c from 0.5 to 16.The experiment sets η to half the standard deviation σ_c.Source: §4.4 · p. 7, Figure 4 · p. 7
- IB-CNN is less sensitive than traditional CNN to learning-rate choice and is more robust to the number of IB-layer input neurons.The latter comparison tested 64, 128, 256, 512, 1042, and 2048 input neurons; B-CNN was also more robust than traditional CNN.Source: §4.4 · pp. 7–8, Figure 5 · p. 8
5 Conclusion and Future Work
IB-CNN integrates boosting into CNN-based AU recognition, accumulating information across mini-batches and fine-tuning with a loss that accounts for strong- and weak-classifier errors. Across four benchmark databases, it improves on traditional CNNs and state-of-the-art CNN methods, with stronger gains for infrequent AUs.
5 Conclusion and Future Work
- On four benchmark AU databases, IB-CNN significantly improves over traditional CNNs and state-of-the-art CNN-based AU recognition methods.The paper also reports greater effectiveness for infrequent AUs with limited training data.Source: §5 · p. 8
- IB-CNN integrates boosting classification into a CNN and accumulates information from multiple batches to address limited positive samples in a mini-batch.Source: §5 · p. 8
- Its fine-tuning loss accounts for errors from both the incremental strong classifier and individual weak classifiers.Source: §5 · p. 8
- The authors describe IB-CNN as adaptable to other learning tasks, especially those with limited training data.Source: §5 · p. 8
- Future work will extend IB-CNN to multitask learning by replacing the binary classifier with a multiclass boosting classifier.Source: §5 · p. 8