Source-linked AI summary
Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks
Zhen-Hua Feng, Josef Kittler, Muhammad Awais, Patrik Huber, Xiao-Jun Wu
TL;DR
The paper addresses robust facial landmark localisation by analysing loss functions and challenges from under-represented extreme poses. It proposes Wing loss, pose-based data balancing, and a two-stage CNN framework, achieving the best reported accuracy on 300W with an almost 20% error reduction versus RAR.
Problem
CNN-based facial landmark localisation lacks systematic loss-function analysis, while large-pose samples are under-represented and unconstrained faces remain challenging.
Method
The paper compares L1, L2, and smooth L1, introduces Wing loss, balances pose distributions through augmentation, and combines these ideas in a two-stage CNN framework.
Results
The two-stage framework with pose-based data balancing and Wing loss outperforms all other state-of-the-art algorithms on 300W, reducing error by almost 20% versus RAR.
Takeaways & Limitations
Wing loss increases the contribution of small and medium errors, while pose balancing and two-stage localisation improve CNN-based landmark-localisation accuracy.
Takeaways & Limitations
The paper leaves discussion of Wing loss in other regression-based computer-vision tasks to future reports.
Abstract
from arXiv · showhide
We present a new loss function, namely Wing loss, for robust facial landmark localisation with Convolutional Neural Networks (CNNs). We first compare and analyse different loss functions including L2, L1 and smooth L1. The analysis of these loss functions suggests that, for the training of a CNN-based localisation model, more attention should be paid to small and medium range errors. To this end, we design a piece-wise loss function. The new loss amplifies the impact of errors from the interval (-w, w) by switching from L1 loss to a modified logarithm function. To address the problem of under-representation of samples with large out-of-plane head rotations in the training set, we propose a simple but effective boosting strategy, referred to as pose-based data balancing. In particular, we deal with the data imbalance problem by duplicating the minority training samples and perturbing them by injecting random image rotation, bounding box translation and other data augmentation approaches. Last, the proposed approach is extended to create a two-stage framework for robust facial landmark localisation. The experimental results obtained on AFLW and 300W demonstrate the merits of the Wing loss function, and prove the superiority of the proposed method over the state-of-the-art approaches.
1. Introduction
Facial landmark localisation seeks robust key-point prediction for unconstrained faces, but existing CNN approaches rely heavily on loss designs that are sensitive to outliers. The paper introduces Wing loss, pose-based data balancing, and a two-stage framework to address these challenges.
- Facial landmark localisation finds predefined semantic key points whose geometric information supports face recognition, emotion estimation, and 3D face reconstruction.
- Unconstrained faces remain difficult because pose, expression, illumination, blur, and occlusion vary substantially.
- Most deep-learning landmark-localisation methods use L2 loss, although L2 is sensitive to outliers and smooth L1 has also been adopted.
- L1 and smooth L1 perform much better than the widely used L2 loss in the paper’s empirical and theoretical comparison.
- Wing loss is designed to improve CNN training for small and medium range errors by amplifying their contribution.
- Pose-based data balancing compensates for the low frequency of large out-of-plane head rotations in training data.
- The paper also proposes a two-stage facial landmark localisation framework for performance boosting.
2. Related work
Related work covers CNN and other deep architectures, strategies for pose variation, and cascaded networks. The paper frames extreme-pose difficulty as a training-data imbalance problem and uses pose-based balancing with a two-stage CNN framework.
- Network Architectures: Regression-based landmark localisation commonly uses CNNs, while FCNs and hourglass networks have also shown promising results.
- Dealing with Pose Variations: Existing pose-variation strategies include multi-view models, 3D face models, pose-varied synthesis, and multi-task learning.
- Dealing with Pose Variations: The paper treats extreme-pose localisation as a training-data imbalance problem and proposes pose-based data balancing.
- Cascaded Networks: Cascaded networks stack multiple models to strengthen regression, but CNNs using global face images do not directly provide the local landmark features needed for such cascades.
- Cascaded Networks: The proposed two-stage CNN uses a fast first network for rough localisation and a second network for fine-grained localisation.
3. CNN-based facial landmark localisation
CNN-based facial landmark localisation learns a nonlinear mapping from a cropped face image to a vector of landmark coordinates. The paper formulates supervised regression training and describes the CNN-6 architecture used for loss-function analysis.
- The CNN maps an input colour image to a shape vector containing the 2D coordinates of L predefined facial landmarks.
- The shape vector is s = [x1, ..., xL, y1, ..., yL]T, with each landmark represented by coordinates (xl, yl).
- Training seeks network parameters that minimise a predefined loss measuring the difference between predicted and ground-truth shape vectors.
- CNN-6 uses a 64×64×3 colour image input and outputs 2L real numbers for the landmark coordinates.
- CNN-6 contains five 3 × 3 convolutional layers, one fully connected layer, and an output layer with nonlinear activation layers.
- The paper also reports advanced architectures, including a two-stage framework and ResNet-50, for performance boosting.
4. Wing loss
The paper analyzes common regression losses for CNN-based facial landmark localisation and proposes Wing loss to emphasize small and medium errors while retaining linear behavior for larger errors. On AFLW, Wing loss improves average normalised error over prior losses and deep-learning baselines.
- Analysis of different loss functions: L1 and smooth L1 perform much better than the widely used L2 loss for CNN-based facial landmark localisation.
- Analysis of different loss functions: For multiple landmarks, L1 and L2 updates are dominated by larger errors, making relatively small displacements difficult to correct.L1 is dominated through step size, whereas L2 is dominated through gradient magnitude.
- The proposed Wing loss: Wing loss combines a logarithmic nonlinear region for |x| < w with an L1-like linear region for larger errors.w controls the nonlinear range, ϵ controls its curvature, and C smoothly links the two pieces.
- The proposed Wing loss: Pose-based data balancing addresses under-represented large out-of-plane rotations by duplicating minority training samples and perturbing them through augmentation.Pose coefficients are obtained by projecting aligned shapes into a one-dimensional pose space.
5. Pose-based data balancing
Pose-based Data Balancing addresses extreme-pose difficulty by treating it as a training-data imbalance problem and augmenting underrepresented pose samples.
- Extreme pose variations challenge robust facial landmark localisation because training data are dominated by near-frontal faces.The authors argue that this imbalance can cause networks to overfit frontal poses and adapt poorly to large poses.
- PDB estimates pose variation by aligning training shapes, projecting them into a pose-space eigenvector, and histogramming the projection coefficients.Procrustes alignment and PCA provide the pose representation used to identify bins with low occupancy.
- PDB balances the data by duplicating samples from low-occupancy bins and perturbing them with rotation, bounding-box changes, and other augmentation.The duplicated samples are modified rather than copied unchanged.
- PDB improves CNN-6 performance on AFLW for every tested loss function.The comparison uses the average normalised error metric reported for the AFLW experiments.
6. Two-stage landmark localisation
The two-stage framework first makes a fast coarse prediction, then uses it to rectify the image and refine landmark coordinates at higher resolution.
- The framework addresses in-plane rotation and inaccurate face bounding boxes, which can degrade landmark localisation alongside out-of-plane pose variation.
- CNN-6 processes 64 × 64 × 3 images first, and its predicted landmarks guide image rectification for CNN-7.Rectification removes in-plane rotation and corrects the bounding box before the second stage.
- CNN-7 performs fine-grained localisation on 128 × 128 × 3 inputs using an expanded convolutional architecture.The second network adds a convolutional, ReLU, and max-pooling layer set relative to CNN-6.
- The two-stage framework further improves accuracy regardless of the loss function used.
7. Experimental results
Experiments on AFLW and 300W compare loss functions, architectures, accuracy, and speed. The proposed CNN-6/7 system with PDB and Wing loss achieves strong accuracy while remaining substantially faster than ResNet-50.
- The evaluation covers AFLW and 300W, comparing the proposed method with state-of-the-art approaches using established protocols and NME-based evaluation.AFLW-Full normalises by face-box width or height, while the 300W protocol uses inter-pupil distance.
- CNN-6/7 with PDB outperforms all other approaches on AFLW even when trained with L2 loss.Switching to L1 or smooth L1 improves performance significantly, and Wing loss improves it further.
- The CNN-6/7 framework with PDB and Wing loss outperforms all state-of-the-art algorithms on 300W, reducing error by almost 20% versus RAR.The comparison uses NME averaged over all test samples.
- Replacing CNN-6/7 with ResNet-50 further improves performance by around 10% on both AFLW and 300W.The improvement is accompanied by much slower training and inference than CNN-6/7.
- ResNet-50 experiments on AFLW further demonstrate Wing loss superiority over other losses for large-capacity networks.
8. Conclusion
The paper concludes that Wing loss, pose-based balancing, and two-stage localisation improve CNN-based facial landmark localisation across benchmark datasets. It also identifies broader regression-based vision use as future work.
- L1 and smooth L1 perform much better in accuracy than L2 for regression-based facial landmark localisation.
- Wing loss increases the contribution of samples with small and medium errors during regression-network training.
- Pose-based data balancing and two-stage localisation are advocated to improve CNN-based facial landmark accuracy further.
- The proposed Wing loss is presented as relevant to other regression-based computer-vision tasks using CNNs, but its extended use is left for future reports.