Source-linked AI summary
Facial Landmark Detection with Tweaked Convolutional Neural Networks
Yue Wu, Tal Hassner, KangGeon Kim, Gerard Medioni, Prem Natarajan
TL;DR
Facial landmark detection must accommodate substantial pose and appearance variation, while prior methods may be limited to variations anticipated in their designs. The paper analyzes intermediate CNN features, uses their alignment information to fine-tune pose-specific processing in TCNN, and reports performance beyond existing state-of-the-art systems. It also identifies boundaries involving appearance coverage and feature-based augmentation.
Problem
Facial landmark detection remains challenging under unconstrained pose, expression, occlusion, and appearance variation, which existing designs may not fully cover.
Method
TCNN clusters intermediate CNN features by alignment and fine-tunes the remaining layers separately for each cluster, while sharing early processing.
Results
TCNN outperforms existing state of the art with an efficient architecture and fewer labels required for training.
Takeaways & Limitations
Intermediate CNN features naturally provide rough head-pose information that supports pose-specific landmark regressors without separately estimating pose first.
Takeaways & Limitations
The approach cannot sample landmark coordinates directly from GMM components because those components are defined on intermediate features.
Abstract
from arXiv · showhide
We present a novel convolutional neural network (CNN) design for facial landmark coordinate regression. We examine the intermediate features of a standard CNN trained for landmark detection and show that features extracted from later, more specialized layers capture rough landmark locations. This provides a natural means of applying differential treatment midway through the network, tweaking processing based on facial alignment. The resulting Tweaked CNN model (TCNN) harnesses the robustness of CNNs for landmark detection, in an appearance-sensitive manner without training multi-part or multi-scale models. Our results on standard face landmark detection and face verification benchmarks show TCNN to surpasses previously published performances by wide margins.
1. Introduction
Facial landmark detection must handle diverse poses, expressions, occlusions, and appearance changes, while existing approaches often target only selected variations. The paper introduces a CNN that uses intermediate representations to apply pose-sensitive processing without multiple part models or auxiliary labels.
- Unconstrained images contain poses, expressions, and occlusions that can alter landmark appearances, locations, or presence.
- Existing methods can be limited by the pose and appearance variations represented in their designs.
- The proposed CNN performs facial landmark regression without multiple part models, using a naturally hierarchical design and no auxiliary labels beyond landmarks.
- Deep landmark-regression features from intermediate layers capture alignment and similar poses more clearly than RGB-based clusters.
- TCNN diverts intermediate features into specialized processing for particular head poses while sharing early convolutional computation.
- The benefits are evaluated on AFLW, AFW, 300W, and Janus CS2 landmark-detection and face-verification benchmarks.
2. A CNN for facial landmark detection
The study examines how a vanilla CNN represents facial landmark information across layers, then uses these representations to guide alignment-sensitive processing. Deeper features, especially at FC5, capture landmark alignment and facial properties, motivating specialized regressors for different feature clusters.
- Vanilla network design: The vanilla CNN uses four convolutional layers, pooling, FC5, and a 2 × m coordinate output for m = 5 facial landmarks.Faces are detected, resized to 40×40 pixels, normalized, and represented using RGB values.
- Research questions: The study's questions concern whether similar features reflect similar facial properties, which properties they encode, and at which layer those properties emerge.These questions motivate adjusting processing midway through the network.
- Intermediate representations: The analysis clusters intermediate representations from each layer with GMMs and examines whether similar features correspond to similar facial properties.The study uses K = 64 clusters per layer and assigns features by highest posterior probability.
- Landmark positions: Average intra-cluster landmark-location variance drops by half from the input to FC5, indicating that deeper layers better represent landmark positions.The authors interpret this as hierarchical, coarse-to-fine feature localization.
- Facial attributes: Higher-layer clusters show reduced variability for smiling and eyeglass attributes, while gender becomes more varied in higher layers.The authors suggest that clustering by pose alone may not be optimal because facial attributes also influence the representations.
- Discussion: At the input to FC5, the network already discriminates differently aligned faces and provides rough head-pose information for pose-specific landmark regressors.Unlike prior two-step approaches, the rough pose emerges naturally from the landmark-regression network.
3. The Tweaked CNN model
TCNN uses intermediate CNN features to route images into pose- and expression-specific final-layer processing, while alignment-sensitive augmentation supports fine-tuning without introducing excessive landmark variation.
- The Tweaked CNN model: Similar intermediate features imply similar landmark positions, allowing the tweaked layers to specialize in facial poses and expressions.The approach exploits the observed specialization of late CNN features for alignment.
- The Tweaked CNN model: TCNN clusters FC5 features and fine-tunes separate versions of the final network layers for images with similar intermediate representations.Earlier layers remain shared and frozen, while each cluster-specific sub-network starts from the vanilla CNN weights.
- The Tweaked CNN model: Fine-tuning a single cluster took approximately 2 minutes, compared with approximately 6.5 hours for training the vanilla CNN.Different clusters can be fine-tuned in parallel, making the additional tweaking time low.
- The Tweaked CNN model: At inference, FC5 features assign each query to a cluster, after which only that cluster’s fine-tuned layers produce the landmark prediction.Horizontally flipped inputs are also processed and their predictions averaged after coordinate mirroring.
- Alignment-sensitive data augmentation: Standard oversampling and mirroring were unsuccessful because they introduce misaligned images and increase landmark-position variability within clusters.Sampling directly from GMM components also cannot produce the landmark coordinates required for training.
- Alignment-sensitive data augmentation: Alignment-sensitive augmentation warps one image toward another image from the same cluster and retains the generated sample only if it remains in that cluster.Accepted samples receive the target image’s landmark labels; fewer than 40% of generated images failed the cluster-membership test.
- Alignment-sensitive data augmentation: The alignment-sensitive method failed to improve vanilla-CNN training when used with a single cluster, so it was not applied to those reported results.For cluster-specific training, the method raised each cluster’s training set to 5,000 images; smaller sets overfit and larger sets added no meaningful gain.
4. Comparison with existing work
The paper positions TCNN as a simpler alternative to multi-part, auxiliary-label, and conventional multi-scale approaches, using internal CNN representations to guide specialized processing.
- Comparison with existing work: TCNN improves network performance on unsupervised augmented subsets of the original data, unlike prior late-layer fine-tuning for domain transfer.The subsets are determined from intermediate CNN features rather than manually specified domains.
- Comparison with existing work: TCNN’s alignment-sensitive augmentation differs from feature-space sampling methods because it generates additional training data.The method is presented as a particular form of sampling from CNN feature space.
- Comparison with existing work: TCNN uses intermediate representations for coarse landmark positions and tweaked final layers for localization, constituting a distinct multi-scale design.The paper states that prior multi-scale designs do not use internal representations to guide processing in this way.
- Comparison with existing work: Unlike multi-task landmark-and-attribute methods, TCNN uses no auxiliary attribute labels and is reported to exceed their performance with a simpler network design.The paper argues that the CNN naturally learns to discriminate some facial attributes.
5. Experiments
Experiments evaluate TCNN on landmark detection and face verification under unconstrained and high-pose-variation conditions, finding improvements over vanilla CNNs and published alternatives.
- Evaluation: Detector error rate normalizes mean landmark distance by inter-ocular distance and is used as the landmark-detection accuracy measure.Lower error rates indicate better landmark localization.
- Landmark detection: TCNN improves accuracy in nearly all feature clusters, with some cluster-level gains reaching several percentage points.The comparison uses validation-set per-cluster error rates for vanilla CNN and TCNN.
- Landmark detection: The largest TCNN improvements occur for poorly performing non-frontal pose clusters, which are underrepresented in training.Near-profile views form the worst-performing cluster, while also receiving the biggest gains from TCNN.
- Landmark detection: The landmark benchmarks contain unconstrained Flickr faces with pose and appearance variation, including 468 AFW faces and 24,386 AFLW faces.Both datasets are evaluated using five manually annotated facial landmarks and the test subsets from prior work.
- Landmark detection: Using more tweaked processes improves performance only up to a point; beyond K = 64 clusters, fine-tuning often overfits, so reported TCNN results use 48 clusters.Too many clusters leave insufficient examples for effective fine-tuning.
- Landmark detection: Tweaking effort grows linearly with the number of clusters, with 79.63 ± 2.9 epochs required per tweaked process.The reported effort is measured in epoch numbers.
- Face recognition: With all other recognition components held equal, TCNN-aligned images are easier to recognize and outperform the larger dedicated DCNN system.The face-verification evaluation uses the Janus CS2 benchmark and compares alignment from four landmark detectors.
6. Conclusions
The paper shows that analyzing intermediate CNN representations can introduce hierarchical, discriminative processing without simply making the model more elaborate, boosting performance beyond more involved state-of-the-art systems. The same analysis also suggests potential benefits for identity-sensitive face recognition.
- 6. Conclusions: Intermediate-layer analysis introduces hierarchical, discriminative processing into an existing facial landmark regression CNN.The approach uses the values produced at intermediate CNN layers rather than relying on progressively more elaborate models and representations.
- 6. Conclusions: The resulting processing boosts performance beyond more involved, state-of-the-art systems.
- 6. Conclusions: The analysis may also improve networks trained for other tasks as an alternative to growing networks or using more data.The conclusion specifically points to face recognition as a potential application of this idea.
- 6. Conclusions: Face-recognition CNN clusters capture identity-related attributes such as facial hair and ethnicity better than the landmark-regression clusters.