Source-linked AI summary
Pixel-in-Pixel Net: Towards Efficient Facial Landmark Detection in the Wild
Haibo Jin, Shengcai Liao, Ling Shao
TL;DR
Facial landmark detection needs accurate, efficient, robust models that generalize across domains despite heatmap regression’s computational cost and limited shape constraints. PIPNet combines low-resolution score-and-offset prediction, neighbor regression, and curriculum self-training, achieving strong benchmark and cross-domain results while remaining lightweight.
Problem
Facial landmark detection seeks accurate, robust, and efficient predictions, but heatmap regression is computationally expensive and cross-domain gaps remain.
Method
PIPNet combines low-resolution PIP regression, neighbor regression for local constraints, and curriculum-based self-training on unlabeled cross-domain data.
Results
PIPNet achieves state-of-the-art results on three of six supervised benchmarks and consistently improves two cross-domain test sets.
Takeaways & Limitations
PIPNet provides an efficient, accurate, and robust facial landmark detector for deployment on lightweight devices.
Takeaways & Limitations
Cross-domain evaluation depends on consistent face crops, while network stride remains a PIP regression hyperparameter.
Abstract
from arXiv · showhide
Recently, heatmap regression models have become popular due to their superior performance in locating facial landmarks. However, three major problems still exist among these models: (1) they are computationally expensive; (2) they usually lack explicit constraints on global shapes; (3) domain gaps are commonly present. To address these problems, we propose Pixel-in-Pixel Net (PIPNet) for facial landmark detection. The proposed model is equipped with a novel detection head based on heatmap regression, which conducts score and offset predictions simultaneously on low-resolution feature maps. By doing so, repeated upsampling layers are no longer necessary, enabling the inference time to be largely reduced without sacrificing model accuracy. Besides, a simple but effective neighbor regression module is proposed to enforce local constraints by fusing predictions from neighboring landmarks, which enhances the robustness of the new detection head. To further improve the cross-domain generalization capability of PIPNet, we propose self-training with curriculum. This training strategy is able to mine more reliable pseudo-labels from unlabeled data across domains by starting with an easier task, then gradually increasing the difficulty to provide more precise labels. Extensive experiments demonstrate the superiority of PIPNet, which obtains state-of-the-art results on three out of six popular benchmarks under the supervised setting. The results on two cross-domain test sets are also consistently improved compared to the baselines. Notably, our lightweight version of PIPNet runs at 35.7 FPS and 200 FPS on CPU and GPU, respectively, while still maintaining a competitive accuracy to state-of-the-art methods. The code of PIPNet is available at https://github.com/jhb86253817/PIPNet.
1 Introduction
Facial landmark detectors must balance accuracy, robustness, and efficiency, but existing detection heads trade these properties against one another and cross-domain generalization remains difficult. PIPNet addresses these challenges with low-resolution score-and-offset prediction, neighbor regression, and curriculum-based self-training.
- Online facial analysis requires landmark detectors that are accurate, robust, and computationally efficient in uncontrolled environments.
- Heatmap regression is accurate but computationally expensive and sensitive to outliers, whereas coordinate regression is fast and robust but less accurate.
- Cross-domain gaps remain a challenge, while supervised generalization depends on human-designed modules and costly annotations.
- PIP regression predicts landmark scores and within-grid offsets on low-resolution feature maps, eliminating repeated upsampling without sacrificing accuracy.
- PIPNet combines PIP regression, neighbor regression, and self-training with curriculum to improve efficiency, robustness, and cross-domain generalization.
- Neighbor regression fuses predictions from neighboring landmarks to add local shape constraints and improve robustness with marginal extra cost.
2 Related Work
Prior facial landmark work centers on coordinate and heatmap regression, while cross-domain methods primarily use supervised learning. PIPNet extends these directions with a low-resolution hybrid head and semi-supervised cross-domain training.
- Coordinate regression directly predicts landmark coordinates but is inaccurate and biased, so it is often cascaded or augmented with additional modules.
- Heatmap regression predicts landmark-location probabilities on high-resolution heatmaps and commonly relies on architectures that generate those resolutions.
- PIP regression differs from prior heatmap-offset methods by operating on low-resolution maps and unifying heatmap and coordinate regression as special cases.
- The neighbor regression module targets landmark consistency, unlike prior offsets designed mainly for accuracy improvement and keypoint grouping.
- Existing cross-domain landmark methods use supervised learning, whereas PIPNet uses generalizable semi-supervised learning to exploit unlabeled data.
- Prior semi-supervised landmark methods use transformation consistency, adversarial training, or teacher-student self-training, while PIPNet follows a different approach.
3 Our Method
PIPNet combines low-resolution score-and-offset prediction with neighbor-based local constraints and curriculum self-training to improve efficiency, robustness, and cross-domain generalization.
- PIP Regression: PIP regression predicts landmark scores and within-grid offsets on low-resolution feature maps, avoiding repeated upsampling while retaining precise localization.Its score and offset predictions are computed independently in a single stage, and a 1 × 1 convolutional layer can implement the head.
- PIP Regression: Ground-truth landmarks assign one positive grid on the score map, while corresponding x- and y-offset maps encode sub-grid localization.For a 256 × 256 image with stride 32, the feature map is 8 × 8 and contains 64 grids per channel.
- PIP Regression: PIP regression generalizes coordinate and heatmap regression by varying network stride and selectively removing score or offset prediction.At image-sized stride it becomes coordinate regression; at stride near one without offsets it becomes heatmap regression.
- Neighbor Regression Module: PIP regression remains vulnerable to inconsistent shapes under extreme poses, motivating neighbor regression to impose local landmark constraints.Coordinate regression preserves more reasonable global shapes but loses detail, whereas heatmap and PIP regression can produce shape inconsistencies in difficult poses.
- Neighbor Regression Module: The neighbor regression module predicts offsets for nearby landmarks and averages their inferred locations with each landmark’s own prediction during inference.Neighbors are selected from the closest landmarks in the training-set mean shape, adding only marginal computational cost.
- Self-Training with Curriculum: Adding NRM improves PIP regression qualitatively and quantitatively, including prediction consistency and NME, while STC addresses remaining cross-domain instability using unlabeled data.The supplied passages also note that consistent face crops matter during cross-domain evaluation or adaptation because benchmark bounding boxes differ in style.
4 Experiments
Section 4 presents the experimental program, covering settings, hyperparameter selection, model analysis, supervised and cross-domain performance, and inference speed.
- The experiments proceed from setup and hyperparameter selection to model analysis, supervised and cross-domain evaluation, and inference-speed comparisons.
4.1 Experimental Settings
The experiments use multiple facial landmark benchmarks, standardized preprocessing and backbones, three cross-domain paradigms, and normalized mean error for evaluation.
- Datasets: CelebA images are used as unlabeled data for the cross-domain experiments.
- Implementation: Images are cropped and resized to 256 × 256, with landmark-dependent bounding-box enlargement and ResNet backbones used by default.ResNet-18 is the default backbone, with ResNet-50 and ResNet-101 also evaluated.
- Cross-Domain Setting: Cross-domain evaluation compares generalizable supervised learning, unsupervised domain adaptation, and generalizable semi-supervised learning using 300W labels and 300W, COFW-68, and WFLW-68 test sets.
- Evaluation: Normalized mean error is the primary metric, with inter-ocular, image-size, or face-diagonal normalization selected by dataset.
4.2 Hyperparameters
The hyperparameter studies vary network stride and neighbor count on a WFLW validation split, identifying a default neighbor count that improves performance consistently.
- Validation Protocol: A WFLW validation set of 1,500 randomly selected training images is used to tune network stride and neighbor count.The remaining training images form the sub-training set.
- Network Stride: Network stride is varied by adding convolutional layers for larger strides or replacing them with deconvolutional layers for smaller strides.
- Neighbor Count: The neighbor regression module consistently improves performance, with the best NME occurring around C = 10.C = 10 is used for the remaining experiments by default.
4.3 Model Analysis
The model analysis compares PIPNet with coordinate- and heatmap-regression baselines through accuracy, variance, and bias-variance behavior across multiple datasets and strides.
- Baselines: The analysis evaluates coordinate regression, heatmap regression, and PIP regression under the generalizable supervised learning paradigm.
- Bias-Variance Trade-Off: Coordinate regression is robust but inaccurate, whereas heatmap regression is accurate on most samples but sensitive to unusual samples.
- Bias-Variance Trade-Off: PIPNets without NRM generally retain lower bias than MapNets as network stride increases, while their variance decreases with larger stride.
- Baseline Comparison: PIPNet achieves the lowest NME across the three evaluated datasets, while PIPNet-32 has variance comparable to CoordNet.The authors attribute the robustness improvement to the neighbor regression module.
- Summary: The analysis characterizes PIP regression with NRM as efficient, accurate, and robust, combining advantages associated with coordinate and heatmap regression.
4.4 Comparison with State of the Arts
PIPNet achieves competitive or state-of-the-art landmark detection across multiple benchmarks, including diverse in-the-wild, large-pose, and video settings. Its strongest results occur with ResNet-101, while lightweight variants remain competitive.
- 3.36 NME: the lightweight ResNet-18 model obtains a competitive result on the full 300W test set.It also outperforms several methods using external training data, including TS3, LaplaceKL, and DeCaFA.
- 3.08 NME: PIPNet with ResNet-101 achieves state of the art on COFW, outperforming HRNet's 3.45 NME.
- 4.31 NME: PIPNet with ResNet-101 achieves state of the art on the full WFLW set and three subsets.Lightweight MobileNetV3 and ResNet-18 models already achieve comparable performance to state-of-the-art methods.
- 1.27 versus 1.20 and 1.89 versus 1.72: PIPNet with ResNet-101 is slightly worse than the Menpo challenge winner on semi-frontal and profile tracks.The PIPNet models were trained without specific Menpo adaptation, whereas the winner used extra detectors for preprocessing.
- All three 300VW categories: PIPNet with ResNet-101 outperforms existing methods without temporal information.The model significantly outperforms prior work on categories 1 and 3, indicating accuracy and robustness advantages.
4.5 Self-Training with Curriculum
Self-training improves cross-domain landmark detection, and curriculum-based self-training yields further gains under both target-domain adaptation and generalized settings. The generalized setting shows that unlabeled data need not come from the target domains.
- 5.9% and 10.0%: STC reduces NME on COFW-68 and WFLW-68 under UDA, exceeding standard self-training's 4.6% and 7.9% reductions.The 300W NME also improves slightly by 0.6%.
- Target-domain unlabeled data is not always available, and target domains may be unknown in real applications.These conditions motivate evaluation beyond the UDA paradigm.
- GSSL: standard self-training consistently outperforms the GSL baseline even when unlabeled CelebA images come from neither target domain.STC outperforms standard self-training on all test sets under GSSL.
4.6 Speed
PIPNet offers a favorable speed-accuracy trade-off by avoiding costly high-resolution detection heads. Its ResNet-18 version is real-time on CPU while retaining competitive accuracy.
- 35.7 FPS: PIPNet with ResNet-18 is the only evaluated model running in real time on CPU while maintaining competitive state-of-the-art accuracy.
- High-resolution feature-map computations make competing approaches slower on GPU, while heavy detection heads slow MapNet on CPU.
- CoordNet is faster than PIPNet, but its accuracy is significantly worse because of biased predictions.
5 Conclusion
PIPNet combines a lightweight pixel-in-pixel detection head, neighbor regression, and curriculum-based self-training to target efficiency, robustness, and cross-domain generalization. Experiments characterize it as an efficient, accurate, and robust detector for lightweight devices.
- PIP regression predicts low-resolution score heatmaps and within-pixel offsets, eliminating repeated upsampling layers.This reduces computational cost, especially on lightweight devices.
- Neighbor regression fuses predictions from neighboring landmarks to improve robustness through local shape constraints.
- Self-training with curriculum gradually increases pseudo-labeling difficulty to introduce fewer errors and improve cross-domain generalization.
- PIPNet is an efficient, accurate, and robust facial landmark detector that can run in the wild on lightweight devices.