Source-linked AI summary
Improving Landmark Localization with Semi-Supervised Learning
Sina Honari, Pavlo Molchanov, Stephen Tyree, Pascal Vincent, Christopher Pal, Jan Kautz
TL;DR
The paper tackles landmark localization when precise landmark labels are scarce but related class or attribute labels are more plentiful. It combines sequential multitasking, which backpropagates auxiliary-task errors through localization, with equivariant landmark transformation training. The techniques achieve accurate landmark prediction with fewer labeled landmarks and new state-of-the-art results on 300W and AFLW.
Problem
Precise landmark locations are available for only a small subset, while related class labels are easier to obtain and more abundant.
Method
The paper combines sequential multitasking through intermediate landmark predictions with unsupervised equivariant landmark transformation training.
Results
The approach achieves high accuracy with far fewer labeled landmark training examples and reports state-of-the-art performance on 300W and AFLW.
Takeaways & Limitations
Effective landmark predictors can be learned from partially annotated datasets by combining auxiliary attributes with transformation-based supervision.
Takeaways & Limitations
On the Blocks dataset, correspondence between classification labels and landmark identities is more tenuous, so labeled ground-truth landmarks are introduced for evaluation.
Abstract
from arXiv · showhide
We present two techniques to improve landmark localization in images from partially annotated datasets. Our primary goal is to leverage the common situation where precise landmark locations are only provided for a small data subset, but where class labels for classification or regression tasks related to the landmarks are more abundantly available. First, we propose the framework of sequential multitasking and explore it here through an architecture for landmark localization where training with class labels acts as an auxiliary signal to guide the landmark localization on unlabeled data. A key aspect of our approach is that errors can be backpropagated through a complete landmark localization model. Second, we propose and explore an unsupervised learning technique for landmark localization based on having a model predict equivariant landmarks with respect to transformations applied to the image. We show that these techniques, improve landmark prediction considerably and can learn effective detectors even when only a small fraction of the dataset has landmark labels. We present results on two toy datasets and four real datasets, with hands and faces, and report new state-of-the-art on two datasets in the wild, e.g. with only 5\% of labeled images we outperform previous state-of-the-art trained on the AFLW dataset.
1. Introduction
The paper addresses scarce precise landmark annotations by using abundant related labels and transformation-based supervision. It proposes sequential multitasking and equivariant landmark learning to improve localization through differentiable, end-to-end training.
- Overall contribution: The combined approach is designed for efficient training when labeled landmark pairs are much fewer than attribute examples and available images.The paper summarizes this regime as S ≪ M ≤ N.
- Motivation: Precise landmark annotation is laborious, while related class labels are more abundant and easier to obtain.Neural networks generally require large labeled datasets to handle natural-image variation, but few landmark datasets provide enough precise annotations.
- Sequential multitasking: Sequential multitasking routes classification predictions through intermediate landmark locations, allowing classification errors to supervise localization during backpropagation.Unlike parallel auxiliary branches, the proposed architecture forces classification information through the landmark-localization components.
- Sequential multitasking: The landmark localizer uses pixel-level heatmaps without pooling or strided convolutions, then extracts coordinates with soft-argmax for full differentiability.Predicted x,y landmark pairs are passed to a multilayer perceptron for attribute regression or classification.
- Equivariant supervision: Equivariant landmark transformation trains the model to transform predicted landmarks consistently with transformations applied to input images, without requiring landmark labels.The method can therefore use images with unlabeled landmarks during semi-supervised training.
2. Sequential Multi-Tasking
Sequential multi-tasking predicts landmarks first and uses those predicted locations to estimate attributes, creating a bottleneck that routes attribute-learning signals through landmark localization. Soft-argmax makes this pipeline differentiable and trainable end-to-end.
- Architecture: Sequential multi-tasking first predicts landmark locations, then uses them as input for attribute classification or regression.The architecture creates a landmark bottleneck between image features and attribute estimation.
- Architecture: The bottleneck forces classification predictions to flow through landmark localization, providing attribute-derived supervision during backpropagation.This differs from parallel multi-task architectures with separate landmark and auxiliary-task branches.
- Differentiable localization: Soft-argmax converts convolutional landmark maps into differentiable 2D coordinates for end-to-end training.It computes each coordinate as a softmax-weighted average of pixel locations.
- Training objective: The attribute objective uses negative log-likelihood for classification or absolute error for regression.The predicted attributes are computed from the landmark coordinates.
3. Equivariant Landmark Transformation
Equivariant landmark transformation trains landmark predictors to produce transformed coordinates consistently when input images undergo affine transformations. The consistency loss can be added across training images, including those without landmark annotations.
- Equivariance constraint: Equivariant landmark transformation requires predictions on transformed images to match the same transformation applied to the original predicted landmarks.For transformation T, the intended relation is L(T ⊙ I) ≈ T ⊙ L(I).
- Training procedure: The method applies small affine coordinate transformations to images and landmark coordinates during unsupervised training.Multiple transformed instances can contribute separate consistency terms to the training cost.
- Composite objective: The composite objective combines attribute loss, equivariant landmark loss, supervised landmark distance, and parameter regularization.The equivariant loss applies to any training image, whereas supervised landmark loss uses only images with landmark labels.
4. Experiments
Experiments on toy and real datasets evaluate sequential multitasking, equivariant landmark transformation, and competing multitask architectures under varying landmark-label availability. The proposed approach improves landmark localization, including with sparse labels, and reaches strong results on 300W and AFLW.
- Shapes dataset: 99% classification accuracy on Shapes accompanied consistent landmark assignments adjacent to the triangle and square.The experiment used the sequential architecture, with landmarks feeding the classification task.
- Blocks dataset: Adding ELT cost made landmark assignments more consistent within Blocks classes, but correspondence did not extend reliably across different classes.The authors therefore introduced ground-truth landmark locations and evaluated localization with different labeled-data percentages.
- Blocks dataset: Seq-MT with class labels improved over landmark-only training, ELT improved results further, and combining landmark, ELT, and class-label costs performed best.The comparison was conducted on the Blocks dataset; the strongest model was also illustrated with only 5% landmark-labeled data.
- Datasets in the wild: On 300W, ELT with soft-argmax reduced RCN error from 5.54 to 5.1, while the RCN+ variant achieved an AFLW state-of-the-art error of 1.59 with 27% relative improvement.With only 5% labeled landmark data, the method also outperformed previous state-of-the-art methods on AFLW.
5. Conclusion
The paper introduces two techniques for semi-supervised landmark localization: equivariant landmark transformation and auxiliary-attribute supervision through landmark localization. Together, they achieve high accuracy with fewer labeled landmarks and new state-of-the-art results on 300W and AFLW.
- The proposed approach combines equivariant landmark transformation with auxiliary attributes such as class labels for semi-supervised landmark localization.The architecture backpropagates attribute errors through landmark-localization components.
- The techniques achieve high accuracy with far fewer labeled landmark examples for hand and face landmark tasks.
- The method achieves new state-of-the-art performance on the 300W and AFLW public benchmarks for fiducial points in the wild.
S.5.1. Comparison on MTFL dataset
The MTFL evaluation compares landmark-localization architectures using a standardized protocol, including re-annotated AFLW and AFW test subsets. The comparison includes fully labeled training data.
- The MTFL dataset provides five facial landmarks: eye centers, the nose tip, and mouth corners.
- The evaluation follows the protocol of [13], using 9,000 training images and 1,000 validation images.
- Testing uses re-annotated AFLW and AFW subsets containing 29,995 and 337 images, respectively.
- Table S1 reports results on MTFL test sets with 100% labeled data.
S.5.2. Selecting auxiliary labels for semi-supervised learning
Auxiliary labels are selected according to their informational overlap with landmark coordinates. Hand-gesture labels and head-pose regression are useful, whereas wearing-glasses labels provide little relevant information.
- Adjusted Mutual Information (AMI) is proposed as a heuristic for selecting attributes before applying sequential multitasking.AMI ranges from 0 to 1 and measures the fraction of statistical overlap.
- The attribute-landmark AMI is computed by discretizing variables into at most 20 uniform levels and summing coordinate-wise AMI under coordinate independence.
- Hand-gesture labels and head-pose regression are among the most effective auxiliary attributes for the method.
- Wearing-glasses labels have little mutual information with landmarks, indicating limited usefulness in this semi-supervised setting.
- Ground-truth landmarks consistently outperform Seq-MT-estimated landmarks for predicting auxiliary attributes.
S.5.3. Comparison of softmax and soft-argmax
Soft-argmax outperforms softmax in the compared landmark-localization models because it can regress continuous coordinate values rather than classify pixels on a fixed grid.
- Soft-argmax outperforms softmax across the compared Heatmap-MT and RCN model variants.
- Softmax limits accuracy to the grid resolution because it performs classification over pixels.
- Soft-argmax can regress to any real number, enabling more accurate landmark coordinates.
- Table S4 directly compares softmax and soft-argmax for Heatmap-MT with landmark and attribute losses.
S.5.4. Supplementary results on Multi-PIE dataset
Supplementary Multi-PIE results examine how landmark supervision affects attribute classification and provide additional localization examples. More labeled landmarks improve emotion classification, while camera classification gains are smaller.
- More labeled landmarks improve emotion classification accuracy even when the number of image–class-label pairs remains unchanged.The reported effect is especially pronounced for emotion classification.
- Camera classification improves only slightly with additional landmark supervision because all models already achieve high accuracy.
- Additional Multi-PIE test examples provide further qualitative localization predictions.
S.5.5. Supplementary results on hands dataset
Supplementary hands results show that landmark supervision and the ELT cost improve classification accuracy. Additional examples illustrate landmark localization performance on the hands dataset.
- Classification accuracy on the hands test set improves with more labeled landmarks and with the ELT cost in Seq-MT models.The ELT component affects both landmark localization and classification accuracy.
- Heatmap-MT achieves better hands classification accuracy than the other multitasking approaches.
- Table S7 reports hands-test classification accuracy in percent, with higher values indicating better performance.
- Additional hands-dataset examples show landmark localization predictions.
S.5.6. Supplementary results on 300W dataset
Supplementary 300W results evaluate ELT with Seq-MT and RCN+ architectures and show improved localization accuracy. RCN+ with ELT and attribute supervision also supports reliable pose estimation from few labeled landmarks.
- S.5.6. Supplementary results on 300W dataset: ELT loss improves landmark localization accuracy in both Seq-MT and RCN+ architectures on 300W samples.
- S.5.6. Supplementary results on 300W dataset: RCN+ (L+ELT+A) achieves pose-estimation results close to a model trained from ground-truth landmarks across labeled-data percentages.Pose error is reported as the average of yaw, pitch, and roll, where lower is better.
- S.5.6. Supplementary results on 300W dataset: Figure S3 documents the RCN architecture used for 300W experiments, including pooling, convolution, upsampling, and feature concatenation layers.
- S.5.6. Supplementary results on 300W dataset: Additional 300W examples include accurate predictions, ELT-related improvements, and difficult cases with high error.The difficult examples identify image regions where landmarks are most affected.