Source-linked AI summary
Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors
Xuanyi Dong, Shoou-I Yu, Xinshuo Weng, Shih-En Wei, Yi Yang, Yaser Sheikh
TL;DR
Facial landmark detectors are limited by imprecise annotations and can jitter across video frames. Supervision-by-Registration adds optical-flow consistency from unlabeled video to detector training, with experiments reporting improved precision on images and video and reduced video jitter.
Problem
Facial landmark detectors can have high variance because training data and human annotations are limited or imprecise, causing jitter in independently processed video frames.
Method
Supervision-by-Registration augments annotation-based detector training with a registration loss that enforces consistency between adjacent-frame detections and optical flow from unlabeled video.
Results
The method improves facial landmark detection on images and video and significantly reduces jitter in video detections.
Takeaways & Limitations
SBR removes dependence on the quantity and quality of human annotations and produces detectors that are more precise and inherently more stable without test-time tracking or post-processing.
Takeaways & Limitations
Performance improvement is not related to face size, while occlusion and head pose may have greater influence.
Abstract
from arXiv · showhide
In this paper, we present supervision-by-registration, an unsupervised approach to improve the precision of facial landmark detectors on both images and video. Our key observation is that the detections of the same landmark in adjacent frames should be coherent with registration, i.e., optical flow. Interestingly, the coherency of optical flow is a source of supervision that does not require manual labeling, and can be leveraged during detector training. For example, we can enforce in the training loss function that a detected landmark at frame$_{t-1}$ followed by optical flow tracking from frame$_{t-1}$ to frame$_t$ should coincide with the location of the detection at frame$_t$. Essentially, supervision-by-registration augments the training loss function with a registration loss, thus training the detector to have output that is not only close to the annotations in labeled images, but also consistent with registration on large amounts of unlabeled videos. End-to-end training with the registration loss is made possible by a differentiable Lucas-Kanade operation, which computes optical flow registration in the forward pass, and back-propagates gradients that encourage temporal coherency in the detector. The output of our method is a more precise image-based facial landmark detector, which can be applied to single images or video. With supervision-by-registration, we demonstrate (1) improvements in facial landmark detection on both images (300W, ALFW) and video (300VW, Youtube-Celebrities), and (2) significant reduction of jittering in video detections.
1. Introduction
Facial landmark detectors can be accurate yet imprecise because of limited training data and inconsistent human annotations, producing jitter across video frames. Supervision-by-Registration augments annotation-based training with optical-flow supervision from unlabeled video to improve precision and temporal stability.
- Motivation: Imprecise landmarks can degrade face alignment and recognition, while temporal jitter can propagate to 3D face meshes and create perceptually jarring animation or reenactment.
- Motivation: Facial landmark detection remains unsolved because detectors may have low bias but high variance, partly from insufficient samples and imprecise annotations.Independent frame-by-frame detection can produce jitter and landmarks that do not remain aligned with anatomically defined points over time.
- Supervision-by-Registration: Supervision-by-Registration augments the training loss with supervision automatically extracted from unlabeled videos, complementing imprecise human annotations.A landmark tracked from frame t−1 with optical flow should coincide with its detection at frame t; the mismatch supplies temporal supervision.
- Supervision-by-Registration: A differentiable Lucas-Kanade operation estimates tracked landmark locations, compares them with current detections, and back-propagates registration-loss gradients end to end.Registration supervision is applied only to landmarks whose optical flow passes a forward-backward reliability check.
- Contributions: SBR can enhance a generic detector on images and video without annotations, use large amounts of unlabeled video, and train with standard gradient back-propagation.
2. Related Work
Prior video landmark methods combine detection with temporal tracking or recurrent modeling, but commonly depend on costly per-frame annotations. SBR instead leverages temporal coherency without video-level annotation and dynamically refines the detector from tracking results.
- Video landmark detection: Frame-by-frame image detectors often yield jittering and unstable video landmarks, motivating methods that use temporal information.
- Existing approaches: Hybrid video methods combine tracking-by-detection with temporal information, including recurrent neural networks and joint detection-tracking optimization.
- Limitations of existing approaches: Existing hybrid methods require per-frame annotations, which are resource-intensive to acquire.
- Supervision-by-Registration: SBR leverages temporal coherency without video-level annotation, enabling detector enhancement from large numbers of unlabeled videos.
- Detector refinement: Unlike approaches using offline-trained static appearance models, SBR dynamically refines the detector based on Lucas-Kanade tracking results.
3. Methodology
SBR combines a general facial landmark detector with a differentiable Lucas–Kanade operation, using labeled-image detection loss and unlabeled-video registration loss to encourage temporal coherence. Forward-backward tracking assesses reliability so only dependable registrations supervise the detector.
- Core framework: SBR back-propagates through a fully differentiable LK operation, enabling detector gradients that encourage temporal coherence across adjacent frames.The detector and LK operation form the two complementary components of the framework.
- LK operation: Feature patches are weighted by distance from the landmark center, while RGB images or convolutional features can serve as LK inputs.The weighting downweights pixels farther from the patch center.
- LK operation: The LK operation estimates motion from adjacent-frame features and a previous landmark location, iteratively updating motion parameters to output the corresponding current-frame coordinates.The displacement warp uses two motion parameters, initialized at zero and updated until convergence.
- Training losses: The training objective combines human-annotation detection loss with unsupervised registration loss, controlled by weight parameter γ.The detection loss is based on predicted and ground-truth landmark locations, while heatmap-based detectors require a differentiable peak-finding operation.
- Registration loss: Registration loss minimizes the distance between LK-tracked landmarks and direct detector predictions in future frames.This forward communication encourages temporal consistency between the two predictions.
- Reliability assessment: A forward-backward check assigns reliability to tracked landmarks, preventing supervision when LK tracking fails.Reliable tracks return to the same prior location after reverse tracking; the reliability indicator βt,i controls their contribution.
4. Evaluation and Results
Experiments evaluate SBR across image and video datasets using CPM and regression-based detectors, showing improved accuracy, temporal coherence, and state-of-the-art results without additional annotations.
- Evaluation on Image Datasets: SBR improves CPM by 7% and Reg by 6% on 300-W without additional annotation.On AFLW, SBR improves CPM by 9% and Reg by 5%.
- Evaluation on Image Datasets: SBR improves both CPM and regression networks on 300-W and AFLW, achieving state-of-the-art performance with CPMs.YouTube-Face supplies the unlabeled training videos, although its smaller faces and compression artifacts may limit tracking quality.
- Evaluation on Video Datasets: SBR improves CPM by 1% on 300-VW, while PAM adds a further 1.2% improvement and achieves state-of-the-art performance.The reported t-test p-values are 0.0316 for SBR and 0.0001 for PAM; neither uses additional annotations.
- Evaluation on Video Datasets: PAM reduces CPM error from 5.26 to 4.74 on YouTube Celebrities, achieving state-of-the-art performance.The baselines are pretrained on 300-W, and PAM is evaluated on six video clips.
- Evaluation on Video Datasets: Qualitative results show CPM predictions can drift across frames, whereas CPM+SBR/PAM produces more stable predictions.The method is evaluated by sampling predictions every 10 frames from 300VW videos.
5. Discussion
The discussion examines how tracking duration, image features, annotation error, and implementation details affect SBR’s registration-based training and evaluation. It also reports limitations in interpreting performance and in the gradients used for training.
- Image Resolution: Higher face resolution usually increased the chance of passing the forward-backward check, while performance improvement was not related to face size.Occlusion and head pose may have greater influence on improvement.
- Temporal Length for Tracking: 5.01 NME with five-frame LK tracking was worse than 4.74 NME with three-frame tracking on YouTube Celebrities.Longer tracking makes the forward-backward check stricter, reducing landmarks included in the registration loss.
- Image Features for Tracking: Using conv-1 features for LK tracking increased YouTube Celebrities error from 4.74 to 5.13 NME.The paper suggests convolutional features may lose information useful for LK tracking.
- Effect of imprecise annotation: SBR and PAM showed small NME and AUC improvements but significant reduction of video jittering.The paper attributes this difference partly to frame-independent metrics and imprecise testing annotations.
- Gradient propagation: Ignoring gradients through the tracked detections caused detector drift and increased YouTube Celebrities error from 4.74 to 5.45 NME.The missing gradients compromise the correctness of the parameter gradient used to generate both direct and tracked detections.
6. Conclusion
The conclusion presents SBR as an annotation-independent way to improve facial landmark detectors by using registration signals from unlabeled video. It emphasizes scalability beyond human annotation limits and end-to-end gradient-based training.
- 6. Conclusion: SBR does not rely on imprecise human annotations and is not limited by their quantity or quality.The conclusion identifies these properties as advantages of the method.
- 6. Conclusion: Back-propagating through the LK layer enables more accurate gradient updates than self-training.This is presented as a further advantage of SBR.