Source-linked AI summary
Binarized Convolutional Landmark Localizers for Human Pose Estimation and Face Alignment with Limited Resources
Adrian Bulat, Georgios Tzimiropoulos
TL;DR
The paper addresses the computational cost and model size of CNN-based landmark localization for human pose estimation and face alignment. It studies binarization and introduces a hierarchical, parallel and multi-scale residual architecture tailored to binary networks. The proposed block improves performance over a same-parameter binary baseline while supporting operation on limited resources.
Problem
CNN landmark-localization methods are accurate but computationally expensive and too large for real-time or mobile applications, motivating efficient binarized CNNs.
Method
The paper evaluates binarization for localization and proposes a hierarchical, parallel and multi-scale residual block that improves binary-network architecture without relying on improved quantization.
Results
About 6% absolute performance improvement is reported over the baseline binary residual block when using the same number of parameters.
Takeaways & Limitations
The proposed architecture enhances representational power and supports efficient landmark localization on limited-resource systems.
Takeaways & Limitations
The study’s wider-block analysis is limited to a moderate increase in channels, up to 256, and excludes the much wider setting examined in related work.
Abstract
from arXiv · showhide
Our goal is to design architectures that retain the groundbreaking performance of CNNs for landmark localization and at the same time are lightweight, compact and suitable for applications with limited computational resources. To this end, we make the following contributions: (a) we are the first to study the effect of neural network binarization on localization tasks, namely human pose estimation and face alignment. We exhaustively evaluate various design choices, identify performance bottlenecks, and more importantly propose multiple orthogonal ways to boost performance. (b) Based on our analysis, we propose a novel hierarchical, parallel and multi-scale residual architecture that yields large performance improvement over the standard bottleneck block while having the same number of parameters, thus bridging the gap between the original network and its binarized counterpart. (c) We perform a large number of ablation studies that shed light on the properties and the performance of the proposed block. (d) We present results for experiments on the most challenging datasets for human pose estimation and face alignment, reporting in many cases state-of-the-art performance. Code can be downloaded from https://www.adrianbulat.com/binary-cnn-landmarks
1. Introduction
The paper targets accurate landmark localization with binarized CNNs that are lightweight enough for resource-limited applications. It studies binarization for human pose estimation and face alignment, then proposes architectural changes to recover performance while retaining efficiency.
- Motivation: CNN landmark-localization methods achieve remarkable accuracy but require high-end GPUs and models occupying hundreds of megabytes.These requirements make them unsuitable for real-time or mobile applications.
- Novelty: The paper is the first to study neural-network binarization for fine-grained landmark localization tasks, including human pose estimation and face alignment.It evaluates design choices, identifies bottlenecks, and investigates multiple ways to improve performance.
- Method: The proposed hierarchical, parallel and multi-scale residual block is designed for binary networks rather than improving the quantization procedure itself.The design increases receptive-field size and improves gradient flow while avoiding 1 × 1 convolutions.
- Results: About 6% absolute performance improvement is reported over the baseline binary residual block when the compared blocks use the same number of parameters.The architecture’s performance boost also generalizes to some extent to real-valued networks.
- Efficiency: The method builds on binarized weights and activations, replacing multiplications with bit-wise XOR operations to support efficient computation.The paper distinguishes this architectural strategy from prior work focused on improving quantization methods.
3. Background
The section introduces the bottleneck residual block used to control parameters and memory, and the hierarchical network that predicts landmark heatmaps through symmetric processing.
- The bottleneck block is designed to reduce parameters and keep the network memory footprint under control.
- This work uses the pre-activation bottleneck variant, placing batch normalization and activation before each convolutional layer.
- The residual block is the main building block of the HG architecture for landmark localization.
- HG predicts one heatmap per landmark in a fully convolutional fashion.
- HG extends earlier work with more symmetric top-down and bottom-up processing.
4. Method
The method analyzes why binarization harms landmark localization and develops architectural changes tailored to binary networks. Its final block combines hierarchical, parallel, and multi-scale processing with improved gradient flow and avoids costly or detrimental design choices.
- 4.1. Binarized HG: The binarized HG uses binary weights and activations while retaining real-valued first and last layers, but suffers a significant performance drop on MPII pose estimation.The final layer remains real-valued because dense heatmap prediction requires higher precision; these real-valued layers contain less than 0.01% of total parameters.
- 4.2. On the Width of Residual Blocks: Widening the bottleneck’s thin 3 × 3 layer from 128 to 256 channels improves performance but increases memory requirements and computational cost.The wider block removes the bottleneck by matching the first and last layers’ channel count.
- 4.3. On Multi-Scale Filtering: Binary filters have limited representational power, motivating multi-scale blocks that enlarge the effective receptive field through larger filters and max-pooling.The multi-scale design uses 3 × 3 filters and decomposed 5 × 5 filters across parallel branches while maintaining similar parameter counts.
- 4.4. On 1 × 1 Convolutions: Removing 1 × 1 convolutions increases performance over the baseline by more than 8% and outperforms the wider block with fewer parameters.The paper attributes the limitation to binary 1 × 1 filters’ restricted learning power and tendency to pass features with little modification.
5. Proposed vs Bottleneck
The proposed block is compared with the original bottleneck under matched-parameter conditions in both binary and real-valued networks. It provides a larger advantage for binary networks, where architectural design matters more than simply increasing width.
- 5. Proposed vs Bottleneck: Both blocks are compared with the same number of parameters in binary and real-valued networks.This controls parameter count while testing whether the proposed architecture itself improves performance.
- 5. Proposed vs Bottleneck: The proposed block largely outperforms a matched-parameter bottleneck in binary networks, while its advantage is smaller in real-valued networks.The comparison is reported across the binary and real-valued cases.
- 5. Proposed vs Bottleneck: Increasing width gives only a moderate gain for the original binary bottleneck despite raising its parameter count substantially.The wider bottleneck is created by increasing the thin layer to 256 channels, but its gain remains far below the proposed block’s performance.
- 5.1. Binary: For real-valued networks, performance can be increased more easily by increasing parameter count, whereas binary networks require a tailored block design.The proposed block still outperforms a similarly sized wider bottleneck in the real-valued comparison.
6. Ablation studies
The ablations show that augmentation, suitable losses, pooling, activations, and other architectural choices materially affect binary landmark-localization performance.
- Augmentation: 4% performance increase follows from data augmentation, which accommodates new poses and scale variations.The reported experiments evaluate augmentation on the binary network.
- Loss: The binary network predicts one heatmap per landmark and compares pixel-wise L2 loss with a sigmoid-based alternative.L2 gradients are usually small and binarization amplifies this issue, motivating the alternative loss.
- Efficiency: Up to 3.5x speedups and 39x memory compression are reported for the binary model versus the stated baselines.The speed comparison is against cuBLAS, while compression is against a single-precision Torch counterpart.
7. Comparison with state-of-the-art
The proposed binary architecture is evaluated across human pose estimation and challenging face-alignment datasets, with reported gains over real-valued state-of-the-art systems.
- Human Pose Estimation: A single real-weight HG network using the proposed block reaches the performance of [22], while the full-sized model slightly surpasses [22].The full-sized model uses 8 HG networks; the binary and real-valued versions still retain a performance gap.
- Face Alignment: The binarized network outperforms current state-of-the-art methods on AFLW, whose competitors use large real-valued CNNs.AFLW contains 25,993 faces annotated with up to 21 landmarks.
- Face Alignment: More than 10% improvement over is reported on AFLW-PIFA when evaluating both visible and occluded points.AFLW-PIFA contains 5,200 images with 34 three-dimensional points and balanced yaw-angle ranges.
- Face Alignment: More than 40% improvement over [40] is reported for large poses ([60°−90°]) on AFLW2000-3D.The method improves over the entire pose range, with a noticeably larger gain for large poses.
8. Conclusion
The paper concludes that its hierarchical, parallel, and multi-scale block improves binary CNN landmark localization while remaining efficient for limited-resource settings.
- Conclusion: The proposed block enhances representational power and enables stronger learned relations without excessively increasing network parameters.The architecture is tailored to binarized CNNs for human pose estimation and face alignment.
A1. Additional ablation studies
Additional ablations identify choices that improve binary-network performance and quantify the resulting computational and memory benefits.
- Pooling: Max-pooling outperforms average pooling by 4% in the binary network.Average pooling can produce slightly worse results in the reported binary-network ablation.
- ReLUs: 2% performance improvement results from adding a ReLU after each convolutional layer.The paper attributes this gain to added non-linearity, particularly useful when training very deep architectures.
- Performance: Up to 3.5x speedups are observed against cuBLAS, although CPU experiments were not conducted.The authors expect speed improvements of about 58x on a single-core CPU based on the same binarization method.
- Memory: 39x memory compression is achieved against the single-precision Torch counterpart by removing biases and packing 32 weights per variable.The authors state that biases have minimal or no performance impact.
A2. Additional face alignment results
Additional face-alignment results are reported on AFLW-PIFA and AFLW2000-3D using normalized mean error comparisons under visible-only and visible-plus-occluded landmark evaluations.
- Additional numerical results are provided for AFLW-PIFA and AFLW2000-3D.
- AFLW-PIFA: AFLW-PIFA results are compared using NME (%) on visible landmarks only.The table includes results for PIFA, RCPR and PAWF taken from prior work.
- AFLW-PIFA: AFLW-PIFA is also evaluated with NME (%) across all 34 landmarks, including visible and occluded points.
A3. Facial part segmentation experiment
The proposed block is tested on facial part segmentation using a seven-class dataset derived from face-alignment keypoints. The reused architecture changes only its output channels, and the binarized proposed block nearly reaches the real-valued network’s performance.
- Facial part segmentation experiment: A seven-class facial-part segmentation dataset was constructed by joining 68 face-alignment keypoints to enclose each facial component.The classes are skin, lower lip, upper lip, inner mouth, eyes, nose and background.
- Evaluation: The network was trained on approximately 3,000 300W images and tested on 600 Indoor and Outdoor competition-test images.
- Dataset construction: The ground-truth mask joins the 68 keypoints, with colors denoting the seven facial-part classes.
- Architecture: The landmark-localization architecture was reused for segmentation, changing only the final layer from 68 to 7 output channels.
- Results: The proposed binarized block significantly outperforms a similarly sized binary bottleneck network and almost matches the real-valued network.The reported advantage is attributed mainly to the block’s higher representation and learning capacity, especially under unusual poses, occlusions and challenging lighting.
- Results: Segmentation results report pixel accuracy, mean accuracy and mean intersection-over-union.
A4. Visual results
Qualitative results are presented for human pose estimation, face alignment and facial part segmentation. On 300W, the proposed binarized network is reported to nearly match the real-valued network and outperform the original binary network.
- Qualitative results cover human pose estimation, face alignment and facial part segmentation.
- Results are shown on AFLW2000-3D and MPII for the corresponding experiments.
- 300W: On 300W Indoor and Outdoor subsets, the proposed binarized network significantly outperforms the original binary network and almost matches the real-valued network.