Source-linked AI summary
Efficient Object Localization Using Convolutional Networks
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, Christopher Bregler
TL;DR
Pooling makes ConvNets efficient and invariant but reduces localization precision. The paper introduces a jointly trained coarse-to-fine cascade with shared convolutional features and a refinement model, achieving new state-of-the-art results on FLIC and MPII-human-pose.
Problem
Pooling reduces computational requirements and promotes invariance, but its loss of spatial resolution limits human-joint localization accuracy.
Method
A jointly trained cascaded ConvNet combines coarse heat-map localization with a fine refinement model using shared convolutional features and parameter sharing.
Results
New state-of-the-art results were achieved on the FLIC and MPII-human-pose datasets.
Takeaways & Limitations
Pooling-related precision loss can be recovered efficiently while retaining the computational benefits of pooling.
Takeaways & Limitations
Wrist accuracy does not improve beyond the coarse approximation under limited convolution sizes and sampling context, while one cascade layer was sufficient within FLIC label noise.
Abstract
from arXiv · showhide
Recent state-of-the-art performance on human-body pose estimation has been achieved with Deep Convolutional Networks (ConvNets). Traditional ConvNet architectures include pooling and sub-sampling layers which reduce computational requirements, introduce invariance and prevent over-training. These benefits of pooling come at the cost of reduced localization accuracy. We introduce a novel architecture which includes an efficient `position refinement' model that is trained to estimate the joint offset location within a small region of the image. This refinement model is jointly trained in cascade with a state-of-the-art ConvNet model to achieve improved accuracy in human joint location estimation. We show that the variance of our detector approaches the variance of human annotations on the FLIC dataset and outperforms all existing approaches on the MPII-human-pose dataset.
1. Introduction
ConvNets advance human-body localization, but pooling improves efficiency and invariance at the cost of spatial precision. The paper introduces a cascaded architecture that combines coarse localization with refinement to retain precision while using more pooling.
- 1. Introduction: Pooling layers reduce spatial resolution and promote invariance to local transformations, especially translations.Their outputs summarize local regions and are invariant to spatial location within each pooling region.
- 1. Introduction: The spatial invariance of pooling limits localization accuracy, creating a trade-off among generalization, model size, and spatial precision.
- 1. Introduction: The proposed ConvNet localizes skeletal joints in monocular RGB images with high spatial accuracy and no significant computational overhead.
- 1. Introduction: The architecture uses increased pooling for computational efficiency while retaining high spatial precision.
- 1. Introduction: A coarse ConvNet produces low-resolution per-pixel joint heat maps, while a jointly trained cascade reuses hidden convolutional features to improve localization accuracy.
- 1. Introduction: The model outperforms recent state-of-the-art methods on standard human-body-pose datasets.
2. Related Work
Prior human-pose systems include deformable-part, template, dependency, and ConvNet-based approaches. The paper distinguishes its heat-map and shared-feature cascade from direct regression and independently trained coarse-to-fine models.
- 2. Related Work: Earlier human-pose work developed deformable-part models, mixture-of-template methods, and cascades of body-part detectors.
- 2. Related Work: High-order part-dependency models use Poselet priors, spatial relationships, or semiglobal classifiers, with some approaches demonstrated only for arms.
- 2. Related Work: ConvNet pose methods include direct continuous regression, low-dimensional representations with spatial models, and multi-resolution heat-map regression.
- 2. Related Work: Direct regression can perform poorly in the high-precision region and has limited output flexibility for multimodal joint locations.
- 2. Related Work: Unlike independently trained coarse-to-fine models, this work jointly trains a shared-feature cascade and samples feature inputs to improve runtime performance.
3. Coarse Heat-Map Regression Model
The coarse model uses a multi-resolution ConvNet sliding-window detector to produce joint heat-maps, while dropout and SpatialDropout address generalization. Training matches predicted heat-maps to Gaussian targets, and torso-conditioned MRF inference selects the annotated person.
- Model Architecture: A multi-resolution ConvNet sliding-window detector uses overlapping contexts to produce a coarse heat-map output.The input is an RGB Gaussian pyramid, and each joint receives a per-pixel likelihood heat-map.
- Model Architecture: Pooling reduces heat-map resolution relative to the input while supporting dimensionality reduction and invariance to noise and local transformations.The architecture uses seven stages, or eleven for the MPII model.
- SpatialDropout: Standard dropout independently drops convolution activations but fails to prevent over-training when feature-map activations are strongly correlated.The method can increase training time while leaving correlated activations and gradients largely dependent.
- SpatialDropout: SpatialDropout performs n_feats dropout trials and extends each dropout value across an entire feature map, improving performance especially on the small FLIC training set.Adjacent pixels in a dropped feature map are therefore either all zero or all active.
- Training and Data Augmentation: Training minimizes MSE between predicted heat-maps and 2D Gaussian targets centered at ground-truth joint locations, with σ ≈1.5 pixels.The predicted and ground-truth heat-maps are defined per joint; inputs are also randomly rotated, scaled, and flipped during training.
4. Fine Heat-Map Regression Model
The fine heat-map model restores spatial precision lost to pooling by refining coarse joint locations with cropped convolutional features. Shared features and joint training reduce cascade cost while regularizing the coarse model.
- Model Architecture: The architecture adds a fine ConvNet to recover spatial accuracy lost through pooling while reusing convolutional features from coarse localization.Reusing features reduces trainable parameters and jointly regularizes the coarse heat-map model.
- Model Architecture: The system crops convolutional features around each coarse joint location, then predicts a local (∆x, ∆y) offset for refinement.Coarse locations come from each joint heat-map maximum before feature cropping and fine-model inference.
- Model Architecture: The crop module preserves contextual size across resolution levels by scaling the cropped area at higher resolutions.Its gradients are added back to the coarse model at the sampled feature locations during back-propagation.
- Model Architecture: The fine model uses a Siamese network with shared weights across joint-specific instances, followed by unshared 1x1 convolutions that produce detailed heat-maps.The MPII configuration uses 14 instances; parameter sharing reduces shared parameters and helps prevent over-training.
- Model Architecture: A single refinement cascade level provides sufficient accuracy within the label-noise level observed on FLIC.Although additional levels are possible, the authors report sufficient accuracy with one level in practice.
- Joint Training: The cascade is pretrained in stages and then jointly optimized with E3 = E1 + λE2, using heat-map regression because argmax is not differentiable.The fine-model loss compares predicted and ground-truth heat-maps, while λ trades off the two subtasks.
5. Results
The experiments show that pooling improves efficiency but reduces fine localization precision, while the cascaded model recovers precision and achieves strong results on FLIC and MPII.
- Pooling and localization: More pooling reduces detection performance at small distance thresholds because coarse quantization limits spatial precision.The coarse heat-map model was evaluated with varying pooling amounts while keeping convolution features constant.
- FLIC results: 8× pooling achieves near-label-noise accuracy with lower forward-propagation time than the 4× cascaded network.The comparison uses FLIC models and reports single-image FPROP time on an Nvidia-K40 GPU.
- FLIC results: Wrist improvements are significant only for the 8× and 16× pooling models.The experiments attribute the wrist behavior to the need for broad spatial context under substantial skeletal deformation and input variability.
- Cascade ablations: Jointly trained shared features outperform a greedily trained cropped-image cascade on the wrist joint.The comparison model was capacity-matched with an additional convolution layer, while the proposed cascade reused coarse-model features.
- Regularization: SpatialDropout improves performance in the high normalized-distance region by reducing strong heat-map outliers.The comparison is reported for the wrist joint.
- Benchmark comparisons: At high precision, the cascaded network outperforms prior state of the art on FLIC, while the model outperforms all existing methods on MPII.FLIC comparisons use averaged wrist-and-elbow PCK; MPII comparisons use PCKh.
- Scale robustness: Training on unnormalized image scales degrades performance but still outperforms state of the art, indicating learned scale invariance.The unnormalized model is evaluated on both normalized and unnormalized MPII test sets.
6. Conclusion
The paper recovers spatial precision lost to pooling while retaining its computational benefits through a cascaded architecture combining coarse and fine convolutional networks.
- Conclusion: The cascaded architecture combines fine- and coarse-scale convolutional networks to recover pooling-related precision efficiently.The paper reports new state-of-the-art results on FLIC and MPII-human-pose.