Source-linked AI summary
Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
Jonathan Tompson, Arjun Jain, Yann LeCun, Christoph Bregler
TL;DR
Human pose estimation in monocular images is difficult because body-part dependencies and occlusions make joint localization challenging. The paper unifies a ConvNet part-detector with an MRF-inspired spatial model and jointly trains them, significantly outperforming existing architectures while remaining close to real time. The approach is less effective for highly articulated poses and datasets with broader pose variation.
Problem
Human pose estimation in monocular RGB images remains challenging because joint dependencies, occlusions, appearance variation, and viewpoint create a high-dimensional input space.
Method
The method combines a ConvNet that outputs per-pixel joint-likelihood heat maps with a higher-level spatial model that constrains joint connectivity and global pose consistency.
Results
The unified architecture significantly outperforms existing state-of-the-art models on FLIC and LSP, while spatial modeling adds 8 to 12% and unified training adds 4-5% detection rate at large radii thresholds.
Takeaways & Limitations
Jointly learning convolutional detection and graphical-model spatial reasoning provides a successful framework for human body pose recognition and runs close to real-time on commodity hardware.
Takeaways & Limitations
The simple spatial model is less effective for datasets with broader pose variation and for highly articulated poses.
Abstract
from arXiv · showhide
This paper proposes a new hybrid architecture that consists of a deep Convolutional Network and a Markov Random Field. We show how this architecture is successfully applied to the challenging problem of articulated human pose estimation in monocular images. The architecture can exploit structural domain constraints such as geometric relationships between body joint locations. We show that joint training of these two model paradigms improves performance and allows us to significantly outperform existing state-of-the-art techniques.
1 Introduction
Human pose estimation in monocular RGB images remains challenging because body structure, occlusion, appearance, and viewpoint create a high-dimensional search problem. The paper combines a ConvNet part-detector with a spatial model to exploit body-part relationships and improve performance.
- Monocular human joint localization remains difficult despite extensive prior work.Challenges include joint inter-dependencies, occlusions, body-shape and appearance variation, and unrestricted viewing angles.
- These factors create a high-dimensional input space in which naive search methods are intractable.
- Existing approaches broadly comprise deformable part models and deep-learning-based discriminative models.
- The proposed unified framework combines a ConvNet Part-Detector with a part-based Spatial-Model and jointly trains both components.The spatial model approximates loopy belief propagation and is back-propagated through during learning.
- The combined architecture improves performance and significantly outperforms existing state-of-the-art human pose recognition models.
2 Related Work
Prior work spans deformable part models, deep convolutional networks, and structured prediction methods. The paper positions its contribution as a successful joint neural-network and graphical-model approach for localizing human body parts.
- Deformable Part Models made pictorial-structure approaches tractable for unconstrained human pose estimation.
- Deep convolutional networks achieve state-of-the-art performance on human pose estimation benchmarks including FLIC and LSP.
- Direct regression of pose vectors can be inaccurate in the high-precision region because the image-to-pose mapping is highly nonlinear and difficult to learn.
- Joint training of neural networks and graphical models had been reported in segmentation and speech or language modeling, but not for detecting and localizing human body parts in images.
3 Model
The model combines a deep ConvNet part detector with a learned spatial model that approximates message passing to enforce pose constraints. The unified network is trained end to end, with fine-tuning reported to further improve performance.
- Convolutional Network Part-Detector: The ConvNet maps an RGB image to per-pixel heat maps representing likelihoods for human-skeleton joint locations.It uses a sliding-window architecture with multi-resolution inputs and overlapping receptive fields.
- Convolutional Network Part-Detector: Efficient dense feature-map computation reduces the redundant convolutions of standard sliding-window evaluation.The efficient architecture combines full-image convolution stages with multi-resolution and overlapping receptive fields.
- Convolutional Network Part-Detector: The practical detector simplifies the lower-resolution stage with a single convolution bank and upscaling, using three resolution banks with minimal empirical performance loss.The simplified design is no longer exactly equivalent to the original sliding-window network because lower-resolution features are decimated and replicated.
- Higher-Level Spatial-Model: The spatial model uses a fully connected pairwise graph whose learned distributions provide spatial constraints for body-part locations.The Part-Detector supplies unary potentials, while the spatial model learns priors and implicitly learns the graph structure.
- Higher-Level Spatial-Model: A single-round sum-product-like message-passing computation can suppress anatomically incorrect detections while retaining spatially plausible joint configurations.Global convergence is not guaranteed because the graph is not tree structured, although the inferred solution is reported as sufficiently accurate on the evaluated datasets.
- Higher-Level Spatial-Model: The spatial-model implementation uses energy-based log-space operations with SoftPlus and ReLU stages, then trains the modified formulation by back-propagation and SGD.The formulation is not exactly an MRF but is intended to encode the spatial constraints while improving numerical stability and gradient behavior.
- Unified Model: End-to-end back-propagation through the trained Part-Detector and Spatial-Model further improves performance during unified fine-tuning.The authors hypothesize that spatial constraints reduce the output dimension of possible heat-map activations, helping the detector localize precise target activations.
4 Results
The model outperforms existing state-of-the-art techniques on FLIC and LSP, while spatial reasoning, joint training, and multiple resolution banks improve detection performance.
- Evaluation Setup: 48 hours train the Part-Detector, 12 hours the Spatial-Model, and 51ms forward-propagation for one image through both networks.These timings were measured using a 12 CPU workstation with an NVIDIA Titan GPU.
- Evaluation Setup: FLIC contains 5003 movie images, while extended-LSP contains 10442 training and 1000 test images with more varied athletic poses.The datasets provide 2D ground-truth joint information for evaluation.
- Evaluation Setup: The FLIC-plus split removes training images sharing scenes with the test set, providing additional samples sufficiently independent of test images.This addresses unfair over-training caused by scene overlap in FLIC-full.
- Overall Performance: The model outperforms existing state-of-the-art techniques on both FLIC and LSP with a considerable margin.Results include FLIC elbow and wrist evaluations and LSP evaluation using person-centric coordinates.
- Ablations: 8 to 12% higher detection for large radii follows from adding the Spatial-Model, while unified training adds another 4-5%.The Spatial-Model has little impact at low radius thresholds; unified training follows independent pre-training.
- Ablations: Multiple resolution banks substantially improve Part-Detector performance, while receptive fields and pooling stages also strongly affect results.Hyperparameters were tuned by coarse meta-optimization under a less-than-100ms forward-propagation budget.
- Qualitative Results: The network produces convincing FLIC predictions with low joint position error, but the Spatial-Model is less effective for some highly articulated poses.Predicted joint locations are shown for FLIC and LSP test-set inputs.
5 Conclusion
The unified ConvNet Part-Detector and MRF-inspired Spatial-Model significantly outperform existing architectures for human body pose recognition. The authors identify greater spatial-model complexity as a future direction, especially for unconstrained datasets like LSP.
- The unified ConvNet Part-Detector and MRF-inspired Spatial-Model significantly outperforms existing architectures on human body pose recognition.
- Training and inference use commodity-level hardware and run at close to real-time frame rates, supporting tractable deployment across application areas.
- Future work will increase the complexity and expressiveness of the simple Spatial-Model, particularly for unconstrained datasets such as LSP.