Source-linked AI summary
LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
Gregory Rogez, Philippe Weinzaepfel, Cordelia Schmid
TL;DR
Single-image human pose estimation must handle multiple people, occlusion, and limited real-world 3D training data. LCR-Net addresses this with jointly trained pose proposals, classification, and regression, outperforming state of the art in controlled 3D evaluation while showing promising real-image results.
Problem
3D pose methods trained with MoCap data perform accurately in controlled environments but often do not generalize well to real images, where occlusion and multiple people remain challenging.
Method
LCR-Net generates pose proposals in candidate regions, scores anchor-poses with classification, refines them with anchor-specific regression, and trains the shared architecture end to end.
Results
LCR-Net++ improves over the initial version by more than 20mm in 3D and 10% in 2D pose accuracy, outperforms state of the art in controlled 3D evaluation, and shows promising real-image results.
Takeaways & Limitations
The proposal-based architecture supports joint 2D-3D full-body pose estimation for multiple people in natural images, including occluded or truncated cases.
Takeaways & Limitations
The authors identify pose-proposal scoring and more accurate real-world training data as important opportunities for improving performance.
Abstract
from arXiv · showhide
We propose an end-to-end architecture for joint 2D and 3D human pose estimation in natural images. Key to our approach is the generation and scoring of a number of pose proposals per image, which allows us to predict 2D and 3D poses of multiple people simultaneously. Hence, our approach does not require an approximate localization of the humans for initialization. Our Localization-Classification-Regression architecture, named LCR-Net, contains 3 main components: 1) the pose proposal generator that suggests candidate poses at different locations in the image; 2) a classifier that scores the different pose proposals; and 3) a regressor that refines pose proposals both in 2D and 3D. All three stages share the convolutional feature layers and are trained jointly. The final pose estimation is obtained by integrating over neighboring pose hypotheses, which is shown to improve over a standard non maximum suppression algorithm. Our method recovers full-body 2D and 3D poses, hallucinating plausible body parts when the persons are partially occluded or truncated by the image boundary. Our approach significantly outperforms the state of the art in 3D pose estimation on Human3.6M, a controlled environment. Moreover, it shows promising results on real images for both single and multi-person subsets of the MPII 2D pose benchmark and demonstrates satisfying 3D pose results even for multi-person images.
1 INTRODUCTION
LCR-Net combines localization, pose classification, and class-specific regression in an end-to-end architecture for joint 2D and 3D multi-person pose detection in natural images. LCR-Net++ improves the initial system and reports stronger controlled-environment 3D and real-image 2D/3D results.
- Occlusions remain a major challenge for 2D pose estimation, while 3D methods trained with MoCap data often generalize poorly to real images.
- Classification captures pose diversity, whereas regression provides smooth refinement; LCR-Net combines both to represent ambiguous, multi-modal pose outputs.
- LCR-Net generates pose proposals across candidate regions, scores them with classification, and refines each anchor-pose independently through regression.
- Localization, classification, and per-anchor regression share convolutional layers and are trained jointly, producing multiple 2D-3D poses per image by aggregating similar hypotheses.
- More than 20mm in 3D and 10% in 2D pose accuracy distinguish LCR-Net++ from the initial version.The 2D metric is PCKh@0.5.
- The method outperforms the state of the art for 3D pose estimation in controlled environments and shows promising 2D and 3D results on real images.
2 RELATED WORK
Prior work estimates 2D pose using local parts or holistic CNNs and approaches 3D pose through 2D intermediates or direct image mappings. LCR-Net instead jointly localizes and estimates 2D and 3D poses for multiple people in real-world images.
- 2D pose methods use local body parts with graphical models or holistic CNNs that directly estimate the full body.
- Local-part methods often require tight human bounding boxes, while heatmap-based systems may need iterative refinement and post-processing to assemble poses.
- Single-image 3D methods either infer 3D pose from 2D joints or directly learn mappings from image features to 3D poses.
- Existing joint 2D-3D approaches include cue fusion, iterative refinement, and quantized pose classification, but related methods can require well-aligned bounding boxes.
- LCR-Net jointly localizes and estimates 2D and 3D poses for multiple people in real-world images.
- Reliable 3D training data usually comes from MoCap systems in constrained environments, limiting the realism of common training and evaluation settings.
3 LCR-NET
LCR-Net detects and estimates joint 2D-3D human poses by generating localized anchor-pose proposals, classifying and refining them with shared features, and integrating compatible hypotheses. Its iterative variant refines predictions, while PPI aggregates proposals using both image overlap and 3D-pose similarity.
- 3 LCR-NET: LCR-Net defines each pose with 2D joint coordinates and torso-centered 3D joint locations, using clustered 2D-3D anchor-poses as hypotheses.The paper considers J = 13 joints and obtains anchors by clustering a large pose set.
- 3 LCR-NET: Training combines localization, classification, and regression losses, with convolutional and later fully connected features shared across components.The regression target normalizes 2D poses within the proposal box, making it independent of image scale and position.
- 3.1 Localization: pose proposals network: The network generates N×K pose proposals by placing K anchor-poses in RPN-generated boxes, then scores and refines them.Localization produces candidate regions, while classification and regression process the resulting pose hypotheses.
- 3.2 Classification and 3.3 Regression: Classification predicts the closest anchor-pose or background, while an anchor-specific regressor estimates 2D and 3D corrections for each joint.The regression output has dimension 5 × J × (K + 1), combining 2D and 3D coordinates.
- Iterative estimation: The iterative variant feeds initial predictions back with RoI features, applies losses after refinements, and returns only the final estimate at test time.The classification and regression branches are repeatedly refined after an initial shared fully connected representation.
4 EXPERIMENTAL RESULTS
Experiments evaluate LCR-Net++ across controlled and natural-image settings, showing strong Human3.6M 3D results, competitive MPII 2D performance, and effective multi-person 3D detection.
- Human3.6M: 53.5 mm average error after PPI versus 54.2 mm after NMS on Human3.6M protocol P1 for 17 joints.After rigid alignment, the corresponding errors are 43.1 mm and 43.7 mm.
- Human3.6M: Rescoring decreases 3D error by 3 mm, while the best pose proposals remain difficult to select because perfect-scoring upper bounds are better.The authors suggest temporal information could help disambiguate and rescore proposals in future work.
- Human3.6M: 65.4 mm on Human3.6M protocol P2 establishes state-of-the-art performance for 13 joints with LCR-Net++, while LCR-Net+ reaches 61.2 mm for 17 joints.The comparison uses the same protocol and notes that competing methods often assume human bounding boxes or external 2D pose detectors.
- Training data and ablations: 13% reduction in 2D-to-3D pose matching error follows from augmenting the MoCap data with additional datasets containing rare poses.Synthetic-data gains are smaller on MPII, where improvement is +0.84%, likely reflecting the difficulty of useful in-the-wild synthesis.
- MPII: 78.2% of MPII validation poses are globally correct, comprising 40% perfectly recognized and 38.2% correct but imprecise poses.The same qualitative analysis reports 20.5% partly incorrect and 1.3% miss-detected poses.
- MPII: 61.7% multi-person mAP@0.5 is obtained with LCR-Net++ on MPII, compared with 54.3% for LCR-Net+.Using head and torso keypoints to define PPI bounding boxes helps avoid unwanted merging of people.
- MuPoTS-3D: 70.6% 3DPCK is achieved on MuPoTS-3D, versus 65.0% for Mehta et al.; performance reaches 74% when evaluating only well-detected persons.The metric counts joints predicted within a 15 cm ball around ground truth and is averaged over available subjects.
5 CONCLUSION
LCR-Net jointly detects 2D and 3D human poses in natural images using pose proposals scored by classification and refined by regression. The authors report state-of-the-art 3D performance in controlled environments and promising real-image results, while identifying proposal scoring and training data as improvement opportunities.
- LCR-Net introduces an end-to-end architecture for joint 2D and 3D human pose detection in natural images.
- Pose proposals are hypothesized at different image locations, scored by classification, refined by regression, and integrated over neighboring hypotheses for final estimation.
- The method outperforms the state of the art in 3D pose estimation in controlled environments and shows promising results on real images.
- The upper-bound analysis indicates that adequately scoring pose proposals could provide a considerable performance boost.
- More accurate real-world training data, such as manually curated data, could improve LCR-Net's 2D and 3D performance.