Source-linked AI summary
Deep Multitask Architecture for Integrated 2D and 3D Human Sensing
Alin-Ionut Popa, Mihai Zanfir, Cristian Sminchisescu
TL;DR
The paper addresses fully automatic human recognition and reconstruction from monocular images despite limited, complementary 2D and 3D training datasets. It proposes a deep multitask architecture that jointly estimates segmentation, body parts, and 2D/3D pose, achieving state-of-the-art results across processing levels and perceptual competitiveness with commercial RGB-D sensing in the wild.
Problem
Monocular human sensing requires integrated recognition and reconstruction, but available 2D and 3D datasets cover complementary components and visual conditions incompletely.
Method
DMHS uses recursive multitask losses to jointly train figure-ground segmentation, pixel-level body-part labeling, and 2D and 3D pose estimation from monocular RGB images.
Results
The system achieves state-of-the-art results at all processing levels and is perceptually competitive with state-of-the-art commercial RGB-D systems in the wild.
Takeaways & Limitations
Combining complementary 2D and 3D datasets enables one fully automatic architecture to perform integrated human sensing across realistic and 3D-focused settings.
Takeaways & Limitations
A complete, fully realistic 2D and 3D training setting remains elusive despite mixed-reality setups in some 3D datasets.
Abstract
from arXiv · showhide
We propose a deep multitask architecture for \emph{fully automatic 2d and 3d human sensing} (DMHS), including \emph{recognition and reconstruction}, in \emph{monocular images}. The system computes the figure-ground segmentation, semantically identifies the human body parts at pixel level, and estimates the 2d and 3d pose of the person. The model supports the joint training of all components by means of multi-task losses where early processing stages recursively feed into advanced ones for increasingly complex calculations, accuracy and robustness. The design allows us to tie a complete training protocol, by taking advantage of multiple datasets that would otherwise restrictively cover only some of the model components: complex 2d image data with no body part labeling and without associated 3d ground truth, or complex 3d data with limited 2d background variability. In detailed experiments based on several challenging 2d and 3d datasets (LSP, HumanEva, Human3.6M), we evaluate the sub-structures of the model, the effect of various types of training data in the multitask loss, and demonstrate that state-of-the-art results can be achieved at all processing levels. We also show that in the wild our monocular RGB architecture is perceptually competitive to a state-of-the art (commercial) Kinect system based on RGB-D data.
1. Introduction
The paper addresses monocular human sensing by integrating 2D and 3D recognition and reconstruction, despite pose, occlusion, depth, and dataset-coverage challenges. Its deep multitask architecture combines complementary 2D and 3D datasets to achieve state-of-the-art results across processing levels.
- 1. Introduction: Monocular human analysis is difficult because pose, body proportions, occlusion, viewing conditions, backgrounds, and missing depth create substantial ambiguity.The need for measurement and prior knowledge motivates joint segmentation, reconstruction, and recognition.
- 1. Introduction: Existing 2D datasets provide realistic visual variability but limited annotations, whereas 3D datasets provide accurate pose information but limited background and clothing diversity.The paper frames combining these complementary strengths as an open training problem.
- 1. Introduction: The proposed model jointly performs figure-ground segmentation, pixel-level body-part labeling, and 2D and 3D pose estimation from monocular RGB images.These components are trained through recursive multitask losses across successive processing stages.
- 1. Introduction: The training design leverages multiple datasets whose partial coverage would otherwise limit model training and expressiveness.The approach combines complex 2D imagery lacking body-part and 3D labels with complex 3D data lacking broad 2D background variability.
- 1. Introduction: Experiments on LSP, HumanEva, and Human3.6M report state-of-the-art results for semantic body-part segmentation and 3D pose estimation.The experiments include ablations and evaluate different types of training data in the multitask loss.
2. Related Work
The paper builds on separate literatures in monocular 2D pose, monocular 3D pose, and semantic segmentation while integrating them into a single trainable architecture. Unlike approaches centered on shape fitting, it jointly estimates segmentation and 2D/3D pose using complementary datasets.
- 2. Related Work: Prior work spans pictorial structures, deformable part models, and deep architectures that combine hierarchical features with spatial constraints for 2D pose estimation.Recent systems commonly use cascaded processing architectures.
- 2. Related Work: Monocular 3D pose research addresses kinematic ambiguity through generative and discriminative methods, increasingly using deep convolutional models trained on HumanEva and Human3.6M.These datasets offer challenging poses but relatively simple backgrounds.
- 2. Related Work: Some recent methods fit an a-priori 3D body model to anatomical joints obtained from an initial deep 2D pose estimate.These approaches add a generative fitting stage to search over admissible body-proportion variations.
- 2. Related Work: DMHS is a fully trainable discriminative model that estimates figure-ground and semantic body-part segmentation together with 2D and 3D pose.Its recurrent multitask stages support end-to-end automatic detection and estimation using complementary 2D and 3D datasets.
- 2. Related Work: The proposed approach complements shape-fitting methods because their models could use its automatic pose and body-part estimates, while DMHS could benefit from their refinement and body-shape estimation.The relationship is presented as complementary rather than mutually exclusive.
3. Methodology
The model integrates recurrent 2D semantic processing and 3D reconstruction into a multitask, multistage architecture. Its losses exploit complementary datasets to train components with complete or partial annotations.
- Multitask multistage architecture: The architecture uses recurrent stages that pass 2D pose into body-part segmentation and both semantic outputs into 3D reconstruction.Each stage reuses earlier estimates and fuses image features with outputs from other tasks.
- 2D Human Body Joint Detection: 2D pose estimation predicts belief maps for anatomically defined joints from RGB images, using image evidence initially and previous-stage belief maps thereafter.The task minimizes squared Euclidean distance between predicted and ground-truth belief maps and can use LSP, HumanEva, or Human3.6M data.
- Semantic Body Part Segmentation: Semantic segmentation assigns every image pixel a body-part or background label, using 2D pose maps and recurrent body-label predictions.The segmentation module predicts per-pixel class probabilities and uses a multinomial logistic loss at full input resolution.
- 3D Pose Reconstruction: 3D reconstruction estimates skeleton joint configurations from monocular images by combining image features, 2D pose, body-part maps, and earlier reconstruction outputs.Its loss is the mean per joint position error, and training requires datasets with 3D ground-truth joints such as HumanEva or Human3.6M.
- Integrated Multi-task Multi-stage Loss: The integrated loss sums task losses across stages, allowing datasets with partial or complete annotations to train available components while missing-ground-truth components can be frozen.Different dataset coverage contributes to different loss terms, supporting joint training despite incomplete annotations.
4. Experiments
Experiments evaluate the multitask architecture across LSP, HumanEva-I, and Human80K, covering semantic labeling, 3D reconstruction, and multistage implementation choices. Jointly using complementary datasets and task outputs improves performance, including state-of-the-art results on the evaluated benchmarks.
- Experimental setup: The experiments use LSP, HumanEva-I, and Human80K, with Human80K providing RGB images, body-part masks, and accurate 2D and 3D pose annotations.HumanEva-I supplies laboratory motion-capture data, while LSP supplies challenging sports imagery and varied appearances.
- Experimental setup: The architecture uses six processing stages per subtask and reports the final stage because validation identifies it as the best-performing stage.The complete recognition and reconstruction pipeline takes approximately 400 ms per frame on an Nvidia TITAN X GPU.
- Body part labeling: More than 10% higher class-normalized average precision than is obtained on Human80K by models trained on Human80K alone or jointly with LSP.These models also solve figure-ground segmentation, whereas assumes the person silhouette is given.
- 3D pose reconstruction: The 3D reconstruction ablation compares pipelines using image features, body-joint belief maps, and body-labeling probability maps as inputs.The D-pipeline omits semantic body-part segmentation and feed-forwards from image features.
- 3D pose reconstruction: 63.35 mm mean error is achieved by the fully integrated DMHSR(J,B,D) system on Human80K, outperforming the evaluated alternatives.Adding joint-belief and body-labeling maps reduces error from 77.56 mm to 63.35 mm, with gains over prior state of the art.
5. Conclusions
DMHS performs fully automatic monocular 2D and 3D human sensing through recurrent multitask processing. It achieves state-of-the-art results across processing levels and remains perceptually competitive with commercial RGB-D Kinect systems in challenging images.
- State-of-the-art results are achieved at all processing levels, including recognition and reconstruction tasks.
- The monocular RGB architecture is perceptually competitive with commercial RGB-D Kinect systems in the wild.
- The system estimates figure-ground segmentation, body-part labels, and 2D and 3D human pose from monocular images.
- Early-stage 2D pose errors can be corrected during later body-part segmentation or 3D pose estimation stages.