PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB

Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, Christian Theobalt

arXiv:1712.03453v3cs.CV

TL;DR

Monocular multi-person 3D pose estimation is difficult under strong occlusions and varied real-world scenes. The paper introduces a single-shot CNN with occlusion-robust pose-maps, trains it using MuCo-3DHP, and evaluates it on MuPoTS-3D, achieving state-of-the-art performance on challenging multi-person scenes.

  • Problem

    Monocular multi-person 3D pose estimation remains insufficiently addressed because scenes contain strong self-, inter-person, and object occlusions.

  • Method

    The paper proposes a single-shot CNN that jointly predicts all persons’ 2D and 3D joints using occlusion-robust pose-maps with fixed outputs and no separate bounding-box proposals.

  • Results

    The method achieves state-of-the-art performance on challenging multi-person scenes where single-person methods completely fail.

  • Takeaways & Limitations

    MuCo-3DHP provides composited multi-person training data, while MuPoTS-3D enables evaluation on diverse real-world scenes with interactions and occlusions.

  • Takeaways & Limitations

    Relative person sizes remain difficult to estimate, and close non-overlapping joints can produce sharp pose-map transitions that lead to inaccurate predictions.

Abstract

from arXiv · show

We propose a new single-shot method for multi-person 3D pose estimation in general scenes from a monocular RGB camera. Our approach uses novel occlusion-robust pose-maps (ORPM) which enable full body pose inference even under strong partial occlusions by other people and objects in the scene. ORPM outputs a fixed number of maps which encode the 3D joint locations of all people in the scene. Body part associations allow us to infer 3D pose for an arbitrary number of people without explicit bounding box prediction. To train our approach we introduce MuCo-3DHP, the first large scale training data set showing real images of sophisticated multi-person interactions and occlusions. We synthesize a large corpus of multi-person images by compositing images of individual people (with ground truth from mutli-view performance capture). We evaluate our method on our new challenging 3D annotated multi-person test set MuPoTs-3D where we achieve state-of-the-art performance. To further stimulate research in multi-person 3D pose estimation, we will make our new datasets, and associated code publicly available for research purposes.

1. Introduction

The paper addresses monocular multi-person 3D pose estimation in cluttered scenes, where occlusions make existing single-person approaches unreliable. It proposes a single-shot CNN with occlusion-robust pose-maps and introduces complementary training and evaluation datasets.

  • Multi-person monocular 3D pose estimation remains difficult because natural scenes combine self-occlusion, inter-person occlusion, and object occlusion.
  • Existing approaches decompose the task into multiple single-person instances, often with redundant predictions and post-processing, while bottom-up joint reasoning remains largely unsolved.
  • The proposed single-shot CNN jointly reasons about all people in one forward pass without requiring separate bounding-box proposals.
  • Occlusion-robust pose-maps encode multiple people with a fixed number of outputs, while hierarchical read-out refines poses using visible joints.
  • MuCo-3DHP supplies composited multi-person training images, and MuPoTS-3D provides a diverse real-world evaluation set with challenging interactions and occlusions.
  • The method achieves state-of-the-art performance on challenging multi-person scenes where single-person methods completely fail.

2. Related Work

Prior work commonly detects or localizes people and then estimates or associates their poses, but these strategies are vulnerable to occlusion and limited 3D data diversity. The paper positions its approach and datasets against these gaps.

  • Multi-person 2D methods often detect people before estimating poses, but detector failures under strong occlusion propagate to pose estimation.
  • Bottom-up 2D methods localize joints and associate them to subjects in post-processing, rather than performing unified multi-person 3D reasoning.
  • Monocular single-person 3D methods perform well on standard datasets but often generalize poorly to varied poses, appearances, backgrounds, and occlusions.
  • Some methods combine 2D and 3D features or losses, use multi-stage belief maps, or refine 2D pose networks for 3D estimation.
  • Prior multi-person 3D work uses bounding boxes, pose classification, refinement, and multiple proposals that must be accumulated.
  • Existing datasets are mostly single-person 3D or multi-person 2D; exceptions have limited simultaneous subjects, interactions, capture volume, or background diversity.

3. Multi-Person Dataset

The paper constructs MuCo-3DHP by compositing annotated single-person captures into diverse multi-person training images, then evaluates generalization on the filmed MuPoTS-3D benchmark.

  • MuCo-3DHP: Compositing-Based Training Set: Multi-person 3D capture under strong occlusions and interactions is challenging, motivating a compositing-based alternative for large-scale training data.
  • MuCo-3DHP: Compositing-Based Training Set: MuCo-3DHP composites MPI-INF-3DHP person images with ground-truth 3D skeletons in a shared space, producing correct depth ordering and overlap for one to four subjects.
  • MuCo-3DHP: Compositing-Based Training Set: MuCo-3DHP increases diversity through varied subject frames, camera views, simulated overlap, and appearance augmentation.
  • MuPoTS-3D: Diverse Multi-Person 3D Test Set: MuPoTS-3D contains 20 filmed real-world scenes, ground-truth 3D pose for up to three subjects, and per-joint occlusion annotations.
  • MuPoTS-3D: Diverse Multi-Person 3D Test Set: The test set spans indoor and outdoor settings, diverse backgrounds, illumination conditions, clothing, poses, interactions, and activities across more than 8000 frames.
  • Evaluation: Evaluation uses 3DPCK, counting a joint correct when its prediction lies within a 15cm ball of the ground-truth location, with occluded and unoccluded breakdowns.

4. Method

The method uses occlusion-robust pose-maps and associated 2D predictions to estimate multiple people’s 3D poses in one fully convolutional forward pass with a fixed output size. Selective redundancy supports fallback read-outs for occluded limbs and overlapping people.

  • 4.1. Formulation: ORPMs provide redundant read-out locations, allowing complete poses at torso joints and limb poses at joints along each limb.The formulation decomposes the body into torso, limbs, and head, with selective rather than universal redundancy.
  • 4.1. Formulation: When read-out locations overlap, the nearer person occupies the shared map location, while redundancy preserves alternative locations for the partially occluded person.This allows multiple partially overlapping people to be encoded without loss of information in the fixed maps.
  • 4.1. Formulation: A fixed set of heatmaps, 3D pose-maps, and part affinity fields jointly encodes all people without output channels that vary with crowd size.Part affinity fields associate detected 2D joints with person identities, enabling per-person pose read-outs.
  • 4.2. Pose Inference: The inference process starts from neck or pelvis base-pose read-outs, then replaces limb components with estimates from valid extremity or upstream joints.A limb is retained from the base pose when all of its candidate read-out joints are invalid.
  • 4.2. Pose Inference: A 2D joint is valid only when its confidence exceeds tC and it is at least tD from other individuals’ corresponding read-out locations.The validation prevents unreliable or conflicting pixel locations from being used for 3D read-out.
  • 4.3. Network and Training Details: The occlusion-aware ORPM inference with limb refinement is designed to estimate strongly occluded body parts while using visible individual limbs.The method uses a ResNet-50 core with separate 2D pose-affinity and 3D pose streams trained on the stated datasets.

5. Results and Discussion

Across multi-person and single-person evaluations, the method is especially robust to occlusion while remaining competitive with specialized single-person approaches. Ablations attribute gains to limb refinement, and comparisons show advantages over LCR-net and VNect under the reported evaluation settings.

  • Multi-person performance: 93% of annotated subjects are detected by our approach versus 86% by LCR-net.
  • Multi-person performance: Despite lacking ground-truth crops, our method exceeds VNect at 65.0 versus 61.1 3DPCK and 30.1 versus 27.6 AUC.VNect operates at a 100% detection rate because it receives ground-truth subject crops.
  • Single-person performance: On MPI-INF-3DHP, multi-person training reaches 73.4 3DPCK versus 75.2 for single-person training, while both outperform Zhou et al. and LCR-net.The multi-person version is marginally worse because training on harder data reduces effective network capacity; performance is on par with VNect and Mehta et al.
  • Single-person performance: On Human3.6m, our method records 69.6mm MPJPE, approximately 17mm better than LCR-net and approximately 10mm better than VNect location-maps.The result is comparable to several recent state-of-the-art methods listed in the comparison.
  • Occlusion robustness: Under synthetic occlusion covering about 14% of joints, both variants outperform VNect on occluded joints while remaining comparable on un-occluded joints.The reported occluded-joint 3DPCK values are 62.8, 64.0, and 53.2; un-occluded values are 67.0, 71.0, and 69.4.
  • Occlusion robustness: With MuPoTS-3D occlusion annotations, our method outperforms LCR-net on occluded joints, 48.7 versus 42 3DPCK, and un-occluded joints, 70.0 versus 57.5 3DPCK.Approximately 23.7% of all annotated joints are occluded by self- or inter-person occlusion.
  • Ablative analysis: Limb refinement improves 3DPCK by approximately 10 points over torso read-out across metrics and training-data variants.The base torso read-out tends toward the training-data mean pose, whereas limb read-outs preserve more detailed articulation near extremities.

6. Limitations and Future Work

The method has several remaining failure modes and open problems, including sharp ORPM transitions, incorrect 2D associations, difficulty estimating relative person sizes, and limited true interaction data.

  • Overlapping same-type joints are supervised only for the person closest to the camera.
  • Closely spaced, non-overlapping same-type joints can cause sharp ORPM transitions that are difficult to regress and may produce inaccurate predictions.
  • Mis-predicted or mis-associated 2D joints are another source of failures.
  • Estimating relative person sizes remains challenging despite accurate root-relative 3D pose estimation.
  • MuCo-3DHP covers plausible scenarios through compositing, but capturing or generating true person-person interactions at scale remains future work.

7. Conclusion

The paper presents ORPM-based monocular multi-person 3D pose estimation trained on synthesized multi-person data. Experiments report strong performance under occlusions and interactions, with generalization to real-world scenes.

  • The method addresses monocular multi-person 3D pose estimation, which previous work had not fully addressed.
  • ORPM enables 3D pose estimation under strong inter-person occlusions and human-human interactions better than previous approaches.
  • MuCo-3DHP is a multi-person dataset synthesized at scale from existing single-person images with 3D pose annotations.
  • Training on MuCo-3DHP generalizes well to real-world scenes in the MuPOTS-3D evaluation set.
  • The proposed training and evaluation datasets are intended to spur further investigation into monocular multi-person pose estimation.

1. Read-out Process

The read-out process detects people from pelvis or neck confidence, reads joint locations from ORPMs, and refines limbs using valid, isolated read-out locations.

  • Algorithm 1 describes the 3D pose inference read-out process.
  • A person is detected when the pelvis or neck confidence exceeds a threshold.
  • For each detected person and joint, the process reads the corresponding ORPM location after scaling the 2D read-out coordinate.
  • The process refines each limb by reading ORPM values for its joints from a valid read-out location.
  • Read-out locations are valid only when the 2D joint confidence is positive and the joint is isolated from corresponding joints of other people.
  • The network architecture includes 2D heatmaps, part affinity maps, 2D heatmaps for MPII, and ORPMs at specified spatial resolutions.

2. Network Details

The implementation uses a pretrained, fully convolutional network with augmented MuCo-3DHP training data and lower-resolution affinity and ORPM outputs, alongside benchmark comparison on MPI-INF-3DHP.

  • The architecture visualization is provided through the Netscope web-based visualization tool.
  • Training uses 12 camera viewpoints to create 400k MuCo-3DHP composite frames, with appearance augmentation applied to half of them.
  • The network is initialized from 2D pose models and trained with Caffe, AdaDelta, momentum 0.9, weight decay multiplier 0.005, and batch size 8.
  • Figure 2 compares joint-wise 3D Percentage of Correct Keypoints (@150mm) with LCR-net on the single-person MPI-INF-3DHP test set.
  • The 3D training stage uses batch size 6 and 360k iterations with a cyclical learning rate from 0.1 to 0.000001.

3. Joint-wise Analysis

The method’s full readout improves joint-wise accuracy over torso-only readout and generally outperforms LCR-net across multi-person sequences.

  • The full readout performs significantly better than torso-only readout for limb joints on the MPI-INF-3DHP test set.LCR-net is comparable to or better than the torso-only readout for elbows, wrists, knees, and ankles.
  • The approach achieves better accuracy than LCR-net for all joint types in most multi-person test sequences.It performs worse only for selected joint types in Test-Seq18, Test-Seq19, and Test-Seq20.

4. Evaluation on Single-person Test Sets

The evaluation covers single-person benchmarks, synthetic occlusion testing, and qualitative comparisons on multi-person and 2D pose datasets. Results are reported with standard 3D pose metrics and include comparisons against LCR-net and related methods.

  • Single-person performance is evaluated on Human3.6m and MPI-INF-3DHP, including comparisons with VNect location-maps trained under the authors’ setup.The setup includes 2D pretraining and 3D pose samples.
  • Synthetic occlusion experiments on MPI-INF-3DHP occlude approximately 14% of evaluated joints and report overall and occlusion-split PCK.The experiment does not account for self-occlusions.
  • The qualitative comparison shows LCR-net tending toward neutral poses, while the method’s full readout addresses the limited limb articulation of its torso-only readout.The comparison concerns the extent of limb articulation in predicted poses.
  • MuPoTS-3D includes varied scenes, activities, and clothing, with ground-truth 3D poses and joint-occlusion annotations available for up to three subjects.The annotations are illustrated for one evaluation frame.
  • Figure 5 compares the method with LCR-net using joint-wise PCK across all 20 sequences and differences in accuracy, with LCR-net mapped to ground-truth bone lengths.The bone-length mapping is used for a fair comparison.
  • The paper also presents qualitative results on the MPI 2D pose dataset and the MuPoTS-3D test set.
Loading 1712.03453v3…