PaperlayerView exact revision on arXiv

Source-linked AI summary

Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image

Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee

arXiv:1907.11346v2cs.CV

TL;DR

Prior 3D pose methods largely address single-person cases and struggle to recover absolute camera-centered coordinates for multiple people from one RGB image. The paper proposes a camera distance-aware top-down framework combining detection, root localization, and root-relative pose estimation, and reports strong multi-person performance without groundtruth inference inputs.

  • Problem

    Previous 3D human pose estimation methods mostly treat a single person, while multi-person estimation requires absolute camera distance in addition to 2D bounding boxes.

  • Method

    The framework uses human detection, absolute 3D root localization, and root-relative 3D single-person pose estimation in a fully learning-based top-down pipeline.

  • Results

    The method significantly outperforms previous 3D multi-person pose estimation methods and achieves comparable performance to state-of-the-art 3D single-person methods without groundtruth information.

  • Takeaways & Limitations

    The framework provides a flexible basis for absolute 3D multi-person pose estimation because existing detection and pose estimation methods can be plugged into it.

  • Takeaways & Limitations

    The distance measure k can be inaccurate because similarly sized image regions may correspond to different human appearances or camera distances.

Abstract

from arXiv · show

Although significant improvement has been achieved recently in 3D human pose estimation, most of the previous methods only treat a single-person case. In this work, we firstly propose a fully learning-based, camera distance-aware top-down approach for 3D multi-person pose estimation from a single RGB image. The pipeline of the proposed system consists of human detection, absolute 3D human root localization, and root-relative 3D single-person pose estimation modules. Our system achieves comparable results with the state-of-the-art 3D single-person pose estimation models without any groundtruth information and significantly outperforms previous 3D multi-person pose estimation methods on publicly available datasets. The code is available in https://github.com/mks0601/3DMPPE_ROOTNET_RELEASE , https://github.com/mks0601/3DMPPE_POSENET_RELEASE.

1. Introduction

The paper addresses the difficulty of extending 3D pose estimation from single people to multiple people, where absolute camera distance is unavailable from cropped images. It proposes a fully learning-based, camera distance-aware top-down framework that combines detection, root localization, and root-relative pose estimation.

  • Research gap: Most prior 3D human pose estimation methods target single-person cases and estimate poses relative to a body root rather than absolute camera-centered coordinates.The final absolute pose requires adding the estimated root coordinates, which commonly depend on bone-length priors or groundtruth information.
  • Research gap: Extending top-down 2D pose estimation to 3D requires both 2D human bounding boxes and each person’s absolute distance from the camera.Existing human detectors provide bounding boxes but not the required camera-distance information.
  • Proposed framework: The proposed framework uses DetectNet for human detection, RootNet for absolute root localization, and PoseNet for root-relative 3D pose estimation.Its components are designed to be compatible with most existing human detection and 3D pose estimation models.
  • Proposed framework: The framework outputs absolute camera-centered coordinates for multiple humans’ keypoints from a single RGB image.RootNet supplies the absolute root location needed to convert root-relative poses into absolute 3D poses.
  • Results: The method significantly outperforms previous 3D multi-person pose estimation methods on several publicly available datasets.It also achieves comparable performance to state-of-the-art 3D single-person methods without groundtruth information during inference.

3. Overview of the proposed model

The proposed top-down system recovers absolute 3D keypoint coordinates by combining detected human regions, localized roots, and root-relative pose estimates. Root depth and pose depth are integrated before back-projection to obtain the final camera-centered pose.

  • System overview: The system targets absolute camera-centered coordinates of multiple persons’ keypoints using DetectNet, RootNet, and PoseNet.DetectNet detects each person, RootNet localizes the root, and PoseNet estimates the corresponding root-relative 3D pose.
  • Root localization: RootNet estimates each human root as R = (xR, yR, ZR), combining image-plane coordinates with absolute depth.xR and yR are pixel coordinates, while ZR is the absolute depth value.
  • Pose estimation: PoseNet estimates the root-relative 3D pose from the same cropped human image used by RootNet.The root-relative pose contains cropped-image coordinates and root-relative depth values.
  • Pose reconstruction: The system adds root depth to each root-relative depth, transforms cropped-image coordinates into the original image space, and obtains absolute 3D pose by back-projection.This conversion combines the outputs of RootNet and PoseNet into camera-centered keypoint coordinates.

4. DetectNet

DetectNet and RootNet form the camera-aware front end: detection supplies human crops, while RootNet estimates absolute camera-centered human-root coordinates by correcting an area-based depth measure with image features.

  • DetectNet: Mask R-CNN is used as DetectNet to propose human bounding boxes through backbone, region proposal, RoIAlign, and classification components.The released COCO-pretrained model is used without fine-tuning on the pose datasets.
  • RootNet: RootNet estimates the human root’s 2D image coordinates and camera distance, then back-projects them into the camera-centered coordinate space.Its output is the absolute root coordinate R = (xR, yR, ZR).
  • Distance measure: k estimates absolute depth from the ratio of a human’s assumed real-space area to its image-space area using camera parameters.The method assumes Areal is 2000mm×2000mm and derives the measure from a pinhole camera model.
  • Distance measure: k can be inaccurate because bounding-box area varies with appearance: humans at equal distances can have different Aimg, while a child and adult can have similar Aimg at different distances.The paper identifies crouching posture and body size as examples of this mismatch.
  • RootNet: RootNet predicts an image-based correction factor γ, applies it to Aimg before recomputing k, and obtains the final depth value from the corrected measure.The image feature can indicate corrections for crouching posture and child body size.
  • RootNet: Camera normalization makes RootNet usable with training data from different camera intrinsics and with in-the-wild images lacking αx and αy.The correction factor is predicted only from the input image.

6. PoseNet

PoseNet estimates a root-relative 3D pose from each cropped human image using a ResNet backbone, deconvolutional heatmap prediction, and soft-argmax coordinate extraction.

  • Pose estimation: PoseNet predicts each joint’s root-relative 3D coordinates from a cropped human image.The model follows prior root-relative pose estimation approaches.
  • Network design: A ResNet backbone extracts global features, while three deconvolutional layers upsample feature maps before joint-specific 3D heatmaps are produced.Batch normalization and ReLU activations are used in the deconvolutional layers.
  • Coordinate extraction: Soft-argmax extracts the 2D image coordinates and root-relative depth values from the predicted joint heatmaps.The extracted outputs are combined as the root-relative pose representation.
  • Training: PoseNet is trained by minimizing the L1 distance between estimated and groundtruth coordinates.The loss is denoted Lpose, with ∗ indicating groundtruth.

7. Implementation details

The implementation uses publicly initialized detection and pose networks, Adam optimization, fixed 256×256 inputs, and geometric and appearance-based augmentation.

  • Initialization: DetectNet uses a publicly released COCO-pretrained Mask R-CNN without fine-tuning on the human pose estimation datasets.RootNet and PoseNet use PyTorch with ResNet-50 backbones initialized from ImageNet pretraining.
  • Optimization: The remaining network weights are initialized from a Gaussian distribution with σ = 0.001 and optimized with Adam using mini-batches of 128.The initial learning rate is 1 × 10^-3 and is reduced by a factor of 10 at the 17th epoch.
  • Data processing: RootNet and PoseNet use 256×256 input images and training augmentation with rotation, horizontal flipping, color jittering, and synthetic occlusion.Horizontal-flip augmentation is also performed during testing for 2D multi-person pose estimation.

8. Experiment

Experiments evaluate the framework on Human3.6M, MuCo-3DHP, and MuPoTS-3D using detection, root-localization, and pose-estimation metrics. Ablations show that detection and root localization affect absolute 3D accuracy, while PoseNet chiefly determines root-relative accuracy; the full system compares favorably with prior methods.

  • Datasets and metrics: Human3.6M provides single-person evaluation with PA MPJPE under Protocol 1 and MPJPE under Protocol 2.Protocol 1 trains on six subjects and tests on S11; Protocol 2 trains on five subjects and tests on S9 and S11.
  • Datasets and metrics: MuPoTS-3D evaluates multi-person performance using 3DPCKrel, AUCrel, and 3DPCKabs on real-world scenes containing up to three subjects.MuCo-3DHP is used for training, while MuPoTS-3D contains 20 outdoor scenes with groundtruth 3D poses.
  • Ablation study: Better human detection improves both 3D root localization and multi-person pose estimation, but groundtruth boxes do not considerably improve overall accuracy.The authors conclude that bounding-box estimation accuracy has limited impact when detection AP is high.
  • Ablation study: Groundtruth roots substantially improve 3DPCKabs, whereas RootNet corrects depth estimates and better root localization is needed for more accurate absolute poses.RootNet settings have similar AUCrel but differ substantially in absolute-pose accuracy.
  • Ablation study: Similar AUCrel across settings indicates that root-relative pose accuracy depends primarily on PoseNet rather than detection boxes or root localization.Using groundtruth boxes and roots does not significantly increase AUCrel for the same PoseNet.
  • Comparison with prior work: The proposed system significantly outperforms prior 3D multi-person methods on MuPoTS-3D and compares favorably with state-of-the-art single-person methods without inference-time groundtruth.On Human3.6M, the paper reports comparisons using groundtruth roots for PoseNet while noting that the full model requires no groundtruth at inference.

9. Discussion

The framework performs strongly against prior multi-person methods, but substantial improvement remains possible. The main boundary is absolute root localization, where groundtruth roots yield a significant 3DPCKabs gain.

  • Discussion: Groundtruth 3D root locations produce significant 3DPCKabs improvement, identifying root localization as the main remaining accuracy boundary.The authors suggest single-image depth estimation as a possible direction for improving RootNet.
  • Discussion: The framework can extend to single-image multi-person mesh reconstruction by replacing PoseNet with an existing human mesh reconstruction model.The proposed substitution preserves the framework's detection and root-localization components.

10. Conclusion

The paper concludes with a flexible three-module framework for single-image 3D multi-person pose estimation. It reports large gains over prior multi-person methods and comparable single-person performance without inference-time groundtruth.

  • Conclusion: The framework combines human detection, 3D root localization, and root-relative single-person pose estimation in a flexible plug-in design.Existing human detection and 3D single-person pose estimation models can be integrated into the framework.
  • Conclusion: The system outperforms previous 3D multi-person methods by a large margin and matches single-person methods comparably without groundtruth during inference.The paper presents the approach as a fully learning-based, camera distance-aware top-down framework compatible with prior models.

Supplementary Material of “Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single

The supplementary material adds experimental results and derives the camera-distance relation using a pinhole camera model. It defines the geometric quantities used in the derivation for the human root and image sensor.

  • Supplementary material: The supplement provides additional experimental results omitted from the main manuscript because of space limitations.These results are presented as supplementary material rather than as a new methodological component.
  • Geometric derivation: Figure 6 uses a pinhole camera model to derive Equation 1 from rays, camera geometry, and the projected human-root-centered axes.The green and blue arrows denote the root-centered x- and y-axes, while yellow lines denote rays.
  • Geometric derivation: In the derivation, d is camera-to-root distance, f is focal length, and l_sensor is the human length on the image sensor.The quantities d and f are measured in millimeters, as is l_sensor.

2. Comparison of 3D human root localization with previous approaches

The paper compares RootNet with distance minimization-based root localization methods and reports stronger performance with greater modularity. RootNet uses RANSAC-based joint selection and can be designed independently of PoseNet.

  • Evaluation: The comparison evaluates previous approaches and RootNet on Human3.6M under protocol 2 using mean root-position errors along the x, y, and z axes.Table 7 defines MRPEx, MRPEy, and MRPEz as mean errors for the three axes.
  • Previous approaches: Previous methods estimate 2D image coordinates and 3D camera-centered root-relative coordinates, then localize the absolute root by minimizing reprojection distance.They use a linear least-squares formulation for optimization.
  • Fitting procedure: RANSAC selects an optimal joint set for fitting across varying numbers of joints instead of relying on a heuristically selected set.Limb joints were excluded during fitting.
  • Results: RootNet significantly outperforms previous approaches on the reported root-localization comparison.The result is reported in Table 7.
  • Design implications: RootNet can be designed independently of PoseNet, whereas previous localization methods require both 2D and 3D predictions, limiting generalizability.The modular design provides flexibility to both models.

3. Running time of the proposed framework

The framework's running time is reported per component on a single TitanX Maxwell GPU, with DetectNet consuming most of the runtime. Additional experiments report absolute 3D pose accuracy and qualitative results on MuPoTS-3D and in-the-wild COCO images.

  • Runtime measurement: Running time is measured in seconds per frame for each framework component using a single TitanX Maxwell GPU.Table 8 reports the component-wise measurements.
  • Runtime profile: DetectNet consumes most of the framework's running time.The paper says direct comparison with previous works is difficult because they did not report runtime.
  • Additional evaluation: The authors report 3DPCKabs for their method because previous works did not report this metric.The reported evaluations appear in Tables 9 and 10 on MuPoTS-3D.
  • Qualitative evaluation: Qualitative results are shown on MuPoTS-3D and COCO, whose in-the-wild images are rarely included in 3D pose-estimation training sets.Figures 7 and 8 present these qualitative evaluations.
Loading 1907.11346v2…