Source-linked AI summary
Coherent Reconstruction of Multiple Humans from a Single Image
Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, Kostas Daniilidis
TL;DR
The paper addresses incoherent multiperson 3D reconstruction from a single image, where independently regressed people may interpenetrate or have incorrect depth ordering. It introduces a top-down SMPL-based feedforward network with collision and depth-ordering losses, and reports better benchmark performance alongside more coherent reconstructions.
Problem
Independent top-down reconstruction of detected people can produce interpenetration and inconsistent depth ordering, motivating coherent multiperson 3D reconstruction from a single image.
Method
The method uses a top-down R-CNN framework with SMPL parameters, an interpenetration loss, and a depth ordering-aware loss trained from rendered and annotated instance segmentation.
Results
The approach outperforms previous multiperson 3D pose-and-shape methods while recovering significantly more coherent results.
Takeaways & Limitations
The proposed losses enable coherent reconstruction of multiple people, including spatially nonintersecting meshes and scene-consistent depth ordering.
Takeaways & Limitations
The paper leaves more detailed modeling of human interactions and additional scene constraints for future work.
Abstract
from arXiv · showhide
In this work, we address the problem of multi-person 3D pose estimation from a single image. A typical regression approach in the top-down setting of this problem would first detect all humans and then reconstruct each one of them independently. However, this type of prediction suffers from incoherent results, e.g., interpenetration and inconsistent depth ordering between the people in the scene. Our goal is to train a single network that learns to avoid these problems and generate a coherent 3D reconstruction of all the humans in the scene. To this end, a key design choice is the incorporation of the SMPL parametric body model in our top-down framework, which enables the use of two novel losses. First, a distance field-based collision loss penalizes interpenetration among the reconstructed people. Second, a depth ordering-aware loss reasons about occlusions and promotes a depth ordering of people that leads to a rendering which is consistent with the annotated instance segmentation. This provides depth supervision signals to the network, even if the image has no explicit 3D annotations. The experiments show that our approach outperforms previous methods on standard 3D pose benchmarks, while our proposed losses enable more coherent reconstruction in natural images. The project website with videos, results, and code can be found at: https://jiangwenpl.github.io/multiperson
1 University of Pennsylvania 2 Zhejiang University
Figure 1 contrasts incoherent top-down multiperson reconstructions with the proposed feedforward approach, which respects spatial and depth constraints.
- Typical top-down regression baselines can predict people in overlapping positions or inconsistent depth orderings.
- The proposed approach is trained to recover a coherent reconstruction of all people in the scene feedforward.
1. Introduction
The paper targets coherent 3D reconstruction of multiple people from a single image by augmenting a top-down SMPL framework with scene-level constraints and two training losses.
- Motivation: Coherent 3D reconstruction of multiple people from single images is identified as a crucial step toward holistic scene understanding.
- Motivation: Top-down methods detect people first and estimate each person's pose, but independent regression can produce overlapping bodies and inconsistent depth ordering.
- Approach: The framework uses an R-CNN pipeline with an SMPL estimation branch to regress pose and shape for all detected people.
- Approach: An interpenetration loss trains the network to avoid regressing meshes that intersect each other.
- Approach: A depth ordering-aware loss promotes reconstructions whose people respect scene depth ordering.
- Reported outcome: The authors report outperforming previous approaches for multiperson 3D pose and shape while recovering significantly more coherent results.
2. Related work
Related work spans single-person mesh regression, multiperson pose estimation, and coherency constraints, positioning this paper as a feedforward top-down SMPL regressor trained with scene supervision.
- Single-person 3D pose and shape: Single-person methods estimate skeletons, nonparametric shape, or full-body pose and shape using parametric models such as SMPL.
- Single-person 3D pose and shape: Recent approaches regress pose and shape parameters directly from images, sometimes using intermediate representations or adversarial priors.
- Multi-person 3D pose: Multiperson top-down methods detect each person before estimating pose, whereas bottom-up methods detect and group body joints.
- Multi-person 3D pose: Prior multiperson pose-and-shape work uses scene constraints, but this paper trains a feedforward regressor to produce coherent estimates at test time.
- Coherency constraints: Existing coherency methods use penalties for human intersections or interactions, often refining estimates iteratively at test time.
- Coherency constraints: The depth ordering-aware loss uses instance segmentation masks to reason about occlusion without explicit depth annotations.
3. Technical approach
The approach augments a top-down R-CNN-style network with SMPL and training losses that target interpenetration and inconsistent depth ordering in multi-person reconstructions.
- SMPL representation: SMPL represents each person as a mesh, enabling the network to reason about occlusion and interpenetration.The model maps pose and shape parameters to a mesh with 6,890 vertices.
- Baseline architecture: The network uses an R-CNN-style architecture with a backbone, Region Proposal Network, detection heads, and an SMPL parameter regression branch.The SMPL branch regresses pose θ and shape β parameters.
- Training and implementation: The proposed losses are added to the jointly trained baseline, whose usual supervision includes 3D losses when available and 2D reprojection losses when only 2D joints are available.The implementation uses PyTorch and mmdetection; the reported forward-pass complexity drops from O(Fwh) to O(F + wh) on average.
- Interpenetration loss: The interpenetration loss samples each person’s differentiable distance field at another person’s mesh vertices and penalizes collisions.Separate tight-box voxelizations permit fine body resolution without voxelizing empty scene space; a robust error reduces instability from large translation gradients.
- Depth ordering-aware loss: The depth ordering-aware loss compares annotated and rendered person indices at pixels where both identify people but disagree, then penalizes their corresponding depths.Depth maps are rendered independently for each mesh, and gradients reach both people, including non-visible vertices.
4. Experiments
The experiments evaluate single-person, multi-person, and coherency performance across several benchmarks. Proposed losses improve multi-person performance, reduce interpenetrations, and improve ordinal depth ordering, including when applied only during training.
- Comparison with the state-of-the-art: Our approach outperforms HMR and Arnab et al. on the single-person Human3.6M benchmark.
- Comparison with the state-of-the-art: On Panoptic, the baseline is comparable to prior methods, while the full model improves across all subsequences and overall.
- Comparison with the state-of-the-art: On MuPoTS-3D, the full model significantly improves over the baseline, while single-person baselines are suboptimal for multi-person 3D pose.
- Ablative studies: The interpenetration loss significantly decreases collisions on MuPoTS-3D and PoseTrack.
- Ablative studies: The depth ordering-aware loss improves the percentage of correctly estimated ordinal depth relations over the baseline.
- Ablative studies: The coherency losses are used during training rather than test time, providing supervision that improves coherence when explicit 3D annotations are unavailable.
- Qualitative evaluation: Qualitatively, the full model produces fewer interpenetrations and more consistent depth ordering than the baseline, though errors remain with large scale differences and no image-plane overlap.
5. Summary
The paper presents an end-to-end approach for coherent multi-person 3D pose and shape estimation. Its losses improve scene coherency, while future extensions target richer interaction and scene context.
- Summary: The approach estimates SMPL-based pose and shape for multiple detected people using a top-down R-CNN framework.
- Summary: Two losses discourage overlapping humans and encourage consistent depth ordering in the reconstructed scene.
- Summary: Qualitative comparisons show that the full model improves interpenetration and depth-ordering errors over the baseline.
- Summary: The method performs competitively on traditional 3D pose metrics and significantly better in qualitative and quantitative scene coherency.
- Summary: Future work proposes modeling interactions and incorporating ground-plane, background, and object constraints for finer scene reconstruction.