Source-linked AI summary
Monocular, One-stage, Regression of Multiple 3D People
Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael J. Black, Tao Mei
TL;DR
Monocular multi-person 3D mesh regression remains difficult under truncation and occlusion because existing methods rely on multi-stage bounding-box processing. ROMP addresses this with one-stage, body-center-guided pixel-level prediction and achieves state-of-the-art benchmark performance with real-time inference. Its collision-aware representation further targets severe person-person overlap, while depth ordering remains limited in extreme crowding.
Problem
Monocular multi-person 3D estimation must handle truncation, person-person occlusion, and environmental occlusion in general scenes.
Method
ROMP predicts a Body Center heatmap and Mesh Parameter map per pixel, samples mesh parameters at body centers, and uses collision-aware representation for severe overlap.
Results
ROMP achieves state-of-the-art performance on multiple benchmarks, demonstrates robustness under person-person occlusion, and runs over 30 FPS on a 1070Ti GPU.
Takeaways & Limitations
ROMP provides a simple one-stage foundation for multi-person 3D tasks such as depth estimation, tracking, and interaction modeling.
Takeaways & Limitations
Body-level depth ordering remains limited in extremely crowded scenes with complex depth relationships.
Abstract
from arXiv · showhide
This paper focuses on the regression of multiple 3D people from a single RGB image. Existing approaches predominantly follow a multi-stage pipeline that first detects people in bounding boxes and then independently regresses their 3D body meshes. In contrast, we propose to Regress all meshes in a One-stage fashion for Multiple 3D People (termed ROMP). The approach is conceptually simple, bounding box-free, and able to learn a per-pixel representation in an end-to-end manner. Our method simultaneously predicts a Body Center heatmap and a Mesh Parameter map, which can jointly describe the 3D body mesh on the pixel level. Through a body-center-guided sampling process, the body mesh parameters of all people in the image are easily extracted from the Mesh Parameter map. Equipped with such a fine-grained representation, our one-stage framework is free of the complex multi-stage process and more robust to occlusion. Compared with state-of-the-art methods, ROMP achieves superior performance on the challenging multi-person benchmarks, including 3DPW and CMU Panoptic. Experiments on crowded/occluded datasets demonstrate the robustness under various types of occlusion. The released code is the first real-time implementation of monocular multi-person 3D mesh regression.
1. Introduction
ROMP replaces failure-prone multi-stage, bounding-box-based regression with a one-stage pixel-level framework for robust monocular multi-person 3D mesh estimation.
- 1. Introduction: Existing methods detect people, extract bounding-box features, and independently regress meshes, but struggle with truncation and multi-person occlusion.Similar overlapping image patches make diverse body meshes difficult to estimate from implicit, ambiguous representations.
- 1. Introduction: ROMP directly predicts a Body Center heatmap and Mesh Parameter map from the whole image to represent multiple people per pixel.Sampling mesh parameters at heatmap-described body centers and applying SMPL yields the final 3D meshes.
- 1. Introduction: Whole-image, body-center-guided learning reduces regression-target ambiguity and learns features suited to bodies affected by occlusion.The holistic representation differs from local bounding-box features and supports robustness in complex multi-person scenes.
- 1. Introduction: Collision-aware representation separates body centers of severely overlapping people through mutual repulsion and shifts centers toward visible body regions.This improves robustness under heavy person-person occlusion.
- 1. Introduction: ROMP achieves superior performance on challenging 3DPW and CMU Panoptic benchmarks and runs over 30 FPS on a 1070Ti GPU.Experiments also demonstrate effectiveness on person-person occlusion datasets and internet or webcam images.
- 1. Introduction: The paper presents ROMP as a first one-stage monocular multi-person 3D mesh regression method with an open-source real-time implementation.The stated contributions also include explicit body-center-guided representation and collision-aware handling of severe overlap.
2. Related Work
ROMP extends pixel-level, one-stage reasoning from multi-person pose estimation to 3D mesh regression using a concise body-center representation and collision-aware refinement.
- 2. Related Work: Single-person mesh regression commonly encodes human bodies with parametric models such as SMPL and regresses their parameters from images.Prior work uses diverse supervision signals including 2D pose, segmentation, geometric priors, motion, temporal coherence, and texture.
- 2. Related Work: Multi-person pose methods include top-down multi-stage systems and one-stage approaches that detect joints and group them into individuals.Other one-stage approaches rely on anchors or depth-aware part affinity fields for occlusion handling.
- 2. Related Work: Prior multi-person mesh methods use intermediate 3D pose, scene constraints, or RoI-aligned features to estimate individual SMPL parameters.These approaches largely retain multi-stage or region-based processing.
- 2. Related Work: Pixel-level center representations avoid dense anchor proposals, inspiring ROMP's fine-grained representation for locating people and regressing meshes.ROMP defines body centers from body joints rather than bounding-box centers.
- 2. Related Work: ROMP uses a concise body-center representation and extends it with collision awareness to address center collisions during severe overlap.This differs from multi-scale grid representations and methods requiring part association or many anchor predictions.
- 2. Related Work: Disambiguation techniques in related fields include guide points, association embeddings, triplet loss, and pose-guided association.These methods address ambiguity in instance segmentation, pose estimation, face recognition, and tracking.
3. Our Approach
ROMP represents multiple 3D people through pixel-level maps rather than bounding boxes, using body-center guidance and collision-aware repulsion to support mesh sampling under overlap.
- Overall Framework: ROMP outputs Body Center, Camera, and SMPL maps from a single RGB image to describe multiple estimated 3D human meshes.The Mesh Parameter map combines camera and SMPL parameters at each image location.
- Body Center Representation: The Body Center heatmap represents each center as a scale-dependent Gaussian distribution, with kernel size computed from the person’s 2D body scale.The default kernel parameters are kl = 2 and kr = 5.
- Body Center Representation: Body centers are defined from visible torso joints, or from the average of visible joints when all torso joints are invisible.This makes the center correspond to visible body content rather than a potentially external bounding-box midpoint.
- CAR: Collision-Aware Representation: Collision-Aware Representation treats nearby overlapping body centers as positive charges and applies mutual repulsion to keep their centers distinguishable.Repulsion is triggered when the center distance is smaller than the two Gaussian kernel sizes plus one, with intensity controlled by γ.
- Parameter Sampling: During inference, local maxima of the Body Center heatmap above threshold tc are ranked, the top N centers are retained, and Mesh Parameter values are sampled at those locations.The sampled parameters are used to generate the corresponding 3D body meshes.
4. Experiments
ROMP is evaluated across standard, multi-person, occlusion, and runtime settings against multi-stage methods. Results show strong accuracy, occlusion robustness, and real-time behavior, while CAR improves crowded-scene performance.
- Implementation Details: ROMP uses ResNet-50 by default, optionally tests HRNet-32, and predicts Body Center, Camera, and SMPL maps from image features.Inputs are resized to 512 × 512, with a 128 × 128 backbone feature map and maximum detection count N = 64.
- Evaluation Setup: 3DPW is divided into person-occluded, object-occluded, and non-occluded/truncated subsets, alongside CMU Panoptic and Crowdpose evaluations.The metrics include PVE, MPJPE, PMPJPE, PCK, AUC, MPJAE, PA-MPJAE, and Crowdpose AP0.5.
- 3DPW: ROMP significantly outperforms competing methods on 3DPW Protocol 1, particularly in MPJPE, PMPJPE, and MPJAE.The comparison uses whole-image input without ground truth during inference, and extra training data further improves ROMP accuracy.
- Comparisons to the State-of-the-Art: ROMP outperforms multi-stage approaches on 3DPW Protocol 2, achieves comparable results on Protocol 3, and outperforms existing methods across all CMU Panoptic activities.Protocol 2 evaluates the test set without fine-tuning, Protocol 3 fine-tunes on the training set, and CMU Panoptic is evaluated without fine-tuning.
- Occlusion Benchmarks: ROMP significantly outperforms previous methods on person-occluded 3DPW-PC and Crowdpose, supporting the effectiveness of pixel-level representation under occlusion.The discussion attributes the difference to holistic pixel-level representation rather than local bounding-box-level representation.
- Runtime Comparisons: ROMP achieves real-time performance, runs faster than competing methods, and maintains roughly constant processing time as the number of people increases.Runtime comparisons use web-camera videos on a GTX 1070Ti desktop system.
5. Conclusion
ROMP is presented as a one-stage network for monocular multi-person 3D mesh regression, combining a body-center-guided representation with collision awareness for occlusion robustness.
- ROMP is a one-stage network for monocular multi-person 3D mesh regression.
- The method uses an explicit body-center-guided representation and its collision-aware version, CAR, for person-person occlusion.
- ROMP achieves state-of-the-art performance on multiple benchmarks with real-time inference speed.
- The framework is proposed as a foundation for depth estimation, tracking, and interaction modeling.
1. Introduction
The supplementary material provides architecture, training, implementation, qualitative-result, and failure-case details.
- The supplementary material details the network architecture design.
- It presents hyper-parameter configurations and implementation details.
- It includes qualitative results and an analysis of failure cases.
2. Ablation study on the architecture
The ablation study examines ROMP’s architecture choices, including head separation, block count, backbone, and the removal of iterative regression.
- Architecture: ROMP builds on a ResNet-50-based architecture with a single head producing SMPL parameters, then progressively changes this design.
- Head design: The study compares single-head and separated multi-head designs, with the latter jointly estimating Camera and SMPL maps through separate branches.
- Backbone: HRNet-32 performs better than ResNet-50 in the reported architecture experiments.
- Head design: Among tested designs, disentangled multi-head architecture generally performs better and is easier to train than the single-head design.
- Block count: Setting the number of blocks to n = 2 balances accuracy and training time.
- Iterative regression: Unlike previous methods, ROMP uses fully convolutional multi-head prediction without iterative regression on cropped person images.
3. Implementation Details
The implementation details cover training inputs, inference controls, visualization depth ordering, datasets, occlusion subsets, and evaluation metrics.
- Training: Training uses both cropped single-person images and whole images to accommodate people at diverse scales.
- Inference: Inference filters detections with the Body Center confidence threshold tc and retains the top N people by confidence.
- Inference: For benchmark evaluation, the confidence threshold is tc = 0.2 and the maximum person number is N = 64.
- Depth Ordering: Depth ordering for visualization is approximated using body-center confidence and projected 2D body area, with larger-area meshes brought forward.
- Datasets: The 3DPW subsets 3DPW-PC and 3DPW-OC evaluate person and object occlusion, respectively.
- Evaluation Metrics: Evaluation includes mesh, pose, keypoint, and multi-person detection metrics such as PVE, MPJPE, PMPJPE, 3D PCK, AUC, MPJAE, AP, and AR.
4. Qualitative Results
ROMP is evaluated qualitatively on benchmark and in-the-wild images, including failure cases involving depth ordering, detection, 3D pose, and severe occlusion. Its depth ordering is limited under extreme crowding, although it produces robust pose estimates in crowded scenes.
- ROMP presents qualitative results on evaluation benchmarks and in-the-wild images, alongside failure cases in depth ordering, detection, and 3D pose.CRMH results are also shown on the failure cases for comparison.
- ROMP’s body-level depth ordering is limited in extremely crowded scenes with complex depth relationships.The authors propose fine-grained depth estimation as future work to address this limitation.
- ROMP produces more robust and accurate pose estimation in crowded scenes than CRMH, while CRMH has an advantage in determining multi-person depth ordering.
- Some qualitative examples involve extremely challenging poses and occlusions that may also challenge human perception.