Source-linked AI summary
Putting People in their Place: Monocular Regression of 3D People in Depth
Yu Sun, Wu Liu, Qian Bao, Yili Fu, Tao Mei, Michael J. Black
TL;DR
Monocular depth estimation for multiple people is ambiguous because image scale reflects both depth and height, especially across ages. BEV addresses this with an inferred bird’s-eye-view representation, age-aware body modeling, and weak supervision, achieving stronger multi-person depth, pose, detection, and child-shape results.
Problem
Existing methods do not explicitly reason about relative depth, while monocular depth remains ambiguous without accounting for age-dependent height variation.
Method
BEV is a unified one-stage method that combines front-view and inferred bird’s-eye-view representations with age-aware body modeling and weakly supervised training.
Results
BEV improves relative depth and pose estimation on RH, 3D pose estimation on CMU Panoptic, and detection and child mesh reconstruction on AGORA.
Takeaways & Limitations
Explicit depth-wise representation and weak supervision support multi-person reconstruction across varied ages and overlapping scenes.
Takeaways & Limitations
BEV is not trained for diverse weights, gender, or ethnicity, assumes a constant focal length, and is not designed for crowds of hundreds of people.
Abstract
from arXiv · showhide
Given an image with multiple people, our goal is to directly regress the pose and shape of all the people as well as their relative depth. Inferring the depth of a person in an image, however, is fundamentally ambiguous without knowing their height. This is particularly problematic when the scene contains people of very different sizes, e.g. from infants to adults. To solve this, we need several things. First, we develop a novel method to infer the poses and depth of multiple people in a single image. While previous work that estimates multiple people does so by reasoning in the image plane, our method, called BEV, adds an additional imaginary Bird's-Eye-View representation to explicitly reason about depth. BEV reasons simultaneously about body centers in the image and in depth and, by combing these, estimates 3D body position. Unlike prior work, BEV is a single-shot method that is end-to-end differentiable. Second, height varies with age, making it impossible to resolve depth without also estimating the age of people in the image. To do so, we exploit a 3D body model space that lets BEV infer shapes from infants to adults. Third, to train BEV, we need a new dataset. Specifically, we create a "Relative Human" (RH) dataset that includes age labels and relative depth relationships between the people in the images. Extensive experiments on RH and AGORA demonstrate the effectiveness of the model and training scheme. BEV outperforms existing methods on depth reasoning, child shape estimation, and robustness to occlusion. The code and dataset are released for research purposes.
1. Introduction
The paper introduces BEV, a unified one-stage method that jointly estimates multiple people’s 3D pose, shape, and relative depth from one RGB image. It combines front-view and bird’s-eye-view reasoning with age-aware shape modeling and weak supervision, improving performance across multi-person benchmarks.
- Motivation: Existing multi-person methods do not explicitly reason about people’s relative depth, limiting scene and interaction understanding.Prior approaches estimate individual or grouped people but primarily reason in the image plane.
- Motivation: Crowded scenes expose failures involving occlusion, diverse body sizes, monocular depth ambiguity, and in-the-wild variation.These failures include missed detections, similar predictions for overlapping people, and predictions with similar heights.
- Method: BEV adds an inferred bird’s-eye-view map to its front-view representation for one-shot, parallel coarse detection and fine localization in depth.The bird’s-eye view is inferred from the RGB image rather than observed directly.
- Age-aware modeling: Depth estimation requires modeling age-related height variation, because monocular perspective makes depth and height ambiguous, especially for children and infants.BEV addresses this with age information and a body model spanning diverse ages.
- Training data and supervision: The Relative Human dataset provides weak age and depth-layer annotations, while weakly supervised losses learn relative depth ordering and age-compatible body heights.The mixed age loss allows substantial height variation within each age group.
- Results: BEV improves relative depth reasoning and pose estimation on RH, 3D pose estimation on CMU Panoptic, and detection and child mesh reconstruction on AGORA.Weakly supervised fine-tuning on RH improves results for all age groups, especially young people.
2. Related Work
Prior work estimates multiple 3D people with multi-stage or one-stage pipelines, while depth reasoning is often added through post-processing or 2D representations. These approaches remain limited for overlapping people and age-dependent depth ambiguity.
- Multi-person mesh regression: Most general multi-person methods use two stages: detecting people first and estimating each person’s mesh parameters separately.This separates person detection from per-person 3D body reconstruction.
- Multi-person mesh regression: One-stage methods such as ROMP estimate multiple people simultaneously using front-view heatmaps and parameter maps.ROMP samples mesh parameters at positions parsed from its 2D body-center heatmap.
- Monocular depth reasoning: Earlier monocular depth methods commonly place bodies in depth through post-processing or 2D-based multi-stage reasoning.Examples use bounding-box depth refinement or instance segmentation to distinguish overlapping people.
3. Method
BEV estimates multiple people’s 3D translations by combining front-view and inferred bird’s-eye-view representations, then regresses age-aware body meshes. Its SMPL+A representation and Relative Human supervision address age-dependent shape and relative-depth ambiguity.
- BEV localization: BEV combines front-view and bird’s-eye-view heatmaps to localize people in 3D, then refines positions using offset vectors and camera anchors.The camera anchor map converts discretized 3D-center coordinates into coarse camera-space translations, while offsets refine them.
- Age-aware mesh regression: BEV samples mesh features at predicted 3D centers to regress SMPL+A pose, shape, and age parameters for each person.The mesh feature map provides a 128-dimensional feature vector at each image position, aligned pixelwise with the input.
- Age-aware mesh regression: SMPL+A maps pose, shape, and age offset parameters to a 3D body mesh, interpolating between adult SMPL and infant SMIL templates.Larger age offsets produce lower mesh-template heights.
- Age-aware mesh regression: For infant bodies, BEV switches from AGORA’s adult shape space to SMIL when the age offset exceeds threshold tα.The adult shape space can produce overly large deformations and distorted infant meshes.
- Relative Human dataset: The Relative Human dataset provides weak depth-layer and age annotations for about 7.6K images containing over 24.8K people.Depth layers group subjects whose depth difference is below one body-width, using γ = 0.3m; more than 21% of subjects are young people.
- BEV localization: The bird’s-eye-view heatmap is inferred from a single RGB image and represents likely body centers in depth rather than metric depth.BEV performs coarse detection and fine localization using front- and bird’s-eye-view maps in parallel.
- Training objectives: BEV’s depth loss constrains penalties within a range, avoiding the tendency of ordinal losses to push remote subjects excessively far away.Age supervision uses ranges of the age offset α derived from statistical height data for each age category.
4. Experiments
Experiments evaluate BEV on RH, CMU Panoptic, and AGORA using depth, pose, detection, and mesh metrics, alongside ablations of its representation, training, and localization components.
- Comparisons to the state-of-the-art methods: BEV improves AGORA detection F1 by 5.2% on the kid subset and 2.2% on the full subset.The comparison includes multi-stage state-of-the-art methods and the one-stage ROMP baseline.
- Comparisons to the state-of-the-art methods: On AGORA kids, BEV reduces matched MVE by 19.6% and all NMVE by 26.9% compared with ROMP.On CMU Panoptic, BEV reduces 3D pose errors by 13.9% compared with multi-person state-of-the-art methods.
- Ablation Studies: Without WST, BEV still exceeds CRMH by 4.1% in PCDR0.2 of all on RH and outperforms ROMP across AGORA detection metrics.BEV without WST also outperforms selected state-of-the-art methods in AGORA NMVE and NMJE.
- Ablation Studies: WST improves depth reasoning, especially for young groups, while using both Ldepth and Lage performs best.The ablations separately evaluate Ldepth and Lage and compare them with training without WST.
- Ablation Studies: OM and FVC improve 3D localization granularity, while Ldepth reduces AGORA 3D translation error, especially in depth.FVC uses the front-view body-centered heatmap as an attention signal during bird’s-eye-view estimation.
5. Conclusion, Limitations, Ethics, Risks
BEV introduces bird’s-eye-view reasoning for monocular 3D reconstruction, using relative depth relations and age-group classification to address complex multi-person scenes. The paper also identifies limits in demographic coverage, camera assumptions, crowd size, and potential misuse.
- BEV is a unified one-stage method for monocular regression and depth reasoning of multiple 3D people.
- The method exploits relative depth relations and age-group classification to reason about complex in-the-wild scenes.
- Limitations: BEV is not trained to capture diverse weights, gender, ethnicity, or very large crowds, and assumes a constant focal length.
- Ethics and risks: Monocular 3D pose and shape estimation may support surveillance, tracking, and behavior analysis that threatens privacy; BEV is released for research only.
**Supplementary Material**
The supplementary material provides additional visual results on Internet images and supports further implementation, dataset, comparison, and failure-mode analysis.
- The supplement includes more qualitative results on Internet images.
1. Introduction
The introduction frames the supplementary material as extending the paper’s technical analysis and empirical comparisons. It also highlights additional visual results and failure modes.
- The supplementary material adds implementation details and analysis of the Relative Human dataset.
- It provides quantitative and qualitative comparisons with state-of-the-art methods.
- Additional visual results examine BEV under different situations and explore its failure modes.
2. Implementation Details
The implementation uses a perspective-camera formulation with normalized scale-based representations, explicit front-view to bird’s-eye-view operations, and a coarse-to-fine one-shot localization pipeline. Training combines multiple datasets, balanced sampling, weak RH annotations, and staged optimization, with sliding windows for high-resolution images.
- 2.1. Normalized Camera Representation: Perspective projection uses a standard-camera assumption with fixed field of view and image size when accurate in-the-wild camera parameters are unavailable.The focal length is defined from image size and FOV; supplied camera parameters can be handled through PnP.
- 2.1. Normalized Camera Representation: A normalized scale-based representation makes estimated 3D translation proportional to a person’s image scale.For FOV=60°, the sensitive scale interval si ∈(0, 2) corresponds to di ∈(0.86, +∞) meters.
- 2.1. Normalized Camera Representation: 3D camera anchor maps voxelize camera space with more maps allocated to the 1m–10m depth range, including 25/32 anchors there.
- 2.2. Network Architecture: BEV estimates all people’s 3D translations in one shot through bird’s-eye-view-based coarse-to-fine localization and front-view/bird’s-eye-view map composition.The architecture generates five 2D maps and two 3D maps; 3D center maps combine front-view and bird’s-eye-view heatmaps.
- 2.2. Network Architecture: Front-view features are converted to bird’s-eye view through height-wise suppression and depth-wise exploration before entering the BVH branch.
- 2.3. Datasets and Training: Training uses AGORA, MuCo-3DHP, RH, Human3.6M, COCO, CrowdPose, and other 2D pose data with balanced dataset and age-group sampling.
- 2.3. Datasets and Training: A two-step strategy first learns monocular 3D pose and shape, then adds RH weak annotations; optional AGORA fine-tuning follows.The stages train for 120, 120, and optionally 80 epochs, respectively.
- 2.4. High-Resolution Images: Sliding-window inference applies BEV to evenly divided image grids to preserve performance on high-resolution images.
3. Relative Human Dataset
The Relative Human dataset combines weakly annotated multi-person images with age-group labels and richer occlusion diversity for evaluating depth reasoning in the wild.
- RH contains about 7,689 images with weak annotations for 24,814 people, split into training, validation, and test groups.The splits contain 5,218, 635, and 1,836 images respectively.
- RH annotates 2D poses using keypoints on the face and feet in addition to LSP’s 14 keypoints.The added keypoints represent facial and foot orientations.
- People are classified into baby, kid, teenager, and adult groups using age ranges of 0–3, 3–8, 8–16, and 16+ years.The dataset reports subject counts and proportions for all four age groups.
- RH includes richer subject diversity and more occlusion cases than existing multi-person 3D pose datasets.This makes it suitable for evaluating depth reasoning in unconstrained images.
- Weak annotations are standardized through labeler training, pre-labeled test samples, and professional double-checking.Labelers who pass the training test are employed for official labeling, and annotations are double-checked by professional testers and the author.
4. Discussion
BEV decomposes multi-person 3D localization into front-view and bird’s-eye-view representations, making depth-wise information easier to learn from available supervision.
- BEV disentangles a 3D heatmap into front-view and bird’s-eye-view maps because direct 3D heatmap learning performs poorly with limited multi-person translation annotations.The separable representation uses abundant 2D in-the-wild data for front-view localization and RH weak annotations for depth reasoning.
- The bird’s-eye-view representation provides a separate depth-wise map for multi-person 3D localization.BEV also estimates a 3D offset map to improve localization granularity.
- BEV differs from prior heatmap refinement and decomposition methods by targeting multiple overlapping people with bird’s-eye-view-based representation.Earlier approaches include offset refinement, coordinate-map prediction, and soft-argmax decomposition, while the proposed representation explicitly isolates depth information.
5. Quantitative and Qualitative Results
BEV is evaluated across AGORA, CMU Panoptic, 3DPW, MuPoTS, and high-occlusion settings, with reported gains in child reconstruction, pose accuracy, detection, depth reasoning, and occlusion robustness.
- Quantitative Comparisons: BEV outperforms prior methods on AGORA kids for child mesh reconstruction.The comparison is reported on the dedicated AGORA kids test set.
- Quantitative Comparisons: BEV is on par with previous methods on MuPoTS and 3DPW despite not being tuned for uncrowded scenes.These are reported as sanity-check comparisons.
- Ablation Studies: On the over-70%-occlusion AGORA-PC subset, BEV’s gains over its version without weakly supervised training come from the 3D representation.This subset excludes kids and provides ground-truth 3D annotations for detailed evaluation.
- Ablation Studies: Adding depth encoding reduces mesh reconstruction error under over-70% occlusion and improves stability under occlusion.Depth encoding transfers people at different depths to individual feature spaces for depth-aware mesh regression.
- Qualitative Results: Qualitative comparisons show better detection, depth reasoning, and occlusion robustness than ROMP and CRMH, especially with children.Observed failure cases include mesh intersections from unmodeled contact, very limited visible parts, and dense small-scale subjects.