PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot

Fabien Baradel, Matthieu Armando, Salma Galaaoui, Romain Brégier, Philippe Weinzaepfel, Grégory Rogez, Thomas Lucas

arXiv:2402.14654v2cs.CV

TL;DR

Multi-HMR addresses the challenge of recovering whole-body 3D meshes for multiple people from one image while estimating their camera-space locations. It combines ViT-based detection with cross-attention prediction, uses CUFFS to strengthen hand predictions, and reports competitive or state-of-the-art performance across body-only and whole-body benchmarks.

  • Problem

    Prior methods had not combined efficient multi-person processing, whole-body recovery, camera-space location estimation, and optional camera-aware prediction, despite these capabilities supporting real-world HMR applications.

    Source: §1 · pp. 2–3

  • Method

    Multi-HMR uses a ViT to detect person centers and a cross-attention head to regress SMPL-X whole-body meshes and camera-space locations, optionally adding camera-ray encodings.

    Source: §3.2 · pp. 6–7

  • Results

    ViT-S at 448×448 is competitive on body-only and whole-body datasets at 30 fps, while ViT-L at 896×896 outperforms the state of the art at 5 fps.

    Source: §1 · p. 4, §5 · p. 14

  • Takeaways & Limitations

    Adding CUFFS, which contains close-up subjects with diverse visible hand poses, boosts performance especially on hand-related metrics.

    Source: §1 · p. 4, §4.1 · p. 11

    Patch-level detection can miss multiple people whose centers fall within the same patch; collisions are very rare in training with resolution above 448×448 and 14×14 patches.

    Source: E Limitations · p. 27

Abstract

from arXiv · show

We present Multi-HMR, a strong sigle-shot model for multi-person 3D human mesh recovery from a single RGB image. Predictions encompass the whole body, i.e., including hands and facial expressions, using the SMPL-X parametric model and 3D location in the camera coordinate system. Our model detects people by predicting coarse 2D heatmaps of person locations, using features produced by a standard Vision Transformer (ViT) backbone. It then predicts their whole-body pose, shape and 3D location using a new cross-attention module called the Human Prediction Head (HPH), with one query attending to the entire set of features for each detected person. As direct prediction of fine-grained hands and facial poses in a single shot, i.e., without relying on explicit crops around body parts, is hard to learn from existing data, we introduce CUFFS, the Close-Up Frames of Full-Body Subjects dataset, containing humans close to the camera with diverse hand poses. We show that incorporating it into the training data further enhances predictions, particularly for hands. Multi-HMR also optionally accounts for camera intrinsics, if available, by encoding camera ray directions for each image token. This simple design achieves strong performance on whole-body and body-only benchmarks simultaneously: a ViT-S backbone on $448{\times}448$ images already yields a fast and competitive model, while larger models and higher resolutions obtain state-of-the-art results.

1 Introduction

Multi-HMR targets single-shot, multi-person whole-body mesh recovery that also estimates camera-space locations and can use camera information. Its ViT and cross-attention design is paired with CUFFS training data; small models are competitive in real time, while larger models achieve state-of-the-art results.

1 Introduction

  • The paper frames expressive poses, variable-person processing, 3D placement, and camera adaptability as key requirements for real-world HMR applications.It links precise face and hand capture to AR/VR communication, and efficient variable-person processing to real-time or computation-constrained settings.Source: §1 · p. 2
  • Previous methods had not integrated efficient multi-person processing, whole-body recovery, camera-space location, and optional camera-aware prediction in one model.The comparison distinguishes single-person crop-based methods from multi-person methods that detect and regress properties in the same network.Source: §1 · pp. 2–3, Table 1 · p. 2
  • A ViT detects person centers and a cross-attention HPH uses each detected token to predict SMPL-X pose, shape, and camera-space location, optionally incorporating camera intrinsics.Camera parameters are encoded as Fourier representations of camera-ray directions and added to image-token features.Source: §1 · p. 3, Figure 2 · p. 3
  • CUFFS adds synthetic close-up full-body subjects with clearly visible, diverse hand poses to support single-shot prediction of small body parts.The paper reports that training with expressive subjects close to the camera improves performance across body parts.Source: §1 · pp. 3–4
  • ViT-S at 448×448 is competitive on body-only and whole-body datasets at 30 fps, while ViT-L at 896×896 outperforms the state of the art at 5 fps.The reported speeds are on an NVIDIA V100 GPU.Source: §1 · p. 4

2 Related work

Related work spans whole-body recovery, multi-person recovery, and synthetic training data. Multi-HMR combines single-shot multi-person processing with whole-body recovery and camera-space location estimation, while CUFFS targets visible, diverse hand poses.

Whole-body Human Mesh Recovery

  • Whole-body methods have generally used multiple crops for face and hand detail, whereas Multi-HMR tackles multi-person whole-body recovery in a single shot without high-resolution crops.OSX and SMPLer-X use single-crop approaches for single-person recovery; Multi-HMR instead uses a nonhierarchical feature extractor.Source: §2 · p. 4

Multi-Person Human Mesh Recovery

  • Earlier multi-person methods commonly used a detector followed by per-person mesh estimation, while single-stage methods include ROMP, BEV, and PSVT.Multi-HMR extends the single-shot setting to whole-body meshes, camera-coordinate 3D locations, and optional camera intrinsics, using a cross-attention head.Source: §2 · p. 5

Synthetic data

  • Large-scale synthetic data can outperform real-world data with pseudo-ground-truth fits, and CUFFS is designed to improve single-shot whole-body prediction, particularly for hands.CUFFS contains people viewed from limited distance with diverse, clearly visible hand poses; the paper reports this type of training data is key for regressing whole-body meshes in one shot.Source: §2 · p. 5

3 Multi-HMR

Multi-HMR maps a single RGB image to centered whole-body human meshes and camera-space locations for each detected person. It uses a ViT for image features and a cross-attention head to predict each person’s mesh and depth.

3 Multi-HMR

  • Multi-HMR outputs a centered whole-body mesh and camera-space 3D location for each person in an RGB image.The mesh is represented with SMPL-X parameters for pose, shape, and expression.Source: §3 · p. 5, §3.1 · p. 6

3.1 Preliminaries

  • SMPL-X represents pose, shape, and facial expression in a 10,475-vertex mesh centered on the head, while the pinhole camera model maps 3D locations to image coordinates.Given a point’s depth, the camera intrinsics can also back-project its image coordinates into 3D.Source: §3.1 · pp. 5–6

3.2 Single-shot architecture

  • Patch-level scores detect primary keypoints, and an MLP refines each detected keypoint’s image coordinates with a residual offset.At inference, patches whose scores exceed a threshold are treated as detections.Source: §3.2 · pp. 6–7
  • The Human Perception Head processes detected people in parallel, with each detection query attending to all image patches before predicting pose, shape, and depth.The queries are updated through cross-attention, query self-attention, and an MLP.Source: §3.2 · p. 7, Figure 3 · p. 7
  • Depth is predicted in log-space with a focal-length-normalized parameterization, and the model directly supervises absolute depth using synthetic data with known depth.The cited depth parameterization is reported to improve robustness to focal-length changes.Source: §3.2 · p. 8
  • When camera intrinsics are available, Multi-HMR encodes patch-level ray directions and concatenates the resulting camera embeddings with image features.The ray directions are computed from patch centers using the inverse intrinsic matrix.Source: §3.2 · p. 8

3.3 Training Multi-HMR

  • Training combines binary cross-entropy detection loss with L1 losses for predicted parameters, human-centered meshes, and mesh reprojections.The final objective adds detection and parameter losses to weighted mesh and reprojection losses.Source: §3.3 · pp. 8–9
  • CUFFS provides 60,000 synthetic close-up views of full-body subjects with diverse hand poses, and adding it during training improves hand-pose predictions without degrading other metrics.Its poses draw on BEDLAM, AGORA, UBody, and additional InterHand2.6M hand poses.Source: §3.3 · p. 9

4 Experiments

The experiments ablate architecture, training data, resolution, and camera conditioning, then compare Multi-HMR with prior work on body-only and whole-body recovery, depth estimation, and inference cost.

4 Experiments

  • Evaluation covers body-only 3DPW, MuPoTs, CMU, and AGORA and whole-body EHF, AGORA, and UBody, using mesh, joint, detection, placement, and compute metrics.The reported metrics include PVE, MPJPE, PCK, F1, MRPE, PCOD, inference time, and MACs.Source: §4 · p. 9

4.1 Ablations on model design and training data

  • Large-scale synthetic training performs better than real-world training, CUFFS particularly boosts hand metrics, and combining real and synthetic data performs worse than synthetic data alone.CUFFS contains close-up synthetic subjects with clearly visible hands in diverse poses.Source: Table 2 · p. 10, §4.1 · p. 11
  • HPH converges faster and performs better than the iterative regressor, and processing queries together is beneficial; two layers are favored for efficiency.Increasing HPH layers slightly improves performance, but the authors choose two layers for better efficiency.Source: Table 3 · p. 10, §4.1 · p. 11
  • Higher input resolution consistently improves performance but increases inference time; high resolutions are particularly important for competitive whole-body metrics with ViT-S or ViT-B.A ViT-L at 896×896 takes approximately 120ms per image.Source: Figure 4 · p. 11, §4.1 · p. 11
  • Camera rays with focal-length normalization improve all reported metrics, while camera-intrinsics conditioning improves depth accuracy more than person-centered reconstruction.The camera-embedding ablation compares embedding types and sensitivity to intrinsics for reconstruction and distance estimation.Source: §4.1 · p. 12, Table 4

4.2 Comparisons with the state of the art

  • Multi-HMR outperforms prior multi-person body-only methods even at lower resolution, with a smaller backbone, or as a universal model, and is competitive with or beats whole-body methods on hands and faces.The whole-body comparisons include competitive hand and face performance alongside multi-person predictions.Source: Table 5 · p. 13, §4.2 · pp. 13–14
  • HPH takes 2.5ms for N=512 queries versus 2.3ms for N=5, while the other model components are independent of the number of people.The authors report that the method scales well with the number of people in an image.Source: Table 6 · p. 14, §4.2 · p. 14
  • The camera-aware variant provides accurate distance predictions across datasets and camera parameters, and Multi-HMR significantly outperforms prior work even without intrinsics.The depth-estimation comparison includes results with and without camera-intrinsics information.Source: Table 6 · p. 14, §4.2 · p. 14

5 Conclusion

The conclusion presents Multi-HMR as a single-shot method for multi-person whole-body mesh recovery, with expressive mesh and position estimates and optional camera adaptation. The supplementary material describes implementation, datasets, metrics, CUFFS generation, additional results, ablations, and limitations.

5 Conclusion

  • Multi-HMR is presented as the first single-shot method for multi-person whole-body recovery, estimating expressive body, face, and hand meshes and scene positions while outperforming prior work on each sub-problem.The method adapts to camera intrinsics when available and uses a ViT backbone with a cross-attention prediction head.Source: §5 · p. 14

Appendix

A.1 Implementation details

The implementation uses 448×448 inputs by default, trains ViT models initialized with DINOv2 weights, and specifies the training schedule and compute setup.

A.1 Implementation details

  • Inputs are resized and zero-padded to 448×448 by default, with random horizontal flipping as the only data augmentation.The longest image side is resized to 448 and the smallest side is zero-padded to preserve aspect ratio.Source: §A.1 · p. 19
  • The models use DINOv2-initialized Small, Base, or Large ViT encoders and train for 400k iterations with batch size 8 and initial learning rate 5e-5.Training uses automated mixed precision.Source: §A.1 · p. 19
  • At 448×448 resolution, training takes around 2 days for ViT-S and 5 days for ViT-L on a single NVIDIA V100 GPU.The default detection threshold is τ=0.5, and the model uses neutral SMPL-X with 10 shape components.Source: §A.1 · p. 19

A.2 Datasets descriptions

The evaluation appendix describes the benchmark datasets, evaluation metrics, and training strategies used for Multi-HMR. It reports that UBody improves in-the-wild robustness in the universal model, while balanced fine-tuning brings gains on CMU and MuPoTs.

A.2 Datasets descriptions

  • The benchmarks span synthetic multi-person, outdoor, controlled multi-camera, scan-based whole-body, and real-life settings, with UBody often showing only upper bodies.BEDLAM, AGORA, and 3DPW are among the described datasets; UBody uses an inter-scene split with 55k training and 2k testing images.Source: §A.2 · pp. 19–20
  • Training-set mixes vary across compared methods, and Table 7 reports the datasets each method uses.Source: §A.2 · p. 20, Table 7 · p. 20

A.3 Metrics Descriptions

  • Evaluation covers mesh reconstruction, detection, and spatial location, using metrics including PVE, MPJPE, PCK, precision, recall, F1, and MRPE.Mesh metrics may separately assess face and hand vertices or use Procrustes alignment; normalized joint and vertex errors divide by F1.Source: §A.3 · pp. 20–21

A.4 Universal model and Fine-tuning strategy

  • The universal model combines BEDLAM, AGORA, CUFFS, and UBody; despite UBody’s noisy ground truths, its inclusion improves in-the-wild robustness with little impact on synthetic benchmarks.Source: §A.4 · p. 21
  • Fine-tuning on AGORA, 3DPW, or UBody follows benchmark-specific training, while equal-sampling fine-tuning on BEDLAM, AGORA, and 3DPW brings substantial gains on CMU and MuPoTs.The authors attribute the latter gains presumably to better alignment between the training mix and those datasets’ distributions.Source: §A.4 · p. 21

B The synthetic CUFFS dataset

CUFFS adds close-up synthetic full-body subjects with diverse hand poses to address distant people and limited hand diversity in existing synthetic training data. Qualitative examples show better hand predictions when CUFFS is included.

B The synthetic CUFFS dataset

  • CUFFS targets existing synthetic data in which people are often distant and hand poses lack diversity, leaving hands represented by few pixels in many single-shot training images.Source: B The synthetic CUFFS dataset · p. 21
  • The dataset uses procedural human models fitted to SMPL-X annotations, rendered as single people near the camera so hands are clearly visible.The subjects face the camera at roughly 2.5 meters; annotations are produced by iteratively minimizing distances between corresponding surface points.Source: B The synthetic CUFFS dataset · p. 22
  • Hand-pose diversity is expanded with InterHand2.6M MANO annotations, converted to right-hand format and mirrored to augment left hands.Source: B The synthetic CUFFS dataset · p. 23
  • CUFFS contains about 60k images, with human shapes equally sampled from BEDLAM, AGORA, UBody, and UBody with increased hand diversity.Source: B The synthetic CUFFS dataset · p. 23
  • Hand predictions are shown as better when synthetic CUFFS data is included in training.The authors highlight a qualitative example in which the hands are significantly better predicted.Source: B The synthetic CUFFS dataset · p. 23

C Additional results

Additional tests report strong results on BEDLAM-test and 3DMPB, with the authors noting limited BEDLAM leaderboard comparisons at submission.

C Additional results

  • Multi-HMR significantly outperforms the other methods reported on BEDLAM-test, although the leaderboard then included only single-person methods.Source: C Additional results · p. 24
  • Multi-HMR reaches state-of-the-art whole-body prediction performance on 3DMPB with ViT-S, ViT-B, and ViT-L backbones.Source: C Additional results · p. 24

D Additional ablations

The additional ablations examine backbone pretraining, the primary detection keypoint, and training-loss choices. DINOv2 performs best early, while the head keypoint gives a small margin over other choices.

D Additional ablations

D.1 Backbone pretraining

  • DINOv2 gives the best final performance and faster convergence, but its advantage decreases with longer training and ViT-Pose eventually reaches comparable results.The authors suggest DINOv2 may be most beneficial when training compute is limited.Source: §D.1 · p. 25

D.2 Choice of primary keypoint

  • Results are robust to choosing the head, pelvis, or spine as primary keypoint, with the head yielding a small performance margin.The authors postulate that the head may be less often occluded and retain it as the primary keypoint.Source: §D.2 · pp. 25–26, Table 10 · p. 26

D.3 Training losses on 3D and 2D

The training-loss comparison finds the best performance with supervision in 3D, 2D, and rotation space; DINOv2 backbone initialization leads to faster convergence.

D.3 Training losses on 3D and 2D

  • Adding supervision in 3D, 2D, and rotation space yields the best performance among the tested reconstruction-loss combinations.The tested losses operate on SMPL-X parameters, generated vertices, or both, with reprojection losses added in one configuration.Source: §D.3 · p. 26
  • The loss comparison tests SMPL-X parameter losses, vertex losses, their combination, and the addition of 2D reprojection losses.Source: §D.3 · p. 26
  • Initializing the backbone with DINOv2 leads to faster convergence.Source: Figure 10 · p. 26

E Limitations

The limitations include patch-level detection collisions, difficulty detecting people whose heads are not visible, and concerns about regressing relative joint rotations with SMPL-X.

E Limitations

  • Multiple person centers in one image patch can collide during training, and some detections are impossible at inference.With images above 448×448 and 14×14 patches, collisions remain very rare during training; multiple queries per patch are suggested as a possible improvement.Source: E Limitations · p. 27
  • Multi-HMR sometimes struggles to detect people when the head is not visible, which may relate to using the head as the primary keypoint and the rarity of such data in training.The authors assume more aggressive cropping augmentation could improve robustness to this truncation.Source: E Limitations · p. 27
  • Relative joint rotations may accumulate pose errors, particularly at the hands and legs, while rotations are difficult to regress in their non-Euclidean space.The authors use a 6D representation for regression and identify alternative pose representations as a direction for future investigation.Source: E Limitations · p. 27
Loading 2402.14654v2…