Source-linked AI summary
PromptHMR: Promptable Human Mesh Recovery
Yufu Wang, Yu Sun, Priyanka Patel, Kostas Daniilidis, Michael J. Black, Muhammed Kocabas
TL;DR
Existing HPS methods struggle to combine scene context, reliable person localization, and auxiliary semantic information in challenging scenes, while recent language-based methods remain below state-of-the-art accuracy. PromptHMR uses full-image transformer features with spatial and semantic prompts to estimate SMPL-X humans, and reports state-of-the-art performance across diverse benchmarks and scenarios. Its prompting framework supports robust localization, shape refinement, interaction modeling, and temporally coherent video reconstruction, but semantic prompts currently require user supply.
Problem
Existing HPS approaches have limited mechanisms for using side information and trade off cropped-image accuracy against whole-image scene context and reliable detection.
Method
PromptHMR combines full-image vision features, spatial and semantic prompt encoding, and transformer decoding to generate SMPL-X body parameters.
Results
PromptHMR achieves state-of-the-art accuracy across diverse benchmarks and supports robust reconstruction in crowded scenes, body-shape prompting, interactions, and videos.
Takeaways & Limitations
Flexible prompting provides a unified way to incorporate scene, spatial, and semantic information into HPS estimation.
Takeaways & Limitations
Shape-description and interaction prompts are not automatically generated and must currently be supplied by the user.
Abstract
from arXiv · showhide
Human pose and shape (HPS) estimation presents challenges in diverse scenarios such as crowded scenes, person-person interactions, and single-view reconstruction. Existing approaches lack mechanisms to incorporate auxiliary "side information" that could enhance reconstruction accuracy in such challenging scenarios. Furthermore, the most accurate methods rely on cropped person detections and cannot exploit scene context while methods that process the whole image often fail to detect people and are less accurate than methods that use crops. While recent language-based methods explore HPS reasoning through large language or vision-language models, their metric accuracy is well below the state of the art. In contrast, we present PromptHMR, a transformer-based promptable method that reformulates HPS estimation through spatial and semantic prompts. Our method processes full images to maintain scene context and accepts multiple input modalities: spatial prompts like bounding boxes and masks, and semantic prompts like language descriptions or interaction labels. PromptHMR demonstrates robust performance across challenging scenarios: estimating people from bounding boxes as small as faces in crowded scenes, improving body shape estimation through language descriptions, modeling person-person interactions, and producing temporally coherent motions in videos. Experiments on benchmarks show that PromptHMR achieves state-of-the-art performance while offering flexible prompt-based control over the HPS estimation process.
1. Introduction
PromptHMR reframes 3D human pose and shape estimation as promptable reconstruction from full images, combining scene context with spatial and semantic side information. This design targets crowded scenes, occlusion, interactions, shape ambiguity, and video motion while achieving reported state-of-the-art performance.
- Motivation: HPS methods should recover every person in complex scenes and place them in a consistent global coordinate frame.The paper identifies this as the broader problem beyond cropped-person regression.
- Method: PromptHMR combines full-image vision features, multimodal prompt encoding, and transformer decoding to generate SMPL-X body parameters.Prompts include spatial inputs such as boxes and masks and semantic inputs such as text or interaction labels.
- Method: Spatial and semantic prompting provides a flexible way to estimate 3D human pose and shape from whole images.The method is designed to retain scene context while using auxiliary information.
- Capabilities: PromptHMR handles face-sized or varied spatial prompts, improves shape predictions with text, models close interactions, and reconstructs people coherently in camera and world space.A temporal transformer further produces temporally stable and smooth video motion, with metric SLAM used for world-coordinate motion.
- Results: Quantitative experiments across EMDB, 3DPW, RICH, Hi4D, CHI3D, and HBW show performance exceeding state-of-the-art approaches and strong baselines.Qualitative in-the-wild images and videos also illustrate robustness and generalization.
- Implication: The paper presents side-information prompting as a route to improving both accuracy and robustness in HPS estimation.It characterizes the approach as combining VLM semantic knowledge with metric 3D human regression.
2. Related Work
Prior HPS methods trade off scene context against reliable detection, while language and interaction methods remain limited or separate from core regression. PromptHMR addresses these gaps with one flexible, promptable model evaluated across standard and specialized benchmarks.
- Image-based HPS: Cropped-person HPS methods are effective for isolated individuals but discard scene context needed under occlusion, overlap, and close interaction.Their typical output is pose and shape in camera coordinates.
- Image-based HPS: Whole-image detection-and-reconstruction methods can perceive occlusions and depth relationships but often fail while jointly detecting people and estimating HPS.PromptHMR uses detection-box prompts to resolve this conflict.
- Video HPS: Video methods estimate world-coordinate motion through SLAM, temporal models, or monocular metric depth, often using multi-stage or direct-regression strategies.These approaches differ in how camera motion and human motion are connected.
- Semantic HPS: ChatPose and PoseEmbroider combine semantic information with 3D human reasoning, but neither achieves state-of-the-art HPS accuracy.Other work separately studies language-based body shape or interaction reconstruction.
- PromptHMR: PromptHMR trains a single flexible-prompt model for standard HPS benchmarks and benchmarks focused on body shape and person-person interaction.This distinguishes it from approaches that handle semantic or interaction information separately or as post-processing.
3. Method
PromptHMR maps full images and per-person spatial or semantic prompts to SMPL-X body parameters, preserving scene context while supporting flexible reconstruction inputs. Its decoder integrates image, prompt, interaction, and temporal information for single-image and video HPS estimation.
- Promptable mesh regression: PromptHMR maps an image and prompts for each person to pose, shape, and camera-space location represented with SMPL-X parameters.Each human is represented by orientation, local body pose, shape, and translation; face and hand parameters are excluded.
- Promptable mesh regression: Spatial prompts include bounding boxes and segmentation masks, while semantic prompts encode body-shape text and close-contact interaction labels.Semantic prompts are optional, but every reconstructed person requires at least one spatial prompt.
- Promptable mesh regression: A vision transformer encodes the full image, prompt encoders convert modalities into tokens, and a transformer decoder attends to image and prompt tokens to estimate body parameters.Camera intrinsics can optionally be embedded with image features, and the image encoder runs once per frame regardless of the number of prompted people.
- Promptable mesh regression: Bounding-box training uses noisy whole-body, face-region, and truncated boxes so inference can accept different box types without identifying their categories.The box prompt is formed from positional encoding and learned embeddings, with Gaussian noise added to both corners during training.
- Promptable mesh regression: Camera-space translation is estimated from focal-length-normalized 2D translation and inverse depth, then transformed into translation rather than regressed directly.The normalized formulation represents image-plane location and uses inverse depth because it relates linearly to human image size.
- PromptHMR video version: Promptable interaction layers add cross-person self-attention only for prompted interacting people, while the video extension applies a temporal transformer to produce motion and translation over time.The video decoder outputs pose, shape, translation, and joint contact probabilities from temporally encoded SMPL-X output tokens.
4. Experiments
PromptHMR is evaluated across camera-space reconstruction, interactions, world-grounded motion, and multimodal prompting. Results show robust performance with varied spatial prompts, improved shape and interaction accuracy from semantic prompts, and state-of-the-art benchmark performance.
- Reconstruction accuracy: PromptHMR matches crop-based methods while outperforming other full-image methods on camera-space reconstruction.PromptHMR and PromptHMR-Vid achieve state-of-the-art performance in Table 1; PromptHMR and CameraHMR use the same training data and have similar performance.
- Reconstruction accuracy: PromptHMR achieves better overall per-person and inter-person accuracy than BUDDI on interaction reconstruction.The comparison uses BUDDI, which is also trained on CHI3D and HI4D.
- Reconstruction accuracy: PromptHMR-Vid achieves state-of-the-art performance among methods estimating human motion in world coordinates.It achieves lower foot skating than TRAM by estimating joint contact probabilities, while using the same metric SLAM transformation.
- Effect of multimodal prompts: PromptHMR remains stable with varying box prompts, including noisy truncated boxes, and uses full-image context to reconstruct crowded scenes.The model accepts combinations of whole-body, face, and truncated boxes from in-the-wild images.
- Effect of multimodal prompts: Masks improve accuracy when closely interacting people overlap and boxes become ambiguous.The HI4D ablation shows that masks as spatial prompts improve accuracy.
- Effect of multimodal prompts: Text prompts improve HBW shape accuracy during training and testing, especially when perspective effects create ambiguity.Training with shape descriptions alone also improves accuracy when prompts are absent at test time.
- Effect of multimodal prompts: The interaction module improves inter-person Pair-PA-MPJPE on HI4D, especially without HI4D training, while not improving per-person PA-MPJPE when trained on HI4D.This indicates out-of-domain generalization for inter-person reconstruction.
- Qualitative results: PromptHMR recovers coherent 3D scenes and relative positions in crowded and close-interaction examples, although closely interacting people can interpenetrate.Qualitative results use face detections as reliable box prompts in crowded scenes and show some interpenetration during close interaction.
5. Limitations
PromptHMR’s shape-description and interaction prompts currently require user-provided inputs, and future work should automate prompting and explore additional side information.
- Shape descriptions and interaction prompts are not automatically generated and must be supplied by the user.
- Future work should integrate PromptHMR with vision-language models to automate prompting.
- Action descriptions, 3D scene context, and body measurements are additional side-information types that may provide benefits in different scenarios.
6. Conclusion
PromptHMR uses full-image context with spatial and semantic prompts to infer 3D humans, achieving state-of-the-art accuracy and strong in-the-wild generalization across challenging scenarios.
- PromptHMR leverages full-image context with spatial and semantic prompts to infer 3D humans in the scene.
- The method demonstrates state-of-the-art accuracy across diverse benchmarks and generalizes well in the wild.
- Flexible prompting with diverse input information enables robustness and adaptability in challenging scenarios.
7. Additional Results
Additional experiments show that interaction prompting improves close-interaction reconstruction, while the interaction module and video module contribute to qualitative and quantitative performance.
- 7. Additional Results: Qualitative results illustrate the effects of interaction prompting and the video module, with additional PromptHMR-Vid results available in supplementary video.
- 7.1. Interaction Prompting: Introducing interaction prompting improves quantitative results on the HI4D interaction-reconstruction dataset.
- 7.1. Interaction Prompting: Without the interaction module, the model does not effectively reconstruct close interactions even when trained with CHI3D interaction data.
- 7.1. Interaction Prompting: Adding the interaction module improves interacting people’s relative position and orientation while reducing penetration.
- 7.1. Interaction Prompting: Training with both CHI3D and HI4D leads to better interaction-reconstruction results.
8. Experiment Details
The experiments use heterogeneous datasets, simulated spatial and textual prompts, transformer-based encoders and decoders, weighted losses, and established metrics for pose, shape, interaction, and motion evaluation.
- Training Setup: Training combines nine datasets with specified sampling rates, using 896×896 padded and resized images plus rotation and color-jitter augmentation.
- Training Setup: Different annotation types determine which losses are computed, with translation loss omitted for some datasets and only 2D reprojection loss used for COCO and MPII.
- Prompt Construction: Training simulates noisy whole-body, face, and truncated bounding boxes by projecting meshes or keypoints and adding Gaussian corner noise.
- Prompt Construction: Shape prompts are generated from SHAPY attribute scores by composing descriptions containing one to three sampled attributes with gender synonyms.
- Model Configuration: The architecture uses a DINOv2-pretrained ViT-L image encoder, MetaCLIP text encoder, and a three-block SMPL-X decoder with separate MLP regression heads.
- Optimization: Image training uses AdamW on eight H100 GPUs, while video training uses two H100 GPUs, 120-frame samples, and an additional binary-cross-entropy contact loss.
- Metrics: Evaluation reports MPJPE, PVE, PA-MPJPE, interaction-specific Pair-PA-MPJPE, world-coordinate motion errors, trajectory error, jitter, and foot skating.
- Metrics: Metric definitions specify units and alignment procedures, including pelvis alignment for MPJPE and PVE, Procrustes alignment for PA-MPJPE, and trajectory alignment for world metrics.