PaperlayerView exact revision on arXiv

Source-linked AI summary

SPEC: Seeing People in the Wild with an Estimated Camera

Muhammed Kocabas, Chun-Hao P. Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, Michael J. Black

arXiv:2110.00620v2cs.CV

TL;DR

Existing 3D human pose and shape methods rely on camera assumptions that often fail in-the-wild images. SPEC estimates perspective camera parameters from a single image and uses them for body reconstruction, improving 3D HPS accuracy over prior methods.

  • Problem

    Existing methods assume weak-perspective projection, large constant focal length, and zero camera rotation, although these assumptions often do not hold for in-the-wild images.

  • Method

    SPEC separately estimates perspective camera parameters from an RGB image with Softargmax-L2 losses, then conditions body reconstruction on the estimated camera information.

  • Results

    SPEC improves 3D HPS accuracy over prior methods on 3DPW and new datasets with challenging camera views and varying focal lengths.

  • Takeaways & Limitations

    Camera geometry can be estimated from images and effectively leveraged to reconstruct more accurate 3D human bodies.

  • Takeaways & Limitations

    CamCalib and SPEC are trained separately because no suitable in-the-wild dataset combines ground-truth bodies with diverse focal lengths and views, while SPEC-MTP is small and used for evaluation.

Abstract

from arXiv · show

Due to the lack of camera parameter information for in-the-wild images, existing 3D human pose and shape (HPS) estimation methods make several simplifying assumptions: weak-perspective projection, large constant focal length, and zero camera rotation. These assumptions often do not hold and we show, quantitatively and qualitatively, that they cause errors in the reconstructed 3D shape and pose. To address this, we introduce SPEC, the first in-the-wild 3D HPS method that estimates the perspective camera from a single image and employs this to reconstruct 3D human bodies more accurately. First, we train a neural network to estimate the field of view, camera pitch, and roll given an input image. We employ novel losses that improve the calibration accuracy over previous work. We then train a novel network that concatenates the camera calibration to the image features and uses these together to regress 3D body shape and pose. SPEC is more accurate than the prior art on the standard benchmark (3DPW) as well as two new datasets with more challenging camera views and varying focal lengths. Specifically, we create a new photorealistic synthetic dataset (SPEC-SYN) with ground truth 3D bodies and a novel in-the-wild dataset (SPEC-MTP) with calibration and high-quality reference bodies. Both qualitative and quantitative analysis confirm that knowing camera parameters during inference regresses better human bodies. Code and datasets are available for research purposes at https://spec.is.tue.mpg.de.

1. Introduction

Single-image 3D human pose and shape estimation is difficult because existing methods simplify camera geometry, assumptions that fail for many real-world images. SPEC estimates perspective camera parameters from in-the-wild RGB images and uses them for body reconstruction.

  • Motivation: Existing HPS methods commonly assume weak-perspective or orthographic projection, a large constant focal length, and zero camera rotation.These assumptions entangle body and camera orientation and reduce the camera model to three translation parameters.
  • Motivation: Perspective effects such as selfie foreshortening violate these assumptions and lead to errors in pose, shape, and global orientation.The assumptions are more plausible when people are far from the camera and roughly perpendicular to its principal axis.
  • SPEC framework: SPEC estimates perspective camera information directly from a single in-the-wild RGB image and uses it to reconstruct 3D human bodies.The framework has separate camera-calibration and body-reconstruction components.
  • Camera calibration: Softargmax-L2 and its asymmetric variant preserve target-space distances and reduce the penalty for overestimated focal length during camera calibration.The asymmetric loss reflects the observation that HPS accuracy is more sensitive to focal-length underestimation than overestimation.
  • Body reconstruction: Estimated camera parameters are integrated into both SMPLify-X optimization and direct HPS regression.For direct regression, the camera enters reprojection loss and is appended to CNN image features to help disentangle camera and body orientation.
  • Datasets: SPEC introduces SPEC-SYN and SPEC-MTP, datasets providing varied camera conditions with ground-truth or reference camera and body annotations.SPEC-SYN is photorealistic and synthetic, while SPEC-MTP is crowdsourced and collected through calibrated videos from different angles.

2. Related Work

Prior calibrated-camera methods depend on known parameters and controlled capture, while monocular HPS methods typically simplify camera geometry. Single-image calibration methods address camera estimation but have used losses that ignore ordering in continuous targets.

  • Calibrated-camera methods: Calibrated-camera 3D human methods assume known intrinsic or extrinsic parameters and generally require offline calibration.These methods are typically limited to controlled settings and may overfit cameras used during training.
  • Monocular HPS: Monocular HPS methods estimate body parameters by fitting image features or regressing directly from pixels, while assuming weak perspective, orthographic projection, or a fixed large focal length.They also assume zero camera rotation, entangling body rotation with camera rotation.
  • Camera assumptions: Some prior work improves results by replacing the focal-length constant with a value closer to ground truth or by jointly estimating camera viewpoints and 3D poses.The cited datasets remain limited in camera viewpoint and focal-length diversity.
  • SPEC: SPEC targets in-the-wild settings with varied camera intrinsics and viewpoints, unlike the preceding methods.Its stated scope is single-image 3D human body reconstruction.
  • Single-image calibration: Single-image camera calibration methods discretize continuous rotation into bins and train with cross-entropy or KL-divergence losses.SPEC proposes losses that retain distance and ordering in the original target space.

3. Method

SPEC replaces simplified camera assumptions with single-image estimates of perspective camera rotation and focal length, then incorporates these parameters into optimization- and learning-based human-body reconstruction.

  • Camera model: Existing methods commonly assume weak-perspective projection, fixed large focal length, and zero camera rotation, which can entangle camera and body orientation.These assumptions fail under significant pitch, smaller focal lengths, and perspective foreshortening.
  • Camera calibration: SPEC estimates camera pitch, roll, and vertical field of view from a full-frame RGB image, assuming zero yaw and equal focal lengths.The field of view is converted to focal length using the image height; camera translation is fixed at the origin while body translation is estimated downstream.
  • Camera calibration: CamCalib uses full-image geometric cues, including vanishing points and lines, to predict camera parameters shared across subjects in the image.The calibration network receives the uncropped image rather than a person-centered crop.
  • Camera calibration: An asymmetric field-of-view loss penalizes overestimates with standard L2 while saturating the penalty for underestimates using a Geman-McClure function.This design reflects the reported greater sensitivity of reconstructed poses to focal-length underestimation than overestimation.
  • Learning-based approach: SPEC: In the learning-based approach, camera parameters are appended to CNN image features and used in the reprojection loss to condition SMPL pose, shape, and translation estimation.This conditioning is intended to disentangle camera rotation from body orientation.
  • Optimization approach: SMPLify-X-cam: In SMPLify-X-cam, estimated camera rotation and intrinsics enter the perspective projection used to compare projected SMPL joints with detected 2D keypoints.The method retains pose and shape priors while modifying the data term to account for perspective camera parameters.

4. Experiments

Experiments evaluate SPEC’s camera calibration, world-coordinate metrics, datasets, and comparisons with prior HPS methods. Results show that explicit camera modeling improves robustness and reconstruction quality, especially under varied viewpoints and focal lengths.

  • Evaluation metrics: W-MPJPE and W-PVE evaluate reconstruction errors in world coordinates, avoiding the camera-rotation masking caused by Procrustes alignment.These metrics are intended to reveal errors that standard camera-coordinate or aligned metrics can hide.
  • Camera calibration: CamCalib predicts camera pitch, roll, and vertical field of view using discretized bins and Softargmax-based losses, with Softargmax-biased-L2 selected as the final variant.The calibration network uses separate prediction heads and is trained on Pano360.
  • Comparison to prior work: SPEC’s gains over prior methods are larger on SPEC-MTP and SPEC-SYN, whose focal lengths and viewpoints vary more than those of 3DPW.The improvement in PA-MPJPE is smaller, suggesting that the largest gains concern world-coordinate placement and orientation rather than articulated pose alone.
  • Robustness: On SPEC-SYN, SPEC remains similar across camera settings, whereas HMR∗ becomes less robust as viewpoints and focal lengths depart from its assumptions.Qualitative results also show more globally coherent SPEC bodies in side-view images.
  • Ablations: SPEC conditions body regression on estimated camera parameters and uses them during projection, improving W-MPJPE and W-PVE over progressively stronger camera-model baselines.The ablation replaces the fixed focal length and identity rotation with estimated camera parameters before adding camera conditioning.

5. Conclusion

The paper concludes that camera geometry can be estimated from images and used to improve 3D human pose and shape accuracy. SPEC addresses simplifying camera assumptions and introduces datasets supporting evaluation and future research.

  • Conclusion: SPEC estimates a perspective camera from a single image and uses it to reconstruct 3D human bodies more accurately.The method targets weak-perspective projection, fixed focal length, and zero camera-rotation assumptions.
  • Conclusion: Estimated camera parameters improve both state-of-the-art camera regression methods and human pose and shape regression methods.The paper also introduces SPEC-MTP and SPEC-SYN with camera and 3D body annotations.

A. Methods

The methods transform subject-relative predictions into full-image camera coordinates and train camera calibration with differentiable, distance-aware losses. Camera rotation is separated from body orientation for projection and downstream visualization.

  • Camera and translation: SPEC transforms camera translation defined relative to a subject bounding box into a vector relative to the original full image.The transformation uses the bounding-box center and dimensions together with the estimated focal length.
  • Camera calibration: CamCalib discretizes pitch, roll, and vertical field of view into 256 bins while retaining continuous distance information through Softargmax-L2 losses.The probability mass from each prediction head is converted into an expectation-valued prediction.
  • Calibration losses: Softargmax provides a differentiable expectation instead of the non-differentiable argmax operation.For pitch and roll, SPEC applies standard L2 loss; for vertical field of view, it uses an asymmetric loss that penalizes underestimation more strongly.
  • Visualization: SPEC places a virtual ground plane just below the SMPL mesh using the minimum mesh height.This simple plane parameterization is feasible because camera rotation and body orientation are disentangled.
  • Optimization-based reconstruction: SMPLify-X-cam replaces the original weak-perspective camera with estimated intrinsic and rotation parameters and initializes optimization from HMR-EFT.The modified optimization runs in two stages using Adam for 100 steps per stage.

B.1. SPEC-MTP Dataset

SPEC-MTP extends the Mimic The Pose protocol to collect multi-view videos and camera-calibration evidence, then fits SMPL-X bodies and cameras through staged optimization.

  • Dataset collection: SPEC-MTP records each posed subject from multiple viewpoints rather than collecting a single image per pose.The protocol follows a Mannequin-Challenge-style video setup.
  • Dataset collection: Participants record a calibration-pattern video and photograph the grid beside a ruler to verify its measured size.
  • Camera-aware fitting: SMPLify-XC-Cam fits SMPL-X pose, shape, camera pitch, roll, yaw, and translation to the collected videos.It extends the three-stage SMPLify-XC optimization routine.
  • Dataset use: SPEC-MTP is used only for evaluation, while its examples show computed SMPL-X fits for collected frames.

C.1. Training Datasets

SPEC trains with synthetic, in-the-wild, and conventional 3D human datasets, using fixed sampling ratios that emphasize SPEC-SYN and COCO.

  • Training datasets: The training corpus includes MPI-INF-3DHP, Human3.6M, COCO, and SPEC-SYN.MPI-INF-3DHP contributes 90K images and the selected Human3.6M subjects contribute 292K images.
  • Training schedule: SPEC is first trained for 175K steps with 50% SPEC-SYN and 50% COCO in each batch.
  • Training schedule: Training then continues for around 50K steps with 20% Human3.6M, 20% MPI-INF-3DHP, 50% SPEC-SYN, and 50% COCO.This phase continues until convergence.
  • CamCalib results: CamCalib qualitatively estimates reasonable camera parameters for most examples, while failures occur in portrait images with little background information.Human bodies still provide useful calibration cues in these difficult cases, which the authors leave for future work.

C.3. SPEC MPJPE/PVE Results and Discussions

SPEC is evaluated against state-of-the-art methods on SPEC-MTP, SPEC-SYN, and 3DPW using pose, vertex, and world-coordinate metrics. It improves aligned and world-coordinate performance, especially on datasets with diverse cameras.

  • Benchmark results: SPEC yields better pure body pose according to improved PA-MPJPE.
  • Benchmark results: SPEC outperforms the cited SOTA methods in MPJPE and PVE, but is on-par or slightly worse than the matched HMR∗ baseline.MPJPE and PVE additionally measure discrepancies in rotation relative to Procrustes-aligned metrics.
  • Metric interpretation: MPJPE and PVE encode dataset-specific camera information because they are typically computed in camera space.The paper therefore advocates W-MPJPE and W-PVE for in-the-wild settings with unknown, diverse cameras.
  • World-coordinate evaluation: W-MPJPE and W-PVE consistently favor SPEC over HMR∗ on SPEC-MTP and SPEC-SYN, which contain diverse camera parameters.These world-coordinate metrics evaluate performance without access to camera information.
  • World-coordinate evaluation: On 3DPW, SPEC attains improved or on-par W-MPJPE and W-PVE results compared with HMR∗ despite the dataset’s single focal length.

C.4. SMPLify-X-cam Results

SMPLify-X-cam improves optimization-based reconstruction by using estimated camera rotation and focal length, while focal-length errors are more damaging when underestimated than overestimated.

  • Camera components: Using CamCalib camera rotation consistently improves W-MPJPE and W-PVE by improving global orientation reconstruction in world coordinates.
  • Camera components: On SPEC-SYN, using the estimated camera matrix K improves PA-MPJPE through more accurate projective geometry.
  • Camera components: 3DPW is unsuitable for evaluating focal-length effects because it was captured with a single focal length of 1962 pixels.On this dataset, the f = 2200 approximation performs well because it is close to the dataset focal length.
  • Focal-length sensitivity: HPS quality is sensitive to underestimated focal lengths and less sensitive to overestimation on SPEC-SYN.
  • Focal-length sensitivity: With CamCalib focal lengths, reconstructions lie closer to the low-error basin around the true focal lengths.The method is relatively robust when estimated focal length ranges from 0.7 to 1.3 times the true value.

C.5. Ablation study on SPEC with 3DPW

The 3DPW ablation shows that estimated camera information improves SPEC over the baseline across multiple metrics, while comparisons reveal metric-specific strengths and weaknesses.

  • Appending estimated cameras to image features improves five metrics on 3DPW over the HMR∗ baseline.
  • Using the estimated focal length also improves results over using only the original image center and estimated focal length combination.The passage compares HMR∗ + c + f against HMR∗ + c.
  • SPEC outperforms Wang et al. [68] on PA-MPJPE but performs poorly on MPJPE when evaluated with estimated camera parameters.

C.7. Qualitative results of SPEC

SPEC produces qualitative reconstructions that are on par with or more physically plausible than the matched baseline, but failures arise for limited-background portraits and uncommon viewpoints or poses.

  • SPEC yields bodies that are on par with or more physically plausible than the identically trained baseline.The paper presents these comparisons as qualitative results in Fig. 13.
  • SPEC failures include portrait images with limited background information and cases involving rarely seen viewpoints or poses.Some errors are partially attributed to errors from CamCalib.
Loading 2110.00620v2…