PaperlayerView exact revision on arXiv

Source-linked AI summary

BodyNet: Volumetric Inference of 3D Human Body Shapes

Gül Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, Cordelia Schmid

arXiv:1804.04875v3cs.CV

TL;DR

Estimating 3D human pose and shape from a single natural image is difficult, while parametric-model regression has representational limitations. BodyNet directly predicts volumetric body shape with auxiliary losses and intermediate supervision, achieving state-of-the-art results on SURREAL and Unite the People while enabling volumetric body-part segmentation.

  • Problem

    Estimating accurate 3D human pose and shape from a single image remains challenging, and predefined body-model parameter regression has nonlinear and ambiguity-related limitations.

  • Method

    BodyNet directly infers voxelized 3D body occupancy from one image using volumetric, multi-view re-projection, and intermediate pose and segmentation supervision.

  • Results

    BodyNet achieves state-of-the-art human-shape estimation on SURREAL and Unite the People, while joint training improves results and the model supports volumetric body-part segmentation.

  • Takeaways & Limitations

    The volumetric, multi-task framework provides a trainable building block for applications using 3D body information and extends single-image inference to 3D body-part labeling.

Abstract

from arXiv · show

Human shape estimation is an important task for video editing, animation and fashion industry. Predicting 3D human body shape from natural images, however, is highly challenging due to factors such as variation in human bodies, clothing and viewpoint. Prior methods addressing this problem typically attempt to fit parametric body models with certain priors on pose and shape. In this work we argue for an alternative representation and propose BodyNet, a neural network for direct inference of volumetric body shape from a single image. BodyNet is an end-to-end trainable network that benefits from (i) a volumetric 3D loss, (ii) a multi-view re-projection loss, and (iii) intermediate supervision of 2D pose, 2D body part segmentation, and 3D pose. Each of them results in performance improvement as demonstrated by our experiments. To evaluate the method, we fit the SMPL model to our network output and show state-of-the-art results on the SURREAL and Unite the People datasets, outperforming recent approaches. Besides achieving state-of-the-art performance, our method also enables volumetric body-part segmentation.

1 Introduction

The paper addresses the underexplored problem of estimating 3D human shape from a single uncontrolled image by proposing BodyNet, an end-to-end volumetric inference network. BodyNet combines volumetric prediction with multi-view re-projection and intermediate supervision, achieving state-of-the-art results while enabling volumetric body-part segmentation.

  • Motivation: Single-view 3D human shape estimation remains challenging because of limited training data, high-dimensional outputs, and the difficulty of choosing suitable 3D representations.Prior work largely studied controlled settings with specialized sensors, while deformable-model parameter mappings are highly nonlinear and difficult to learn.
  • Applications: BodyNet also enables volumetric body-part segmentation and can serve as a subnetwork for applications such as virtual cloth change or re-enactment.The predicted volumetric representation supports both body shape and body-part outputs.
  • Method: BodyNet directly predicts likelihoods on a person’s 3D occupancy grid using a volumetric representation.The network is designed for single-image inference rather than regressing predefined human shape-model parameters.
  • Method: BodyNet combines a volumetric 3D loss with a multi-view re-projection loss and intermediate supervision for 2D pose, 2D body-part segmentation, and 3D pose.These auxiliary losses regularize the network and support end-to-end training.
  • Evaluation: The method is evaluated on SURREAL and Unite the People by fitting SMPL to BodyNet outputs, and it demonstrates state-of-the-art single-view 3D human shape estimation performance.The paper reports improvements over previous regression-based and recent methods, including gains from end-to-end training and auxiliary losses.

2 Related work

Prior work estimates 3D human shape by optimizing deformable-body-model parameters or predicting them with neural networks, but these approaches face challenges from nonlinear mappings and 2D detection quality. BodyNet instead uses volumetric reconstruction, distinguishes itself from 2.5D methods, and employs intermediate multi-task cues for shape estimation.

  • 3D human body shape: Earlier methods optimize SCAPE or SMPL pose and shape parameters, with some incorporating silhouettes, temporal priors, or multi-view volumetric representations.These optimization-based approaches include single-image and multi-view video settings.
  • 3D human body shape: Optimization-based methods can produce compelling results but remain limited by the quality of 2D detections.The limitation is stated for methods recovering shape using optimization techniques.
  • 3D human body shape: Neural approaches learn priors from data by predicting 3D shape or SMPL body parameters, although parameter prediction requires learning a highly nonlinear mapping.An early silhouette-based method assumes a frontal input view, while later methods predict SMPL parameters from images.
  • 3D human body shape: Unlike methods estimating 2.5D projections or image-to-surface correspondences, BodyNet generates a full 3D body reconstruction.Related methods include DenseReg, DensePose, and quantized depth maps for SMPL bodies.
  • Multi-task neural networks: BodyNet uses 2D pose, 2D body-part segmentation, and 3D pose as intermediate cues, with 3D pose serving as an auxiliary task for final 3D shape estimation.The paper reports that a joint loss across these tasks increases the performance of all subnetworks.

3 BodyNet

BodyNet directly infers volumetric 3D human body shape from a single image using voxel occupancy, multi-view re-projection, and intermediate 2D/3D representations. Its output can also support 3D body-part segmentation and fitting to the SMPL model.

  • Architecture and training: BodyNet combines four subnetworks to predict 2D pose, 2D body-part segmentation, 3D pose, and 3D shape, trained independently before joint optimization.The full training procedure progressively trains intermediate tasks, 3D shape, re-projection losses, and finally all network weights end-to-end.
  • Volumetric representation: The shape subnetwork represents the body as an occupancy map on a fixed voxel grid centered approximately at the hip joint.Ground-truth SMPL meshes are voxelized into the grid, and the network is trained with binary cross-entropy after sigmoid activation; experiments use 128 × 128 × 128 voxels.
  • Segmentation and model fitting: The volumetric formulation extends naturally to 3D body-part segmentation with six parts and seven classes including background, and its output can be converted and fitted to SMPL.The fitting procedure extracts an isosurface from the occupancy map, then optimizes SMPL pose and shape parameters against the predicted surface and joints.
  • Multi-view re-projection loss: Multi-view re-projection losses address low-confidence limb predictions by increasing the importance of boundary voxels through silhouette supervision.The losses compare projected predictions with ground-truth front and side-view silhouettes, and are added after initial training with the volumetric loss.
  • Intermediate representations: BodyNet’s 3D shape input combines RGB with predicted 2D pose, body-part segmentation, and 3D pose, while intermediate tasks use stacked hourglass subnetworks.The 2D pose network predicts 16 joints, the 2D segmentation network predicts 15 body parts, and the 3D pose network consumes aligned RGB and heatmap modalities.

4 Experiments

Experiments evaluate BodyNet on SURREAL and UP-3D, studying intermediate inputs, supervision and re-projection losses against alternative methods. BodyNet also produces volumetric body-part segmentations directly from images with fast inference.

  • Datasets and evaluation: SURREAL provides large-scale synthetic training and testing data with ground-truth segmentation, 2D/3D pose, and SMPL parameters.Evaluation uses voxel IOU and projected-silhouette quality for 3D shape estimation.
  • Alternative inputs: Adding intermediate cues, especially 3D pose and 2D segmentation, improves 3D shape estimation over single-input alternatives and RGB.The 3D pose network performs best among single-input variants, while combining more cues improves performance.
  • Losses: Re-projection losses improve volumetric predictions under both front-view and front-and-side-view constraints, with multi-view supervision producing better SMPL fits.The voxel network without additional loss already outperforms the described baselines.
  • UP dataset: On UP-3D, BodyNet is fine-tuned from SURREAL pretraining and evaluated with end-to-end multi-view re-projection loss against recent approaches.The dataset’s 3D ground truth is described as imprecise, and comparisons include methods.
  • Body-part segmentation: BodyNet extends volumetric prediction to 3D body-part labeling, achieving 58.9 foreground IOU when parts are combined versus 58.1 in Table 1.The network directly predicts foreground and per-limb voxels in 0.28s and 0.58s per image, respectively, on modern GPUs.

5 Conclusion

The conclusion presents BodyNet as a fully automatic, end-to-end multi-task network that predicts 3D human body shape from a single image. It highlights improvements from intermediate-task training and the effectiveness of volumetric regression with multi-view re-projection loss.

  • BodyNet is a fully automatic, end-to-end multi-task network for predicting 3D human body shape from a single image.
  • Joint training with intermediate tasks significantly improves the results.
  • Volumetric regression together with a multi-view re-projection loss effectively represents human bodies.

BodyNet: Volumetric Inference of 3D Human Body Shapes Supplemental Material · A Qualitative analysis

The supplemental material lists the paper’s authors and their institutional affiliations. The supplied passages do not provide substantive content from the qualitative analysis.

  • BodyNet: Volumetric Inference of 3D Human Body Shapes Supplemental Material: The paper lists G¨ul Varol, Duygu Ceylan, and Bryan Russell as authors.
  • BodyNet: Volumetric Inference of 3D Human Body Shapes Supplemental Material: The author list also includes Ersin Yumer, Ivan Laptev, and Cordelia Schmid.
  • BodyNet: Volumetric Inference of 3D Human Body Shapes Supplemental Material: The authors are affiliated with Inria in France and Adobe Research in the USA.

A.1 Volumetric shape results · A.2 Predicted silhouettes versus manual segmentations on UP

BodyNet produces volumetric 3D body-shape predictions visualized through thresholded outputs, fitted surfaces, and texture mapped from input images. On UP, silhouette comparisons are qualitatively good but limited by occlusions, annotation imprecision, clothing and hair, and imperfect evaluation references.

  • A.1 Volumetric shape results: BodyNet outputs real-valued 3D volumetric body shapes from original RGB images for visualization.The output is thresholded at 0.5 to produce the displayed fitted surface.
  • A.1 Volumetric shape results: The visualization displays fitted surfaces reconstructed from thresholded BodyNet volumetric outputs.The threshold used for the real-valued 3D output is 0.5.
  • A.1 Volumetric shape results: Reconstructed-body texture is automatically segmented and mapped from the original images.
  • A.2 Predicted silhouettes versus manual segmentations on UP: Projected voxel-prediction silhouettes are compared with manually annotated segmentations used as ground truth for Table 3.
  • A.2 Predicted silhouettes versus manual segmentations on UP: The predicted silhouettes are generally good but often disagree with manual annotations because BodyNet completes occluded bodies and predicts clothing and hair.The passage also attributes inconsistencies to imprecise annotations.
  • A.2 Predicted silhouettes versus manual segmentations on UP: Manual segmentation is not used during training; reprojection supervision instead uses SMPL silhouettes, whose overlap with manual segmentation is imperfect.Consequently, 2D metrics have an upper bound, and UP evaluation relies mostly on qualitative results.

A.3 SMPL error · A.4 Failure modes

BodyNet’s SMPL surface errors are concentrated at articulated extremities, where articulation and voxel-grid resolution limit fine-detail recovery. Its failures commonly arise from depth ambiguity and multi-person scenes through pose-estimation errors, while UP annotations introduce additional noise and silhouette evaluation is application-dependent.

  • A.3 SMPL error: 65.8mm surface error is analyzed by body location using color-coded SMPL per-vertex errors.The analysis uses the network from Tab. 2, line 10.
  • A.3 SMPL error: Projected voxel silhouettes are difficult to evaluate because occlusions and clothing make the target definition application-dependent.Applications may prioritize anatomical bodies or full clothing deformation.
  • A.3 SMPL error: The highest errors occur at body extremities because articulation and limited voxel-grid resolution hinder fine-detail capture.The passage identifies these factors as explanations for the spatial error pattern.
  • A.3 SMPL error: Hands and feet contribute the most to SMPL surface error, followed by other articulated body parts.The per-vertex error visualization is reported on SURREAL.
  • A.4 Failure modes: Depth ambiguity and multi-person images often cause pose-estimation failures that propagate to the voxel output.These failure cases are illustrated on UP, including rotated views for depth ambiguity.
  • A.4 Failure modes: UP ground-truth errors can be learned by BodyNet when training on UP, limiting interpretation of failure cases.The annotations are produced by a semi-automatic protocol and may be inaccurate.
  • A.4 Failure modes: The paper provides detailed architecture diagrams for BodyNet’s volumetric shape-estimation and shape-parameter-regression subnetworks.The volumetric subnetwork figure annotates spatial resolutions in red, while the regression subnetwork is described in Sec. 4.2.

B Architecture details … C Performance of intermediate tasks

The paper details BodyNet’s volumetric shape architecture, an alternative SMPL shape-parameter regression subnetwork, and a multi-view-trained 3D body-part segmentation extension. The supplied passages specify their multimodal inputs, bottleneck and fully connected layers, and segmentation-loss training procedure.

  • B.1 Volumetric shape network: The volumetric shape network uses two hourglasses, with each supervised by the same type of losses.Its input differs from other BodyNet subnetworks by combining multiple modalities at different resolutions.
  • B.1 Volumetric shape network: Its first branch concatenates RGB, 2D pose, and segmentation inputs, each specified at 256 × 256 resolution.The passage gives channel configurations of RGB (3 × 256 × 256), 2D pose (16 × 256 × 256), and segmentation (15×256×256).
  • B.2 Shape parameter regression network: The shape-parameter regression network retains the Fig. A.4 input and produces a 10-dimensional SMPL β vector.It vectorizes a 128×4×4 bottleneck output, yielding a 2048-dimensional representation before regression.
  • B.2 Shape parameter regression network: The regression head comprises fully connected layers fc1(2048, 1024), fc2(1024, 512), and fc3(512, 10).These layers map the vectorized bottleneck representation to the SMPL shape parameters.
  • B.3 3D body part segmentation network: The 3D body-part segmentation network initializes from shape-network weights trained without re-projection loss.It is first trained for 3D body parts and then fine-tuned with additional multi-view re-projection losses.
  • B.3 3D body part segmentation network: Segmentation training applies one re-projection loss per part and view, totaling 7*2=14 binary cross-entropy losses.The losses cover 6 parts plus 1 background across frontal and side views.

C.1 Effect of multi-task training

End-to-end multitask fine-tuning improves intermediate 2D pose, segmentation, and 3D pose performance on SURREAL. Multi-task loss balancing is crucial for learning these tasks equally, particularly 3D pose.

  • Effect of multi-task training: Segmentation is evaluated using mean IoU over 14 foreground parts, excluding background, while 2D pose uses PCKh@0.5.These metrics follow the cited evaluation protocols for segmentation and 2D pose.
  • Effect of multi-task training: 3D pose error is averaged over 16 joints in millimeters after centering predictions and ground truth at the root joint.The reported 3D pose error uses root-centered predictions and ground truth.
  • Effect of multi-task training: Joint end-to-end training improves all evaluated intermediate tasks: 2D pose, segmentation, and 3D pose.The comparison is performed before and after multitask fine-tuning on SURREAL.
  • Effect of multi-task training: Loss balancing is crucial for equally learning all tasks, especially 3D pose, which is balanced with a factor 106.Training curves compare models with and without multi-task loss balancing.

C.2 Balancing multi-task losses … D SMPLify++ objective

The supplementary sections describe balanced multi-task weighting, adaptation of BodyNet’s segmentation subnetwork to UP, cue ablations for 3D pose, and the SMPLify++ fitting objective. Together, they detail implementation choices, dataset handling, input-cue effects, and iterative model fitting.

  • C.2 Balancing multi-task losses: Multi-task loss weights are calibrated by equalizing average gradient magnitudes after 100 training iterations.All initial weights are equal and normalized to sum to 1 before relative gradient ratios rescale the individual losses.
  • C.3 2D segmentation subnetwork on the UP dataset: The UP segmentation subnetwork is fine-tuned from SURREAL pretraining and evaluated against.BodyNet’s segmentation network requires 15 classes: 14 body parts plus background.
  • C.3 2D segmentation subnetwork on the UP dataset: UP provides 5,703 UP-3D training images with 31 projected SMPL-part labels, while manual six-part segmentation covers 639 LSP images.The manual annotations come from 639 of the full 1,000-image LSP set, and not all have SMPL ground truth.
  • C.4 Effect of additional inputs for 3D pose: Adding RGB, 2D pose, and 2D segmentation cues progressively improves 3D pose estimation across SURREAL and Human3.6M.The experiment uses networks pretrained on SURREAL and evaluates Human3.6M under protocol 1.
  • C.4 Effect of additional inputs for 3D pose: Combining all intermediate input cues yields the best 3D pose error and achieves state-of-the-art performance.The reported comparison uses 3D pose error in millimeters; the method does not apply a rotation transformation before evaluation.
  • D SMPLify++ objective: SMPLify++ fits SMPL pose and shape by minimizing weighted closest-point distances between predicted and projected silhouette contours.The objective searches for {θ⋆, β⋆} using the predicted contour Sn and model silhouette Ss(θ, β).
  • D SMPLify++ objective: Camera parameters are initialized from hip and shoulder joints, then optimized using 2D/3D joint correspondences for projection.Contour-point weights wn represent pixel distances between predicted contour points and their correspondences.
  • D SMPLify++ objective: The joint-objective weight is set to λj = 100, and contour correspondences are updated iteratively during optimization.The weight is chosen by visual inspection, and the authors note that multiple objectives make tuning difficult.

E Effect of using manual segmentations for re-projection

This section compares manual UP segmentations with SMPL projections as supervision for the front-view re-projection loss. Manual annotations produce significantly better-aligned silhouettes than SMPL projections, according to the reported experiment.

  • Re-projection supervision: The experiment compares manual UP segmentations in the M-network against SMPL projections in the S-network for front-view re-projection supervision.The main-paper experiments did not use UP manual segmentations for training, although 2D evaluation used this ground truth.
  • Re-projection supervision: Manual annotations yield significantly better-aligned silhouettes than SMPL projections on the UP dataset.The comparison is summarized using 2D metrics in Table A.4 and illustrated in Fig. A.8.
  • Qualitative results: The section also presents qualitative volumetric shape predictions on UP.These qualitative results are shown in Fig. A.9.
Loading 1804.04875v3…