PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepHuman: 3D Human Reconstruction from a Single Image

Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, Yebin Liu

arXiv:1903.06473v2cs.CV

TL;DR

Single-image reconstruction of clothed humans must infer invisible geometry and visible surface detail despite depth ambiguities and self-occlusions. DeepHuman combines SMPL-derived semantic representations with image-guided volumetric and normal-refinement networks, trained on THuman, and reports improved quality and efficiency over state-of-the-art approaches.

  • Problem

    Single-image clothed-human reconstruction must recover invisible parts and visible geometry, whereas prior methods often rely on multiple views or temporal images and SMPL lacks normal-clothing surface detail.

  • Method

    DeepHuman uses SMPL-derived semantic volumes and maps as structured input to an image-guided volume-to-volume CNN with multi-scale volumetric feature transformation and visible-surface normal refinement.

  • Results

    DeepHuman reports improved efficiency and quality over current state-of-the-art approaches and demonstrates robustness on monocular videos.

  • Takeaways & Limitations

    A real-world THuman dataset of about 7000 clothed human meshes supports training for single-image reconstruction that generalizes to natural images.

  • Takeaways & Limitations

    The method depends on accurate SMPL estimation, oversmooths invisible areas, and does not recover fine-scale facial-expression or hand-motion details.

Abstract

from arXiv · show

We propose DeepHuman, an image-guided volume-to-volume translation CNN for 3D human reconstruction from a single RGB image. To reduce the ambiguities associated with the surface geometry reconstruction, even for the reconstruction of invisible areas, we propose and leverage a dense semantic representation generated from SMPL model as an additional input. One key feature of our network is that it fuses different scales of image features into the 3D space through volumetric feature transformation, which helps to recover accurate surface geometry. The visible surface details are further refined through a normal refinement network, which can be concatenated with the volume generation network using our proposed volumetric normal projection layer. We also contribute THuman, a 3D real-world human model dataset containing about 7000 models. The network is trained using training data generated from the dataset. Overall, due to the specific design of our network and the diversity in our dataset, our method enables 3D human model estimation given only a single image and outperforms state-of-the-art approaches.

1. Introduction

DeepHuman targets clothed-human 3D reconstruction from a single RGB image, where invisible geometry and visible details remain difficult to recover. It combines semantic body representations, image-guided volumetric reconstruction, multi-scale feature fusion, normal refinement, and the THuman dataset.

  • Motivation: Single-image clothed-human reconstruction remains challenging because invisible parts and visible-surface geometry must both be inferred.Existing approaches commonly require multiview or temporal images, while SMPL captures minimally clothed body shape and pose rather than normally clothed geometry.
  • Method: DeepHuman uses SMPL-derived 3D semantic volumes and 2D semantic maps to constrain output degrees of freedom and provide network initialization.The semantic representation is generated after estimating the parametric body model’s shape and pose from the input image.
  • Method: An image-guided volume-to-volume CNN fuses multiscale image information into 3D volumes to recover geometry such as hairstyles and clothing contours.The proposed volumetric feature transformation integrates image guidance at different scales.
  • Dataset: THuman contains about 7000 human meshes covering approximately 230 kinds of clothes and randomly sampled poses for training-data generation.The dataset was captured from real people wearing casual clothes and provides outer geometry with medium-level surface detail and texture.
  • Results: The authors report improved efficiency and quality over current state-of-the-art approaches, with robustness demonstrated on monocular videos.The network is trained on a corpus synthesized from THuman and is reported to generalize to natural images.

2. Related Work

Prior human-model reconstruction methods use multiview or temporal imagery, while single-image parsing typically estimates pose, shape, or sparse structure. Existing single-view reconstruction methods face limitations in geometry detail or generalization, motivating DeepHuman’s real-world clothing dataset and design.

  • Human Models from Multiview Images: Multiview systems recover human models using silhouette, stereo, shading, and related cues, but require complicated camera calibration, synchronization, lighting, or large camera arrays.Sparse-view systems reduce setup demands but still use about four camera views for coarse surface detail.
  • Human Models from Temporal Images: Temporal-image methods use single RGB or RGBD cameras but aggregate multiple frames to generate full-body models.RGBD approaches improve fusion robustness through color, shading, articulated priors, topology handling, and human shape priors.
  • Human Parsing from a Single Image: Single-image human parsing spans sparse 2D skeleton estimation, sparse 3D skeleton estimation, dense 2D parsing, and dense 3D shape-and-pose estimation.SMPL and SCAPE enable automatic estimation of body shape and pose from one image through fitting or CNN regression.
  • Single-View Human Model Reconstruction: Earlier single-view reconstruction work was limited by SMPL-like geometry from synthesized training imagery or constrained training sets that hindered pose generalization.These limitations leave a need for detailed reconstruction that generalizes across human poses and clothing.
  • 3D Human Body Datasets: Existing 3D human datasets often provide skeletons or multiview video but lack a 3D model for each frame and show limited apparel diversity.This dataset gap motivates constructing a real-world clothed-human mesh dataset.

3. Overview

DeepHuman reconstructs a clothed person’s full-body surface from one image through a coarse-to-fine pipeline. It estimates a parametric body, converts it into dense semantic and occupancy representations, fuses image features volumetrically, and refines visible details with normals trained using THuman data.

  • 3. Overview: The pipeline addresses depth ambiguity, self-occlusion, and high output freedom by performing reconstruction in a coarse-to-fine manner.It begins with body estimation before reconstructing surface geometry and visible details.
  • Body Estimation: HMR supplies a plausible initial SMPL estimate, while Simplify improves alignment with detected image keypoints.The method combines them sequentially because their predictions have complementary characteristics.
  • Semantic Representation: A dense semantic representation assigns each SMPL vertex a semantic code based on its rest-pose spatial coordinate.This representation makes the parametric mesh usable as structured input to the neural reconstruction pipeline.
  • Volume Reconstruction: The network reconstructs a 128 × 192 × 128 occupancy volume from the semantic volume with RGB-image guidance using multi-scale volumetric feature transformation.The representation combines knowledge from the 2D image and 3D volume to recover surface geometry.
  • Normal Refinement: A projected frontal normal map refined by a U-net enriches visible geometry beyond voxel resolution while reducing memory requirements.The normal representation is designed to capture fine details such as clothing wrinkles on the visible surface.
  • Training and Output: THuman provides real-world clothed-human meshes for supervised training, and the trained network predicts occupancy and frontal normal outputs from an RGB image.The final polygon mesh is extracted from occupancy and refined using the frontal normal map.

4. Approach

DeepHuman reconstructs a 3D human from one RGB image by translating semantic and image features into an occupancy volume, then refining visible normals. Multi-scale volumetric fusion preserves geometric information while differentiable projection connects volume reconstruction to detailed normal refinement.

  • Network Architecture: The network combines an image feature encoder, a volumetric U-Net, and a normal refinement U-Net in an end-to-end architecture.The encoder processes RGB and semantic inputs; the volume network predicts occupancy, and the refiner improves projected normals.
  • Multi-scale Volumetric Feature Transformer: VFT transforms multi-scale 2D image features into 3D feature volumes, preserving local structure while avoiding extra convolutions or bottleneck-only fusion.It uses affine modulation and can operate on original or downsampled feature maps and volumes.
  • Multi-scale Volumetric Feature Transformer: Skip-connections propagate encoder-side transformation information to the decoder, helping preserve geometry details compared with bottleneck latent concatenation.The paper links this multi-scale transformation to improved preservation of surface geometry details.
  • Volume-to-normal Projection: A differentiable volume-to-normal layer projects depth from occupancy, converts it to a vertex map, computes derivatives, and produces normals for refinement.Depth is obtained through a differentiable occupancy-to-depth transformation rather than a nondifferentiable nearest-occupied-voxel operation.
  • Loss Functions: Training combines occupancy reconstruction, silhouette re-projection regularization, and normal-map refinement losses.The formulation includes extended BCE for occupancy, front- and side-view silhouette terms, and cosine distance for normals.

5. THuman: 3D Real-world Human Dataset

THuman is built through a two-stage, single-depth-camera capture pipeline designed to collect textured human meshes in casual clothing and diverse poses. The resulting dataset contains about 7,000 items from 230 subjects and approximately 30 poses per subject.

  • Capture pipeline: A DoubleFusion-based RGBD system captures detailed outer geometry while incorporating an SMPL-derived motion prior.The system simultaneously solves skeleton motion and non-rigid deformation from depth observations, then fuses observed surfaces into a reference volume.
  • Capture pipeline: The two-stage procedure reduces texture blur from fast motion and obtains more complete surface geometry before texture capture.Subjects first rotate slowly and perform surface-completion motions, followed by texture capture and pose imitation.
  • Pose diversity: Random MOSH poses are presented every six seconds for performers to imitate, yielding natural but diverse pose coverage.Raw meshes are subsequently processed with hole filling, remeshing, and isolated-artifact removal.
  • Dataset scale: Approximately 70 hours of capture produced 230 subject characters with about 30 poses each, totaling 7,000 THuman data items.Each item contains a textured surface mesh, a Kinect RGBD image, and an aligned SMPL model.
  • Training corpus: Training data are synthesized by rendering four random viewpoints per model and generating semantic maps, volumes, occupancy volumes, and normal maps, producing about 28K images.The network training uses the textured mesh and accompanying SMPL model from THuman.

6.1. Results

DeepHuman reconstructs clothed human geometry and visible surface details from synthetic and natural images, while also supporting frame-independent reconstruction from monocular video. The reported examples include wrinkles, belts, dress hems, varied poses, and robust results without temporal smoothing.

  • Image results: DeepHuman reconstructs 3D human models and surface details such as cloth wrinkles from synthetic and natural images.The tested natural images are sampled from LIP, while the synthetic subjects are held out from training.
  • Image results: The method recovers surface discontinuities including belts and dress hems while using input images and detected SMPL models.These examples are presented as reconstruction results on natural images.
  • Video application: Applied independently to each monocular-video frame, DeepHuman handles various human poses without temporal smoothness.The video application is described as demonstrating robust performance.

6.2. Comparison

DeepHuman is compared with HMR and BodyNet using matched qualitative inputs and quantitative 3D IoU evaluation. The reported comparisons emphasize more detailed reconstruction, greater robustness under occlusion, and the highest numerical reconstruction accuracy.

  • Qualitative comparison: Figure 6 compares input images with front- and side-view renderings from HMR, BodyNet, and DeepHuman on synthetic and natural images.The methods are color-coded orange, green, and blue, respectively.
  • Competing approaches: HMR regresses SMPL shape and pose parameters from an RGB image, whereas BodyNet directly infers a 128 × 128 × 128 occupancy volume.Both pretrained networks are fine-tuned using the same synthetic training data used for DeepHuman.
  • Qualitative comparison: DeepHuman produces more detailed reconstructions than HMR and BodyNet and is more robust to occluded body parts than BodyNet.The qualitative comparison uses the same images for all networks and converts outputs into triangular meshes.
  • Quantitative comparison: Mean 3D IoU is evaluated on the synthetic test set after resolution matching and z-axis alignment for the compared volumes.HMR outputs are converted to 128×192×128 occupancy volumes, while BodyNet outputs are upsampled by 1.5 and cropped.
  • Quantitative comparison: DeepHuman achieves the most accurate reconstruction numerically, while BodyNet receives the lowest score because it occasionally produces broken bodies.The quantitative metric is the maximum mean Intersection-over-Union after searching for the best z-axis alignment.
  • Representation evaluation: Figure 8 evaluates dense semantic maps/volumes against body-joint heat maps/volumes using training and validation combined reconstruction losses.The caption states that solid lines represent training error and dashed lines validation error.

6.3. Ablation Study

The ablation studies evaluate dense semantic representations, multi-scale volumetric feature transformation, and normal refinement. Results indicate that dense semantic inputs and multi-scale transformation improve reconstruction, while normal refinement reduces surface-normal error.

  • Semantic Volume/Map Representation: Dense semantic maps and volumes reconstruct 3D models more accurately than sparse joint heat maps and volumes.Both representations use ground-truth SMPL inputs, isolating the representation effect from SMPL estimation errors.
  • Multi-scale Volumetric Feature Transformation: Multi-scale volumetric feature transformation reconstructs details such as the girl’s hair bun.The comparison also reports that the finest-scale baseline is more accurate but contains noise.
  • Normal Refinement: Normal refinement lowers the prediction error of surface normals compared with removing the normal projection and refinement modules.The comparison is reported using the synthetic dataset in Table 4 and Figure 11.

7. Discussion

The discussion identifies dependence on SMPL estimation and remaining limitations in invisible-area smoothness and fine-scale detail recovery.

  • Limitations: Erroneous SMPL estimation prevents accurate reconstruction because the method relies on its dense semantic representation.The authors illustrate this failure with an example in Figure 12.
  • Limitations: Invisible regions are over-smoothed, while facial expressions and hand motion remain unrecovered.The paper suggests adversarial training for invisible-area details and specialized face or hand methods for fine-scale features.
Loading 1903.06473v2…