Source-linked AI summary
SiCloPe: Silhouette-Based Clothed People
Ryota Natsume, Shunsuke Saito, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, Shigeo Morishima
TL;DR
Single-view reconstruction of complete clothed human bodies is difficult because clothing introduces substantial shape variation and the back view is unobserved. The paper addresses this with silhouette synthesis, deep visual-hull reconstruction, and front-to-back texture generation, producing textured 3D bodies from one frontal image. The authors report effectiveness on synthetic and real data, while noting limitations from biased synthetic training data, pose failures, uncovered accessories, and non-rigged output.
Problem
Reconstructing complete clothed 3D humans from one image is challenging because clothing varies widely and only the frontal projection is observed.
Method
The method synthesizes novel-view silhouettes guided by 3D joints, reconstructs geometry with a deep visual hull, and predicts back-view texture using conditional image-to-image translation.
Results
The authors demonstrate effectiveness on synthetic and real inputs and compare the approach with single-view and multi-view reconstruction methods.
Takeaways & Limitations
The framework provides a non-parametric way to reconstruct textured clothed 3D human geometry from a single frontal image without a parametric model or pre-captured template.
Takeaways & Limitations
The implementation relies on limited, potentially biased synthetic training data; pose estimation can fail, accessories may be unsupported, and the output mesh is not rigged for direct animation.
Abstract
from arXiv · showhide
We introduce a new silhouette-based representation for modeling clothed human bodies using deep generative models. Our method can reconstruct a complete and textured 3D model of a person wearing clothes from a single input picture. Inspired by the visual hull algorithm, our implicit representation uses 2D silhouettes and 3D joints of a body pose to describe the immense shape complexity and variations of clothed people. Given a segmented 2D silhouette of a person and its inferred 3D joints from the input picture, we first synthesize consistent silhouettes from novel view points around the subject. The synthesized silhouettes which are the most consistent with the input segmentation are fed into a deep visual hull algorithm for robust 3D shape prediction. We then infer the texture of the subject's back view using the frontal image and segmentation mask as input to a conditional generative adversarial network. Our experiments demonstrate that our silhouette-based model is an effective representation and the appearance of the back view can be predicted reliably using an image-to-image translation network. While classic methods based on parametric models often fail for single-view images of subjects with challenging clothing, our approach can still produce successful results, which are comparable to those obtained from multi-view input.
1. Introduction
The paper presents a non-parametric pipeline for reconstructing textured 3D clothed humans from a single frontal image. It combines novel-view silhouette synthesis, deep visual-hull reconstruction, and front-to-back texture prediction.
- Motivation: Single-view reconstruction of fully textured, clothed 3D humans is difficult because only a 2D projection is observed and the subject’s back is missing.Clothing creates a wide range of possible shapes and appearances, making complete inference from one image challenging.
- Evaluation: The authors evaluate synthetic and real inputs, ablate major design decisions, and compare against single-view and multi-view reconstruction methods.These experiments are used to assess the method across varied inputs and design choices.
- Contribution: The method is a non-parametric solution that predicts clothed geometry and texture from a single frontal-view image without a parametric model or pre-captured template.It is designed to generalize to new subjects and capture fine-level clothing geometry.
- Method: The shape pipeline predicts multi-view 2D silhouettes from one segmentation, then reconstructs a mesh with a deep visual hull and optimized view sampling.The approach uses 3D pose to guide consistent novel-view silhouette synthesis before visual-hull reconstruction.
- Method: A conditional image-to-image translation framework infers the full-body texture by synthesizing the subject’s unseen back view from the frontal image.The paper treats geometry and appearance as complementary stages of the reconstruction pipeline.
2. Related Work
Prior work uses visual hulls, parametric body repositories, and learned volumetric models to make clothed-human reconstruction more tractable. These approaches remain constrained by sparse-view geometry, calibration requirements, or limited clothing detail.
- Multi-view reconstruction: Visual-hull methods are efficient and robust for multi-view reconstruction but struggle with concavities and fine-scale details when views are limited.Garment deformation and self-occlusion make clothed-human geometry especially complex.
- Multi-view reconstruction: Recent sparse-view volumetric methods reduce the number of required images but still require carefully calibrated input views.Deep networks encode shape priors to model body and garment deformation from highly sparse views.
- Single-view reconstruction: Parametric repositories make single-view body reconstruction more tractable, but deformable models are not suitable for fine-scale clothing details.Later methods incorporate 2D or 3D pose to recover clothed geometry without relying on a template mesh.
3. Method
The method represents clothed human shape through synthesized novel-view silhouettes conditioned on an input silhouette and 3D pose, then reconstructs geometry and completes texture from a single frontal image.
- 3. Method: The system extracts a 2D silhouette and 3D joints, synthesizes silhouettes from novel viewpoints, and feeds them to visual-hull reconstruction.The synthesized views use known camera projections and 3D pose information to guide reconstruction.
- 3.1. Multi-View Silhouette Synthesis: Silhouette synthesis combines binary-cross-entropy reconstruction loss with a patch-based adversarial loss.The adversarial term is reported as critical for producing sharp, detailed silhouettes; without it, outputs become blurry.
- 3.2. Visual Hull Reconstruction: Because conditioned 3D joints may not resolve details such as fingers and garment wrinkles, naive visual hull carving can excessively erode the reconstruction.The method addresses this inconsistency through learned reconstruction and view selection rather than directly applying conventional visual hull algorithms.
- 3.2. Visual Hull Reconstruction: Greedy view sampling selects mutually consistent silhouettes from 12 yaw-oriented view bins before deep visual-hull reconstruction.The process iteratively updates the visual hull as one candidate view is selected from each bin.
- 3.3. Front-to-Back Texture Synthesis: A conditional image-to-image translation network predicts back-view texture from the frontal image and silhouette to complete the reconstructed mesh.Training uses feature-matching, perceptual, and adversarial losses, with λFM = λVGG = 10.0 in the reported experiments.
4. Experimental Results
Experiments evaluate the silhouette-based representation, deep visual hull reconstruction, view sampling, and comparisons with single- and multi-view methods. Results show improved reconstruction quality for clothed bodies, including complex clothing, from single-view input.
- Qualitative Results: The method reconstructs fully textured 3D clothed bodies from varied single-view DeepFashion images, poses, body shapes, and clothing styles.The networks were trained only on synthetically rendered images.
- Silhouette Representation: Silhouette-based representation outperforms direct voxel prediction in matching side-view silhouettes and aligning reconstructed geometry with ground truth.The evaluation reports 2D side-view IoU and 3D Chamfer-distance error.
- Visual Hull Reconstruction: The deep visual hull algorithm outperforms a naive visual hull approach, while greedy view sampling significantly reduces reconstruction error.Using 12 inferred silhouettes, the strategy also performs better than 69% of randomly selected views for the deep visual hull algorithm.
- Visual Hull Reconstruction: Greedy view sampling always outperforms random selection for the naive visual hull method and exceeds 69% of random selections for the deep visual hull method.The comparison uses repeated random selection and Chamfer-distance reconstruction errors.
- Comparisons: Single-view input produces visually comparable reconstructions to native visual hull and Huang et al. methods using multiple views.The comparison uses eight views for the native visual hull and four views for Huang et al.
- Comparisons: Compared with HMR and BodyNet, the approach produces more faithful results for cases involving complex clothed geometry.The comparison is qualitative and concerns methods that focus on body shape using parametric models.
5. Discussion and Future Work
The framework combines novel-view silhouette synthesis, deep visual hull reconstruction, greedy view selection, and front-to-back texture synthesis for monocular reconstruction. Its main limitations include synthetic-data bias, failures on pose or unseen accessories, non-rigged meshes, and no explicit cloth–body separation.
- Framework: The framework predicts textured 3D clothed body geometry from one image without requiring a parametric model or pre-captured template.Its components include adversarial novel-view silhouette synthesis, deep visual hull reconstruction with greedy view selection, and front-to-back texture synthesis.
- Limitations: Limited synthetic training data may be biased toward synthetic rather than real images and can cause failures in pose estimation or handling additional accessories.The authors identify realistic training data as useful but potentially tedious and costly to acquire.
- Limitations: The output mesh is not rigged, so it cannot be directly used for animation.The method also does not explicitly separate clothing geometry from the human body.