PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

• Single-image 3D inference remains limited in resolution and accuracy because existing representations are ineffective for detailed domain-specific modeling.Source: §1 · p. 2
• PIFu represents clothed-human surfaces with pixel-aligned implicit functions that connect local image features to global 3D context.The method learns per-pixel features and evaluates whether queried points along camera rays lie inside or outside the surface.Source: §1 · p. 2
• The representation preserves input-image details while inferring plausible geometry in largely unseen regions.Source: §1 · p. 2
• PIFu produces high-resolution models of complex hairstyles and clothing, including skirts, scarves, high-heels, and wrinkles matching the image at pixel level.Source: §1 · p. 2
• Regressing RGB values at queried surface points extends PIFu to complete texture inference, while multi-view constraints incorporate additional information from arbitrary numbers of views.Source: §1 · p. 2
• The method is evaluated on challenging real-world clothed-human images and high-end photogrammetry scan datasets, including dynamic reconstructions from video.Source: §1 · p. 2

• Parametric human models capture body measurements and motion but generally produce naked bodies, ignoring clothing, hair, and accessories.Source: §2 · pp. 2–3
• Displacement-based clothing models handle some skin-tight garments but fail on complex topologies such as dresses, skirts, and long hair.Source: §2 · p. 3
• Voxel-based methods directly generate 3D people from images, but their memory requirements limit fine-scale detail.Source: §2 · p. 3
• Sparse-view reconstruction methods combine multiple views but also rely on memory-intensive voxel representations, with some requiring at least two views.Source: §2 · p. 3
• View-synthesis approaches can predict back views for texturing, but they cannot handle self-occluding regions and side views.Source: §2 · p. 3

PIFu Representation

• PIFu combines a fully convolutional image encoder with a continuous MLP-based implicit function whose surface is a level set.The representation avoids explicitly storing the embedded 3D space.Source: §3 · p. 3
• Pixel-aligned image features preserve local image detail while supporting memory-efficient geometry with arbitrary topology.The continuous representation can generate detailed geometry without voxel discretization.Source: §3 · p. 3

Digitization Pipeline

• Surface PIFu predicts a continuous inside/outside probability field, from which an iso-surface is extracted at threshold 0.5.Inference densely samples the field and uses Marching Cubes to extract the implicit surface.Source: §3 · p. 4, §3.1 · p. 4

Texture Inference

• Tex-PIFu extends the implicit output from a scalar to an RGB vector field, directly predicting colors on surface geometry with arbitrary topology and self-occlusion.Texture training samples colors around surface points using normal offsets, rather than restricting color prediction to the exact surface.Source: §3.2 · p. 5

Multi-View PIFu

• Multi-view PIFu decomposes f into embedding and reasoning networks, aggregates view-specific embeddings in 3D world coordinates, and supports more than three views.The aggregated features inform surface and texture predictions from arbitrary available views.Source: Figure 3 · p. 5, §3.3 · pp. 5–6

Datasets

• PIFu is evaluated on RenderPeople, BUFF, and DeepFashion, including ground-truth measurements and diverse complex clothing.The experiments use multiple datasets to assess reconstruction performance across varied settings.Source: §4 · p. 6

Metrics

• The evaluation measures P2S distance, Chamfer distance, and normal reprojection error between reconstructed and ground-truth surfaces.P2S is measured in centimeters from reconstructed vertices to the ground-truth surface.Source: §4.1 · p. 6

Quantitative Results

• PIFu produces pixel-aligned high-resolution surfaces that capture hairstyles and clothing wrinkles, with higher fidelity than voxel-based representations under the same encoder architecture.The comparison attributes the fidelity difference to the greater expressiveness of implicit representations.Source: §4.1 · p. 6

Texture and Multi-View Reconstruction

• PIFu predicts textures directly on the surface mesh, avoiding the projection distortion and silhouette artifacts reported for SiCloPe.The multi-view experiments compare PIFu with LSM and Huang et al. using three-view inputs and examine explicit depth conditioning.Source: §4.1 · p. 6

Qualitative Results

• On DeepFashion images, PIFu handles skirts, jackets, and dresses while recovering high-resolution local details, plausible unseen surfaces, and complete single-image textures.The reconstructed textures support viewing the models from 360 degrees, including dynamic clothed-human performances from a single 2D input video.Source: §4.2 · p. 6

• PIFu spatially aligns pixel-level image information with 3D shape for learning-based geometry and texture inference of clothed humans.Source: §5 · pp. 6–7
• PIFu infers plausible geometry in largely unseen regions while preserving high-frequency details present in the image.Source: §5 · p. 7
• PIFu directly predicts surface textures, avoiding projection artifacts and enabling texture completion in unseen, concave, and side regions.Source: Figure 6 · p. 8, §5 · p. 8
• PIFu handles additional views and improves surface and texture predictions as more views are added.Source: Figure 7 · p. 8, Figure 8 · p. 8, §5 · p. 8
• PIFu produces high-resolution output without voxel representations’ high memory requirements and supports arbitrary-topology textured surfaces.Source: §5 · p. 8
• The authors identify higher-resolution appearance, scale-factor inference, and reconstruction under occlusion or partial framing as future work.Source: §5 · p. 8

Experimental Setup

• The High-Fidelity Clothed Human Dataset contains 491 textured meshes, split into 442 training subjects and 49 test subjects.Each mesh contains about 100,000 triangles and spans varied clothing, shapes, and poses.Source: Appendix I · p. 11
• Precomputed radiance transfer simulates global and local light transport, including ambient occlusion, for higher-fidelity real-image reconstructions.Visibility is precomputed on each surface and represented with spherical harmonics.Source: Appendix I · p. 11

Network Architecture

• PIFu permits any fully convolutional image encoder; the final surface model adapts a stacked hourglass network and uses group normalization for small-batch training.The architecture is not tied to a specific encoder.Source: Appendix I · p. 11
• Surface reconstruction uses a six-layer MLP with skip connections from the image feature and depth, while multi-view features are aggregated by average pooling.The surface MLP ends with sigmoid activation after leaky-ReLU layers.Source: Appendix I · p. 12

Training Procedure

• Surface reconstruction is trained before texture inference, which conditions the texture module on learned surface-reconstruction image features.RMSProp trains surface reconstruction, while Adam trains texture inference.Source: Appendix I · p. 12

Spatial Sampling

• Combining adaptive sampling with σ = 5 cm and uniform sampling produces the best qualitative and quantitative surface-reconstruction results.Uniform sampling alone blurs the decision boundary, whereas smaller surface offsets sharpen it but increase artifacts.Source: Appendix II. Additional Evaluations · p. 12

Network Architecture

• The appendix compares stacked hourglass, ResNet, and VGG-based encoder architectures on the evaluation data.The associated figures visualize reconstructed geometry and point-to-surface error.Source: Appendix II. Additional Evaluations · pp. 12–13, Figure 10 · p. 12
• ResNet performs slightly better in-domain but overfits and generalizes poorly to BUFF and DeepFashion, motivating the stacked-hourglass final model.The comparison uses three surface-reconstruction architectures.Source: Appendix II. Additional Evaluations · p. 13

Voxel Comparison

• PIFu aligns reconstructions with image pixels at higher resolution than voxel regression and supports texturing arbitrary-topology, self-occluding shapes.The voxel representation suffers misalignment from limited precision.Source: Appendix III. Additional Results · p. 13, Figure 12 · p. 13

General Objects

• Preliminary class-agnostic ShapeNet experiments reveal new challenges when extending PIFu from clothed humans to general objects.The authors speculate that greater shape variety makes globally coherent reconstruction difficult from pixel-level features alone.Source: Figure 13 · p. 13, Appendix III. Additional Results · p. 14

Video Sequences

• PIFu reconstructs video frames independently yet produces reasonably temporally coherent results after fixed cropping and scaling per sequence.The video ground truth uses eight views with controlled lighting, while PIFu uses a single view.Source: Appendix III. Additional Results · p. 14, Figure 14 · p. 15