Source-linked AI summary
Learning to Reconstruct People in Clothing from a Single RGB Camera
Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, Gerard Pons-Moll
TL;DR
The paper addresses practical, detailed 3D reconstruction of clothed people from a monocular video using only a few frames. Octopus predicts canonical-pose body shape and clothing detail with fused pose-invariant representations and bidirectional bottom-up/top-down processing. It achieves 4mm accuracy in under 10 seconds across three datasets while supporting fully automatic reconstruction.
Problem
Detailed personalized 3D reconstruction from a single monocular RGB camera is challenging because existing practical methods can miss clothing detail or require costly optimization.
Method
Octopus predicts SMPL parameters and canonical T-pose vertex offsets from fused pose-invariant frame codes, then refines them through bottom-up and top-down streams using image cues.
Results
4mm reconstruction accuracy is achieved in less than 10 seconds across LifeScans, PeopleSnapshot, and a third dataset.
Takeaways & Limitations
The method enables fully automatic clothed-person reconstruction from 1–8 monocular video frames, reducing test-time images and computation relative to prior work.
Takeaways & Limitations
Clothing far from the body, including skirts and coats, and hairstyles such as ponytails require a different formulation.
Abstract
from arXiv · showhide
We present a learning-based model to infer the personalized 3D shape of people from a few frames (1-8) of a monocular video in which the person is moving, in less than 10 seconds with a reconstruction accuracy of 5mm. Our model learns to predict the parameters of a statistical body model and instance displacements that add clothing and hair to the shape. The model achieves fast and accurate predictions based on two key design choices. First, by predicting shape in a canonical T-pose space, the network learns to encode the images of the person into pose-invariant latent codes, where the information is fused. Second, based on the observation that feed-forward predictions are fast but do not always align with the input images, we predict using both, bottom-up and top-down streams (one per view) allowing information to flow in both directions. Learning relies only on synthetic 3D data. Once learned, the model can take a variable number of frames as input, and is able to reconstruct shapes even from a single image with an accuracy of 6mm. Results on 3 different datasets demonstrate the efficacy and accuracy of our approach.
1. Introduction
Octopus addresses the cost, speed, and personalization limits of monocular 3D human reconstruction by predicting clothed human shape from a few frames. Its canonical-pose representation and bidirectional prediction streams enable fast, automatic reconstruction using synthetic training data.
- Detailed 3D human shape and appearance are needed for applications including VR/AR, gaming, virtual try-on, and cinematography.
- Scanner and multi-view setups are costly and large, while single-image body-model methods cannot capture personalized shape detail or clothing geometry.
- Octopus predicts 3D human shapes in canonical pose from a few frames of a person rotating before a single RGB camera.
- Bidirectional bottom-up and top-down streams provide 50ms-per-view predictions that are refined using the same images in 10 seconds.
- Predicting canonical T-shape offsets for clothing, hair, and details encourages pose-invariant latent codes that can fuse information across frames.
- Across LifeScans, PeopleSnapshot, and another dataset, Octopus achieves 4mm reconstruction accuracy in less than 10 seconds.
2. Related Work
Prior reconstruction methods trade practicality, flexibility, or detail: many require multiple cameras, depth sensing, careful motion, or expensive optimization. Hybrid approaches combine bottom-up inference with top-down fitting, but conventional body-model fitting remains limited in clothing and detail capture.
- Top-down methods: Top-down methods either deform free-form meshes or volumetric representations, commonly relying on multiple RGB or depth cameras.
- Top-down methods: Top-down model-based methods use parametric body models to regularize fitting, but jointly optimizing one shape and multiple poses is expensive.
- Top-down methods: Depth-based and template-based methods can model clothing offsets, but related systems may require a pre-scanned template or combine SMPL with volumetric tracking.
- Bottom-up methods: Bottom-up learning methods have improved free-form performance capture, yet typically require multiple views and face limited training data and image-feedback challenges.
- Hybrid methods: Hybrid methods combine bottom-up and top-down approaches, while fitting a 3D body model to 2D poses cannot capture clothing or details beyond model space.
- Hybrid methods: The paper's hybrid design predicts clothing, hair, and personalized shapes from a single camera.
3. Method
Octopus reconstructs an animatable clothed human from a few monocular RGB frames by predicting canonical SMPL+D shape and per-frame pose, then refining predictions with differentiable image and joint losses. Its architecture combines pose-invariant multi-view shape encoding with bottom-up and top-down processing, and can train with 3D or weak 2D supervision.
- Overview: Octopus targets fast, fully automatic reconstruction of body shape, hair, and clothing from a few frames of a monocular RGB video.The cooperative setting uses people rotating before a camera while holding a rough A-pose.
- Shape representation: The model represents each subject with SMPL parameters for undressed shape and pose plus vertex offsets for clothing, hair, and other details.SMPL+D remains differentiable with respect to pose, shape, and free-form deformations, enabling integration into the CNN.
- Model and data representation: The CNN predicts body shape, personal and scene-specific features, per-image 3D poses, and positions from semantic images and corresponding 2D joints.RGB inputs are simplified into semantic segmentations and 2D keypoints so the network can train on synthetic data and generalize to real data.
- Model and data representation: Images from different poses are encoded into latent codes and fused into a single shape code that predicts canonical T-pose SMPL shape and free-form deformations.Predicting the T-shape encourages the per-frame latent codes to be pose-invariant and supports varying numbers of views.
- Loss functions: Training combines canonical-space and posed-space vertex losses with joint, projection, silhouette, pose, undressed-body, and differentiable rendering losses.The undressed-body loss keeps offsets from explaining the subject’s overall body shape.
- Instance-specific top-down optimization: At test time, selected latent layers are optimized using 2D image and joint projection losses to refine bottom-up predictions without 3D annotations.This top-down refinement balances the learned shape manifold against instance-specific features not learned by the network.
4. Experiments
Experiments evaluate Octopus’s architecture, reconstruction accuracy, view-count and optimization trade-offs, supervision requirements, and comparisons with prior methods across multiple datasets.
- Network architecture: The network separates pose-dependent and pose-invariant latent codes, predicting per-frame pose and translation alongside T-pose body shape and per-vertex clothing and hair offsets.A graph-convolution decoder predicts the free-form offsets, and the model is end-to-end trainable.
- View-count analysis: After optimization, reconstruction error saturates at around 8 input images under the fixed 10-second budget, while additional views provide finer details and more accurate texture calculation.Before optimization, performance saturates at around five images; more views reduce the optimization steps available within the time budget.
- Type of supervision: Training with 10%, 20%, or 50% full supervision produces almost identical results, suggesting that strong pose predictions can support training with minimal full supervision.The experiment uses ground-truth poses to isolate the effect of the amount of full supervision.
- Qualitative comparisons: Compared with an optimization-based method using 120 frames and 122 minutes, Octopus uses eight images and remains visually comparable while requiring a fraction of the data and time.The comparison uses approximately 20 seconds of optimization for Octopus and reports a difference of several magnitudes in speed.
5. Discussion and Conclusion
Octopus reconstructs clothed people from a few monocular video frames, combining fast feed-forward prediction with top-down refinement for accurate, image-aligned results. The method supports practical avatar acquisition while remaining constrained for garments and hairstyles far from the body.
- Accuracy and refinement: 4mm reconstruction error is achieved after top-down refinement, improving on 4.5mm feed-forward predictions and aligning the model with input silhouettes.The refinement also supports texture mapping by improving silhouette overlap.
- Accuracy and refinement: 8 RGB images produce reconstructions visually on par with an RGB-D method despite using only monocular color input.
- Limitations: Clothing with geometry far from the body, including skirts and coats, and hairstyles such as ponytails require a different formulation.
- Practical acquisition: Fast automatic reconstruction from a few images supports personalized 3D avatars for VR and AR applications.