Source-linked AI summary
DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor
Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, Yebin Liu
TL;DR
Single-depth-camera human capture is challenged by occlusion, fast motion, loop closure, real-time constraints, and missing pre-scanned templates. DoubleFusion combines a parametric inner body with a fused outer surface, jointly tracking and optimizing them online. The system reconstructs clothing and inner body shape in real time while improving fast-motion tracking and surface loop closure over prior methods.
Problem
Single-depth-camera reconstruction remains limited by occlusions, computational constraints, loop closure, controlled slow motions, and the need for pre-scanned templates.
Method
DoubleFusion uses a double-layer representation with an SMPL-based inner body and fused outer surface, jointly optimizing body pose, shape, and non-rigid deformation.
Results
DoubleFusion reconstructs clothing, inner body shape, pose, and outer-surface motion in real time from one depth camera, with improved fast-motion tracking and surface loop closure.
Takeaways & Limitations
The system provides visually plausible real-time inner-body reconstruction alongside general clothing capture without pre-scanning efforts.
Takeaways & Limitations
The system over-estimates body size with thick clothing, struggles with very wide cloth and separated outer-surface geometry, and cannot handle human-object interactions.
Abstract
from arXiv · showhide
We propose DoubleFusion, a new real-time system that combines volumetric dynamic reconstruction with data-driven template fitting to simultaneously reconstruct detailed geometry, non-rigid motion and the inner human body shape from a single depth camera. One of the key contributions of this method is a double layer representation consisting of a complete parametric body shape inside, and a gradually fused outer surface layer. A pre-defined node graph on the body surface parameterizes the non-rigid deformations near the body, and a free-form dynamically changing graph parameterizes the outer surface layer far from the body, which allows more general reconstruction. We further propose a joint motion tracking method based on the double layer representation to enable robust and fast motion tracking performance. Moreover, the inner body shape is optimized online and forced to fit inside the outer surface layer. Overall, our method enables increasingly denoised, detailed and complete surface reconstructions, fast motion tracking performance and plausible inner body shape reconstruction in real-time. In particular, experiments show improved fast motion tracking and loop closure performance on more challenging scenarios.
1. Introduction
DoubleFusion addresses single-depth-camera capture challenges by combining an inner parametric body layer with a fused outer surface layer for real-time reconstruction and tracking.
- Single-depth-camera dynamic reconstruction remains constrained by occlusions, real-time computation, loop closure, and the absence of pre-scanned templates.
- DoubleFusion jointly reconstructs general clothing geometry, non-rigid motion, inner body shape, and pose using a double-layer representation.The inner layer is complete and parametric, while the observed outer layer is progressively fused and deformed.
- The inner body layer supplies correspondences and fusion constraints, while the outer surface supplies increasingly strong constraints for online body-shape and pose updates.
- A predefined on-body node graph and a dynamically changing far-body graph preserve body priors while representing surface deformations away from the body.
- Joint tracking uses correspondences from both the inner body and fused outer layer to support fast motion tracking and robust geometry reconstruction.
- Volumetric shape-pose optimization fits SMPL parameters directly in the canonical TSDF volume, improving the body embedding used for outer-surface tracking.
2. Related Work
Prior work spans template-based, model-based, and free-form dynamic reconstruction, with trade-offs among generality, clothing representation, setup requirements, and real-time performance.
- Static template based dynamic reconstruction: Template-based methods convert reconstruction into motion tracking and surface deformation but require scanning templates for different identities or apparel.
- Model-based dynamic reconstruction: Model-based methods use shape priors such as SCAPE or SMPL, but these models cannot represent arbitrary geometry from varied apparel.
- Free-form dynamic reconstruction: Free-form reconstruction avoids geometric priors but couples motion estimation with geometry estimation, making accurate tracking and complete fusion mutually dependent.
- Free-form dynamic reconstruction: Recent free-form systems achieve real-time capture from a single depth camera through hierarchical node graphs, GPU solvers, shading cues, or temporal correspondences.
3. Overview
DoubleFusion represents the performer with an observable outer surface and a parametric SMPL-based inner body, then updates both through a sequential real-time pipeline. Its double node graph combines semantically anchored on-body nodes with flexible far-body nodes for broader surface deformations.
- Double-layer representation: The output is a double-layer surface whose outer layer contains observable regions while the inner layer is a parametric SMPL human shape and skeleton.The system takes a depth stream from a single consumer-level depth sensor.
- Double-layer representation: On-body nodes are predefined on the SMPL model and bind nearby outer regions to semantic body structure.Geodesic distances construct the on-body graph, whose nodes are inherently bound to SMPL skeleton joints.
- Double-layer representation: Far-body nodes extend deformation modeling to outer regions that the on-body graph cannot accurately represent.This preserves flexibility for surface regions distant from the inner body.
- Inner body model: SMPL uses 6890 vertices and 24 joints, with shape and pose parameters defining the body before general blend skinning produces posed vertices.The model uses learned shape and pose-dependent deformations and ten shape blendshapes.
- Initialization: Initialization estimates initial shape and pose from the first depth frame, creates the double node graph, and extracts a triangle mesh from the volume.The performer is assumed to begin in a rough A-pose, and the camera position is fixed.
- Per-frame pipeline: Each frame sequentially performs joint motion tracking, geometric fusion, and volumetric shape-pose optimization.The latter directly optimizes body shape and pose in the increasingly complete canonical fused volume.
4. Joint Motion Tracking
Joint motion tracking optimizes skeletal motion and non-rigid node deformation using correspondences from both the inner body and fused outer surface. Binding and graph regularization preserve articulated on-body motion while allowing independent far-body deformation.
- Graph constraints: The binding term is enforced on on-body nodes, while far-body nodes retain independent non-rigid deformations regularized within their graph.This prevents non-articulated on-body motion without constraining far-body clothing-like regions to the skeleton.
- Tracking objective: The tracking objective combines data, binding, regularization, and pose-prior energies.The data term measures fit between the reconstructed double-layer surface and the depth map.
- Correspondences: Correspondences may lie on either the body shape or fused surface, and body-shape correspondences support fast and robust tracking.Indicator functions distinguish fused-surface, body-shape, and on-body-neighbor cases.
- Motion parameterization: Each canonical vertex is warped by neighboring node transformations, while skeleton-skinned positions and normals provide the articulated motion component.Node transformations use dual quaternion blending, whereas skeletal motion uses linear blend skinning.
- Pipeline context: The broader pipeline places joint motion tracking before geometric fusion and volumetric shape-pose optimization in every frame.This ordering connects motion estimation with subsequent geometry completion and body-model refinement.
- Correspondence search: For each depth point, the system searches for body-shape and fused-surface correspondences and selects using Euclidean distance and normal affinity.Correspondences beyond the 0.1m search radius are eliminated.
- Graph constraints: Graph regularization applies locally as-rigid-as-possible constraints while reducing its effect near joints to permit articulated deformations.Neighboring skinning-weight differences control the reduction around joint regions; far-body nodes use their own graph regularization.
5. Volumetric Fusion & Optimization
DoubleFusion fuses depth into a canonical volume, then jointly optimizes body shape and pose against the fused geometry. Temporal and pose regularization stabilize the fit, while a simplified skinning formulation improves the optimization landscape.
- Volumetric Fusion: Depth observations are non-rigidly integrated into a reference volume after warping voxels to the live frame and updating their TSDF values.Collided voxels are handled to prevent erroneous fusion results.
- Shape-Pose Optimization: The method jointly optimizes SMPL shape parameters and the initial embedding pose directly in the canonical volume after fusion makes the geometry more complete.The optimization addresses mismatch between initial parameters and new observations.
- Energy Formulation: The shape energy combines a volumetric data term, temporal regularization, and a prior that prevents unnatural poses.The data term measures reference-volume misalignment, while temporal regularization maintains consistency with previous parameters.
- Energy Formulation: Bilinear TSDF sampling is valid only when a point’s neighboring nodes are on-body, preventing the body from fitting exterior objects such as backpacks.The transformed body point is evaluated through shape and pose deformation followed by linear blend skinning.
- Optimization Simplification: Ignoring the pose blend shape makes the skinning formulation linear in shape and pose parameters, producing a better sampling-based energy landscape.The paper states that pose blend shapes contribute much less than shape blend shapes to the modified body shape.
6. Results
DoubleFusion reconstructs varied human shapes, motions, and clothing in real time using a double node graph and asynchronous processing. Evaluations report improved tracking, loop closure, and fast-motion performance relative to prior systems.
- Performance: DoubleFusion reconstructs varied body shapes, challenging motions, and different cloth types in loop-closed models.Example results are presented in Figure 5.
- Performance: 32 ms per frame enables real-time operation, with joint motion tracking taking 21 ms, geometric fusion 6 ms, and volumetric shape-pose optimization 3 ms.The pipeline runs on one NVIDIA TITAN X GPU; preprocessing and mesh extraction add less than 1 ms asynchronously.
- Ablations: The double node graph avoids erroneous connections and captures loose geometry beyond the control area of an on-body graph alone.The evaluation contrasts standard uniform surface sampling, on-body nodes alone, and the proposed double node graph.
- Ablations: Using both body and fused-surface correspondences produces more plausible tracking than using either representation alone in the evaluated joint-motion and on-body correspondence studies.Figure 6 also evaluates adding non-rigid registration and full energy terms.
- Ablations: Volumetric shape-pose optimization is evaluated through per-frame tracking error and reconstructed shape-mesh overlap, with and without optimization.The corresponding shape reconstruction error along the sequence is shown in Figure 10.
- Comparison: DoubleFusion obtains smaller per-frame maximum and average tracking errors than BodyFusion, especially during fast motions.The comparison uses BodyFusion’s public Vicon dataset; average numerical errors are reported for the entire sequence.
- Comparison: Qualitative comparison with DynamicFusion and BodyFusion reports improved tracking and loop closure performance for DoubleFusion.DynamicFusion uses general non-rigid registration, whereas BodyFusion uses a human skeletal constraint.
7. Discussion
DoubleFusion reconstructs clothing and inner body shape in real time from a single depth sensor, with improved non-rigid tracking and surface loop closure. The system remains limited by thick clothing, wide cloth, separated surfaces, and human-object interactions.
- The system tends to over-estimate body size with thick clothing, while very wide cloth remains challenging.
- The method cannot handle outer-surface geometry separations or human-object interactions.The authors identify key-volume updates and future work as possible directions for these limitations.
- The system reconstructs both clothing and inner body shape in real time from a single depth sensor.
- DoubleFusion achieves better non-rigid tracking and surface loop closure performance than state-of-the-art methods.