Source-linked AI summary
SimulCap : Single-View Human Performance Capture with Cloth Simulation
Tao Yu, Zerong Zheng, Yuan Zhong, Jianhui Zhao, Qionghai Dai, Gerard Pons-Moll, Yebin Liu
TL;DR
Single-view human performance capture can produce results that remain far from realistic, particularly for clothing and occluded regions. SimulCap combines a layered body-and-garment representation with physics-based capture, producing realistic visible-region results and plausible occluded cloth dynamics while supporting clothing retargeting.
Problem
Single-view performance-capture results remain far from realistic, and existing methods have difficulty representing cloth-body interactions and occluded clothing deformations.
Source: §1 · p. 1Method
SimulCap digitizes separate undressed-body and garment surfaces, then tracks body motion and cloth through simulation and iterative depth fitting for each frame.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Separate body and clothing capture with physical constraints supports retargeting captured clothing to different bodies.
Source: §1 · p. 2, §6 · p. 8Occluded-region realism depends on simulator quality, very thick clothing remains challenging, and topology changes, faces, hands, and soft tissue are not faithfully reconstructed.
Source: §6 · p. 8
Abstract
from arXiv · showhide
This paper proposes a new method for live free-viewpoint human performance capture with dynamic details (e.g., cloth wrinkles) using a single RGBD camera. Our main contributions are: (i) a multi-layer representation of garments and body, and (ii) a physics-based performance capture procedure. We first digitize the performer using multi-layer surface representation, which includes the undressed body surface and separate clothing meshes. For performance capture, we perform skeleton tracking, cloth simulation, and iterative depth fitting sequentially for the incoming frame. By incorporating cloth simulation into the performance capture pipeline, we can simulate plausible cloth dynamics and cloth-body interactions even in the occluded regions, which was not possible in previous capture methods. Moreover, by formulating depth fitting as a physical process, our system produces cloth tracking results consistent with the depth observation while still maintaining physical constraints. Results and evaluations show the effectiveness of our method. Our method also enables new types of applications such as cloth retargeting, free-viewpoint video rendering and animations.
1. Introduction
Single-view performance capture remains short of realism because existing representations struggle with garment motion, cloth-body interaction, and occluded deformations. SimulCap combines layered body and clothing surfaces with physics-based simulation and depth fitting to capture cloth separately while maintaining physical constraints.
1. Introduction
- Existing capture methods often connect skin and clothing in one surface, limiting representation of layering, sliding, detailed cloth motion, and occluded deformations.Their results may also be non-editable and over-smoothed.Source: §1 · p. 1
- SimulCap combines a multi-layer body-and-clothing representation with physics-based simulation to capture cloth separately, model cloth-body interactions, and match observed data.The simulation is constrained by cloth physics and recovers motion in occluded regions.Source: §1 · p. 1
- Simulating cloth over the tracked body initializes fitting of visible cloth and predicts occluded cloth more accurately than surface skinning or non-rigid warping.Depth fitting is formulated as a physical process to reconstruct observed wrinkles while preserving simulation constraints.Source: §1 · pp. 1–2
- The pipeline digitizes separate undressed-body and garment surfaces, then tracks body skeleton motion and detailed cloth deformation using simulation and iterative depth fitting.The subject turns around once for digitization; capture results are realistic in visible regions and plausible in invisible regions.Source: §1 · p. 2
- The method demonstrates cloth retargeting and is presented as the first live capture method to track body and clothing separately with physical constraints.The introduction motivates single-camera, low-budget capture as a challenging task with potential applications including augmented reality and virtual dressing.Source: §1 · pp. 1–2
2. Related Work
Prior work captures human performance, simulates cloth, or reconstructs cloth geometry, but these lines of work leave gaps in separate garment modeling and single-view recovery of occluded clothing. SimulCap uses simulation during monocular RGBD capture to address the non-visible cloth regions.
2. Related Work
- Many performance-capture methods use a single connected body-and-clothing surface, while some require pre-scanned templates or multicamera setups.These approaches limit detailed clothing deformation and cloth-body interaction capture, or cannot capture occluded regions.Source: §2 · p. 2
- Parametric-body and RGBD methods can capture shape or motion, but the cited approaches do not separate garments from the body or predict occluded cloth deformations.Some body-model methods are restricted to the model's shape space, while some RGB-video methods restrict motion to rotation around the camera.Source: §2 · p. 2
- Cloth simulation can produce dynamics, but realism, controllability, and speed remain open problems, while example-based methods face challenges generalizing to new motions, shapes, and fabrics.Dynamic cloth reconstruction has typically used multiview studios and focused on a garment rather than the person wearing it.Source: §2 · p. 2
- Data-driven clothing models learn deformation over the body, but some cannot separate garments and others require garment-specific learning.SimulCap instead uses simulation during capture to reconstruct occluded deformations from a single camera.Source: §2 · p. 3
- ClothCap jointly estimates body shape, pose, and cloth deformation with separate meshes, but its demonstrated input is 4D scans without occlusion.SimulCap addresses monocular RGBD capture and uses physics-based simulation for non-visible parts.Source: §2 · p. 3
3. Multi-Layer Avatar Digitization
Avatar digitization builds a double-layer surface and segments garments to create separate clothing meshes. The procedure combines depth-based reconstruction with image parsing and volumetric fusion, under a garment-color distinction assumption.
3. Multi-Layer Avatar Digitization
- Double-layer reconstruction produces dressed and undressed body surfaces using DoubleFusion, followed by hole-filling surface reconstruction and remeshing for a manifold suitable for segmentation and simulation.The subject turns around once in front of a depth camera.Source: §3 · p. 3
- Multi-layer avatar generation segments garments from the dressed surface, requiring garment colors to be sufficiently different for robust automatic parsing.The method combines learning-based image parsing with volumetric fusion.Source: §3 · p. 3
- The segmentation initializes from human parsing, estimates garment colors with K-means, and fuses projected labels and color into a parsing volume for 3D segmentation.It uses three labels—upper cloth, lower cloth, and skin—and assumes upper cloth lies outside pants when resolving clothing occlusion.Source: §3 · p. 3
- The undressed-body head is deformed to fit the fused head on the dressed surface to enhance avatar realism.Source: §3 · p. 3
4. Physics-Based Performance Capture
The capture pipeline tracks the undressed body, simulates cloth, then fits it iteratively to depth. It combines physical constraints with depth observations to produce plausible cloth motion, including in invisible regions.
4.1. Body Tracking
- Body tracking uses ICP with a double-layer surface data term, an interpenetration term, and a pose prior to track the undressed body from dressed-depth input.The interpenetration term limits body-depth overlap; the authors say severe overlap can deteriorate subsequent cloth tracking.Source: §4.1 · p. 4
4.2. Cloth Tracking
- Simulated cloth supplies a plausible initial state in visible and invisible regions, while direct depth fitting can cause discontinuities and violate physical constraints.The authors formulate fitting as a physical process to maintain those constraints.Source: §4.2 · p. 4
4.2.1 Cloth Simulation
- The cloth simulator extends a force-based mass-spring model with edge springs for stretching, torsion springs for bending, and collision handling.Spring stiffness can be adjusted to approximate different cloth materials; the system uses explicit Euler integration and handles body-cloth and cloth-cloth collisions.Source: §4.2.1 · pp. 4–5
4.2.2 Iterative Depth Fitting
- Depth fitting acts as an external force on cloth vertices, with nearby depth points blended and a smooth weight scaling the force.The fitting is performed by simulating again with the body static; physical constraints filter non-physical depth observations such as large noise.Source: §4.2.2 · pp. 5–6
- Smooth blending reduces depth-fitting force near cloth boundaries to smooth the transition between simulated and fitted regions.The authors separate cloth simulation from iterative depth fitting because the body-driving and depth-fitting forces may conflict and produce unnatural results.Source: §4.2.2 · p. 6
5. Results
The results present faithful cloth dynamics and compare SimulCap with DoubleFusion and a multi-layer baseline, including in invisible regions. The paper also demonstrates cloth retargeting and free-viewpoint rendering applications.
5. Results
- SimulCap reconstructs faithful cloth dynamics in the system results.Figure 8 presents reconstruction results; the paper highlights the reconstructed cloth dynamics.Source: §5 · p. 6, Figure 8 · p. 7
- A smooth blending mask assigns visible cloth vertices depth-fitting weights that taper around the visibility boundary.The mask is generated from cloth visibility and a distance transform; weights range from 0 (black) to 1 (white).Source: Figure 7 · p. 6
- The physics-based performance-capture step takes 56 ms, including 17 ms for body tracking, 14 ms for cloth simulation, and 20 ms for iterative depth fitting.The implementation uses one NVIDIA TITAN Xp GPU; avatar generation takes 10 s, with 7.9 s for post-geometry processing.Source: §5 · p. 6
5.1. Evaluation
- SimulCap produces more realistic cloth-body interactions, including sliding and separation, than DoubleFusion and the multi-layer baseline.DoubleFusion uses one geometry for body and clothing, while the multi-layer baseline’s ICP-based tracking struggles with these motions under single-view observations and a real-time budget.Source: §5.1 · p. 7
- SimulCap captures more realistic cloth dynamics than the comparison methods, which have limited non-rigid tracking detail or static template details.The paper attributes its results to cloth simulation and iterative depth fitting; DoubleFusion’s temporal fusion can delay or smooth details during fast motion.Source: §5.1 · p. 7
- SimulCap produces more plausible armpit results than the comparison methods, whose results show typical artifacts in that region.The paper attributes those artifacts to erroneous surface fusion, inaccurate skeleton embedding, and smooth warping weights.Source: §5.1 · p. 7
- Iterative depth fitting makes reconstructed cloth dynamics more accurate and more consistent with the depth input.Figure 11 color-codes depth-fitting error for results with and without iterative depth fitting.Source: §5.1 · p. 7, Figure 11 · p. 8
- SimulCap infers plausible cloth dynamics in invisible regions, unlike the previous methods shown for comparison.Figure 10 evaluates the invisible area using an additional RGBD camera for reference; that camera is not used by any method.Source: §5.1 · p. 7, Figure 10 · p. 8
6. Conclusion
SimulCap uses physics-based simulation with a multi-layer body-and-clothing representation to capture cloth-body interactions and infer occluded surfaces from partial observations. Its realism remains constrained by simulator quality, and several clothing and body details remain challenging to reconstruct.
6. Conclusion
- SimulCap models skeletal motion deforming the body and clothing layered on top, enabling cloth-body interaction tracking and surface reconstruction in occluded regions.The paper reports realistic live capture with a single-view RGBD camera and retargeting of captured clothing to different bodies.Source: §6 · p. 8
- Occluded-region realism is limited by simulator quality, and tracking very thick clothing such as sweaters remains challenging.The authors suggest advanced cloth simulators and sewing-pattern information as possible ways to increase realism.Source: §6 · p. 8
- Topology changes, faces, hands, and soft tissue cannot be faithfully reconstructed by SimulCap.The paper notes that capturing natural hand/arm interactions with cloth requires more accurate physics-based collision models.Source: §6 · p. 8