Source-linked AI summary
Real-time Deep Dynamic Characters
Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt
TL;DR
Realistic, controllable full-body characters are difficult to learn without expensive 3D capture or to animate with dynamic clothing and appearance. This paper learns differentiable geometry and texture from weak multi-view supervision, achieving real-time, free-viewpoint animation with motion-dependent surface detail. Its scope remains bounded by approximate clothing physics, fixed topology, tracking omissions, training cost, and foreground-segmentation quality.
Problem
Existing character pipelines require manual intervention or expensive equipment, while prior multi-view methods omit motion-dependent geometry or view-dependent appearance.
Method
A differentiable, person-specific character model learns full-body non-rigid deformation and motion- and view-dependent texture from skeletal motion using weak multi-view image supervision.
Results
The method produces real-time, controllable, free-viewpoint avatars with dynamic surface deformations and textures, and reports higher visual quality than state-of-the-art approaches.
Takeaways & Limitations
Skeleton motion can drive video-realistic full-body character animation with dynamic clothing and appearance without ground-truth 3D supervision.
Takeaways & Limitations
Clothing dynamics are data-driven approximations, the model cannot handle apparent topology changes, and it omits facial-expression and hand tracking.
Abstract
from arXiv · showhide
We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit space-time coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.
1 INTRODUCTION
The paper targets controllable, realistic digital humans learned from multi-view video, addressing missing clothing dynamics, dynamic textures, and difficult 3D supervision. It introduces a real-time learning-based character model that predicts motion- and view-dependent geometry and appearance.
- Motivation: Realistic, controllable digital characters remain difficult to create because conventional pipelines require manual intervention or expensive equipment.Existing characters can be difficult to control or insufficiently realistic, motivating learning directly from multi-view video.
- Limitations of existing pipelines: Traditional skinning is efficient and controllable but omits non-rigid clothing motion, while physics simulation and physically based rendering require expertise or high computation.Examples include skirt swinging, articulated-pose glitches, manual parameter tuning, and time-consuming rendering.
- Limitations of prior learning-based methods: Prior learned methods rely on registered 3D geometry, have limited realism or clothing coverage, or obtain training data from simulations whose realism depends on the physics engine.These approaches commonly require scanners and challenging template registration.
- Research gap: Multi-view character methods generally lack motion-dependent deforming geometry and view-dependent appearance changes, although texture dynamics are crucial for photorealism.Monocular neural rendering methods also struggle with resolution, quality, and consistency across pose and viewpoint.
- Contribution: The proposed differentiable character predicts full-body non-rigid deformation and dynamic texture from skeletal motion using only weak multi-view image supervision.Analysis-by-synthesis and backpropagation replace precomputed 3D mesh registrations, while geometry carries coherent deformation and texture represents fine details.
- Reported outcome: The system supports motion capture or interactive skeleton editing and produces motion- and view-dependent effects in real time.The authors report qualitative and quantitative results as a step toward photo-realistic, animatable full-body avatars.
2 RELATED WORK
Related work spans video-based character rendering, learned cloth deformation, differentiable rendering, and neural rendering. The paper distinguishes its approach by combining full-body dynamic geometry with motion- and view-dependent textures from multi-view imagery.
- Video-based Characters: Video-based character methods can replay multi-view video but generally cannot generate new human poses.Neural approaches improve realism but full-body photo-realistic rendering remains challenging.
- Video-based Characters: Compared with fixed-camera image-to-image methods, the proposed approach targets arbitrary viewpoints and explicitly models view-dependent appearance without 2D refinement.The cited alternatives can produce missing limbs and other image-space artifacts.
- Learning Based Cloth Deformation: Physics-based cloth simulation is computationally expensive and often requires manual parameter tuning.Learning cloth deformation from skeletal pose or body shape provides a data-driven alternative, but prior representations have limitations across clothing types.
- Learning Based Cloth Deformation: The paper factors cloth deformation into low-frequency graph-based motion and high-frequency per-vertex wrinkles, supporting loose as well as tight clothing.Unlike the cited methods, it also predicts a dynamic texture map for video-realistic controllable characters.
- Differentiable Rendering and Neural Rendering: Differentiable rendering enables learning 3D scene parameters from 2D image supervision, whereas neural rendering learns the rendering process with fewer physical assumptions.The paper uses differentiable character rendering to connect weak multi-view supervision with learned geometry and texture.
3 METHOD
The method learns a poseable full-body character from multi-view 2D images, predicting dense motion- and view-dependent geometry and textures from skeleton motion and camera view. It uses temporal motion information, coarse-to-fine deformation regression, and differentiable rendering without detailed 3D ground truth.
- Method overview: The model takes the current pose and a frame window of past motions to predict motion-dependent geometry and texture.Past motions provide velocity- and acceleration-derived information for clothing deformation.
- Method overview: The character is driven by posing a skeleton and defining a camera view, while training uses only multi-view 2D supervision.This removes the need for detailed 3D ground-truth geometry and annotations.
- Deformation regression: Non-rigid deformation is regressed coarse to fine, beginning with rotations and translations of an embedded graph.The embedded deformation stage is trained using multi-view foreground images as supervision.
3.1 Character Deformation Model
The character deformation model combines a person-specific template, skeletal skinning, embedded graph deformation, and per-vertex displacements in canonical pose. Its differentiable formulation separates pose from learned non-rigid surface deformation.
- Template and skeleton: The person-specific model begins with a scanned, textured T-pose template mesh manually segmented and rigged to a skeleton.The reconstructed mesh is downsampled to approximately 5000 vertices and uses automatically computed skinning weights.
- Embedded Deformation: Dual quaternion skinning applies skeletal pose to the template, but embedded deformation is added to represent non-rigid effects such as skirt swinging.The embedded graph contains around 500 nodes in the reported experiments.
- Embedded Deformation: Each original vertex receives a weighted combination of movements from nearby embedded-graph nodes.Weights are based on geodesic distance and become zero beyond a threshold.
- Vertex Displacements: Per-vertex displacements model fine-scale deformations after the embedded graph captures large deformations.The coarse stage reduces the complexity of learning many fine displacement parameters.
- Character Deformation Model: The model first deforms the template in canonical pose and then applies skeletal posing to obtain the global posed character.This ordering reduces ambiguities caused by pose variation and simplifies learning non-rigid deformation.
- Differentiable representation: Equations 2 and 3 are differentiable with respect to pose, graph deformation, and vertex displacements, enabling gradient-based learning.The representation jointly supports skinning and coarse-to-fine non-rigid surface deformation while disentangling pose from deformation.
3.2 Data Capture and Motion Preprocessing
The method captures synchronized multi-view video, foreground masks, and tracked skeletal motion, then normalizes motion windows to separate dynamic geometry from irrelevant global transforms.
- Training uses 120 synchronized, calibrated 4K cameras recording at 25 frames per second, with color keying producing foreground masks and distance transforms.
- The system records actors performing varied motions at different speeds to expose diverse clothing deformations caused by body motion.
- A multi-view markerless motion-capture system tracks each frame's skeletal motion for model training.
- Motion is normalized over temporal windows so global translation and rotation around the y axis do not influence geometry generation.
3.3 Embedded Deformation Regression
Embedded deformation regression maps normalized skeletal motion to coarse, spatially structured graph deformations. Weak multi-view losses and regularization keep the predicted geometry aligned, plausible, and material-aware.
- Embedded Deformation Regression: EGNet maps a temporal window of normalized skeletal motion to embedded-graph rotations and translations for coarse non-rigid deformation.
- Embedded Deformation Regression: The motion is first posed onto the embedded graph, converting skeleton-to-graph regression into graph-to-graph translation.
- Structure Aware Graph Convolution: SAGC uses node-specific kernels and biases, with distance-weighted aggregation across each node's R-ring neighborhood to preserve spatial properties.
- Structure Aware Graph Convolution: SAGCN processes graph feature matrices with residual blocks and skip connections, while EGNet outputs six deformation parameters per graph node.
- Weakly Supervised Losses: The weakly supervised objective combines multi-view image-based terms with spatial regularization, including silhouette alignment and as-rigid-as-possible constraints.
- Weakly Supervised Losses: Material-aware rigidity weights allow skirt regions to deform more freely than skin-attached regions.
- Weakly Supervised Losses: Optimized lighting improves dense rendering supervision for learning per-vertex displacements rather than serving as the final appearance result.
3.4 Lighting Estimation
Lighting estimation fits camera-specific illumination while holding coarse geometry and static texture fixed. The optimized lighting improves image agreement and enables dense supervision of finer deformations.
- Lighting Estimation: The renderer assumes Lambertian reflectance and represents smooth illumination with spherical-harmonics coefficients for each camera.
- Lighting Estimation: Rendered color depends on geometry, texture, and camera lighting coefficients, enabling gradient backpropagation through the differentiable renderer.
- Lighting Estimation: Lighting optimization fixes the static texture and coarse posed geometry, then estimates camera-specific coefficients across training frames.
- Lighting Estimation: The optimized lighting matches global appearance better than rendering without explicit lighting and supports dense supervision of per-vertex displacements.
3.5 Vertex Displacement Regression
DeltaNet predicts fine, motion-dependent per-vertex displacements on top of coarse deformation. Dense chroma supervision and Laplacian regularization target detailed appearance while limiting geometric distortion.
- Vertex Displacement Regression: DeltaNet maps a temporal motion sequence to a three-dimensional displacement for every template vertex in canonical pose.
- Vertex Displacement Regression: The network uses the SAGCN architecture because its mesh graph preserves local structures during displacement prediction.
- Weakly Supervised Losses: Weak supervision combines silhouette alignment with a dense image-rendering loss after adding predicted displacements.
- Weakly Supervised Losses: The chroma loss compares rendered and ground-truth UV channels in YUV space, reducing sensitivity to shadow effects that the renderer cannot model.
- Weakly Supervised Losses: A Laplacian regularizer keeps local mesh neighborhoods similar before and after displacement, reducing distorted geometry.
3.6 Dynamic Texture Regression
The dynamic texture model regresses motion- and view-dependent texture maps from pose-derived normal maps and camera information. TexNet is trained photometrically by rendering the predicted texture and comparing it with the real image.
- Texture goal: The model regresses a 1024 × 1024 × 3 dynamic texture that depends on motion and viewpoint.The static scan texture contains pose-specific wrinkles and cannot represent view-dependent effects.
- Texture input: Normalized motion is converted into stacked global-normal maps, providing a texture-space encoding of the posed geometry.The normal maps are generated for each of the F + 1 poses in the motion window.
- Texture input: Camera position and orientation are encoded per texture pixel and concatenated with the normal maps to form TexNet’s input.The resulting camera tensor has six channels at 1024 × 1024 resolution.
- Texture regression: TexNet adapts a UNet to output dynamic textures from the combined motion and camera encoding.This representation allows the network to learn both motion-dependent and view-dependent appearance effects.
- Training: The photometric loss renders the predicted texture with the predicted geometry and compares it against the real image using an eroded foreground mask.Texture supervision is applied only to the conditioning camera view, with identity lighting.
3.7 Implementation Details
Training uses Adam, samples available camera views for multi-view losses, and employs separate rendering resolutions for geometry-related training and TexNet.
- Optimization: The experiments use Adam and randomly sample 40 camera views, when available, for all multi-view losses.Distance-transform images use 350 × 350 resolution, while differentiable rendering uses 512 × 512 or 1024 × 1024 depending on the trained component.
4 RESULTS
The experiments evaluate the method on DynaCap and DeepCap across apparel, motions, viewpoints, and geometry or texture metrics. Results show real-time inference, strong generalization to unseen motions, plausible clothing dynamics, and improved geometry over the cited baseline.
- Runtime: 38 frames per second at inference enables interactive applications on the reported hardware.The first test frame reuses its pose as the previous motion-window frames.
- Datasets: DynaCap contains 5 sequences with 4 subjects wearing 5 apparel types, separate training and testing recordings, and substantially different motions.Sequences are recorded at 25 fps with approximately 20,000 training frames and 7,000 testing frames each.
- Datasets: DeepCap evaluation uses three external subjects with calibrated, segmented, synchronized multi-view imagery and 11–14 camera views.The external data includes ground-truth pose tracking and rigged template meshes at 1024 × 1024 resolution and 50 fps.
- Qualitative results: Across 8 sequences, the method produces motion- and view-dependent deformations for tight and loose clothing without physics simulation or dense 3D supervision.The reported qualitative results include challenging fast motions and viewpoint-consistent textures at novel viewpoints.
- Generalization: The geometry overlays precisely on unseen-motion test frames, while rendered textures appear almost identical to ground truth.The ground-truth frame is not supplied as input; the method uses skeletal motion.
- Comparison: Compared with prior approaches, the method models motion-dependent geometry and loose-apparel dynamics that skinning- or retrieval-based methods cannot represent reliably.Earlier methods also generalize poorly to differing motions or lack explicit, space-time coherent geometry.
- Comparison: The supplemental comparison reports improved visual quality over methods exhibiting temporal inconsistency, texture jumps, or missing body parts.These differences are attributed to explicit geometry and learned dynamic appearance in the proposed representation.
- Geometry evaluation: On DeepCap S4, the method is reported as more accurate than MVBL despite using only skeletal motion and not seeing testing multi-view imagery.MVBL directly optimizes pose and surface fitting on the testing sequence using image cues.
4.5 Ablation
The ablations show that fine vertex displacements, graph-based deformation modeling, canonical-space prediction, chroma supervision, and dynamic textures improve reconstruction and rendering. Sparse multi-view training remains plausible, while additional cameras improve geometry and texture quality; applications include real-time motion retargeting and interactive editing.
- Geometry networks: Vertex displacements improve reconstruction accuracy by capturing finer wrinkles and deformations beyond embedded deformation and skinning.Embedded deformation still outperforms pure skinning, but limited graph resolution prevents it from fully matching ground-truth silhouettes.
- Graph architecture: The proposed node-specific graph convolution performs better than fully connected and unstructured graph-convolution baselines because it encodes locality and individual surface deformation behavior.Fully connected networks lack locality and generalize poorly, while shared weights in the unstructured operator cannot represent node-specific behaviors.
- Displacement representation: Canonical-pose displacement prediction improves generalization and foreground-overlay accuracy by disentangling pose from deformation.
- Chroma supervision: 38.20 versus 44.53 MSE shows that chroma loss improves test-time accuracy over silhouette-only supervision by disambiguating visual-hull drifts.The chroma term penalizes drifts that silhouette supervision alone cannot track well.
- Texture networks: Dynamic motion- and view-dependent textures achieve the lowest error, capturing effects that optimized lighting with static texture still misses.
- Amount of data: Five training cameras already produce plausible unseen-viewpoint results, while additional cameras further improve geometry and texture quality.The experiments used 5, 10, 25, and 49 cameras in a dome-like arrangement, with camera 46 held out for MSE evaluation.
- Applications: The model supports real-time motion retargeting, free-viewpoint video synthesis, and interactive skeletal-pose and camera editing.A source actor can drive the character, and users can freely change skeletal pose and 3D viewpoint.
- Limitations: Physics-based clothing animation could further improve the data-driven approximation, and the method cannot handle taking off apparel or other topological changes.
5 CONCLUSION
The paper presents a real-time, user-controllable method for animating realistic 3D avatars with dynamic surface deformation and texture from skeleton motion. Using multi-view imagery and analysis-by-synthesis losses instead of ground-truth 3D supervision, it reports improved surface detail and textural appearance for character animation, free-viewpoint video, and neural video synthesis.
- The method animates realistic 3D avatar deformation and texture in real time under freely controlled skeleton motion and arbitrary 3D viewpoints.
- It learns dynamic surface deformations and textures from multi-view imagery using analysis-by-synthesis losses, without ground-truth 3D supervision.
- The reported results outperform the state of the art in surface detail and textural appearance, supporting video-realistic character animation, controllable free-viewpoint video, and neural video synthesis.