Source-linked AI summary
Driving-Signal Aware Full-Body Avatars
Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, Jason Saragih
TL;DR
The paper addresses full-body avatars that must be animated from incomplete driving signals while remaining faithful to the available information. It introduces a variational model that separates observed driving factors from missing factors, and reports improved generalization and fidelity over other approaches in two information-deficient scenarios.
Problem
The work targets full-body avatars driven by commodity sensors while remaining faithful to information-deficient signals and representing plausible missing configurations.
Source: §1 · pp. 1–2Method
A variational model separates reliably observed driving factors from missing factors and uses localized conditioning to support generalization.
Source: §1 · p. 2, §3.3 · p. 6Results
The model demonstrates improved generalization and fidelity compared with other approaches in two scenarios with information-deficient driving signals.
Source: §1 · p. 2, §6 · p. 16Takeaways & Limitations
The representation can generate diverse plausible configurations that agree with the information contained in the driving signal.
Source: §1 · p. 2The model relies on tracked meshes and fixed-topology UV representations, constraining use with complex loose clothing that changes topology.
Source: §5 · p. 14
Abstract
from arXiv · showhide
We present a learning-based method for building driving-signal aware full-body avatars. Our model is a conditional variational autoencoder that can be animated with incomplete driving signals, such as human pose and facial keypoints, and produces a high-quality representation of human geometry and view-dependent appearance. The core intuition behind our method is that better drivability and generalization can be achieved by disentangling the driving signals and remaining generative factors, which are not available during animation. To this end, we explicitly account for information deficiency in the driving signal by introducing a latent space that exclusively captures the remaining information, thus enabling the imputation of the missing factors required during full-body animation, while remaining faithful to the driving signal. We also propose a learnable localized compression for the driving signal which promotes better generalization, and helps minimize the influence of global chance-correlations often found in real datasets. For a given driving signal, the resulting variational model produces a compact space of uncertainty for missing factors that allows for an imputation strategy best suited to a particular application. We demonstrate the efficacy of our approach on the challenging problem of full-body animation for virtual telepresence with driving signals acquired from minimal sensors placed in the environment and mounted on a VR-headset.
1 INTRODUCTION
The paper targets full-body avatars driven by commodity sensors, where incomplete signals leave unobserved appearance and pose factors ambiguous. It proposes separating observed and missing factors so avatars remain faithful to the signal while representing plausible missing information.
1 INTRODUCTION
- Information asymmetry makes driving a one-to-many problem: naive models can average appearance or overfit to chance correlations in the training set.Prior approaches may prescribe a single imputation strategy and operate post-training, limiting their ability to address existing over- or under-fitting.Source: §1 · p. 2
- The model separates reliably observed factors from missing factors, aiming to preserve fidelity to information-deficient driving signals while representing plausible missing configurations.The latent space captures missing information; localized conditioning reduces correlations among observed factors, while coarse effects are modeled explicitly.Source: §1 · p. 2
- The approach combines localized conditioning, coarse long-range effects, and disentanglement to generalize to novel inputs and produce high-quality reconstructions.The paper demonstrates the approach in performance capture with different attire and VR telepresence animation.Source: §1 · p. 2
2 RELATED WORK
Prior work builds expressive avatars with statistical, geometric, and image-synthesis methods, but challenges remain in handling missing driving information and generalizing across poses and viewpoints. Disentangled representations motivate separating predefined observed factors from learned missing factors.
2.1 Avatar Modeling
- Avatar modeling spans statistical deformation, skinning, localized deformation, textured models, and unified face-hand-body representations, but generated results can remain underfit.Deep generative avatar models model geometry and appearance, yet methods that do not explicitly account for missing information can produce artifacts under deficient driving signals.Source: §2.1 · p. 3
- Image-translation approaches can produce plausible images but may generalize poorly across poses, fail on hands or facial expressions, and lack arbitrary-view rendering.Concurrent approaches address animatable avatars or novel-view synthesis, but cited methods have limitations in appearance modeling, pose generalization, or inference cost.Source: §2.1 · p. 3
2.2 Disentangled Representations
- Disentanglement research offers supervised and unsupervised approaches, while this work separates predefined observed factors from learned missing factors rather than discovering all factors unsupervised.Related methods include β-VAE, mutual-information objectives, and human-modeling approaches for separating attributes, gaze, pose, or shape.Source: §2.2 · pp. 3–4
3 METHOD
Sparse driving signals leave body appearance and other states ambiguous, so models may average unobserved states or learn chance correlations. The method aims to keep the driving signal faithful while using a disentangled latent space for information the signal does not contain.
3 METHOD
- Sparse keypoints may not distinguish clothing, hair, or facial-expression states, creating a many-to-one mapping from driving signal to body appearance.A model may smooth over unobserved states or overfit to dataset-specific correlations; an unconstrained latent code can also absorb control information and compromise fidelity at test time.Source: §3 · p. 4
- The latent space should contain only information absent from the driving signal, while the model remains faithful to that signal during animation.The paper attributes generalization and drivability to complementary architectural elements designed to encourage this separation.Source: §3 · p. 4
3.1 Overview
The model jointly uses driving signals and viewing direction to generate geometry and view-dependent texture. Its design localizes signal effects, explicitly models coarse long-range effects, and reserves a disentangled latent space for missing information.
3.1 Overview
- The architecture takes driving signals and viewing direction and generates geometry and view-dependent texture for rendering into an avatar.Its three generalization components are localized conditioning, coarse long-range effects, and information disentanglement.Source: §3.1 · p. 4
3.1.1 Spatially Localized Conditioning.
- Location-specific low-dimensional embeddings and late conditioning localize each signal's footprint in output geometry and texture, reducing spurious long-range correlations.The embeddings extract information relevant to each output location.Source: §3.1.1 · p. 4
3.1.2 Coarse Long Range Effects.
- Linear-blend skinning models rigid limb motion, while an ambient-occlusion-driven gain map models shadowing to capture coarse long-range effects.These branches complement localized conditioning; the ambient occlusion map substitutes for a shadow map under roughly uniform capture-space illumination.Source: §3.1.2 · p. 4, Figure 2 · p. 5
3.1.3 Information Disentanglement.
- A spatially global latent vector captures information absent from the driving signal, and training encourages it to contain as little driving-signal information as possible.The jointly trained conditional variational autoencoder uses disentanglement strategies to support drivability.Source: §3.1.3 · pp. 4–5, Figure 2 · p. 5
3.2 Variational Autoencoder
The cVAE is trained to reconstruct multiview images from registered geometry, pose, and facial signals, with a latent code for information unavailable from those driving signals. At animation time, the decoder uses the driving signals, an imputed latent code, and viewpoint to synthesize geometry and texture.
3.2 Variational Autoencoder
- The decoder is the component used for animation, while the encoder only facilitates training and is discarded afterward.Source: §3.2 · p. 5
- During training, an encoder samples a 512-dimensional latent code from unposed geometry, while the decoder reconstructs geometry and view-dependent texture from that code, pose, facial keypoints, and viewpoint.The encoder's role is to facilitate learning and structure the latent space; it can be discarded after training.Source: §3.2 · p. 5
- The reconstructed geometry is posed with LBS and the texture is combined with a quasi-shadow map before differentiable rendering and comparison with the captured image.The system is trained end-to-end with image error and additional supervision and regularization losses.Source: §3.2 · p. 5
- At animation time, the driving signals come from sensors and the missing latent code can be sampled or supplied by a temporal model before decoding and rasterization.The latent code represents information not contained in the often-insufficient driving signal.Source: §3.2 · pp. 5–6
3.3 Spatially Localized Driving Signals
Limited capture data can teach models spurious dependencies between body parts, undermining realistic outputs for new combinations of pose and expression. The method uses localized driving-signal encodings to reduce such dependencies while retaining spatially coherent control.
3.3 Spatially Localized Driving Signals
- Sparse range-of-motion data can produce spurious correlations, such as a model linking a fixed facial expression to hand gestures and making pose-dependent clothing effects follow expression.Source: §3.3 · p. 6, Figure 3 · p. 6
- Localization is intended for local geometric and appearance effects; global effects such as shadowing must be modeled separately.The approach assumes sufficiently reliable driving signals and does not require preserving dependencies among all their components.Source: §3.3 · p. 6
- A fully convolutional decoder receives a separate encoding map for each driving source to encourage conditional spatial independence.Each map is formed by repeating the signal over a 32×32 grid, applying location-dependent compression, and masking channels to approximate each parameter's region of influence.Source: §3.3 · p. 6
- The masks use downsampled LBS skinning weights, and varying a control produces spatially and semantically coherent changes in its corresponding output region.The authors report no improvement from learning the masks in their experiments.Source: §3.3 · p. 6
3.4 Coarse Long Range Effects
The method supplements localized conditioning with parametric mechanisms for long-range effects, which can otherwise limit reconstruction accuracy or encourage overfitting. LBS handles coarse rigid limb motion, while an approximate self-shadowing branch models long-range appearance effects.
3.4 Coarse Long Range Effects
- Because localization can limit reconstruction of long-range effects and naive global influence can overfit, the method handles rigid limb motion and self-shadowing with parametric forms intended to generalize to novel poses.Source: §3.4 · p. 6
3.4.1 Rigid Limb Motion.
- The geometric model combines localized corrections with an LBS skeleton model that captures large rigid limb transformations through joint-linked motion.LBS limitations such as the candy-wrapper effect are described as localized and addressed with the localized corrective model.Source: §3.4.1 · p. 7
3.4.2 Self-Shadowing.
- Self-shadowing accounts for long-range appearance effects, especially when limbs touch or approach, leaving the localized model to handle effects such as wrinkles and clothing shifts.Source: §3.4.2 · p. 7
- The shadow branch transforms an ambient-occlusion map from LBS-posed template geometry into a quasi-shadow map, which is multiplied with the texture output.The approximate AO map is efficient but less precise; the branch is supervised implicitly through final rendered-image comparison, and a low-resolution shadow map suffices for plausible soft shadows.Source: §3.4.2 · p. 7
- The approximate shadowing branch is not explicitly supervised, yet its test-sample output captures global shadowing.Source: Figure 5 · p. 7
3.5 Handling Information Deficiency
Body pose and facial keypoints omit appearance information, while an unconstrained latent code can retain driving-signal information and undermine faithful animation. The section presents three strategies for separating observed controls from missing factors, with different applicability to the data.
3.5 Handling Information Deficiency
- Pose and facial keypoints omit clothing, hair, and mouth-state information, while direct regression can overfit to non-generalizable signal-output correlations.A latent code added naively may also retain driving-signal information, allowing the decoder to ignore or distribute information across the latent and driving variables.Source: §3.5 · p. 8
3.5.1 Variational Preferencing.
- In a cVAE, variational preferencing encourages the decoder to use shared information from the driving signal, squeezing that information from the latent code.The mechanism is imperfect, and increasing KL weight can improve disentanglement while reducing reconstruction accuracy.Source: §3.5.1 · p. 8
3.5.2 Mutual Information Minimization.
- Mutual-information minimization uses MINE to estimate dependence between latent z and driving signal c, then adds an adversarial disentangling term to the training objective.The estimate compares statistics-network scores for paired data and shuffled or randomly sampled embeddings.Source: §3.5.2 · p. 8
3.5.3 Perturbation Consistency
- Perturbation consistency penalizes changes to decoder outputs corresponding to driving-signal components when z is sampled, encouraging those outputs to remain consistent with c.It applies when driving-signal components can be directly matched to decoder outputs.Source: §3.5.3 · p. 9
- The authors recommend starting with variational preferencing, adding mutual-information minimization if needed, and using perturbation consistency when direct correspondence is available.Perturbation consistency is not always applicable because some driving-signal elements lack direct correspondence to decoder outputs.Source: §3.5.3 · p. 9
3.6 Training Details
Training combines image and mask inverse-rendering losses with mesh, Laplacian, variational, and disentangling terms. Mesh supervision provides an initial phase before inverse rendering further refines shape and correspondence.
3.6 Training Details
- 1.0e-3 initial learning rate, batch size 8, and approximately 1.5−2 days of training are reported for full-body models on 8 NVidia V100 GPUs with 32GB RAM.Models are trained with Adam until convergence.Source: §3.6 · p. 9
- The composite objective combines inverse-rendering losses for images and foreground masks with mesh, Laplacian, variational KL, and disentangling losses.The Laplacian term encourages smoothness; the KL term is the standard VAE KL-divergence penalty.Source: §3.6 · p. 9
- Mesh supervision is used only for the first 2000 iterations, followed by inverse-rendering losses that improve shape and correspondence estimates and decoder reconstruction accuracy.The transition addresses artifacts in the initial tracked mesh arising from human-body surface tracking challenges.Source: §3.6 · p. 9
4 EXPERIMENTS
Experiments evaluate reconstruction and driving with information-deficient signals across held-out poses, changed clothing, and headset occlusion. They compare model variants and report that the full method generalizes better than variants without disentanglement, localized embeddings, or the shadow branch.
4.1 Data
- The dataset uses 140 synchronized cameras and includes three subjects, held-out test frames, and additional changed-clothing and VR-headset sequences for one subject.Each subject contributes 50–70k frames; roughly 5,000 frames per subject were left out for testing.Source: §4.1 · pp. 9, 12
4.2.1 Reconstruction.
- Optimizing pose, face parameters, and latent codes reconstructs held-out poses with subtle expression and pose details across three identities.The reconstruction figure includes frames not observed during training.Source: Figure 9 · p. 10, §4.2.1 · p. 12
4.2.2 Driving.
- With pose and facial keypoints as driving signals and the latent code set to zero, pose and expressions match well, but reconstruction quality drops around clothing.Comparisons also show visual artifacts in baselines that omit the latent space or do not disentangle it, especially in dynamic sequences.Source: §4.2.2 · p. 12, Figure 11 · p. 13
- For changed attire and VR-headset driving, the model remains faithful to the available signals while imputing missing information plausibly with the latent code set to zero.These tests address a domain gap between modeling and driving conditions, including changed clothing and facial occlusion.Source: §4.2.2 · p. 12
- Disentangling the latent code from driving signals permits different imputation strategies, including random sampling and a temporal model, but further latent-code synthesis study is left for future work.Source: §4.2.2 · p. 14
4.3 Quantitative Evaluation
- On the test set, the full method has a clear advantage over pose+face+latent without disentanglement; removing localized embeddings or the shadow branch also degrades test performance.The version without disentangling fits training data but has the worst performance among the model variants.Source: Table 1 · p. 14, §4.3 · p. 14
5 LIMITATIONS AND FUTURE WORK
The method is limited by tracking quality, latent capacity, and fixed-topology UV meshes; its evaluation and imputation strategy also leave important cases unresolved.
5 LIMITATIONS AND FUTURE WORK
- Tracked-mesh supervision makes performance dependent on tracking quality, while fixed-topology UV meshes constrain applicability and can produce seam artifacts.Implicit surfaces may address topology constraints; mesh convolutions may address seams but cost more.Source: §5 · p. 14
- With constant-code imputation, both baselines struggle to capture shadows and avoid ghost artifacts.Source: Figure 14 · p. 15
- Limited latent-space capacity can lose detail, possibly explaining why reconstructions miss some clothing deformations.The authors suggest hierarchical generative models as a potential direction for improving representational power.Source: §5 · p. 15
- Constant latent codes fix clothing state during animation, although pose-dependent deformations remain capturable.Temporal models or style-dependent generation may enable more semantically meaningful imputations; uncertainty visualization may matter for authenticity-focused telepresence.Source: §5 · p. 15
- Evaluation focuses on natural conversations, leaving generalization to poses far from the training distribution unclear.Source: §5 · p. 15
6 CONCLUSION
The paper builds full-body avatars around the driving signal available at use time, with a disentangled latent space for missing information. Across two information-deficient scenarios, it reports improved generalization and fidelity over other approaches.
6 CONCLUSION
- The method integrates the available driving-signal modality during avatar construction and uses a disentangled latent space to generate diverse plausible configurations faithful to that signal.Source: §6 · pp. 15–16
- The approach demonstrates improved generalization and fidelity compared with other approaches in two information-deficient driving-signal scenarios.Source: §6 · p. 16
- The demonstrated scenarios involve driving signals with deficient information.Source: §6 · p. 16