PaperlayerView exact revision on arXiv

Source-linked AI summary

Single Camera based Simulation and Try-on Ready Avatar Modeling

Honghu Chen, Bo Peng, Jun Xiang, Jingtao Zhou, Xujing Chen, Juyong Zhang

arXiv:2610.34223v1cs.GR

TL;DR

Monocular avatar reconstruction remains difficult because a single view leaves body and garment regions unobserved, while generated image sequences can be inconsistent. The paper combines a double-layer representation, generated observations, and alignment strategies; it reports try-on and novel-pose animation with plausible cloth dynamics from monocular inputs.

  • Problem

    Monocular inputs leave body and garment regions unobserved, and single-layer representations do not distinguish their motion characteristics for try-on-ready reconstruction.

    Source: §I · p. 1

  • Method

    The method reconstructs a double-layer clothed avatar from monocular inputs and generated observations, separating geometric and appearance inconsistencies during alignment.

    Source: §I · p. 1, 2) Base Representation · p. 5

  • Results

    The reconstructed avatars generalize to novel poses and clothing and support virtual try-on animations with visually plausible cloth dynamics from monocular inputs.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    A disentangled avatar with physical simulation supports virtual try-on and novel-pose animation using monocular input.

    Source: §I · p. 2, §V · p. 14

    Physical fidelity is not guaranteed: inverse parameter estimation, template-to-ground-truth geometry gaps, uncertain wrinkles, and variable generative quality constrain results.

    Source: §I · p. 13

Abstract

from arXiv · show

Reconstructing and rendering high-fidelity, animatable, and try-on-ready virtual avatars using only a single camera is crucial for many applications. However, existing try-on-ready human avatar reconstruction methods struggle to achieve this from monocular videos or single images, primarily due to two issues: (1) existing approaches typically use a single layer to represent both garments and the underlying body, ignoring their distinct motion characteristics; and (2) input captured from a single camera often fails to fully cover both the garments and the underlying body. To address these challenges, we propose a physically integrated, double-layer GS-Mesh representation for reconstructing decoupled 3D clothed human avatars from monocular videos or even single images by incorporating 2D generated data. While 2D generation methods can produce video sequences from given pose sequences, they struggle to maintain inter-frame consistency. We decompose the inter-frame inconsistencies in the generated data into geometric and appearance inconsistencies. Based on our 3D representation, we ensure that the final reconstruction and rendering results are both 3D-consistent and temporally coherent. Compared with existing 3D garment transfer methods that rely on multi-view reconstruction and most 2D approaches that use diffusion models for image-based garment editing without novel-view or novel-pose synthesis, our method requires only a monocular video or even a single image while producing visually plausible cloth dynamics during virtual try-on and novel-pose animation. This represents a significant advancement toward virtual try-on with visually plausible cloth dynamics using only a single camera.

I. INTRODUCTION

Monocular avatar reconstruction is challenged by unobserved body and garment regions and by the mismatch between single-layer representations and clothing changes. The paper combines a double-layer avatar with generated observations and alignment strategies to support try-on and novel-pose animation from monocular inputs.

I. INTRODUCTION

  • Single-layer monocular avatars struggle with clothing changes and unseen poses, while single-view inputs leave body and garment regions unobserved.Generated observations can add missing information but introduce temporal inconsistencies that must be addressed for reliable rendering.Source: §I · p. 1
  • The resulting avatar supports virtual try-on and novel-pose animations with visually plausible cloth dynamics from monocular inputs.The paper reports generalization to novel poses and clothing using monocular inputs and generated image sequences.Source: §I · pp. 1–2, Figure 1 · p. 2
  • The method reconstructs a disentangled garment and plausibly completed underlying body, then separates geometric from appearance inconsistencies in generated data.Separate deformation representations and pose re-tracking address geometric inconsistencies; SH coefficients and perceptual loss align appearance.Source: §I · pp. 1–2

A. Reconstruction from Monocular Input.

  • Mesh methods can lack detail or depend on clothing-specific templates, while implicit methods support varied clothing but often have slow rendering and limited editability.3D Gaussian Splatting methods improve rendering speed and editability compared with NeRF-based approaches.Source: A. Reconstruction from Monocular Input · p. 2
  • Unlike most single-image avatar methods that treat clothing and body as a whole, this method targets a disentangled try-on-ready representation and uses diffusion priors for both.It separately generates key-pose clothed-human and underlying-body observations, supporting garment transfer and subsequent cloth simulation.Source: A. Reconstruction from Monocular Input · p. 2

B. Virtual Try-on

Image-based try-on can produce high-quality images but has difficulty across diverse poses and consistent sequences. 3D approaches model garment dynamics or use generative priors, while multi-view and video generation methods address consistency and motion with distinct limitations.

B. Virtual Try-on

  • Image-based try-on methods can produce high-quality images but struggle across diverse poses and continuous, consistent sequences.Source: B. Virtual Try-on · p. 3
  • Physics-integrated 3D try-on methods can generate plausible cloth dynamics, but the cited approaches require multi-view sequences that are not readily accessible to the general public.Other 3D methods model dynamic garment deformation in detail but may lack high-fidelity rendering, while diffusion-guided approaches do not model physics-related deformation.Source: B. Virtual Try-on · p. 3

C. Consistent multi-view and video generation

  • Multi-view generators use synchronization and cross-view attention to improve view consistency, but 2D-trained generators still exhibit cross-view inconsistencies.Generators trained on 3D data are typically limited by the scarcity of high-quality 3D datasets.Source: C. Consistent multi-view and video… · p. 3
  • Human video-generation methods address identity, detail, pose control, motion stability, or environment coherence, but generally do not provide outfit swapping.Physics-conditioned latent diffusion and disentangled Gaussian representations are among approaches addressing dynamic clothing.Source: C. Consistent multi-view and video… · p. 3

A. Algorithm Pipeline

The pipeline builds a decoupled, double-layer clothed avatar from monocular inputs, supplementing unobserved regions with generated data and addressing its temporal inconsistencies. It uses personalized body and clothing templates, a mesh-anchored 2DGS representation, and physical simulation for novel-pose animation and virtual try-on.

A. Algorithm Pipeline

  • The pipeline jointly trains on monocular and generated data, applying geometric and appearance consistency constraints to address generated-data inconsistencies.Generated data supplements regions unobserved in real inputs.Source: A. Algorithm Pipeline · p. 3
  • The reconstructed double-layer avatar generalizes to novel clothing and poses for high-fidelity virtual try-on.The pipeline integrates physical simulation for the try-on application.Source: A. Algorithm Pipeline · p. 3

1) Personalized Template Generation:

  • Decoupled body and clothing template meshes are initialized by removing pose and shape deformations from the reconstructed clothed-human mesh.The resulting X0 is used to obtain separate body and clothing templates.Source: 1) Personalized Template Generation · p. 4

2) Base Representation:

  • Mesh-anchored 2DGS combines global properties that follow template faces with learnable local properties that add geometric detail.The global-local representation addresses the mesh-resolution limit of mesh-parameterized Gaussian properties.Source: 2) Base Representation · p. 4
  • Separate MLPs model body and clothing non-rigid deformation, while real and generated data share X0 but use independent deformations.The separation is intended to isolate generated-data deformations and address their geometric inconsistencies.Source: 2) Base Representation · p. 5
  • The renderer combines a 2DGS albedo map with shading computed from spherical harmonics and learnable lighting coefficients.The method uses SH shading rather than a learned U-Net because view- and pose-dependent shading is challenging to learn from monocular inputs without suitable priors.Source: 2) Base Representation · p. 5

C. Key Pose Extraction

The method selects a compact set of generated-data poses to balance motion representativeness, pose diversity, and visibility. It uses an MMR score to account for these criteria while limiting the need to generate and train on large pose datasets.

C. Key Pose Extraction

  • The compact pose set addresses the computational infeasibility of generating sequences and training directly on large-scale pose data.The generated human image sequences are intended to supplement reconstruction in novel poses and garments.Source: C. Key Pose Extraction · p. 5
  • Pose selection combines representativeness, similarity to already selected poses, and rendering visibility in an MMR-based score.The method uses KDE for pose representativeness, cosine similarity for diversity, and summed SMPL-X vertex visibility for rendering visibility.Source: C. Key Pose Extraction · p. 5
  • 500 key poses are selected from 3.45 million AMASS CMU-subset poses using MMR.The selected set balances representativeness, visibility, and diversity.Source: C. Key Pose Extraction · p. 5

D. Data Generation

The method generates posed clothed-human sequences to supplement monocular inputs and improve information about unseen regions. It separately generates minimally clothed body data to complete regions hidden by garments or absent from the input.

D. Data Generation

  • Generated clothed-human sequences for selected poses provide information about regions unseen in single-image input or occluded in monocular video.The sequences are produced using a human animation generation method.Source: D. Data Generation · p. 5
  • Body completion targets artifacts that can appear on exposed arms or other body parts revealed by clothing changes or motion.The motivating examples include changing long sleeves to short sleeves and garment movement during motion.Source: D. Data Generation · p. 6
  • To complete unobserved body regions, the method inpaints a minimally clothed person and generates 360-degree body data with estimated SMPL-X parameters.The inpainting redraws the clothed person as minimally clothed before 3D human generation.Source: D. Data Generation · p. 6

E. Generated Data Alignment.

The method addresses generated-sequence inconsistencies in geometry and appearance through separate alignment steps. During training, it uses different appearance supervision for real and generated data and adds smoothing to reduce high-frequency noise.

E. Generated Data Alignment.

  • Generated-sequence inconsistencies include shape discrepancy and pose misalignment, as well as style variation and shading incoherence.These issues can introduce reconstruction artifacts when generated data is used directly.Source: E. Generated Data Alignment · p. 6
  • Shape parameters are re-optimized to match generated data to real data, an additional non-rigid MLP adapts frame-varying geometry, and poses are re-tracked.The MLP addresses inter-frame geometric inconsistency, while re-tracking corrects pose alignment.Source: E. Generated Data Alignment · p. 6
  • Per-frame global SH coefficients model unstable generated-data lighting, while perceptual-loss supervision mitigates reconstruction errors from style variation.The global coefficients avoid overfitting to individual frames; perceptual loss addresses inaccurate 3D alignment and potential per-pixel-loss failure under style discrepancies.Source: E. Generated Data Alignment · p. 6

F. Training

  • Real data uses L1, SSIM, and perceptual appearance losses, whereas generated data uses only perceptual loss because its color style differs from ground truth.Label and geometry losses support correct body/clothing representation and smooth geometry.Source: F. Training · p. 6
  • A self-supervised smoothing loss filters high-frequency noise associated with generated-data frame inconsistencies.The training objective also regularizes anchored 2DGS toward alignment with mesh faces.Source: F. Training · p. 6

G. Physical Simulation

The method animates reconstructed avatars with a pre-trained cloth simulator, then fits its parameters to captured motion when video input is available. Garment dynamics are simulated separately from body animation.

G. Physical Simulation

  • A pre-trained ContourCraft simulator generates garment dynamics, while the reconstructed avatar’s 2D Gaussians are transferred to the simulated mesh for novel-pose and novel-view rendering.The simulator predicts garment-vertex acceleration frame by frame with a GNN.Source: G. Physical Simulation · pp. 6–7
  • The simulator uses garment-vertex acceleration, velocity, and position, body-vertex positions, and per-vertex garment material coefficients as inputs.The material coefficients include mass, bending coefficient, and Lamé coefficient.Source: G. Physical Simulation · pp. 6–7
  • The paper compares monocular-video and single-image novel-pose animation, with ExAvatar’s single-image reconstruction shown as a reference.The figure’s left side uses monocular video input; the right side uses one image.Source: Figure 3 · p. 7
  • The fitting stage optimizes garment material coefficients and simulator weights against reconstructed clothing motion while freezing avatar geometry and appearance.The optimized parameters are garment material coefficients ρ and learnable simulator weights ϕ.Source: G. Physical Simulation · p. 7

IV. EXPERIMENTS

The experiments evaluate reconstruction, novel-pose animation, and virtual try-on from monocular video or single-image inputs. The pipeline combines Mesh-GS optimization with generated data and, for video inputs, fitted cloth simulation.

IV. EXPERIMENTS

  • The experiments assess reconstruction quality, novel-pose animation, ablations, virtual try-on, and relighting across monocular-video and single-image settings.The paper uses public-dataset subjects wearing different clothes for quantitative reconstruction evaluation.Source: §IV · p. 7

A. Implementation Details

  • Mesh-GS optimization trains deformation and appearance compensation networks with Gaussian attributes for 100K iterations, taking about 10 hours on an RTX 3090 for 200 real and 200 MMR-extracted frames.For video inputs, physics fitting adds about one hour; single-image inputs bypass fitting and use the pre-trained simulator’s default parameters.Source: A. Implementation Details · pp. 7–8

B. Dataset

  • Virtual try-on results compare the method with IDM-VTON using the same body, clothing, and pose.The monocular-video reconstruction results are presented as a qualitative comparison.Source: Figure 4 · p. 8, Figure 5 · p. 9
  • The dataset evaluation uses PeopleSnapshot, SelfRecon, and REC-MV subjects, comparing monocular-video reconstruction with iHuman and PICA and single-image reconstruction with MimicMotion and ExAvatar.The experiments include three PeopleSnapshot subjects and two skirt-wearing subjects from SelfRecon and REC-MV.Source: B. Dataset · p. 8

C. Quantitative Evaluation

The method performs strongly in quantitative reconstruction evaluations and produces natural novel-pose animations with generated data and physical simulation. A user study also favors its novel-pose results over iHuman on the evaluated sequences.

C. Quantitative Evaluation

  • The method achieves the best performance on all reported metrics for monocular-video reconstruction and superior performance across all metrics for single-image reconstruction.The reported metrics are SSIM, PSNR, and LPIPS; the single-image comparison includes MimicMotion and ExAvatar.Source: C. Quantitative Evaluation · p. 9

D. Qualitative Evaluation

  • For novel poses, the method generates natural animations without artifacts, while iHuman’s LBS can deform loose clothing unnaturally and other baselines show issues with unseen regions or consistency.The comparison includes PICA, MimicMotion, and ExAvatar; LHM’s LBS can distort loose clothing under challenging leg movements, while the proposed simulation maintains structural integrity.Source: D. Qualitative Evaluation · pp. 9–10, Figure 6 · p. 10

E. Generated Data Visualization

  • Generated body and key-pose data are described as reasonable and able to compensate for information missing from monocular inputs.The paper also notes that video-generation models can exhibit temporal and spatial inconsistencies, and says its method can migrate to a superior model when available.Source: E. Generated Data Visualization · p. 10

F. User Study on Novel Poses

  • Participants preferred the method over iHuman in 96.79% of visual-clarity judgments and 97.95% of dynamic-clothing-accuracy judgments on the evaluated novel-pose sequences.The study collected 343 valid clarity judgments and 341 valid dynamic-clothing judgments from 49 questionnaires.Source: F. User Study on Novel Poses · p. 10, Table III · p. 10

G. Ablation Study

The ablations show that generated data, key-pose selection, learnable local properties, data alignment, and physics-based animation each contribute distinct benefits. Generated data trades some held-out reconstruction metrics for better appearance in unseen regions during novel-pose animation.

G. Ablation Study

  • Without body-generated data, exposed regions show artifacts; adding it supplies information for unseen regions and improves reconstruction quality.The comparison is presented in Figure 7(a).Source: G. Ablation Study · p. 10, Figure 7 · p. 11
  • Without key-pose generated data, occluded areas such as the underarm show artifacts in new poses; adding it fills in appearance details.The comparison is presented in Figure 7(b).Source: G. Ablation Study · p. 10, Figure 7 · p. 11
  • MMR extracts a more evenly distributed pose set than uniform sampling, and 200 MMR-generated frames perform slightly below 1,000 uniformly sampled frames while carrying more information in fewer frames.The selected key-pose data can also fill information missing from ground-truth data.Source: G. Ablation Study · pp. 10–11, Table IV · p. 11
  • Learnable local geometric properties produce richer geometric and shading details than the representation without them.The comparison is shown in Figure 12.Source: G. Ablation Study · p. 11, Figure 12 · p. 12
  • Removing any generated-data alignment component—shape consistency, style consistency, pose alignment, or SH coefficients—reduces reconstruction quality.Quantitative results are reported in Table V.Source: G. Ablation Study · p. 11, Table V · p. 12
  • Adding generated data to PICA lowers its PSNR from 30.92 to 29.05, while the method without generated data achieves PSNR 33.67 versus PICA’s 30.92.The authors attribute the degradation to generated-data inconsistencies acting as noise without alignment strategies and the dual-layer formulation.Source: G. Ablation Study · p. 12
  • Generated data slightly lowers held-out reconstruction metrics versus the full pipeline without it, but provides supervision for plausible appearances in unseen regions during novel-pose animation.Removing generated data produces more visible artifacts in previously unseen or occluded regions under novel poses.Source: G. Ablation Study · p. 12
  • Physics simulation avoids the severe unnatural stretching that LBS causes for loose garments such as skirts during wide stances or novel motions.The simulation models collisions and non-rigid dynamics; fitted parameters better approximate garment-specific behavior, though absolute physical properties are intractable to recover from under-constrained monocular video.Source: G. Ablation Study · pp. 12–13, Figure 13 · p. 12, Figure 14 · p. 13

1) Virtual Try-On:

The method combines clothing transfer with physical simulation for try-on, preserving garment texture and style while producing visually plausible cloth dynamics. The paper also demonstrates relighting and identifies limits from unseen-region errors and physical-simulation fidelity.

1) Virtual Try-On:

  • Try-on preserves clothing texture and style while producing visually plausible cloth dynamics, but the comparison method sometimes changes clothing style, lacks frame consistency, and cannot handle novel poses.The comparison uses front-facing clothing images and demonstrates results only on real captured data because IDM-VTON requires the body to be specified for each frame.Source: 1) Virtual Try-On · p. 13

2) Relighting:

I. Limitations and Future Work.

  • Initialization errors in body or garment geometry, pose, segmentation, or generated unseen regions can propagate; alignment cannot fully correct semantic hallucinations in unseen regions.Such errors may affect garment-body collision and physical simulation.Source: §I · p. 13
  • Physical fidelity is not guaranteed because parameter estimation, template geometry, clothing wrinkles, generated appearance, and multilayer interactions remain challenging.Sophisticated multilayer garments can exhibit unresolved cross-layer interactions, subtle geometric inter-penetration, and rendering artifacts during dynamic driving.Source: §I · p. 13

V. CONCLUSION

  • The conclusion describes a double-layer Mesh-GS avatar reconstructed from real and generated data, with geometric and appearance inconsistencies in generated sequences addressed separately.The method integrates physical simulation and generalizes to novel poses and clothing.Source: §V · pp. 13–14
  • The authors report high-fidelity virtual-try-on avatars from monocular input, with visually plausible cloth dynamics for novel garments and poses.They describe the single-camera, physics-integrated reconstruction as a technical foundation for animation and virtual try-on applications.Source: §V · p. 14

Related papers

Loading 2610.34223v1…