Source-linked AI summary
GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos
Yiqian Wu, Rawal Khirodkar, Egor Zakharov, Timur Bagautdinov, Lei Xiao, Zhaoen Su, Shunsuke Saito, Xiaogang Jin, Junxuan Li
TL;DR
Creating photorealistic 3D avatars typically requires inaccessible multi-view images or long monocular videos, while native 3D generation has not effectively used in-the-wild data. GenLCA uses a reconstruction tokenizer and visibility-aware training to learn from million-scale real-world videos, achieving high-quality, animatable avatars and outperforming existing methods.
Problem
Photorealistic 3D avatar creation usually requires inaccessible multi-view images or long monocular videos, and no native 3D avatar generator effectively uses in-the-wild data.
Method
GenLCA uses a pretrained reconstruction model as a 3D tokenizer and visibility-aware training to train a full-body 3D diffusion model from partially observable, million-scale 2D data.
Results
GenLCA outperforms state-of-the-art methods in semantic alignment, visual quality, and human preference while generating full-body, animatable avatars from incomplete observations.
Takeaways & Limitations
GenLCA demonstrates that imperfect real-world videos can support scalable 3D avatar diffusion training, improving model diversity and generalizability.
Takeaways & Limitations
GenLCA relies on Linear Blend Skinning inherited from its reconstruction model, which can produce unrealistic deformations for loose clothing under extreme poses.
Abstract
from arXiv · showhide
We present GenLCA, a diffusion-based generative model for generating and editing photorealistic full-body avatars from text and image inputs. The generated avatars are faithful to the inputs, while supporting high-fidelity facial and full-body animations. The core idea is a novel paradigm that enables training a full-body 3D diffusion model from partially observable 2D data, allowing the training dataset to scale to millions of real-world videos. This scalability contributes to the superior photorealism and generalizability of GenLCA. Specifically, we scale up the dataset by repurposing a pretrained feed-forward avatar reconstruction model as an animatable 3D tokenizer, which encodes unstructured video frames into structured 3D tokens. However, most real-world videos only provide partial observations of body parts, resulting in excessive blurring or transparency artifacts in the 3D tokens. To address this, we propose a novel visibility-aware diffusion training strategy that replaces invalid regions with learnable tokens and computes losses only over valid regions. We then train a flow-based diffusion model on the token dataset, inherently maintaining the photorealism and animatability provided by the pretrained avatar reconstruction model. Our approach effectively enables the use of large-scale real-world video data to train a diffusion model natively in 3D. We demonstrate the efficacy of our method through diverse and high-fidelity generation and editing results, outperforming existing solutions by a large margin. The project page is available at https://onethousandwu.com/GenLCA-Page.
GenLCA
GenLCA is a diffusion-based model for generating and editing full-body 3D Gaussian avatars from text and image inputs. It targets realistic, input-consistent avatars with facial and full-body animation.
- GenLCA generates and edits full-body 3D Gaussian avatars from text and image inputs.
- The model produces avatars consistent with input face identity and text descriptions while supporting high-fidelity facial and full-body animations.
- GenLCA supports multimodal editing using text, RGB images, and scribbles as control signals.
- The model generates diverse 3D avatars from text inputs.
1 Introduction
GenLCA addresses the difficulty of training scalable 3D avatar diffusion models from real-world, partially observed data. It combines a reconstruction tokenizer with visibility-aware training to support large-scale, realistic, animatable avatar generation and editing.
- 3D avatar creation often requires inaccessible multi-view images or long monocular videos, motivating more efficient and scalable generation from user-friendly inputs.
- Synthesized data can suffer a domain gap from real-world humans, while calibrated multi-view datasets are costly and limited in coverage.
- GenLCA trains a full-body 3D diffusion model from partially observable, million-scale 2D data using a feed-forward avatar reconstruction network as tokenizer.
- The tokenizer converts video-derived observations into 3D tokens, which are compressed into compact latents for efficient diffusion-model training.
- A flow-based diffusion model conditions generation and editing on text, segmented body-part images, and scribble images.
- GenLCA captures semantic relationships between visible tokens and generates full-body, animatable avatars while outperforming state-of-the-art methods by a significant margin.
2.2 Zero-shot and one-shot 3D human creation
Prior 3D human generation methods rely on 2D diffusion guidance or face ambiguities when modeling from single-view data. GenLCA instead uses real-world data and a reconstruction model to train a native 3D diffusion model at scale.
- SDS-based methods generate 3D content using guidance from 2D diffusion models followed by multi-step optimization.
- GenLCA operates natively in 3D and is trained on real-world data, avoiding issues of blurriness and low realism.
- Directly modeling 3D implicit distributions from single-view 2D collections introduces ambiguities and degrades quality.
- GenLCA addresses the lack of native 3D avatar generators that effectively use in-the-wild data by extracting training samples from large-scale video through reconstruction.
- Its pipeline compresses 3D tokens and restricts training to observable, reliable information using a visibility mask.
3.1 3D avatar tokenizer
GenLCA uses a pretrained reconstruction model, LCA, as a tokenizer that converts 2D images into a structured 3D avatar representation. LCA encodes images into 3D tokens and decodes them into Gaussian splats.
- LCA is a pretrained reconstruction model used to obtain structured and unified 3D avatar representations from 2D images.
- The tokenizer encodes multiple body and face images into 3D tokens, while the detokenizer interprets those tokens as Gaussian splats.
- LCA extracts image tokens, combines them with query points sampled on a shared template body mesh, and produces Gaussian-splat attributes for rendering.
3.2 GenLCA
GenLCA trains a flow-based 3D diffusion model in a compact latent space, using reconstructed Gaussian-splat tokens and multimodal text or image conditioning. Its architecture combines compression, masked training for imperfect observations, and separate latent and conditional processing.
- GenLCA is a flow-based diffusion model trained with a rectified flow objective.
- 3.2.1 Token compressor: The compressor uses MLPs for downsampling or upsampling, self-attention for feature fusion, and query-point positional embeddings.
- 3.2.1 Token compressor: A compressor encodes high-dimensional Gaussian-splat tokens into lower-dimensional latents while preserving the token count.The latent dimension is D_Z = 8, and reconstruction uses L1 and KL-divergence losses.
- 3.2.2 Visibility-aware training: GenLCA avoids learning from corrupted unobserved regions by applying visibility-aware masked training to partially observable video-derived tokens.The strategy uses a visibility mask to handle blurry or incomplete information from monocular frames.
- 3.2.3 GenLCA architecture: Each GenLCA block processes latent and conditional features through separate branches while using query points, time steps, and positional encoding.
- 3.2.4 Conditional inputs: The model supports text-only, image-only, and text-plus-image conditioning, using CLIP for text and DINOv2 for scribble and body-part images.Body-part conditioning concatenates embeddings from five body-part images.
3.3 Visibility mask
The visibility mask identifies unreliable regions caused by incomplete views and lets GenLCA retain only trustworthy token information. Valid tokens reconstruct visible content realistically, while invalid tokens commonly appear blurry or transparent.
- Monocular videos provide incomplete views that cause blurry or transparent artifacts in unobserved body regions.Frontal-only views produce blurry backs, while upper-body-only views produce transparent lower bodies.
- The visibility mask is computed by rendering decoded Gaussian splats with the input camera and body poses, then measuring each splat’s image contribution.
- Filtering invalid tokens leaves valid Gaussian splats that reconstruct visible image regions with high quality.Invalid splats are typically blurry or transparent, whereas valid splats appear realistic.
- A token is marked visible when at least two of its eight decoded splats are visible in at least one input view.
4 Implementation Details
GenLCA builds a large 3D token dataset from monocular and captured videos, then evaluates generation quality against prior methods using quantitative metrics, user studies, and ablations.
- Training dataset: Frames from monocular videos are encoded into structured 3D tokens using the tokenizer.Body and face images are processed to obtain GS tokens for training.
- Training dataset: 1,113,476 in-the-wild monocular videos and captured multi-view data form the basis of GenLCA’s training token dataset.The final token dataset comprises 1,117,411 identities, while 1,000 captured identities are sampled for evaluation.
- Training dataset: Each input image undergoes body segmentation and background removal before multimodal labels are added for text, scribble, and body-part conditioning.The labels support multimodal generation and editing experiments.
- Evaluation: GenLCA is compared with SDS-based methods and text-conditioned 3D human diffusion models using the same text prompts and full-body and facial views.The evaluation includes quantitative tables, user studies, and visual comparisons.
- Evaluation: GenLCA produces stronger semantic alignment, color accuracy, facial detail, and overall fidelity than the compared methods in visual examples.SDS-based methods show unrealistic styles, while TeRA and SIGMAN show semantic or visual-quality weaknesses.
- Ablation studies: Ablations remove visibility-aware training, learnable placeholders, and in-the-wild training data to evaluate the effectiveness of each component.The study is reported in Fig. 7.
6 Conclusion
GenLCA uses scalable training on imperfect real-world videos to achieve high-quality 3D avatars, while its animation quality remains constrained by inherited Linear Blend Skinning.
- Conclusion: GenLCA achieves state-of-the-art quality by scaling training with imperfect real-world videos and visibility-aware token training.The approach uses a reconstruction model as tokenizer and filters unreliable signals from partial observations.
- Limitations: Linear Blend Skinning can cause unrealistic deformations, particularly for loose clothing under extreme poses.This limitation is inherited from the reconstruction model used for animation.