Source-linked AI summary
Ani3DHuman: Photorealistic 3D Human Animation with Self-guided Stochastic Sampling
Qi Sun, Can Wang, Jiaxiang Shang, Yingchun Liu, Jing Liao
TL;DR
Ani3DHuman addresses the trade-off between kinematics-based animation's limited non-rigid motion and diffusion-based methods' artifacts and identity loss. It combines layered motion representation with self-guided stochastic sampling to restore realistic, identity-preserving videos, and experiments show photorealistic animation that surpasses state-of-the-art methods.
Problem
Existing kinematics-based methods lack realistic non-rigid dynamics, while video-diffusion approaches can suffer from low-quality generation and identity loss.
Method
Ani3DHuman combines mesh-rigged motion with a residual deformation field and uses self-guided stochastic sampling to restore coarse out-of-distribution renderings into identity-preserving videos for 4D optimization.
Results
Ani3DHuman surpasses state-of-the-art methods, achieving best perceptual results and an 18.8 FID improvement with competitive reconstruction metrics.
Takeaways & Limitations
The framework demonstrates that combining stochasticity for photorealistic quality with self-guidance for identity fidelity can produce high-quality supervision for photorealistic human animation.
Takeaways & Limitations
The 4D Gaussian Splatting representation is not strictly lossless and may smooth extremely high-frequency texture details despite PSNR ≈35 dB.
Abstract
from arXiv · showhide
Current 3D human animation methods struggle to achieve photorealism: kinematics-based approaches lack non-rigid dynamics (e.g., clothing dynamics), while methods that leverage video diffusion priors can synthesize non-rigid motion but suffer from quality artifacts and identity loss. To overcome these limitations, we present Ani3DHuman, a framework that marries kinematics-based animation with video diffusion priors. We first introduce a layered motion representation that disentangles rigid motion from residual non-rigid motion. Rigid motion is generated by a kinematic method, which then produces a coarse rendering to guide the video diffusion model in generating video sequences that restore the residual non-rigid motion. However, this restoration task, based on diffusion sampling, is highly challenging, as the initial renderings are out-of-distribution, causing standard deterministic ODE samplers to fail. Therefore, we propose a novel self-guided stochastic sampling method, which effectively addresses the out-of-distribution problem by combining stochastic sampling (for photorealistic quality) with self-guidance (for identity fidelity). These restored videos provide high-quality supervision, enabling the optimization of the residual non-rigid motion field. Extensive experiments demonstrate that \MethodName can generate photorealistic 3D human animation, outperforming existing methods. Code is available in https://github.com/qiisun/ani3dhuman.
1. Introduction
ANI3DHUMAN combines kinematics-based animation with video diffusion priors to preserve human structure and identity while restoring realistic non-rigid dynamics. Its self-guided stochastic sampling addresses out-of-distribution renderings, and restored videos supervise residual-motion optimization.
- Video diffusion methods can model non-rigid dynamics but may produce low-quality videos or lose identity, while kinematics-based methods struggle with clothing and other deformations.
- ANI3DHUMAN layers mesh-rigged motion with a deformation field, using kinematics as a structural and identity prior before diffusion restoration.
- The method restores coarse mesh-rigged renderings into realistic videos that synthesize non-rigid dynamics onto existing human structure rather than inventing an identity.
- Self-guided stochastic sampling combines stochasticity for photorealistic quality with self-guidance for identity fidelity when deterministic ODE sampling fails on out-of-distribution inputs.
- Diagonal view-time sampling reduces generative trajectories to provide a coherent optimization signal for sharp reconstruction despite inconsistency across samples.
- ANI3DHUMAN achieves photorealistic animation with high-fidelity non-rigid dynamics and significantly surpasses state-of-the-art methods.
2. Related Work
Prior work spans mesh-based kinematic animation, physics-based simulation, score distillation, and direct reconstruction from generated videos. ANI3DHUMAN builds on mesh structure while using diffusion priors to improve non-rigid realism and identity preservation.
- Kinematics-based methods: Mesh-driven kinematic methods efficiently control skeletal motion and provide explicit structure for animation and motion retargeting.
- Kinematics-based methods: ANI3DHUMAN incorporates mesh-rigged motion as a rigid-motion prior while adding a residual field for non-rigid dynamics.
- Physics-based animation: Physics-based methods model clothing dynamics through garment-body interaction simulation, often requiring a separate garment mesh.
- Score distillation sampling: Score-distillation approaches use video diffusion models to supervise dynamic-scene reconstruction but can require lengthy optimization.
- Reconstruction from generated videos: Direct reconstruction from generated videos follows multi-view or pose-driven paths, but limited 4D training data and identity loss constrain quality.
3. Preliminary: Flow Matching
Flow Matching learns a velocity field that transports noise to the data distribution along a prescribed path. Sampling reverses this transport with an ODE solver, while endpoint estimates support deterministic updates.
- Flow Matching learns a velocity field vθ that transports a prior distribution p1 to the data distribution p0.
- Rectified Flow defines a linear interpolation xt = (1 − σt)x0 + σtx1 between data x0 and Gaussian noise x1.
- The target velocity for this path is the constant ut = x1 − x0, which the model vθ is trained to predict.
- Sampling starts from Gaussian noise x1 and integrates the learned velocity field backward from t = 1 to t = 0 using an ODE solver.
- Intermediate states can provide estimates of the posterior mean x̂0 and posterior noise x̂1 by rearranging the path and approximating vθ ≈ ut.
- A deterministic ODE solver uses the endpoint predictions to reinterpolate the linear trajectory for each update from t to t_next.
4. Proposed Method
Ani3DHuman combines mesh-rigged rigid motion with a residual non-rigid motion field, then restores coarse renderings using self-guided stochastic sampling. Diagonal view-time sampling supplies coherent supervision for optimizing the motion field.
- Pipeline: The framework animates a 3D human by modeling rigid body pose and non-rigid deformations for photorealistic rendering from arbitrary viewpoints.It uses layered motion representation, self-guided stochastic sampling, and progressive optimization.
- Layered Motion Representation: Layered motion combines explicit SMPL-based mesh rigging for rigid motion with an implicit residual field for non-rigid motion.The residual field is parameterized with Hexplane features and an MLP decoder.
- Self-guided Stochastic Sampling: Deterministic Flow-ODE sampling fails because noised coarse renderings are out-of-distribution relative to the flow model’s training marginal.The resulting trajectory follows inaccurate velocity predictions and produces low-quality samples.
- Self-guided Stochastic Sampling: Stochastic sampling corrects OOD trajectories by adding a diffusion term that pulls samples toward the target marginal during reverse-time evolution.The practical discretization adds noise to the clean-noise prediction before interpolation.
- Self-guided Stochastic Sampling: Self-guidance pulls posterior predictions toward masked input regions, combining stochastic quality with identity preservation.The guidance uses a data-space L2 loss with a binary preserved-region mask and efficient closed-form gradient.
- Progressive 4D Optimization: Diagonal view-time sampling jointly evolves camera view and time to minimize inter-trajectory inconsistency during residual-field optimization.Using fewer generative trajectories reduces conflicting signals and supports sharper reconstruction.
5. Experiments
Experiments evaluate Ani3DHuman against existing human-animation and video-rerendering methods using quantitative metrics, user studies, comparisons, and ablations. The results support improvements in quality, identity preservation, non-rigid motion, and sampling behavior.
- Ablations: Removing stochastic sampling causes a significant quality drop, while removing self-guidance greatly reduces identity preservation.Replacing the personalized diffusion prior with a general prior introduces slight degradation and artifacts.
- Experimental Settings: Experiments use 10 ActorsHQ cases reconstructed from single-view images and extracted motion sequences, with pixel, perceptual, image-quality, video-quality, and user-study measures.Metrics include PNSR, SSIM, LPIPS, CLIP-Image, FID, and FVD.
- Comparisons to the State-of-the-art Methods: Ani3DHuman is the only compared method reported to achieve high quality, identity preservation, and realistic non-rigid motion simultaneously.Disco4D and SV4D 2.0 show quality problems, PERSONA loses identity, and LHM lacks realistic clothing dynamics.
- Comparisons to the State-of-the-art Methods: 18.8 FID improvement is reported over state-of-the-art methods, alongside competitive reconstruction metrics and the best user-preference scores.The comparison covers four humans with different motion sequences.
- Ablations: Diagonal sampling reconstructs sharp details, whereas baseline sampling methods suffer from significant floaters and spikes.The sampling ablation is shown in Fig. 9.
- Sampling Analysis: Only self-guided stochastic sampling resolves initial artifacts while generating high-quality video and preserving human identity across compared sampling methods.The comparison uses the same base video model, noise level t0 = 0.6, and 30 denoising steps.
- Ablations: Layered motion captures intricate hand transformations that the single-layer motion-field baseline fails to model.The motion-field ablation is illustrated in Fig. 8.
6. Conclusion
ANI3DHUMAN combines layered motion representation with self-guided stochastic sampling to generate photorealistic animation that preserves identity while capturing non-rigid motion. Diagonal view-time sampling supports coherent 4D optimization, while lengthy video-diffusion sampling remains the key limitation.
- ANI3DHUMAN combines mesh-rigged motion with a residual field to model complex non-rigid human motion.
- Balancing stochasticity for quality with self-guidance for fidelity enables photorealistic results, while diagonal view-time sampling provides coherent 4D optimization.
- The key limitation is the lengthy sampling time of the video diffusion prior, motivating few-step generation as future work.
B. Proof
The appendix presents theoretical and implementation details for stochastic diffusion correction, approximate identity guidance, the video backbone, preserved-area masking, and 4D optimization components.
- For the binary masking operator M, the spectral norm is determined by its maximum singular value, supporting the stated error bound.
- The gradient approximation error is bounded by C · E[∥x0 − x̂0|t∥] and tends to zero as the diffusion process approaches the clean-data manifold.
- The stochastic sampler discretizes a reverse-time SDE by adding an explicit diffusion term to the deterministic Rectified Flow ODE.
- Stochasticity perturbs the predicted noise with Gaussian noise scaled by scheduling parameter γ, separating deterministic drift from diffusion.
- Efficient identity guidance approximates the diffusion Jacobian as a scalar identity, reducing the gradient to a masked residual.
D.4. Baseline Implementation
The baseline implementation compares representative human-animation and diffusion-restoration methods, adapting unavailable or DDPM-based components for fair evaluation.
- The study selects representative baselines with official implementations and re-implements methods when key repository components are unavailable.
- Disco4D combines single-view driving-video MSE supervision with Zero-123 score-distillation guidance.
- SV4D 2.0 reconstructs synchronized multiview videos but exhibits severe identity shifts attributed to its synthetic-object training domain.
- LHM provides an efficient kinematics-based baseline but lacks clothing deformation because it relies purely on kinematics-based deformation.
- The restoration comparison includes Vanilla SDEdit, MCS, NC-SDEdit, and FlowEdit as methods for converting low-quality inputs into high-quality targets.
- NC-SDEdit replaces high-frequency posterior components with those from the source reference while allowing diffusion to refine low-frequency content.
E.1. Quantitative Ablation
Ablations show that the full configuration best balances visual fidelity, identity preservation, temporal coherence, and motion realism. Removing self-guidance slightly improves FID but substantially harms identity consistency.
- Removing self-guidance marginally improves Frechet Inception Distance but substantially decreases CLIP-Identity, validating its role in identity preservation.
- Stochastic sampling and personalized diffusion are essential for temporal coherence and motion realism, with the full model attaining the lowest Frechet Video Distance.
- The full model achieves the most robust performance across all evaluation metrics.
- Qualitative comparisons expose artifacts in mesh-rigged inputs and identity loss or hallucinated accessories under direct generation.
E.3. Sensitivity Analysis of Initial Noise Strength
The initial noise strength t0 controls the trade-off between correcting OOD artifacts and preserving fidelity to the coarse rendering. Higher noise levels improve restoration, while self-guidance preserves identity even at high noise.
- t0 governs the trade-off between restoration capability and fidelity to the initial coarse rendering.The sensitivity analysis varies t0 from 0.2 to 0.8.
- At t0 ∈ {0.2, 0.4}, the sampling trajectory is too short to effectively correct OOD artifacts.
- At higher noise levels, stochastic sampling more effectively restores plausible details and corrects non-rigid dynamics.
- Self-guidance preserves identity even at the high noise strength t0 = 0.8.
E.4. More Ablations in 4D Optimization
The ablations show that adaptive densification, mask regularization, dataset updates, and the sampling configuration each address distinct reconstruction challenges. Together, these components support sharp dynamic details while controlling geometry and optimization inconsistency.
- Sampling strategy: The sampling strategy must balance restoration quality with identity fidelity when correcting coarse renderings.Self-guided stochastic sampling is presented as the mechanism for addressing this restoration challenge.
- Adaptive densification: Adaptive densification is crucial for capturing high-frequency texture dynamics that canonical-geometry deformation alone cannot model.Without densification, the model lacks sufficient primitives in dynamically changing regions such as shifting wrinkles.
- Mask loss regularization: Mask loss regularization is essential for constraining the reconstructed geometry.
- Dataset update: Dataset updates mitigate over-smoothing from inconsistent supervision and sharpen fine-grained features such as dress wrinkles.The update mechanism lets optimization reject inconsistent noise and converge toward high-fidelity results.
- Optimization cost: The full optimization uses 30k iterations, updates pseudo-ground truth every 5k iterations, and takes about 19 minutes overall.Each video rerendering step takes about 67 seconds on average.
F.2. Discussion with Image-based Animation methods.
Compared with image-based animation methods, ANI3DHUMAN targets a stronger combination of motion fidelity, identity preservation, and 3D view consistency. Results across benchmark comparisons and diverse camera trajectories support this broader animation capability, while 4DGS introduces a reconstruction boundary.
- Image-based animation: Image-based methods can achieve competitive rendering quality but struggle to maintain identity consistency, especially from side views.Their lower CLIP-Identity scores contrast with the identity priors used by ANI3DHUMAN.
- Image-based animation: ANI3DHUMAN achieves a superior balance between motion fidelity and identity preservation compared with state-of-the-art image-driven animation methods.
- 4DGS distillation: Distilling the pose-controlled diffusion model into 4DGS enables high-fidelity real-time rendering from arbitrary viewpoints after optimization.Rasterization-based rendering avoids iterative denoising during subsequent sequence generation.
- ActorsHQ evaluation: The method maintains identity preservation and temporal consistency while handling challenging articulation and plausible non-rigid dynamics on ActorsHQ.The evaluation uses only a single-view image as input; spatial misalignment is attributed to inaccurate SMPL parameters from raw video.
- Additional qualitative results: Dynamic 360-degree camera trajectories across diverse subjects and complex motions maintain high visual fidelity and temporal consistency from all viewing angles.
- Limitations: 4DGS reconstruction is not strictly lossless and may mildly smooth extremely high-frequency texture details despite PSNR ≈35 dB.