Structured Local Radiance Fields for Human Avatar Modeling

• Traditional high-quality avatar reconstruction is costly and tedious because of clothing dynamics, and commonly presumes a subject-specific template accurately registered to input frames.Source: §1 · p. 1
• Loose garments expose a motion-modeling gap: body-joint motion misses wrinkles and non-rigid deformation, while direct pose-to-cloth regression can produce averaged, blurry appearance.Earlier methods had not demonstrated animatable characters wearing skirts or dresses.Source: §1 · pp. 1–2
• Structured local radiance fields attached to SMPL nodes represent local shape and appearance while decomposing garment motion into skeleton movement, residual translations, and dynamic detail embeddings.These form a coarse-to-fine hierarchy from body pose to node motion to time-varying detail within each field.Source: §1 · p. 2
• Per-field conditional variational auto-encoders infer node-related variables from pose conditions, using a latent space to balance fitting training frames with generalization to unseen poses.The method also uses timestamps as encoder input; the text says this helps distinguish similar poses across frames and supports temporal smoothness.Source: §1 · p. 2
• The method automatically creates animatable avatars with varied garments from RGB video without pre-scanning, and reports better qualitative and quantitative results than state-of-the-art approaches.The paper reports the first demonstration of automatic avatar creation for dresses.Source: §1 · p. 2

• Earlier human reconstruction methods use multi-view, RGB(D), sparse-view, or single-view images; scan-trained approaches can yield plausible results but have reconstruction errors and weak generalization, whereas this method learns avatars directly from RGB videos.Source: §2 · p. 2
• The proposed representation attaches structured local radiance fields to SMPL nodes and models garment deformation through body poses, node residual translations, and dynamic detail embeddings.Source: Figure 2 · p. 3
• Neural scene methods represent static scenes with feature grids, continuous functions, or NeRFs and extend to dynamic scenes through deformation learning, but many human-motion methods only replay training sequences.This work instead targets animation that generalizes to new poses.Source: §2 · pp. 2–3
• Animatable avatar pipelines progressed from subject-specific meshes and simulation or learned deformation to general-body-template methods and neural textures, while recent neural-scene methods still showed tightly fitting garments.Source: §2 · p. 3

3. Representation

• Each local radiance field models shape and appearance around a node sampled on the canonical SMPL surface.The nodes are sampled using farthest point sampling.Source: §3 · p. 3
• Pose-dependent linear blending skinning moves each node, while a time-varying residual translation lets nodes shift independently for non-rigid cloth deformation.Each node has skinning weights for the body joints, and its residual translation is defined in canonical space.Source: §3 · p. 3
• A dynamic detail embedding conditions each node’s local radiance field on fine-grained changes beyond what node translations represent.The local features are blended and used to compute color and density for volume rendering.Source: §3 · p. 4
• The representation allows more motion and geometry freedom for different cloth topologies, despite sampling its nodes from SMPL.It does not explicitly define a global canonical field or require backward skinning, which the authors describe as ambiguous near contacting body parts.Source: §3 · p. 4

4. Method

• Training estimates node residual translations and dynamic-detail embeddings alongside radiance-network weights from image sequences, assuming known body poses for training frames.The images may come from a multi-view or monocular system.Source: §4 · p. 4

4.1. Network Architecture

• Per-node conditional VAEs use time and a pose condition based on skinning weights and attention to produce node translations and dynamic-detail embeddings.The pose-conditioned latent code is decoded into those two node-related variables.Source: §4.1 · pp. 4–5
• Time input distinguishes similar poses at different instants, while the KL penalty encourages the decoder to rely mainly on pose when predicting node-related variables.For unseen poses, the encoder is omitted and the latent code is set to zero.Source: §4.1 · p. 5

4.2. Training Loss

• The objective combines rendered-pixel reconstruction error with regularization of node translations, dynamic-detail embeddings, and the latent distribution.Embedding regularization encourages embeddings to encode information that node position cannot represent.Source: §4.2 · p. 5
• The local radiance networks and cVAEs use parallel tiny MLPs implemented as group 1D convolutions, and only a small portion of nodes influence each point’s color and density.Training takes about 25 hours for 500k iterations on one NVIDIA 3090 GPU; rendering a 512×512 image typically takes 5 seconds on one NVIDIA 3080TI GPU.Source: §4.2 · p. 5

5. Experiments

• Evaluation uses dress, sweater, ZJU-MoCap, and collected multi-view sequences, with PSNR and SSIM as quantitative metrics.The dataset descriptions specify camera counts and selected views for several sequences.Source: §5 · pp. 5–6

5.1. Results

• Across varied poses and clothing styles, the method produces realistic dynamic wrinkles and handles different cloth types.The examples are shown in Figures 1 and 5.Source: §5.1 · p. 6, Figure 5 · p. 6
• A monocular video of a person rotating 360 degrees in an A-pose can support avatar learning, novel-pose generation, and free-view synthesis.The PeopleSnapshot example is shown in Figure 6.Source: §5.1 · p. 6, Figure 6 · p. 7

5.2. Comparison

• Our method reports higher accuracy than Neural Body [56] in both metrics and higher-quality novel-pose results than Animatable NeRF [55].The comparisons are reported in Tables 2 and 1, respectively.Source: §5.2 · p. 7, Table 2 · p. 7, Table 1 · p. 7

5.3. Ablation Study

• Ablations show that node residual translations recover dress-hem deformation, dynamic detail embeddings improve appearance fidelity, and deterministic pose regression misses garment wrinkles.The deterministic baseline is compared with the proposed cVAE on training-frame reconstruction.Source: §5.3 · p. 8
• Replacing timestamps with per-frame latent codes produces noisy node motion, while the proposed timestamp input yields smoother, more physically plausible trajectories.The trajectory comparison is shown in Figure 10.Source: §5.3 · p. 8, Figure 10 · p. 8

Conclusion

• The representation combines neural scene modeling with a motion hierarchy to model appearance and motion across more general clothing styles.The authors report realistic dynamic details compared with existing methods.Source: §6 · p. 8

Limitation

• Pose generalization depends on training-data variance; animations starkly different from training poses may produce implausible results..Source: §6 · p. 8
• Complex garment physics may exceed the model’s representation capacity, and inaccurate training pose estimates can cause rendering artifacts, particularly with monocular video..Source: §6 · p. 8

Potential Social Impact

• The authors note that the avatar-generation method could be combined with Deep Fake algorithms to create fake videos, a risk to address before deployment..Source: §6 · p. 8

A. Overview

• The supplement includes further discussion, network architecture, data and experiment details, additional experiments, and limitations.It also points readers to a supplementary video for visualizations.Source: A. Overview · p. 12

B. More Discussion

• Neural Body [56] struggles with new-pose synthesis; the authors contrast its 3D-convolution approach with their combination of localized radiance fields.The paper says this construction enables avatar animation by design.Source: B. More Discussion · p. 12
• Animatable NeRF [55] can synthesize unseen poses, but its motion representation may yield static textures and artifacts on loose clothes.The authors contrast this with their coarse-to-fine treatment of non-rigid cloth deformation.Source: B. More Discussion · p. 12
• Neural Actor [36] models pose-dependent appearance details but works best for clothing topologically similar to the body and requires multi-view training input.The authors say their method does not explicitly rely on SMPL topology and can use partial input such as monocular video.Source: B. More Discussion · p. 12
• DDC [22] handles loose clothes but requires a pre-scanned actor template, which was unavailable in the authors’ experiment setting.The authors position their method as modeling general garments without pre-scanning.Source: B. More Discussion · p. 12
• The authors distinguish their 3D approach from 2D avatar methods that may lack view consistency, and from MVP, which targets facial motion rather than body motion and novel poses.MVP also assumes scene-geometry estimates and accurate tracking over time.Source: B. More Discussion · p. 12

C.1. Architecture Details

• The architecture uses sinusoidal encoding for coordinates, view directions, and timestamps, with the timestamp normalized by total frame count.The stated encoding values are m=6 for coordinates, m=4 for view directions, and m=12 for timestamps.Source: §C.1 · pp. 12–13
• For fair comparison with baseline methods, the model uses one network with uniform sampling rather than NeRF’s coarse-and-fine networks and hierarchical sampling..Source: §C.1 · p. 13

C.2. Network Acceleration

• Custom CUDA kernels query only influential local networks; the optimized implementation reduces memory consumption by about 85% and running time by a factor of 4.The method rearranges valid point samples into a tensor sized for the maximum required queries per local MLP.Source: §C.2 · p. 13

D.1. Dataset

• Experiments use two dress sequences, “Ling” and “FranziBlue”; each has about 20,000 training frames captured with 100 cameras, from which 20 views are manually selected for computational efficiency.Source: §D.1 · p. 13
• The “Lan” sweater sequence contains about 30,000 training frames captured from 11 cameras.Source: §D.1 · p. 13
• CoreView387 and CoreView392 each contain about 300 training frames from 23 viewpoints, but experiments use 4 viewpoints for comparison with Neural Body.Source: §D.1 · p. 13
• The authors’ multi-view system uses 24 uniformly distributed cameras at 30 Hz and 1024×1024 resolution, recording three subjects for 2,500–5,000 frames each.Source: §D.1 · p. 13
• SMPL(-X) is registered to video frames, and BackgroundMattingV2 is used for foreground segmentation.Source: §D.1 · p. 13

D.2. Training Details

• The learning rate decays exponentially every 20k iterations, and implementation and training hyperparameters are reported in Table A.Source: §D.2 · p. 13
• Table B reports averaged PSNR across frames for images generated at training poses under different node-related-variable settings.Source: §D.2 · p. 14
• Table C compares training-frame reconstruction PSNR when the cVAE is replaced with a deterministic regression network.Source: Table C · p. 14
• Training runs use 100k iterations for People Snapshot, 300k for ZJU-MoCap, and 500k for other multi-view sequences; baselines use author-provided code and default settings.Source: §D.2 · p. 14

D.3. Metrics

• Evaluation reports PSNR and SSIM on images cropped to a 2D bounding box estimated from the projected SMPL model.Source: §D.3 · p. 14

E. More Experiments

• The cVAE design is reported as critical for reconstructing realistic details in training frames, and its ablation also favors sharper wrinkle details on novel poses.Source: E. More Experiments · p. 14
• Table B evaluates node-related variables using averaged PSNR across training frames, while Table C evaluates the cVAE design.Source: E. More Experiments · p. 14
• Figure B shows testing results for the same pose with z_i set to zero or assigned random Gaussian noise.Source: E. More Experiments · p. 14
• For novel-pose synthesis, the latent z_i is set to zero by default, but changing it produces different wrinkle patterns and could support multiple plausible animation sequences.Source: E. More Experiments · p. 14

F. Limitation and Future Work

• Poses that starkly differ from training poses may yield implausible results because neither subject-specific templates nor the SMPL surface regularize shape learning.The authors suggest clothed-human geometrical priors as a possible remedy.Source: F. Limitation and Future Work · p. 14
• Sparse nodes may not capture complex garment deformation and wrinkle physics; denser nodes could help but increase computational burden.The authors identify combining learning-based implicit representations with physics-based cloth simulation as future work.Source: F. Limitation and Future Work · p. 14