Source-linked AI summary
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans
Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou
TL;DR
Neural Body addresses sparse-view novel view synthesis for dynamic human performers, where conventional approaches struggle with limited observations, wide camera baselines, and self-occlusion. It shares deformable-mesh-anchored latent codes across video frames to integrate observations, achieving state-of-the-art results on ZJU-MoCap and demonstrating monocular-video capture on People-Snapshot.
Problem
Sparse-view novel view synthesis for dynamic humans is challenging because image-based methods require dense views while reconstruction methods suffer from wide-baseline matching and self-occlusion.
Method
Neural Body anchors latent codes to vertices of a deformable human model and generates each frame’s implicit fields from the same codes, jointly learning them across video frames.
Results
Neural Body achieves the best PSNR and SSIM among compared methods, exceeding prior work by at least 4.20 PSNR and 0.047 SSIM, while also demonstrating monocular-video capture on People-Snapshot.
Takeaways & Limitations
The approach enables photorealistic novel views of performers in complex motions from sparse multi-view video and supports 3D reconstruction.
Abstract
from arXiv · showhide
This paper addresses the challenge of novel view synthesis for a human performer from a very sparse set of camera views. Some recent works have shown that learning implicit neural representations of 3D scenes achieves remarkable view synthesis quality given dense input views. However, the representation learning will be ill-posed if the views are highly sparse. To solve this ill-posed problem, our key idea is to integrate observations over video frames. To this end, we propose Neural Body, a new human body representation which assumes that the learned neural representations at different frames share the same set of latent codes anchored to a deformable mesh, so that the observations across frames can be naturally integrated. The deformable mesh also provides geometric guidance for the network to learn 3D representations more efficiently. To evaluate our approach, we create a multi-view dataset named ZJU-MoCap that captures performers with complex motions. Experiments on ZJU-MoCap show that our approach outperforms prior works by a large margin in terms of novel view synthesis quality. We also demonstrate the capability of our approach to reconstruct a moving person from a monocular video on the People-Snapshot dataset. The code and dataset are available at https://zju3dv.github.io/neuralbody/.
1. Introduction
Neural Body targets novel view synthesis of dynamic humans from sparse multi-view video by sharing structured latent codes across frames and anchoring them to a deformable model. It achieves strong novel-view results and supports monocular-video reconstruction.
- Motivation: Sparse-camera human view synthesis is difficult because dense-view methods do not apply, wide baselines hinder stereo, and self-occlusion causes incomplete reconstructions.These limitations produce noisy geometry and heavy rendering artifacts.
- Motivation: Neural representations learned from highly sparse views are ill-posed, while independently learned per-frame latent codes provide insufficient constraints for fusing video observations.The paper motivates aggregating observations across frames instead.
- Method: Neural Body generates different-frame implicit fields from the same latent codes, anchored to deformable-model vertices whose locations follow the human pose.The shared codes form a latent-variable model that integrates observations across video frames.
- Method: The deformable model supplies rough surface geometry, providing geometric guidance for more efficient implicit-field learning.This prior complements the shared latent-code representation.
- Results: The method synthesizes photorealistic novel views of performers in complex motions from sparse multi-view video and reconstructs moving humans from monocular RGB video.The evaluations use ZJU-MoCap and People-Snapshot.
2. Related work
Related work spans image-based rendering, traditional and template-based human performance capture, and neural representations. These approaches differ in their requirements for camera density, depth sensing, templates, and learned human priors.
- Image-based rendering: Image-based rendering synthesizes views from densely sampled images, with depth proxies extending viewpoint range through image warping and blending.Light-field interpolation can produce impressive results but has limited renderable viewpoints.
- Human performance capture: Traditional human performance-capture methods rely on depth sensors or dense camera arrays for high-fidelity reconstruction.Neural networks can compensate for some geometric artifacts in the rendering pipeline.
- Human performance capture: Template-based methods reconstruct sparse-view dynamic humans by deforming pre-scanned human models, but the resulting geometries may be unrealistic and templates are often unavailable.These methods assume access to a pre-scanned human shape.
- Human performance capture: Learned human-prior methods recover 3D geometry and texture from single images but struggle with photorealistic view synthesis and unseen complex poses.Their limitations are especially relevant for dynamic humans under sparse observations.
- Neural representation-based methods: Neural representation methods learn scene representations from 2D images using differentiable renderers across voxels, point clouds, textured meshes, multi-plane images, and implicit functions.SRN and NeRF exemplify implicit neural representations in this related-work landscape.
3. Neural Body
Neural Body represents dynamic humans with structured latent codes anchored to a deformable model, then diffuses and decodes them into frame-dependent density and color fields for volume rendering. Training jointly uses observations across all video frames to synthesize novel views and reconstruct performers.
- Structured latent codes: Neural Body attaches structured latent codes to a deformable human model and generates each frame’s geometry and appearance from the same codes.The shared-code latent-variable formulation integrates observations across video frames while pose-dependent deformation controls code locations.
- Code diffusion: SparseConvNet diffuses surface-anchored codes into nearby 3D space, enabling latent-code queries at continuous locations.The method voxelizes the SMPL bounding box at 5mm × 5mm × 5mm resolution, processes sparse codes at multiple scales, and interpolates the resulting volumes.
- Density and color regression: MLP networks decode queried latent codes into volume density and color, with color additionally conditioned on viewing direction, spatial location, and frame-specific appearance embeddings.The density model uses only the queried latent code, whereas the color model incorporates positional encodings and a 128-dimensional frame embedding.
- Volume rendering: Volume rendering samples 64 points along each camera ray, predicts density and color at those points, and accumulates them into pixel colors.Ray bounds are estimated from the SMPL model, and adjacent-sample distances determine the rendering weights.
- Training: The model is optimized by minimizing total squared error between rendered and observed images across all cameras and video frames.Unlike frame-wise reconstruction, this optimization uses all video images; training a four-view, 300-frame video typically takes about 200k iterations, or 14 hours, on four 2080 Ti GPUs.
- Applications: The trained representation supports free-viewpoint video synthesis and 3D reconstruction, with generated videos exhibiting high inter-frame and inter-view consistency.Novel views are produced through volume rendering, while reconstruction discretizes the scene using 5mm × 5mm × 5mm voxels.
4. Experiments
Experiments evaluate Neural Body on sparse multi-view and monocular videos, showing stronger view synthesis and reconstruction than prior methods. Ablations indicate benefits from shared latent embeddings, more camera views, and video-frame integration, while excessively long training videos can hurt performance.
- Experimental setup: The ZJU-MoCap dataset contains 9 dynamic human videos captured with 21 synchronized cameras, using four cameras for training and the remaining views for testing.Sequences span 60–300 frames and include twirling, Taichi, arm swings, warmup, punching, and kicking.
- Novel view synthesis: For both PSNR and SSIM, Neural Body achieves the best results, exceeding prior methods by at least 4.20 PSNR and 0.047 SSIM.Comparisons include Neural Volumes, Neural Textures, and NHR.
- Novel view synthesis: Neural Body produces photorealistic, inter-frame- and inter-view-consistent free-viewpoint videos, whereas competing methods show inaccurate geometry, blur, or poor viewpoint control.The qualitative comparison uses four input cameras and two selected novel views.
- 3D reconstruction: Neural Body reconstructs accurate human geometry from four views, while COLMAP, DVR, and PIFuHD fail to recover reasonable or pose-consistent shapes on complex motions.PIFuHD also generalizes poorly, especially for complex poses and clothing.
- Monocular videos: On People-Snapshot monocular videos, Neural Body renders more appearance details and geometric detail than the baseline, including loose clothing, hair, blouses, and pants.The method also handles strong outdoor illumination variations in the reported qualitative results.
- Ablation studies: Removing per-frame latent embeddings lowers performance from 30.56 PSNR to 30.03 PSNR, a 0.53 PSNR improvement for the complete model.Training on more camera views improves novel view synthesis, but excessively long videos can decrease performance because they are harder to fit.
5. Conclusion
The paper introduces Neural Body for novel view synthesis of dynamic humans from sparse multi-view videos. It anchors shared latent codes to a deformable human model, integrates observations across frames, and reports superior view synthesis quality on ZJU-MoCap and People-Snapshot.
- Conclusion: Neural Body represents dynamic humans with latent codes anchored to vertices of a deformable human model, generating implicit fields across video frames from shared codes.The representation is learned over the video with volume rendering.
- Conclusion: The shared latent-code formulation incorporates observations across video frames while the deformable model supplies local geometry and appearance structure.The paper describes this as a latent variable model for dynamic human representation.
- Conclusion: Neural Body achieves superior view synthesis quality to prior work on the newly collected ZJU-MoCap dataset and the People-Snapshot dataset.