Source-linked AI summary
NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras
Xin Suo, Yuheng Jiang, Pei Lin, Yingliang Zhang, Kaiwen Guo, Minye Wu, Lan Xu
TL;DR
Sparse multi-view RGB methods still fall short of recovering fine geometry and texture together at the level of input images. NeuralHumanFVV combines hierarchical implicit geometry inference with neural blending and normal refinement. The paper reports high-quality novel-view geometry and photo-realistic texture, including challenging poses, clothing, and topology changes.
Problem
Recent RGB-based approaches do not recover fine geometry and texture together with the detail present in input images, despite the importance of free-viewpoint human rendering for immersive VR/AR.
Source: §1 · p. 1Method
NeuralHumanFVV uses hierarchical coarse-to-fine implicit geometry inference, neural blending of adjacent-view images, and blended normals for geometry refinement.
Source: §1 · p. 2Results
Takeaways & Limitations
The system generates novel-view geometry and texture at 12 fps from six RGB camera streams for interactive immersive applications.
Source: §5 · p. 6When test images deviate substantially from training images, generated geometry may be plausible but physically inaccurate.
Source: §6 · p. 8
Abstract
from arXiv · showhide
4D reconstruction and rendering of human activities is critical for immersive VR/AR experience.Recent advances still fail to recover fine geometry and texture results with the level of detail present in the input images from sparse multi-view RGB cameras. In this paper, we propose NeuralHumanFVV, a real-time neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of human activities in arbitrary novel views. We propose a neural geometry generation scheme with a hierarchical sampling strategy for real-time implicit geometry inference, as well as a novel neural blending scheme to generate high resolution (e.g., 1k) and photo-realistic texture results in the novel views. Furthermore, we adopt neural normal blending to enhance geometry details and formulate our neural geometry and texture rendering into a multi-task learning framework. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality geometry and photo-realistic free view-point reconstruction for challenging human performances.
1. Introduction
Immersive VR/AR increases demand for free-viewpoint human reconstruction, but existing approaches leave a gap in detailed geometry and texture under lightweight, real-time RGB capture. NeuralHumanFVV combines sparse-view geometry inference with neural blending to render high-quality geometry and photo-realistic textures in novel views.
1. Introduction
- Sparse multi-view RGB methods still struggle to recover fine geometry and texture together while supporting lightweight, real-time capture.Depth-camera approaches are less cheap and ubiquitous than color cameras, while some existing RGB methods do not produce compelling photo-realistic texture.Source: §1 · p. 1
- NeuralHumanFVV targets this gap with high-quality geometry and photo-realistic textures in arbitrary novel views from sparse surrounding RGB cameras.The paper reports significant superiority to existing state-of-the-art methods.Source: §1 · p. 2
- Hierarchical coarse-to-fine sampling along camera rays enables real-time implicit geometry inference with fine detail.The method reasons about underlying geometry in the novel view.Source: §1 · p. 2
- Neural blending maps adjacent input views to high-resolution texture, while blended normals refine geometry and support multi-task geometry-texture learning.The paper describes high-resolution output such as 1k views.Source: §1 · p. 2
2. Related Work
Prior human-capture methods trade among capture hardware, occlusion, detail, and real-time performance. NeuralHumanFVV combines implicit geometry modeling and neural blending for novel-view rendering from six RGB cameras.
Human Performance Capture
- High-end capture uses hundreds of cameras and controlled imaging, while monocular RGBD methods can miss motion in self-occluded regions.Lightweight multi-view alternatives still use three to eight RGBD streams.Source: §2 · p. 2
Data-Driven Human Modeling
- Data-driven human modeling predicts attributes and implicit geometry from RGB, but prior methods still fail to recover input-level fine geometry and texture simultaneously.Some approaches use parametric human models or implicit occupancy for textured geometry and real-time inference.Source: §2 · p. 2
Neural Rendering
- Neural scene methods may require per-scene training, while image-to-view methods can rely on heavy networks or complicated 3D proxies unsuitable for real time.Human novel-view methods using latent geometry also face limited representation for complex texture output.Source: §2 · pp. 2–3
- NeuralHumanFVV combines neural blending with implicit geometry in a lightweight multi-RGB, real-time pipeline for novel views.Its pipeline takes six surrounding RGB videos and uses neural geometry generation followed by neural blending.Source: §2 · p. 3, Figure 2 · p. 3
3. Overview
NeuralHumanFVV takes six surrounding RGB videos and produces novel-view rendering with geometry and texture. Its geometry-generation and neural-blending stages respectively infer view geometry and synthesize appearance.
3. Overview
- The system takes six RGB videos surrounding the performer and produces high-quality novel-view synthesis across varied poses, clothing, and topology changes.The pipeline marries implicit volumetric modeling with neural texture rendering.Source: §3 · p. 3
Neural Geometry Generation
- Shape-from-silhouette priors and pixel-aligned implicit functions support complete geometry, while hierarchical ray sampling enables real-time novel-view inference.The inferred geometry also serves as a proxy for the subsequent blending stage.Source: §3 · p. 3
Neural Blending
- Neural blending uses occlusion analysis and learned weights to map adjacent-view images into target-view texture, with boundary-aware upsampling for high-resolution output.The same blending strategy restores target-view normals, refining geometry and coupling geometry and texture generation in multi-task learning.Source: §3 · p. 3
4. NeuralHumanFVV Method
NeuralHumanFVV reconstructs novel-view human geometry from sparse RGB cameras and blends adjacent-view information for detailed, photo-realistic rendering. Its method combines coarse-to-fine geometry inference, neural texture and normal blending, and training on synthetic and pipeline-generated data.
4.1. Neural Geometry Reconstruction
- A coarse SfS shape guides novel-view MVIFu ray sampling, and hierarchical depth fine-tuning recovers details beyond the smoothed geometry estimate.Figure 3 depicts coarse reconstruction, MVIFu, and refinement in successive stages.Source: §4.1 · pp. 3–4, Figure 3 · p. 4
4.2. Neural Blending
- The blending pipeline warps two adjacent input views into the target view and learns pixel-wise weights to combine their local detail with multi-view information.It uses warped source depths to obtain occlusion information before blending.Source: §4.2 · p. 4, Figure 4 · p. 5
- Boundary-aware depth recomputation upsamples 256×256 depth to 1K, while blended normals drive a refinement network that adjusts target depth.The boundary area is recalculated using the geometry pipeline; the normal-refinement network predicts depth displacement from blended normals and upsampled depth.Source: §4.2 · pp. 4–5
4.3. Data and implementation Details
- The feature extractor, implicit function, depth fine-tuner, texture blender, and normal refiner use U-Net or MLP architectures, with the implicit network reduced for real-time performance.The feature extractor outputs 64-channel feature maps; the blending and normal-refinement networks adopt U-Net structures.Source: §4.3 · p. 5
- Training uses 1,820 Twindom scans augmented with rigged poses, six fixed input views, and 180 sampled virtual target views per model.The dataset includes RGB images, normal maps, and depth maps for the views and models.Source: §4.3 · p. 5
- Texture blending is trained against RGB appearance and normal targets with a shared blending map and perceptual loss; normal refinement uses self-supervision from pipeline-generated real data.The RGB and normal tasks share the blending map; the normal-refinement network is not trained with the same synthetic dataset.Source: §4.3 · p. 5
5. Experimental Results
NeuralHumanFVV is evaluated on challenging human sequences, with comparisons against prior methods and ablations of its geometry, texture, and camera-view components. The reported experiments assess both reconstruction quality and real-time performance.
5. Experimental Results
- With six live RGB cameras, the system produces novel-view geometry and texture at 12 fps, including for interactions with topology changes.The pipeline takes about 80 ms per frame: 64 ms for geometry generation and 16 ms for neural blending.Source: §5 · p. 6
5.1. Comparison
- On complex clothing such as a floral dress, NeuralHumanFVV produces better texture and more detailed geometry than the compared methods.Using Multi-PIFu geometry with per-vertex texture mapping results in more frequent image blurring.Source: §5 · p. 6, Figure 6 · p. 7
- NeuralHumanFVV achieves lower overall MAE than the compared methods across all scenarios in the real testing dataset.The comparison uses one-camera and six-camera settings.Source: Table 1 · p. 6, §5.1 · p. 7
- Across a 600-frame synthetic sequence and 90 target views, NeuralHumanFVV maintains the lowest MAE throughout.Figure 7 includes the methods’ error curves and qualitative outputs.Source: §5.1 · p. 7, Figure 7 · p. 7
5.2. Ablation Study
- Normal refinement improves geometry beyond the mid-level wrinkles produced without it, yielding details close to those in the input images.Without refinement, results remain especially smooth on faces; shape-from-silhouette provides only coarse geometry priors.Source: §5.2 · p. 7, Figure 8 · p. 8
- The full neural texture scheme avoids the block artifacts, seam-region inferiority, oversmoothing, and coarse boundaries seen in the tested alternatives.The alternatives include per-vertex texturing, AGI, low-resolution neural texturing, and neural texturing without boundary optimization.Source: §5.2 · p. 7, Figure 9 · p. 8
- The component comparison reports fewer artifacts for the complete system than for variants changing geometry or texture, with per-frame mean errors also evaluated.The variants use Multi-PIFu geometry, per-vertex texture mapping, or both.Source: §5.2 · p. 7
- Reducing the number of input cameras increases average error and produces geometry and blending artifacts; six cameras are reported as a good compromise.The camera-count experiment evaluates the influence of input views in the multi-view setting.Source: §5.2 · p. 8
6. Discussion
NeuralHumanFVV is presented as a real-time system for high-quality geometry and photo-realistic novel-view textures from sparse RGB cameras. Its limitations include resolution- and segmentation-related issues, and geometry may be physically inaccurate when testing images differ substantially from training images.
6. Discussion
- Thin structures such as fingers are difficult to reconstruct at limited input resolution, and segmentation inaccuracies leave incomplete synthesized regions.Texture results also depend on input image resolution.Source: §6 · p. 8
- Figure 10 compares reconstructions with neural geometry and texture components removed and includes numerical error curves.Source: Figure 10 · p. 8
- Figure 11 evaluates input-camera counts with a mean-absolute-error cumulative distribution and texture reconstructions using six, four, and two cameras.Source: Figure 11 · p. 8
- Geometry can be physically inaccurate when testing images deviate substantially from the training images.Source: §6 · p. 8
- The system generates high-quality geometry and photo-realistic textures of human activities in novel views using sparse multiple RGB cameras.The authors report effectiveness in challenging scenarios with varied poses, clothing types, and topology changes.Source: §6 · p. 8