PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuMan: Neural Human Radiance Field from a Single Video

Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, Anurag Ranjan

arXiv:2203.12575v2cs.CV

TL;DR

NeuMan addresses the challenge of reconstructing and reposing a human with its scene from only one in-the-wild video. It trains human and scene NeRFs using estimated geometry, canonical-space warping, geometry refinement, and error correction. The resulting models render detailed humans under novel poses and views together with the background, while dynamics beyond SMPL remain a limitation.

  • Problem

    Reconstructing a human and static scene with novel-pose and novel-view rendering from one in-the-wild video remains difficult without multi-camera setups or manual annotations.

  • Method

    NeuMan trains separate human and scene NeRFs initialized by off-the-shelf geometry estimates, using SMPL-based canonical warping, end-to-end SMPL optimization, and error correction.

  • Results

    NeuMan renders detailed, reposed humans from novel views together with the scene, learning cloth and subject-specific geometric details from less than 40 images.

  • Takeaways & Limitations

    The framework supports composing reposed human and scene NeRFs, including combining multiple human models in one scene without additional training.

  • Takeaways & Limitations

    Dynamics beyond SMPL, including changing hand gestures and garment dynamics, cannot be modeled by the static NeRF and may degenerate to average shape or color.

Abstract

from arXiv · show

Photorealistic rendering and reposing of humans is important for enabling augmented reality experiences. We propose a novel framework to reconstruct the human and the scene that can be rendered with novel human poses and views from just a single in-the-wild video. Given a video captured by a moving camera, we train two NeRF models: a human NeRF model and a scene NeRF model. To train these models, we rely on existing methods to estimate the rough geometry of the human and the scene. Those rough geometry estimates allow us to create a warping field from the observation space to the canonical pose-independent space, where we train the human model in. Our method is able to learn subject specific details, including cloth wrinkles and accessories, from just a 10 seconds video clip, and to provide high quality renderings of the human under novel poses, from novel views, together with the background.

1 Introduction

NeuMan reconstructs a human and static scene from one moving-camera video, enabling pose-driven human rendering and novel views without multi-camera setups or manual annotations. It addresses inaccurate geometry estimates through end-to-end refinement and error correction while jointly training human and scene NeRFs.

  • Problem and contribution: NeuMan targets human-and-scene reconstruction from a single in-the-wild video, avoiding expensive multi-camera setups and manual annotations.Existing approaches require conditions such as multiple cameras, consistent lighting, clean backgrounds, accurate geometry, or manual masks.
  • Capability: NeuMan is presented as the only compared method that renders both a scene and a reposed human from a single video.The comparison also distinguishes NeuMan from methods that cannot drive humans by pose, require multiple videos, or cannot model the background.
  • Framework: The framework trains separate human and scene NeRF models to render humans under novel poses and views together with the background.The scene model is guided by segmentation masks and depth estimates, while the human model supports pose-driven rendering.
  • Framework: NeuMan uses off-the-shelf estimates of human pose, shape, masks, camera poses, sparse scene structure, and depth to initialize reconstruction.The human NeRF is trained in a pose-independent canonical volume based on SMPL, while the scene NeRF is regularized with fused depth estimates.
  • Geometry correction: End-to-end SMPL optimization and an error-correction network compensate for erroneous human geometry and warping errors during training.These components address misalignment between estimated SMPL geometry and the underlying human, helping preserve human details.

2 Related Work

Prior work extends NeRF toward controllable human rendering, but existing approaches often depend on multi-camera supervision or impose limitations on novel-pose synthesis. This paper builds on canonical-space human modeling and NeRF volume rendering to address single-video reconstruction.

  • Neural Radiance Fields (NeRF): NeRF represents scenes as radiance fields that support novel-view rendering through volume rendering.The field predicts color and density from 3D locations and viewing directions, and ray samples are integrated to obtain pixel colors.
  • Human-conditioned NeRF: Human-conditioned NeRF methods use explicit models such as SMPL or 3D skeletons to represent and control human appearance and pose.Examples include associating latent codes with SMPL vertices, warping observations into canonical space, and learning blending fields for novel poses.
  • Human-conditioned NeRF: Many prior human-rendering methods require expensive multi-camera setups for ground-truth bounding boxes, 3D poses, or SMPL estimates.This dependence limits their suitability for reconstructing humans from a single in-the-wild video.
  • Human-conditioned NeRF: ST-NeRF avoids precise human-geometry estimation but cannot extrapolate to unseen poses because it depends on frame time.Its representation separates the human into 3D bounding boxes and models dynamics within each box.
  • Single-video human rendering: HumanNeRF also targets free-viewpoint human rendering from a single video, but relies on manual masks and represents motion with skeletal and non-rigid transformations.The comparison motivates automated human-scene decomposition and a motion representation intended to mitigate ambiguity under novel poses.

3 Method

NeuMan separates the background and human into two NeRF models, using estimated masks, depths, and camera geometry to train them. The human model is pose-independent through canonical-space warping, refined SMPL alignment, and rendering-specific ray integration.

  • Framework: The framework trains a scene NeRF for the background first, followed by a human NeRF conditioned on the trained scene model.The scene model is trained from background pixels identified by human segmentation masks.
  • 3.1 The Scene NeRF Model: Scene NeRF training combines RGB reconstruction on background rays with depth-based empty-space regularization using fused multi-view and monocular depth estimates.The fused depth map fills holes in multi-view depth while preserving its scale; α = 0.8 provides slack for inaccurate depth estimates, and λempty = 0.1 controls the emptiness regularizer.
  • 3.2 The Human NeRF Model: The human NeRF is trained in a pose-independent canonical volume defined by the Da-pose SMPL mesh, which avoids leg volume collisions during warping.Observation-space points are transformed using the rigid motion of their closest SMPL-mesh points, then extended through a volume warping field.
  • 3.2 The Human NeRF Model: Jointly optimizing SMPL parameters with the human NeRF corrects pose-estimation misalignment, while an error-correction network models details absent from SMPL.The error-correction network is used only during training and discarded for validation and novel-pose rendering, allowing it to overfit frame-specific warping errors.
  • Rendering: Rendering evaluates human and scene rays together, sorts their samples by depth, and integrates the combined colors and densities; viewing directions are recomputed along warped canonical rays.The warped observation-space ray becomes curved in canonical space, so viewing angles account for the actual trajectory of successive canonical samples.
  • 3.2 The Human NeRF Model: The human loss combines RGB, LPIPS, mask, SMPL, hard-surface, and canonical-edge terms to constrain appearance, segmentation, geometry, and opacity.The hard-surface and canonical-edge losses encourage terminating weights and accumulated canonical alpha values toward binary 0-or-1 behavior.

4 Experiments

Experiments evaluate NeuMan’s dataset, scene and human reconstructions, qualitative capabilities, comparisons, and ablations. The method reconstructs backgrounds and detailed, reposable humans, while geometry correction and Lsmpl improve rendering quality.

  • 4.1 Dataset: The NeuMan dataset contains mobile-phone videos of single-person walking sequences with camera motion enabling multi-view reconstruction.The experiments introduce a dataset because existing motion-capture and video datasets do not match the single-video setting.
  • 4.2 Qualitative Results: Scene NeRF reconstructs consistent background geometry from background pixels and effectively removes the dynamic human.Figure 5 reports high-quality novel-view background renderings despite limited scene coverage.
  • 4.2 Qualitative Results: Human NeRF captures cloth patterns and subject-specific geometry, supports novel poses and views, and composes the reposed human with the scene.Reported details include sleeves, collars, and zippers beyond SMPL expressiveness; walking training data also supports cartwheeling motion.
  • 4.2 Qualitative Results: NeuMan enables telegathering by combining multiple human NeRF models in one scene without additional training.Figure 7 shows SMPL meshes, a scene point cloud overlay, and novel-view renderings.
  • 4.3 Quantitative Results: NeuMan faithfully reconstructs drastically dynamic human scenes where NeRF-T and HyperNeRF fail to reconstruct and interpolate across space and time.The comparison uses the same training data for NeRF-T and densely sampled frames for HyperNeRF.
  • 4.4 Ablation Studies: Geometry correction preserves human texture and geometric details, while Lsmpl encourages a clean canonical volume and suppresses rendering halos.Without geometry correction, the canonical volume overfits observations; disabling Lsmpl introduces fog around the human.

5 Conclusions

NeuMan reconstructs human and scene NeRF models from a single in-the-wild video for novel human poses and views. The conclusion highlights learned clothing and geometric detail, while identifying limits from unmodeled dynamics, extreme warping, and ground-contact assumptions.

  • 5 Conclusions: NeuMan reconstructs human and scene NeRF models renderable under novel human poses and views from a single in-the-wild video.Off-the-shelf 2D and 3D geometry estimates provide initialization.
  • 5 Conclusions: The human model learns cloth patterns and details such as sleeves, collars, and zippers from less than 40 images.These details include both texture and geometry.
  • 5 Conclusions: Dynamics beyond SMPL cannot be modeled by the static NeRF and may degenerate to average shape or color, especially for changing hand gestures.The paper suggests more expressive body and garment models as future work.
  • 5 Conclusions: The SMPL-based warping function can cause volume collisions in extreme cases, limiting generalization under extreme poses.The paper identifies collision-aware or learned warping as a needed improvement.
  • 5 Conclusions: The method assumes at least one human ground contact point to estimate scale relative to the scene.Jumping or uneven-ground videos require smarter geometric reasoning.

A.1 Dataset Details

The dataset details section supplies the dataset’s frame-count reference for training, validation, and testing.

  • A.1 Dataset Details: Table 3 reports the number of frames used for training, validation, and testing in each dataset.The supplied passage identifies the table’s organization but does not include the frame counts.

A.2 SMPL Refinement

SMPL refinement combines detected human evidence with differentiable rendering to refine noisy body and camera estimates. Different silhouette sources are used for preprocessing and NeRF training because they serve different representation goals.

  • A.2 SMPL Refinement: HigherHRNet and DensePose regress 2D joints and human masks, while ROMP estimates the SMPL mesh and parameters.The mesh includes vertices, faces, and 3D joints.
  • A.2 SMPL Refinement: Soft-rasterizer refinement minimizes silhouette and projected-joint discrepancies between observations and the rendered SMPL estimate.The objective compares m with m̂ and j2d with ĵ2d.
  • A.2 SMPL Refinement: DensePose silhouettes guide preprocessing, whereas Mask-RCNN masks guide human NeRF training to capture geometry beyond the SMPL model.DensePose matches naked-SMPL correspondences, while Mask-RCNN better estimates extra 2D geometry details.

A.3 Network Architecture

The architecture uses separate coarse and fine scene NeRF sub-models, while the human NeRF uses one sub-model because prior SMPL geometry is available. These components share the architecture shown in Figure 10.

  • A.3 Network Architecture: Figure 10 distinguishes raw inputs, positional encodings, hidden layers, and final outputs by box color and labels feature dimensions.Concatenation is marked with a plus sign, and ReLU follows each hidden layer.
  • A.3 Network Architecture: The scene NeRF contains coarse and fine sub-models, whereas the human NeRF uses only one sub-model based on SMPL geometry estimates.The error-correction network shares the same architecture as each sub-model.

A.4 Comparison with previous works

Comparisons with prior methods show that the proposed approach produces stronger novel-view renderings in monocular and single-camera settings, including better generalization to unseen subject regions and poses.

  • A.4 Comparison with previous works: NeuralBody overfits training observations in the monocular Seattle sequence, while the proposed method renders the subject’s back more faithfully.The comparison uses a pose rendered from the left training image and contrasts novel-view results from both methods.
  • A.4 Comparison with previous works: On the ZJU Mocap dataset, the proposed method renders high-quality novel views and extrapolates in pose space.The comparison includes HumanNeRF and NeuralBody; the proposed method and HumanNeRF use one camera view, while NeuralBody uses multiple views.

A.5 Error Correction Network and Scene Model Conditioning

The error-correction network and scene-NeRF conditioning preserve details and separate human appearance from background content. Removing either component causes visible canonical-rendering artifacts.

  • A.5 Error Correction Network and Scene Model Conditioning: Without error correction, the canonical NeRF loses cloth and facial details and produces fog around the human when hallucinating clothing dynamics.The full model, the model without error correction, and the model without scene-NeRF conditioning are compared in Figure 13.
  • A.5 Error Correction Network and Scene Model Conditioning: Training the human NeRF alone can encode background pixels because of segmentation errors, leading to fog around the human and hallucinated background colors.Conditioning on the scene NeRF addresses the interaction between human reconstruction and background content.
Loading 2203.12575v2…