PaperlayerView exact revision on arXiv

Source-linked AI summary

KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints

Marko Mihajlovic, Aayush Bansal, Michael Zollhoefer, Siyu Tang, Shunsuke Saito

arXiv:2205.04992v2cs.CV

TL;DR

Sparse-view image-based human reconstruction still faces difficulty with fine details, and spatial encodings can overfit training distributions. KeypointNeRF encodes spatial information relative to sparse 3D keypoints and combines it with pixel-aligned image features. It reports state-of-the-art head reconstruction, comparable body reconstruction, and transfer to unseen iPhone captures, while noting sensitivity to occlusions and difficulty with thin geometry and highly articulated motion.

  • Problem

    Sparse-view reconstruction of unseen human faces and bodies can lack fine detail, while existing spatial encodings may overfit training distributions and multi-view stereo constraints are less robust with sparse, widely spaced views.

    Source: §1 · p. 2, §2 · p. 4

  • Method

    The method encodes query-point spatial information relative to sparse 3D keypoints and fuses it with pixel-aligned image features to model volumetric humans.

    Source: §1 · p. 3, §4.2 · p. 7

  • Results

    The method achieves state-of-the-art results for head reconstruction, performance comparable to prior work for body reconstruction, and generalizes from studio training to unseen iPhone captures.

    Source: §1 · p. 3, §7 · pp. 13–14

  • Takeaways & Limitations

    The keypoint-based approach supports reconstruction from as few as two views and can be applied to bodies without a parametric template mesh.

    Source: §7 · pp. 13–14

    The method is sensitive to occlusions, has difficulty reconstructing thin geometries such as glasses, and is less robust to highly articulated motion.

    Source: E Limitations and Future Work · p. 24

Abstract

from arXiv · show

Image-based volumetric humans using pixel-aligned features promise generalization to unseen poses and identities. Prior work leverages global spatial encodings and multi-view geometric consistency to reduce spatial ambiguity. However, global encodings often suffer from overfitting to the distribution of the training data, and it is difficult to learn multi-view consistent reconstruction from sparse views. In this work, we investigate common issues with existing spatial encodings and propose a simple yet highly effective approach to modeling high-fidelity volumetric humans from sparse views. One of the key ideas is to encode relative spatial 3D information via sparse 3D keypoints. This approach is robust to the sparsity of viewpoints and cross-dataset domain gap. Our approach outperforms state-of-the-art methods for head reconstruction. On human body reconstruction for unseen subjects, we also achieve performance comparable to prior work that uses a parametric human body model and temporal feature aggregation. Our experiments show that a majority of errors in prior work stem from an inappropriate choice of spatial encoding and thus we suggest a new direction for high-fidelity image-based human modeling. https://markomih.github.io/KeypointNeRF

1 Introduction

The paper targets personalized volumetric humans from only two or three views, avoiding subject-specific methods that require days of training. It analyzes spatial encodings and proposes keypoint-relative encoding, reporting strong reconstruction and cross-domain generalization.

1 Introduction

  • The approach aims to make personalized reconstruction scalable, in contrast to subject-specific learning-based methods that require days of training per person.The stated goal is reconstruction from two or three snaps taken from different views with a phone.Source: §1 · p. 2
  • Camera-depth encodings overfit the training distribution, while multi-view stereo constraints are less robust with sparse views and large baselines.The authors’ analysis finds that spatial-encoding choice affects reconstruction quality and generalization to novel identities and views.Source: §1 · p. 2
  • The method encodes query locations relative to sparse 3D keypoints, which are obtained by detecting 2D keypoints and triangulating views.The authors report state-of-the-art reconstruction of unseen subjects from two or three widely spaced views, with comparable full-body performance to Neural Human Performer.Source: §1 · p. 3
  • A model trained on studio captures generalizes without modification to never-before-seen iPhone captures, including in-the-wild inputs.The authors report this as a result of training on only a studio-captured dataset.Source: §1 · p. 3

2 Related Work

Prior neural rendering methods improve volumetric reconstruction but often need dense multi-view supervision or struggle with sparse-view detail. The paper positions relative keypoint depth encoding as a flexible alternative for faces and bodies.

2 Related Work

  • Neural volumetric methods handle regions where geometry is difficult to estimate, but person-specific variants require dense multi-view supervision and 3–4 days of training per model.Earlier mesh-based approaches also face difficulty reconstructing hair, mouth, and teeth where accurate geometry is hard to obtain.Source: §2 · pp. 3–4
  • Pixel-aligned sparse-view methods combine multi-view constraints with NeRF, but struggle to produce fine details for unseen faces and bodies from sparse views.The paper’s stated target is high-fidelity reconstruction of unseen subjects from as few as two views.Source: §2 · pp. 3–4
  • Relative depth encoding with 3D keypoints is presented as sufficient for sparse-view learning across faces and bodies, with state-of-the-art face results and competitive body reconstruction against prior work.The authors contrast their reliance on keypoints alone with methods using human-face or body priors, including SMPL and temporal feature aggregation.Source: §2 · pp. 4–5

3 Preliminaries: Neural Radiance Fields

NeRF represents a scene with a volumetric radiance field that predicts color and density from spatial position and viewing direction. Pixel-aligned methods condition that field on image features and spatial encodings to generalize across scenes.

3 Preliminaries: Neural Radiance Fields

  • NeRF predicts color and density at 3D points from position and viewing direction, then integrates those values along camera rays to estimate pixel color.The ray integral uses near and far bounds, defined as t_n and t_f.Source: §3 · p. 5
  • Pixel-aligned neural fields condition NeRF on image features to generalize to novel scenes, while spatial encodings help address ray-depth ambiguity.Prior encodings include camera-space depth and coordinates relative to the head; the paper argues these global encodings are suboptimal compared with localized relative encoding.Source: §3 · p. 5

4 KeypointNeRF

KeypointNeRF combines relative spatial information from triangulated 3D keypoints with pixel-aligned image features to condition a volumetric radiance field. Its components fuse per-view evidence to predict density and color for novel-view rendering.

4 KeypointNeRF

  • The radiance field takes a query point and viewing direction alongside calibrated input images, and predicts color and density through four network components.Those components are the spatial keypoint encoder, convolutional image encoders, feature-fusion network, and density and color prediction networks.Source: §4 · p. 5

4.1 Relative Spatial Keypoint Encoding

  • Relative spatial encoding uses triangulated 3D keypoints and each query point’s depth difference from them in each input view.The encoding combines positional encoding of relative depth with a Gaussian distance kernel; α controls each keypoint’s impact and is set to 5 cm for facial keypoints and 10 cm for the human skeleton.Source: §4.1 · pp. 6–7

4.2 Convolutional Pixel-aligned Features

  • Two convolutional image encoders extract pixel-aligned features, including a human geometric prior for density estimation and a separate appearance pathway.Query points are projected onto feature planes and feature values are bilinearly interpolated.Source: §4.2 · p. 7

4.3 Multi-view Feature Fusion

  • The method transforms per-view spatial encodings, blends them with deep pixel-aligned features, and combines the result with shallow features into per-view vectors.Mean and variance pooling fuse the multi-view features into a single 128-dimensional vector.Source: §4.3 · p. 7

4.4 Modeling Radiance Fields

  • Decoupled MLPs predict view-independent density and view-dependent color, with color formed by blending input pixel values.The color network uses geometry features, appearance features, pixel values, and view-direction information to predict blending weights.Source: §4.4 · pp. 7–8

5 Novel View Synthesis

Novel views are rendered by integrating a radiance field between proxy-defined ray bounds, using a shared network for coarse-to-fine rendering. Training uses rendered patches and combines RGB and VGG losses.

5 Novel View Synthesis

  • Novel views use coarse-to-fine volume rendering with shared network weights and near/far bounds from ray intersections with a geometric proxy.Head experiments use a 30 cm-radius sphere centered on keypoints; body experiments use a 3D bounding box.Source: §5 · p. 8

5.1 Training and Implementation Details

  • Training renders patches from 64 coarse and 64 fine ray samples, then minimizes the mean RGB ℓ1-distance and VGG loss against ground-truth patches.The VGG loss is applied over rendered and ground-truth image patches.Source: §5.1 · p. 8
  • The combined loss is optimized with Adam at a learning rate of 1e−4 and batch size one; input-image backgrounds are removed with an off-the-shelf matting network.For inference with two input images, the maximum dot product is clipped to 0.8 for more temporally coherent novel-view synthesis.Source: §5.1 · p. 8

6 Experiments

The experiments evaluate head and body reconstruction from sparse views, including transfer from studio training to in-the-wild iPhone captures. They report sharper studio head renderings, cross-dataset generalization, and body performance on par with a more complex method.

6 Experiments

  • The experiments cover studio head reconstruction, in-the-wild head reconstruction, and body reconstruction, reporting SSIM and PSNR.Studio evaluation uses 38 novel identities and reconstructs at inference from two or three input views.Source: §6 · p. 9, §6.1 · p. 9

6.1 Reconstruction of Human Heads from Studio Data

  • For unseen identities reconstructed from two studio images, rendered results are sharper and higher quality than the baselines, and distance weighting improves reconstruction quality.The pipeline without spatial encoding is third-best in the studio setting but does not generalize well to other capture systems.Source: §6.1 · pp. 9–10, Table 1 · p. 9, Figure 3 · p. 10
  • Keypoint-relative encoding degrades more slowly under keypoint noise than canonical-space encoding, and the neutral-face-trained model generalizes to dynamic expressions while outperforming baselines.The expression evaluation covers 38 test subjects performing eight expressions.Source: §6.1 · p. 11, Table 2 · p. 11, Table 3 · p. 11

6.2 Reconstruction from In-the-wild Captures

  • With three iPhone images and no retraining from studio data, the proposed method reliably reconstructs heads while the baselines produce blur and cloudy artifacts.The improvement is quantitatively supported by SSIM and PSNR on novel held-out views of four visualized subjects.Source: Figure 4 · p. 12, §6.2 · p. 13, Table 4 · p. 13

6.3 Reconstruction of Human Bodies

  • On unseen identities and 438 novel views, the method is on par with Neural Human Performer while using skeleton keypoints rather than SMPL registration and temporal feature fusion.The body reconstruction setup uses three input views and does not rely on parametric-body template fitting.Source: §6.3 · p. 13

7 Conclusion

KeypointNeRF uses relative spatial information from 3D keypoints to generate high-fidelity volumetric humans from as few as two input images. The paper reports strong head-reconstruction and out-of-domain results, and body-reconstruction performance comparable to more complicated prior work.

7 Conclusion

  • Relative spatial information from 3D keypoints underpins high-fidelity volumetric humans from as few as two input images.The method does not require a parametric template mesh and can be applied to body reconstruction without modification.Source: §7 · pp. 13–14
  • The method outperforms state-of-the-art methods for head reconstruction and better generalizes to challenging out-of-domain inputs, including iPhone selfies captured in the wild.The conclusion says the method reliably reconstructs heads; the human-body comparison is scoped to unseen identities in the ZJU-MoCap dataset.Source: §7 · p. 13, Figure 5 · p. 14
  • For body reconstruction, performance is comparable to prior work that relies on parametric human body models and temporal feature aggregation.Table 5 compares the method with pixelNeRF, PVA, and NHP; its caption also identifies a variant without relative-encoding weighting.Source: §7 · p. 14, Table 5 · p. 14

using Relative Spatial Encoding of Keypoints — Supplementary Material —

The supplementary material provides implementation and baseline details, additional results, and discussion of limitations and future work.

Supplementary Material

  • The supplement provides additional implementation details.These are in Section B.Source: A Overview · p. 20
  • It includes information about the baseline methods.This material is in Section C.Source: A Overview · p. 20
  • It presents more qualitative and quantitative results and reflects on limitations and future work.These topics are covered in Sections D and E, respectively.Source: A Overview · p. 20

B Implementation Details

The implementation combines image encoders with multi-view feature fusion, density estimation, and view-dependent color prediction. The supplement also reports qualitative comparisons and specifies the sources and setup of baseline methods.

B Implementation Details

  • A single HourGlass image encoder produces deep and shallow feature maps, while an independent convolutional encoder provides an appearance pathway.The deep map uses four convolutional blocks and an HourGlass block; the appearance encoder is a 15-layer residual network.Source: B Implementation Details · p. 20
  • A four-layer MLP fuses multi-view features using mean-variance pooling to form geometry features for density prediction.A separate color pathway uses pooled view features and a nine-layer residual MLP to predict blending weights for input pixel colors.Source: B Implementation Details · pp. 20–21
  • With two input views, the method produces sharper reconstructions with fewer artifacts than prior work in the studio-capture results.The comparison is shown in Fig. B.1.Source: Figure B.1 · p. 21
  • For keypoint noise from 1 to 20 mm, rendered images can blur around keypoints such as the eyes at noise levels above 10 mm.The perturbation examples are shown in Fig. D.2.Source: Figure D.2 · p. 22

C Baseline Methods

  • Baseline comparisons use released MVSNeRF and IBRNet code with default parameters, a reimplementation of PVA, and public NHP results for ZJU-MoCap.The authors state that PVA code was not public.Source: C Baseline Methods · p. 22

D Additional Results

Additional experiments examine baseline renderings, keypoint-noise sensitivity, camera-calibration robustness, and feature-encoder choice. The reported findings include blurring at high keypoint noise, a small calibration drop, and consistent gains from HourGlass features.

D Additional Results

  • HourGlass consistently improves reconstruction quality over the U-Net encoder in studio and in-the-wild experiments.The comparisons are reported in Tables D.2 and D.3.Source: D Additional Results · p. 22, Table D.2 · p. 23, Table D.3 · p. 23
  • Noisier iPhone camera calibration produces a negligible performance drop of -0.04 PSNR and -0.5 SSIM.Table D.1 compares IBRNet, the method without spatial encoding, and the method with keypoint encoding.Source: D Additional Results · p. 22, Table D.1 · p. 23
  • Keypoint perturbations from 1 to 20 mm can blur rendered regions around keypoints, including eyes, when noise exceeds 10 mm.The noise-level examples are shown from 1 mm through 20 mm in Fig. D.2.Source: D Additional Results · p. 22
  • Among the compared studio-capture baselines, IBRNet produces blur and fog, PVA produces overly smooth renderings, and MVSNeRF struggles with widely spread input views.The qualitative comparisons are shown in Fig. B.1.Source: D Additional Results · p. 22

E Limitations and Future Work

The image-based rendering formulation makes color prediction sensitive to occlusions, and the method struggles with thin geometries and highly articulated motions. Future work includes integrating learnable 3D lifting with relative spatial encoding for end-to-end training.

E Limitations and Future Work

Loading 2205.04992v2…