PaperlayerView exact revision on arXiv

Source-linked AI summary

pixelNeRF: Neural Radiance Fields from One or Few Images

Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa

arXiv:2012.02190v3cs.CVcs.GRcs.LG

TL;DR

NeRF-based novel-view synthesis is costly because each scene must be optimized independently from many calibrated views. pixelNeRF conditions a NeRF on spatial image features, learns scene priors across scenes, and predicts views feed-forward from one or a few images. Across synthetic and real settings, the paper reports superior performance to current baselines, while retaining limitations in rendering efficiency and background handling.

  • Problem

    NeRF requires many posed images and lengthy per-scene optimization, limiting sparse-view reconstruction and preventing shared prior knowledge across scenes.

  • Method

    pixelNeRF uses fully convolutional spatial image conditioning to predict a continuous NeRF from one or a few images, trained across scenes with image supervision and no explicit 3D supervision.

  • Results

    pixelNeRF outperforms current state-of-the-art baselines across novel-view synthesis and single-image 3D reconstruction experiments spanning ShapeNet, multi-object scenes, and real datasets.

  • Takeaways & Limitations

    The learned scene prior enables feed-forward novel-view synthesis from sparse inputs, including single images and unseen object categories, without test-time optimization.

  • Takeaways & Limitations

    Like NeRF, pixelNeRF has slow rendering, runtime increases linearly with additional input views, and its rendering cannot be converted reliably to meshes.

Abstract

from arXiv · show

We propose pixelNeRF, a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. The existing approach for constructing neural radiance fields involves optimizing the representation to every scene independently, requiring many calibrated views and significant compute time. We take a step towards resolving these shortcomings by introducing an architecture that conditions a NeRF on image inputs in a fully convolutional manner. This allows the network to be trained across multiple scenes to learn a scene prior, enabling it to perform novel view synthesis in a feed-forward manner from a sparse set of views (as few as one). Leveraging the volume rendering approach of NeRF, our model can be trained directly from images with no explicit 3D supervision. We conduct extensive experiments on ShapeNet benchmarks for single image novel view synthesis tasks with held-out objects as well as entire unseen categories. We further demonstrate the flexibility of pixelNeRF by demonstrating it on multi-object ShapeNet scenes and real scenes from the DTU dataset. In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction. For the video and code, please visit the project website: https://alexyu.net/pixelnerf

1. Introduction

pixelNeRF addresses sparse-view novel view synthesis by predicting a NeRF from one or a few images in a feed-forward, learned framework. It uses image-conditioned neural rendering to learn scene priors across scenes and supports diverse synthetic and real-scene settings.

  • NeRF produces photorealistic novel views but requires many posed images and lengthy per-scene optimization.
  • pixelNeRF predicts NeRFs from one or several images in a feed-forward manner by conditioning the network on spatial image features aligned to pixels.The framework trains across multi-view images to learn scene priors for sparse-view synthesis.
  • The framework supports multi-view training without explicit 3D shape or object-mask supervision, predicts in the input camera coordinate system, and accepts variable numbers of posed views at test time without optimization.Its fully convolutional design preserves spatial alignment between image features and the output 3D representation.
  • Experiments cover single-image synthesis for category-specific and unseen-category settings, multi-object ShapeNet scenes, real car images, and DTU scenes.The authors report that pixelNeRF outperforms prior approaches on the multi-object benchmark and generates plausible novel views on DTU despite training on under 100 scenes.

2. Related Work

Prior novel-view and 3D-reconstruction methods either fit scenes independently, rely on restricted 2.5D or canonical representations, or require stronger supervision. pixelNeRF instead combines a learned volumetric scene prior with image-only supervision, view-space prediction, and spatially aligned features.

  • Learning-based 3D reconstruction: The proposed approach learns one- or few-view neural scene representations using only multi-view 2D image supervision, without requiring a consistent canonical space.
  • Novel View Synthesis: Scene-specific neural representations require many input views and substantial optimization time per scene.
  • Novel View Synthesis: Few-view methods using 2.5D representations limit the range of camera motions, whereas volumetric NeRF representations support synthesis from larger baselines.
  • Novel View Synthesis: pixelNeRF is fully feed-forward at test time and requires only relative camera poses, unlike a learned continuous 3D feature prior requiring further optimization with absolute poses.
  • Learning-based 3D reconstruction: Many learning-based reconstruction methods require ground-truth 3D models or object masks, while pixelNeRF can be trained from images alone and applied to scenes containing two objects.
  • Learning-based 3D reconstruction: Spatially aligned local image features preserve image details, addressing limitations of global features and complementing methods that require 3D supervision.
  • Viewer-centric 3D reconstruction: Canonical-space prediction can inhibit performance on unseen categories and multi-object scenes because those settings lack a well-defined canonical pose.

3. Background: NeRF

NeRF represents a scene as a continuous volumetric radiance field that maps 3D points and viewing directions to density and color. It renders camera rays through this field and optimizes image agreement, but does so independently for each scene without shared prior knowledge.

  • A NeRF encodes a scene as a continuous volumetric radiance field mapping a 3D point and viewing direction to differential density and RGB color.The field is represented by f(x, d) = (σ, c).
  • A target camera ray is parameterized as r(t) = o + td and evaluated between predefined near and far depth bounds.Rendering uses numerical quadrature over sampled points along each pixel ray.
  • The rendered pixel value for each target-view ray is compared with its ground-truth pixel value to define the NeRF rendering loss.
  • NeRF optimizes each scene independently using geometric consistency as its sole signal, so it cannot use shared world knowledge for acceleration or shape completion from single or extremely sparse views.

4. Image-conditioned NeRF

pixelNeRF conditions a NeRF on pixel-aligned image features, predicting density and color for query points in the input view’s camera space. With multiple views, it transforms queries into each view, processes them independently, and aggregates the resulting representations before rendering.

  • Single-Image pixelNeRF: pixelNeRF encodes each input image into a feature volume and conditions NeRF on the corresponding feature for each spatial query.The image encoder produces pixel-aligned features, which are sampled at projected query locations and supplied to the NeRF network.
  • Single-Image pixelNeRF: The single-view formulation represents positions and camera rays in the input image’s view space rather than a separate canonical coordinate system.Query coordinates and view directions are specified in the input view coordinate system before being processed by the network.
  • Single-Image pixelNeRF: The NeRF network outputs density and color from positional encodings, viewing direction, and sampled image features, with features incorporated residually at each layer.The output is then used in the volume-rendering pipeline to compare rendered pixels with target pixels.
  • Incorporating Multiple Views: Multiple input views are supported with only relative camera poses, and the formulation is independent of the world-space choice and input-view order.Each query point and viewing direction is transformed into every input view’s coordinate frame.
  • Incorporating Multiple Views: For multiple views, pixelNeRF processes coordinates and image features independently in each view frame, then aggregates the intermediate representations inside NeRF.The initial layers operate per view, while later layers process the aggregated views to predict density and color.

5. Experiments

The experiments evaluate pixelNeRF across ShapeNet reconstruction benchmarks, unseen categories, multi-object scenes, real car images, and DTU scenes. Results show strong reconstruction quality, including category-agnostic generalization, while retaining feed-forward inference and avoiding test-time optimization.

  • Experimental setup: Experiments compare pixelNeRF with SRN and DVR on category-specific and category-agnostic ShapeNet view-synthesis benchmarks.The evaluation also includes SoftRas in the category-agnostic setting and NeRF on DTU sparse-view scenes.
  • Experimental setup: PixelNeRF uses a feature pyramid image encoder and residual feature modulation to condition the NeRF network on spatially aligned image information.The encoder uses a ResNet34 backbone, while image features are injected at each ResNet block.
  • Category-specific reconstruction: PixelNeRF achieves superior results in category-specific reconstruction despite using feed-forward prediction, relative rather than absolute test-time poses, and view-space prediction.The benchmark covers one- and two-view reconstruction of ShapeNet chairs and cars using fixed novel-view evaluation protocols.
  • Category-specific reconstruction: Using local features instead of a global code and providing view directions each significantly improves category-specific ShapeNet chair reconstruction.These comparisons are reported as ablations for both single-view and two-view settings.
  • Category-agnostic reconstruction: In category-agnostic reconstruction, one model trained on the 13 largest ShapeNet categories recovers fine details and thin structures more effectively than strong baselines.The reported examples include monitor images and tabletop textures that global-latent baselines fail to preserve; Table 4 provides quantitative evaluations.
  • Generalization and transfer: The framework also performs on unseen categories, randomly placed multi-object scenes, real car images without fine-tuning, and held-out DTU scenes without test-time optimization.For DTU, training 60 separate NeRFs took 14 hours, whereas pixelNeRF applies immediately to new scenes.

6. Discussion

pixelNeRF learns scene priors for reconstructing NeRFs from one or a few images, while retaining several unresolved efficiency, scale, and data-coverage challenges.

  • Rendering remains slow, and runtime increases linearly with the number of input views.NeRF-based representations also cannot be converted to meshes reliably for subsequent fast rendering and manipulation.
  • PixelNeRF requires manually tuned ray-sampling bounds and positional-encoding scale, leaving scale-invariant NeRF methods as an open challenge.
  • DTU evaluation is constrained by controlled capture conditions, shared camera poses across scenes, and limited viewpoints.
  • Improving NeRF efficiency could enable real-time applications.

A. Additional Results

The supplemental section adds qualitative and quantitative results for several key experiments and directs readers to the video and website for richer animated results.

  • The section provides additional qualitative and quantitative results for several key experiments.The authors also point readers to the video and website for a richer, animated presentation.

A.1. Category-agnostic ShapeNet: Random Results

Additional results expand category-agnostic ShapeNet evaluation with random samples, category-level metric breakdowns, two-object scenes, and DTU comparisons against NeRF.

  • A.1. Category-agnostic ShapeNet: Random Results: Six random objects from each of the 13 largest ShapeNet categories are shown with baseline comparisons for category-agnostic view synthesis.Two random views among 24 available views serve as source and target views.
  • A.1. Category-agnostic ShapeNet: Random Results: Table 6 provides a detailed metric breakdown by category for generalization to unseen categories.
  • A.1. Category-agnostic ShapeNet: Random Results: The two-object dataset includes randomly sampled images and results when two input views are provided.The supplemental also reports performance as the number of inference views increases from one to three.
  • A.1. Category-agnostic ShapeNet: Random Results: Figure 16 reports quantitative results for each DTU scene and renders all test scenes omitted from the main paper.
  • A.1. Category-agnostic ShapeNet: Random Results: Table 8 reports DTU means and standard deviations for pixelNeRF and NeRF with 1, 3, 6, and 9 views.

B.1. Implementation Details

The supplemental implementation details describe the encoder, NeRF architecture, sampling and encoding choices, dataset settings, and aggregate DTU comparison protocol.

  • B.1. Implementation Details: The architecture was not extensively tuned because of high compute cost, so alternative configurations or smaller models may perform better.
  • B.1. Implementation Details: The image encoder uses ResNet34 feature maps, bilinear upsampling, and concatenation into a 512-channel pixel-aligned feature volume.For 64 × 64 inputs, the first pooling layer is skipped to preserve resolution.
  • B.1. Implementation Details: The NeRF network is a width-512 fully connected ResNet with five blocks and average pooling across source views after block 3.Coarse and fine networks share this architecture, while the encoder is not duplicated.
  • B.1. Implementation Details: Hierarchical sampling uses stratified uniform, importance, and near-surface fine samples to improve sampling efficiency.
  • B.1. Implementation Details: Positional encoding is applied to spatial coordinates but not viewing directions, with dataset-specific scaling and manually selected sampling bounds.
  • B.1. Implementation Details: The supplemental includes synthetic two-object samples and aggregate DTU metrics comparing pixelNeRF with NeRF.For DTU, higher PSNR and SSIM are better, whereas lower LPIPS is better; NeRF is optimized for 14 hours per scene and view count.

B.2. Experimental Details

This section specifies shared evaluation metrics, training settings, and supplemental dataset and architecture references used across experiments.

  • Experiments use PSNR, SSIM, and VGG-based LPIPS, with LPIPS inputs normalized to [−1, 1].
  • Training uses a learning rate of 10−4, batch size 4, and 128 rays.
  • Supplemental figures document additional DTU scenes and a split designed to prevent overlap between training and test scans.
  • The multi-view architecture applies positional encoding with six exponentially increasing frequencies and ReLU activations after each linear layer.
  • Table 9 provides abbreviations for ShapeNet categories.

B.2.1 Single-category ShapeNet

The experiments evaluate single- and few-view reconstruction across ShapeNet, multi-object scenes, real cars, and DTU, using specified training protocols and baselines.

  • Single-category experiments train pixelNeRF for 400000 iterations, using two views initially and then randomly switching between one and two views.Training took roughly 6 days on a single Titan RTX.
  • SRN comparisons use latent inversion for 170000 iterations in both one-view and two-view chair evaluations.
  • Category-agnostic evaluation uses a random input view shared across baselines, while testing held-out objects from the provided split.
  • Multi-object ShapeNet: The multi-object ShapeNet benchmark contains two randomly placed chairs, with 2715 training instances, 1101 test instances, and 20 or 50 rendered views per instance.
  • Multi-object ShapeNet: The two-object model is evaluated with one, two, and three input views after 1 million training iterations and 150000 latent-inversion iterations per setting.
  • Sim2Real on Real Car Images: Real-car inputs are foreground-normalized using PointRend masks, bounding boxes, translation, and rescaling before evaluation with the single-category car model.
  • DTU: DTU evaluation uses 1, 3, 6, or 9 informative input views, measuring image metrics on remaining views while excluding bad-exposure views from testing.
  • DTU: DTU comparison trains 60 NeRFs separately by scene and input-view count; five cases failed to converge with six or nine views.
Loading 2012.02190v3…