PaperlayerView exact revision on arXiv

Source-linked AI summary

F$^{2}$-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories

Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, Wenping Wang

arXiv:2303.15951v1cs.CVcs.GR

TL;DR

Fast grid-based NeRF methods struggle with unbounded scenes and arbitrary camera trajectories because existing warping functions target specialized paths. F2-NeRF introduces perspective warping with adaptive space subdivision, achieving high-quality rendering across trajectory types while training in minutes.

  • Problem

    Existing grid-based NeRF methods are limited to bounded scenes, while NDC and inverse-sphere warping support only forward-facing or 360° object-centric trajectories.

  • Method

    F2-NeRF uses perspective warping based on projected image coordinates and PCA, with adaptive subdivision for foreground and background regions.

  • Results

    F2-NeRF renders high-quality images on forward-facing, 360° object-centric, and free-trajectory datasets, with Free-dataset training taking approximately 12 minutes on a 2080Ti GPU.

  • Takeaways & Limitations

    A single perspective-warping scheme enables fast grid-based NeRF training across arbitrary and specialized camera trajectories.

Abstract

from arXiv · show

This paper presents a novel grid-based NeRF called F2-NeRF (Fast-Free-NeRF) for novel view synthesis, which enables arbitrary input camera trajectories and only costs a few minutes for training. Existing fast grid-based NeRF training frameworks, like Instant-NGP, Plenoxels, DVGO, or TensoRF, are mainly designed for bounded scenes and rely on space warping to handle unbounded scenes. Existing two widely-used space-warping methods are only designed for the forward-facing trajectory or the 360-degree object-centric trajectory but cannot process arbitrary trajectories. In this paper, we delve deep into the mechanism of space warping to handle unbounded scenes. Based on our analysis, we further propose a novel space-warping method called perspective warping, which allows us to handle arbitrary trajectories in the grid-based NeRF framework. Extensive experiments demonstrate that F2-NeRF is able to use the same perspective warping to render high-quality images on two standard datasets and a new free trajectory dataset collected by us. Project page: https://totoro97.github.io/projects/f2-nerf.

1. Introduction

Fast grid-based NeRF methods train quickly but struggle with unbounded scenes and arbitrary camera trajectories because existing warping functions assume specialized trajectory patterns. F2-NeRF addresses this with perspective warping and adaptive subdivision, achieving high-quality rendering across diverse trajectories with minutes of training.

  • Motivation: Grid-based methods accelerate NeRF training to minutes, but their memory grows cubically with scene size and they process only bounded scenes without warping.Voxel pruning, tensor decomposition, and hash indexing reduce memory but do not remove the bounded-scene restriction in Euclidean grids.
  • Motivation: NDC warping supports forward-facing scenes, while inverse-sphere warping supports 360° object-centric scenes; neither handles arbitrary camera trajectories.Free trajectories can be long and include multiple foreground objects, making them particularly challenging.
  • Motivation: Free trajectories waste grid capacity because regularly tiled grids represent empty regions that are invisible to input views.This imbalanced spatial allocation contributes to degraded performance even when pruning, tensor decomposition, or hash indexing is used.
  • Contribution: F2-NeRF is a fast NeRF training method for large, unbounded scenes with free camera trajectories, built on Instant-NGP’s hash-grid representation.The method is designed to retain fast convergence while accommodating diverse trajectories.
  • Contribution: Perspective warping concatenates projected 2D coordinates, maps them into a compact 3D subspace with PCA, and adaptively assigns coarse background and fine foreground grids.The method generalizes NDC and inverse-sphere warping and can degenerate to them for their specialized trajectory settings.
  • Results: F2-NeRF renders high-quality images on forward-facing, 360° object-centric, and free-trajectory datasets using the same perspective warping, with Free-dataset training taking approximately 12 minutes on a 2080Ti GPU.The experiments compare against fast grid-based NeRF baselines across the three trajectory patterns.

2. Related Works

Prior work advances novel view synthesis through explicit scene representations, neural fields, and fast grid-based encodings. F2-NeRF extends this fast grid-based line toward flexible space warping and arbitrary camera trajectories.

  • Novel view synthesis: Novel view synthesis methods interpolate input images or reconstruct scenes with meshes, voxels, point clouds, depth maps, and multi-plane images.These approaches seek higher-quality synthesized views through explicit scene or image-based representations.
  • Neural scene representations: Neural scene representations support novel view synthesis, relighting, new-scene generalization, shape representation, and multiview reconstruction.Representations may use neural networks alone or hybrid parametric encodings with space subdivisions.
  • Neural scene representations: F2-NeRF subdivides scenes for flexible space warping and uses a hybrid neural representation.This positions the method within neural representations that combine learned components with spatial subdivision.
  • Fast NeRF training with space warping: Grid-based representations accelerate NeRF training by storing voxel values or interpolated features, but unbounded scenes require additional space warping.Plenoxels store density and color directly, while Instant-NGP, TensoRF, and DVGO use feature grids with tiny MLPs.
  • Large-scale Neural Radiance Fields: Large-scale NeRF methods decompose scenes into separately trained blocks, while F2-NeRF targets small-scale scenes and may serve as a backbone for one block.The paper distinguishes its scope from large-scale block-based reconstruction.

3. Our Approach

F2-NeRF builds a grid-based representation for unbounded scenes by warping space according to camera visibility and projected image coordinates. Its perspective warping supports arbitrary trajectories while enabling perspective sampling and shared hash-grid features.

  • Space subdivision: F2-NeRF subdivides an unbounded region into local subregions and constructs a perspective warping function from the cameras visible to each subregion.An octree stores the subdivided regions and supports retrieval of visible cameras.
  • Perspective warping: In the 2D analysis, grids aligned with projected camera coordinates align with camera rays, unlike axis-aligned grids in the original Euclidean space.The misalignment of the original grids becomes more severe farther from the cameras.
  • Perspective warping: Perspective warping represents a 3D point through its projections in input images, then uses PCA to map those coordinates into a compact 3D warp space.The PCA projection preserves sampled projection distances as much as possible.
  • Perspective warping: Perspective warping relates existing warps: it behaves like NDC warping for narrow forward-facing views and becomes more Euclidean as camera angles widen.Inverse-sphere warping corresponds to the Euclidean inner-sphere case and NDC-like outer-space behavior.
  • Grid representation: F2-NeRF uses one shared hash table while different local warping functions and hash functions represent features across subregions.Densities and colors are decoded from interpolated scene features using a tiny MLP.
  • Rendering: Perspective sampling uses the warping Jacobian to choose non-uniform Euclidean samples that are approximately uniform on images, improving sampling efficiency and convergence stability.The next sample distance is scaled by the Jacobian applied to the camera-ray direction.

4. Experiments

Experiments evaluate F2-NeRF on free, forward-facing, and 360-degree trajectories against fast and MLP-based NeRF baselines. F2-NeRF achieves strong Free-dataset quality and comparable results on specialized trajectories using one perspective-warping scheme.

  • Experimental Settings: The evaluation uses the Free, LLFF, and NeRF-360-V2 datasets, alongside voxel-, hash-grid-, and MLP-based NeRF baselines.The Free dataset contains narrow, long trajectories and focused foreground objects; training times are measured on a 2080Ti GPU.
  • Comparative Studies: F2-NeRF achieves the best rendering quality among fast-training NeRFs on the Free dataset.DVGO and Plenoxels produce blurred images, while Instant-NGP is sharper but remains unclear because of unbalanced scene-space organization.
  • Comparative Studies: F2-NeRF uses perspective warping on all datasets and achieves comparable results to other fast NeRF methods on LLFF and NeRF-360-V2.Baseline methods use NDC warping on LLFF and inverse sphere warping on NeRF-360-V2, whereas F2-NeRF uses the same perspective warping throughout.
  • Ablation Studies: Perspective warping improves performance substantially over inverse sphere warping in the Free-dataset ablation.Adding perspective sampling after perspective warping produces the best quantitative result in the ablation.
  • Comparative Studies: Perspective warping renders clearer images than inverse sphere warping in the visual comparison.The comparison directly evaluates the two warping techniques on rendered imagery.

5. Conclusion

The paper proposes perspective warping and F2-NeRF for high-quality novel-view synthesis with arbitrary camera trajectories in unbounded scenes. Experiments show this capability with training requiring only a few minutes, while the paper notes a possible use in misleading fake-image generation.

  • Conclusion: F2-NeRF combines perspective warping with a grid-based NeRF framework to handle arbitrary input camera trajectories in unbounded scenes.The method addresses trajectories not covered by NDC or inverse sphere warping.
  • Conclusion: F2-NeRF renders high-quality images with arbitrary trajectories while requiring only a few minutes of training.This is the paper's concluding experimental claim.
  • Conclusion: The authors identify misleading fake-image generation as a possible negative societal impact of F2-NeRF.This is stated as a potential use-related risk rather than an evaluated failure mode.

A. Additional Implementation Details

The supplementary details construct perspective-warping matrices from projected samples, normalize their axes through Jacobian constraints, adapt ray-marching steps, and define reconstruction and regularization losses.

  • Camera rectification: Camera rectification rotates camera views so selected cameras fully cover each subdivided region before perspective-warping computation.The strategy addresses cameras whose frustums intersect a region without fully covering it.
  • Principal component analysis: 323 uniformly sampled points are projected into selected cameras, and PCA on their concatenated coordinates supplies the first three projection directions.The covariance eigendecomposition produces M′ from the three eigenvectors with the largest eigenvalues.
  • Computing the axis length: Axis scaling forms M = SM′ so unit distances in warp space approximately align with image-space distances.The scale parameters are averaged across sampled points after constraining Jacobian columns.
  • Ray marching: Ray marching chooses original-space steps δ_i so warped samples are separated by target distance l, using the local Jacobian norm.The method sets δ_i = l/||J_i d|| through a linear approximation.
  • Loss functions: The training loss combines color reconstruction with disparity and total-variation regularization.Disparity discourages excessively large inverse depth, while total variation encourages neighboring octree-border points to have similar densities and colors.
  • Loss functions: Total variation samples 8192 octree-border points and penalizes differences between feature vectors fetched from neighboring nodes.The feature vectors are obtained through two hash functions conditioned on the neighboring octree nodes.

A.5. More implementation details

The implementation follows Instant-NGP’s compact hash-grid design, trains for 20k steps in roughly 10–15 minutes on a single GPU, and analyzes perspective warping’s relation to established warps.

  • Architecture details: F2-NeRF uses a 16-level hash table, 2-dimensional features per level, and small MLPs for scene, density, and rendering prediction.Fetched 32-dimensional hash features feed the network architecture.
  • Training details: Training uses Adam with scheduled learning rates, 256k point samples per batch, and 20k steps for every experimental scene.The learning rate warms up to 1 × 10^-1 and decays to 10^-2 with cosine scheduling.
  • Training details: Most scenes train in 10–15 minutes on a single Nvidia 2080Ti GPU.Training time depends on scene complexity.
  • Warp-coordinate analysis: Perspective warping uses projected coordinates from selected cameras as warp coordinates and can coincide with NDC or inverse-sphere warping under specific parameter choices.The equivalences hold with a constant offset when camera separation matches the stated NDC or sphere-radius conditions.
  • Additional experiments: Tables 5 and 6 compare longer-training results for Instant-NGP and F2-NeRF on the Free and NeRF-360-V2 datasets.The supplied passages identify the table scope but do not provide the table values.

C.1. Training for longer steps

Longer training improves both F2-NeRF and Instant-NGP, but on the Free dataset F2-NeRF reaches better rendering quality in fewer steps than a longer-trained Instant-NGP.

  • Training for longer steps: Longer training improves the rendering quality of both F2-NeRF and Instant-NGP on the Free and NeRF-360-V2 datasets.The comparison is reported in Tables 5 and 6.
  • Training for longer steps: 15 minutes and 50k steps of Instant-NGP on Free do not outperform F2-NeRF trained for 12 minutes and 20k steps.The passage reports this comparison without specifying the table’s metric values.
  • Compatibility with MLP-based NeRF: 250k-step MLP experiments show perspective warping and NDC outperform inverse-sphere warping in the forward-facing setting.Mean PSNR is 26.29 for perspective warping, 26.31 for NDC warping, and 26.02 for inverse-sphere warping.
  • Compatibility with MLP-based NeRF: Perspective warping produces more visual details than the compared methods in the LLFF Room result and is compatible with MLP-based NeRF.The qualitative comparison is shown in Fig. 14.
  • View extrapolation: In Lego view extrapolation, perspective warping achieves a result similar to using the original Euclidean space.Training uses elevation angles below 30° and tests the remaining views.

C.4. Additional ablations

Additional ablations examine hash-table sharing and regularization, showing that a global hash table preserves quality better while regularization suppresses foggy artifacts.

  • Single versus multiple hash tables: Using multiple hash tables with the same parameter budget clearly degrades quality compared with a single global hash table.The experiment assigns one hash table to each octree node.
  • Additional comparisons: Perspective warping is evaluated visually on the LLFF Room case and through extrapolated-view results summarized in the supplied figure and table captions.The captions identify visual comparison and PSNR-based view-extrapolation evaluation, but provide no outcomes by themselves.
  • Single versus multiple hash tables: A global hash table offers more flexibility in allocating representation capacity across different regions.This is given as the reason for its better quality in the ablation.
  • Effect of regularization losses: Removing regularization losses produces foggy artifacts and unclear rendering, especially in pure-color regions.The effect is reported in Fig. 17.

C.5. Per-scene results

Per-scene results are reported using PSNR across the Free, NeRF-360-V2, and LLFF datasets. On longer Free-dataset trajectories, perspective warping with perspective sampling performs relatively better than inverse-sphere warping.

  • C.5. Per-scene results: Per-scene results for the Free, NeRF-360-V2, and LLFF datasets are reported using PSNR.The results are provided in Tables 10, 12, and 11, respectively.
  • C.5. Per-scene results: Longer trajectories, including “stair” and “grass,” favor perspective warping with perspective sampling over inverse-sphere warping.This comparison is reported for the Free dataset.
  • C.5. Per-scene results: The Bonsai case includes a visual comparison without and with regularization losses on the NeRF-360-V2 dataset.

C.6. More visual results

The paper provides additional visual comparisons on the Free dataset and scene-breakdown tables for the Free, LLFF, and NeRF-360-V2 datasets. These materials complement the reported per-scene and visual evaluations.

  • C.6. More visual results: Additional visual comparisons are provided for the Free dataset.
  • C.6. More visual results: Table 9 presents ablation studies comparing warping and sampling methods.
  • C.6. More visual results: Scene-breakdown results are reported for the Free, LLFF, and NeRF-360-V2 datasets.The corresponding results appear in Tables 10, 11, and 12.
Loading 2303.15951v1…