GNeRF: GAN-based Neural Radiance Field without Posed Camera

Quan Meng; Anpei Chen; Haimin Luo; Minye Wu; Hao Su; Lan Xu; Xuming He; Jingyi Yu

Source-linked AI summary

GNeRF: GAN-based Neural Radiance Field without Posed Camera

Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, Jingyi Yu

arXiv:2103.15606v3Mar 29, 2021cs.CV

TL;DR

NeRF reconstruction generally depends on accurate camera poses, while existing pose-optimization methods remain constrained by initialization, trajectory, or scene setting. GNeRF jointly estimates poses and radiance fields from random initialization through adversarial coarse estimation and photometric refinement, achieving favorable results on challenging repeated-pattern and low-texture scenes.

Problem
Most NeRF methods require accurate camera poses, while prior pose-optimization methods are limited to settings with short trajectories, reasonable initialization, or roughly forward-facing scenes.
Method
GNeRF uses a two-phase end-to-end framework that first jointly estimates coarse poses and radiance fields with adversarial training, then refines them with photometric loss and iterative optimization.
Results
GNeRF demonstrates effectiveness across synthetic and natural scenes and performs favorably in scenes with repeated patterns or low textures.
Takeaways & Limitations
The framework supports pose and radiance-field reconstruction in complex scenes from randomly initialized camera poses.
Takeaways & Limitations
GNeRF requires a reasonable camera-pose sampling distribution that is not far from the true distribution for different datasets.

Abstract

from arXiv · show

We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field (NeRF) reconstruction for the complex scenarios with unknown and even randomly initialized camera poses. Recent NeRF-based advances have gained popularity for remarkable realistic novel view synthesis. However, most of them heavily rely on accurate camera poses estimation, while few recent methods can only optimize the unknown camera poses in roughly forward-facing scenes with relatively short camera trajectories and require rough camera poses initialization. Differently, our GNeRF only utilizes randomly initialized poses for complex outside-in scenarios. We propose a novel two-phases end-to-end framework. The first phase takes the use of GANs into the new realm for optimizing coarse camera poses and radiance fields jointly, while the second phase refines them with additional photometric loss. We overcome local minima using a hybrid and iterative optimization scheme. Extensive experiments on a variety of synthetic and natural scenes demonstrate the effectiveness of GNeRF. More impressively, our approach outperforms the baselines favorably in those scenes with repeated patterns or even low textures that are regarded as extremely challenging before.

1. Introduction

GNeRF addresses NeRF’s dependence on accurate camera poses by jointly estimating poses and radiance fields from random initialization in complex scenes. Its two-phase, hybrid optimization targets challenging conditions including repeated patterns and low textures.

1. Introduction: Most NeRF methods rely heavily on accurate camera poses, whose estimation becomes difficult with repeated patterns, varying lighting, or few keypoints.Pose-estimation uncertainty can also increase uncertainty during NeRF training.
1. Introduction: Recent pose-optimization methods require short camera trajectories and reasonable initialization, with NeRF−− additionally limited to roughly forward-facing scenes.iNeRF focuses on pose estimation and assumes a trained NeRF.
1. Introduction: GNeRF jointly estimates camera poses and neural radiance fields from randomly initialized poses in complex scenarios.The framework is designed for complex outside-in settings rather than requiring rough pose initialization.
1. Introduction: Its first phase uses adversarial training for coarse pose and radiance-field estimation, while the second jointly refines them with photometric loss.A hybrid and iterative optimization scheme is used to avoid local minima.
1. Introduction: The end-to-end differentiable pipeline performs well in scenes with repeated patterns, low textures, noise, and even texture-less gray-mask views.These conditions are presented as challenging for COLMAP-based methods.

2. Related Works

Prior work reduces dependence on known camera poses through perturbation, inversion, or joint optimization, but retains initialization, trajectory, data, or scene-complexity constraints. GNeRF instead targets complex real scenes with limited data through coarse generative modeling and photometric refinement.

2. Related Works: IDR, iNeRF, and NeRF−− reduce pose dependence but use ground-truth perturbations, trained-field inversion, or identity initialization with forward-facing restrictions.The compared methods differ in how camera poses are initialized and optimized.
2. Related Works: Traditional pose estimation commonly uses SfM to extract local descriptors, match correspondences, estimate candidate poses, and select hypotheses with RANSAC.This pipeline relies on local feature matching and geometric correspondence.
2. Related Works: Learning-based pose methods can regress viewpoints without tracking, but they require large labeled datasets or abundant training images.SSV estimates viewpoints self-supervisedly from unlabeled images but still needs abundant training images.
2. Related Works: 3D-aware image-synthesis methods combine differentiable rendering and implicit 3D representations, but can entangle latent variables and destroy multi-view consistency.GRAF, GIRAFFE, and pi-GAN focus on disentangled shape and appearance modification for generated objects.
2. Related Works: GNeRF handles complex real scenes with limited data by learning a coarse generative network and refining it with photometric constraints.This contrasts with related methods requiring abundant data and focusing on simplistic objects such as faces and cars.

3. Preliminary

The preliminary formulation represents camera poses with translation and continuous rotation embeddings and scenes with NeRF’s continuous volumetric radiance field. Standard NeRF learns a per-scene representation from calibrated images, poses, intrinsics, and scene bounds using photometric loss.

3. Preliminary: Camera translation is represented by a 3D Euclidean embedding t ∈R3, while rotation uses a continuous 6D embedding r ∈R6.The continuous rotation representation is chosen because quaternions and Euler angles are discontinuous and difficult for neural networks to learn.
3. Preliminary: The original rotation matrix is recovered from the 6D embedding through a Gram-Schmidt-like process, with the final column computed by a generalized cross product.The rotation vector is formed by dropping the last column of the rotation matrix.
3. Preliminary: NeRF represents a scene as a continuous volumetric radiance field whose MLP maps 3D location and viewing direction to emitted color and volume density.The inputs are x ∈R3 and d ∈[−π, π]2, and the outputs are c ∈R3 and σ.
3. Preliminary: For each scene, NeRF learns a separate representation from RGB images, corresponding camera poses and intrinsic parameters, scene bounds, and a differentiable volume renderer.The representation is optimized with a photometric loss.

4. Methods

GNeRF jointly learns radiance fields and unknown camera poses from uncalibrated images using adversarial pose-free estimation followed by photometric refinement. A hybrid, iterative strategy interleaves these phases to mitigate non-convex optimization and improve both estimates.

Overall framework: GNeRF treats camera poses as latent variables and jointly estimates them with a NeRF model from uncalibrated single-scene images.The method is designed for random pose initialization in complex scenarios.
Pose-free NeRF Estimation: Phase A uses a NeRF generator, discriminator, and inversion network to learn coarse radiance fields and camera poses from randomly sampled poses and image patches.The discriminator matches real and generated patch distributions, while the inversion network predicts poses from synthetic patches.
Pose-free NeRF Estimation: The inversion network is trained self-supervised on synthetic image patches paired with their sampled camera poses, then predicts pose estimates for real images after convergence.The generator's improving image quality supports applying the learned pose predictor to real patches.
NeRF Refinement: Phase B jointly refines the NeRF model and pose embeddings by minimizing photometric reconstruction error from the coarse initialization.The initial estimates are not fully accurate because of sparse patch sampling and limited inversion-network capacity, but they provide useful initialization.
Regularized Learning Strategy: The hybrid strategy interleaves phases A and B and combines photometric loss with an L2 pose-prediction penalty to regularize non-convex optimization.Training follows an iterative pattern such as A → AB ... AB → B, using inversion-network predictions as a constraint.
Implementation: Dynamic 16 × 16 patch sampling, progressively adjusted camera intrinsics, and shared hierarchical-sampling MLPs are used to improve training efficiency, stability, and pose-space alignment.The generator uses 64 coarse and 64 importance-sampled points, while RMSprop and Adam optimize the networks and poses.

5. Experiments

GNeRF is evaluated for novel-view synthesis, camera-pose estimation, rendering quality, component contributions, and optimization strategies across synthetic and real scenes. It performs especially well in challenging low-texture or repetitive scenes and benefits from its full iterative design.

Novel View Synthesis: GNeRF generates novel views on par with COLMAP-based NeRF while remaining more robust on challenging scenes where COLMAP-based NeRF fails.Evaluation uses PSNR, SSIM, and LPIPS on Synthetic-NeRF and DTU.
Novel View Synthesis: GNeRF outperforms C+n on challenging scenes such as scan48, scan104, lego, and drums, while achieving similar results on regular scenes.The challenging scenes lack enough keypoints for reliable pose estimation, which harms NeRF synthesis.
Rendering Quality: GNeRF produces more natural images than IDR, which shows more artifacts and fewer fine details in rendering-quality comparisons on DTU.Both methods are evaluated with jointly optimized camera extrinsics on 49 training images per scene using PSNR over the whole image and within the mask.
Camera Poses: GNeRF outperforms COLMAP on camera-pose accuracy for the drums and lego scenes but is less accurate on scenes with enough reliable keypoints.The comparison reports mean translation and rotation differences on the Synthetic-NeRF training set.
Ablation Study: The full combination of adversarial training, the inversion network, and photometric loss achieves the best performance in ablation studies.Removing adversarial loss or the inversion network prevents correct geometry, while removing photometric loss limits the model to coarse radiance fields.
Optimization Schemes: The iterative optimization pattern A, AB...AB, B achieves higher image quality and camera-pose accuracy than the two-stage pattern A, B.It also produces finer edge geometry and camera poses closer to ground truth, helping overcome local minima.

6. Discussion and Conclusion

GNeRF jointly reconstructs neural radiance fields and estimates completely unknown camera poses in complicated scenes through adversarial and photometric optimization. Its hybrid iterative framework performs well on repeated-pattern and low-texture scenes, while discussion identifies a remaining pose-distribution requirement and pose-accuracy gap.

Although initialization-independent, GNeRF requires a reasonable camera pose sampling distribution and remains less accurate than COLMAP when many reliable keypoints are available.
GNeRF reconstructs neural radiance fields and estimates camera poses when poses are completely unknown and scene conditions are complicated.
The first phase jointly optimizes 3D representation and camera poses with GAN-based training, while the second phase refines them using photometric loss.
Interleaving the two phases in a hybrid iterative scheme further refines results robustly.
Experiments demonstrate promising results on scenes with repeated patterns or low textures, previously regarded as extremely challenging.

A. Pose Distribution Analysis

The pose-distribution analysis evaluates how changing camera sampling parameters affects novel view synthesis quality on the Synthetic-NeRF chair scene.

The analysis varies camera sampling space by changing radius, elevation, azimuth, and lookat point individually.
Novel view synthesis quality is reported for the chair scene in the Synthetic-NeRF dataset under these altered sampling spaces.

B. Additional Results

The image-size analysis compares COLMAP registration using 400×400 and 800×800 input images on Synthetic-NeRF. COLMAP performs satisfactorily at the original size but struggles after downsampling.

108 Synthetic-NeRF training images are evaluated at 400×400 and 800×800 resolutions for COLMAP registration.
COLMAP works satisfactorily with 800×800 images but struggles to register as many downsampled images.
COLMAP-based NeRF results therefore use registered poses from images sized 800×800.

C. Applications

Additional applications test GNeRF on unposed masks, mismatched pose distributions, reduced image counts, and intense image noise. The method learns 3D representations from masks and remains effective under strong noise, while pose-distribution and image-count settings are explicitly analyzed.

3D Reconstruction from Unposed Masks: GNeRF learns a 3D representation and camera poses from unposed masks by jointly optimizing radiance fields and camera poses.
3D Reconstruction from Unposed Masks: The mask is treated as a one-channel image, rendered through volume rendering, and converted into a 3D representation with marching cubes.
Pose Distribution Analysis: Pose-distribution analysis changes radius, elevation, azimuth, and lookat point to measure novel view synthesis quality on the Synthetic-NeRF chair scene.
Additional Results: Image-count analysis reduces training images and compares GNeRF with COLMAP-based NeRF.
Image Noise Analysis: With Gaussian noise N(0, 0.52), COLMAP-based NeRF methods fail to estimate camera poses and learn radiance fields, whereas GNeRF still renders less-noisy novel views and depth maps.