INeRF: Inverting Neural Radiance Fields for Pose Estimation

Lin Yen-Chen; Pete Florence; Jonathan T. Barron; Alberto Rodriguez; Phillip Isola; Tsung-Yi Lin

Source-linked AI summary

INeRF: Inverting Neural Radiance Fields for Pose Estimation

Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, Tsung-Yi Lin

arXiv:2012.05877v3Dec 10, 2020cs.CVcs.RO

TL;DR

iNeRF asks whether NeRF can enable mesh-free, RGB-only 6DoF pose estimation without object meshes. It inverts a NeRF by optimizing pose to reduce rendered–observed image residuals, and demonstrates pose estimation for scenes, objects, and unseen category instances while also improving NeRF training through estimated poses.

Problem
RGB-only 6DoF pose estimation commonly relies on 3D meshes, but obtaining suitable watertight models is difficult and some complex objects are poorly represented by mesh formats.
Method
Starting from an initial pose, iNeRF backpropagates photometric residuals between NeRF-rendered and observed pixels to optimize the camera pose, using sampled rays for efficiency.
Results
iNeRF estimates poses for complex scenes and objects without meshes, supports category-level estimation for unseen instances from a single-view inferred NeRF, and reduces required labeled images by 25% while maintaining reconstruction quality.
Takeaways & Limitations
NeRF inversion provides a mesh-free RGB route to both object pose estimation and camera localization, while estimated poses can supply additional supervision for NeRF training.
Takeaways & Limitations
Lighting and occlusion can severely affect performance, and approximately 20 seconds are required for 100 optimization steps, preventing real-time use.

Abstract

from arXiv · show

We present iNeRF, a framework that performs mesh-free pose estimation by "inverting" a Neural RadianceField (NeRF). NeRFs have been shown to be remarkably effective for the task of view synthesis - synthesizing photorealistic novel views of real-world scenes or objects. In this work, we investigate whether we can apply analysis-by-synthesis via NeRF for mesh-free, RGB-only 6DoF pose estimation - given an image, find the translation and rotation of a camera relative to a 3D object or scene. Our method assumes that no object mesh models are available during either training or test time. Starting from an initial pose estimate, we use gradient descent to minimize the residual between pixels rendered from a NeRF and pixels in an observed image. In our experiments, we first study 1) how to sample rays during pose refinement for iNeRF to collect informative gradients and 2) how different batch sizes of rays affect iNeRF on a synthetic dataset. We then show that for complex real-world scenes from the LLFF dataset, iNeRF can improve NeRF by estimating the camera poses of novel images and using these images as additional training data for NeRF. Finally, we show iNeRF can perform category-level object pose estimation, including object instances not seen during training, with RGB images by inverting a NeRF model inferred from a single view.

I. INTRODUCTION

iNeRF addresses RGB-only 6DoF pose estimation without mesh models by inverting a NeRF through differentiable rendering and pose refinement. It also uses selective ray sampling to reduce rendering demands and supports scene, object, and category-level pose estimation.

Differentiable-rendering pose methods typically require high-quality watertight meshes, which are difficult to obtain and unsuitable for some complex materials and shapes.NeRF offers an alternative representation for complex 3D and optical structures without mesh models during training or testing.
iNeRF estimates 6DoF pose from an observed RGB image, an initial pose, and a NeRF by minimizing differences between rendered and observed pixels.Residual gradients are backpropagated through the NeRF to update the pose until the images align.
Interest point-based ray sampling identifies informative pixels for pose optimization, enabling accurate estimation with two orders of magnitude fewer pixels than full-image sampling.The approach exploits NeRF’s ability to render individual rays and pixels rather than requiring complete-image rendering for every update.
iNeRF can estimate camera poses for additional images and use them as training data, reducing the number of required labeled images by 25% while maintaining reconstruction quality.The method thereby supports self-supervision of NeRF training through pose estimation.
The framework performs category-level pose estimation for unseen object instances using RGB images and a NeRF inferred from a single object view.This extends the method beyond pose estimation for scenes or object instances represented by optimized NeRFs.

II. RELATED WORKS

The paper builds on implicit neural 3D representations, NeRF view synthesis, and RGB pose-estimation methods while positioning iNeRF as a mesh-free inverse-rendering approach. NeRF’s continuous representation supports both optimized and predicted scene models for pose estimation.

Neural 3D shape representations: Implicit neural representations parameterize geometric or appearance properties of continuous 3D points with neural networks and can represent complex topologies at high resolution with low memory usage.Prior work includes signed-distance, occupancy, texture, and neural scene representations.
Pose Estimation from RGB Images: Classical and deep-learning RGB pose methods use known 3D models, keypoints, direct CNN predictions, or PnP-RANSAC, while differentiable mesh renderers also require object models.These model requirements motivate alternatives based on learned scene representations.
Pose Estimation from RGB Images: Because NeRF and iNeRF require posed images rather than object meshes, the same formulation can address object pose estimation and camera localization in challenging real-world scenes.The paper contrasts this broader scene-localization capability with prior category-level work.
Neural Radiance Fields: NeRF represents a scene with volumetric density and view-dependent color, enabling novel-view synthesis for unseen camera poses from continuous 3D coordinates and viewing directions.An MLP predicts density and color, which are converted to pixels through neural volume rendering.
Neural Radiance Fields: NeRF extensions can predict continuous scene representations from one or a few images, and iNeRF uses either optimized or predicted NeRF models for 6D pose estimation.This supports pose estimation without requiring a separately optimized NeRF for every test scene.

IV. INERF FORMULATION

iNeRF solves pose estimation as the inverse of NeRF training: with a fixed NeRF and known camera intrinsics, it optimizes the unknown camera pose to match an observed image.

Given a trained NeRF, known camera intrinsics, and an observed image with unknown pose, iNeRF recovers the camera transformation in the NeRF coordinate frame.The NeRF parameters remain fixed while the camera pose is treated as the optimization variable.
The method renders an image from an estimated pose, evaluates NeRF’s photometric loss, and updates the pose rather than the NeRF weights to minimize that loss.This reverses the usual NeRF optimization direction from representation parameters to camera pose.

A. Gradient-Based SE(3) Optimization

iNeRF performs gradient-based optimization on the SE(3) pose using exponential coordinates and iteratively differentiates the rendering loss through the fixed NeRF.

The estimated pose is parameterized with exponential coordinates so gradient-based updates remain on the SE(3) manifold.The pose is represented relative to an initial estimate using screw-axis coordinates.
Screw-axis coordinates encode the update using axis S and magnitude θ, with K(S, θ) defining the associated translational component.The formulation uses the skew-symmetric matrix of the rotational component within the SE(3) exponential map.
At each iteration, gradients of the loss with respect to the exponential-coordinate update are backpropagated through the MLP and optimized with Adam.The update begins near the initial pose by sampling the perturbation around zero.

B. Sampling Rays

iNeRF samples subsets of rays to reduce expensive NeRF rendering and obtain informative pose gradients. Interest-region sampling avoids uninformative background and aligned pixels, improving accuracy and convergence, especially with small ray batches.

Computational cost: NeRF requires O(bn) MLP forward/backward passes for b sampled rays and n sampled points per ray.Rendering all HW image pixels is expensive, motivating ray subsampling.
Batch size: With 2048 rays per gradient step, iNeRF fits within GPU memory and achieves 150× faster gradient steps on a 640 × 480 image.The batch corresponds to a single forward/backward pass within GPU memory.
Random Sampling: Random sampling often selects flat background pixels that provide little or no pose gradient, making it ineffective with small ray batches.Many sampled background pixels contribute no information for matching.
Interest Point Sampling: Interest-point sampling reduces stochasticity and accelerates convergence but can become trapped in local minima by considering only observed-image interest points.Already aligned points may provide little information for further refinement.
Interest Region Sampling: Interest-region sampling dilates masks around interest points to include nearby pixels and prevent the local minima associated with point-only sampling.A 5 × 5 morphological dilation is applied for I iterations; sufficiently large I approaches random sampling.

C. Self-Supervising NeRF with iNeRF

iNeRF can use estimated camera poses to incorporate unknown-pose images into NeRF training. The resulting procedure trains NeRF in a semi-supervised setting.

Pipeline: A NeRF is first trained from RGB images with known camera poses to obtain training parameters.These parameters initialize pose estimation for additional images.
Pipeline: iNeRF estimates poses for additional observed images whose camera poses are unknown.The estimated poses provide self-supervised pose labels.
Pipeline: The additional images and their estimated poses are added to the training set for NeRF.This expands supervision beyond the initially posed images.
Outcome: This feedback procedure allows NeRF to be trained in a semi-supervised setting.

V. RESULTS

Experiments cover synthetic NeRF scenes, real-world LLFF scenes, and self-supervised NeRF training. Larger ray batches and interest-region sampling improve synthetic pose optimization, while LLFF performance is lower and estimated poses improve NeRF training.

Evaluation scope: Experiments evaluate iNeRF on synthetic scenes and complex real-world LLFF scenes, studying ray batch size, sampling strategy, and pose-based NeRF augmentation.
A. Synthetic Dataset: On the synthetic dataset, larger ray batches improve pose accuracy and convergence speed, while interest-region sampling improves accuracy and efficiency at fixed batch size.Pose accuracy is measured by the percentage of predictions below 5° rotation or 5cm translation error.
B. LLFF Dataset: 71% to 55% of predictions remain below 5° rotation error, and 73% to 39% remain below 5cm translation error when comparing synthetic and LLFF results at batch size 1024.The authors suggest NDC space or scene-content differences may explain the dataset gap.
C. Self-Supervising NeRF with iNeRF: NeRF models retrained with additional images assigned poses by iNeRF perform better than models trained without those additional data.The Fern experiment uses 25%, 50%, or 100% of training data and reports PSNR.

C. ShapeNet-SRN Cars

For category-level pose estimation on ShapeNet cars, iNeRF aligns an unseen object instance using a NeRF predicted from one reference view. The method uses relative pose between images and outperforms a SuperGlue feature-based baseline with fewer outliers.

Method: At test time, pixelNeRF predicts a NeRF conditioned on one image, and iNeRF aligns a second image to estimate relative pose.
Results: iNeRF achieves lower rotation and translation errors than the SuperGlue feature-based baseline.The estimate is defined relative to a reference object view rather than a canonical pose.
Results: iNeRF produces 8.7% outliers compared with 33.3% for the baseline.Outliers are defined according to Table II.

D. Sim2Real Cars

The sim-to-real evaluation applies a pixelNeRF inferred from one real image to estimate relative poses for unseen toy cars, with qualitative tracking results and broader pose-optimization comparisons.

Evaluation: Table II reports mean and median translation and rotation errors, treating predictions with translation or rotation error above 20° as outliers.
Setting: The evaluation uses 10 unseen real-world cars and applies a ShapeNet-trained pixelNeRF without extra fine-tuning to infer an object NeRF from one real image.iNeRF then estimates the relative pose to a target real image.
Pose tracking: Qualitative tracking uses the previous frame to condition pixelNeRF, while iNeRF estimates each subsequent object pose and visualizes reconstructed frames.
Sampling and batching: Interest Region Sampling improves synthetic-dataset accuracy by 15% across batch sizes, while larger LLFF ray batches can improve accuracy and reduce required gradient steps.

VI. LIMITATIONS AND FUTURE WORK

iNeRF demonstrates mesh-free pose estimation and tracking for real and novel object instances, but lighting, occlusion, and runtime remain important limitations.

Limitations: Lighting and occlusion can severely affect iNeRF because the current formulation does not model them.
Limitations: The method takes around 20 seconds for 100 optimization steps, preventing practical real-time use in its current form.
Conclusion: The conclusion reports accurate gradient-based pose estimation across synthetic and real datasets, including category-level pose estimation and tracking for novel object instances.

APPENDIX

The appendix specifies optimization and loss details, and reports LLFF pose-error distributions together with the importance of sufficient pose-label initialization for NeRF training.

LineMOD loss: For LineMOD, rendered and observed images are transformed from RGB to YUV, and the Y channel is excluded from the loss.
Pose-error histogram: With batch size 2048, more than 70% of LLFF perturbation cases achieve below 5° rotation and 5 cm translation error after iNeRF optimization.The perturbations span [−40, 40] degrees in rotation and [−0.1, 0.1] meters along each translation axis.
NeRF supervision: Using only 10% of labeled Fern camera poses worsens PSNR from 18.5 to 15.64, indicating that enough labels are important for initialization.