Source-linked AI summary
BARF: Bundle-Adjusting Neural Radiance Fields
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, Simon Lucey
TL;DR
NeRF requires accurate camera poses, creating a joint reconstruction-and-registration problem when poses are imperfect or unknown. BARF connects NeRF to classical image alignment and applies coarse-to-fine bundle adjustment, enabling scene learning alongside recovery of substantial pose misalignment. The method learns scene representations from scratch and supports view synthesis and localization from unknown poses.
Problem
NeRF’s requirement for accurate camera poses leaves open how to jointly reconstruct 3D scenes and register cameras from imperfect or unknown poses.
Method
BARF uses view synthesis as a photometric bundle-adjustment objective, with coarse-to-fine registration motivated by classical image alignment and applied during NeRF training.
Results
BARF learns NeRF scene representations from scratch while resolving large camera pose misalignment at the same time.
Takeaways & Limitations
The approach supports view synthesis and localization of video sequences from unknown camera poses.
Takeaways & Limitations
BARF retains NeRF’s slow optimization and rendering, rigidity assumption, and sensitivity to dense 3D sampling, while also relying on heuristic coarse-to-fine scheduling.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) have recently gained a surge of interest within the computer vision community for its power to synthesize photorealistic novel views of real-world scenes. One limitation of NeRF, however, is its requirement of accurate camera poses to learn the scene representations. In this paper, we propose Bundle-Adjusting Neural Radiance Fields (BARF) for training NeRF from imperfect (or even unknown) camera poses -- the joint problem of learning neural 3D representations and registering camera frames. We establish a theoretical connection to classical image alignment and show that coarse-to-fine registration is also applicable to NeRF. Furthermore, we show that naïvely applying positional encoding in NeRF has a negative impact on registration with a synthesis-based objective. Experiments on synthetic and real-world data show that BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time. This enables view synthesis and localization of video sequences from unknown camera poses, opening up new avenues for visual localization systems (e.g. SLAM) and potential applications for dense 3D mapping and reconstruction.
1. Introduction
BARF addresses the joint problem of reconstructing 3D scenes and registering camera poses when NeRF’s usual accurate-pose prerequisite is unavailable. It connects NeRF training to classical image alignment and uses coarse-to-fine registration to learn representations from imperfect or unknown poses.
- Motivation: Joint reconstruction and camera registration form a chicken-and-egg problem because reconstruction needs known poses while localization needs reliable reconstructed correspondences.Classical SfM and SLAM can fall into suboptimal solutions when local registration is poor, while sparse point clouds limit dense geometric reasoning.
- Motivation: NeRF synthesizes high-fidelity novel views by representing scenes as continuous neural volume densities, but it requires accurate camera poses for the input images.The representation maps 3D points to color and volume density and uses differentiable volume rendering.
- Approach: BARF jointly optimizes scene reconstruction and camera registration from imperfect or unknown poses using view synthesis as a photometric bundle-adjustment objective.Unlike traditional bundle adjustment, BARF can learn the scene representation from randomly initialized network weights.
- Approach: Positional encoding improves high-frequency fitting but makes registration more susceptible to suboptimal results, motivating coarse-to-fine optimization.The paper establishes a theoretical connection between NeRF registration and classical image alignment.
- Outcome: BARF recovers scene representations from imperfect camera poses and supports view synthesis and localization of video sequences from unknown poses.The paper presents these capabilities as applications of the proposed joint registration and reconstruction framework.
2. Related Work
Related work spans explicit-geometry SfM and SLAM, neural view synthesis, and NeRF-based representations. BARF differs by learning a NeRF representation while recovering imperfect or unknown camera poses through coarse-to-fine bundle adjustment.
- Structure from Motion and SLAM: SfM and SLAM recover 3D structure and sensor poses using indirect keypoint methods or direct photometric consistency methods.Direct methods can use every pixel and integrate naturally with image-reconstruction losses.
- View Synthesis: View synthesis generates images from novel viewpoints and has used depth maps, proxy geometry, layered representations, and neural scene models.The task remains closely connected to 3D reconstruction because synthesized views depend on scene-aware intermediate representations.
- Neural Radiance Fields: NeRF models a continuous radiance field with a multilayer perceptron and differentiable volume rendering to synthesize novel views from posed images.Its simplicity and performance have motivated extensions for relighting, dynamic scenes, and single-image inference.
- BARF: Most NeRF-based view-synthesis methods require posed images, whereas BARF learns the scene representation while recovering imperfect camera poses.BARF also handles unknown poses in video sequences through coarse-to-fine bundle adjustment.
3. Approach
BARF connects neural scene reconstruction with classical image alignment by jointly optimizing NeRF representations and camera poses. It uses coarse-to-fine control of signal frequencies to make pose updates more coherent during registration.
- Planar Image Alignment (2D): Classical image alignment jointly minimizes photometric error through a warp, updating transformation parameters iteratively with gradient-based optimization.The warp maps image coordinates under parameters p, and updates follow p ← p + ∆p.
- Planar Image Alignment (2D): Coherent per-pixel predictions improve alignment, whereas complex signals and poor initialization can produce suboptimal registration.Classical coarse-to-fine methods address this by blurring images at earlier optimization stages.
- Planar Image Alignment (2D): Coordinate-based image networks replace numerical image gradients with the network Jacobian, enabling principled control of signal smoothness.This makes coarse-to-fine control generalizable from images to 3D scene representations.
- Neural Radiance Fields (3D): NeRF represents each 3D point with RGB color and volume density, renders pixel colors along sampled camera rays, and jointly optimizes scene and pose parameters.The rendering function is differentiable and deterministic, allowing photometric errors to backpropagate through camera transformations and the scene network.
- On Positional Encoding and Registration: Positional encoding improves high-frequency representation but can make registration gradients incoherent, causing updates to cancel across sampled 3D points.Its frequency-dependent gradients amplify signals by 2^kπ while changing direction, making naïve encoding a double-edged choice for joint registration and reconstruction.
- Bundle-Adjusting Neural Radiance Fields: BARF applies a smooth mask to positional-encoding bands, progressively increasing frequencies so registration proceeds from coarse to fine.The progress parameter α controls frequency weights; when a weight is zero, that frequency and all higher-frequency gradient contributions are removed.
4. Experiments
Experiments evaluate BARF from planar image alignment through synthetic and real-world NeRF scenes, testing whether coarse-to-fine optimization jointly recovers representations and camera poses from imperfect initialization.
- Planar image alignment: BARF recovers precise homography warps while optimizing the neural representation of an image from five perturbed patches initialized as center crops.The first patch warp is anchored to identity, and homographies are parametrized with the sl(3) Lie algebra.
- Planar image alignment: BARF combines accurate geometric alignment with high-fidelity image reconstruction, whereas full positional encoding misaligns and no encoding blurs reconstructions.The planar alignment experiment compares naïve full encoding, no encoding, and BARF.
- Synthetic NeRF scenes: On synthetic chair scenes, BARF realigns camera frames, while NeRF with naïve positional encoding remains stuck at suboptimal solutions.Camera poses are initialized with perturbations corresponding to 14.9° rotation and 0.26 translational standard deviations.
- Synthetic NeRF scenes: BARF achieves near-perfect synthetic-scene registration and view synthesis comparable to reference NeRF models trained with perfect camera poses.Table 2 reports less than 0.2° rotation error; translation errors are scaled by 100.
- Real-world NeRF scenes: On real-world LLFF scenes with identity-initialized cameras, BARF poses agree with SfM estimates and recover high-fidelity 3D representations.Naïve positional encoding diverges to incorrect camera poses and poor view synthesis.
5. Conclusion
BARF trains NeRF from imperfect camera poses by using coarse-to-fine registration, jointly learning scene representations and resolving substantial camera misalignment. The approach retains NeRF’s formulation but also retains important computational and modeling limitations.
- Conclusion: BARF establishes that coarse-to-fine registration is necessary for joint registration and reconstruction with coordinate-based scene representations.The conclusion connects this strategy to a theoretical relationship with classical image alignment.
- Conclusion: BARF can learn 3D scene representations from scratch while resolving large camera pose misalignment simultaneously.The fern visualization reports optimized poses that highly agree with SfM.
- Limitations: BARF retains NeRF’s slow optimization and rendering, rigidity assumption, and sensitivity to dense 3D sampling, while relying on heuristic coarse-to-fine scheduling.These limitations constrain efficiency, scene flexibility, sampling requirements, and scheduling principledness.
A. Visualizing the Basin of Attraction
The planar alignment experiment examines how positional encoding shapes registration’s basin of attraction. BARF widens this basin while preserving high-fidelity reconstruction through coarse-to-fine optimization.
- The experiment jointly optimizes an image network and translational warp across target offsets to measure alignment behavior.It searches the pixel grid until the target patch no longer overlaps the initialized center box.
- Naïve positional encoding creates a more nonlinear alignment landscape and a smaller basin of attraction.
- BARF widens the registration basin while reconstructing the image representation with high fidelity.The result supports coarse-to-fine registration for the 3D NeRF setting.
B. Additional NeRF Details & Results
This section provides further details and results from the paper’s NeRF experiments, particularly for real-world scenes.
- The section reports additional NeRF experiment details and results, with particular attention to real-world scenes.
B.1. Evaluation Details
Registration quality is evaluated after aligning optimized camera poses with reference poses under the inherent 3D similarity ambiguity. Procrustes analysis on camera locations supplies this alignment.
- Optimized scene and camera solutions are evaluated after pre-aligning them to reference poses with Procrustes analysis.References are ground-truth poses for synthetic objects and SfM-estimated poses for real-world scenes.
- The evaluation algorithm represents reference poses as camera extrinsic matrices and computes aligned pose parameters.
22 R = UV⊤∈R3×3
The evaluation aligns camera poses before measuring rotation error and visualizes the resulting trajectories. Real-world novel-view evaluation is complicated by baseline pose differences and requires coordinate transformation.
- Rotation error between SfM and aligned NeRF/BARF poses is measured using angular distance based on quaternion inner products.
- Figure 10 provides a detailed visualization of optimized camera poses for the LLFF dataset.
- Novel-view evaluation transforms test views into the optimized coordinate system using Procrustes-derived scale, rotation, and translation.Baseline NeRF’s large pose differences make its corresponding transformed test views far from plausible, so additional qualitative views are provided.
B.2. Real-World Scenes (LLFF Dataset)
The LLFF evaluation tests NeRF view synthesis and pose recovery on real-world forward-facing scenes with future frames held out. BARF remains competitive under a shared coarse-to-fine schedule, while depth parametrization substantially affects performance.
- Dataset: The LLFF dataset contains 8 forward-facing scenes captured as sequential RGB images with hand-held cameras.
- Dataset: Holding out the last 10% of frames evaluates prediction of future viewpoints from the first 90%, rather than interpolation between training views.
- Quantitative comparison: BARF and the baseline without positional encoding are competitive across different LLFF metrics, while full positional encoding performs poorly for unknown camera poses.
- Quantitative comparison: An averaged similar or better performance is achieved by BARF despite using a shared coarse-to-fine schedule, although optimal scheduling would be scene-dependent.Exhaustively searching for the best schedule is outside the paper’s scope.
- Depth parametrization: All compared methods perform better when 3D points are sampled in inverse-depth space than in regular metric-depth space.In regular depth space, the baseline with full positional encoding still performs poorly on all metrics.
- Qualitative results: BARF produces additional novel views by perturbing training-set camera viewpoints, but these qualitative views lack ground-truth RGB images for direct comparison.