PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Calibrating Neural Radiance Fields

Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Animashree Anandkumar, Minsu Cho, Jaesik Park

arXiv:2108.13826v2cs.CV

TL;DR

The paper addresses self-calibration for generic cameras when conventional methods rely on limited camera models and sparse geometric evidence. It jointly learns scene geometry and camera parameters with a differentiable NeRF framework, using photometric consistency and projected ray distance. The model learns intrinsics and poses from scratch without COLMAP initialization and improves NeRF-based rendering robustness when poses are given.

  • Problem

    Self-calibration methods are limited by simple camera models, sparse geometric correspondences, and the lack of jointly learned scene geometry.

  • Method

    The method jointly learns NeRF scene geometry and camera parameters using pinhole, radial-distortion, and generic nonlinear-distortion models with photometric and projected-ray geometric consistency.

  • Results

    The model learns geometry, intrinsics, and extrinsics from scratch without COLMAP initialization and improves NeRF and NeRF++ robustness when camera poses are given.

  • Takeaways & Limitations

    Differentiable generic camera modeling can be used as a modular addition to NeRF variants for self-calibration and improved novel-view synthesis.

  • Takeaways & Limitations

    Accurate camera learning depends on geometry quality, so the method sequentially learns geometry and a linear camera model before complex camera parameters.

Abstract

from arXiv · show

In this work, we propose a camera self-calibration algorithm for generic cameras with arbitrary non-linear distortions. We jointly learn the geometry of the scene and the accurate camera parameters without any calibration objects. Our camera model consists of a pinhole model, a fourth order radial distortion, and a generic noise model that can learn arbitrary non-linear camera distortions. While traditional self-calibration algorithms mostly rely on geometric constraints, we additionally incorporate photometric consistency. This requires learning the geometry of the scene, and we use Neural Radiance Fields (NeRF). We also propose a new geometric loss function, viz., projected ray distance loss, to incorporate geometric consistency for complex non-linear camera models. We validate our approach on standard real image datasets and demonstrate that our model can learn the camera intrinsics and extrinsics (pose) from scratch without COLMAP initialization. Also, we show that learning accurate camera models in a differentiable manner allows us to improve PSNR over baselines. Our module is an easy-to-use plugin that can be applied to NeRF variants to improve performance. The code and data are currently available at https://github.com/POSTECH-CVLab/SCNeRF.

1. Introduction

Self-calibration is needed when calibration objects are unavailable, but conventional methods are limited by linear camera models, sparse non-differentiable correspondences, and unlearned scene geometry. The proposed differentiable NeRF-based framework jointly learns generic camera parameters and geometry, adding photometric and projected-ray geometric consistency.

  • Calibration connects incoming rays to image pixels and supports applications including autonomous driving, robotics, and augmented reality.
  • Self-calibration estimates camera parameters from uncalibrated images without external objects such as checkerboards.Conventional calibration places objects with known geometry in the scene, which can be unavailable or interfere with perception tasks.
  • Traditional methods assume linear pinhole cameras, despite real lenses exhibiting complex nonlinear distortions that affect mapping accuracy.The linear model is computationally efficient but trades away accuracy.
  • Conventional self-calibration relies on sparse correspondences and geometric constraints, whereas photometric consistency supplies many physically based constraints.Photometric consistency forces the same 3D point to have the same color across valid viewpoints.
  • The proposed method jointly learns scene geometry and camera parameters end-to-end using Neural Radiance Fields and a generic model with radial and nonlinear distortion.The framework is differentiable, allowing improved geometry to refine camera parameters.
  • Projected ray distance provides geometric consistency for the camera model, while experiments learn intrinsics and extrinsics without COLMAP initialization and apply the model to NeRF variants.The model is tested on fish-eye images and is described as modular for NeRF and NeRF++.

3. Preliminary

This section introduces NeRF-based scene representations and the differentiable camera model used to learn rays, including pinhole, radial-distortion, and generic nonlinear components.

  • Neural Radiance Fields: NeRF represents scene transparency and view-dependent color implicitly, enabling volumetric rendering of rays through learned 3D geometry.Ray color is approximated by a weighted sum of colors sampled along the ray; accuracy depends on sampling choices.
  • NeRF++: NeRF++ separates foreground and background representations and reparameterizes background rays to bound otherwise unbounded outdoor scenes.This reduces the computational burden caused by the increasing sample count required for unbounded space.
  • Differentiable Camera Model: The camera model maps image pixels to 3D rays using differentiable pinhole unprojection followed by radial and generic nonlinear ray distortions.The model focuses on unprojection because projected ray distance requires pixel-to-ray conversion.
  • Camera Parameterization: The model uses bounded residual updates for intrinsics and a 6-vector representation to maintain valid rotations while optimizing camera parameters.The final intrinsic matrix is K = K0 + ∆K, while rotation columns are orthonormalized from the 6-vector representation.
  • Differentiable Camera Model: Pinhole rays are unprojected with K^-1 and transformed into world coordinates using rotation R and translation t.The resulting ray direction and origin are rd = R K^-1 p and ro = t, allowing gradients to reach camera-parameter residuals.
  • Distortion Models: Fourth-order radial distortion models lens warping, while locally interpolated ray residuals capture generic nonlinear optical aberrations.Direction and origin offsets are learned at discrete image locations and continuously extracted with bilinear interpolation.

5. Geometric and Photometric Consistency

The method combines photometric consistency from a differentiable NeRF with projected ray distance, a geometric loss designed for generic non-linear camera models. Projected ray distance directly compares corresponding rays, normalizes their separation through image-plane projection, and averages only valid distances.

  • Geometric Consistency: Generic camera models make conventional geometric losses unsuitable because epipolar distance assumes a pinhole camera and reprojection error requires non-differentiable 3D reconstruction.The proposed loss is designed for camera models with generic non-linear distortions.
  • Geometric Consistency: Projected ray distance directly measures the discrepancy between corresponding rays, which should intersect at the generating 3D point when cameras are calibrated.Camera-parameter misalignment is measured through the shortest distance between the rays.
  • Projected Ray Distance: The loss projects the shortest ray-segment endpoints onto image planes so correspondence distances are normalized across different camera distances.The projection function equalizes each correspondence's contribution regardless of its distance from the cameras.
  • Geometric Consistency: Projected ray distance differs from epipolar distance and reprojection error because it models non-linear distortions without an intermediate 3D reconstruction.It therefore supplies a direct geometric discrepancy between rays within the camera-learning framework.
  • Projected Ray Distance: Only ray configurations whose corresponding points lie in front of both cameras contribute to the averaged geometric loss.This avoids minimizing invalid distances when large distortion and a small camera baseline place the shortest segment behind a camera.
  • Photometric Consistency: Photometric consistency is learned through a neural radiance field that reconstructs 3D occupancy and color for differentiable volumetric rendering.The rendering loss compares observed pixel colors with colors rendered along their corresponding rays.

6. Optimizing Geometry and Camera

Geometry and camera parameters are optimized with NeRF and the camera model, but learning proceeds through a curriculum because unstable or coarse geometry can prevent accurate calibration. The procedure progressively adds camera parameters and periodically applies projected ray distance using image correspondences.

  • Joint Optimization: NeRF parameters and camera parameters are learned jointly to optimize scene geometry and camera calibration.Camera parameters determine the ray positions and directions used for NeRF learning.
  • Curriculum Learning: The optimization first learns geometry and a linear camera model, then introduces complex camera-model parameters because accurate calibration requires sufficiently good geometry.The paper states that unknown or overly coarse geometry makes accurate camera-parameter learning impossible.
  • Curriculum Learning: Training initializes focal lengths and focal centers at half the image width and height before learning coarse geometry.The camera curriculum adds subsets of parameters to reduce the complexity of jointly learning cameras and geometry.
  • Curriculum Learning: The algorithm progressively adds camera-model parameters through a curriculum-learning parameter schedule.The final procedure is presented as Algorithm 1.
  • Projected Ray Distance Training: At scheduled iterations, the method samples a target image, extracts correspondences, computes projected ray distance, and adds it to the training loss.Target images are heuristically selected within a maximum 30° from the source view.

7. Experiment

The experiments evaluate self-calibration without camera priors, improvement over NeRF and NeRF++, fish-eye reconstruction, and component ablations. Across these settings, the model learns camera information and generally improves rendering quality, including under nonlinear distortion.

  • 7.2. Self-Calibration: Training from scratch produces reliable rendering without calibrated camera information, with some scenes outperforming NeRF initialized using COLMAP.The evaluation uses identity rotation, zero translation, and image dimensions for focal-length initialization.
  • 7.3. Improvement over NeRF: With COLMAP initialization, the model consistently improves rendering quality over NeRF and yields much lower projected ray distance.The learned nonlinear distortion is visualized in Figure 4.
  • 7.4. Improvement over NeRF++: On Tanks and Temples, the model improves rendering qualities and reduces training projected ray distance relative to NeRF++.The comparison is reported for the Tanks and Temples dataset and visualized in Figure 6.
  • 7.5. Fish-eye Lens Reconstruction: For fish-eye scenes, the experiments compare the learned model with a NeRF++ baseline using camera parameters that include radial distortion.The baseline implementation is modified to incorporate fish-eye distortion during ray computation.
  • 7.6. Ablation Study: The ablation study evaluates learnable camera parameters, nonlinear distortion, and projected ray distance loss across curriculum-learning phases.The authors report clearer images with model extensions, while projected ray distance can increase for some scenes.

8. Conclusion

The paper concludes with an end-to-end method that jointly learns scene geometry and camera parameters using a model combining pinhole, radial, and nonlinear distortion. It reports learning from scratch and improved robustness for NeRF and NeRF++ when camera poses are provided.

  • 8. Conclusion: The proposed camera model combines a pinhole model, radial distortion, and nonlinear distortion to capture real lens noise.Projected ray distance is introduced to improve accuracy and enable fine-grained correspondences.
  • 8. Conclusion: The method jointly learns scene geometry and camera parameters end-to-end.The framework uses learned geometry to support self-calibration.
  • 8. Conclusion: The model learns geometry and camera parameters from scratch when poses are unavailable, and improves NeRF and NeRF++ robustness when poses are given.These conclusions are supported by the reported experiments and rendered-image comparisons.

Self-Calibrating Neural Radiance Fields: Supplementary Materials

The supplementary material reports a comparison between NeRF and the proposed method with camera information initialized by COLMAP.

  • Self-Calibrating Neural Radiance Fields: Supplementary Materials: The supplementary comparison reports PSNR, SSIM, LPIPS, and PRD for NeRF and the proposed method on the training dataset.The table concerns camera information initialized with COLMAP.

1. Implementation Details

Implementation uses curriculum learning and staged camera-parameter expansion, with separate training settings for NeRF and NeRF++. The projected ray distance loss is thresholded to remove distant correspondence pairs.

  • 1. Implementation Details: NeRF uses batches of 1024 rays, while NeRF++ uses batches of 512 rays, with an initial learning rate of 0.0005 for both.Learning rates decay exponentially, and learnable camera parameters are extended every 200K iterations.
  • 1. Implementation Details: Projected ray distance pairs above threshold η are removed to address large distance changes caused by small directional changes.The experiments set η to 5.0.

2. Calibration with COLMAP initialization

With learnable camera parameters initialized by COLMAP information, the model consistently improves rendering quality over NeRF on additional LLFF scenes.

  • The model shows a consistent improvement from NeRF when learnable camera parameters are initialized by COLMAP camera information.

3. Calibration without COLMAP initialization

Without COLMAP initialization, NeRF fails to render the additional LLFF scenes reliably, whereas the proposed model does.

  • NeRF fails to render the additional LLFF scenes reliably, whereas the proposed model does.
  • The experiments extend the ablation study to other scenes in the LLFF dataset and report quantitative results.

5. Qualitative Results

Qualitative results compare the proposed model with NeRF and NeRF++ across standard, fish-eye, and uncalibrated-camera settings, while visualizing learned distortion and ablation stages.

  • On the tanks and temples dataset, error maps compare NeRF++ with the proposed model, with PSNR shown for each subfigure.
  • The model produces clearer renderings than NeRF++ on the FishEyeNeRF dataset.
  • For the Trex scene, adding intrinsic/extrinsic learning, non-linear distortion, and projected ray distance loss raises PSNR from 26.5 to 32.0.
  • The ablation visualization uses colored error maps for NeRF, intrinsic/extrinsic learning, non-linear distortion, and projected ray distance loss.
  • Without calibrated camera information, the model shows performance comparable to NeRF trained with COLMAP camera information.
  • Across LLFF scenes, the trained camera model captures circular non-linear distortions as ray offset and ray direction distortions.
Loading 2108.13826v2…