PaperlayerView exact revision on arXiv

Source-linked AI summary

BA-Net: Dense Bundle Adjustment Network

Chengzhou Tang, Ping Tan

arXiv:1806.04807v3cs.CV

TL;DR

SfM requires jointly recovering scene depth and camera motion, while dense per-pixel depth makes optimization computationally difficult. BA-Net addresses this with differentiable feature-metric BA and learned basis depth maps, and it outperforms conventional and recent deep-learning methods on real-data evaluations.

  • Problem

    Dense SfM must recover scene depth and camera motion, but direct per-pixel depth introduces too many optimization parameters and makes network training difficult.

  • Method

    BA-Net uses a differentiable feature-metric BA layer to jointly optimize depth and camera motion, while a network generates basis depth maps whose linear-combination weights are optimized.

  • Results

    BA-Net outperforms conventional BA and recent deep-learning methods, including supervised and unsupervised methods on KITTI.

  • Takeaways & Limitations

    The system combines hard-coded multi-view geometry with learned feature representations and basis-depth generation in an end-to-end trainable dense SfM pipeline.

Abstract

from arXiv · show

This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error. The whole pipeline is differentiable so that the network can learn suitable features that make the BA problem more tractable. Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth. The network first generates several basis depth maps according to the input image and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA. The basis depth maps generator is also learned via end-to-end training. The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem. Experiments on large scale real data prove the success of the proposed method.

1 INTRODUCTION

BA-Net embeds differentiable feature-metric bundle adjustment into a learnable SfM pipeline and uses learned basis depth maps to make dense depth optimization tractable. The approach jointly learns features and depth-map generation, and outperforms conventional and deep-learning baselines on ScanNet and KITTI.

  • BA-Layer: BA-Net formulates bundle adjustment as a differentiable layer that bridges classic optimization and deep learning.An MLP predicts the Levenberg-Marquardt damping factor, enabling back-propagation through the optimization.
  • Feature-metric BA: Feature-metric BA optimizes aligned CNN feature distances while jointly estimating scene structure and camera motion.The learned features are trained to make the SfM optimization more suitable.
  • Dense depth parameterization: Dense depth is represented as a linear combination of basis depth maps generated from each input image.The combination coefficients are optimized jointly with camera motion in the BA-Layer.
  • Dense depth parameterization: The basis-depth parameterization reduces unknowns, supports dense BA, and promotes smooth depth maps with good object-boundary consistency.The basis generator is learned through gradients back-propagated from the BA-Layer.
  • Novelty: The learned basis generator is trained end-to-end for SfM, unlike CodeSLAM’s separately learned generator and standalone optimization component.A shared encoder-decoder backbone supports joint feature and basis-depth learning.
  • Results: BA-Net outperforms DeMoN, LS-Net, and several conventional baselines on ScanNet and KITTI.The paper reports these evaluations on large-scale real datasets.

2 RELATED WORK

Prior work estimates monocular depth or SfM with CNNs, but BA-Net combines depth and camera estimation in a geometry-constrained, feature-metric framework. This design targets dense depth parameterization, multi-image reconstruction, and greater robustness than photometric objectives.

  • Monocular Depth Estimation Networks: Monocular depth estimation is ill-posed because infinitely many scenes can produce the same image.Earlier methods used MRFs, semantic segmentation, manually designed features, or CNN-based architectures.
  • Monocular Depth Estimation Networks: BA-Net uses a monocular depth network to generate basis depth maps, then improves the final depth through optimization.The network does not directly produce the final optimized depth map.
  • Structure-from-Motion Networks: Earlier SfM networks separately estimate depth and camera motion or apply differentiable direct methods after depth estimation.Examples include jointly trained photometric-loss networks and pipelines that compute camera motion after predicting depth.
  • Structure-from-Motion Networks: BA-Net simultaneously predicts scene depth and camera motion from CNN features while explicitly enforcing multi-view geometry constraints.The hard-coded constraints support reconstruction of more than two images.
  • Structure-from-Motion Networks: Feature-metric error is used instead of photometric error to enhance robustness.The paper contrasts this objective with prior methods that minimize photometric error.

3 BUNDLE ADJUSTMENT REVISITED

Classical geometric BA jointly optimizes camera poses and scene structure by minimizing reprojection error, but it uses limited matched features and suffers from outliers. Photometric BA uses aligned pixels instead, yet remains sensitive to initialization, appearance changes, and moving objects.

  • Geometric BA: Classical geometric BA jointly optimizes camera poses and 3D scene points by minimizing reprojection error.Levenberg-Marquardt is used to compute parameter updates during optimization.
  • Geometric BA: Geometric reprojection error measures the difference between a projected scene point and its corresponding feature point.The projection function maps scene points into image space.
  • Geometric BA: Geometric BA uses only information conforming to selected feature types and requires feature matching that can produce outliers.RANSAC-based outlier rejection cannot guarantee a correct result.
  • Photometric BA: Photometric BA eliminates feature matching by minimizing pixel-intensity differences between aligned pixels.It uses pixels with sufficient gradient magnitude and has shown strong performance in less textured scenes.
  • Photometric BA: Photometric BA is sensitive to initialization, camera exposure and white-balance changes, and outliers such as moving objects.The paper attributes initialization sensitivity to increased non-convexity from the photometric error.

4 THE BA-NET ARCHITECTURE

BA-Net combines learned feature representations, compact basis-depth parameterization, and a differentiable feature-metric BA layer to jointly optimize dense depth and camera poses. Its differentiable LM procedure enables end-to-end feature and depth-basis learning for multi-view geometry optimization.

  • Feature-Metric Bundle Adjustment: Feature-metric BA minimizes aligned CNN feature distances across multiple images, rather than geometric or photometric errors.The formulation retains scene-depth and camera-motion optimization while using feature pyramids as inputs.
  • Overall Architecture: The complete architecture combines a DRN-54 backbone, basis-depth generator, feature-pyramid constructor, and BA-Layer in an end-to-end pipeline.The backbone processes multiple images, while the generator produces basis maps for the reference image and the BA-Layer jointly optimizes depth and poses.
  • Feature Pyramid: The network constructs multi-scale feature pyramids whose learned features are smoother and support precise pixel alignment.The pyramid uses top-down context propagation, while its higher resolution facilitates alignment.
  • Differentiable BA-Layer: The BA-Layer makes LM optimization differentiable by fixing iteration structure and predicting the damping factor λ with an MLP instead of using discrete updates.Back-propagation can therefore pass through pose and depth optimization to the feature pyramids.
  • Basis Depth Maps: Dense depth is represented as a non-negative linear combination of 128 generated basis depth maps, reducing optimization variables while preserving smoothness and object boundaries.The basis matrix is fixed during BA, while its combination weights are optimized jointly with camera motion.

5 EVALUATION

BA-Net is evaluated on ScanNet and KITTI using depth and camera-pose metrics, with comparisons against learned and conventional SfM methods. It outperforms the reported baselines on both datasets.

  • ScanNet: ScanNet contains 1,513 sequences across 706 scenes, with metric-scale depth from a depth camera but imperfect poses and depths estimated by BundleFusion.
  • ScanNet: Training and testing use 1,413 and 100 ScanNet sequences, respectively, yielding 547,991 training pairs and 2,000 testing pairs after sampling.
  • KITTI: KITTI provides 61 street scenes across city, residential, and road categories, using the Eigen et al. split for fair comparison with prior methods.
  • Metrics: ScanNet evaluation measures depth errors and camera rotation, translation-direction, and absolute-position errors.
  • Results: BA-Net consistently outperforms DeMoN and conventional geometric and photometric BA on ScanNet, while geometric BA suffers from difficult indoor feature matching and photometric BA from non-convex optimization.
  • Results: On KITTI, BA-Net outperforms supervised and unsupervised methods and achieves more accurate camera trajectories than Zhou et al. and Wang et al.

6 CONCLUSIONS AND FUTURE WORKS

The paper presents BA-Net as a differentiable feature-metric bundle-adjustment network that jointly estimates scene depth and camera motion. Its learned features and basis-depth representation combine geometric constraints with end-to-end learning and outperform conventional BA and recent deep-learning methods.

  • BA-Net jointly optimizes scene depths and camera motion through feature-metric bundle adjustment in a differentiable, end-to-end trainable pipeline.
  • Dense depth is represented as a linear combination of basis depth maps generated by the network.
  • The system combines hard-coded multi-view geometry with learned feature representations and a learned basis-depth generator.
  • BA-Net outperforms conventional bundle adjustment and recent deep-learning-based methods.

APPENDIX A: IMPLEMENTATION DETAILS

The implementation uses a DRN-54 backbone, a basis-depth decoder, and a differentiable BA layer. Reconstructing two images takes 95.21 ms, while the BA layer remains the main computational bottleneck.

  • Figure 5 documents the DRN-54 backbone and basis-depth generator architectures.
  • The basis-depth generator uses five up-projection blocks to produce 128 half-resolution basis depth maps.
  • Evaluation Time: 95.21 ms are required to reconstruct two 320 × 240 images, compared with DeMoN’s 110 ms for two 256 × 192 images.
  • Evaluation Time: The BA-Layer is the current computation bottleneck because it performs many matrix operations, with direct CUDA implementation identified as a possible speedup.

APPENDIX B: ABLATION STUDIES

Ablations show that learned features, joint BA optimization, and differentiable Levenberg–Marquardt improve performance. Predicted damping factors outperform constant values because λ must adapt across data and iterations.

  • Learned Features vs Pre-trained Features: Pre-trained ImageNet features produce larger errors than the learned feature pyramid.
  • Bundle Adjustment Optimization vs SE(3) Pose Estimation: Without joint BA optimization, both depth maps and camera poses become less accurate than with joint optimization.
  • Differentiable Levenberg–Marquardt vs Gauss–Newton: Vanilla Gauss–Newton produces much larger error than differentiable Levenberg–Marquardt because the feature-metric objective is non-convex and convergence is not guaranteed.
  • Predicted vs Constant λ: Camera rotation and translation errors decrease as constant λ increases up to λ = 0.5, then increase.
  • Predicted vs Constant λ: Depth errors decrease with larger constant λ, potentially because smaller updates keep the final depth closer to the initialization.
  • Predicted vs Constant λ: Constant λ values consistently perform worse than MLP-predicted λ because no single λ is optimal across data and iterations.

APPENDIX C: EVALUATION ON DEMON DATASET

On the DeMoN dataset, the method generally outperforms DeMoN and improves scene-depth accuracy over LS-Net, while multi-view reconstruction benefits from additional views.

  • The appendix reports quantitative comparisons on the DeMoN dataset and multi-view reconstruction on ScanNet.
  • The method consistently outperforms DeMoN on camera motion and scene depth, except on the Scenes11 dataset.The paper attributes the exception to synthesized images with random ShapeNet objects and physically incorrect scale.
  • Compared with LS-Net, the method achieves similar camera-pose accuracy but better scene-depth accuracy.The paper presents this as evidence that feature-metric BA with learned features is superior to LS-Net’s photometric BA.
  • Accuracy consistently improves as more views are included in ScanNet multi-view reconstruction.The evaluation limits sequences to five views because of 12G GPU memory.

APPENDIX E: QUANTITATIVE COMPARISONS WITH CODESLAM

Against CodeSLAM, the method combines end-to-end learning with feature-metric BA and achieves substantially lower trajectory error, while its basis depth maps capture scene structure.

  • Unlike CodeSLAM’s separately learned depth auto-encoder and standalone photometric BA, the method learns features and basis-depth generation end-to-end through feature-metric BA.CodeSLAM uses a variational auto-encoder, whereas the method uses a standard encoder-decoder.
  • The quantitative comparison uses maximum and minimum error bars, with orange and blue boxes denoting CodeSLAM and the method’s median errors.
  • CodeSLAM’s trajectory error is roughly 1 m over 9 m traveled, compared with about 0.4 m for the method.The method’s median error is less than half of CodeSLAM’s error on EuroC MH02.
  • Visualization shows basis depth maps responding differently to close objects, far backgrounds, and smoothly varying scene layouts.The authors interpret these responses as evidence that the learned maps capture latent scene structures.

APPENDIX G: QUALITATIVE COMPARISONS WITH OTHER METHODS

Qualitative comparisons evaluate recovered depth maps against DeMoN, Wang et al., and Godard et al. across ScanNet and other datasets.

  • On ScanNet, the method recovers more shape details than DeMoN in regions highlighted by red circles.The paper states that this agrees with the quantitative results in Table 1.
  • Figure 11 compares recovered depth maps from the method, Wang et al., and Godard et al.
  • Figure 10 presents qualitative comparisons with DeMoN on DeMoN’s dataset.
Loading 1806.04807v3…