Source-linked AI summary
DeepSFM: Structure From Motion Via Deep Bundle Adjustment
Xingkui Wei, Yinda Zhang, Zhuwen Li, Yanwei Fu, Xiangyang Xue
TL;DR
Deep learning for SfM has faced limitations in enforcing geometric structure and handling uncertain camera poses. DeepSFM uses paired cost volumes to iteratively refine depth and pose, and the paper reports state-of-the-art performance on both tasks across multiple datasets.
Problem
Deep SfM methods have limited explicit structural constraints, and cost-volume approaches often assume accurate camera poses.
Source: §1 · p. 1Method
DeepSFM uses depth- and pose-based cost volumes to iteratively refine depth and camera pose with photo-consistency and geometric-consistency constraints.
Source: §1 · p. 2Results
DeepSFM outperforms conventional bundle adjustment and state-of-the-art deep SfM methods on depth and pose estimation, with experiments reporting state-of-the-art performance across several datasets.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
The framework combines learnable priors with geometric rules in a deep-learning-based bundle-adjustment approach.
Source: §5 · p. 14DeepSFM assumes initial depth maps and camera poses are available, although they need not be accurate.
Source: §3 · p. 5
Abstract
from arXiv · showhide
Structure from motion (SfM) is an essential computer vision problem which has not been well handled by deep learning. One of the promising trends is to apply explicit structural constraint, e.g. 3D cost volume, into the network. However, existing methods usually assume accurate camera poses either from GT or other methods, which is unrealistic in practice. In this work, we design a physical driven architecture, namely DeepSFM, inspired by traditional Bundle Adjustment (BA), which consists of two cost volume based architectures for depth and pose estimation respectively, iteratively running to improve both. The explicit constraints on both depth (structure) and pose (motion), when combined with the learning components, bring the merit from both traditional BA and emerging deep learning technology. Extensive experiments on various datasets show that our model achieves the state-of-the-art performance on both depth and pose estimation with superior robustness against less number of inputs and the noise in initialization.
1 Introduction
Conventional SfM and many deep-learning approaches have limitations in handling correspondence, generalization, or camera-pose estimation. DeepSFM iteratively refines depth and camera pose with cost volumes, and the paper reports state-of-the-art performance across several datasets.
1 Introduction
- Conventional SfM depends on cross-view correspondences, limiting it when view coverage or overlap is inadequate and on textureless or non-Lambertian surfaces.Selecting suitable input views and scenes can require excessive care, even for experienced users.Source: §1 · p. 1
- Earlier deep methods directly regress structure and motion without explicit geometric constraints, while cost-volume methods commonly require known camera motion.Direct motion regression is also described as vulnerable to generalization problems.Source: §1 · p. 2
- DeepSFM iteratively improves depth and camera pose using cost volumes that measure photo-consistency and geometric-consistency.D-CV updates depth with current poses; P-CV updates poses with current depth estimates, and rough pose initialization is sufficient.Source: §1 · pp. 2–3
- BA-Net [40] integrates LM optimization but uses few unrolled iterations and a depth regressor; DeepSFM is presented as less prone to overfitting.The paper attributes BA-Net’s iteration limit to memory and computational inefficiency and says its direct depth regressor relies heavily on training-data priors.Source: §1 · p. 3
- DeepSFM outperforms state-of-the-art methods on DeMoN datasets, ScanNet, ETH3D, and Tanks and Temples.The authors report experiments across all four datasets.Source: §1 · p. 3
3 Architecture
DeepSFM alternates depth- and pose-based cost volumes to refine scene structure and camera motion from initial estimates that need not be accurate. Its architecture combines learned features with photometric and geometric consistency constraints.
3 Architecture
- DeepSFM starts from initial depth maps and camera poses, then iteratively rebuilds cost volumes from updated predictions to refine both.The initial estimates need not be accurate and can be obtained from direct regression methods.Source: §3 · pp. 4–5
3.1 2D Feature Extraction
- A shared 2D feature extractor uses a seven-layer CNN and spatial pyramid pooling to produce 32-channel features for both cost volumes.The pooling blocks use four scales: 4 × 4, 8 × 8, 16 × 16, and 32 × 32.Source: §3.1 · p. 5, §3 · p. 5
3.2 Depth based Cost Volume (D-CV)
- The D-CV combines target features, warped source features, and depth-consistency maps across uniformly sampled inverse-depth planes.Its geometric channels include warped initial depth maps and virtual depth planes transformed into the source-view coordinate system.Source: §3.2 · p. 6
3.3 Pose based Cost Volume (P-CV)
- The P-CV samples candidate camera poses around the initialization and scores warped source features and depth maps for photometric and geometric consistency.Rotation and translation are sampled separately while the other is held unchanged.Source: §3.3 · pp. 7–8
3.4 Cost Aggregation and Regression
- Depth costs are refined with a context network and regressed by soft-argmax; pose costs also use soft-argmax to estimate rotation and translation.Source: §3.4 · p. 8
3.5 Training
- Training uses supervised pose and depth losses, with DeMoN initial estimates and four iterations that reuse each iteration's predictions.Translation vectors and depth maps are scaled by the norm of the ground-truth camera translation to keep correct scale.Source: §3.5 · p. 8
4 Experiments
DeepSFM is evaluated on DeMoN, ScanNet, ETH3D, and Tanks and Temples, with comparisons covering accuracy, generalization, and robustness to initialization. The experiments report stronger depth and pose results than traditional baselines and DeMoN, with further analyses of iterative refinement, pose estimation, and performance with fewer views.
4.1 Datasets
- Across DeMoN, ETH3D, and Tanks and Temples evaluations, the datasets provide varied indoor, outdoor, synthetic, and real-world scenes, including laser-scanned ground truth.DeMoN-trained models are tested on MVS without training there, ETH3D depth maps are projected from laser-scanned points, and Tanks and Temples also supplies laser-scanned point clouds.Source: §4.1 · p. 9
4.2 Evaluation
- DeepSFM outperforms traditional baselines and DeMoN on depth and camera poses, and beats LS-Net and BA-Net on most metrics across four datasets.On RGB-D, results are comparable to the state of the art; the authors attribute this to higher noise in the ground-truth data.Source: §4.2 · pp. 9–10, Table 1 · p. 11
- On ETH3D, DeepSFM has the best overall performance; COLMAP does better on abs_diff, but outliers hurt its abs_rel and sq_rel results.The reported qualitative sequence also shows a good-quality point cloud from estimated depth and trajectories closer to ground truth than the baseline.Source: §4.2 · p. 11, Table 2 · p. 12, Figure 5 · p. 12
- As input quality falls, COLMAP and R-MVSNet degrade while DeepSFM maintains performance within a range; COLMAP fails when images are subsampled to 1/16.The Tanks and Temples evaluation adds noise by downscaling images and subsampling temporal frames.Source: §4.2 · pp. 11–12
4.3 Model Analysis
- With DeMoN initialization, four iterations reduce both depth and pose errors, with the process resembling coordinate descent.The authors report that improving depth and pose benefits each other by building more accurate cost volumes.Source: §4.3 · p. 13, Figure 7 · p. 13
- DeepSFM’s pose cost volume yields lower depth and pose errors than a direct visual-odometry pose baseline across iterations, including with noisy early depth.Both methods use the same depth prediction and DeMoN initialization.Source: §4.3 · p. 13
- With fewer ETH3D views, DeepSFM’s depth error is half of COLMAP’s, or it achieves similar error with half as many views.Both methods improve with more images, but DeepSFM’s performance drops more slowly as views decrease.Source: §4.3 · p. 14, Figure 8 · p. 14
5 Conclusions
DeepSFM embeds photometric consistency, geometric consistency, and camera-motion constraints in a deep SfM network through depth- and pose-based cost volumes. The authors report that it outperforms conventional BA and state-of-the-art deep-learning SfM methods.
5 Conclusions
- DeepSFM explicitly enforces photometric consistency, geometric consistency, and camera-motion constraints within a deep SfM network.Its two key components are the depth-based and pose-based cost volumes, D-CV and P-CV.Source: §5 · p. 14
- D-CV and P-CV measure photometric and geometric errors by hypothetically moving reconstructed scene points or the camera, respectively.Source: §5 · p. 14
- The authors report that DeepSFM outperforms conventional BA and state-of-the-art deep-learning methods for SfM.Source: §5 · p. 14
1 Implementation Details
Implementation uses paired target-source inputs and separate depth- and pose-based cost volumes, with specified training schedules, sampling ranges, and network components.
1 Implementation Details
- Training used 160k sequences, batch size 4, and 3 NVIDIA TITAN GPUs for 6 days, with two-image sequences and staged depth-map inputs.Pretrained 2D CNN feature-extraction weights were frozen for the first two epochs; ground-truth source depth was then replaced by predicted depth.Source: §1 · p. 18
- The loss weights were λr = 0.8, λt = 0.1, and λd = 0.1, with λ = 0.7 for balancing the depth loss.The authors set loss terms to similar gradient magnitudes and used a relatively large λr because rotation accuracy had greater impact on depth accuracy than vice versa.Source: §1 · p. 18
- The feature extractor maps 4W × 4H × 3 images to W × H × 32 feature maps for constructing D-CV and P-CV.Source: §1 · p. 18
- P-CV channels concatenate reference and warped source features with warped source depth and projected reference depth; D-CV channels use sampled depth planes and warped source features.The cost volumes contain P sampled camera poses and L sampled depth planes, respectively.Source: §1 · pp. 18–19
- The post-D-CV 3D convolution input has 66 channels, while the P-CV network adds three 3D average-pooling layers and global spatial pooling.Source: §1 · p. 19
2 Evaluation on ScanNet
On ScanNet, models trained on DeMoN significantly outperform prior methods in the reported evaluation, which the authors interpret as evidence of generalization to indoor environments.
2 Evaluation on ScanNet
- DeepSFM significantly outperforms all previous work on ScanNet in the evaluation of models trained on DeMoN.The evaluation uses 2,000 pairs filtered from 100 sequences; Table 3 reports results on ScanNet.Source: §2 · p. 19, Table 3 · p. 19
- The authors interpret the ScanNet results as indicating that DeepSFM generalizes well to general indoor environments.Source: §2 · p. 19
3 Evaluation on Tanks and Temples
The Tanks and Temples evaluation compares DeepSFM with COLMAP and R-MVSNet using all seven training sequences and average F-score. DeepSFM is reported to be more robust to initialization quality.
3 Evaluation on Tanks and Temples
- DeepSFM is reported to be more robust to initialization quality than COLMAP and R-MVSNet on Tanks and Temples.The evaluation adds noise to COLMAP poses by down-scaling images, subsampling temporal frames, or adding random Gaussian noise.Source: §3 · p. 20
- The Tanks and Temples evaluation uses all seven provided training sequences and calculates F-score as an average.Source: §3 · p. 20
4 Computational costs
This section reports computational costs on the DeMoN dataset. It identifies peak testing memory as the memory-cost measure.
4 Computational costs
- Memory cost is measured as peak memory usage during testing on a TiTAN X GPU.The passage does not specify the other computational-cost measures.Source: §4 · p. 20
- Computational costs are reported for DeMoN in Table 4.The supplied passage does not include the table’s values.Source: Table 4 · p. 20, §4 · p. 20
5 More Ablation Study
The ablations examine test-time iteration, depth-map warping, geometric consistency, initialization noise, depth smoothness, and pose sampling. The reported findings include convergence at six test iterations, a slight drop with bilinear warping, and no generalization benefit from initialization-noise augmentation.
5.1 More Iterations for Testing
- DeepSFM converges at the 6th test iteration after being evaluated with up to 20 iterations.Training uses up to four iterations, with each test iteration initialized from the previous depth maps and camera poses.Source: §5.1 · pp. 20–21
5.2 Bilinear Interpolation vs Nearest Neighbor Sampling
- Bilinear interpolation causes a slight performance drop; nearest-neighbor sampling better preserves geometric consistency at depth boundaries.The authors attribute potential side effects of bilinear interpolation to discontinuous depth values and smoothed boundaries.Source: §5.2 · p. 21, Table 6 · p. 21
5.3 Geometric consistency
- Geometric consistency is especially helpful in regions with weak photometric consistency, including textureless and specular-reflection regions.The ablation compares depth accuracy with and without geometric consistency using the same ground-truth poses.Source: §5.3 · p. 21, Table 7 · p. 21
5.4 Initialization data augmentation
- Gaussian-noise augmentation of DeMoN initializations slightly decreases performance on the unseen MVS dataset and does not improve generalization.The authors attribute this to the network overfitting the noise distribution.Source: §5.4 · p. 22, Table 8 · p. 22
5.5 Smoothness on depth map
- Adding scale-invariant gradient loss produces qualitative depth-map improvements in some samples but no significant improvement on depth evaluation metrics.The loss weight is set to 1.5 times the L1-loss weight, following DeMoN.Source: §5.5 · p. 22, Table 9 · p. 22
5.6 Pose sampling
- Varying rotation and translation bin sizes without retraining is used to assess pose-sampling generalization and robustness.The authors report that the physical-driven model shows good generalization ability.Source: Table 9 · p. 22, §5.6 · p. 23
6 Discussion with DeepV2D
DeepV2D and DeepSFM both update depth and camera motion alternately, but use different motion modules and geometric-consistency inputs.
6 Discussion with DeepV2D
- DeepV2D uses Gauss–Newton iterations to minimize photometric reprojection error, whereas DeepSFM uses P-CV and 3D convolution to learn correspondences from photometric and geometric features.DeepV2D’s depth cost volume is similar to DeepSFM’s except that DeepSFM introduces geometric consistency; the authors note that differing settings prevent a fair comparison.Source: §6 · p. 23
7 Visualization
The visualizations compare DeepSFM with DeMoN and COLMAP across depth maps and challenging scenes. The reported comparisons highlight scene details and completeness, while several figure captions identify comparisons without stating their outcomes.
7 Visualization
- DeepSFM reconstructs more scene details than DeMoN and produces more complete depth maps than COLMAP, whose outputs are often incomplete in textureless areas.The cited examples include plants, keyboards, and table legs in DeMoN datasets, and ETH3D results for COLMAP.Source: §7 · p. 23, Figure 14 · p. 27, Figure 15 · p. 28
- Figures 10 and 11 identify four components in D-CV or P-CV and 3D convolutional layers after P-CV.The captions identify these architectural details without further explanation.Source: Figure 10 · p. 25, Figure 11 · p. 26
- Figure 12 compares DeepSFM with COLMAP and R-MVSNet using noisy input and states that DeepSFM is less sensitive to initialization.Source: Figure 12 · p. 26
- Figure 13 presents qualitative depth comparisons with and without scale-invariant gradient loss, but the supplied caption does not state the comparison’s outcome.Source: Figure 13 · p. 27
- Figure 16 compares COLMAP and DeepSFM on textureless, poorly illuminated, reflective, and transparent scene materials.The caption names textureless ground and walls, poor illumination, and reflective or transparent glass.Source: Figure 16 · p. 29