MVDepthNet: Real-time Multiview Depth Estimation Neural Network

1. Introduction

• Monocular cameras suit compact robots and allow variable observation baselines, while stereo requires a fixed baseline and careful calibration.Large baselines can triangulate distant objects, and short baselines can improve robustness to repeated patterns.Source: §1 · p. 1
• Monocular multiview estimation must search arbitrary epipolar lines, encode geometry dependent on pixel locations and camera parameters, and preserve geometric constraints during augmentation.Unconstrained pixel search expands the search space and can reduce accuracy; conventional image scaling and flipping cannot be directly applied.Source: §1 · p. 2
• A cost volume lets the network avoid pixel matching across images and work across camera types and image counts, enabling real-time depth estimation.Geometric augmentation transforms the cost volume and ground-truth depth together to maintain per-pixel consistency.Source: §1 · p. 2
• MVDepthNet correctly estimates objects difficult for RGB-D cameras and objects far from the camera in the illustrated indoor and outdoor examples.The examples highlight tiny structures and reflective surfaces, as well as distant objects.Source: §1 · p. 2

2. Related Work

• Related methods trade flexibility or speed: DeMoN uses two images and fixed intrinsics, DeepMVS is not real-time, and single-image prediction is dataset-limited and scale-ambiguous.Traditional multiview methods use handcrafted matching, aggregation, estimation, and refinement stages; MVDepthNet instead directly estimates depth from multiple posed images without post-processing and supports larger-FOV cameras.Source: §1 · p. 1, §2 · pp. 2–3

3. System Overview

• MVDepthNet converts input images into a cost volume whose elements record pixel observations across views at a given distance, then uses that volume and the reference image to estimate inverse depth.The inputs are multiple images with known camera intrinsics and poses.Source: §3 · p. 3
• The cost-volume approach is presented as adaptable to different camera configurations and real-time monocular depth estimation from a moving camera.The following sections apply the trained network in a system that continuously generates depth maps and compare its performance with other methods.Source: §3 · p. 3
• The paper applies the trained network in a monocular dense mapping system that continuously generates depth maps from a single localized camera.Its performance is compared with a variety of state-of-the-art methods.Source: §3 · p. 3

4. MVDepthNet

• MVDepthNet follows multiview stereo’s cost-volume-and-reference-image structure but learns depth estimation, aggregation, and refinement directly from data.Absolute-difference costs are described as efficient to calculate and extend across measurement images, while avoiding a supporting window preserves detail.Source: §4 · p. 3, §4.2 · pp. 3–4

4.1. Preliminaries

• Each reference pixel is projected using camera geometry, and its sampled-depth matching costs encode observations from measurement images.More measurement images make the matching cost more distinct and robust to occlusion.Source: §4.1 · p. 3, §4.2 · p. 4

4.2. Network Architecture

• The network combines a reference image and cost volume in an encoder-decoder, using skip connections and four-resolution inverse-depth outputs bounded by a scaled sigmoid.Skip connections preserve low-level per-pixel information while the decoder upsamples global and high-level information.Source: §4.2 · p. 4, Figure 3 · p. 4

4.3. Geometric Data Augmentation

• Geometric augmentation scales depth and camera translation by 0.5–1.5 to extend short-range training supervision, and transforms cost volumes, reference images, and ground-truth depths.The latter transformation uses random flips and scales between 1.0 and 1.2.Source: §4.3 · pp. 4–5

5. Monocular Depth Estimation System

• MVDepthNet is applied to image-pose sequences from a localized moving camera to continuously generate depth maps.For each input image, two measurement frames from the preceding sequence form the cost volume; the resulting inverse-depth map is inverted to obtain dense depth.Source: §5 · p. 5
• Measurement frames are selected when their view-angle difference or baseline length from the former measurement frame is sufficient.The view angle reflects overlap between frames.Source: §5 · p. 5

6.1. MVDepthNet

• Although the network supports multiple images, it is trained using two-image pairs because its inverse-depth predictions use the reference image and cost volume.The implementation uses PyTorch and Adam, with β1 = 0.9 and β2 = 0.999.Source: §6.1 · p. 5
• The loss sums average L1 errors between estimated and ground-truth inverse-depth maps at different resolutions.At each scale, only pixels with valid ground-truth depth measurements are counted.Source: §6.1 · p. 5
• Training runs for 1140k iterations with 64 sampled depth values and batch size 8; images are scaled to 320 × 256 without cropping.The learning rate starts at 1e-4 for 300k iterations and is halved when error plateaus afterward.Source: §6.1 · p. 5

6.2. Monocular Depth Estimation System

• The monocular system selects measurement frames at a view-angle change of 15 degrees or baseline translation of 0.3 meters.It estimates depth maps for an image sequence.Source: §6.2 · p. 5

7.1. Datasets

• Training uses 436,928 image pairs and testing uses 13,394 pairs selected from multiple open-source and synthetic datasets.Pair selection requires at least 70% reference-image visibility and average photometric consistency error below 80.Source: §7.1 · p. 6

7.2. Error Metrics

• The evaluation uses L1-rel, L1-inv, sc-inv, and correctly estimated depth percentage (C.P.).C.P. is the percentage of depth estimates with relative error within 10%.Source: §7.2 · p. 6

7.3. Ablation Studies

• MVDepthNet-64 halves L1-inv and doubles C.P. compared with MVDepthNet-0; even 16 sampled depths double C.P. against the no-cost-volume variant.The cost volume encodes multiview information; FPS drops mainly because of the time needed to construct it.Source: Table 2 · p. 6, §7.3 · p. 6, Figure 4 · p. 6
• With only the cost volume, MVDepthNet retains high-quality estimates and performance drops by 3% to 5%; the reference image helps smooth surfaces and locate edges.Cost-volume-only estimates also work well in textureless regions.Source: §7.3 · pp. 6–7, Table 3 · p. 7, Figure 5 · p. 7
• Geometric data augmentation improves depth estimation on test-image orientations absent from training, including vertical flips and ceiling scenes.The comparison is between MVDepthNet-64 and an otherwise similarly trained network without geometric augmentation.Source: Table 4 · p. 7, §7.3 · p. 7, Figure 6 · p. 7
• Using more than two images improves depth estimates, including in textureless regions and distant areas.The example processes up to six images, including the reference image.Source: §7.3 · p. 7, Figure 7 · p. 7

7.4. Comparison Using Image Pairs

• MVDepthNet outperforms other methods on most datasets and error metrics, while DeepMVS performs best on the MVS dataset.The comparison evaluates all pixels with ground-truth depth; DeMoN and DeepMVS are not tested on TUM RGB-D because some images overlap with their training data.Source: Table 5 · p. 8, §7.4 · p. 8

7.5. Comparison Using Image Sequences

• Our method is best in all real-world experiments, whereas VI-MEAN estimates depth most accurately on synthetic datasets.The evaluation uses six TUM RGB-D sequences and four ICL-NUIM sequences; the authors conclude their learning-based method is more suitable for real-world data with image and pose noise.Source: Table 6 · p. 8, §7.5 · p. 8

• MVDepthNet generates depth maps from multiview images and corresponding camera poses captured by a monocular camera.Source: §8 · p. 8
• Its cost volume encodes multiview observations so the network can find pixel correspondences and triangulate depth efficiently.Source: §8 · p. 8
• The method supports more than two images and differing camera intrinsics, uses geometric augmentation for limited datasets, and serves as the core solver in a continuous monocular dense-mapping system.Source: §8 · p. 9

1. Cost Volume Computation

• Perspective warping transforms reference-image pixels into a new image, allowing the cost volume to be calculated across all pixels at a sampled depth.Source: §1 · p. 11
• At sampled depth d, the reference pixel is projected into the measurement frame using the camera intrinsic matrix and relative camera pose.Source: §1 · p. 11
• The warped reference image is compared with the measurement image by computing their absolute difference.Source: §1 · p. 11

2. Datasets

• Many image pairs are used to train the network, and most real datasets, including SUN3D, TUM RGB-D, and SceneNN, are captured by RGB-D cameras.Source: §2 · p. 11
• Scenes11 provides perfect poses and larger-range depth maps but is not photorealistic; its depth range is cropped to 0.5 m to 50.0 m to match sampled-depth coverage.Source: §2 · p. 11

3.1. Model Saliency

• Saliency maps indicate that MVDepthNet uses information from large image regions and focuses on related objects when estimating a pixel's depth.Source: §3.1 · p. 11

3.2. Comparison Using Image Pairs

• MVDepthNet and DeMoN produce smooth, accurate depth maps using multiview observations, while FCRN performs well indoors but does not generalize to outdoor environments.Source: §3.2 · p. 11
• MVDepthNet uses camera poses to search and triangulate depth more precisely; DeMoN estimates depth with ego-motion and takes more time searching correspondences in 2D images.Source: §3.2 · pp. 11–12
• DeepMVS works well on the MVS dataset's high-quality images and poses but handles occlusion poorly, as in the chair sequence.Source: §3.2 · p. 12

3.3. Comparison Using Image Sequences

• MVDepthNet-based mapping outperforms the other methods on all TUM RGB-D sequences, while VI-MEAN performs best on ICL-NUIM in accuracy.The comparison evaluates only valid depth estimates for VI-MEAN and REMODE, but evaluates all pixels for MVDepthNet.Source: Table 2 · p. 12, §3.3 · p. 12