Source-linked AI summary
Dense Hybrid Recurrent Multi-view Stereo Net with Dynamic Consistency Checking
Jianfeng Yan, Zizhuang Wei, Hongwei Yi, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, Yu-Wing Tai
TL;DR
Deep-learning MVS remains constrained by memory, resolution loss, and fixed depth-fusion rules. D2HC-RMVSNet combines dense-resolution DRENet features, HU-LSTM regularization, and dynamic consistency checking; it ranks first on Tanks and Temples, is competitive on DTU, and uses 19.4% of R-MVSNet memory.
Problem
Deep-learning MVS methods face memory limitations, resolution loss from downsampling, and heuristic fixed-parameter depth-map fusion that can reduce completeness.
Method
D2HC-RMVSNet uses DRENet for dense multi-scale features, HU-LSTM for recurrent 3D cost-volume regularization, and dynamic consistency checking for depth-map fusion.
Results
The method ranks 1st on Tanks and Temples, achieves competitive DTU performance, and uses 19.4% GPU memory of R-MVSNet.
Takeaways & Limitations
The proposed network and fusion strategy support dense point-cloud reconstruction with competitive benchmark performance and substantially lower memory consumption.
Takeaways & Limitations
Downsampled feature extraction in related methods may lose high-resolution information and influence reconstructed-result accuracy.
Abstract
from arXiv · showhide
In this paper, we propose an efficient and effective dense hybrid recurrent multi-view stereo net with dynamic consistency checking, namely $D^{2}$HC-RMVSNet, for accurate dense point cloud reconstruction. Our novel hybrid recurrent multi-view stereo net consists of two core modules: 1) a light DRENet (Dense Reception Expanded) module to extract dense feature maps of original size with multi-scale context information, 2) a HU-LSTM (Hybrid U-LSTM) to regularize 3D matching volume into predicted depth map, which efficiently aggregates different scale information by coupling LSTM and U-Net architecture. To further improve the accuracy and completeness of reconstructed point clouds, we leverage a dynamic consistency checking strategy instead of prefixed parameters and strategies widely adopted in existing methods for dense point cloud reconstruction. In doing so, we dynamically aggregate geometric consistency matching error among all the views. Our method ranks \textbf{$1^{st}$} on the complex outdoor \textsl{Tanks and Temples} benchmark over all the methods. Extensive experiments on the in-door DTU dataset show our method exhibits competitive performance to the state-of-the-art method while dramatically reduces memory consumption, which costs only $19.4\%$ of R-MVSNet memory consumption. The codebase is available at \hyperlink{https://github.com/yhw-yhw/D2HC-RMVSNet}{https://github.com/yhw-yhw/D2HC-RMVSNet}.
1 Introduction
The paper addresses memory, resolution, and depth-fusion limitations in deep-learning MVS by combining a lightweight hybrid network with dynamic consistency checking. It reports first-place Tanks and Temples performance, competitive DTU results, and substantially lower memory use.
- Motivation: Deep-learning MVS methods face memory limits, information loss from downsampling, and heuristic depth-map fusion criteria that can reduce completeness.These limitations motivate a method that preserves dense resolution while reducing resource demands and adapting fusion to the scene.
- Approach: D2HC-RMVSNet combines a lightweight feature extractor, a hybrid LSTM–U-Net regularizer, and dynamic consistency-based depth-map fusion.The architecture targets dense depth regression, while postprocessing aggregates matching consistency across neighboring views.
- Contributions: DRENet extracts dense feature maps for dense point-cloud reconstruction.The contribution is presented as a lightweight backbone designed for dense depth-map processing.
- Contributions: DHU-LSTM combines LSTM and U-Net to reduce memory cost while maintaining reconstruction accuracy.This hybrid architecture is designed to absorb the respective merits of both components.
- Contributions: Dynamic consistency checking filters depth estimates using matching consistency aggregated across views rather than fixed fusion parameters.The method is intended to retain accurate and reliable dense points in the final reconstruction.
- Results: 1st place on Tanks and Temples was reported, while DTU performance was competitive with the state-of-the-art method.The method also dramatically reduces memory consumption relative to the comparison context described by the authors.
2 Related Work
Related work shows that deep-learning MVS improves reconstruction but commonly loses resolution, consumes substantial memory, and relies on fixed consistency rules. The paper responds with dense-resolution feature extraction, hybrid recurrent regularization, and dynamic filtering.
- Deep-learning MVS: Deep-learning MVS methods commonly use downsampled backbone features, preventing direct output at input-image resolution and potentially losing high-resolution information.The cited discussion connects this design pattern with possible effects on reconstructed-result accuracy.
- Deep-learning MVS: Plane-sweep volumes are memory-consuming, while MVSNet and DPSNet construct 3D cost volumes through differentiable homography warping for end-to-end training.Other related methods use patch-wise matching confidence or coarse-to-fine processing.
- Depth fusion: Existing methods fuse depth maps using pre-defined consistency checks that may be non-robust across scenes and miss points observed by few images.This postprocessing follows geometric-consistency procedures inherited from non-learning MVS pipelines.
- Proposed response: DRENet outputs input-sized feature maps with large receptive fields, HU-LSTM reduces memory while maintaining reconstruction accuracy, and dynamic checking targets accurate complete point clouds.Together, these components address resolution, regularization, and fusion limitations identified in prior work.
3 Reconstruction Pipeline
The reconstruction pipeline estimates dense depth maps for each reference image from calibrated multi-view imagery, then filters and fuses them through dynamic consistency checking. The network processes warped multi-view features as 3D cost volumes with HU-LSTM.
- Depth estimation: Each calibrated multi-view image is treated as a reference and processed with neighboring images to regress its dense depth map.The pipeline assumes camera parameters calculated from Structure-from-Motion.
- Cost-volume construction: DRENet features undergo differentiable homography warping and mean square variance to form 3D cost volumes.This figure describes the conversion from multi-view 2D features to the matching representation used by the network.
- Depth regularization: HU-LSTM processes the 3D cost volume sequentially along the depth direction for training or depth prediction.The recurrent regularizer operates after cost-volume construction in the depicted architecture.
- Evaluation: The method evaluates DTU and Tanks and Temples, and extends reconstruction to large-scale aerial scenes in Blend-MVS.These evaluations are presented as tests of efficacy, practicality, and generalization.
4 Dense Hybrid Recurrent MVSNet
DH-RMVSNet combines dense, resolution-preserving feature extraction with hybrid recurrent cost-volume regularization to produce accurate depth maps efficiently. DRENet preserves multi-scale context without downsampling, while HU-LSTM combines LSTM and U-Net advantages for dense original-size processing.
- Network overview: DH-RMVSNet combines 3DCNN accuracy with recurrent-unit efficiency to generate dense depth maps and point-cloud reconstructions on large-scale datasets.The network processes 3D data accurately while sequential recurrent processing improves efficiency.
- Image Feature Extractor: DRENet concatenates features from dilated convolutions to aggregate multi-scale context while preserving the input resolution.Three dilated convolutional layers use dilation ratios 2, 3, and 4 to extract context without sacrificing resolution.
- Cost volume construction: The cost volume is built by differentiable homography warping between views and mean-square-variance aggregation.Extracted feature maps are warped across views and aggregated into one cost volume C.
- Hybrid Recurrent Regularization: HU-LSTM combines LSTMConvCells in a U-Net architecture to aggregate multi-scale context while processing dense original-size cost volumes efficiently.It addresses the trade-off between 3DCNN U-Net context aggregation and recurrent processing efficiency, using sequential LSTMConvCell propagation.
- Hybrid Recurrent Regularization: HU-LSTM costs 19.4% of the GPU memory used by R-MVSNet while retaining multi-scale aggregation for dense depth estimation.The regularized cost volume is converted into a probability volume through softmax for training and depth estimation.
- Training Loss: Training uses cross-entropy loss between the probability volume and ground-truth depth, while testing processes depth sequentially with winner-take-all selection.The loss is evaluated over valid pixels, and the full probability map need not be stored during testing.
5 Dynamic Consistency Checking
The method replaces fixed multi-view geometric consistency checks with a dynamic strategy that aggregates matching consistency across neighbor views. It filters pixel-wise outliers to retain more accurate and complete depth values for point-cloud fusion.
- Motivation: Previous MVS methods use fixed geometric-consistency parameters, which are not robust across different scenes and may discard points seen by few images.The paper identifies fixed consistency checking as a limitation of prior depth-map fusion methods.
- Dynamic consistency criterion: The dynamic consistency check treats a depth as reliable when reprojection error is very low in few views or lower across a majority of views.This criterion explicitly accounts for both reprojection error and the number of supporting views.
- Geometric matching: For each reference pixel, the method back-projects its depth into 3D, projects it into a neighbor view, and reprojects the neighbor depth back to the reference image.The procedure compares the original pixel and depth with the reprojected pixel and depth.
- Consistency aggregation: Dynamic matching consistency combines reprojection errors between two views and aggregates them across all neighbor views into global consistency Cgeo(p).A weighting parameter λ balances the two reprojection-error metrics before multi-view aggregation.
- Filtering and reconstruction: Pixels with Cgeo(p) < τ are filtered as outliers, yielding depth maps with more accurate and complete values than fixed-threshold filtering.The resulting filtering improves the robustness, completeness, and accuracy of reconstructed point clouds.
6 Experiments
Experiments evaluate D2HC-RMVSNet on DTU, Tanks and Temples, and BlendedMVS, alongside ablations of its architecture and resource use. The method achieves competitive or leading reconstruction quality, scales to challenging scenes, and benefits from its hybrid modules and dynamic consistency checking.
- Experimental Method: The method combines dense original-size feature extraction, hybrid recurrent cost-volume regularization, and dynamic geometric consistency checking for point-cloud filtering and fusion.Depths are first filtered by probability and then by the proposed dynamic global geometric consistency algorithm before 3D fusion.
- DTU Dataset: D2HC-RMVSNet achieves competitive reconstruction performance on DTU, with the best completeness and competitive overall quality among evaluated methods.It improves accuracy and completeness over MVSNet and R-MVSNet.
- Tanks and Temples Benchmark: 1st on Tanks and Temples, D2HC-RMVSNet outperforms all existing methods without fine-tuning and demonstrates strong generalization to complex outdoor scenes.Its mean f-score increases from 50.55 to 59.20 compared with Dense R-MVSNet.
- BlendedMVS: D2HC-RMVSNet reconstructs large-scale scenes and small cars in BlendedMVS, while R-MVSNet fails on the same validation setting without fine-tuning.Original-size depth estimation, DRENet, HU-LSTM, and dynamic consistency checking support dense reconstruction without specific adjustment.
- Ablation Study: The ablation study shows HU-LSTM substantially improves accuracy over DRENet with 3D GRU, while LSTM is more accurate and robust than GRU.DRENet also slightly improves accuracy while reducing inference time and memory relative to 2DCNNFeatNet.
7 Discussion
D2HC-RMVSNet scales dense reconstruction to larger scenes and higher-resolution inputs while using substantially less memory than R-MVSNet.
- 6.6GB processes multi-view images at 1600 × 1200 resolution, supporting practical dense point cloud reconstruction.
- 19.4% memory consumption of R-MVSNet is achieved while generating depth maps at the input image size.
- 2.6s per view adds little inference time relative to R-MVSNet, which requires an additional 6.2s refinement stage.
- The method extends to aerial photos and large architectural scenes, producing denser, more accurate, and more complete point clouds.
8 Conclusions
The paper presents D2HC-RMVSNet for dense, accurate point cloud reconstruction, combining hybrid recurrent modeling with dynamic consistency checking. It reports first place on Tanks and Temples, competitive DTU performance, and sharply reduced memory use.
- D2HC-RMVSNet combines DRENet feature extraction, HU-LSTM recurrent regularization, and dynamic consistency checking for dense point cloud reconstruction.
- 1st on Tanks and Temples is reported alongside competitive DTU results and 19.4% of R-MVSNet memory consumption.