Source-linked AI summary
ZipMVS: Multi-View Stereo with Compressed Cost Volumes
Guanglin Jin, Hongshan Yu, Javier Civera, Zhaoxin Li
TL;DR
MVS provides accurate dense 3D reconstruction but its cost-volume memory demands hinder deployment in resource-constrained settings. ZipMVS compresses cost volumes using a pixel-adaptive, exploratory depth-hypothesis strategy and lightweight components. On DTU and Tanks and Temples, it achieves competitive reconstruction quality with substantially lower memory than non-iterative baselines and a favorable trade-off against iterative methods.
Problem
The substantial memory requirements of MVS cost volumes limit deployment in embedded, real-time, aerospace, and autonomous-system settings.
Method
ZipMVS uses pixel-wise adaptive depth ranges, central-densely sampling, and a GRU-based Depth Speculator to generate fewer, structured depth hypotheses.
Results
ZipMVS achieves competitive reconstruction quality on DTU and Tanks and Temples while using substantially less memory than non-iterative baselines and offering a favorable trade-off against iterative methods.
Takeaways & Limitations
The results support cost-volume compression through depth-hypothesis sampling as an efficient route to high-fidelity MVS reconstruction.
Abstract
from arXiv · showhide
Multi-view stereo (MVS) methods typically deliver highly accurate 3D reconstructions from multiple registered RGB images, thanks to the highly informative, geometric constraints between them. However, their substantial memory requirements remain a major obstacle for deployment in domains such as aerospace and autonomous systems, where resource efficiency is critical. In this work, we introduce ZipMVS, an MVS method specifically designed for efficient high-quality reconstruction. We propose a novel depth-hypothesis strategy that enables substantial compression of the cost volume, hence greatly reducing GPU memory consumption while preserving reconstruction accuracy. Experiments on the DTU and Tanks and Temples datasets show that ZipMVS achieves competitive reconstruction quality compared with other efficiency-oriented MVS methods, while achieving a competitive balance between reconstruction quality and GPU memory usage. The code is available at https://github.com/JihnGlyn/ZipMVS
1 INTRODUCTION
ZipMVS targets the memory bottleneck of learning-based MVS by compressing cost volumes through fewer, deliberately generated depth hypotheses. Its adaptive and exploratory sampling strategy aims to reduce memory while preserving reconstruction fidelity.
- MVS recovers dense 3D scene geometry from registered RGB images, camera poses, and camera intrinsics.
- Learning-based MVS offers robustness and accuracy in challenging environments but has a substantial computational footprint that limits deployment on resource-constrained systems.
- 3D-convolutional plane-sweep cost volumes consume substantial memory, motivating compression through fewer depth hypotheses despite potentially narrower depth-range exploration.
- ZipMVS combines conservative coarse-to-fine refinement with exploratory neighboring-pixel alternatives to avoid exhaustive depth sweeping and local-minimum errors.
- The method introduces pixel-wise adaptive depth sampling with a differentiable scaling network and a GRU-based Depth Speculator for reliable hypotheses with fewer samples.
2 RELATED WORK
Related MVS research improves efficiency by simplifying architectures or reducing cost-volume spatial, depth, and channel dimensions. ZipMVS follows this efficiency direction by jointly adopting reductions in spatial, depth, and channel dimensions.
- Classical MVS methods include patch-based, surface-based, voxel-based, and depth map-based families, with the first three often hindered by complex optimization and computational demands.
- Depth map-based methods estimate per-view depth maps and fuse them into point clouds, balancing accuracy and efficiency while improving robustness through adaptive sampling and structural priors.
- Hypothesis-evaluation methods use planar priors, deformable regions, or semantic depth-edge aggregation to improve robustness in ambiguous or textureless regions.
- Learning-based MVS commonly warps multi-view features into cost volumes and regularizes them with 3D convolutions, whose memory overhead often forces reduced-resolution volumes.
- Efficiency strategies include sequential GRU processing, coarse-to-fine cascades, uncertainty-guided volumes, feature and cost-volume pyramids, PatchMatch structures, and lightweight probability estimators.
- The main reduction axes are network architecture, spatial dimensions H and W, depth dimension D, and channel dimension C; ZipMVS jointly adopts the latter three.
3 METHOD
ZipMVS builds a hierarchical MVS pipeline around compressed, adaptively generated depth hypotheses. Its DDR and GDS modules combine confidence-guided range sampling with spatially inferred alternatives before cost-volume construction and refinement.
- Architecture: Multi-view images pass through a feature pyramid, with depth estimation starting at level 3 and subsequent levels generating hypotheses from prior confidence and depth maps.The final depth map is refined at full resolution using the RGB reference image.
- Architecture: At the coarsest level, ZipMVS uniformly samples the full depth range; finer levels use DDR and GDS to generate fewer, structured hypotheses.The resulting hypothesis set combines DDR-generated depths with one speculative depth, yielding n + 1 candidates per pixel.
- Differentiable Depth Range: DDR predicts pixel-wise depth ranges from previous-level confidence, using narrower intervals for confident regions and wider intervals for uncertain regions.Its central-dense strategy places more samples near the interval center and progressively fewer toward the boundaries.
- Differentiable Depth Range: DDR samples n hypotheses in inverse-depth space around the upsampled previous-level depth, with a learnable range scale and ARNet-predicted quadratic spacing.The final depth is recovered by converting inverse depth back to depth within global bounds.
- GRU-based Depth Speculator: GDS processes depth cues sampled from neighboring pixels along 4 or 8 directions with a convolutional GRU to infer speculative depth hypotheses.The outermost sample initializes the sequence, and deformable convolution learns its offset before inward prediction.
4.1 Datasets
ZipMVS is evaluated on three public MVS datasets and an additional synthetic space-environment dataset. The evaluation includes indoor scenes, large-scale benchmarks, and a simulated International Space Station scenario.
- 4.2 Evaluation Datasets: The evaluation uses three public datasets plus an in-house synthetic dataset simulating space-application conditions.The public datasets include DTU and Tanks and Temples, while the additional dataset targets space-environment evaluation.
- 4.2 Evaluation Datasets: DTU contains 124 indoor environments captured under seven lighting conditions, with standard training, validation, and testing splits.Each environment includes multiple views and corresponding camera poses.
- 4.2 Evaluation Datasets: Tanks and Temples provides intermediate and advanced subsets evaluated through the official online point-cloud submission server.Performance is measured after submitting reconstructed point clouds to the benchmark server.
- 4.2 Evaluation Datasets: The synthetic space scenario places NASA’s International Space Station model at the coordinate origin and automates image acquisition and camera-parameter extraction.The setup uses Blender, a 53.70mm focal-length pinhole camera, and 900 × 1200 output images.
4.2 Implementation Details
The model is trained on resized DTU images using five views and a three-level pyramid with specified feature groups and depth hypotheses. Training uses Adam on one RTX 4090, while evaluation requires input dimensions compatible with repeated downsampling.
- 4.2 Implementation Details: Training uses DTU images resized to 512 × 640 with N = 5 views and feature groups of 8, 8, 8 across three pyramid levels.The depth-hypothesis counts are 4, 4, and 48 for levels l = 1, 2, 3.
- 4.2 Implementation Details: The depth-hypothesis counts are 4, 4, and 48 for pyramid levels l = 1, 2, 3, with uniform full-range discretization at level l = 3.The GDS starting radii are 6 and 4, with 8 and 4 sampling neighbors at the corresponding levels.
- 4.2 Implementation Details: The network is optimized for 24 epochs with Adam in PyTorch, using an initial learning rate of 0.001 and batch size 4 on one NVIDIA RTX 4090 GPU.The learning rate is reduced by a factor of 1.5 after epochs 12, 14, 16, 18, 20, and 22.
- 4.2 Implementation Details: Evaluation settings match training, and the three-level architecture requires input resolution dimensions divisible by 2^6 to avoid upsampling size mismatches.Each level downsamples feature maps by half, including the three-level U-Net cost-volume regularizer.
4.3 Experimental Results
ZipMVS delivers competitive reconstruction quality across DTU and Tanks and Temples while balancing accuracy with memory and runtime efficiency. Its qualitative results and synthetic evaluation further support high-quality reconstruction with reduced resource consumption.
- Results on DTU: 0.327mm overall error ranks second on DTU, reflecting a favorable balance between accuracy and completeness.ZipMVS records 0.369mm accuracy and 0.284mm completeness.
- Results on Tanks and Temples: 54.46 F-score on Tanks and Temples Intermediate is below CasMVSNet and IterMVS, while 33.03 on Advanced is competitive with IterMVS and exceeds PatchmatchNet.On Intermediate, ZipMVS favors completeness with the highest value, 74.79; on Advanced, recall reaches 42.38.
- Memory Usage and Runtime: ZipMVS uses 1322MB and 0.112s per inference versus IterMVS at 886MB and 0.093s, while running faster than PatchmatchNet.The comparison uses five input images, batch size 1, a single RTX 4090, and 1152×1600 resolution.
- Memory Usage and Runtime: ZipMVS is surpassed only by MVSFormer++ in the DTU quality-memory trade-off, while requiring over 3× less GPU memory.Points nearer the lower-left corner represent better reconstruction quality and memory trade-offs.
- Evaluation on Synthetic Data: 824MB and 0.073s per depth map enable ZipMVS to outperform PatchmatchNet qualitatively while using fewer resources on synthetic data.PatchmatchNet requires 1024MB and 0.078s per depth map.
4.4 Ablation Study
The ablation studies show that ZipMVS’s adaptive sampling components and input-view choice contribute to reconstruction quality and efficiency. GDS-derived hypotheses outperform compensated alternatives, while performance saturates at four views.
- Adaptive Range and Central Densely Modules: Removing either the AR or CD module, or both, reduces overall performance relative to the full ZipMVS model.All variants were trained for 24 epochs with identical hyperparameters.
- GRU-based Depth Speculator: GDS-derived depth hypotheses consistently outperform compensated hypotheses from preceding levels.The comparison replaces missing GDS hypotheses with estimates from the previous level to maintain a fair depth-hypothesis count.
- Number of Views: Performance saturates after N = 4 views, while additional views increase GPU memory consumption and inference time.Most competing approaches require five views to achieve comparable performance.
5 CONCLUSION
ZipMVS is an efficient MVS framework that reduces cost-volume size through pixel-adaptive depth sampling while preserving reconstruction fidelity. On DTU and Tanks and Temples, it achieves competitive accuracy with substantially lower memory than non-iterative baselines and a favorable trade-off against iterative methods.
- ZipMVS uses a differentiable depth-sampling strategy that adaptively refines each pixel’s search range and propagates depth hypotheses across levels.
- The pixel-adaptive mechanism substantially reduces cost-volume size while preserving high reconstruction fidelity.
- Experiments on DTU and Tanks and Temples show competitive accuracy with substantially less memory than non-iterative baselines.
- ZipMVS offers a favorable reconstruction-quality and memory trade-off compared with iterative methods.