Source-linked AI summary
BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, Long Quan
TL;DR
Learning-based MVS is constrained by limited training data and therefore struggles to generalize to unseen scenes. BlendedMVS reconstructs textured meshes, renders aligned color images and depth maps, and blends rendered images with inputs to preserve lighting; experiments report significantly better generalization than other MVS datasets.
Problem
MVS training data is limited because obtaining large-scale ground-truth 3D structures requires expensive active scanners and labor-intensive processing, restricting generalization to unseen scenes.
Method
BlendedMVS reconstructs textured meshes from images, renders them into color images and depth maps, and blends rendered images with input images to generate training inputs.
Results
Models trained on BlendedMVS achieve significantly better generalization ability than models trained on other MVS datasets.
Takeaways & Limitations
BlendedMVS provides more than 17,000 high-quality training images covering varied scenes for learning-based multi-view depth estimation.
Takeaways & Limitations
MegaDepth training can suffer from misalignment between input images and reconstructed depth maps and from lower-quality crowdsourced images.
Abstract
from arXiv · showhide
While deep learning has recently achieved great success on multi-view stereo (MVS), limited training data makes the trained model hard to be generalized to unseen scenarios. Compared with other computer vision tasks, it is rather difficult to collect a large-scale MVS dataset as it requires expensive active scanners and labor-intensive process to obtain ground truth 3D structures. In this paper, we introduce BlendedMVS, a novel large-scale dataset, to provide sufficient training ground truth for learning-based MVS. To create the dataset, we apply a 3D reconstruction pipeline to recover high-quality textured meshes from images of well-selected scenes. Then, we render these mesh models to color images and depth maps. To introduce the ambient lighting information during training, the rendered color images are further blended with the input images to generate the training input. Our dataset contains over 17k high-resolution images covering a variety of scenes, including cities, architectures, sculptures and small objects. Extensive experiments demonstrate that BlendedMVS endows the trained model with significantly better generalization ability compared with other MVS datasets. The dataset and pretrained models are available at \url{https://github.com/YoYo000/BlendedMVS}.
1. Introduction
BlendedMVS addresses limited and narrow MVS training data by generating aligned rendered images and depth maps from textured meshes, while preserving ambient lighting. Its diverse dataset improves trained models' generalization across scenes.
- Existing learning-based MVS models trained mainly on DTU generalize poorly because DTU contains small objects captured along a fixed camera trajectory.
- BlendedMVS generates training images and depth maps by rendering textured 3D models reconstructed from input images instead of using expensive active scanners.The pipeline reconstructs a textured mesh, renders it from different viewpoints, and produces corresponding depth maps.
- Rendered images are blended with input images so training inputs retain rendered visual cues and realistic ambient lighting while remaining aligned with rendered depth maps.
- The dataset contains 113 reconstructed 3D models and more than 17,000 images spanning cities, architectures, sculptures, and small objects.Each scene contains 20 to 1,000 input images.
- Benchmarks show that models trained on BlendedMVS have significantly better generalization ability than models trained on other MVS datasets.The experiments train MVSNet, R-MVSNet, and PointMVSNet on several MVS datasets and evaluate them on different validation sets.
2. Related Works
Prior MVS datasets provide limited training ground truth and often target evaluation rather than network training. BlendedMVS follows synthetic-data practice while using image blending to incorporate lighting effects into rendered training data.
- Earlier learning-based MVS methods use neural networks for cost learning, voxel classification, or camera-aware surface prediction.
- MVSNet builds a camera-frustum cost volume with differentiable homography and predicts per-view depth, while R-MVSNet replaces 3D CNN regularization with recurrent regularization for high-resolution MVS.
- Existing benchmarks such as DTU, Tanks and Temples, and ETH3D provide limited training data, with DTU commonly used for learning-based MVS training.
- BlendedMVS provides more than 17,000 images with ground truth depth maps across diversified scenes to support MVS network training.
- Synthetic datasets can generate large amounts of ground truth at low cost, but BlendedMVS blends rendered images with original images to recover lighting effects that are difficult to model through material properties.
3. Dataset Generation
BlendedMVS generates aligned training images and depth maps by reconstructing textured meshes, rendering them, and blending rendered visual cues with lighting from input images. This pipeline targets realistic lighting while retaining reliable rendered-image textures and geometry.
- The pipeline reconstructs textured meshes from input images, renders each mesh viewpoint into color images and depth maps, and uses the depth maps as training ground truth.
- Training samples must keep images aligned with depth maps while reflecting view-dependent lighting for real-world generalization.
- Manual mesh-material and lighting setup is labor-intensive, making large-scale MVS dataset construction difficult.
- The blending method extracts high-frequency visual cues from rendered images and low-frequency ambient lighting from input images before fusing them.
- The blended image preserves rendered-image texture details while retaining realistic environmental lighting from the input image.
4. Scenes and Networks
BlendedMVS contains diverse reconstructed scenes and camera trajectories, then evaluates several learning-based MVS networks trained on the dataset. The section also describes the network implementations and training setup.
- Scenes: BlendedMVS includes 113 reconstructed models spanning architectures, street views, sculptures, and small objects, with 17,818 images total.
- Scenes: Its scenes use varied, unstructured camera trajectories rather than a fixed robot-arm path, modeling different image-capturing styles.
- Scenes: Training samples use a unified resolution of H × W = 1536 × 2048, with resized images cropped centrally and corresponding camera parameters adjusted.
- Networks: The study trains and evaluates MVSNet, R-MVSNet, and Point-MVSNet on BlendedMVS.
- Networks: MVSNet extracts image features, builds warped 3D feature volumes and a variance-based cost volume, regularizes it with multi-scale 3D CNNs, and regresses depth.
- Networks: R-MVSNet sequentially regularizes 2D cost maps through depth with a recurrent network, reducing memory consumption for high-resolution reconstruction.
- Networks: Point-MVSNet converts a coarse MVSNet depth map into a point cloud and iteratively refines it using estimated depth residuals.
5. Experiments
Experiments compare models trained on DTU, ETH3D, MegaDepth, and BlendedMVS across depth-map and point-cloud benchmarks. BlendedMVS training consistently provides stronger cross-dataset generalization, with additional benefits from blended inputs and online augmentation.
- Depth Map Validation: Models are compared using EPE, > 1 pixel error, and > 3 pixel error across validation sets.EPE is the average L1 depth error; the other metrics measure pixels exceeding one or three depth-wise pixels.
- Depth Map Validation: DTU-trained models perform well on DTU but produce high errors on BlendedMVS and ETH3D, indicating overfitting to small indoor scenes.The authors relate this behavior to DTU’s limited object-category diversity and fixed camera trajectory.
- Depth Map Validation: ETH3D-trained models show high validation errors on every validation set, including ETH3D, indicating insufficient training data.The ETH3D training split contains only five scenes.
- Depth Map Validation: MegaDepth-trained models remain inferior to BlendedMVS-trained models despite having more training samples.The authors attribute this to inconsistent alignment between input images and reconstructed depth maps, plus variable quality in crowdsourced images.
- Depth Map Validation: BlendedMVS-trained models achieve the best validation results on BlendedMVS and ETH3D, and the second-best result on DTU, close to the best.The results indicate strong generalization across different scene types.
- Point Cloud Evaluation: R-MVSNet trained on BlendedMVS outperforms models trained on the other three datasets across all Tanks and Temples scenes.The average f score increases from 0.475 with DTU training data to 0.532 with BlendedMVS.
- Ablation Study: Blended images with online photometric augmentation produce the best DTU validation result, while rendered images alone produce the worst.The ablation compares input, rendered, and blended images with and without online augmentation.
- Discussions: BlendedMVS also supplies occlusion and normal ground truth and can support image retrieval, feature learning, pose regression, depth, normal, and disparity estimation.The dataset is presented as applicable beyond MVS network training, with a later extension containing over 110k training samples.
6. Conclusion
BlendedMVS provides over 17k high-quality training samples for multi-view depth estimation, generated from reconstructed textured meshes and blended image inputs. Models trained on it achieve better generalization than models trained on other MVS datasets.
- BlendedMVS provides more than 17k high-quality training samples covering diverse scenes for multi-view depth estimation.
- The dataset pipeline reconstructs textured meshes, renders color images and depth maps, and blends rendered images with inputs to form training images.
- Models trained on BlendedMVS demonstrate significantly better generalization abilities than models trained on other MVS datasets.
1. Imperfect Reconstruction
Some mesh reconstructions are imperfect in reflective water and thin key-ring regions, producing incomplete blended images and rendered depth maps. The authors state that these defects do not affect training because the training inputs remain aligned with rendered depth maps.
- Reflective water and a thin key ring were not reconstructed successfully, leaving corresponding blended images and rendered depth maps incomplete.
- The authors state that these reconstruction defects do not affect training because original images are excluded and blended inputs remain aligned with rendered depth maps.
2. Input v.s. Blended v.s. Rendered Images
Blended images combine the lighting characteristics of input images with detailed visual cues from rendered images. The corresponding rendered depth maps are visualized alongside the three image types.
- The comparison distinguishes input images, blended images, and rendered images, with corresponding rendered depth maps shown jointly.
- Blended images resemble input-image lighting while inheriting detailed visual cues from rendered images.
3. Scenes
BlendedMVS includes diverse textured models organized into large-scale scenes, small-scale objects, and high-quality sculptures, with seven validation scenes. The figures illustrate these scene categories and camera trajectories.
- The textured models are roughly categorized as large-scale scenes, small-scale objects, and high-quality sculptures.
- Seven validation scenes are shown with their camera trajectories.
- Figure 1 compares input images, blended images, and rendered images.