Source-linked AI summary
Learning a Multi-View Stereo Machine
Abhishek Kar, Christian Häne, Jitendra Malik
TL;DR
The paper addresses whether multi-view stereo can be learned while preserving the problem’s geometric constraints. It introduces LSMs, which differentiate feature projection and unprojection between images and metric 3D space, combining learned priors with geometry. On ShapeNet, LSMs reconstruct effectively from fewer images than classical systems and use camera poses better than recent learning-based approaches, while remaining limited by coarse grid resolution.
Problem
The paper asks whether a learned multi-view stereo system can reconstruct 3D geometry from images and known camera poses while exploiting geometric cues.
Method
LSMs use differentiable projection and unprojection to transfer features between 2D images and a metric 3D grid, recurrently matching views before 3D CNN processing.
Results
LSMs reconstruct objects from fewer images than classical approaches and use camera pose information more effectively than recent learning-based reconstruction methods as views increase.
Takeaways & Limitations
The system supports voxel or per-view depth reconstruction from few views, including a single image, while using shape priors to predict unseen regions.
Takeaways & Limitations
The current system is constrained by the coarse 32^3 world-grid resolution, below the higher resolutions commonly used by classical algorithms.
Abstract
from arXiv · showhide
We present a learnt system for multi-view stereopsis. In contrast to recent learning based methods for 3D reconstruction, we leverage the underlying 3D geometry of the problem through feature projection and unprojection along viewing rays. By formulating these operations in a differentiable manner, we are able to learn the system end-to-end for the task of metric 3D reconstruction. End-to-end learning allows us to jointly reason about shape priors while conforming geometric constraints, enabling reconstruction from much fewer images (even a single image) than required by classical approaches as well as completion of unseen surfaces. We thoroughly evaluate our approach on the ShapeNet dataset and demonstrate the benefits over classical approaches as well as recent learning based methods.
1 Introduction
The paper asks whether multi-view stereo can be learned while retaining geometric constraints, and introduces LSMs to reconstruct metric 3D geometry from few views. LSMs combine learned shape priors with projective geometry and outperform direct learning approaches in using camera pose information.
- MVS reconstructs 3D world geometry from images with known camera poses, traditionally using dense correspondences and triangulation.
- The paper asks whether a neural multi-view stereo system can learn depth and reconstruction from image observations.
- LSMs reconstruct voxel occupancy grids or per-view depth maps from few views, including a single image.
- LSMs embed projective geometry through differentiable feature unprojection and projection, enabling metric 3D reasoning.
- Unlike methods built around one cue, LSMs learn instance-relevant cues while using shape priors to predict unseen regions.
- Camera-pose input gives LSMs substantial improvements as views increase, whereas straightforward pose-augmented volumetric prediction does not yield expected gains.
2 Related Work
Prior work uses geometric cues, learned shape priors, or binocular geometry, but often restricts geometry to training or single-view inference. LSMs extend differentiable geometric reasoning to multiple views and retain geometric constraints during inference.
- Classical methods use silhouettes, photo-consistency, optimization, or explicit viewing-ray reasoning to recover 3D structure.
- Learning-based reconstruction has used shape priors, semantic surface priors, image-patch similarity, and end-to-end disparity regression.
- Related systems use multi-view supervision but apply photo-consistency and silhouette consistency only during training, predicting shape from one image at inference.
- Unlike complementary single-view systems, LSM uses geometric constraints during inference and can also infer 3D from a single image.
- LSM generalizes binocular geometric learning to multiple views by tracing rays through a discretized grid and matching views recurrently.
- Its differentiable projection learns a nonlinear feature combination rather than relying on differentiable arg-min, which is susceptible to multiple modes.
3 Learnt Stereo Machines
Learnt Stereo Machines build an end-to-end stereo system around metric 3D grids, using differentiable projection and unprojection to incorporate camera geometry. Recurrent fusion and 3D grid reasoning combine multi-view evidence with learned shape priors, supporting voxel or depth reconstruction.
- 3.1 System overview: LSMs encode input images, unproject their features into metric 3D grids using known camera poses, and align corresponding features for local matching.The shared image encoder produces dense feature maps before pose-guided unprojection into 3D feature grids.
- 3.2 Differentiable unprojection: Differentiable unprojection copies image features along viewing rays into a global 3D grid, placing features from corresponding views at shared world locations.Projecting grid points into image space and bilinearly sampling feature maps provides a differentiable alternative to analytically tracing rays.
- 3.2 Differentiable unprojection: For single-image prediction, LSMs append depth values and ray directions because features from multiple images are unavailable for matching.These geometric features facilitate predicting where surfaces should be placed along viewing rays.
- 3.3 Recurrent grid fusion: Recurrent grid fusion sequentially combines per-image 3D feature grids into a joint grid, using 3D convolutions to exploit neighborhood information while limiting parameters.The recurrent hidden state acts like a running belief over matching scores from the observations processed so far.
- 3.4 3D grid reasoning: A 3D UNet transforms the fused grid into an output grid by combining feature matches and silhouettes with priors such as smoothness, symmetry, and object-class knowledge.This grid-reasoning stage models contextual smoothing and can produce complete shapes from partial observations.
- 3.5 Differentiable projection: Differentiable projection samples 3D grid features at continuous world points along viewing rays, concatenates samples in depth order, and produces 2D feature maps.Trilinear interpolation is differentiable, while nearest-neighbor interpolation is used in practice for computational efficiency.
- 3.6 Architecture details: V-LSM outputs voxel occupancy grids, whereas D-LSM projects the final grid into per-view depth maps; both use shared UNet-based components.D-LSM learns ray-wise reduction functions with 1x1 convolutions and uses skip connections to retain high-frequency image information.
4 Experiments
The experiments evaluate LSMs on ShapeNet for voxel reconstruction, category generalization, depth prediction, and comparison with plane sweeping stereo. Results show improved use of multiple views, generalization to unseen categories, consistent depth geometry, and lower depth error than plane sweeping.
- Experimental setup: LSMs are evaluated on ShapeNet for voxel reconstruction, unseen-category generalization, and per-view depth prediction against classical and learning-based baselines.The evaluation uses posed image sets, ground-truth occupancy and depth maps, and 13 ShapeNet categories.
- Multi-view reconstruction: Mean voxel IoU increases with additional views for all methods, but V-LSM benefits more than 3D-R2N2 variants.Naively injecting camera pose into 3D-R2N2 does not improve over its base version, whereas V-LSM is designed to exploit geometric multi-view cues.
- Generalization: The performance gap on unseen object categories decreases as the number of views increases for V-LSM, indicating use of category-agnostic shape cues.3D-R2N2 w/pose fails to generalize with more views; V-LSMs trained for four steps are evaluated for up to eight.
- Multi-view depth prediction: Depth-LSM predicts thin structures and metrically consistent geometry across views, with unprojected point clouds aligning well in world coordinates.The qualitative evaluation includes challenging chair and table examples and uses shading from a shifted viewpoint.
- Comparison to plane sweeping: With fewer input images, Depth-LSM produces cleaner point clouds than plane sweeping stereo and remains robust in texture-less regions by using shape priors.Plane sweeping completely fails with fewer images in the qualitative comparison; quantitative evaluation with 10 views reports mean depth-map errors of 0.051 for PS and 0.024 for D-LSM.
5 Discussion
LSMs demonstrate the benefits of differentiable projective geometry while remaining limited by coarse world-grid resolution and object-focused evaluation. The projection modules also support broader 2D–3D applications beyond the evaluated setup.
- LSMs differentiably transfer features between 2D images and the 3D world, supporting metric multi-view stereopsis.The system is end-to-end learned and uses projective geometry in both directions.
- The system generalizes to new object categories and produces reconstructions with fewer images than classical systems.
- The current world grid has coarse resolution (323), whereas classical algorithms often use higher resolutions and structures such as octrees.The authors identify higher-resolution volume prediction as a direction for future work.
- Although experiments use known-pose multi-view reconstruction, the projection modules can move information between image and world frames for other representations and view synthesis.Potential outputs include silhouettes, pixel-wise part labels, and projections into unobserved views.
A.1 Sensitivity to noisy camera pose and segmentations
V-LSM performance degrades under noisy camera poses, but remains better than the baseline at 10 deg noise; training with perturbed poses improves robustness. Random image backgrounds cause only a very small performance drop.
- At 10 deg camera-pose noise, models trained with perfect poses degrade but remain better than the baseline.
- Training with synthetically perturbed camera extrinsics substantially improves robustness to noisy poses while maintaining competitive performance.The comparison is illustrated in Figure 7.
- Random image backgrounds produce only a very small performance drop compared with training on white backgrounds.The result indicates that the method does not rely heavily on perfect segmentations.
A.2 Detailed Results
The detailed-results section reports per-category ShapeNet test-set evaluations for voxel occupancy and depth-map reconstruction across the compared systems.
- Table 2 reports mean Voxel IoU for V-LSM across all ShapeNet test-set classes.
- Table 3 reports mean Voxel IoU for 3D-R2N2 w/pose across all ShapeNet test-set classes.
- Table 4 reports mean depth map error using L1 distance between predictions and ground truth at valid ShapeNet test-set pixels.