Source-linked AI summary
SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis
Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, Lu Fang
TL;DR
Multiview stereopsis traditionally uses separate, manually engineered stages, motivating an end-to-end method that learns reconstruction from images and camera parameters. SurfaceNet encodes these inputs in colored voxel representations and predicts 3D surface occupancy; on DTU, it achieves accuracy and completeness comparable to existing methods.
Problem
Existing multiview stereopsis pipelines use separate stages, limiting end-to-end learning of photo-consistency and geometric context.
Method
SurfaceNet is a 3D convolutional network that encodes images and camera parameters in colored voxel cubes and predicts binary surface occupancy.
Results
SurfaceNet achieves better accuracy than camp and furu, better completeness than tola and Gipuma, and overall reconstruction quality comparable to other methods.
Takeaways & Limitations
The framework directly learns photo-consistency and geometric context for detailed multiview surface reconstruction without separate depth fusion.
Takeaways & Limitations
The method is currently comparable to the state of the art, while accuracy could be further improved with more advanced post-processing.
Abstract
from arXiv · showhide
This paper proposes an end-to-end learning framework for multiview stereopsis. We term the network SurfaceNet. It takes a set of images and their corresponding camera parameters as input and directly infers the 3D model. The key advantage of the framework is that both photo-consistency as well geometric relations of the surface structure can be directly learned for the purpose of multiview stereopsis in an end-to-end fashion. SurfaceNet is a fully 3D convolutional network which is achieved by encoding the camera parameters together with the images in a 3D voxel representation. We evaluate SurfaceNet on the large-scale DTU benchmark.
1. Introduction
Multiview stereopsis traditionally relies on separate processing stages, while SurfaceNet replaces this pipeline with end-to-end learning from images and camera parameters to 3D surfaces.
- Multiview stereopsis reconstructs a dense 3D object model from images with known camera parameters.
- Existing pipelines detect and propagate sparse features or compute and fuse multiview depth maps through separate stages.
- Deep learning had improved individual components, but the full pipeline remained outside end-to-end learning.
- SurfaceNet is proposed as an end-to-end 3D convolutional framework that learns reconstruction from images and camera parameters.
- Camera parameters are encoded through colored voxel cubes, and surface probabilities are converted into reconstructions by binarization with optional thinning.
2. Related Works
Prior MVS methods primarily use volumetric or depth-map-fusion pipelines with manually engineered stages. SurfaceNet instead predicts detailed 3D surfaces directly from multiview images.
- MVS approaches are broadly categorized into volumetric methods and depth-map-fusion algorithms.
- Depth-map-fusion pipelines recover pairwise depth maps and fuse them into 3D reconstructions using steps such as matching, denoising, and view-pair selection.
- SurfaceNet directly infers detailed 3D surfaces from multiview images without manually engineered separate processing steps.
- Patch-based reconstruction expands sparse keypoint matches but can leave holes around curved surfaces because rectangular patches lack sufficient detail.
- CNNs had been used to predict depth-map normals for improved fusion, while other CNN reconstruction methods remained coarse with limited geometric detail.
3. Overview
SurfaceNet voxelizes the solution space and predicts which voxels belong to the object surface, extending the approach from two views to multiple views.
- SurfaceNet takes images and camera parameters as input and predicts a binary surface attribute for each voxel.
- The network reconstructs a 2D surface embedded in a 3D voxel space, analogous to predicting a 1D boundary from 2D images.
- The method is introduced first for two views and then generalized to multiple views.
- Additional implementation details include early rejection of empty volumes to speed computation.
4. SurfaceNet
SurfaceNet reconstructs voxelized 3D surfaces end-to-end from multiview images by encoding camera geometry in colored voxel cubes. Its 3D CNN predicts per-voxel surface confidence, which is converted into a surface by thresholding and optional thinning.
- SurfaceNet: SurfaceNet estimates whether each voxel lies on the surface from images and known camera parameters.The target is a binary surface attribute for every voxel in the voxelized solution space.
- SurfaceNet: Colored voxel cubes encode camera parameters implicitly by projecting each voxel into an image and storing its RGB value.Voxels sharing a projection ray receive the same color, producing a view-specific stripe pattern.
- SurfaceNet Architecture: The network takes two colored voxel cubes and predicts a surface confidence for every voxel using a fully convolutional 3D architecture.Its design includes dilated convolutions and multi-scale feature aggregation, while the output remains the same spatial size as the input cubes.
- Training: Training compares per-voxel predictions with ground truth using a class-balanced cross-entropy loss because most voxels are non-surface.Training samples contain a cropped ground-truth cube and colored voxel cubes from two randomly selected views.
- Inference: Inference processes the scene in cubes, predicts surface probabilities, thresholds them, and optionally thins the resulting surface by ray pooling.When both operations are used, a voxel must exceed the threshold and receive votes from both views.
5. Multi-View Stereopsis
For multiple views, SurfaceNet combines predictions from selected view pairs rather than treating all pairs equally. Pair selection uses learned patch dissimilarity and viewpoint geometry, followed by view voting for surface extraction.
- Multi-View Stereopsis: With multiple views, SurfaceNet computes colored voxel cubes and surface predictions for a selected subset of view pairs.The framework extends the two-view procedure by combining pairwise predictions for each voxel.
- Multi-View Stereopsis: Predictions from view pairs are combined by a weighted average because reconstruction accuracy varies with viewpoint difference and occlusions.The method selects the Nv view pairs with the highest learned weights instead of using every pair equally.
- View-Pair Selection: A triplet network embeds 64 × 64 image patches into a compact 128D Euclidean space, where distances measure patch dissimilarity.The dissimilarity is combined with the angle between the projection rays to assign relative view-pair weights.
- Surface Extraction: During multi-view extraction, a voxel becomes a surface voxel when at least γ = 80% of views vote for it and its probability exceeds the threshold.This extends the two-view ray-pooling procedure to multiple views.
- Training: Training averages predictions from Nv random view pairs, with Nv train = 6 chosen as a memory-related trade-off.Larger values increase sampled-cube memory and require reducing the training batch size.
6. Implementation Details
Implementation details accelerate reconstruction and improve consistency across cube boundaries. Empty cubes are rejected using learned patch similarity, while adaptive thresholds favor overlapping surface voxels between neighboring cubes.
- Empty-Cube Rejection: Logistic regression predicts whether patches from each view pair are similar and rejects cubes with insufficiently similar view pairs.A cube is rejected when fewer than Nmin view pairs have predicted similarity probability at least 0.5.
- Adaptive Thresholding: Adaptive binarization initializes each cube threshold at 0.5 and optimizes it within [0.5, 1) using neighboring cubes.The estimated surface SC(τC) is defined after binarization with the cube-specific threshold.
- Adaptive Thresholding: The adaptive energy penalizes disagreement in overlapping regions while encouraging neighboring cubes to share surface voxels.The formulation avoids trivially high thresholds that remove most overlapping surface voxels.
- Implementation: The implementation used the Lasagne library, and the code and trained model were made publicly available.
7. Experiments
Experiments on DTU and cross-dataset tests examine parameter choices, reconstruction quality, efficiency, and generalization. SurfaceNet achieves competitive accuracy and completeness while requiring minimal post-processing.
- Benchmark: The DTU benchmark contains 80 scenes with varied objects, materials, viewpoints, and lighting conditions, using structured-light reference models.
- Parameter impact: Threshold τ trades accuracy against completeness: τ = 0.9 discards surface regions, whereas τ = 0.5 produces noisy, inaccurate reconstructions.
- Parameter impact: Thinning with γ = 100% improves accuracy for every threshold τ while only slightly affecting completeness.
- Parameter impact: Adaptive thresholding with β = 6 converges after a few iterations and improves mean accuracy, although mean completeness is slightly worse than constant thresholding.
- Parameter impact: Using five best view pairs substantially improves accuracy over one pair, while additional fusion slightly improves accuracy at a small completeness cost.
- Comparison: On 22 DTU test models, 64^3 cubes improve both metrics for constant thresholding; SurfaceNet beats selected baselines on complementary accuracy and completeness measures.
- Efficiency: Inference takes 50ms for one view pair and a 32^3 cube, versus 400ms for a 64^3 cube, while rejected cubes reduce the processed volume.
- Generalization: With unseen camera views, accuracy remains relatively stable but completeness error increases because fewer views are available.
8. Conclusion
SurfaceNet is presented as an end-to-end multiview stereopsis framework with colored voxel cubes that combine image and camera information for standard 3D convolutions. Evaluation on a large-scale MVS benchmark showed accurate 3D surface reconstruction, while accuracy remained comparable to state of the art and could improve through advanced post-processing.
- SurfaceNet provides an end-to-end learning framework for multiview stereopsis.
- Colored voxel cubes combine image and camera information in a representation processable by standard 3D convolutions.
- Evaluation on a large-scale MVS benchmark demonstrated accurate reconstruction of 3D object surfaces.
- The method was comparable to the state of the art, with accuracy potentially improvable through more advanced post-processing methods.