Source-linked AI summary
MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis
Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, Gim Hee Lee
TL;DR
Single-image novel view synthesis needs dense geometry and occluded-content completion, while existing MPI and NeRF representations have complementary limitations. MINE predicts continuous-depth RGB-density planes conditioned on one image, rendering reconstructed frusta into novel RGB and depth views. The method significantly outperforms existing view-synthesis methods and achieves near-state-of-the-art depth estimation without dense ground-truth supervision.
Problem
Single-image view synthesis requires accurate dense geometry and occluded-content inpainting, while MPI has discrete-depth limitations and NeRF is trained separately for each scene.
Method
MINE conditions on a single image and arbitrary depth values to predict four-channel RGB-and-volume-density planes forming a continuous camera-frustum representation.
Results
MINE significantly outperforms existing single-image view-synthesis methods and achieves near-state-of-the-art depth estimation without dense ground-truth depth supervision.
Takeaways & Limitations
The reconstructed and inpainted frustum supports rendering novel RGB images and depth maps through differentiable rendering.
Takeaways & Limitations
Depth scale is ambiguous from a single image, requiring scale calibration; the sparse disparity loss is optional and unavailable when dataset scale is fixed.
Abstract
from arXiv · showhide
In this paper, we propose MINE to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our approach is a continuous depth generalization of the Multiplane Images (MPI) by introducing the NEural radiance fields (NeRF). Given a single image as input, MINE predicts a 4-channel image (RGB and volume density) at arbitrary depth values to jointly reconstruct the camera frustum and fill in occluded contents. The reconstructed and inpainted frustum can then be easily rendered into novel RGB or depth views using differentiable rendering. Extensive experiments on RealEstate10K, KITTI and Flowers Light Fields show that our MINE outperforms state-of-the-art by a large margin in novel view synthesis. We also achieve competitive results in depth estimation on iBims-1 and NYU-v2 without annotated depth supervision. Our source code is available at https://github.com/vincentfung13/MINE
1. Introduction
Single-image novel view synthesis requires both precise geometry and completion of occluded scene content. MINE addresses these challenges with a continuous-depth representation that reconstructs and inpaints the camera frustum.
- Single-image novel view synthesis is challenging because it requires precise geometry understanding and inpainting occluded geometry and textures.
- Naive SfM or monocular/multiview depth estimation cannot reliably recover dense 3D geometry or occluded content, causing distortions and artifacts.
- MPI improves view synthesis but represents depth with discrete planes, limiting continuous 3D representation and conversion to meshes or point clouds.
- MINE generalizes MPI toward NeRF by conditioning on image features and arbitrary depth to predict a four-channel RGB-and-density plane.
- MINE performs continuous, occlusion-inpainted 3D reconstruction from one image and significantly outperforms existing methods in view synthesis and depth estimation.
2. Related Work
Prior work spans explicit layered or volumetric representations and implicit neural representations for view synthesis and depth estimation. MINE combines continuous NeRF-like modeling with image-conditioned generalization to unseen scenes.
- Layered depth images and MPIs explicitly model occlusions for view synthesis but use limited layered or planar representations.
- Self-supervised depth methods use image reconstruction, epipolar geometry, consistency, or monocular videos instead of ground-truth depths.
- Implicit 3D methods map coordinates to shape functions or occupancy, while NeRF maps continuous coordinates and viewing directions to RGB and volume density.
- MINE predicts RGB-σ planes at arbitrary depths, enabling continuous dense 3D reconstruction while conditioning on the input image to generalize to unseen scenes.
3. Our Approach
MINE represents a camera frustum with planar neural radiance fields whose RGB and density functions can be queried continuously. Differentiable volume rendering projects this representation into the input or novel camera views.
- The method takes a single image and outputs a 3D representation of the source camera frustum.
- Arbitrary-depth planes contain RGB values and volume densities, allowing continuous reconstruction of every frustum position within the selected depth range.
- For convenience, the continuous field is discretized into N four-channel image planes, while arbitrary 3D queries remain available through depth sampling.
- Input-image rendering uses accumulated transmittance and distances between planes in classical volume rendering.
- Depth maps are rendered similarly to RGB views, and disparity samples are drawn across the frustum to support continuous field learning.
- Novel-view rendering homography-warps each source plane into the target camera, computes target-ray distances, and applies volume rendering.
3.2. Network and Training Design
The network predicts depth-conditioned RGB-density planes from a single image and trains them through rendered target views. Scale calibration and composite image, smoothness, and optional sparse-depth losses support the reconstruction.
- Network: The encoder-decoder predicts four-channel planes from a single image and a depth or disparity value, with disparity encoded before decoding.
- Training: Training renders predicted RGB and depth into target views and compares the synthesized RGB image with the ground-truth target image.
- Scale Calibration: Single-image depth has an unknown global scale, so the method calibrates camera translation using a predefined frustum range and SfM-derived sparse points.
- Loss Functions: The total loss combines RGB L1, RGB SSIM, edge-aware disparity smoothness, and an optional sparse disparity term weighted by hyperparameters.
- Loss Functions: Edge-aware smoothness penalizes disparity changes in smooth image regions and aligns disparity edges with image edges.
- Loss Functions: The sparse disparity loss is used when SfM resolves scale ambiguity but is unnecessary when dataset scale is fixed, such as KITTI.
3.4. Our Relation to NeRF
MINE uses a continuous frustum representation that extends MPI beyond discrete planes, while requiring fewer inference passes than pixel-wise alternatives such as pixelNeRF and GRF.
- Relation to MPI: MINE represents RGB and volume density at arbitrary spatial positions, whereas MPI is a special case using discrete depth planes.The MPI equivalence follows by setting sampled depths to bin edges and applying the resulting compositing operation.
- Relation to pixelNeRF and GRF: MINE directly models and reconstructs the source-camera frustum per plane, while pixelNeRF and GRF model the entire 3D space per ray.This distinction makes MINE’s representation camera-frustum-specific rather than globally space-specific.
- Relation to pixelNeRF and GRF: MINE requires N_planes forward passes to obtain its representation, followed by homography warping for each novel view.The decoder is fully convolutional, so the representation is produced plane by plane before rendering.
- Relation to pixelNeRF and GRF: PixelNeRF and GRF require forward passes proportional to output resolution, sampled points along rays, and the number of target views.Both methods render output pixels individually, unlike MINE’s frustum-wide reconstruction.
- Relation to pixelNeRF and GRF: MINE’s frustum-focused design makes inference significantly more efficient than pixelNeRF and GRF.The efficiency difference follows from reconstructing the frustum per plane instead of rendering each output pixel per ray.
4. Experiments
Experiments evaluate MINE for novel view synthesis across KITTI, RealEstate10K, and Flowers Light Fields, and for depth estimation on iBims-1 and NYU-Depth V2. Across these settings, MINE improves reconstruction quality, benefits from denser depth sampling, and achieves strong depth estimation without dense depth supervision.
- Experimental setup: MINE is evaluated with SSIM, PSNR, and LPIPS for view synthesis on RealEstate10K, Flowers Light Fields, and KITTI, plus depth estimation on iBims-1 and NYU-Depth V2.The depth model is trained with RealEstate10K and evaluated against systems using ground-truth depth supervision.
- KITTI view synthesis: 0.822 SSIM is achieved by MINE on KITTI versus 0.733 for MPI [51].Both 32- and 64-plane models outperform the compared methods, while qualitative results show fewer artifacts and shape distortions.
- Ablations and Flowers Light Fields: Increasing N consistently improves results on KITTI, RealEstate10K, and Flowers Light Fields.The KITTI ablation attributes this trend to learning more complex scene geometry through denser depth sampling while model capacity remains fixed.
- Ablations and Flowers Light Fields: The edge-aware depth smoothness loss yields marginal quantitative gains but enables better disparity maps qualitatively on KITTI.Figure 4 provides the qualitative comparison associated with this design choice.
- RealEstate10K view synthesis: MINE outperforms [56] and [51] by a large margin on all three RealEstate10K criteria.Increasing the number of planes from 32 to 64 improves results, and qualitative comparisons show sharper images, fewer artifacts, and better dis-occluded-region inpainting.
- Depth estimation: MINE achieves comparable depth-estimation performance to 3DKenBurns [32] and significantly outperforms other compared methods without ground-truth depth supervision.The evaluation uses iBims-1 and NYU-Depth V2, with scale and bias calibration before scoring.
5. Conclusion
MINE performs single-image novel view synthesis and depth estimation by densely reconstructing and inpainting the camera frustum, then rendering novel RGB and depth views differentiably.
- MINE jointly reconstructs the camera frustum densely and inpaints occluded contents from a single image.
- The reconstructed frustum is rendered into novel-view RGB images and depth maps using differentiable rendering.
- MINE significantly outperforms existing state-of-the-art single-image view synthesis methods.
- MINE achieves near-state-of-the-art depth estimation without dense ground-truth depth supervision.
Supplementary Materials
The supplementary materials identify the authors and their affiliations.
- The paper lists Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, and Gim Hee Lee as authors.
- The authors are affiliated with ByteDance and the National University of Singapore.
A. MINE vs. pixelNeRF and GRF
MINE differs from pixelNeRF and GRF by reconstructing the source-camera frustum per plane, enabling more efficient rendering and dense supervision strategies.
- MINE predicts RGB-σ planes conditioned on input images, while pixelNeRF samples projected feature vectors and GRF assumes multiple test-time views.
- MINE directly models the source-camera frustum, whereas pixelNeRF and GRF model the entire 3D space.
- MINE reconstructs the source-camera frustum per plane, while pixelNeRF and GRF reconstruct the entire 3D space per ray.
- MINE requires only Nplanes fully convolutional decoder passes to obtain the representation, followed by homography warping for each novel view.
- MINE uses one encoder inference for a single input image, whereas GRF requires Nviews encoder inferences for multiple input images.
- MINE's complexity is independent of Ntargets, while pixelNeRF and GRF complexity is proportional to Ntargets.
- Dense rendering allows MINE to use supervision signals such as SSIM and edge-aware smoothness loss during training.
- MINE is evaluated on large-scale real-world datasets including KITTI, NYU-V2, and RealEstate10k, unlike pixelNeRF and GRF in the cited comparison.
B. Additional Implementation Details
Implementation details cover MINE's encoder-decoder architecture and dataset-specific preprocessing for Flowers Light Fields.
- MINE uses a standard ResNet50 encoder and a decoder producing a 4-channel RGB-σ image.
- For Flowers Light Fields, the disparity range is set to [3.0, 0.03], with a constant principal-point shift following prior work.
- Flowers Light Fields training and testing use an adjacent-grid distance of 0.00128, with translation between views and no rotation.
C. Additional Qualitative Results
Supplementary qualitative and video results evaluate MINE on unseen indoor, outdoor, and complex-geometry scenes. The results include RGB, disparity, and architectural materials across several datasets.
- Cross-dataset qualitative results: MINE generalizes to unseen KITTI, RealEstate10K, and Flowers Light Fields scenes spanning outdoor, indoor, and complex flower geometries.The supplementary image results cover Figures 7–9 and emphasize generalization across scene types.
- Video results: Supplementary videos provide RGB and disparity renderings from single-image inputs across RealEstate10K, KITTI, and iBims-1.Each scene is rendered into multiple novel views, including outdoor and indoor scenes with complex geometries and textures.
- Video results: Under large camera motion, MINE produces temporally consistent realistic images and smooth, accurate disparity maps on unseen scenes.The video results report this behavior for both RGB and disparity sequences.
- Depth decoder architecture: The depth decoder uses upconvolution blocks with convolution, batch normalization, activation, and 2× nearest-neighbor upsampling, while downconvolution blocks use stride-2 max pooling.Table 5 also specifies convolution followed by activation in the downconv blocks.