PaperlayerView exact revision on arXiv

Source-linked AI summary

MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis

Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, Gim Hee Lee

arXiv:2103.14910v3cs.CVcs.GRcs.LG

TL;DR

Single-image novel-view synthesis requires dense geometry and occluded-content inpainting, while straightforward SfM or depth recovery is insufficient for both. MINE extends MPI with continuous depth-conditioned RGB and density planes, significantly outperforming prior view-synthesis methods and achieving near-state-of-the-art depth estimation.

  • Problem

    Single-image novel-view synthesis requires dense scene geometry and occluded-content inpainting, while straightforward SfM or depth recovery is insufficient for both.

    Source: §1 · p. 1

  • Method

    MINE conditions on an input image and arbitrary depth to predict RGB-volume-density planes, extending MPI into continuous, dense 3D scene reconstruction.

    Source: §1 · pp. 1–2, §2 · p. 2

  • Results

    MINE significantly outperforms state-of-the-art single-image view-synthesis methods and achieves near-state-of-the-art depth estimation; on KITTI, SSIM rises from 0.733 to 0.822 versus.

    Source: §4.1 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    The reconstructed frustum can be rendered as novel-view RGB images and depth maps using differentiable rendering.

    Source: §5 · p. 8

    Frustum reconstruction is restricted to a pre-defined depth range, set to zn = 1 and zf = 1000.

    Source: §3.3.1 · p. 4

Abstract

from arXiv · show

In this paper, we propose MINE to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our approach is a continuous depth generalization of the Multiplane Images (MPI) by introducing the NEural radiance fields (NeRF). Given a single image as input, MINE predicts a 4-channel image (RGB and volume density) at arbitrary depth values to jointly reconstruct the camera frustum and fill in occluded contents. The reconstructed and inpainted frustum can then be easily rendered into novel RGB or depth views using differentiable rendering. Extensive experiments on RealEstate10K, KITTI and Flowers Light Fields show that our MINE outperforms state-of-the-art by a large margin in novel view synthesis. We also achieve competitive results in depth estimation on iBims-1 and NYU-v2 without annotated depth supervision. Our source code is available at https://github.com/vincentfung13/MINE

1. Introduction

Novel view synthesis from images is challenging because it requires accurate geometry and inpainting of occluded content. MINE addresses this by generalizing MPI to continuous 3D reconstruction from a single image.

Problem

  • SfM and monocular or multiview depth estimation [11] can miss dense geometry and occluded content, while 2.5D approaches such as LDI and MPI have limited resolution for representing the full scene.These shortcomings can produce distortion and artifacts in rendered novel views.Source: §1 · p. 1

Related work

  • MPI is limited by its discrete planes and cannot represent continuous 3D space effectively, while NeRF [29] produces high-quality structures but requires a separate MLP for each scene.MPI depth-wise resolution depends on the number of planes; NeRF’s MLP predicts RGB and volume occupancy density from 3D position and viewing direction.Source: §1 · p. 1

Contribution

  • MINE generalizes MPI into a continuous 3D representation inspired by NeRF [29], enabling occlusion-inpainted reconstruction of the camera frustum from one image.The paper also identifies continuous, occlusion-inpainted reconstruction from a single image as a contribution.Source: §1 · pp. 1–2

Method

  • Given image features and an arbitrary depth, MINE predicts an RGB-and-volume-density plane parallel to the input camera, allowing reconstruction across the full camera frustum.The paper states that MPI is a limited special case of MINE.Source: §1 · pp. 1–2

2. Related Work

Prior work explores layered and implicit 3D representations for view synthesis, as well as self-supervised depth estimation. MINE combines continuous-depth rendering with input-image conditioning to reconstruct scenes and generalize to unseen scenes.

Explicit 3D representations for view synthesis

  • MPIs model occluded contents with RGB-α planes at fixed depths, but sparse depth discretization limits performance; LDIs allow arbitrary layers but can produce cracks or require slow inpainting [28] [46] [42].LDI rendering in can produce cracks; the iterative depth-edge inpainting in [42] is described as prohibitively slow for real-world applications.Source: §2 · p. 2

Self-supervised depth estimation

  • Self-supervised depth methods use image reconstruction error, including paired-image disparity prediction and monocular-video supervision, but strong depth estimation does not guarantee strong view synthesis, or vice versa [7] [11] [25].Method [11] also adds a left-right consistency term to improve disparity-map quality.Source: §2 · p. 2

Implicit 3D representations for view synthesis

  • Implicit 3D shape methods map coordinates to signed distance functions or occupancy, but require ground-truth geometry; RGB-supervised alternatives do not render photorealistically in complex scenes [3] [17] [10] [27] [33] [9] [35] [31].NeRF [29] maps continuous 3D coordinates and viewing directions to RGB and volume density, but requires per-scene optimization; PixelNerf and GRF [50] do not report large-scale real-world dataset experiments.Source: §2 · p. 2

MINE

  • MINE predicts RGB-σ planes at arbitrary depths, combining NeRF and MPI for continuous, dense scene reconstruction while conditioning on the input image to generalize to unseen scenes.Unlike NeRF, MINE does not encode scene geometry in network weights.Source: §2 · p. 2

3. Our Approach · 3.1. 3D Representation

MINE maps a single input image to a continuous planar radiance field representing the camera frustum. The representation supports volume rendering of the source image and novel views, as well as depth rendering.

3. Our Approach

  • MINE takes a single image as input and outputs a 3D representation, with its network design and training pipeline introduced in later sections.Source: §3 · p. 2

3.1.1 Planar Neural Radiance Field

  • Arbitrary-depth planes store RGB values and volume densities, allowing the frustum’s radiance and density to be queried continuously at any position in the depth range.The density represents the differential probability of a ray terminating at an infinitesimal particle.Source: §3.1.1 · pp. 2–3

3.1.2 Volume Rendering

  • In practice, the field is discretized into N four-channel image planes, yet arbitrary plane depths and sub-pixel sampling preserve RGB and density queries at any 3D position.Source: §3.1.2 · p. 3
  • Source-image rendering uses classical volume-rendering principles, with accumulated transmittance representing the probability that a ray travels between planes without hitting an object.Source: §3.1.2 · p. 3
  • Sampling disparity uniformly across N bins exposes the network to every frustum depth during training, enabling it to learn a continuous planar radiance field.Source: §3.1.2 · p. 3
  • Novel-view rendering first uses inverse homography to map target pixels to source planes, then applies volume rendering using density and distances between ray intersections.Source: §3.1.2 · p. 3

3.2. Network and Training Design

MINE uses an encoder-decoder to predict RGB and density planes from a single image and depth values. It encodes disparity before decoding each plane because directly feeding disparity performs poorly.

Network Overview

  • The network takes an input image and depth values to reconstruct the source camera frustum, which can then be rendered into a novel view.Depth or disparity samples are randomly drawn according to Eq. 3.Source: §3.2 · p. 3, Figure 2 · p. 4

Encoder-Decoder Structure

  • The encoder processes each image once, while the decoder uses each plane’s disparity to generate its 4-channel RGB-and-density image N times.The encoder uses ResNet-50 [15], and the decoder design is similar to Monodepth2.Source: §3.2 · p. 4

Disparity Encoding

  • Directly feeding disparity to the decoder gives poor performance, so MINE applies an encoding function to disparity first.The passage notes this finding is consistent with [29] [36].Source: §3.2 · p. 4

3.3. Supervision with RGB Videos

MINE trains from multi-view images or RGB videos by rendering a source image into a target camera view and comparing the synthesized RGB image with the target. It calibrates translation to resolve depth-scale ambiguity and combines RGB, smoothness, and optional sparse-disparity losses.

3.3. Supervision with RGB Videos

  • Training renders each source image into a target view using the camera rotation and calibrated translation, then supervises the synthesized RGB image against the ground-truth target.The network also renders a target depth map alongside the RGB image.Source: §3.3 · p. 4

3.3.1 Scale Calibration

  • Because single-image depth has ambiguous scale, MINE fixes the frustum range to [1, 1000] and scales camera translation using SfM points and synthesized depth, setting t′ = t · s.COLMAP provides sparse 3D points for estimating the scale; the representation itself is not scaled.Source: §3.3.1 · p. 4

3.3.2 Loss Functions

  • The loss combines RGB L1 and SSIM terms with edge-aware disparity smoothness, weighted by hyperparameters; smoothness encourages disparity edges to align with image edges [11].The disparity is mean-normalized, with predicted disparity defined as the inverse of predicted depth.Source: §3.3.2 · pp. 4–5
  • Sparse disparity loss is optional when SfM resolves scale ambiguity, but is inapplicable when scale is fixed to s = 1, as in KITTI [6].The disparity maps must be scaled consistently with calibrated translation and depth.Source: §3.3.2 · p. 5

3.4. Our Relation to NeRF · 3.5. Our Relation to MPI · 3.6. Our Relation to pixelNeRF and GRF

MINE predicts RGB and volume density over a source-camera frustum, making it a continuous counterpart to MPI and distinguishing its scene representation and inference from NeRF, pixelNeRF, and GRF. The sections also describe MINE’s advantages and limitations relative to NeRF.

3.4. Our Relation to NeRF

  • MINE shares NeRF’s RGB-and-volume-density representation at arbitrary positions, generalizes to unseen scenes, and uses about 32 network inferences versus NeRF’s millions for novel-view rendering.Source: §3.4 · p. 5
  • Because MINE uses one input image and no viewing-direction input, it cannot reconstruct an object from all 360° or model complex view-dependent effects.Source: §3.4 · p. 5

3.5. Our Relation to MPI

  • MPI is a special case of MINE’s representation: setting depth samples to bin edges and defining per-depth alpha maps yields MPI-identical compositing.Source: §3.5 · p. 5
  • With depth samples set to bin edges rather than randomly sampled, the interval δ_zi is constant.Source: §3.5 · p. 5

3.6. Our Relation to pixelNeRF and GRF

  • MINE models the source-camera frustum per plane, whereas pixelNeRF and GRF [50] model all 3D space and reconstruct it per ray; MINE also evaluates large-scale indoor and outdoor datasets.The listed datasets are RealEstate10K, NYUv2, and KITTI; the passage says pixelNeRF and GRF do not present experiments on large-scale real-world datasets.Source: §3.6 · p. 5
  • MINE’s per-plane frustum reconstruction requires Nplanes decoder forward passes, followed by homography warping for any novel view, unlike pixelNeRF and GRF’s pixel-by-pixel rendering.For pixelNeRF and GRF, forward-pass counts scale with output resolution, points along each ray, and target views.Source: §3.6 · p. 5

4. Experiments

Experiments evaluate MINE on novel view synthesis across KITTI, RealEstate10K, and Flowers light fields, and on depth estimation using iBims-1 and NYU-Depth V2. The reported comparisons cover quantitative metrics and qualitative results.

4.1. View Synthesis on KITTI

  • On KITTI at 384 × 128 test resolution, MINE significantly improves SSIM from 0.733 to 0.822 versus, producing more realistic images with fewer artifacts and shape distortions.Source: Table 1 · p. 6, §4.1 · p. 6
  • KITTI ablations show moderate gains from ImageNet encoder pre-training, marginal quantitative gains from edge-aware depth smoothness, and consistent improvements as N increases.With model capacity held constant, the N-related gains are attributed to learning more complex scene geometry through denser depth sampling; the smoothness loss qualitatively improves synthesized disparity maps.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2. View Synthesis on RealEstate10K

  • MINE outperforms on all three RealEstate10K criteria, with further improvements when increasing N from 32 to 64.Qualitative comparisons show sharper, more realistic target images, fewer artifacts and distortions, and improved inpainting of disoccluded areas.Source: Table 2 · p. 7, §4.2 · pp. 7–8, Figure 5 · p. 7

4.3. Depth Estimation on iBims-1 and NYU-V2

  • MINE achieves performance comparable to 3DKenBurns [32] and significantly outperforms the other compared methods on iBims-1 and NYU-Depth V2 without ground-truth depth supervision.Against MPI, MINE produces smoother and more accurate disparity maps on texture-rich surfaces, while MPI is biased toward image textures.Source: Table 3 · p. 7, §4.3 · p. 8

4.4. View Synthesis on Flowers Light Fields

  • On Flowers light fields, MINE improves upon, with consistent gains as N increases.The comparison uses the central 8 × 8 angular grid, with its center image as source and four corner images as targets.Source: Table 4 · p. 8, §4.4 · p. 8

5. Conclusion

MINE generalizes MPI to continuous depth with NeRF, reconstructing and inpainting a camera frustum from one image for differentiable rendering of novel views and depth maps. The paper reports stronger single-image view synthesis results and near-state-of-the-art depth estimation without dense ground-truth depth supervision.

5. Conclusion

  • MINE significantly outperforms existing state-of-the-art single-image view synthesis methods and achieves near-state-of-the-art depth estimation without dense ground-truth depth supervision.Source: §5 · p. 8
  • MINE extends MPI to continuous depth by introducing NeRF, jointly reconstructing the camera frustum and inpainting occluded contents from a single image.Source: §5 · p. 8
  • Differentiable rendering converts the reconstructed frustum into novel-view RGB images and depth maps.Source: §5 · p. 8

A. MINE vs. pixelNeRF and GRF

MINE differs from pixelNeRF [59] and GRF [50] by reconstructing the source-camera frustum plane by plane, enabling efficient rendering and full-image training. The section also describes its input and viewing-direction handling and evaluates it on large-scale real-world datasets.

A. MINE vs. pixelNeRF and GRF

  • MINE models the source-camera frustum and reconstructs it per plane, while pixelNeRF [59] and GRF [50] model all 3D space and reconstruct it per ray.Source: A. MINE vs. pixelNeRF and GRF · p. 12
  • MINE needs Nplanes decoder passes to reconstruct the frustum, then renders each novel view by homography warping, whereas pixelNeRF [59] and GRF [50] require per-pixel, per-ray-point decoding.MINE's rendering complexity is independent of the number of target views; pixelNeRF [59] and GRF [50] scale with that number.Source: A. MINE vs. pixelNeRF and GRF · p. 12
  • MINE and pixelNeRF [59] take a single input image and run the encoder once, whereas GRF [50] takes multiple images and requires Nviews encoder inferences.Source: A. MINE vs. pixelNeRF and GRF · p. 12
  • MINE does not use viewing direction as input, but the paper proposes concatenating target-view per-ray directions after warping and predicting view-dependent radiance with a lightweight network.This extension adds Nviews network inferences in total and is described as still significantly faster than pixelNeRF [59] and GRF [50].Source: A. MINE vs. pixelNeRF and GRF · p. 12
  • Because MINE renders the full target image and disparity map during training, it can use dense supervision such as SSIM and edge-aware smoothness loss [11].The paper argues this supports generalization to large-scale real-world datasets, while NeRF-like methods cannot feasibly render the full image during training.Source: A. MINE vs. pixelNeRF and GRF · p. 12
  • MINE is verified on KITTI, NYU-V2, and RealEstate10k, while pixelNeRF [59] and GRF [50] do not present experiments with large-scale real-world data.Source: A. MINE vs. pixelNeRF and GRF · p. 13

B. Additional Implementation Details

The implementation uses a ResNet50 encoder and an expanded depth-decoder design to predict RGB and volume density, with multi-scale losses. Flowers Light Fields preprocessing fixes disparity and camera-shift settings to accommodate missing shift metadata.

Network architecture

  • The decoder extends the depth decoder in with two downsampling and two upsampling blocks, producing a 4-channel RGB-σ image from a ResNet50 encoder [15].RGB uses a Sigmoid layer and σ is the absolute value of the final output channel; the method is not restricted to this network architecture.Source: B. Additional Implementation Details · p. 13
  • LL1 and LSSIM apply only to output1, while Lsmooth applies to all four outputs under the multi-scale training strategy in.Source: B. Additional Implementation Details · p. 13

Pre-processing for Flowers Light Fields

  • For Flowers Light Fields, disparity spans [3.0, 0.03], and camera principal-point shifts are fixed following because metadata is absent.Views translate without rotation, and adjacent views in the extracted 8×8 grid are spaced 0.00128.Source: B. Additional Implementation Details · p. 13

C. Additional Qualitative Results

MINE’s additional qualitative results cover unseen outdoor and indoor scenes, complex geometries and textures, and flowers; supplementary videos show realistic novel views and smooth disparity maps under large camera motion.

C. Additional Qualitative Results

  • On unseen KITTI, RealEstate10K, and Flowers Light Fields scenes, MINE generalizes across outdoor and indoor environments and flowers with complex geometry.The qualitative results are shown in Figures 7–9.Source: C. Additional Qualitative Results · p. 13, Figure 7 · p. 14, Figure 8 · p. 15
  • Under large camera motion, supplementary videos for unseen RealEstate10K, KITTI, and iBims-1 [21] scenes show temporally consistent realistic images and smooth, accurate disparity maps.Each video is generated by rendering multiple novel views from a single input image and includes RGB and disparity-map videos.Source: C. Additional Qualitative Results · p. 13

Related papers

Loading 2103.14910v3…