Source-linked AI summary
3D Photography using Context-aware Layered Depth Inpainting
Meng-Li Shih, Shih-Yang Su, Johannes Kopf, Jia-Bin Huang
TL;DR
Single-image RGB-D novel-view synthesis must fill parallax-induced disocclusions without relying on rigid layers or inefficient representations. The paper completes a connectivity-aware LDI through local, context-aware color-and-depth inpainting, producing efficiently renderable 3D photos with fewer reported visual artifacts than state-of-the-art techniques, while remaining limited by difficult depth estimation and reflective or transparent surfaces.
Problem
Single RGB-D images leave occluded regions missing, causing gaps or stretched content under parallax and challenging fixed-layer or inefficient representations.
Method
The method recursively inpaints local LDI regions using explicit pixel connectivity, spatially adaptive context, and learned edge, color, and depth synthesis.
Results
The method produces 3D photos with synthesized texture and structures in occluded regions, fewer visual artifacts than state-of-the-art novel-view methods, and efficient standard-engine rendering.
Takeaways & Limitations
Adaptive layered representations support 3D photography across varied everyday scenes without requiring a fixed number of layers.
Takeaways & Limitations
The method struggles with complex structures when estimated depth is unsatisfactory and does not handle reflective or transparent surfaces well.
Abstract
from arXiv · showhide
We propose a method for converting a single RGB-D input image into a 3D photo - a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show fewer artifacts compared with the state of the arts.
1. Introduction
The paper targets single-image 3D photography, where parallax exposes missing content in disoccluded regions. It introduces a connectivity-aware, context-conditioned LDI inpainting method that adapts layer structure to scene depth complexity.
- Motivation: Single-image RGB-D capture makes 3D photography more effortless than elaborate multi-image or specialized-hardware setups.Dual-lens phones can provide the RGB-D input needed for novel-view rendering.
- Problem: Naive depth warping leaves gaps or stretches content in parallax-induced disocclusions, motivating better extrapolation methods.Disocclusions are identified as the most salient features in rendered novel views.
- Approach: The method uses an LDI with explicit pixel connectivity and solves many local inpainting problems iteratively instead of applying a global CNN.Spatially adaptive context regions are extracted from local LDI connectivity, and synthesized regions are fused back recursively.
- Contribution: Unlike rigid fixed-layer representations, the method generates a varying number of layers that adapts to local depth complexity.The LDI remains compact while supporting arbitrary depth complexity.
- Outcome: The resulting 3D photos synthesize texture and structures in regions occluded from the input and were validated across varied photos.The introduction reports validation on a wide variety of photos captured in different situations.
2. Related Work
Prior work spans light fields, multi-plane images, layered depth images, and image inpainting, but faces efficiency, locality, or extrapolation limitations. This paper distinguishes its method by locally adapting both hole and context regions while jointly inpainting color, depth, and discontinuities.
- Novel-view representations: Light fields can render photorealistic novel views but generally require many input images.This limits their suitability when capture should use fewer images.
- Novel-view representations: Multi-plane images use fixed-depth RGB-α layers, but sloped surfaces often reproduce poorly and excessive planes increase memory, storage, and rendering costs.The fixed depth discretization is the source of the sloped-surface limitation.
- Layered representations: Fixed-number layered representations do not preserve locality well because layer content changes abruptly across depth discontinuities.Such abrupt changes destroy locality in convolutional receptive fields.
- Image inpainting: The proposed inpainting operates locally around each depth discontinuity using adaptive hole and context regions rather than static image-wide regions.The context follows directly connected LDI pixels and halts at silhouettes.
- Depth inpainting: Unlike conventional visible-surface depth completion, the method inpaints hidden-surface depth together with color and missing depth discontinuities.Its model builds on two-stage structure-content inpainting approaches.
3. Method
The method converts an RGB-D image into a connectivity-aware LDI, detects linked depth edges, and recursively inpaints occluded background regions. Local context and synthesis regions support CNN-based generation of aligned edge, color, and depth content.
- Representation: The LDI stores potentially many pixels at each lattice position and explicitly links neighboring pixels except across depth discontinuities.Each LDI pixel stores color and depth plus up to one neighbor in each cardinal direction.
- Image preprocessing: The pipeline initializes a fully connected single-layer LDI, sharpens depth, detects discontinuities, and groups them into connected depth edges.Short segments are removed after connected-component cleanup.
- Robustness: Dilating the synthesis region by 5 pixels compensates for imperfect depth-edge alignment and helps reduce artifacts.The adjustment targets misalignment around occlusion boundaries.
- Context and synthesis regions: For each edge, the algorithm cuts cross-edge connections, retains the background silhouette for inpainting, and creates a contiguous synthesis region.The synthesis region begins next to disconnected silhouette pixels and expands through unvisited neighboring coordinates.
- Context and synthesis regions: The context region follows connected LDI links and stops at silhouettes, restricting the inpainting network to directly relevant known content.Alternating flood-fill expansion produces separate context and synthesis regions.
- Inpainting model: A three-network model inpaints edge structure, color, and depth so the generated depth remains aligned with generated color.The extracted regions are locally image-like, allowing standard image-inpainting architectures.
4. Experimental Results
Experiments evaluate the inpainting pipeline through visual comparisons, ablations, quantitative testing, and varied depth inputs. The method produces plausible content around difficult depth structures and improves perceptual quality.
- Evaluation setup: The experiments include visual comparisons, quantitative evaluation on RealEstate10K, ablations, and tests with depth maps from different sources.The evaluation covers implementation, state-of-the-art comparisons, quantitative metrics, model components, and depth-map robustness.
- Ablation and visual analysis: Edge-guided depth inpainting produces more accurate structure, particularly in depth-complex regions such as T-junctions.The synthesized novel view is highlighted in the blue box.
- Visual comparisons: Our method synthesizes plausible structures around depth boundaries, while MPI-based baselines exhibit artifacts and LLFF shows ghosting during view extrapolation.The comparison uses challenging RealEstate10K examples and DPSNet-generated input depth maps.
- Visual comparisons: Our results generally fill synthesis regions with visually plausible contents and structures, whereas Facebook 3D Photo artifacts become visible for larger occluded regions.Facebook 3D Photo performs well when small or thin occluded regions are revealed.
- Ablation and visual analysis: The proposed color inpainting model yields better perceptual quality, while edge-guided depth inpainting provides only minor numerical-metric improvement.The color-inpainting ablation is illustrated in Figure 11.
- Depth-map robustness: The method handles depth maps from MegaDepth, MiDas, and Kinect reasonably well.The resulting 3D photos are shown for all three depth sources.
5. Conclusions
The paper presents an algorithm that creates 3D photography from a single RGB-D image through context-aware color and depth inpainting. Across varied everyday scenes, it produces fewer visual artifacts than state-of-the-art novel-view synthesis techniques.
- Conclusion: The algorithm creates 3D photography from a single RGB-D image using a completed layered depth image built through context-aware color and depth inpainting.The method is validated on a wide variety of everyday scenes.
- Conclusion: The experimental results show considerably fewer visual artifacts than state-of-the-art novel-view synthesis techniques.The authors suggest this could broaden access to immersive 3D photography.
3D Photography using Context-aware Layered Depth Inpainting Supplementary Material
The supplementary material provides an additional quantitative comparison on the RealEstate10K dataset.
- Quantitative results: Table 4 reports a quantitative comparison on the RealEstate10K dataset.The supplied passage identifies the table but provides no metric values or outcomes.
6. Additional Quantitative Results
The paper further evaluates PB-MPI using different numbers of depth layers and reports the results in Table 4.
- Additional quantitative results: PB-MPI is evaluated with various numbers of depth layers, with results reported in Table 4.The supplied passage does not include the corresponding metric values.
7. Visual Results
The paper presents comparisons with state-of-the-art novel-view synthesis methods and includes ablations isolating the contributions of its components.
- Comparisons with the state-of-the-arts: The visual results compare rendered 3D photos against state-of-the-art novel-view synthesis algorithms.The collection also demonstrates novel-view synthesis for legacy photos.
- Ablation studies: Ablation studies render 3D photos under the same settings to examine how each proposed component contributes to synthesized-view quality.The ablations follow the settings described in Section 4.4.
8. Implementation Details
The implementation uses U-Net-based color and depth inpainting, a separate edge model, region-aware reconstruction objectives, and distinct losses for color, depth, and edges.
- Model architectures: The color and depth inpainting models use the U-Net architecture, while the edge model uses a separate generator-discriminator design.Inputs are adapted by changing channels, and synthesis-region values are masked for the models.
- Training objective: The reconstruction losses separately supervise the context and synthesis regions using binary masks and the ground-truth image.The masks select pixels for the corresponding loss terms.
- Training objective: The color model combines context and synthesis reconstruction with perceptual, style, and total-variation losses.Its objective is L = Lcontext + 6Lsynthesis + 0.05Lperceptual + 120Lstyle + 0.01Ltv.
- Training objective: The depth model uses only context and synthesis reconstruction losses, while the edge model follows the training protocol of prior work.The models therefore use different objective configurations.
9. Failure cases
Failures arise when estimated depth cannot represent complex thin structures or reflective and transparent surfaces, limiting plausible rendered motion parallax.
- Depth-estimation failures: Single-image depth estimation struggles with thin and complex structures, producing overly smooth depth maps.The failure examples use depth estimated by MegaDepth.
- Depth-estimation failures: For complex structures, the method may fail to produce satisfactory results with plausible motion parallax.This limitation follows from the difficulty of estimating depth or disparity from a single image.
- Surface limitations: Because the method uses an explicit depth map, it does not handle reflective or transparent surfaces well.The paper illustrates such cases among its failure examples.