Source-linked AI summary
Layer-structured 3D Scene Inference via View Synthesis
Shubham Tulsiani, Richard Tucker, Noah Snavely
TL;DR
The paper addresses the limitation that single-view 2.5D representations do not capture hidden scene content. It predicts a layered scene representation from one image and trains it through differentiable novel-view synthesis, with evaluations showing hidden-structure inference and competitive depth prediction.
Problem
Single-view 2.5D representations model visible pixels but do not capture hidden scene content, limiting reasoning about novel views and navigable regions.
Method
A CNN predicts a layered depth image from one image, and differentiable novel-view rendering enforces correspondence between predictions and observed views.
Results
The method predicts plausible occluded geometry and textures, while achieving an Absolute Relative error of 0.1856 versus 0.2079 by on KITTI depth evaluation.
Takeaways & Limitations
Layered representations extend single-view scene inference beyond visible-pixel depth to include occluded structures using indirect multi-view supervision.
Takeaways & Limitations
Because training relies only on multi-view supervision, learned geometry is restricted by the available motion across training views, and the method remains far from full 3D understanding.
Abstract
from arXiv · showhide
We present an approach to infer a layer-structured 3D representation of a scene from a single input image. This allows us to infer not only the depth of the visible pixels, but also to capture the texture and depth for content in the scene that is not directly visible. We overcome the challenge posed by the lack of direct supervision by instead leveraging a more naturally available multi-view supervisory signal. Our insight is to use view synthesis as a proxy task: we enforce that our representation (inferred from a single image), when rendered from a novel perspective, matches the true observed image. We present a learning framework that operationalizes this insight using a new, differentiable novel view renderer. We provide qualitative and quantitative validation of our approach in two different settings, and demonstrate that we can learn to capture the hidden aspects of a scene.
1 Introduction
The paper targets single-image scene representations that model hidden as well as visible content, using layered depth images and view synthesis to learn without direct 3D supervision.
- Motivation: Common 2.5D representations model only visible pixels, limiting novel-view prediction and reasoning about navigable scene regions.Depth maps are easy to learn because they preserve a one-to-one pixel mapping, but that same structure prevents extrapolation beyond visibility.
- Layered representation: The method predicts a layered depth image from one image, storing multiple ordered depths and associated colors along each viewing ray.LDIs represent multiple intersections with scene geometry, including foreground objects and backgrounds behind them.
- Learning framework: Multi-view images provide indirect supervision: the predicted LDI is rendered from a novel view and enforced to match the observed target image.This view-synthesis proxy addresses the lack of training data with direct layered-3D annotations.
- Validation: A differentiable forward-splatting renderer operationalizes the training procedure, which is validated qualitatively and quantitatively on synthetic data and an outdoor driving dataset.The evaluations use synthetic scenes with known ground-truth 3D and a real outdoor driving dataset.
2 Related Work
Prior work spans single-view geometric prediction, view-synthesis supervision, multi-view 3D reconstruction, and layered scene representations. This approach combines single-image inference with layered representations that reason about occluded content using indirect multi-view supervision.
- Single-view scene inference: Earlier depth and surface-normal methods primarily estimated geometry for visible pixels, whereas this work targets a richer layered scene representation.
- View-synthesis supervision: View-synthesis methods learn single-view depth from stereo or monocular-video supervision, while this work applies the same learning philosophy to a different layered representation and renderer.
- Multi-view supervised reconstruction: Single-view 3D reconstruction methods have shifted from full 3D supervision toward multi-view masks or depth, but volumetric occupancy is less suited to general scenes.
- Layered scene representations: Layer-based representations are used for parsimony, efficiency, and descriptive power, including approaches that group visible content into layers or capture hidden surfaces.
- Single-view scene inference: Unlike classical multi-image inference and prior single-view methods, the approach predicts a layered representation from one image while reasoning about occluded scene aspects.
3 Learning LDI Prediction
The method learns a layered depth image from one input image using multi-view supervision and differentiable forward rendering. Its layered representation models visible and occluded scene structure, enabling novel-view synthesis rather than only re-rendering visible pixels.
- Learning setup: Training uses source-target image pairs from the same scene with known camera transformations, without direct supervision for the scene’s 3D structure.The setup can use calibrated camera rigs or an agent’s egomotion to obtain the relative camera transformation.
- Representation: An LDI stores texture and inverse depth across ordered layers, with the first layer representing visible content and later layers capturing occluded structure.Depth increases across layers, while disparity decreases; the method restricts experiments to two layers but supports the general formulation.
- View synthesis supervision: Given a source image, the CNN predicts an LDI whose rendered target view is compared with the observed target image to learn meaningful layered structure.The target view can expose disoccluded content, so the predicted LDI must represent more than the structure visible in the source image.
- Novel-view rendering: Unlike prior inverse-warping approaches that re-render the source view and constrain only jointly visible pixels, this method forward-renders novel views from the layered representation.Forward rendering is required to learn both visible and occluded scene structure.
- Differentiable rendering: The differentiable renderer treats each LDI pixel as a textured point, forward-projects it, resolves occlusions with soft z-buffering, and averages projected colors.The forward projection, depth-dependent weighting, and final color computation are differentiable, allowing the renderer to serve as the learning signal.
- Training objective: The view-synthesis loss ignores pixels near image boundaries and adds a minimum-per-layer loss that helps the background layer learn despite foreground occlusion.The boundary heuristic removes target pixels that cannot be explained from the source image, while the minimum-per-layer term requires at least one layer to explain each observed target pixel.
4 Experiments
Experiments test single-image LDI prediction on synthetic scenes and KITTI stereo data, evaluating novel-view synthesis, hidden geometry, and visible-depth accuracy.
- Synthetic data: Synthetic scenes vary camera viewpoint enough to expose content hidden behind foreground objects, while diverse object shapes and textures increase reconstruction difficulty.Training uses procedurally generated room-like scenes with sampled PASCAL VOC objects and SUN 2012 wall textures.
- Synthetic data: The learned model predicts foreground geometry and uses a background layer to infer occluded geometry and plausible textures, with errors where content remains unseen.Reported errors include incorrect background predictions beneath wide objects and spurious details outside the room.
- Synthetic data: Two-layer LDI view synthesis slightly outperforms a single-layer model overall, with a more significant advantage on dis-occluded target pixels.The comparison uses mean pixel-wise ℓ1 error between ground-truth novel views and rendered views.
- Synthetic data: Synthetic geometry evaluation compares predicted inverse depths with ground truth, using foreground predictions as the single-layer model’s background reference.This is explicitly a harsh comparison because the single-layer model does not infer hidden background depth.
- KITTI: On KITTI, the model predicts foreground depth and hallucinates occluded geometry and texture, but uses the background layer less extensively than in the synthetic setting.The inferred background mainly covers thin structures or boundaries of larger objects, consistent with the small stereo viewpoint change.
- KITTI: KITTI evaluation finds slightly better novel-view reconstruction with two layers, especially for dis-occluded pixels, and Absolute Relative error 0.1856 versus 0.2079 for Zhou et al.The authors describe competitive depth prediction as encouraging but not their central goal.
5 Discussion
The method advances beyond 2.5D scene representations toward reasoning about occluded structures, but its learned geometry remains constrained by available training-view motion and does not constitute full 3D understanding.
- The learned layer-structured representation extends beyond common 2.5D representations by supporting reasoning about occluded scene structures.
- Because training uses only multi-view supervision, the learned geometry is restricted by the extent of motion available across training views.
- Future extensions include grouping, semantics, and semantic priors such as treating roads as flat.
- The authors characterize the work as a step beyond 2.5D prediction while remaining far from full 3D scene understanding.
A1. Additional Visualizations
The KITTI evaluation uses stereo-derived dis-occlusion masks to assess predictions in regions hidden in one view, especially around thin objects and object boundaries.
- KITTI dis-occlusion masks are obtained with an occlusion-aware stereo algorithm and used to evaluate view synthesis error for estimated dis-occluded pixels.
- The masks primarily cover thin objects such as poles and object boundaries such as cars.
- Because the masks are stereo-derived approximations, they can sometimes be erroneous.
A2. Ablations
Ablations show that min-view-synthesis training, disjoint layer-specific prediction branches, and smoothness regularization are important for meaningful layered predictions, especially the background layer.
- The ablations remove the min-view-synthesis loss, disjoint layer branches, or smoothness loss to test these training-design choices.
- Without disjoint prediction branches, the model cannot learn meaningful background amodal texture, although it can learn simpler amodal geometry.
- Including the min-view-synthesis term and separate layer-specific branches is important for meaningful background-layer prediction.
- The smoothness prior prevents undesirable artifacts.