Source-linked AI summary
PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image
Chen Liu, Jimei Yang, Duygu Ceylan, Ersin Yumer, Yasutaka Furukawa
TL;DR
Single-image piece-wise planar reconstruction requires structured plane geometry that ordinary depth predictors do not provide. PlaneNet directly predicts plane parameters, probabilistic masks, and non-planar depth, using ScanNet-derived supervision. The authors report better plane segmentation and improved single-image depth prediction, while noting failures under low texture and clutter.
Problem
Single-image piece-wise planar depth reconstruction lacks sufficient methods because it must infer structured plane parameters and segmentation masks with unknown plane count and ordering.
Method
PlaneNet is an end-to-end neural architecture that directly predicts plane parameters, probabilistic segmentation masks, and non-planar depth from a single RGB image.
Results
PlaneNet significantly outperforms competing baselines in plane segmentation and advances the state of the art in single-image depth prediction.
Takeaways & Limitations
The paper demonstrates that a single RGB image can support structured piece-wise planar reconstruction with competitive plane segmentation and depth accuracy.
Takeaways & Limitations
PlaneNet can generate co-planar duplicates, confuse similarly colored objects and walls, and miss structures under clutter or limited texture.
Abstract
from arXiv · showhide
This paper proposes a deep neural network (DNN) for piece-wise planar depthmap reconstruction from a single RGB image. While DNNs have brought remarkable progress to single-image depth prediction, piece-wise planar depthmap reconstruction requires a structured geometry representation, and has been a difficult task to master even for DNNs. The proposed end-to-end DNN learns to directly infer a set of plane parameters and corresponding plane segmentation masks from a single RGB image. We have generated more than 50,000 piece-wise planar depthmaps for training and testing from ScanNet, a large-scale RGBD video database. Our qualitative and quantitative evaluations demonstrate that the proposed approach outperforms baseline methods in terms of both plane segmentation and depth estimation accuracy. To the best of our knowledge, this paper presents the first end-to-end neural architecture for piece-wise planar reconstruction from a single RGB image. Code and data are available at https://github.com/art-programmer/PlaneNet.
1. Introduction
Piece-wise planar understanding is important for interpreting indoor scenes and supporting robotics and augmented-reality applications, but single-image reconstruction remains difficult because it requires structured geometry. PlaneNet addresses this gap with an end-to-end network that predicts planes, masks, and non-planar depth from one RGB image.
- Motivation: Piece-wise planar geometry captures dominant indoor surfaces such as floors, walls, ceilings, and table-tops.The representation supports scene understanding for robotics and augmented reality, including movement planning and virtual-object placement.
- Problem: Single-image piece-wise planar reconstruction has received little attention despite progress in depth inference and room-layout estimation.The task is challenging because the number and output ordering of planes are unknown.
- Approach: PlaneNet directly predicts plane parameters and probabilistic plane segmentation masks from a single RGB image.Its loss is order-agnostic, masks can deactivate unused plane slots, and a separate branch predicts non-planar depth.
- Results: Qualitative and quantitative evaluations report significantly better plane segmentation and competitive or superior single-image depth accuracy.The depth comparison is against techniques specifically trained for single-image depth inference.
2. Related work
Prior approaches reconstruct piece-wise planar geometry through multi-view inference, generic depth prediction, or specialized layout estimation. PlaneNet instead directly predicts planes and segmentations from a single RGB image for general piece-wise planar scenes.
- Multi-view piece-wise planar reconstruction: Traditional multi-view methods reconstruct 3D points, fit plane hypotheses, and solve a global inference problem.Their task is to infer plane parameters and assign a plane ID to each pixel.
- PlaneNet: PlaneNet differs by directly inferring plane parameters and plane segmentations from one RGB image.This contrasts with multi-view pipelines that first reconstruct precise 3D points and generate plane hypotheses.
- Learning based depth reconstruction: Most learning-based depth methods output depth values without detecting or segmenting individual planes.A closer method enforces planarity but predicts only a binary planar-surface mask, not plane parameters or instance-level segmentation.
- Layout estimation: RoomNet jointly predicts room-layout types and corner locations but does not apply to general piece-wise planar scenes.Room-layout methods commonly rely on hand-crafted image features and global optimization.
- Line analysis: Line-based reconstruction methods use line directions, junctions, or image segments to recover 3D structure from drawings or real images.These approaches target architectural scenes or indoor panoramas among other settings.
3. PlaneNet
PlaneNet uses a shared dilated-residual backbone with three coordinated outputs: planar parameters, non-planar depth, and probabilistic segmentation masks. Its design handles variable plane counts, ordering ambiguity, and joint depth reconstruction.
- Architecture: A Dilated Residual Network backbone provides high-resolution features for global and pixel-wise prediction tasks.Three output branches are composed from the final feature maps.
- Outputs: The network predicts K plane parameters, a non-planar depthmap, and K+1 probabilistic segmentation masks.The non-planar surface is treated as the additional output surface alongside the K planes.
- Architecture: Figure 2 summarizes the three prediction outputs produced from a single RGB image.The outputs are plane parameters, probabilistic segmentation masks, and a non-planar depthmap.
- Plane parameter branch: A fixed K=10 output accommodates unknown plane counts by allowing unused plane predictions to have zero segmentation masks.The training data contains at most 10 planes for most examples.
- Plane parameter branch: An order-agnostic Chamfer-distance loss trains the regressed plane parameters despite unknown plane ordering.Each plane parameter encodes the 3D coordinate of the point nearest the camera center on that plane.
- Non-planar depth branch: The non-planar depth branch uses probability-weighted squared depth errors between the ground-truth depth and predicted planes or non-planar depth.The loss exploits the entire ground-truth depthmap rather than defining a loss only for non-planar regions.
4. Datasets and implemenation details
The authors construct piece-wise planar supervision from ScanNet meshes and implement PlaneNet as a high-resolution deep network trained on RGB indoor scenes. The dataset contains 51,000 generated examples split into training and testing sets.
- Dataset: 51,000 ground-truth piece-wise planar depthmaps were generated from ScanNet, with 50,000 for training and 1,000 for testing.ScanNet is a large-scale indoor RGB-D video database.
- Ground-truth generation: Ground-truth depthmaps are produced by fitting planes to consolidated meshes and projecting them back to individual frames.The process also exploits associated semantic annotations to address holes and far-distance quality degradation in raw depthmaps.
- Ground-truth generation: Plane extraction repeatedly applies RANSAC to semantic sub-meshes until 90% of points are covered, using a 5 cm inlier threshold.Training and testing examples are sampled from separate ScanNet scene splits.
- Implementation: The implementation uses TensorFlow and DeepLab with a 101-layer dilated-convolution ResNet, Adam optimization, and 256x192 outputs.The learning rate is initialized at 0.0003, and outputs include plane masks and a non-planar depthmap.
5. Experimental results
PlaneNet directly predicts piece-wise planar structure from a single RGB image and is evaluated against 3D-point baselines, depth predictors, and room-layout methods. It achieves stronger plane segmentation and depth accuracy, while retaining consistent ordering for major surfaces but exhibiting identifiable failure modes.
- Plane segmentation accuracy: PlaneNet significantly outperforms competing methods on both plane-recall metrics when inferred depthmaps are used, and surpasses some methods using ground-truth depthmaps.The metrics measure correctly predicted ground-truth planes and pixels within correctly predicted plane overlaps.
- Plane segmentation accuracy: Qualitatively, PlaneNet produces better plane boundaries and fewer redundant planes than methods operating on noisy inferred depthmaps.Existing methods often miss boundaries where intensity edges are weak.
- Depth reconstruction accuracy: The network can also produce a standard per-pixel depthmap variant, PlaneNet (Depth rep.), by disabling the plane-parameter and segmentation branches.This variant predicts the entire depthmap through the non-planar depth branch.
- Depth reconstruction accuracy: PlaneNet outperforms state-of-the-art single-image depth inference techniques on NYUv2, with larger improvements when its piece-wise planar representation is enforced.Evaluation covers planar regions, boundary regions, and the entire valid image area.
- Plane ordering consistency: PlaneNet learns consistent ordering for major surfaces such as floors and walls, enabling a simple room-layout estimation procedure, although ordering degrades for small objects or extreme camera angles.Room layouts are constructed from selected ceiling, floor, and wall planes and evaluated on NYUv2 303, where performance is comparable to existing techniques.
- Failure modes: Reported failure modes include splitting nearly co-planar surfaces, confusing white floor objects with walls, and missing wall columns near occluding objects.These examples identify concrete visual conditions where reconstruction remains unreliable.
6. Applications
The paper demonstrates texture insertion and replacement applications using its piece-wise planar representation, with predicted plane normals supporting surface-aligned texture coordinates.
- The piece-wise planar representation enables texture insertion and replacement for augmented-reality image editing.These applications are illustrated with input images, plane segmentation results, and edited images.
- Predicted plane normals are used with Manhattan directions to define UV coordinates for each planar region.The second UV axis is computed as the cross product of the first axis and the plane normal.
- New textures can be inserted by alpha-blending or used to completely replace existing textures.
7. Conclusion and future work
PlaneNet is presented as the first deep neural architecture for piece-wise planar depthmap reconstruction from a single RGB image. It directly infers plane parameters and probabilistic segmentation masks, outperforming competing baselines in plane segmentation and advancing single-image depth prediction.
- PlaneNet is presented as the first deep neural architecture for piece-wise planar depthmap reconstruction from a single RGB image.
- PlaneNet directly infers a set of plane parameters and their probabilistic segmentation masks.
- The approach significantly outperforms competing baselines in plane segmentation and advances the state of the art in single-image depth prediction.