Source-linked AI summary
PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image
Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, Jan Kautz
TL;DR
Piecewise planar reconstruction from a single RGB image is difficult because it is ill-posed and texture-poor surfaces require global scene understanding. PlaneRCNN combines detection, joint mask refinement, and view-consistency training, and introduces a fine-grained benchmark where it significantly outperforms current state-of-the-art methods across detection, segmentation, and reconstruction metrics.
Problem
Single-image piecewise planar reconstruction is ill-posed, while texture-poor planar structures require global scene understanding and prior methods miss small surfaces, use fixed plane counts, or generalize poorly.
Method
PlaneRCNN detects arbitrary numbers of planar regions, jointly refines their masks, and uses a warping loss enforcing consistency with another view during training.
Results
PlaneRCNN significantly outperforms current state-of-the-art methods across plane detection, segmentation, and reconstruction metrics on a new fine-grained benchmark.
Takeaways & Limitations
PlaneRCNN advances robust plane extraction by combining flexible detection, global mask refinement, and view-consistent training.
Takeaways & Limitations
Future work is needed to process image sequences during inference, including learning correspondences between plane detections.
Abstract
from arXiv · showhide
This paper proposes a deep neural architecture, PlaneRCNN, that detects and reconstructs piecewise planar surfaces from a single RGB image. PlaneRCNN employs a variant of Mask R-CNN to detect planes with their plane parameters and segmentation masks. PlaneRCNN then jointly refines all the segmentation masks with a novel loss enforcing the consistency with a nearby view during training. The paper also presents a new benchmark with more fine-grained plane segmentations in the ground-truth, in which, PlaneRCNN outperforms existing state-of-the-art methods with significant margins in the plane detection, segmentation, and reconstruction metrics. PlaneRCNN makes an important step towards robust plane extraction, which would have an immediate impact on a wide range of applications including Robotics, Augmented Reality, and Virtual Reality.
1. Introduction
PlaneRCNN addresses the challenges of reconstructing piecewise planar structure from a single RGB image with a detection-and-refinement architecture and a new fine-grained benchmark. It outperforms current state-of-the-art methods across plane detection, segmentation, and reconstruction metrics.
- Single-image piecewise planar reconstruction is ill-posed, and texture-poor surfaces require global scene understanding.Prior methods also miss small surfaces, require a fixed maximum number of planes, and generalize poorly across scene types.
- PlaneRCNN detects an arbitrary number of planar regions, estimates plane normals and per-pixel depths, and reconstructs 3D planes using camera intrinsics.Its detection network is built upon Mask R-CNN and produces an instance mask for each planar region.
- A refinement network jointly optimizes all segmentation masks to produce a more coherent scene interpretation.The module is designed to handle an arbitrary number of regions.
- A warping-loss module enforces reconstruction consistency with another view during training and improves plane-parameter and depthmap accuracy end-to-end.
- The new benchmark contains 100,000 ScanNet images with an average of 14.7 plane instances per image, compared with roughly 6 in the existing benchmark.Ground truth is generated from associated 3D scans, providing finer-grained plane annotations.
- PlaneRCNN outperforms the current state-of-the-art with significant margins on plane detection, segmentation, and reconstruction metrics.The paper reports that it can detect small planar surfaces and generalize well to new scene types.
2. Related Work
Prior approaches rely on multiple views, depth, fixed plane counts, or brittle refinement strategies. PlaneRCNN addresses these limitations with detection-based extraction and joint optimization of an arbitrary number of segmentation masks.
- Traditional 3D plane methods generally require multiple views or depth information, fitting planes to 3D points before assigning proposals to pixels.
- PlaneNet and PlaneRecover use pixel-wise segmentation with fixed plane counts, limiting reconstruction expressiveness and generalization across scene types.PlaneNet uses 10 planes and PlaneRecover uses 5.
- Detection-based methods for 3D understanding can represent objects through boxes, wire-frames, or template compositions, but coarse representations struggle with complex cluttered indoor scenes.
- Low-level CRF refinement localizes boundaries, while RNNs add global context; neither alone resolves the challenges of instance-aware joint segmentation refinement.
- Traditional graph-based methods jointly optimize instance masks through heuristic sequences that are often not robust.The paper proposes a refinement network that jointly optimizes an arbitrary number of masks on top of a detection network.
3. Approach
PlaneRCNN detects planar regions and their 3D parameters from a single RGB image, then jointly refines masks and uses nearby-view consistency during training. Its design handles arbitrary numbers of planes while improving geometric reconstruction through differentiable warping.
- Framework: PlaneRCNN combines a Mask R-CNN-based plane detector, joint mask-refinement network, and nearby-view warping-loss module.The three components respectively detect regions and parameters, jointly optimize masks, and enforce reconstruction consistency during training.
- Plane detection: Mask R-CNN enables detection of an arbitrary number of planes instead of requiring a preset maximum number.The paper contrasts this flexibility with PlaneNet's maximum of 10 planes and PlaneRecover's maximum of 5.
- Plane detection: Plane normals are estimated by selecting among 7 K-means-derived anchor normals and regressing a residual 3D vector.The residual is added to the selected anchor and normalized to produce a unit-length normal.
- Segmentation refinement: The refinement network jointly optimizes masks using information from all detected planes, while ConvAccu aggregates features across them.This avoids fixed-size mask concatenation that does not scale to many planes and can miss small planes.
- Warping loss: The warping loss builds depth and 3D coordinate maps, projects nearby-view points into the current frame, and penalizes transformed 3D distances.Nearby views are 20 frames ahead in ScanNet training videos; the warping module is used only during training, while testing uses a single image.
4. Benchmark construction
The benchmark is built from ScanNet RGB-D videos with modifications designed to preserve finer-grained planar instances. It averages 14.7 plane instances per image, more than twice the 6.0 instances in the PlaneNet dataset.
- Benchmark construction: 14.7 plane instances per image are represented on average, compared with 6.0 in the PlaneNet dataset.The benchmark is constructed from ScanNet RGB-D videos and is designed to recover more fine-grained planar regions.
- Fine-grained annotations: The benchmark retains smaller planar regions by reducing the area threshold from 1% of the image to 0.16%, or 500 pixels.Small planes are also retained when the total number of planes exceeds 10.
- Fine-grained annotations: Co-planar planes from different objects remain separate instance masks instead of being merged under one plane label.This preserves semantic distinctions that merging would remove.
- Quality control: Images with average planar-region depth discrepancy above 0.1m are removed to reduce camera-pose and ground-truth misalignment.The filtering targets tracking failures that degrade ScanNet camera poses.
5. Experimental results
Experiments evaluate PlaneRCNN on plane detection, geometric reconstruction, segmentation, ablations, and unseen datasets. The results show strong reconstruction accuracy, generalization, and benefits from the refinement, warping-loss, and occlusion-reasoning components.
- Ablation studies: PlaneRCNN recovers planar surfaces for small objects and can infer occluded surfaces, including floors behind tables and chairs, using layered depthmaps.The extended model adds a mask-prediction module for complete plane-instance masks.
- Qualitative evaluations: On unseen datasets, PlaneRCNN reconstructs planes across varying scene types, sizes, shapes, and textures without fine-tuning.Results are consistently better than both competitors in indoor scenes and better than PlaneRecover for near-camera planes in KITTI.
- Plane reconstruction accuracy: PlaneRCNN significantly outperforms competing methods in plane detection, except at small depth thresholds where MWS-G benefits from ground-truth depths.MWS-G nevertheless has lower recall in general because it does not robustly extract planar regions.
- Geometric accuracy: PlaneRCNN generalizes better without fine-tuning on NYUv2 and outperforms PlaneNet across every geometric metric after fine-tuning with ground-truth depths.The evaluation combines plane-parameter differences with standard depthmap metrics.
- Ablation studies: The segmentation refinement network fills gaps between adjacent planes, while the warping loss improves reconstruction accuracy using a second view.The ablation study reports positive contributions from all four PlaneRCNN components.
6. Conclusion and future work
PlaneRCNN detects planar regions, regresses plane parameters and instance masks, refines segmentation globally, and uses a neighboring training view to improve reconstruction. It outperforms competing methods on a fine-grained plane-annotation benchmark, while sequence processing remains future work.
- PlaneRCNN detects planar regions, regresses plane parameters and instance masks, globally refines segmentation masks, and uses a neighboring view during training.
- PlaneRCNN outperforms competing methods by a large margin on a new benchmark with fine-grained plane annotations.
- A simple modification enables PlaneRCNN to infer occluded surfaces and reconstruct layered depthmap models for new-view synthesis.
- Processing an image sequence during inference remains a future direction requiring learned correspondences between plane detections.
A. Refinement network architecture
The segmentation refinement network jointly optimizes extracted masks using an architecture detailed in Figure 10 and described in the refinement section.
- Figure 10 provides the detailed architecture of the segmentation refinement network.
B. More qualitative results
The paper reports additional qualitative PlaneRCNN results on ScanNet and comparisons with PlaneNet and PlaneRecover across unseen datasets without fine-tuning.
- Additional qualitative results for PlaneRCNN are shown on test scenes from the ScanNet dataset.
- Figure 10 shows the refinement network taking global and instance-specific information to refine an instance mask with a U-Net architecture.
- Figures 13 and 14 compare PlaneRCNN with PlaneNet and PlaneRecover on unseen datasets without fine-tuning.