PaperlayerView exact revision on arXiv

Source-linked AI summary

FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans

Chen Liu, Jiaye Wu, Yasutaka Furukawa

arXiv:1804.00090v1cs.CV

TL;DR

FloorNet addresses automatic floorplan reconstruction from RGBD video captured while walking through large indoor spaces, where existing approaches struggle with the required 3D extent and semantics. It combines 3D-point, top-down, and RGB branches with cross-branch feature sharing, and introduces a 155-unit smartphone benchmark. The authors report effectiveness over competing methods, while identifying missing corners and the need for higher-level constraints and more scans as remaining limitations.

  • Problem

    Automatic reconstruction of concise, semantically meaningful floorplans from RGBD video across large 3D indoor spaces remains challenging.

  • Method

    FloorNet fuses PointNet 3D points, a top-down point-density CNN, and an RGB CNN, sharing features before Integer Programming recovers vector-graphics floorplans.

  • Results

    Extensive qualitative and quantitative evaluations demonstrate effectiveness over competing methods on a benchmark of 155 residential units.

  • Takeaways & Limitations

    The hybrid architecture and benchmark provide a basis for vector-graphics floorplan reconstruction from smartphone RGBD video.

  • Takeaways & Limitations

    Missing room corners from clutter or incomplete scans can cause reconstruction failures, motivating higher-level constraints and additional training scans.

Abstract

from arXiv · show

The ultimate goal of this indoor mapping research is to automatically reconstruct a floorplan simply by walking through a house with a smartphone in a pocket. This paper tackles this problem by proposing FloorNet, a novel deep neural architecture. The challenge lies in the processing of RGBD streams spanning a large 3D space. FloorNet effectively processes the data through three neural network branches: 1) PointNet with 3D points, exploiting the 3D information; 2) CNN with a 2D point density image in a top-down view, enhancing the local spatial reasoning; and 3) CNN with RGB images, utilizing the full image information. FloorNet exchanges intermediate features across the branches to exploit the best of all the architectures. We have created a benchmark for floorplan reconstruction by acquiring RGBD video streams for 155 residential houses or apartments with Google Tango phones and annotating complete floorplan information. Our qualitative and quantitative evaluations demonstrate that the fusion of three branches effectively improves the reconstruction quality. We hope that the paper together with the benchmark will be an important step towards solving a challenging vector-graphics reconstruction problem. Code and data are available at https://github.com/art-programmer/FloorNet.

1 Introduction

FloorNet addresses the challenge of reconstructing concise, semantically meaningful floorplans from RGBD video spanning large 3D spaces. It combines three complementary branches and introduces a 155-unit benchmark with complete floorplan annotations.

  • Motivation: Existing indoor scanners produce detailed geometry but not concise floorplans that preserve scene segmentation and semantics.Floorplans encode both geometric structure and semantic information such as room types and object arrangements.
  • Challenges: Large multi-room floorplan reconstruction challenges existing methods because projections lose height, PointNet lacks local neighborhoods, and multi-view methods struggle with large scenes and complex motions.These limitations motivate combining representations rather than relying on a single domain.
  • Approach: FloorNet fuses PointNet over 3D points, a top-down point-density CNN, and an RGB CNN to predict floorplan geometry and semantics before vector reconstruction.The branches exchange features so the architecture can process full RGBD videos covering large 3D scenes with complex camera motions.
  • Benchmark: The benchmark contains RGBD streams and complete floorplan annotations for 155 residential houses or apartments acquired with Google Tango phones.Annotations include architectural structures, icons, and room types.
  • Contributions: The paper contributes a hybrid RGBD architecture and a vector-graphics floorplan benchmark intended for applications including digital mapping, real estate, and civil engineering.The benchmark targets a reconstruction problem that existing indoor scene databases do not address.

2 Related work

Prior work provides detailed scans, primitive-based reconstructions, semantic understanding, and raster-floorplan recovery, but lacks a robust learning-based method for vector-graphics floorplans from large RGBD videos. FloorNet is positioned to address this gap with consumer-grade smartphone data and a hybrid representation.

  • Indoor scene reconstruction: Dense indoor reconstructions preserve detail but fall short as CAD models because they lack concise geometry and appropriate architectural segmentation or semantics.This distinction separates detailed scanning from floorplan-oriented reconstruction.
  • Indoor scene reconstruction: Primitive-based methods can produce promising results but depend critically on detecting geometric primitives from noisy and incomplete 3D data.The cited approach instead performs global analysis of the entire scene.
  • Vector-graphics reconstruction: Prior semantic and scene-understanding methods do not provide a robust learning-based solution for vector-graphics floorplan reconstruction.The paper presents both a method and benchmark with ground-truth annotations for this target.
  • Raster-floorplan recovery: Raster-floorplan recovery uses an Integer Programming formulation, whereas FloorNet must process RGBD video covering a large 3D space with complex camera motions.FloorNet shares the reconstruction target and final formulation but requires a different input-processing architecture.
  • 3D deep learning: Volumetric CNNs face data sparsity and computational-cost challenges, while multi-view CNNs remain limited by regular camera arrangements and are difficult to extend to larger scenes.These constraints motivate combining 3D, top-down, and image-based representations.
  • 3D deep learning: PointNet exploits 3D coordinates without quantization errors but lacks explicit local spatial reasoning, while hierarchical point methods add computation and are limited at object scale.Scene-scale processing by splitting into small regions can hurt global reasoning.
  • Indoor scan datasets: Existing indoor scan datasets often contain only one or two rooms or focus on semantic annotations rather than vector-graphics reconstruction.Matterport-based datasets also require specialized hardware, while FloorNet targets smartphone RGBD capture.

3 FloorNet

FloorNet converts RGBD video with camera poses into pixel-wise floorplan geometry and semantics, then applies Integer Programming to recover vector graphics. Its hybrid design shares features among 3D-point, top-down, and image branches.

  • Intermediate representation: The intermediate representation encodes geometry with heatmaps for room corners, icon corners, and door/window endpoints, and semantics with room-type and icon-type probability maps.The representation is converted into a final floorplan using an Integer Programming formulation.
  • Triple-branch hybrid design: FloorNet combines unmodified PointNet, a fully convolutional top-down point-density branch, and an image branch based on semantic segmentation and room-layout CNNs.The contribution is the hybrid design and feature sharing rather than new branch architectures.
  • Feature sharing: PointNet features are pooled into corresponding top-down cells, while floorplan features are unpooled back to the associated 3D points.These operations connect unordered 3D features with spatially organized floorplan features.
  • Feature sharing: Image features are unprojected into 3D using depthmaps and camera poses, then pooled into the corresponding floorplan cells.Image-branch pooling is performed every 10 frames in the video sequence.
  • Prediction and losses: The network predicts 256 × 256 geometry heatmaps and semantic maps using sigmoid cross entropy for geometry and pixel-wise softmax cross entropy for classification.Geometry targets assign value 1.0 within an 11-pixel-radius disk around each ground-truth pixel.

4 Floorplan reconstruction benchmark

The benchmark pairs smartphone RGBD scans of residential units with complete vector-floorplan annotations and evaluates reconstruction at low, mid, and high levels of geometric and semantic consistency. Its metrics cover local detections, object and room overlap, and architectural relationships.

  • Dataset: The benchmark retains 155 residential scans after quality filtering and annotates room corners, walls, object icons, doors, windows, and related floorplan information.The scans were acquired with Google Tango phones from roughly two hundred residential units.
  • Evaluation: Low-level evaluation measures room-corner precision and recall, counting a detection when it is within 10 pixels of the ground truth and closest among other corners.This metric targets local geometric accuracy.
  • Dataset: The benchmark visualization aligns subsampled video frames, colored 3D point clouds, and ground-truth vector-graphics floorplans.Rooms are rendered by type and objects by canonical icons.
  • Evaluation: Mid-level evaluation measures precision and recall for openings, object icons, and rooms using endpoint distance or IOU thresholds.Success requires less than 10 pixels for opening endpoints, IOU above 0.5 for objects, and IOU above 0.7 for rooms.
  • Evaluation: High-level evaluation measures the ratio of rooms with correct neighboring-room connections, neighboring-room overlap, and room type.A correct relationship requires door-based connectivity, IOU above 0.5 for the room and neighbors, and the correct room type.
  • Dataset statistics: Dataset statistics report average and standard deviation for room counts, icon counts, opening counts, room-corner counts, and total area.The table organizes these five statistics from left to right.

5 Implementation details

FloorNet training uses augmented RGBD scans, staged loss optimization, and heuristic post-processing for noisy large-scale sensor data. The model trains efficiently and generalizes from a relatively small scan collection, while icon-loss training remains prone to overfitting.

  • DNN Training: 135 scans are split into 135 training and 20 testing examples, with random scaling and four-way z-axis rotation applied during training.Scaling uses factors from 0.5 to 1.5; rotations are 0°, 90°, 180°, or 270°.
  • DNN Training: All loss functions are not trained jointly because icon loss causes overfitting; instead, losses are optimized one by one with early stopping for icon training.Icon-loss training is capped at 600 iterations and monitored using testing loss.
  • DNN Training: 1,620,000 augmented training samples are consumed on average, while training takes around 2 hours on a TitanX GPU.The sample count is 135 samples × 600 iterations × 20 batches.
  • DNN Training: Despite the small number of 3D scans, FloorNet generalizes well, likely because its pixel-wise predictions are mostly low-level vision tasks and training uses augmentation.Each scan contains about 10 object icons, 10 openings, and a few dozen room corners.
  • Post-processing: The Integer Programming formulation is augmented with primitive-generation and confidence-weighting heuristics to handle noisy large-scale sensor predictions.Room-corner components yield wall candidates, while wall and opening weights are based on semantic heatmap confidence minus 0.5.

6 Experiments

FloorNet reconstructs complex vector-graphics floorplans, while experiments show that noisy or incomplete 3D data and missing corners remain important failure sources. Comparisons indicate that combining branches and sharing features improves reconstruction quality across metrics.

  • Qualitative results: FloorNet successfully recovers complex floorplan geometries and door connectivities, but icon detection is a major failure mode.The authors attribute icon errors to overfitting in the icon loss and the greater data requirements of object detection.
  • Qualitative results: Poor-quality or noisy 3D points produce fake rooms and inaccurate reconstructed room shapes.These intermediate outputs illustrate the sensitivity of reconstruction to input-point quality.
  • Comparisons: FloorNet with all branches produces more complete and accurate floorplans than OctNet and single-branch variants.The comparison includes OctNet and variants using only one FloorNet branch.
  • Quantitative comparisons: The floorplan branch is the most informative, whereas PointNet-only and image-only variants do not work well.A PointNet-only model trained on separate 1m × 1m blocks performs much worse after merging, while OctNet performs poorly on the high-level metric.
  • Ablation: Feature sharing across branches improves results overall, especially on mid- to high-level metrics.The ablation disables intra-branch pooling and unpooling operations to evaluate this design choice.
  • Commercial comparison: 1.66 pixels versus 2.72 pixels: FloorNet has lower average wall-line distance than the Tango Navigator App.The comparison evaluates geometry only because the commercial app does not provide room segmentations, room types, object detection, or CAD-quality geometry.

7 Conclusion

FloorNet addresses vector-graphics floorplan reconstruction from RGBD videos and introduces a benchmark for this problem. The authors identify integrating higher-level constraints within the network and collecting more scans as future directions.

  • FloorNet reconstructs vector-graphics floorplans from RGBD videos with camera poses using a hybrid of three neural architectures.The approach exploits complementary strengths to process large 3D spaces with complex camera motions.
  • The paper introduces a benchmark for vector-graphics floorplan reconstruction, a problem missing from recent indoor scene databases.
  • Future work includes learning higher-level constraints inside DNNs instead of applying them through separate postprocessing.The authors note that learning these constraints likely requires more training data and additional scans.
  • More than 90% of houses in North America lack floorplans, motivating reconstruction from ordinary smartphone walkthroughs.
Loading 1804.00090v1…