Source-linked AI summary
HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation
Cheng Sun, Chi-Wei Hsiao, Min Sun, Hwann-Tzong Chen
TL;DR
Room-layout estimation from a single panoramic image requires recovering 3D structure despite limited annotations and complex non-cuboid layouts. HorizonNet predicts a compact 1D layout representation, uses fast post-processing, and adds Pano Stretch augmentation. It outperforms previous state-of-the-art methods, improves with augmentation, and supports qualitative general-layout results after fine-tuning.
Problem
The paper addresses estimating 3D room layouts from a single panoramic image, where precise annotations are limited and complex layouts may be represented ambiguously.
Method
HorizonNet predicts floor-wall, ceiling-wall, and wall-wall layout information as three 1D vectors, followed by geometric post-processing and Pano Stretch Data Augmentation.
Results
HorizonNet outperforms previous state-of-the-art methods with fewer computation resources, while Pano Stretch improves accuracy under all experimental settings.
Takeaways & Limitations
The compact representation and efficient post-processing support fast room-layout recovery, while the augmentation can potentially benefit other panorama-based tasks.
Takeaways & Limitations
General-layout recovery assumes perpendicular intersecting walls, parallel floor and ceiling, a 1.6-meter camera height, and correct floor alignment; non-cuboid training data were limited to 65 relabeled rooms.
Abstract
from arXiv · showhide
We present a new approach to the problem of estimating the 3D room layout from a single panoramic image. We represent room layout as three 1D vectors that encode, at each image column, the boundary positions of floor-wall and ceiling-wall, and the existence of wall-wall boundary. The proposed network, HorizonNet, trained for predicting 1D layout, outperforms previous state-of-the-art approaches. The designed post-processing procedure for recovering 3D room layouts from 1D predictions can automatically infer the room shape with low computation cost - it takes less than 20ms for a panorama image while prior works might need dozens of seconds. We also propose Pano Stretch Data Augmentation, which can diversify panorama data and be applied to other panorama-related learning tasks. Due to the limited data available for non-cuboid layout, we relabel 65 general layout from the current dataset for finetuning. Our approach shows good performance on general layouts by qualitative results and cross-validation.
1. Introduction
HorizonNet addresses room-layout estimation by combining a compact 1D representation with geometry-aware augmentation and recurrent modeling. The approach targets accuracy improvements without relying on substantially more annotations or larger networks.
- Motivation: Limited precise annotations constrain both training data and the feasible size of deep networks because larger models may overfit.Complex layouts may be approximated as cuboid-shaped or L-shaped layouts, creating ambiguities for training and testing.
- Contributions: The method uses a 1D O(W) representation to encode whole-room layouts with fewer parameters and less computation than prior approaches.This representation is designed to remove redundant degrees of freedom and focus learning on critical layout properties.
- Contributions: Pano Stretch Data Augmentation generates panorama images on the fly and improves accuracy under all experimental settings.The mechanism is designed around the geometric constraints of panorama-based layout prediction.
- Contributions: RNNs improve layout prediction by capturing long-range geometric patterns in room layouts.The contribution focuses on using recurrent modeling to exploit relationships across image columns.
- Contributions: The efficient representation and post-processing procedure enable low computation cost and extension beyond cuboid-shaped or L-shaped scenes.The paper positions this efficiency as a way to handle more complex room layouts without increasing model complexity.
2. Related Work
Earlier room-layout methods used geometric assumptions, hand-crafted cues, dense predictions, and post-processing to infer layouts from perspective or panoramic images. HorizonNet instead predicts three layout quantities for each panorama column, reducing the target from O(HW) to O(W).
- Geometric methods: Many earlier methods impose the Manhattan world assumption, aligning room layouts and furniture with three principal axes.This assumption supplies geometric constraints and supports vanishing-point-based image rectification and feature extraction.
- Perspective and dense prediction: Perspective-image approaches use cues such as boundary probabilities, geometric context, orientation maps, surface labels, object interactions, and corner heatmaps.Several methods post-process dense predictions or geometric hypotheses to generate room layouts.
- Panorama methods: Panorama-based methods extend vanishing-point, hypothesis-scoring, object, superpixel, and constraint-graph techniques to 360° imagery.These approaches adapt geometric or semantic cues to panoramic inputs, including equirectangular views.
- HorizonNet: HorizonNet predicts floor-wall and ceiling-wall boundary positions plus wall-wall boundaries for each equirectangular image column.Its three-value-per-column output reduces the representation from dense O(HW) prediction to O(W), while using an RNN across image columns.
3. Approach
HorizonNet estimates panoramic room layouts with three per-column 1D outputs, then reconstructs 3D geometry through efficient post-processing and supports stretched panorama augmentation.
- HorizonNet: HorizonNet regresses an O(W) target for each panorama column instead of using conventional O(HW) dense prediction.The network outputs ceiling-wall and floor-wall boundary positions plus wall-wall boundary existence.
- HorizonNet: The three-channel output records ceiling-wall position, floor-wall position, and wall-wall boundary existence for every image column.Boundary positions are normalized to [−π/2, π/2], while wall-wall values use distance-based targets with c = 0.96.
- HorizonNet: An RNN captures long-range geometric dependencies across image columns, with bidirectional processing using information from both sides.The implementation predicts four columns per time step to reduce computation without loss of accuracy.
- Post-processing: The post-processing procedure recovers floor, ceiling, and wall planes from the predicted boundaries and wall-wall peaks under Manhattan-layout assumptions.It averages projected floor-ceiling distances, uses principal components and voting to construct orthogonal wall planes, and intersects adjacent planes for corners.
- Post-processing: O(W) post-processing takes less than 20ms on average, while Pano Stretch augmentation changes room length and width by stretching panoramas along x or z.The augmentation maps stretched points back to the sphere and uses bilinear interpolation for image sampling.
- Pano Stretch Data Augmentation: Pano Stretch Data Augmentation improves quantitative results across experiment settings and may also support other panorama tasks.The paper specifically mentions semantic-segmentation maps and object-detection bounding boxes as potential applications.
4. Experiments
Experiments evaluate HorizonNet’s training setup, layout representation, augmentation, recurrent module, cuboid benchmarks, and non-cuboid fine-tuning. The method achieves strong accuracy with reduced computation and supports general layouts despite limited non-cuboid annotations.
- Experimental Setup: HorizonNet is trained on PanoContext and extended Stanford 2D-3D using 3 × 1 × 1024 ground-truth vectors and the LayoutNet split.Training uses Adam for 300 epochs with batch size 24 and learning rate 0.0003.
- Evaluation: Three metrics evaluate predictions: 3D IoU, normalized Corner Error, and Pixel Error.Corner Error is normalized by image diagonal length, while Pixel Error compares predicted and ground-truth surface classes pixelwise.
- Cuboid Room Results: Our approach outperforms existing methods under all settings on both PanoContext and Stanford-2D3D benchmarks.The comparisons are summarized in Tables 1 and 2.
- Ablation Study: The 1D O(W) representation performs better than dense O(HW) prediction while reducing parameters and computation.The ablation study compares the proposed representation with a dense prediction model using the same backbone.
- Ablation Study: Pano Stretch Data Augmentation consistently boosts performance during training.The augmentation is generated on the fly; stretching factors are sampled from U[1, 2] and optionally inverted.
- Ablation Study: RNNs improve layout prediction by capturing long-range geometric patterns, including in ceiling beams, missing polar regions, and occluded areas.Raw RNN outputs are highly consistent with the Manhattan world even without post-processing.
- Non-cuboid Room Results: Fine-tuning on 65 re-annotated non-cuboid rooms with 13-fold cross-validation shows that the approach works on general room layouts with occluded corners.The original datasets label non-cuboid rooms as cuboids, so fine-tuning supplies non-cuboid supervision.
5. Conclusion
The paper concludes that HorizonNet’s 1D representation enables accurate panorama room-layout estimation with fewer computational resources. Fast post-processing, Pano Stretch augmentation, and non-cuboid reconstruction extend the approach beyond standard cuboid settings.
- HorizonNet uses a new 1D representation to estimate room layouts from panoramas while outperforming previous state-of-the-art methods with fewer computation resources.
- The post-processing method rapidly recovers 3D layouts and works for complex rooms with occluded corners.
- Pano Stretch Data Augmentation further improves results and may also benefit other panorama tasks.
A. Pano Stretch Augmentation for Semantic Segmentation
The paper evaluates Pano Stretch Data Augmentation beyond layout estimation on Stanford 2D3D semantic segmentation. The reported mIoU evaluation indicates potential for addressing limited training data in other panorama tasks.
- Pano Stretch Augmentation is evaluated for semantic segmentation using PSPNet on the Stanford 2D3D benchmark.The model is trained on a subsampled training set and tested on the full testing set.
- The semantic-segmentation experiment reports the standard mIoU (%) metric and suggests potential to mitigate limited training data for other tasks.
B. More Qualitative Results of Cuboid Room Layout Reconstruction
Qualitative cuboid-layout results on PanoContext and Stanford 2D-3D are grouped by corner-error range. Ground-truth layouts are shown in green and estimated layouts in orange, including the worst-error examples.
- PanoContext qualitative results are arranged from best to worst corner-error groups across four rows, with four worst-error cases in the last row.
- Stanford 2D-3D qualitative results use the same corner-error grouping, with green ground truth and orange estimated layouts.
C. More Qualitative Results of Non-Cuboid Room Layout Reconstruction
Figures 11–23 compare original ground-truth room-layout annotations with HorizonNet estimates, including cases with occluded walls.
- Green lines denote the original ground-truth room-layout annotation, while blue lines denote the layout estimated by HorizonNet.
- The figures provide qualitative comparisons between annotated and estimated layouts across multiple reconstructed room examples.
- Figures 13–15 and 22–23 mark occlusion walls in black alongside the ground-truth and estimated layouts.