Source-linked AI summary
DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama
Shang-Ta Yang, Fu-En Wang, Chi-Han Peng, Peter Wonka, Min Sun, Hung-Kuo Chu
TL;DR
Estimating complex Manhattan-world room layouts from a single RGB panorama is difficult because clutter can hide structural cues and 2D boundary inference is ill-posed. DuLa-Net combines panorama and ceiling-view branches with feature fusion, and reports higher accuracy than the current state of the art, especially for rooms with more than four corners, while taking less time to compute layouts.
Problem
Single-panorama room-layout estimation remains challenging because clutter occludes corners and edges, while inference from 2D boundaries is ill-posed.
Method
DuLa-Net uses two connected encoder-decoder branches for equirectangular panorama and perspective ceiling views, jointly predicting floor plans and layout height.
Results
DuLa-Net outperforms the current state of the art in prediction accuracy, especially for rooms with more than four corners, and takes less time to compute final layouts.
Takeaways & Limitations
The dual-projection architecture and Realtor360 dataset support room-layout prediction for Manhattan-world rooms with varied complexity beyond cuboid layouts.
Takeaways & Limitations
The method may be confused by mirrors or large occluding objects and remains constrained by heuristics, assumptions, and the Manhattan-world setting.
Abstract
from arXiv · showhide
We present a deep learning framework, called DuLa-Net, to predict Manhattan-world 3D room layouts from a single RGB panorama. To achieve better prediction accuracy, our method leverages two projections of the panorama at once, namely the equirectangular panorama-view and the perspective ceiling-view, that each contains different clues about the room layouts. Our network architecture consists of two encoder-decoder branches for analyzing each of the two views. In addition, a novel feature fusion structure is proposed to connect the two branches, which are then jointly trained to predict the 2D floor plans and layout heights. To learn more complex room layouts, we introduce the Realtor360 dataset that contains panoramas of Manhattan-world room layouts with different numbers of corners. Experimental results show that our work outperforms recent state-of-the-art in prediction accuracy and performance, especially in the rooms with non-cuboid layouts.
1. Introduction
DuLa-Net targets difficult single-panorama room-layout estimation by combining two projections and introducing data for more complex Manhattan-world rooms.
- Existing methods struggle with occluded corners, edges, and complex layouts because clutter obscures cues and 2D boundary inference is ill-posed.
- DuLa-Net analyzes equirectangular panorama and perspective ceiling views with two connected encoder-decoder branches.Feature fusion connects the branches, which are jointly trained.
- The method outperforms the current state of the art, especially for rooms with more than four corners, while requiring less post-processing.
- The dual-projection architecture can infer room layouts with shapes beyond cuboids and L-shapes.
- Realtor360 contains 2573 panoramas depicting rooms with 4 to 12 corners for learning complex layouts.
2. Related Work
Prior room-layout methods vary in assumptions and input types, while DuLa-Net retains the Manhattan-world assumption and permits arbitrary corner counts using panorama and ceiling-view reasoning.
- Room-layout methods differ mainly in layout assumptions, input images, and inference methods.
- The Manhattan-world assumption aligns walls to a global coordinate system, whereas cuboid rooms restrict layouts to exactly four corners.
- DuLa-Net adopts Manhattan-world geometry while allowing arbitrary numbers of room corners.
- Panorama-based methods exploit a wider field of view, and LayoutNet predicts on the panorama as a whole to extract global information.
- The related floor-view approach converts panoramas into a 2D closed loop, motivating DuLa-Net’s alternative ceiling-view projection.
3. Overview
The framework aligns a panorama to Manhattan-world coordinates, creates a perspective ceiling view, and combines both branches to recover a 3D layout.
- The input panorama is aligned with a global coordinate system before equirectangular-to-perspective conversion creates the ceiling view.
- Panorama-view and ceiling-view images enter two encoder-decoder branches connected through E2P-based feature fusion.
- The system extrudes a fused 2D Manhattan floor plan using predicted layout height to determine the final 3D layout.
4. E2P conversion
The E2P conversion maps pixels from a perspective ceiling-view image back to corresponding positions in an equirectangular panorama. It uses pinhole-camera geometry, axis rotation, unit-sphere projection, and differentiable interpolation.
- 4. E2P conversion: E2P converts a perspective image into corresponding equirectangular-panorama coordinates for pixel-value sampling.The perspective pixel is rotated according to whether the camera looks toward the ceiling or floor, projected onto a unit sphere, and mapped to panorama coordinates.
- 4. E2P conversion: The perspective projection uses a square image of width w and focal length f = 0.5 ∗w ∗cot(0.5 ∗FoV).
- 4. E2P conversion: The conversion is differentiable, allowing the transformed panorama samples to participate in backpropagation.
5. Network architecture
DuLa-Net uses two encoder-decoder branches to analyze panorama-view and ceiling-view images, predicting floor-ceiling structure, floor plans, and layout height. E2P-based feature fusion connects the branches, while the outputs are trained with classification and height-regression losses.
- 5. Network architecture: Two encoder-decoder branches analyze panorama-view and ceiling-view inputs, predicting floor-ceiling maps, ceiling-view floor plans, and layout height.The panorama branch outputs the floor-ceiling probability map and height, while the ceiling branch outputs the floor-plan probability map.
- 5. Network architecture: Feature fusion improves prediction accuracy, partly because panorama-view features are less distorted near ceiling-view image boundaries.
- 5. Network architecture: E2P feature fusion merges panorama-branch features into the ceiling branch before its first five decoder layers.The panorama features are converted into the ceiling view and combined using decay coefficients before being passed to the next ceiling-branch layer.
- 5. Network architecture: The network applies binary cross-entropy losses to floor-ceiling and floor-plan maps and L1 loss to layout height.
- 5. Network architecture: The floor-plan probability maps are fused, thresholded at 0.5, polygon-fitted, and converted into axis-aligned grid cells for the final floor-plan shape.
6. 3D layout estimation
The 3D layout is reconstructed by estimating a 2D Manhattan floor plan from ceiling- and floor-derived probability maps, then extruding it using the predicted height. Realtor360 provides annotated panoramas spanning layouts with varied corner counts.
- 6. 3D layout estimation: The reconstruction first estimates a 2D Manhattan floor plan from probability maps, then extrudes that shape according to layout height.
- 6. 3D layout estimation: Ceiling- and floor-derived maps are registered using the scale factor 1.6/(H −1.6) before producing the fused floor-plan probability map.The constant 1.6 represents the camera-to-ceiling distance.
- 6. 3D layout estimation: The fused map is thresholded at 0.5, simplified into a polygon, and partitioned by axis-aligned lines into grid cells.The final floor-plan shape is the union of cells whose floor-plan area ratio exceeds 0.5.
- 6. 3D layout estimation: Realtor360 is presented as a dataset of panoramas with annotated room layouts for training and testing layout estimation.
7. Realtor360 dataset
Realtor360 addresses the lack of complex room-layout data by providing over 2,500 annotated panoramas with varying corner counts. Its annotations combine automated geometric initialization with interactive refinement.
- Dataset motivation: The dataset is designed for training and testing layouts of different complexities, unlike existing datasets dominated by simple cuboid rooms.
- Annotation tool: Annotations begin with automatically extracted depth and line segments, then users refine the 3D layout by editing, merging, splitting, and snapping walls.
- Dataset evaluation: Table 2 compares DuLa-Net with LayoutNet on Realtor360 and also evaluates alternative DuLa-Net configurations.
- Dataset evaluation: Table 3 evaluates LayoutNet against DuLa-Net after training on a cuboid-only subset while retaining the broader testing set.
8. Experiments
Experiments compare DuLa-Net with LayoutNet across Realtor360, PanoContext, and Stanford 2D/3D, including ablations and timing. DuLa-Net performs especially well on complex layouts and computes final layouts faster.
- Realtor360 evaluation: DuLa-Net achieves an overall performance gain of approximately 14% in both 2D and 3D metrics over LayoutNet on Realtor360.Its advantage is especially pronounced as the number of room corners increases.
- Realtor360 evaluation: When both methods are trained only on cuboid rooms, DuLa-Net still outperforms LayoutNet across room types in the unchanged test set.
- Realtor360 evaluation: Qualitatively, LayoutNet tends to predict cuboid rooms, whereas DuLa-Net directly predicts Manhattan-world floor plans without fixing the number of corners.The authors conjecture that this difference helps explain DuLa-Net’s stronger results on rooms with more than four corners.
- Ablation study: Jointly training the panorama-view and ceiling-view branches improves performance, and feature fusion provides an additional improvement.
- Cross-dataset evaluation: On PanoContext, DuLa-Net outperforms LayoutNet by a small margin in 3D IoU.
- Timing: DuLa-Net takes about 13.4 seconds end to end versus LayoutNet’s 43.9 seconds, a 3.28X speedup.LayoutNet’s additional 3D optimization step takes 30.5 seconds, whereas DuLa-Net’s prediction and fitting steps take milliseconds.
9. Conclusion
DuLa-Net estimates Manhattan-world room layouts from a single RGB panorama using jointly trained dual-projection branches and feature fusion. It outperforms prior work, particularly on rooms with more than four corners, but remains limited by missing object semantics and Manhattan-world assumptions.
- Method: DuLa-Net uses two encoder-decoder branches for equirectangular and perspective ceiling views, connected by feature fusion and jointly trained to predict floor plans and layout height.
- Dataset: Realtor360 provides 2,573 indoor panoramas of Manhattan-world room layouts with varied complexity.
- Results: The method outperforms the current state of the art in prediction accuracy, especially for rooms with more than four corners, while computing final layouts faster.
- Limitations: Failure cases arise when mirrors or large occluding objects confuse the network because object semantics are unavailable.
- Limitations: The layout-estimation procedure uses heuristics and assumptions that can misestimate floor-plan probabilities and restrict outputs to Manhattan-world rooms.
A.1. Re-training LayoutNet
The supplementary comparison re-trains LayoutNet under cuboid-room conditions and documents its evaluation setup and additional qualitative comparisons.
- Re-training setup: LayoutNet is re-trained on Realtor360 using only four-corner rooms before applying its cuboid-room optimization module.
- Evaluation: The supplementary material reports quantitative evaluation of LayoutNet across different training sets.
- Qualitative evaluation: Additional figures provide further qualitative comparisons with LayoutNet.
B. More Visual Results of DuLa-Net
This section presents additional visualizations of DuLa-Net’s estimated 3D room layouts, emphasizing layouts with different numbers of corners and their complexity in 3D.
- B. More Visual Results of DuLa-Net: DuLa-Net generates 3D room layouts with different numbers of corners, which are shown in Figures 10 and 11.The paper notes that layout complexity is best viewed in 3D.
- B. More Visual Results of DuLa-Net: Figures 8 and 9 qualitatively compare DuLa-Net with LayoutNet using ground-truth, LayoutNet, and DuLa-Net layout lines in panorama view.Blue denotes ground truth, green denotes LayoutNet, and orange denotes DuLa-Net.
- B. More Visual Results of DuLa-Net: Figures 10 and 11 display estimated layouts in equirectangular panorama view and textured 3D renderings, with checkerboards marking textures missing because of occlusion.The two views provide complementary visualizations of the estimated layouts and their texture mapping.