Source-linked AI summary
LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image
Chuhang Zou, Alex Colburn, Qi Shan, Derek Hoiem
TL;DR
Single-image room-layout estimation has largely focused on special cases such as cuboid rooms and limited image types. LayoutNet predicts layouts directly from perspective or panoramic RGB images using geometric alignment, joint layout prediction, and constrained Manhattan fitting. It reports strong panorama speed and accuracy, second-best perspective accuracy while being fastest, and support for non-cuboid Manhattan layouts.
Problem
Existing room-layout methods address special cases, including cuboid-shaped layouts from perspective or panoramic images, motivating a more general single-image approach.
Method
LayoutNet aligns images using vanishing points, jointly predicts corners and boundaries with a CNN, regresses 3D layout parameters, and fits a Manhattan-constrained layout.
Results
The method compares well in panoramic speed and accuracy, achieves second-best perspective accuracy while being fastest, and handles non-cuboid Manhattan layouts.
Takeaways & Limitations
A single RGB image can support layout reconstruction across perspective and panoramic views, including cuboid and more general Manhattan room shapes.
Takeaways & Limitations
The layout assumes Manhattan structure, with neighboring vertices sharing coordinate values to reduce free parameters.
Abstract
from arXiv · showhide
We propose an algorithm to predict room layout from a single image that generalizes across panoramas and perspective images, cuboid layouts and more general layouts (e.g. L-shape room). Our method operates directly on the panoramic image, rather than decomposing into perspective images as do recent works. Our network architecture is similar to that of RoomNet, but we show improvements due to aligning the image based on vanishing points, predicting multiple layout elements (corners, boundaries, size and translation), and fitting a constrained Manhattan layout to the resulting predictions. Our method compares well in speed and accuracy to other existing work on panoramas, achieves among the best accuracy for perspective images, and can handle both cuboid-shaped and more general Manhattan layouts.
1. Introduction
LayoutNet targets single-image room-layout estimation across perspective and panoramic images, while extending beyond cuboid rooms to general Manhattan layouts. It combines geometric alignment, joint layout-element prediction, and constrained 3D fitting, with strong reported speed and accuracy.
- Existing methods mainly address special cases such as cuboid-shaped layouts from perspective or panoramic images.
- LayoutNet estimates indoor 3D room layouts from a single perspective or panoramic image and extends to non-cuboid Manhattan rooms such as L-shaped rooms.
- Vanishing-point alignment levels the image, making wall-wall boundaries vertical and substantially reducing error in experiments.
- Panoramic results compare well in speed and accuracy, while perspective results are second best and fastest among compared methods.
- The network jointly predicts corner and boundary probability maps, then optimizes 3D layout parameters under Manhattan constraints.
- The method demonstrates gains from vanishing-point cues, geometric constraints, postprocessing, and direct 3D-parameter regression.
2. Related Work
Prior work estimates indoor layouts from single perspective or panoramic RGB images using surface, boundary, corner, geometric, and object cues, but generally targets cuboid layouts. This paper distinguishes itself by supporting both image types and extending prediction to non-cuboid Manhattan layouts.
- Single-view layout methods commonly estimate surfaces, boundaries, corners, or combinations of these elements under Manhattan-world assumptions.These assumptions place walls at right angles and perpendicular to the floor; cuboid layouts are a special case.
- Most prior methods target cuboid-shaped layouts from perspective RGB images, while only a few operate on panoramic images.Panoramic approaches include PanoContext, edge-based methods, and methods using surface orientations or object hypotheses.
- PanoContext adapts perspective-image frameworks to panoramas by projecting each panorama into multiple overlapping perspective images before combining features.Its estimates use vanishing points, Orientation Maps, Geometric Context, and object hypotheses.
- Compared with RoomNet, this method adds image alignment, multitask prediction of boundaries, corners, and 3D cuboid parameters, and constrained Manhattan-layout inference.The paper reports that these differences improve results, while RoomNet's RNN-refined 2D corners may not correspond to a consistent 3D cuboid.
- The paper claims the first method, to its knowledge, applicable to both perspective and panoramic images and easily extendable to non-cuboid Manhattan layouts.It presents this combination as making the approach more general and effective for single-image indoor layout estimation.
3. Approach
LayoutNet aligns images using vanishing points, predicts corners, boundaries, and 3D parameters with a multitask CNN, then optimizes a Manhattan layout. The pipeline combines learned predictions with explicit geometric constraints and candidate-layout scoring.
- 3.1. Alignment: The pipeline aligns the image using vanishing-point cues before predicting layout elements and fitting the final 3D layout.For panoramas, the method estimates the floor direction, rotates the scene, and reprojects it into equirectangular form.
- 3.2. Network structure: An encoder-decoder CNN predicts boundary and corner probability maps, with skip connections and additional connections from the boundary branch to the corner branch.The boundary map covers wall-wall, ceiling-wall, and wall-floor boundaries, including visible and occluded boundaries.
- 3.2. Network structure: A 3D regressor consumes the predicted 2D maps and estimates six cuboid layout parameters, while its loss contribution is intended to improve corner and boundary predictions.The direct regressor is inaccurate because small 2D shifts can produce large 3D-shape differences, but including it in the objective slightly improves network predictions.
- 3.2. Network structure: The training objective combines binary cross-entropy losses for boundary and corner maps with Euclidean loss for 3D parameters.The loss weights are α = β = 1 and τ = 0.01.
- 3.3. Training details: Training proceeds in stages because joint training from random initialization sometimes fails to converge.The boundary network and 3D regressor are initialized separately before end-to-end joint training.
- 3.4. 3D layout optimization: Layout optimization extracts candidate corners from corner maps, recovers camera and layout geometry under Manhattan constraints, and scores candidates using corner and boundary evidence.The score uses projected corner probabilities and maximum boundary likelihoods, with weights 1.0, 0.5, and 1.0 for junction, ceiling, and floor terms.
4. Experiments
Experiments evaluate LayoutNet on panorama and perspective benchmarks, including cuboid and non-cuboid Manhattan layouts, using accuracy metrics, ablations, runtime comparisons, and qualitative results.
- Evaluation setup: LayoutNet is evaluated on cuboid panoramas, non-cuboid panoramic layouts, and perspective images using public datasets and an additional labeled Stanford 2D-3D dataset.The evaluation reports test-set results after training on dataset training splits and tuning hyperparameters on validation data.
- Metrics: The evaluation uses 3D IoU, normalized corner error, and pixel error to compare predicted layouts with ground truth.3D IoU measures volumetric overlap; corner error is normalized by image diagonal; pixel error measures layout pixel-wise accuracy.
- Panoramic cuboid layouts: LayoutNet outperforms the PanoContext state of the art on all three reported metrics for PanoContext cuboid layouts.The qualitative comparison shows accurate pixel-level predictions, while the paper notes that small 2D errors can affect 3D IoU.
- Ablations: The full configuration performs best across ablations, while joint corner-boundary prediction, alignment, geometric constraints, and layout optimization improve the system.Cross-entropy performs better than L2 for boundary and corner prediction; the 3D regressor has a small impact and is omitted for perspective images.
- Runtime and comparisons: On Yang et al.’s 88 indoor panoramas, LayoutNet outperforms their method in L2 depth-distribution distance but is slightly worse in cosine distance.A forward pass takes 39 ms, while CPU alignment and layout optimization take 13.73 s and 30.5 s, respectively.
- Perspective and non-cuboid layouts: LayoutNet predicts complex non-cuboid Manhattan layouts such as L-shaped rooms and ranks second on the Hedau perspective benchmark.On perspective images, it processes one image in 39 ms, compared with 52 ms for RoomNet basic and 168 ms for RoomNet recurrent.
5. Conclusion
The paper concludes that LayoutNet predicts room layouts from single panoramic or perspective images while relaxing the cuboid assumption. It uses geometric alignment and Manhattan constraints and handles non-cuboid layouts such as L-shaped rooms.
- 5. Conclusion: LayoutNet predicts room layouts from a single panorama or perspective image and supports non-cuboid Manhattan layouts such as L-shaped rooms.The method operates directly on panoramic images rather than decomposing them into perspective images.
- 5. Conclusion: Pre-alignment using vanishing points and Manhattan constraints substantially improve quantitative results.The paper identifies these geometric components as important additions to the layout prediction pipeline.
- 5. Conclusion: Future work includes arbitrary room layouts, object detection for room-shape estimation, and complete 3D indoor-model recovery from single images.
A. Quantitative Results on LSUN layout Challenge [9]
The LSUN results place LayoutNet below recurrent RoomNet on keypoint error and below the leading methods on pixel error. The paper attributes the lower pixel accuracy to its simplified keypoint representation.
- Quantitative results: LayoutNet ranks second in Keypoint Error (%) and third in Pixel Error (%) on the LSUN dataset.RoomNet recurrent 3-iter ranks ahead of LayoutNet in both reported rankings.
- Analysis: The lower pixel-error accuracy mainly results from LayoutNet’s simplified room keypoint representation.LayoutNet directly predicts eight keypoints and selects among them according to room type.
- Analysis: Applying the paper’s layout optimization step could possibly further enhance perspective-image performance.
B.1. Non-cuboid layout from panorama
Additional panorama samples demonstrate LayoutNet’s qualitative reconstruction of non-cuboid room layouts using images collected by Yang et al., excluding overlaps with PanoContext.
- B.1. Non-cuboid layout from panorama: LayoutNet qualitatively reconstructs non-cuboid room layouts from panorama samples collected by Yang et al.Samples overlapping with the PanoContext dataset are excluded.
B.2. Cuboid layout from panorama
The paper presents additional qualitative cuboid-layout comparisons on the PanoContext and Stanford 2D-3D annotation datasets, using ground-truth layouts as references.
- PanoContext examples compare the method’s qualitative predictions with the state-of-the-art.
- Stanford 2D-3D examples compare predicted layouts with the authors’ ground-truth annotations.
B.3. Perspective images
The paper provides qualitative perspective-image and non-cuboid-layout results, including comparisons with ground truth and examples of L-shaped rooms.
- The perspective-image qualitative results are presented on the LSUN Layout Challenge dataset against ground-truth annotations.
- Non-cuboid examples demonstrate predicted layouts for L-shaped rooms.
- PanoContext figures compare the method and the state-of-the-art using predicted layouts and ground-truth layouts.
- Stanford 2D-3D figures present randomly sampled cuboid-layout predictions against ground truth, with smaller vertical field of view and more occlusion than PanoContext.
- Perspective-image results show the RGB input, predicted boundary and corner maps, and the final layout against ground truth.