Source-linked AI summary
HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features
Cheng Sun, Min Sun, Hwann-Tzong Chen
TL;DR
Holistic understanding of high-resolution indoor 360° panoramas requires efficient modeling across layout, depth, and semantic modalities, but prior latent horizontal features were limited to per-column outputs. HoHoNet redesigns the feature architecture and adds horizon-to-dense recovery from compact LHFeat, achieving fast inference and strong results across these tasks.
Problem
Prior latent horizontal feature methods were limited to per-column modalities, constraining their use for tasks requiring per-pixel predictions.
Method
HoHoNet uses efficient height compression to form compact LHFeat and a horizon-to-dense module to recover 2D dense modalities.
Results
HoHoNet runs at 52 FPS with ResNet-50 and 110 FPS with ResNet-34 on 512 × 1024 panoramas, with state-of-the-art-level layout and segmentation results and stronger dense-depth performance.
Takeaways & Limitations
The framework supports layout, dense depth, and semantic segmentation from compact LHFeat within one fast, versatile system.
Takeaways & Limitations
The model is vulnerable to non-gravity-aligned views: 10° pitch or roll rotation increases MAE from 28.45cm to more than 44cm.
Abstract
from arXiv · showhide
We present HoHoNet, a versatile and efficient framework for holistic understanding of an indoor 360-degree panorama using a Latent Horizontal Feature (LHFeat). The compact LHFeat flattens the features along the vertical direction and has shown success in modeling per-column modality for room layout reconstruction. HoHoNet advances in two important aspects. First, the deep architecture is redesigned to run faster with improved accuracy. Second, we propose a novel horizon-to-dense module, which relaxes the per-column output shape constraint, allowing per-pixel dense prediction from LHFeat. HoHoNet is fast: It runs at 52 FPS and 110 FPS with ResNet-50 and ResNet-34 backbones respectively, for modeling dense modalities from a high-resolution $512 \times 1024$ panorama. HoHoNet is also accurate. On the tasks of layout estimation and semantic segmentation, HoHoNet achieves results on par with current state-of-the-art. On dense depth estimation, HoHoNet outperforms all the prior arts by a large margin.
1. Introduction
HoHoNet targets holistic modeling of indoor 360° panoramas with compact latent horizontal features, extending per-column representations to dense per-pixel prediction. It combines efficient architecture with horizon-to-dense recovery for layout, depth, and semantic segmentation.
- HoHoNet addresses holistic scene modeling from a single high-resolution equirectangular image capturing a 360° panorama.
- LHFeat flattens feature height and encodes compact representations for both per-column and per-pixel modalities.The framework processes ERP features with an efficient height compression module before prediction.
- The horizon-to-dense module recovers 2D per-pixel modalities from LHFeat while maintaining overall efficiency.
- DCT-based modeling replaces linear interpolation in dense prediction because the spatially blended LHFeat motivates frequency-domain recovery.
- 52 FPS and 110 FPS are achieved for dense modalities on 512 × 1024 panoramas with ResNet-50 and ResNet-34, respectively.
- Layout reconstruction and semantic segmentation are on par with recent state-of-the-art, while dense depth estimation outperforms prior methods by a margin.
2. Related work
Prior work addresses indoor 360° scene modeling across datasets, projections, depth, and segmentation, while HoHoNet builds on latent horizontal features to support dense modalities. Its design uses ERP input and compact feature processing to broaden the original per-column formulation.
- Indoor 360 datasets: Indoor 360° scene modeling uses real-world and synthetic datasets providing depth, semantic, layout, and structural annotations.
- Input 360 format: ERP preserves all captured information in one image but introduces distortion that can degrade conventional convolutional processing.
- Input 360 format: HoHoNet uses ERP inputs with classical convolutions for simplicity and efficiency rather than adding distortion-aware techniques with extra computational overhead.
- Depth estimation on 360 imagery: Prior 360° depth methods commonly use distortion-aware architectures, multiple projections or modalities, and sometimes multiple backbones with cascaded training.
- Semantic segmentation on 360 imagery: Recent semantic-segmentation methods often use icosahedral representations, while tangent-image approaches support high-resolution panoramas.
- Latent horizontal features: Unlike HorizonNet's layout-specific, horizontal-only formulation, HoHoNet improves LHFeat extraction and uses horizon-to-dense prediction for dense-modality modeling.
3. Approach
HoHoNet encodes high-resolution ERP panoramas into compact Latent Horizontal Features, then predicts either per-column or dense per-pixel modalities. Its horizon-to-dense module recovers 2D outputs from flattened features using interpolation or IDCT.
- Framework overview: HoHoNet uses ERP inputs, a ResNet feature pyramid, and efficient height compression to construct compact LHFeat representations.The EHC module reduces feature height to one and fuses the resulting 1D features.
- Framework overview: Multi-head self-attention replaces bidirectional LSTM refinement because it runs faster and improves accuracy.The recurrent layer accounted for 22% of deep-network processing time in the authors’ comparison.
- Per-column prediction: HoHoNet predicts per-column modalities by upsampling LHFeat and applying three Conv1D layers to produce the target channels.The convolution kernels are 3, 3, and 1, with batch normalization and ReLU between layers.
- Dense prediction: The horizon-to-dense module reshapes predictions into r components per column and recovers dense outputs with linear interpolation or IDCT.The unified form is X = Mx, where the basis matrix M determines the reconstruction operation.
- Dense prediction: IDCT constantly outperforms linear interpolation because frequency-domain basis functions help characterize row-dependent information blended into LHFeat.This addresses the challenge of recovering dense modalities after vertical features have been flattened.
4. Experiments
HoHoNet is evaluated through ablations and comparisons across dense depth estimation, semantic segmentation, and layout estimation. The experiments show improved speed–accuracy tradeoffs, strong depth results, competitive segmentation and layout performance, and sensitivity to non-gravity-aligned views.
- Architecture ablations: HoHoNet’s overall LHFeat architecture improves depth MAE from 0.3002 to 0.2835 while increasing speed from 38 to 52 FPS.The comparison uses ResNet-50 and IDCT with r = 64 for dense prediction.
- Horizon-to-dense ablations: IDCT consistently outperforms linear interpolation for dense depth prediction, while r = 64 provides the best tested accuracy with nearly unchanged FPS.The two operations have negligible computational cost relative to the deep network.
- Backbone ablations: ResNet-34 almost doubles FPS compared with ResNet-50 while causing only a small accuracy decrease.This establishes a faster backbone option for HoHoNet’s dense prediction framework.
- Dense depth estimation: HoHoNet outperforms BiFuse by a large margin on real-world indoor 360 depth estimation while using one backbone instead of two.The comparison follows the Matterport3D and Stanford2D3D evaluation protocol with depth clipped to 10 meters and without median alignment.
- Dense depth estimation: HoHoNet achieves the best accuracy under the same training protocol on the Stanford2D3D depth subset and remains superior despite using one backbone and only depth training.GeoReg360 uses two backbones and additional layout and semantic annotations, whereas HoHoNet uses a single ResNet-50.
- Segmentation and layout estimation: HoHoNet matches or approaches state-of-the-art results across segmentation and layout while offering substantial layout speed gains.It achieves similar high-resolution mAcc with significantly better mIoU, matches AtlantaNet while being 22× faster, and exceeds HorizonNet by +0.77 3D IoU and +0.61 2D IoU while being 3.5× faster.
- Non-gravity-aligned views: A 10° pitch or roll rotation increases MAE from 28.45cm to more than 44cm, revealing sensitivity to non-gravity-aligned inputs.Rotation augmentation improves robustness to non-canonical views but raises gravity-aligned test MAE from 28.35cm to 30.92cm.
5. Conclusion
HoHoNet is presented as a fast, versatile, and accurate framework for holistic modeling of omnidirectional images across layout, depth, and semantic segmentation tasks.
- HoHoNet targets layout reconstruction, depth estimation, and semantic segmentation with accuracy on par with or better than the state-of-the-art.
- The work received support from Taiwan’s MOST, the MOST Joint Research Center for AI Technology, and All Vista Healthcare.
A. Network architecture diagram
The architecture diagram specifies HoHoNet’s tensor-shape conventions and input dimensions, while contrasting the proposed EHC block with the prior HC block.
- The detailed architecture assumes an input panorama of height 512 and width 1024, with tensor shapes written as channels, height, and width.D and E are hyperparameters.
- The ConvSqueezeH layer uses depthwise convolution with a kernel matching the known input feature height, producing output height 1.
- The height-compression design compares HoHoNet’s Efficient Height Compression block with the HC block from prior work.
C. Detailed semantic segmentation results
Semantic-segmentation evaluation reports detailed per-class IoU and accuracy, with strong class-level and overall performance on Stanford2D3D.
- HoHoNet achieves the best IoU on 10 of 13 classes and superior overall mIoU.
- HoHoNet achieves the best accuracy on 7 of 13 classes and comparable overall mAcc.
- Table 5 provides detailed quantitative per-class results on Stanford2D3D using RGB-D input.
- Table 6 provides a detailed quantitative comparison for room-layout estimation under different numbers of ground-truth corners.
E. More qualitative comparisons for depth estimation
Qualitative depth comparisons place HoHoNet alongside BiFuse and visualize per-pixel MAE differences between the two methods.
- Figure 9 presents qualitative comparisons of HoHoNet’s estimated dense depth with BiFuse.BiFuse results come from its official model trained on Matterport3D.
- The Advantage column reports the MAE difference between HoHoNet and BiFuse.
- Blue indicates that HoHoNet is better, whereas red indicates that BiFuse is better.
F. Qualitative results for semantic segmentation
HoHoNet’s semantic-segmentation results on Stanford2D3D are presented qualitatively, but no comparison with the prior high-resolution method is shown because its public release could not be built.
- Qualitative semantic-segmentation results are shown on the Stanford2D3D dataset.
- The prior high-resolution panorama method could not be built from its public release, so only HoHoNet’s results are shown.
G. Qualitative comparisons for layout estimation
HoHoNet produces promising room-layout estimates comparable to AtlantaNet while running substantially faster, with both methods visualized against ground truth.
- 22× faster: HoHoNet runs substantially faster than AtlantaNet while achieving promising, comparable room-layout results.
- HoHoNet uses the post-processing algorithm from [21] to produce Manhattan layouts, whereas AtlantaNet generates less restrictive Atlanta layouts.
- Figure 11 compares ground-truth layouts with AtlantaNet’s and HoHoNet’s predictions.