PaperlayerView exact revision on arXiv

Source-linked AI summary

UniFuse: Unidirectional Fusion for 360$^{\circ}$ Panorama Depth Estimation

Hualie Jiang, Zhe Sheng, Siyu Zhu, Zilong Dong, Rui Huang

arXiv:2102.03550v2cs.CVcs.RO

TL;DR

Spherical panorama depth estimation must handle the complementary limitations of equirectangular and cubemap projections. UniFuse unidirectionally feeds cubemap features into equirectangular features during decoding and adds a CEE fusion module. It establishes state-of-the-art performance on four datasets while also showing lower complexity and higher generalization capability.

  • Problem

    Equirectangular panoramas provide full scene coverage but become distorted toward the poles, while cubemaps are distortion-free but discontinuous at edges and limited in field-of-view.

  • Method

    UniFuse unidirectionally feeds cubemap features to the equirectangular branch only during decoding and uses the CEE fusion module.

  • Results

    UniFuse establishes state-of-the-art performance on four datasets, including reducing Matterport3D Abs Rel from 0.2048 to 0.1063 versus BiFuse.

  • Takeaways & Limitations

    The model combines effective panoramic feature fusion with lower model complexity and higher generalization capability than previous works.

Abstract

from arXiv · show

Learning depth from spherical panoramas is becoming a popular research topic because a panorama has a full field-of-view of the environment and provides a relatively complete description of a scene. However, applying well-studied CNNs for perspective images to the standard representation of spherical panoramas, i.e., the equirectangular projection, is suboptimal, as it becomes distorted towards the poles. Another representation is the cubemap projection, which is distortion-free but discontinued on edges and limited in the field-of-view. This paper introduces a new framework to fuse features from the two projections, unidirectionally feeding the cubemap features to the equirectangular features only at the decoding stage. Unlike the recent bidirectional fusion approach operating at both the encoding and decoding stages, our fusion scheme is much more efficient. Besides, we also designed a more effective fusion module for our fusion scheme. Experiments verify the effectiveness of our proposed fusion strategy and module, and our model achieves state-of-the-art performance on four popular datasets. Additional experiments show that our model also has the advantages of model complexity and generalization capability.The code is available at https://github.com/alibaba/UniFuse-Unidirectional-Fusion.

I. INTRODUCTION

The paper targets single-image depth estimation for spherical panoramas, whose full scene coverage is offset by projection-specific distortions and discontinuities. It proposes a simpler fusion framework that uses cubemap features to support equirectangular prediction.

  • I. INTRODUCTION: Spherical panoramas support more complete and potentially more accurate, scale-consistent scene reconstruction than perspective images.The motivation includes applications in 3D reconstruction, robot navigation, and virtual or augmented reality.
  • I. INTRODUCTION: Equirectangular projection covers the full scene but becomes increasingly distorted toward the poles, whereas cubemaps are distortion-free but discontinuous at face edges and field-of-view limited.These complementary weaknesses make direct CNN-based panoramic depth estimation challenging.
  • I. INTRODUCTION: BiFuse combines the two projections bidirectionally during encoding and decoding, but its many added modules make the architecture over-complicated.The cited design also includes spherical padding and a refinement network.
  • I. INTRODUCTION: UniFuse feeds cubemap features into equirectangular features only during decoding, using U-Net skip connections to minimize coupling with the backbones.The framework is designed around the final equirectangular depth output rather than a separate cubemap depth objective.
  • I. INTRODUCTION: The paper introduces a CEE fusion module and reports state-of-the-art performance together with advantages in model complexity and generalization ability.CEE is designed to use cubemap features to enhance equirectangular features.

II. RELATED WORK

Prior work spans perspective-image depth estimation, panoramic methods, and projection-aware operations. The paper positions UniFuse against complex bidirectional fusion and specialized convolutions or padding that add complexity without improving its performance.

  • II. RELATED WORK: Perspective-image depth estimation evolved from graphical models to CNN-based supervised regression or classification, with some later unsupervised approaches using stereo images or monocular videos.The cited passage notes that unsupervised performance is inferior to supervised approaches.
  • II. RELATED WORK: Panoramic depth methods use full field-of-view images, but several earlier experiments rely on virtual datasets that may transfer poorly to real applications.The related work discusses 3D60, PonoSUNCG, and other virtual-data approaches before real-dataset experiments.
  • II. RELATED WORK: BiFuse combines ERP and CMP through bidirectional fusion, spherical padding, and refinement, but its complex structure may hinder learning equirectangular features.UniFuse instead uses cubemap features to enhance equirectangular feature learning.
  • II. RELATED WORK: Specialized convolutions and padding methods address projection distortion or discontinuity, but UniFuse does not adopt them because experiments show they add complexity without improving performance.Examples include SC, DaC, CirP, CuP, and SP.

III. METHODOLOGY

The methodology introduces equirectangular and cubemap representations and their conversion. Cubemap-to-equirectangular reprojection maps cubemap contents back onto the equirectangular grid through spherical coordinates, face selection, and inverse projection.

  • A. Preliminaries: Equirectangular projection samples a spherical surface uniformly in longitude and latitude on a rectangular grid.The grid width is twice its height, with angular coordinates φ and θ.
  • A. Preliminaries: Cubemap projection maps the spherical surface onto six perspective-image faces of an inscribed cube.The faces correspond to back, down, front, left, right, and up directions.
  • A. Preliminaries: Each cubemap face uses a rotation matrix to relate its coordinate system to the spherical surface before projection.The face-specific matrices are denoted R_fi.
  • A. Preliminaries: C2E reprojects cubemap RGB values or features onto the equirectangular grid using inverse wrapping.It converts the angular position to the spherical surface, selects the nearest cubemap face, and computes the corresponding face position by inverting the cubemap projection.

B. The Unidirectional Fusion Network

UniFuse fuses ERP and CMP in one direction because the desired output is an equirectangular depth map. It performs fusion only during decoding, using cubemap features as support while avoiding a cubemap decoder and reverse fusion.

  • B. The Unidirectional Fusion Network: The network feeds distortion-free cubemap features into full-view equirectangular features as a supporting component.The equirectangular depth map is the ultimate output target.
  • B. The Unidirectional Fusion Network: UniFuse omits reverse fusion because CMP has limited field-of-view, cubemap spherical padding is time-consuming, and cubemap decoding increases complexity.Optimizing cubemap depth maps could also distract from equirectangular depth learning.
  • B. The Unidirectional Fusion Network: Fusion occurs only at the decoding stage within U-Net skip connections, preserving the learning of well-encoded backbone features.This design is intended to avoid disturbing the backbone.

C. The Fusion Modules

The paper compares concatenation, Bi-Projection, and CEE fusion modules within UniFuse. CEE uses cubemap features to enhance equirectangular features while addressing cubemap-boundary inconsistencies.

  • Concatenation: Concatenation casts cubemap features to equirectangular form, concatenates them with equirectangular features, and reduces channels with a 1 × 1 convolution.Its parameter count is 2C^2.
  • Bi-Projection: Bi-Projection generates a sigmoid-scaled mask from both feature branches before adding the masked modification to the target branch.In UniFuse, the reverse E2C path is omitted because fusion is unidirectional; the module has 18C^2 + 4C + 1 parameters.
  • CEE: CEE is designed to use distortion-free cubemap features to enhance equirectangular features through a more elaborate concatenation.The module is intended to facilitate fusion in the unidirectional decoding-stage framework.
  • CEE: CEE first generates a residual feature map from concatenated features and adds it to the cubemap-to-equirectangular features to reduce boundary inconsistency.The residual block uses 1 × 1 convolution for channel squeezing and 3 × 3 convolution for residual generation.
  • CEE: The visualization shows cracks between cube faces in Fc2e disappearing after residual modulation in F′c2e.This illustrates the intended boundary-correction effect of the residual feature map.

A. Experimental Settings

Experiments use four panoramic depth datasets spanning real-world and rendered settings. The implementation uses an ImageNet-pretrained ResNet18 for most experiments, standard optimization, and dataset-specific validation and training schedules.

  • Datasets: Experiments cover Matterport3D, Stanford2D3D, 3D60, and PanoSUNCG.Matterport3D and Stanford2D3D are real-world datasets, while 3D60 and PanoSUNCG are rendered or virtual settings.
  • Datasets: Matterport3D provides raw depth, Stanford2D3D derives depth from reconstructed 3D models, and 3D60 is rendered from two realistic datasets.Matterport3D misses top and bottom depth, while some Stanford2D3D depth is inaccurate.
  • Implementation Details: Most experiments use an ImageNet-pretrained ResNet18 backbone, Adam with default parameters, and a constant learning rate of 0.0001.Training also uses random color adjustment, left-right flipping, and random yaw rotation.
  • Implementation Details: Training lasts 100 epochs for real datasets and 30 epochs for virtual datasets, with dataset-specific validation subsets.The input size is set to 512 following BiFuse.

3) Evaluation Metrics:

The evaluation uses standard depth-error and accuracy metrics to compare UniFuse with baselines across four datasets. Reported results show state-of-the-art performance, stronger fusion gains than BiFuse on many metrics, and fewer qualitative artifacts than the equirectangular baseline.

  • Evaluation Metrics: Evaluation reports MAE, Abs Rel, RMSE, RMSElog, and accuracy percentages for δ thresholds below 1.25, 1.252, and 1.253.RMSElog uses log10 to match the BiFuse comparison protocol.
  • Performance Comparison: UniFuse establishes state-of-the-art performance on all four datasets and reduces Matterport3D Abs Rel from 0.2048 to 0.1063 versus BiFuse.Its δ < 1.25 accuracy also improves by 4.45%.
  • Performance Comparison: UniFuse reduces error metrics by over 10% on average from its equirectangular baseline, compared with about 4% for BiFuse.UniFuse is especially stronger than BiFuse on the tighter accuracy metrics, including δ < 1.25.
  • Performance Comparison: On Stanford2D3D, UniFuse performs best on most metrics, while Jin et al. use only a restricted Manhattan-structure subset.UniFuse is not limited to that specific scene structure.
  • Performance Comparison: On 3D60, UniFuse improves more than BiFuse on MAE, Abs Rel, RMSElog, and δ < 1.25, while improving slightly less on the other metrics.UniFuse significantly outperforms BiFuse, whereas a competing model requires an extra depth camera and calibration.
  • Qualitative Results: Qualitative results show that UniFuse produces accurate depth maps with fewer artifacts than the equirectangular baseline.The examples are drawn from the test sets of the four datasets.

2) Ablation Study:

The ablations show that pretraining, unidirectional fusion, and the CEE module each contribute to UniFuse’s performance, with CEE outperforming Bi-Projection.

  • Pretraining: ImageNet pretraining benefits both the equirectangular baseline and UniFuse, with disabling it degrading depth-estimation performance.For the baseline, Abs Rel error increases by 8.36% and δ < 1.25 accuracy drops by over 2%.
  • Fusion strategy: 5.14% lower Abs Rel and 1.3% higher δ < 1.25 accuracy show that simple unidirectional feature concatenation improves over the equirectangular baseline.The strategy feeds cubemap features into the equirectangular branch and remains compatible with Bi-Projection.
  • Fusion module: 10.52% lower Abs Rel and 3.03% higher δ < 1.25 accuracy show that CEE substantially outperforms Bi-Projection under unidirectional fusion.CEE remains markedly better than Bi-Projection even when its SE block is disabled.

3) Complexity Analysis:

UniFuse improves performance with substantially lower complexity than BiFuse, while special padding and convolution methods add cost without improving UniFuse.

  • Model complexity: BiFuse’s fusion scheme nearly quadruples parameters and increases inference time to about 10 times the baseline.Its bidirectional modules, refinement network, and cubemap processing contribute to the added complexity.
  • Model complexity: UniFuse on ResNet-18 doubles baseline parameter and time complexity while remaining real-time at over 30fps.A similarly complex ResNet-50 baseline performs far worse, and MobileNetV2 reduces memory and parameters while retaining real-time inference.
  • Padding and convolution: Special padding methods increase inference time by over 10 times for CuP and over 20 times for SP without improving UniFuse performance.The paper therefore finds them unnecessary for UniFuse.
  • Padding and convolution: DaC improves the baseline but remains far inferior to UniFuse, while its dense interpolation produces much higher space and time complexity.Replacing the first layer with SC slightly improves the baseline but reduces UniFuse performance.

4) Generalization Capability:

UniFuse generalizes better than BiFuse in the reported comparisons, fitting Matterport3D more effectively and transferring better to Stanford2D3D.

  • Generalization comparison: BiFuse’s Abs Rel is 3 times UniFuse’s, while UniFuse achieves 8.0% higher δ < 1.25 accuracy.These results indicate better Matterport3D fitting for UniFuse.
  • Generalization comparison: UniFuse’s stronger Matterport3D fit is not overfitting because it also transfers better to Stanford2D3D.Visualization results further support this comparison, including plausible predictions where top and bottom ground-truth depth is unavailable.
Loading 2102.03550v2…