PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniFusion: 360 Monocular Depth Estimation via Geometry-Aware Fusion

Yuyan Li, Yuliang Guo, Zhixin Yan, Xinyu Huang, Ye Duan, Liu Ren

arXiv:2203.00838v2cs.CV

TL;DR

Spherical distortion and independent tangent-patch predictions complicate consistent depth estimation from 360 images. OmniFusion combines tangent-image predictions with geometry-aware fusion, transformer aggregation, and iterative refinement, and reports state-of-the-art performance on three benchmark datasets.

  • Problem

    ERP distortion varies vertically and may degrade regular convolutional layers, while independent tangent-image predictions can produce inconsistent depth scales and merging artifacts.

    Source: §1 · pp. 1–2

  • Method

    OmniFusion predicts depth from tangent-image patches, fuses geometric features with image features, globally aggregates patch information with a transformer, and iteratively refines geometry.

    Source: §1 · p. 2, §3.1 · p. 3, §3.5 · p. 5

  • Results

    OmniFusion outperforms state-of-the-art methods by a significant margin on Stanford2D3D, Matterport3D, and 360D.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that OmniFusion effectively mitigates distortion in 360 depth estimation and improves depth quality with more structural details.

    Source: §5 · p. 8

Abstract

from arXiv · show

A well-known challenge in applying deep-learning methods to omnidirectional images is spherical distortion. In dense regression tasks such as depth estimation, where structural details are required, using a vanilla CNN layer on the distorted 360 image results in undesired information loss. In this paper, we propose a 360 monocular depth estimation pipeline, OmniFusion, to tackle the spherical distortion issue. Our pipeline transforms a 360 image into less-distorted perspective patches (i.e. tangent images) to obtain patch-wise predictions via CNN, and then merge the patch-wise results for final output. To handle the discrepancy between patch-wise predictions which is a major issue affecting the merging quality, we propose a new framework with the following key components. First, we propose a geometry-aware feature fusion mechanism that combines 3D geometric features with 2D image features to compensate for the patch-wise discrepancy. Second, we employ the self-attention-based transformer architecture to conduct a global aggregation of patch-wise information, which further improves the consistency. Last, we introduce an iterative depth refinement mechanism, to further refine the estimated depth based on the more accurate geometric features. Experiments show that our method greatly mitigates the distortion issue, and achieves state-of-the-art performances on several 360 monocular depth estimation benchmark datasets.

1. Introduction

ERP distortion can undermine depth estimation, while independent tangent-image predictions can yield inconsistent depth scales and merging artifacts. OmniFusion combines geometric and image features, global patch aggregation, and iterative refinement, and reports state-of-the-art results across three benchmarks.

1. Introduction

  • ERP distortion varies vertically and may degrade regular convolutional layers designed for perspective images.The paper identifies this as a challenge for 360-image depth estimation.Source: §1 · p. 1
  • Independent tangent-image predictions lack information exchange, and inconsistent depth scales can create artifacts when their outputs are merged.The same object may also appear differently across perspective views.Source: §1 · pp. 1–2
  • OmniFusion encodes spherical geometry to compensate for discrepancies between patch-wise image features.The geometric embedding combines encoded 3D points with patch-center coordinates and adds the resulting features to corresponding 2D features.Source: §1 · p. 2
  • A self-attention transformer aggregates patch-wise information globally, while iterative refinement feeds more accurate predicted 3D information back into geometric embedding.The transformer is intended to improve global depth-scale estimation and consistency; refinement further improves depth quality.Source: §1 · p. 2
  • OmniFusion outperforms state-of-the-art methods by a significant margin on Stanford2D3D, Matterport3D, and 360D.The introduction reports results on all three benchmarks.Source: §1 · p. 2

2. Related Work

Prior monocular depth work explores network design, supervision, and additional modalities, while 360-depth research addresses spherical distortion through specialized convolutions and alternative image representations. OmniFusion follows tangent-image approaches and adds geometry-aware fusion and transformer aggregation to address patch discrepancies.

2.1. Monocular depth estimation

  • Monocular depth research has explored network architecture, supervision, and unsupervised stereo or video, while sensor-assisted methods add computation or power demands.The passage notes sensitivity to camera-intrinsic changes and generalization challenges for unsupervised approaches.Source: §2.1 · p. 2

2.2. 360 depth estimation

  • 360-depth methods include spherical or distortion-aware convolutions, cubemap representations, and tangent images, whose cross-view discrepancies can degrade merged results.OmniFusion follows tangent-image work and adds geometry-aware fusion and a transformer to address discrepancies.Source: §2.2 · p. 2

2.3. Transformer

  • Self-attention can exploit long-range context, but ERP distortion may weaken its use of pairwise patch correlations.OmniFusion feeds its transformer distortion-free, geometry-aware inputs for global patch aggregation.Source: §2.3 · p. 3

3. Method

OmniFusion converts 360 ERP images into tangent patches, then fuses patch-wise depth estimates into an ERP output. Geometry-aware features, global transformer aggregation, confidence-weighted merging, and iterative refinement address patch discrepancies and improve depth consistency.

3. Method

  • OmniFusion projects ERP images into tangent patches, predicts patch-wise depth, and fuses the estimates into an ERP depth output.The pipeline also globally aggregates patch features and iteratively refines depth using updated spherical coordinates.Source: §3 · p. 3

3.1. Depth estimation from tangent images

  • Tangent images use gnomonic projection to provide less-distorted views that can be processed with regular encoder-decoder CNNs.The experiments sample 18 patches across four latitudes, using fewer patches near the poles to limit extreme overlap.Source: §3.1 · pp. 3–4

3.2. Geometry-aware feature fusion

  • Geometric embeddings combine spherical pixel coordinates and patch-center coordinates with image features to reduce patch-wise feature discrepancies.The spherical coordinates identify global position, while patch-center coordinates help distinguish patches and align their image features.Source: §3.2 · pp. 4–5, Figure 5 · p. 5

3.3. Global aggregation with transformer

  • A self-attention transformer globally aggregates patch features to improve consistency and estimate depth scale across the broader field of view.Encoder feature maps are reduced, flattened into patch tokens, and combined with learned positional embeddings.Source: §3.3 · p. 5

3.4. Depth merging with learnable confidence map

  • Each patch predicts depth and confidence, and confidence scores weight the patch depths when merging them into the ERP output.Both maps are inverse-projected to the ERP domain before merging.Source: Figure 4 · p. 4, §3.4 · p. 5

3.5. Iterative depth refinement

  • After the first iteration, estimated depth updates the spherical coordinate ρ, providing more accurate geometry for the next embedding step.The first iteration uses unit-sphere coordinates because depth is initially unavailable.Source: §3.5 · pp. 5–6

4. Experiments

OmniFusion is evaluated on three 360-depth benchmarks, with its two-iteration setting outperforming prior methods across datasets. Ablations examine component contributions, patch configurations, and encoder and iteration choices.

4.1. Datasets

  • The evaluation covers Stanford2D3D, Matterport3D, and 360D, using the datasets’ stated official or default train-test splits.The reported input resolutions are 512 × 1024 for Stanford2D3D and Matterport3D, and 256 × 512 for 360D.Source: §4.1 · p. 6

4.2. Implementation details

  • Training uses a pre-trained ResNet encoder, end-to-end optimization, BerHu depth supervision, and summed depth losses across iterations.The default configuration uses 18 patches; Stanford2D3D is trained for 80 epochs, and Matterport3D and 360D for 60 epochs.Source: §4.2 · p. 6

4.3. Overall performance

  • With 2-iter, OmniFusion beats BiFuse in Abs Rel by 21.4% on Stanford2D3D, 56.1% on Matterport3D, and 30% on 360D, achieving state-of-the-art performance across all three datasets.The paper also reports improvements over UniFuse in Abs Rel on each dataset.Source: §4.3 · p. 6, Table 1 · p. 7

4.4. Ablation studies

  • On Stanford2D3D, geometry-aware fusion improves Abs Rel by 9.7% with fewer than 2K added parameters, while the transformer adds a further 5.7% improvement.Together, the components improve Abs Rel by 15.4% with 1-iter and 16.4% with 2-iter.Source: §4.4 · p. 6, Table 2 · p. 7
  • The patch study finds that patches cannot be too small for depth-scale estimation, while using 26 or more patches can worsen performance as overlap intensifies discrepancy.The selected balance is 18 patches at 256 × 256 resolution.Source: §4.4 · pp. 6–7, Table 3 · p. 8
  • ResNet34 outperforms ResNet18, and additional test-time iterations improve results through three iterations but provide no gain from three to four.The authors choose 1-iter or 2-iter to balance performance and speed.Source: §4.4 · pp. 7–8, Table 4 · p. 8

5. Conclusion

OmniFusion uses tangent images, geometric feature fusion, a transformer, and iterative refinement for 360 monocular depth estimation. The paper reports that it mitigates distortion and achieves state-of-the-art performance on several datasets.

5. Conclusion

  • OmniFusion uses gnomonic-projection tangent images, geometry-aware fusion, transformer aggregation, and iterative refinement to address distortion and improve patch-wise depth estimation.The geometric features are fused with image features, while the transformer globally aggregates patch information and refinement updates spherical coordinates.Source: §5 · p. 8

A. Gnomonic projection

OmniFusion uses gnomonic tangent images to represent 360 imagery with less distortion. Forward and inverse mappings connect ERP pixels, sphere points, and tangent-image pixels.

A. Gnomonic projection

  • Gnomonic projection maps points on a sphere onto a flat plane tangent to a point on the sphere, forming the tangent-image representation.The paper uses this less-distorted representation to address irregular 360-image distortion.Source: A. Gnomonic projection · p. 11
  • An ERP pixel is first mapped to its corresponding point on the unit sphere, then projected to a tangent-image pixel.The projection uses the tangent plane center’s spherical coordinates.Source: A. Gnomonic projection · p. 11
  • Forward and inverse gnomonic transformations provide one-to-one mappings between ERP-image pixels and tangent-image pixels.The inverse transformation maps tangent-image coordinates back to the sphere and ERP representation.Source: A. Gnomonic projection · p. 11

B. Geometry-aware feature fusion

Geometry-aware feature fusion combines patch-wise image and geometric features to address discrepancies between tangent views. The geometric embedding uses global point position and patch-center coordinates to compensate for those differences.

B. Geometry-aware feature fusion

  • Patch-wise 2D image features are fused with geometric features, and the merged ERP feature map appears more locally consistent after fusion.The paper expects this local consistency to carry over to depth results.Source: B. Geometry-aware feature fusion · p. 11
  • With fixed spherical-coordinate inputs, geometric features remain fixed after learning; in the second iteration, image-dependent geometry is recomputed using lightweight MLPs.The first iteration therefore requires no extra inference computation for those fixed geometric features.Source: B. Geometry-aware feature fusion · p. 11
  • The same sphere point can produce different image features in different tangent views, while its global position and each patch center provide geometric cues for compensation.The point encoding uses the sphere point for global position and the patch center to distinguish patches.Source: B. Geometry-aware feature fusion · p. 11

C. Transformer Architecture and Ablation Study

The architecture uses multi-head self-attention to aggregate features, with the transformer configuration selected through an ablation study. The accompanying feature-map visualization contrasts merged patch features with and without geometric fusion.

C. Transformer Architecture and Ablation Study

  • The figure compares merged ERP feature maps without and with geometric fusion, with the fused map appearing more locally consistent.The pipeline visualization shows tangent-image feature maps and geometric feature maps being fused patch-wise.Source: Figure 9 · p. 12
  • The geometric features in the figure are used to compensate for image-feature discrepancies attributed to gnomonic transformation.The illustration depicts geometry-aware compensation in feature space.Source: Figure 10 · p. 12
  • Multi-head self-attention computes attention-weighted values from query, key, and value projections of its input features.The transformer output is reshaped, expanded with a 1×1 convolution, and added as a residual.Source: C. Transformer Architecture and… · pp. 12–13
  • The ablation study selected 6 transformer blocks and 4 heads as the default, a configuration reported to tend toward fewer errors and higher inlier ratios.These configuration experiments used ResNet18 rather than the ResNet34 final pipeline.Source: Table 5 · p. 12, C. Transformer Architecture and… · p. 13

D. Loss Function

The network is trained end-to-end with BerHu loss for depth predictions from all iterations. Its residual uses a mask for invalid pixels and a batch-dependent threshold.

D. Loss Function

  • BerHu loss optimizes the depth predictions from all iterations in the end-to-end training objective.The final loss combines the losses from the iterations.Source: D. Loss Function · p. 13
  • BerHu applies absolute error when the residual is at most c and a quadratic form when it exceeds c.The paper defines c as 0.2 times the maximum residual per batch.Source: D. Loss Function · p. 13
  • The residual is the masked absolute difference between ground-truth and predicted depth, with invalid depth pixels excluded.The mask is binary, and the threshold c is set to 20% of the maximum residual per batch.Source: D. Loss Function · p. 13

E. Generalization

Cross-dataset evaluation found that OmniFusion generalizes better than the compared state-of-the-art methods when trained on Matterport3D and tested on Stanford2D3D.

E. Generalization

  • All methods were trained on the Matterport3D training set and evaluated on the Stanford2D3D test set.Source: E. Generalization · p. 13
  • Official pre-trained models and evaluation code from UniFuse and HoHoNet were used for a fair comparison.Source: E. Generalization · p. 13
  • OmniFusion showed superior generalization ability to the compared state-of-the-art methods in cross-dataset evaluation.Table 6 reports evaluation on Stanford2D3D after training on Matterport3D.Source: Table 6 · p. 13, E. Generalization · p. 13

F. Additional qualitative comparisons

Qualitative comparisons cover Stanford2D3D, Matterport3D, and 360D, with additional OmniFusion results on Matterport3D. The comparisons report more structural detail, sharper edges, smoother surfaces, and fewer errors in OmniFusion's depth maps.

F. Additional qualitative comparisons

Related papers

Loading 2203.00838v2…