Source-linked AI summary
Decoupling Spherical Reasoning from Dense Prediction for 360 Depth Estimation
Zhijie Shen, Chunyu Lin, Shuai Zheng, Feng Li, Runmin Cong, Huihui Bai, Yao Zhao
TL;DR
ERP distortion makes geometry-consistent contextual modeling difficult for panoramic depth estimation. The paper uses FSG for spherical reasoning and SCC to transfer that context to dense ERP features, reporting improved accuracy over compared methods on three benchmarks.
Problem
ERP’s latitude-dependent distortion makes pixel-grid neighborhoods inconsistent with spherical spatial relationships, challenging geometry-consistent feature modeling.
Source: §1 · p. 1Method
FSG models contextual relationships on compact Fibonacci nodes over the sphere, while SCC transfers spherical context to dense, pixel-aligned ERP features.
Source: §1 · p. 2Results
The method improves depth-estimation accuracy over compared methods on Matterport3D, Stanford2D3D, and Structured3D.
Source: §5 · p. 9Takeaways & Limitations
The findings support dedicated spherical relational modeling alongside ERP prediction as a promising strategy for panoramic depth estimation.
Source: §5 · p. 9
Abstract
from arXiv · showhide
The equirectangular projection (ERP) is widely used for panoramic depth estimation, but its spatially varying distortion makes geometry-consistent feature modeling challenging. We revisit panoramic depth estimation by decoupling contextual modeling in native spherical space from dense ERP prediction. To this end, we propose a Fibonacci Spherical Graph (FSG) as an intermediate reasoning space to lift ERP features onto quasi-uniform Fibonacci nodes on the sphere and capture local and long-range dependencies through complementary spherical neighborhoods. The resulting spherical discretization distributes graph nodes approximately uniformly over the spherical surface, reducing the over-representation of highly stretched regions during relational modeling. Operating on a compact set of Fibonacci nodes also avoids the computational burden of constructing and processing a graph at full ERP resolution. To bridge spherical reasoning and dense prediction, we propose a Spherical Context Conditioning (SCC) module that adaptively modulates dense ERP features with the enhanced spherical representation, allowing spherical context to guide pixel-aligned depth prediction. Extensive experiments on three benchmarks demonstrate that the proposed method consistently achieves superior depth accuracy over existing approaches.
1 INTRODUCTION
ERP supports dense panoramic prediction, but its latitude-dependent distortion makes pixel-grid neighborhoods inconsistent with spherical relationships. The paper separates spherical contextual reasoning from dense ERP prediction using FSG and SCC, and reports improved accuracy on three benchmarks.
1 INTRODUCTION
- ERP’s latitude-dependent distortion makes equal-sized image-grid neighborhoods correspond to different spherical extents, complicating geometry-consistent feature modeling.Prior approaches adapt ERP sampling or add planar projections, but their feature interactions remain organized in projection spaces.Source: §1 · pp. 1–2
- A spherical reasoning space models feature relationships on the sphere while complementing, rather than replacing, ERP’s dense pixel-aligned prediction.The motivation is to aggregate contextual relationships according to their spherical geometry.Source: §1 · p. 2
- FSG provides a compact Fibonacci-node space for modeling contextual dependencies directly over spherical geometry, independently of ERP’s latitude-dependent deformation.This decouples geometry-aware reasoning from the pixel-aligned ERP representation.Source: §1 · p. 2
- SCC transfers geometry-aware spherical context to dense ERP features while preserving the spatial correspondence needed for depth estimation.FSG handles spherical scene reasoning; ERP features remain responsible for spatially aligned prediction.Source: §1 · p. 2
- The proposed method consistently improves depth-estimation accuracy over existing approaches on three panoramic depth benchmarks.Ablations also verify the effectiveness of spherical relational modeling and context conditioning.Source: §1 · p. 2
2 RELATED WORK
Prior panoramic depth methods address ERP distortion with adapted operations or complementary projections, while other work explores spherical representations and prediction. This paper instead uses a compact spherical space for relational reasoning while retaining ERP for dense prediction.
2 RELATED WORK
- ERP-based methods use distortion-aware convolutions, adaptive sampling, and geometry-aware receptive fields, but still define feature interactions on the distorted ERP grid.ERP remains popular for its dense rectangular layout and compatibility with standard CNN- and Transformer-based architectures.Source: §2 · pp. 2–3
- Multi-projection methods combine ERP with cubemap, tangent, or other views, yet relationships remain within projection domains or are formed through cross-projection fusion.These approaches improve distortion handling and global context.Source: §2 · p. 3
- Spherical depth methods use meshes, equal-area grids, or point representations, exploring the sphere as a prediction domain or auxiliary representation.Spherical CNNs, equivariant networks, and graph models also provide geometry-aware operators or spherical-neighborhood processing.Source: §2 · p. 3
- FSG differs by using Fibonacci nodes as a compact spherical reasoning space, while retaining ERP for dense pixel-aligned prediction.Its design organizes relational modeling according to spherical geometry rather than using the sphere as the final prediction representation.Source: §2 · p. 3
3 METHODOLOGY
The method separates spherical contextual reasoning from dense ERP prediction: FSG models relationships on Fibonacci nodes, while SCC uses spherical context to condition pixel-aligned ERP features.
3.1 OVERVIEW
- UniFuse’s ERP features are refined at four scales, from 1/4 to 1/32 resolution, before corresponding fusion stages: FSG maps each feature to spherical context, which SCC uses to produce its refined feature.The design retains UniFuse’s dual-projection encoders, fusion modules, and ERP decoder.Source: §3.1 · p. 3
3.2 FIBONACCI SPHERICAL REPRESENTATION
- Fibonacci nodes provide approximately uniform spherical coverage, and bilinear sampling maps ERP features at their viewing directions onto those nodes.The node count can be chosen independently of ERP resolution, while the original dense features remain available for prediction.Source: §3.2 · p. 4
3.3 MULTI-RELATION SPHERICAL GRAPH REASONING
- FSG connects each node to its k nearest nodes, the next k by distance, and k nodes nearest its antipode; learned node-wise weights adapt their combination.The relation affinities are fixed softmax-normalized cosine similarities, while the learned weights let feature responses determine each relation’s contribution.Source: §3.3 · pp. 4–5
3.4 SPHERICAL CONTEXT CONDITIONING
- SCC interpolates enhanced graph features into an ERP context map, then gates scale-and-shift modulation and context injection while retaining the original dense feature pathway.The ERP feature remains the carrier of dense information because interpolation does not guarantee recovery of its original variations.Source: §3.4 · p. 5
3.5 OBJECTIVE FUNCTION
- Training combines BerHu depth regression over valid pixels with spherical-gradient supervision of spatial depth variations, using c = 0.2 and λsg = 0.5.The BerHu threshold is 0.2; the spherical-gradient loss weight follows PGFuse and HUSH.Source: §3.5 · pp. 5–6
4 EXPERIMENTS
Experiments on three panoramic-depth benchmarks compare the method with prior approaches and evaluate its components. The reported results favor the complete method and support the roles of spherical node support, graph reasoning, SCC, and complementary spherical relations.
4.1 EXPERIMENTAL SETUP
- Evaluation covers real indoor panoramas from Matterport3D and Stanford2D3D and synthetic indoor scenes from Structured3D, using prior methods’ protocols and four depth metrics.The metrics are Abs Rel, Sq Rel, RMSE, and threshold accuracies δt for t ∈ {1, 2, 3}; implementation uses a ResNet-34 backbone and Adam.Source: §4.1 · pp. 6–7
4.2 COMPARISON WITH STATE-OF-THE-ART METHODS
- The method achieves the lowest errors and highest threshold accuracies among compared approaches on Matterport3D, Stanford2D3D, and Structured3D.The authors also report consistent performance in RMSE and δ metrics.Source: §4.2 · p. 7
- Qualitative comparisons show more coherent large-surface and room-scale depth structure while preserving local transitions around object and structural boundaries.Examples span all three benchmarks.Source: §4.2 · pp. 7–8, Figure 2 · p. 7
4.3 ABLATION STUDIES AND ANALYSIS
- Fibonacci node support reduces RMSE and improves δ1 over matched ERP-grid support, with gains across latitude regions and the largest improvement near the equator.The authors attribute this pattern to more uniform spherical support that reduces latitude-dependent redundancy in ERP-grid sampling.Source: §4.3 · pp. 8–9, Table 2 · p. 8, Table 4 · p. 9
- Adding graph reasoning to Fibonacci features improves both RMSE and δ1, indicating that balanced spherical sampling alone does not provide the full benefit.The ablation compares variants with and without graph reasoning.Source: §4.3 · p. 8, Table 2 · p. 8
- SCC improves all evaluation metrics over direct residual fusion and better preserves spatially localized depth structures through selective context transfer.The ablation retains graph-derived spherical context while changing how it is incorporated into ERP features.Source: §4.3 · p. 8, Table 2 · p. 8
- Combining local, intermediate-range, and long-range spherical relations achieves the best reported relation configuration, while adding only dilated relations to the local graph does not improve it.Adaptive relation fusion integrates the context paths.Source: Table 3 · p. 8, §4.3 · p. 9
- The method adds moderate computational overhead over UniFuse but uses fewer FLOPs and less training memory and has lower inference latency than PGFuse and HUSH at comparable accuracy.Spherical reasoning runs on compact Fibonacci nodes rather than full-resolution ERP features; the comparison uses 512 × 1024 inputs and batch size 1.Source: §4.3 · p. 9, Table 5 · p. 9
5 CONCLUSION
The conclusion presents FSG as an intermediate spherical reasoning space and SCC as the way to incorporate its context into dense ERP features. It reports improved depth accuracy on three benchmarks and identifies spherical sampling and combined graph reasoning and conditioning as beneficial.
5 CONCLUSION
- Experiments on Matterport3D, Stanford2D3D, and Structured3D demonstrate improved depth accuracy over the compared methods.The conclusion reports this outcome across all three benchmarks without specifying metric values.Source: §5 · p. 9
- Controlled node-support comparisons and component ablations support the benefits of spherical sampling and the joint use of graph reasoning and conditioning.The conclusion presents a dedicated representation for contextual interaction as a promising strategy for incorporating panoramic geometry into dense prediction pipelines.Source: §5 · p. 9
- The work focuses on monocular panoramic depth estimation, uses existing research datasets, and releases no new dataset.Experiments follow the datasets’ established evaluation protocols.Source: ETHICS STATEMENT · p. 10
- The paper describes its methods, datasets, metrics, implementation settings, and ablations, with additional details in the supplement, and states that source code will be released.The listed methodological details include Fibonacci spherical representation, multi-relation graph construction, spherical context conditioning, and the training objective.Source: REPRODUCIBILITY STATEMENT · p. 10
A.1 FIBONACCI NODES AND ERP SAMPLING
This appendix section describes fixed unit-sphere node generation and how node directions are mapped to ERP sampling coordinates. It also specifies padding and sampling behavior.
A.1 FIBONACCI NODES AND ERP SAMPLING
- The Fibonacci construction generates fixed nodes on the unit sphere, using a half-step offset to avoid placing nodes exactly at the poles.Source: §A.1 · p. 13
- Node directions are converted to latitude φ and longitude λ, with increasing φ corresponding to increasing ERP row index in the implementation’s coordinate convention.The section then defines pixel-center coordinates.Source: §A.1 · p. 13
- ERP features are bilinearly sampled after circularly padding one column on each horizontal side; node coordinates remain fixed and sampling is differentiable with respect to feature values.Sampling uses align_corners=False and border handling vertically.Source: §A.1 · p. 13
A.2 RELATION CONSTRUCTION AND GRAPH BLOCKS
This section constructs local, dilated, and antipodal graph neighborhoods and describes geometric weighting and relation fusion. Graph blocks update node features with residual normalization and feed-forward operations before producing enhanced representations.
A.2 RELATION CONSTRUCTION AND GRAPH BLOCKS
- Each node receives k messages from each of three relations: a local neighborhood including itself, a dilated neighborhood, and an antipodal neighborhood.The per-node neighbor lists are used without additional symmetrization.Source: §A.2 · p. 13
- Relative geometry encodes each propagation pair, and neighbor indices, geometry encodings, and aggregation weights are precomputed from node coordinates and fixed during training.The displacement and angular separation are computed from the actual propagation pair, including antipodal edges.Source: §A.2 · p. 13
- Within each relation, aggregation weights depend on spherical node coordinates, while learned feature-adaptive weights fuse the relation-specific responses.The post-aggregation operator applies PReLU and LayerNorm to the message plus a learned relation-specific bias; feature and geometric projections and relation-scoring functions are independent.Source: §A.2 · p. 13
- A graph block adds the fused responses to its input, applies LayerNorm, then adds a feed-forward network’s output after another residual step.The feed-forward network expands features from C to 2C, applies GELU, and projects back to C; two graph blocks are used at each scale, and the final projection yields enhanced node features for ERP back-projection.Source: §A.2 · p. 13
A.3 SPHERICAL BACK-PROJECTION
This section maps ERP pixels to unit-sphere directions and interpolates enhanced spherical node features back to ERP locations. It caches interpolation indices and weights while recomputing node features for each forward pass.
A.3 SPHERICAL BACK-PROJECTION
- Each ERP pixel at row v and column u is assigned a unit-sphere direction for spherical back-projection.Source: §A.3 · p. 14
- Back-projection selects spherical nodes associated with each pixel and interpolates their enhanced features.Source: §A.3 · p. 14
- Interpolation indices and weights are cached because they depend only on the node set and ERP resolution; input-dependent node features are interpolated on each forward pass.Sampling and back-projection use the same spherical coordinate convention.Source: §A.3 · p. 14
A.4 SCC OPERATORS AND INITIALIZATION
SCC separately normalizes ERP and context inputs, then combines affine modulation with gated local and context updates. The affine stage begins as an identity, while the gate is initially biased toward smaller values.
A.4 SCC OPERATORS AND INITIALIZATION
- The affine predictor maps C to 2C channels, splitting its outputs into S0 and T, and bounds only scale as S = 0.5 tanh(S0).ERP and context inputs are separately normalized by GroupNorm.Source: §A.4 · p. 14
- The gate applies a 1×1 convolution, GroupNorm, GELU, and a second convolution to the concatenation of normalized ERP features, context, and their difference, then sigmoid produces M.Source: §A.4 · p. 14
- Local and context branches form updates that are normalized before residual addition; the local depth-wise convolution uses horizontal circular and vertical replicate padding.All GroupNorm operators use eight groups for the channel configurations in this model.Source: §A.4 · p. 14
- Zero-initializing the affine predictor makes the affine stage an identity initially, while the final gate convolution’s −2 bias favors smaller gate values at training start.Source: §A.4 · p. 14
A.5 MODULE CONFIGURATIONS
Table 6 summarizes FSG and SCC configurations across feature scales. The node count adapts to feature resolution, while graph relations are independently parameterized at each scale.
A.5 MODULE CONFIGURATIONS
- Table 6 summarizes the spherical modeling module configurations at different feature scales.Source: Table 6 · p. 14
- The node count N is selected per ERP feature resolution as the nearest Fibonacci number to its node budget, decreasing from fine to coarse scales.C denotes feature dimension, k the neighborhood size per spherical relation, and k_b the neighbor count for spherical-to-ERP back-projection.Source: §A.5 · p. 14
- Each scale has independently parameterized local, dilated, and antipodal relations; two graph blocks are used at every scale, while the original UniFuse 1/2-scale fusion branch is retained.Each cubemap face has side length H_in/2, where H_in is the input ERP height.Source: §A.5 · p. 15