PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

Hao Ai, Zidong Cao, Jinjing Zhu, Haotian Bai, Yucheng Chen, Lin Wang

arXiv:2205.10468v2cs.CV

TL;DR

Omnidirectional vision offers wider fields of view and richer spatial information than planar imagery, but its distortions, projections, and annotation costs complicate deep learning. This paper systematically surveys methods, datasets, taxonomies, learning strategies, applications, and open problems, synthesizing more than 200 representative works and proposing future research directions. It connects current approaches across tasks while identifying data efficiency, multimodal learning, distortion handling, and wider-context understanding as continuing opportunities.

  • Problem

    Omnidirectional vision provides richer spatial information but requires methods that address spherical distortion, projection differences, costly annotations, and task-specific challenges.

  • Method

    The paper conducts a comprehensive survey of deep learning methods, datasets, hierarchical taxonomies, learning strategies, applications, challenges, and future directions for omnidirectional vision.

  • Results

    The survey synthesizes over 200 representative works and analyzes recent trends, methods, applications, and unresolved problems across omnidirectional vision.

  • Takeaways & Limitations

    The synthesis identifies data-efficient learning, distortion-aware multimodal processing, and broader spatial-context modeling as promising directions for omnidirectional vision.

  • Takeaways & Limitations

    Omnidirectional vision remains constrained by scarce task-specific datasets, projection distortion, expensive annotation, and incomplete spatial-context understanding.

Abstract

from arXiv · show

Omnidirectional image (ODI) data is captured with a 360x180 field-of-view, which is much wider than the pinhole cameras and contains richer spatial information than the conventional planar images. Accordingly, omnidirectional vision has attracted booming attention due to its more advantageous performance in numerous applications, such as autonomous driving and virtual reality. In recent years, the availability of customer-level 360 cameras has made omnidirectional vision more popular, and the advance of deep learning (DL) has significantly sparked its research and applications. This paper presents a systematic and comprehensive review and analysis of the recent progress in DL methods for omnidirectional vision. Our work covers four main contents: (i) An introduction to the principle of omnidirectional imaging, the convolution methods on the ODI, and datasets to highlight the differences and difficulties compared with the 2D planar image data; (ii) A structural and hierarchical taxonomy of the DL methods for omnidirectional vision; (iii) A summarization of the latest novel learning strategies and applications; (iv) An insightful discussion of the challenges and open problems by highlighting the potential research directions to trigger more research in the community.

1 INTRODUCTION

Omnidirectional vision has gained momentum as 360° cameras and deep learning enable broad applications and strong results, while ODI-specific distortions and representations distinguish it from planar vision. This survey organizes the field’s methods, applications, learning strategies, and open problems.

  • Motivation: ODIs capture a 360° × 180° field of view and richer spatial information than planar images, but ERP distortion and CP discontinuities create domain-specific challenges.ODI data supports applications including augmented and virtual reality, autonomous driving, and robot navigation.
  • Motivation: Public datasets and deep learning have enabled breakthroughs and often state-of-the-art performance across omnidirectional vision tasks.Representative datasets include SUN360, Salient 360!, Stanford2D3D, Pano-AVQA, and PanoContext.
  • Survey scope: The survey comprehensively reviews ODI imaging, representation learning, datasets, a hierarchical taxonomy, applications, and differences from planar image data.It presents this coverage as the first comprehensive survey of deep learning methods for omnidirectional vision.
  • Contributions: It analyzes recent trends across more than 200 representative papers, summarizes novel learning strategies and applications, and discusses challenges and future directions.The survey also provides analytical category-level insights and an open-source repository with taxonomy and code links.
  • Organization: The paper proceeds from imaging principles, convolutions, and datasets to task taxonomies, novel learning paradigms, applications, open problems, and conclusions.Covered learning paradigms include unsupervised, transfer, and reinforcement learning.

2 BACKGROUND

ODI acquisition and representation rely on spherical imaging and projections such as ERP, CP, and tangent formats, each shaping distortion and computation. Deep learning methods therefore adapt planar convolutions or operate directly on spherical structures, while datasets vary in source and scale.

  • Acquisition: 360° cameras capture views beyond a hemisphere using fisheye, dual-fisheye, or multi-lens designs, with stitching potentially causing edge blurring.Practical cameras may omit top and bottom regions because of dead angles.
  • Projection representations: ERP uniformly samples the sphere, CP maps it onto six cube faces, and tangent projection maps spherical points onto a tangent plane.These representations provide different sphere-to-plane mappings for omnidirectional processing.
  • Projection representations: Tangent projection supports one-to-one forward and inverse mappings between spherical coordinates and tangent-image pixels, while icosahedron projection reduces spherical distortion through sphere-surface approximation and subdivision.Increasing icosahedron subdivisions can further mitigate distortion when lower-distortion representations are needed.
  • Spherical stereo: Spherical stereo is more irregular than pinhole stereo: a vertical baseline leaves longitude unchanged, whereas a horizontal baseline changes both angular coordinates.The baseline is represented in Cartesian coordinates and induces angular differences on the sphere.
  • Convolution methods: Because planar CNNs poorly handle spherical distortions, ODI methods either apply adapted 2D filters to projections or directly use spherical-domain convolutions.ERP-based methods address severe polar distortion with coordinate-dependent filters, while spherical approaches include harmonic, graph, and rotation-equivariant designs.
  • Datasets: ODI datasets differ in size, resolution, source, and annotation availability, and include both real-world and synthetically rendered data for varied vision tasks.Table 1 summarizes representative image and video datasets, with N/A denoting unavailable information and GT denoting ground truth.

3.1 Image/Video Manipulation

Image and video manipulation methods address ODI-specific generation, cross-view translation, compression, illumination, super-resolution, upright adjustment, and quality assessment challenges caused by spherical geometry and high resolution.

  • Image Generation: ODI image generation restores or synthesizes complete, clean panoramas from partial or noisy data through depth, semantic-map, completion, and view-synthesis directions.OmniNeRF projects an ODI into 3D using depth and gradient information before re-projecting a novel viewpoint and learning with NeRF.
  • Cross-view Synthesis and Geo-localization: Cross-view synthesis and geo-localization connect ground-view panoramas with satellite imagery through staged synthesis, contrastive embeddings, retrieval, and attention-based vision transformers.TransGeo removes uninformative satellite patches and surpasses previous CNN-based methods, while many methods assume precisely centered reference images.
  • Compression: ODI compression is difficult because omnidirectional data combines richer geometry, higher resolution, and wider FoV than perspective images, motivating adaptive, region-based, graph, and spherical-domain coding.A spherical convolution approach defines convolution directly on the sphere and outperforms similar learnable compression models operating on ERP images.
  • ODI Super-Resolution (SR): Super-resolution methods target HMD-scale output by adapting processing to latitude, continuous spherical queries, and multi-frame information, including LAU-Net, SphereSR, and SMFN.Current camera systems cannot directly capture the at least 21600x10800 pixels required by existing HMD devices.
  • Upright Adjustment and Quality Assessment: Upright adjustment estimates the ODI pole or up-vector and rotates the panorama, while ODI and ODV quality assessment extends conventional metrics or aggregates viewport-specific spatial and temporal quality.ODV-QA can propose viewports, score each viewport, and combine the scores using a weighted average.

3.2 Scene Understanding

Scene-understanding research adapts detection, segmentation, depth, summarization, and optical-flow methods to spherical distortion, scarce annotations, irregular projections, and the need for globally coherent predictions.

  • Object Detection: ODI object detection must address both irregular projection grids and spherical evaluation geometry, motivating distortion-aware networks and spherical bounding boxes with SphIoU or FoV-IoU.FoV-IoU better approximates exact IoU between FoV bounding boxes than SphIoU without requiring extra movement.
  • Semantic Segmentation: ODI semantic segmentation faces projection distortion, object deformation, computational cost, and scarce labels, so methods synthesize panoramas, transfer perspective knowledge, and use domain adaptation.PASS avoids fully dense panoramic annotations by aggregating features represented by conventional perspective images; DS-PASS further adapts perspective-domain knowledge and enhances spatial detail.
  • Monocular Depth Estimation: Monocular depth estimation combines distortion-aware representations, multiple projections, vision transformers, geometric priors, and synthesized viewpoints to improve panoramic predictions.Geometric priors include edges, planes, surface normals, boundaries, and uncertainty, while multi-view methods exploit synthesized stereo viewpoints for self-supervision.
  • Monocular Depth Estimation: Most panoramic depth methods focus on indoor scenes because geometric priors may fail outdoors and outdoor scale ambiguity and varied object shapes increase difficulty.The discussion specifically identifies plane assumptions as ineffective in the wild and sky regions as approximately infinite.
  • Video Summarization: ODV summarization remains less developed than 2D video summarization, spanning offline event linking and online navigation or normal-view generation based on viewer-like behavior.Pano2Vid requires observing the whole video and is therefore less capable for streaming, whereas Deep360Pilot selects important objects online and AutoCam generates NFoV videos.

3.3 3D Vision

ODI 3D-vision methods use panoramic context for room-layout recovery, stereo depth, and SLAM while adapting representations and geometric reasoning to spherical projections.

  • Room Layout Estimation and Reconstruction: Room-layout estimation and reconstruction combines edge alignment, layout-element prediction, and 3D recovery to support holistic scene understanding from a single ODI.Panoramas provide wider surrounding context than perspective images, while representative pipelines use multiprojection or compact 1D layout representations.
  • Room Layout Estimation and Reconstruction: Final room-layout recovery optimizes mappings between walls and corners, and HorizonNet handles boundary occlusions using adjacent-wall PCA orientation and long-term global geometry.The general recovery strategy uses weighted probability-map losses for floors, ceilings, and corners.
  • Stereo Vision: Omnidirectional stereo requires spherical correspondence modeling because its geometry is more irregular than pinhole stereo; 360SD-Net adds polar-angle coordinates and a learnable cost volume.The method takes top-bottom ODI pairs as input in an end-to-end trainable network.
  • SLAM: SLAM combines camera modalities and onboard sensors to reconstruct environments and estimate agent pose in real time, with stereo keypoints or depth maps providing indispensable information.The passage situates SLAM in applications including real-time navigation and reality augmentation.

3.4 Human Behavior Understanding

Human-behavior understanding in omnidirectional vision includes saliency prediction, gaze behavior, audio-visual scene understanding, and VQA, using projection-aware and multimodal learning to exploit wider scene context.

  • Saliency Prediction: ODI saliency methods progress from projected 2D models toward ODI-aware convolutions, viewport aggregation, multi-projection fusion, and attention-based learning.Representative categories include planar projections, ODI-aware CNNs, viewport saliency aggregation, mixed projection inputs, and attention mechanisms.
  • Saliency Prediction: ERP and CP inputs are fused in SalBiNet360 to predict global and local saliency maps, addressing ERP pole distortion and CP boundary discontinuity.Multi-domain approaches additionally combine spherical-harmonic frequency components with ERP viewports.
  • Gaze Behavior: GazeFollow360 is the first ODI gaze-following dataset, while ODI gaze prediction combines temporal saliency, spatial saliency, and gaze-history paths with CNN-LSTM models.ODI gaze tasks use the sphere’s wider scene context to locate gaze targets in local and distant regions.
  • Audio-Visual Understanding and VQA: ODI audio-visual understanding uses spatial audio cues for full-scene awareness, while VQA 360° introduces a cubemap-based framework for omnidirectional question answering.These tasks exploit the wider FoV and spatial information of ODIs and ODVs.

4 NOVEL LEARNING STRATEGIES

Novel learning strategies address limited ODI annotations and domain differences through semi-supervised, adversarial, attention-based, transfer, reinforcement, and multitask learning.

  • Unsupervised/Semi-supervised Learning: Semi-supervised and unsupervised learning exploit abundant unlabeled ODI data to mitigate insufficient and costly panorama annotations.Mean-Teacher learning uses labeled and unlabeled data for 3D room-layout reconstruction.
  • GAN: GAN-based methods reduce perspective-to-ODI domain divergence for semantic segmentation and synthesize RGB-D indoor panoramas from camera and depth-sensor configurations.They introduce adversarial learning into both domain adaptation and panorama generation.
  • Attention Mechanism: Attention mechanisms remove uninformative patches or model spatial relationships, reallocating computation toward informative regions and improving omnidirectional representations.ViT-based non-uniform cropping and self-attention are used in localization and saliency prediction.
  • Transfer Learning: Transfer learning adapts knowledge from labeled perspective-image models to ODI tasks including semantic segmentation and depth estimation.Deformable CNNs or MLPs can be designed on pretrained perspective models to improve ODI capability.
  • Deep Reinforcement Learning: Deep reinforcement learning selects actions such as head-fixation policies, adaptive ERP upscaling factors, and object-related decisions under omnidirectional geometry.These applications treat head movements as discrete actions or allocate resources according to uneven ERP pixel density.

5 APPLICATIONS

Omnidirectional vision supports immersive AR/VR, telepresence, remote navigation, and heading estimation by capturing complete surrounding environments in a single view.

  • AR and VR: ODIs support AR/VR by providing 360° × 180° coverage for immersive experiences and personalized interaction with real or simulated environments.Their complete surrounding view is central to interactive scenarios.
  • Telepresence: ODV telepresence systems enable remote environment exploration and more natural interaction, including outdoor activity scenarios.The systems aim to overcome spatial constraints through realistic omnidirectional scene capture.
  • Navigation: Panoramic imagery can reduce computation relative to NFoV images by providing complete FoV in one shot, supporting obstacle avoidance and heading-direction estimation.A lightweight uncalibrated-camera framework formulates heading estimation as a sequence of classification tasks.

6 DISCUSSION AND NEW PERSPECTIVES

The survey identifies projection trade-offs, data-efficiency needs, physical and multimodal opportunities, security risks, and application directions for future omnidirectional vision research.

  • Projection Formats: ERP offers wide planar FoV but suffers increasing polar stretching, whereas CP and tangent images reduce distortion at the cost of computation, discrepancy, and discontinuity.These trade-offs motivate distortion-aware filters and more efficient processing across projection planes.
  • Data-efficient Learning: Data-efficient learning is needed because large, high-quality ODI datasets are expensive; domain adaptation, knowledge distillation, and self-supervised pretraining are proposed remedies.These approaches transfer information from labeled perspective data or pretrained models to panoramic learning.
  • Physical Constraint: Physical constraints and rendering models could improve global lighting and reflectance estimation by leveraging the complete surrounding scene captured by ODIs.The survey highlights ray tracing, rendering, and lighting-transport approaches as future directions.
  • Multi-modal Omnidirectional Vision: Multimodal omnidirectional learning should exploit complete FoV while accounting for acquisition discrepancies and maintaining consistency across modalities such as RGB, depth, LiDAR, text, and audio.Knowledge transfer and multitask learning are suggested for modalities with unequal availability.
  • Security and Applications: Omnidirectional models face emerging security and application questions, including sparse tangent-image attacks and opportunities in metaverse and smart-city systems.Future applications include high-fidelity digital models, behavior analysis, urban planning, and interior design.

7 CONCLUSION

The survey synthesizes deep-learning progress in omnidirectional vision, organizes methods and applications, and identifies challenges and promising future directions.

  • The survey reviews recent deep-learning methods for omnidirectional vision and analyzes their research progress.
  • It introduces omnidirectional imaging principles, convolution methods, and datasets before constructing a hierarchical and structural taxonomy.
  • For each taxonomy task, it summarizes the research status and identifies opportunities and challenges.
  • The survey also reviews novel learning strategies and applications, connects existing approaches, and proposes promising future research directions.

1 COMPLETE DATASETS OF ODI AND ODV

ODI and ODV datasets are publicly available in real-world and synthetic forms, vary in size, resolution, and source, and remain unevenly distributed across tasks.

  • Public ODI and ODV datasets support deep-learning development and differ in size, resolution, and data source.
  • ODI datasets include real-world and synthetic data, with real-world collections concentrated on indoor tasks such as layout reconstruction, detection, and depth estimation.
  • Real-world optical-flow and gaze-flow datasets remain comparatively limited because acquisition is difficult.
  • ODV datasets are mostly real-world sequences obtained from Vimeo and YouTube, with several VQA datasets but fewer datasets for other tasks.

2 COMPARISON OF THE REPRESENTATIVE METH-

Comparisons across representative omnidirectional-vision methods show gains from distortion-aware projections, tailored architectures, transformers, and efficient modeling, while annotation and benchmark limitations remain.

  • 2.1 Comparison of Semantic Segmentation: For semantic segmentation, icosahedron projection reduces spherical distortion, transformer methods outperform CNN methods, and deformable models improve distorted-ODI modeling.
  • 2.2 Comparison of Monocular Depth Estimation: OmniFusion outperforms other monocular-depth methods on most metrics, while tangent images and attention-based transformers improve structure prediction but conversion reduces efficiency.
  • 2.2 Comparison of Monocular Depth Estimation: Compact methods such as UniFuse and HoHoNet achieve real-time speeds above 30 FPS, while UniFuse outperforms BiFuse in both efficiency and accuracy.
  • 2.3 Comparison of Room Layout Reconstruction: Led2-net achieves the best overall room-layout performance, whereas AtlantaNet surpasses it on 2D and 3D IOU in more-corner scenarios.
  • 2.3 Comparison of Room Layout Reconstruction: Later room-layout methods improve speed and accuracy on challenging layouts, whereas earlier methods struggle with many corners and distortion.
  • 2.4 Comparison of Saliency Prediction: Saliency prediction still lacks a large-scale comparable ODV benchmark, and predictions remain more continuous and blurred than discrete ground-truth saliency points.
Loading 2205.10468v2…