PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-IGN: Onboard Image-Goal Navigation for Drones with Omnidirectional Vision

Zichen Yan, Rui Huang, Shenao Wang, Yucong Wang, Nuthasith Gerdpratoom, Lin Zhao

arXiv:2610.36968v1cs.RO

TL;DR

Drone ImageNav must coordinate exploration, goal-directed flight, and collision avoidance despite limited onboard resources. Omni-IGN combines quad-fisheye perception, an attention-based continuous-action policy, and stereo-depth-supported safety correction. It performs best among continuous-action baselines on Habitat benchmarks and demonstrates zero-shot sim-to-real transfer in real-world flights.

  • Problem

    Aerial ImageNav methods rely on single-view observations, while drones must coordinate exploration, goal seeking, and collision avoidance within limited onboard computing resources.

    Source: §I · p. 1

  • Method

    Omni-IGN is a fully onboard aerial ImageNav system that adaptively fuses goal-conditioned features across four views and uses stereo depth with a safety layer to correct unsafe actions.

    Source: §I · pp. 1–2

  • Results

    Omni-IGN achieves state-of-the-art performance among continuous-action baselines on Habitat benchmarks, and real-world flights demonstrate zero-shot sim-to-real transfer without fine-tuning.

    Source: §I · p. 2

  • Takeaways & Limitations

    Panoramic observations improve navigation efficiency and robustness, and the system transfers to real-world flights without fine-tuning.

    Source: §VII · p. 10

Abstract

from arXiv · show

Image-goal navigation (ImageNav) requires an agent to reach a location specified by a goal image in an unknown environment. For drones, this task demands exploration, goal- directed navigation, and collision avoidance under limited onboard computing resources. Existing aerial ImageNav methods are built on single-view observations, requiring additional rotation and exploration to search for goal-relevant cues outside the field of view. In this paper, we present Omni-IGN, a fully onboard ImageNav system that integrates quad-fisheye perception, an attention-based reinforcement learning (RL) policy, and a collision-aware safety layer. The policy adaptively fuses goal-relevant features across multiple views to provide directional guidance for continuous navigation. The quad-fisheye RGB images support both ImageNav and obstacle avoidance without a dedicated depth sensor. On Habitat benchmarks, Omni-IGN achieves the best performance among continuous-action baselines. Moreover, flight tests in IsaacSim validate the effectiveness of integrating learning-based depth estimation with the action safety layer for collision avoidance. Finally, real-world experiments demonstrate zero-shot sim-to-real transfer without any fine-tuning or external localization. The code and model checkpoints will be released upon acceptance.

I. INTRODUCTION

Omnidirectional ImageNav could help drones find image-specified goals, but requires coordinating exploration, goal seeking, and collision avoidance within onboard computing limits. Omni-IGN combines four-view goal-conditioned navigation with stereo depth and safety correction in a fully onboard system.

I. INTRODUCTION

  • ImageNav requires searching an unknown 3D environment for the location where a goal image was captured, without a prescribed map or trajectory.The image-specified destination can reduce manual preparation in unknown or unstructured environments.Source: §I · p. 1
  • The central challenge is coordinating exploration, goal seeking, and collision avoidance when goal cues may be ambiguous, absent, or outside the forward view.Indiscriminate processing of all views can add computation and visual distraction under limited onboard resources.Source: §I · p. 1
  • Omni-IGN combines a quad-fisheye RGB rig, an end-to-end RL policy, and stereo-depth-supported safety correction in a fully onboard aerial ImageNav system.The safety layer corrects unsafe actions before they reach the flight controller.Source: §I · pp. 1–2
  • Omni-IGN adaptively fuses goal-conditioned features from four views to guide continuous navigation and reduce repeated exploration.Overlapping camera views also support stereo depth estimation for obstacle avoidance, with a safety layer correcting unsafe actions.Source: §I · p. 2
  • Omni-IGN achieves state-of-the-art performance among continuous-action baselines on Habitat benchmarks, and real-world flights demonstrate zero-shot sim-to-real transfer without fine-tuning.IsaacSim experiments validate the depth-estimation and safety-layer approach for collision avoidance.Source: §I · p. 2

A. End-to-end ImageNav Algorithms

Prior ImageNav approaches trade panoramic coverage and efficiency against continuous control and real-time onboard computation. Omni-IGN uses four goal-conditioned camera views, adaptive attention, recurrent state, and continuous actions for aerial navigation.

A. End-to-end ImageNav Algorithms

  • Panoramic ImageNav methods use graph-based memories or episodic attention to improve path efficiency, but their discrete actions hinder smooth continuous control and multiple views increase onboard computation.Single-view methods span reinforcement learning, goal-conditioned supervised learning, and world models.Source: A. End-to-end ImageNav Algorithms · pp. 2–3

B. Aerial ImageNav

  • Aerial ImageNav methods address exploration and continuous velocity control, but primarily rely on front-view observations.Omni-IGN instead combines broader visual coverage with continuous control for onboard deployment.Source: B. Aerial ImageNav · p. 3

C. Drone Navigation with Omnidirectional Perception

  • Omnidirectional drone systems address mapping, exploration, obstacle avoidance, or point-goal navigation; Omni-IGN targets ImageNav without oracle goal information and outputs continuous velocities.OMNI-DRL uses relative goal-direction information and discrete steering actions.Source: C. Drone Navigation with… · p. 3

A. Task Definition

  • The task conditions continuous body-frame forward and angular velocities on panoramic observations while searching for a goal RGB image in an unknown environment.Success means reaching within 1 m of the goal viewpoint.Source: A. Task Definition · p. 3

B. Navigation Model

  • The navigation model encodes each directional RGB view paired with the goal image, then adaptively fuses per-view features with self-attention for goal-relevant directional guidance.A GRU integrates the fused representation with the previous action before the actor-critic predicts continuous actions; training uses PPO.Source: Figure 2 · p. 3, B. Navigation Model · pp. 3–4

C. Reward Design and Evaluation Metrics

The reward encourages progress toward a goal viewpoint and alignment of the front view with the goal image, while evaluation measures task completion and path efficiency.

C. Reward Design and Evaluation Metrics

IV. DEPTH ESTIMATION FOR COLLISION AVOIDANCE

Habitat training does not make the navigation policy collision-aware, so the authors add depth-based action correction for safer deployment despite having only RGB cameras.

IV. DEPTH ESTIMATION FOR COLLISION AVOIDANCE

  • Habitat benchmarks reward goal reaching but do not penalize collisions, leaving the trained navigation policy not safety-aware.Source: §IV · p. 4
  • A safety layer intervenes on raw navigation actions only when it detects a potential collision.Source: §IV · p. 4
  • Because the drone has quad-fisheye RGB cameras without direct depth measurements, the authors develop real-time stereo depth estimation for action correction.Source: §IV · p. 4

A. Learning-Based Stereo Depth Estimation

The system recovers frontal metric depth from overlapping quad-fisheye views by rectifying stereo pairs, estimating disparity, and stitching the resulting depth maps.

A. Learning-Based Stereo Depth Estimation

B. Obstacle Avoidance via Action Safety Layer

The safety layer predicts collision risk from a compact depth representation and iteratively adjusts policy actions that exceed a risk threshold.

B. Obstacle Avoidance via Action Safety Layer

  • The layer derives a 16-bin depth vector from the central 90° horizontal field of view and a central band spanning 40% of the stitched depth map’s height.It retains each bin’s minimum valid range, clips ranges at 3 m, and normalizes them to [0, 1].Source: B. Obstacle Avoidance via Action… · p. 5
  • Actions pass through unchanged when predicted collision probability is at or below the threshold; otherwise, gradient descent iteratively adjusts the action to reduce risk.Source: B. Obstacle Avoidance via Action… · p. 5
  • The threshold controls the safety–efficiency tradeoff, with lower values producing more conservative action correction.Source: B. Obstacle Avoidance via Action… · p. 5

A. Simulation Settings

Omni-IGN is trained on Gibson and evaluated both there and on MP3D and HM3D, with continuous-action performance compared against other methods. The evaluation reports gains on Gibson and the highest SR under continuous control across the three benchmarks.

1) Datasets:

  • The model is trained on Gibson, using 72 training scenes and 14 testing scenes, then evaluated on MP3D and HM3D for cross-domain robustness.Scenes across the datasets vary in backgrounds and room layouts.Source: 1) Datasets · p. 5

2) Agent Configurations:

  • Simulation uses four RGB cameras at 90° intervals for 360° coverage, with 90° FoV and 128×128 images; continuous actions specify forward and angular velocities.An early stop implements stopping when both velocities fall below specified thresholds.Source: 2) Agent Configurations · p. 6

3) Training Details:

  • The lightweight ResNet9 model is trained with PPO for 50M steps on Gibson, using eight parallel environments and 4096-step rollout batches.The encoder starts from pretrained weights.Source: 3) Training Details · p. 6

1) Evaluation Results:

  • Omni-IGN improves SR by +1.5% and SPL by +10.9% over SIGN on Gibson, and by +3.3% SR and +10.1% SPL over 4-RGB FGPrompt.The paper reports comparable performance to the discrete-action SOTA while enabling continuous flight control without a map.Source: 1) Evaluation Results · p. 6, Table I · p. 6
  • Across MP3D and HM3D, Omni-IGN surpasses 4-RGB FGPrompt on HM3D by +2.4% SR and +3.8% SPL, and achieves the highest SR under continuous control across all three benchmarks.Results are averaged across easy, medium, and hard settings, each containing 1000 episodes.Source: 1) Evaluation Results · p. 6

2) Benefits of Omnidirectional Perception:

Omnidirectional perception is associated with more efficient navigation paths, while the self-attention ablation indicates that adaptively weighting views matters for SR. IsaacSim tests evaluate the depth-based safety layer and report both successful avoidance and a limited-field-of-view failure mode.

2) Benefits of Omnidirectional Perception:

  • Compared with the single-view baseline, Omni-IGN gains more in SPL than SR, and its trajectories are more direct with fewer exploration loops and backtracking.The paper suggests broader visual coverage reduces exploration and the likelihood of missing the goal image.Source: 2) Benefits of Omnidirectional… · p. 7

3) Ablation of Self-Attention:

  • On Gibson, adding self-attention improves SR by +3.3% and SPL by +1.1%; across MP3D and HM3D, it improves SR by +2.6% and +5.2%, respectively, while maintaining comparable SPL.Without self-attention, panoramic input still gains +9.8% SPL over SIGN but changes SR by -1.8%.Source: 3) Ablation of Self-Attention · p. 7

C. Reliability Analysis of Depth Estimation and Safety Layer

  • The safety layer iteratively corrects actions when collision probability exceeds 0.65, using depth-based deceleration and turning toward the side with greater depth.Corrections are limited to five iterations; if the threshold remains unmet, linear speed is set to zero.Source: C. Reliability Analysis of Depth… · p. 8
  • Across 90 IsaacSim flights, the drone reached the goal in 86 cases, with a 95.6% success rate and 0.91 average SPL; four side-collision failures were linked to limited depth-scan FoV.The average action correction rate was 23.3%; the stereo-derived range scan had 0.14 m mean depth error and 1.13 m maximum error.Source: C. Reliability Analysis of Depth… · p. 8

VI. REAL-LIFE EXPERIMENTS

Real-world experiments run perception, inference, and control onboard using a quad-fisheye camera rig. The attention visualizations show view weights shifting with goal cues, from broadly distributed attention during exploration toward the forward view on approach.

1) Hardware Configuration:

  • The drone executes perception, inference, and control onboard; four 220° fisheye cameras provide overlapping panoramic RGB views used by both the ImageNav policy and stereo depth estimation.A LiDAR supplies state estimates for the low-level controller, rather than depth measurements for the policy or safety pipeline.Source: 1) Hardware Configuration · p. 8

2) Acceleration of Onboard Computation:

  • The depth pipeline runs asynchronously at approximately 3 Hz while the ImageNav policy runs at approximately 10 Hz, with both accelerated using FP16 TensorRT.The asynchronous depth worker does not block policy execution.Source: 2) Acceleration of Onboard Computation · p. 8

B. Verification of Attention Mechanism

  • Attention shifts toward views containing goal cues: in one lab flight the forward-view weight rises to 55% and 62% during final approach, while office exploration begins broadly distributed before the front view reaches 62%.The lab examples also show attention moving from left to right as landmarks change views; the office example shows weights near-uniform when no distinctive cue is visible.Source: B. Verification of Attention Mechanism · pp. 8–9

C. Verification of Learning-Based Depth Estimation

The paper reports dense, spatially coherent stereo depth predictions in lab and office scenes, including textureless cylindrical obstacles. Its conclusion summarizes Omni-IGN as a fully onboard quad-fisheye system with strong continuous-action benchmark performance and zero-shot transfer.

C. Verification of Learning-Based Depth Estimation

  • The learned stereo pipeline produces dense, spatially coherent depth estimates that separate nearby obstacles from the background, including textureless cylinders.The paper reports more complete spatial coverage than raw conventional depth-camera measurements and says calibrated estimates are sufficiently reliable for obstacle avoidance.Source: C. Verification of Learning-Based… · p. 9, Figure 10 · p. 10

VII. CONCLUSIONS

  • Omni-IGN achieves state-of-the-art performance among continuous-action baselines on Habitat benchmarks and zero-shot sim-to-real transfer without fine-tuning.The conclusion also reports that panoramic observations improve navigation efficiency and robustness.Source: §VII · p. 10
Loading 2610.36968v1…