PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniNav: Robust Long-Horizon Target Navigation in Dynamic Environments

Yujie Tang, Meiling Wang, Jinhao Jiang, Sibo Zuo, Yinan Deng, Xinyu Zhang, Yufeng Yue

arXiv:2609.08159v1cs.RO

TL;DR

Long-horizon navigation must maintain scene validity, revise target beliefs, and choose interaction-feasible endpoints as observations and physical conditions change. OmniNav couples these tasks through continual inference and feedback, with evaluations reporting greater robustness and efficiency and real-world pick-and-place success increasing from 53.3% to 71.7%.

  • Problem

    Scene evidence can become stale, unsuccessful searches should revise target-location beliefs, and nearby navigation goals may be infeasible for physical interaction.

    Source: §I · p. 2

  • Method

    OmniNav performs continual inference over coupled scene validity, target belief, and interaction feasibility, updating object memory and using search and execution feedback in subsequent decisions.

    Source: §I · p. 2

  • Results

    Evaluations across navigation benchmarks, dynamic long-horizon tasks, and real-world mobile manipulation demonstrate greater robustness and efficiency; real-world pick-and-place success rises from 53.3% to 71.7%.

    Source: §I · pp. 2–3

  • Takeaways & Limitations

    Long-horizon execution is treated as a coupled process in which scene validity, target belief, and interaction feasibility are continually revised.

    Source: §IX · p. 19

    Memory and fine-grained confirmation depend on visual-language perception, depth sensing, and camera-pose estimation; scene changes are detectable only after affected regions become observable again.

    Source: §IX · p. 19

Abstract

from arXiv · show

Long-horizon target navigation requires a robot to sustain task execution across evolving observations, decisions, and physical interactions. This requires three coupled capabilities: maintaining valid scene memory, revising target beliefs under partial observability, and selecting interaction-feasible navigation endpoints. However, the state underlying each capability is only conditionally valid: scene representations become stale when objects move or disappear, unsuccessful searches alter beliefs over target locations, and geometrically convenient endpoints may still be infeasible for manipulation. To address these challenges, we present OmniNav, which formulates long-horizon navigation as continual inference over a factorized task state posterior coupling scene validity, target belief, and interaction feasibility. For representation, OmniNav incrementally constructs an updatable 3D object scene memory, preventing stale scene evidence from propagating to subsequent decisions. For exploration, it introduces an evidence-aware Bayesian belief-revision mechanism that derives dependency-aware region priors from semantic context, incorporates unsuccessful searches as negative evidence, and updates them for posterior-guided frontier selection. For interaction, OmniNav incorporates manipulation reachability and collision constraints into navigation-endpoint selection and propagates execution feedback through hierarchical closed-loop recovery. Extensive experiments demonstrate that OmniNav achieves the highest success rates among the compared methods on semantic ObjectNav and fine-grained instance navigation benchmarks, remains robust to target relocation, and improves real-world pick-and-place success from 53.3% to 71.7% over an adapted open-loop baseline. The project page of OmniNav is available at https://omni-nav.github.io/.

I. INTRODUCTION

Long-horizon navigation couples scene memory, target search, and physical interaction, but the states supporting each can become invalid as scenes change and searches fail. OmniNav treats these stages as continual inference, revising scene validity, target belief, and interaction feasibility.

II. RELATED WORK

  • Existing frameworks cover complementary capabilities, but provide limited mechanisms for evidence from one stage to revise another during dynamic long-horizon execution.Source: §I · p. 2

I. INTRODUCTION

  • OmniNav formulates long-horizon execution as factorized probabilistic inference coupling scene memory, target search, and interaction feasibility, without requiring a pre-built map or task-specific training.Source: §I · p. 2
  • OmniNav updates each mapped instance’s Bayesian persistence from supporting, empty-space, and occlusion evidence, removing it only when posterior validity falls below a threshold.Source: §I · p. 2
  • OmniNav groups dependent context cues before aggregating priors, updates region probabilities with unsuccessful searches, and verifies candidates against scene context and egocentric VLM observations.Source: §I · p. 2
  • Interaction-feasible base poses are optimized using a differentiable dual-ellipsoid workspace approximation and collision penalties; execution feedback can trigger retries, re-optimization, or target re-search.Source: §I · p. 2
  • Real-world pick-and-place success rises from 53.3% to 71.7%; evaluations across navigation benchmarks, dynamic long-horizon tasks, and mobile manipulation demonstrate greater robustness and efficiency.Source: §I · pp. 2–3

A. Scene Representation and Memory

  • Scene graphs support target grounding and relational reasoning, while many such approaches assume static scenes; dynamic mapping methods have limited instance-level validity inference from ambiguous observations.Source: A. Scene Representation and Memory · p. 3

B. Target-Oriented Navigation Strategies

OmniNav revises target beliefs from unsuccessful searches, links navigation to interaction feasibility, and maintains scene memory as objects change. Its framework feeds updated scene and execution evidence back into subsequent decisions.

B. Target-Oriented Navigation Strategies

  • Prior strategies use frontier coverage, semantic relevance, or unified goal frameworks; OmniNav revises dependency-aware semantic priors with unsuccessful-search evidence.Target-absent observations become negative evidence for region target probabilities.Source: B. Target-Oriented Navigation Strategies · pp. 3–4

C. Interaction-Aware Mobile Manipulation

III. OVERVIEW

  • The framework continually revises scene understanding, target judgment, and physical feasibility from posed RGB-D observations, without requiring a pre-built test-environment map.Its factorized task state comprises scene memory, target search, and interaction feasibility; execution feedback triggers re-evaluation of affected factors.Source: §III · p. 4

A. Persistent Instance Memory in Dynamic Environments

  • OmniNav models instance persistence with Bayesian inference from depth consistency, checking movable everyday objects whenever observable and structural objects at a low fixed frequency.Invalid instances are removed from the global map so stale objects do not affect downstream semantic querying.Source: A. Persistent Instance Memory in… · pp. 5–7

IV. DYNAMIC OBJECT-CENTRIC SCENE MEMORY

  • Each remembered instance’s surface points are classified as support, empty, or occluded in depth observations, and increasingly contradictory observations reduce its persistence belief.The memory uses a two-state persistence model with an absorbing invalid state; an invalidated instance is not reactivated at its old location.Source: Figure 3 · p. 6, A. Persistent Instance Memory in… · p. 6

B. Hierarchical Scene Graph Abstraction

OmniNav organizes active instances into a lightweight hierarchical scene graph, linking object properties and spatial relations to grouped scene regions.

B. Hierarchical Scene Graph Abstraction

  • The hierarchical graph contains active-instance nodes, spatially grouped region nodes, and relational edges, conditioned on the object map and inferred persistence.Source: B. Hierarchical Scene Graph Abstraction · p. 7
  • Instance nodes aggregate semantics, geometry, persistence state, and observation statistics; their attributes include color, material, shape, and size.Source: B. Hierarchical Scene Graph Abstraction · p. 7
  • Nearby instances are grouped into regions using centroid distance and obstacle-free connections, while removed instances leave their region and empty regions are deleted.Source: B. Hierarchical Scene Graph Abstraction · p. 7
  • Relationship edges encode pairwise spatial predicates, and the graph is synchronized after memory updates by inserting new instances, removing invalid ones, and reevaluating affected assignments and relations.Natural-language relations in queries are normalized to supported predicates.Source: B. Hierarchical Scene Graph Abstraction · p. 7

C. 2D Exploration Progress Map

OmniNav tracks spatial observation progress in a 2D grid independently of its object-centric memory, then aggregates progress over scene-graph regions for exploration decisions.

C. 2D Exploration Progress Map

  • Each grid cell is Unknown, Observed, or Explored, distinguishing cells not covered by valid depth rays, traversed by a ray, or containing an endpoint seen in the central camera field of view.Source: C. 2D Exploration Progress Map · p. 7
  • Valid depth pixels are projected and ray-cast onto the grid, promoting traversed cells from Unknown to Observed and central-50° endpoint cells to Explored.Cell states update monotonically from Unknown to Observed to Explored.Source: C. 2D Exploration Progress Map · p. 7
  • The map records observation progress separately from object memory and aggregates cell states over scene-graph regions for exploration–exploitation reasoning.Source: C. 2D Exploration Progress Map · p. 7

V. TASK-GROUNDED EXPLORATION–EXPLOITATION NAVIGATION

OmniNav represents target search as inference over region-level target existence and a target hypothesis. It grounds queries into object or fine-grained specifications and reuses updated scene memory across sequential goals.

V. TASK-GROUNDED EXPLORATION–EXPLOITATION NAVIGATION

  • The target-search state combines binary region-level target-existence variables with a target hypothesis, instantiating the target-search factor conditioned on revised scene memory and accumulated evidence.Source: §V · p. 7

A. Query Grounding

  • Object-goal queries specify a target category, while fine-grained queries specify visual attributes and/or spatial relations to identify a particular instance.Source: A. Query Grounding · pp. 7–8
  • For fine-grained instructions, an LLM parses the query once at initialization into a graph of target and optional reference-object nodes, attributes, and normalized spatial-relation edges.Supported attributes include color, material, shape, and size; the resulting constraints support candidate matching without further online LLM calls.Source: A. Query Grounding · p. 8
  • For sequential targets, OmniNav grounds each query when activated, reuses object and region memories, and excludes invalidated instances from later candidate consideration.Persistent scene memory tracks observable changes between successive targets, avoiding a full scene-representation rebuild.Source: A. Query Grounding · p. 8

B. Region Prior Estimation

OmniNav estimates region-level target priors from object co-occurrence while accounting for dependencies among contextual objects. It revises those priors with negative evidence from explored regions where the target was not found.

B. Region Prior Estimation

  • Offline co-occurrence statistics and observed object context provide the semantic evidence for estimating target-existence priors over regions.The co-occurrence knowledge base is computed from image-level Visual Genome statistics.Source: B. Region Prior Estimation · p. 8

C. Exploration-Progress-Aware Posterior Update

B. Region Prior Estimation

  • Dependency-Aware Region Prior Aggregation groups correlated objects and selects representative evidence to avoid overestimating region priors from redundant cues.The resulting dependency groups are treated as approximately independent when estimating the memory-conditioned semantic prior.Source: B. Region Prior Estimation · p. 8

C. Exploration-Progress-Aware Posterior Update

  • As more of a region is explored without finding the target, its posterior target probability decreases monotonically, redirecting search toward less-explored, semantically plausible regions.Under the uniform search model, the posterior equals the prior at zero exploration progress and falls to zero at full progress.Source: C. Exploration-Progress-Aware… · p. 9

D. Navigation Decision

OmniNav selects reachable frontier waypoints by balancing regional target posteriors against travel efficiency. For fine-grained queries, it confirms candidates using scene memory and current visual evidence, then chooses whether to continue exploring or exploit a hypothesis.

D. Navigation Decision

  • Frontier utility balances the posterior of the associated region against travel distance, and the highest-scoring frontier becomes the next waypoint.Frontiers are reachable, traversable cells on the boundary between Explored and Observed areas.Source: D. Navigation Decision · p. 9

E. Fine-Grained Confirmation and Exploration–Exploitation Transition

  • Fine-grained candidate confirmation fuses query-constraint matches from persistent scene memory and current egocentric visual evidence.Memory evidence gains weight with more candidate observations; current-view evidence contributes more for candidates with limited observation history.Source: E. Fine-Grained Confirmation and… · pp. 9–10
  • Candidates exceeding the confirmation threshold are accepted; otherwise, OmniNav continues exploration or exploits the highest-scoring hypothesis according to its transition criterion.The transition criterion incorporates search progress and penalizes premature commitment.Source: E. Fine-Grained Confirmation and… · p. 10

VI. REACHABILITY-AWARE CLOSED-LOOP MOBILE MANIPULATION

For pick-and-place, OmniNav refines navigation endpoints for manipulation feasibility and uses execution feedback for closed-loop recovery.

VI. REACHABILITY-AWARE CLOSED-LOOP MOBILE MANIPULATION

  • Reachability-aware base alignment refines the final navigation pose for manipulation feasibility and collision safety, while closed-loop recovery can correct grasps, re-optimize the base pose, or renew exploration.The task setting is navigating to a target, grasping it, navigating to a receptacle, and placing the object.Source: §VI · p. 10

A. Reachability-aware Base Alignment

OmniNav models manipulation reachability offline and uses that model in online base-pose refinement. The resulting navigation goal accounts for target alignment, reachability, and obstacle clearance rather than target proximity alone.

A. Reachability-aware Base Alignment

  • A dual-ellipsoid shell fitted offline to sampled inverse-kinematics reachability provides a smooth surrogate for online base-pose optimization.The outer ellipsoid bounds the reachable workspace, while the inner ellipsoid excludes near-base or low-reachability regions.Source: A. Reachability-aware Base Alignment · pp. 10–11, Figure 6 · p. 11
  • The reachability score is the fraction of sampled orientations with a valid inverse-kinematics solution, with higher scores indicating reachability across more orientations.A valid hypothesis satisfies joint-limit and self-collision constraints.Source: A. Reachability-aware Base Alignment · p. 11
  • Online base-pose optimization combines target-facing alignment, manipulation reachability, workspace clearance, and collision safety.The objective weights alignment, reachability, and local obstacle clearance; obstacle intrusion into the shell is penalized.Source: A. Reachability-aware Base Alignment · p. 11
  • Two-stage coarse-to-fine particle swarm optimization selects the final navigation goal as an interaction-feasible base pose rather than merely a pose near the target.The optimized objective is evaluated online.Source: A. Reachability-aware Base Alignment · pp. 11–12

B. Robust Hierarchical Closed-Loop Recovery

OmniNav uses hierarchical, closed-loop recovery to address execution failures at the lowest level that restores progress. Feedback can revise interaction feasibility and, for task-level failures, update scene memory and subsequent decisions.

B. Robust Hierarchical Closed-Loop Recovery

  • OmniNav resolves execution anomalies at the lowest level that restores execution, rather than restarting the task after every failure.Feedback re-evaluates interaction feasibility; task-level failures additionally revise scene memory, which propagates to target search and feasibility.Source: B. Robust Hierarchical Closed-Loop… · p. 12
  • Grasp slippage or failed closure triggers local re-perception and grasp or placement-pose updates, followed by a retry without global restart.This manipulation-level recovery uses gripper-state and wrist-camera feedback.Source: B. Robust Hierarchical Closed-Loop… · p. 12
  • When the target remains identified but the base-arm configuration becomes unsuitable, OmniNav re-optimizes the base pose and resumes manipulation.This reachability-level recovery can be triggered by target displacement or inverse-kinematics failure.Source: B. Robust Hierarchical Closed-Loop… · p. 12
  • When the task state is invalidated, OmniNav updates scene memory, candidate hypotheses, and object states before re-entering search, confirmation, or replanning.Execution ends if recovery cannot identify a feasible candidate, reachable pose, or valid plan.Source: B. Robust Hierarchical Closed-Loop… · p. 12

A. Experimental Setup

Across semantic ObjectNav and fine-grained instance navigation, OmniNav is evaluated in Habitat benchmarks with comparisons to training-based and training-free methods. Results report the strongest overall ObjectNav performance on MP3D and HM3D and the highest instance-navigation success rate, alongside ablations of exploration and confirmation components.

B. Semantic Object Navigation

  • OmniNav achieves the best overall ObjectNav performance among compared methods on MP3D (0.473 SR, 0.180 SPL) and HM3D (0.546 SR, 0.285 SPL).On MP3D, the best baseline is 0.420 SR and 0.164 SPL; on HM3D, the best baseline is 0.545 SR and 0.260 SPL.Source: Table III · p. 12, B. Semantic Object Navigation · p. 12
  • The ObjectNav evaluation uses Habitat HM3D (2,000 episodes, 20 scenes, 6 categories) and MP3D (2,195 episodes, 11 scenes, 21 categories).The compared methods include training-based and training-free approaches; success rate and success weighted by path length measure success and efficiency.Source: B. Semantic Object Navigation · p. 12
  • ObjectNav progress modeling down-weights searched-but-unsuccessful regions, reducing redundant revisits and supporting the exploration–exploitation trade-off.Qualitative examples report sofa SPL increasing from 0.28 to 0.89 and cabinet SPL from 0.31 to 0.54.Source: B. Semantic Object Navigation · pp. 12–13
  • Removing either exploration-progress modeling or dependency-aware region-prior aggregation consistently reduces ObjectNav SR and SPL.Without progress modeling, unsuccessful searches cannot serve as negative evidence; without dependency grouping, correlated context objects can inflate target likelihood.Source: Table IV · p. 13, B. Semantic Object Navigation · p. 13

C. Fine-grained Instance Navigation

  • Fine-grained navigation tests instance grounding across 1,000 HM3D episodes using long-form descriptions combining object attributes and relational context.The benchmark covers 36 scenes and 6 target categories.Source: C. Fine-grained Instance Navigation · p. 13
  • OmniNav reaches 0.271 SR in fine-grained instance navigation, exceeding GLMap by 4.6 and UniGoal by 6.9 percentage points.Its SPL is lower than the strongest baselines; the paper notes that confirmation and exploration can increase path length.Source: Table V · p. 13, C. Fine-grained Instance Navigation · p. 13
  • For fine-grained queries, OmniNav checks candidate attributes and relations using both accumulated scene memory and current egocentric evidence.Qualitative examples show this distinguishing the described chair from a same-category distractor that UniGoal selects.Source: C. Fine-grained Instance Navigation · p. 14, Figure 8 · p. 14
  • Fine-grained SR falls to 23.1% without egocentric VLM evidence, to 25.6% with Llama3.2-Vision, and to 26.5% without the exploration–exploitation transition.Removing egocentric evidence causes the largest reported degradation; the framework remains usable with the alternative VLM.Source: Table VI · p. 14, C. Fine-grained Instance Navigation · pp. 14–15

D. Dynamic Long-Horizon Object Navigation

OmniNav is tested on sequential navigation tasks with relocated targets in simulation and real-world settings, as well as real-world long-horizon pick-and-place tasks. Across these settings, results highlight scene-memory revision after relocation and stronger manipulation success as navigation and execution disturbances increase.

D. Dynamic Long-Horizon Object Navigation

  • The simulation tasks use four scenes each from HM3D and Gibson, with four sequential subtasks and the fourth target moved after the first subtask.Methods explore online without a pre-built map; outcomes average subtask SR and SPL across three starting positions.Source: D. Dynamic Long-Horizon Object… · p. 15
  • After final-target relocation, OmniNav achieves higher fourth-subtask SR and SPL than OmniNav*, which retains the stale location; UniGoal's average SR also declines in later subtasks.The dynamic-navigation comparison reports subtask-level SR and SPL in Gibson and HM3D.Source: D. Dynamic Long-Horizon Object… · p. 15

A. Dynamic Long-Horizon Object Navigation

  • In three real-world dynamic scenarios, OmniNav succeeds in every trial for the first two subtasks and in 8 of 9 trials for the relocated target.Each task sequences three object targets and is repeated three times with different initial robot positions.Source: Table VII · p. 17, A. Dynamic Long-Horizon Object… · p. 17
  • Real-world examples show OmniNav removing the old target instance from memory after relocation, then resuming search with the updated scene representation.The real-world navigation setup includes a laboratory, home office, and spacious corridor.Source: A. Dynamic Long-Horizon Object… · pp. 16–17, Figure 12 · p. 17

B. Long-Horizon Mobile Manipulation Tasks

  • At mobile-manipulation Level 2, OmniNav improves success by 20 percentage points over OK-Robot*; at Level 3, the gap is 30 points.Level 2 requires exploration and navigation to an initially unseen target; Level 3 adds unannounced perturbations. The methods perform comparably at Level 1.Source: B. Long-Horizon Mobile Manipulation… · pp. 17–18, Table VIII · p. 18
  • Removing either reachability-aware base alignment or hierarchical recovery reduces overall mobile-manipulation success.The comparison is an ablation of the real-world long-horizon mobile-manipulation system.Source: Table IX · p. 18, B. Long-Horizon Mobile Manipulation… · p. 18
  • OmniNav records 92.8% alignment success, with higher grasp (80.0% vs. 69.2%) and place (100.0% vs. 88.9%) success than OK-Robot*.The step-wise breakdown focuses on execution after target localization because both methods share the perception and navigation stack.Source: Table X · p. 18, B. Long-Horizon Mobile Manipulation… · p. 18
  • Real-world pick-and-place examples combine target search, reachability-aware base-pose refinement, and closed-loop recovery, including after target relocation.The tasks require locating and grasping an object, navigating to a receptacle, and placing it.Source: B. Long-Horizon Mobile Manipulation… · pp. 17–18

C. Runtime Analysis

OmniNav's online runtime is distributed across per-frame scene-memory updates and intermittent or event-triggered decision modules. The conclusion summarizes its navigation and interaction capabilities while identifying perception dependence, delayed change detection, and limited recovery coverage as boundaries.

C. Runtime Analysis

  • Scene-memory updates take 0.30 s per frame, frontier selection 0.52 s per decision, target confirmation 4.58 s per candidate, and base-pose optimization 2.85 s per call.Only scene-memory updates run per frame; other modules run intermittently or upon candidate observation or physical interaction.Source: C. Runtime Analysis · p. 18
  • Offline co-occurrence-prior construction takes 15.71 s, while reachability-shell construction takes 1125.40 s and scales approximately with sampled workspace points.The reachability shell is built once per manipulator model and can be reused across subsequent tasks.Source: C. Runtime Analysis · pp. 18–19

IX. CONCLUSION

  • OmniNav couples scene validity, target belief, and interaction feasibility through continual inference, combining scene memory, evidence-aware search, reachability-aware endpoints, and hierarchical recovery.The conclusion reports experiments across navigation benchmarks, dynamic long-horizon tasks, and real-world mobile manipulation.Source: §IX · p. 19
  • Scene memory and fine-grained confirmation depend on visual-language perception, depth sensing, and camera-pose accuracy.Scene changes can be detected only after the affected region becomes observable again, potentially delaying memory revision in large or sparsely revisited environments.Source: §IX · p. 19
  • The recovery mechanism covers only a limited set of manipulation failures and target-relocation events.The paper identifies active change detection and broader recovery strategies as future work.Source: §IX · p. 19
Loading 2609.08159v1…