PaperlayerView exact revision on arXiv

Source-linked AI summary

RTNav: Towards Real-Time Zero-Shot Object Navigation

Easop Lee, Lingyu Zhang, Boyuan Chen

arXiv:2608.26496v1cs.ROcs.AIcs.CV

TL;DR

Foundation-model navigation suffers from inference latency that synchronous benchmarks ignore, even though wall-clock time matters for real-world operation. RTNav addresses this with modular asynchronous execution and on-demand reasoning, improving success rate by up to 11% and Success weighted by Completion Time by up to 5.1 points over prior methods. The study primarily evaluates static environments, leaving dynamic scenes and tightly coupled embodied tasks for future work.

  • Problem

    Synchronous benchmarks ignore inference latency, although model queries can take over one second per step and real-world task budgets include computation time.

    Source: §I · pp. 1–2

  • Method

    RTNav uses independently running perception, mapping, planning, and navigation modules with an on-demand VLM under bounded compute and latency constraints.

    Source: §I · p. 2

  • Results

    RTNav improves success rate by up to 11% and Success weighted by Completion Time by up to 5.1 points over prior methods across HM3D-v1, HM3D-v2, and HM3D-OVON.

    Source: §VI · p. 7

  • Takeaways & Limitations

    Wall-clock efficiency and asynchronous execution are important considerations for deploying foundation-model navigation agents in realistic robotic settings.

    Source: E. Real-Time Evaluation Results · p. 6, §VI · p. 7

    The study primarily considers static environments and leaves dynamic scenes and tightly coupled tasks such as locomotion and manipulation for future work.

    Source: §VI · p. 7

Abstract

from arXiv · show

Navigation in unknown environments to find unforeseen objects has become increasingly feasible with capable vision and language foundation models. However, these models also introduce non-negligible inference latency, which becomes an important concern when agents must operate continuously in the real world. Most state-of-the-art methods are still developed in synchronous simulators, where the environment waits for the agent to act and inference time is effectively free. As a result, agents are often designed around the sequential execution of perception, reasoning, and action, with little regard for time constraints. Under real-time execution, where wall-clock time counts towards the task budget, the inefficiencies of these architectures become clear. We show that recent zero-shot object navigation methods suffer consistent performance degradation under such realistic timing conditions. Motivated by this observation, we propose RTNav, a simple but effective architecture that treats inference latency, asynchronous environment stepping, and bounded compute as explicit design considerations. Evaluated on real-time variants of HM3D-v1, HM3D-v2, and HM3D-OVON, RTNav improves the success rate by up to 11% and the Success weighted by Completion Time by up to 5.1 points over prior work.

I. INTRODUCTION

Foundation-model navigation improves zero-shot generalization but introduces inference latency that synchronous benchmarks largely ignore. RTNav addresses this gap with asynchronous, wall-clock-aware execution and improves real-time performance across three benchmarks.

III. REAL-TIME OBJECTNAV EVALUATION

Real-time ObjectNav decouples simulator progression from agent computation while preserving standard tasks and scenes. This makes inference latency part of the task budget and exposes stale-command effects that synchronous evaluation hides.

  • Real-time ObjectNav scores task progress in wall-clock time while preserving standard benchmark tasks and datasets.Source: B. Real-Time ObjectNav · p. 3
  • A fixed 1/30-second simulation interval makes episode completion time equal wall-clock time.Source: B. Real-Time ObjectNav · p. 3
  • In asynchronous execution, the simulator advances under the previous command while inference runs, then applies the new command after its computation completes.Source: B. Real-Time ObjectNav · p. 3
  • Latency reduces progress because the simulator advances fδ_i steps under a stale command during inference.Source: B. Real-Time ObjectNav · p. 3
  • A ROS2 layer separates the environment and agent into processes that exchange observations and velocity commands without making the simulator wait for computation.Source: C. Implementation · p. 3

IV. RTNA V: REAL-TIME OBJECTNAV AGENT

RTNav is a modular real-time agent whose components run independently and communicate through shared state. Fast perception and control continue while an on-demand VLM supplies semantic reasoning when needed.

  • Different navigation modules operate at different natural frequencies, with low-level control updating frequently and subgoal planning updating less often.Source: §IV · p. 3
  • RTNav runs perception, mapping, planning, and navigation in independent threads that read the latest shared state without waiting for a complete cycle.Source: §IV · p. 3
  • An on-demand VLM server provides expensive vision-language reasoning only when needed, allowing fast perception and control loops to continue.Source: §IV · p. 3, Figure 2 · p. 4
  • Owlv2 runs at around 15Hz for high-frequency open-vocabulary candidate proposal, while Qwen3.5-9B performs semantic verification.Source: A. Modules · p. 3
  • The mapping module maintains a 2D metric map of obstacles, explored regions, frontiers, and confirmed target locations.Source: A. Modules · p. 4
  • The VLM is queried for frontier selection when replanning is necessary and when the object detector proposes candidates.Source: B. Execution Flow · p. 4, Figure 2 · p. 4

V. EXPERIMENTS

The experiments evaluate representative open-source zero-shot navigation methods across three benchmarks using success, path-efficiency, distance, and completion-time metrics. Baselines were rerun under unified synchronous and real-time protocols for reproducible comparison.

Benchmarks

  • Evaluation covers HM3D-v1, HM3D-v2, and HM3D-OVON, spanning known and novel object-goal categories.HM3D-v1 includes 2,000 episodes across 20 scenes, HM3D-v2 includes 1,000 episodes across 20 scenes, and OVON includes 3,000 episodes across 36 scenes.Source: A. Evaluation Benchmarks · p. 4

Metrics

  • The study reports Success Rate, Success weighted by Path Length, Distance to Goal, and Success weighted by Completion Time.SCT incorporates wall-clock completion time, including detours and computation pauses.Source: B. Evaluation Metrics · pp. 4–5

Baselines

  • The comparison includes open-source methods spanning map-based, open-vocabulary, belief-based, and sequential-planning architectures.Evaluated baselines include VLFM, GAMap, L3MVN, OpenFMNav, Trihelper, and BeliefMapNav.Source: C. Baselines · p. 5

Protocol

  • Baseline evaluation used reproducible code and conventional configurations, with synchronous action sequences verified against official implementations.The verification was performed on the HM3D-v1 val-mini split before real-mode evaluation.Source: C. Baselines · p. 5

D. Implementation Details

Under a unified real-time evaluation protocol, RTNav achieved the strongest overall performance across all three benchmarks. Its advantages were especially pronounced for success rate and completion-time efficiency.

Protocol

  • The real-time protocol evaluates methods on LocoBot or Stretch with benchmark-specific success thresholds and a single RTX A6000 GPU.Thresholds were 0.1m for HM3D-v1, 0.2m for HM3D-v2, and 0.25m for HM3D-OVON.Source: D. Implementation Details · p. 5

Results

  • RTNav surpassed the strongest synchronous baseline by 6.4%, 10.5%, and 9.1% in success rate on OVON, HM3D-v1, and HM3D-v2, respectively.All comparisons used the same 500s real-time evaluation budget.Source: E. Real-Time Evaluation Results · p. 5
  • RTNav achieved the highest success rate while maintaining competitive path efficiency and the lowest Distance to Goal across all three benchmarks.These results were reported under the unified evaluation protocol.Source: E. Real-Time Evaluation Results · p. 5

Time efficiency

  • RTNav scored 2.45, 5.17, and 4.17 higher Success weighted by Completion Time on OVON, HM3D-v1, and HM3D-v2, respectively.SCT captures both navigation success and wall-clock task completion time.Source: E. Real-Time Evaluation Results · pp. 5–6

F. Real-Time Execution Analysis

Real-time analysis shows that synchronous designs lose performance because sequential modules create idle time and limit visual processing. RTNav’s asynchronous execution reduces these delays and increases visual coverage.

Performance degradation

  • Baseline agents consistently degraded when transferred from synchronous evaluation to realistic real-time timing.Lightweight methods showed smaller drops, while slower methods were more affected as inference time consumed the wall-clock budget.Source: F. Real-Time Execution Analysis · p. 6

Idle time

  • RTNav showed 14%+ less idle time than all other baselines.Idle time includes latency, collision-related stalls, and stationary camera movements.Source: Figure 4 · p. 6

Visual processing

Architecture

  • Asynchronous execution lets perception, reasoning, mapping, and navigation proceed concurrently while the robot continues controlling its motion.This reduces waiting between module completion and control commands.Source: F. Real-Time Execution Analysis · pp. 6–7

Real-world example

  • RTNav’s real-world example combines semantic frontier selection, high-frequency detection, and VLM verification to navigate toward a sink.The figure presents representative snapshots from a successful trajectory.Source: Figure 7 · p. 7

G. Sensitivity Analysis

RTNav maintains strong real-time navigation across compute hardware and VLM choices, while slower edge compute reduces wall-clock efficiency and insufficient model capacity degrades performance. Real-world examples illustrate its asynchronous exploration, detection, and verification behavior.

Compute hardware

  • RTNav achieved nearly identical SR and SPL across RTX A6000, L40S, and Jetson Thor platforms.SCT decreased moderately on the slower edge-compute platform.Source: G. Sensitivity Analysis · p. 7

Vision-language models

  • Qwen3.5-9B achieved the best overall performance, slightly outperforming the larger Gemma4-12B.Reducing the model to Qwen3.5-4B substantially degraded all metrics.Source: G. Sensitivity Analysis · p. 7
  • RTNav did not require the largest available VLM, but sufficient semantic reasoning capacity remained important for reliable frontier selection.Source: G. Sensitivity Analysis · p. 7

Real-world behavior

  • Semantic frontier selection guides exploration, while high-frequency detection captures peripheral targets and VLM verification filters false positives in a real-world episode.Source: H. Qualitative Results · p. 7

Overall performance

  • Across HM3D-v1, HM3D-v2, and HM3D-OVON, RTNav improves SR by up to 11% and SCT by up to 5.1 points over prior methods.Source: §VI · p. 7

Limitations

  • Static environments and tightly coupled embodied tasks remain outside the study’s evaluated scope.The authors identify locomotion, manipulation, and dynamic scenes as future directions.Source: §VI · p. 7

APPENDIX

The appendix documents the ROS-based evaluation setup, action conversion, baseline reproduction, and implementation adjustments used to compare synchronous and asynchronous execution.

Execution modes

  • In real-time mode, env.step() runs continuously at 30Hz using the latest available action and updated timestamp.Synchronous mode calls env.step() whenever the agent produces a new action.Source: A. Real-Time Environment Implementation · p. 10
  • The synchronous benchmark waits indefinitely for agent output, whereas the real-time benchmark runs environment and agent nodes independently at their own frequencies.Source: Figure 8 · p. 10

Action mapping

Baseline reproduction

Implementation fixes

Baseline reproduction

C. RTNav Implementation Details

RTNav combines open-vocabulary perception, persistent object memory, VLM verification, semantic frontier selection, and asynchronous execution to support real-time object navigation.

Perception and mapping

  • RTNav generates up to two target synonyms and combines them with filtered LVIS categories to construct detector candidates.The detector vocabulary contains 405 common indoor categories after excluding six small fixture categories.Source: C. RTNav Implementation Details · pp. 11–12
  • The OWLv2 detector uses 960 × 960 resolution, a 0.25 confidence threshold, seven text templates, and NMS IoU threshold 0.6.Source: C. RTNav Implementation Details · p. 11

Object memory and verification

  • Persistent object-graph nodes become confirmed after observations from three distinct viewpoints.Distinct viewpoints require at least 0.1 m translation or approximately 3° yaw difference.Source: C. RTNav Implementation Details · p. 11
  • RTNav submits up to eight object candidates in parallel to a VLM and accepts candidates only when p_yes > 0.90.Competing detector or graph labels are included when available; three confident rejections blacklist a node.Source: C. RTNav Implementation Details · p. 11
  • MobileSAM refines accepted masks to obtain object centroids and nearest-surface navigation goals.Far or truncated detections remain candidate targets until a better view enables verification within 3 meters.Source: C. RTNav Implementation Details · p. 11

Implementation efficiency

  • Perception, mapping, frontier detection, decision-making, and navigation execute asynchronously, with cached embeddings and batched inference improving efficiency.Source: C. RTNav Implementation Details · p. 11

VLM prompts

D. Task Details

The evaluation uses standard HM3D and OVON validation tasks with fixed motion and stopping rules, while category visualizations report RTNav’s navigation metrics only for sufficiently represented categories.

Task and simulator configurations

Category-level evaluation

  • Fig. 10 reports category-level SR, SPL, and SCT for RTNav, omitting OVON categories with fewer than 25 episodes.The number of evaluated episodes is shown for each displayed category.Source: Figure 10 · p. 13

E. Additional Results and Analysis

RTNav’s performance varies substantially across object categories and benchmarks, with failures dominated by timeouts and exploration challenges. The real-world system combines RGB-D sensing with SLAM-based mapping and waypoint control.

Performance by Goal Category

  • Chairs, beds, couches, refrigerators, and pianos are generally easier targets, while televisions, potted plants, hangers, blinds, and flowerpots are harder to locate.The reported differences are associated with target size, visual distinctiveness, and predictable room contexts.Source: E. Additional Results and Analysis · p. 13

Failure Analysis

  • Most failures on HM3D-v1 and HM3D-v2 are timeouts rather than incorrect STOP decisions.HM3D-v1 additionally has more exploration failures because targets may lie on another floor or be missed during exploration.Source: E. Additional Results and Analysis · p. 13
  • HM3D-OVON failures are more dispersed across categories, consistent with its more challenging open-vocabulary targets.A narrower, taller camera field of view can also make some objects easier to miss during exploration.Source: E. Additional Results and Analysis · p. 13

Real-World Experiments

  • Real-world experiments used RGB-D from a head-mounted RealSense D435i camera and global pose from onboard HectorSLAM with a 2D LiDAR.The deployed system fused SLAM occupancy with RGB-D-reconstructed obstacles for mapping.Source: F. Real-World Experiment Details · p. 13
  • For real-world navigation, RTNav replaced Habitat’s PointNav policy with a modified FMM planner and used a continuous PD controller to track generated waypoints.The replacement addressed the mismatch between Habitat-trained PointNav and real-world deployment.Source: F. Real-World Experiment Details · p. 14

Related papers

Loading 2608.26496v1…