Source-linked AI summary
RTNav: Towards Real-Time Zero-Shot Object Navigation
Easop Lee, Lingyu Zhang, Boyuan Chen
TL;DR
Foundation-model navigation suffers from inference latency that synchronous benchmarks ignore, even though wall-clock time matters for real-world operation. RTNav addresses this with modular asynchronous execution and on-demand reasoning, improving success rate by up to 11% and Success weighted by Completion Time by up to 5.1 points over prior methods. The study primarily evaluates static environments, leaving dynamic scenes and tightly coupled embodied tasks for future work.
Problem
Synchronous benchmarks ignore inference latency, although model queries can take over one second per step and real-world task budgets include computation time.
Method
RTNav uses independently running perception, mapping, planning, and navigation modules with an on-demand VLM under bounded compute and latency constraints.
Results
RTNav improves success rate by up to 11% and Success weighted by Completion Time by up to 5.1 points over prior methods across HM3D-v1, HM3D-v2, and HM3D-OVON.
Takeaways & Limitations
Wall-clock efficiency and asynchronous execution are important considerations for deploying foundation-model navigation agents in realistic robotic settings.
Takeaways & Limitations
The study primarily considers static environments and leaves dynamic scenes and tightly coupled tasks such as locomotion and manipulation for future work.
Abstract
from arXiv · showhide
Navigation in unknown environments to find unforeseen objects has become increasingly feasible with capable vision and language foundation models. However, these models also introduce non-negligible inference latency, which becomes an important concern when agents must operate continuously in the real world. Most state-of-the-art methods are still developed in synchronous simulators, where the environment waits for the agent to act and inference time is effectively free. As a result, agents are often designed around the sequential execution of perception, reasoning, and action, with little regard for time constraints. Under real-time execution, where wall-clock time counts towards the task budget, the inefficiencies of these architectures become clear. We show that recent zero-shot object navigation methods suffer consistent performance degradation under such realistic timing conditions. Motivated by this observation, we propose RTNav, a simple but effective architecture that treats inference latency, asynchronous environment stepping, and bounded compute as explicit design considerations. Evaluated on real-time variants of HM3D-v1, HM3D-v2, and HM3D-OVON, RTNav improves the success rate by up to 11% and the Success weighted by Completion Time by up to 5.1 points over prior work.
I. INTRODUCTION
Foundation-model navigation improves zero-shot generalization but introduces inference latency that synchronous benchmarks largely ignore. RTNav addresses this gap with asynchronous, wall-clock-aware execution and improves real-time performance across three benchmarks.
- A single foundation-model query may take hundreds of milliseconds to several seconds, directly affecting responsiveness and task completion time on real robots.
- Synchronous benchmarks hide inference costs by waiting for actions and measuring success by environment steps or path length rather than wall-clock time.
- Sequential architectures tie perception, reasoning, planning, and action to one cycle, so the slowest module determines system reactivity.
- Real-time evaluation exposes performance degradation from inference latency, with slower methods suffering the largest drops.
- RTNav uses independently running perception, mapping, planning, and navigation modules to improve wall-clock efficiency while preserving navigation performance.Its design makes asynchronous execution and bounded latency explicit system-level considerations.
III. REAL-TIME OBJECTNAV EVALUATION
Real-time ObjectNav decouples simulator progression from agent computation while preserving standard tasks and scenes. This makes inference latency part of the task budget and exposes stale-command effects that synchronous evaluation hides.
- Real-time ObjectNav scores task progress in wall-clock time while preserving standard benchmark tasks and datasets.
- A fixed 1/30-second simulation interval makes episode completion time equal wall-clock time.
- In asynchronous execution, the simulator advances under the previous command while inference runs, then applies the new command after its computation completes.
- Latency reduces progress because the simulator advances fδ_i steps under a stale command during inference.
- A ROS2 layer separates the environment and agent into processes that exchange observations and velocity commands without making the simulator wait for computation.
IV. RTNA V: REAL-TIME OBJECTNAV AGENT
RTNav is a modular real-time agent whose components run independently and communicate through shared state. Fast perception and control continue while an on-demand VLM supplies semantic reasoning when needed.
- Different navigation modules operate at different natural frequencies, with low-level control updating frequently and subgoal planning updating less often.
- RTNav runs perception, mapping, planning, and navigation in independent threads that read the latest shared state without waiting for a complete cycle.
- An on-demand VLM server provides expensive vision-language reasoning only when needed, allowing fast perception and control loops to continue.
- Owlv2 runs at around 15Hz for high-frequency open-vocabulary candidate proposal, while Qwen3.5-9B performs semantic verification.
- The mapping module maintains a 2D metric map of obstacles, explored regions, frontiers, and confirmed target locations.
- The VLM is queried for frontier selection when replanning is necessary and when the object detector proposes candidates.
V. EXPERIMENTS
The experiments evaluate representative open-source zero-shot navigation methods across three benchmarks using success, path-efficiency, distance, and completion-time metrics. Baselines were rerun under unified synchronous and real-time protocols for reproducible comparison.
- Benchmarks: Evaluation covers HM3D-v1, HM3D-v2, and HM3D-OVON, spanning known and novel object-goal categories.HM3D-v1 includes 2,000 episodes across 20 scenes, HM3D-v2 includes 1,000 episodes across 20 scenes, and OVON includes 3,000 episodes across 36 scenes.
- Metrics: The study reports Success Rate, Success weighted by Path Length, Distance to Goal, and Success weighted by Completion Time.SCT incorporates wall-clock completion time, including detours and computation pauses.
- Baselines: The comparison includes open-source methods spanning map-based, open-vocabulary, belief-based, and sequential-planning architectures.Evaluated baselines include VLFM, GAMap, L3MVN, OpenFMNav, Trihelper, and BeliefMapNav.
- Protocol: Baseline evaluation used reproducible code and conventional configurations, with synchronous action sequences verified against official implementations.The verification was performed on the HM3D-v1 val-mini split before real-mode evaluation.
D. Implementation Details
Under a unified real-time evaluation protocol, RTNav achieved the strongest overall performance across all three benchmarks. Its advantages were especially pronounced for success rate and completion-time efficiency.
- Protocol: The real-time protocol evaluates methods on LocoBot or Stretch with benchmark-specific success thresholds and a single RTX A6000 GPU.Thresholds were 0.1m for HM3D-v1, 0.2m for HM3D-v2, and 0.25m for HM3D-OVON.
- Results: RTNav surpassed the strongest synchronous baseline by 6.4%, 10.5%, and 9.1% in success rate on OVON, HM3D-v1, and HM3D-v2, respectively.All comparisons used the same 500s real-time evaluation budget.
- Results: RTNav achieved the highest success rate while maintaining competitive path efficiency and the lowest Distance to Goal across all three benchmarks.These results were reported under the unified evaluation protocol.
- Time efficiency: RTNav scored 2.45, 5.17, and 4.17 higher Success weighted by Completion Time on OVON, HM3D-v1, and HM3D-v2, respectively.SCT captures both navigation success and wall-clock task completion time.
F. Real-Time Execution Analysis
Real-time analysis shows that synchronous designs lose performance because sequential modules create idle time and limit visual processing. RTNav’s asynchronous execution reduces these delays and increases visual coverage.
- Performance degradation: Baseline agents consistently degraded when transferred from synchronous evaluation to realistic real-time timing.Lightweight methods showed smaller drops, while slower methods were more affected as inference time consumed the wall-clock budget.
- Idle time: RTNav showed 14%+ less idle time than all other baselines.Idle time includes latency, collision-related stalls, and stationary camera movements.
- Visual processing: RTNav reached over 20× the unique detection frames per second of synchronous baselines.Its perception thread continued processing without being blocked by other modules.
- Architecture: Asynchronous execution lets perception, reasoning, mapping, and navigation proceed concurrently while the robot continues controlling its motion.This reduces waiting between module completion and control commands.
- Real-world example: RTNav’s real-world example combines semantic frontier selection, high-frequency detection, and VLM verification to navigate toward a sink.The figure presents representative snapshots from a successful trajectory.
G. Sensitivity Analysis
RTNav maintains strong real-time navigation across compute hardware and VLM choices, while slower edge compute reduces wall-clock efficiency and insufficient model capacity degrades performance. Real-world examples illustrate its asynchronous exploration, detection, and verification behavior.
- Compute hardware: RTNav achieved nearly identical SR and SPL across RTX A6000, L40S, and Jetson Thor platforms.SCT decreased moderately on the slower edge-compute platform.
- Vision-language models: Qwen3.5-9B achieved the best overall performance, slightly outperforming the larger Gemma4-12B.Reducing the model to Qwen3.5-4B substantially degraded all metrics.
- Vision-language models: RTNav did not require the largest available VLM, but sufficient semantic reasoning capacity remained important for reliable frontier selection.
- Real-world behavior: Semantic frontier selection guides exploration, while high-frequency detection captures peripheral targets and VLM verification filters false positives in a real-world episode.
- Overall performance: Across HM3D-v1, HM3D-v2, and HM3D-OVON, RTNav improves SR by up to 11% and SCT by up to 5.1 points over prior methods.
- Limitations: Static environments and tightly coupled embodied tasks remain outside the study’s evaluated scope.The authors identify locomotion, manipulation, and dynamic scenes as future directions.
APPENDIX
The appendix documents the ROS-based evaluation setup, action conversion, baseline reproduction, and implementation adjustments used to compare synchronous and asynchronous execution.
- Execution modes: In real-time mode, env.step() runs continuously at 30Hz using the latest available action and updated timestamp.Synchronous mode calls env.step() whenever the agent produces a new action.
- Execution modes: The synchronous benchmark waits indefinitely for agent output, whereas the real-time benchmark runs environment and agent nodes independently at their own frequencies.
- Action mapping: Discrete agent actions were converted into corresponding continuous commands for the real-time simulation design.
- Action mapping: With instant inference, 500 synchronous steps and 500 asynchronous seconds provide effectively equivalent task budgets.Termination conditions are checked at every 30Hz simulation step.
- Baseline reproduction: The evaluation reproduced original synchronous results while documenting compatibility changes, discrepancies, and outdated dependencies.
- Implementation fixes: Minor fixes handled malformed VLM responses and constrained map updates to predefined spatial bounds without modifying the underlying navigation method.
- Baseline reproduction: Pose changes for SemExp-based methods were computed from simulator GPS and compass readings to prevent map drift from asynchronous execution and ROS delays.
- Baseline reproduction: ROS-wrapped baselines were checked against official codebases on HM3D-v1, with reproduced results compared against reported results.
C. RTNav Implementation Details
RTNav combines open-vocabulary perception, persistent object memory, VLM verification, semantic frontier selection, and asynchronous execution to support real-time object navigation.
- Perception and mapping: RTNav generates up to two target synonyms and combines them with filtered LVIS categories to construct detector candidates.The detector vocabulary contains 405 common indoor categories after excluding six small fixture categories.
- Perception and mapping: The OWLv2 detector uses 960 × 960 resolution, a 0.25 confidence threshold, seven text templates, and NMS IoU threshold 0.6.
- Object memory and verification: Persistent object-graph nodes become confirmed after observations from three distinct viewpoints.Distinct viewpoints require at least 0.1 m translation or approximately 3° yaw difference.
- Object memory and verification: RTNav submits up to eight object candidates in parallel to a VLM and accepts candidates only when p_yes > 0.90.Competing detector or graph labels are included when available; three confident rejections blacklist a node.
- Object memory and verification: MobileSAM refines accepted masks to obtain object centroids and nearest-surface navigation goals.Far or truncated detections remain candidate targets until a better view enables verification within 3 meters.
- Implementation efficiency: Perception, mapping, frontier detection, decision-making, and navigation execute asynchronously, with cached embeddings and batched inference improving efficiency.
- VLM prompts: The VLM is called on demand for synonym selection, frontier selection, and object verification.
- VLM prompts: Frontier selection prefers semantically relevant directions or traversable areas while avoiding walls, closed doors, dead ends, and stairs.
D. Task Details
The evaluation uses standard HM3D and OVON validation tasks with fixed motion and stopping rules, while category visualizations report RTNav’s navigation metrics only for sufficiently represented categories.
- Task and simulator configurations: All methods were evaluated on HM3D-v1, HM3D-v2, and OVON validation splits in Habitat.
- Task and simulator configurations: Agents use a 0.25 m forward step and 30° turn angle, with sliding disabled across benchmarks.
- Task and simulator configurations: An episode succeeds when the agent calls STOP within the benchmark-specific success distance of a target instance.
- Category-level evaluation: Fig. 10 reports category-level SR, SPL, and SCT for RTNav, omitting OVON categories with fewer than 25 episodes.The number of evaluated episodes is shown for each displayed category.
E. Additional Results and Analysis
RTNav’s performance varies substantially across object categories and benchmarks, with failures dominated by timeouts and exploration challenges. The real-world system combines RGB-D sensing with SLAM-based mapping and waypoint control.
- Performance by Goal Category: Chairs, beds, couches, refrigerators, and pianos are generally easier targets, while televisions, potted plants, hangers, blinds, and flowerpots are harder to locate.The reported differences are associated with target size, visual distinctiveness, and predictable room contexts.
- Failure Analysis: Most failures on HM3D-v1 and HM3D-v2 are timeouts rather than incorrect STOP decisions.HM3D-v1 additionally has more exploration failures because targets may lie on another floor or be missed during exploration.
- Failure Analysis: HM3D-OVON failures are more dispersed across categories, consistent with its more challenging open-vocabulary targets.A narrower, taller camera field of view can also make some objects easier to miss during exploration.
- Real-World Experiments: Real-world experiments used RGB-D from a head-mounted RealSense D435i camera and global pose from onboard HectorSLAM with a 2D LiDAR.The deployed system fused SLAM occupancy with RGB-D-reconstructed obstacles for mapping.
- Real-World Experiments: For real-world navigation, RTNav replaced Habitat’s PointNav policy with a modified FMM planner and used a continuous PD controller to track generated waypoints.The replacement addressed the mismatch between Habitat-trained PointNav and real-world deployment.