Source-linked AI summary
OptiSight: Bridging Semantic Reasoning and Geometric Control for Embodied Navigation
Alperen Avan, Jordi Sanchez-Riera
TL;DR
Autonomous indoor navigation must combine semantic understanding with precise, efficient geometric control. OptiSight uses selective VLM reasoning, Grounded-SAM target grounding, camera projection, and deterministic visual servoing within a finite-state architecture. Across 24 AI Habitat scenarios, it handles diverse navigation conditions while requiring only limited VLM queries.
Problem
Indoor navigation systems must reconcile semantic instruction understanding and open-vocabulary perception with precise geometric control and resource constraints.
Method
OptiSight combines finite-state selective VLM reasoning, Grounded-SAM target grounding, visual-to-3D geometric projection, and deterministic visual servoing.
Results
Across 24 AI Habitat indoor scenarios, OptiSight handles obstacle avoidance, semantic ambiguity, partial observability, and challenging viewpoints while requiring limited VLM queries.
Takeaways & Limitations
The evaluation supports combining semantic reasoning with lightweight geometric control for efficient embodied navigation.
Abstract
from arXiv · showhide
Autonomous indoor navigation requires both semantic understanding and precise geometric control. We propose OptiSight, a hybrid framework that combines Vision-Language Model reasoning with deterministic visual servoing through a finite-state Chain-of-Thought architecture. Grounded-SAM localizes open-vocabulary targets, while camera projection geometry converts visual observations into navigation commands without requiring dense mapping. The VLM is queried only at key decision points, reducing computational overhead while geometric control handles continuous navigation. Experiments in AI Habitat demonstrate reliable zero-shot navigation across diverse indoor scenarios, including obstacle avoidance and semantic ambiguity, while operating within an 8~GB VRAM budget. The source code is available at https://github.com/avanalperen/OptiSight-Python-Multimodal-CoT-for-Visual-Reasoning.
I. INTRODUCTION
OptiSight addresses indoor navigation by combining semantic scene understanding with precise geometric control. Its hybrid architecture uses selective VLM reasoning and deterministic visual servoing to support efficient navigation without dense mapping.
- Indoor navigation requires both geometric obstacle avoidance and semantic understanding of objects, spaces, and natural-language goals.
- The framework organizes navigation into Perceive, Think, and Navigate stages, converting high-level instructions into geometric control commands.This design separates semantic reasoning from low-level navigation.
- OptiSight translates semantic observations into perspective-correct navigation angles using Grounded-SAM and camera projection geometry.The approach avoids dense mapping and neural scene reconstruction.
- A finite-state Chain-of-Thought architecture invokes the VLM only at critical transitions while deterministic visual servoing controls continuous motion.The states are Search, Find, Scan, Navigate, and Recover.
- Experiments in AI Habitat evaluate OptiSight as a computationally efficient framework for semantic navigation in complex indoor environments.The reported evaluation emphasizes resource-constrained robotic deployment.
A. Vision-Language Models for Embodied Navigation
Vision-Language Models expand embodied agents’ semantic and zero-shot capabilities, but OptiSight distinguishes semantic reasoning from continuous motion generation. This avoids end-to-end policy learning and delegates control to deterministic geometric methods.
- Vision-Language Models provide zero-shot perception, open-vocabulary recognition, contextual understanding, and natural-language instruction interpretation.
- Recent embodied foundation models and reasoning-driven navigation methods extend semantic capabilities across diverse robotic domains and multi-step tasks.
- OptiSight uses pretrained VLMs for semantic perception and high-level reasoning without end-to-end policy learning or large-scale navigation datasets.
B. Semantic and Object-Goal Navigation
Semantic and object-goal navigation broaden navigation beyond predefined coordinates, while memory and hierarchical methods address reasoning efficiency. OptiSight instead limits semantic inference to decision points and uses lightweight visual servoing continuously.
- Object-goal navigation requires locating objects specified by semantic categories rather than predefined coordinates.OVON expanded this setting to more than 15,000 object instances.
- Memory-augmented and hierarchical approaches reduce repeated reasoning or decompose complex instructions into navigational subgoals.
- Repeated LLM or VLM inference increases computational overhead, whereas OptiSight invokes reasoning only at critical decision points.
- OptiSight maintains continuous navigation through lightweight visual servoing, substantially reducing latency.
C. Semantic Grounding and Chain-of-Thought Reasoning
Reasoning-driven embodied systems use explicit intermediate reasoning for planning, grounding, and navigation, but many repeatedly query large VLMs. OptiSight combines state-driven reasoning with deterministic low-level visual servoing.
- Chain-of-Thought methods extend explicit intermediate reasoning from language tasks into robotic perception, planning, and control.
- Recent navigation systems integrate Chain-of-Thought reasoning into policies for long-horizon planning and unified 3D task execution.
- Reasoning paradigms have also been applied to manipulation through ReAct-style planning, executable code generation, and tool use.
- OptiSight invokes semantic reasoning at finite-state transitions while deterministic visual servoing handles continuous low-level motion.
III. METHODOLOGY
OptiSight integrates VLM reasoning, open-vocabulary grounding, geometric projection, and deterministic motion control in a closed-loop navigation framework. Its finite-state execution decomposes high-level instructions into semantic reasoning and geometric navigation stages.
- OptiSight integrates VLM reasoning, open-vocabulary object grounding, geometric projection, and deterministic motion control.
- The finite-state machine governs execution through five states: Search, Find, Scan, Navigate, and Recover.Each state specifies a subtask, perception module, and transition conditions.
- The nominal pipeline follows Search → Find → Scan → Navigate, with the VLM queried only at semantic decision points.State-specific prompt templates preserve execution context without requiring long conversational histories.
- Grounded-SAM localizes targets, projection geometry estimates their spatial locations, and geometric planning produces direct or collision-free bypass trajectories.The pipeline projects segmented targets into the 3D camera frame and uses obstacle observations during Scan.
- During Navigate, proportional steering follows 3D waypoints, while Recover re-invokes the VLM when targets are lost or paths become invalid.Waypoints are discarded after the robot approaches the current target waypoint.
A. Finite State Machine (FSM)
The FSM combines semantic search and target localization with geometric obstacle scanning and waypoint-based visual servoing. It periodically reassesses traversability and can recover when semantic or geometric execution fails.
- Search: Search uses the VLM to determine whether a semantic target is visible and recommends the next navigation action.If the target is absent, the robot performs nine 10° rotations separated by 0.3-second intervals.
- Find: Find applies Grounded-SAM open-vocabulary localization and up to three refinement passes before projecting the target into 3D camera coordinates.The refinement passes produce a consistent low-noise bounding region.
- Scan: Scan tilts the camera downward by 40° to inspect traversable floor and projects segmented obstacles into 3D space.An obstacle within the predefined 0.50 m safety margin triggers a collision-free bypass trajectory.
- Navigate: Navigate follows the generated trajectory through 3D waypoints using proportional visual servoing and switches waypoints below a 0.30 m distance threshold.The controller scales steering commands according to each waypoint’s projected size.
- Navigate: The navigation process returns to Scan every 15 control steps to reassess floor traversability and update the trajectory when obstacles change.
B. Dual-Isolated System Architecture
OptiSight separates semantic inference from geometric navigation into independent runtime environments connected by asynchronous socket communication. This design supports heterogeneous software stacks and operation within an 8 GB VRAM budget.
- The architecture runs geometric navigation and VLM-based semantic processing in two independent runtime environments.The navigation environment hosts Habitat-Sim, projection, planning, waypoint generation, and motion control; the second hosts the VLM and Grounded-SAM.
- The environments communicate asynchronously through a lightweight socket interface, exchanging observations, prompts, and semantic predictions when reasoning is required.Geometric navigation continues while the VLM performs inference in the background.
- Separating semantic inference from geometric control isolates framework dependencies and enables modern multimodal models to work with legacy robotics software.
- The complete system operates within an 8 GB VRAM budget for resource-constrained edge platforms such as NVIDIA Jetson.
IV. EXPERIMENTS
OptiSight is evaluated in AI Habitat across diverse indoor navigation scenarios designed to test semantic grounding, obstacle avoidance, perception, and geometric ambiguity. The automated evaluation reports mean performance over repeated independent runs using task, safety, and operational metrics.
- The evaluation covers 24 indoor scenarios, with 10 independent runs per scenario and two 2B VLMs used across scenario subsets.Qwen3.5-2B is used for the first 12 scenarios and Moondream2-2B for the remaining 12; Grounded-SAM is used throughout.
- The scenarios test single- and dual-obstacle avoidance, semantic disambiguation, partial observability, perception, and geometric ambiguities.The first three scenario types emphasize navigation and semantic grounding, while the latter three emphasize perception and geometric reasoning.
- Results are reported as means over 10 independent runs using an automated pipeline that records quantitative metrics and step-by-step execution traces.Traces include state transitions, VLM requests, parsing events, collisions, and recovery actions.
- The evaluation measures mission success, success rate, execution time, total distance, total steps, collisions, recoveries, VLM requests, and minimum obstacle distance.These metrics jointly characterize navigation effectiveness, safety, computational behavior, and operational behavior.
A. Results
Across 24 indoor scenarios, OptiSight completes many missions with few VLM requests, while collisions, recovery cycles, and constrained clearances mark harder cases. Execution time increases when geometric navigation and obstacle avoidance become more demanding.
- 8 of 12 scenarios in Experiments 01–12 satisfy the mission-success criterion, with successful cases generally requiring few VLM requests and no collisions or recovery actions.Experiments 03, 07, and 09 achieve 100% success rates.
- Collisions occur in Experiments 05, 06, 08, 10, 13, 14, 16, 17, and 22, and collision-prone experiments also trigger recovery actions.Experiment 22 records four collisions and two recovery procedures.
- 7 of 12 scenarios in Experiments 13–24 satisfy the mission-success criterion, including four experiments with 100% success rates.Experiments 19, 20, 23, and 24 achieve 100% success rates.
- Experiments 13–24 generally take 46.5–118.1 s, with extended durations arising primarily from geometric navigation, obstacle avoidance, or repeated scanning rather than continuous VLM inference.Experiments 13, 17, and 22 take 97.3, 98.9, and 118.1 s, respectively.
- Successful experiments operate with clearances as small as 0.25 m, whereas Experiment 22 reaches 0.15 m and experiences multiple collisions.The results associate constrained obstacle proximity with more difficult navigation conditions.
- One to four VLM requests are typical across many experiments, while challenging scenarios require more interactions and longer execution times.Experiments 05, 06, and 08 require 8, 6, and 6 requests, with execution times from 59.0 to 68.7 s.
V. CONCLUSIONS
OptiSight combines selective VLM reasoning with deterministic geometric navigation through finite-state control, target grounding, and visual-to-3D projection. Across 24 AI Habitat scenarios, it handles diverse indoor conditions with limited VLM queries, while future work targets broader tasks and physical-robot validation.
- OptiSight combines selective VLM reasoning with deterministic geometric navigation through a finite-state machine, Grounded-SAM target grounding, and visual-to-3D geometric projection.
- Across 24 AI Habitat scenarios, OptiSight handles obstacle avoidance, semantic ambiguity, partial observability, and challenging viewpoints while requiring only a limited number of VLM queries.
- Future work will extend OptiSight to a broader range of navigation tasks and validate it on physical robotic platforms.