PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-Semantic Reasoning using Large Language Models for Efficient UAV Search Operations

Marin Maletic, Marijana Peti, Tamara Petrovic, Stjepan Bogdan

arXiv:2608.28270v1cs.ROcs.AI

TL;DR

The paper addresses efficient UAV ObjectNav by targeting the limitations of computationally heavy, simulator-oriented navigation approaches. It combines real-time detection, 3D spatial mapping, LLM semantic reasoning, and feasible trajectory planning to prioritize likely search regions. Across simulated and real-world evaluations, the framework reports high search success with shorter missions and paths than conventional lawn-mower search.

  • Problem

    Existing navigation methods can be computationally unsuitable for onboard real-time inference and may rely on simulator-constrained action spaces rather than realistic UAV trajectories.

  • Method

    The framework integrates real-time object detection, 3D spatial representations, LLM-guided semantic relevance reasoning, and trajectory planning for UAV ObjectNav.

  • Results

    The framework achieved 100% mission success across diverse scenarios, reducing average mission duration by up to 36% and path length by over 50% versus conventional lawn-mower patterns.

  • Takeaways & Limitations

    LLM-guided semantic search supports targeted, context-aware UAV exploration with consistent localization across varying object classes, sizes, and occlusions.

Abstract

from arXiv · show

We present a real-time semantic navigation framework for Unmanned Aerial Vehicles (UAVs) focused on improving time efficiency in the Object Goal Navigation (ObjectNav) task. Central to our approach is a Large Language Model (LLM) that interprets user-provided natural language instructions and performs semantic reasoning over detected objects and spatial context to prioritize high-probability search regions. The system combines real-time object detection, 3D spatial mapping, and polynomial spline interpolation for smooth and feasible UAV trajectory planning. Unlike prior methods that rely on offline reasoning or simulator-constrained action spaces, our framework can operate in real time, continuously updating semantic relevance based on new observations. Experiments in both simulated and real-world settings demonstrate reductions in mission duration while maintaining high search accuracy, underscoring the effectiveness of LLM-guided reasoning for time- efficient UAV-based ObjectNav.

I. INTRODUCTION

ObjectNav requires UAVs to locate semantically specified objects in unseen environments, but existing approaches can be computationally demanding, simulator-constrained, or insufficiently focused on real-time mission efficiency. The proposed framework uses lightweight LLM reasoning, real-time perception, spatial information, and smooth trajectories to prioritize likely search regions.

  • ObjectNav asks agents to search unseen environments for objects specified by semantic categories and optional contextual descriptions.
  • Existing VLM-based methods improve semantic navigation but often impose computational loads unsuitable for onboard real-time inference.
  • Simulator-based approaches commonly use discretized actions and do not model realistic continuous robot maneuvers, limiting transferability to real-world systems.
  • The proposed UAV framework dynamically interprets user instructions and prioritizes high-likelihood search regions using real-time object detection and LLM-based reasoning.
  • Polynomial spline interpolation generates smooth, dynamically feasible 3D trajectories while the UAV continuously scans and refines its reasoning during flight.

II. METHODOLOGY

The methodology frames UAV ObjectNav as searching for a user-specified target in an unknown environment and combines semantic reasoning with spatially informed planning. Its four stages represent the environment, ground detected objects, reason over relevance, and plan trajectories.

  • The approach uses four stages: 3D environment representation, object detection and semantic grounding, LLM-guided decision logic, and trajectory planning.
  • The system searches autonomously for a target object specified by a natural-language instruction containing a semantic label and optional contextual description.
  • The objective is to guide the UAV toward regions where the target is more likely to occur, reducing search time and path length.
  • Depth-aware environmental representation supports safe UAV navigation and accurate 3D localization of detected objects.

1) Volumetric Space Representation Module:

The framework represents the environment with an OctoMap that separates navigable, occupied, and unknown space, while assuming an initial map and precise UAV localization. Camera-based real-time perception detects and semantically labels objects for downstream processing.

  • Volumetric Space Representation: OctoMap classifies each voxel as free, occupied, or unknown, defining navigable space, obstacles and objects, and unexplored space.The free, occupied, and unknown voxel sets are denoted S_free, S_occ, and S_unk.
  • Volumetric Space Representation: The method assumes the environment is mapped beforehand and the complete space S is known in advance.
  • Volumetric Space Representation: LiDAR and incremental mapping can remove the pre-mapped-environment assumption during real-time exploration.The UAV is also assumed to localize itself precisely from an arbitrary initial pose.
  • Volumetric Space Representation: A camera-based perception and semantic-grounding pipeline supplies detected objects to the LLM-guided search process.The pipeline combines visual perception with 3D object-position estimation using volumetric information.
  • Volumetric Space Representation: Each incoming image frame is processed in real time by a YOLO-based detector that outputs objects with bounding boxes, semantic labels, and probabilities.Bounding boxes encode center pixels, widths, and heights.

2) 3D Position Estimation Module:

The 3D position estimation module projects detected image objects into occupied OctoMap voxels, then consolidates repeated observations into distinct labeled objects. DBSCAN clustering groups detections by spatial proximity and shared semantic labels to produce consistent representations.

  • 3D Position Estimation Module: The estimator uses occupied OctoMap depth information to estimate 3D positions for objects detected in each image frame.
  • 3D Position Estimation Module: Camera intrinsics convert each detected object’s center pixel into a 3D viewing ray from the optical center.The computed direction accounts for camera resolution, lens distortion, and field of view.
  • 3D Position Estimation Module: The estimator represents each detected object with a 3D occupied-space position and its corresponding semantic label.
  • 3D Position Estimation Module: Ray casting from the UAV identifies the first occupied OctoMap voxel along each computed direction as the object’s location.
  • 3D Position Estimation Module: Repeated detections accumulated across frames can produce spatially scattered position estimates, especially for larger objects.
  • 3D Position Estimation Module: DBSCAN groups detections using spatial proximity and shared semantic labels, consolidating observations into distinct objects with consistent positions and labels.The final object set contains far fewer objects than the total accumulated detections.

1) Probabilistic Relevance Module:

The Probabilistic Relevance Module uses an LLM to estimate which detected object is the best place to search for a user-specified target, updating decisions as search data accumulate. Thresholds determine whether the UAV performs a local search, gathers more data through fallback scanning, or terminates after finding the target.

  • Decision logic: If exhaustive exploration fails to locate the target, the Bnone hypothesis becomes dominant and the search ends unsuccessfully.Bnone represents the hypothesis that the target is absent from the explored space.
  • Probabilistic relevance estimation: The module estimates a posterior relevance probability for each detected object using the target and accumulated search data.The data include user instructions, detected objects, spatial relations, and visited areas.
  • Probabilistic relevance estimation: The LLM selects the detected object with the maximum estimated posterior as the next relevant search location.This selection is expressed as O∗ = arg max P(Oi | D, T).
  • Decision logic: If the selected object’s probability is below the minimum threshold, the UAV defaults to a scanning strategy such as wall following to gather more data.The decision logic uses the accumulated observations to reassess subsequent actions.
  • Decision logic: When relevance exceeds the search threshold, the UAV conducts a thorough search around the selected object while continuing to estimate probabilities for newly detected objects.A sufficiently high estimated probability indicates that the target has been found and terminates the search.

2) Text prompt:

The text prompt gives the LLM both the search intent and structured environmental context so it can identify the detected object most likely to be near the target. Its response includes a relevance flag, the selected object with coordinates, and an explanation.

  • Prompt components: The prompt has system and user information components that specify the reasoning task and mission context.The system instruction asks the LLM to infer a probable target location from descriptions and observed objects.
  • Prompt components: Input data combine the natural-language target description, detected objects with 3D coordinates, and previously visited regions.Visited regions identify areas already thoroughly scanned during the search.
  • Response format: The required response flags whether any object exceeds the minimal relevance threshold and identifies the most relevant object.The selected object is specified by its label and 3D coordinates.
  • Response format: The prompt also requires an explanation of the LLM’s reasoning for selecting the identified object.This makes the semantic basis of the selected search region explicit in the returned response.

E. Trajectory planning and thorough search

The UAV performs a viewpoint-rich local search around the selected object while combining obstacle-aware 3D path planning with smooth continuous trajectory generation. The inspection path adapts to environmental constraints, including nearby walls.

  • Thorough search: A circular inspection trajectory around the selected object provides multiple viewpoints for detecting partially or fully occluded targets.The trajectory uses a specified radius and fixed altitude to scan the region from all angles.
  • Trajectory planning: The first planning stage uses 3D A* search to reach the region of interest and checks OctoMap for obstacle-free space.The map query verifies that the planned path does not intersect obstacles before circular scanning begins.
  • Trajectory planning: The second stage generates continuous trajectories over 3D waypoints using 7th-order polynomial interpolation.The resulting flight paths satisfy constraints on position, velocity, and acceleration.
  • Thorough search: When the relevant object lies near walls, the circular inspection path is adapted to a semicircle while maintaining safe obstacle distance.The illustrated case moves from (1.9, 8.1, 2.0) toward an object at (9.0, 2.5, 1.1) near x = 10 and y = 0.
  • Thorough search: During local search, newly gathered visual data and visited areas update the relevance module for iterative action selection.If the target is not found, the scene is reassessed to determine the next best action.

III. RESULTS

The evaluation analyzes semantic-guided search in simulation and real-world experiments using a Crazyflie 2.1 nano-UAV. The setup relies on pre-mapped environments, offboard YOLO11 detection, and OctoMap-based spatial grounding.

  • Evaluation setup: The experiments compare the proposed semantic-guided search framework with a traditional non-semantic search strategy.The evaluation includes both simulation and real-world experiments.
  • Evaluation setup: The real-world platform is a Crazyflie 2.1 nano-UAV equipped with an AI Deck and monochrome camera.Its limited payload prevents carrying advanced sensors such as LiDAR for real-time mapping.
  • Mapping and perception: Both simulated and real environments were pre-mapped with Cartographer SLAM using point clouds acquired from an external LiDAR sensor.The pre-built map supported navigation and semantic grounding.
  • Evaluation setup: The proposed framework was evaluated across three object-search scenarios against a baseline lawn mower scanning strategy.The scenarios are presented as simulation test cases in Fig. 5.
  • Mapping and perception: During flight, camera images were processed offboard with YOLO11, and detected objects were grounded in the map using OctoMap ray casting.The reported implementation separates object detection from the UAV’s onboard sensing payload.

A. Simulation results

The study evaluates LLM-guided semantic search against lawn-mower scanning across three simulated ObjectNav scenarios, using success rate, search-time improvement, and path-length reduction. Evaluation covers 120 flights from randomized starting positions and reports aggregate improvements only where both methods succeeded.

  • Evaluation metrics: Success Rate measures the percentage of trials in which the UAV successfully localized the target object.
  • Evaluation metrics: Search Time Improvement measures the percentage reduction in elapsed time from takeoff to successful localization relative to the baseline.
  • Evaluation metrics: Path Length Reduction measures the percentage reduction in distance traveled from takeoff to object localization relative to the baseline.
  • Aggregate results: Aggregate comparisons report success rate alongside mean search-time and path-length reductions with standard deviations, computed only where both methods localized the target.
  • Scenario I - Banana: In the banana scenario, LLM-guided search achieved 100% success versus 65% for lawn-mower scanning, while reducing search time by 36.02% and path length by 52.26%.The reported standard deviations were σ = 30.52 for search time and σ = 38.16 for path length.

1) Scenario I - Banana:

In the computer-mouse scenario, semantic priors directed the UAV toward office-related objects and achieved perfect localization while reducing both search time and path length. In the toy-zebra scenario, detailed user context supported similarly efficient search despite ambiguity.

  • Scenario II - Computer Mouse: Computer-mouse search achieved 100% success with LLM guidance versus 75% for the baseline, while reducing search time by 24.83% and path length by 43.50%.The system prioritized desks, monitors, and other office-related regions based on semantic priors.
  • Scenario III - Toy Zebra: Toy-zebra search achieved 100% success with LLM guidance versus 95% for the baseline, while reducing search time by 29.24% and path length by 51.93%.The instruction linked the toy zebra to sleep, enabling inference of bedrooms or resting areas as likely search regions.
  • Scenario III - Toy Zebra: The toy-zebra result attributes semantic effectiveness to detailed user context, because the object alone lacks an inherent connection to an indoor environment.
  • Overall interpretation: Across the reported scenarios, the framework reduced mission duration and trajectory complexity while maintaining perfect search accuracy in occluded or context-dependent cases.

B. Experimental results

The framework was validated on a lightweight Crazyflie nano-UAV in a real environment using semantic reasoning over mapped scene objects. The real-world experiment successfully localized an occluded computer mouse, supporting practical real-time feasibility while motivating future improvements in prompts, priors, and open-set detection.

  • Real-world validation: The system was validated on a Crazyflie 2.1 nano-UAV equipped with an AI Deck and monochrome camera in an 8 × 10 meter flight arena.OptiTrack supported localization, while LiDAR and Cartographer SLAM generated an OctoMap before flight.
  • Real-world validation: The UAV detected a table, chair, and monitor, then focused its search around the monitor and successfully localized the computer mouse.
  • Real-world validation: The experiment demonstrates feasibility and effectiveness for real-time semantic search on a lightweight UAV in a real environment.
  • Conclusion: The framework combines real-time detections, spatial maps, and natural-language instructions to transform exhaustive sweeps into targeted, context-aware exploration.
  • Future work: Future work will refine prompt engineering and semantic priors and integrate open-set object detectors for faster inference and zero-shot generalization in unseen environments.
Loading 2608.28270v1…