PaperlayerView exact revision on arXiv

Source-linked AI summary

LAPF: LLM-Agent-Based Path Finder Using the UAVScenes Dataset

Yousef Emami, Mohammadhossein Homaei, Hao Zhou, Miguel Gutiérrez Gaitán, Atefeh Hajijamali Arani, Rui Zhang

arXiv:2608.15175v1cs.ROcs.AI

TL;DR

Autonomous UAV navigation needs adaptable decision-making beyond approaches limited by predefined models, task-specific training, or weak agentic capabilities. LAPF integrates perception, memory, planning, and tool-enforced action in a closed-loop architecture, reducing mean path length by 17.2% and 15.6% versus CoT prompting while maintaining zero clamp events.

  • Problem

    Existing UAV navigation approaches rely on environment modeling or task-specific training and lack unified integration of heterogeneous perception, reasoning, memory, and action.

  • Method

    LAPF integrates multimodal perception, episodic memory, planning, and tool-enforced action into a closed-loop architecture for adaptive waypoint generation and hazard response.

  • Results

    17.2% and 15.6%: LAPF reduces mean path length relative to CoT prompting across open-field and obstacle-injected scenarios while maintaining zero clamp events.

  • Takeaways & Limitations

    LAPF’s memory-grounded feedback and tool-enforced action distinguish it from prompting-only methods as task complexity increases.

  • Takeaways & Limitations

    The evaluation is limited to UAVScenes, a single scene, one backbone, and three trials per configuration, excluding dense urban settings and pedestrian interactions.

Abstract

from arXiv · show

Uncrewed aerial vehicles (UAVs) are increasingly deployed for autonomous navigation in complex outdoor environments, where dynamic conditions and mission requirements require intelligent adaptive decision-making. Existing optimization-based, Machine Learning (ML), and Reinforcement Learning (RL) approaches often rely on predefined models or task-specific training, limiting their generalization and adaptability in uncertain scenarios. Recent Large Language Model (LLM)-assisted approaches offer promising reasoning capabilities but remain constrained by limited agentic functionality, including insufficient memory, planning, and tool interaction mechanisms.This paper proposes an LLM-Agent-Based Path Finder (LAPF) framework for autonomous UAV navigation in town-scale outdoor environments. LAPF extends LLM-assisted navigation by integrating perception, memory, planning, and action modules into a closed-loop cognitive architecture. The proposed agent leverages prior navigation experiences, performs Chain-of-Thought (CoT) reasoning, couples each detected hazard to a bounded corrective action, and dynamically refines waypoint decisions based on environmental feedback.The three independent trials per method demonstrate that LAPF achieves mean path lengths of 512.83 m and 506.37 m, compared to the straight-line optimum of 497.33 m, corresponding to path length reductions of 17.2% and 15.6% relative to CoT prompting and absolute path efficiencies of 97.1% and 98.1% in open-field and obstacle-injected scenarios, respectively. Furthermore, LAPF is the only evaluated approach that couples every detected hazard to a bounded, metric-neutral corrective action while maintaining near-goal stability, with zero clamp events in both scenarios, whereas CoT prompting increases from 9.7 to 14.0 events.

I. INTRODUCTION · II. RELATED WORK · A. Classical Path Planning

The paper motivates LAPF by limitations in conventional UAV navigation and standalone LLM prompting, then positions it against classical path-planning methods and evaluates its closed-loop multimodal agent architecture on UAVScenes.

  • I. INTRODUCTION: UAVs support three-dimensional mobility, dynamic Line-of-Sight establishment, and high data throughput for complex outdoor applications including public safety, monitoring, agriculture, and delivery.Their trajectories can adapt to mission requirements and dense urban environments where infrastructure may be obstructed or unavailable.
  • I. INTRODUCTION: Optimization-based, ML, and RL approaches are limited by predefined models, task-specific training, insufficient generalization, and difficulty handling dynamic, uncertain environments.Existing methods may also require extensive environment modeling, offer limited interpretability, and struggle to integrate heterogeneous perception information.
  • I. INTRODUCTION: Conventional text-based LLMs cannot directly interpret heterogeneous UAV inputs such as RGB, LiDAR, depth, and 6-DoF localization data.Reliable navigation additionally requires memory, adaptive planning, long-term spatial reasoning, continual environmental interaction, and tool integration.
  • I. INTRODUCTION: LAPF integrates multimodal perception, episodic memory, tool-enforced action, adaptive waypoint generation, and autonomous replanning within a closed-loop perception–reasoning–action architecture.It combines native RGB input to an open-weight VLM with symbolic LiDAR and 6-DoF pose representations in one inference call.
  • I. INTRODUCTION: 17.2% and 15.6%: LAPF reduces mean path length relative to CoT prompting in open-field and obstacle-injected scenarios, respectively.The evaluation uses the UAVScenes AMtown03 sequence, identical multimodal observations, and the same Qwen2-VL-7B backbone for all methods.
  • I. INTRODUCTION: 97.05% and 98.09%: LAPF reaches these path efficiencies in open-field and obstacle-injected scenarios, respectively, while maintaining zero near-goal clamp events in both.These results indicate that the reported advantage is preserved across both evaluated scenarios.
  • II. RELATED WORK: The related-work section reviews classical UAV path planning, LLM-agent frameworks for wireless networks, and emerging LLM-assisted UAV navigation before identifying the research gap motivating LAPF.The review frames LAPF as a response to limitations in existing approaches.
  • A. Classical Path Planning: Classical studies address urban UAV safety, efficiency, autonomy, collision-aware navigation, obstacle-dense planning, urban mapping, risk assessment, and trajectory optimization.Examples include V2X-based dynamic risk maps with RRT, RRT for obstacle-dense environments, SLAM-assisted mapping, risk-cost path planning, and improved particle swarm optimization.

B. LLM Agent for Wireless Networks · C. LLMs for UAV Networks

Prior work spans agentic wireless-network management and UAV-specific LLM applications, but the reviewed approaches leave a gap between architectural agency, multimodal grounding, and safety-critical physical navigation. LAPF addresses this gap through explicit memory, tool-enforced action, no fine-tuning, and multimodal UAVScenes data.

  • B. LLM Agent for Wireless Networks: WirelessAgent combines perception, memory, planning, and action through LangGraph workflows for autonomous wireless-network management.It achieves higher bandwidth utilization than prompt-based methods while approaching rule-based optimization performance.
  • B. LLM Agent for Wireless Networks: WirelessLLM-Agent uses semantic state serialization and MoE-LoRA for efficient knowledge sharing in multi-task wireless optimization.Its framework includes a two-stage training strategy, although the supplied passage truncates the remaining details.
  • C. LLMs for UAV Networks: Emami et al. establish a unified framework for LLM-assisted UAV systems, positioning LLMs as high-level reasoning modules for autonomous aerial networks.Their survey summarizes architectures, methodologies, and applications.
  • C. LLMs for UAV Networks: LLM-assisted UAV studies support SAR scheduling and public-safety control by adapting data collection, trajectory planning, and velocity to operational conditions.The SAR work considers dynamic channel conditions, battery constraints, and queue states; other work addresses trajectory planning and adaptive velocity control.
  • C. LLMs for UAV Networks: AutoFly extends UAV reasoning with an end-to-end VLA model that navigates unknown environments using coarse guidance rather than predefined trajectories.The approach incorporates pseudo-depth spatial reasoning, progressive multimodal training, and a real-world autonomous-navigation dataset.
  • C. LLMs for UAV Networks: LAPF combines explicit memory, tool-enforced action, and no fine-tuning while grounding decisions in RGB, LiDAR, and 6-DoF pose from UAVScenes.The contrast is with wireless agents lacking multimodal grounding and aerial safety-critical design, and UAV-specific works relying on prompting alone.
  • C. LLMs for UAV Networks: The literature gap separates domain-general agentic frameworks from architecturally shallow UAV-specific systems.Wireless agents provide memory and/or tool use but target network management without multimodal sensor grounding; UAV works optimize scheduling or trajectory parameters through prompting alone without persistent navigation memory.

III. SYSTEM MODEL

LAPF models a single UAV interacting with an onboard or edge-based LLM agent through closed-loop path planning, using UAVScenes observations for evaluation. Its synchronized sensing and localization suite supports multimodal navigation, while dense urban settings and some annotated modalities remain outside the present implementation.

  • System architecture: LAPF comprises a single UAV, an onboard or edge-based LLM agent, and UAVScenes observations connected through closed-loop interaction for adaptive path planning.The UAV and LLM agent interact continuously during navigation.
  • Sensor suite: The UAV uses a synchronized Livox-Avia LiDAR and high-resolution RGB camera at 2448×2048 pixels and 10 Hz, supplemented by GNSS/IMU with RTK support.The camera–LiDAR configuration provides 2D texture and 3D spatial measurements.
  • Operating environments: UAVScenes covers towns, valleys, airports, and islands, but excludes high-rise buildings, narrow streets, and pedestrian crowds from the current benchmark.Extension to dense urban environments is identified as future work.
  • Localization: The localization system supplies 6-DoF poses (x, y, z, ψ, θ, ϕ) reconstructed via SfM and aligned with the 3D map.Median positional accuracy on benchmark splits is approximately 6.1 m, so benchmarked methods do not achieve submeter localization accuracy.
  • Planner inputs: Although UAVScenes annotates dense depth maps and semantic masks, the present work does not supply Dt or St to the planner.The instantiated planner therefore uses a subset of the full UAVScenes sensor suite.

IV. PROBLEM FORMULATION · A. Decision Process and Observation Model

LAPF formulates UAV navigation as a partially observable sequential decision process that minimizes projected mission length while enforcing hazard responses. Its observation model combines RGB imagery, 6-DoF pose, and summarized LiDAR clearance, with conservative observation-level safety enforcement.

  • IV. PROBLEM FORMULATION: LAPF minimizes projected mission length while ensuring an enforced hazard response during autonomous navigation.The agent uses RGB images, summarized LiDAR returns, and 6-DoF localization states from UAVScenes for LLM-assisted path planning.
  • A. Decision Process and Observation Model: The mission is modeled as a discrete-time, partially observable sequential decision process over horizon Tmax.At decision step t, the agent holds commanded state wpt ∈ R3, initialized at wp0 = p0, and receives an observation.
  • A. Decision Process and Observation Model: Each observation includes an RGB frame It, a 6-DoF pose Lt = (Rt, pt) ∈ SE(3), and a compact LiDAR statistic σt.The observation pathways preserve the RGB frame as an image while reducing LiDAR and pose information to symbolic state before multimodal fusion.
  • A. Decision Process and Observation Model: The LiDAR statistic captures return geometry through boresight coordinates, return count nt, Euclidean-range bounds, lateral bounds, and minimum boresight coordinate ct.The minimum boresight coordinate ct lower-bounds Euclidean range for the corresponding return.
  • A. Decision Process and Observation Model: The clearance test is conservative by construction because ct lower-bounds Euclidean range and therefore cannot under-declare a hazard.The enforcement mechanism couples the registered scalar hazard signal to a bounded response at every decision step.
  • A. Decision Process and Observation Model: The instantiated hazard channel is boresight clearance ct, which certifies separation below the platform rather than clearance along the commanded-motion direction.The test therefore guarantees a prescribed response to registered clearance violations, not horizontal collision avoidance.
  • A. Decision Process and Observation Model: Because LiDAR resolves range but not bearing, waypoint-level clearance cannot be evaluated from σt without an uncertified bearing assumption.The observation-level test is conservative; lateral scene structure remains available through the nadir RGB frame, but enforcement excludes stochastic visual inference and applies identically to all three methods.

B. Memory-Conditioned Decision Policy  · C. Enforced Hazard Response and Stability Regulation

LAPF conditions waypoint decisions on recent episodic memory and trailing path-length feedback, then shields execution with guaranteed hazard checks, bounded corrective responses, and terminal-phase stability regulation. The model retains waypoint authority in hazard-free states, while hazard response takes precedence over step-magnitude regulation.

  • B. Memory-Conditioned Decision Policy \b: The policy conditions decisions on the k most recent episodic episodes, their execution outcomes y_j, generated reflections ρ_j, mission specification G, and trailing-window path-length feedback ϕ_t.The policy also incorporates a proposed waypoint c_wpt and requested tool set Tools_t.
  • C. Enforced Hazard Response and Stability Regulation: The proposed waypoint c_wpt is not executed directly; a hazard indicator is evaluated at every decision step using minimum admissible clearance d_safe.This check is guaranteed to run independently of whether the corresponding tool is named in a particular Toolst sampling.
  • C. Enforced Hazard Response and Stability Regulation: When h_t = 1, the model-authored waypoint proposal is discarded and replaced by a deterministic corrective operator based only on the commanded state and mission goal.The operator provides a prescribed, bounded reaction in critical states rather than optimizing the trajectory.
  • C. Enforced Hazard Response and Stability Regulation: The corrective operator functions strictly as a low-level failsafe guardrail, while memory and continuous reasoning help the agent anticipate hazards and minimize activation of that guardrail.Its stated role is not intelligent planning or trajectory optimization.
  • C. Enforced Hazard Response and Stability Regulation: A second operator regulates step magnitude during the terminal phase, preventing independently resampled coordinates from otherwise being charged as travelled distance.Its parameters include goal tolerance ε and maximum near-goal displacement δ_max.
  • C. Enforced Hazard Response and Stability Regulation: A hold-position operator is invoked when the agent selects WaypointAdjust, contributing to the terminal-phase execution safeguards.The supplied passage identifies this operator as part of the stated composition.
  • C. Enforced Hazard Response and Stability Regulation: Equation (9) defines a shielded policy π_safe: the language model retains waypoint authority in hazard-free steps, while C constrains only model-authored proposals.C is deliberately not applied to R, so hazard response has strict precedence over step-magnitude regulation.

D. Objective and Reported Estimators

The section defines travelled length, straight-line optimum, path efficiency, detour ratio, and clamp count, then uses a residual-distance estimator to prevent early termination from appearing efficient. It also establishes that enforcement is metric-neutral, while evaluation remains tied to goal progress over a fixed horizon.

  • Objective definitions: The framework defines travelled length and the straight-line optimum L⋆, alongside projected mission length, path efficiency, detour ratio, and clamp count.These quantities form the reported objective and estimator set.
  • Efficiency estimator: The estimator eL_t adds residual distance to the goal, preventing trajectories from receiving credit for terminating early away from p_goal.By the triangle inequality, eL_t ≥ L⋆ and η_t ≤ 1; equality requires a collinear, monotone start–goal sequence.
  • Efficiency estimator: Reported efficiencies are means of per-trial ratios η_t rather than ratios of mean path lengths, so Jensen’s inequality makes the two quantities differ slightly.The distinction applies to how efficiency results are aggregated and reported.
  • Metric neutrality: The enforcement pathway contributes zero to η_t and is scored through N_h, separating safety enforcement from trajectory-quality measurement.The decomposition is an exact identity on the realized trajectory, and the shield is designed to remain metric-neutral.
  • Metric neutrality: The enforcement response strictly reduces residual goal distance without adding an evaluation penalty, but this establishes no geometric reachability guarantee.The claim is limited to metric neutrality and monotone residual reduction.
  • Evaluation conditions: All methods are evaluated over an identical fixed step budget, with the terminal goal-reaching condition assessed at the horizon rather than used as a stopping rule.The residual term penalizes trajectories that have not met the goal-reaching condition.

V. PROPOSED LAPF

LAPF is an LLM-agent framework for autonomous UAV navigation that combines perception, memory, planning, action, and feedback in a closed-loop architecture. Its workflow uses recent navigation episodes and guaranteed hazard handling to refine and safely execute waypoint decisions.

  • Architecture: LAPF deploys an LLM agent on an edge server to support a UAV navigating the environment and updating navigation memory.The system comprises the UAV and an edge server hosting the agent and supporting memory updates.
  • Operational workflow: The closed-loop workflow comprises perception, memory, planning, action, and feedback stages that process observations, execute decisions, and update subsequent reasoning.Perception forms the observation, memory retrieves recent episodes, planning proposes a waypoint, action applies safeguards, and feedback closes the loop.
  • Memory: k = 3 recent episodes provide temporal context, with each episode containing an executed waypoint, outcome, and reflection for subsequent reasoning.Retrieval is recency-based and includes previously encountered trajectories, failures, and successful outcomes.
  • Action: Guaranteed hazard evaluation precedes execution, replacing unsafe waypoints with a bounded corrective operator and applying stability or hold-position operators when required.The hazard test runs at every decision step, and the resulting waypoint is transmitted to the UAV after action-stage modification.
  • Tool interaction: LAPF uses external tools—including obstacle checking, trajectory validation, localization, simulation, vector retrieval, and knowledge-base access—to evaluate candidate trajectories against safety and feasibility constraints.Tool invocation is mediated through a dedicated external tool suite.
  • Planning: The agent combines the mission goal, symbolic state, retrieved memory, feedback, system prompt, and RGB image to produce a reasoning trace, waypoint, and tool calls.The planning module passes this multimodal context to the agent backbone in a single inference call.

VI. NUMERICAL RESULTS AND DISCUSSIONS

This section evaluates LAPF through numerical experiments on the UAVScenes dataset, comparing baseline methods across navigation scenarios using path efficiency, safety, and overall navigation performance.

  • Evaluation Setup: The evaluation uses numerical experiments on the UAVScenes dataset to assess the proposed LAPF framework.The section first establishes implementation settings and baseline methods.
  • Comparative Evaluation: Comparative results are presented across different navigation scenarios.These comparisons are used to evaluate LAPF against baseline methods.
  • Evaluation Criteria: The experiments assess path efficiency, safety, and overall navigation performance.These criteria define the section’s evaluation focus.

A. Implementation of LAPF

LAPF is implemented as a fully local Python framework using Qwen2-VL-7B-Instruct on a MacBook Air. Its multimodal pipeline resizes RGB inputs, enforces structured JSON outputs, and monitors fallback handling during evaluation.

  • Implementation platform: LAPF runs in Python on a MacBook Air with an Apple M5 processor and 16 GB unified memory under macOS.
  • Model deployment: Qwen2-VL-7B-Instruct serves as the open-weight vision-language backbone, deployed locally through mlx-vlm without proprietary large-scale models.
  • Planning and fallback: The Planning module parses returned JSON into Thought, Waypoint, and ToolCalls, while geometric fallback activity is monitored through the llm_fallback_steps log.
  • Input processing: 1024×768 RGB inputs are produced from 2448×2048 frames before inference to control the visual token budget.
  • Prompt and output handling: The structured prompt template requires JSON-based outputs, with invalid responses handled through retries.

B. Baselines Description

The baselines compare direct LLM navigation with step-by-step reasoning, while all three methods use identical model, inputs, decoding, output, and stability settings.

  • Baselines: Pure LLM directly generates navigation decisions from the input state without external feedback, memory, or iterative action correction.
  • Baselines: CoT LLM performs step-by-step reasoning before generating navigation actions but remains non-agentic.
  • Common evaluation setup: All three methods use the same Qwen2-VL-7B backbone, decoding settings, multimodal observations, symbolic fields, output schema, and near-goal operators.They are additionally evaluated with the same near-goal stability operator C of (7) and hold-position operator H of (8).

C. Open-Field Navigation (AMtown03) · D. Obstacle-Injected Navigation (AMtown03)

Across open-field and obstacle-injected AMtown03 trials, LAPF produced the shortest, most efficient trajectories and maintained zero clamp events. Its memory, feedback, and action-constraint mechanisms supported stable navigation under both ordinary and injected environmental conditions.

  • C. Open-Field Navigation (AMtown03): 512.83 ± 26.78 m was LAPF’s shortest mean open-field path, near the 497.33 m straight-line optimum.T1 and T2 measured 497.33 m and 497.40 m, respectively.
  • C. Open-Field Navigation (AMtown03): 97.05 ± 5.09% was LAPF’s highest open-field mean efficiency, with the lowest detour ratio at 1.032 ± 0.056.Pure LLM reached 89.43 ± 2.45% efficiency, while CoT LLM reached 80.37 ± 3.85%.
  • C. Open-Field Navigation (AMtown03): Zero clamp events occurred for LAPF and Pure LLM in open-field trials, whereas CoT LLM averaged 9.7 events.LAPF couples LLM decisions with explicit action constraints and execution feedback to regulate waypoint generation.
  • C. Open-Field Navigation (AMtown03): 100% efficiency is admissible when the commanded sequence is collinear with and monotone along the start–goal segment.The metric rewards geometric directness but does not separately certify per-step kinematic realizability.
  • D. Obstacle-Injected Navigation (AMtown03): 506.37 ± 16.68 m was LAPF’s shortest mean obstacle-injected path, closest to the 497.33 m optimum.Pure LLM reached 581.51 ± 35.20 m, while CoT LLM reached 599.89 ± 37.20 m.
  • D. Obstacle-Injected Navigation (AMtown03): LAPF maintained efficient, controlled navigation under environmental perturbations by perceiving conditions, preserving memory, enforcing constraints, and refining decisions iteratively.The obstacle-injected scenario required adaptation beyond single-pass trajectory generation.
  • D. Obstacle-Injected Navigation (AMtown03): 98.09 ± 3.17% was LAPF’s obstacle-injected efficiency, with a 1.020 ± 0.034 detour ratio.LAPF’s path-length advantages over CoT LLM and Pure LLM were 93.5 m and 75.1 m, respectively.

E. Discussion · VII. CONCLUSION

LAPF’s discussion attributes its navigation gains to the combined effects of episodic memory, feedback, and tool-enforced action within a closed loop. The conclusion presents this architecture as an adaptive cognitive layer for semantic, replanning-intensive UAV navigation, while noting limits on evaluated environments.

  • E. Discussion: LAPF’s waypoint decisions jointly condition on episodic memory M_t and feedback signal ϕ_t, unlike non-agentic baselines with M_t = ∅ and ϕ_t = ∅.Because all methods receive byte-identical observations and the same operators C and H, the measured gap isolates memory and feedback rather than reasoning depth or perception quality.
  • E. Discussion: Recency-based retrieval over the k = 3 most recent episodes conditions each proposal on how prior commands actually resolved.Each episode carries an executed waypoint, its outcome, and a reflection on their discrepancy.
  • E. Discussion: 98.09% efficiency is preserved under obstacle injection because prior hazard episodes bias subsequent waypoints toward directions anticipating the shield.In open-field navigation, the resulting command sequence is near-collinear, with T1 coincident with L⋆ at N_h = 0.
  • E. Discussion: The guaranteed-invocation Obstacle Checker tests clearance independently at every step, coupling each registered hazard to the bounded response of (6).The discretionary waypoint adjuster regulates step magnitude near the goal.
  • E. Discussion: Memory and feedback shorten and stabilize trajectories, while tool-enforced action guarantees hazard responses; neither mechanism alone reproduces the full behavior.The discussion identifies reasoning depth alone as insufficient when generated actions are not tested through feedback or enforced hazard-action pathways.
  • E. Discussion: LAPF targets environments requiring semantic interpretation, adaptive replanning, and heterogeneous information integration rather than replacing classical geometric planners for accurate maps and deterministic objectives.The framework provides a higher-level cognitive layer that formulates, revises, and validates navigation strategies.
  • VII. CONCLUSION: LAPF integrates multimodal perception, episodic memory, planning, and tool-enforced action in a closed-loop UAV-navigation architecture using observations from UAVScenes.It combines native RGB input with symbolic LiDAR and 6-DoF pose representations through a compact open-weight VLM for edge-scale navigation on a single consumer laptop.
  • VII. CONCLUSION: Zero clamp events under obstacle injection accompanies LAPF’s higher path efficiency, shorter trajectories, and improved navigation stability versus conventional and CoT prompting.The conclusion identifies memory-grounded feedback paired with tool-enforced action, rather than reasoning depth alone, as the decisive agentic advantage.
Loading 2608.15175v1…