PaperlayerView exact revision on arXiv

Source-linked AI summary

UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility

Yonglin Tian, Fei Lin, Yiduo Li, Tengchao Zhang, Qiyao Zhang, Xuan Fu, Jun Huang, Xingyuan Dai, Yutong Wang, Chunwei Tian, Bai Li, Yisheng Lv, Levente Kovács, Fei-Yue Wang

arXiv:2501.02341v2cs.ROcs.AI

TL;DR

UAVs need greater autonomy for complex, dynamic operations, while LLMs offer capabilities relevant to reasoning and adaptation. This paper surveys their integration, supporting resources, tasks, and applications, then proposes Agentic UAVs as a forward-looking framework. It also identifies computational, latency, and safety challenges that constrain deployment.

  • Problem

    Current UAV operations remain largely human-controlled and have limited autonomy for complex environments and tasks.

  • Method

    The paper surveys UAV systems, foundation models, datasets, integration studies, applications, and methodologies, then proposes the Agentic UAVs framework.

  • Results

    The paper proposes Agentic UAVs as a framework for autonomous perception, reasoning, memory, and tool utilization.

  • Takeaways & Limitations

    The survey provides a reference roadmap for integrating LLMs with UAV systems across complex tasks and applications.

  • Takeaways & Limitations

    Large model size and computational demands can limit real-time UAV operation, while latency is problematic for time-sensitive missions.

Abstract

from arXiv · show

Low-altitude mobility, exemplified by unmanned aerial vehicles (UAVs), has introduced transformative advancements across various domains, like transportation, logistics, and agriculture. Leveraging flexible perspectives and rapid maneuverability, UAVs extend traditional systems' perception and action capabilities, garnering widespread attention from academia and industry. However, current UAV operations primarily depend on human control, with only limited autonomy in simple scenarios, and lack the intelligence and adaptability needed for more complex environments and tasks. The emergence of large language models (LLMs) demonstrates remarkable problem-solving and generalization capabilities, offering a promising pathway for advancing UAV intelligence. This paper explores the integration of LLMs and UAVs, beginning with an overview of UAV systems' fundamental components and functionalities, followed by an overview of the state-of-the-art in LLM technology. Subsequently, it systematically highlights the multimodal data resources available for UAVs, which provide critical support for training and evaluation. Furthermore, it categorizes and analyzes key tasks and application scenarios where UAVs and LLMs converge. Finally, a reference roadmap towards agentic UAVs is proposed, aiming to enable UAVs to achieve agentic intelligence through autonomous perception, memory, reasoning, and tool utilization. Related resources are available at https://github.com/Hub-Tian/UAVs_Meet_LLMs.

1. Introduction

UAVs improve perception and mobility across applications but remain constrained by human-dependent control and the challenges of complex environments. This paper systematically reviews UAV–LLM integration and proposes Agentic UAVs as a roadmap toward autonomous perception, reasoning, memory, and tool use.

  • UAVs support monitoring, transportation, logistics, agriculture, and inspection by improving intelligent systems’ perception and decision-making.
  • UAV Challenges: Flexible viewing angles, variable altitudes, three-dimensional mobility, swarming, and diverse environments complicate perception, reasoning, planning, and coordination.
  • Human-dependent flight control creates labor costs, safety risks, and limited scalability in complex environments.
  • LLMs offer reasoning, generalization, adaptation, and responsiveness that may enhance UAV autonomy in dynamic environments.
  • Paper Scope: The paper reviews UAV and foundation-model systems, datasets, integration methodologies, and applications across navigation, perception, and planning.
  • Paper Contribution: Agentic UAVs are proposed as a framework combining autonomous perception, reasoning, memory, and tool utilization.

2. Systematic Overview of UAV Systems

The paper presents UAV systems through their functional modules and embodied configurations, linking core system structure to UAV designs and applications.

  • UAV system foundations are organized around functional modules and embodied configurations.
  • Functional Modules: The functional view covers perception, planning, communication, control, navigation, human–drone interaction, and payload modules.
  • Embodied Configurations: The embodied-configuration view focuses on UAV structural characteristics, designs, and applications.

2.1. Functional Modules of UAVs

UAV functional modules divide responsibilities across sensing, planning, navigation, control, communication, and interaction. Together, these modules support autonomous flight, coordination, and human collaboration.

  • Perception Module: The Perception Module fuses onboard sensing and computer vision to interpret environments and support detection, segmentation, motion estimation, and swarm awareness.
  • Navigation Module: The Navigation Module converts planned trajectories into flight paths through state estimation, sensor fusion, and localization methods such as visual SLAM.
  • Planning Module: The Planning Module transforms mission objectives into adaptive, collision-free trajectories while coordinating routes among multiple UAVs.
  • Control Module: The Control Module converts navigation and planning references into actuator commands that regulate altitude, velocity, orientation, and attitude.
  • Communication Module: Communication modules exchange telemetry, control, and payload information across UAVs, ground stations, satellites, edge devices, and cloud services.
  • Interaction Module: Interaction modules support communication with human operators and other agents through interfaces including voice, gestures, augmented reality, and visualization.

2.2. Embodied Configurations of UAVs

UAVs use diverse embodied configurations with distinct trade-offs in speed, endurance, maneuverability, payload, noise, and operating conditions.

  • Fixed-Wing UAVs: Fixed-wing UAVs provide high speed, long endurance, and stable flight but cannot hover and require advanced piloting skills.
  • Multirotor UAVs: Multirotor UAVs offer low cost, easy operation, VTOL, and hovering, but have limited endurance and relatively low speed.
  • Unmanned Helicopters: Unmanned helicopters provide VTOL, hovering, maneuverability, payload capacity, endurance, and wind resistance, with relatively low speed as a limitation.
  • Hybrid UAVs: Hybrid UAVs combine VTOL with the long endurance and high speed associated with fixed-wing designs.
  • Flapping-Wing UAVs: Flapping-wing UAVs mimic birds or insects, offering quiet operation, efficiency, maneuverability, and compact size alongside lower payload capacity and greater control complexity.
  • Unmanned Airships: Unmanned airships are cost-effective and quiet but have limited agility and speed and are vulnerable to wind effects.

2.3. UAV Swarm

UAV swarms coordinate multiple vehicles through task allocation, communication, path planning, and formation control. These functions balance centralized coordination, decentralized autonomy, optimization quality, computational cost, and scalability.

  • Swarm foundations: UAV swarms collaboratively pursue shared objectives through decentralized decisions that adapt vehicle behavior to peers and environmental changes.The approach offers redundancy, scalability, and efficiency compared with individual UAV operations.
  • Task allocation: Task allocation distributes missions among UAVs, but its NP-hard complexity grows exponentially with swarm size and task count.Common formulations include TSP, VRP, MILP, and CMTAP.
  • Task allocation: Allocation methods include heuristic optimization, mathematical programming, reinforcement learning, neural networks, and market-based protocols.Heuristics search efficiently, mathematical programming provides precise solutions at higher computational cost, AI methods adapt dynamically, and auctions support distributed allocation.
  • Communication architecture: Swarm communication uses infrastructure-based or FANET architectures, trading centralized computation and optimization against single-point failures, infrastructure dependence, dynamic topologies, and reliability demands.FANETs allow direct UAV-to-UAV coordination while maintaining at least one ground-base or satellite link.
  • Path planning: Swarm path planning selects routes to targets while maintaining predefined inter-UAV distances for collision avoidance and optimizing mission-specific criteria.Relevant criteria include path length, travel time, energy consumption, and other event-specific constraints.
  • Formation control: Formation control enables UAVs to form, maintain, switch, or rebuild formations through centralized, decentralized, or distributed algorithms.Centralized control simplifies coordination but has communication overhead and single-point failure risks, whereas decentralized control is flexible but lacks global information.

3. Preliminaries on FMs

Foundation models provide general-purpose language, vision, and multimodal capabilities relevant to UAV systems. The section surveys their core abilities, representative models, and zero-shot vision applications for UAV imagery.

  • Overview: The section surveys LLMs, VFMs, and VLMs to establish foundations for integrating these models with UAV systems.It organizes the discussion around model characteristics and technical advantages.
  • LLMs: LLMs trained on large, diverse corpora support strong generalization, including zero-shot and few-shot transfer to new tasks.Prompt design and in-context examples guide task adaptation without explicit retraining.
  • LLMs: LLMs address complex problems by generating intermediate reasoning steps that decompose challenges into sequential sub-tasks.The Chain of Thought framework exemplifies this structured reasoning approach.
  • VLMs: VLMs integrate visual and textual information for tasks such as visual question answering and image captioning.Representative systems include GPT-4V, GPT-4o, LLaVA, and related models emphasizing perception, reasoning, efficiency, or deployability.
  • VFMs: VFMs extract expressive image features and transfer across vision tasks, with CLIP aligning visual and textual embeddings through large-scale image-text training.Their capabilities include zero-shot object detection, image segmentation, and depth estimation.
  • VFMs: Figure 3 illustrates UAV-domain zero-shot VFM applications using SynDrone imagery for car detection, whole-image segmentation, and depth generation.The figure pairs Grounding DINO with a natural-language “car” prompt, SAM for segmentation, and ZoeDepth for depth.

4. Datasets and Platforms for UAVs

The survey organizes publicly available UAV datasets and simulation resources by data modality, task, and application domain. These resources support training, evaluation, perception, navigation, tracking, action recognition, and domain-specific UAV research.

  • Resources: Public UAV datasets and 3D simulation platforms provide essential resources for developing, testing, and validating FM-based UAV systems.Datasets supply diverse training data, while simulation enables safe and controlled experimentation.
  • Resources: The survey catalogs publicly accessible datasets in Tables 3–9, recording update year and default RGB image or video formats.The collection includes datasets verified as publicly downloadable.
  • Data modalities: UAV datasets span video, RGB, LiDAR, infrared, depth, and textual data, with video and RGB images predominating.Some datasets add subtitles or question-answering functionality to support multimodal research.
  • General-domain datasets: General-domain datasets are grouped by environmental perception, event recognition, object tracking, action recognition, and navigation.Environmental perception includes detection, segmentation, and depth estimation, while related tasks are listed separately for clarity.
  • Task datasets: Representative datasets support multimodal UAV perception, tracking, action recognition, navigation, and localization across complex environments.Examples include AirFisheye for multimodal urban perception, WebUAV-3M for tracking, MEVA and UAV-Human for actions, and CityNav for language-guided navigation.
  • Domain-specific datasets: Domain-specific datasets target transportation, remote sensing, agriculture, industrial applications, emergency response, military operations, and wildlife.Transportation resources address traffic monitoring and detection, while remote-sensing datasets support detection, classification, localization, and image analysis.

5. Advances of FM-based UAV systems

The survey reviews how foundation models are integrated into UAV perception, navigation, planning, control, and infrastructure tasks. It emphasizes multimodal and language-based capabilities while noting limitations of relying on large models alone.

  • Motivation: Traditional UAV AI models often depend on task-specific datasets, limiting generalization and robustness when deployment conditions differ from training distributions.The survey presents this limitation as a motivation for foundation-model integration.
  • Motivation: LLMs, VFMs, and VLMs add natural-language understanding, zero-shot adaptation, multimodal collaboration, and human–machine interaction to UAV systems.These capabilities are described as new intelligent functions for UAV applications.
  • Task taxonomy: FM-based UAV tasks are categorized into visual perception, vision-language navigation, planning, flight control, and infrastructure.The categories span environmental understanding, mission execution, low-level control, and supporting frameworks or datasets.
  • Comparative overview: The survey compares representative methods in Table 10, while noting that some base-model identities and model details are unavailable in the cited references.The comparison is therefore partly constrained by the source descriptions.
  • Visual perception: FM-based methods support UAV object detection, segmentation, depth estimation, captioning, and VQA through multimodal features, zero-shot adaptation, and domain-specific fine-tuning.Examples combine models such as Grounding DINO, CLIP, GPT-4V, LLaVA, and Depth Anything for UAV-oriented perception.
  • Visual perception: Relying solely on VFMs or VLMs can limit object-detection performance because of hallucinations or insufficient task-specific adaptability.The survey describes large-model and small-model collaboration as a potential alternative.

5.2. VLN

UAV vision-language navigation combines visual understanding and language instruction with three-dimensional flight planning. Surveyed systems address indoor and outdoor navigation, localization, tracking, and target search through multimodal models and structured reasoning.

  • VLN foundations: UAV VLN requires path planning in three-dimensional space while accounting for flight altitude and visual observations.Large-scale multimodal pretraining improves task understanding in dynamic and complex environments.
  • Indoor VLN: Neuro-LIFT uses an LLM to assess human maneuver feasibility and translate language into high-level planner commands for indoor UAV navigation.The system links human instructions with UAV planning.
  • Indoor VLN: NaVid combines visual and geometric tokens with Vicuna-7B to generate navigation actions from monocular video without maps, odometry, or depth.Historical observations provide spatiotemporal context for real-time low-level navigation reasoning.
  • Outdoor VLN: AerialVLN formulates outdoor navigation as reaching target locations from natural-language instructions and first-person visual perception without preconstructed navigation maps.Its baseline extends conventional cross-modal alignment navigation methods.
  • Outdoor VLN: CityNav and related systems use language models and visual models to interpret landmarks, construct navigation maps, generate trajectories, and perform geolocalization.GOMAA-Geo additionally combines language, ground imagery, aerial imagery, and CLIP for multimodal active geolocalization.
  • VLT: Vision-language tracking continuously follows targets from multimodal inputs while adjusting flight paths under occlusion and environmental interference.The UAVNLT dataset supports UAV natural-language tracking with CLIP-based initial visual localization.
  • Target search: Target search combines vision-language navigation, object detection, and efficient path planning for multimodal perception and mission planning.NEUSIS uses neural-symbolic methods and foundation models for perception, localization, and 3D reasoning under uncertainty.

5.3. Planning

UAV mission planning must coordinate heterogeneous vehicles while respecting sensing, energy, and collision constraints in dynamic environments. LLM-based planners address this through task decomposition, language interaction, and multi-agent coordination.

  • Planning challenges: Traditional multi-UAV planning must account for each vehicle’s capabilities, limitations, sensing modes, energy consumption, and collision avoidance.These requirements complicate mission allocation and path planning in dynamic environments.
  • LLM-based planning: LLMs can decompose complex missions into executable subtasks and generate planning strategies through chain-of-thought, in-context learning, and few-shot learning.The survey presents these capabilities as mechanisms for adapting plans to diverse mission requirements.
  • LLM-based planning: Natural-language understanding and generation support real-time collaboration between operators and UAV mission-planning systems.Language instructions provide an interaction channel for planning and operational flexibility.
  • Applications: Representative systems use LLMs for heterogeneous-agent coordination, HAPS communication management, and natural-language mission-script generation.Examples include AutoHMA-LLM, ACMA, and TypeFly.

5.4. Flight Control

UAV flight control is progressing from conventional learning and rule-based approaches toward LLM-enabled strategies for adaptive planning, interaction, and autonomous control, including swarm operations.

  • Single-UAV control increasingly uses imitation and reinforcement learning, but these methods rely on large-scale annotated data and still face unresolved challenges.
  • Communication delays, computational complexity, and limited global optimization remain challenges for UAV flight-control approaches.
  • LLM-based flight control uses few-shot and in-context learning to adapt to task requirements and generate high-level strategies from task environments.
  • Natural-language interaction with LLM-based controllers supports mission planning, real-time decision-making, and adaptation to complex environments.
  • Swarm-GPT combines GPT-3.5 waypoint generation with model-based safety planning to satisfy physical constraints and collision avoidance requirements.

5.5. Infrastructures

UAV infrastructure research spans multimodal data construction, synthetic aerial imagery, benchmarking platforms, and task-specific frameworks that support UAV intelligence and deployment.

  • High-quality datasets and processing workflows are foundational for applying LLM, VLM, and VFM technologies to UAV tasks.
  • DTLLM-VLT generates multi-granularity text annotations from segmentation masks and visual descriptions for visual-language tracking.
  • GPG2A addresses the high cost of aerial-image acquisition by synthesizing aerial imagery from ground images through a two-stage Ground-to-Aerial framework.
  • AeroVerse integrates simulators, datasets, task definitions, and evaluation methods into a benchmark suite for UAV-agent perception, cognition, planning, and decision-making.
  • Related frameworks address UAV safety assessment, emergency communication optimization, air-combat simulation, and voice control for human-machine interaction.

6. Application scenarios of FMs-based UAVs

FM-enhanced UAV applications are organized around surveillance, logistics, and emergency response, targeting environmental perception, autonomous decision-making, and human interaction.

  • The reviewed application categories are surveillance, logistics, and emergency response, representing effective but non-exhaustive integration areas for UAVs and FMs.
  • Surveillance: In surveillance, VLMs and LLMs support traffic-sign monitoring, urban patrolling, identification, tracking, and agricultural crop analysis.
  • Logistics: Logistics applications cover decision-making, route planning, and delivery, while FMs are positioned to improve navigation, real-time analysis, planning, and human interaction.
  • Logistics: REAL uses LLM prior knowledge and zero-shot prompting to improve UAV adaptability, positional control, and real-time task decision-making.
  • Emergency response: Emergency-response UAVs support customized disaster relief, communication-network deployment, and mobile-access-point operations in disrupted or remote areas.
  • Emergency response: LLM-driven swarm deployment reduces iterations while maintaining network connectivity and quality of service through precise UAV positioning.

7. Agentic UAV: The General Pipeline Integrating FMs with UAV Systems

Agentic UAVs integrate foundation models with data, knowledge, tools, and agent workflows to support adaptive perception, planning, action, and reflection. The framework also faces practical limits involving computation, security, response latency, and infrastructure.

  • The Agentic UAV framework contains data, knowledge, tools, foundation-model, and agent modules for integrating FMs with UAV systems.
  • Agent module: The agent module combines perception, planning, action, feedback-based reflection, and multi-agent designs for UAV tasks and swarms.
  • Agentic UAVs continuously learn and adapt, using real-time environmental data to plan and execute tasks independently in response to unforeseen changes.
  • Data module: The data module converts UAV data into captions, question-answering, or chain-of-thought formats for FM training and fine-tuning.
  • FM module: Model selection and optimization distinguish language, vision-language, and 3D models according to task modality and UAV operating conditions.
  • FM module: Prompting, few-shot learning, instruction fine-tuning, LoRA, and layer freezing adapt foundation models to UAV-specific requirements.
  • Knowledge module: RAG supplies UAV systems with retrieved environmental and domain knowledge to improve real-time planning, navigation, and decision-making.
  • Challenges: Large model size and computational demands can restrict real-time UAV operations, while response delays threaten time-sensitive missions such as search and rescue.

8. Conclusion

The paper frames LLM–UAV integration as a path toward more capable UAV decision-making, perception, and reasoning. It reviews foundational components, research progress, datasets, applications, and a forward-looking Agentic UAV framework.

  • LLMs are positioned to enhance UAV decision-making, perception, and reasoning capabilities.
  • The paper establishes integration foundations by reviewing UAV system components and the underlying principles of large models.
  • It reviews UAV systems enhanced by foundational LLMs, including their classification, research progress, and application scenarios.
  • Key UAV-related datasets are highlighted as resources supporting the development of intelligent UAV systems.
  • Agentic UAVs are proposed as a forward-looking framework in which multi-agent systems integrate knowledge and tool modules for complex tasks in dynamic environments.
  • The paper identifies model pruning, edge computing, and coordinated air–land–sea unmanned systems as future development directions.
Loading 2501.02341v2…