Source-linked AI summary
UAV-VLA: Vision-Language-Action System for Large Scale Aerial Mission Generation
Oleg Sautenkov, Yasheerah Yaqoot, Artem Lykov, Muhammad Ahsan Mustafa, Grik Tadevosyan, Aibek Akhmetkazy, Miguel Altamirano Cabrera, Mikhail Martynov, Sausar Karaf, Dzmitry Tsetserukou
TL;DR
UAV-VLA addresses the difficulty of communicating complex aerial missions and the limited generalization of existing VLA/VLN approaches. It generates flight paths and actions from language instructions and satellite imagery, achieving human-comparable performance while producing trajectories 21.6% longer but 6.5 times faster than human-created ones.
Problem
Traditional UAV interaction relies on complex programming or manual controls, while many VLA and VLN approaches require paired instruction-behavior datasets and struggle with new environments and global-scale understanding.
Method
UAV-VLA generates path-action sets from text-based mission requests using satellite images and zero-shot model capabilities without additional training.
Results
21.6% longer trajectories were generated 6.5 times faster than human-created ones, with performance comparable to human-level path and action generation.
Takeaways & Limitations
Language-based path planning enables UAVs to autonomously execute mission plans from natural-language inputs and simplifies human-UAV interaction.
Abstract
from arXiv · showhide
The UAV-VLA (Visual-Language-Action) system is a tool designed to facilitate communication with aerial robots. By integrating satellite imagery processing with the Visual Language Model (VLM) and the powerful capabilities of GPT, UAV-VLA enables users to generate general flight paths-and-action plans through simple text requests. This system leverages the rich contextual information provided by satellite images, allowing for enhanced decision-making and mission planning. The combination of visual analysis by VLM and natural language processing by GPT can provide the user with the path-and-action set, making aerial operations more efficient and accessible. The newly developed method showed the difference in the length of the created trajectory in 22% and the mean error in finding the objects of interest on a map in 34.22 m by Euclidean distance in the K-Nearest Neighbors (KNN) approach.
I. INTRODUCTION
UAV-VLA addresses the need for more accessible human–UAV communication by generating path-action sets from text-based mission requests and satellite imagery. It introduces a global-scale benchmark and reports performance comparable to human-level path and action generation.
- Traditional UAV interaction often relies on complex programming, manual controls, or extensive piloting experience.
- Many VLA and VLN approaches require paired language–behavior datasets, generalize poorly to new environments, and lack global-scale understanding.
- UAV-VLA generates path-action sets from text-based mission requests using satellite images.
- UAV-VLPA-nano-30 provides a global-scale benchmark for evaluating VLA systems.
- The system was validated on UAV-VLPA-nano-30 with performance comparable to human-level path and action generation.
II. RELATED WORK
Related work develops multimodal models, aerial visual-language navigation systems, map representations, simulators, and robot action-generation models. These efforts establish components relevant to UAV-VLA but commonly depend on substantial training data or structured environments.
- Vision Transformers and multimodal models expanded processing across text, images, video, audio, and vision.
- Aerial VLN, VLDN, and related systems addressed aerial navigation, operator dialogue, and combined VLA-VLN tasks.
- STMR represented maps as matrices for language-model input, while OpenUAV provided realistic environments, flight simulation, and algorithmic support.
- RT-1 generated 11-dimensional robot actions from a large collected dataset, and RT-2 integrated RT-1 with a Visual-Language Model.
III. DATA AND BENCHMARK
UAV-VLPA-nano-30 is a standardized benchmark of real satellite imagery for evaluating language-conditioned aerial mission generation. It covers diverse U.S. environments and includes metadata supporting geographic flight-plan generation.
- UAV-VLPA-nano-30 contains 30 high-resolution satellite images from USGS EarthExplorer for evaluating linguistic instruction interpretation and navigation-plan generation.
- The benchmark spans urban, suburban, rural, and natural U.S. environments, including buildings, water bodies, transportation infrastructure, fields, and parking lots.
- The satellite imagery has approximately 1.5 meters per pixel resolution and covers roughly 760 sq. meters per image.
- Image metadata supports calculating identified points in latitude and longitude for flight-plan generation.
- Real-world satellite imagery provides authentic representation of environments and scenarios encountered in practical UAV applications.
B. Manual Flight Plan Generation
The benchmark uses manually created operator plans as a reference and describes UAV-VLA’s pipeline from natural-language instructions to geographic points and executable actions. The pipeline combines goal extraction, object search, coordinate transformation, and action generation.
- B. Manual Flight Plan Generation: An experienced drone operator created flight plans for all 30 images using the instruction to fly over buildings inside the violet square.
- B. Manual Flight Plan Generation: The operator completed all benchmark flight plans in 35 minutes, producing 63.89 km total and 2.13 km average trajectory length.
- IV. METHODOLOGY: The framework comprises goal-extracting GPT, object-search VLM, and actions-generation GPT modules.
- IV. METHODOLOGY: The goal-extracting GPT module parses the language instruction into task-specific goals.
- IV. METHODOLOGY: The object-search VLM identifies those goals in satellite imagery and outputs processed image points.
- IV. METHODOLOGY: The system transforms points into global latitude–longitude coordinates and uses mission details with MAVProxy to generate UAV actions.
V. EXPERIMENTS
The evaluation examines UAV-VLA performance through flight-plan creation and a novel effectiveness metric.
- The experiment evaluates the UAV-VLA system using the benchmark introduced earlier.
- The evaluation focuses on creating flight plans.
- A novel metric is used to assess system effectiveness.
A. Evaluation Metrics
The evaluation measures both path length and trajectory error, using RMSE with sequential, temporal, and spatial alignment methods.
- The evaluation metric considers total path length and error between system-generated and human-generated trajectories.
- Three error methods compare trajectories: Sequential alignment, Dynamic Time Warping, and K-Nearest Neighbors.Sequential alignment follows point order, DTW permits nonlinear alignment, and KNN matches points by spatial proximity.
- RMSE quantifies trajectory error using the system-generated and ground-truth points.
B. System Setup and Procedure
The system setup evaluates UAV-VLA flight plans against human-generated plans using a specified command and computing environment.
- The system receives a command to fly around each building at 100 m, return home, and land at take-off.
- The experiment runs on a PC with an RTX 4090 and Intel Core i9-13900K processor.
- Due to memory constraints, the experiment uses the quantized Molmo-7B-D BnB 4-bit model.
- UAV-VLA flight plans are compared with human-generated plans, with Fig. 4 showing an example comparison.
VI. EXPERIMENTAL RESULTS
UAV-VLA produces flight plans with trajectories longer than the human ground truth on average, while KNN provides the smallest reported trajectory error and the system generates plans faster than humans.
- 77.74 km is the general UAV-VLA trajectory length, 21.6% longer than the experienced pilot’s ground-truth trajectory.The difference is 13.85 km.
- 23% of cases produced UAV-VLA trajectories shorter than the ground-truth trajectory.This occurred in 7 out of 30 cases.
- 409.54 m is the largest mean error for sequential RMSE, while DTW reduces it to 307.27 m.
- KNN produces the smallest mean error by ignoring sequence and focusing on spatial proximity.
- 5 minutes 24 seconds is the UAV-VLA processing time, making it 6.5 times faster than human-generated flight plans.The time includes 2 minutes for point identification and 3 minutes 24 seconds for mission-file generation.
VII. CONCLUSION
The paper presents a global-scale UAV mission-generation approach and benchmark for converting natural-language requests into executable flight paths and actions. The method reports longer generated paths, faster planning than humans, and simplified language-based UAV interaction.
- The UAV-VLPA-nano-30 benchmark provides a standardized framework for evaluating global path-planning techniques.
- 21.6% longer paths and 6.5 times faster generation than human-created plans demonstrate the method’s reported efficiency.
- Language-based path planning enables UAVs to execute mission plans autonomously from natural-language inputs.
VIII. FUTURE WORK
Future work will improve satellite map-based path planning through specialized data and develop an end-to-end system for autonomous mission generation.
- A specialized dataset for satellite map-based path planning is planned to improve model precision and efficiency.
- An end-to-end model is planned to generate mission plans from high-level goals.
- The planned system will integrate action generation, path planning, and decision-making across diverse environments.