Source-linked AI summary
AerialVLN: Vision-and-Language Navigation for UAVs
Shubo Liu, Hongsheng Zhang, Yuankai Qi, Peng Wang, Yaning Zhang, Qi Wu
TL;DR
Ground-based VLN does not address UAV navigation in the sky, where flying height and complex spatial relationships matter. AerialVLN introduces a city-level UAV task, simulator, dataset, and CMA-based baselines; baseline performance remains far below human performance, underscoring the task’s challenge.
Problem
Existing VLN tasks are designed for ground-based agents, leaving UAV navigation in the sky with its greater height and spatial-reasoning requirements underrepresented.
Method
AerialVLN combines a UAV-based city-level navigation task, a near-realistic simulator spanning 25 city-level environments, a pilot-generated dataset, and extended CMA-based baselines.
Results
Baseline performance falls far behind human performance, with golden Seq2Seq and CMA models achieving only 1.0%–1.6% success rates on unseen full-dataset splits versus approximately 80% for humans.
Takeaways & Limitations
AerialVLN provides a challenging research setting for studying complex language understanding, visual-textual alignment, and navigation in the sky.
Takeaways & Limitations
Success rates are strongly affected by path length and stopping errors, with long paths reaching only 1.8% success versus 7.4% for short paths.
Abstract
from arXiv · showhide
Recently emerged Vision-and-Language Navigation (VLN) tasks have drawn significant attention in both computer vision and natural language processing communities. Existing VLN tasks are built for agents that navigate on the ground, either indoors or outdoors. However, many tasks require intelligent agents to carry out in the sky, such as UAV-based goods delivery, traffic/security patrol, and scenery tour, to name a few. Navigating in the sky is more complicated than on the ground because agents need to consider the flying height and more complex spatial relationship reasoning. To fill this gap and facilitate research in this field, we propose a new task named AerialVLN, which is UAV-based and towards outdoor environments. We develop a 3D simulator rendered by near-realistic pictures of 25 city-level scenarios. Our simulator supports continuous navigation, environment extension and configuration. We also proposed an extended baseline model based on the widely-used cross-modal-alignment (CMA) navigation methods. We find that there is still a significant gap between the baseline model and human performance, which suggests AerialVLN is a new challenging task. Dataset and code is available at https://github.com/AirVLN/AirVLN.
1. Introduction
Existing VLN tasks focus on ground navigation, leaving UAV navigation in the sky underrepresented. AerialVLN addresses this gap with a challenging city-level task featuring richer spatial reasoning, larger action spaces, complex environments, and a substantial dataset.
- Existing VLN tasks support vision-language understanding, cross-modal matching, path planning, and reasoning, but restrict agents to indoor or ground-based outdoor navigation.
- AerialVLN adds aerial-specific actions, including rising and panning down, while allowing lateral movement without changing heading.
- Its city-level outdoor environments require bird’s-eye spatial reasoning and support interactive changes such as landing, weather, and illumination variation.
- Instructions contain about 9.7 referred objects on average, over 2.6 times the amount in R2R, while agents must avoid obstacles in 3D space.
- The dataset includes 25 city-level environments, more than 870 object types, 8,446 pilot-generated flying paths, three instructions per path, and fine-grained sub-path alignment.
2. Related Work
Prior UAV language-navigation studies and ground-based VLN tasks do not capture the scale, openness, and aerial challenges targeted by AerialVLN. The proposed dataset is positioned as a larger city-level alternative for UAV navigation.
- Dataset Comparison: The comparison table characterizes AerialVLN as a city-level open-environment dataset for aerial, instruction-based navigation, with path lengths measured in meters.
- UAV Navigation: Earlier UAV language-navigation studies used a single closed virtual field with horizontal-only movement, whereas AerialVLN targets broader aerial navigation.
- UAV Navigation: AerialVLN provides 25k instructions averaging 83 words, a 4,470-word vocabulary, and 870 object types, exceeding the cited earlier UAV dataset’s scale.
- Ground-based VLN Tasks: Ground-based VLN tasks address indoor or terrestrial navigation, so they do not reflect multirotor-specific challenges in the sky.
3. The AerialVLN Task
AerialVLN asks a virtual multirotor to reach a destination by following natural-language instructions and front-view visual observations in a continuous, open environment. Success requires stopping within 20 units of the target.
- The agent navigates toward a destination using natural-language instructions and first-person visual perceptions from the simulator.
- Unlike prior VLN tasks, AerialVLN provides no pre-built navigation graph, so every unoccupied point is navigable.
- Each episode begins with an agent pose containing position and orientation, while the instruction is represented as a sequence of word tokens.
- The baseline agent receives front-view RGB and depth observations and must rotate to obtain other views.
- Navigation ends after a Stop action or a maximum action limit, and success requires stopping less than 20 units from the target.
4. Simulator
The AerialVLN simulator provides continuous movement and front-view sensing in a dynamic outdoor environment. It combines RGB and depth observations with changing weather, illumination, traffic, and other environmental effects.
- The simulator outputs front-view RGB and depth images while allowing an embodied agent to move freely through a continuous outdoor environment.
- Its depth sensor perceives up to 100 meters ahead, and semantic segmentation is available for future use.
- Dynamic conditions include blowing leaves, running cars, changing illumination, and sun, rain, snow, or fog.
5. Dataset
AerialVLN combines human-generated UAV flight paths with crowdsourced language instructions, then analyzes their linguistic and navigation complexity across full and small-scene settings.
- Data collection: AOPA-licensed multirotor pilots generate flight paths, which are smoothed and discretized into meta actions for training.The resulting trajectories preserve human remote-pilot behavior while enabling model training.
- Data collection: AMT annotators describe drone-flight videos three times each, and another worker group manually checks the collected instructions.Referring expressions are required to reduce ambiguity caused by similar landmarks.
- Dataset analysis: AerialVLN combines long paths and many actions with substantial reference, sequencing, spatial-relationship, and direction phenomena, increasing navigation difficulty.Most paths contain 50–240 actions, while instructions range from 50–130 words.
- Dataset splits: The dataset is split across training, seen-validation, unseen-validation, and test scenes, with test goals withheld and trajectories scored through an evaluation server.The test split uses unseen scenes and does not release goal locations.
- Dataset variants: AerialVLN-S provides a smaller-scene variant, reducing average path length by 51.5% while retaining the same split structure.The variant is intended as a benchmark for general first-person-view 3D aerial VLN, while full AerialVLN targets long-horizon unseen-environment navigation.
6. Experiment and Results
The experiments compare established and adapted navigation baselines on AerialVLN, showing very low success rates relative to humans and identifying training, path-length, stopping, and modality effects.
- Evaluation Metrics: The evaluation uses Success Rate, Oracle Success Rate, Navigation Error, and success weighted by Normalised Dynamic Time Warping.Success requires stopping within 20 meters, while oracle success requires passing within 20 meters at any trajectory point.
- Baselines: The compared systems include Random, Action Sampling, LingUNet, Seq2Seq, CMA, and the CMA-based LAG extension.LAG combines CMA with look-ahead guidance for generating training actions after deviations from the reference path.
- Results: 0% success for Random and below 1% for Action Sampling show that uninformed action selection rarely reaches the destination.The oracle success rate remains below 3% on unseen splits for these settings.
- Results: 1.0%–1.6% unseen success for Seq2Seq and CMA on the full dataset remains far below human performance of approximately 80%.On AerialVLN-S, these golden baselines reach 2.2%–3.9% success, while oracle success rises to 5% and 16%, respectively.
- Results: Dataset Aggregation improves performance by about 6% on seen splits and about 1% on unseen splits.The reported comparison links this improvement to training with actions sampled from model predictions rather than only ground-truth actions.
- Analysis: Aligning visual and textual landmarks and understanding rotation commands are associated with successful navigation, while removing RGB or depth reduces success.The ablation study reports that both vision and language inputs play important roles, with RGB removal causing the larger vision-side drop.
- Failure Analysis: Success falls from 7.4% on short paths to 1.8% on long paths, and agents often pass the goal without stopping correctly.The reported average path lengths are 326.9m for short paths and 813.2m for long paths.
7. Conclusion
The paper introduces AerialVLN as a new UAV-based vision-and-language navigation task and dataset for navigation in the sky. Widely adopted baselines perform significantly worse on this task than humans, leaving substantial room for further research.
- AerialVLN introduces a new task and large-scale dataset for exploring vision-and-language navigation in the sky.
- The dataset presents challenges in complex language understanding and visual-textual alignment.
- Baseline performance drops significantly on AerialVLN and falls far behind human performance.