PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology

Xiangyu Wang, Donglin Yang, Ziqin Wang, Hohin Kwan, Jinyu Chen, Wenjun Wu, Hongsheng Li, Yue Liao, Si Liu

arXiv:2410.07087v2cs.CVcs.RO

TL;DR

UAV vision-language navigation is less explored than ground-based VLN, and existing UAV benchmarks often use discrete actions that miss the differences in flight dynamics and aerial task complexity. The paper introduces a realistic platform, dataset, assistant-guided benchmark, and hierarchical navigation LLM; its method outperforms baselines, though human operators remain substantially ahead.

  • Problem

    UAV VLN is comparatively underexplored, and existing benchmarks often rely on discrete ground-based actions that do not reflect continuous UAV dynamics or complex aerial navigation.

    Source: §1 · pp. 1–2

  • Method

    The paper builds OpenUAV and a realistic 6-DoF trajectory dataset, introduces assistant-guided UAV-Need-Help, and proposes an LLM for hierarchical trajectory generation.

    Source: §1 · pp. 2–3, §6 · p. 7, §6.1 · p. 8

  • Results

    On the test seen set, the method outperforms across metrics at different difficulty levels, with SR improving by an average of 5% over CMA across assistant levels.

    Source: §7.2 · p. 9

  • Takeaways & Limitations

    The platform, dataset, benchmark, and methodology establish a unified framework for realistic UAV VLN research.

    Source: §8 · p. 11

Abstract

from arXiv · show

Developing agents capable of navigating to a target location based on language instructions and visual information, known as vision-language navigation (VLN), has attracted widespread interest. Most research has focused on ground-based agents, while UAV-based VLN remains relatively underexplored. Recent efforts in UAV vision-language navigation predominantly adopt ground-based VLN settings, relying on predefined discrete action spaces and neglecting the inherent disparities in agent movement dynamics and the complexity of navigation tasks between ground and aerial environments. To address these disparities and challenges, we propose solutions from three perspectives: platform, benchmark, and methodology. To enable realistic UAV trajectory simulation in VLN tasks, we propose the OpenUAV platform, which features diverse environments, realistic flight control, and extensive algorithmic support. We further construct a target-oriented VLN dataset consisting of approximately 12k trajectories on this platform, serving as the first dataset specifically designed for realistic UAV VLN tasks. To tackle the challenges posed by complex aerial environments, we propose an assistant-guided UAV object search benchmark called UAV-Need-Help, which provides varying levels of guidance information to help UAVs better accomplish realistic VLN tasks. We also propose a UAV navigation LLM that, given multi-view images, task descriptions, and assistant instructions, leverages the multimodal understanding capabilities of the MLLM to jointly process visual and textual information, and performs hierarchical trajectory generation. The evaluation results of our method significantly outperform the baseline models, while there remains a considerable gap between our results and those achieved by human operators, underscoring the challenge presented by the UAV-Need-Help task.

1 INTRODUCTION

UAV VLN remains less explored than ground-based VLN, while existing UAV benchmarks often inherit discrete ground-navigation actions that do not represent continuous flight or complex aerial search. The paper addresses these gaps with OpenUAV, a realistic trajectory dataset, an assistant-guided benchmark, and a hierarchical navigation LLM.

1 INTRODUCTION

  • Ground-based discrete actions poorly capture UAVs’ continuous 3D flight, while complex aerial settings and shifting views make target descriptions alone insufficient for precise localization.The paper identifies movement dynamics and navigation-task complexity as the two main mismatches.Source: §1 · pp. 1–2
  • OpenUAV combines realistic environments, AirSim flight simulation, continuous trajectory control, and algorithmic support for UAV VLN.It integrates 22 scenarios and 89 objects, with APIs for object and scenario configuration.Source: §1 · p. 2
  • The target-oriented dataset is the first realistic UAV VLN dataset with 6-DoF motion and contains approximately 12k trajectory-instruction pairs.Human annotators recorded continuous flights; GPT-4 generated target descriptions followed by manual quality checks.Source: §1 · pp. 2–3
  • UAV-Need-Help defines assistant-guided object search, using target descriptions, environmental information, and assistant guidance at varying levels.The assistant supplies action guidance in specific scenarios.Source: §1 · p. 3
  • The proposed UAV navigation LLM uses multimodal understanding for hierarchical trajectory generation, alongside backtracking sampling-based data aggregation.The paper reports closed-loop evaluations across various settings as validation of the approach.Source: §1 · p. 3
  • The authors aim for the platform, benchmark, and methodology to promote research on continuous realistic UAV trajectories and support transfer to real-world applications.Source: §1 · p. 3

2 RELATED WORK

Prior UAV simulators and VLN datasets provide foundations for aerial navigation, but the reviewed work leaves limitations in realism, continuous trajectory support, and action-space fidelity. Related LLM navigation research spans ground robots, autonomous driving, and UAV code or coordinate generation.

Embodied Simulator

  • Existing UAV platforms are described as limited in realism, scalability, or scenario diversity, and some UE–AirSim systems lack continuous-trajectory VLN support.The review contrasts these gaps with OpenUAV’s realistic environments, flight simulation, and algorithmic support.Source: §2 · p. 3

Vision-Language Navigation Datasets

  • Prior UAV VLN datasets include aerial dialogue or discrete 4-DOF trajectories, whereas this work collects continuous 6-DOF UAV trajectories.The review says earlier trajectories were collected by modifying UAV position, which deviates from real UAV flights.Source: §2 · p. 4

LLM-based Navigation Agent

  • Related LLM navigation methods cover goal-conditioned ground robots, language-guided autonomous driving, UAV navigation-code generation, and drone target-coordinate optimization.The paper positions its own contribution as hierarchical trajectory generation for UAV object search.Source: §2 · p. 4

3 OPENUAV SIMULATION PLATFORM

OpenUAV integrates environment composition, realistic flight simulation, and algorithmic support for realistic UAV VLN. Its scenarios, sensors, and continuous control support data collection and closed-loop navigation experiments.

3 OPENUAV SIMULATION PLATFORM

  • OpenUAV is an open-source platform for realistic UAV VLN with environment composition, flight simulation, and algorithmic-support modules.Source: §3 · p. 4

3.1 ENVIRONMENT COMPOSITION

  • UE4-rendered environments include 22 urban, rural, and natural scenarios, dynamic effects, and configurable placement of diverse object assets.Objects can be placed through the scenario editor or runtime API.Source: §3.1 · p. 4

3.2 FLIGHT SIMULATION

  • AirSim enables physics-based 6-DoF flight control, while configurable payloads include multi-view RGB and depth cameras and 360-degree LiDAR.The UAV pose records position and pitch, roll, and yaw angles; users can adjust sensor configurations.Source: §3.2 · p. 4

3.3 ALGORITHMIC SUPPORT

  • Algorithmic support combines asynchronous sensor collection, closed-loop feedback, DAgger, collision-recovery backtracking, and parallel simulation.With 8 NVIDIA A100 GPUs, single-UAV simulation reaches 16× speedup and 160–1600 fps depending on payload data captured.Source: §3.3 · p. 5

4 TARGET-ORIENTED REALISTIC UAV VLN DATASET

The paper constructs a target-oriented realistic UAV VLN dataset on OpenUAV, described as the first to capture complex UAV flight dynamics. Its collection and analysis span 12,149 trajectories, multiple difficulty levels, target categories, and seen and unseen test splits.

4 TARGET-ORIENTED REALISTIC UAV VLN DATASET

  • The dataset is presented as the first realistic UAV VLN dataset to accurately capture complex flight dynamics.It is constructed using the OpenUAV platform.Source: §4 · p. 5

4.1 DATA COLLECTION

  • Human experts manually flew trajectories, recorded UAV states at fixed intervals, and collected sensor data afterward; collision trajectories were discarded, yielding 12,149 valid trajectories.The trajectory recording ended within 5 meters of the target.Source: §4.1 · p. 6

4.2 DATA ANALYSIS

  • Trajectories shorter than 250 meters are classified as easy and those exceeding 250 meters as hard, while target distances span 50 to 400 meters.The paper describes this range as representing the environment’s spatial scale.Source: §4.2 · p. 6
  • The dataset covers 89 target-object categories, including vehicles, humans, animals, and other objects; common descriptions include buildings, trees, and cars.The paper says descriptions provide contextual information for estimating object locations through visual cues.Source: §4.2 · p. 6
  • The four splits are Train (9,152 trajectories), Test Seen (1,410), Test Unseen Map (958), and Test Unseen Object (629).The unseen-map test uses two unseen scenes, and the unseen-object test uses 13 unseen objects.Source: §4.2 · p. 6

5 UAV-NEED-HELP BENCHMARK

UAV-Need-Help formulates assistant-guided UAV object search using target descriptions, environment information, and guidance. It distinguishes three assistant tiers by how often and under what conditions they intervene.

5.1 TASK FORMULATION

  • UAV-Need-Help asks a UAV to navigate to a target object using its description, environment information, and assistant guidance.The task formulation supplies the initial pose and target description, then updates the UAV with state and multi-view RGB and depth observations.Source: §5.1 · p. 6
  • In closed-loop simulation, the UAV predicts 6-DoF trajectories and succeeds by landing within a 20-meter radius of the target.The platform’s flight API executes predicted positions while updating observations and adhering to flight dynamics.Source: §5.1 · p. 7

5.2 ASSISTANT MECHANISM

  • The L1 assistant continuously guides the UAV along the oracle path, L2 intervenes for collision risk or excessive deviation, and L3 provides obstacle avoidance in hazardous scenarios.The three settings represent differing levels of guidance, as depicted in Figure 3(a).Source: §5.2 · p. 7, Figure 3 · p. 7

6 UAV NAVIGATION LLM

The UAV Navigation LLM processes multimodal inputs and generates trajectories hierarchically, from a target pose to fine-grained paths. The method also uses backtracking sampling during data aggregation to obtain more successful obstacle-avoidance trajectories.

6 UAV NAVIGATION LLM

  • The UAV Navigation LLM combines image and text inputs with Vicuna-7B and uses a hierarchical trajectory decoder to produce target poses and refined pose sequences.The method also introduces backtracking sampling-based data aggregation to enhance obstacle avoidance in complex scenarios.Source: §6 · p. 7

6.1 HIERARCHICAL TRAJECTORY GENERATION

  • Multimodal tokenization combines visual tokens from multi-view images with task-description and assistant-instruction tokens.Each image yields one global context token and 16 local content tokens.Source: §6.1 · p. 8
  • The high-level MLLM decoder uses a learnable trajectory token to extract trajectory-specific features and decode the UAV’s next target pose.It plans using environmental data and detailed language instructions.Source: §6.1 · p. 8
  • The fine-grained path decoder combines front-view visual tokens with processed pose features and passes the combined features through an MLP to generate trajectories.Grounding DINO detects the target in multi-view images to initiate landing.Source: §6.1 · p. 8

6.2 BACKTRACKING SAMPLING-BASED DATA AGGREGATION

  • When a model-predicted action causes a collision during DAgger collection, backtracking restores the UAV to its state from two frames earlier before it follows the teacher trajectory.The mechanism is intended to produce more successful obstacle-avoidance trajectories.Source: §6.2 · p. 8

7 EXPERIMENTS

The experiments compare the proposed method with baselines across assistant levels and test sets, and include a human-operated comparison and qualitative examples. The reported results cover overall performance, generalization, data scalability, and realistic flight challenges.

7.1 EXPERIMENTAL SETUP

  • The evaluation uses SR, OSR, SPL, and NE, and compares against random, fixed-action, and CMA baselines.The baselines represent unstructured pose selection, assistant instructions mapped to predefined actions, and a modified grounded-VLN model.Source: §7.1 · pp. 8–9, Table 2 · p. 9

7.2 QUANTITATIVE RESULT

  • The method outperforms baselines across metrics and difficulty levels on the test-seen set, with average SR 5% higher than CMA.Backtracking sampling-based data aggregation also improves SR over the original model.Source: §7.2 · p. 9
  • Under L1 guidance, human operators achieve high success rates but sometimes take longer paths than the UAV method.The authors suggest humans may choose more cautious or exploratory routes.Source: §7.2 · p. 10
  • As assistant guidance decreases, the proposed method and CMA show declines in success rate and other metrics, underscoring the challenge of long-term UAV VLN.The proposed method achieves the highest success rate under continuous L1 guidance.Source: §7.2 · p. 10
  • On unseen scenes, the method’s performance declines but remains higher than other methods; on unseen objects, its success rate slightly exceeds the seen-set result.The authors attribute the unseen-object outcome to Grounding DINO generalization and relatively simpler data.Source: §7.2 · p. 10
  • Performance improves as the amount of training data increases in the L1-assistant setting.The authors identify larger, more diverse datasets as a potential way to improve model understanding and decision-making.Source: §7.2 · p. 10

7.3 QUALITATIVE RESULT

  • Qualitative examples show successful instruction-following among buildings and a forest collision caused by insufficient altitude.Attitude changes shift the camera view during flight.Source: §7.3 · p. 10, Figure 4 · p. 10

8 CONCLUSION

The conclusion presents platform, benchmark, and methodology contributions for realistic UAV VLN. It identifies stronger autonomous navigation with minimal guidance and better simulation-to-reality transfer as future directions.

8 CONCLUSION

  • The paper combines OpenUAV, a target-oriented realistic UAV VLN dataset, UAV-Need-Help, and a UAV navigation LLM with backtracking-based data augmentation.The authors describe these contributions as a unified framework for realistic UAV VLN research.Source: §8 · p. 11
  • The authors identify autonomous navigation with minimal guidance and improved transfer from simulation to real-world deployment as future research directions.These directions target complex environments and real-world UAV applications, respectively.Source: §8 · p. 11
  • The paper reports that its platform, benchmark, LLM, and augmentation strategy enhance performance on realistic trajectory-based VLN tasks.Source: §8 · p. 11

A PLATFORM SCENARIOS

The appendix describes OpenUAV’s scenario collection and the multi-view prompting used to generate object descriptions and environmental information.

A PLATFORM SCENARIOS

B PROMPT ENGINEERING IN DESCRIPTION COLLECTION

  • The description-collection prompt combines multiple camera views and generates environmental colors, shapes, and spatial relations to support target identification and localization.The views include top, front, rear, left, and right perspectives.Source: B PROMPT ENGINEERING IN DESCRIPTION… · p. 14

C EXPERIMENTAL DETAILS

The experimental details specify model training choices, instruction construction, and component ablations. The ablation results identify LoRA as the most consequential tested component.

C EXPERIMENTAL DETAILS

  • Training updates only the visual projector, trajectory prediction head, and LoRA layers, which comprise 4% of model parameters; most MLLM parameters remain frozen.Pose angles use cosine-similarity loss, while predicted waypoints and completed trajectories use L1 loss.Source: C EXPERIMENTAL DETAILS · p. 14
  • The instruction-generation process derives target direction from UAV start and end coordinates and uses GPT-4 to generate object descriptions and environmental information from multi-view inputs.Source: Figure 6 · p. 15
  • The MLLM is trained for two epochs on eight NVIDIA A100 GPUs with batch size 128; fine-grained and CMA models train for ten epochs with batch size 128.All models use Adam and a one-cycle learning-rate schedule with maximum learning rate 5e-4.Source: C EXPERIMENTAL DETAILS · p. 15
  • Removing LoRA decreases OSR by 8% and SR by 4%, while the complete model performs best across all metrics.Removing cosine loss or the learnable query also reduces performance, but less substantially.Source: C EXPERIMENTAL DETAILS · p. 15, Table 5 · p. 15

D ADDITIONAL QUALITATIVE RESULTS

The section provides additional visualizations of the method in closed-loop evaluation. Examples of UAV object search illustrate longer trajectories and improved navigation across various scenarios.

D ADDITIONAL QUALITATIVE RESULTS

Loading 2410.07087v2…