PaperlayerView exact revision on arXiv

Source-linked AI summary

Openfly: A comprehensive platform for aerial vision-language navigation

Yunpeng Gao, Chenhui Li, Zhongrui You, Junli Liu, Zhen Li, Pengan Chen, Qizhi Chen, Zhonghan Tang, Liansheng Wang, Penghui Yang, Yiwen Tang, Yuhang Tang, Shuai Liang, Songyi Zhu, Ziqin Xiong, Yifei Su, Xinyi Ye, Jianan Li, Yan Ding, Dong Wang, Xuelong Li, Zhigang Wang, Bin Zhao

arXiv:2502.18041v7cs.CVcs.RO

TL;DR

Aerial VLN remains underexplored, and existing aerial datasets are relatively small. OpenFly combines diverse rendering resources and automated data generation with a 100k-trajectory benchmark and a keyframe-aware agent; the authors report superior performance for the agent and improvements over uniform sampling.

  • Problem

    Aerial VLN remains underexplored, with existing methods constrained by limited scene diversity, labor-intensive collection, and datasets of about 10k trajectories.

    Source: §1 · p. 2

  • Method

    OpenFly integrates multiple rendering engines and an automated data-generation toolchain, builds a 100k-trajectory dataset across 18 scenes, and proposes a keyframe-aware VLN agent.

    Source: §1 · pp. 2–3

  • Results

    OpenFly-Agent outperforms comparison methods by a large margin, while its adaptive frame-level sampling improves performance and reduces computation versus uniform sampling.

    Source: §1 · p. 3, §6.3 · p. 8

  • Takeaways & Limitations

    OpenFly provides a platform for aerial VLN benchmarking and customized data collection across diverse scenes.

    Source: §1 · p. 3, §7 · p. 10

Abstract

from arXiv · show

Vision-Language Navigation (VLN) aims to guide agents by leveraging language instructions and visual cues, playing a pivotal role in embodied AI. Indoor VLN has been extensively studied, whereas outdoor aerial VLN remains underexplored. The potential reason is that outdoor aerial view encompasses vast areas, making data collection more challenging, which results in a lack of benchmarks. To address this problem, we propose OpenFly, a platform comprising various rendering engines, a versatile toolchain, and a large-scale benchmark for aerial VLN. Firstly, we integrate diverse rendering engines and advanced techniques for environment simulation, including Unreal Engine, GTA V, Google Earth, and 3D Gaussian Splatting (3D GS). Particularly, 3D GS supports real-to-sim rendering, further enhancing the realism of our environments. Secondly, we develop a highly automated toolchain for aerial VLN data collection, streamlining point cloud acquisition, scene semantic segmentation, flight trajectory creation, and instruction generation. Thirdly, based on the toolchain, we construct a large-scale aerial VLN dataset with 100k trajectories, covering diverse heights and lengths across 18 scenes. Moreover, we propose OpenFly-Agent, a keyframe-aware VLN model emphasizing key observations during flight. For benchmarking, extensive experiments and analyses are conducted, evaluating several recent VLN methods and showcasing the superiority of our OpenFly platform and agent. The toolchain, dataset, and codes will be open-sourced.

1 INTRODUCTION

Aerial VLN remains underexplored, with existing approaches constrained by limited data diversity, costly collection, and small datasets. OpenFly combines varied rendering resources and an automated toolchain with a large benchmark and a keyframe-aware agent.

1 INTRODUCTION

  • Existing aerial VLN methods face limited scene diversity, labor-intensive collection, and datasets of only about 10k trajectories.Prior approaches rely on AirSim and Unreal Engine assets, while pilots generate trajectories and instructions require manual annotation.Source: §1 · p. 2
  • OpenFly-Agent uses keyframe-aware sampling to emphasize instruction-related landmarks, improving performance and reducing computation versus uniform sampling.The authors report superior performance in simulated and real-world settings.Source: §1 · p. 3
  • OpenFly integrates multiple rendering engines and an automated toolchain to generate diverse aerial VLN data efficiently.Its resources include UE, GTA V, Google Earth, and 3D Gaussian Splatting, which supports real-to-sim scenes reconstructed from UAV images.Source: §1 · pp. 2–3
  • The benchmark contains 100k trajectories across 18 scenes, which the authors describe as the largest aerial VLN benchmark to date.Users can also collect customized data using the platform.Source: §1 · p. 3

2 RELATED WORKS

Prior VLN datasets largely concern ground-based navigation, while aerial datasets and methods have begun to address navigation in aerial settings. The related work distinguishes dataset designs from approaches to navigation.

2.1 VISION-LANGUAGE NAVIGATION DATASETS

  • Aerial datasets include dialogue-based ANDH, map-assisted CityNav, and simulator-based AerialVLN, OpenUAV, and CityNavAgent.Earlier datasets span graph-based navigation and continuous environments; aerial work includes bird-view images, point clouds, and UAV simulators.Source: §2.1 · p. 3

2.2 VISION-LANGUAGE NAVIGATION METHODS

  • Navigation methods range from predefined-graph and continuous-space approaches to LLM-driven reasoning and recent aerial methods.AerialVLN uses lookahead guidance, STMR uses matrix representations for spatial reasoning, and OpenUAV uses human feedback with ground-truth trajectories.Source: §2.2 · p. 4

3 AUTOMATIC DATA GENERATION

OpenFly combines multiple rendering resources with a toolchain for acquiring scene geometry, identifying landmarks, generating trajectories, and producing instructions. Its stages accommodate different scene types and automate much of data creation.

3.1 RENDERING ENGINES AND DATA RESOURCES

  • The platform combines UE urban scenes, a Los Angeles-like GTA V city, Google Earth regions, and 3D Gaussian Splatting reconstructions of real-world campuses.The resources provide varied environments and landmarks for aerial VLN.Source: §3.1 · p. 4

3 AUTOMATIC DATA GENERATION

  • OpenFly unifies rendering engines and interfaces for agent movement and sensor acquisition, then streamlines point-cloud, segmentation, trajectory, and instruction generation.The section introduces the resources and toolchain as the framework for automatic data generation.Source: §3.2 · p. 4, §3 · p. 4

3.2 TOOLCHAIN FOR AUTOMATIC DATA COLLECTION

  • Point clouds come from rasterized sampling in UE and GTA V or sparse COLMAP reconstruction in 3D GS scenes.Landmarks are identified through top-down scene understanding, projected point-cloud contours with GPT-4o captions, or manual annotation when point-cloud quality is low or finer segmentation is needed.Source: §3.2 · pp. 4–5
  • A* searches a global voxel map for collision-free trajectories between randomly selected landmarks using a customized action space.Trajectory points yield UAV-egocentric visual observations, and repeated endpoints can form complex trajectories.Source: §3.2 · p. 5
  • Instruction generation groups trajectory segments by action transitions, sends key actions and the final three images of each segment to a VLM, then combines sub-instructions with an LLM.The authors report a 91% qualification rate after manually inspecting 3K sampled items; all test data were manually inspected and low-quality items removed.Source: §3.2 · p. 5

4 DATASET ANALYSIS

The dataset is larger and more environmentally diverse than compared VLN datasets, with intentionally shorter average paths and instructions. Its trajectories vary in length, altitude, and action, and the data are divided into train, seen-test, and unseen-test splits.

4 DATASET ANALYSIS

  • OpenFly has more trajectories and greater environmental diversity than compared VLN datasets, while its average trajectory and instruction lengths are relatively short by design.The authors argue that short- and medium-range instructions better reflect natural usage and may benefit aerial VLN.Source: §4 · pp. 5–6

4.1 TRAJECTORY AND INSTRUCTION ANALYSIS

  • Trajectory lengths range from 0 to 300 meters and heights from 0 to 210 meters, with discrete actions and forward steps of 3, 6, or 9 meters.Figure 3 summarizes height, length, action, and instruction-word distributions; trajectories also include waypoint information and the platform supports continuous-waypoint generation and DAgger augmentation.Source: §4.1 · p. 6, Figure 3 · p. 6
  • The dataset’s instruction vocabulary is 15.6K, and instructions average 59 words; building, windows, and skyscraper are its most common references, while proceed and turn are the most-used VLN verbs.Figure 3d shows word clouds of nouns and verbs.Source: §4.1 · p. 6

4.2 DATASET SPLIT

  • Training data draw mainly from seven UE scenes (75.7%) and four 3D GS scenes (nearly 20%), while Google Earth contributes 4.46%.Test Seen contains 1,800 trajectories from 11 seen scenarios; Test Unseen contains 1,200 trajectories from three unseen scenes.Source: §4.2 · p. 6

5 OPENFLY-AGENT

OpenFly-Agent builds on OpenVLA to use observation histories and select keyframes containing action changes or instruction-related landmarks. It merges and compresses visual tokens to retain key observations while reducing computational burden.

5 OPENFLY-AGENT

  • OpenFly-Agent extends the OpenVLA baseline to take an image sequence as observation history and predict six UAV-specific action types.The action decoder discretizes predicted tokens into Forward, Turn Left, Turn Right, Move Up, Move Down, and Stop.Source: §5 · pp. 6–7

5.1 KEYFRAME SELECTION

  • Keyframe selection combines UAV movement change points with landmark grounding to retain frames likely to contain instruction-related landmarks.At testing, the model estimates landmark boxes and selects keyframes using box area and adjacent frames around significant motion changes.Source: §5.1 · p. 7

5.2 VISUAL TOKEN PRUNING

  • For token merging, the largest-box frame serves as reference; similar tokens in adjacent frames are averaged while corresponding comparison-frame tokens are discarded.The process is repeated across the keyframe set.Source: §5.2 · p. 7
  • A FIFO memory bank retains recent keyframes, whose historical tokens are compressed with grid pooling while current-frame tokens remain uncompressed.The current frame is left uncompressed to preserve the latest observation for action prediction.Source: §5.2 · p. 8

6 EXPERIMENTS

OpenFly-Agent achieves the strongest reported performance on the test set and in real-world experiments, while ablations examine keyframe selection and visual token merging. The evaluation also reports substantial degradation on unseen tasks across methods.

6.1 IMPLEMENTATION AND TRAINING DETAILS

  • OpenFly-Agent uses 256 tokens for the current frame and compresses historical keyframes into one token, with a memory-bank capacity of two images.The keyframe threshold is set to 0.25 times the input-image size.Source: §6.1 · p. 8

6.2 EVALUATION METRICS

  • Evaluation uses NE, SR, OSR, and SPL; SR requires stopping within 20 m of the target, and any collision counts as failure.OSR counts a trajectory as successful if any point comes within 20 m of the target.Source: §6.2 · p. 8

6.3 QUANTITATIVE RESULTS

  • OpenFly-Agent outperforms comparison methods by a large margin on the test set and is best on test-unseen, although all methods degrade significantly there.Table 2 reports test-set comparisons, with all models retrained using the OpenFly dataset.Source: §6.3 · p. 8, Table 2 · p. 9

6.5 ABLATION STUDY

  • SR rises from 16.6% to 34.3% with keyframe selection, and the KS versus KS + VTM comparison indicates a contribution from visual token merging.The ablations evaluate keyframe selection and visual token merging against the OpenVLA baseline.Source: §6.5 · p. 9

7 CONCLUSION

OpenFly integrates rendering engines and an automatic toolchain for aerial-VLN data collection. Its dataset contains 100k trajectories across 18 scenes, and experiments support OpenFly-Agent as a keyframe-aware model and establish a benchmark for future work.

7 CONCLUSION

  • OpenFly combines multiple rendering engines with an automatic toolchain to collect diverse aerial-VLN data.The platform is designed for large-scale data collection.Source: §7 · p. 10
  • The dataset comprises 100k trajectories across 18 scenes, spanning a wide range of altitudes and trajectory lengths.The conclusion describes it as significantly larger than existing aerial-VLN datasets.Source: §7 · p. 10
  • OpenFly-Agent identifies frames with critical observations for flight-action prediction, and experiments establish a comprehensive aerial-VLN benchmark.The paper presents the model as keyframe-aware.Source: §7 · p. 10

A MORE DETAILS OF RENDERING ENGINES AND DATA RESOURCES

OpenFly combines varied rendering resources with procedures for collecting 3D scenes and unified interfaces for agent movement and sensing. Its 3D Gaussian Splatting (3D GS) workflow balances reconstruction quality through altitude choices.

A MORE DETAILS OF RENDERING ENGINES AND DATA RESOURCES

  • Unreal Engine supplies eight urban scenes, while GTA V contributes a dynamic, realistic Los Angeles cityscape.UE scenes include varied architectural styles and assets; the UE4 time setting can vary lighting conditions.Source: A MORE DETAILS OF RENDERING ENGINES… · p. 15
  • Google Earth provides four urban regions across 53.60km2, and 3D GS reconstructs real scenes, including five university campuses.Google Earth Studio supports route-and-angle tour videos; the platform uses SIBR with an API to render 3D GS scenes from arbitrary poses.Source: Figure 7 · p. 15, A MORE DETAILS OF RENDERING ENGINES… · p. 16

B DETAILS OF 3D GS DATA COLLECTION

  • For 3D GS data collection, oblique photography at approximately twice average building height balances point-cloud initialization and Gaussian-splatting detail.Landmark buildings substantially taller or shorter than average receive additional targeted collection at twice their height.Source: B DETAILS OF 3D GS DATA COLLECTION · p. 16

C UNIFIED INTERFACES

  • Three unified interfaces standardize agent movement, lidar acquisition, and image acquisition across scenes and rendering engines.CoorTrans aligns coordinates to a meter-based FLU convention; lidar sources include sensors, depth-map back-projection, and image feature matching, while image requests support flexible resolutions and viewpoints.Source: C UNIFIED INTERFACES · p. 17

D RESULTS OF POINT CLOUD ACQUISITION AND SEMANTIC SEGMENTATION

OpenFly provides scene-specific point-cloud and segmentation methods, then uses maps, landmarks, and path search to generate trajectories. Its instruction-generation examples extract landmark features and fuse sub-instructions into fluent directions.

D RESULTS OF POINT CLOUD ACQUISITION AND SEMANTIC SEGMENTATION

  • UE and GTA V scenes use rasterized point-cloud sampling, whereas 3D GS scenes use COLMAP image-based reconstruction that provides sufficient coverage despite sparse points.The figure also shows rasterized and image-based reconstruction, semantic segmentation, and point-cloud projection with contour extraction.Source: D RESULTS OF POINT CLOUD ACQUISITION… · p. 17
  • Scene segmentation can use granular Octree-Graph results or point-cloud projection and contour extraction for ground-contacting building and tree instances.OpenFly describes three segmentation methods overall, while these results present methods other than manual annotation.Source: D RESULTS OF POINT CLOUD ACQUISITION… · p. 17

E A COMPREHENSIVE INTRODUCTION TO AUTOMATIC TRAJECTORY GENERATION

  • OpenFly generates collision-free trajectories with A* from point-cloud maps, selecting landmarks, starting points, and nearby endpoints under a customized action space.For Google Map data, it instead rasterizes an area and selects images from grid points and orientations because arbitrary-pose image retrieval is unavailable.Source: E A COMPREHENSIVE INTRODUCTION TO… · p. 18

F DETAILS OF INSTRUCTION GENERATION

  • Instruction generation extracts target-landmark features from selected images, then fuses sub-instructions into fluent directions using pronouns for repeated or similar landmarks.The examples prompt a VLM to characterize a target building by color, feature, size, and type, and to combine actions and landmarks into a sentence.Source: F DETAILS OF INSTRUCTION GENERATION · p. 18

G DATA QUALITY CONTROL

OpenFly filters damaged images and implausible or extreme trajectories, refines repeated-landmark descriptions, and manually checks a sample of generated data. The sampled instructions were mostly judged suitable, though some landmark ambiguity remained.

G DATA QUALITY CONTROL

  • Trajectories with fewer than 2 or more than 150 actions are removed, along with damaged images and flights below tree height.The authors exclude extremely short or long trajectories because they consider them unhelpful for model training.Source: G DATA QUALITY CONTROL · p. 18
  • To address repeated-landmark captions that can add complexity and ambiguity, refinement merges similar descriptions and replaces them with referential pronouns.The process parses landmark captions, embeds them with a sentence-transformer, and identifies similarity using dot products.Source: G DATA QUALITY CONTROL · pp. 18–19
  • 91% of 3K randomly sampled instructions were judged qualified, while some remaining landmark descriptions were ambiguous but not wholly unacceptable to examiners.The authors conclude that most generated data are good quality for aerial VLN.Source: G DATA QUALITY CONTROL · p. 19

H MORE DATASET ANALYSES

The dataset’s training data are concentrated in UE scenes, while the test split includes both seen and unseen data.

H MORE DATASET ANALYSES

  • 60% of test data are seen and 40% unseen; in training, 7 UE scenes account for 75.7% of the 100K data, 4 3D GS scenes nearly 20%, and Google Earth 4.46%.Figure 9 shows scene-level data-volume distributions for the Train and Test sets.Source: H MORE DATASET ANALYSES · p. 19, Figure 9 · p. 19

I QUALITATIVE EXPERIMENTAL RESULTS

OpenFly-Agent successfully follows instructions in both UE and 3D GS examples, including scenes with substantially different image styles, flight heights, and viewpoints. The visualizations also show failure cases and illustrate how VTM affects attention to the current observation.

Loading 2502.18041v7…