PaperlayerView exact revision on arXiv

Source-linked AI summary

AerialDojo-200K: A Large-Scale Benchmark Suite for Open-World Aerial Object-Goal Search

Tongtong Feng, Xin Wang, Haoran Hou, Ren Wang, Weiran Wang, Shaokai Zhu, Ziqi Jia, Hao Wang, Yu-Wei Zhan, Zongyuan Wu, Jinghao Cui, Wenwu Zhu

arXiv:2609.36066v1cs.CVcs.AIcs.ETcs.MMcs.RO

TL;DR

Open-world aerial object-goal search asks agents to explore unstructured 3D environments and find objects from semantic or image goals, but existing benchmarks are small and heterogeneous. AerialDojo-200K provides a larger benchmark and unified evaluation framework; current MLLMs remain far from reliable generalist aerial agents under its evaluation.

  • Problem

    Small, environment-specific benchmarks and heterogeneous action spaces and formats limit large-scale training and cross-benchmark evaluation for open-world aerial object-goal search.

    Source: §1 · pp. 1–2

  • Method

    AerialDojo-200K constructs a large-scale aerial object-goal search benchmark with annotated scenes and tasks, reference trajectories, and a unified evaluation framework.

    Source: §1 · pp. 2–3

  • Results

    Current MLLMs remain far from reliable generalist aerial agents; the best reported SR is 3.20% on Base tasks at the primary 3 m threshold.

    Source: §5 · p. 10

  • Takeaways & Limitations

    AerialDojo-200K provides a common testbed for developing and evaluating methods that integrate exploration, instance-level target grounding, precise stopping, and collision avoidance.

    Source: §5 · p. 10

    Aggregate metrics cannot identify the individual causes of failures to convert target-region visits into valid completion.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Open-world aerial object-goal search is a foundational yet challenging task, requiring aerial agents to autonomously explore large-scale, unstructured three-dimensional environments and reach target objects specified by semantic descriptions or reference images, rather than following route-specific instructions. However, research in this task remains at a nascent stage and relies on small, environment-specific benchmarks with heterogeneous action spaces and data formats. These limitations hinder large-scale training and cross-benchmark evaluation, constraining the scalability and generalizability of aerial agents. To address this problem, we propose AerialDojo-200K, a large-scale benchmark suite for open-world aerial object-goal search, with 3 times as many scenes and 18.7 times as many task instances as the largest existing benchmark for this task. Specifically, we construct 42 simulation scenes spanning four scene families and 21 scene types, including 18 urban, 12 natural, six infrastructure, and six disaster scenes. To ensure data quality, 12 annotators spent two months manually annotating 109 landmarks, 2099 target objects, and 2099 object anchors across these scenes. We further construct 205,732 task instances, comprising over 100K semantic-goal and over 100K image-goal instances across Base, Standard, and Long-Horizon settings. Each task instance includes a collision-free reference trajectory and corresponding multi-view video recordings. We also develop a unified evaluation framework with a scene partition comprising 21 in-distribution scenes and 21 out-of-distribution scenes. Finally, our evaluation of five open-source and four closed-source multimodal large language models reveals that there is still a long way to go toward achieving general-purpose aerial agents. All can be found at https://fengtt42.github.io/AerialDojo/.

1 INTRODUCTION

Open-world aerial object-goal search requires autonomous exploration and target identification, but small, environment-specific benchmarks and heterogeneous formats impede large-scale training and cross-benchmark evaluation. AerialDojo-200K expands benchmark scale and provides a unified evaluation framework, while evaluations of multimodal models still indicate a gap to general-purpose aerial agents.

1 INTRODUCTION

  • AerialOGS requires agents to explore large, unstructured 3D environments and decide when an observed object matches a semantic description or reference image.Unlike route-instruction navigation, it specifies a high-level goal without a detailed travel route.Source: §1 · p. 1
  • Existing small, environment-specific benchmarks and heterogeneous action spaces and data formats hinder large-scale training and cross-benchmark evaluation.These constraints limit aerial agents’ scalability and generalizability.Source: §1 · p. 2
  • 205,732 task instances and 42 simulation scenes make AerialDojo-200K 3× larger in scenes and 18.7× larger in tasks than the largest existing AerialOGS benchmark.The scenes span four families and 21 types; tasks cover semantic and image goals across three settings, with collision-free reference trajectories and multi-view videos.Source: §1 · p. 2
  • 12 annotators spent two months documenting 109 landmarks, 2,099 target objects, and 2,099 object anchors across the benchmark.Each task instance also includes a collision-free reference trajectory and corresponding multi-view video recordings.Source: §1 · p. 3
  • Evaluations of five open-source and four closed-source MLLMs reveal that substantial progress remains before general-purpose aerial agents are achieved.The benchmark provides a unified evaluation framework for this task.Source: §1 · p. 3

2 RELATED WORKS

Related work covers aerial object-goal search methods and aerial navigation benchmarks. It describes approaches combining spatial representations with semantic reasoning, while noting that limited benchmark scale and heterogeneous evaluation protocols constrain systematic assessment across diverse environments.

2.1 AERIAL OBJECT-GOAL SEARCH

  • AerialOGS approaches use spatio-semantic representations to organize memory and plan, while multimodal models interpret goals, assess candidates, and guide search decisions.The review also identifies potential applications in search and rescue, infrastructure inspection, and autonomous delivery.Source: §2.1 · p. 3

2.2 AERIAL BENCHMARK

  • AerialOGS is an autonomous-exploration and target-identification setting within ObjNav, unlike VLN’s instruction-guided navigation.Existing aerial benchmarks largely focus on VLN tasks.Source: §2.2 · p. 3
  • UAV-ON provides 11K instance-level ObjNav tasks across 14 scenes, while CityAVOS provides 2.4K OGS tasks across six urban scenes.The review concludes that existing benchmarks remain limited in task scale and environmental coverage, with heterogeneous action spaces and data formats hindering large-scale training and cross-benchmark evaluation.Source: §2.2 · p. 4

3 AERIALDOJO-200K

AerialDojo-200K brings together 42 simulated scenes, 205K task instances, 4,115 km of flight trajectories, and a unified evaluation framework. It specifies how scenes, tasks, trajectories, and benchmark evaluation are constructed.

3 AERIALDOJO-200K

  • AerialDojo-200K comprises 42 simulated scenes, 205K task instances, 4,115 km of flight trajectories, and a unified evaluation framework.Source: §3 · p. 4

3.1 SIMULATOR

  • 42 scenes span four families and 21 types, covering urban, natural, infrastructure, and disaster environments.Source: §3.1 · p. 4
  • Twelve annotators spent two months labeling 109 landmarks, 2,099 target objects, and 2,099 object anchors; each anchor is a collision-free viewpoint 1 m outside its object.Source: §3.1 · p. 5

3.2 TASK AND DATASET

  • Each task pairs a start pose with an instance-level goal containing a target name, nearest landmark, relative direction, and semantic description or reference image.Source: §3.2 · p. 6
  • Hybrid A* plans collision-free reference trajectories, and synchronized multi-view RGB-D recordings are collected along them.Source: §3.2 · pp. 6–7
  • 205,732 task instances comprise 102,866 SemanticOGS and 102,866 ImageOGS instances across 42 scenes.Source: §3.2 · p. 7

3.3 BENCHMARK

  • The benchmark splits 42 scenes evenly into in-distribution and out-of-distribution sets, reserving OOD tasks for adaptation and held-out testing.Source: §3.3 · p. 7
  • Five complementary metrics assess navigation success, oracle success, distance to success, path efficiency, and collision rate.Source: §3.3 · p. 7

4 EXPERIMENTS

The experiments evaluate nine multimodal large language models across aerial object-goal search settings and scene families. Completion remains low, with model comparisons also varying by success radius and scene family.

4.1 BASELINES AND SETUP

  • Nine MLLMs are tested across Base, Standard, and Long-Horizon settings; Base and Standard reference lengths are [5, 30) m and [30, 60) m, with action budgets of 90 and 180.Overall results equally average map-level scores within each family, then equally average the four family means.Source: §4.1 · p. 8

4.2 EVALUATION RESULTS

  • GPT-5.6 Sol leads Natural and Infrastructure SR, Claude Opus 5 leads Disaster SR, and neither model dominates all scene families and metrics.Figure 5 plots Base-task performance across four scene families; outward indicates better performance for each metric.Source: Figure 5 · p. 9, §4.2 · p. 9
  • At 5 m, Gemini 3.8 Flash's Base SR rises from 2.40% to 11.16%, while success-radius changes also reverse model rankings on Base and Standard tasks.The paper retains the prompt-aligned 3 m threshold as the primary measure of precise task completion.Source: §4.2 · p. 9
  • At 3 m, the best SR falls from 3.20% on Base tasks to 0.49% on Standard tasks; Claude Opus 5's Base OSR of 7.84% accompanies a 58.96% collision rate.On Standard tasks, GPT-5.6 Sol and Claude Opus 5 achieve SRs of 0.26% and 0.49%, respectively.Source: §4.2 · pp. 8–9

5 CONCLUSION

AerialDojo-200K provides 205,732 semantic- and image-goal instances across 42 simulated scenes and three task settings. Evaluation finds low MLLM success rates and concludes current models remain far from reliable generalist aerial agents.

5 CONCLUSION

  • The benchmark comprises 205,732 semantic-goal and image-goal task instances across 42 simulated scenes and three task settings.It uses a unified observation, action, and evaluation interface.Source: §5 · p. 9
  • At the primary 3 m threshold, success rates reach only 3.20% on Base tasks and 0.49% on Standard tasks.The evaluation also reports frequent collisions and gaps between target-region visits and successful completion.Source: §5 · p. 10
  • AerialDojo-200K offers a common testbed for methods integrating exploration, instance-level target grounding, precise stopping, and collision avoidance.The reported findings place current MLLMs far from reliable generalist aerial agents under the evaluated framework.Source: §5 · p. 10

A PROMPT

The evaluation prompt specifies the observations, actions, embodiment configurations, and response format for SemanticOGS. It defines task completion through an explicit stop meeting distance, collision, and action-budget conditions.

A PROMPT

  • The prompt specifies front-facing RGB-D inputs, depth encoding, eight available actions, and aerial embodiment configurations.Figure 6 presents the evaluation prompt.Source: A PROMPT · p. 12, Figure 6 · p. 12
  • The prompt requires a single-action JSON response and specifies termination conditions.Source: A PROMPT · p. 12
  • Success requires an explicit Stop within 3 m of the target anchor, without collisions and within the assigned action budget.Source: A PROMPT · p. 12

B TASK COMPOSITION AND REFERENCE LENGTHS

The task set spans Base, Standard, and Long-Horizon settings, with instance counts varying across scene families. Figure 7 separates task volume, train/test allocation, and reference lengths.

B TASK COMPOSITION AND REFERENCE LENGTHS

  • AerialDojo-200K contains 79,974 Base, 82,424 Standard, and 43,334 Long-Horizon task instances.These represent 38.87%, 40.06%, and 21.06% of the release, respectively.Source: B TASK COMPOSITION AND REFERENCE LENGTHS · p. 13
  • Urban scenes contribute 60.79% of instances, followed by Natural at 24.22%, Infrastructure at 9.63%, and Disaster at 5.36%.Mall scenes alone contribute 28.82%, and scene counts range from 190 to 40,214.Source: B TASK COMPOSITION AND REFERENCE LENGTHS · p. 13
  • Figure 7 reports goal-conditioned counts by family and setting, Train/Test counts for ID and OOD groups, and mean reference lengths weighted by navigation-task count.The figure separates task volume from reference-route length.Source: Figure 7 · p. 13

C SCENE PARTITIONS AND TARGET COVERAGE

The partition places one instance of each scene type in ID and the other in OOD, yielding 21 scenes per group. Task counts and Train/Test allocations differ substantially between the groups.

C SCENE PARTITIONS AND TARGET COVERAGE

D INVENTORY

Table 7 inventories AerialDojo-200K’s scenes, annotations, task instances, and reference-trajectory coverage.

Loading 2609.36066v1…