AirPlayground: Benchmarking Aerial Navigation across Diverse Reconstructed Environments

1 INTRODUCTION

• Offline demonstrations scale policy learning without training-time environment interaction, but accurate imitation does not guarantee reliable navigation under action-shaped deployment states.Small errors can compound through feedback, motivating systematic closed-loop evaluation across diverse environments.Source: §1 · p. 2
• The resource gap is twofold: dense flight demonstrations across diverse scenes and standardized closed-loop evaluation across varied environments remain limited.Indoor, urban outdoor, and vegetation settings differ in visibility and obstacle configurations.Source: §1 · p. 2
• AirPlayground pairs dense demonstrations with photorealistic 3DGS scenes and aligned navigation and collision geometry for learning and matched closed-loop execution.The benchmark contains 94 scenes and nearly 100K demonstrations with 46.5M synchronized samples at 50 Hz.Source: §1 · p. 2
• Its API supports shared-data training and unified evaluation across three navigation settings while preserving each policy’s native representations and inference structure.The benchmark also examines how offline prediction errors relate to closed-loop outcomes.Source: §1 · p. 2

Aerial Navigation Datasets

• Flight datasets have expanded scene and task coverage, while automated planning and fine-grained demonstrations have advanced collection; temporally dense experience across environments remains limited.That limits continuous-control experience for training diverse policies.Source: §2 · pp. 2–3

Reconstructed Environments and Navigation Benchmarks

• 3DGS benchmarks add metric geometry and collision-aware interaction, but broader reconstructed coverage of indoor, urban outdoor, and vegetation scenes is still needed.OpenFly spans 18 scenes and HUGE provides four reconstructed outdoor environments.Source: §2 · p. 3

Navigation Policy Evaluation

• Offline prediction errors and closed-loop task outcomes are evaluated through different protocols, and systematic understanding of their gap in aerial navigation remains limited.AirPlayground pairs demonstrations with executable worlds for trajectory replay and policy evaluation in corresponding scenes.Source: §2 · p. 3

3.1 SCENE COLLECTION

• The benchmark collects 94 scenes across indoor, urban outdoor, and vegetation settings, using a handheld scanner that captures metric geometry and multi-view imagery in one walk-through.The corpus includes 18 indoor, 48 urban outdoor, and 28 vegetation scenes.Source: §3.1 · p. 3

3.2 EXECUTABLE SCENE RECONSTRUCTION

• Each scene pairs a 3DGS visual asset with a shared-coordinate planning grid and 3D clearance field for free-space planning and collision queries.The grid includes obstacle inflation for path margins; the clearance field preserves vertical structure without inflation.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 SCENE-ADAPTIVE DEMONSTRATION GENERATION

• Scene-adaptive generation samples start–goal pairs with separation and obstacle-clearance constraints, retaining pairs only when a planned path connects them.When natural obstacle interaction is limited, box obstacles diversify trajectories; an A* path is shortened before the expert follows it in closed loop at 50 Hz.Source: §3.3 · pp. 4–5

3.4 UNIFIED CROSS-POLICY API

• Training and stateful control adapters map shared trajectories to policy-specific inputs and outputs while preserving each method’s execution schedule and planning mechanisms.The API supports pointgoal, image-goal, and goal-free navigation, and new policies can reuse the task generator, simulator, and evaluator.Source: §3.4 · p. 5

3.5 RESOURCE STATISTICS AND VALIDATION

• 94 executable worlds contain 94,000 successful trajectories and 46,530,557 synchronized RGB–state–action samples recorded at 50 Hz.The corpus totals roughly 259 flight hours and 1,800 km; all released trajectories are accounted for with no missing or orphaned files.Source: §3.5 · p. 5
• AirPlayground combines 94 executable worlds with 94K trajectories and continuous body-frame control supervision at 50 Hz, unlike resources with discrete supervision or lower-rate recordings.Across 20 scenes, rendering validation reports PSNR 19.19 dB, SSIM 0.655, and LPIPS (AlexNet) 0.223; Figure 3 pairs rendered observations with mesh-derived obstacle depth.Source: §3.5 · pp. 5–6, Table 1 · p. 6, Figure 3 · p. 6

4.1 EXPERIMENTAL SETUP

• The experiments compare seven baselines under a shared protocol across seen and unseen scenes, with fixed episodes and scene-equal weighting.The benchmark evaluates three navigation settings and uses matched scenes for offline-to-closed-loop comparisons.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 UNIFIED POLICY BENCHMARK

• NWM leads seen-scene success rate and A∗-SPL with lowest collision rate, while unseen-scene leaders vary by metric; ImageGoal improves success for three of seven baselines, whereas goal-free input lowers it for all seven.Comparable seen-scene success rates mask different behaviors: NWM has lower ControlVar and Turn/m than OmniVLA.Source: §4.2 · p. 7, Table 2 · p. 7
• Across four training-data scales, ACT median success rose from 30.6% to 41.2% on seen scenes and from 16.8% to 29.4% on unseen scenes.The evaluation used three training seeds, identical tasks across scales and seeds, and 100 fixed episodes per scene.Source: §4.2 · p. 8, Figure 6 · p. 8

4.3 FROM PREDICTION TO CONSEQUENCE

• Directional and turning-subset errors associate more strongly with collision rate than mean FDE, with signed correlation differences reaching +0.33.Yaw direction error, lateral direction error, and turning-subset FDE correlate with collision rate at ρ = 0.51, 0.37, and 0.31, respectively.Source: §4.3 · p. 8, Figure 5 · p. 8
• ACT success rate rises from 0% at H = 1 to 45% at longer horizons, while NavMorph peaks at an intermediate horizon and ViNT remains unchanged.The same checkpoints and tasks are evaluated across horizons; ACT executes only the first H controls of each newly queried action chunk.Source: §4.3 · pp. 8–9, Figure 7 · p. 8
• When predicted clearance decreases, local passage rate is 58.1% versus 80.3% for clearance increase, and reference-obstacle collision rate is 37.2% versus 13.1%.The groups have similar average prediction errors and expert clearances, but minimum clearance is 5.2 versus 15.7 cm.Source: §4.3 · p. 9, Table 3 · p. 9

5 CONCLUSION

• AirPlayground combines dense flight experience and scene geometry with executable real-world reconstructions in a unified aerial-navigation benchmark.Its API supports shared-corpus training and standardized evaluation across diverse environments while accommodating different policy formulations.Source: §5 · p. 9
• Seven baselines across four policy paradigms are evaluated in three navigation settings on seen and unseen scenes.The paper says it will publicly release the flight dataset, cross-policy API, and benchmark tools.Source: §5 · p. 9
• The analyses identify an offline-to-closed-loop mismatch involving prediction-error structure, execution protocols, and interactions with scene geometry.The conclusion frames the benchmark as supporting performance comparison and deployment diagnosis in corresponding environments.Source: §5 · p. 9

AI USE STATEMENT

• Generative AI tools were used only to improve grammar and clarity; the authors report no use for ideation, method design, implementation, experiments, or analysis.The authors reviewed the assisted text and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A DEMONSTRATION COLLECTION PROTOCOL

• A collection episode samples a scene-specific PointGoal task, plans a reference path, and executes a feedback controller to record flight observations and commands.Each episode also includes its task configuration and reference path.Source: A DEMONSTRATION COLLECTION PROTOCOL · p. 12

A.1 SCENE CONFIGURATION AND NAVIGABLE SPACE

• Scene settings define the planning grid inputs, and the grid marks filtered, inflated obstacle cells and locations outside the navigable polygon as occupied.A feasible grid path is followed by a closed-loop rollout before an episode is accepted, so grid feasibility and successful execution are separate checks.Source: §A.1 · pp. 12–13
• A separate 3D clearance field preserves height structure, while runtime queries read clearance and subtract the robot radius to signal collision.The field, planning grid, and rendered asset share a map transform; points outside the baked volume are treated as free.Source: §A.1 · p. 13

A.2 TASK SAMPLING AND INITIALIZATION

• Start–goal pairs are drawn from candidate locations only when their separation is at least max(dmin, 2δ), then one pair is selected uniformly and its roles randomly exchanged.Failed sampling or subsequent planning triggers task resampling.Source: §A.2 · p. 14

A.3 REFERENCE PATH PLANNING

• A* searches an eight-connected occupancy grid, rejects corner-cutting diagonals, and uses greedy line-of-sight shortcutting to reduce intermediate waypoints.Endpoint snapping and the bounded expansion budget can also cause a task to be rejected.Source: §A.3 · p. 14
• The reference polyline’s mean executed-to-straight-line path-length ratio is 1.06 across the released corpus.The collector stores both waypoint sequence and reference length.Source: §A.3 · p. 14

A.4 CLOSED-LOOP EXPERT EXECUTION

• At each control step, the expert tracks the active waypoint using the current simulated pose and outputs normalized planar commands plus yaw rate.The wrapper inserts zero vertical velocity, while the simulator applies the configured translational scale; only successful episodes are retained.Source: §A.4 · p. 15
• 98.0% of rollouts in the most recent collection run were kept; 64,000 of its 65,000 successful episodes entered the released corpus.The 1,332 rollouts that failed to reach the goal and two simulator-error aborts were excluded; rejected tasks that never reached rollout are not counted as failures.Source: §A.4 · p. 15

A.5 RECORDED MODALITIES AND EPISODE STORAGE

• Each record contains forward RGB, a 13-dimensional state vector, a local occupancy patch, and a three-dimensional action.The patch is yaw-aligned geometric supervision; its presence does not prescribe policy inputs.Source: §A.5 · p. 15
• Every one of 94,000 metadata episodes resolves to a valid shard, all 940 shards are referenced, and the seven baselines consumed the corpus end to end.The checks found no missing or orphaned files and confirmed the arrays load aligned across modalities for policy learning.Source: §A.5 · p. 16

A.6 BEHAVIOR DIVERSITY MECHANISMS

• Optional diversity mechanisms vary speed near obstacles and bends, sample near-obstacle starts, introduce disturbances followed by replanning, and vary planning margins or camera field of view.Disturbance metadata distinguishes those segments from subsequent expert recovery.Source: §A.6 · pp. 16–17

A.7 COLLECTION ACROSS SCENES

• Independent scene processes collect and write shards, while orchestration packs processes onto GPUs and promotes validated outputs to durable storage.The common recording schema allows outputs from separately collected scenes to be aggregated for policy training.Source: §A.7 · p. 17

B.1 EVALUATION PROTOCOL AND METRICS

• Evaluation uses 16 seen and 16 unseen scenes with 50 fixed episodes per scene, reusing scene IDs, poses, goals, geometry, and seeds across policies and settings.The scaling study instead uses five shared seen scenes and five unseen scenes, with 100 episodes per scene and three training seeds.Source: §B.1 · pp. 17–18
• Offline prediction diagnostics use shared expert anchors and a 0.2 s window, but additional queries use recorded expert observations rather than predicted-state observations.These are teacher-forced offline predictions, and lower position errors are better.Source: §B.1 · pp. 18–19, Table 7 · p. 19
• Closed-loop reporting includes completion, collision, obstacle, timeout, terminal-distance, and path-efficiency outcomes, with most metrics averaged equally across scenes.A*-SPL is relative to the grid-based planner rather than an exact continuous-space optimum; TTS averages simulation seconds over successes.Source: §B.1 · p. 20

B.2 BASELINE CONFIGURATIONS

• The shared API supports PointGoal, ImageGoal, and Goal-free modes where applicable; the main benchmark uses PointGoal without maps, A* paths, or simulator goal RGB.Training adapters preserve native policy inputs and targets, and the evaluated policies use adapted execution settings rather than universal original-method defaults.Source: §B.2 · p. 20
• Direct-control baselines include CNN-BC, ACT, and Diffusion Policy, while waypoint and world-model adapters convert their native predictions into the shared flight-control interface.ACT disables temporal ensembling by default; Diffusion Policy uses 100 denoising steps per decision.Source: §B.2 · pp. 21–22

B.3 ADDITIONAL NAVIGATION RESULTS

• Image Subgoal views are placed along the A* route every 0.5 m, and the supplied subgoal advances when the drone comes within 3 m.The final view is positioned at max(L/2, L−1 m), and the protocol does not require arrival at every subgoal.Source: §B.3 · p. 22
• The three goal settings reuse initial poses, target locations, and scene geometry while changing goal conditioning.The protocol covers 16 seen and 16 unseen scenes with 50 episodes per scene for each supported policy-setting pair.Source: §B.3 · p. 22
• Image Subgoal yields higher SR than PointGoal for DP, ViNT, and OmniVLA, but lower SR for the other four methods.Withholding goal input lowers SR for all seven methods; Goal-free CNN-BC has 19.8% CR but 73.3% OB.Source: §B.3 · p. 22

• Across seven policy scores, Spearman correlations use equally weighted seen and unseen split scores rather than averaging split-specific correlations.Source: §B.4.1 · p. 22
• The protocol averages scores over common valid scenes within each split and specifies 16 scenes and 50 closed-loop episodes per scene in each split.Source: §B.4.1 · p. 22
• For collision rates, Δρ is the correlation with execution error minus that with FDE, computed on identical valid policies and scene coverage.The passage characterizes this as a descriptive cross-policy association, not an estimate of predictive accuracy.Source: §B.4.1 · p. 22

B.4.2 EXECUTION HORIZON

• The main horizon evaluation reuses fixed tasks, checkpoints, prediction lengths, decoders, and history sampling, with 500 episodes per policy and horizon.Scores are averaged within scenes and then equally within each seen and unseen split; combined scores weight the splits equally.Source: §B.4.2 · p. 22
• Matched two-scene sweeps found ACT improves beyond H = 1, NavMorph favors intermediate intervals, and ViNT retains the same SR and CR through H = 50.Source: §B.4.2 · p. 23
• All four ACT tasks with measurable turn decay showed longer decay times under H = 1 than H = 50, including 0.40 to 1.28 s in one task.One task still succeeded despite delayed decay; early-terminated runs had no measured decay time.Source: §B.4.2 · p. 23

B.4.3 CLEARANCE-BASED DIAGNOSTICS

• Table 3 summarizes 180 independent local rollouts, one per sample across seven methods and ten scenes, comparing expert and predicted minimum clearances to a fixed reference obstacle over the same 0.2 s window; ∆c denotes their difference.Source: §B.4.3 · p. 23
• Local passage requires reaching an exit within 5 s without collision or leaving the navigable region; collision counts use the full group size as denominator.Source: §B.4.3 · p. 25

B.4.4 LOCAL GEOMETRY

• Local interventions apply fixed-direction 0.5 m/s velocity pulses after shared observation prefixes; paired clearance contrasts use the shortest window common to all branches.The analysis uses two scenes and task-cluster bootstrap intervals.Source: §B.4.4 · p. 26
• Toward reduces common-window clearance versus Clean and Away increases it in all six policy–bin groups, while collision contrasts vary.Source: §B.4.4 · p. 26, Figure 14 · p. 27