AeroEval: Staged Program and Execution Validation for AI-Generated Drone Missions

1 Introduction

• Generated drone programs must satisfy platform interfaces, user intent, sensed-event responses, and environmental spatio-temporal constraints, not just syntax.Failures such as obstacle collisions, incorrect survey orientation, incomplete coverage, or missed search-to-tracking transitions may depend on geometry, sensing events, and realized trajectories.Source: §1 · pp. 1–2
• AeroEval improves navigation success over CLG from 11/20 to 19/20 tasks and analytical run-level success over one-shot AeroGen from 17/50 to 44/50 runs.With up to three regeneration cycles, Code Validator-only, Trajectory Validator-only, and full AeroEval achieve mean run-level success rates of 44%, 56%, and 88%, respectively.Source: Figure 1 · p. 2, §1 · p. 2
• Existing generation workflows provide limited evidence that a plausible program realizes its mission and do not systematically localize failures across code, intent, and physical behavior.Prior systems rely on prompt guardrails or simulator feedback, while many autonomous-system validation frameworks assume authored application logic and testable properties.Source: §1 · p. 2
• AeroEval combines deterministic syntax and API checks with context-grounded mission-intent validation, then evaluates simulated behavior against mission and environmental context.For analytical missions without a unique reference path, a bounded agent checks properties including obstacle avoidance, altitude, coverage, orientation, and event-driven transitions; failed stages return structured diagnoses for refinement.Source: §1 · p. 2
• The paper contributes a staged lifecycle with stage-specific feedback and context-grounded behavioral validation for analytical missions without a unique reference trajectory.The evaluation spans 20 navigation tasks and five analytical mission types, using two generators, two simulators, and two reasoning models.Source: §1 · p. 3

2 Related Work

• AeroEval examines how program analysis, mission context, and execution observations can be combined to validate and repair AI-generated drone applications.The paper situates this combination at the intersection of drone program synthesis, autonomous-system validation, and middleware for pervasive cyber-physical applications.Source: §2 · p. 3

2.0.1 AI-based drone program synthesis

• Drone code-generation approaches use prompts, simulator feedback, or mission-generation pipelines, but organize validation around execution outcomes or specified navigation behavior.AeroEval separates syntax and API checks, intent validation, and post-execution behavior, including mission-specific properties for analytical missions with multiple valid trajectories.Source: §2.0.1 · p. 3

2.0.2 Validation of autonomous systems

• Autonomous-system validation commonly uses specifications, simulation, digital twins, scenario tests, and runtime monitoring, typically assuming authored application logic and specified properties.AeroEval instead validates generated program structure, interfaces, intent, and realized behavior; formal verification and runtime assurance remain complementary.Source: §2.0.2 · pp. 3–4

2.0.3 Middleware for pervasive drone applications

• Drone middleware coordinates navigation, sensing, analytics, and resource placement, whereas AeroEval checks whether generated mission logic meets program-level and physical-context requirements before execution.Prior mobile sensing and analytics systems illustrate context-dependent behavior across mobility, sensing, communication, and computing, while assuming authored application logic.Source: §2.0.3 · p. 4

2.0.4 Summary

• AeroEval connects program synthesis, autonomous-system validation, and middleware through a lifecycle that separates program validity, intent alignment, and realized physical behavior.Its technical distinction is context-grounded validation of analytical missions without a unique reference trajectory, coupled with stage-specific feedback for repair.Source: §2.0.4 · p. 4

3.1 Architecture and Lifecycle

• AeroEval processes candidate missions through program validation, simulated execution, and behavioral validation, returning failed validation records for regeneration.A candidate must pass both mandatory stages; acceptance is limited to the implemented checks for the evaluated mission and environment.Source: §3.1 · pp. 4–5, Figure 2 · p. 5

3.2 Agents, Context, and Tools

• Validation agents use structured instructions, mission context, and observations to return records; they do not modify or execute mission code.Robot and world descriptions supply platform and environmental constraints, while deterministic tools include syntax parsing, API metadata checks, and numerical path comparison.Source: §3.2 · pp. 5–6

3.3 Code Validator

• The Code Validator checks syntax, platform API usage, and mission intent before simulator resources are allocated.The API checks identify unsupported operations and assess call ordering; the Intent Validator checks mission phases, constraints, and event transitions, not realized physical behavior.Source: §3.3 · p. 6

3.4 Trajectory Validator

• The Trajectory Validator compares navigation traces with reference paths or uses contextual criteria to assess analytical-mission behavior.Analytical criteria include obstacle and corridor compliance, altitude, coverage, orientation, waypoint completion, and event-driven transitions; multiple valid paths need not reproduce one expected trajectory.Source: §3.4 · pp. 6–7

3.5 Optional Deployment Advisor

• An optional deployment advisor recommends resource placements or waypoint schedules, but does not determine mission acceptance.Its evaluation is limited to an extensibility mechanism under constructed resource contention.Source: §3.5 · pp. 7–8

• The evaluation covers four dimensions: complementary failure detection, validator consistency, mission success, and latency and model costs.These dimensions assess both validation behavior and the practicality of the complete lifecycle.Source: §4 · p. 8
• Navigation missions use reference trajectories, while analytical missions depend on sensing, environmental context, and event-driven transitions.Source: §4 · p. 8
• The evaluation tests whether validation stages identify complementary failure classes and whether agent-based validators provide consistent decisions.Source: §4 · p. 8

4.1.1 Workloads

• 20 CLG navigation tasks with reference trajectories run in AirSim, while five AeroGen analytical mission types run in Gazebo.The navigation tasks include waypoint traversal, altitude and yaw control, and geometric flight patterns; analytical missions combine navigation with sensing, analytics, coverage, and event-driven behavior.Source: §4.1.1 · p. 8

4.1.2 Experimental environment

• The analytical experiments use Gazebo Ignition Fortress with a PX4-based quadrotor and isolated analytical-service containers.The shared workstation has an NVIDIA RTX 3090 GPU with 24 GB VRAM and 124 GB RAM; navigation experiments use CLG and AirSim on the same workstation.Source: §4.1.2 · p. 8
• Figure 3 reports stagewise mean SRrun across five analytical mission types and validator-agreement measures, with 10 runs per mission and a three-regeneration-cycle limit.Source: Figure 3 · p. 9

4.1.3 Metrics

• SRrun measures independent runs accepted by both mandatory stages within budget, whereas SRcandidate measures individual programs accepted by both stages.The study reports iterations-to-success, end-to-end latency, token usage, and Wilson-score 95% confidence intervals for SRrun; stagewise configurations use 10 independent runs per mission and at most three regeneration cycles.Source: §4.1.3 · p. 9

4.2.1 Program-level and execution-grounded validation address complementary failure classes.

• 88% mean run-level success for the complete configuration exceeds Code Validator-only validation at 44% and Trajectory Validator-only validation at 56%.The three configurations use independently generated initial candidates, so the comparison is across configuration-level outcomes rather than paired outcomes for identical generations.Source: §4.2.1 · pp. 9–10
• Without trajectory validation, Delivery collides with a building in 8/10 runs, Farm Survey starts from the wrong edge in 10/10, and Radio Tower Inspection violates requirements in 10/10.These failures depend on realized trajectories and environmental interaction, so program-level checks do not catch them.Source: §4.2.1 · p. 10
• Without Code Validation, Farm Survey has a false acceptance in 1 run and reaches the refinement limit in 3/10; Radio Tower Inspection has insufficient altitude or excessive lane spacing in 8/10.Search and Track also has invalid API usage in 4/10 runs and unnecessary yaw operations in 2/10; failure categories can overlap.Source: §4.2.1 · p. 10

4.2.2 Many code-valid candidates subsequently fail behavioral validation.

• Across Delivery, Farm Survey, and Radio Tower Inspection, 103 candidates pass Code Validation but fail behavioral validation because their realized behavior violates mission requirements.Cable Inspection and Search and Track have no observed code-pass/behavior-fail candidates in these results.Source: §4.2.2 · p. 10

4.2.3 The Trajectory Validator provides empirical rather than formal validation evidence.

• Among 15 expert-labelled outcomes, the validator has 3 false acceptances and no false rejections, while 14/15 traces receive consistent labels across three repetitions.The false-accept rate is 3/12 = 25%; the small, imbalanced set motivates reporting raw cross-model agreement rather than a chance-corrected coefficient.Source: §4.2.3 · p. 10
• A 25% false-accept rate means the Context-Grounded Agent is not a standalone assurance mechanism; AeroEval combines it with deterministic checks, iterative repair, and human intervention for non-converged runs.Source: §4.2.3 · p. 10

4.3.1 AeroEval improves convergence for navigation tasks that already use simulator feedback.

• Across 20 navigation tasks, CLG +AeroEval completes 19/20 (95%) versus native CLG’s 11/20 (55%), a net gain of eight completed tasks.AeroEval resolves nine tasks that fail under native CLG, but one task that succeeds under CLG does not converge with AeroEval; successful tasks average 3.3 iterations versus 2.1 for native CLG.Source: §4.3.1 · pp. 10–11, Figure 4 · p. 11

4.3.2 AeroEval improves run-level reliability across the analytical missions.

• Across five analytical mission types, AeroEval completes 44/50 runs (88%) versus AeroGen’s 17/50 (34%), a 54-percentage-point increase.Per-mission Wilson score 95% confidence intervals are wide because each estimate uses 10 runs; the aggregate intervals are [0.22, 0.48] for AeroGen and [0.76, 0.94] for AeroEval.Source: §4.3.2 · p. 11, Figure 4 · p. 11
• Radio Tower Inspection has the lowest AeroEval success, with a mean of 12.8 refinement iterations across 10 runs, while Cable Inspection has a more constrained workflow.Radio Tower Inspection combines coverage, altitude, lane-spacing, detection, inspection, and search-resumption requirements.Source: §4.3.2 · p. 11

4.3.3 Candidate-level acceptance is distinct from run-level reliability.

• Candidate-level acceptance is a lifecycle statistic, not run-level reliability: AeroEval accepts 44/293 candidates (15.02%), while run-level success measures whether a request eventually succeeds within budget.Candidate counts vary with the number of refinement iterations, and candidates within one run are not independent.Source: §4.3.3 · p. 11, Figure 5 · p. 12

4.4.1 Program-level validity does not imply correct mission behavior.

• Of 293 candidates, 103 pass syntax, API, and intent validation but fail during execution or behavioral validation; only 48/293 pass the complete lifecycle.Late failures primarily involve physical and mission context, including obstacle or corridor violations, incorrect altitude or orientation, incomplete coverage, and missing event-driven transitions.Source: §4.4.1 · p. 12

4.4.2 Accepted programs execute on the evaluated physical platform under supervision.

• Three missions execute successfully on a DJI Ryze Tello after mandatory simulation validation and expert review, with no manual intervention during flight beyond setup and teardown.The physical runs use scaled trajectory extents and travel distances because of the Tello’s limited battery life; mission logic and validation methodology remain unchanged.Source: §4.4.2 · pp. 12–13
• Figure 6 decomposes median analytical-mission refinement latency into generation, Code Validation, simulated execution, and Trajectory Validation; Table 3 summarizes practicality and model sensitivity over five mission types.Source: Figure 6 · p. 13, Table 3 · p. 13
• The physical evidence is limited to one drone platform and a small set of mission and environment conditions, so it does not establish general sim-to-real transfer or safety across other contexts.Source: §4.4.2 · p. 13

4.5.1 Validation overhead is smaller than the end-to-end generation and simulation cost for most missions.

• Validation contributes at most 30% and 25% of end-to-end latency for the Code and Trajectory Validators, respectively, while generation and simulated execution dominate the remaining latency.Median analytical refinement iteration time excluding simulation ranges from 49 to 585 s across the five mission types.Source: §4.5.1 · p. 13
• Across 50 runs of the five analytical missions, o3-mini uses 8.51M input tokens, 0.47M output tokens, and 1.60M reasoning tokens; approximately 83% of input context is cached.The average cost is approximately $0.03 per analytical refinement iteration.Source: §4.5.1 · pp. 13–14

4.5.2 The reliability benefit persists across the two evaluated reasoning models.

• AeroGen succeeds in 21/50 independent runs (42%), while AeroEval succeeds in 47/50 runs (94%) using Gemini-3.1-Pro, an increase of 52 percentage points.AeroEval also succeeds in 44/50 runs (88%) using o3-mini; overlapping confidence intervals do not establish that one model is more reliable.Source: §4.5.2 · p. 14

4.5.3 Optional deployment advice

• The optional advisor uses resource telemetry to recommend alternative analytics placements or waypoint schedules, but its recommendations do not determine mission acceptance.It is evaluated only as an extensibility mechanism under constructed resource contention.Source: §4.5.3 · p. 14

4.5.4 Summary

• The evaluation finds complementary failures detected by program and behavioral validation, with scoped physical evidence and mission- and model-dependent overhead rather than a general assurance guarantee.The physical experiments provide scoped evidence of applicability on one supervised drone platform.Source: §4.5.4 · p. 14

5 Conclusion and Future Works

• AeroEval separates syntax, API, and intent checks from post-simulation behavioral validation, returning stage-specific diagnoses for iterative regeneration.For analytical missions without a reference trajectory, its Context-Grounded Agent evaluates spatio-temporal and event-driven properties using mission, program, environment, and execution information.Source: §5 · p. 14
• AeroEval increases navigation success from 11/20 (55%) to 19/20 tasks (95%) and analytical run-level success from 17/50 (34%) for one-shot AeroGen to 44/50 (88%) using o3-mini.Code Validator-only and Trajectory Validator-only configurations achieve mean run-level success rates of 44% and 56%, respectively, while their composition achieves 88%.Source: §5 · p. 14
• AeroEval provides empirical validation evidence, not formal guarantees of safety, liveness, or adversarial robustness, and its evaluation covers two generators, two simulators, two reasoning models, five analytical mission types, and one physical drone platform.The authors also note that using the same model for generation and validation may produce correlated errors.Source: §5 · p. 14