PaperlayerView exact revision on arXiv

Source-linked AI summary

AERIAL: Mission-Aware LLM Selection for Language-Driven UAV Control in Flying Networks

Diana Nunes, Ricardo Amorim, Pedro Ribeiro, André Coelho, Rui Campos

arXiv:2610.31029v1cs.NI

TL;DR

The paper asks whether mission-aware LLM selection can improve on static model assignment amid variation in performance across UAV mission types. AERIAL benchmarks models, routes missions using empirical scenario-model mappings, and reports mission-dependent results plus a placement-algorithm integration case study.

  • Problem

    The paper asks whether mission-aware LLM selection produces better outcomes than static single-model assignment, given performance variation across mission types.

    Source: §1 · p. 2

  • Method

    AERIAL benchmarks candidate LLMs across UAV mission categories, builds an empirical scenario-model mapping, and uses it for hybrid edge-cloud execution.

    Source: §1 · p. 1

  • Results

    Performance varied by mission-model pairing: Gemma 2 showed high reliability in basic navigation, Qwen 2.5 Coder performed better in obstacle scenarios, and both Qwen 2.5 Coder and Gemini 2.5 Flash achieved 70% reliability in structured function invocation.

    Source: §6.4 · p. 10, §7.4 · p. 11

  • Takeaways & Limitations

    The authors conclude that adaptive selection improves reliability and operational robustness, and the case study suggests support for parameter-dependent optimization workflows.

    Source: §7.4 · p. 11, §9 · p. 12

    Evaluation was limited to simulation and single-UAV missions, and mappings were not tested with cross-validation or held-out prompts.

    Source: §8.3 · p. 12

Abstract

from arXiv · show

Unmanned Aerial Vehicles (UAVs) in Flying Networks (FNs) increasingly require mechanisms to translate high-level natural language objectives into executable missions. Although Large Language Models (LLMs) have demonstrated the feasibility of language-driven UAV control, existing approaches typically rely on static model configurations and overlook mission-aware performance variability. This paper presents AERIAL, a benchmark-driven framework for adaptive LLM selection in hybrid edge-cloud UAV control. AERIAL separates systematic model benchmarking from mission execution, deriving empirical scenario-model mappings and applying them at runtime for mission-aware selection. Evaluation across 720 automated simulation runs spanning navigation, spatial reasoning, and algorithm invocation revealed significant performance variability depending on scenario-model pairing. A case study integrating AERIAL with a FNs placement algorithm further validated the framework's ability to interface with domain-specific optimization modules through natural language. These results validate the effectiveness of AERIAL's adaptive model selection in improving reliability and operational robustness for autonomous UAV control in FNs.

1. INTRODUCTION

AERIAL addresses fixed-model UAV control with a benchmark-driven framework that selects LLMs by mission category. Its evaluation reports mission-dependent model performance and includes integration with a UAV placement algorithm.

1. INTRODUCTION

  • Prior LLM-based UAV control demonstrated executable mission generation but relied on fixed model configurations despite performance varying across mission types.The cited FLUC work also did not consider cloud-assisted reasoning or systematic model benchmarking.Source: §1 · p. 1
  • AERIAL separates systematic model benchmarking from execution and selects a model based on empirical performance for the classified mission type.It supports local inference for offline autonomy and cloud-based reasoning when connectivity permits.Source: §1 · p. 1
  • Across 720 automated simulation runs, LLM performance differed by mission scenario; a placement-algorithm case study also tested natural-language integration with a domain-specific module.The evaluation covered diverse UAV scenarios, and the case study examined integration with a state-of-the-art UAV placement algorithm.Source: §1 · p. 2

2. RELATED WORK

Prior work establishes natural-language UAV control and LLM code generation, but the reviewed systems leave gaps in systematic evaluation and mission-aware selection. The related-work section positions AERIAL against those gaps.

2.1 LLM-Based UAV Control

  • Several UAV-control systems demonstrate language-driven control, but the reviewed approaches do not provide systematic benchmarking across heterogeneous mission categories.The cited works include conceptual reviews, cloud-dependent systems, and architectures using local models without cross-category benchmarking.Source: §2.1 · p. 2

2.2 LLM-Based Code Generation for Networking Systems

  • Networking code-generation benchmarks report model variability, motivating structured benchmarking for UAV mission synthesis because both tasks translate natural language into constrained executable code.SIMCODE [18] evaluates network simulation code generation, while the paper notes shared API, correctness, and safety or performance constraints with UAV missions.Source: §2.2 · pp. 2–3

2.3 Adaptive Model Selection and Hybrid Edge-Cloud Execution

  • Related work on adaptive selection and hybrid inference addresses distributed model use, but the reviewed approaches do not combine heterogeneous-task model selection with UAV control.The cited UAV-networking works focus on coordination or networking intelligence, while the cited embodied-robotics approach relies on cloud-hosted models.Source: §2.3 · p. 3

2.4 Summary

  • The paper identifies a gap in combining automated LLM benchmarking, performance-based model learning, adaptive mission-aware selection, and hybrid edge-cloud execution for UAV control.The related-work comparison says existing frameworks focus on either local or cloud execution and mostly use fixed model configurations.Source: §2.4 · p. 3

3. AERIAL FRAMEWORK

AERIAL separates empirical model evaluation from mission execution, using scenario-model performance profiles to route natural-language UAV missions. Its architecture combines runtime components with hybrid edge-cloud operation and automated benchmarking.

3. AERIAL FRAMEWORK

  • AERIAL benchmarks LLMs across mission categories and routes incoming prompts to models according to the resulting performance profiles.The two-phase design separates empirical evaluation from operational deployment.Source: §3 · p. 3

3.1 Two-Phase Architecture

  • During benchmarking, models generate and execute simulated missions whose outcomes and metrics form a scenario-model mapping; execution classifies prompts and uses deterministic lookup without runtime model comparison.Recorded metrics include latency, token usage, and mission success rate.Source: §3.1 · p. 3

3.2 System Components

  • AERIAL’s modular components handle model selection, autopilot communication, telemetry supervision, and storage of benchmarking results and mappings.The Model Selection Engine supports local and cloud inference, while the Runtime Supervision Layer applies scenario-specific validation during benchmarking.Source: §3.2 · p. 3

3.3 Workflow

  • The workflow first benchmarks predefined scenarios and records response time, token use, prompt count, and success; at execution, a classifier routes the mission through the stored mapping to generated code.The figure depicts benchmarking as the source of profiles used to allocate incoming missions during execution.Source: §3.3 · pp. 3–4, Figure 1 · p. 4

3.4 Key Architectural Features

  • AERIAL supports local or cloud inference, empirically driven deterministic selection, reproducible simulation benchmarking, and modular compatibility with standard UAV control stacks.Its mapping can be updated as models or performance profiles change, and components can be extended independently.Source: §3.4 · pp. 4–5

4. IMPLEMENTATION

The implementation integrates local and cloud model inference with ArduPilot simulation and modular Python components. It records structured logs and benchmarking data for use across the framework’s two phases.

4.1 Software Stack

  • The software stack combines local inference through Ollama, cloud APIs, ArduPilot SITL, MAVProxy, and OpenStreetMap location resolution.SITL provides simulated mission execution, while MAVProxy bridges AERIAL and ArduPilot communication.Source: §4.1 · p. 5

4.2 Core Components

  • The Python components implement control orchestration, six-category prompt classification, automated tests and validation, metric aggregation, model-map generation, code checks, and autopilot communication.The classifier’s categories include coordinates, location, diagonal, waypoints, obstacle, and algorithm.Source: §4.2 · p. 5

4.3 Data Storage and Logging

  • AERIAL stores session histories, aggregated performance metrics, and the scenario-model recommendations used during execution.The records include model, latency, token usage, prompt count, and success statistics.Source: §4.3 · p. 6

5. EVALUATION METHODOLOGY

AERIAL evaluates local and cloud-hosted LLMs across UAV missions, then uses measured performance for mission-aware model selection. The methodology records latency, token usage, and validated mission success in simulation.

5.1 LLMs Under Evaluation

  • The evaluation compares three locally deployed models with one cloud-hosted model to examine performance and hybrid edge-cloud trade-offs.The models represent different trade-offs in reasoning capability, inference efficiency, and deployment flexibility.Source: §5.1 · p. 6, §5 · p. 6

5.2 Benchmarking Pipeline

  • Each model-scenario pair was tested in 30 automated ArduPilot SITL runs, recording response latency, token usage, and mission success.The pipeline included prompt generation, code execution, telemetry monitoring, and scenario-specific validation.Source: §5.2 · p. 6

5.3 Mission Scenarios and Validation Criteria

  • Six mission categories tested coordinate and named-location navigation, relative diagonal movement, waypoint traversal, obstacle avoidance, and SUPPLY algorithm invocation.The scenarios probe different natural-language interpretation and control tasks.Source: §5.3 · p. 6, §5.3.1 · p. 6, §5.3.2 · p. 6
  • Success criteria included 15 m horizontal and 2 m vertical target accuracy for coordinate and location missions, and at least 90% waypoint visitation within 15 m.Diagonal movement used distance and direction-ratio tolerances; obstacle and algorithm missions required obstacle clearance or validated SUPPLY parameters and waypoint traversal.Source: §5.3.1 · p. 6, §5.3.2 · p. 6, §5.3.3 · p. 6

5.4 Evaluation Metrics

  • The study measured LLM response latency, input-plus-output token usage, and success rate based on error-free execution meeting scenario validation criteria.Success was computed from successful missions and total attempts; UAV propulsion energy and onboard power consumption were outside the study’s scope.Source: §5.4 · p. 7

6. EVALUATION RESULTS

Benchmark results vary substantially by model and mission: no model leads across all tasks in success, latency, and token use. Simpler navigation often favors different models than waypoint or obstacle tasks.

6.1.1 LLM Response Latency Analysis

  • 3.6 s for LLaMA 3.2 location missions and 4.9 s for coordinate navigation; waypoint latency rose to 31.2 s.Gemini 2.5 Flash ranged from 2.1–13.3 s and was fastest for algorithm invocation at 2.1 s; Qwen 2.5 Coder reached 114.1 s for waypoint sequencing.Source: §6.1.1 · p. 7

6.1.2 Token Efficiency Analysis

  • 21.9–91.9 tokens for Gemma 2 and 26.0–78.4 tokens for Gemini 2.5 Flash, while LLaMA 3.2 used 337.5 tokens for waypoint sequencing.Qwen 2.5 Coder used 298.5 tokens for waypoints and 140.7 tokens for obstacle scenarios.Source: §6.1.2 · p. 7

6.1.3 Success Rate Analysis

  • 93–100% success for Gemma 2 on coordinate, location, and diagonal missions contrasted with 0% waypoint and 3% obstacle success.Qwen 2.5 Coder led obstacle success at 57% and reached 37% for waypoints; Gemini 2.5 Flash reached 90% for coordinates and 87% for diagonal movement, but 23% for waypoints.Source: §6.1.3 · p. 7

6.2 Performance Characteristics

Model performance depends strongly on mission type: models trade speed, output length, and success, while waypoint sequencing remains difficult across the evaluated models.

6.2.1 Model-specific Observations

  • LLaMA 3.2 responded quickly (3.6–9.6 seconds) but achieved at most 53% success and 0% on location missions.The passage attributes its difficulty with complex spatial reasoning to its 3B parameter count.Source: §6.2.1 · p. 8
  • Gemma 2 achieved 93–100% success on basic navigation but 0% on waypoints and 3% on obstacle avoidance.Its token usage was 21.9–91.9 tokens; the paper identifies limitations in multi-step planning and conditional logic.Source: §6.2.1 · p. 8
  • Qwen 2.5 Coder led obstacle reasoning at 57% success, nearly 20× Gemma 2, despite 17.2–114.1-second latency and 140.7–298.5-token outputs.The paper links its constraint-reasoning strength to code-specialized training.Source: §6.2.1 · p. 8
  • Gemini 2.5 Flash achieved 40–90% success, with 90% for coordinate navigation and 2.1-second algorithm invocation at 70% success.It is described as a viable cloud-based alternative when network connectivity permits.Source: §6.2.1 · p. 8

6.2.2 Scenario-specific Observations

  • Among 397 failed runs, location navigation had 77.9% name-related errors, while 74.7% of 87 obstacle failures were validation errors.Waypoint failures included unsupported helper functions; LLaMA 3.2 also had recurring math-module and undefined-variable errors.Source: §6.2.2 · pp. 8–9
  • No evaluated model consistently achieved high reliability across mission types, and waypoint sequencing remained challenging across local and cloud models.No local model exceeded 37% waypoint success; obstacle success ranged from 3–57%.Source: §6.2.2 · pp. 8–9

6.3 Mission-aware Model Selection Results

AERIAL derives a scenario-to-model mapping from benchmarked success, latency, and token usage, then uses it to route missions. The execution example illustrates classification, selection, code generation, and runtime supervision.

6.3 Mission-aware Model Selection Results

  • The selection policy weights success rate at 0.65, latency at 0.20, and token count at 0.15, with configurable weights and ties favoring higher success.Latency and token count are min-max normalized within each mission category and inverted; the mapping remains fixed during execution.Source: §6.3 · p. 9

6.3.2 Benefits of Adaptive Selection

  • The data-driven mapping routes missions to comparatively stronger models, reserves larger models for structured reasoning, and can be recomputed as benchmarks or models change.The default mapping prioritizes local models for offline capability, with cloud models available when connectivity permits.Source: §6.3 · p. 9, §6.3.2 · p. 9

6.3.1 Learned Model-Selection Mapping

  • The learned mapping assigns Gemma 2 to coordinates, location, and diagonal missions, and Qwen 2.5 Coder to waypoints, obstacle, and algorithm missions.The mapping is a snapshot of the evaluated models and scenarios.Source: §6.3.1 · p. 9

6.3.3 Execution Phase Example

  • For the diagonal command, the system classified the task, selected Gemma 2 from the mapping, generated sequential movement code, and monitored stability across five readings.Position changes fell from 0.42 m to 0.00 m, after which the system confirmed movement completion.Source: §6.3.3 · pp. 9–10

6.4 Summary

AERIAL’s evaluation found that model reliability depends on the mission scenario, supporting adaptive selection rather than uniform model use. The task classifier routed benchmark prompts with 100% accuracy within the structured prompt set used for system testing.

6.4 Summary

  • Gemma 2 handled basic navigation most reliably, while Qwen 2.5 Coder performed better on obstacle scenarios.The evaluation benchmarked four LLMs across six mission scenarios.Source: §6.4 · p. 10
  • The task classifier achieved 100% accuracy on structured benchmarking prompts designed as distinct instances of six predefined mission categories.This result applies to the tested prompt distribution.Source: §6.4 · p. 10
  • Benchmark-driven model selection provided more consistent reliability and improved resource allocation than static single-model configurations under the evaluated conditions.Source: §6.4 · p. 10

7. CASE STUDY: VALIDATION WITH THE SUPPLY ALGORITHM

The SUPPLY case study tested whether AERIAL could route natural-language, parameter-dependent missions to models that reliably invoke a domain-specific optimization module. Results showed differing success rates and persistent function-invocation challenges across the evaluated models.

7.1 SUPPLY Algorithm Overview

  • SUPPLY computes energy-aware 3D UAV trajectories that minimize propulsion energy while satisfying ground-user QoS requirements; its internal optimization behavior was outside this evaluation’s scope.The study focused on parameter extraction, invocation, and mission execution through AERIAL.Source: §7.1 · p. 10

7.2 Test Design

  • Across 200 runs, AERIAL’s SUPPLY integration tested 50 configurations once per each of four LLMs.Configurations varied two ground users’ spatial coordinates and traffic demands.Source: §7.2 · p. 10
  • A successful run required correct parameter parsing and function invocation, valid waypoints, and mission completion without runtime errors.Source: §7.2 · p. 11

7.3 Results

  • 70% success was achieved by both Qwen 2.5 Coder and Gemini 2.5 Flash, compared with 66% for Gemma 2 and 44% for LLaMA 3.2.The results cover the SUPPLY case study; the table caption identifies metrics as mean ± std.Source: §7.3 · p. 11, Table 3 · p. 11
  • LLaMA 3.2 had the lowest local-model response latency at 6.58 s on average but the highest token usage at 99.3 tokens.The passage attributes the token usage likely to output inflation from failed or incomplete invocations.Source: §7.3 · p. 11
  • Across 75 failed runs, LLaMA 3.2 accounted for 28 failures, Gemma 2 for 17, and Gemini 2.5 Flash and Qwen 2.5 Coder for 15 each.The reported error patterns included name errors and generic execution exceptions.Source: §7.3 · p. 11
  • Structured parameter extraction and function invocation remained challenging across models under strict API constraints.Source: §7.3 · p. 11

7.4 Summary

  • The results indicated that AERIAL’s mission-aware selection routed algorithm-invocation tasks to comparatively reliable models and supported parameter-dependent optimization workflows under the evaluated conditions.Source: §7.4 · p. 11

8. DISCUSSION

AERIAL’s evaluation found mission-specific differences in model reliability and showed that its mapping-based selection can support both UAV missions and SUPPLY function invocation. The discussion also identifies limits in evaluation scope, networking conditions, and the framework’s role in safe deployment.

8.1 Key Findings

  • Across six scenarios and four LLMs, results showed mission-aware performance variation, with navigation scenarios favoring Gemma 2 and obstacle tasks favoring Qwen 2.5 Coder.The findings supported separating benchmarking from mission execution.Source: §8.1 · p. 11
  • In 200 SUPPLY runs, Qwen 2.5 Coder and Gemini 2.5 Flash each achieved 70% success, while Gemma 2 achieved 66%.The case study indicated that LLM control layers can interface with domain-specific optimization modules under defined API constraints, with reliability below deterministic software baselines.Source: §8.1 · p. 11

8.2 Practical Implications for FNs

  • AERIAL reduces manual model configuration through data-driven selection and supports local offline models alongside cloud models when network access permits.It is a natural-language interface above, not a replacement for, rule-based systems, reinforcement-learning planners, or traditional mission planners.Source: §8.2 · p. 12

8.3 Limitations

  • Evaluation was limited to ArduPilot SITL simulation, so real-world actuation delays and environmental variability were not assessed.Source: §8.3 · p. 12
  • The six mission categories and single-UAV setting do not cover the full FN operational spectrum; mappings were not tested with held-out prompts or cross-validation.Execution results therefore reflect benchmarking prompts, and automated remapping triggers were not implemented.Source: §8.3 · p. 12
  • The hybrid architecture was not evaluated under degraded connectivity, cloud outages, handover, fallback behavior, or varying communication overhead.Source: §8.3 · p. 12
  • AERIAL is not a safety-control mechanism, and complex-mission performance must improve before deployment beyond research contexts.It does not replace deterministic collision-avoidance or mission-planning modules required for safe operational deployment.Source: §8.3 · p. 12

9. CONCLUSIONS

AERIAL combines systematic benchmarking, adaptive LLM selection, and hybrid edge-cloud execution for natural-language UAV control in Flying Networks. Its evaluation found mission-dependent performance differences, while future work targets broader validation and more complex settings.

9. CONCLUSIONS

  • AERIAL separates model evaluation from mission execution and selects models using empirical performance rather than fixed manual configuration.Source: §9 · p. 12
  • The evaluation found that LLM performance varies across mission types and that adaptive selection improves reliability and operational robustness.Source: §9 · p. 12
  • Future work includes real-world UAV validation, coordinated multi-UAV missions, broader mission categories, realistic FN networking conditions, and online policy adaptation.Source: §9 · p. 12
Loading 2610.31029v1…