PaperlayerView exact revision on arXiv

Source-linked AI summary

$α^3$-Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents over 6G Networks

Mohamed Amine Ferrag, Abderrahmane Lakas, Merouane Debbah

arXiv:2601.03281v1eess.SYcs.AI

TL;DR

Existing evaluations provide limited evidence about whether LLM-based UAV agents remain safe, protocol compliant, and effective under dynamic network constraints. α3-Bench addresses this gap by evaluating multi-turn conversational UAV control with protocol-mediated actions and a composite metric across 113k episodes. Results show high mission success and safety compliance for several models but substantial variation in degraded-network robustness, reliability, latency, and token efficiency.

  • Problem

    Existing evaluations rarely assess safe, protocol-compliant, and effective LLM-based UAV autonomy under continuous interaction and realistic, fluctuating 6G conditions.

  • Method

    α3-Bench evaluates LLM-driven UAV autonomy as a multi-turn language-mediated control problem with MCP and A2A actions, dynamic network context, and six composite metric pillars.

  • Results

    Several frontier models achieve near-perfect mission completion and safety compliance, while robustness, protocol adherence, reliability, latency, and token efficiency differ substantially under degraded 6G conditions.

  • Takeaways & Limitations

    α3-Bench provides a reproducible, failure-aware basis for comparing networked LLM agents by both achieved outcomes and the reliability and efficiency of achieving them.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used as high level controllers for autonomous Unmanned Aerial Vehicle (UAV) missions. However, existing evaluations rarely assess whether such agents remain safe, protocol compliant, and effective under realistic next generation networking constraints. This paper introduces $α^3$-Bench, a benchmark for evaluating LLM driven UAV autonomy as a multi turn conversational reasoning and control problem operating under dynamic 6G conditions. Each mission is formulated as a language mediated control loop between an LLM based UAV agent and a human operator, where decisions must satisfy strict schema validity, mission policies, speaker alternation, and safety constraints while adapting to fluctuating network slices, latency, jitter, packet loss, throughput, and edge load variations. To reflect modern agentic workflows, $α^3$-Bench integrates a dual action layer supporting both tool calls and agent to agent coordination, enabling evaluation of tool use consistency and multi agent interactions. We construct a large scale corpus of 113k conversational UAV episodes grounded in UAVBench scenarios and evaluate 17 state of the art LLMs using a fixed subset of 50 episodes per scenario under deterministic decoding. We propose a composite $α^3$ metric that unifies six pillars: Task Outcome, Safety Policy, Tool Consistency, Interaction Quality, Network Robustness, and Communication Cost, with efficiency normalized scores per second and per thousand tokens. Results show that while several models achieve high mission success and safety compliance, robustness and efficiency vary significantly under degraded 6G conditions, highlighting the need for network aware and resource efficient LLM based UAV agents. The dataset is publicly available on GitHub : https://github.com/maferrag/AlphaBench

I. INTRODUCTION

α3-Bench addresses gaps in evaluating LLM-based UAV autonomy by modeling missions as multi-turn, language-mediated control loops under dynamic 6G conditions. It combines protocol-aware interaction, network adaptation, and a composite assessment of task, safety, reliability, and efficiency.

  • Motivation: Existing benchmarks rarely model continuous, interactive, safety-critical UAV missions under fluctuating networks or account for reasoning latency and token usage.These omissions limit evaluation of real-time, resource-constrained deployment.
  • Benchmark formulation: α3-Bench formulates UAV mission execution as multi-turn conversational control between an LLM-based agent and a human operator.The benchmark extends UAVBench scenarios into validated JSON dialogues encoding UAV states, airspace constraints, mission policies, and 6G context.
  • Network-aware evaluation: The benchmark integrates UAV dynamics, airspace constraints, mission policies, and a 6G-aware context to evaluate adaptation to latency, packet loss, and edge-computing variation.This formulation targets reasoning and control under dynamically changing operational conditions.
  • Protocol-aware interaction: MCP tool calls and A2A communication form a dual-protocol action layer for assessing protocol compliance, tool consistency, and multi-agent coordination.The interaction layer is embedded within each dialogue turn rather than treated as a separate capability.
  • Evaluation and results: α3 aggregates Task Outcome, Safety Policy, Tool Consistency, Interaction Quality, Network Robustness, and Communication Cost with reliability- and efficiency-normalized scores.The benchmark also reports comparative results for 17 state-of-the-art LLMs under realistic UAV mission conditions.

A. Conversational and Tool-Use Benchmarks for LLM Agents

α3-Bench extends conversational-agent and UAV benchmarks into end-to-end conversational autonomy under network-constrained UAV missions. Its workflow combines structured dialogue, protocol-mediated actions, state evolution, safety constraints, and 6G network conditions in one evaluation setting.

  • Conversational and tool-use benchmarks: τ2-Bench evaluates dual-control conversational agents through tool-mediated environment interaction, while experiments show agents struggle when collaborating with an active user.This motivates conversational evaluation beyond single-control settings.
  • Conversational and tool-use benchmarks: AGENTBENCH evaluates multi-round instruction following, reasoning, and action execution across eight interactive environments but omits UAV-specific constraints and communication dynamics.Its actions are also evaluated independently of latency, reliability, and resource efficiency.
  • UAV reasoning benchmarks: UAVBench provides 50,000 structured, safety-validated flight scenarios covering mission objectives, UAV configurations, environmental factors, and quantitative risks.α3-Bench uses UAVBench scenarios as the basis for conversational mission episodes.
  • Positioning of α3-Bench: α3-Bench jointly evaluates conversational decision-making, network-aware adaptation, and resource efficiency for UAV missions under 6G conditions.Its network context includes slicing, latency, jitter, packet loss, throughput, and edge-compute load.
  • Workflow and environment: The workflow initializes validated UAVBench scenarios, runs structured human–LLM dialogue with MCP or A2A actions, updates the environment, and terminates on completion or constraint-related conditions.The environment couples UAV dynamics, airspace constraints, 6G QoS attributes, and continuous UAV state.

1) UAV Dynamics and Sensors:

The UAV model combines flight state, multimodal sensing, airspace restrictions, dynamic 6G conditions, and mission policies to constrain high-level control.

  • UAV state: The UAV state includes Cartesian position, velocity, yaw direction, and remaining battery energy.
  • Sensors: The sensor set comprises LiDAR, RGB, Thermal, and IMU modalities whose outputs inform the next intent or action.
  • Airspace and geofencing: Airspace constraints specify minimum and maximum altitudes plus cylindrical or polygonal no-fly zones defined by centers and radii or boundaries.
  • 6G network context: The 6G context tracks service slice, latency, jitter, packet loss, throughput, and edge-server load, which evolve during dialogue.
  • Mission policies: Mission policies constrain permissible actions through altitude, energy, collision-avoidance, and communication-reliability requirements.

5) State Evolution and Kinematics:

α3-Bench models UAV missions as structured, alternating dialogue turns embedded in coupled physical, network, and policy dynamics, with intents, actions, observations, and network state driving evolution.

  • State evolution: The environment couples UAV kinematics, stochastic disturbances, network feedback, and mission policies into a unified evaluation testbed.Disturbances include wind turbulence and sensor noise modeled through a multivariate Gaussian process.
  • Conversational control: Each mission is a multi-turn dialogue where the LLM interprets UAV and network state and proposes the next high-level action.
  • Dialogue representation: Dialogue turns encode speaker role, high-level intent, optional MCP or A2A action, structured observation, and current 6G network state.
  • Dialogue representation: Strict speaker alternation preserves the human-in-the-loop interaction structure, while observations record task, source, status, and payload fields.
  • Observed agent behavior: Across more than 113k episodes, frequent intents emphasize mission initialization, telemetry checks, and confirmation before execution.

2) User Speaker Modeling:

The benchmark’s action layer combines MCP tool calls with A2A coordination so agents can operate, communicate, and adapt under changing 6G conditions.

  • Example trajectory: The illustrated mission uses MCP for telemetry and thermal scanning, A2A for collision avoidance, and switching among eMBB, URLLC, and mMTC slices.
  • A2A: A2A actions coordinate the UAV with other autonomous agents through recipient, task, and payload fields, producing structured coordination feedback.
  • Evaluation layer: The dual-protocol design evaluates structured tool use, multi-agent coordination, and protocol compliance in UAV workflows.
  • MCP: MCP tools connect the LLM controller to mission services such as sensing, navigation, configuration, and telemetry.Telemetry observations include UAV position, velocity, yaw, battery, and 6G link-quality indicators.
  • Network adaptation: Network context conditions both reasoning and action evolution, enabling adaptation to degraded connectivity and fluctuating edge-computing capacity.

6) LLM Policy Function:

α3-Bench models the LLM as a policy mapping UAV state, dialogue history, and 6G context to structured mission or reasoning actions, whose observations update a closed-loop control process. The benchmark evaluates MCP tool use and A2A coordination under controlled physical and network stochasticity.

  • LLM Policy Function: The LLM policy maps UAV state, dialogue history, and current 6G network state to mission-level or reasoning-level actions.Examples include changing altitude, switching network slices, verifying navigation plans, and checking telemetry consistency.
  • Observation and Feedback: Each dialogue turn produces a structured observation from the UAV state, network context, and invoked action semantics, enabling a language-mediated perception–reasoning–action loop.MCP observations represent tool outputs, while A2A acknowledgements report coordination outcomes and feed subsequent reasoning without directly modifying local UAV state.
  • MCP and A2A Usage: Over 113k episodes, read_telemetry exceeded 21% of MCP calls, while network-aware, sensing, planning, and safety tools formed the dominant usage pattern.The reported distribution includes set_waypoint, activate_sensor, switch_network_slice, check_geofence, and land.
  • Network-Aware Actions: Control-critical MCP actions concentrated in URLLC, sensing actions in eMBB, and non-time-critical monitoring in mMTC, indicating slice-dependent action selection.Several control actions exceeded 80% of invocations under URLLC, while switch_network_slice was almost exclusively executed there.
  • Stochastic Effects: Controlled stochastic perturbations model wind disturbances, latency and jitter variation, packet loss, and sensor noise while preserving deterministic action–observation schemas.The resulting execution logic combines deterministic semantics with bounded uncertainty in physical and network variables.
  • Multi-Agent Coordination: A2A coordination appeared in 99.998% of episodes at approximately 2.05 calls per episode, with collision avoidance accounting for over 26% of A2A calls.Other tasks support environmental awareness, information sharing, and swarm-level coordination.

8) Dialogue Termination and Success Signal:

α3-Bench terminates dialogue episodes on mission completion, safety or structural violations, degraded communication, or turn-budget exhaustion, and uses a binary completion signal as the primary Task Outcome component. Its composite objective combines task, safety, protocol, interaction, network, and communication dimensions while also measuring computational efficiency.

  • Termination Conditions: Episodes terminate when mission objectives are achieved, a safety or structural constraint is violated, or communication quality falls below an operational threshold.The broader closed-loop process also allows termination through exhaustion of the turn budget.
  • Success Signal: The binary success indicator equals 1 when the mission is completed and 0 otherwise, forming the primary component of the Task Outcome score.Task Outcome is one pillar of the overall α3 composite evaluation metric.
  • Optimization Objective: The agent optimizes expected composite performance subject to schema compliance, safety constraints, and strict dialogue alternation while adapting to dynamic 6G conditions.The objective treats linguistic coherence and high-level planning as part of a language-mediated control problem.
  • Composite Evaluation Function: The α3 metric combines Task Outcome, Safety Policy, Tool Consistency, Interaction Quality, Network Robustness, and Communication Cost.The six pillars cover mission performance, operational correctness, interaction quality, degraded-network resilience, and communication efficiency.
  • Metric Weights: Task Outcome receives weight 0.30, Safety Policy and Tool Consistency 0.20 each, Interaction Quality 0.15, Network Robustness 0.10, and Communication Cost 0.05.The paper states that sensitivity analysis across degraded network ranges preserved model ranking stability.
  • Efficiency Metrics: The benchmark supplements α3 with efficiency-adjusted scores based on wall-clock generation time and total prompt-plus-completion tokens.These measures capture trade-offs among reasoning quality, inference latency, and token consumption.

3) Definition of the Six Pillars:

α3-Bench defines six pillars covering mission completion, safety, tool consistency, interaction quality, network robustness, and communication cost, while also recording generation efficiency metrics. These components evaluate valid, grounded UAV dialogue under safety and dynamic 6G constraints.

  • Task Outcome measures whether the UAV completes its mission while satisfying environmental, policy, and structural constraints.
  • Safety Policy penalizes altitude, no-fly-zone, separation, and hard battery violations, with the score clamped to [0, 1].
  • Tool Consistency measures the fraction of structured MCP or A2A actions that produce logically consistent, schema-compliant observations.
  • Interaction Quality combines dialogue length, strict speaker alternation, and grounding in prior observations, tool outputs, or mission-state variables.The optimal dialogue length is the median number of turns among successful episodes, and components are normalized to [0, 1].
  • Network Robustness evaluates resilience to degraded 6G conditions using hard-state exposure, behavioral adaptation, and network-slice awareness.Hard conditions include latency above 40 ms, packet loss of at least 1%, throughput below 5 Mbps, or edge load above 0.8.
  • 7.0 ms median latency and 1.11 ms average jitter are reported for URLLC across more than 113k conversational UAV episodes.The benchmark also reports slice-dependent latency, packet loss, throughput, and edge-load behavior under large-scale 6G dynamics.
  • Communication Cost rewards concise reasoning and minimal protocol overhead relative to token and tool-call budgets.The benchmark uses budgets of 10,000 tokens and 25 tool actions, chosen from successful mission upper bounds.
  • Generation Efficiency records reasoning delivered per second and per thousand tokens using wall-clock latency and total prompt-plus-completion tokens.

4) Model-Level Aggregation and Reliability:

Model-level aggregation computes composite scores over valid UAV dialogue episodes and adjusts them for generation reliability, requested-episode coverage, and the number of attempts needed.

  • Composite scores are computed over valid UAV dialogue episodes satisfying the JSON schema and mission constraints.
  • Reliability-adjusted aggregation penalizes invalid generations, failure to produce the requested number of valid episodes, and repeated attempts per valid episode.The call-efficiency term uses total attempts across generation attempts, while reliability and coverage capture distinct failure modes.
  • The resulting score is explicitly described as reliability- and efficiency-adjusted.

5) Efficiency-Normalized Scores:

α3-Bench reports efficiency-normalized performance alongside constraints for safe, valid, and network-aware UAV episodes. These measures relate composite performance to wall-clock generation time and token usage.

  • Efficiency-normalized metrics report composite performance per second and per thousand tokens under a fixed per-domain episode budget.
  • Episodes must satisfy physical safety constraints, JSON schema validation, adaptive network behavior, and valid communication-safe actions under degraded conditions.Schema failures are discarded, and failure to adapt lowers Network Robustness and Task Outcome scores.
  • Under degraded networks, communication-safe actions include switching to URLLC, buffering commands, or postponing sensor activation.

4) Dialogue Alternation and Turn Integrity:

Dialogue episodes require strict agent–user alternation, finite turn counts, and a consistent safe final state. These constraints make evaluation reflect interactive reasoning and operational validity rather than language fluency alone.

  • Every episode must alternate between agent and user roles; deviations or disallowed roles invalidate the episode.
  • Missions must terminate within 8–12 dialogue turns with a final UAV state aligned with mission policies and safety flags.
  • The benchmark embeds structural, physical, safety, and network constraints so scores reflect meaningful reasoning and safe control under 6G uncertainty.
  • α3(D∗) →1, gen_fail_rate →0, and success_rate →1 define the stated target behavior for the optimized policy.The paper further describes valid JSON, mission completion across UAVBench scenarios, safety, and efficiency despite 6G variability.
  • An effective agent is expected to maintain schema validity, safety awareness, network adaptivity, conversational coherence, and resource efficiency.Network adaptation includes reducing communication, prioritizing URLLC, or deferring non-critical actions when conditions degrade.

IV. EXPERIMENTAL RESULTS

The experiments evaluate LLM UAV agents with controlled conversational episodes, fixed cross-model sampling, and efficiency-aware α3 metrics. Results show that mean task performance remains high, but composite scores vary substantially across models.

  • Evaluation Setup: 113k conversational UAV episodes were generated from standardized UAVBench scenarios for evaluation.The corpus represents complete dialogue-based control loops with repeated executions over controlled initial states.
  • Evaluation Setup: 17 state-of-the-art LLMs spanning proprietary and open-weight families were comparatively evaluated.The candidate set includes Claude, GPT, Gemini, DeepSeek, Qwen, and Mistral model families.
  • Evaluation Setup: 50 episodes per UAVBench scenario were reused identically across all models to balance robustness, feasibility, and fairness.The evaluation used the same sampled conversational trajectories for every candidate model.
  • Evaluation Setup: Deterministic decoding, fixed seeds, a 10,000-token cap, and up to three schema-repair attempts controlled reproducibility.Failed episodes were retained as failure stubs rather than discarded, preserving generation failures in the statistics.
  • Overall Results: All models exceeded 0.93 mean score, with GPT-5.1-chat highest at 0.991, but ChatGPT-4o-latest led α3 macro at 0.976.GPT-5.1-chat’s α3 macro score was 0.825, while GPT-5.2-chat reached 0.514, showing that mean task score and composite performance diverge.
  • Overall Results: α3 macro scores exposed large cross-model differences in reasoning quality, reliability, coverage, and efficiency despite broadly high mean scores.The composite score integrates these dimensions and reveals trade-offs not captured by average task accuracy alone.

B. Reliability, Coverage, and Failure Analysis

Reliability and failure analyses show that nominal mission success can conceal unstable execution, while efficiency analyses expose substantial latency and token-cost differences. Strong reasoning quality therefore does not uniformly translate into robust, efficient UAV operation.

  • Reliability and Coverage: Claude-Sonnet-4.5, GPT-4.1-mini, and Qwen3-Max each achieved 1.00 reliability, 1.00 coverage, and 1.00 success rate.These models represent the strongest fully stable mission-execution results reported in the reliability analysis.
  • Reliability and Coverage: DeepSeek-v3.2 reached approximately 0.66 reliability and coverage, while Gemini-3-Pro-Preview fell to approximately 0.52 despite nominal success rates of 1.00.The results indicate instability across complete mission executions that is not reflected by nominal success alone.
  • Generation Failures: Gemini-3-Pro-Preview had the highest generation failure rate at 0.48, followed by DeepSeek-v3.2 at 0.34 and DeepSeek-v3.2-exp at 0.20.The reported failures directly explain the reduced reliability and coverage of the affected models.
  • Generation Failures: GPT-5.1-chat, ChatGPT-4o-latest, Claude-Haiku-4.5, Claude-Sonnet-4.5, and Qwen3-235B-A22B-2507 recorded zero generation failure rates.The analysis identifies these models as robust across the evaluated conversational trajectories.
  • Efficiency Trade-offs: ChatGPT-4o-latest combined a mean score near 0.98 with the highest temporal efficiency at nearly 0.09 α3 per second.GPT-5.1-chat followed with approximately 0.99 mean score and 0.043 α3 per second.
  • Computational Cost: ChatGPT-4o-latest completed episodes in approximately 10.6 seconds, whereas DeepSeek-v3.2 and Kimi-K2-Thinking reached up to 314.0 seconds.The timing spread illustrates the practical latency trade-off among evaluated agents.
  • Computational Cost: GPT-5.1-chat used approximately 2,854 tokens per episode, while stronger reasoning generally introduced substantial resource overhead.Token consumption was treated as both a computational and economic cost relevant to scalability and real-time deployment.
Loading 2601.03281v1…