PaperlayerView exact revision on arXiv

Source-linked AI summary

AeroCopilotBench: A Two-Tier Benchmark for Evaluating LLM Agents as Aviation Copilots in an Interactive Virtual Cockpit Environment

Yuchen Yuan, Zhenghuang Wu, Yuangan Li, Liang Ma, Ke Li

arXiv:2608.16349v1cs.AI

TL;DR

Existing question-answering evaluations do not comprehensively test aviation agents’ procedural execution and safety compliance in interactive settings. The paper introduces a two-tier benchmark and virtual cockpit environment, finding that Tier-2 performance varies substantially and the highest success rate is 72.6%.

  • Problem

    Conventional question-answering evaluations insufficiently assess aviation agents’ domain knowledge, procedural execution, and safety compliance in safety-critical interactive settings.

  • Method

    The paper combines 1,200 aviation knowledge questions with 73 interactive cockpit tasks that test state-dependent execution against final-state goals and hard safety constraints.

  • Results

    72.6% is the highest Tier-2 success rate, while similar Tier-1 accuracies and success rates conceal substantial differences in interactive performance and safety.

  • Takeaways & Limitations

    Aviation-agent evaluation should consider task completion, trajectory safety, and failure processes together rather than relying on static knowledge scores or aggregate success rates alone.

  • Takeaways & Limitations

    The benchmark covers two aircraft, 12 scenario templates, and 73 tasks, while ACOE omits several real-world flight dynamics and operational pressures.

Abstract

from arXiv · show

Large language model (LLM) agents may assist flight crews with complex decisions and task execution, but existing aviation evaluations centered on static knowledge do not support systematic testing of procedural execution and safety compliance in interactive environments. This paper presents the AeroCopilot Operational Environment (ACOE), a reproducible interactive virtual-cockpit test environment, and AeroCopilotBench, a two-tier aviation agent evaluation benchmark. Tier-1 evaluates aviation knowledge using 1,200 multiple-choice questions, while Tier-2 comprises 73 emergency and abnormal tasks derived from the manufacturers' Pilot's Operating Handbooks (POHs) and instantiated in ACOE. ACOE converts natural-language procedures into executable state transitions, final-state goal conditions, and hard safety constraints, enabling models to interpret cockpit state, diagnose faults, and operate aircraft systems through standardized tool interfaces. We establish a safety-gated evaluation framework in which a trajectory succeeds only when all task goals are achieved without violating any hard safety constraint, while safe goal progress and trajectory safety are measured separately. Across 12 models, the highest Tier-2 success rate is 72.6%, while static knowledge performance does not consistently translate into procedural execution. Analysis of 451 failed episodes from 3 representative models identifies recurring failures in procedural completeness, use of state feedback, and long-horizon execution management. These findings motivate state-aware agent orchestration, joint assessment of task completion and trajectory safety, and repeated regression testing. ACOE and AeroCopilotBench provide a reproducible foundation for testing knowledge application, interactive execution, and operational safety in aviation agents.

1. Introduction

The paper argues that aviation copilots require not only domain knowledge but also interactive procedural execution and trajectory-level safety compliance. It introduces ACOE and AeroCopilotBench to evaluate these capabilities, revealing a gap between static knowledge and operational performance.

  • Motivation: A competent aviation copilot must combine aviation knowledge, correct multi-step interaction with a stateful cockpit environment, and safety-compliant execution.Required capabilities include diagnosing faults, selecting procedures, invoking tools, executing operations, and complying with safety constraints.
  • Benchmark and environment: ACOE is a reproducible interactive virtual-cockpit environment, while AeroCopilotBench combines 1,200 Tier-1 knowledge questions with 73 Tier-2 interactive tasks.Tier-2 tasks are instantiated in ACOE from emergency and abnormal procedures derived from authoritative aviation sources.
  • Evaluation framework: The evaluation uses safety-gated success rate (SR): an episode succeeds only if all task goals are achieved without violating any hard safety constraint throughout the trajectory.Safety-gated outcome measures goal completion under the safety gate, while safety compliance rate measures compliance over the complete trajectory.
  • Results: 72.6% is the highest Tier-2 SR among 12 models, and safe-but-incomplete failures are more common than unsafe failures for every model.Similar SRs of 59.4% and 58.9% correspond to unsafe-episode shares of 0.5% and 9.6%, respectively.
  • Results: Similar Tier-1 accuracies of 86.3% and 85.9% coexist with Tier-2 success rates of 18.7% and 46.1%, exposing a gap between knowledge and procedural execution.Analysis of 451 failed episodes from 3 representative models identifies missing procedural steps, erroneous semantic priors, state-gating failures, and long-horizon execution drift.

2. Related Work

Prior aviation LLM evaluations largely measure knowledge or provide advisory recommendations without directly testing execution correctness. AeroCopilotBench extends interactive agent evaluation to aviation by assessing knowledge, procedural actions, and hard safety constraints in state-dependent environments.

  • Aviation LLM evaluation: Aviation LLM research spans knowledge assessment, real-time advisory, and operational-environment evaluation, with many domain-model evaluations emphasizing what models know.Examples include OpenAviation, ALUE, CAMB, and AviationGPT, whose evaluations remain largely centered on question answering, summarization, and information extraction.
  • Aviation LLM evaluation: Advisory systems combine flight data, weather, and manuals to recommend emergencies, but crews still select procedures and operate aircraft systems.LeRAAT therefore does not directly test execution correctness.
  • Interactive agent benchmarks: Interactive-agent benchmarks formalize executable environments, controlled tasks, tool interfaces, and verifiable outcomes across reasoning, function calls, multi-turn APIs, and repeated trials.AgentBench, BFCL, τ-bench, and τ 2-bench provide related evaluation paradigms for interactive systems.
  • AeroCopilotBench positioning: AeroCopilotBench extends interactive testing to safety-critical aviation by separating Tier-1 aviation knowledge from Tier-2 procedural actions under hard safety constraints.Tier-2 evaluates translating knowledge into correct actions in a state-dependent environment while satisfying hard safety constraints.
  • Safety-critical agent evaluation: Safety-critical evaluation studies show that static knowledge performance does not directly imply interactive execution capability, while long trajectories complicate judging temporal constraints and prohibited actions.AgentClinic reports declines when static questions become sequential decisions, and MANTRA identifies representational mismatch between manuals and tool-call trajectories.

3. Benchmark Construction

AeroCopilotBench is constructed as a two-tier progression from authoritative aviation knowledge assessment to state-dependent, safety-constrained procedural execution. Its interactive Tier-2 evaluation is enabled by ACOE, which converts POH procedures into executable cockpit tasks with standardized interaction and verifiable grading.

  • Tier-1 construction: Tier-1 freezes 1,200 multiple-choice questions generated from authoritative FAA, CFR, AIM, handbook, and POH sources with item-level provenance.The corpus spans 8 major categories, and each item retains its source document, section, and supporting quotation.
  • Tier-2 construction: Tier-2 converts 73 emergency and abnormal POH procedures into interactive cockpit tasks evaluated by final-state goals and hard safety constraints.It tests whether models translate knowledge into correct multi-turn operations rather than merely restating procedures.
  • ACOE environment: ACOE provides a reusable virtual cockpit in which agents query instruments, operate controls, and interact with evolving hidden conditions through standardized interfaces.Its architecture separates world definitions, task specifications, runtime execution and grading, and the global tool interface.
  • ACOE interaction: The model acts as Pilot Monitoring in a read–diagnose–act loop, observing permitted instrument values, operating writable controls, and indirectly affecting hidden variables through environment transitions.The runtime validates legal targets and values before applying task-specific state-transition rules.
  • Extensibility: ACOE supports extensibility by allowing new aircraft types and scenarios to be added mainly through world definitions or task specifications while reusing execution and interface components.The same translation from authoritative procedures to executable states and verifiable criteria may extend to other safety-critical domains.

4. Formal Evaluation Framework

The framework models Tier-2 as deterministic, partially observable cockpit interaction and evaluates episodes through separate goal-attainment and trajectory-safety criteria. It combines safety-gated performance metrics with diagnostics for tool use, ineffective actions, and interaction cost.

  • 4.1. Interactive Task Formulation: Tier-2 tasks use deterministic state transitions, shared ACOE state and action spaces, hidden variables, and model-visible observations returned through 12 standardized tools.Models receive task instructions and local tool observations rather than complete cockpit state; episodes end upon submit or after 48 decision turns.
  • 4.2. Safety-Gated Task Evaluation: Task success requires achieving final-state goals while satisfying every hard safety constraint throughout the complete interaction trajectory.Unsafe actions remain violations even if later recovery restores the system or satisfies final-state goals.
  • 4.2. Safety-Gated Task Evaluation: Primary-goal failure sets goal attainment to 0, while hidden task-critical variables can be reached only by satisfying their corresponding transition conditions.Goal attainment otherwise measures the fraction of final-state goal conditions satisfied.
  • 4.3. Performance Metrics: Tier-1 uses accuracy, whereas Tier-2 uses safety-gated success rate, with equal weight assigned to each evaluation unit.The overall Tier-2 success rate is a task-balanced macro-average over empirical task success rates and is the primary leaderboard metric.
  • 4.3. Performance Metrics: Safety-gated outcome measures safe goal progress by assigning zero to safety-violating episodes while retaining goal attainment for safe-but-incomplete episodes.Safety compliance rate separately reports the proportion of episodes violating no hard safety constraint.
  • 4.4. Interaction Diagnostic Metrics: Diagnostic metrics supplement primary performance measures by evaluating tool selection, ineffective actions, and interaction cost.Tool-selection scoring excludes the neutral submit tool and penalizes irrelevant tools or failure results.
  • 4.4. Interaction Diagnostic Metrics: The ineffective-action rate counts harmless but operationally ineffective system-write attempts, including no-op rewrites identified through rule-based trajectory replay.Write attempts include all set system calls, even those rejected by the environment.

5. Experimental Evaluation · 5.1. Experimental Setup

The evaluation covers 12 models under a common API-based protocol, with six models participating in both benchmark tiers for cross-tier comparison. Tier-1 uses 1,200 questions, while Tier-2 uses 73 interactive tasks repeated three times per model under frozen evaluation settings.

  • 5. Experimental Evaluation: The study evaluates 12 models spanning multiple providers, parameter scales, and model generations.The models include systems from Alibaba Cloud, DeepSeek-AI, MiniMax, Qwen Team, OpenAI, Google DeepMind, and Z.AI.
  • 5. Experimental Evaluation: Six models participate in both Tier-1 and Tier-2 and support the cross-tier comparison in Section 5.3.These models are qwen3.7-max, deepseek-v4-pro, deepseek-v4-flash, MiniMax-M2.5, Qwen3.5-397B-A17B, and Qwen3.5-122B-A10B.
  • 5.1. Experimental Setup: All models are accessed through OpenAI-compatible APIs under a common protocol summarized in Table 2.Formal Tier-2 results use the native function-calling evaluation path.
  • 5.1. Experimental Setup: Tier-1 contains 1,200 multiple-choice questions, with each question answered once by each Tier-1 model.This establishes the static-question component of the evaluation protocol.
  • 5.1. Experimental Setup: Tier-2 contains 73 interactive tasks, each run independently three times for every model.The repeated runs are part of the formal interactive evaluation procedure.
  • 5.1. Experimental Setup: 219 complete episodes per model result from running the 73 Tier-2 tasks three times independently.This figure is the resulting Tier-2 episode count per model.
  • 5.1. Experimental Setup: Task snapshots, system prompts, tool definitions, and interaction budgets remain frozen during evaluation.Freezing these components preserves a consistent evaluation configuration across runs.

5.2. Tier-2 Procedural Execution Analysis

Tier-2 performance varied substantially across models: success required complete goal attainment without hard-safety violations, and the highest success rate was 0.726. Safe-but-incomplete outcomes dominated failures, while recurring errors involved procedural omissions, incorrect semantic priors, state-gating failures, and long-horizon execution drift.

  • Model-level performance: 0.726 was the highest Tier-2 success rate, achieved by GPT-5.6-sol, while model-level SR ranged from 0.123 to 0.726 across 12 models.GPT-5.6-sol still failed 27.4% of episodes; GLM-5.1 led open-weight models at SR 0.530.
  • Safety-gated outcomes: 24.7%–67.6% of episodes were safe but incomplete, compared with 0.5%–22.8% unsafe, making safe-but-incomplete outcomes the larger failure category for every model.SR requires complete goal attainment and compliance with all hard safety constraints; the decomposition distinguishes success, safe incompleteness, and unsafe outcomes.
  • Execution discipline: 0.055–0.189 was the model-level ineffective-action-rate range, and GPT-5.6-sol combined the highest SR with IAR 0.168, the second-highest value.gemini-3.5-flash and deepseek-v4-pro had the lowest IAR values, 0.055 and 0.057, whereas GLM-5.1 had the highest at 0.189.
  • Task-level variation: 0.00–0.97 was the range of per-task mean success rates; 31 tasks were at or below 0.25, while only 6 exceeded 0.75.No task was completed in all 36 trials, and only one task was unsuccessful in every trial, indicating broad difficulty coverage concentrated toward harder tasks.
  • Failure analysis: The four recurring failure modes were missing critical procedural steps, erroneous semantic priors, state-gating failures, and long-horizon execution drift.Examples include omitting standby-battery or cowl-flap actions, failing to revise judgments after feedback, acting despite contradictory state observations, and losing procedural completeness near submission.

5.3. The Knowing–Doing Gap in Aviation Knowledge

Across the six models evaluated on both tiers, static aviation knowledge varied within a relatively narrow range, while procedural-execution performance differed much more widely. Similar Tier-1 scores could therefore accompany substantially different Tier-2 outcomes, and knowledge and execution rankings did not consistently coincide.

  • Knowledge-performance range: Tier-1 accuracy ranged from 0.7442 to 0.8917, with five of six models between 0.8250 and 0.8917.Most models therefore exhibited relatively similar levels of static aviation knowledge.
  • Knowledge–execution divergence: Tier-1 accuracy spanned 0.744–0.892, whereas Tier-2 success rates ranged from 0.123–0.589.The concentrated knowledge range corresponded to a much wider procedural-execution range.
  • Knowledge–execution relationship: The model-level Pearson correlation between Tier-1 accuracy and Tier-2 success rate was r = 0.57.The paper interprets this association descriptively for the evaluated model set.
  • Comparable knowledge, divergent execution: Qwen3.5-397B-A17B and deepseek-v4-pro differed by only 0.0041 in Tier-1 accuracy but by 0.274 in Tier-2 success rate.Their Tier-1 accuracies were 0.8633 and 0.8592, while their Tier-2 success rates were 0.187 and 0.461, respectively.
  • Ranking mismatch: Qwen3.5-397B-A17B ranked second among the six models on Tier-1 but second from last on Tier-2.This example shows that knowledge and execution rankings need not coincide among the evaluated models.

5.4. Implications for Agent-System Design and Testing

The section argues for state-aware orchestration, joint evaluation of task completion and trajectory safety, and repeated controlled testing to improve aviation-agent reliability. It also identifies model adaptation, knowledge calibration, or external procedural support as responses to procedural-knowledge failures.

  • Agent orchestration: Agent harnesses should track plans, observations, completed actions, outstanding steps, request critical-operation readbacks, and audit procedural completeness.These mechanisms target state-gating and long-horizon execution failures.
  • Agent orchestration: Procedural-knowledge failures may require model adaptation, knowledge calibration, or external procedural support.
  • Safety-aware assessment: 59.4% and 58.9% success rates masked unsafe-episode shares of 0.5% and 9.6% for gemini-3.5-flash and qwen3.7-max, respectively.The section recommends examining SR and SCR jointly, with IAR as supplementary evidence about system-write discipline.
  • Repeated testing: 30.6% of 876 groups of three repeated trials contained both successful and unsuccessful episodes, showing that single-trial success may be inconsistent.Because ACOE fixes task specifications and environment transitions, the same task set supports controlled comparisons across model versions and orchestration mechanisms.

6. Conclusion

ACOE and AeroCopilotBench provide a reproducible two-tier framework for evaluating aviation knowledge, state-dependent procedural execution, and safety compliance in LLM agents. Results show substantial gaps between static knowledge and interactive performance, while the benchmark’s scope and realism remain limited.

  • Contributions: ACOE and AeroCopilotBench evaluate aviation knowledge, state-dependent procedural execution, and safety compliance through a reproducible interactive virtual-cockpit environment.Tier-1 contains 1,200 multiple-choice questions, while Tier-2 instantiates 73 emergency and abnormal tasks from manufacturers’ POHs.
  • Findings: 72.6% is the highest Tier-2 success rate among 12 models, meaning even the strongest evaluated model fails in more than one-quarter of episodes.Safe-but-incomplete episodes are more common than unsafe episodes for every model.
  • Findings: Tier-1 accuracy varies within a relatively narrow range across six models, whereas Tier-2 performance varies substantially, showing that similar success rates can conceal safety differences.The conclusion separately emphasizes safe-but-incomplete and unsafe episodes when interpreting model performance.
  • Limitations and future work: The current benchmark covers two aircraft, 12 scenario templates, and 73 emergency and abnormal tasks, limiting the breadth of its evaluation.ACOE omits continuous aerodynamics, sensor noise, system hysteresis, uncertain fault evolution, and real-world flight-operation time pressure; some interfaces also abstract cockpit operations.

Appendix A. Evaluation Scope

Appendix A defines AeroCopilotBench as a closed-book evaluation of lower-frequency Pilot Monitoring functions rather than continuous flight-path control. Tier-2 covers 12 scenario templates instantiated from manufacturer POH procedures.

  • Evaluation scope: ACOE excludes flight-path control and evaluates lower-frequency Pilot Monitoring functions, including state interpretation, anomaly diagnosis, long-horizon execution, and system operation.The captain/PF retains responsibility for continuous control.
  • Evaluation scope: Three to four orders of magnitude greater inference latency than conventional trajectory-prediction models contributes to excluding continuous-control loops.Existing models also degrade markedly on continuous trajectory-prediction tasks during highly dynamic flight phases.
  • Closed-book condition: ACOE provides no procedure- or checklist-retrieval interface, keeping Tier-2 focused on interpreting cockpit state, diagnosing faults, and executing internalized procedures.The final-state goals and hard safety constraints are translated from applicable POH procedures.
  • Tier-2 task coverage: 12 scenario templates define Tier-2 coverage, with source procedures, principal instantiation dimensions, and frozen task-instance counts listed in Table A.1.POH references identify the applicable manufacturer handbook section and page.

Appendix B. Tool-Selection Diagnostic

Tool-selection performance is uniformly strong across Tier-2 models, with scores concentrated near 1.000. Most models achieve at least 0.990, and task-irrelevant or failed calls are uncommon.

  • Tool-selection performance: Tool-selection scores range from 0.936 to 1.000, with a mean of 0.986 and a median of 0.995.These scores use the Tier-2 tool-selection metric defined in Section 4.4.
  • Tool-selection performance: 9 of the 12 models score at least 0.990.
  • Tool-selection performance: Task-irrelevant or failed calls constitute only a small share of the calls issued by most models.

Appendix C. Failure Trajectory Case Study

The case study shows how omitting a C172S-specific standby-battery step initiates a cascading failure: the model violates safety constraints despite observing that ELECTRICAL FIRE remains active. The episode demonstrates that procedural omission and inadequate state gating can jointly cause task failure.

  • Failure cascade: The model omitted the C172S-specific standby-battery step, initiating the subsequent failure cascade.The reference trajectory included stby batt=OFF, which the model omitted.
  • Safety violations: 5 violating writes involved 2 types of hard safety constraints after the model restored ventilation and electrical power.These actions occurred while ELECTRICAL FIRE remained active.
  • State gating: Repeated observations that ELECTRICAL FIRE remained active failed to constrain subsequent actions, producing a state-gating failure.The episode was assigned to the state-gating category under the mutually exclusive dominant-mechanism rule.
Loading 2608.16349v1…