PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding

Yujing Chang, Thinh Pham, Van-Phat Thai, Chunyao Ma, Yash Guleria, Pham Nhut Huy, Sameer Alam

arXiv:2608.24621v1cs.CL

TL;DR

Safety-critical language models may achieve strong semantic scores while mishandling information with asymmetric operational consequences. The paper evaluates this problem with a consequence-aware framework and controlled ATC benchmark, finding that semantic metrics overestimate reliability and that risk-aware fine-tuning narrows but does not close the gap.

  • Problem

    Standard semantic metrics can treat harmless differences and dangerous misses similarly, making them incomplete measures of reliability in asymmetric-risk ATC communication.

  • Method

    The paper evaluates ATC understanding with consequence-sensitive action and slot scoring plus directional readback-risk metrics on a controlled benchmark grounded in aviation standards and expert input.

  • Results

    Nearly identical zero-shot AR-Geo scores of 0.338 and 0.343 coincided with a ninefold DDR difference of 0.459 versus 0.049.

  • Takeaways & Limitations

    Consequence-aware evaluation is a necessary complement to standard NLP metrics before safety-critical deployment claims.

  • Takeaways & Limitations

    Task 2 uses structured perturbations rather than live readback variation, and the study is not evidence of deployment readiness.

Abstract

from arXiv · show

Can language models be trusted in safety- critical operations? In such settings, strong per- formance on semantic metrics does not guaran- tee operational reliability: a misread altitude, a dropped execution condition, or a confused call- sign may score well under standard F1 yet carry sharply asymmetric operational consequences. We study this problem in air traffic control (ATC), where controller-pilot communication demands near-zero error tolerance, and use consequence-aware evaluation to test whether semantic scores misstate operational reliabil- ity. The framework is instantiated in a con- trolled diagnostic ATC benchmark grounded in aviation standards and feedback from 40 air traffic controllers across three countries. Evaluating 8 models, we uncover a system- atic semantic-safety gap: conventional scores give substantially higher performance estimates than consequence-aware evaluation, even for models that appear reliable under standard met- rics. Risk-aware fine-tuning narrows but does not close this gap, showing that consequence- aware evaluation is a necessary complement to standard NLP metrics before any real safety- critical deployment claim

1 Introduction

Safety-critical language understanding requires evaluating operational consequences, not only semantic similarity. The paper introduces a consequence-aware ATC framework and benchmark to measure this semantic-safety gap.

  • Standard semantic metrics treat errors uniformly even when action targets, control values, and execution conditions have different operational consequences.
  • The study targets ATC communication, where misreading an altitude, callsign, or execution condition can contribute to serious operational events.
  • The framework maps language-understanding errors to action structure, critical information units, and consequence-sensitive scores grounded in aviation standards and expert validation.
  • Conventional semantic metrics can substantially overestimate operational reliability in asymmetric-risk communication.
  • The controlled benchmark tests structured language understanding and real-operation error detection using expert-reviewed cases grounded in aviation standards.
  • Across prompting, fine-tuning, and model scales, the evaluation demonstrates substantial divergence between semantic correctness and operational safety.

2 Related Work

Prior language-understanding work largely optimizes semantic robustness, while safety-oriented evaluation broadens assessment beyond aggregate accuracy. This paper focuses on mapping structured semantic errors to concrete operational risk.

  • Task-oriented language understanding has traditionally emphasized intent accuracy, slot-level F1, exact match, and robustness to noisy or complex input.
  • Safety-oriented evaluation studies behavioral tests, severity-aware metrics, uncertainty, prompting, and outcome-aware failures beyond aggregate accuracy.
  • Existing safety-oriented studies typically address general behavioral failures, harmful content, uncertainty, or domain-level labels rather than concrete operational consequences.
  • ATC language-processing research has primarily focused on corpora, automatic speech recognition, and domain-specific spoken-language understanding.

3 Dual-Task Dataset

The paper constructs a controlled, expert-grounded dual-task ATC benchmark. It combines structured instruction recovery with safety judgment over deliberately perturbed readbacks.

  • The dataset is grounded in aviation standards and validated by real air traffic controllers across structured recovery and readback safety tasks.
  • The evaluation samples 500 structurally complete utterances from U.S. and European controller-pilot communications after filtering incomplete or non-operational exchanges.
  • Five controllers annotate the evaluation set with utterance-level actions and chunk-level operational spans, while separate splits support fine-tuning only.
  • Task 2 tests whether controlled pilot-readback variants preserve the safety-critical meaning of their original clearances.
  • Task 2 perturbations target critical fields, lower-criticality values, execution constraints, and harmless formatting or filler variations.
  • The Task 2 evaluation is balanced across ten readback types and four risk levels: CORRECT, HIGH, CRITICAL, and EXTREME.

4 Consequence-Aware Evaluation Framework

The evaluation framework scores ATC understanding according to operational criticality and penalizes dangerous risk underestimation. It covers weighted action recovery, entity matching, and readback safety judgment.

  • The framework combines weighted semantic matching, nonlinear completeness scoring, and ordinal downgrade penalties.
  • Criticality weights derive from aviation procedures and severity ratings by 40 air traffic controllers across China, Singapore, and India.
  • Task 1: Structured Operational Understanding: Action scoring binds recovered slots to each gold action schema and assigns zero when the action type itself is not predicted.
  • Task 1: Structured Operational Understanding: AR-Geo uses non-compensatory geometric aggregation so many lower-risk matches cannot average away a missed high-consequence field.
  • Task 1: Structured Operational Understanding: AR-Geo has the strongest tested correlation with controller-rated acceptability, and matches controller preferences more often than AR-Lin: 0.76 vs. 0.64.
  • Task 2: Readback Safety Judgment: Task 2 evaluates correctness, error type, affected slot, and ordered risk level for each instruction-readback pair.
  • Task 2: Readback Safety Judgment: Dangerous Downgrade Rate and Weighted Downgrade Severity measure lower-risk predictions relative to gold risk, with lower values preferred.

5 Experimental Setup

The experiments evaluate eight models across structured understanding and readback safety judgment, comparing prompting, fine-tuning, and consequence-aware metrics. The protocol targets critical slots, risk calibration, and dangerous under-estimation through four research questions.

  • Evaluation protocol: Task 1 evaluates eight commercial and open-weight models for broad zero-shot coverage, while Task 2 uses a four-model subset spanning API and open-weight settings.Unless fine-tuned, models use standard inference interfaces with greedy decoding; Task 1 has 500 held-out examples and Task 2 has 1000 readback examples.
  • Prompting conditions: Four prompting conditions compare schema-only zero-shot, operational knowledge, few-shot examples, and Full-Aligned rules-plus-examples prompting.Task 2 additionally evaluates Few-shot+CoT to test whether explicit reasoning changes risk calibration.
  • Fine-tuning baseline: Consequence-weighted LoRA fine-tuning compares standard cross-entropy with Risk-Loss objectives that up-weight critical slots and higher-severity readback samples.Critical Task 1 slots include altitude, heading, runway, and callsign; Task 2 weights risk level, error type, and severity.
  • Evaluation metrics: AR-Geo is the primary Task 1 metric, while Task 2 emphasizes error-type F1, risk-level accuracy, Dangerous Downgrade Rate, and Weighted Downgrade Severity.DDR and WDS measure dangerous under-estimation and are lower-is-better metrics.
  • Research questions: The research questions measure the semantic-safety gap, its slot and readback drivers, adaptation effects, and whether structured understanding transfers to safety judgment.The design explicitly tests whether models recover operationally critical information and judge its safety significance.

6 Results and Analysis

Results show that semantic metrics systematically overestimate operational reliability: consequence-aware scores expose failures in critical values, conditions, and risk calibration. Risk-aware adaptation improves performance, but structured extraction and safety judgment remain distinct capabilities.

  • Semantic-safety gap: 0.737 NER-F1 for gpt-5.4 and 0.308 NER-F1 versus 0.109 AR-Geo for qwen3-8b show that surface span recovery can overstate operational completeness.Missing callsigns or shared conditions can collapse AR-Geo despite substantial NER-F1, while lower-risk misses are more tolerated.
  • What drives the gap: 56.0pp higher strict heading recall and 35.0pp higher condition recall from zero-shot to Full-Aligned prompting identify exact values and execution conditions as major gap drivers.Heading errors include strict boundary failures and complete value disappearance; route elements also remain difficult.
  • Human alignment: AR-Geo correlates more closely with controller judgments than NER-F1, with Pearson r = 0.68 versus r = 0.44.Controller preference comparisons support the same pattern when outputs trade surface overlap against safety-critical information.
  • Adaptation effects: 0.686 AR-Geo for Risk-LoRA versus 0.515 for CE fine-tuning shows that consequence weighting substantially improves structured extraction.Matched models gain from near-zero baselines to 0.65–0.70 under Risk-LoRA, while prompting introduces a trade-off between few-shot extraction gains and broad-knowledge reasoning.
  • Risk miscalibration: DDR ranges from 0.459 for gpt-4o-mini to 0.049 for qwen-plus, while qwen-plus has 0.530 risk-level accuracy, revealing unstable risk calibration.Models may identify error mechanisms yet differ sharply in dangerous under-estimation.
  • Error-type structure: Condition-related readback errors remain weaker than explicit value and target substitutions, requiring sensitivity to dangerous omissions and invariance to equivalent paraphrases.Risk-aware fine-tuning gives the best reported overall Task 2 performance, while Full-Aligned prompting reaches 0.878 risk-level accuracy and new open-weight runs reach 0.930–0.936.
  • Cross-task transfer: 0.338 versus 0.343 zero-shot AR-Geo can coincide with a ninefold DDR difference, 0.459 versus 0.049, showing that extraction quality does not determine risk calibration.Figure 3 presents risk calibration as an additional capability beyond Task 1 extraction.
  • Implications: Execution conditions remain the clearest cross-task failure because nearly identical wording can authorize different actions at different times.The paper concludes that standard metrics are necessary but insufficient and that closing the gap requires consequence-aware objectives.

7 Conclusion

The paper demonstrates a semantic-safety gap in safety-critical language understanding: standard semantic evaluation can systematically overestimate operational reliability when errors have asymmetric consequences. Its consequence-aware framework and controlled ATC benchmark make this gap measurable, while prompting and risk-aware fine-tuning reduce but do not eliminate it.

  • Standard semantic evaluation can systematically overestimate reliability when language-understanding errors carry asymmetric operational consequences.
  • The framework and controlled ATC benchmark measure the gap across structured instruction understanding and readback safety judgment.
  • Prompting and risk-aware fine-tuning reduce but do not close the semantic-safety gap.

Limitations

The study’s limitations constrain how its ATC findings should be interpreted and generalized. The benchmark and adaptation experiments do not support deployment claims or broad conclusions about live operational performance.

  • ATC-specific slot weights and Task 2 taxonomy require new expert input for adaptation, which may change quantitative results.
  • Task 2 structured perturbations support controlled diagnosis but do not fully capture live pilot readback variation or estimate deployment readiness.
  • The study does not exhaustively evaluate current commercial models or broad fine-tuning recipes, and fine-tuning uses one 8B open-weight backbone.
  • Improved scores remain far from deployment-grade ATC reliability.

Ethics Statement

The paper documents its evaluation materials, analysis procedures, and reproducibility boundaries. It provides supporting artifacts and sensitivity analyses while noting that provider-hosted model aliases may not be immutable.

  • Sensitivity analyses test smoothing constants, slot weights, action-schema strictness, and alternative nonlinear penalties.
  • The appendix documents expert validation, dataset construction, metric definitions, error taxonomies, prompting protocols, and fine-tuning details.
  • The framework can transfer beyond ATC by replacing the domain action inventory, critical-slot mapping, and error taxonomy while preserving the scoring principle.
  • Provider-hosted aliases may change over time because the outputs lack immutable provider-side snapshot hashes.
  • Varying the geometric smoothing constant across four orders of magnitude leaves reported scores and rankings unchanged at the paper’s stated precision.

B.2 Slot Weight Perturbation

The slot-weight analysis tests whether consequence-aware rankings depend on expert-derived coefficients or strict action schemas. Results indicate that weight and schema perturbations change absolute scores modestly but preserve the main qualitative ordering and semantic-safety gap.

  • B.2 Slot Weight Perturbation: Expert-derived weights are perturbed using Uniform-Geo and ±20% changes to high-consequence slots, including callsign, runway, altitude, heading, waypoint, taxiway, and condition.
  • B.2 Slot Weight Perturbation: Absolute values move only slightly and model ordering remains stable under slot-weight perturbations.
  • B.2 Slot Weight Perturbation: Changing coefficients alters how strongly misses are penalized but does not erase the pattern that models missing action-critical values and conditions remain weaker.
  • Action-schema sensitivity: Relaxing execution-condition and contextual-slot requirements raises absolute AR-Geo scores, but the qualitative ordering remains stable.
  • Diagnostic benchmark design: The dataset prioritizes safety-relevant examples, multi-action utterances, readback perturbations, and expert-auditable labels over broad coverage.
  • Alternative nonlinear penalty: Geo and ERP expose the same broad semantic-safety gap, while AR-Lin is less sensitive to critical misses.

C.4 Task 1: Structured Operational Understanding

Task 1 evaluates whether models recover structured operational information from ATC utterances using exact span matching and action-conditioned consequence-aware scores. The benchmark uses controller-annotated held-out utterances and distinguishes complete recovery from partial or high-consequence misses.

  • Benchmark and annotations: Task 1 contains 500 held-out ATC utterances manually annotated by five air traffic controllers with utterance-level actions and chunk-level semantic labels.The action inventory includes eight operational actions, and utterances may contain multiple actions.
  • Metrics: The evaluation includes exact span metrics, weighted NER scores, action-risk metrics, and Strict complete-recovery scoring.Matching requires normalized exact text-span and label agreement; partial overlap is not counted as a strict match.
  • Metrics: AR-Geo conditions slot recovery on the gold action schema, assigning zero when a gold action is not predicted and penalizing missing required slots geometrically.Unlike linear scoring, the geometric score is designed to make missing required operational components sharply reduce performance.
  • Metrics: Strict measures the fraction of utterances with every gold action and action-conditioned slot recovered, but it does not distinguish near-complete recovery from catastrophic misses.The paper therefore treats Strict as a complete-recovery indicator rather than the primary metric.
  • Consequence-aware scoring: A worked example shows that missing a heading value is operationally more consequential than missing a frequency, even when both predictions contain one error.AR-Geo sharply penalizes missing required high-weight components, whereas linear scoring assigns moderate credit based on weighted miss mass.

E Task 1 Additional Error Analysis

Additional analysis shows that strict span errors can overstate operational information loss, while AR-Geo exposes action instances that collapse because of one or more missed safety-relevant slots. Task 2 separately evaluates whether readbacks preserve operational meaning and assigns directional risk judgments.

  • Value-aware diagnostics: Among 93 held-out examples with explicit numeric headings, strongest zero-shot models preserve the heading somewhere in the predicted heading region in over 90% of cases despite much lower strict heading-slot recall.This separates exact-boundary failure from complete disappearance of the operational value.
  • Collapsed action instances: 3,330 of 5,864 gold action instances across eight zero-shot models are collapsed with action-conditioned AR-Geo below 0.10.Of these, 61.1% result from a single missed slot and 38.9% from multiple missed slots.
  • Collapsed action instances: 61.1% of collapsed action instances are caused by a single missed slot, showing that AR-Geo often identifies a one-field operational failure rather than accumulated boundary errors.The remaining 38.9% involve multiple missed slots.
  • Readback safety judgment: Task 2 tests readback safety judgment by labeling correctness, error type, risk level, and the affected slot rather than relying on string similarity.The affected slot is the highest-consequence wrong or missing field when one dominates.
  • Readback safety judgment: The taxonomy treats equivalent number formatting and harmless filler as operationally correct, while simultaneous independent critical errors receive EXTREME risk.MULTI_ERROR is considered less recoverable through a single clarification.

G.2 Task 1 Fine-Tuning

Task 1 fine-tuning compares standard cross-entropy with consequence-weighted training for structured slot extraction. Risk-aware loss substantially improves AR-Geo and heading recall by concentrating training emphasis on safety-critical fields.

  • Training comparison: Heading recall increases from 20.8% to 67.7% under the risk-aware objective.The improvement is attributed to higher token weighting for heading spans.
  • Risk-aware objective: Task 2 applies the same principle at the output-field and sample-severity levels, up-weighting risk judgment fields and dangerous under-recognition cases.Risk-level, error-type, and affected-slot outputs receive elevated weights, with EXTREME examples multiplied by 2.0.
  • Risk-aware objective: The risk-aware trainer replaces mean cross-entropy with per-token weighted cross-entropy derived from operational criticality.Altitude, heading, callsign, and runway receive the highest Task 1 slot weights, while lower-risk fields receive smaller weights.
  • Expert validation: The expert questionnaire uses 40 controller responses to validate slot weights, action criticality assumptions, readback taxonomy, and nonlinear metric design.Ratings are normalized within question blocks and discretized into reproducible criticality tiers.
  • Expert validation: Controllers’ responses support preserving critical units over relying only on surface span overlap for operational acceptability.The paper reports this conclusion alongside the questionnaire’s support for nonlinear, risk-proportional penalties.

H.4 Recognition-Error and AR-Geo Preference

Expert ratings establish a severity ordering for recognition errors and support a nonlinear consequence-aware metric. Prompt ablations favor aligned prompting, while domain transfer requires new expert-defined schemas and validation rather than direct ATC score transfer.

  • Recognition-error severity: Wrong or missed altitude has the highest mean severity at 8.45, followed by heading at 8.20 and callsign at 7.20.Speed and frequency are rated lower, at 3.85 and 3.62 respectively.
  • Metric preference: Controllers prefer nonlinear, risk-proportional penalties, with mean preferences of 5.45 for nonlinear scoring and 5.50 for risk-proportional penalties on a 1–7 scale.These responses support AR-Geo but do not establish it as the only possible nonlinear form.
  • Prompt ablations: Full-Aligned prompting produces the best error-type F1 and the lowest downgrade rates across four models.The Knowledge prompt has comparable risk-level accuracy.
  • Domain transferability: Consequence-aware evaluation can be adapted to other domains by defining domain actions, mapping slots to criticality roles, assigning risk weights, and aligning failure taxonomies.The scoring formula remains unchanged, but domain-specific definitions are required.
  • Domain transferability: The driving-command illustration is not a validated cross-domain benchmark and provides no cross-domain model scores.It is presented as a recipe requiring redefinition and expert validation before use elsewhere.

K Prompt Templates and Inference Protocol

The protocol specifies JSON-only outputs for two ATC tasks, with constrained action and slot labels, structured spans, and safety judgments. It documents zero-shot and knowledge-augmented prompt conditions, validation rules, and an illustrative transfer recipe beyond ATC.

  • K.1 Task 1 Output Contract: Task 1 requires one JSON object containing the original utterance, one or more actions, and labeled token spans.The schema includes utterance, utterance_action, and tokens fields.
  • K.1 Task 1 Output Contract: Task 1 uses fixed action labels and semantic slot labels, including callsign, runway, altitude, speed, heading, frequency, condition, relation, attribute, and O.Functional relation and attribute tokens are mapped to O, with the same normalization and scorer used for all systems.
  • K.2 Task 2 Output Contract: Task 2 returns a JSON safety judgment with correctness, error type, risk level, affected slot, and explanation fields.Its taxonomy contains ten error types and four risk levels: CORRECT, HIGH, CRITICAL, and EXTREME.
  • K.2 Task 2 Output Contract: All protocol conditions require valid JSON without free-form text outside the object.The requirement applies across the documented prompting conditions.
  • K.3 Task 1 Zero-Shot Prompt: Zero-shot Task 1 prompting defines action meanings, requires array-form actions, and permits multiple actions while forbidding labels outside the allowed set.The prompt also specifies chunk-level spans copied from the utterance, with non-overlap, original order, and full coverage.
  • K.3 Task 1 Zero-Shot Prompt: The appendix lists the prompt inventory and gives an illustrative driving-command transfer recipe without reporting a second benchmark or cross-domain scores.The recipe is intended to show what must be redefined and expert-validated for another domain.
  • K.3 Task 1 Zero-Shot Prompt: The zero-shot prompt requires continuous, non-overlapping chunks in left-to-right order that cover every word exactly once.A silent self-check verifies the original utterance, allowed labels, span constraints, and multi-intent ordering before output.
  • K.4 Task 1 Knowledge Prompt: The knowledge-augmented Task 1 prompt preserves the JSON schema and allowed actions while adding aviation-oriented definitions for callsigns, traffic, taxiways, waypoints, headings, conditions, and attributes.These definitions distinguish directly addressed aircraft from referenced traffic and clarify operational slot meanings.
Loading 2608.24621v1…