Source-linked AI summary
Coverage-Driven Verification for Safety-by-Design in AI-Based Collision Avoidance Systems
Thomas Stefani, Johann Maximilian Christensen, Elena Hoemann, Frank Köster, Sven Hallerbach
TL;DR
Safety-critical aviation AI needs evidence that development and verification data represent the ODD, yet a structured engineering process for defining target distributions and assessing this property remains underdeveloped. The paper proposes a two-stage, EASA-aligned assessment using target-distribution identification, an ODD population gate, and complementary divergence measures. In the collision-avoidance case study, chi-squared was unsuitable for large simulation data sets, whereas KL divergence and Cramér’s V supported representativeness assessment.
Problem
A structured engineering process for defining ODD target distributions and evaluating data representativeness for safety-critical aviation AI remains insufficiently defined despite EASA requirements.
Method
The paper proposes an EASA-aligned two-stage method that identifies target distributions and combines an ODD population gate with KL divergence and Cramér’s V.
Results
The case study found that chi-squared was unusable as a decision criterion for large simulation-derived data sets, motivating KL divergence and Cramér’s V as retained measures.
Takeaways & Limitations
Representativeness assessment must examine sampling proportions and use complementary measures rather than relying on ODD range population or a single statistic.
Takeaways & Limitations
The demonstration defines target distributions from expert knowledge because representative real-world or dedicated simulation data may be time-intensive or infeasible during early investigations.
Abstract
from arXiv · showhide
Artificial Intelligence (AI) offers significant potential for future aviation systems; however, its integration into safety-critical applications requires compliance with the aviation sector's stringent safety standards. For AI and Machine Learning (ML)-based systems, the European Union Aviation Safety Agency (EASA) emphasizes the need to demonstrate the representativeness and completeness of the Operational Design Domain (ODD) and the associated data distributions used during development and verification. Despite this requirement, a structured engineering process for defining target distributions and evaluating representativeness within ODDs remains largely unexplored. This work presents a method for representativeness assessment of AI/ML constituent ODDs in the context of aviation safety assurance. Starting from the methodical identification of suitable target distributions, a process flow is proposed that guides developers from ODD definition and parameter distribution modeling to the quantitative assessment and interpretation of coverage results with respect to EASA's learning assurance objectives. As quantitative measures, the chi-squared goodness-of-fit test is examined and found unsuitable for the large data sets arising in this setting, leading to the adoption of the Kullback--Leibler divergence and Cramér's $V$ for the representativeness assessment. The method is demonstrated using the example of AI-based airborne collision avoidance, employing experimental data from previous Horizontal Collision Avoidance System (HCAS) and Vertical Collision Avoidance System (VCAS) simulations. The results illustrate how statistical distribution comparison methods can support the assessment of representativeness for safety-critical AI applications and contribute toward a systematic Safety-by-Design AI engineering process aligned with emerging EASA guidance.
1. Introduction
AI is expanding into safety-critical aviation applications, while EASA guidance makes representativeness and completeness central to learning assurance. A structured process for assessing representativeness within ODDs remains insufficiently defined.
- AI-based aviation applications promise substantial efficiency gains but introduce new challenges for safety assurance and certification.
- Collision avoidance is a prominent safety-critical use case, including simplified ACAS X systems such as HCAS and VCAS.
- Previous work derived ODD-based scenarios and integrated HCAS and VCAS into pyCASX for evaluation in the FlightGear simulator.
- The resulting experimental data provide a foundation for investigating representativeness assessment methods for AI-based collision avoidance systems.
2. State of the Art
Prior work connected aviation ODD descriptions with scenario generation and explored geometric or density-based input-space coverage metrics. These approaches quantify spatial coverage rather than whether sampled data follow a prescribed distribution.
- EASA’s AI roadmap and concept paper established ODD coverage within a W-shaped learning-assurance framework, but its translation into an engineering process remained loosely defined.
- Prior coverage methods examined convex hulls, per-sample hyperspheres, and kernel-density estimation to quantify how much of the input space is covered.
- These approaches address input-space extent rather than whether the collected data follow the target distribution prescribed for the ODD.
3. Method for ODD Coverage Assessment
The method operationalizes EASA ODD coverage objectives through a structured assessment loop that separates representativeness from completeness and evaluates distributions quantitatively. It derives target distributions from available evidence and compares observed data with those targets using complementary statistics.
- 3.1 Method Overview: The coverage assessment takes a formally defined AI/ML Constituent ODD and a dataset as inputs, then evaluates coverage through an iterative two-step process.The process addresses representativeness and completeness as distinct properties.
- 3.2 Data Representativeness: Representativeness assesses whether data are sampled according to the distribution prescribed for each ODD parameter, while completeness concerns coverage of the joint parameter space.The proposed method formalizes the representativeness property; combinatorial completeness is outside this work’s scope.
- 3.2.2 Representativeness via Kullback–Leibler Divergence: Target distributions are identified systematically from real-world data, simulation data, statistical or physical assumptions, and expert knowledge.The process begins with a formally defined ODD parameter and selects among available evidence sources.
- 3.2.2 Representativeness via Kullback–Leibler Divergence: KL divergence quantifies the difference between target and observed distributions, but discrete evaluation requires binning and regularization of empty observed bins with ε = 10^-6.The observed distribution is renormalized after assigning the floor probability.
- 3.2.3 Representativeness via Chi-Squared: The chi-squared test compares observed and expected bin counts, but its statistic grows with sample size, causing large data sets to reject negligible deviations.The test uses expected counts n·p_i and a conventional significance level α = 0.05.
- 3.2.4 Representativeness via Cramér’s V: Cramér’s V normalizes χ2 by sample size and table dimensionality, yielding an effect-size measure that supports magnitude-based interpretation alongside KL divergence.The goodness-of-fit framing uses q = 2, and V is interpreted relative to a fixed bin resolution.
4. Case Study: Horizontal and Vertical AI-Based Collision Avoidance
The case study applies the representativeness assessment to HCAS and VCAS simulation data, defining constituent ODDs from neural-network input variables and comparing coverage against target distributions. It uses a bin-coverage gate together with KL divergence and Cramér’s V, showing that range or bin coverage alone can hide distributional mismatch.
- 4.1 ODD Definition: HCAS and VCAS define their AI/ML constituent ODDs from the neural networks’ immediate state-variable input spaces.HCAS uses horizontal-encounter variables, while VCAS uses vertical-encounter variables; the previous advisory is omitted for simplification.
- 4.2 Data Generation: The case study uses previously executed, highly automated HCAS and VCAS simulations that generate randomized trajectories and translate advisories into avoidance maneuvers about 30 s before CPA.The simulation outputs contain the corresponding HCAS or VCAS state variables used for coverage assessment.
- 4.2 Target Distributions: Target distributions are defined from expert knowledge and engineering assumptions, with their rationale and sources documented for traceability under EASA MOC DM-07-2.This choice supports an exemplary demonstration when representative operational data or dedicated simulation data are time-intensive or infeasible to obtain.
- 4.3 Coverage Results for Representativeness: The Pearson χ2 test is discarded as a representativeness criterion because large-sample significance depends on sample size and bin granularity, whereas Cramér’s V remains stable and below 0.1 across the bin sweep.KL divergence and Cramér’s V are therefore retained as complementary distributional measures in the coverage reports.
- 4.3 Coverage Results for Representativeness: VCAS achieves 100 % bin coverage for τ but only 64.35 % for ownship vertical rate, while HCAS fully covers ρ, v_int, and τ and reaches 99.41 % for θ and 95.29 % for ψ.Distributional results distinguish broad agreement from mismatch: ψ is rated Fail by both measures with Cramér’s V = 0.5563, while several other parameters receive less severe ratings.
- 4.3 Coverage Results for Representativeness: The two-stage criterion first requires 100 % bin coverage and then evaluates distributional agreement using KL divergence and Cramér’s V.The bin count is fixed by sensitivity analysis rather than adjusted to satisfy the coverage gate.
- 4.3 Coverage Results for Representativeness: The altitude example demonstrates that high coverage can mask substantial distributional mismatch, so populating the ODD range does not by itself establish representativeness.Both coverage and distributional agreement are needed to assess the coupled expectations of EASA’s learning-assurance objectives.
5. Discussion
The work formalizes ODD representativeness as one pillar of EASA-aligned coverage assessment and combines complementary statistical measures to evaluate distributional agreement. It finds that χ2 is unsuitable for the large data sets considered and identifies scope and calibration limitations for certification use.
- The method formalizes representativeness assessment while leaving combinatorial completeness of the joint ODD parameter space outside this work.
- Cramér’s V normalizes χ2 into a bounded, sample-size-independent effect size, while KL divergence remains sensitive to shape, tails, and unrealized target probability mass.
- Requiring both KL divergence and Cramér’s V to remain within tolerance guards against failure modes that either measure alone could miss.
- Current limitations include a minimal bin-population gate, bin-count sensitivity, and qualitative Cramér’s V tolerances not yet tied to parameter safety relevance.
6. Conclusion and Outlook
The conclusion presents a two-stage, reusable representativeness criterion for safety-critical AI ODDs and demonstrates it on neural-network-based HCAS and VCAS. It identifies sample-size-robust, complementary metrics as necessary and outlines extensions toward completeness, reliable bin sizing, and certification-oriented thresholds.
- The proposed criterion couples an ODD population gate with KL divergence and Cramér’s V to assess representativeness in HCAS and VCAS.
- χ2 loses usability on the large simulation-derived data sets, so representativeness decisions require sample-size-robust effect-size measures.
- Bin coverage populates the ODD range, whereas KL divergence and Cramér’s V capture complementary distributional properties; their joint use gives a more reliable quality assessment than either alone.
- The assessment loop is generic, reusable, and directly mapped to EASA learning-assurance expectations for data representativeness.
- Future work should formalize completeness, strengthen bin gates with power-based sample requirements, and replace qualitative thresholds with safety-criticality-specific acceptance ranges.
8. Copyright Statement
The copyright statement confirms ownership or permission for the original and third-party material included in the paper.
- The authors confirm copyright ownership or publication permission for the paper’s original and third-party material.