Source-linked AI summary
Coverage-Driven Verification for Safety-by-Design in AI-Based Collision Avoidance Systems
Thomas Stefani, Johann Maximilian Christensen, Elena Hoemann, Frank Köster, Sven Hallerbach
TL;DR
Aviation AI assurance requires evidence that ODD data are representative, yet a structured way to assess distributional representativeness remains largely unexplored. This paper proposes a two-stage ODD coverage assessment using KL divergence and Cramér’s V; in the simulations, χ2 was unusable at large sample sizes while Cramér’s V stayed below 0.1 across the bin-count sweep.
Problem
EASA objectives require representative data, while ODD range coverage alone does not establish that the data follow the prescribed distribution.
Source: §3.2 · p. 4, §5 · p. 14Method
The paper proposes a two-stage representativeness criterion combining an ODD-partition population gate with KL divergence and Cramér’s V.
Source: §5 · p. 14, §6 · p. 15Results
χ2 was unusable at large simulation-derived sample sizes, while Cramér’s V stayed below 0.1 across the bin-count sweep.
Source: §4.3 · pp. 10–11, §6 · p. 15Takeaways & Limitations
ODD range population alone does not establish representativeness; the proportions in which the input space is sampled must also be assessed.
Source: §5 · p. 14The current bin-coverage gate requires each bin to be populated but does not ensure that per-bin frequencies are estimated with sufficient statistical precision.
Source: §5 · p. 15
Abstract
from arXiv · showhide
Artificial Intelligence (AI) offers significant potential for future aviation systems; however, its integration into safety-critical applications requires compliance with the aviation sector's stringent safety standards. For AI and Machine Learning (ML)-based systems, the European Union Aviation Safety Agency (EASA) emphasizes the need to demonstrate the representativeness and completeness of the Operational Design Domain (ODD) and the associated data distributions used during development and verification. Despite this requirement, a structured engineering process for defining target distributions and evaluating representativeness within ODDs remains largely unexplored. This work presents a method for representativeness assessment of AI/ML constituent ODDs in the context of aviation safety assurance. Starting from the methodical identification of suitable target distributions, a process flow is proposed that guides developers from ODD definition and parameter distribution modeling to the quantitative assessment and interpretation of coverage results with respect to EASA's learning assurance objectives. As quantitative measures, the chi-squared goodness-of-fit test is examined and found unsuitable for the large data sets arising in this setting, leading to the adoption of the Kullback--Leibler divergence and Cramér's $V$ for the representativeness assessment. The method is demonstrated using the example of AI-based airborne collision avoidance, employing experimental data from previous Horizontal Collision Avoidance System (HCAS) and Vertical Collision Avoidance System (VCAS) simulations. The results illustrate how statistical distribution comparison methods can support the assessment of representativeness for safety-critical AI applications and contribute toward a systematic Safety-by-Design AI engineering process aligned with emerging EASA guidance.
1. Introduction
Aviation’s growing use of AI makes verification-data representativeness within the defined ODD central to EASA learning assurance. The paper proposes a statistical assessment process and demonstrates it with HCAS and VCAS simulation data.
1. Introduction
- EASA requires evidence that AI/ML development and verification data are complete and representative of distributions within the constituent ODD.The ODD defines the operational conditions and input space in which the system is intended to operate safely.Source: §1 · p. 1
- Prior HCAS and VCAS scenario-generation work provides experimental data, while developers still lack a clear process for defining target distributions and assessing representativeness.The paper proposes a process flow and investigates the chi-squared goodness-of-fit test, KL divergence, and Cramér’s V for the assessment.Source: §1 · p. 2
- The paper reviews ODD coverage methods, introduces a two-step assessment loop, applies it to HCAS and VCAS, and discusses findings before concluding with future-work directions.The discussion focuses on the limited suitability of chi-squared testing at large scale and the complementary roles of KL divergence and Cramér’s V.Source: §1 · p. 2
2. State of the Art
EASA’s learning-assurance agenda centers on demonstrating full ODD coverage, but a usable process for doing so remains loosely defined. Existing work largely measures input-space completeness rather than whether data follow ODD-prescribed distributions.
2. State of the Art
- Prior aviation work combined the W-shaped process with DevOps, while Werner methodically derived the AI/ML constituent OD and ODD.For AI-based systems, ODD coverage shifts attention from structural code metrics to the neural network’s input space, with completeness and representativeness as complementary properties.Source: §2 · p. 2
- Weissensteiner used boundary-constrained k-means to reduce scenario counts, but did not account for attribute interactions and resulting high dimensionality.The approach proposes n-dimensional scenario-parameter sampling to support automated-driving-system validation.Source: §2 · p. 2
- Diemert used combinatorial testing and expert judgment to reduce the modeled ODD parameter space, without quantifying how representatively scenarios populated it.The CACTus framework systematically models the ODD input space and derives a manageable scenario suite.Source: §2 · p. 2
- Hirschle found convex-hull and hypersphere metrics limited by poor high-dimensional scaling or overcoverage, then proposed a scalable KDE metric that localizes low-coverage regions.The hypersphere metric could yield coverage values above one depending on the chosen radius.Source: §2 · p. 3
- ForMuLA assessed flight-regime durations against Gaussian ODD targets using goodness-of-fit tests, but reduced representativeness to a significance-level pass/fail verdict.The authors noted that visually matching data could be rejected, changing the significance level required justification, and small or empty expected bins or limited data weakened reliability.Source: §2 · p. 3
- The literature still lacks a comprehensive process for assessing completeness and representativeness together, particularly robustly across a high-dimensional ODD.The ForMuLA assessment demonstrated statistical representativeness testing in an EASA context but used a single low-dimensional feature and a single significance-test outcome.Source: §2 · p. 3
3. Method for ODD Coverage Assessment
The method structures ODD coverage verification around distributional representativeness and parameter-combination completeness, addressing EASA’s high-level objectives with a parameter-level representativeness process and quantitative measures.
3. Method for ODD Coverage Assessment
- The method addresses the gap in scalable, quantitative ways to demonstrate ODD completeness and representativeness for AI/ML constituents under EASA guidance.Prior work proposes a completeness method, but the section identifies representativeness assessment as lacking a generic, scalable process.Source: §3 · p. 3
3.1 Method Overview
- The two-step assessment checks representativeness first and completeness second, iterating to augment data or target uncovered parameter combinations until both criteria are met.Representativeness must precede completeness because adding data to address distribution gaps changes the dataset’s combinatorial mix.Source: §3.1 · pp. 3–4, Figure 1 · p. 4
3.2 Data Representativeness
- The process operationalizes EASA’s DA-04, DM-07-2, and DM-08 objectives by comparing each ODD parameter’s observed data with its prescribed univariate distribution.It treats pairwise parameter combinations as the separate completeness pillar, assumes sampling independence between data sets, and does not characterize joint ODD density beyond the two coupled properties.Source: §3.2 · p. 4
3.2.1 Target Distribution
- Target distributions are derived, in order, from representative real-world data, reference simulations, physically motivated assumptions, or expert knowledge and standards; unavailable sources leave a parameter undefined.The rationale and origin of each chosen distribution are documented for traceability under EASA MOC DM-07-2.Source: §3.2.1 · p. 5, Figure 2 · p. 6
3.2.2 Representativeness via Kullback–Leibler Divergence
- For discrete samples, KL divergence compares observed and target distributions; empty bins receive ε = 10^-6 before renormalization to keep the divergence finite and reproducible.Lower D_KL indicates a better fit; proposed interpretation ranges are engineering guidance, not regulatory limits.Source: §3.2.2 · p. 5
3.2.3 Representativeness via Chi-Squared
- The chi-squared goodness-of-fit test compares observed and target counts across bins, using α = 0.05 and requiring expected counts commonly of at least 5 per bin.Although applicable to varied target distributions, its statistic scales with sample size, so large datasets can make negligible deviations significant.Source: §3.2.3 · p. 7
3.2.4 Representativeness via Cramér’s V
- Cramér’s V normalizes χ^2 by sample size to measure deviation magnitude, avoids large-sample inflation, and provides an indicator independent in construction from KL divergence.V = 0 indicates matching distributions; it is unbounded above for this goodness-of-fit framing and must be interpreted at fixed bin resolution, with Cohen-based guidance rather than regulatory thresholds [24, 25].Source: §3.2.4 · pp. 7–8
4. Case Study: Horizontal and Vertical AI-Based Collision Avoidance
The case study applies ODD coverage assessment to HCAS and VCAS, showing that broad input-space coverage does not necessarily mean the observed data match the intended distributions.
4.1 ODD Definition
- The constituent ODDs are defined by HCAS’s seven and VCAS’s five neural-network input state variables, excluding the previous advisory for simplicity.The analysis focuses on the networks’ immediate input spaces; parameter values were adapted from prior work and adjusted for the demonstration’s scenario settings.Source: §4.1 · pp. 8–9
4.2 Simulation Setup and Data Collection
- Each HCAS and VCAS dataset contains 1.97 million simulated state-variable rows from randomized trajectories beginning about 55 seconds before CPA, with avoidance maneuvers initiated around 30 seconds before CPA.The prior simulations were not designed to demonstrate representativeness; their data serve as an exemplary basis for the assessment method.Source: §4.2 · p. 9
4.3 Coverage Results for Representativeness
- χ² p-values stayed far below 0.05 at every VCAS τ bin count, whereas Cramér’s V stayed below 0.1; large-sample sensitivity made p-values unusable, so KL divergence and Cramér’s V were retained.Cramér’s V stabilized as bin counts increased, supporting its use as a normalized effect size alongside KL divergence.Source: §4.3 · pp. 10–11, Figure 4 · p. 11
- HCAS relative heading ψ was the sole parameter rated Fail by both DKL (1.1599) and Cramér’s V (0.5563), while broad-support parameters were Excellent on both.The broad-support parameters were VCAS τ and HCAS ρ, vint, and τ.Source: Table 6 · p. 11, Table 7 · p. 12, §4.3 · p. 12
- For VCAS, h and ˙hown reached 92.96% and 87.80%; for HCAS, ρ, vint, and τ exceeded baseline ranges at 173.95%, 469.36%, and 272.62%, while θ and ψ each attained 100%.Source: §4.3 · p. 11
- Against a normal target across the ODD range, ODDVCAS h had DKL=1.0382 (Fail) despite 92.96% range utilization and 89.65% bin coverage; against a uniform target within 0–40 s, τ had Cramér’s V=0.0013 (Excellent).Source: §4.3 · p. 12
- Representativeness requires 100% bin coverage first, then both DKL and Cramér’s V must meet the parameter’s safety-tier tolerance; failure of the coverage gate precludes divergence evaluation.Bin counts are fixed by the sensitivity analysis rather than adjusted to obtain full coverage.Source: §4.3 · p. 13
5. Discussion
The method formalizes ODD-data representativeness as distinct from combinatorial completeness, using a population gate followed by distributional checks rather than relying on range coverage or a single statistic.
5. Discussion
- Method formalizes ODD-data representativeness for EASA learning assurance, while combinatorial completeness is out of scope and developed separately.The two properties together address EASA’s coupled representativeness and completeness expectations.Source: §5 · pp. 13–14
- Because range coverage can mask distorted sampling proportions, the method gates on populated bins before comparing distributions; large-sample χ2 was rejected even for well-matched τ as p-values collapsed toward zero.The χ2 test rejected the null hypothesis for every parameter in this setting, leaving it without discriminatory power as a representativeness gate.Source: §5 · p. 14
- KL divergence captures missing target mass and shape or tail mismatches, while bounded Cramér’s V summarizes overall deviation; requiring both guards against failures either measure alone could miss.They concur for every parameter passing the bin-coverage gate. With unpopulated bins, both rate ψ Fail, while KL fails h, ḣown, and θ and Cramér’s V rates them only Good to Moderate.Source: §5 · pp. 14–15
- The current bin gate ensures range sampling but not reliable frequency estimates, and Cramér’s V tolerance bands are not yet tied to each parameter’s safety-relevance tier.Choosing the number of bins is important for the required granularity; safety-tier anchoring is needed before the criterion can support a formal certification argument.Source: §5 · p. 15
- Representativeness findings depend on assumed targets: broad normal references partly drive Fail ratings for h and ḣown, while operational surveillance distributions could strengthen the claim.The reusable assessment loop is aligned with EASA guidance, and combining bin coverage with KL divergence and Cramér’s V gives a more complete picture than any single metric.Source: §5 · p. 15
6. Conclusion and Outlook
The paper proposes a two-stage assessment of ODD data representativeness for safety-critical AI, demonstrated on neural-network-based HCAS and VCAS. It finds that complementary coverage and distribution metrics are needed, and identifies completeness and more safety-specific criteria as future work.
6. Conclusion and Outlook
- The case study finds χ2 unusable on large simulation-derived data, while bin coverage, KL divergence, and Cramér’s V jointly provide a more reliable assessment than any measure alone.Bin coverage establishes that the ODD range is populated; KL divergence and Cramér’s V capture complementary aspects of distributional agreement.Source: §6 · p. 15
- The proposed two-stage criterion couples a data-population gate over the ODD partition with KL divergence and Cramér’s V, mapping to EASA learning-assurance expectations.The assessment addresses representativeness required by Objective DA-04 and Anticipated MOC DM-07-2, verified with completeness at the DM-08 data-verification step.Source: §6 · p. 15
- Future work includes formalizing completeness for combinatorial coverage of the joint ODD space and strengthening the bin-coverage gate with power-calibrated per-bin sample-size requirements.The present criterion does not address completeness; power-calibrated occupancy requirements would support reliable per-bin frequency estimates for distributional measures.Source: §6 · p. 16
- Further refinements would set safety-criticality-specific acceptance ranges, use empirical operational-data references, and investigate metrics better suited to ordered continuous variables.Tighter tolerances for safety-relevant parameters are a precondition for a formal certification argument.Source: §6 · p. 16
7. Contact Author Email Address · 8. Copyright Statement
The authors affirm ownership or permission for all original and third-party material and authorize publication and distribution of the paper in the ICAS proceedings or as off-prints.
8. Copyright Statement
- The authors affirm copyright ownership or permission for included material and authorize its publication and distribution through the ICAS proceedings or as individual off-prints.They confirm that permission has been obtained from third-party copyright holders and from the paper's copyright holder where needed.Source: §8 · p. 16