PaperlayerView exact revision on arXiv

Source-linked AI summary

JTA: Joint Testability Architecture for Scenario-Based Validation of Safety-Critical Software

Wenyao Xue, Jiandi Wang, Yichen Wang

arXiv:2608.05594v1cs.SEcs.RO

TL;DR

Existing testability research is largely artifact-centric, while safety-critical validation also depends on scenario construction, evidence alignment, and fault attribution. JTA analyzes the scenario, test system, and system under test together, using capability dimensions and bridge-oriented design. In the ArduPilot illustration, link-loss validation is comparatively mature, while GCS-timeout and EKF-anomaly scenarios retain substantial blind spots.

  • Problem

    Existing testability research typically treats the software artifact as its analysis object, leaving joint scenario-based validation capability across the scenario, test system, and system under test insufficiently addressed.

    Source: §1 · p. 1, §2.4 · p. 2

  • Method

    JTA models the scenario, test system, and system under test as a joint entity and uses scenario contracts, three capability dimensions, and bridge-oriented actions to assess and address gaps.

    Source: §1 · pp. 1–2, §3.3 · p. 3, §3.7 · p. 5

  • Results

    In the ArduPilot assessment, current test assets cover RC link loss reasonably well, but substantial joint-adequacy blind spots remain for GCS timeout and EKF anomaly scenarios.

    Source: S3 EKF · p. 8

  • Takeaways & Limitations

    The case suggests that validation difficulty can reflect mismatches among scenario requirements, test-system capability, and exposed system hooks, with bottlenecks differing by scenario.

    Source: §5.5 · p. 9

    The work is a method-and-architecture study rather than a report of gains after bridge actions, and its illustrative H/M/L ratings involve engineering judgment.

    Source: §6.3 · p. 9

Abstract

from arXiv · show

Validation adequacy in safety-critical software depends on more than the system under test. Critical scenarios must be constructed under controlled conditions, execution evidence must be aligned into verdict-ready form, and abnormal outcomes must be attributable to actionable causes. Existing testability research remains largely artifact-centric and offers little architectural support for reasoning about the combined capability of the scenario, the test system, and the system under test. Joint Testability Architecture (JTA) addresses this gap by treating those three elements as a single object of analysis and design. It characterizes validation capability along three dimensions--controllability, observability, and isolability--and organizes them through three domains, three bridges, and an analysis-design-evaluation-refinement loop. JTA also introduces scenario contracts, joint capability assessment, validation blind-spot identification, and bridge-oriented design actions that map capability gaps to concrete improvements in control points, evidence organization, and attribution boundaries. An illustrative analysis of ArduPilot failsafe validation shows that link-loss scenarios are comparatively mature, whereas state-estimation anomaly scenarios remain harder to validate because evidence alignment and attribution semantics are weaker. JTA is not a replacement for existing testing or safety-analysis techniques; it provides an architectural basis for modeling, designing, and assessing scenario-based validation in safety-critical software.

1. INTRODUCTION

Safety-critical validation depends on more than test-case availability: scenarios must be constructed, evidence organized for verdicts, and abnormal outcomes attributed to actionable boundaries. JTA treats the scenario, test system, and system under test as a joint, scenario-conditioned capability to assess and improve.

1. INTRODUCTION

  • Validation difficulties often concern stable scenario construction, verdict-ready evidence, and actionable attribution rather than a simple lack of test cases.The paper frames these as system-level capabilities shaped jointly by the scenario, test system, and system under test.Source: §1 · p. 1
  • Artifact-centric testability is insufficient to determine whether existing systems can adequately validate failsafe and other safety-critical scenarios.The paper asks whether the test system, scripts, simulation, and exposed hooks jointly support scenario-based validation.Source: §1 · p. 1
  • The research questions address unifying the scenario, test system, and system under test, operationalizing three capability dimensions, and translating gaps into design actions.The dimensions are controllability, observability, and isolability.Source: §1 · p. 1
  • JTA defines the joint testability system as a scenario-instantiated entity composed of the scenario, test system, and system under test.This expands the validation object beyond an isolated software artifact.Source: §1 · p. 1
  • JTA combines scenario contracts and joint adequacy assessment with bridge-oriented design actions, illustrated through ArduPilot failsafe validation.The actions target control, evidence, and attribution bridges; the case identifies blind spots in link-failure and state-estimation anomaly scenarios.Source: §1 · p. 2

2. BACKGROUND AND RELATED WORK

Prior work contributes artifact-level testability, safety-scenario analysis, and explicit test modeling, but does not jointly assess scenario, test-system, and system-under-test capability. JTA positions itself as an architectural response that operationalizes adequacy and maps blind spots to design actions.

2.1 Artifact-Centric Testability Research

  • Artifact-centric testability research emphasizes controllability, observability, and design intervention, while usually treating the software artifact as the analysis object.The cited foundations include early testability studies, ISO/IEC 25010, design-level work,,, and MBSE/DFT.Source: §2.1 · p. 2

2.2 Safety-Critical Verification and Fault-Driven Assessment

  • Safety-verification methods address hazards, scenarios, and evidence, but do not directly establish whether existing test systems and software jointly support adequate validation.JTA is presented as complementary: fault injection can construct abnormalities but does not itself ensure evidence structuring or attribution convergence.Source: §2.2 · p. 2

2.3 Test Architecture, Model-Based Testing, and UTP

  • MBT, UTP, standards, and contract-based validation model test artifacts or activities; JTA instead targets joint capability gaps and architectural actions.UTP may provide a representation path, but modeling-notation completeness is not JTA’s methodological core.Source: §2.3 · p. 2

2.4 Research Positioning

  • JTA addresses the stated gap by jointly analyzing the scenario, test system, and system under test, operationalizing adequacy, and mapping blind spots to architectural actions.The positioning names these as three requirements not simultaneously met by existing approaches.Source: §2.4 · pp. 2–3

3. ARCHITECTURE AND FORMAL MODEL

JTA models scenario-based validation as a joint architecture spanning the scenario, test system, and system under test, then assesses capability gaps and directs bridge-level design actions. Its contracts and review protocol make controllability, observability, and isolability assessable against scenario requirements.

3.1 Architecture Definition

  • JTA formalizes the architecture as three domains, three bridges, and a closed loop for analysis, design, evaluation, and refinement.The domains represent scenario intent, test-system capability, and system-under-test exposure; the bridges connect them.Source: §3.1 · p. 3, §3.7 · p. 5

3.2 Joint Testability System

  • For a critical scenario, the joint testability system includes the participating test capabilities and the relevant system-under-test boundary, rather than an isolated software artifact.At project level, it is defined over the critical-scenario set, the test system, and the system under test.Source: §3.2 · p. 3

3.3 Scenario Contracts

  • A scenario contract specifies objectives, constraints, stimulation, minimum evidence, verdict rules, fault sources to distinguish, and capability thresholds.Explicitly including the fault-source set makes isolability a design constraint tied to attribution obligations and actionable evidence.Source: §3.3 · p. 3

3.4 Operationalizing the Three Capability Dimensions

  • JTA assesses controllability, observability, and isolability by comparing required scenario obligations with the control, evidence, and attribution points exposed by the system.Observability concerns whether relevant facts can be seen; isolability concerns whether those facts distinguish candidate fault sources into actionable causes.Source: §3.4 · pp. 3–4

3.5 Scoring and Review Protocol

  • A trace-matrix review extracts required and exposed points, maps system-side coverage to H/M/L ratings, and keeps expert judgment auditable.The rubric assigns H at 80% or greater coverage with no critical point missing, M at 50–80% or partly ambiguous coverage, and L below 50% or with a missing critical point.Source: §3.5 · p. 4, Table I · p. 4

3.6 Joint Capability, Scenario Adequacy, and Validation Blind Spots

  • Joint capability uses the minimum across test-system and system-under-test capabilities, making the weaker side a bottleneck; adequacy and capability gaps identify validation blind spots.The project-level measure summarizes criticality-weighted adequacy coverage across critical scenarios.Source: §3.6 · pp. 4–5

3.7 Three Domains, Three Bridges, and a Closed Loop

  • The Scenario Domain organizes contracts and requirements, the Test-System Domain supplies validation functions, and the System-Under-Test Domain exposes control, evidence, and responsibility points.JTA assigns distinct roles to control, evidence, and attribution bridges because cross-boundary mismatches require different design responses.Source: §3.7 · p. 5
  • Bridge actions target the identified gap: control mechanisms produce repeatable stimuli, evidence mechanisms align evidence with verdict rules, and attribution mechanisms align sources, causes, and boundaries.The analysis-design-evaluation-refinement loop rechecks adequacy after bridge additions and feeds remaining gaps into the next modeling round.Source: §3.7 · p. 5

4. METHODOLOGY

The methodology identifies critical scenarios, assesses their joint capabilities and gaps, and maps those gaps to bridge-oriented design actions. It supports both retrofitting existing systems and co-construction during system design.

4.1 Five-Step Analysis Workflow

  • The five-step workflow identifies and prioritizes critical scenarios, instantiates their contracts, assesses test-system and system capabilities, computes gaps, and applies bridge design with re-evaluation.Scenarios with nonzero capability gaps enter design; re-evaluation continues until minimum requirements are met or residual risk is explicitly explained.Source: §4.1 · pp. 5–6

4.2 Capability Gaps and Bridge Design Patterns

  • Table III maps representative symptoms of the three capability-gap categories to corresponding design patterns.The workflow uses gap components to select actions on the control, evidence, or attribution bridge.Source: §4.2 · p. 6, §4.1 · p. 6

4.3 Two Adoption Modes

  • JTA can retrofit bridge mechanisms around critical scenarios in existing systems or use scenario contracts early to constrain system design.The reported case study uses retrofitting, while the method supports both adoption modes.Source: §4.3 · p. 6

5. CASE: ARDUPILOT

The ArduPilot case applies JTA to three failsafe scenarios using scenario-scoped assets, ordinal capability judgments, and bridge-level improvement paths. It finds link-related validation comparatively mature, while GCS-timeout and EKF-anomaly scenarios retain distinct capability gaps.

5.1 Case Objectives and Analysis Protocol

  • The case assesses failsafe-related assets with a unified H/M/L rubric and reports capability judgments and design paths, not post-action performance gains.Source: §5.1 · pp. 6–7

5.2 Scenario Contract Instantiation

  • Each scenario contract records its objective, stimulation mode, minimum evidence, and explicit fault-source set; EKF anomalies receive greater criticality weight (κS3 = 1.5; κS1 = κS2 = 1.0).Source: §5.2 · p. 7

S2 GCS

  • For GCS heartbeat timeout, the central gap is attribution: logs and cause semantics do not reliably distinguish a genuine timeout from test-toolchain control error.Source: §5.3 · p. 7

S3 EKF

  • EKF anomalies require finer perturbation control and cross-component semantic attribution, exposing controllability and isolability bottlenecks.Source: §5.3 · p. 7, S3 EKF · p. 8

5.3 Joint Capability Assessment

5.4 Representative Capability Gaps and Bridge Design Actions

  • Bridge-oriented actions make capability gaps traceable to engineering changes; for S3, the focus is controllable anomaly construction and clearer responsibility boundaries, not indiscriminate logging.Source: §5.4 · p. 8, Table VI · p. 9

5.5 Findings and Discussion

  • The case links validation difficulty to mismatch among scenario requirements, test-system capability, and exposed hooks, supporting scenario-conditioned rather than single system-level testability.Source: §5.5 · p. 9

6. DISCUSSION

JTA's novelty is a scenario-conditioned joint adequacy review that embeds fault-source obligations and maps capability gaps to bridge-level design actions. It shifts validation toward proactive architectural design, while the study remains bounded by illustrative assessment and the ArduPilot case.

6.1 Novelty and Research Positioning

  • JTA's novelty is a scenario-conditioned joint adequacy review that embeds fault-source obligations and maps capability gaps to control, evidence, and attribution bridge actions.The contribution is not fault-source tracing alone, but its use within a joint review of the scenario, test system, and SUT.Source: §6.1 · p. 9

6.2 Implications for Safety-Critical Software Engineering

  • JTA treats validation as an architectural concern, organizing the test system, evidence, and responsibility boundaries proactively through scenario contracts.The paper connects this shift to assurance-case engineering, shift-left validation, cross-team collaboration, and lifecycle-oriented V&V.Source: §6.2 · p. 9

6.3 Limitations and Threats to Validity

  • The study demonstrates JTA's analysis and design organization, not performance gains or certification benefits after bridge actions are implemented.Researcher judgment remains in defining scenario boundaries and minimum evidence sets.Source: §6.3 · pp. 9–10
  • The current H/M/L assessment is an illustrative discretization; control-interface, log, and cause-code indicators remain candidates for further automation.The paper identifies automation of these indicators as future work.Source: §6.3 · p. 10
  • ArduPilot's openness and testing-infrastructure maturity exceed those of many industrial projects, limiting direct generalization of the case conclusions.Applicability to closed systems with restricted control interfaces, logging, or evidence disclosure requires further case-based validation.Source: §6.3 · pp. 9–10
  • JTA is a meso-level architectural method that complements, rather than replaces, formal verification, certification processes, and concrete testing frameworks.It provides a unified perspective for organizing validation capability across them.Source: §6.3 · p. 10

7. CONCLUSION

JTA treats testability as a scenario-conditioned capability shared by the scenario, test system, and SUT, then maps capability gaps to bridge-level design actions. Its ArduPilot analysis identifies different bottlenecks for link-related and EKF anomaly scenarios, supporting a joint rather than code-only view of validation.

7. CONCLUSION

  • JTA defines a joint testability system, scenario contracts, capability assessment, adequacy, blind-spot identification, and bridge-oriented actions for control, evidence, and attribution gaps.The framework organizes these through three domains, three bridges, and a closed loop.Source: §7 · p. 10
  • Link-related scenarios are bottlenecked by evidence structuring and explicit cause semantics, while EKF anomaly scenarios face anomaly-construction and cross-component-attribution gaps.The ArduPilot analysis describes distinct bottlenecks across the scenario types.Source: §7 · p. 10
  • The analysis indicates that safety-critical software's testing difficulty is not merely code-level, but involves mismatches among scenario requirements, testing infrastructure, and exposed system hooks.Source: §7 · p. 10
  • Future work will refine capability indicators in additional industrial cases, validate bridge-action benefits, and explore integration with UTP, assurance cases, and model-driven engineering.Source: §7 · p. 10
Loading 2608.05594v1…