PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Comprehensive Risk Assessments and Assurance of AI-Based Systems

Heidy Khlaaf

arXiv:2606.29390v1cs.CY

TL;DR

AI risk research has not fully addressed the limitations of borrowed safety and security methods, and inconsistent terminology can mislead safety claims. This paper aligns assurance terminology and proposes an ODD-based framework with an AI-specific taxonomy to operationalize risk assessment. It also identifies limits on determining AI failure likelihood and on inferring behavior from model design.

  • Problem

    AI risk work has adopted safety and security methods whose limitations and terminology differences can hinder their correct application and mislead stakeholders about safety claims.

  • Method

    The paper proposes consistent assurance terminology and an end-to-end AI risk framework integrating ODDs and a taxonomy for general AI systems.

  • Results

    The framework uses concrete operational envelopes to support determination of AI-system hazards, harms, risks, and required mitigations within defined conditions.

  • Takeaways & Limitations

    ODDs provide a basis for operationalizing AI risks under selected application domains so developers and auditors can assess safety risks.

  • Takeaways & Limitations

    Determining hazard or harm likelihood remains beyond the paper’s scope and requires sophisticated risk metrics tailored to specific subject domains.

Abstract

from arXiv · show

Novel safety, socio-economic, and ethical harms arising from the deployment of AI-based systems have led to a breadth of work seeking to map, measure, and mitigate against newly found risks. These works have heavily leveraged techniques and terminology from the fields of System Safety Engineering and Cybersecurity, yet they have fallen short in accounting for the limitations and nuances that reduce the efficacy and correct application of adopted methodologies. Furthermore, misuse of terminology entailing compliance with established safety and security properties can mislead stakeholders with regard to the claims an AI system satisfies and provide a false sense of safety. In this paper, we seek to align overlapping, AI-adjacent communities on a consistent and comprehensive assurance terminology crucial for the safe deployment of AI-based systems. We outline why previous attempts to adapt risk assessment techniques and terminology from the safety and security fields have been insufficient. We then propose a novel end-to-end AI risk framework that integrates the concept of an Operational Design Domains (ODD), initially introduced for ADS (Automated Driving Systems) [1], for more general AI-based systems. The purpose of an ODD is to provide a description of the specific operating conditions for which an AI-system is designed to properly behave, thus outlining the safety envelope for which system hazards and harms can be determined against. We believe that by defining a more concrete operational envelope, developers and auditors can better assess potential risks and required safety mitigations for AI-based systems.

Executive Summary

AI risk work has adopted safety and security methods without fully accounting for their limitations, while inconsistent terminology can mislead stakeholders about system safety. The paper proposes consistent assurance terminology and an ODD-based risk framework to define operating conditions and support risk assessment.

  • AI risk assessments have borrowed System Safety Engineering and Cybersecurity techniques but often overlook limitations affecting their efficacy and correct application.
  • The framework integrates Operational Design Domains into AI risk assessment and introduces an AI-relevant ODD taxonomy for general multimodal models.
  • ODDs describe the operating conditions under which an AI system is designed to behave properly, defining a safety envelope for assessing hazards and harms.
  • The paper distinguishes safety from alignment, arguing that safety concerns harm arising from system intent rather than merely satisfying intended values.
  • Hardware safety techniques based on random failures are unsuitable for AI systems whose systematic failures arise from design issues.
  • The paper recommends system-level frameworks such as MIL-STD-882e because they address hazards, harms, systematic failures, and emergent behaviors relevant to AI.

1. Introduction

AI systems have shown limited robustness, difficult-to-measure capabilities, and harms associated with deployment, while terminology has produced contradictory safety approaches. The paper proposes ODD-based risk assessment to explore application-specific scenarios and risks.

  • Despite increasing general-domain performance, machine-learning models have demonstrated limited robustness and capabilities that are difficult to measure.
  • Insufficient requirements and risk analyses have accompanied AI deployment with safety hazards and novel socio-economic and ethical harms.
  • Inconsistent use of safety terminology can equate safety measures with meeting system intent, producing contradictory risk and hazard assessments.
  • The paper proposes ODDs and a novel AI-based taxonomy to explore scenarios and associated risks across application domains.

2. Distinguishing Value Alignment, Safety, and Risk

The paper distinguishes value alignment from safety, arguing that safety addresses harms arising from system intent or failures, while risk assessment evaluates hazards by probability and severity.

  • Distinguishing Value Alignment and Safety: AI literature uses “safety” inconsistently, sometimes referring to accident prevention and sometimes to alignment with human values.
  • Distinguishing Value Alignment and Safety: Value Alignment is a system requirement because it expresses goals and behaviors intended to satisfy subjective human values.
  • Distinguishing Value Alignment and Safety: Safety concerns hazards or harms caused by a system meeting its specifications or failing while attempting to meet them.
  • Distinguishing Value Alignment and Safety: Alignment alone is insufficient for safety because risk assessments and additional oversight remain necessary even when system intent is intended to prevent harm.
  • Risk Terminology: Hazards are harmful conditions, risks combine hazard probability and severity, and threats concern adversarial effects on system confidentiality, integrity, or availability.
  • Risk Terminology: Risk assessments identify, evaluate, and report system risks, whereas risk management continuously addresses organizational risks and lies beyond this paper’s scope.
  • Risk Terminology: The paper adopts distinct meanings for errors, faults, failures, and failure modes to support consistent risk analyses.Errors are erroneous states or actions; faults manifest errors in software; failures prevent required function or specified performance; failure modes are function manifestations of failure.

3. Pitfalls in Existing Adoptions and Approaches

Existing AI risk work often adopts safety and cybersecurity techniques without accounting for their intended scope, assumptions, or limitations. The paper argues for operationalized, system-level risk assessment and assurance methods suited to AI’s systematic, emergent, and nondeterministic harms.

  • Cross-cutting pitfalls: AI risk studies have adopted techniques including FMEA, threat modeling, bug bounties, and red teaming, but community silos and implicit practitioner knowledge hinder correct application.The paper emphasizes that techniques are often transferred without the broader processes and contexts for which they were designed.
  • Hardware safety techniques: Hardware safety techniques such as FMEA are ill-suited to ML because they assume random component failures and do not capture systematic, emergent, or system-level behavior.Hardware methods rely on deterministic functions and measured failure rates, whereas ML behavior is nondeterministic and increasingly complex.
  • Cybersecurity techniques: Cybersecurity threat modeling is misapplied when used to measure harms imposed by AI systems, because it protects systems from adversarial environments rather than analyzing system-originated hazards.The paper also notes that DREAD and CVSS score vulnerabilities and can produce subjective risk scores rather than systematic AI hazard assessments.
  • Cybersecurity techniques: AI red teaming commonly probes LLMs for harmful outputs, but this is safety-oriented boundary or stress testing rather than cybersecurity red teaming of adversarial readiness.The distinction matters because cybersecurity red teaming evaluates an organization’s ability to detect and respond to staged attacks.
  • System and software safety: System safety and software assurance methods face AI-specific limits because ML design determines how a model learns, not what it learns, while novel nondeterministic failures resist quantification.Static analysis, formal verification, and comprehensive testing are described as non-transferable or still under development for ML models.
  • Operationalization and assurance: Prior risk frameworks and hazard catalogs remain difficult to operationalize across applications because their hazard sources are too general to systematically identify domain-specific hazards, harms, and consequences.The proposed blueprint aims to operationalize risk modeling so assurance claims, criticality, harms, and appropriate mitigations can be determined more systematically.

4. Unifying Risk Assessment and Safety Justification

The paper adapts system-level risk assessment for AI-based systems by defining operational envelopes through ODDs and a baseline taxonomy. This framework supports hazard identification across applications, users, vectors, protected characteristics, and assets, while acknowledging that AI hazard likelihood remains unresolved.

  • The proposed AI risk assessment guides system design and mitigation by defining tolerable risk criteria for identified hazards and harms.The approach aims to help developers and auditors build confidence that safety risks have been addressed in implementation and deployment as far as reasonably practicable.
  • AI Operational Design Domain: ODDs describe the operating domains in which an AI system is designed to properly operate, establishing an envelope for assessing risks and required safety functionality.The paper extends ODDs from automated driving systems to more general AI-based systems, including multi-modal models.
  • AI Operational Design Domain: A baseline AI ODD taxonomy organizes operational conditions into Application/Domain, Users/Agents, Vector, Protected Characteristics, and Assets.The categories provide a starting point that organizations can extend with domain-specific categories and subcategories.
  • AI Operational Design Domain: The taxonomy covers varied deployment contexts, human roles, attack surfaces, protected groups, and system assets across the AI system life cycle.Examples include applications such as automotive, manufacturing, marketing, education, infrastructure, science, and advocacy; users include data labelers and human-in-the-loop agents.
  • Operationalizing Risk Assessments: The paper treats determining hazard likelihood in AI systems as outside its scope because model design determines how a model learns, not what it learns.Developing adequate risk metrics requires techniques spanning machine learning and social science for specific subject domains.
  • Operationalizing Risk Assessments: Risk identification considers every defined ODD category within operational scenarios, even when a category is not initially perceived as risky.Risk assessments are carried out across abstraction levels and by multidisciplinary teams to improve coverage of potential harms and hazards.

5. Conclusive Remarks

The paper argues that AI assurance requires consistent terminology and techniques, because existing adaptations from hardware safety, cybersecurity, and system safety do not directly transfer. It proposes ODD-based assurance to define concrete operational envelopes for assessing risks and mitigations.

  • Consistent assurance terminology is needed to align objectives and avoid equating safety measures with requirements or value alignment.
  • The proposed ODD taxonomy covers Application/Domain, Users/Agents, Vector, Protected Characteristics, and Assets to explore scenarios and associated risks.The taxonomy is intended for AI technologies, including general multi-modal models.

Appendix A

The appendix provides reference materials for hazard severity, hazard risk indexing, and structuring risk assessments with ODD categories.

  • Table 1 presents Hazard Severity Categories associated with general multi-modal models.
  • Table 2 presents a Hazard Risk Index that considers hazard frequency against severity category.
  • Table 3 provides a risk assessment template using ODD categories as fields.
Loading 2606.29390v1…