PaperlayerView exact revision on arXiv

Source-linked AI summary

Making AI Compliance Evidence Machine-Readable

Rodrigo Cilla Ugarte, Miguel Ángel Patricio Guisado, Antonio Berlanga de Jesús, José Manuel Molina López

arXiv:2604.13767v1cs.CY

TL;DR

AI governance frameworks specify what organizations must assure but lack executable formats and infrastructure for producing current evidence. This paper extends OSCAL with 16 AI-specific properties and implements a three-layer compliance architecture that captures and evaluates evidence during ML runs. Across two EU AI Act high-risk systems, the SDK produces machine-readable, schema-validated OSCAL evidence and linked remediation artifacts, while its scope remains limited to selected domains and Articles 9–15.

  • Problem

    AI governance frameworks define assurance obligations but lack executable formats, standardized evidence artifacts, and ML-pipeline architectures for producing that evidence.

  • Method

    The paper extends OSCAL with AI lifecycle, enforcement, traceability, and risk-acceptance properties and implements policy, evidence, and enforcement layers.

  • Results

    A single pipeline execution generates machine-readable OSCAL Assessment Results, POA&M documents, and independently verifiable probe artifacts, validated against the NIST JSON schema.

  • Takeaways & Limitations

    Compliance evidence can be produced as a version-controlled byproduct of training rather than reconstructed after an audit, enabling earlier feedback on data quality and fairness.

  • Takeaways & Limitations

    The architecture covers Articles 9–15 and tabular or volumetric imaging scenarios, leaving organizational requirements, NLP, LLMs, recommender systems, and GPAI systemic-risk models outside scope.

Abstract

from arXiv · show

AI Assurance -- producing the machine-readable evidence required to demonstrate compliance with AI governance frameworks -- has mature policy scaffolding but lacks the infrastructure to operationalize it. Organizations building high-risk AI systems under the EU AI Act face a gap: frameworks such as the EU AI Act, ISO/IEC 42001, and NIST AI RMF specify what to assure but provide no executable format for how. This paper proposes OSCAL -- the NIST standard adopted for FedRAMP cybersecurity compliance -- as a candidate interchange format for AI governance, complementing rather than replacing the emerging JTC21 standards stack. We define 16 property extensions covering lifecycle phases, enforcement semantics, risk traceability, and risk-acceptance justification, and present a three-layer Compliance-as-Code architecture (policy, evidence, enforcement) that generates assurance evidence as a byproduct of model training. The SDK produces native OSCAL Assessment Results validated against the NIST JSON schema. We test the approach on two Annex III high-risk systems: a credit scoring model and a medical imaging segmentation system. The architecture and reference implementation are open-source under Apache 2.0.

INTRODUCTION

AI governance frameworks define what organizations should assure but lack executable evidence formats and infrastructure. This paper adapts OSCAL, extends it for AI assurance, and validates the approach on two EU AI Act high-risk systems.

  • AI governance frameworks specify assurance obligations but not executable controls, evidence artifacts, or architectures integrated with ML pipelines.Teams commonly rely on manually authored documentation disconnected from training code, leaving audit evidence outdated.
  • OSCAL provides a cybersecurity precedent for version-controlled controls and structured evidence, while AI governance lacks an equivalent infrastructure.FedRAMP adoption established OSCAL as a compliance-as-code standard, and RegTech adoption reportedly reduced compliance costs by 40–70%.
  • The paper extends OSCAL with 16 AI lifecycle properties and builds policy, evidence, and enforcement layers that generate compliance evidence during model training.The approach is tested on a credit scoring model and a medical imaging segmentation pipeline under the EU AI Act.

BACKGROUND

AI assurance must reconcile overlapping frameworks, binding evidence obligations, and existing documentation or platform tools that are not interoperable. The paper positions a standardized interchange format as the missing layer.

  • A single AI deployment may face overlapping EU AI Act, DORA, NIS2, GDPR, ISO/IEC 42001, and NIST AI RMF obligations with different authorities and deadlines.Table 1 illustrates how one concern, bias, receives different terminology and documentation expectations across frameworks.
  • EU AI Act compliance requires providers to demonstrate evidence, while ISO/IEC 42001 supplies organizational governance rather than machine-readable technical evidence.The first harmonised standard is not yet cited in the Official Journal, so providers cannot claim presumption of conformity under Article 40.
  • Model Cards, FactSheets, and governance platforms address documentation or workflow needs but do not provide portable, continuous evidence streams tied to ML training runs.The paper identifies an interchange format as the missing layer between governance requirements and auditor-verifiable pipeline evidence.

OSCAL AS LINGUA FRANCA FOR AI GOVERNANCE

The paper uses OSCAL as an extensible, serializable compliance language for AI governance, adding properties that encode lifecycle behavior, enforcement, traceability, and defensible thresholds. A single profile can serve multiple consumers and frameworks while remaining compatible with OSCAL tooling.

  • OSCAL AS LINGUA FRANCA FOR AI GOVERNANCE: OSCAL provides six serializable models—catalog, profile, component definition, assessment plan, assessment results, and POA&M—for the compliance lifecycle.These models support version control alongside source code, and FedRAMP mandates OSCAL for cloud service authorizations.
  • An OSCAL Profile for AI Assurance: The paper introduces an AI OSCAL profile because equivalent catalogs, profiles, and assessment plans for EU AI Act high-risk systems do not exist.The profile adds 16 properties expressing what to evaluate, when and how, why, and how thresholds are justified.
  • Why Not a New Format?: Extending OSCAL preserves existing validation, conversion, and institutional adoption while allowing one file to reference EU AI Act, NIST AI RMF, and ISO/IEC 42001 controls.Unknown properties are ignored, preserving backward compatibility across consumers.
  • Lifecycle Semantics: Lifecycle properties let one control apply across training, validation, monitoring, and incident phases while consumers select the subset matching their scope.Training checks operate on raw data, validation checks on predictions, monitoring checks on production traffic, and incident controls feed Article 73 reporting.
  • Lifecycle Semantics: Evaluation windows and enforcement modes encode when controls run and whether failure blocks execution, warns operators, or logs silently.Per-run is the default for training and validation; periodic and sliding-window evaluation serve runtime monitoring.
  • An OSCAL Profile for AI Assurance: Traceability properties connect each technical control to policy, objectives, risks, treatments, and corrective actions within an ISO/IEC 42001 PDCA hierarchy.A failed metric can trace from its observed value and threshold through remediation, risk, objective, and policy.

ARCHITECTURE

The architecture separates policy, evidence, and enforcement for pre-market assurance under Articles 9–15. During training or inference, probes capture artifacts, enforcement evaluates OSCAL controls, and failures become structured risks and remediation items.

  • ARCHITECTURE: The three-layer architecture targets pre-market evidence for Articles 9–15, while post-market monitoring and incident reporting remain outside the work’s infrastructure scope.Figure 1 places policy, evidence, and enforcement around the pre-market pipeline.
  • ARCHITECTURE: OSCAL assessment plans separate data policies evaluated before training from model policies evaluated after training, with metrics, operators, thresholds, lifecycle phases, and enforcement modes.Figure 2 illustrates a training-time gender disparate-impact control using the Four-Fifths Rule and blocking enforcement.
  • Evidence layer: The evidence layer activates seven concurrent probes covering code, data integrity, software supply chain, environment, hardware, carbon, and enforcement verification.The probes map to EU AI Act Articles 9–12 and 15 and address documented gaps in ML evidence capture.
  • Evidence layer: A context-managed training run captures AST traces, SHA-256 hashes, a CycloneDX bill of materials, and environment fingerprints without modifying the training code.The monitor activates probes in parallel, while enforcement loads the YAML policy and caches evaluation results.
  • Enforcement layer: Evidence is stored as native OSCAL Assessment Results validated against the NIST JSON schema, with failed controls generating linked POA&M risk items.The enforcement engine emits Observations and Findings, then records metric values, thresholds, operators, and affected demographic groups in risks.

VALIDATION

The architecture was validated on credit-scoring and medical-imaging systems, using the same OSCAL-based monitoring and enforcement flow to generate machine-readable assurance evidence. In credit scoring, mitigation improved gender parity while an age-disparity failure produced traceable remediation artifacts; the medical scenario generated cohort-level evaluation evidence for a pre-trained model.

  • Credit scoring: 0.012 gender demographic parity was achieved after reweighting while accuracy held at 0.795, but age disparity remained at 0.286 against a 0.50 threshold.The age failure would halt training under block enforcement and required a subsequent mitigation cycle.
  • Credit scoring: The failed age control generated an OSCAL Risk, detailed metric facets, an open POA&M item, and per-group rates of 35.16%, 27.68%, and ratio 0.787.These breakdowns provide evidence beyond a pass/fail result for audit verification.
  • Medical imaging: The medical-imaging pipeline evaluated Dice, sensitivity, and specificity across patient cohorts, producing 37 OSCAL Observation rows.The same flow captured model-weight hashes, dependency inventory, and CUDA environment metadata for a pre-trained MONAI/PyTorch model.
  • Cross-scenario validation: Both scenarios used the same monitor() integration, OSCAL policy format, probes, and enforcement engine despite different training and inference pipelines.Only policy content and lifecycle targeting changed between the tabular credit model and volumetric imaging model.
  • Generated evidence: A single execution generated locally stored OSCAL Assessment Results, POA&M documents, BOMs, hashes, environment fingerprints, and carbon traces that auditors could independently verify.The evidence bundle was machine-readable and colocated with the run artifacts.

DISCUSSION

The architecture is presented as infrastructure for shifting compliance toward pipeline-generated evidence and scalable review, with extensions toward runtime monitoring and agentic systems. Its current scope remains limited to pre-market Arts. 9–15 and two validated application domains.

  • Implications for engineering teams: Evidence generated during pipelines can provide immediate fairness and data-quality feedback, enabling remediation before market placement rather than after an audit.The architecture frames compliance as an amortized pipeline cost instead of recurring ex-post reconstruction.
  • Implications for regulators and auditors: Machine-readable OSCAL results could support programmatic conformity assessment and auditor dashboards once recognized in sector-specific guidance or notified-body procedures.Structured Findings, Risks, and POA&M items can surface failing controls and affected demographic groups without parsing raw JSON.
  • Implications for standards bodies: A future NIST OSCAL Profile for AI could unify NIST AI RMF sub-categories with EU AI Act articles in one catalog.The proposed extensions are positioned as a starting point for standardization alongside emerging CEN/CENELEC work.
  • From pre-market to post-market monitoring: The lifecycle_phase and evaluation_window properties provide hooks for extending the same OSCAL source of truth from pre-market assurance to post-market monitoring and incident reporting.A runtime layer could emit the same Assessment Results structure for sliding-window drift and demographic-parity checks.
  • Limitations: Validation covers only tabular and volumetric imaging scenarios, while organizational requirements, multiregulation, NLP, LLMs, recommenders, and systemic-risk GPAI remain outside scope.The paper also notes that OSCAL JSON requires a parallel human-readable layer for public accountability.

CONCLUSION

The paper positions AI compliance as an infrastructure problem and shows OSCAL-based evidence generation across two high-risk AI systems. Its machine-readable, version-controlled outputs are produced during training, with implications for engineering teams, regulators, and standards bodies.

  • OSCAL-based evidence is machine-readable, version-controlled, and auditor-ready, produced as a byproduct of training rather than a separate exercise.The SDK produces native Assessment Results validated against the NIST JSON schema.
  • The approach uses the same OSCAL policies, probes, and enforcement engine for credit scoring and medical imaging systems.
  • The paper frames compliance as an infrastructure problem for engineering teams and argues that machine-readable evidence can scale for regulators.
  • The OSCAL extensions are presented as a concrete starting point for standards bodies working on prEN 18286.
Loading 2604.13767v1…