Developing Security Metrics for Automated Compliance Checking

• Changing security states make technical representations of requirements necessary for continuous, automated conformance checking.Standards and policies need requirements expressed as measurable technical properties.Source: §1 · p. 1
• Inconsistent data models, abstraction levels, and reference points make metrics hard to compare or combine, impede completeness, and create redundancy and duplicated mapping effort.Source: §1 · p. 2
• The paper proposes a general requirements model, a metric data model, and an ontology-based elicitation method linking requirements and metrics.It builds on an expanded resource ontology and discusses connecting the approach with structured standards such as OSCAL.Source: §1 · p. 2

2.1 Terminology

• Here, a standard broadly means an authoritative document prescribing product requirements, while a security metric reproducibly maps observable product properties to a value indicating an aspect of security level.The term standard includes certification schemes, benchmarks, and control catalogues.Source: §2.1 · p. 2

2.2 Security Metrics

• Prior security-metric work examines quality criteria and classifies existing metrics, whereas this paper engineers new metrics from standards using a resource- and feature-centric ontology.The paper’s model and elicitation process target structured, machine-readable representation for automated, continuous certification.Source: §2.2 · pp. 3–4

2.3 Continuous Certification and Measurement Approaches

• Continuous-certification projects established foundations, but their metric definitions and links to standards remained project-specific rather than using a reusable model and systematic elicitation method.Earlier work also demonstrated technical measurements and combined indicators, but defined metrics ad hoc and without a generic data model.Source: §2.3 · p. 4

3.1 A Requirement Decomposition Pattern

• Requirements are decomposed into six elements, termed a systematized requirement; target-resource selection can be ambiguous or implicit.The elements are designation, action, target resource type, means, scope, and purpose.Source: §3.1 · p. 5

3.2 Modeling Metrics

• The data model supports logical combinations of target properties and distinguishes long-lived metadata from shorter-lived configuration data.It also calls for purpose and context descriptions and versioned identifiers.Source: §3.2 · p. 6
• The semi-structured metric model requires ontology references while remaining human-readable, and each metric can have a machine-readable Rego specification.The two representations support stakeholder-specific interfaces and cross-checking between documentation and executable logic.Source: §3.2 · pp. 7–8

3.3 Elicitation Steps

• Elicitation decomposes requirements, maps relevant properties to ontology concepts, and formalizes evaluation logic; some requirements may still need manual evidence.The process identifies properties, extends the ontology when needed, and implements evaluation in Rego; the authors do not claim every requirement can be translated comprehensively into technical metrics.Source: §3.3 · pp. 8–9
• Ontology references support metrics at comparable abstraction levels, reveal coverage gaps, and help integrate metrics from different sources.The method also aims to prevent duplicate metrics for the same property.Source: §3.3 · p. 9

3.4 Example

• In the C5 example, an implicit resource type is made explicit as resources offering TransportEncryption, and the metric defines a configurable protocol target.The cloud-customer-data and public-network limitation is handled by the applying compliance system rather than embedded in the metric definition.Source: §3.4 · pp. 9, 11

3.5 A Collaborative Security Metrics Repository

• The ontology-based method supports collaborative metric development through an open-source repository governed by the Cybersecurity Certification Hub.Multiple EU-funded projects contribute, and the repository is being extended with metrics related to several standards.Source: §3.5 · p. 11

4.1 Practical Implications

• The repository is integrated into Confirmate for automated processing; top-down metrics favor requirement coverage, while meaningful bottom-up metrics can map to existing controls.The discussion also notes that EUCS calls for regular audits and C5 acknowledges automation potential in many requirements.Source: §4.1 · p. 11

4.2 Limitations

• Automated metrics may invite Goodhart-style optimization and cannot fully capture complex security, while human audits may overlook less tangible aspects.Metrics can stop measuring security effectively when auditees optimize for them as key performance indicators; human auditors may favor easily verifiable indicators.Source: §4.2 · p. 12
• Flexible metric granularity supports varied assessment contexts, but automated checks cannot encompass all prose requirements; comparing coverage with traditional audits remains future empirical work.Metrics range from low-level configuration checks to high-level aggregators evaluating complex testing procedures.Source: §4.2 · p. 12

4.3 Complementing OSCAL

• OSCAL's control and assessment models can connect machine-readable controls and human-oriented assessment activities to ontology-linked technical measurements.Its control statements suit atomic requirements, while assessment activities can link to metric IDs and asset types can link to the ontology; integrating the models would span high-level controls to technical measurements.Source: §4.3 · pp. 12–13

4.4 Towards a Semi-Formalized Certification Scheme

• EUCS CKM-03 could become a documentation requirement plus an enabled-at-rest-encryption check on storage holding CSC data.The example separates documenting encryption mechanisms from configuring AtRestEncryption as enabled on Storage resources.Source: §4.4 · p. 13
• Ontology-based formulations retain human readability and contextual flexibility while specifying target resources and properties, simplifying metric derivation and supporting automated conformance tools.Structured resource descriptions in ontologies enable automated tooling for conformance checking.Source: §4.4 · p. 13

5 Conclusions

• Ontology-based standardization addresses inconsistent data models, abstraction differences, metric redundancy, and duplicated mapping across standards.The paper presents a metric data model, an elicitation methodology, and an open-source repository as parts of this approach.Source: §5 · p. 13
• Collaborative metric development is a benefit of the framework, and the authors have begun that work in an open-source repository.The paper also proposes a path toward semi-formalized standards using the ontological basis.Source: §5 · p. 13
• Future work will develop more metrics, map them to standards, evaluate the approach in practice, and explore LLM-assisted translation of prose requirements.The authors identify the structured templates, data models, ontology, and methodology as the basis for this planned work.Source: §5 · p. 13