PaperlayerView exact revision on arXiv

Source-linked AI summary

CCR: Towards a Common, Quality-Gated CACAO Integrations Registry for European Cybersecurity Automation

Mateusz Zych, Vasileios Mavroeidis, Gudmund Grov

arXiv:2609.27567v2cs.CR

TL;DR

CACAO represents cybersecurity playbooks in a portable, machine-readable form, but reusable product-specific bindings are still needed to invoke external tools. The paper introduces CCR and a hybrid OpenAPI-to-CACAO pipeline; its evaluation finds that rule-based translation preserves mechanical API structure at least as well, while the LLM adds bounded semantic enrichment.

  • Problem

    CACAO represents playbook logic, but authors and automation environments still need reusable, inspectable product-specific bindings to connect actions to tool capabilities.

    Source: §1 · p. 2

  • Method

    The paper introduces CCR, a provenance-aware registry of CACAO connector envelopes, seeded by a hybrid pipeline that combines deterministic structural processing with bounded LLM semantic decisions.

    Source: §1 · p. 3, §3 · p. 6

  • Results

    The rule-based mapper equals or exceeds the LLM pipeline on mechanical checks, while the LLM contributes bounded semantic enrichment; all 713 envelopes passed CACAO v2 schema validation.

    Source: §5.1 · p. 11, §7 · p. 17

  • Takeaways & Limitations

    CCR provides an initial common foundation for quality-gated CACAO integration candidates, with acceptance evidence recorded separately from operational readiness.

    Source: §7 · p. 17

    Back-validation and the rule-based comparison assess artefacts against the same source OpenAPI document, and independent expert assessment of operational adequacy was not completed.

    Source: §6 · p. 16

Abstract

from arXiv · show

Standardised, machine-readable cybersecurity playbooks provide a basis for portable, shareable, and reusable incident-response logic. OASIS CACAO provides a vendor-neutral representation for such playbooks, but not the product-specific integration artefacts needed to invoke external products and services. We introduce the Common CACAO Registry (CCR), an open, provenance-aware registry of CACAO HTTP-API connector envelopes. Each envelope captures an API operation's command, inputs, target, authentication-related information, provenance, validation evidence, and maturity metadata. CCR is quality-gated, with acceptance requiring both CACAO v2 schema validity and a mean back-validation score of at least 0.8 against the source OpenAPI operation, while a six-level maturity model records progressively stronger evidence and distinguishes gate acceptance from operational readiness. To seed CCR, we develop a hybrid OpenAPI-to-CACAO pipeline. Deterministic code extracts source-derived interface facts, generates identifiers, wires cross-references, and validates structure, while a constrained LLM provides bounded semantic enrichment, including action naming, authentication interpretation, and CACAO activity annotation. Evaluation across eight security APIs yields 713 CACAO-schema-valid envelopes with a mean back-validation score of 91.5%, of which 675 produce well-formed, dispatchable HTTP requests in a local harness. Comparison with a deterministic rule-based baseline shows that mechanical API structure is preserved more reliably through rule-based translation, while the LLM contributes bounded semantic enrichment, most notably CACAO activity annotation. Together, these results support CCR as reusable integration infrastructure for CACAO action steps and as an initial foundation for a broader common European registry.

1 Introduction

Incident-response playbooks need maintainable product-specific bindings to invoke external security tools, while CACAO supplies the portable playbook representation but not those bindings. The paper introduces CCR and a hybrid generation pipeline to create and quality-gate reusable CACAO integrations.

1 Introduction

  • Integrations bind playbook actions to product interfaces, commands, inputs, targets, and authentication, creating a maintenance burden as security products evolve.These bindings connect manual or automated response activities to concrete capabilities in organisational environments.Source: §1 · p. 2
  • CCR is an open, versioned, provenance-aware registry of CACAO HTTP-API connector envelopes, with shared quality controls and contribution processes.Each envelope represents one API operation and records provenance, validation evidence, assurance metadata, and maturity information; local users complete product-specific configuration.Source: §1 · p. 3
  • The hybrid pipeline assigns structural tasks to deterministic code and bounded semantic decisions to an LLM, then compares the pipeline with a rule-based baseline.Semantic decisions include action naming, variable enrichment, authentication interpretation, and CACAO activity annotation.Source: §1 · pp. 2–3
  • The paper specifies CCR with explicit quality gates, a six-level maturity model, an open contribution model, and a layered validation framework.The validation layers include schema validation, back-validation, a deterministic baseline, dispatch testing, and limited pipeline execution.Source: §1 · p. 3
  • The authors connect CCR to European cybersecurity policy and argue that shared integration infrastructure can reduce duplicated engineering effort for NIS2- and resilience-relevant capabilities.They frame this contribution in relation to wider EU interoperability objectives.Source: §1 · pp. 3–4

2 Motivation and Background

Security-tool integrations are costly to build and maintain, and their assumptions can become outdated as APIs change. The section frames reusable CACAO bindings as a technical and policy-relevant response to this problem.

2 Motivation and Background

  • 251 of 2,224 studied OpenAPI specifications introduced breaking changes, and 87.3% of those changes did not first deprecate the superseded operation.The section uses this finding to illustrate the maintenance challenge for product-specific integrations as security tools and APIs change.Source: §2 · p. 4
  • A reusable integration translates provider-oriented OpenAPI operations into CACAO bindings that preserve interface facts while expressing actions for playbook use.HTTP methods, paths, and declared parameters can often be derived mechanically; variables, authentication, targets, and operational semantics require additional decisions.Source: §2 · p. 4
  • The European policy context includes cybersecurity risk management, incident reporting, resilience, product cybersecurity, coordinated response, interoperability, and reuse of digital capabilities.The section cites NIS2 and other EU instruments, while noting that these objectives do not prescribe a particular platform, standard, or registry.Source: §2 · pp. 4–5
  • CCR applies the shared-infrastructure principle of the European Vulnerability Database to reusable integration knowledge maintained across organisations and automation environments.Table 1 maps selected EU regulatory and policy objectives to CCR capabilities.Source: §2 · p. 5, Table 1 · p. 5

3 The CCR Generation Pipeline

The pipeline divides OpenAPI-to-CACAO translation between deterministic structural work and bounded LLM semantic decisions, then validates generated envelopes against the schema and source operations.

3 The CCR Generation Pipeline

  • The hybrid design keeps identifier generation, cross-reference construction, and validation deterministic while using the LLM for semantic decisions not directly determined by OpenAPI.The LLM handles choices such as variable naming, type enrichment, and vocabulary-grounded activity annotation.Source: §3 · p. 6
  • Stage 1 normalises extracted endpoints, and Stage 2 uses CACAO context and a typed output schema to produce semantic content.The model is routed by endpoint complexity; identifiers remain under deterministic control.Source: §3 · p. 6
  • Stages 3–5 construct the envelope, validate CACAO v2 schema conformance, and compare it with the source OpenAPI operation on nine dimensions.Schema failures can trigger up to two correction rounds; failures after correction are flagged for manual review.Source: §3 · p. 7
  • Each endpoint produces one self-contained JSON connector envelope containing CACAO objects, back-validation scores, and generation telemetry.Complete playbooks can combine multiple envelopes into workflows.Source: §3 · p. 7

3.1 Worked Example

  • The CVE lookup example passes all nine applicable back-validation checks after first-attempt schema validation, with its identifier supplied as a CACAO variable.The operation defines no security requirement, so the generated envelope contains no authentication object.Source: §3.1 · p. 8
  • Request-body composition is bounded to depth 6; flattening anyOf/oneOf branches over-approximates their alternatives, and callback-dependent operations remain unsupported.Flattening is exact for allOf, but the pipeline unions fields across anyOf/oneOf branches without using discriminator fields.Source: §3.1 · p. 8

4 The Common CACAO Registry

CCR is a public, versioned registry of CACAO HTTP-API connector envelopes with shared acceptance criteria and evidence-based maturity levels. Its seed corpus and demonstrations establish schema conformance and dispatch evidence, while operational governance and third-party SOAR integration remain beyond the evaluation.

4 The Common CACAO Registry

  • CCR is a public, versioned repository of CACAO HTTP-API connector envelopes, but the evaluation does not establish end-to-end integration with a third-party orchestration engine.The demonstrated scope is CACAO conformance and direct dispatch of underlying HTTP requests.Source: §4 · p. 8
  • CCR provides one envelope per endpoint alongside source specifications, hashes, validation evidence, an assurance schema, and per-API manifests.Contributors can submit generated envelopes or manually author or correct them under the same acceptance criteria; artefacts and materials use Apache-2.0 licensing.Source: §4 · pp. 8–9
  • Acceptance requires CACAO v2 schema validity and an overall back-validation score of at least 0.8; 82.5% (588/713) of envelopes qualify.The remaining envelopes are retained with a needs-review status.Source: §4 · p. 8
  • The seed corpus contains 713 CACAO-schema-valid envelopes across eight security APIs, with recorded back-validation evidence and a common open license.The APIs span vulnerability intelligence, case management, SOAR, network policy, and SIEM/EDR domains.Source: §4 · p. 9
  • The six-level maturity model assigns each artefact the highest level supported by its evidence; most envelopes reach Level 3 through dispatch testing.Live-tested CVE operations and the playbook demonstration reach Level 4, while higher levels require separate evidence and are not implied by the 91.5% corpus mean.Source: §4 · p. 9, Figure 2 · p. 9
  • The assurance block records provenance, validation, credentials and target constraints, impact class, review status, lifecycle, and known limitations, but does not itself provide operational governance.The threat model includes malicious or erroneous target URLs, excessive credential scope, and misclassification of destructive operations; governance controls remain future work.Source: §4 · p. 10
  • A two-action CACAO playbook combining a live CVE lookup with a mock TheHive alert creation passes schema validation and returns HTTP 200 and HTTP 201 in the local harness.The playbook was not executed through a CACAO-enabled SOAR platform.Source: §4 · p. 10

5 Evaluation

The evaluation finds that all 713 generated envelopes pass CACAO schema validation, with a 91.5% mean back-validation score; 675 also produce dispatchable requests in a local harness. The comparison supports deterministic mapping for mechanical API structure and bounded LLM enrichment for activity annotation, while execution and scope limitations qualify the results.

5 Evaluation

  • The evaluation covers 713 endpoints from eight heterogeneous security APIs and uses nine source-operation checks averaged into the back-validation score.The checks cover HTTP method and path, parameter coverage and placement, CACAO types and flags, authentication presence, and base-URL match; the headline score covers evidence layers 1–2.Source: §5 · pp. 10–11, §5.1 · p. 11, Table 4 · p. 11

5.1 Results

  • 713/713 envelopes passed CACAO v2 schema validation, and the mean back-validation score was 91.5%.The headline results summarize source-agreement checks; first-attempt validity was 95.5% (681/713), with the remaining envelopes passing after correction.Source: §5.1 · pp. 11–12, Table 6 · p. 12
  • Required-flag accuracy was 99.1% and authentication-presence accuracy was 99.9% across six observed security patterns; the check records whether an authentication object exists when the source declares a security requirement.It does not evaluate header placement, token format, or credential values.Source: §5.1 · p. 12
  • The raw path-check rate was 86.7%, but understates true path-level correctness: all 93 failures (all but one in TheHive) stemmed from mixed-case parameter names rejected by lowercase-only normalization.The regular expression matches OpenAPI {param} to CACAO __param__:value only for lowercase variable names, so equivalent endpoint representations receive different path scores.Source: §5.1 · p. 12

5.2 Analysis

  • Rule-Based Baseline matched or outperformed Bridge on each mechanical check, while Bridge annotated 52.7% of endpoints with an activity label and RBB assigned none.The comparison supports deterministic mapping for mechanical structure and bounded LLM enrichment, most clearly through activity annotation.Source: §5.2 · p. 13, Table 8 · p. 13
  • Back-validation compares artefacts with their source OpenAPI document, so it cannot alone establish operational usability or human-judged adequacy.Dispatch tests found defects missed by back-validation, and no independent expert assessment was conducted; repeated-run scores also varied for 12 of 22 sampled endpoints.Source: §5.2 · pp. 13–15
  • 675/713 (94.7%) envelopes produced well-formed, dispatchable requests locally, while all five live CVE requests returned HTTP 200 with the correct record.The other seven APIs required credentials or deployments unavailable to the study; request construction was validated locally for these APIs.Source: §5.2 · p. 14
  • 461/713 (64.7%) envelopes had target addresses that were not directly usable as absolute URLs, a condition the source-agreement base-URL check cannot detect.TheHive inherited a literal slash and Wazuh an unresolved URL template from their specifications.Source: §5 · p. 11, §5.2 · p. 14

6 Discussion and Related Work

Prior work addresses playbook representation, API recommendation, and API-to-task mapping, but not the construction of executable CACAO bindings. CCR combines that integration layer with an openly licensed, quality-gated registry, while the evaluation remains limited in representativeness, regeneration, and governance evidence.

6 Discussion and Related Work

  • APIRO ranks candidate APIs and IRP2API maps response-plan tasks to APIs, whereas this work generates and validates executable CACAO bindings for selected operations.The binding includes authenticated requests, parameter bindings, and target configuration, addressing a subsequent integration stage.Source: §6 · p. 15
  • The stated novelty combines AI-assisted OpenAPI-to-CACAO generation, schema-matching-style evaluation, an openly licensed quality-gated registry, and EU cybersecurity policy grounding.The authors state that they know of no prior work combining these elements.Source: §6 · p. 16
  • The observed inference cost was $0.026 per endpoint, excluding review and governance costs; applying one gate to AI-generated and manual contributions supports a multi-contributor registry.The authors frame this as support for a broader EU-wide registry, not as a measured effort-reduction result.Source: §6 · p. 16
  • The headline findings come from a single generation run, and full corpus regeneration is not possible because the prototype generator was not released.The artefact package supports inspection and reproduction of the registry artefacts, validation results, and execution traces.Source: §6 · p. 16
  • The eight-API convenience sample does not establish representativeness of security APIs generally, and the scope excludes GraphQL, gRPC, and proprietary protocols.Observed defect proportions are evidence that a defect class is present, not population-level estimates.Source: §6 · pp. 16–17
  • CCR governance has not been evaluated at scale, and the work does not establish legal compliance with NIS2, CER, or the CRA.External-contributor adoption and quality-gate robustness against adversarial submissions remain to be evaluated in a public registry.Source: §6 · p. 17

7 Conclusion

The evaluation supports a division of labour in OpenAPI-to-CACAO translation: rules preserve mechanical interface facts, while the LLM contributes bounded semantic enrichment. CCR records quality-gate results separately from stronger execution evidence and is seeded with candidates generated from vendor OpenAPI specifications.

7 Conclusion

  • CCR is an open repository of quality-gated CACAO integration candidates, initially seeded from vendor OpenAPI specifications and relevant to capabilities promoted by NIS2, CER, and the CRA.Its regulatory discussion uses a scoped mapping between requirements and technical capabilities; open licensing and contributions support European interoperability, reuse, and digital autonomy.Source: §7 · p. 17
  • The rule-based mapper equals or exceeds the LLM pipeline on mechanical checks, while the LLM’s principal contribution is bounded semantic enrichment, especially activity annotation.The evaluation also found a 5.3% dispatch-failure rate and no independent expert assessment was completed.Source: §7 · p. 17
  • CCR separates automated gate results from execution evidence across six maturity levels, reflecting defects found through full-corpus dispatch testing and limits of a single back-validation score.The assurance-envelope schema provides a common structure for accumulating stronger evidence as the registry develops.Source: §7 · p. 17
  • Future work includes expert ratings, normalised path comparison, evaluation of seven additional APIs when sandboxed deployments are available, and governance testing with external contributors.The authors also plan to execute the demonstration playbook through SOARCA and investigate CACAO activity-vocabulary extensions.Source: §7 · p. 18
  • The artefact package includes 713 seed envelopes, source specifications and hashes, assurance metadata, validation materials, raw results, and the demonstration playbook and trace; the prototype generator is excluded.Expert-rating results will be added after that study is completed.Source: Data and Artefact Availability · p. 18
Loading 2609.27567v2…