Source-linked AI summary
Continuous Assurance of Agentic Security Auditors for Software Delivery Decision Gates
Guy Lupo, Nguyen Hung Nguyen, Viet Vo, M. A. P. Chamikara, Guangdong Bai, Nazatul Haque Sultan, Alsharif Abuadbba
TL;DR
SDLC decision gates need current assurance for LLM-based auditors whose evidence and policy context can change. The paper separates policy, evidence, and execution in TAIP, then evaluates posture recomputation over retained RepoAudit evidence. In the tested single-host conditions, the reported recomputation times were below the predeclared 5 s gate budget.
Problem
Existing approaches do not provide continuous, context-bound assurance for SDLC decision gates, where technical changes can alter evidence and policy changes can reinterpret it.
Source: §2.4 · p. 7Method
The Policy–Evidence–Execution Separation Pattern binds admissible evidence and policy to a versioned Posture Tree while keeping assurance execution stable across models, contexts, and policy profiles.
Source: §1 · p. 3Results
The maximum observed single-context policy-to-posture latency was 1.1 ms, and full policy-triggered recomputation across 1,000 contexts with one worker recorded a maximum aggregate refresh of 1.62 s, below the 5 s budget.
Source: §1 · p. 4, §4.4 · p. 14Takeaways & Limitations
TAIP can recompute and publish a current posture from retained evidence after policy, evidence, or model-context changes without re-executing the underlying security control.
Source: §1 · p. 4The evaluation is limited to posture recomputation over one Python NPD condition, one pinned RepoAudit revision, two model configurations, one Evidence Window design, and one local execution environment.
Source: §4.5 · p. 16
Abstract
from arXiv · showhide
Large language model (LLM)-based repository auditors are increasingly deployed as security controls within continuous integration (CI) pipelines, where their findings admit, block, or delay software changes. As Agentic Software Development Life Cycle (SDLC) Security Controls, their non-deterministic behaviour changes the evidence, while organisational risk appetite and jurisdictional or data-sovereignty policy change its interpretation. Point-in-time audits therefore cannot maintain current assurance for merge decisions. We propose the Policy-Evidence-Execution Separation Pattern, implemented by the Trustworthy AI Posture (TAIP) Assurance Engine and operated as Continuous Control Posture Assurance (CCPA). By separating policy from stable execution and binding admitted evidence to a versioned Posture Tree, the same assurance logic operates across models, environments, and policy profiles. We evaluate the approach using unmodified RepoAudit on a fixed Python Null Pointer Dereference benchmark. The retained evidence repository contains 80 RepoAudit executions across two OpenAI model configurations, gpt-4o-mini and gpt-4.1. TAIP recomputes assurance posture after policy, evidence, and model-context changes and is evaluated across increasing numbers of independent Decision Gateway contexts. The maximum observed policy-to-posture latency was 1.1 ms across three policy-class cycles in one execution. At 1,000 independent assurance contexts, full policy-triggered recomputation with one worker recorded a maximum aggregate refresh of 1.62 s, below the predeclared 5 s Decision Gateway budget. These single-host measurements concern assurance over retained evidence and exclude RepoAudit execution and provider inference.
1 Introduction
LLM-based repository auditors can influence merge decisions, but their behavior and the policies governing them can change independently, leaving point-in-time assurance insufficient for current gate decisions. The paper introduces a policy–evidence–execution separation pattern and evaluates TAIP’s posture recomputation over retained RepoAudit evidence.
1 Introduction
- LLM-backed controls can change evidence through technical events, while policy changes can alter how unchanged evidence is interpreted.Observability alone does not establish that a control remains adequate, effective, and sustainable for the active gate.Source: §1 · p. 2
- Prior approaches address parts of assurance, but do not provide continuous, context-bound assurance for SDLC decision gates.The paper identifies a need for current verdicts tied to governing policy and admissible evidence within each gate’s decision budget.Source: §1 · pp. 2–3
- The Policy–Evidence–Execution Separation Pattern binds admissible evidence and policy to a versioned Posture Tree while keeping assurance execution stable across models, contexts, and policy profiles.TAIP implements the pattern around unmodified RepoAudit and recomputes posture after admitted evidence, model-context, or policy-profile changes.Source: §1 · p. 3
- 1.1 ms was the maximum observed single-context policy-to-posture latency, while 1,000-context full recomputation took at most 1.62 s, below the 5 s Decision Gateway budget.These measurements concern the tested conditions and evaluate assurance responsiveness rather than RepoAudit execution.Source: §1 · p. 4
- The evaluation is bounded to assurance responsiveness, not RepoAudit detection accuracy, distributed execution, comparisons with alternative approaches, or production-scale deployment.Those questions remain future work.Source: §1 · p. 4
1.1 Contributions
- The contributions comprise a separation pattern, a TAIP implementation with a CCPA event loop, and a decision-gate evaluation over retained RepoAudit evidence.The implementation supports evidence admission, policy re-evaluation, context invalidation, posture publication, and replay; the measurements isolate assurance computation from live RepoAudit execution.Source: §1.1 · p. 5
2 Related Work and Assurance-Layer Gap
Prior work supplies lifecycle review, automated auditing, evidence infrastructure, and continuous monitoring, but the paper identifies an unfilled assurance-layer role: turning context-bound control evidence into a current, budgeted gate decision.
2 Related Work and Assurance-Layer Gap
- The comparison frames related work by whether it binds evidence to active context and policy, aggregates a current claim decision, represents the absence of a defensible decision, and meets the gate budget.These are the assurance-layer properties used to position prior work.Source: §2 · p. 5
2.1 Lifecycle audit and assurance processes
- Lifecycle audit frameworks support adoption review and periodic reassessment, but rerunning expert-led engagements after each change does not provide a bounded event-to-verdict path for every affected merge gate.Evidence-based assurance cases link claims, arguments, and evidence; the remaining question is how to recompute and publish a verdict across SDLC gates with measured latency.Source: §2.1 · pp. 5–6
2.2 Agentic audit tools and runtime controls
- Agentic audit tools produce findings and target-specific validation evidence, but typically stop before assessing whether the auditor remains fit to influence a merge gate under active policy.AgentSentinel performs local interception and blocking, rather than longitudinal posture assessment of the control.Source: §2.2 · p. 6
2.3 Evidence infrastructure and continuous posture precedents
- Provenance and accountability mechanisms support traceability, replay, and forensics, but remain policy-neutral until another mechanism selects the claim, checks evidence scope and freshness, and publishes a gate consequence.Continuous monitoring provides a temporal precedent, but not a context-bound assurance object for a non-deterministic SDLC auditor or a policy-specific gate action within a declared waiting budget.Source: §2.3 · p. 6
2.4 Assurance-layer integration gap
- The assurance-layer gap is not another defect detector, but a layer that binds observations to scope and configuration, rejects stale or incompatible evidence, retains failed or inconclusive branches, and publishes a policy-governed decision within the gate budget.TAIP converts control-test evidence into a forward-looking posture for an SDLC decision.Source: §2.4 · p. 7
3 RQ1: The Policy–Evidence–Execution Separation Pattern
The Policy–Evidence–Execution Separation Pattern keeps policy, evidence, and execution independently changeable while representing assurance as a context-bound Posture Tree evaluated recursively.
3.1 Pattern Structure: Composite Posture and Recursive TEVV
- A posture instance P = (T, π, κ) binds a versioned rooted Posture Tree, policy profile, and active control-context fingerprint.The root is the accountable claim; internal nodes represent objectives and risks, while leaves represent control tests that terminate at evidence interfaces.Source: §3.1 · p. 8
- Evidence binding admits observations only when scope, configuration, provenance, freshness, and integrity match the leaf and active context.Model or evidence-source changes affect E or κ, while risk-appetite or sovereignty changes affect T or π; neither changes ASSURE.Source: §3.1 · p. 8
- Recursive TEVV uses post-order traversal: tests invoke or replay the control, evaluation admits observations, verification applies policy, and root validation checks the SDLC claim.Missing required evidence yields Inconclusive; mandatory failure or a missed threshold yields Fail, and exceptions remain attached to the root.Source: §3.1 · p. 8
- Full evaluation costs O(M + N), while changed-leaf updates have a theoretical O(∆M + kD) bound; RQ2 scale conditions instead require full recomputation.The complexity expressions describe different workloads and do not establish a measured speedup over other approaches.Source: §3.1 · p. 9
3.2 TAIP Assurance Engine Implementation over RepoAudit
- TAIP wraps unmodified RepoAudit with a process boundary, adapter, context gate, and stable engine, mapping raw outputs into five policy-neutral evidence families.The adapter is the only component that understands RepoAudit formats; stale, incomplete, integrity-failed, or mismatched records are retained but not admitted.Source: §3.2 · pp. 9–10
4 RQ2: Can TAIP Recompute Current Posture Fast Enough for an SDLC Decision Gate?
TAIP recomputes and publishes a current posture after policy, evidence, or observable control-context changes, measuring whether the update fits the Decision Gateway’s waiting budget. In the tested single-host conditions, the maximum refresh across 1,000 contexts was below the 5 s budget, with results bounded to assurance computation over retained evidence.
4.1 Decision-gate use case
- The gate use case requires an updated Posture Card after new evidence, policy changes, or context changes, before the pipeline reaches its declared waiting limit.RepoAudit supplies evidence as the representative detective control, while TAIP maintains and publishes the corresponding posture for the Decision Gateway.Source: §4.1 · p. 11
4.2 Experiment setup
- RepoAudit ran on one fixed Python null-pointer-dereference benchmark; its execution and provider latency were excluded from the posture-recomputation timing.The workload was a single-host Python process in an isolated sandbox, with assurance operations running locally over retained evidence.Source: §4.2 · p. 11
4.3 Posture-recomputation test
- The test times accepted changes until a replacement Posture Card is published, checks the expected output, and scales assurance computation using replicated evidence across up to 1,000 trees.Evidence volume increased to a 40-run window per model; replicated contexts used one frozen evidence bundle and did not represent 1,000 live RepoAudit executions.Source: §4.3 · pp. 12–13
4.4 Results
- 1.62 s was the maximum full-fleet refresh for 1,000 independent Contexts with one worker, below 5 s; the single-context maximum was 1.1 ms.The single-context value came from three policy-class cycles in one sequence execution; the fleet value was the maximum across 30 repetitions.Source: §4.4 · pp. 14, 16
- With eight workers, lower median refresh rose from 1.3 ms at one Context to 288.9 ms at 1,000 Contexts, and all reported maxima remained below 5 s.Table 6 and Figure 4 report this series; the elevated ten-Context maximum is retained, but the timings do not identify its cause.Source: §4.4 · p. 14, Table 6 · p. 15, Figure 4 · p. 15
- Strict policy moved the root posture from Pass with Exceptions to Fail on unchanged evidence; a model-context change withdrew prior evidence and published Inconclusive.Replay was also reproduced, and trials were accepted only when TAIP published the state required by the triggering change.Source: §4.4 · p. 15
4.5 Limitations and claim boundary
- The findings are descriptive single-host timing results, not worst-case guarantees, distributed-performance evidence, or proof of universal detection accuracy or production suitability.The evaluation is bounded to one Python NPD condition, one pinned RepoAudit revision, two model configurations, one Evidence Window design, and one local environment.Source: §4.5 · p. 16
5 Conclusion
The paper presents TAIP as an architecture for continuous, context-bound assurance at software-delivery gates, with measured refresh times below the declared gate budget in the evaluated single-host conditions. Future work includes broader evaluations and incremental recomputation.
5 Conclusion
- 1.62 s was the maximum aggregate refresh across 1,000 independent assurance contexts with one worker, below the predeclared 5 s Decision Gateway budget.The reported single-context policy-to-posture latency was 1.1 ms (rounded).Source: §5 · p. 17
- TAIP implements continuous, context-bound assurance by separating stable execution from versioned policy and admissible evidence, without rerunning the underlying auditor.The contribution checks whether evidence remains current and context-compatible before a software-delivery decision.Source: §5 · p. 17
5.1 Future directions
- Future evaluations should span other agentic security controls, repository types, model providers, and security tasks, while assessing evidence admission, adversarial evidence, and workflow effects.The proposed assessments include false holds, exceptions, and inconclusive decisions.Source: §5.1 · p. 17
- Future work could test whether dependency-aware recomputation reduces latency and memory use at enterprise scale.The proposed approach updates only affected Posture Tree paths using shared evidence and policy dependencies.Source: §5.1 · p. 17