PaperlayerView exact revision on arXiv

Source-linked AI summary

A Governance Methodology Layer for AI-Assisted Software Development: Defect Taxonomy, Controlled Ablation, and Process-Over-Capability Evidence

Sungjin Kwon

arXiv:2609.04218v1cs.SE

TL;DR

AI coding agents pass syntactic checks while semantic boundaries, invariants, and contracts remain insufficiently assessed. This paper introduces a structured governance methodology, portable file-based gate, and controlled ablation; structured review records higher ground-truth recall and more consistent severity grading than token-matched unstructured review. The evidence is directional and bounded by a small sample and stated validity threats.

  • Problem

    AI-assisted development closes a syntactic correctness loop while leaving semantic design boundaries, security invariants, and maintainability contracts insufficiently assessed.

  • Method

    The paper develops a defect taxonomy, runtime-decoupled governance gate, methodology-as-code artifact, and controlled ablation comparing structured with token-matched unstructured review.

  • Results

    Structured review achieves higher ground-truth recall than unstructured review: 62% versus 50% lenient and 25% versus 0% strict, with more consistent severity grading.

  • Takeaways & Limitations

    Within the study’s scope, the results are consistent with review-process design, rather than reviewer-model capability, driving critical-defect coverage.

  • Takeaways & Limitations

    Claims are directional because the experiment uses a small sample, requires replication with N=20+, and includes threats from adversarial inputs or reviewing-model behavior.

Abstract

from arXiv · show

Autonomous coding agents produce output that passes syntactic checks -- compilation, type safety, CI -- at high velocity. Yet syntactic correctness does not imply semantic correctness: design boundaries, security invariants, and maintainability contracts remain structurally invisible to automated pipelines. This paper makes four contributions toward closing this gap. First, we present a defect-class taxonomy grounded in five AI agent permission and governance modules, distinguishing defects structurally detectable by static analysis from those requiring semantic review. Second, we describe a runtime-decoupled governance gate -- a file-based protocol that reads generator output and emits a structured verdict without API coupling, making it portable across code-generation tools. Third, we formalize methodology-as-code: expressing a verification protocol as a version-controlled, executable, cross-platform artifact with a two-layer architecture separating portable methodology from host-specific automation. Fourth, we report a controlled ablation experiment (E-ablation, N=5 artifacts, 8-item independent ground truth) comparing harness-structured review against a token-matched unstructured review prompt on the same artifacts. The structured condition records 62% lenient ground-truth recall against 50% for the unstructured condition, with 25% strict recall against 0%, and applies more consistent severity grades where the unstructured condition inflates severity. Both conditions miss document-quality defects identified by a human QA reviewer, indicating a complementarity between structured AI review and human process inspection. Within the bounds of this sample, the results are consistent with the thesis that review-process design, rather than reviewer-model capability, is the dominant factor in critical-defect coverage for AI-assisted software development.

1. Introduction

AI coding agents accelerate implementation but leave semantic correctness gaps in design boundaries, security invariants, and maintainability contracts. The paper studies whether structured, portable governance methodology improves semantic review coverage beyond model capability alone.

  • The Semantic Gap in AI-Assisted Development: AI agents can close the syntactic development loop while leaving design coherence, security boundaries, permission correctness, and maintainability contracts structurally open.CI, compilation, type checking, and unit tests verify well-formedness and selected input-output behavior, not all semantic properties.
  • Research Questions: The paper tests whether methodology structure, rather than any sufficiently detailed prompt, is the operative factor in review coverage.A controlled ablation directly compares structured governance review with a token-matched unstructured review prompt.
  • Contributions: The paper contributes a defect taxonomy, runtime-decoupled governance gate, methodology-as-code pattern, and E-ablation evidence linking process design to coverage.The contributions span defect classification, portable gate architecture, version-controlled review methodology, and controlled comparison.
  • The Semantic Gap in AI-Assisted Development: At agent velocity, conventional human review becomes harder to sustain because larger diff volumes reduce the marginal value of deep inspection.The paper contrasts approximately 200 human-written lines per day with 2,000 agent-written lines per day.
  • Artifact Domain: The study focuses on semantic correctness in permission-enforcement code from active AI agent frameworks, where governance defects directly affect system security posture.The artifact domain includes OpenCode, Hermes, Goose, and OpenHuman modules.
  • Research Object and Scope: The governance methodology treats a natural-language specification as a version-controlled software artifact subject to verification, drift control, and maintenance.This SE framing measures defect coverage and addresses how the specification is verified and maintained over time.

3. Defect-Class Taxonomy (C1)

The taxonomy identifies defect classes that CI and SAST do not structurally assess because they concern intended security models, contracts, and semantic behavior. Its coverage is demonstrated on five artifacts but is not established beyond that artifact set.

  • Motivation: CI and SAST assess syntax, style, types, and known vulnerability patterns, but do not ask whether permission boundaries reflect the intended security model.The paper distinguishes syntactic checks from semantic review of boundaries and contracts.
  • Taxonomy: Four defect classes are identified as structurally beyond CI/SAST reach.The taxonomy is grounded in five AI agent permission and governance modules.
  • Taxonomy Validity: 49 structured-review findings classify into D1–D4 without residual.This is reported as a taxonomy-validity result for the reviewed findings.
  • Limitation: The taxonomy is exhaustive only within the five demonstration artifacts, not across held-out data.The paper calls for validation on an independent artifact corpus.
  • Empirical Grounding: The reviewed defects occurred in committed, CI-integrated files whose projects reported passing automated checks, supporting the distinction between syntactic and semantic coverage.The authors did not independently execute the CI pipelines.

4. Runtime-Decoupled Governance Gate (C2)

The governance gate reads generator-written files and emits structured severity-graded verdicts without coupling to the generator, runtime, or language toolchain. Its central signal is a PENDING verdict when CI passes but semantic defects remain.

  • Architecture: The gate reads files, applies a review protocol, and emits PASS, PENDING, or BLOCKED verdicts with S/A/B severity grades.Its architecture separates generator output from the governance decision and downstream CI.
  • Decoupling: The gate requires no API coupling to the generator, language-specific toolchain, or access to the executing system.It operates on static artifacts rather than the running program.
  • Portability: Runtime decoupling makes the same gate portable across TypeScript, Markdown, Python, and Rust artifacts without language-specific runtimes or compilers.Five languages or formats were reviewed using one protocol in one session.
  • Integration Signal: A PENDING verdict identifies unresolved semantic defects in code that passed automated checks.In the controlled trial, arity.ts received PENDING with two A-grade findings despite passing the complete CI suite and developer self-review.

5. Methodology-as-Code (C3)

Methodology-as-code expresses review as a version-controlled, executable specification rather than an informal prompt. Its portable methodology layer can operate independently of host-specific automation and can itself be reviewed by the same process.

  • The SKILL.md Pattern: The SKILL.md artifact defines the review protocol, axes, structured output format, and explicit completion condition.These sections make the verification procedure executable and human-readable.
  • Methodology-as-Code: Methodology-as-code makes the review protocol a first-class artifact subject to version control, diff, review, and testing.The protocol is maintained like other software-development artifacts.
  • Two-Layer Portability: The methodology layer is portable across LLM CLIs, while host-specific automation is optional and accelerates execution without being required.The separation supports use with Claude Code, Codex, Gemini CLI, and GitHub Copilot.
  • Practical Consequence: Adoption requires no toolchain change, only the protocol document and LLM access.Automation remains an optional execution aid.
  • Self-Referential Governance: The protocol is governed by the same review process it applies to code, enabling it to reveal defects in its own specification.A companion steel-quench pass found phantom references and structural gaps missed by the authors.

6. E-Ablation: Process vs. Capability (C4)

The controlled ablation compares a structured, axis-based two-wave governance review with a token-matched unstructured prompt on the same five artifacts. Structured review improves strict ground-truth recall, while both conditions show complementary coverage and miss document-quality defects.

  • Experimental design: The ablation tests whether harness structure adds value beyond a similarly detailed unstructured prompt.Both conditions used the same artifacts and comparable prompt length, with independent agents per artifact.
  • Experimental design: Condition A applies four review axes and two waves; Condition B asks for design and security defects without axis specification or multi-wave discipline.The structured protocol covers design correctness, security, API contract, and maintainability before producing findings.
  • Results: Condition B assigns more S-grades than Condition A, 6S versus 3S, while reporting fewer A-grade issues, 13A versus 19A.Examples include the same findings receiving A or B under Condition A but S under Condition B.
  • Results: 25% versus 0% strict ground-truth recall favors structured review, concentrated in dead-code and exception-handling findings.The structured axes explicitly direct reviewers toward dead code and silent failure modes.
  • Coverage complementarity: Both AI conditions miss document truncation, formatting inconsistency, and non-executable placeholders found by the human QA reviewer.The AI reviews focused on runtime correctness and security, whereas human QA identified document-quality issues.
  • Coverage complementarity: Neither condition strictly dominates total coverage: each finds unique issues, with structured review emphasizing semantic-boundary and silent-failure defects.Unstructured review uniquely finds performance and annotation issues.
  • SAST comparison: Default SAST and governance review target different defect spaces in this artifact set.SAST missed design and semantic defects, including dead or mathematically unreachable code, while Ruff mainly flagged a broad-except pattern and style issues.

7. Supporting Evidence (Shared with Companion Paper)

Supporting experiments report benefits from structured review across independent sessions, verdict flips on CI-passing code, and model-tier variation. Together, these results support process-focused evidence, though the experiments are summarized as companion-paper evidence.

  • Governance verdict flip: A governance review changed a CI-passing, developer-approved TypeScript artifact to PENDING with two A-grade findings.Neither finding was surfaced by CI, the linter, or developer self-review.
  • Model-tier independence: S-grade critical-defect detection was equivalent across Haiku-4.5, Sonnet-4.6, and Opus-4.8 on a bounded artifact.This model-tier result is reported as supporting evidence for process over capability.
  • Multi-session coverage: 57% versus 84% defect coverage compares single-session review with a three-persona cross-session panel across N=5 artifacts.The 27-percentage-point increase came at negligible marginal cost.

8. Threats to Validity

The evidence is constrained by shared-session contamination, small and selected samples, a single human ground-truth rater, limited SAST breadth, and model-based verdict provenance. The paper calls for independent-session replication and broader evaluation.

  • Experimental threats: Both ablation conditions ran in the same AI session, so Condition B may have retained findings from Condition A.Separate API calls with no shared context are proposed for cleaner replication.
  • Ground truth: The eight-item ground truth came from one QA engineer, making it subject to perspective bias.The proposed mitigation is an inter-rater ground truth with a second independent reviewer.
  • Experimental threats: N=5 artifacts provides insufficient statistical power for significance tests, so the ablation claims are directional.Replication with N=20+ is identified as needed for venue submission.
  • Sampling: The five artifacts were selected from AI-agent governance code, which may not represent other AI-agent code.A random sample from a larger corpus would reduce selection bias.
  • Sampling: One artifact is itself a governance artifact, potentially giving the methodology domain-specific advantages.This self-referential case may not generalize to unrelated code.
  • SAST baseline: The SAST comparison used one curated ruleset set and omitted standalone clippy and eslint.A maximally tuned, multi-tool sweep over a larger corpus would strengthen the comparison.
  • Verdict provenance: The gate's accept/flag signal remains a model judgment rather than a verdict bound to non-model evidence.The proposed robustness direction is to anchor findings to literal source spans or observed execution results.

9. Future Work

Future work targets broader empirical validation, dynamic behavior, and cross-modal requirements that the current static, single-modality evaluation does not cover.

  • Empirical expansion: A larger multi-tool sweep is proposed to test whether the observed non-overlap between governance review and default SAST generalizes.The current evidence comes from five artifacts; broader tools and corpora are identified as the next empirical step.
  • Empirical expansion: Independent sessions and a larger, stratified artifact corpus would test robustness and variation across domains.The proposed replication removes shared context, while expansion to N=20 artifacts enables within-stratum recall measurement.
  • Human comparison: A developer study would compare governance review with unaided human review rather than only unstructured AI review.The proposed design presents the same five artifacts to 10–20 developers and compares defect recall across conditions.
  • Human comparison: Inter-rater replication would measure agreement between security and QA raters and use their intersection as conservative ground truth.This addresses uncertainty in the pre-read ground-truth record.
  • Scope extensions: The static gate cannot assess defects that depend on realized execution state, so dynamic observation is required for that class.Such defects involve values, timing, environment, and interactions absent from the source artifact.
  • Scope extensions: Cross-modal review is needed for inconsistencies spanning text specifications, visual mockups, and API contracts.The proposed approach pairs textual and visual review before cross-validating the modalities.
  • Scope extensions: The paper treats cross-modal extension as future work and does not claim a cross-modal contribution.The evaluated contribution remains limited to the single-modality D1–D4 set.

10. Related Work

Related work positions this paper at the intersection of structured inspection, AI review, static analysis, methodology-as-code, and agent safety. Its distinctive focus is measuring governed methodology as an independent layer over the model substrate.

  • Structured review: Structured inspection has historically been associated with higher defect detection than unstructured walkthroughs, and this work extends that comparison to AI review.The E-ablation applies the structured-versus-unstructured distinction in the software review domain.
  • Structured review: Structured AI review with explicit axes outperforms unstructured AI review on ground-truth recall, while the paper treats the harness as a governed artifact.The related natural-language harness work studies task resolution; this paper measures defect coverage, making the two lines complementary.
  • LLM-based software engineering: LLM code-review research generally treats the model as the primary agent, whereas this paper measures methodology effects independently of the substrate.The LLM is presented as executing a methodology rather than defining the review process itself.
  • Testing and oracles: Test-generation and oracle systems often rely on model judgments, while this methodology is orthogonal to those systems.The paper emphasizes anchoring verdicts to non-model evidence and describes non-model backstops for structurally invalid outputs.
  • Testing and oracles: A non-model backstop is proposed to reject structurally invalid outputs before model consensus is trusted.This addresses the failure mode in which several models agree on the same plausible-but-wrong structure.
  • Agent safety: The taxonomy supplies a structured vocabulary for defect classes that AI-agent safety and permission systems are designed to prevent.The D1–D4 classes connect agent safety concerns with governance review targets.
  • Methodology-as-code: Methodology-as-code applies version-control principles from infrastructure-as-code to executable review protocols.The SKILL.md pattern makes the review procedure subject to a software development lifecycle.
  • Static analysis: Default SAST and governance review target different defect spaces on the five-artifact set.The reported baseline measures empirical non-overlap between D1–D4 design or semantic classes and default SAST coverage.

11. Conclusion

The paper argues that a structured governance methodology addresses semantic defects that automated pipelines miss and evaluates that claim through controlled comparison. On five artifacts, structured review achieved higher recall and more consistent severity grading, while human QA exposed complementary document-quality defects.

  • Contribution and evaluation: The proposed governance layer targets semantic boundary violations, silent failure modes, missing contract guards, and dead correctness code beyond automated pipeline reach.The claim is grounded in a five-artifact external evaluation of AI-agent permission and governance code.
  • Contribution and evaluation: The controlled ablation compares structured and unstructured AI review on the same artifacts against independent human ground truth.The experiment uses five artifacts and an explicitly contrasted review design.
  • Findings: 62% versus 50% lenient recall and 25% versus 0% strict recall favor structured review over token-matched unstructured review.The structured condition also produces more consistent severity grades, while the rival explanation remains under evaluation in the supplied conclusion text.
  • Findings: Both review conditions miss document-quality defects identified by a human QA reviewer.This indicates that AI governance review and human process inspection cover non-overlapping defect classes.
  • Implication: Combining AI governance review with human process inspection achieves higher combined recall than either alone.The conclusion presents the two review modes as complementary rather than interchangeable.
  • Implementation: The methodology uses version-controlled, executable, CLI-portable artifacts that can wrap any code-generation tool without API coupling.The gate is decoupled from the runtime environment of the reviewed code.
  • Scope: The first E-baseline finds design and semantic defect classes empirically non-overlapping with default SAST, while a larger sweep remains future work.The conclusion identifies broader tooling and sample size as needed next steps.

Appendix A — E-ablation Raw Findings

The appendix provides access information for the raw E-ablation findings and the full ground-truth record.

  • Raw findings: Full per-artifact findings for Conditions A and B are available from the author on request.The appendix directs readers to the raw condition-level findings.
  • Ground truth: The full ground-truth record with rater notes is available from the author on request.This supplements the reported ablation results with the underlying adjudication record.
Loading 2609.04218v1…