PaperlayerView exact revision on arXiv

Source-linked AI summary

A-SAW: The Agentic Software Architecture Workflow

Finn Klatt, Luca Di Grazia, Johannes Schneider

arXiv:2610.13647v1cs.SE

TL;DR

Architecture documentation maintenance remains underserved even though Architecture as Code makes artifacts machine-readable. A-SAW adds a persistent, repository-grounded agent that drafts documents under mandatory human review. Across matched documents, scores became more uniform and intrinsic quality increased, while traceability remained the weakest dimension.

  • Problem

    AI assistance in software engineering focuses on implementation and verification, while architecture-document maintenance remains comparatively underserved and Architecture as Code does not reduce upkeep effort.

    Source: §1 · p. 1

  • Method

    A-SAW adds a persistent agent that drafts from version-controlled C4 models, a typed element catalog, and instrumented templates, with every draft subject to human review.

    Source: §1 · p. 2

  • Results

    Document-score standard deviation fell from 19.2 to 3.8, while intrinsic quality rose by a median of +19 points across 13 matched pairs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Practitioners’ trust remained conditional: 11 of 13 insisted on human review, and their effort shifted from authoring toward verification.

    Source: §1 · p. 2

    Traceability remained the lowest-scoring dimension at 79%, with decision auditability and implementation linkage among its residual gaps.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

AI assistance in software engineering focuses on implementation and verification, leaving the design phase, and in particular the maintenance of architecture documentation, comparatively underserved. Architecture as Code makes architectural artifacts versionable, diffable, and machine-readable, but it does not reduce the effort of keeping them updated, so design documentation still ends up as a fragmented, author-dependent, and only partially trusted record of why a system looks the way it does. In this paper we introduce the Agentic Software Architecture Workflow (A-SAW), a process-level artifact that extends Architecture as Code with a persistent agent layer: an IDE-integrated agent drafts architecture documents from a version-controlled C4 model, a typed element catalog, and instrumented templates. Every draft passes a mandatory human review gate before it is accepted. We evaluate it in a large industrial organization along two tracks: a rubric assessment of 45 architecture documents including 13 matched source-regeneration pairs, and 13 task-anchored practitioner interviews. Our results show that the workflow collapses the between-author variance that made quality unpredictable. Intrinsic quality also rises, by a median of +19 points across the matched pair. The cohort has a mean of 58% and 89% bundle templates, catalog, agent, and a richer regeneration context. So they bound the agent's contribution from above rather than isolating it. In the agent-generated documents, completeness (100%), consistency (93%), and machine-readability (99%) reach or approach the rubric maximum, while traceability reaches 79%. Eleven of the 13 participants rejected agent output that no human had reviewed, and one warned that polished output may lead reviewers to skip the review. Our work supports keeping human review mandatory and concentrating it where the agent must infer beyond the repository.

1 Introduction

Architecture as Code makes architecture artifacts machine-readable but does not solve their maintenance, leaving design documentation fragmented and only partly trusted. A-SAW adds a persistent agent to draft architecture documents under mandatory human review; its evaluation reports more uniform quality and a median matched-pair gain, alongside conditional practitioner trust.

1 Introduction

  • AI assistance in software engineering focuses on implementation and verification, while architecture documentation maintenance remains comparatively underserved and Architecture as Code does not reduce upkeep effort.The paper frames the remaining problem as maintenance: architectural rationale, alternatives, and trade-offs can disappear as systems evolve.Source: §1 · pp. 1–2
  • A-SAW extends Architecture as Code with a persistent agent that drafts from a version-controlled C4 model, typed element catalog, and instrumented templates, with mandatory human approval.Structural safeguards include catalog-grounded references, scoped context, explicit gap marking, automated checks, and a two-role approval gate.Source: §1 · p. 2
  • Document-score standard deviation fell from 19.2 to 3.8, while intrinsic quality rose by a median of +19 points across 13 matched pairs.The cohort means of 58% and 89% combine templates, catalog, agent, and richer regeneration context, so they do not isolate the agent’s contribution.Source: §1 · p. 2
  • Practitioners’ trust remained conditional: none expressed unconditional confidence in generated diagrams, and 11 of 13 insisted on human review.The evaluation included 45 documents and 13 task-anchored practitioner interviews.Source: §1 · p. 2
  • Measured artifact quality and practitioner trust can diverge, and polished output may cause reviewers to relax scrutiny; whether trust converges under sustained use remains unanswered.The single-session design cannot answer that longitudinal question.Source: §1 · p. 3

2 Background

The background frames architecture documentation as views with quality requirements, describes Architecture as Code and C4 as a machine-readable foundation, and distinguishes process-level agentic assistance from autonomous decision-making.

2.1 Architecture documentation and its quality

  • Architecture descriptions should be complete, internally consistent across views, map rationale to motivating requirements, and remain usable for intended readers.The rubric operationalizes these four properties; the paper produces ADDs and DDDs with embedded decisions rather than one-record-per-decision Architecture Decision Records.Source: §2.1 · p. 3

2.2 Architecture as Code and the C4 model

  • Architecture as Code treats architectural artifacts with source-code rigor, while C4 supplies hierarchical Context, Container, Component, and Code views.Text-based PlantUML supports machine readability for agent-assisted maintenance, trading formal analytic power for readability and low authoring friction.Source: §2.2 · p. 3

2.3 Agentic AI and the design phase

  • A-SAW uses agentic assistance at the organizational process level, while each generation remains a bounded pass and humans retain decision authority.Its automation profile is roughly level 4 of Parasuraman et al.’s ten-point scale: the computer suggests an alternative for a human to accept, revise, or reject.Source: §2.3 · p. 4

3 Our Approach

The approach was designed and iterated in an industrial setting around documented maintenance and trust problems. A-SAW adds an agent to a version-controlled architecture repository, scaffolds generation with shared models and templates, and makes human review the approval gate.

3.1 Research Method and Setting

  • The study follows Design Science Research: it builds a real artifact, deploys it, and evaluates it in use through three refinement rounds before summative measurement.The naturalistic evaluation uses real users in an organization; the embedded researcher’s dual role was addressed through triangulation, standardized protocols, and interview-pool exclusions.Source: §3.1 · p. 4

3.2 Problem Analysis and Requirements

  • The identified problems include deferred updates, fragmented documentation, inconsistent templates, author- and age-dependent trust, and lost design context at handovers.The exploratory conversations served problem discovery rather than evidence collection; these themes were later corroborated against the rubric baseline and summative interviews.Source: §3.2 · pp. 4–5

3.3 Conceptual Model: Adding an Agent Layer to Architecture as Code

  • A-SAW makes the repository an authoritative context source: an on-demand agent drafts from it, and human-approved output returns to the artifact layer for later generations.The three layers are the version-controlled repository, agent orchestration, and mandatory human review; approved output grows the available context over time.Source: §3.3 · p. 5, Figure 1 · p. 6

3.4 Architecture-as-Code Foundation

  • The workflow maintains C1/C2 as system-level ground truth, selectively generates C3, excludes the Code view, and uses a typed element catalog as the naming authority across views.The repository consolidates system diagrams, catalog, and templates in a shared location, while feature folders hold their documents, context inputs, and PlantUML sources.Source: §3.4 · p. 6

3.5 Templates and Agentic Generation

  • ADD and DDD templates standardize architecture and detailed-design content, while refinement showed that specific expected-content guidance and worked examples improved semantic substance.The templates also require alternatives and rejection rationale in design decisions; generation assembles context from the catalog, system diagrams, template, and feature context folder.Source: §3.5 · p. 7
  • Generation rules require every template section, explicit gap markers, catalog identifiers, prefix-matched macros, feature-scope legends, and structured design decisions.Automated checks flag catalog mismatches, phantom identifiers, macro misuse, and unresolved placeholders, leaving reviewers to assess plausibility and rationale.Source: §3.5 · p. 7

3.6 Human-in-the-Loop Governance

  • Five structural measures constrain hallucination, while a two-role approval gate ensures every draft requires validation rather than bypassing human decision-making.The measures include catalog-grounded references, scoped context, explicit gap marking, automated checks, and mandatory approval; non-deterministic drafts are treated as first drafts for validation.Source: §3.6 · p. 8

4 Evaluation

The evaluation combines a rubric assessment of 45 architecture documents, including 13 matched source–regeneration pairs, with 13 practitioner interviews. It finds higher, more consistent measured quality alongside residual human-dependent properties and conditional practitioner trust.

4.1 Experimental setup

  • The evaluation scores 45 documents across three cohorts and pairs 13 unreviewed regenerations with named sources, alongside task-anchored interviews with 13 practitioners.Regeneration uses already-rich feature context and does not test novel design from a blank slate; interviews use a fixed feature and approached saturation. The rubric separates intrinsic from process-state-dependent properties, and the analysis notes that confidence intervals omit rater variance.Source: §4.1 · pp. 8–10

4.2 RQ1: How Much Does Documentation Quality Change?

  • Intrinsic quality averages rise from 58% in Cohort A to 89% in C, while intrinsic-mean SD falls from 19.2 and 21.0 to 3.8 (A→C ratio 5.0).Across the 13 matched pairs, median improvement is +19 points (p = 0.001); 11 improve, one ties, and one regresses by four points.Source: §4.2 · pp. 10–11, Figure 2 · p. 10, Table 3 · p. 11

4.3 RQ2: Where Does the Agent’s Reach End?

  • 100% C4 element completeness, 93% consistency, and 99% machine-readability reach or approach the ceiling; traceability remains lowest at 79%.Consistency is the gain most clearly attributable to the agent because catalog validation is not performed by a template alone; completeness and format gains conflate template enforcement with agent capability. Requirements linkage depends on external systems, and decision evolution requires iterative human authorship.Source: §4.3 · pp. 12–13

4.4 RQ3: How Do Practitioners Experience and Govern the Workflow?

  • 11 of 13 practitioners reject unreviewed agent output; 10 rate context access better, and 12 of 13 rate it at least as good as before.All participants reported faster first drafts, but verification became harder and shifted toward correctness checks; participants described ownership as arising through review and sign-off rather than generation.Source: §4.4 · pp. 13–14

4.5 RQ4: Where Do Measurement and Perception Diverge?

  • Three of four measurement–perception pairings converge, but 93% measured consistency coexists with conditional practitioner trust.The rubric scores controlled regenerations, whereas trust reports concern unscored session drafts; trust also concerns abstraction level, accumulated reliability, polished-output complacency, and whether the recorded design is right. Other convergences concern context and traceability, friction and process maturity, and ownership and auditability.Source: §4.5 · pp. 14–15

5 Discussion

The discussion separates measured documentation quality from practitioner trust, which also depends on abstraction, reliability over time, and the correctness of recorded design. It proposes transferable workflow principles while emphasizing mandatory, expert review and the structural conditions that bound the findings.

5.1 Structural Quality and Trust

  • High rubric scores do not establish trust: practitioners also care about abstraction, repeated reliability, and whether the recorded design is right.The discussion aligns these conditions with performance, process, and purpose trust, while warning that polished output may reduce vigilance.Source: §5.1 · p. 15

5.2 Transferable Design Principles

  • Five principles combine persistent context, a typed naming contract, separated workflow layers, competence-calibrated review, and trust earned before autonomy expands.P2 and P4 are empirically derived; P1, P3, and P5 are literature-supported but untested, and context-rich targets leave P1's premise assumed.Source: §5.2 · p. 16

5.3 Practical Implications and Boundary Conditions

  • Output quality is expected to depend strongly on feature-context completeness, while building the catalog and ground truth requires a multi-month infrastructure phase.The authors identify issue-tracker integration plus a feature-to-ticket convention as a high-value extension; access alone cannot identify which ticket belongs to which feature.Source: §5.3 · p. 16
  • Review should be non-optional, budgeted, and performed by domain experts, with saved drafting time reallocated to review rather than promised as a net reduction.For complex features, review effort may approach or exceed the drafting effort saved.Source: §5.3 · p. 16
  • The findings are bounded to settings with a maintained container-level C4 model, readable version-controlled diagrams, a naming authority, and enough repositories to justify centralization.The evaluation did not test greenfield design; in small single-repository settings, co-locating architecture with code may be more proportionate, and traceability and trust remained human-dependent.Source: §5.3 · pp. 16–17

6 Threats to Validity

The evaluation has construct and internal-validity constraints, including single-rater scoring and confounding between the agent, templates, and richer regeneration context. Its external validity is limited by one organizational setting and a task-bound interview design.

6.1 Construct and internal validity.

  • The rubric only partly operationalizes design-phase effectiveness, and one unblinded rater leaves measurement error and possible self-preference bias unresolved.Even matched pairs cannot separate the agent's contribution from instrumented templates and richer regeneration context; cross-cohort comparisons remain associational.Source: §6.1 · p. 17

6.2 External validity and reliability.

  • One organization and stack at one C4 maturity level support transferability, not statistical generalization, so reported magnitudes are feasibility evidence rather than predictions.The interview time box, one shared feature, participant unfamiliarity, and model dependence further constrain interpretation and replication.Source: §6.2 · p. 17

7 Related Work

The related work positions A-SAW as a process-level architecture-documentation workflow that combines persistent architectural context with explicit naming constraints and review. It also connects the evaluation to prior work on trust and generated-artifact assessment.

7.1 Generative AI for architecture and design documentation

  • A-SAW differs from prior architecture-generation work by maintaining a multi-artifact production repository and a persistent identifier contract, while producing practitioner-invoked first drafts.The approach targets gaps in verification, terminology consistency, and requirements-to-architecture handover, though full requirements linkage remains future work.Source: §7.1 · pp. 17–18

7.2 Architecture documentation, decisions, and Architecture as Code

  • The paper operationalizes established views-based documentation properties and targets rationale loss, while centralizing artifacts rather than co-locating them with code.It treats C4 as machine-readable context, not primarily as a comprehension aid, and does not claim conformance to the cited viewpoint standard.Source: §7.2 · p. 18

7.3 Trust, automation, and the evaluation of generated artifacts

  • The discussion interprets trust through performance, process, and purpose dimensions and treats polished-output complacency and observed-error sensitivity as reasons to retain review.It adopts LLM-as-judge evaluation while treating reliability and self-preference as live measurement threats addressed through triangulation.Source: §7.3 · p. 18

8 Conclusion

A-SAW extends Architecture as Code with an agent that drafts from repository-held models and catalogs, while requiring human approval of every draft. In an industrial evaluation, matched-pair quality improved and score variability fell, but rubric quality did not by itself establish practitioner trust.

8 Conclusion

  • A-SAW adds a persistent agent to Architecture as Code, drafting from a version-controlled C4 model and typed catalog under mandatory human review.Source: §8 · p. 18
  • Median intrinsic quality rose by +19 points across 13 matched pairs, while document-score standard deviation fell from 19.2 to 3.8.Traceability remained the lowest-scoring dimension.Source: §8 · p. 18
  • Practitioners based trust on properties outside the rubric, so high rubric scores alone do not show that a document can be trusted.Source: §8 · p. 19

Related papers

Loading 2610.13647v1…