Source-linked AI summary
Recovering Software Architecture Intent from Historical Work Items using Generative AI: A Mixed-Methods Industry Case Study
Dominik Storck, Tobias Eisenreich, Stefan Wagner
TL;DR
The paper asks whether architectural intent can be recovered from fragmented historical work items rather than eroded source code. It implements and evaluates a five-stage LLM pipeline that generates upper-level C4 diagrams, finding useful, accurate baselines with stable entities but less stable relationships. The workflow supports architectural-drift detection, while remaining bounded by its input data and requiring human curation for external use.
Problem
Architectural intent is often difficult to recover because source code reflects erosion and historical work items are not directly suited to architectural analysis.
Method
A five-stage LLM-based pipeline transforms historical Azure DevOps work items into C4 System Context and Container diagrams.
Results
Practitioners found the generated diagrams accurate and understandable, while stability was higher for architectural entities than for relationships.
Takeaways & Limitations
The workflow mirrors documented intent and can help practitioners identify architectural drift when recovered artifacts are compared with the implemented system.
Takeaways & Limitations
External validity is limited by the single-company case study, three practitioners, one LLM, young projects, and context-window constraints for massive systems.
Abstract
from arXiv · showhide
Software architecture is often only partially captured in code, while much of the design intent lives in evolving project artifacts. In agile projects, work items, user stories, and related tracking documents preserve valuable traces of that intent, but they rarely support direct architectural analysis. This work investigates the recovery of C4 architecture diagrams from historical agile work items using an LLM-based pipeline. The semi-automatic five-step workflow employs a prompt chain, bidirectional traceability, and Chain-of-Thought reasoning to transform unstructured Azure DevOps work items into visual artifacts. Evaluated on two industry projects, we use a mixed-methods design combining qualitative expert interviews with a quantitative stability analysis. Practitioners perceive the generated architectural baselines as accurate and highly useful for system comprehension. Strictly bound by their input data, the artifacts mirror the documented intent, thereby surfacing discrepancies and architectural drift when compared to the implemented reality. Quantitatively, the workflow exhibits high stability for architectural entities but lower stability for their relationships, with relative variance compounding across generation steps. The proposed workflow demonstrates the practical viability of LLM-assisted architectural recovery based on development process artifacts.
1 Introduction
The paper addresses recovering architectural intent from fragmented project artifacts rather than relying solely on eroded source code. It proposes an LLM-based workflow that transforms historical Azure DevOps work items into upper-level C4 diagrams for architectural analysis.
- Motivation: Architectural erosion and staff turnover can make implemented systems diverge from their original design intent.Understanding that intent remains important for continued maintenance.
- Problem: Code-level architecture recovery often captures implementation workarounds instead of the system’s logical design, while documentation-based recovery requires substantial tacit architectural knowledge.These limitations motivate recovering intent from other historical project artifacts.
- Approach: The workflow deliberately extracts documented architectural intent from historical work items while bypassing noise in eroded codebases.Its goal is to demonstrate the practical viability of Generative AI for architecture rediscovery.
- Approach: The end-to-end pipeline translates historical Azure DevOps work items into structural C4 diagrams.Its technical scope is limited to textual work items and C4 System Context and Container diagrams.
- Research questions: The study evaluates practitioner perceptions of accuracy and understandability, workflow stability across iterations, and practical integration and utility.These dimensions correspond to the paper’s research questions.
2 Background
The background frames architecture erosion as divergence between intended and implemented systems and motivates abstraction-based recovery. It presents C4 as an accessible way to represent system boundaries and containers, while describing LLM prompting techniques for complex extraction tasks.
- Architecture erosion: Architecture erosion is the growing divergence between implemented as-built architecture and intended as-planned architecture.Design decisions can be lost as changing requirements produce architectural workarounds.
- Recovery challenges: Traditional source-code recovery uses clustering or dependency analysis but often produces inaccurate representations that do not match human architectural partitioning.Even accurate dependency information may not yield meaningful models for large codebases.
- C4 model: The C4 model visualizes software at distinct abstraction levels, with its upper tiers defining system boundaries and grouping functional capabilities.These abstractions support communication and system comprehension.
- C4 model: A System Context diagram represents the entire software system within its surrounding context.It is the most abstract C4 diagram.
- C4 model: A Container diagram divides a system into separately deployable units that execute code or store data.Examples include databases, microservices, and mobile applications.
- LLM prompting: Chain-of-Thought prompting and prompt chaining are presented as ways to reduce omissions and hallucinations in complex architectural generation.They introduce intermediate reasoning and multiple linked LLM queries.
3 Related Work
Related work shows growing interest in LLM-assisted architecture recovery and diagram generation, but limited evidence for fragmented historical work items. This paper addresses that gap with a sequential pipeline evaluated through stability analysis and expert feedback.
- LLM-based recovery: LLMs applied directly to source code struggle with precise structural modeling and fine-grained relationship extraction.Architectural erosion in code causes these approaches to inherit source-level noise.
- LLM-based recovery: Deductive Chain-of-Thought reasoning has been used to classify architectural components from source-code indicators in a human-understandable way.The present workflow applies deductive reasoning directly to historical work items instead.
- Prompting strategies: Sequential prompting can introduce error propagation because small inaccuracies early in a chain may compound across later steps.A literature review also found that 70% of surveyed research relied on basic zero-shot prompting.
- Research gap: Existing diagram-generation studies mainly use curated requirements or comprehensive system briefs rather than fragmented and inconsistent historical artifacts.Those pre-structured inputs do not represent the operational backlog setting examined here.
- Research gap: Prior research lacks a comprehensive evaluation of whether LLM workflows can reliably derive architecture diagrams from historical agile work items.The gap concerns reliability in a realistic industry-data setting.
- Study design: This study evaluates sequentially chained prompting on industry backlog items using stability analysis and expert feedback on accuracy, understandability, and utility.The evaluation covers the paper’s three research-question dimensions.
4 Concept and Implementation
The implementation is a five-stage pipeline that transforms a project backlog into formalized C4 diagrams. It decomposes architectural reasoning into System Context and Container generation, validates structured outputs, and separates semantic generation from visual rendering.
- Pipeline design: The five-stage workflow transforms raw project-backlog data into formalized architecture diagrams through sequential processing.The stages span extraction, preprocessing, two LLM generation steps, validation, and rendering.
- Data preparation: Azure DevOps responses are filtered and converted into hierarchical Markdown to reduce irrelevant content and token overhead.The extraction targets Epics, Features, User Stories, and Tasks.
- Architectural generation: The first LLM stage processes the Markdown context to identify the System Context and its boundaries.This generates C4 Level 1 elements.
- Architectural generation: The second LLM stage receives the validated System Context and must place containers within its boundary while reusing external element IDs.This dependency supports referential integrity and structural consistency.
- Validation and rendering: A Pydantic validation layer checks both JSON outputs before converting them to PlantUML for visual rendering.The validation layer is intended to prevent downstream rendering failures.
- Prompt engineering: The prompting strategy combines schema-guided one-shot prompting, structured Chain-of-Thought, negative constraints, and linked work-item provenance.The linked_work_items array provides bidirectional traceability to source tickets.
5 Methodology
The study combines expert interviews, traceability analysis, and repeated pipeline executions to evaluate recovered C4 diagrams from two real-world Azure DevOps projects. Stability is measured structurally through repeated node and edge counts under fixed generation settings.
- The mixed-methods design combines semi-structured expert interviews, targeted traceability analysis, and quantitative variance analysis.
- The evaluation uses Project A with 89 work items and Project B with 193 work items, while an independent third project supports pipeline development and calibration.
- Three senior practitioners evaluated fixed baseline diagrams for perceived accuracy, understandability, and practical utility through semi-structured interviews and thematic synthesis.
- The pipeline ran 10 independent times per project under identical GPT-5.1, temperature 0.01, and fixed-seed settings, producing 40 diagrams.
- Output stability is defined as structural similarity across repeated generations from identical input and is operationalized using node and edge counts.
- For Container diagrams, counting includes only internal elements within the System_Boundary, isolating newly generated container-level structure from inherited context elements.
6 Results
Experts found the recovered diagrams accurate, useful, and semantically understandable, while traceability exposed gaps between documented intent and implementation. Quantitative analysis found greater stability for nodes than edges, with instability increasing across chained generation steps.
- Qualitative Expert Evaluation: All three interviewed practitioners supported four themes concerning extraction, utility, understandability, and continuous consistency or auditing.
- Qualitative Expert Evaluation: Practitioners considered extraction highly accurate despite vague work items, but outputs remained strictly bounded by outdated content and organizational quirks.
- Qualitative Expert Evaluation: The workflow reduced manual architectural-baseline creation from hours to minutes, while external or client-facing use required manual validation and curation.
- Qualitative Expert Evaluation: Semantic extraction was consistently comprehensible; reported cognitive friction arose from PlantUML visual differentiation and spatial layout.
- Qualitative Expert Evaluation: Continuous documentation, milestone canvases, and auditing exposed missing or misrepresented elements, documentation debt, and structural drift.
- Architectural Discrepancies: Traceability showed that the workflow extracted unexecuted intent but remained blind to undocumented implementations such as Cloud Storage in Project A.
- Quantitative Structural Stability: System Context nodes were more stable than edges: Project A nodes had CV = 6.6%, whereas Project B edges reached CV = 33.0%.
- Quantitative Structural Stability: Container nodes remained more stable than internal edges, with Project A nodes at CV = 9.6% and Project B internal edges peaking at CV = 38.3%.
7 Discussion
The workflow produces useful, accurate architectural baselines by mirroring documented intent, while stability varies between entities and relationships and decreases across sequential steps. Its practical use remains bounded by human curation, evaluation limitations, and case-specific validity constraints.
- Accuracy and Understandability (RQ1): Practitioners perceived the generated C4 diagrams as accurate reflections of projects and historical work items despite ambiguity and sparse input.Rendering technology, rather than semantic reasoning, limited understandability.
- Accuracy and Understandability (RQ1): The LLM mirrors documented backlog structures rather than generating an idealized technical architecture, including organizationally shaped boundaries.A project diagram split deployment units along team competency lines because tasks followed resource allocation rather than technical architecture.
- Accuracy and Understandability (RQ1): Comparing recovered intent with the codebase can expose architectural drift, but practitioners must apply tacit domain knowledge to validate discrepancies.The workflow shifts effort from manual diagramming toward validating structural differences.
- Workflow Stability (RQ2): The workflow shows high structural stability and tends to regenerate a dominant baseline of architectural nodes.Automated regeneration can return unsatisfactory outputs toward this distinct baseline.
- Workflow Stability (RQ2): Architectural entities are identified more consistently than relationships, while stability degrades as the sequential pipeline progresses.Passing context-step outputs as ground truth lets small foundational differences amplify structural inconsistency in later generation.
- Utility, Integration, and Limits (RQ3): The artifacts reduce baseline creation from hours to minutes and can support technical kickoffs, infrastructure checks, and onboarding.Experts proposed using them as starting canvases for critical milestones.
- Utility, Integration, and Limits (RQ3): Raw outputs are useful for internal engineering and exploration but require manual curation before external, pre-sales, or client-facing use.Curation resolves rendering friction, fills gaps, and validates semantic mapping.
- Utility, Integration, and Limits (RQ3): Recovered artifacts can surface documentation debt and architectural drift while providing accurate representations when documented intent aligns with implementation.This dual purpose connects visual recovery with ongoing documentation practices.
8 Conclusion
The paper presents a semi-automatic five-step workflow for recovering C4 System Context and Container diagrams from historical agile work items. Evaluation indicates that the workflow mirrors documented intent and provides useful baselines, while requiring human curation and remaining limited by undocumented implementations.
- 8 Conclusion: The study evaluates semi-automatic recovery of C4 System Context and Container diagrams from historical agile work items.It assesses structural accuracy, statistical stability, and practical utility using a deductive five-step workflow.
- 8 Conclusion: The pipeline accurately mirrors input data and reflects the project’s documented as-planned state.It remains blind to undocumented implementations while rendering unexecuted intent.
- 8 Conclusion: The workflow provides rapid baseline discovery and helps practitioners expose architectural drift while preserving historical design decisions.Its artifacts offer a starting canvas for understanding systems.
- 8 Conclusion: Raw outputs are effective for internal use, but external or client-facing applications require human curation.The trust boundary reflects the workflow’s dependence on review before external dissemination.