PaperlayerView exact revision on arXiv

Source-linked AI summary

The Code of Theseus: Can Agents Rebuild Software from Recovered Design Documents?

Cezar Sas, Sandra Mitrović, Fabio Barth, Georg Rehm, Matteo Salani

arXiv:2610.20068v2cs.SE

TL;DR

Data connecting intent, requirements, and design to code is scarce, while source code is abundant. The paper recovers aligned artefacts at scale and tests them in a round-trip rebuild. The artefacts do not significantly change held-out test performance, but bring rebuilt module structure closer to the original.

  • Problem

    Data connecting intent, requirements, and design to code is scarce, limiting evidence for whether recovered artefacts carry enough design to support rebuilding.

    Source: §I · p. 1

  • Method

    The paper generates aligned intent, requirements, and architecture from source-code repositories and evaluates them through a coding-agent rebuild and artefact-quality assessment.

    Source: A. Statistics · p. 5, 2) Statistical analysis · p. 6, §IX · p. 10

  • Results

    The artefacts do not significantly change held-out test performance, but significantly bring rebuilt module structure closer to the original.

    Source: §IX · p. 10

  • Takeaways & Limitations

    The round trip indicates that recovered artefacts convey design information not captured by the contract and tests, including when the artefacts are concise.

    Source: §IX · p. 10, §VII · p. 10

    The round-trip evaluation uses small Python libraries and a single agent, so results may not generalise to larger systems, other languages, or other agents.

    Source: §VIII · p. 10

Abstract

from arXiv · show

LLM agents increasingly design a system before implementing it, yet the data to train and evaluate them centres on implementation and testing: requirements and design artefacts are often missing, outdated or private. Source code, in contrast, is abundant. Prior work recovers architecture documentation by feeding LLMs the source code or letting agents explore it, which saturates the context, does not scale to many repositories, and yields a single artefact rather than aligned ones. We present a framework that recovers aligned stakeholder intent, requirements and architecture from a compact repository representation, a static dependency graph and key contextual files, and release the artefacts of 10,764 Java, Python, C/C++ and C# repositories as a dataset. An LLM judge rates the artefacts of three generators as clear and internally consistent, but less reliably correct and at the right level of abstraction. To test whether the artefacts carry the design, we pose a software Ship of Theseus: a coding agent rebuilds 29 Commit0 libraries from the intent alone or with the recovered requirements and architecture, given only an interface contract and part of the tests. The artefacts do not significantly change the held-out pass rate (33.7% vs. 35.8%), but bring the rebuilt module structure significantly closer to the original (module F1 from 0.54 to 0.63), with either generator and even with concise artefacts. Recovered artefacts thus convey design that tests do not capture, offering a scalable source of design-level data for agents that design before they implement, and the round trip measures this without human references.

I. INTRODUCTION

LLM agents increasingly design software before implementation, but datasets rarely connect intent, requirements, design, and code. This paper recovers aligned artefacts at scale and evaluates their quality and whether they carry enough design to rebuild software.

I. INTRODUCTION

  • Existing benchmarks emphasize implementation and testing, while requirements engineering and software design are sparsely represented; agents without early-stage artefacts may produce monolithic programs.The cited benchmark shares are 61% for implementation, 12% for testing, 5% for requirements engineering, and 3% for software design.Source: §I · p. 1
  • Data linking intent, requirements, and design to code is scarce because early-stage artefacts may be missing, incomplete, outdated, inconsistent, or private.This scarcity also limits comprehensive benchmarking of multi-agent systems on intermediate development steps.Source: §I · p. 1
  • Prior source-code recovery leaves open whether generation can scale across repositories, align multiple artefacts, and recover enough design to support rebuilding.CIAO reconstructs architecture documentation from source code and directory structure, rather than producing aligned intent, requirements, and architecture.Source: §I · p. 1
  • The framework uses a compact repository representation, a static dependency graph, and contextual files to generate aligned stakeholder intent, requirements, and architecture.The paper releases a dataset pairing these artefacts with code for more than 10,000 repositories.Source: §I · pp. 1–2
  • The evaluation combines an LLM judge’s assessment of three generators with a round-trip rebuild of 29 Python libraries using artefacts from two generators.The rebuild is assessed with held-out tests and structural metrics against the original implementation; the reported contribution is improved module-level structural similarity.Source: §I · pp. 1–2

A. Existing Datasets

Existing resources tend to cover isolated software-engineering tasks or lifecycle stages rather than align intent, requirements, architecture, and implementation. Related architecture-recovery and requirements-extraction methods demonstrate useful capabilities, but their scope differs from repository-level lifecycle alignment.

A. Existing Datasets

  • Existing datasets rarely align stakeholder intent, requirements, architecture, and implementation, and therefore offer limited support for agents reasoning across the full software lifecycle.The authors state that, to their knowledge, no publicly available dataset systematically aligns all four.Source: A. Existing Datasets · p. 2

B. Software Architecture Recovery

  • Architecture-recovery methods range from graph clustering to LLM-based repository exploration, but prior work described here does not establish large-scale aligned lifecycle datasets.CIAO generates system-level architecture documentation from source code and folder structure and was reviewed on 22 small repositories, mostly by repository contributors.Source: B. Software Architecture Recovery · pp. 2–3

C. Software Requirements Extraction

  • Requirements-extraction research often classifies existing text or infers requirements from snippets and small programs, rather than reconstructing repository-level specifications.Examples include hybrid static-analysis/LLM methods, user-story generation from code inputs below 350 NLOC, and evidence-selection pipelines.Source: C. Software Requirements Extraction · p. 3

D. Software Summarization

The paper situates its work alongside software summarization, then describes how it curates repositories and extracts dependency graphs to build the seed dataset.

D. Software Summarization

  • Software summarization spans bug reports, pull requests, code snippets, and README files, with approaches including templates, information retrieval, deep learning, and LLM-based methods.Examples include LLM summarization of code snippets, README summarization, and the Metagente multi-agent system.Source: D. Software Summarization · p. 3

III. SEED REPOSITORY DATASET

  • The seed dataset is curated for diversity, coverage, and quality through project selection, dependency-graph construction, and repository representation.Candidate repositories come from language-specific awesome lists and projects previously analysed by Arcan.Source: §III · p. 3, A. Projects List · p. 3

A. Projects List

  • Filtering for recent activity and at least 50 stars retains 14,525 of 31,438 non-archived candidate repositories (46%).The retained projects had at least one push in the previous five years.Source: A. Projects List · p. 3

B. Dependency Graph

  • Arcan extracts typed dependency graphs and language-agnostic units; applying graph-size filters leaves 10,764 repositories for the dataset.Arcan completes analysis for 12,664 selected repositories, and the retained graphs have 12–1,493 nodes.Source: B. Dependency Graph · p. 3

C. Repository Representation

The approach represents repositories compactly using dependency information and contextual files, then conditions an LLM on that representation to recover design artefacts. A coding agent uses the artefacts to rebuild the library for comparison with the original.

C. Repository Representation

  • Compression is motivated by context-budget limits: even after filtering very large repositories, dependency graphs plus contextual documents can exceed practical budgets.Figure 1 compares prompt-token distributions for raw and compressed representations against context budgets.Source: C. Repository Representation · p. 4, Figure 1 · p. 4
  • The repository representation combines a unit-level dependency edge list with mappings from units to their files.Context documents include auxiliary project files such as dependency manifests, README files, and Dockerfiles; the representation can reduce raw graph context size by up to an order of magnitude.Source: C. Repository Representation · p. 4

IV. PROPOSED APPROACH

  • The LLM receives a static-analysis dependency graph and project documents as fixed context for generating artefacts; a coding agent then rebuilds the library for comparison with the original.The workflow is presented as a software Ship of Theseus.Source: §IV · p. 4, A. Input Representation · p. 4, Figure 2 · p. 4

B. Description of Generated Documents

The framework generates architecture, requirements, and stakeholder-intent documents from repository context, then uses generated artefacts to guide code reconstruction.

1) Architecture:

  • The architecture document uses C4 System Context, Container, and Component views, excluding the Code view because it focuses on implementation details.Source: 1) Architecture · p. 4

2) Requirements:

  • Requirements comprise functional requirements describing expected behaviour and non-functional requirements capturing quality attributes and constraints.Source: 2) Requirements · p. 4

3) Stakeholder Intent:

  • Stakeholder intent is a freeform initial specification covering desired functionality, constraints, business objectives, user needs, and operational expectations.It guides the derivation of requirements, architecture, and implementation-related outputs.Source: 3) Stakeholder Intent · p. 4

C. Artefacts Generation

  • Each artefact is generated independently with a zero-shot prompt using the dependency graph and project documents, without conditioning on earlier generated artefacts.Source: C. Artefacts Generation · p. 4

D. Prompt Design

  • Structured inference prompts specify the artefact type, output schema, and available context, but include no demonstrations or example outputs.Source: D. Prompt Design · p. 4

E. Agent Code Regeneration

  • To assess whether the artefacts convey design, a coding agent rebuilds each repository from scratch, with the original implementation serving as ground truth.Source: E. Agent Code Regeneration · p. 5

V. VALIDATION METHODOLOGY

The validation combines reference-free document statistics, LLM-based ratings, and agent-based reconstruction against original implementations.

V. VALIDATION METHODOLOGY

  • Because architecture, specification, and intent have no unique reference in code, validation rates the artefacts and tests them through code reconstruction.Source: §V · p. 5

A. Statistics

  • The study samples 1,000 repositories stratified by language, size, and popularity, and uses medians and IQRs to describe generator outputs.Measures include document length, code units named, and diagram validity.Source: A. Statistics · p. 5

B. LLM-as-a-Judge

  • An LLM judge evaluates three generators’ artefacts against repository representations as evidence, treating uncalibrated ratings as a comparative proxy rather than ground truth.Each artefact is rated from 1 to 5 across dimensions adapted to its type; three runs are averaged into quality scores.Source: B. LLM-as-a-Judge · p. 5
  • The judge’s reliability is assessed across runs, and document coherence is measured by linking requirements to intent items and architecture elements.The study compares linkage shares with those obtained using requirements from a random repository.Source: B. LLM-as-a-Judge · p. 5

C. Agent-based Repository Reconstruction

  • For reconstruction, agents rebuild 29 COMMIT0 libraries with an interface contract and 30% of tests visible; the remaining 70% grade the code.INTENT supplies stakeholder intent, while FULL adds requirements and architecture; runs that could not be graded are excluded.Source: C. Agent-based Repository Reconstruction · p. 5

1) Metrics:

  • Correctness is the share of held-out tests passed, while structural outcomes compare rebuilt and reference modules beyond the contract and assess co-location agreement.Co-location agreement is the adjusted Rand index for shared public classes and functions; secondary measures include tokens, prompt length, and artefact modules built.Source: 1) Metrics · pp. 5–6

2) Statistical analysis:

The analysis compares FULL, which adds requirements and architecture, with INTENT alone using paired library-level tests and robustness checks.

2) Statistical analysis:

  • The primary comparison asks whether FULL improves rebuilt code over INTENT, averaging runs across replicates and generators to yield one paired observation per library.It uses paired Wilcoxon signed-rank tests, bootstrap 95% confidence intervals, and Holm-adjusted p-values for three primary outcomes.Source: 2) Statistical analysis · p. 6
  • A mixed-effects model of all runs checks robustness to run and generator variation, with library as a random intercept.The analysis also repeats paired tests by generator; condition-by-generator interaction tests are secondary.Source: 2) Statistical analysis · p. 6
  • Because the contract and visible tests already reveal some structure, the analysis relates FULL’s gain to contract coverage and artefact coverage of remaining modules.It uses correlations, a median split, and a condition-by-coverage interaction.Source: 2) Statistical analysis · p. 6
  • A separate analysis tests whether artefacts naming more modules or using longer prompts correspond to greater gains, comparing generators within libraries.It models 48 library–generator units, with predictors standardised and gains averaged over replicates.Source: 2) Statistical analysis · p. 6

A. Artefacts Statistics

The generated documents differ in verbosity, code-unit coverage, diagram validity, and judged quality across models.

A. Artefacts Statistics

1) Length:

  • Documents grow longer from stakeholder intent to requirements to architecture; GEMA is most concise, while QW35’s architecture documents are about twice GEMA’s length.Lengths vary little across repositories, consistent with the prompts’ fixed structure.Source: 1) Length · p. 6

2) Units named:

  • GEMA names a median of 13% of code units, compared with 29% for QW35 and 31% for GPTO, with almost all names coming from architecture.Source: 2) Units named · p. 6

3) Diagram validity:

  • GEMA diagrams render at 94.3%, with all diagrams rendering in 82.4% of its documents; QW35 reaches 83.8% and 53.4%, and GPTO 57.8% and 20.0%.Most failures involve relationships to undeclared elements rather than syntax; the generators differ in verbosity, not in following the prompted form.Source: 3) Diagram validity · pp. 6–7

1) Quality of the generators:

  • GEMA receives the highest overall quality score, Q = 3.48, versus 3.02 for QW35 and 2.84 for GPTO; artefacts are clear and consistent but less reliably correct and appropriately abstract.GEMA leads on most dimensions, while longer QW35 and GPTO architecture documents score slightly higher on completeness.Source: 1) Quality of the generators · p. 7

2) Patterns across generators:

Across generators, the judge found the artefacts clearest and most consistent, but less reliably correct and appropriately abstract; in reconstruction, requirements and architecture improved module structure without significantly changing held-out test performance.

2) Patterns across generators:

  • The judge rated clarity and consistency highest, but correctness, abstraction, and usefulness lowest; architecture scored below intent, and non-functional requirements below functional ones.These ratings indicate that the documents were coherent but less reliable at grounding claims in code and choosing an appropriate abstraction level.Source: 2) Patterns across generators · p. 7

3) Evaluator reliability:

  • The requirements refined 63–72% of intent items and were allocated to 78–87% of architecture elements, versus 5% and 10% for another repository’s requirements.This coherence check tests whether independently generated artefacts fit together as a body of documentation.Source: 2) Patterns across generators · p. 7
  • Across independent runs, evaluator reliability was ICC(3, 1) = 0.86 and ICC(3, 3) = 0.95, and its holistic ratings tracked Q (mean Spearman ρ = 0.89).The evaluator also ranked the models identically across runs.Source: 3) Evaluator reliability · p. 7

1) Correctness:

  • FULL did not significantly change held-out tests passed: 33.7% in INTENT versus 35.8% in FULL (∆= +2.1 percentage points (pp), p = 0.160).The effect ranged from −21.9 to +24.3 pp across libraries, and replicate outcomes varied substantially.Source: 1) Correctness · p. 7, Table V · p. 8

2) Architecture:

  • Module F1 on modules not named by the contract rose from 0.54 in INTENT to 0.63 in FULL (∆= +0.09), while held-out test performance did not change significantly.The module F1 result was significant (p = 0.005); the held-out test comparison was not (p = 0.160).Source: 1) Correctness · p. 7, 2) Architecture · p. 8, Table V · p. 8

3) Cost:

Adding requirements and architecture substantially increased prompt text but did not significantly change run cost. Both generators showed the same direction of primary effects, with no significant generator interaction.

3) Cost:

  • FULL supplied 4.8 k tokens of task text versus 1.0 k in INTENT, but output tokens and steps did not change significantly.The conditions used 869 and 858 steps on average, respectively.Source: 3) Cost · p. 8

4) Generator:

  • Both generators showed significantly higher module F1 with FULL, but no significant change in correctness or co-location agreement.Neither generator nor its interaction with FULL was significant for any primary outcome.Source: 4) Generator · p. 8, Table VI · p. 9
  • QW35 had a larger module F1 gain than GEMA, but the difference was not significant (interaction p = 0.484), and generators often disagreed on the direction across libraries.The per-library figure shows FULL-over-INTENT gains for module F1 and held-out tests, with markers for each generator.Source: 4) Generator · pp. 8–9, Figure 3 · p. 9

5) Contract coverage:

The module F1 gain did not depend significantly on contract coverage, artefact coverage, or artefact length. The discussion connects recovered design to module decomposition while noting limits on what code can reveal and what this experiment establishes about behavior.

5) Contract coverage:

  • Contract coverage ranged from 7% to 100% of library modules (median 67%), but did not significantly moderate the module F1 gain.The correlation was positive but not significant (ρ = +0.29, p = 0.162); the condition × coverage interaction was also not significant (p = 0.099).Source: 5) Contract coverage · p. 9, Table VII · p. 9

6) Artefact coverage and length:

  • Neither the share of non-contract modules named nor artefact length predicted module F1 gain; the verbose generator’s larger coverage did not produce a significantly larger gain.Within the tested range, receiving requirements and architecture mattered more than how many modules those artefacts or the contract named.Source: 6) Artefact coverage and length · p. 9

VII. DISCUSSION

  • The authors suggest that the decomposition described by the artefacts, rather than module names alone, may steer the agent’s structure; fidelity is bounded by what code reveals, and effects on less-specified behavior remain open.Dependency graphs and build files rarely specify performance or security targets, while the interface in this experiment reveals tested behavior and fixes part of the structure.Source: §VII · pp. 9–10

VIII. THREATS TO VALIDITY

The paper discusses threats to validity and concludes that recovered artefacts convey design beyond what the contract and tests capture. It also identifies limitations in the round-trip evaluation and outlines future tests of the artefacts’ use.

VIII. THREATS TO VALIDITY

  • An LLM judge may diverge from human judgement or favour verbose or same-vendor text; a concrete-defect rubric and three high-agreement runs mitigate this concern.The ranking runs against both proposed biases: the most concise generator scores best and same-vendor GPTO scores worst; RQ2–RQ3 do not depend on the judge.Source: §VIII · p. 10
  • Pretraining recall could affect reconstruction, but paired conditions share the agent, library, contract and test split, and audited sandbox runs had no access to copies.The agent and QW35 share a model family, which could favour QW35, although its larger gain is not significant.Source: §VIII · p. 10
  • The round trip uses small Python libraries and a single agent, so findings may not generalise to larger systems, other languages or other agents.The study mitigates this boundary with libraries spanning several domains and two generators from different families and verbosity levels, which give the same results.Source: §VIII · p. 10
  • The judge and repository statistics cover a stratified sample of 1,000 repositories by language, size and popularity.The sample is drawn from the larger repository collection.Source: §VIII · p. 10
  • With 29 libraries and variable runs, small effects are hard to detect; the study uses three runs per condition, multiple-test adjustment, a mixed model and confidence intervals.Coverage analyses are labelled exploratory.Source: §VIII · p. 10

IX. CONCLUSIONS

  • The framework recovers aligned stakeholder intent, requirements and architecture from source code and releases the resulting dataset of more than 10k repositories in four languages.Source: §IX · p. 10
  • FULL does not significantly change held-out test performance, but brings rebuilt module structure significantly closer to the original, carrying design information absent from the contract and tests.The conclusion frames this as the software Ship of Theseus evaluation.Source: §IX · p. 10
  • Future work will fine-tune models on aligned artefacts and code, use artefacts as multi-agent hand-offs, and test reconstruction without runnable tests or a prescribed interface.The authors also propose using structural similarity from the round trip as a reward for training generators.Source: §IX · p. 10

Related papers

Loading 2610.20068v2…