Source-linked AI summary
Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study
Amey Karan, Rudra Dhar, Mohamed Soliman, Karthik Vaidhyanathan
TL;DR
Recovering architectural design decisions from commits matters because commits are often available while architectural rationale is implicit and undocumented. This preliminary study evaluates four LLMs with zeroshot and fewshot prompting on 30 developer-written ADDs using automated metrics and manual review. All models achieved BERT-F1 above 0.81 and fewshot prompting improved alignment, but outputs often remained implementation-focused and omitted rationale.
Problem
Recovering ADDs from commits is underexplored because commits expose implementation changes while architectural rationale and context are often implicit.
Method
The study compares four LLMs with zeroshot and fewshot prompting on 30 developer-written ADDs, using similarity metrics and manual review.
Results
All four models achieved BERT-F1 above 0.81, and fewshot prompting improved alignment, including Gemini’s increase from 0.828 to 0.847.
Takeaways & Limitations
ADD extraction from commits is a higher-level architectural reasoning task rather than purely summarising implementation changes.
Takeaways & Limitations
The dataset contains 30 ADDs from three Apache repositories, is biased toward Java-based systems, and includes manual inspection only for Gemini outputs.
Abstract
from arXiv · showhide
Context: Architectural Design Decisions (ADDs) capture the rationale behind the structure and evolution of software systems but are rarely documented explicitly, and are often hidden inside source code commits. Recovering them is important for Architectural Knowledge Management (AKM). Problem: Extracting ADDs from commits is challenging due to their implicit and unstructured nature. Large Language Models (LLMs) have shown strong capabilities in understanding code and text, yet their effectiveness for this task remains underexplored. Study: We present a preliminary study using four LLMs (Gemini 3 Pro, DeepSeek R1, Kimi K2, Qwen3) with zeroshot and fewshot prompting on 30 developer-written ADDs from open-source projects. We score outputs with ROUGE-L, BLEU, METEOR, and BERTScore, and one author manually reviews the Gemini outputs. Results: All models reach a BERT-F1 above 0.81, and fewshot prompting improves alignment (Gemini BERT-F1: 0.828 to 0.847). However, the generated ADDs are often too long, implementation-focused, and miss the rationale behind the decision. This highlights opportunities for architecture-aware LLM systems and automated AKM.
1 Introduction
Architectural Design Decisions are central to architectural knowledge but are rarely documented explicitly, while commits preserve architectural evolution yet primarily expose low-level changes. This study therefore examines whether LLMs can recover ADDs and their rationale from commits.
- Architectural Design Decisions shape software structure, behaviour, and evolution but are rarely written down.
- Existing extraction approaches depend mainly on issue trackers and mailing lists, which are often incomplete or missing.
- Commits are consistently available records of system evolution, but recovering ADDs requires interpreting low-level modifications as broader design intent and rationale.
- LLMs are promising for this task because they work well on code and natural language, although their ability to recover implicit architectural rationale remains unclear.
- The study compares zeroshot and fewshot prompting across four LLMs on 30 developer-written ADDs, using automated metrics and manual review of Gemini outputs.
2 Related Work
Prior AKM approaches recover architectural knowledge from textual project artifacts, but those artifacts are often incomplete or unavailable. This work addresses the underexplored use of consistently available source code commits for ADD extraction.
- Issue discussions can contain recoverable architectural knowledge, and tools such as ADeX extract ADDs from project history artifacts.
- Existing approaches depend on detailed textual documentation that is often incomplete or unavailable in practice.
- Recovering ADDs directly from source code commits remains underexplored despite commits being consistently available records of architectural evolution.
- The study evaluates how effectively LLMs recover ADDs from commits alone and how prompting and input structuring affect extraction quality.
3 Study Design
The study uses a small, curated dataset of developer-written ADDs and compares four LLMs under zeroshot and dynamically retrieved fewshot prompting. Automated similarity metrics are complemented by structured manual inspection.
- Study Design: The study defines alignment as how closely a generated ADD reflects documented architectural intent, abstraction, and semantic meaning.
- Study Design: It asks how zeroshot extraction aligns with documented ADDs and how fewshot prompting changes that alignment.
- Dataset Curation: Researchers manually selected 30 commit–ADD pairs from Cassandra, Hadoop, and Tajo using clarity, single-commit mapping, and 60–100-word length criteria.
- Dataset Curation: Four LLMs were selected according to leaderboard ranking, size, and availability, including Gemini 3 Pro, DeepSeek R1, Kimi K2, and Qwen3.
- Prompting: Zeroshot uses a commit message and git diff, whereas fewshot adds dynamically retrieved documented ADD examples to guide abstraction and structure.
- Evaluation: Outputs are evaluated with ROUGE, BLEU, METEOR, and BERTScore, plus manual coding of abstraction level, verbosity, and rationale-related quality.
4 Results
LLMs achieved substantial semantic alignment with documented ADDs, and fewshot prompting generally improved automated scores. However, outputs frequently remained verbose, implementation-focused, and incomplete about decision rationale.
- Zeroshot Results: All four models reached BERT-F1 above 0.81, while Gemini led zeroshot performance with BERT-F1 = 0.828.
- Zeroshot Results: Low word-overlap scores indicate that generated ADDs often used different wording from documented decisions despite covering the same idea.
- Manual Inspection: Manual inspection found verbosity, incorrect abstraction level, and missing rationale as the main problems; about two thirds of outputs were 2 to 10 times longer than documented ADDs.
- Fewshot Results: Fewshot prompting improved scores across most metrics, with Gemini reaching BERT-F1 = 0.847 and ROUGE-L = 0.152.
- Fewshot Results: Gemini’s METEOR decreased from 0.175 to 0.165, showing that improved semantic alignment did not uniformly improve wording overlap.
- Fewshot Results: Fewshot examples shortened outputs and improved style, but implementation details and missing rationale persisted because the examples contained decisions rather than their reasoning.
5 Discussion
The discussion finds that commit-based ADD extraction requires architectural reasoning beyond summarization. LLM outputs can support AKM, but they often omit rationale and retain implementation details.
- Generated decisions usually identify changes but omit the rationale motivating them.The rationale often resides in issue discussions, design documents, or code reviews rather than commits.
- Fewshot examples teach expected ADD length and style but do not consistently produce the correct architectural abstraction.They may shorten outputs without teaching the model how to abstract over a diff.
- Commit-based ADD extraction is an architectural reasoning task rather than pure summarization.
- LLM-generated ADDs can serve as first drafts for AKM, but require rationale checking and removal of implementation details.
6 Threats to Validity
The study’s validity is constrained by metric fit, evaluation subjectivity, prompting and leakage risks, and a small, Java-biased dataset. These limitations make the findings preliminary.
- Automated similarity metrics may miss architectural meaning when equivalent ADDs use different wording or detail levels.A single author supplemented the metrics with manual review, using fixed criteria applied uniformly.
- Prompt wording can affect LLM outputs, while public project data creates a possible pretraining-data leakage risk.The study used a common prompt template and combined metrics with meaning-focused manual inspection.
- The dataset contains 30 ADDs from three Apache repositories and is biased toward Java-based systems.Selection was limited to clearly written ADDs tied to single commits, and manual review covered only Gemini outputs.
7 Conclusion and Future Directions
The study finds that LLMs can recover some architectural information from commits, while often producing implementation-focused ADDs without rationale. Future work will add contextual artifacts and broaden evaluation.
- All four models achieved BERT-F1 above 0.81, and fewshot prompting improved alignment.The generated ADDs nevertheless often remained implementation-focused and omitted rationale.
- Commit-only extraction is a higher-level architectural reasoning problem rather than purely summarization.
- Future work will incorporate issue discussions, code reviews, repository context, larger evaluations, and improved assessment methods.