Source-linked AI summary
Automated Extraction and Analysis of Developer's Rationale in Open Source Software
Mouna Dhaouadi, Bentley Oakes, Michalis Famelis
TL;DR
Manually tracing commit-message history to avoid conflicts with past decisions is laborious, and no implemented automated mechanism had been presented for detecting reasoning conflicts. The paper instantiates Kantara to extract and analyze rationale, finding decision relationships and potential inconsistencies in OOM-Killer and applying the approach to other projects. Its findings support rationale analysis across the studied projects, while the English-only training dataset limits confidence about other linguistic and cultural contexts.
Problem
Manually reviewing related commit messages to avoid conflicts with past decisions is tedious, time-consuming, and error-prone, and prior work had not implemented automated reasoning-conflict detection.
Source: §1 · p. 2Method
The authors instantiate Kantara with pretrained models, LLMs, and graph-based mechanisms to extract and analyze decisions and rationale from commit messages.
Source: §1 · p. 2, §3 · p. 4, §4.1.1 · p. 6Results
The approach found 29 similar and 396 contradictory decisions in OOM-Killer, with a manually validated portion judged reasonable and worthy of developer inspection.
Source: §4.1.2 · p. 9Takeaways & Limitations
The approach detected 85 OOM-Killer cases of contradictory decisions with similar rationales and generalized to other Linux modules and open-source projects.
Source: §8 · p. 20The classifiers were trained on a limited-size, English-only OOM-Killer dataset, so generalization to other cultural and linguistic contexts may be weaker.
Source: §6.2 · p. 17
Abstract
from arXiv · showhide
Contributors to open source software must deeply understand a project's history to make coherent decisions which do not conflict with past reasoning. However, inspecting all related changes to a proposed contribution requires intensive manual effort, and previous research has not yet produced an automated mechanism to expose and analyze these conflicts. In this article, we propose such an automated approach for rationale analyses, based on an instantiation of Kantara, an existing high-level rationale extraction and management architecture. Our implementation leverages pre-trained models and Large Language Models, and includes structure-based mechanisms to detect reasoning conflicts and problems which could cause design erosion in a project over time. We show the feasibility of our extraction and analysis approach using the OOM-Killer module of the Linux Kernel project, and investigate the approach's generalization to five other highly active open source projects. The results confirm that our automated approach can support rationale analyses with reasonable performance, by finding interesting relationships and to detect potential conflicts and reasoning problems. We also show the effectiveness of the automated extraction of decision and rationale sentences and the prospects for generalizing this to other open source projects. This automated approach could therefore be used by open source software developers to proactively address hidden issues and to ensure that new changes do not conflict with past decisions.
1 INTRODUCTION
The paper addresses the effort and risk involved in manually tracing commit-message rationale to avoid conflicts with past decisions. It instantiates Kantara to extract and analyze rationale, then evaluates feasibility and generalizability across Linux modules and other open-source projects.
1 INTRODUCTION
- Manually tracing related commit messages to avoid conflicts with past decisions is tedious, time-consuming, and error-prone.Commit messages preserve rationale that can erode or evaporate as contributors enter and leave long-lived projects.Source: §1 · pp. 1–2
- The paper addresses the lack of an implemented automated mechanism for detecting reasoning conflicts, despite existing frameworks for rationale extraction and management.It extends Kantara, which structures decisions and rationale in a knowledge graph for analyses such as relationships, conflicts, and inconsistencies.Source: §1 · p. 2
- The authors define rationale information as sets of decision-rationale pairs and reasoning conflicts as mutually exclusive decisions.They also examine rationale presence in commit structure over time, while noting that further work is needed to distinguish types and levels of rationale.Source: §1 · p. 2
- The approach uses Semantic Role Labeling, pretrained models, and LLMs, assuming rationale is present in commit messages and focusing on its presence without AI optimization.The authors frame the work as applying AI techniques to Software Engineering rather than making a technical AI contribution.Source: §1 · p. 2
- The study demonstrates feasibility on Linux OOM-Killer commits and investigates generalization to two other Linux modules and five other open-source projects.Its contributions are an automated method to extract, structure, and contextualize decisions and rationale, and rationale knowledge from the studied projects.Source: §1 · p. 2
2 BACKGROUND
Kantara structures decision-rationale information from text as a graph whose relationships support rationale analyses. The paper addresses the gap between earlier preliminary work and a concrete automated implementation, using an OOM-Killer dataset and excluding supporting facts from its analysis.
2 BACKGROUND
- Kantara’s inference component turns sentences containing decisions and their rationale into a Rationale and Decision Graph, whose decision nodes are linked by relationships such as Similar and Contradicts.An analysis interface uses these relationships to examine inconsistencies and conflicts.Source: §2 · p. 3
- The current article addresses the lack of a concrete, fully automated implementation left by Kantara’s earlier preliminary ideas and small commit example.Kantara was introduced as a high-level architecture with information inference and an analysis interface.Source: §2 · p. 3
- The study uses a manually labeled OOM-Killer dataset of 2,234 sentences from 404 commit messages, tagged for Decision, Rationale, and Supporting Facts.The OOM-Killer selects and kills a task to free memory when the machine is running out of memory.Source: §2 · p. 3
- Supporting Facts, which capture information about system state at commit time, are excluded to remove temporal dependencies that could bias the evaluation.The study treats Linux sub-projects as separate projects because they have distinct concerns and committers.Source: §2 · p. 3
3 METHODOLOGY AND EVALUATION OVERVIEW
The methodology instantiates Kantara, analyzes the resulting graph, automates sentence extraction, and tests generalization to unseen projects. Its evaluation addresses feasibility and generalizability through detection counts, classifier performance, and manual or metric-based correctness checks.
3 METHODOLOGY AND EVALUATION OVERVIEW
- The methodology has four components: graph construction, rationale analyses, automated sentence extraction, and generalization to unseen data.The first three components address feasibility questions; the fourth addresses generalizability.Source: §3 · pp. 3–4
- Graph construction extracts decision and rationale entities from 774 jointly labeled OOM-Killer sentences and detects Similar and Contradicts relationships.The analysis mechanisms use graph relationships to identify similar or contradictory decisions and inconsistencies that may indicate graph-construction errors or reasoning problems.Source: §3 · p. 4
- The study evaluates whether the approach finds similar and contradictory decisions and inconsistencies, reporting detection counts and manually validating samples.It also measures processing time on the OOM-Killer corpus.Source: §3 · pp. 4–5
- Kantara operates only on sentences containing both decision and rationale, so it does not handle cases where they span separate or multiple sentences.This is why extracting decision-containing and rationale-containing sentences is necessary to apply the approach beyond OOM-Killer.Source: §3 · p. 5
- Two binary classifiers, trained on the OOM-Killer dataset, automate extraction of decision-containing and rationale-containing sentences.The study compares candidate models using cross-validation and standard classification metrics, with default hyperparameters and comparison to human performance.Source: §3 · p. 5
- Generalization is evaluated on two other Linux modules through manual sample validation and inter-rater agreement, and on five other open-source projects against original labels using standard metrics.The study also applies the approach to extract, structure, and contextualize rationale and presents detected inconsistencies in those projects.Source: §3 · p. 5
4 OUR INSTANTIATION OF KANTARA
The Kantara instantiation builds a decision-and-rationale knowledge graph from OOM-Killer commit sentences, then uses extracted relationships to analyze conflicts and inconsistencies. Its analyses found 86 inconsistencies, while entity extraction and graph construction had limitations in accuracy and runtime.
4.1 Rationale and Decision Graph construction
- The graph represents decision and rationale entities linked in pairs, with separate relationships between decisions, using 774 labeled sentences after removing two nonsensical entries.Construction requires extracting and linking entities from sentences, then detecting relationships between decisions.Source: §4.1 · p. 6
4.1.1 Entities Extraction.
- With Cause, Purpose, and Manner tags, structured-prediction-srl-bert detected 317 sentences at 41% accuracy, but SRL missed more than half of rationale-bearing sentences.The structured-prediction-srl model detected 300 sentences at 39% accuracy using those tags.Source: §4.1.1 · p. 6
- ChatGPT produced 527 non-null decision-rationale tuples, but at least one entity was null for 247 of the 774 sentences and some tuples contained noise.The authors continued with ChatGPT because it outperformed SRL on non-missed sentences, 527 to 317.Source: §4.1.1 · pp. 7–8
4.1.2 Relationship extraction.
- At a 0.9 threshold, the models found 29 similar and 396 contradictory relationships among 138,601 decision pairs.The authors manually validated a portion of the detections as reasonable and worthy of developer inspection.Source: §4.1.2 · pp. 8–9
4.2 Rationale Analyses
- 86 inconsistencies were detected, including 85 cases where contradictory decisions had similar rationales.The mechanisms flag both similar decisions with contradictory rationales and contradictory decisions with similar rationales.Source: §4.2 · pp. 10–11
4.3 Performance
- Building the OOM-Killer graph took about three hours; the authors consider this reasonable if construction occurs once per project and is maintained incrementally.Runtime depends on model inference and available hardware, and may be considerably longer for larger projects.Source: §4.3 · p. 11
5 AUTOMATION AND GENERALIZATION
The Bi-LSTM classifiers identify decision- and rationale-containing sentences with more than 90% accuracy and human-like performance on OOM-Killer data. They generalize to other Linux modules and OSS projects, where the approach also surfaces potentially conflicting decisions for developer review.
5.1 Automating the extraction of decision and rationale sentences
- Bi-LSTM achieves 92% accuracy for Decision and 90% for Rationale, with performance comparable to human raters and better than the best human performance for rationale.The models were evaluated with 10-fold cross-validation; hyperparameters were not optimized.Source: §5.1 · pp. 12–13, Table 5 · p. 12, Table 8 · p. 13
5.2.1 Classifiers generalization.
- Using already pre-processed commit messages, the classifiers show reasonable generalization across the five OSS projects considered together, with overall F1-scores of 0.70 for decisions and 0.75 for rationale.The messages omit identifier information, which may affect the analysis.Source: §5.2.1 · p. 14
5.2 Investigation of Generalization
- The approach generalizes without fine-tuning the LLM for decision-rationale tuples or the pretrained models for similar and contradictory relationships.The reusable pipeline combines binary classifiers, LLM entity extraction, and pretrained-model relationship extraction.Source: §5.2.1 · pp. 14–15
5.2.2 Application and inconsistency detection in other contexts.
- The classifiers and Kantara instance identify potentially conflicting decisions in other projects, including pairs with similar decisions but contradictory rationales and contradictory decisions with similar rationales.The examples include a possible identifier typo needing correction elsewhere; developer evaluation is still necessary to determine whether flagged cases are truly problematic.Source: §5.2.2 · pp. 15–16, Table 12 · p. 16
6 DISCUSSION
The discussion outlines how rationale analysis could support developers and researchers, while identifying limits involving data, assumptions, evaluation choices, and model selection.
6.1 Implications
- Embedding rationale identification in development tools could encourage rationale-aware commit messages and help developers take ownership of AI-generated code.The authors also link preserving rationale to countering design evaporation.Source: §6.1 · p. 16
- Rationale graphs can help developers reuse similar decisions and inspect contradictions or inconsistencies to address hidden issues proactively.The authors also connect these tools with countering design erosion and taking preventive action before changes are submitted.Source: §6.1 · p. 17
- The approach could create rationale repositories for research on project characteristics and support automated rationale-aware comments, commit messages, and documentation.CoMRAT is described as able to replicate the OOM-Killer analysis for any GitHub module.Source: §6.1 · p. 17
6.2 Threats to Validity
- The limited, English-only OOM-Killer training dataset may constrain extractor generalization to other cultural and linguistic contexts.The authors report validation on other Linux modules and OSS projects but identify different developer communication and documentation styles as a boundary.Source: §6.2 · p. 17
- The approach assumes rationale is consistently present in commits and confined to one sentence, limiting its scope when either condition fails.The authors plan to replicate the approach in other contexts and test original commit messages because preprocessing may have influenced extracted relationships.Source: §6.2 · p. 17
- Using the union of three raters’ labels as ground truth may bias classifier training toward including rationale, though the authors consider this threat minor.They base that assessment on classifiers’ reported generalization and human-like performance.Source: §6.2 · p. 17
- Default hyperparameters and limited model comparisons leave room for improvement through optimization or alternative pretrained models.The authors note that optimization may be computationally costly and that fine-tuned BERT could outperform Bi-LSTM.Source: §6.2 · pp. 17–18
- Heuristically selected thresholds and residual temporal dependencies may affect relationship interpretation and evaluation.The authors plan user-study threshold investigation and further study of temporal validation techniques.Source: §6.2 · p. 18
7 RELATED WORK
Related work spans rationale extraction, management, and Linux development studies; the paper distinguishes its focus on commit messages and automated rationale analysis from these lines of research.
Related work
- The authors describe their work as the first to use LLMs for developer rationale extraction and management.Other cited LLM studies address model completion [5], program repair, test scenarios, and class-level code generation.Source: §7 · p. 18
- Linux studies address logging rationale [37], developer motivation [53], uncivil communication [14], and patch-submission language; earlier work studied OOM-Killer rationale characteristics [9].This paper instead uses Linux kernel commit messages for automated analysis and extraction training.Source: §7 · pp. 18–19
- This work focuses on Linux kernel commit messages both as material for automated rationale analysis and as training data for extraction.It differs from prior extraction work by examining rationale in its natural textual form rather than only predefined categories or state transitions.Source: §7 · p. 19
- Other rationale research examines end-user reviews, design documents, and developer decision processes through interviews and surveys [27], rather than software developers’ commit-message rationale.The cited developer studies report 15 rationale components and links between implementation decisions and higher-level design [27].Source: §7 · p. 19
- Prior What/Why commit-message studies focused on extraction, not rationale management, and did not investigate pretrained models for extraction.Tian et al. [52] studied What and Why information; Li et al. also considered link content.Source: §7 · p. 19
- Existing rationale-management systems use different representations and techniques and do not propose automated detection of reasoning inconsistencies.Examples include Condec Tools, the patent rationale system, AdeX, and TREx.Source: §7 · pp. 19–20
- Compared with their earlier Kantara implementation, this work uses advanced neural models and LLMs, an ML-based implementation, and investigates generalizability.The earlier approach used basic ML models and an ontology-based implementation with openCAESAR.Source: §7 · p. 20
8 CONCLUSION
The paper concludes that its Kantara-based approach can analyze rationale in commit messages, identify contradictory decisions, and generalize beyond OOM-Killer. It proposes developer validation and tool development as future work.
8 CONCLUSION
- The approach identified 85 OOM-Killer cases of contradictory decisions with similar rationales, which can help identify reasoning problems in project history.The authors also report detecting similar decisions and inconsistencies.Source: §8 · p. 20
- The authors report that the approach generalizes to other Linux modules and other open source projects.They describe this as opening the possibility of wider application.Source: §8 · p. 20
- Future work includes validating extracted relationships with knowledgeable developers, packaging the implementation as a GitHub bot, and improving models and prompts.The authors also propose developing a research tool for creating rationale datasets.Source: §8 · p. 20