Source-linked AI summary
End-to-End Rationale Reconstruction
Mouna Dhaouadi, Bentley James Oakes, Michalis Famelis
TL;DR
Design rationale is valuable, but prior automatic extraction approaches are context-specific and no complete end-to-end pipeline captures its historical evolution. Kantara proposes an NLP/ML-based graph approach with validation mechanisms; a preliminary Linux Kernel example yielded promising results.
Problem
Prior automatic rationale-extraction techniques are context-specific, and prior work lacks an end-to-end pipeline for extraction, structuring, and management that captures decisions’ temporal evolution.
Source: §1 · p. 1Method
Kantara uses NLP and ML to extract decisions and rationales from textual artifacts, structure them in a graph, and propose mechanisms to check graph consistency.
Source: §1 · pp. 1–2Results
The preliminary Linux Kernel example produced promising results for the pipeline’s decision-rationale extraction and decision-relationship steps.
Source: §6 · p. 4, §3.3 · p. 4Takeaways & Limitations
The graph is intended to help stakeholders understand decision history and relationships and check development-process consistency.
Source: §1 · p. 1Kantara recovers recorded rationale, not the full actual rationale, which may never be recorded.
Source: §3.2.1 · p. 3
Abstract
from arXiv · showhide
The logic behind design decisions, called design rationale, is very valuable. In the past, researchers have tried to automatically extract and exploit this information, but prior techniques are only applicable to specific contexts and there is insufficient progress on an end-to-end rationale information extraction pipeline. Here we outline a path towards such a pipeline that leverages several Machine Learning (ML) and Natural Language Processing (NLP) techniques. Our proposed context-independent approach, called Kantara, produces a knowledge graph representation of decisions and of their rationales, which considers their historical evolution and traceability. We also propose validation mechanisms to ensure the correctness of the extracted information and the coherence of the development process. We conducted a preliminary evaluation of our proposed approach on a small example sourced from the Linux Kernel, which shows promising results.
1 INTRODUCTION
Design rationale can preserve useful experience, but it is implicit and scattered across artifacts, while prior automated approaches lack broad applicability, an end-to-end pipeline, and temporal modeling. Kantara proposes an NLP/ML-based approach that structures decisions and rationales in a graph and includes mechanisms to check graph consistency.
1 INTRODUCTION
- Rationale can preserve past experience for similar problems and best practices, but manual capture effort and delayed benefits have limited its adoption.The paper motivates automatic, non-intrusive extraction as a way to address both barriers.Source: §1 · p. 1
- Rationale is difficult to extract because it is implicit, poorly organized, and scattered across different artifacts.Existing automated NLP techniques are context-specific; prior work also lacks a complete extraction, structuring, and management pipeline and does not capture decisions’ temporal evolution.Source: §1 · p. 1
- Kantara aims to extract decisions and rationales from textual sources and artifacts, then structure them as a graph-based representation.The approach leverages NLP and ML and targets context-independent use.Source: §1 · p. 1
- The proposed contributions include a decision-and-rationale graph that accounts for historical evolution and artifact traceability, an end-to-end NLP-based extraction pipeline, and mechanisms to detect graph inconsistencies.The graph is intended to support understanding across time and space and checking whether proposed decisions conflict with prior decisions or repeat past attempts.Source: §1 · pp. 1–2
2 MOTIVATING EXAMPLE
The motivating example uses OOM-Killer commits and discussions to illustrate rationale reconstruction in a compact but semantically rich setting. The selected commits trace different approaches to reclaiming memory from a dying task.
2 MOTIVATING EXAMPLE
- The OOM-Killer selects a task to kill and signals or forces it to release memory when the kernel faces memory exhaustion.The authors use the component and related commits and discussions as their running example and preliminary evaluation object.Source: §2 · p. 2
- The example focuses on reclaiming used memory from the OOM victim, using manually selected commits from the OOM-Killer Git history.The component was chosen for its comprehensible scope, semantically rich decisions, and small number of commits and code.Source: §2 · p. 2
- Linux kernel patches include motivation or rationale, and links to LKML discussions are encouraged, making the kernel and LKML repositories of decision/rationale information.Patch summaries also provide a short description of each change.Source: §2 · p. 2
- The five selected commits move from raising dying-task priority, to reverting that approach after conflicts, to a dedicated reclamation thread and a user-triggered system call.The table gives each commit’s summary phrase, date, and description.Source: §2 · p. 2
3 END-TO-END RATIONALE RECOVERY
Kantara aims to create a context-independent graph of decisions, rationales, and relationships from textual artifacts, while preserving decision history and artifact traceability. The approach combines NLP and ML extraction stages, illustrated and preliminarily evaluated on Linux OOM-Killer commits.
3.1 Rationale and Decision Graph
- The graph represents decisions and rationales as nodes, links decisions to source artifacts for traceability, and groups decisions by topic.It also records history, semantic similarity, and contradiction relationships between decisions.Source: §3.1 · p. 2
3.2 Approach Overview
- Kantara’s two pipeline stages extract decision–rationale triples and then relationships between decisions from textual artifacts.The graph is intended to preserve historical evolution and interconnections between decisions.Source: §3.2 · p. 2
3.2.1 Extracting Decision-Rationale Triples.
- The decision classifier labeled D3, D4, and D5 correctly but missed D1 and D2; sentence-level SRL extracted rationales for D1 and D2.SRL may capture incomplete rationale when it is scattered across sentences, while D3’s rationale occurs in a different sentence; the authors also propose investigating manner information for D4 and D5.Source: §3.2.1 · pp. 2–3
- The approach only recovers recorded rationale, not the full actual rationale, which may never have been recorded.The authors warn that decisions based on partial rationale could have negative effects.Source: §3.2.1 · p. 3
3.2.2 Extracting Decision-Decision Relationships.
- Decision relationships include topic relatedness, semantic similarity, history, and contradiction, but the relatedness classifier performed poorly on the example and contradiction detection missed D3’s conflicts.The authors plan to investigate class-imbalance techniques for relatedness and heuristics for identifying historical links and reversions.Source: §3.2.2 · pp. 3–4
3.3 Preliminary Evaluation Summary
- The preliminary evaluation summary reports results for different Kantara pipeline steps and describes them as promising.The supplied passage does not give the individual evaluation values.Source: §3.3 · p. 4, Table 2 · p. 4
4 GRAPH VALIDATION MECHANISMS
The proposed validation mechanisms check graph and development-process consistency, and look for potential conflicts when a new decision is introduced.
4 GRAPH VALIDATION MECHANISMS
- The authors propose checking whether rationales are consistent with the relationships between their decisions.D1 and D2 are similar, and their rationales had semantic similarity of 0.92, which the authors describe as a strong indicator of consistency.Source: §4 · p. 4
- For a new decision, the proposed check uses its similarity and the graph’s history and contradiction links to flag earlier approaches and possible conflicts.For D5, the example finds similarity of 0.87 with D1 and would surface the earlier approaches and D3’s conflict with them.Source: §4 · p. 4
5 RELATED WORK
Prior rationale-extraction work differs from Kantara in its representation, source context, or technique.
5 RELATED WORK
- Prior work used ISAL-based design-document extraction, recovered but did not structure developer-discussion information, summarized code-change reasons, or used semantic grammars.The work in does not consider decision history, while focuses on code changes.Source: §5 · p. 4
6 CONCLUSION
The paper presents Kantara as an initial end-to-end, graph-based approach to rationale reconstruction. Its small Linux OOM-Killer example produced promising preliminary evaluation results, and the authors plan to develop an OD3 response-generation component.
6 CONCLUSION
- Kantara’s initial evaluation uses a small set of Linux OOM-Killer commits to illustrate and assess extraction of decision–rationale triples and decision–decision relationships.The authors describe the preliminary results as promising and outline plans concerning observed limitations.Source: §6 · p. 4
- The authors are building the OD3 response-generation component, with a GitHub bot prototype envisioned to assist end-users during development.The bot would use the knowledge graph created by Kantara.Source: §6 · p. 4