DRILS: Dependency-based Requirement Change Impact Analysis Using LLMs

1 Introduction

• Change can propagate across interdependent requirements, making analysis necessary to keep documentation aligned with system changes.Failure to synchronize documentation risks ambiguity, inconsistency, and a disconnect from the system's actual state.Source: §1 · p. 2
• Large specifications, especially in safety-critical systems, make CIA time- and resource-intensive and increase the risk of overlooking impacts or misjudging scope.Source: §1 · p. 2
• Existing LLM-based CIA work remains limited, and the authors identify no prior approach combining LLMs, dependency graphs, and technical knowledge from requirements specifications.Earlier dependency, model-based, and NLP approaches have distinct limitations, including insufficient semantic coverage, dependence on formalisms, or shallow linguistic features.Source: §1 · pp. 2–3
• DRILS uses RE-RAG for concise domain context, semantic clustering, two generator agents guided by a Requirement Dependency Graph, and a judge agent to select the final impact set.Generators also provide rationales, change locations, and confidence scores; the judge produces selection rationales for analysts.Source: §1 · p. 3
• 81% average F1 for DRILS versus 55% for ProReFiCIA [13], with precision rising from 41.33% to 76.3%.The authors report that the higher precision reduces falsely predicted impacts analysts must review.Source: §1 · p. 3
• The paper introduces three newly constructed CIA datasets to address the lack of publicly available datasets for the task.Source: §1 · p. 4

2 Approach

• DRILS takes system requirements, their Requirement Dependency Graph (RDG), and a change request through four stages to produce an impact set with supporting rationales.The stages are RE-RAG, clustering, impact-set identification, and impact-set validation.Source: §2 · pp. 4–5

2.1 Step 1: Requirement-Enrichment RAG (RE-RAG)

• RE-RAG resolves referential ambiguity using the five preceding requirements, then enriches each clarified requirement with domain context selected from summarized SRS sections.The enrichment agent selects the three most semantically similar summaries and generates concise, independent context sentences.Source: §2.1.1 · p. 6, §2.1 · p. 6, §2.1.2 · p. 7

2.2 Step 2: Requirements Clustering

• Clustering groups requirements by semantic similarity without domain knowledge, forming smaller batches while aiming to keep dependencies together for impact analysis.The method uses DBSCAN; the authors motivate clustering because random batches may split highly dependent requirements.Source: §2.2 · p. 8

2.3 Step 3: Impact-set Identification

• Two heterogeneous generation agents independently process each cluster with the RDG and change request, returning candidate impacts with rationales, suggested change locations, and confidence scores.A requirement counts as impacted when it should be updated or rewritten because it no longer accurately describes the functionality.Source: §2.3 · pp. 8, 9, 11
• The RDG represents Requires, Implements, Conflicts, and Details relationships as triples, supplying dependency information relevant to assessing change impacts.Requires, Implements, and Details are directional; Conflicts is not.Source: §2.3 · p. 9

2.4 Step 4: Impact-set Validation

• A Judge Agent reviews both generators’ outputs, removes requirements it considers false positives, and produces the final impact set with rationales.The judge is instructed to base its decision solely on information supplied by the Generation Agents.Source: §2.4 · pp. 11–12

3 Empirical Evaluation

• The evaluation proceeds backward from the Judge Agent to upstream components, then benchmarks DRILS against recent baseline methods.Research questions test auxiliary information, RE-RAG, clustering, and the requirement dependency graph before comparing DRILS with baselines.Source: §3 · pp. 12–13

3.2 Datasets

• Three automotive safety-critical systems provide separate datasets with dependency graphs; change rationales and impact sets were developed and adjudicated by two analysts.The systems are Automated Parking Assist, Traffic Jam Assist, and Adaptive Driving Beam; analysts refined rationales and reached consensus on impact sets. Mean Impact Density is below 7% in every dataset.Source: §3.2 · pp. 14–15

3.3 Baselines

• Baselines are selected to cover varied automated techniques without human intervention or large task-specific training sets.The comparison includes LSI, SBERT, LLM-Vec, and ProReFiCIA; similarity-based methods use two threshold strategies, while ProReFiCIA combines LLM ranking with NLI filtering.Source: §3.3 · p. 15, §3.3.2 · p. 16, §3.3.3 · p. 17

3.4 Implementation

• The implementation uses LLaMa 3.1 70B for RE-RAG, Claude Sonnet and GPT-4o as generators, and Gemini as the judge.Baseline implementations include LSI, SBERT, LLM-Vec, and ProReFiCIA.Source: §3.4 · pp. 17–18

3.5 Metrics

• Performance is measured with recall, precision, and F1, while operational measures capture macro-averaged effectiveness and the proportion of requirements requiring analyst review.Fisher’s Exact test assesses whether approaches differ significantly in recall and precision.Source: §3.5 · p. 18, §3.5.2 · pp. 18–19, §3.5.1 · p. 18

4.1 RQ1: Effect of Auxiliary Information Produced by the Generator Models on DRILS Performance

• DRILS averaged 80.9% F1 and 76.3% precision, versus 72.5% F1 and 63.53% precision without rationale; rationale inclusion also corresponded to lower cost.The authors attribute the precision benefit to the Judge Agent using the Generators’ justifications to assess whether proposed impacts are logically supported.Source: Table 2 · p. 20, §4.1 · p. 20, §4.1.1 · p. 21
• Removing change location raised average F1 from 80.9% to 83.1% but lowered recall from 87.1% to 84.1%, a tradeoff the authors reject because CIA prioritizes recall.The authors suggest change location can reveal impacts that are implicit in requirement text; DRILS also averaged 89.7% effectiveness versus 87.6% without it.Source: Table 2 · p. 20, §4.1.2 · p. 21
• Including confidence score yielded average F1 of 80.9% versus 79.8% without it, and effectiveness of 89.7% versus 86.8%; its overall metric effects were smallest.The authors describe confidence as an implicit quality signal that helps the Judge Agent assess the reliability of accompanying rationale and change-location information.Source: Table 2 · p. 20, §4.1.3 · pp. 21–22
• Removing rationale, change location, and confidence score reduced average F1 to 70.2%, recall to 71.6%, precision to 69.7%, and effectiveness to 79.8%, versus DRILS’s 80.9%, 87.1%, 76.3%, and 89.7%.The two settings had similar costs; the authors report the largest performance declines when all auxiliary information was removed.Source: Table 2 · p. 20, §4.1.4 · pp. 22–23

4.2 RQ2: Effect of pre-processing stages on DRILS performance

• DRILS with RE-RAG averaged 80.9% F1, versus 77.6% without RE-RAG and 78.1% with standard RAG, and achieved the highest recall across datasets, above 80% each.Its average effectiveness was 89.6%, compared with 84.1% without RE-RAG and 85.8% with standard RAG; average precision was approximately 76% across all three settings.Source: Table 3 · p. 23, §4.2.1 · p. 24
• Without RE-RAG, the Generator Agents’ union recall was 94.5%, versus 91.0% with DRILS, yet final average recall was 80.0% versus DRILS’s 87.1%.The authors suggest RE-RAG’s focused domain context improves auxiliary information for the Judge Agent, which can offset the smaller candidate pool.Source: Table 4 · p. 25, §4.2.1 · pp. 26–27
• DRILS’s deviations were at most 0.05 across all 15 metric–dataset combinations; its average deviation was 0.02, compared with 0.07 without RE-RAG and 0.05 with standard RAG.Across RQ1 configurations and datasets, average deviation was 0.027, and DRILS had deviations below 0.05 for 14 of 15 combinations.Source: §4.1.4 · p. 23, §4.2.1 · p. 25
• DRILS achieved 87.1% average recall, versus 82.3% without clustering and 84% with fixed batching, while average precision was 76.3%, versus 83.3% and 79.7%.No setting led F1 on all datasets; DRILS had the highest average effectiveness, while the no-clustering setting had the lowest cost.Source: Table 5 · p. 27, §4.2.2 · pp. 27–28
• DRILS-Batch has higher average F1 than DRILS (34.2% versus 27.0% precision), but DRILS has higher average recall (84.6% versus 83.6%).Batching yields higher union recall than DRILS (92.5% versus 91.0%), while DRILS’s higher TP and lower FP confidence scores may help the Judge retain true impacts; clustering also reduces variability, with average deviation 0.026 versus 0.038 without clustering and 0.037 for batching. Agent2 generally outperforms Agent1 and is less sensitive to input size.Source: §4.2.2 · pp. 28–29
• Integrating RE-RAG and clustering yields higher overall recall than excluding these techniques or using alternative configurations, primarily benefiting the Judge Agent.The authors attribute this to the higher quality of auxiliary information provided to the Judge Agent.Source: §4.2.2 · p. 29

4.3 RQ3: Effect of Requirement Dependency Graph on DRILS performance

• With the RDG, DRILS’s average F1 rises from 76.7% to 81% and recall from 81.2% to 87.1%, with precision also improving on D1 and D3.The RDG explicitly links requirements, helping Generator Agents find related impacted requirements; incorporating it also improves effectiveness while keeping average cost similar.Source: §4.3.1 · pp. 30–31, Table 7 · p. 30
• Adding the RDG improves effectiveness while keeping average cost similar, and DRILS’s cost deviation is lower than without the graph.Average effectiveness exceeds 90% on D1 and D2 with the RDG; average cost is 6.0% with RDG and 5.8% without it, while cost deviations average 0.026 and 0.05, respectively.Source: §4.3.1 · pp. 30–31, §4.3.2 · p. 31
• RDG inclusion gives DRILS the highest union recall on all three datasets, although average Generator Agent F1 results are mixed across settings.RDG inclusion is associated with higher TP confidence (4.73 versus 4.69) and lower FP confidence (3.60 versus 3.65); it also increases the gap between individual and union recall to 57.5 percentage points versus 53.1 without RDG. Agent2 outperforms Agent1 across evaluated settings and datasets.Source: §4.3.3 · pp. 31–32, §4.3.2 · p. 31

4.4 RQ4: Comparison of DRILS against Baseline Approaches

• DRILS leads 11 of 15 metric–dataset comparisons, averaging 87.1% recall, 76.3% precision, and 80.9% F1 across the three datasets.Against ProReFiCIA, DRILS’s precision rises from 44.5% to 64.9% for APA, 48.8% to 86.2% for TJA, and 30.7% to 77.8% for ADB. Table 9 reports the baseline comparison.Source: Table 9 · p. 33, §4.4.3 · p. 34
• For the semantic-similarity baselines, T2 yields higher average effectiveness and lower average cost than T1, while ProReFiCIA is the strongest baseline by F1 and cost.T2 averages 88.8% effectiveness and 26.7% cost versus T1’s 69.6% and 39.5%; ProReFiCIA averages 55.0% F1 and has 10.9% cost. Semantic similarity alone leaves each approach below 50% F1.Source: §4.4.1 · p. 33, §4.4.2 · pp. 33–34
• DRILS’s precision advantage over ProReFiCIA is statistically significant on APA, TJA, and ADB, while recall differences are not significant.DRILS averages 89.7% effectiveness at 6.0% cost, the lowest cost across all three datasets; the precision p-values are 2e-2, 3.3e-3, and 1e-4, while recall p-values exceed 0.05.Source: §4.4.3 · pp. 34–35

• DRILS consistently outperformed state-of-the-art baselines in recall, F1 score, effectiveness, and cost, while achieving substantially higher precision.Across three baseline categories, its precision was nearly twice that of the best-performing baseline, reducing analysts’ review effort.Source: §5 · pp. 35–36
• Rationales, change locations, and confidence scores provide complementary signals; rationales produced the most pronounced gains among the evaluated auxiliary information.Including all three together yielded more pronounced improvements across evaluation metrics.Source: §5 · p. 35
• RE-RAG outperformed chunk-based RAG and no RAG, particularly in recall, while unconstrained chunk context performed similarly to no RAG.Its benefits appeared especially during Judge Agent validation; the authors suggest concise, structured context may reduce noise.Source: §5 · pp. 35–36
• Clustering requirements yielded higher recall than fixed-size batching or processing the full list at once.The authors suggest semantic grouping can preserve relevant context while reducing information processed simultaneously; clustering particularly benefited Judge Agent validation.Source: §5 · p. 36
• Adding the requirement dependency graph improved overall recall, precision, F1 score, and effectiveness, benefiting both impact-set identification and validation.The graph adds relationship information that can help identify further impacted requirements and support better-grounded auxiliary information.Source: §5 · p. 36
• Although manual validation remains necessary, DRILS can automate a large part of CIA and reduce analysts’ verification effort.Rationales explaining selected requirements facilitate the subsequent manual validation of the impact set.Source: §5 · p. 36

• Internal validity may be affected by architecture design choices, attribution of performance differences, LLM stochasticity, and random variation in baseline comparisons.The authors used controlled ablations, temperature 0, three runs with negligible standard deviation, and Fisher’s exact test.Source: §6 · pp. 36–37
• External validity is bounded by three automotive datasets whose requirement counts may not reflect larger industrial systems, and by evaluation using GPT, Claude, and Gemini.The authors note that other domains and LLMs may yield different results and plan further model evaluation.Source: §6 · p. 37
• Manual annotation of ground-truth impact sets may introduce subjectivity and bias, despite independent annotation and consensus discussions by two researchers.Source: §6 · p. 37
• Reproducibility is a stated threat; detailed implementation reporting and publicly available artifacts support independent replication under identical conditions.Source: §6 · p. 37

7 Related Works

• CIA approaches span model-based, requirement dependency-based, NLP-based, and LLM-based techniques.The paper frames the categories against requirements that are complex, interdependent, and ambiguous in natural language.Source: §7 · p. 37

7.1 Model-Based Approaches

• Model-based methods use formal artifacts and structural traversal or slicing, sometimes combining candidates with NLP ranking, and depend on specific formalisms.Examples include SysML-to-design analysis, UML dependencies, and URN traceability.Source: §7.1 · p. 38, §7.5 · p. 39

7.2 Requirement Dependency-Based Approaches

• Dependency-based approaches propagate typed requirement links through logic or OWL reasoning; hybrid variants transform SysML diagrams into ontologies or require meta-model alignment.The approaches differ in how requirement relationships are formalized and traversed.Source: §7.2 · pp. 38–39

7.3 NLP-based Approaches

• NLP-based CIA uses lexical and syntactic features, with analyst-defined Boolean propagation conditions enabling phrase-level, context-aware impact scoring.Source: §7.3 · p. 39

7.4 LLM-Based Approaches

• LLM-based examples include GPT-4 with RAG for impact and implication analysis and ProReFiCIA’s prompting, refinement, and NLI-filtering pipeline.Source: §7.4 · p. 39

7.5 Limitations and DRILS Contribution

• Prior approaches face formalism dependence, sparse or implicit links, shallow NLP, and basic LLM prompting or RAG; DRILS combines multi-agent reasoning, RE-RAG, and dependency graphs as complementary evidence.The paper positions dependencies as complementing semantic reasoning rather than serving as the sole basis for analysis.Source: §7.5 · pp. 39–40

8 Conclusion

• DRILS uses RE-RAG, two Generator Agents guided by a requirement dependency graph, and a Judge Agent that filters false positives to improve precision.Source: §8 · p. 40
• 81% average F1 across three automotive datasets, with 87% average recall and 76% precision; removing RE-RAG, clustering, or the dependency graph reduced recall and F1.Removing all generator-agent auxiliary information produced an 18% relative decrease in recall.Source: §8 · p. 40
• Evaluating generalizability beyond the automotive domain and across diverse datasets is future work.The authors also plan to assess different LLMs and extend CIA across additional software artifacts.Source: §8 · p. 40