PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Requirements Engineering: A Cross-Task Empirical Evaluation

Jacek Dąbrowski, Manjeshwar Aniruddh Mallya, Alessio Ferrari, Mohammad Amin Zadenoori, Yijun Yu

arXiv:2608.21531v1cs.SE

TL;DR

Empirical evidence on LLMs for Requirements Engineering remains fragmented across tasks, leaving limited understanding of how findings generalise. This paper combines two empirical studies spanning five RE activities. It finds that effectiveness is strongly task-dependent, with no universally superior model or prompting strategy.

  • Problem

    Empirical LLM evidence in RE remains fragmented across single-task studies, with limited cross-task generalisability and rare joint assessment of technical performance and practical applicability.

    Source: §3.2 · p. 8, §3.3 · p. 8

  • Method

    Two complementary empirical studies integrate evidence across five representative RE activities.

    Source: §8 · p. 31

  • Results

    LLM effectiveness is strongly task-dependent; no model or prompting strategy is universally superior.

    Source: §8 · p. 31

  • Takeaways & Limitations

    Adoption should match models and prompts to each RE activity, while retaining human expertise for engineering judgement, factual grounding, and validation.

    Source: §8 · p. 31

    The industrial traceability findings are exploratory rather than definitive because analysed artefacts were limited and no complete ground truth existed.

    Source: §6.5 · p. 29

Abstract

from arXiv · show

Requirements-related information is scattered across heterogeneous artefacts such as user feedback, developer discussions, and software repositories, making the extraction of actionable requirements knowledge labour-intensive and hard to scale. Large Language Models (LLMs) can support many Requirements Engineering (RE) activities, from classification and traceability identification to specification and explanation generation, but existing evidence is fragmented across tasks, artefact types, and evaluation settings, and studies rarely offer cross-task evaluations or replication packages. We present two complementary empirical studies evaluating LLMs across five RE-related activities. The first is a controlled experiment on five lightweight open-source LLMs for feedback-driven requirements classification and specification generation. The second is an exploratory industrial case study on two frontier LLMs for traceability link identification and traceability explanation generation using real project artefacts. Classification and traceability identification were assessed with quantitative metrics, and generation tasks through human evaluation. LLM performance is strongly task-dependent, ranging from moderate to high, and no single model consistently outperformed the others, indicating that effective adoption depends on selecting models and prompting strategies per task. Our contributions are: (i) the first cross-task empirical evaluation of LLMs spanning five RE-related activities, (ii) replication materials supporting reproducibility, and (iii) a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.

1 Introduction

The paper addresses fragmented evidence on LLMs in Requirements Engineering with two complementary empirical studies spanning five activities, and provides replication materials to support reproducibility. It situates this work against the challenge of extracting actionable knowledge from diverse project artefacts.

Motivation

  • RE practitioners must extract actionable requirements knowledge from high-volume, diverse, unstructured information distributed across artefacts such as feedback, issue tickets, discussions, and repository documentation.These artefacts provide complementary evidence about stakeholder needs, implementation decisions, and software evolution.Source: §1 · p. 2

LLMs for RE

  • Earlier NLP and ML approaches often require substantial labelled data and generalise poorly, whereas LLMs can support diverse analytical and generative RE activities through natural-language prompts.The described activities include classification, traceability analysis, specification, and explanation generation.Source: §1 · pp. 2–3

Research gap

  • Evidence on LLMs in RE remains fragmented across tasks, artefacts, and evaluation settings, while few studies provide cross-task evaluations or public replication materials.This limits understanding of capabilities, limitations, cross-context generalisation, and cumulative empirical knowledge.Source: §1 · p. 3

Studies and scope

  • The paper combines five lightweight open-source LLMs on three feedback-driven tasks with two frontier LLMs on traceability identification and explanation using heterogeneous project artefacts.The first study covers non-functional requirements classification, user request classification, and specification generation; the exploratory industrial case study covers traceability link identification and explanation generation.Source: §1 · p. 3

Contributions

  • The paper contributes the first cross-task evaluation spanning five RE activities, replication materials, and broader empirical understanding of current LLM capabilities and limitations.Source: §1 · p. 3

2 Background

The background frames LLMs as prompt-adaptable tools for diverse RE tasks and explains how model capabilities and prompting strategies shape their application. It motivates evaluating lightweight models on feedback-driven tasks and frontier models on traceability across project artefacts.

2.1 Large Language Models

  • LLMs support reasoning, classification, summarization, and generation, and can adapt to diverse RE tasks through natural-language prompts rather than task-specific models [1].Their broad capabilities motivate their growing use in Requirements Engineering [1].Source: §2.1 · p. 4
  • Lightweight LLMs run locally with lower resource needs, while frontier models offer stronger reasoning and larger context windows but may raise cost and privacy concerns [1].The paper associates lightweight models with shorter artefacts such as user feedback and frontier models with reasoning across multiple software artefacts.Source: §2.1 · p. 4
  • Study I examines five locally deployable lightweight models on feedback-driven RE tasks, while Study II uses GPT-5.5 and DeepSeek-R1 for traceability analysis across project artefacts.Local execution enabled controlled, reproducible experimentation; lightweight models were deemed impractical for Study II’s long, heterogeneous artefacts because of limited context windows and reasoning capabilities.Source: §2.1 · pp. 4–5

2.2 Prompting Strategies

  • Prompt formulation influences model behaviour and output quality; appropriate prompting can improve reasoning, consistency, and clarity for RE tasks requiring accurate interpretation and structured outputs [1] [14].Prompts are natural-language instructions specifying a task and expected output [1].Source: §2.2 · p. 5
  • The study considers zero-shot, few-shot, chain-of-thought, constraint-based, and role-based prompting, evaluating all five in Study I and zero-shot and chain-of-thought in Study II [1] [14].The strategies respectively use task descriptions, labelled examples, explicit reasoning, rules or templates, and a specified professional perspective; Study II assesses frontier-model reasoning in an industrial setting rather than optimising prompts.Source: §2.2 · pp. 5–6

3 Related Works

Related work traces RE’s shift toward data-driven analysis and LLM-supported activities, while identifying fragmented cross-task evidence, limited comparisons across models and settings, and incomplete replication support. These gaps motivate the paper’s cross-task evaluation and replication package.

3.1 Modern Requirements Engineering

  • Modern RE has shifted from stakeholder-centric practices toward DDRE, which complements stakeholder knowledge with evidence from feedback, issue reports, repositories, and developer discussions.AI4RE applies ML, information retrieval, knowledge-based techniques, and NLP to activities including classification, information extraction, traceability recovery, and defect detection.Source: §3.1 · pp. 6–7
  • NLP4RE focuses on textual software artefacts, but earlier AI/NLP approaches often target individual tasks and artefact types or require task-specific models and labelled data [1] [43].LLMs extend this landscape by supporting analytical and generative RE activities through natural-language prompting across diverse artefacts [1].Source: §3.1 · p. 7

3.2 Large Language Models for Requirements Engineering

  • LLM-for-RE research spans analytical tasks such as classification and traceability recovery and generative tasks such as specification, refinement, and explanation; recent work also explores elicitation, validation, testing, modelling, and design support [1] [43].Studies increasingly use prompt-driven and generative approaches [1] [43].Source: §3.2 · p. 7
  • Evaluations commonly rely on frontier models and zero- or few-shot prompting, while lightweight models and advanced prompting receive less attention; few studies combine controlled experiments with industrial case studies [1].Advanced prompting approaches discussed include chain-of-thought reasoning, RAG, and agent-based workflows.Source: §3.2 · p. 8

3.3 Research Gap

  • Most LLM-for-RE studies examine a single task, while analytical and generative tasks and controlled and industrial settings are rarely assessed together, limiting generalisability and practical insight [1] [43].Studies also often consider one LLM family, leaving limited evidence on trade-offs between lightweight and frontier models [1].Source: §3.3 · p. 8
  • Replication support is often incomplete or unavailable [1] [14]; this study addresses the gap with cross-task evaluation of lightweight and frontier LLMs in controlled and industrial settings, plus a complete replication package.The evaluation spans multiple analytical and generative RE tasks.Source: §3.3 · p. 9

4 Motivating Scenarios

The section presents two complementary RE scenarios: deriving requirements from stakeholder feedback and tracing requirements across heterogeneous project artefacts. Together, they motivate studies of LLM support for these lifecycle activities.

4.1 Scenario 1: From User Feedback to Requirements

  • Scenario 1 uses noisy, high-volume app reviews to motivate classifying actionable needs and NFRs and drafting requirements; it motivates Study I on lightweight open-source LLMs.Classifications could support elicitation, prioritisation, and software evolution; generated statements or user stories would be initial drafts, not replacements for a complete specification, and could preserve traceability to feedback.Source: §4.1 · pp. 9–10

4.2 Scenario 2: Requirements Across Project Artefacts

  • Scenario 2 links Rust roadmap goals, GitHub issues, and Zulip discussions to support traceability, evidence-grounded rationale explanations, and impact analysis; it motivates Study II.The relationships could help contributors follow how features evolve and anticipate effects on implementation activities, technical dependencies, and downstream systems such as operating systems and embedded software.Source: §4.2 · pp. 10–11

5 Study I: From User Feedback to Requirements

This study evaluates five lightweight open-source LLMs on app-review classification and requirements-specification generation. Performance varies by task and prompting strategy; the models show promise but still require human oversight.

5.1 Study Context

  • The study treats app reviews as feedback instances and evaluates request-type and NFR classification plus transformation of classified feedback into structured requirements specifications.Request labels distinguish feature requests, bug reports, and other; NFR labels follow ISO/IEC 25010 categories.Source: §5 · p. 11, §5.1 · p. 12

5.2.3 Evaluation Metrics and Criteria

5.2.2 Datasets

  • Two public annotated datasets support the evaluation: an NFR dataset with 4,000 labeled sentences and a request dataset with 2,912 reviews, sampled across apps and domains.For specification generation, 90 annotated reviews were sampled from both datasets; generated outputs were assessed by the first author using five criteria.Source: §5.2.2 · p. 13, Evaluation Procedures · p. 15

5.2.4 Experimental Setup and Procedure

  • Classification prompts used zero-shot, few-shot, and chain-of-thought strategies; specification generation also used role-based and constraint-based prompts.Models ran three times per task at temperature 0 with a fixed seed and no hyperparameter tuning; classification scores were averaged over runs.Source: Computational Setup · pp. 14–15, Evaluation Procedures · p. 15

5.3 Results

  • For classifying feedback by NFR type, LLMs achieve moderate accuracy (F1≈0.47–0.51); chain-of-thought prompting performs best, with Gemma leading overall at P=0.53, R=0.57, and F1=0.55.Source: §5.3 · p. 16
  • Few-shot prompting yielded the best request-classification average (F1=0.68), with Llama 3 highest (P=0.72, R=0.75, F1=0.74).Across strategies, average F1 ranged from 0.59 to 0.68; chain-of-thought averaged 0.64.Source: §5.3 · pp. 16–17, Table 7 · p. 17
  • Generated specifications were moderate in quality overall (mean=3.1; SD=0.8); Llama 3 with chain-of-thought and Mistral with few-shot tied highest (mean=3.6).Both top configurations scored SA=4 and CL=5; few-shot and chain-of-thought improved structure and clarity, while constraint-based prompting offered limited gains.Source: §5.3 · p. 17, Table 8 · p. 18

5.4 Discussion

  • Lightweight models can assist with feedback filtering and initial drafting but are not yet reliable for unsupervised industrial use; generated text may lack formal structure or factual grounding.Validity is limited by single-evaluator subjective generation assessment, a small specification sample without inferential tests, and use of app reviews rather than industrial artefacts.Source: §5.4 · pp. 17–19, §5.5 · p. 19

6 Study II: Requirements Across Project Artefacts

Study II explores whether frontier LLMs can support requirements traceability across heterogeneous project artefacts through an industrial case study at Huawei Research Centre.

6 Study II: Requirements Across Project Artefacts

  • The exploratory industrial case study evaluates two frontier LLMs on traceability link identification and explanation generation [1].It is based on observations from a research visit at Huawei Research Centre.Source: §6 · p. 20

6.1 Industrial Study Context

The industrial study examines requirements information in Rust project goals, GitHub tracking issues, and Zulip messages, where distributed evidence makes reconstructing requirement evolution labour-intensive. It evaluates identifying traceability links and generating evidence-grounded explanations.

6.1 Industrial Study Context

  • The study uses Rust project goals, GitHub tracking issues, and Zulip messages because reconstructing requirement evolution across these artefacts is labour-intensive.Together, the artefacts capture project objectives, implementation activities, technical decisions, and project evolution.Source: §6.1 · p. 20
  • Project goals state objectives and expected outcomes, tracking issues document their implementation, and Zulip messages capture related technical discussions and decisions.A traceability link connects a goal or tracking issue with a Zulip message describing the same development objective or implementation activity.Source: §6.1 · p. 20
  • The study examines traceability link identification and evidence-grounded explanation generation for Rust project artefacts.Identification finds Zulip messages related to a project goal or tracking issue; explanation generation describes the identified link and its supporting evidence.Source: §6.1 · p. 20
  • For a Rust-for-Linux compiler-support goal, identification selects a message reporting successful nightly builds, while explanation connects that progress to the goal.The generated explanation grounds the link in the message that remaining compiler changes were merged and Rust-for-Linux builds successfully on nightly.Source: §6.1 · p. 20

6.2 Empirical Study Design

Study II uses an industrial case study to evaluate frontier LLMs for traceability-link identification and evidence-grounded explanation generation across heterogeneous software artefacts. It combines quantitative retrieval assessment with human evaluation of explanations.

6.2.1 Research Questions

  • RQ4 asks how well LLMs identify links between Project Goals or GitHub tracking issues and Zulip discussions; RQ5 asks how well they explain those links.Links are manually validated, and explanations are assessed by human judgment against predefined criteria.Source: §6.2.1 · p. 21

6.2.2 Dataset

  • The dataset contains nine Rust Project Goals cases from 2024–2026, each pairing a goal with its GitHub tracking issue and associated Zulip discussion.The discussions total 80,642 messages and the cases span compiler engineering, tooling, language design, verification, infrastructure, and project governance.Source: §6.2.2 · p. 21

6.2.3 Pilot Study

  • The pilot found that evidence-oriented and chain-of-thought prompting improved technical grounding, while finer-grained artefacts improved retrieval precision and whole-thread retrieval added unsupported contextual noise.Goal–Zulip and Issue–Zulip retrieval presented different levels of ambiguity, so the study evaluated them separately and retained manual validation against surrounding discussions and Rust documentation.Source: §6.2.3 · p. 22

Evaluation Metrics for Traceability Identification (RQ4).

  • For RQ4, the study reports Precision@5 and NDCG@5 separately for Goal–Zulip and Issue–Zulip retrieval, but omits recall because complete link ground truth is unavailable.The top-five cutoff limits time-intensive manual validation while focusing on the highest-ranked candidates.Source: Evaluation Metrics for Traceability… · p. 23

Evaluation Criteria for Explanation Generation (RQ5).

6.2.5 Experimental Setup and Procedure

  • The experiment covers 72 configurations of GPT-5.5 and DeepSeek-R1, with two prompting strategies, two document types, and nine topics; each configuration runs once.The prompts use zero-shot as a baseline and chain-of-thought for multi-step reasoning; a second author’s review yielded kappa values of 0.83 for link validation and 0.71 for explanation ratings.Source: §6.2.5 · pp. 23–25

6.3 Results

Frontier LLMs achieved moderate-to-high traceability-link identification and generated high-quality traceability explanations, with results varying by artefact pair, model, and prompting strategy.

RQ4: Traceability link identification

  • Frontier LLMs achieved moderate-to-high traceability-link identification; ChatGPT led on Issue–Zulip links under chain-of-thought prompting (P@5=0.91; NDCG@5=0.98).For Goal–Zulip links, chain-of-thought raised average P@5 from 0.59 to 0.73 and average NDCG@5 from 0.87 to 0.90; both models reached P@5=0.73, while ChatGPT achieved NDCG@5=0.94. Issue–Zulip links had higher average P@5 than Goal–Zulip links (0.77 versus 0.73); DeepSeek performed best on Issue–Zulip links with zero-shot prompting, with performance decreasing under chain-of-thought.Source: §6.3 · pp. 25–26, Table 13 · p. 26

RQ5: Traceability explanation generation

  • Frontier LLMs generated high-quality traceability explanations, averaging 4.15 for Goal–Zulip and 4.21 for Issue–Zulip.DeepSeek with chain-of-thought achieved the highest Goal–Zulip mean (4.27; CR=4.21; EG=4.33), while ChatGPT had the highest Goal–Zulip correctness score (CR=4.23). ChatGPT with zero-shot prompting achieved the highest Issue–Zulip mean (4.35; CR=4.35; EG=4.34); chain-of-thought improved Goal–Zulip explanations for both models, while DeepSeek showed only modest improvement on Issue–Zulip explanations.Source: §6.3 · p. 26, Table 14 · p. 27

6.4 Discussion

The discussion presents frontier LLMs as promising aids for traceability link recovery and explanation across project artefacts, but stresses the need for human review and practical adoption tradeoffs.

A) Traceability Link Identification

  • Issue–Zulip links were identified more reliably than Goal–Zulip links, while chain-of-thought particularly improved Goal–Zulip identification; downstream use still requires human verification.Issue–Zulip links are semantically more direct, whereas Goal–Zulip links span strategic objectives and implementation discussions, requiring more reasoning.Source: §6.4 · p. 27

B) Traceability Explanation Generation

  • Frontier LLMs generated generally correct, evidence-grounded explanations, with chain-of-thought benefiting Goal–Zulip explanations more than Issue–Zulip explanations.The explanations can help developers understand artefact relationships during repository exploration, but should support rather than determine interpretations of project history.Source: §6.4 · p. 27

C) Implications for Requirements Engineering

  • Traceability identification and explanations can reduce manual reconstruction effort and support rationale understanding, impact analysis, and developer onboarding.Commercial frontier LLMs raise vendor lock-in, privacy, and cost concerns, while lightweight models' limited context windows and slow local execution hinder large-repository analysis; frontier models should augment, not replace, engineers.Source: §6.4 · pp. 27–28

6.5 Threats to Validity

The study’s findings are exploratory and context-bound: manual interpretation, informal artefacts, and limited ground truth constrain the reliability, generalisability, and strength of its conclusions.

Internal Validity

  • Manual traceability-link and explanation assessment required Rust-specific interpretation; predefined criteria, independent subset review, substantial agreement, and Rust references mitigated but did not eliminate bias.Source: §6.5 · p. 28

External Validity

  • The single Rust ecosystem case enabled transparent evaluation, but other projects may differ, and the study claims illustration rather than direct generalisability.The case was selected for its publicly available project goals, tracking issues, and developer discussions; comparable industrial datasets are rarely public due to confidentiality.Source: §6.5 · p. 28

Construct Validity

  • Informal community artefacts may only partially instantiate RE constructs; established measures and predefined correctness and evidence-grounding dimensions mitigated but did not remove underspecification.Source: §6.5 · p. 28

Conclusion Validity

  • Limited artefacts, no complete ground truth, and costly manual validation make conclusions exploratory, not causal or definitive evidence of effectiveness.The study reports observations from applying off-the-shelf LLMs in a realistic open-source setting, rather than benchmarking model performance.Source: §6.5 · p. 29

7 Cross-study Discussion

LLM effectiveness and prompting vary by RE task, and human verification remains necessary. The discussion calls for task-matched adoption and future research that improves grounding and evaluates complementary RE activities.

7.1 Cross-Task Perspective on LLM Performance in RE

  • No model or prompting strategy consistently led across RE activities; performance ranged from moderate to high, supporting task-specific adoption rather than universal model rankings.The synthesis recommends selecting models according to the characteristics and reasoning demands of the target task, with human expertise retained for engineering judgment and validation.Source: §7.1 · p. 29, Table 15 · p. 30

7.2 Prompting Matters, but Depends on the RE Task

  • Chain-of-Thought generally benefited NFR classification and traceability tasks, whereas few-shot prompting best supported user-request classification, making prompt choice specific to task demands.The passages identify traceability identification and explanation generation as tasks benefiting from Chain-of-Thought prompting.Source: §7.2 · pp. 29–30, Table 15 · p. 30

7.3 Human Oversight Remains Essential

  • Well-defined classification and traceability decisions were generally reliable but still need verification; multi-artefact reasoning and generated artefacts risk unsupported or incomplete outputs, so LLMs augment practitioners.Grounding and consistency issues become more likely when tasks synthesize information across artefacts, preserve semantic consistency, or generate new text.Source: §7.3 · p. 30

7.4 Practical Implications for Requirements Engineering

  • Practitioners should jointly select models and prompts for task demands and validate outputs before use, rather than relying on model popularity or rankings.Lightweight open-source models performed well on short-context feedback analysis but were impractical for heterogeneous-artefact traceability analysis, leading the study to use frontier models; all classifications, links, specifications, and explanations require validation.Source: §7.4 · pp. 30–31

7.5 Future Research Directions

  • Future research should improve factual grounding, investigate retrieval and fine-tuning approaches, and evaluate complementary RE activities with common datasets, protocols, and replication packages.The proposed hybrid approaches include retrieval-augmented generation, repository-aware retrieval, and fine-tuning on RE-specific corpora.Source: §7.5 · p. 31

8 Conclusion

The paper evaluates LLMs across five RE activities through two complementary studies, finding that effectiveness depends on the task and that human expertise remains essential. It offers cross-task evidence and replication materials to support practical adoption and future research.

8 Conclusion

  • The two complementary studies provide the first cross-task empirical evaluation spanning five representative RE activities, addressing fragmented evidence about LLM capabilities and limitations.Source: §8 · p. 31
  • LLM effectiveness depends on the RE task, artefacts, and required outputs; no model or prompt strategy is universally superior, and human expertise remains essential for judgement, grounding, and validation.Source: §8 · p. 31
  • Evaluating complementary RE activities together offers practical guidance for integrating LLMs into engineering workflows and clarifies where they are effective and where limitations remain.Source: §8 · pp. 31–32
  • The replication package includes datasets, prompts, scripts, and experimental materials to support transparency and reproducibility, while the authors encourage future studies spanning multiple RE activities.Source: §8 · p. 32

Supplementary information.

  • Study 1 was partly conducted within Mallya’s MSc thesis, Study 2 largely during Dąbrowski’s research visit to Huawei Ireland Research Center, and the research was jointly funded by Research Ireland and the European Union.Source: Acknowledgements · p. 32

Related papers

Loading 2608.21531v1…