PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models (LLMs) for Requirements Engineering (RE): A Systematic Literature Review

Mohammad Amin Zadenoori, Jacek Dąbrowski, Waad Alhoshan, Liping Zhao, Alessio Ferrari

arXiv:2509.11446v1cs.SE

TL;DR

Research on LLMs in RE has expanded, but the evidence is spread across studies examining varied tasks and practices. This SLR analyzes 74 primary studies to characterize that landscape. It finds a broadening of applications alongside persistent gaps in rigor, openness, evaluation, and integration into practice.

  • Problem

    The paper addresses the need to characterize how LLMs are being applied in RE across tasks, methods, and evaluation, an area not specifically covered by the related reviews discussed.

    Source: §8 · p. 55

  • Method

    The authors conduct a systematic literature review of 74 primary studies on LLMs in RE, categorizing their tasks, prompting, resources, and evaluation.

    Source: §4.3 · p. 14, §8 · p. 55

  • Results

    LLM4RE spans tasks from elicitation and validation to modeling and tracing, and extends into software engineering activities such as testing and code generation.

    Source: §8 · p. 56

  • Takeaways & Limitations

    The authors identify opportunities in underexplored RE tasks and call for more rigorous evaluation, reproducibility, broader artifacts, and integration into industrial workflows.

    Source: §8 · p. 56

    The preliminary review has not systematically cross-checked the retrieved and analyzed studies, and the authors plan reliability measures for future versions.

    Source: §4.2 · p. 11

Abstract

from arXiv · show

Large Language Models (LLMs) are finding applications in numerous domains, and Requirements Engineering (RE) is increasingly benefiting from their capabilities to assist with complex, language-intensive tasks. This paper presents a systematic literature review of 74 primary studies published between 2023 and 2024, examining how LLMs are being applied in RE. The study categorizes the literature according to several dimensions, including publication trends, RE activities, prompting strategies, and evaluation methods. Our findings indicate notable patterns, among which we observe substantial differences compared to previous works leveraging standard Natural Language Processing (NLP) techniques. Most of the studies focus on using LLMs for requirements elicitation and validation, rather than defect detection and classification, which were dominant in the past. Researchers have also broadened their focus and addressed novel tasks, e.g., test generation, exploring the integration of RE with other software engineering (SE) disciplines. Although requirements specifications remain the primary focus, other artifacts are increasingly considered, including issues from issue tracking systems, regulations, and technical manuals. The studies mostly rely on GPT-based models, and often use Zero-shot or Few-shot prompting. They are usually evaluated in controlled environments, with limited use in industry settings and limited integration in complex workflows. Our study outlines important future directions, such as leveraging the potential to expand the influence of RE in SE, exploring less-studied tasks, improving prompting methods, and testing in real-world environments. Our contribution also helps researchers and practitioners use LLMs more effectively in RE, by providing a list of identified tools leveraging LLMs for RE, as well as datasets.

1. Introduction

The review focuses on generative LLMs for RE, whose language-generation capabilities extend beyond encoder-only models. It surveys 74 studies and finds that LLM4RE emphasizes elicitation and validation, with evaluation and shared datasets still limited.

1. Introduction

  • Traditional rule-based and statistical NLP approaches struggled with the ambiguity and complexity of real-world requirements because they relied on predefined rules or shallow statistical models.The review situates its topic within longstanding NLP research in RE.Source: §1 · p. 2
  • The review examines generative pretrained transformers for RE rather than encoder-only models such as BERT, focusing on text understanding and production.The authors describe LLM4RE as the use of LLMs to support RE-relevant activities.Source: §1 · p. 3
  • 74 studies show LLM4RE concentrates on elicitation and validation, while evaluation is mostly laboratory-based and shared datasets remain scarce.The review also identifies underexplored tasks, including requirements retrieval and terminology extraction.Source: §1 · p. 3
  • The paper proceeds from background and related work through methodology, results, discussion, threats to validity, and conclusions.The authors provide extracted and synthesized data in supplementary spreadsheets.Source: §1 · p. 4

2. Background

The background frames RE as a lifecycle of language-intensive activities and LLMs as generative language systems. Its conceptual scheme organizes how surveyed studies apply and evaluate LLMs across tasks, artifacts, prompts, and resources.

2.1. Requirement Engineering

  • RE covers eliciting, documenting, analyzing, verifying, validating, and managing requirements, with natural language serving as a widely understood medium among stakeholders.The described lifecycle proceeds from elicitation and analysis through modeling, management, and verification and validation, before downstream SE activities.Source: §2.1 · pp. 4–5
  • RE tasks and artifacts extend beyond specifications to include user stories, feedback, and legal or regulatory documents, creating interpretation, integration, and consistency challenges.Tasks such as classification can be studied across different RE phases and artifact types.Source: §2.1 · p. 5, §2.3 · p. 8

2.2. LLMs

  • LLMs use transformer self-attention to capture contextual information and can generate coherent text, while their outputs may still involve bias or misinformation.The background also describes transfer learning as a way to adapt models to specialized domains.Source: §2.2 · p. 6

2.3. Scope of the Study: Conceptual Scheme

  • The study treats a primary study as its unit of analysis and classifies its demographics, RE usage, resources, and evaluation methods.RE usage connects tasks and phases to input and output artifacts; resources include models, datasets, code, prompts, and possible tools or solutions.Source: §2.3 · pp. 7–8

3. Related Reviews

Prior reviews cover LLM applications across software engineering and selected subfields, but the authors find no earlier review specifically targeting RE. They position this SLR as a continuation of research on NLP for RE.

3. Related Reviews

  • Existing reviews examine LLM applications across software engineering and identify supported tasks, strategies, challenges, and future research needs.The cited surveys also consider LLM-based agents and multi-agent systems.Source: §3 · p. 9
  • Reviews of software testing and software architecture describe LLM uses in testing and repair, as well as architecture tasks, while identifying underused areas [24].The software architecture review covers 18 articles and notes underuse in code generation and conformance checking [24].Source: §3 · p. 9
  • The authors state that none of the discussed reviews specifically targets RE and frame their study as a continuation of Zhao et al.’s review of 404 NLP-for-RE studies through 2019.Source: §3 · p. 9

4. Research Methodology

The authors follow a systematic review process to identify and categorize LLM-for-RE studies. They define research questions across publication demographics, RE usage, prompting, resources, and evaluation, then search, select, extract, and synthesize study data.

4. Research Methodology

  • The review follows Kitchenham’s guidelines, moving from research questions and a protocol through study selection, data extraction, consolidation, and reporting.Source: §4 · pp. 9–10

4.1. Research Questions

  • The research questions cover publication demographics, RE tasks and artifacts, prompting, resources and availability, and evaluation strategies.The questions also examine co-occurrences between input and output artifacts and how LLM outputs are assessed.Source: §4.1 · pp. 10–11

4.2. Literature Search and Selection

  • The September 11, 2024 query returned 244 studies; title-and-abstract screening yielded 136 papers, inclusion and exclusion criteria retained 62, and a December 18 venue-based search added 12, yielding 74 studies.Source: §4.2 · p. 12

4.3. Data Extraction and Synthesis

  • The first and last authors designed an extraction form mapped to the research questions and developed classification schemes from prior literature or inductively when needed.The first author performed extraction, with the last author consulted on doubts; systematic reliability checks were deferred to a future version.Source: §4.3 · pp. 14–15

5. Results

The review maps 74 LLM4RE studies published in 2023–2024, finding strong attention to elicitation and validation alongside growth in the range of RE artifacts and tasks studied. Publications increased in 2024, while the distribution of inputs, outputs, and input–output pairs indicates a varied research landscape.

5.1. RQ1: Demographics

  • 30 more studies appeared in 2024 than in 2023 (a 136% rise), bringing LLM4RE to 74 studies across 2023–2024.The review reports no LLM4RE papers before 2023 in its search results.Source: §5.1 · pp. 16, 18, Figure 3 · p. 17
  • Conferences account for 49 studies (66%), versus workshops’ 17 (23%) and journals’ 8 (11%).The RE conference leads with 21 studies (28%); 32 studies (43%) are scattered across one or two venues, including non-RE and non-SE venues.Source: §5.1 · pp. 16–17, Figure 4 · p. 17, Table 4 · p. 18

5.2.1. RQ2.1: RE Tasks

  • Elicitation and Validation each account for 15 studies (20%), while defect detection and classification rank eighth and fifth, respectively, in LLM4RE.Software Engineering Tasks include generating test cases from requirements; Requirements Retrieval and Terminology Extraction each appear in one study (1%).Source: §5.2.1 · p. 19, Table 5 · p. 20

5.2.2. RQ2.2: RE Phases

  • Elicitation and Validation & Verification each cover 16 studies (22%); Analysis covers 14 (19%), unlike the Analysis-led distribution in earlier NLP4RE work.The authors caution that elicitation involves stakeholder interaction and that V&V can be critical for compliance with standards or regulations.Source: §5.2.2 · pp. 21–22, Table 6 · p. 22

5.2.3. RQ2.3: Input Artifacts

  • Specifications are the most common input (28 studies, 38%); Issues and User Feedback account for 9 (12%), and Regulatory and Legal Artifacts for 7 (9%).Specifications made up 60% of inputs in the cited earlier NLP4RE survey, compared with 38% here.Source: §5.2.3 · p. 23, Table 7 · p. 24

5.2.4. RQ2.4: Output Artifacts

  • Requirements Specifications are the most common output (15 studies, 20%), followed by Analysis Reports (14, 19%) and Models and Designs (10, 14%).Outputs also include Compliance Documents and Legal Artifacts, User Profiles, Code, Testing and Traceability Artifacts, Interview Scripts and Questionnaires, and Process Descriptions.Source: §5.2.4 · pp. 25–26, Table 8 · p. 26

5.2.5. RQ2.5: Input and Output Artifacts Mapping

  • The 74 studies contain 27 input–output pairs, with Specifications–Analysis Reports the most frequent (6 studies, 8%).Specifications also pair with themselves in five studies (7%), often for completeness improvements or template translation.Source: §5.2.5 · pp. 27–28, Table 9 · p. 28, Figure 9 · p. 29

5.3. RQ3: Prompt Engineering Approaches

The review maps prompt strategies, prompt-selection practices, and formal guidelines, finding simple prompts prevalent and prompt selection often undocumented.

5.3.1. RQ3.1: Prompt Engineering Strategies

  • Zero-shot (28 studies, 38%) and K-shot (19 studies, 26%) dominate the eight prompt-engineering categories.The review notes that one paper may use more than one strategy.Source: §5.3.1 · p. 31, Table 10 · p. 32
  • Because papers could use multiple strategies, reasoning and task decomposition was the most common of these approaches (16 studies, 22%), compared with RAG-driven (5, 7%) and interactive strategies (3, 4%).Source: §5.3.1 · p. 31

5.3.2. RQ3.2: Prompt Selection Strategies

  • Prompt selection is Not-specified in 29 studies (39%), leaving a substantial share of selection practices undocumented.Source: §5.3.2 · p. 33, Table 11 · p. 34
  • Among specified selection methods, Criteria-based Refinement leads (19 studies, 26%), followed by Trial and Error (15, 20%), Predefined Guidelines (6, 8%), Domain Knowledge (4, 5%), and Automatic Refinement (3, 4%).Source: §5.3.2 · p. 33, Table 11 · p. 34

5.3.3. RQ3.3: Prompt Engineering Guidelines

5.4. RQ4: Resources

The review maps datasets, shared materials, model families, and model sizes used in LLM4RE studies, finding limited public resources and substantial reliance on GPT-family models.

5.4.1. RQ4.1 Datasets

  • Only 10 public datasets were identified, used by 12 of 74 studies (16%); many studies use private datasets, constraining reproducibility and comparisons.The datasets span tasks including regulatory compliance, defect detection, requirements modeling, validation, and elicitation.Source: §5.4.1 · p. 36, Table 13 · p. 37

5.4.2. RQ4.2: Source Code and Prompts

  • Source code, prompts, or both were provided by 36 of 74 studies (48%), while 38 (52%) provided neither.The review calls for more work to ensure replicability and verifiability.Source: §5.4.2 · p. 37

5.4.3. RQ4.3: Large Language Models

  • GPT-family models appear in 57 studies (77%), compared with LLaMA in 9 (12%), BERT in 8 (11%), and Mixtral/Mistral and CodeLLaMA in 3 each (4%).The review treats BERT as a baseline family, not an LLM family.Source: §5.4.3 · p. 38, Table 14 · p. 39
  • The median study used one distinct LLM, although 29 studies (39.19%) used more than one; the review links model obsolescence to validity and reproducibility risks.Studies ranged from one to six distinct models.Source: §5.4.3 · p. 38
  • Many studies use models above 70B parameters, which can depend on external services and specialized hardware, creating a gap with practical settings involving sensitive requirements data.The review points to smaller locally deployable or privacy-preserving models as potential alternatives; targeted comparisons can help control costs and resource demands.Source: §5.4.3 · pp. 38–39, Figure 12 · p. 40

5.5. RQ4.4: Tools and Solutions

The review identifies a small set of LLM tools and solutions across RE and SE tasks, with most studies using models out of the box rather than in complex workflows.

5.5. RQ4.4: Tools and Solutions

  • The review identifies 21 tools and solutions: 17 (80%) support RE tasks and 4 (20%) support SE tasks, including code generation and testing.Reported RE uses range from elicitation to modeling and defect detection.Source: §5.5 · p. 40, Figure 13 · p. 41, Table 15 · pp. 41–42
  • The studies usually use LLMs out of the box rather than integrating them into complex workflows; the review calls for combinations with AI techniques, formal methods, or domain-specific tools.The review characterizes many RE tasks as “hairy” and argues that meaningful progress requires richer workflows.Source: §5.5 · pp. 40, 43
  • Only 10 public datasets were identified, used in 16% of studies, and source code or prompts were shared by 48% of studies.The review says reliance on private datasets limits replicability and comparability, while greater sharing would improve transparency.Source: §5.5 · p. 43
  • GPT-family models dominate (77%), while many studies rely on models above 70B parameters, raising cost, infrastructure, and privacy concerns.The review calls for smaller, locally deployable alternatives.Source: §5.5 · p. 43

5.6. RQ5: Evaluation

The review examines both how LLM outputs are assessed and which empirical strategies studies use. Quantitative and mixed output assessment are common, while laboratory experiments dominate study designs.

5.6.1. RQ5.1: LLM Output Evaluation

  • Quantitative and Mixed Methods each assess LLM output in 30 studies (42%), while Qualitative Methods are used in 12 (16%).Generative tasks can lack a ground truth defined a priori, and rich outputs may require inspection by multiple judges.Source: §5.6.1 · pp. 43–44, Table 16 · p. 44, Figure 14 · p. 45

5.6.2. RQ5.2: Empirical Strategy

  • Laboratory Experiments dominate empirical strategies (56 studies, 76%); Judgment Studies account for 10 (14%), Field Studies 5 (7%), and Experimental Simulations 3 (4%).Sample Studies and Field Experiments were not used in the reviewed publications.Source: §5.6.2 · p. 45, Table 17 · p. 46, Figure 15 · p. 47
  • The dominance of laboratory experiments leaves limited evidence on real-world applicability and practitioner views across companies, while systematic field evaluations are lacking.Source: §5.6.2 · pp. 45, 48
  • Elicitation and Software Engineering Tasks have been evaluated using all empirical strategy types, while other RE tasks need more diverse assessment.Laboratory experiments have been applied across all tasks, especially elicitation (14.94%) and requirements validation (13.79%).Source: Figure 16 · p. 47, §5.6.2 · p. 48

6. Discussion

LLM4RE has expanded rapidly from NLP4RE’s analytic focus into generative RE and adjacent SE tasks, while the discussion identifies gaps in task diversity, prompting, openness, integration, and real-world evaluation.

6.1. LLM4RE as a Paradigm Shift

  • 74 LLM4RE studies appeared in two years, with 2024 publications up 136% over 2023, marking a rapid shift from NLP4RE’s slower analytic trajectory.Source: §6.1 · p. 49

6.2. LLM4RE Expands the Scope of RE

  • 20% of studies bridge RE and SE through tasks such as test or code generation, while over 57% appear in diverse AI, NLP, and HCI venues.The authors also connect RE expertise in requirement quality with prompt engineering, framing prompts as goals specified for an LLM or agent.Source: §6.2 · pp. 49–50

6.3. Rethinking LLM Applications in RE Tasks

  • Terminology extraction, relation extraction, and change-impact analysis remain underrepresented, while requirements specifications dominate inputs and diverse outputs and transformations remain scarce.User feedback and legal texts are less represented as inputs; exploratory transformations such as user stories to test cases appear in only 1% of studies each.Source: §6.3 · pp. 50–51
  • LLMs may complement classification and defect detection with generated justifications, addressing ambiguous ground truths and potentially increasing user acceptance.Source: §6.3 · p. 51

6.4. Toward Smarter Prompts for Requirements Engineering

  • Prompt practice is heterogeneous and often ad hoc: over 62% of studies combine strategies, while interactive and RAG methods remain limited and prompt choice is often underreported.The authors call for systematic prompt-design frameworks and empirical benchmarks across representative RE scenarios.Source: §6.4 · pp. 51–52

6.5. Openness and Reproducibility

  • Only 12 of 74 studies shared public datasets, which were rarely reused; 39% shared no supplementary materials, limiting reproducibility and cross-study comparison.The authors recommend shared repositories and availability standards, including adaptation of the NLP4RE study ID card.Source: §6.5 · pp. 52–53

6.6. Model Usage and Practical Deployment

  • The GPT family was cited in 77% of studies, while only a minority explored open-source alternatives; just 21 tools were reported and most applications remained out-of-the-box.The authors call for integrated workflows combining LLMs with other AI techniques, formal methods, and domain-specific tools.Source: §6.6 · pp. 53–54

6.7. Evaluation in Practice and Novel Frameworks

  • 76% of studies used laboratory experiments versus 7% field studies, and the authors recommend evaluating practical utility alongside output quality.They propose measures such as time savings, error reduction, stakeholder communication, usefulness, acceptance, and cognitive workload, alongside mixed methods and field studies.Source: §6.7 · p. 54

7. Threats to Validity

The authors identify threats in study selection, data extraction, and classification, and describe steps taken or planned to address them.

  • Search selection may miss relevant papers because the search string and single Scopus database constrain retrieval, despite a piloted query and secondary venue search.Source: §7 · p. 55
  • Data extraction remains vulnerable to subjective bias: expert cross-checking covered only a sample, with full cross-checking deferred to a future version.Source: §7 · p. 55
  • Some classification categories were author-provided, introducing subjectivity, though two authors developed the schemes iteratively; a defined protocol and open analyses support research validity.Source: §7 · p. 55

8. Summary and Conclusion

This review synthesizes 74 studies of LLMs in RE from 2023–2024 and describes a rapid expansion across RE and SE tasks. It identifies opportunities alongside gaps in openness, reproducibility, evaluation, and practical integration.

  • The review synthesizes 74 primary studies from 2023–2024 and characterizes LLM4RE as a rapid, expansive shift beyond NLP4RE.Source: §8 · p. 55
  • LLMs have been applied to RE tasks from elicitation and validation to modeling and tracing, and to SE activities such as testing and code generation.Source: §8 · p. 56
  • LLM4RE re-centers requirements as a foundation for SE and expands generative and interpretative work, but terminology extraction, traceability, and compliance remain underexplored.The authors identify openness, reproducibility, evaluation, and practice integration as critical gaps in a field still dominated by exploratory proof-of-concept studies.Source: §8 · p. 56
  • Advancing LLM4RE requires rigorous evaluation, reproducible practices, systematic prompt engineering, broader tasks and artifacts, and end-to-end workflows reflecting industrial practice.Source: §8 · p. 56
  • The authors frame prompt engineering as a form of requirements specification and identify an opportunity for RE to influence broader AI reliability, including in safety-critical domains.Source: §8 · p. 56

Appendix A. Code Availability

Appendix A presents a table of RE tasks and code availability as a resource for locating implementations.

  • Appendix A.18 summarizes requirements-engineering task categories alongside code availability.Source: Appendix A · p. 71, Table A.18 · p. 71
  • The table lists each study citation, its mapped broad RE task, and a repository or dataset link, including only studies with available code.Source: Appendix A · p. 71
  • The listing serves as a lookup resource for researchers and practitioners seeking code implementations for RE-related tasks.Source: Appendix A · p. 71
Loading 2509.11446v1…