PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Requirements Engineering: A Cross-Task Empirical Evaluation

Jacek Dąbrowski, Manjeshwar Aniruddh Mallya, Alessio Ferrari, Mohammad Amin Zadenoori, Yijun Yu

arXiv:2608.21531v1cs.SE

TL;DR

Requirements knowledge is distributed across heterogeneous artefacts, while evidence about LLMs for RE remains fragmented across tasks and evaluation settings. The paper combines a controlled experiment and an industrial case study across five RE activities, finding that LLM effectiveness is strongly task-dependent and requires task-specific model and prompt selection.

  • Problem

    Evidence on LLMs for RE is fragmented across tasks, artefact types, and evaluation settings, with few cross-task evaluations or replication materials.

  • Method

    The paper combines a controlled experiment on five lightweight open-source LLMs with an exploratory industrial case study on two frontier LLMs across five RE-related activities.

  • Results

    LLM effectiveness is strongly task-dependent, with performance shaped by task reasoning demands, artefact characteristics, and required outputs.

  • Takeaways & Limitations

    Effective RE adoption requires selecting models and prompts for the specific activity, while human expertise remains essential for judgement, grounding, and validation.

  • Takeaways & Limitations

    The traceability case study is exploratory: analysed artefacts are limited, complete ground truth is unavailable, and conclusions are not causal benchmarks of effectiveness.

Abstract

from arXiv · show

Requirements-related information is scattered across heterogeneous artefacts such as user feedback, developer discussions, and software repositories, making the extraction of actionable requirements knowledge labour-intensive and hard to scale. Large Language Models (LLMs) can support many Requirements Engineering (RE) activities, from classification and traceability identification to specification and explanation generation, but existing evidence is fragmented across tasks, artefact types, and evaluation settings, and studies rarely offer cross-task evaluations or replication packages. We present two complementary empirical studies evaluating LLMs across five RE-related activities. The first is a controlled experiment on five lightweight open-source LLMs for feedback-driven requirements classification and specification generation. The second is an exploratory industrial case study on two frontier LLMs for traceability link identification and traceability explanation generation using real project artefacts. Classification and traceability identification were assessed with quantitative metrics, and generation tasks through human evaluation. LLM performance is strongly task-dependent, ranging from moderate to high, and no single model consistently outperformed the others, indicating that effective adoption depends on selecting models and prompting strategies per task. Our contributions are: (i) the first cross-task empirical evaluation of LLMs spanning five RE-related activities, (ii) replication materials supporting reproducibility, and (iii) a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.

1 Introduction

Requirements-related information is dispersed across heterogeneous artefacts, while existing LLM evidence remains fragmented across RE tasks and evaluation settings. This paper addresses the gap with two complementary studies spanning five RE activities and different LLM categories.

  • Motivation: Heterogeneous artefacts make extracting actionable requirements knowledge labour-intensive and difficult to scale.Practitioners must analyse information distributed across user feedback, issue tickets, developer discussions, and repository documentation.
  • Research gap: Earlier NLP and ML approaches commonly target individual tasks, require labelled data, and generalise poorly across projects and artefact types.
  • LLM opportunity: LLMs extend RE support from analytical activities such as classification and traceability to generative activities such as specification and explanation generation.They can be adapted through natural-language prompts rather than task-specific models.
  • Research gap: Empirical evidence on LLMs in RE remains fragmented, with few cross-task evaluations or publicly available replication materials.This limits broader understanding, reproducibility, and accumulation of empirical knowledge.
  • Approach: The paper presents two complementary empirical studies covering five RE-related activities.Study I evaluates five lightweight open-source LLMs on feedback-driven classification and specification; Study II evaluates two frontier LLMs on traceability analysis and explanation generation.
  • Contributions: The work contributes a cross-task evaluation, replication materials, and broader evidence about current LLM capabilities and limitations for RE.

2 Background

The paper compares lightweight and frontier LLMs and examines prompting strategies as alternative ways to support RE tasks. Lightweight models are studied in controlled feedback tasks, while frontier models address longer, heterogeneous traceability artefacts.

  • Large Language Models: LLMs can support RE through reasoning, classification, summarization, and text generation using natural-language prompts.
  • Model categories: Lightweight LLMs require fewer resources and can run locally, but generally offer weaker reasoning and smaller context windows than frontier LLMs.Local execution supports privacy-sensitive environments, controlled experimentation, and resource-constrained projects.
  • Models evaluated: Study I evaluates five lightweight open-source models, whereas Study II evaluates two frontier models for traceability analysis across software project artefacts.
  • Study I: Study I uses representative locally deployable models to examine task performance rather than identify a single universally best model.Local execution also provides control over the experimental environment and supports reproducibility.
  • Study II: Lightweight models were considered impractical for long, heterogeneous traceability artefacts because of limited context windows and reasoning capabilities.Study II therefore evaluates GPT-5.5 and DeepSeek-R1 as frontier reasoning models with larger context windows.
  • Prompting Strategies: The study uses zero-shot prompting as a baseline and chain-of-thought prompting to support multi-step reasoning, while intentionally excluding more complex prompt designs.

3 Related Works

Prior LLM-for-RE research spans analytical and generative tasks and increasingly diverse artefacts, but evidence remains difficult to generalise across tasks, models, settings, and studies. This work addresses these gaps through a cross-task evaluation and complete replication package.

  • Evolution of RE research: AI4RE and NLP4RE research has progressed from task-specific analysis toward general-purpose language models for textual software artefacts.
  • Existing LLM research: Existing LLM studies cover analytical tasks such as classification and traceability recovery alongside generative tasks such as specification and explanation generation.
  • Research gap: Most empirical studies still evaluate a single RE task or application scenario, leaving cross-task generalisation unclear.
  • Research gap: Previous evaluations often separate analytical from generative tasks and controlled experiments from industrial case studies, limiting combined insight into technical performance and practical applicability.They also provide limited comparison between lightweight and frontier LLMs.
  • Reproducibility: Reproducibility support is frequently incomplete or unavailable, limiting independent validation, cross-study comparisons, and cumulative evidence.
  • Contribution: This study combines multiple analytical and generative tasks, lightweight and frontier models, a controlled experiment and an industry-motivated case study, and a complete replication package.

4 Motivating Scenarios

The paper motivates two complementary RE scenarios: extracting and specifying stakeholder needs from noisy user feedback, and tracing requirements across heterogeneous project artefacts. Together, they cover different stages of the RE lifecycle.

  • Scenario I: User feedback: Mobile app reviews contain feature requests, bug reports, and software-quality concerns, but their volume and noise make manual analysis challenging.
  • Scenario I: User feedback: Automated classification can organise reviews by request type and non-functional requirement type, supporting requirements elicitation and classification.
  • Scenario I: User feedback: Automated generation can transform classified feedback into requirement statements, user stories, or an initial Software Requirements Specification.For example, a request for dark mode can be reformulated as a requirement statement or user story.
  • Scenario I: User feedback: Study I investigates lightweight open-source LLMs for feedback classification and requirements specification.
  • Scenario II: Project artefacts: Requirements information in project roadmaps, GitHub issues, and developer discussions supports traceability, rationale understanding, and impact analysis but is difficult to reconstruct manually.
  • Scenario II: Project artefacts: Automated traceability links and evidence-grounded explanations can connect project goals, issues, and discussions while clarifying how features evolve.
  • Scenario II: Project artefacts: Study II investigates state-of-the-art LLMs for traceability across heterogeneous software project artefacts.
  • Lifecycle coverage: The two scenarios cover complementary lifecycle stages: eliciting and documenting requirements from feedback, then tracing and explaining them across project artefacts.

5 Study I: From User Feedback to Requirements

Study I evaluates five lightweight open-source LLMs on classifying mobile-app feedback and generating requirements specifications. Performance is moderate to high depending on task, model, and prompting, while further customization and human supervision remain necessary for reliable industrial use.

  • Study scope: Five lightweight open-source LLMs were evaluated on user-request classification, NFR classification, and requirements specification generation from app reviews.The study used human-annotated feedback datasets and assessed classification quantitatively and specification quality through human judgment.
  • Feedback classification: NFR classification achieved moderate performance, with average F1 increasing from 0.47 under zero-shot to 0.51 under chain-of-thought prompting.Gemma achieved the highest overall F1 of 0.55 under chain-of-thought, while Llama 3 achieved the highest recall of 0.59.
  • Feedback classification: User-request classification achieved average F1 values of 0.59, 0.68, and 0.64 for zero-shot, few-shot, and chain-of-thought prompting, respectively.Llama 3 led under few-shot prompting with P=0.72, R=0.75, and F1=0.74.
  • Specification generation: Specification generation produced moderate quality overall (mean=3.1; SD=0.8), with Llama 3 and Mistral reaching the highest mean score of 3.6.Models generally performed better on clarity and completeness than on fidelity and conciseness.
  • Practical implications: Lightweight models can assist with feedback filtering, classification, and initial drafting, but their moderate accuracy, structural failures, and occasional fabrication require human review.The study positions them as drafting and documentation aids rather than autonomous specification generators.
  • Threats to validity: Results may not generalize beyond app reviews or lightweight models, and specification quality was manually assessed by a single evaluator without reliability validation.The authors identify potential subjectivity and limited external validity as threats.

6 Study II: Requirements Across Project Artefacts

Study II is an exploratory industrial case study evaluating frontier LLMs for traceability analysis across heterogeneous software project artefacts. It examines both traceability-link identification and evidence-grounded explanation generation.

  • Study II overview: The exploratory industrial case study evaluates two frontier LLMs on traceability link identification and traceability explanation generation.It extends the controlled experiment in Study I with a realistic project setting.

6.1 Industrial Study Context

Study II analyzes requirements-related information distributed across Rust project goals, GitHub tracking issues, and Zulip developer discussions. It treats traceability identification and evidence-grounded explanation generation as complementary tasks.

  • Artefact context: The industrial context combines project goals, GitHub tracking issues, and Zulip messages from the Rust open-source ecosystem.These artefacts represent objectives, implementation activities, technical decisions, and project evolution.
  • Study tasks: Traceability link identification retrieves Zulip messages related to a project goal or tracking issue, while explanation generation describes the link and supporting evidence.The example connects completed compiler changes in a Zulip message to a Rust-for-Linux project goal.

6.2 Empirical Study Design

Study II evaluates frontier LLMs on traceability retrieval and explanation generation using a Rust dataset of linked project artefacts. The design combines pilot-informed prompting, manual validation, retrieval metrics, and rubric-based human assessment.

  • Study design: The study evaluates two frontier LLMs in a realistic industrial case study addressing traceability-link identification and explanation generation.The research questions assess link retrieval and explanation quality across project artefacts.
  • Dataset: The dataset contains nine Rust traceability cases linking Project Goals, GitHub tracking issues, and Zulip discussions, with 80,642 associated messages.Cases span compiler engineering, tooling, language design, verification, infrastructure, and project governance.
  • Pilot study: A pilot refined prompting, artefact selection, and manual validation before the full study.It used two Rust project goals and their associated issues and discussions.
  • Pilot findings: Evidence-oriented and chain-of-thought prompting produced more technically grounded explanations, while finer-grained artefacts improved retrieval precision.Entire discussion threads added contextual noise, and plausible links still required manual verification.
  • Evaluation: Traceability identification was evaluated with manually validated Precision@5 and NDCG@5, while explanations were scored for correctness and evidence grounding on five-point Likert scales.The top-five cutoff reflects the time-intensive nature of manual validation and practical focus on highly ranked candidates.
  • Experimental procedure: The experiment comprised 72 executions across two models, two prompting strategies, two document types, and nine project topics, with each configuration run once.The prompting comparison used zero-shot as a baseline and chain-of-thought for multi-step reasoning.

6.3 Results

Frontier LLMs achieved moderate-to-high performance in traceability link identification and high-quality explanation generation, with outcomes varying by model, link type, and prompting strategy.

  • Traceability link identification: Chain-of-thought increased average Goal–Zulip P@5 from 0.59 to 0.73 and average NDCG@5 from 0.87 to 0.90.
  • Traceability link identification: ChatGPT achieved the highest Issue–Zulip scores under chain-of-thought prompting, with P@5=0.91 and NDCG@5=0.98.
  • Traceability link identification: Issue–Zulip identification outperformed Goal–Zulip identification on average, with P@5 of 0.77 versus 0.73.
  • Traceability explanation generation: Mean explanation scores ranged from 3.94 to 4.35, averaging 4.15 for Goal–Zulip and 4.21 for Issue–Zulip links.
  • Traceability explanation generation: DeepSeek with chain-of-thought achieved the highest Goal–Zulip mean score at 4.27, while ChatGPT with zero-shot prompting reached 4.35 for Issue–Zulip.

6.4 Discussion

Frontier LLMs show promise for reconstructing traceability across heterogeneous artefacts, but industrial adoption remains constrained by verification needs and practical deployment trade-offs.

  • Issue–Zulip links performed better than Goal–Zulip links because issues provide more direct semantic connections to developer discussions than high-level goals.
  • Chain-of-thought particularly improved Goal–Zulip identification, suggesting explicit reasoning helps bridge strategic objectives and implementation discussions.
  • Traceability identification and explanation generation can reduce manual effort in reconstructing relationships among goals, issues, and developer discussions.
  • Frontier models raise vendor lock-in, privacy, and operational-cost concerns, while lightweight models remain impractical for large repositories.
  • Human verification remains necessary because incorrect links or explanations could propagate errors into impact analysis and release planning.

6.5 Threats to Validity

Validity is limited by interpretive manual evaluation, a single Rust-based case study, limited artefact counts, and incomplete ground truth.

  • Internal validity: Manual validation required technical interpretation, including Rust-specific concepts, so some evaluator bias may remain despite predefined criteria and substantial inter-rater agreement.
  • External validity: External validity is constrained because the study uses a single case study from the Rust ecosystem, selected for its unusually transparent artefacts.
  • Conclusion validity: The study does not establish causal relationships or benchmark model performance; it reports observations from off-the-shelf LLMs in a realistic open-source setting.
  • Conclusion validity: Manual-validation effort limited the number of analysed artefacts, and the selected artefacts lacked complete ground truth.

7 Cross-study Discussion

Across the two studies, LLM performance depended on the RE task and prompting strategy rather than on a universally best model, while human oversight remained necessary.

  • Cross-task findings: The cross-study synthesis covers requirements classification, requirements specification, traceability identification, and traceability explanation generation.
  • Cross-task findings: No single model or prompting strategy consistently achieved the best performance across the evaluated RE tasks.
  • Prompting strategies: Chain-of-thought generally benefited reasoning-intensive tasks, whereas few-shot prompting was most effective for user request classification.
  • Prompting strategies: Prompting strategies should be selected according to each task’s reasoning demands and expected outputs rather than applied uniformly.
  • Human oversight: Tasks requiring multi-artefact synthesis or generated artefacts were more susceptible to unsupported, incomplete, or insufficiently grounded outputs.
  • Practical implications: LLM adoption should match models and prompting strategies to task characteristics, with outputs validated as decision-support artefacts before engineering use.
  • Future research: Future research should improve factual grounding through retrieval-augmented generation, repository-aware retrieval, or RE-specific fine-tuning.

8 Conclusion

The paper provides a cross-task view of LLM capabilities and limitations in RE, finding that effectiveness depends on the specific activity rather than one universally superior model or prompting strategy. Its replication package supports reproducibility and future cumulative evaluation.

  • LLM effectiveness is strongly task-dependent, varying with reasoning demands, artefact characteristics, and required outputs.The findings support selecting models and prompts for specific RE activities rather than adopting a universally superior option.
  • The paper contributes cumulative evidence by evaluating LLMs across complementary RE activities instead of examining them in isolation.This cross-task perspective broadens understanding of where current models are effective and where limitations remain.
  • The study offers practical guidance for integrating LLMs into RE and software-engineering workflows.Human expertise remains essential when engineering judgement, factual grounding, and validation are required.
  • A replication package containing datasets, prompts, scripts, and experimental materials supports transparency and reproducibility.The materials are intended to encourage future empirical studies spanning multiple RE activities.
Loading 2608.21531v1…