PaperlayerView exact revision on arXiv

Source-linked AI summary

A Retrieval-Augmented Automated Stakeholder for Requirements Elicitation Education: A Comparative Study

Manal Binkhonain, Ohoud Mosa Alharbi

arXiv:2609.12576v1cs.SE

TL;DR

Requirements elicitation practice calls for interviewing skills, but role-playing can be constrained and LLM simulations may return hallucinated or inconsistent responses. This study evaluates a LangChain-based RAG stakeholder in controlled experiments with 69 students; students rated human stakeholders as more engaging, while the automated stakeholder yielded stronger elicitation outcomes.

  • Problem

    Requirements elicitation practice is challenging, and LLM interview simulations remain susceptible to hallucinated and inconsistent responses.

    Source: §1 · pp. 2–3

  • Method

    The study used a LangChain-based RAG stakeholder grounded in instructor materials and compared it with invited human stakeholders in two experiments involving 69 students.

    Source: §1 · p. 3, §7 · p. 23

  • Results

    Students perceived the automated stakeholder as less engaging and less preferable, but it produced higher-quality elicitation questions and identified more features.

    Source: §7 · p. 23

  • Takeaways & Limitations

    Automated stakeholders may serve as an instructional scaffold in early elicitation training, where lower interactional complexity lets students focus on analytical tasks and develop elicitation skills.

    Source: §7 · p. 23

    Individual differences in prior elicitation experience, conversational-AI familiarity, and technical proficiency may have affected performance, despite the within-subjects design in Experiment 2.

    Source: §6.2 · p. 22

Abstract

from arXiv · show

Developing the skills required for requirements engineering students to conduct effective requirements elicitation interviews is critical yet challenging, as it requires the development of soft skills in addition to technical knowledge. Role-playing is widely adopted in requirements engineering education to support the development of these skills but is often constrained by time and resource limitations. Although recent advances in large language models (LLMs) enable automated and interactive stakeholder simulations for role-playing, their application in requirements engineering education remains limited by hallucinations and inconsistent responses. To address these limitations, this study investigates the use of retrieval-augmented generation (RAG), implemented using the LangChain framework, to support requirements elicitation activities in a requirements engineering course. We conducted controlled experiments with 69 students, comparing cohorts who interacted with non-technical faculty role-players and those who engaged with RAG-based simulated stakeholders. The results indicate that while students perceived invited stakeholders as more realistic and engaging, the RAG-based automated stakeholder produced higher-quality elicitation questions and more complete feature identification.

1 Introduction

Requirements elicitation education must develop interviewing and interpersonal skills, but role-playing can be difficult to organize at scale. This study proposes a LangChain-based RAG stakeholder and evaluates it in controlled comparisons with human role-players.

1 Introduction

  • Interviewing requires analysts to ask focused questions, clarify ambiguities, negotiate priorities, and interpret stakeholder perspectives using both experience and soft skills.The section identifies interviews as a prevalent elicitation technique for gathering contextual information about system goals, constraints, and design needs.Source: §1 · p. 2
  • Role-playing supports experiential learning and develops RE skills, but time, resource, and participant constraints make it difficult to integrate effectively into courses.Students typically alternate stakeholder and analyst roles, practicing communication, negotiation, and empathy alongside technical knowledge.Source: §1 · p. 2
  • LLM-based interview simulations offer controlled practice, but hallucinations, inconsistent responses, and context drift can undermine realism and educational effectiveness.Source: §1 · p. 3
  • The study investigates a LangChain-implemented RAG conversational agent that grounds simulated stakeholder responses in instructor-provided materials and manages conversation history for contextual recall.The automated stakeholder is intended to support question formulation, engagement and learning experience, and scalable independent interview practice.Source: §1 · p. 3
  • The experiments assess perceived realism, engagement, and learning experience, along with elicitation-question quality and feature-identification completeness, while investigating benefits and limitations for course integration.The study compares students engaging with the automated stakeholder and human role-players.Source: §1 · pp. 3–4

2 Background and Related Work

Prior work uses role-playing to develop RE skills and ranges from structured automated practice to open-ended LLM stakeholder simulations. This study distinguishes its RAG approach through instructor-material grounding and a comparison with human role-playing, then describes the LangChain workflow supporting retrieval and generation.

2.1 Role-Playing in teaching requirements elicitation

  • In RE education, role-playing offers practical learning and engagement, while studies have examined learning outcomes, professional skills, and recurring interview mistakes.A study of 412 students linked strong role-playing project performance with higher written-exam scores; other work examined motivation, communication, and interview errors.Source: §2.1 · p. 4
  • SaPeer and ReverseSaPeer use self- and peer-review of interview mistakes, with role reversal reported to improve analytical understanding and emotional engagement.Observational studies identified recurring errors including vague questions, poor interview closure, weak customer rapport, and implicit goals.Source: §2.1 · pp. 4–5

2.2 Automated Stakeholder Interviews

  • Automated interview tools range from guided multiple-choice practice to LLM-based open-ended simulations, but fixed responses limit realism and LLMs can produce unreliable or boundary-violating replies.RoboREIT and VICO use predefined interactions; C-LEIA sometimes generated imprecise, incomplete, or inconsistent responses, while LLMREI elicited up to 73.7% of requirements in 33 simulated interviews and sometimes hallucinated or requested personal information.Source: §2.2 · pp. 5–6
  • The study differs from prior LLM approaches by grounding stakeholder responses in instructor-provided scenario materials and comparing automated and human role-players in the same educational setting.The comparison includes student perceptions and objective elicitation outcomes.Source: §2.2 · p. 6

2.3 LangChain Framework

  • LangChain supports workflows that retrieve relevant external information before an LLM generates a grounded response, rather than relying only on the model’s internal parameters.The framework’s RAG capability is described as reducing hallucinations and improving factual accuracy and controllability.Source: §2.3 · pp. 6–7
  • The architecture separates indexing from retrieving and generating: documents are split into chunks, embedded, and stored for similarity search.The vector database supports retrieval of relevant segments instead of entire documents.Source: §2.3 · pp. 6–7, Figure 2 · p. 7
  • At interaction time, retrieved segments are passed with the user query and conversation history to the LLM to produce a context-aware response.Source: §2.3 · p. 7

3 Research Design and Implementation

The study uses two controlled experiments to compare RAG-based and human stakeholders in requirements elicitation education, examining student perceptions and elicitation performance.

3.1 Goals and Research Questions

  • The study compares a RAG-based automated stakeholder with an invited human stakeholder across elicitation performance, perceived experience, and engagement.Source: §3.1 · pp. 7–8

3.2 Methodology

  • Two complementary experiments use between-subjects and within-subjects designs; Experiment 1 explores feasibility, while Experiment 2 adds expert assessment.Source: §3.2 · p. 8
  • Students prepared and conducted interviews, submitted questions and features, and completed feedback; two instructors independently assessed elicitation artifacts (ICC = .74).Experiment 1 randomly assigned students to conditions; Experiment 2 also gathered comparative reflections. Final expert ratings averaged the two instructors’ scores.Source: §3.2 · pp. 8–9

3.3 Independent and Dependent Variables

  • The independent variable is stakeholder type; outcomes cover subjective learning experience and expert-rated question quality and feature identification.Perceptions use 10 five-point items; elicitation artifacts are assessed with a nine-item question rubric and four feature criteria.Source: §3.3 · pp. 9–10

3.4 Statistical Analysis and Hypotheses

  • Tests were selected by normality and experiment design, with item-level hypotheses evaluated using two-tailed tests at α ≤ 0.05.The analysis uses independent-samples or Mann–Whitney tests for Experiment 1 and paired-samples or Wilcoxon tests for Experiment 2.Source: §3.4 · p. 10

3.5 Participants

  • The 69 participants comprised 50 third-year undergraduates in Experiment 1 and 19 first-semester master’s students in Experiment 2.Experiment 1 took place in late 2023; Experiment 2 was conducted in 2025.Source: §3.5 · p. 11

3.6 Implementation

  • LangChain Python chatbots used RAG with 300-token chunks, ada-002 embeddings, FAISS top-four retrieval, and GPT-3 or GPT-4 at temperature 0.1.Experiment 1 used GPT-3 and Experiment 2 used GPT-4.Source: §3.6 · pp. 11–12, Figure 4 · p. 12, Figure 5 · p. 13
  • Experiment 1 processed queries independently; Experiment 2 added sliding-window memory and persona prompting, while both limited responses to the instructor-provided scenario.The approximately 700-word source described an iOS consultation-support system and was reviewed by two instructors for domain and learning-objective alignment.Source: §3.6 · p. 12

4 Results

Across the two experiments, invited stakeholders were often perceived as more realistic, while the automated stakeholder received stronger expert ratings for elicitation questions and identified features. Students also valued the automated stakeholder’s efficiency but noted limits to its realism and interaction.

4.1.1 Quantitative Results

  • ISG scored higher across all ten perceived-experience items, but differences were significant on only four, including realism and engagement.No significant differences were found for the other six items; the authors describe perceived learning experience as broadly comparable across most dimensions.Source: §4.1.1 · pp. 13–14

4.1.2 Qualitative Data Analysis

  • Students associated ISG with realistic human interaction and authentic elicitation, while ASG was valued for speed and flexibility but criticized for limited realism and repetitive or unhelpful responses.Students said interacting with a real stakeholder exposed them to natural human cues and ambiguous requests; ASG interaction could depend heavily on prompting skills.Source: §4.1.2 · p. 14

4.2.1 Quantitative Results

  • In Experiment 2, perceived-experience ratings were broadly comparable: no statistically significant differences appeared on any of the ten items.ISG had marginally higher values on most items, but none reached statistical significance.Source: §4.2.1 · p. 15

4.2.2 Blind Paired Expert Review

  • ASG students made fewer mistakes than ISG students across all nine criteria; average mistake scores were 2.52 for ASG and 3.63 for ISG.The difference was statistically significant, and lower scores indicate fewer mistakes.Source: §4.2.2 · pp. 15–16, Figure 8 · p. 16
  • ASG received higher feature-quality scores than ISG (3.75 vs. 2.62), with higher ratings on all four feature rubric items.All four item comparisons were statistically significant at p < .001.Source: §4.2.2 · p. 16, Figure 9 · p. 17

4.2.3 Qualitative Data Analysis

  • Students valued ASG’s immediate responses and continuous practice, while ISG offered more natural dialogue; 64% saw ASG as a possible substitute under certain conditions.Overall, 43% preferred ASG and 57% preferred ISG. Students also cited ASG’s lack of human cues and reliance on carefully worded questions as limitations.Source: §4.2.3 · pp. 17–18

5 Discussion

Invited and automated stakeholders offer different learning experiences: human interactions provide social realism, while automation offers greater structure and consistency. The discussion recommends using them as complementary tools, with structured practice for foundational skills and human interactions for socially situated skills.

5.1 Experiences with Invited and Automated Stakeholders

  • Students valued invited stakeholders’ realism and professional relevance, while describing automated interactions as enjoyable, controlled, efficient, and easy to review.Automated interviews avoided scheduling constraints and logged interactions; human sessions supported negotiation, empathy, and interpretation of verbal and nonverbal cues.Source: §5.1 · p. 19
  • Invited stakeholders can be difficult to schedule and keep within class time, and their responses may vary in ways that complicate assessment of feature completeness.Automated stakeholders address these logistical and consistency challenges, but may struggle with complex questions or specifications that omit requested information.Source: §5.1 · p. 19
  • Students generally preferred invited stakeholders, though many saw the approaches as complementary and suggested prompting instruction and automated practice before real interviews.The discussion also raises a research question about whether prompting proficiency supports interview skills.Source: §5.1 · pp. 19–20

5.2 Comparison of Learning Outcomes

  • Structured, consistent automated responses reduce conversational variability and ambiguity, helping novices focus on question formulation, sequencing, and coverage.The discussion connects this controlled environment with fewer common elicitation mistakes among novice analysts.Source: §5.2 · p. 20
  • Automated stakeholders yielded more structured, consistent practice, while invited stakeholders offered greater realism and exposure to negotiation and uncertainty.The authors frame this as a trade-off between social realism and analytical clarity, with the two approaches supporting complementary learning objectives.Source: §5.2 · p. 20

5.3 Implications for Curriculum Design

  • The authors recommend introducing automated stakeholders early for foundational analytical practice, then invited stakeholders for interpersonal, negotiative, and ambiguous interactions.They position automation as a scalable, repeatable complement to—not a replacement for—human role-playing.Source: §5.3 · p. 21
  • A project-document-based automated stakeholder could let junior analysts rehearse questions before meeting real clients, but professional use still needs empirical validation.The authors also describe configuration around domain-specific documentation as a potential way to prepare analysts for unfamiliar domains.Source: §5.3 · p. 21

6 Threats to validity

The study addresses threats to validity through complementary subjective and expert-assessed measures, design controls, and explicit acknowledgment of model, setting, and sample constraints. These constraints qualify how confidently the findings can be attributed to the learning approaches or generalized beyond the study.

6.1 Construct Validity

  • Measures combined questionnaire ratings of learning experience, engagement, and realism with blind expert assessment of elicitation performance using a published taxonomy and ISO criteria.The authors say this triangulation reduces mono-method bias.Source: §6.1 · p. 21

6.2 Internal Validity

  • Participant differences may have influenced performance; Experiment 1 randomly assigned students, while Experiment 2 used a within-subjects design to reduce individual variability.Potential differences included prior elicitation experience, familiarity with conversational AI, and technical proficiency.Source: §6.2 · p. 22
  • GPT-3 in Experiment 1 and GPT-4 in Experiment 2 may have affected outcomes independently of condition, so the experiments are reported separately rather than compared directly.Standardized tasks, instructions, and time constraints were used to minimize procedural variability.Source: §6.2 · p. 22
  • Counterbalancing condition order mitigated potential learning and fatigue effects in Experiment 2, but residual effects cannot be ruled out.The authors also identify possible order effects in the within-subjects design.Source: §6.2 · p. 22

6.3 External Validity

  • The academic student setting and intentionally limited domain and specification may constrain generalization to complex, real-world elicitation.The limited scope supported experimental control, alignment with course objectives, and comparability between conditions.Source: §6.3 · p. 22
  • Experiment 2 had 19 participants versus 50 in Experiment 1, increasing sensitivity to variability and potentially reducing estimate precision.The authors note that the within-subjects design partly improves statistical efficiency, but recommend replication with larger samples.Source: §6.3 · p. 22

7 Conclusion

This study evaluates a retrieval-augmented automated stakeholder against invited human stakeholders in two controlled experiments with 69 students. Although students perceived automation less favorably, it produced higher-quality elicitation questions and feature identification, supporting its use as an instructional scaffold.

Conclusion

  • Two complementary controlled experiments with 69 students compared student performance and learning experiences with RAG-based automation and invited human stakeholders.The evaluation examined subjective perceptions and objective elicitation outcomes.Source: §7 · p. 23
  • RAG-based automated interactions resulted in higher-quality elicitation questions and identified features, despite students’ lower engagement and preference compared with invited stakeholders.The authors suggest reduced interactional complexity lets students focus on core analytical tasks during early elicitation training.Source: §7 · p. 23
  • The work contributes a RAG-based automated stakeholder, controlled-experiment evidence, and implications for using automation alongside human role-playing.The reported comparison indicates improved elicitation quality alongside a trade-off in engagement and realism.Source: §7 · p. 23
  • Future evaluation should examine richer knowledge bases, multimodal interaction, complex real-world scenarios, and hybrid exercises combining automated and human stakeholders.The current implementation used an instructor-created requirements document as its RAG knowledge base.Source: §7 · p. 23

A Appendix

The appendix includes tables describing evaluation items for stakeholder perceptions, elicitation-question quality, and feature identification.

A Appendix

  • Table 2 lists items for evaluating perceptions and learning experience.Source: Table 2 · p. 28
  • Table 3 lists items and descriptions for evaluating question quality.Source: Table 3 · p. 28
  • Table 4 lists items and descriptions for evaluating features.Source: Table 4 · p. 28
Loading 2609.12576v1…