PaperlayerView exact revision on arXiv

Source-linked AI summary

Engineering Requirements for Generative AI Agents: The BARD Process Theory

Alfonso Cannavale, Antonio Della Porta, Gianmario Voria, Gilberto Recupito, Maria Teresa Baldassarre, Gemma Catolino, Fabio Palomba, Andrea De Lucia, Stefano Lambiase

arXiv:2610.29741v1cs.SE

TL;DR

Research had not examined the practical RE process for GenAI Agent systems, whose behavior cannot be fully anticipated at design time. The authors used Socio-Technical Grounded Theory, iteratively collecting and analyzing practitioner interviews to build a substantive process theory. BARD describes requirements as progressively discovered through building the agent, observing its behavior, and revising what the team asks for.

  • Problem

    Research had not examined the practical RE process for GenAI Agent systems, whose behavior cannot be fully anticipated at design time.

    Source: §I · p. 1

  • Method

    The authors used Socio-Technical Grounded Theory, iteratively collecting and analyzing practitioner interviews to build a substantive process theory.

    Source: §III · p. 3

  • Results

    BARD describes requirements as progressively discovered through building the agent, observing its behavior, and revising what the team asks for.

    Source: §V · p. 8

  • Takeaways & Limitations

    BARD is modifiable by construction because it focuses on the process rather than any specific technology.

    Source: C. Discussing Parsimony and… · p. 10

    Interviews were the sole evidence channel, so the theory relies on practitioner self-reports without observation of practice or examination of artifacts.

    Source: §VI · p. 10

Abstract

from arXiv · show

How teams engineer requirements for systems built around Generative AI (GenAI) Agents remains empirically uncharted. Existing Requirements Engineering (RE) research mainly addresses AI and ML-based systems broadly but provides no process account for agentic systems, whose non-deterministic behavior cannot be anticipated at design time. To understand how teams actually proceed, we applied Socio-Technical Grounded Theory across 36 semi-structured interviews with practitioners, interleaving data collection and analysis so that emerging findings drove our theoretical sampling and shaped successive rounds of coding. The result is Behavioral Agent Requirement Discovery (BARD), the process theory of RE for GenAI Agents. BARD holds that requirements are not fixed up front but discovered through a recurring cycle of four activities: teams scope the need, build the agent, observe its behavior, and revise what they ask for. Through this four-phase loop, the requirement evolves from a fixed specification towards behavioral and evaluation criteria. Four tensions drive transitions between phases, and two conditions frame the loop. BARD provides researchers with a framework for iteratively refining RE workflows, while giving practitioners a structured basis to embed into their development processes.

I. INTRODUCTION

The paper addresses the lack of an empirical process account for how practitioners engineer requirements for GenAI Agent systems, whose behavior cannot be fully anticipated at design time. It develops a substantive process theory intended to connect narrower research concerns and provide a shared frame for researchers and practitioners.

I. INTRODUCTION

  • Because agent behavior cannot be fully anticipated at design time, requirements engineering must account for systems with new roles, activities, and practices.The paper situates this shift within the broader evolution of RE for AI-enabled systems.Source: §I · p. 1
  • No empirical study had examined how requirements for GenAI Agent systems are elicited, negotiated, refined, and validated in practice.Existing work addressed narrower concerns, leaving no shared process frame for connecting them.Source: §I · p. 1
  • A process theory can connect activities over time; without one, the field risks disconnected reports and teams rediscovering trade-offs in isolation.Source: §I · p. 1
  • The authors develop a substantive process theory to explain how practitioners engineer requirements for GenAI Agent systems.They describe the review as deliberately limited to justifying the research gap and verifying novelty, to avoid biasing grounded-theory coding.Source: §I · pp. 1–2, §II · p. 2
  • The paper asks how practitioners engineer requirements for software systems built around GenAI Agents.Source: §I · p. 2
  • Using Socio-Technical Grounded Theory and 36 practitioner interviews, the study offers BARD, an empirically grounded account of requirements, artifacts, and decisions.The authors describe it as, to their knowledge, the first substantive theory of the RE process for GenAI Agents.Source: §I · p. 2

A. Generative AI Agents

The background distinguishes GenAI Agents from earlier AI and ML systems and from pre-LLM agent-oriented RE. It identifies the lack of empirical, process-level theorization of requirements engineering for GenAI Agent systems.

A. Generative AI Agents

  • GenAI Agents combine LLM reasoning, tool use, and delegated goals; this study covers both single-agent integrations and multi-agent orchestrations.Source: A. Generative AI Agents · p. 2

B. Requirements Engineering for AI and ML Systems

  • Prior RE-for-AI and ML studies address broader engineering practices or narrower facets, while Wan et al. examine ML-related development changes across the lifecycle.This paper instead focuses on workflows and activities in RE for GenAI Agent systems.Source: B. Requirements Engineering for AI and… · p. 2

C. Agent-Oriented Requirements Engineering (Pre-LLM)

  • Pre-LLM goal-oriented RE traditions assume hand-crafted reasoning and explicit plans, whereas GenAI Agents use stochastic LLM reasoning shaped by prompts and tool descriptions.Source: C. Agent-Oriented Requirements… · p. 2
  • The related literature lacks an empirical study of the GenAI Agent RE process and a grounded theory explaining that process.Adjacent work addresses narrower facets, and no STGT or comparable qualitative theorization had been applied to this object.Source: C. Agent-Oriented Requirements… · p. 2

III. RESEARCH METHOD

The study uses Socio-Technical Grounded Theory to build a process theory from iterative practitioner interviews and analysis. Its sample includes 36 practitioners, recruited with attention to variation in work and organizational contexts.

III. RESEARCH METHOD

  • STGT was selected because software development is socio-technical and the study asks a process question for which existing theory is absent.Its iterative collection and theoretically grounded sampling fit a field whose relevant categories were not yet known.Source: §III · p. 3
  • The researchers iterated recruitment, interviews, and basic-stage coding until saturation, then linked concepts in advanced analysis and sought participant feedback on the resulting theory.Source: §III · p. 3

A. Participants and Sample Diversity

  • Participants were practitioners involved in agent design, integration, or development and in requirements elicitation or analysis; recruitment sought diversity across context, organization, country, and seniority.Convenience, snowball, and self-selection sampling were complemented by theoretical sampling guided by undersaturated categories.Source: A. Participants and Sample Diversity · p. 3
  • The sample comprised 36 practitioners, and its profile corresponded closely with cited developer-population benchmarks on several reported dimensions.The authors interpret those correspondences as evidence of sample plausibility and transferability.Source: A. Participants and Sample Diversity · p. 3

B. Data Collection

Semi-structured interviews provided the study’s data, and iterative coding and theoretical sampling shaped the analysis. The researchers used open coding, constant comparison, memoing, and an audit trail to develop categories and the theory.

B. Data Collection

  • Semi-structured interviews lasted 60–90 minutes, were recorded with consent, and followed a protocol inspired by scoping, analysis, and validation in traditional RE.Two to three researchers conducted each interview, and participants were encouraged to describe their work in their own vocabulary.Source: B. Data Collection · p. 3

C. Data Analysis

  • Basic-stage analysis used line-by-line open coding, constant comparison within and across cases, and memos on tentative concepts.Rotating coding groups and asynchronous review supported discussion and independent feedback; the shared Miro board held coding artifacts and an audit trail.Source: C. Data Analysis · p. 4
  • Theoretical sampling used gaps and undersaturated properties from coding to guide later recruitment; saturation meant no new properties or relationships in the last 11 interviews.The researchers completed three data-collection and analysis iterations.Source: C. Data Analysis · p. 4
  • Memos, codebook versions, and decision rules were maintained as a versioned audit trail, while each researcher spent about 30 person-hours on coding and theory-building.Source: C. Data Analysis · p. 4

D. The Research Team and Reflexivity

The research team used collaborative coding and explicit reflexivity to balance differing perspectives. Trustworthiness was supported through consensus, an audit trail, contextual reporting, and participant feedback.

D. The Research Team and Reflexivity

  • All authors conducted interviews and coding, with rotating groups discussing every coding decision in real time.Surfacing differing interpretations was intended to prevent any single author’s lens from dominating.Source: D. The Research Team and Reflexivity · p. 4
  • The team acknowledged disciplinary, attitudinal, and epistemological differences and treated disagreement as input to analysis.They preferred consensus-building over Cohen’s κ, while recording unresolved disagreements for later resolution.Source: D. The Research Team and Reflexivity · p. 5

E. Trustworthiness of the Theory Building Process

  • Trustworthiness was addressed through multiple coders, detailed interviews, a shared-board audit trail, and contextual reporting.The reported context included sample composition, organizational settings, and technical scope.Source: E. Trustworthiness of the Theory… · p. 5
  • Feedback from six randomly selected practitioners confirmed the integrated theory, with minor clarifications to phase transitions.Participants had opted in to follow-up feedback.Source: E. Trustworthiness of the Theory… · p. 5

IV. BARD: A SUBSTANTIVE PROCESS THEORY OF REQUIREMENTS ENGINEERING FOR GENAI AGENTS

The section presents seven categories that describe agent requirements and the practices around them. Requirements target behavior and evolve through iteration, with prompts, technology choices, team roles, non-determinism, and evaluation shaping the process.

A. Constituent Categories

  • Agent requirements target behavior rather than exact outputs, include agent-specific choices such as reasoning depth, and may remain partly implicit.The paper reports that 24 participants contributed to this category.Source: A. Constituent Categories · p. 5
  • Teams iteratively build an initial version, observe agent behavior, and revise requirements rather than fixing them at the outset.The paper reports that 24 participants contributed to this category.Source: A. Constituent Categories · p. 5
  • The system prompt serves as both a requirement and implementation artifact, and continuous prompt refinement records changing requirements.Teams refine prompts as testing reveals needs; the paper reports 23 participant contributions.Source: A. Constituent Categories · p. 5
  • Technology selection specifies required capabilities rather than a particular model, balancing task difficulty, capability, and cost.The paper reports 14 participant contributions; technology names appeared in requirements about half the time, according to one participant’s estimate.Source: A. Constituent Categories · pp. 5–6
  • Requirements work involves growing sets of project roles and layered communication, while client or management authority commonly shapes decisions.The paper reports 21 participant contributions to the socio-technical configuration category.Source: A. Constituent Categories · p. 6
  • Because agents are non-deterministic and opaque, teams add quality and control requirements, then judge behavior with test sets and edge cases before human sign-off.The paper reports 20 contributions to the non-determinism category and 23 to behavioral evaluation and acceptance.Source: A. Constituent Categories · p. 6
  • The process anatomy distinguishes phases, steps, techniques, and artifacts, while treating maturity and requirement typology as separate dimensions.The section first introduces the seven categories and then cross-cutting dimensions.Source: A. Constituent Categories · pp. 5–6

B. Behavioral Agent Requirement Discovery (BARD)

BARD models requirements engineering for GenAI Agents as a goal-directed, four-phase loop shaped by tensions and contextual conditions. Across the loop, requirements move from initial specifications toward behavioral and evaluation criteria, while failures can reveal previously unstated requirements.

B. Behavioral Agent Requirement Discovery (BARD)

  • BARD describes a four-phase requirements loop in which requirements move from fixed specifications toward behavioral and evaluation criteria.The theory combines a goal-directed process with tensions whose resolution moves work between phases.Source: B. Behavioral Agent Requirement… · p. 6
  • The loop comprises scoping and feasibility, building, evaluating behavior, and discovering latent requirements, with steps, techniques, and artifacts identified within phases.The phases recombine categories rather than mirroring them one-to-one.Source: B. Behavioral Agent Requirement… · p. 6
  • The loop is common but not universal, and its depth varies with client maturity, organization type, and team formality.A small, well-scoped project was reported as a boundary case in which requirements stabilized up front.Source: B. Behavioral Agent Requirement… · p. 7
  • Requirements flowing through the loop include traditional, agent-specific, chain-of-thought, prompt-specific, interaction-based, and data requirements.Their most concrete carrier is often the system prompt, and the requirement changes form across phases.Source: B. Behavioral Agent Requirement… · p. 7
  • Scoping begins with a vague need, assesses whether an agent is appropriate, and often produces an MVP rather than a written specification.Client maturity shapes how deeply feasibility and requirement splitting are developed.Source: B. Behavioral Agent Requirement… · p. 7
  • During building, teams encode requirements in prompts and select models and tools; evaluation uses criteria, test sets, checklists, edge cases, repeated runs, and client sign-off.The control-versus-non-determinism tension requires operational boundaries calibrated to task risks while preserving generative utility.Source: B. Behavioral Agent Requirement… · pp. 7–8
  • Failures and repeated runs can expose latent behavioral requirements, which teams express as evaluation criteria for the next scoping cycle.The fixed-specification-versus-failure-driven-requirement tension closes the loop.Source: B. Behavioral Agent Requirement… · p. 8
  • The loop is framed by socio-technical configuration and non-determinism, which makes behavior require judgment across runs and foregrounds qualities such as guardrails, factual correctness, latency, and token cost.Client or management authority typically shapes decisions, while compliance or regulatory constraints may block or redesign requirements before behavioral evaluation.Source: B. Behavioral Agent Requirement… · p. 8

V. DISCUSSION

BARD treats agent requirements as progressively discovered through iterative building, observing, and revising, rather than fixed upfront. Its cycle blurs requirements with evaluation and implementation under non-determinism and divided acceptance authority.

V. DISCUSSION

  • Requirements take shape through a recurring cycle of building the agent, observing its behavior, and revising what the team asks for.The requirement shifts from a fixed specification toward behavioral and evaluation criteria.Source: §V · p. 8
  • BARD blurs requirements with evaluation criteria, requirements with implementation, and stakeholder elicitation with discovery from agent behavior.A running agent’s failure can reveal a requirement, while the system prompt carries both requirements and implementation.Source: §V · p. 8
  • The process centers on evaluation because the agent’s black-box reasoning core cannot be specified and verified once, and loop runners rarely accept its results.These are the two conditions framing BARD’s cycle: agent non-determinism and a socio-technical configuration with separate operators and accepters.Source: §V · p. 8

A. Discussing Novelty

BARD is presented as the first substantive theory of RE for GenAI agents, and distinguishes its process account from adjacent theories of requirements and system development.

A. Discussing Novelty

  • BARD is presented as the first substantive theory of the RE process for GenAI agents.The paper assesses theory maturity using novelty, usefulness, parsimony, and modifiability; credibility and rigor assess method application.Source: A. Discussing Novelty · p. 8
  • Unlike RE-for-AI and ML accounts,, BARD shifts the unit of analysis from requirements to their formation over time and includes agent-specific requirements.These include requirements around tool use, the system prompt, and reasoning depth.Source: A. Discussing Novelty · p. 9
  • Unlike KAOS and Tropos,, BARD concerns stochastic agents whose behavior is shaped indirectly through prompts, not rational agents with explicitly engineered reasoning logic.Source: A. Discussing Novelty · p. 9
  • Unlike requirements reflection for self-adaptive systems, BARD addresses behavioral uncertainty discovered and revised by human teams during development, not environmental uncertainty handled at runtime.The paper therefore treats BARD as a different phenomenon, not a special case of requirements reflection.Source: A. Discussing Novelty · p. 9
  • BARD shifts Twin Peaks’ requirements-architecture pairing toward requirements-implementation, with the system prompt merging both.The prompt is written like code while specifying agent behavior.Source: A. Discussing Novelty · p. 9

B. Discussing Usefulness

BARD explains how iteration transforms requirements into behavioral and evaluation criteria, and offers implications for practitioners and researchers.

B. Discussing Usefulness

  • BARD’s contribution is explaining how iteration changes the requirement itself and which tensions drive those transitions.The authors note that iteration is familiar; their account specifies how an initial specification becomes behavioral and evaluation criteria.Source: B. Discussing Usefulness · p. 9

1) Usefulness for Practitioners:

  • Practitioners can use BARD as a process map and shared vocabulary for planning discovery and making prompt and acceptance practices explicit.Suggested practices include planning evaluation cycles, versioning and reviewing prompts with stakeholders, tracing requirements decisions into prompts, agreeing on sign-off across repeated runs, test sets, and edge cases, and explicitly splitting AI from traditional requirements.Source: 1) Usefulness for Practitioners · p. 9

2) Usefulness for Researchers:

  • Researchers can use BARD to organize narrower findings by the phase and tension they address and to test propositions about its conditions.The paper identifies system prompts, splitting matrices, and evaluation suites as units for future empirical work that could corroborate the theory and address its single-channel limitation.Source: 2) Usefulness for Researchers · p. 9

C. Discussing Parsimony and Modifiability

BARD compresses requirements engineering into a four-phase loop that moves requirements from fixed specifications toward behavioral and evaluation criteria. The theory is process-focused, while its evidence and transferability are bounded by sample composition, interview-only data, changing technology, and the contexts studied.

C. Discussing Parsimony and Modifiability

  • BARD’s four-phase loop moves requirements from fixed specifications toward behavioral and evaluation criteria, driven by four tensions under two conditions.Interviewed practitioners confirmed the theory’s results and reported that it is understandable.Source: C. Discussing Parsimony and… · p. 9, §VI · p. 10
  • BARD is modifiable by construction because its unit of analysis is the requirements process, not a specific technology.The authors expect new contexts, including regulated domains and large organizations with formal requirements governance, to extend areas where the theory is thinner.Source: C. Discussing Parsimony and… · p. 10

VI. LIMITATIONS

  • The study’s sample is uneven in gender and geography, despite diversity-seeking across roles, application domains, organizational types, and seniority.The authors call for larger studies to robustly extend the results over the original context.Source: §VI · p. 10
  • Researcher-bias checks remained team-internal, so a blind spot shared across software engineering researchers could survive the analysis.The authors report that interviewed practitioners confirmed the theory and consider this evidence that the risk was mitigated.Source: §VI · p. 10
  • Interviews were the study’s only evidence source, leaving practitioner self-reports uncorroborated by observed practice or artifacts such as prompts and evaluation suites.The authors note that self-reports can smooth over what teams actually do.Source: §VI · p. 10
  • The theory is time-bound: models, tool interfaces, and prompting patterns may date faster than its process account.Its unit of analysis is how requirements form and stabilize rather than a specific technology.Source: §VI · p. 10
  • Transferability is claimed for human teams that iteratively discover, refine, and validate agent behavior, mostly in small-to-medium consultancy/B2B or internal-product settings.The theory may thin out in heavily regulated or safety-critical domains and very large organizations with formal requirements governance; more diverse contexts need investigation.Source: §VI · p. 10

VII. CONCLUSION

The paper proposes BARD, a substantive theory in which teams discover agent requirements through an iterative sequence of scoping, building, observing, and revising. It offers researchers a process frame and practitioners phases to organize work, with future work planned on measurement and extending the theory.

VII. CONCLUSION

  • BARD describes requirements for GenAI agent systems as discovered through teams scoping the need, building the agent, observing behavior, and revising what they ask for.The theory offers researchers a frame for positioning narrower findings and practitioners vocabulary and phases for organizing often-ad-hoc work.Source: §VII · p. 10
  • The authors plan quantitative instruments to measure developers’ well-being and AI-readiness across the process phases.Source: §VII · p. 10
  • The authors also plan to extend BARD to other development steps, including design and testing, toward a more comprehensive contribution.Source: §VII · p. 10
Loading 2610.29741v1…