Agent4RE: A Self-Refining Multi-agent Framework for End-to-End Software Requirements Engineering and Benchmarking

1 Introduction

• Existing LLM-based RE studies mainly address isolated tasks, and no systematic end-to-end solution or comprehensive open-source benchmark evaluates the full RE lifecycle.RE work is labor-intensive, and defects originating in this phase are significantly more expensive to fix later in development.Source: §1 · p. 1
• Agent4RE coordinates specialized agents to simulate stakeholder interactions and iteratively elicit, generate, and traceably refine requirements.It offers sequential, self-refining, and human-in-the-loop modes.Source: §1 · p. 1, Figure 1 · p. 2
• RE-E2E is built from full-scale, human-written specifications and spans initial project descriptions through finalized requirement specifications.The authors describe it as the first end-to-end dataset for complete RE processes, to their knowledge.Source: §1 · p. 1
• All Agent4RE variants outperform a context-augmented prompting baseline by an average 8% in text-based metrics; the enhanced variants exceed two baselines by approximately 0.8 points on a four-point scale.The evaluation integrates eight frontier LLMs and examines effectiveness and robustness across model capabilities and computational budgets.Source: §1 · p. 2

2.1 Large Language Models for RE-related Tasks

• BERT variants target ambiguity detection and requirements classification, while few-shot prompting can match fine-tuned classification performance without training overhead [8].The related work also cites transformer foundations and ambiguity detection in interdisciplinary projects.Source: §2.1 · p. 2
• LLMs have been used for RE elicitation, analysis, specification, and validation, but cited systems address specific aspects rather than complete production-ready specifications [19].Source: §2.1 · p. 2

2.2 LLM-based Agentic Systems for Automated RE Process

• Agentic RE systems include role-based collaboration, modular requirements generation, elicitation interviews, and iReDev [18], whose evaluation focused mainly on intermediate steps.For iReDev, final specifications were assessed only with subjective completeness, correctness, and consistency metrics.Source: §2.2 · p. 3

2.3 Baseline Selection

• Because the study targets the complete RE lifecycle, it selects a prompting approach tailored to RE stages and iReDev [18] as its two baselines.Source: §2.3 · p. 3

3 Agent4RE

• Agent4RE offers sequential, self-refining, and human-feedback workflows, each with distinct advantages for requirements engineering.The sequential version streamlines the process; the other versions add iterative improvement or domain-expert guidance.Source: §3 · p. 3

3.1 Sequential Workflow without Feedback

• The sequential workflow uses an interviewer and stakeholder to conduct elicitation rounds, with an autonomous exit tool and a maximum of 10 iterations before reporting results.The stakeholder uses the orchestrator's project summary and its own knowledge to answer questions.Source: §3.1 · pp. 3–4
• The generation agent turns the elicitation report into an initial specification, applies required templates, and returns it to the orchestrator for validation and corrective action.Templates are included in the system message to guide structural compliance.Source: §3.1 · p. 4

3.2 Agentic Workflow with Self-refinement

• The self-refinement workflow adds a refactoring agent that checks specifications against criteria including unambiguity, completeness, and consistency, then sends improvement suggestions for generation-agent revisions.The loop can run multiple times until the specification meets the demanded quality standards.Source: §3.2 · p. 4

3.3 Agentic Workflow with Human Feedback

• The human-feedback workflow adds developers' domain-specific insights to orchestrator verification comments before refactoring, with feedback limited to concise bullets or high-level structural hints.Participants are experienced software engineers with access to the dataset's ground-truth specifications.Source: §3.3 · p. 4

3.4 Implementation and Configuration

• Agent4RE is implemented with Google's ADK, using independent agents with specialized prompts and tool configurations, plus task-specific temperature settings.The orchestrator prompt is shown in Figure 3, and remaining prompt templates are in supplementary materials.Source: §3.4 · p. 4, Figure 3 · p. 4, Table 3 · p. 4

4.1 RE-E2E Dataset

• RE-E2E contains 50 complete real-world requirement specifications, with documents organized into IEEE 830 and IEEE 29148 structures and section mismatches reviewed by annotators.Semantic matching below 0.7 cosine similarity prompted human review.Source: §4.1 · p. 5

4.2 Selected Language Models

• The evaluation spans eight LLMs: four frontier API models and four open-source alternatives.The selected models cover different performance tiers.Source: §4.2 · p. 5

4.3 Metrics

• Evaluation combines text and semantic similarity metrics with LLM-as-a-judge and human ratings on a 1-to-4 scale.LLM judging was limited to four top-tier models because compact models often struggled with judging quality.Source: §4.3 · p. 5, Figure 4 · p. 6

4.4 RE Specification Generation

• Human-feedback refinement led semantic metrics at 0.251 METEOR, 0.273 Rouge-L, 0.063 BLEU-1, 0.838 BERT F1, and 0.392 Sentence-BERT similarity.Human evaluation scored 3.29 out of 4; pure self-refinement could degrade semantic metrics, and GPT-5 used 33.74K tokens per specification at an estimated cost of 0.127$.Source: §4.4 · p. 6, Table 5 · p. 7, Table 6 · p. 7

4.5 Comparison among Agentic Approaches

• All three Agent4RE variants outperform both baselines; human-feedback refinement leads average scores, improving METEOR by approximately 4% and human and LLM evaluation by 0.8 points over prompting.Figure 5 averages performance across models and displays the min-max range.Source: §4.5 · p. 7, Figure 5 · p. 8

4.6 Study on Self-refining Loops

• Large models executed about 2.5 elicitation loops with variance below 0.5, while compact models were less stable and Gemma4-4B required up to 7 loops.The most variable models performed worst in most metrics; the authors identify iteration stability as a stronger performance indicator than iteration count.Source: §4.6 · p. 8, Figure 6 · p. 8

5 Conclusions and Future Work

• Agent4RE uses five collaborative self-refining agents and significantly outperforms existing baselines on RE-E2E.The authors report that human expertise at critical decision points benefits the agentic system, while less capable models may experience semantic drift during repeated refinement.Source: §5 · p. 8
• The authors call for more end-to-end benchmarks with structured requirements annotations and propose testing ambiguous briefs.RE-E2E uses well-formed project descriptions to establish a controlled baseline.Source: §5 · p. 8
• Future research could explore alternative self-refining architectures and benchmark newer models to examine how capability changes relate to specification generation.The authors identify model stability, or lower variance across iterations, as a stronger performance indicator than the number of refinement iterations.Source: §5 · p. 8

A System Prompt for Specialized Agents

• Figure 1 contains the interviewer agent's instruction prompt.The appendix also identifies prompts for the stakeholder, refactoring, and specification-generation agents.Source: A System Prompt for Specialized Agents · p. 10
• Figure 2 contains the stakeholder agent's instruction prompt.The appendix identifies separate prompts for the interviewer, refactoring, and specification-generation agents.Source: A System Prompt for Specialized Agents · p. 10
• Figures 3 and 4 contain the refactoring and specification-generation agents' instruction prompts, respectively.The appendix lists these alongside prompts for the interviewer and stakeholder agents.Source: A System Prompt for Specialized Agents · p. 10

2 Resource Consumption

• Compact Ollama models had no API cost; Qwen3-8B took 43.7 seconds without feedback and 51.0 seconds with human feedback.Its token consumption was 4.37K without feedback and 5.10K with human feedback.Source: §2 · p. 11
• Without feedback, GPT-5 used 17.30K tokens per specification, cost 0.065$, and took 17.3 seconds.Its average output length was 4.50K tokens.Source: §2 · p. 11
• With human feedback, GPT-5 used 26.74K tokens per specification, cost 0.100$, and took 26.7 seconds.Its average output length was 4.64K tokens.Source: §2 · p. 11

• The example SRS follows IEEE 830 and specifies the system overview, functional requirements, environment characteristics, and quality attributes.Source: §3 · p. 12

1.1 Definitions

• The SRS defines AHES as Automated Household Energy Control System and UI, IoT, and API as interface, connectivity, and software terms used in the specification.Source: §1.1 · p. 12

1.2 Background

• The system is intended to help users manage household energy use efficiently and conveniently, promoting energy conservation and cost savings.Its stated scope includes developing hardware and software for remote control of lighting, heating, cooling, and appliances.Source: §1.2 · p. 12, §1.3 · p. 12

2.1.1 System Interfaces

• The system interfaces with web, mobile, and voice-controlled user interfaces, household sensors and devices, external services, and standard communication protocols.Listed connections include smart meters, thermostats, plugs, and sensors; weather APIs, energy databases, and notification services; and Wi-Fi, Bluetooth, Zigbee, and MQTT.Source: §2.1.2 · pp. 12–13, §2.1.3 · p. 13, §2.1.4 · p. 13

2.2.2 Site Adaptation Requirements

• The system is intended for residential buildings, apartments, and single-family homes, with compatibility for standard electrical systems and home-automation infrastructure.It functions as a standalone system while interacting with sensors, actuators, and user interfaces, and may integrate with existing home-automation systems and IoT platforms.Source: §2.1.1 · p. 12, §2.2.2 · p. 13

2.3 Product Functions

• The system monitors household energy use, controls lighting, heating, cooling, and appliances remotely, offers optimization recommendations, and alerts users to abnormal use or malfunctions.It also supports schedules, real-time individual-device monitoring, remote internet access, and recommendations based on historical data and user preferences.Source: §2.3 · p. 13, §2.5 · pp. 14–15

2.4 User Characteristics

• Homeowners, renters, and property managers are potential users with varying technical expertise and interaction preferences.The system is intended to operate with smart meters, thermostats, plugs, motion sensors, and light sensors; voice-controlled devices such as Alexa and Google Assistant may interact through supported APIs.Source: §2.4 · p. 14, §6.1 · p. 16, §6.2 · p. 16

2.5 Constraints, Assumptions and Dependencies

• The specification assumes stable internet access and users with basic smart-device knowledge, and depends on third-party API availability and reliable sensors and actuators.It also calls for device and platform compatibility, standards compliance, privacy and security consideration, scalability, and accessible interfaces.Source: §2.2.1 · p. 13, §2.5 · pp. 14–15

5.1 Functional Partitioning

• Functions are partitioned into real-time monitoring, remote device control, optimization recommendations, scheduling, and alerts and notifications.Descriptions specify smart-meter integration and energy-use visualization; secure protocols and device compatibility; personalized recommendations; recurring or one-time schedules; and configurable alerts through channels such as email or SMS.Source: §5.1 · p. 15, §5.2 · pp. 15–16

7.1 Reliability

• The system must maintain at least 99% uptime, excluding scheduled maintenance, and undergo regular testing for reliability and stability.Source: §7.1 · p. 17

7.2 Availability

• Support services must be available 24/7, with response times and service-level agreements determined by severity and user subscription tier.Source: §7.2 · p. 17

7.3 Security

• Security requirements include encrypted device-server communication, user authentication, and activity logs with audit trails.The stated purposes include preventing unauthorized access and supporting accountability and forensic analysis.Source: §7.3 · p. 17

7.4 Maintainability

• Maintenance includes bug fixes, performance optimizations, security updates, and feature enhancements informed by user feedback and evolving requirements.Regular updates are intended to address identified issues, improve performance, and introduce capabilities.Source: §7.4 · p. 17

7.5 Portability

• The system must support Apple HomeKit, Google Home, and Amazon Alexa, and provide multilingual user interfaces.Source: §7.5 · p. 17

• The case study presents the target system, human feedback used to guide iterative SRS refinement, and the final result.Source: §4 · p. 17

Project Summary as Input

• The project summary presents household energy monitoring, remote control, scheduling, alerts, and energy-saving recommendations across web, mobile, and voice interfaces.It also identifies third-party APIs, device compatibility, internet access, and hardware reliability as risks or unknowns.Source: Project Summary as Input · p. 18

Human Feedback

• The SRS organizes the system description, key terms, scope, user and environment context, detailed requirements, and supporting requirements into seven main sections.Its contents include interfaces, functional behavior, design constraints, quality attributes, security, and system modules.Source: Human Feedback · pp. 18–19

Delivery Timeline

• Other requirements cover security, greater-than-99% uptime, MVP performance baselines, and a three-sprint plan from backend integration through testing and release preparation.The plan assigns security and feedback to Sprint 3.Source: §7 · p. 19, §7.1 · p. 22, §7.2 · p. 23

1.1 Objectives

• The six-week MVP centers on real-time monitoring, remote device control, scheduling, alerts, and integrations with selected smart devices.Its stated scope includes residential energy optimization and convenience.Source: §1.1 · p. 19, §1.3 · pp. 19–20, §2.2 · p. 20

2.3 User Classes

• The identified users are homeowners and a technical support team; assumptions include reliable connectivity, documented third-party APIs, and MVP capacity for 100 households with 10 devices each.The stated network requirements include a minimum speed of 2 Mbps.Source: §2.3 · p. 20, §2.4 · p. 20, §5.5 · p. 22

3.1 External Interfaces

• The SRS specifies mobile interfaces, compatible smart plugs and thermostats, cloud and manufacturer APIs, HTTPS with OAuth 2.0 tokens, sub-five-second data loading, and quality attributes.The quality targets include greater than 99% uptime and modular maintainability.Source: §3.1 · pp. 20–21, §3.2 · p. 21, §3.3 · p. 21

5.1 Real-Time Monitoring

• Monitoring must refresh within five seconds, remote device commands must execute within two seconds, and scheduling includes one-time operations with recurring schedules optional.Monitoring requirements also call for household- and device-level statistics and connectivity-failure notifications.Source: §5.1 · p. 21, §5.2 · p. 21, §5.3 · p. 22

5.4 Alerts System

• The requirements include alerts for excessive consumption and forgotten devices, plus secure integration with Nest, Ecobee, TP-Link Kasa, and Wemo.Alerts may use visual or audio cues in the mobile application.Source: §5.4 · p. 22, §5.5 · p. 22