Source-linked AI summary
ReqEvolve: User-Oriented Software Self-Evolution through Automatic Requirement Interpretation
Md Asif Iqbal Fahim, Alessio Ferrari
TL;DR
User requests are often nontechnical, forcing translation before software can evolve and delaying behavioural requirements validation. ReqEvolve combines automatic requirements engineering with test-driven runtime generation to turn such requests into executable functionality. Across 72 cases in 18 projects, it achieved 89.2% Pass@1 and outperformed both comparison systems.
Problem
End-user functionality requests are rarely expressed as technical specifications, so existing self-evolution approaches require developer translation before users can observe the requested behaviour.
Method
ReqEvolve combines automatic requirements engineering and test-driven development to clarify requests, decompose specifications, generate tests, synthesise code, and integrate functionality at runtime.
Results
89.2% Pass@1 across 72 cases in 18 projects, outperforming SpecFix by 18.8% and the ablation baseline by 32.6%.
Takeaways & Limitations
The results provide initial evidence that user-driven self-evolution can support immediate requirements validation before developer verification.
Takeaways & Limitations
The logger was not evaluated in isolation, so logging inaccuracies could affect failure-analysis interpretations and audit conclusions remain empirically unvalidated.
Abstract
from arXiv · showhide
The paradigm of software self-evolution enables systems to autonomously extend and reconfigure their own capabilities during execution in response to technical specifications. Yet requests for new functionality often originate from end users and are rarely expressed in technical terms. As a result, developers must translate user needs into technical specifications before the system can evolve, delaying early validation of the requested functionality by preventing users from immediately observing the resulting behaviour. To address this gap, we present ReqEvolve, a runtime code generation system that enables user-driven self-evolution by accepting high-level user requests. The system integrates automatic requirements engineering (RE) and test-driven development (TDD) to transform these requests into executable functionality through clarification, specification decomposition, test generation, and runtime integration. We evaluate ReqEvolve on 72 software evolution cases across 18 projects against two baselines: SpecFix, an RE-focused code generation approach, and an ablation variant of our system. ReqEvolve achieves 89.2% Pass@1, outperforming SpecFix by 18.8% (p < 0.01, r = 0.79, large effect) and the ablation baseline by 32.6% (p < 0.001, r = 0.88, large effect). These results provide initial evidence that user-driven self-evolution is a viable paradigm for autonomously extending software capabilities from user requests, thereby accelerating requirements validation prior to developer verification.
1 Introduction
Conventional evolution requires translating user needs into technical specifications before implementation, delaying behavioural validation. ReqEvolve addresses this gap by generating and integrating functionality from end-user requests, achieving strong benchmark performance.
- Problem: Existing self-evolution approaches require developer-reformulated implementation specifications even when capability requests originate from end users.User intent is often expressed in domain terms rather than technical language.
- Contribution: User-driven self-evolution lets users express needs directly and observe generated behaviour before developers verify correctness and non-functional properties.This supports earlier validation of whether the intended software is being built.
- Results: 89.2% Pass@1, outperforming SpecFix by 18.8% and the ablation baseline by 32.6%.The evaluation covered 72 software evolution cases across 18 projects; both comparisons were statistically significant with large effects.
- Approach: ReqEvolve combines automatic requirements engineering with test-driven development to transform user requests into runtime functionality.The pipeline clarifies needs, refines specifications, generates tests, synthesises functionality, and integrates it during execution.
2 User-Driven Self-Evolution
User-driven self-evolution shifts capability requests from developer-mediated translation toward direct natural-language interaction. The system interprets underspecified requests, generates sandboxed functionality, and lets users refine it through observed behaviour before developer hardening.
- 2 User-Driven Self-Evolution: Conventional evolution routes user needs through analysts and developers before returning a prototype for requirements validation.The process may include elicitation, clarification, implementation, feedback sessions, and later production integration.
- 2 User-Driven Self-Evolution: User stories provide the structured interaction format, expressing a requested capability without prescribing its implementation.The paper uses user stories as a human-machine communication artefact despite their limited naturalness for direct user interaction.
- 2 User-Driven Self-Evolution: Users submit needs through a natural-language interface, receive generated behaviour, and iteratively refine the capability through feedback.Developers subsequently inspect and harden the implementation because generated functionality may contain security or non-functional defects.
- 2 User-Driven Self-Evolution: Because requests are often underspecified, the system formulates plausible interpretation hypotheses before generating functionality.Users can assess the resulting behaviour and provide further feedback as interpretations are refined.
3 Background and Related Work
The paper positions ReqEvolve at the intersection of user-oriented requirements engineering, runtime self-evolution, LLM-based code generation, and realistic benchmarking. Its contribution is runtime generation from user stories supported by a benchmark of complex evolution cases.
- Requirements Engineering: User stories encode role, requested means, and intended ends without prescribing implementation.AQUSA assesses these artefacts using 13 syntactic, semantic, and pragmatic quality criteria.
- Requirements Engineering: Acceptance criteria translate user stories into testable GIVEN-WHEN-THEN specifications for workflows and outcomes.They can describe multi-step sequences involving context, actions, and expected results.
- LLM-based Code Generation: Prior LLM code-generation research spans single-agent planning and reflection methods as well as multi-agent workflows with specialised roles.These approaches address synthesis complexity but generally operate on implementation-oriented tasks.
- Code Generation Benchmarks: Class-level and context-dependent benchmarks expose limitations of small, self-contained function evaluations.The cited comparison reports GPT-4 falling from 85.4% on HumanEval to 37.0% on class-level tasks.
- Related Work: ReqEvolve accepts user stories for runtime code generation, unlike surveyed systems that use developer-oriented or formal inputs.It incorporates the SelfEvolve test-driven pipeline while uniquely targeting end-user requirements.
- Code Generation Benchmarks: The benchmark contains 72 evolution cases across 18 projects requiring runtime integration into existing codebases.Each case integrates functions of approximately 62 LOC with codebases ranging from 40 to 3,100 LOC.
4 ReqEvolve Architecture
ReqEvolve uses an iterative architecture that first interprets and decomposes user requests, then generates and validates code through TDD and runtime integration. Its design supports reuse, clarification, test feedback, and failure analysis, with explicit limitations around matching and logging.
- Pipeline Overview: The pipeline has an RE stage for intent clarification and acceptance-criteria decomposition followed by iterative test-driven code generation.A persistent knowledge base supports pre-existing and generated functions throughout the process.
- Pipeline State: Each iteration tracks the request, code context, defects, interpretation hypotheses, acceptance criteria, signature, tests, generated code, and feedback.This state enables refinement based on test feedback.
- Capability Matching: The dispatcher reuses semantically matching existing functions and generates new code only when no match is found.For new functionality, it selectively retrieves relevant project context instead of loading entire codebases.
- Automatic Requirements Engineering: Clarification autonomously generates and ranks plausible answers to missing or vague stakeholder, scope, outcome, and example information.These hypotheses guide subsequent specification decomposition rather than requiring immediate user questioning.
- Limitations: Capability matching occurs only on the initial request, so clarification can reveal missed reuse opportunities.The framework also always attempts an answer, even for incomplete or ambiguous stories, motivating future input validation and safeguards.
- Test-Driven Code Generation: The generator creates tests and implementations that can reuse functions from the knowledge base as subroutines.The running example generates tests for output shape, empty results, missing data, and a discover_friends function.
- Logger Component: Logger outputs support failure analysis but are not evaluated in isolation, limiting conclusions about their reliability and practical audit sufficiency.The authors leave empirical validation of developer audit use for future work.
5 Evaluation
ReqEvolve is evaluated across a diverse benchmark against a requirements-focused baseline and an ablated variant, using semantic judging alongside execution-based checks. It substantially outperforms both baselines overall, while failures concentrate in code-generation and integration defects.
- Experimental Setup: The benchmark contains 72 evolution cases across 18 projects, with synthetic Python codebases ranging from 40 to 3,100 LOC and spanning multiple domains.Projects were designed to approximate realistic test settings and include greater structural complexity than the earlier SelfEvolve codebases.
- Evaluation Methodology: LLM-as-judge evaluation accepts implementations against binary semantic criteria, complementing execution-based checks without requiring one canonical output.This protocol addresses user stories that permit multiple valid implementations, although it does not fully capture validation through observed user behaviour.
- RQ1: ReqEvolve Performance: 89.2% Pass@1 (257/288) exceeds SpecFix’s 70.5% (203/288) by 18.8%, with statistical significance and a large effect.The comparison covers 72 user stories across 18 projects with four independent runs per system; Wilcoxon testing gives p < 0.01 and r = 0.79.
- RQ1: ReqEvolve Performance: 35.4% separates the approaches on compositional tasks, highlighting integration of previously generated functions as a distinguishing capability and a remaining challenge.The evaluation identifies improved change-impact analysis and an evolving architectural model as possible ways to improve compositional performance.
- RQ3: Failure Analysis: 31 failed ReqEvolve runs were classified using an error–fault–failure chain, with incorrect code despite correct inputs dominating errors at 64.5%.Integration-related defects account for 45.2% of faults, while 83.9% of failures involve criteria not being passed.
6 Discussion and Future Plans
ReqEvolve frames user-driven self-evolution as a broader requirements-engineering and software-process paradigm, while identifying evaluation, generalizability, safety, and deployment boundaries. Its results support early user validation through generated behaviour, but technical verification and broader empirical validation remain necessary.
- Threats to Validity: The evaluation is bounded by controlled Python benchmarks, non-mission-critical settings, and exclusive use of GPT-4.1.The authors state that real-world studies with end users and codebases are still needed, while safety-critical domains remain out of scope.
- Threats to Validity: Binary Pass@1, LLM-based judging, benchmark defect distributions, and predefined criteria constrain how completely evaluation quality and acceptable implementations are represented.The predefined criteria mean reported results are a lower bound of acceptable solutions.
- Implications for RE Research: ReqEvolve embeds quality assessment, clarification, specification decomposition, and code generation into one runtime RE pipeline.This integration shifts RE from an isolated activity toward an embedded component of LLM-based software-engineering pipelines.
- Implications for Software Engineering Practice: User-driven self-evolution supports individualized software and a two-tier validation model separating user intent validation from developer technical certification.Users can validate experimental behaviour, while developers remain responsible for certifying correctness before broader diffusion.
- Future Directions: Future work includes multimodal requests, less constrained user studies, and deployment-oriented validation of the two-tier model.Proposed studies would examine how effectively end users express needs and whether generated behaviour improves requirements validation.
- Conclusion: 89.2% Pass@1 across 72 cases and 18 projects exceeded SpecFix by 18.8% and the ablation baseline by 32.6%.The authors interpret these results as evidence that structured RE support is foundational for user-story-driven runtime generation.