Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study

1 Introduction

• Existing benchmarks focus on narrow software-development aspects, leaving comprehensive lifecycle evaluation unaddressed.Earlier tasks centered on single files or methods, while newer repository-level benchmarks still cover only parts of development.Source: §1 · p. 1
• DevEval evaluates software design, environment setup, implementation, acceptance testing, and unit testing across 22 repositories in four languages.The languages are Python, C/C++, Java, and JavaScript; the repositories cover domains including machine learning, web services, and command-line utilities.Source: §1 · p. 1, Figure 1 · p. 2
• Less than 10% on repository-level implementation: GPT-4-Turbo struggles, and even the more complicated tasks leave it struggling to exceed 40%.The introduction also reports oracle test scores below 40%, while executable generated tests achieve up to 79.4% code coverage.Source: §1 · pp. 1–2
• External information and execution feedback can yield notable, consistent prompting improvements.The introduction also identifies difficulties with Makefile and Gradle, function-argument configuration, and advanced object-oriented programming.Source: §1 · p. 2

2 Related Work

• Early datasets focused on self-contained Python completion [Chen et al., 2021; Austin et al., 2021].Later work expanded language coverage, execution-based test coverage, dependencies, interactive environments, and short-form task suites [Zheng et al., 2023b; Liu et al., 2023a; Lu et al., 2021].Source: §2 · p. 2
• DevEval aligns with recent repository-scale coding works [Jimenez et al., 2023; Liu et al., 2023b], yet evaluates LLMs’ ability to create entire codebases from extensive natural-language descriptions.Source: §2 · p. 2
• Unlike agent frameworks such as MetaGPT, DevEval provides structured evaluation across all software development lifecycle phases [Hong et al., 2023].The paper also notes the growing trend in agent-based frameworks targeting software engineering [Yang et al., 2024; Zhang et al., 2024; Liu et al., 2024; Wang et al., 2024a].Source: §2 · p. 3

3.1 Task 1: Software Design

• Software design tasks generate UML class and sequence diagrams and a file-tree architecture, evaluated for general principles and faithfulness to instructions.General principles include cohesion, decoupling, and practicability, with criteria specific to each design task.Source: §3.1 · p. 3

3.2 Task 2: Environment Setup

• Environment setup generates dependency files from the PRD and design artifacts, then evaluates whether example code runs in the installed environment.The task uses Conda for Python, Gradle for Java, and NPM for JavaScript; DevEval omits C/C++ because there is no broadly acknowledged, user-friendly dependency manager.Source: §3.2 · p. 3

3.3 Task 3: Implementation

• Implementation generates repository files from detailed requirements and design artifacts, one file per interaction, following a predefined dependency-based partial order.Language-specific reference acceptance and unit tests determine pass rates; model-generated planning is left for future work.Source: §3.3 · p. 4

3.4 Task 4: Acceptance Testing

• Acceptance tests use the PRD and design artifacts, optionally with implementation code, and are checked against the benchmark implementation.Tests exercise CLI behavior or library APIs and compare actual outputs with model-predicted expected outputs.Source: §3.4 · pp. 4–5

3.5 Task 5: Unit Testing

• Unit tests are evaluated on reference code for oracle correctness and statement coverage, measuring executed statements against all statements.The tests are generated from the PRD and design artifacts.Source: §3.5 · p. 5

3.6 Dataset

• The dataset combines curated, executable repositories with cleaned files, prepared UML diagrams, and architecture designs.Annotators set up environments, verified functionality, and cleaned repositories before preparing the design documents.Source: §3.6 · p. 5

4.1 Setup

• The study evaluates GPT, CodeLlama, and DeepSeek-Coder models using a ChatDev-based baseline extended for UML diagrams, architecture design, environment setup, and multi-language execution.It compares No-Review, Normal-Review, and Execution-Feedback prompting; the latter adds runtime results, errors, and performance metrics to the review process.Source: §4.1 · pp. 5–6

4.2 Main Results

• For implementation, Execution-Feedback performs best overall and improves GPT-4-Turbo especially on acceptance tests, while Normal-Review shows no significant gain over No-Review.CodeLlama-34B-Instruct and DeepSeek-Coder-33B show no improvement with review; the authors note rigorous, sparse automated tests may understate review gains.Source: §4.2 · p. 6
• Testing results differ by task: Normal-Review enhances acceptance testing, but review offers no clear stable benefit for unit testing, where longer inputs can challenge long-context comprehension.Adding source code as input dramatically increases acceptance-testing performance, although models can barely generate executable acceptance tests without it.Source: §4.2 · pp. 6–7
• All models remain far from satisfactory on DevEval; implementation pass rates stay below 10%, and GPT-4-Turbo reaches 7.1% on reference acceptance tests and 8.0% on unit tests.The main results use the optimal prompting method for each task.Source: Table 6 · p. 7, §4.2 · p. 7
• Across repositories, overall Oracle Test and Coverage scores are quite low; among generated testing code that executes successfully with a non-zero Oracle Test score, coverage scores are relatively high.Instruct achieves the highest score among open-sourced models and approaches GPT-4-Turbo in acceptance testing, while generating meaningless but executable testing code.Source: §4.2 · p. 8

4.3 Analysis

• Models struggle with Makefiles and Gradle files, function redefinitions, multi-file references, parameter and type handling, naming, variable scope, and fabricated variables or misread data files.Source: §4.3 · p. 8

4.2 Main Results

• GPT-4-Turbo dominantly outperforms GPT-3.5-Turbo in software-design comparisons; larger open-source models improve design quality but remain relatively weaker on faithfulness.Without ties, GPT-4-Turbo's judgments agree with human majorities 79.2% on general principles and 83.2% on faithfulness.Source: Table 7 · p. 8, §4.2 · p. 8

• DevEval assesses LLMs across design, environment setup, implementation, and testing through a multi-stage evaluation process.Source: §5 · p. 8
• The study identifies difficulties understanding complex repository structures and handling the nuanced demands of comprehensive software development.Source: §5 · p. 8
• The findings point to critical pathways for future model development.Source: §5 · p. 8

6 Limitations

• Only 22 curated repositories across four programming languages are included, limiting the breadth of the evaluation.Source: §6 · p. 8
• The selected repositories may not fully capture the variety of challenges and complexities in real-world software development.Source: §6 · p. 8
• The authors identify additional repositories as a way to make future evaluation more comprehensive.Source: §6 · p. 8

A Software Engineering Tasks with a Running Example

• The Actor Relationship Game is a Java application that uses TMDB data to build an actor graph and find the shortest relationship path between actors.Source: A Software Engineering Tasks with a… · p. 11

A.1 Software Design

• Design documents translate requirements into a PRD and UML diagrams that specify structures, architecture, interfaces, and modules.Source: §A.1 · p. 11

A.1.1 Class Diagrams

• The example class diagram maps Actor and Movie entities, an ActorGraph management layer, and utility classes to their associations and operations.Source: §A.1.1 · pp. 12–13, Figure 2 · p. 12

A.1.2 Sequence Diagrams

• The sequence diagram traces graph creation, TMDB data retrieval, serialization, loading, and actor-connection lookup.Source: Figure 3 · p. 12, §A.1.2 · p. 13

A.1.3 Architecture Design

• A hierarchical file-tree representation shows repository organization, including test classes for Java classes.Source: §A.1.3 · p. 13

A.2 Software Development

• The development lifecycle includes environment setup and implementation, with setup preparing tools and implementation following repository design documents.Source: §A.2 · pp. 13–14

A.3 Quality Assurance

• QA examples distinguish acceptance tests that compare generated actor lists with reference outputs from unit tests checking Actor identity and movie-ID behavior.Source: §A.3 · p. 14

B.1 Dataset Construction

• Dataset preparation selects manageable repositories, checks and cleans their code and tests, then creates UML and architecture documents aligned with each repository.Source: §B.1 · p. 15

B.2 Dataset Statistics

• 22 curated repositories span Python, C/C++, Java, and JavaScript across diverse domains and multi-file structures.Source: §B.2 · p. 15
• Python repositories average about two files and 276 lines of code, while JavaScript repositories have the most files and lines.Source: §B.2 · p. 15
• Extensive reference acceptance and unit tests are prepared for every repository to evaluate implementation tasks.Source: §B.2 · p. 15

C The Baseline System

• The baseline system is introduced as a DevEval system built on ChatDev’s foundations.The implementation uses LMDeploy for CodeLlama and DeepSeek-Coder, 32K context, task-specific temperatures, default remaining hyperparameters, and Docker sandboxes.Source: C The Baseline System · pp. 15–16
• ChatDev organizes waterfall development into design, coding, testing, and documentation, using role-based agents and multi-turn chat chains for smaller tasks.Programmers, reviewers, and testers propose and validate solutions through context-sensitive dialogues.Source: C The Baseline System · p. 15
• The baseline extends ChatDev to cover DevEval’s PRD-based design, environment setup, acceptance testing, and multi-language execution-feedback needs.It includes UML class and sequence designs and repository architecture planning.Source: C The Baseline System · pp. 15–16

D Software Design Evaluation

• The study compares 192 response pairs across eight repositories, eight models, and three subtasks, with GPT-4-Turbo judging against GPT-3.5-Turbo.Each pair is judged twice in different orders; inconsistent LLM decisions count as ties, yielding 384 judgments.Source: D Software Design Evaluation · p. 16
• Human evaluators see shuffled response orders and annotate each pair three times, with the majority determining the result.Source: D Software Design Evaluation · p. 16
• The LLM judge uses a customized prompt for guideline-based pairwise comparisons, and a regular expression extracts its choice.The guidelines focus on general principles and faithfulness.Source: D Software Design Evaluation · p. 16

E.1 Model Capacity

• Approximately 90% of C/C++ compilation and execution errors and Java errors are attributed to deficient Makefiles or Gradle files; even GPT-4-Turbo sometimes makes basic syntax errors in them.Generated build files may omit source files, dependencies, or essential tasks.Source: §E.1 · p. 16
• In multi-file repositories, models redundantly redefine global functions, and C/C++ outputs may mishandle header and implementation files, producing conflicting declarations or implementations.Source: §E.1 · p. 16
• Models often make file-reference errors, confuse global functions with class methods, and can struggle to correct complex reference mistakes through review.Larger models generally handle basic Python imports and C++ includes, but errors remain common without review.Source: §E.1 · pp. 16–17

E.2 Instruction Following

• GPT-4-Turbo can generate syntactically correct code yet alter required function or variable names, and review does not adequately fix these naming errors.The passage gives an example of replacing the instructed stack method top() with getTop().Source: §E.2 · pp. 18–19

Function Parameters and Overloading Errors

• Models, including GPT-4-Turbo, may omit required function parameters or overlook overloaded constructors and methods with different parameter sets.In the area_calculation example, models create parameterized but not parameterless constructors, and may resist correction during review.Source: Function Parameters and Overloading… · p. 19
• Type-conversion errors, including const and pointer misuse, are common in strongly typed languages such as C/C++; these issues are less critical but still present in Python.Source: Function Parameters and Overloading… · p. 19
• Models misuse variables beyond their scope or lifecycle, including loop variables and private class members accessed from outside the class.Source: Function Parameters and Overloading… · p. 19

E.3 Hallucination

• Models fabricate nonexistent local variables, an issue typically corrected during review, and may misinterpret data files as Python libraries and attempt invalid imports.The passage links variable fabrication to sequential generation and describes data-file misinterpretation as causing reference errors.Source: §E.3 · p. 19

E.4 Limitations in Testing

• Acceptance tests are evaluated by execution without manual quality assessment, an approach under which smaller models may set arbitrary criteria and always give positive feedback.Source: §E.4 · pp. 19–20
• GPT-4-Turbo may follow original repository methods rather than modified versions, causing import errors, and can retain original import logic after __init__.py files are removed.The authors associate this behavior with training-data bias and reluctance to adjust to changed circumstances.Source: §E.4 · p. 20
• Structured test templates with explicit instructions and incomplete assertions are proposed as a way to guide models toward meaningful test generation.Source: §E.4 · p. 20

F Repositories statistics in DevEval

• Table 8 presents repository statistics for DevEval.Source: F Repositories statistics in DevEval · p. 20, Table 8 · p. 21