PaperlayerView exact revision on arXiv

Source-linked AI summary

RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository

Zhiyuan Peng, Xin Yin, Pu Zhao, Fangkai Yang, Lu Wang, Ran Jia, Xu Chen, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

arXiv:2601.13943v3cs.SE

TL;DR

Existing benchmarks do not adequately assess generating complete microservice repositories from scratch. RepoGenesis provides a multilingual benchmark and evaluates agents and IDEs across correctness, API coverage, and deployability; the best-performing system reaches only 23.67% Pass@1, while fine-tuned GenesisAgent-8B performs comparably to GPT-5 mini.

  • Problem

    Existing benchmarks focus on isolated code generation, modifying existing codebases, or repository generation with limited microservice relevance or scale.

    Source: §1 · p. 1

  • Method

    RepoGenesis benchmarks repository-level web microservice generation with 106 repositories and black-box tests, evaluated using Pass@1, API Coverage, and Deployment Success Rate.

    Source: §1 · pp. 1–2

  • Results

    The best-performing system achieves only 23.67% Pass@1, while GenesisAgent-8B fine-tuned on RepoGenesis (Train) performs comparably to GPT-5 mini.

    Source: §7 · p. 8

  • Takeaways & Limitations

    RepoGenesis provides a benchmark for model improvement, with GenesisAgent-8B’s GPT-5 mini-comparable performance after fine-tuning serving as evidence of that use.

    Source: §7 · p. 8

    RepoGenesis covers RESTful web microservices implemented in Python and Java, not other architectural patterns or programming paradigms.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models and agents have achieved remarkable progress in code generation. However, existing benchmarks focus on isolated function/class-level generation (e.g., ClassEval) or modifications to existing codebases (e.g., SWE-Bench), neglecting complete microservice repository generation that reflects real-world 0-to-1 development workflows. To bridge this gap, we introduce RepoGenesis, the first multilingual benchmark for repository-level end-to-end web microservice generation, comprising 106 repositories (60 Python, 46 Java) across 18 domains and 11 frameworks, with 1,258 API endpoints and 2,335 test cases verified through a "review-rebuttal" quality assurance process. We evaluate open-source agents (e.g., DeepCode) and commercial IDEs (e.g., Cursor) using Pass@1, API Coverage (AC), and Deployment Success Rate (DSR). Results reveal that despite high AC (up to 73.91%) and DSR (up to 100%), the best-performing system achieves only 23.67% Pass@1 on Python and 21.45% on Java, exposing deficiencies in architectural coherence, dependency management, and cross-file consistency. Notably, GenesisAgent-8B, fine-tuned on RepoGenesis (train), achieves performance comparable to GPT-5 mini, demonstrating the quality of RepoGenesis for advancing microservice generation. We release our benchmark at https://github.com/pzy2000/RepoGenesis.

1 Introduction

Existing benchmarks leave complete microservice repository generation from scratch under-evaluated. RepoGenesis addresses this gap, and its evaluation finds that functional correctness remains difficult even as benchmark-driven fine-tuning reaches GPT-5 mini-level performance.

1 Introduction

  • Existing benchmarks emphasize isolated function or class generation, repository modification, or repository generation outside microservices or at limited scale and diversity.These paradigms do not adequately evaluate generating complete microservice repositories from scratch.Source: §1 · p. 1
  • RepoGenesis is introduced as a benchmark for repository-level web microservice generation, targeting architectural design, dependency management, and system-level consistency.It comprises 106 repositories across 18 domains and 11 frameworks, with 1,258 API endpoints and 2,335 test cases.Source: §1 · pp. 1–2
  • Its quality assurance uses a review-rebuttal loop with three LLM reviewers and a human Area Chair to support test-case reliability and reduce potential bias.The benchmark pairs requirement documents with black-box test suites and draws on real-world and expert-curated implementations.Source: §1 · p. 2
  • The evaluation methodology uses Pass@1 for functional correctness, API Coverage for implementation completeness, and Deployment Success Rate for deployability.The paper evaluates open-source coding agents and commercial IDEs.Source: §1 · p. 2
  • 23.67% Pass@1 is the best-performing agent’s result, with failures primarily involving cross-file consistency (50.2%), architectural coherence (26.0%), and dependency management (23.8%).The evaluation also reports a gap between commercial IDE deployability and functional correctness, while GenesisAgent-8B achieves GPT-5 mini-level performance after fine-tuning on RepoGenesis.Source: §1 · pp. 2–3

2 Benchmark – RepoGenesis

RepoGenesis defines repository generation from a README as a task requiring complete, deployable microservices that satisfy black-box tests. Its benchmark and pipeline cover repository scope, code generation, execution, and evaluation across three quality dimensions.

2.1 Statistics of RepoGenesis

  • The verified benchmark contains 30 repositories across 18 domains and 11 frameworks; the broader collection adds 76 training repositories and totals 1,258 API endpoints and 2,335 test cases.The verified set combines six real-world GitHub repositories with 24 expert-supervised repositories; difficulty levels are based on code complexity metrics.Source: §2.1 · p. 3

2.2 Task Definition and Challenges

  • The task is to generate a fully deployable repository from requirements specifying service behavior, API schemas, authentication, and operational constraints, then pass black-box tests.Challenges include architecture and module decomposition, cross-file consistency, and real-world complexity such as authentication, database interactions, and validation.Source: §2.2 · p. 3

2.3 Evaluation Pipeline

  • The evaluation pipeline has two stages: repository-level code generation followed by post-generation evaluation.The figure presents this flow from requirements through agent-generated repositories to evaluation.Source: §2.3 · p. 3, Figure 2 · p. 4

2.3.1 Repository-level Code Generation

  • Agents receive a README describing requirements and generate a complete repository from scratch, including source code, configuration files, and dependency specifications.Source: §2.3.1 · p. 3

2.3.2 Post-Generation Evaluation

  • Generated repositories are evaluated through deployment and black-box testing using Pass@k, API Coverage, and Deployment Success Rate.These dimensions assess functional correctness, implementation completeness, and deployability, respectively.Source: §2.3.2 · pp. 3–4, Figure 2 · p. 4

3 Benchmark Construction

RepoGenesis is built through repository collection and preprocessing, test-suite construction with iterative review and human oversight, and difficulty classification. Its separate training set omits human adjudication, creating a stated reliability boundary.

3 Benchmark Construction

  • Construction comprises repository collection, preprocessing, test-case quality assurance, and difficulty classification.Source: §3 · p. 4

3.1 Repository Collection

  • The collection combines deployable GitHub microservices with expert-supervised repositories designed to extend domain coverage beyond available open-source projects.The authors initially identified 28 GitHub projects and retained nine after deployment screening; two researchers authored 97 requirement documents for expert-supervised repositories.Source: §3.1 · p. 4

3.2 Repository Preprocessing

  • Preprocessing refined real-world READMEs, removed implementation code, verified tests, and checked API specifications and requirement-functionality consistency in expert-supervised repositories.The process yielded 106 repository specifications ready for evaluation.Source: §3.2 · p. 4

3.3 Test Case & Quality Assurance

  • Test suites cover API compliance, error handling, edge cases, and integration, with Gemini-generated cases refined through multi-model review and human oversight.Reviewers score cases independently; score disagreements can trigger Area Chair review, while consistent but low-scoring cases can be refined iteratively, up to five iterations.Source: §3.3 · pp. 4–5
  • Across 150 test-case evaluations, Krippendorff’s α rose from 0.032 under initial zero-shot scoring to 0.69 after iterative rubric refinement; 78% showed agreement within two points.Only test cases with mean scores ≥7 were included.Source: §3.3 · p. 5

3.4 Difficulty Classification

  • Difficulty scores weight LOC at 25%, cyclomatic complexity at 30%, file count at 15%, API endpoints at 15%, and functions or classes at 15%; verified repositories are mostly Medium.The verified set has 6 Easy, 16 Medium, and 8 Hard repositories.Source: §3.4 · p. 5

3.5 RepoGenesis for Training

  • RepoGenesis (Train) contains 76 repositories but omits human Area Chair adjudication, discarding conflicting cases; approximately 23% of initially generated test cases were discarded.The authors note that this may remove challenging edge cases and exclude the training set from coding-agent evaluation; the reported α = 0.69 applies only to the verified set.Source: §3.5 · pp. 5–6

4 Experimental Setup

The experimental setup evaluates open-source coding agents and commercial IDEs using Pass@1, API Coverage, and Deployment Success Rate. Results are averaged across five independent runs.

4.1 Evaluated Systems

  • The evaluation includes four open-source agents—DeepCode, MetaGPT, MS-Agent, and Qwen-Agent—and three commercial IDEs—Antigravity, Cursor, and Copilot.The systems are described as the evaluated agents and IDEs for the study.Source: §4.1 · p. 6

4.2 Evaluation Metrics

  • Generated repositories are scored with Pass@1 for functional correctness, API Coverage for implementation completeness, and Deployment Success Rate for deployability.Results are averaged across five independent runs.Source: §4.2 · p. 6

5 Evaluation

The evaluation finds a large gap between repository-level functional correctness and deployability or API coverage, while fine-tuning GenesisAgent-8B reaches performance comparable to GPT-5 mini. Its empirical lessons identify capabilities associated with successful generation and recurring infrastructure, compatibility, and tooling failures.

5.1 Results of Coding Agents and IDEs

  • Copilot achieves 23.67% Pass@1 on Python and 21.45% on Java, while commercial IDEs outperform open-source agents and performance degrades as repository difficulty rises.Python performance generally exceeds Java; Qwen-Agent with GPT-5.1 falls from 25.00% (Easy) to 4.86% (Hard) on Python.Source: §5.1 · p. 6
  • Copilot reaches 95.45% DSR on Python and 62.50% on Java, while MetaGPT reaches 73.04% AC on Python and 74.14% on Java.Cursor's DSR is 68.18% on Python and 75.00% on Java; most open-source agents achieve below 20% DSR, and broad API skeletons can fail to initialize.Source: §5.1 · p. 7

5.2 Fine-Tuning Qwen3 with RepoGenesis

  • GenesisAgent-8B, fine-tuned on 16,396 successful repository-generation trajectories, achieves performance comparable to GPT-5 mini.The training samples were filtered for complete, compilable, executable repositories; Qwen3-8B was fine-tuned for 3 epochs at a learning rate of 2e-4.Source: §5.2 · p. 7

5.3 Empirical Lessons

  • Successful generation combines multilingual requirement decomposition, proactive validation, domain-specific algorithms, test-driven debugging, and fit-for-purpose data structures.The analysis describes these as five capabilities observed in success cases.Source: §5.3 · p. 7
  • Failures include missing or malformed deployment scripts, framework-version incompatibilities, and file-persistence problems that can leave no generated artifacts.The reported incompatibilities include deprecated Javalin parameter syntax; the phantom-generation example is Cursor with GPT-4.1.Source: §5.3 · p. 8

6 Related Works

The related work frames microservice generation as a system-level task requiring both functional correctness and architectural coherence. It contrasts earlier function-level benchmarks and newer repository-level efforts with RepoGenesis's multilingual, end-to-end microservice scope.

6.1 Microservices and Modern Software Architecture

  • Microservices use independently deployable services that communicate through lightweight mechanisms, typically RESTful APIs, making their generation a test of functional correctness and architectural coherence.RESTful APIs provide uniform interfaces for stateless communication, while frameworks such as Flask and Django abstract routing and request handling.Source: §6.1 · p. 8

6.2 Code Generation

  • Earlier code-generation benchmarks target function- or class-level problems, spanning Python, Java, multilingual programming tasks, data-science libraries, and open-source Java problems.Examples include HumanEval, AiXBench, MultiPL-E, DS-1000, and Concode.Source: §6.2 · p. 8
  • Repository-aware methods use cross-file context, while issue-based benchmarks and engineering agents target closed-loop requirements, code changes, testing, and delivery.The cited examples include A3-CodGen, RepoCoder, CatCoder, SWE-bench, SWE-PolyBench, SWE-agent, MAGIS, and OpenHands.Source: §6.2 · p. 8
  • Among prior natural-language repository-generation benchmarks, CodeS has 19 Python tasks, RPG has 6 Python repositories, and NL2Repo-Bench has 104 Python tasks evaluated solely by pytest pass rate.RepoGenesis differs by evaluating multilingual end-to-end generation of complete microservice repositories.Source: §6.2 · p. 8

7 Conclusion

The conclusion presents RepoGenesis as a multilingual benchmark for complete web microservice repository generation and reports persistent correctness and deployment challenges alongside a fine-tuning result comparable to GPT-5 mini. It proposes expanding benchmark scope, measuring code quality, and investigating ways to bridge the API-coverage and deployment gap.

7 Conclusion

  • RepoGenesis is a multilingual benchmark for repository-level web microservice generation, with 106 repositories across 18 domains and 11 frameworks, 1,258 API endpoints, and 2,335 test cases.The benchmark targets complete repository generation from natural-language requirements.Source: §7 · p. 8
  • The best-performing system achieves only 23.67% Pass@1, while open-source agents can produce broad API skeletons that fail to deploy and GenesisAgent-8B reaches performance comparable to GPT-5 mini.The authors present the fine-tuning result as evidence of RepoGenesis's quality for model improvement.Source: §7 · p. 8
  • Future work includes adding languages and architectural patterns, measuring design-pattern adherence and maintainability, and investigating agent architectures to bridge the AC-DSR gap.Source: §7 · p. 8

Limitations

RepoGenesis evaluates RESTful web microservices in Python and Java under clear, structured requirements, while leaving other system types, languages, and ambiguity resolution outside its scope. Passing tests does not establish that generated code is maintainable or follows software-engineering standards.

Limitations

  • The benchmark covers RESTful web microservices, not event-driven systems, real-time WebSocket applications, GraphQL APIs, or gRPC services.Source: Limitations · p. 9
  • Its language and framework coverage is limited to Python and Java, excluding paradigms such as C++ and Go and mobile development such as Swift.Source: Limitations · p. 9
  • RepoGenesis assumes comprehensive, structured requirements, leaving ambiguous, incomplete, or iterative requirements engineering outside its evaluation.Source: Limitations · p. 9
  • Test-based evaluation does not measure maintainability, readability, modular design, idiomatic framework usage, or adherence to industry standards.Source: Limitations · p. 9

Ethics Statement

This section documents RepoGenesis’s end-to-end microservice generation task and its evaluation procedures, including IDE prompts, metrics, agent design, and repository and difficulty structure. It also addresses risks and safeguards and explains the rationale for using pom.xml in Java generation.

Ethics Statement

  • RepoGenesis uses public repositories and documentation, reports no human-subject studies or newly collected PII, and applies license and content screening while acknowledging residual risks.The authors identify dual-use and environmental risks and recommend human review, sandboxing, and security scanning for downstream use.Source: Ethics Statement · p. 9

B Task Definition

  • The input is a README-style requirements document and language-specific dependency file; the output is a complete, deployable repository with a one-command startup script.Requirements can specify API schemas, authentication, error handling, data models, and operational constraints.Source: B Task Definition · p. 12
  • Pass@k requires passing every test, API Coverage measures correctly implemented endpoints, and Deployment Success Rate records successful initialization without runtime errors.The three metrics capture all-tests correctness, functional completeness, and basic executability, respectively.Source: B Task Definition · p. 13
  • Repository generation challenges models to design architectures, maintain cross-file consistency, and implement real-world concerns such as authentication, databases, validation, and error handling.These requirements distinguish the task from function- or file-level code generation.Source: B Task Definition · p. 13

C.1 Scoring Mechanism.

  • Test cases are independently scored on correctness, coverage and completeness, readability and maintainability, and relevance.The four dimensions use a 0–10 scale, with respective weights of 0–3, 0–3, 0–2, and 0–2.Source: §C.1 · p. 13

D.1 Evaluated Agents

  • The evaluated open-source agents are DeepCode, MetaGPT, MS-Agent, and Qwen-Agent, using their respective repository-generation workflows and tools.These include MetaGPT’s generate_repo interface, DeepCode’s CodeImplementationWorkflow, MS-Agent’s LLMAgent, and Qwen-Agent’s Assistant with code_interpreter.Source: §D.1 · pp. 13–14

D.2 Agent Prompts

  • Open-source agents share a standardized system prompt, with execution prompts adapted to each framework; the instructions require complete project files and a single-command startup script.The prompt variants specify language-specific dependency files and service-binding requirements.Source: §D.2 · p. 14, §D.2.1 · pp. 14–15

D.2.2 Agent-Specific Configurations

  • Agent-specific configurations varied by framework and language, and the study also used Antigravity, Cursor, and Copilot with prompts for generating Java and Python repositories.For example, DeepCode enabled read tools for Java but disabled them for Python; MetaGPT used five rounds and code review.Source: §D.2.2 · p. 15, §D.3 · p. 15

D.3 IDE Prompts

  • IDE prompts require systems to inspect repository subdirectories, generate needed application, configuration, dependency, and documentation files, provide a single-command start.sh, bind web services to 0.0.0.0, and write documented production-ready code.Source: §D.3 · p. 16

D.4.1 Pass@k Calculation and Its Necessity for Estimation

  • Pass@k counts a task as solved when at least one of k generated repositories passes all tests; the study uses an unbiased estimator from n = 10 samples to reduce variance and computation.The estimator follows Kulal et al. (2019), with k ≤ 10.Source: §D.4.1 · p. 16

D.5 API Coverage (AC) / D.6 Deployment Success Rate (DSR)

  • API Coverage (AC) detects implemented endpoints through static routing-signature analysis, independently of deployment or test execution, while Deployment Success Rate (DSR) checks whether a repository installs, initializes, and starts without runtime errors.AC excludes test directories and measures structural completeness; DSR is a binary deployability metric.Source: §D.5 · p. 17, §D.6 · p. 17

E Details of GenesisAgent

  • GenesisAgent extends MS-Agent with language-specific generation and execution strategies, including dependency-first Python generation and Java framework detection from pom.xml.The Python agent selects Flask or FastAPI based on requirements; the Java agent generates framework-appropriate start.sh commands and allows a 30-second startup timeout.Source: E Details of GenesisAgent · p. 17

F.1 Repository Structure

  • Each repository provides a README.md specification as the evaluated system’s sole input, a golden oracle implementation, and black-box test suites for functionality, API compliance, error handling, and business logic.The oracle is the original GitHub codebase for real-world repositories and an expert-supervised Gemini 3 Pro implementation for supervised repositories.Source: §F.1 · pp. 17–18

F.2 Difficulty Classification Methodology / F.2.1 Classification Metrics

  • Repository difficulty is classified using a weighted composite of five code-complexity metrics, with cyclomatic complexity weighted highest at 30%.The metrics are cyclomatic complexity, lines of code, file count, API endpoints, and functions and classes.Source: §F.2 · p. 18, §F.2.1 · p. 18

F.2.2 Scoring Algorithm / F.2.3 Difficulty Thresholds

  • The composite difficulty score assigns metric points by thresholds and groups repositories as Easy (S < 4.5), Medium (4.5 ≤ S < 7.0), or Hard (S ≥ 7.0).The final score ranges approximately from 2.0 to 10.0.Source: §F.2.2 · p. 19, §F.2.3 · p. 19

F.3 Why using pom.xml for Java repositories?

  • Java’s pom.xml constrains dependency and framework selection, avoiding guessed libraries that can break compilation; Python’s standard library and dynamic execution allow greater resilience to missing dependencies.Java’s compilation barrier means one unresolved import can prevent tests from running, while Python may still execute code paths that do not encounter a missing dependency.Source: §F.3 · pp. 19–20

G Framework-level Analysis

Framework deployability varies sharply, with Spring Boot at 100% and Quarkus and Spark at 25.00%. The supplied section also describes the hardware, test environment, coverage checks, evaluation controls, and case-study scope.

G Framework-level Analysis

  • Spring Boot achieves 100% deployability, versus Javalin at 93.75% and Quarkus and Spark at 25.00%.The authors relate these outcomes to framework design: Spring Boot's convention-over-configuration, compared with additional failure points in Quarkus and Spark.Source: G Framework-level Analysis · p. 20

H.1 Hardware and Testing Environment

  • Non-training evaluation and preprocessing ran on an Apple M4 Pro MacBook Pro, while supervised fine-tuning used two NVIDIA A100 80GB GPUs.Agent inference used the Azure API over approximately two weeks.Source: §H.1 · p. 20

H.2 Testing Environment

  • Repositories were tested in isolation on Ubuntu 20.04 LTS with Python 3.9, Node.js 18.x, and Conda 25.5.1, with a five-minute execution limit per repository.Fresh conda environments were used to prevent cross-contamination.Source: §H.2 · p. 20

I.1 API Coverage Calculation

  • API Coverage extraction handles explicit endpoint lines, Markdown tables, and feature lists, then checks method-and-path signatures in source files using framework-specific patterns.Test directories are excluded from implementation verification.Source: §I.1 · p. 20

I.2 Evaluation Infrastructure

  • Evaluation infrastructure terminates whole process groups and waits ten seconds after startup to avoid orphaned workers and premature tests.The warm-up period accounts for initialization overhead in heavier frameworks, especially Spring Boot.Source: §I.2 · pp. 20–21

J Detailed Case Study

  • The case-study section provides examples of successful and failed repository-generation attempts referenced in Section 5.3.Source: J Detailed Case Study · p. 21

J.1 Success Case: Data_Rank_Searcher

The appendix contrasts a repository-generation success with recurring structural and framework-version failures, including Cursor with GPT-4.1 repeatedly promising autonomous work without creating files.

J.1 Success Case: Data_Rank_Searcher

  • Copilot with claude-sonnet-4.5 passed 11/11 tests on Data_Rank_Searcher, handling four REST endpoints, compound queries, validation, and pagination.The agent parsed a Chinese README and corrected an off-by-one pagination calculation after tests exposed it.Source: §J.1 · p. 21

J.2 Common Failure Modes

  • Common failures include missing startup scripts and outdated Javalin 4 path syntax, which can prevent service initialization or cause runtime exceptions.DeepCode often omitted a valid start.sh entry point; agents also used legacy colon-prefixed parameters instead of brace-enclosed parameters.Source: §J.2 · p. 21

J.2.1 How Cursor with GPT-4.1 fails to generate a repository?

  • Cursor with GPT-4.1 entered a planning-and-acknowledgment loop, while the authors hypothesize workflow misalignment or misinterpreted autonomy instructions as possible causes.Despite repeatedly promising to proceed autonomously, it performed no file-system operations or code synthesis.Source: §J.2.1 · p. 22

K Additional Experiments

Extended experiments compare model, agent, language, difficulty, and IDE configurations, while the appendices document artifact terms, AI assistance, annotation procedures, and data practices.

K Additional Experiments

  • Pass@1 declined sharply with repository difficulty; Qwen-Agent with claude-sonnet-4.5 fell from 26.50% on Easy Python repositories to 6.45% on Hard ones.Qwen3-30B scored 0.00% on Python Hard tasks across all agents.Source: K Additional Experiments · p. 22, Table 13 · p. 23
  • Performance varied by language and model: Qwen-Agent with claude-sonnet-4.5 led open-source agents at 12.65% Pass@1 on Python and 7.80% on Java.Both Qwen3-30B variants mostly scored below 1%, while DeepCode scored 0.00% across most configurations.Source: K Additional Experiments · pp. 22–23, Table 14 · p. 23, Table 15 · p. 23
  • MetaGPT with claude-sonnet-4.5 reached 73.04% API Coverage on Python but only 13.00% Deployment Success Rate, while MS-Agent with GPT-5.1 reached 37.50% DSR and 47.41% AC on Java.Increasing MS-Agent from GPT-5 mini to GPT-5.1 raised DSR from 4.55% to 18.18% on Python and from 25.00% to 37.50% on Java.Source: K Additional Experiments · p. 23, Table 16 · p. 24
  • Copilot with GPT-5 mini achieved 100.00% DSR on Java and 90.91% on Python; Cursor with GPT-4.1 scored 0.00% on both DSR and AC.The reported IDE-model results also show language-specific DSR differences for claude-sonnet-4.5 pairings.Source: K Additional Experiments · p. 23, Table 17 · p. 24
  • Copilot with Grok Code Fast 1 scored 1.50% (Python) and 3.12% (Java) Pass@1, below claude-sonnet-4.5’s 23.67% and 21.45% baseline scores.The extended results also report Gemini 3 Pro outperforming Gemini 3 Flash by 0.85pp on Python and 0.51pp on Java.Source: K Additional Experiments · p. 24, Table 18 · p. 25

L The License For Artifacts

N Human Annotations

  • Two PhDs with at least three years of coding experience annotated test quality, and the review process uses reviewer consensus, human adjudication, and iterative refinement.When reviewer scores differ by more than three points, the Area Chair evaluates the case; refinement may run for up to five iterations.Source: N Human Annotations · p. 26, §N.1 · pp. 26–27

N.2 Consent for Data Usage

Q Potential Risks

The paper identifies risks from insecure generated services, training compute, narrow data coverage, and possible misuse, and notes the benchmark’s web-microservice scope.

Q Potential Risks

  • Generated repositories may contain faulty or insecure code that could be exploited if deployed in production, motivating security evaluation and comprehensive testing.The authors also note possible misuse for illicit purposes such as unauthorized data access or system compromise.Source: Q Potential Risks · p. 28
  • Training and fine-tuning large models require substantial GPU hours and energy, creating environmental impact that future work should mitigate.The authors suggest improving model efficiency or exploring more energy-efficient training approaches.Source: Q Potential Risks · p. 28
  • A narrow training-data source could produce code poorly suited to diverse or marginalized developer groups; the authors cite broad real-world repository coverage as a response.They acknowledge the risk of exacerbating existing biases or inequalities in software development.Source: Q Potential Risks · p. 28
  • Rapid repository generation could be misused to create malicious services or fraudulent systems, and unvalidated insecure code could cause unintended consequences in production.The authors recommend validation before generated code is used in production environments.Source: Q Potential Risks · p. 29
  • The work focuses on web microservices and specific frameworks, which may exclude communities using other technology stacks or programming paradigms.The authors call for future support for more diverse frameworks and languages.Source: Q Potential Risks · p. 29

Related papers

Loading 2601.13943v3…