PaperlayerView exact revision on arXiv

Source-linked AI summary

Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents

Kaixin Wang, Tianlin Li, Xiaoyu Zhang, Chong Wang, Weisong Sun, Yang Liu, Aishan Liu, Xianglong Liu, Chao Shen, Bin Shi

arXiv:2505.05283v3cs.SEcs.AI

TL;DR

Comprehensive reviews of CodeLLM benchmarks from an SDLC perspective are lacking, leaving open whether benchmarks cover the lifecycle and assess realistic capabilities. This paper applies a tiered analysis to 178 benchmarks and finds coverage concentrated in implementation, with requirements engineering and design underrepresented.

  • Problem

    It remains unclear whether existing benchmarks cover all SDLC phases and faithfully measure capabilities required in realistic software engineering settings.

    Source: §I · p. 1

  • Method

    The paper systematically reviews 178 benchmarks using a tiered framework that combines general and phase-specific dimensions.

    Source: §I · pp. 1–2

  • Results

    Approximately 61% of benchmarks focus on software implementation, while requirements engineering and design together account for less than 8%.

    Source: §VIII · p. 16

  • Takeaways & Limitations

    The paper identifies benchmark deficiencies and outlines directions intended to align evaluation with realistic engineering scenarios.

    Source: §I · p. 2

    The authors identify subjectivity in paper selection and data extraction, along with ambiguity in taxonomy classification, as threats to validity.

    Source: §VII · p. 16

Abstract

from arXiv · show

Code large language models (CodeLLMs) and agents are increasingly being integrated into complex software engineering tasks spanning the entire Software Development Life Cycle (SDLC). Benchmarking is critical for rigorously evaluating these capabilities. However, despite their growing significance, there remains a lack of comprehensive reviews that examine these benchmarks from an SDLC perspective. To bridge this gap, we propose a tiered analysis framework to systematically review 178 benchmarks from 461 papers, comprehensively characterizing them from the perspective of the SDLC. Our findings reveal a notable imbalance in the coverage of current benchmarks, with approximately 61\% focused on the software implementation phase in SDLC, while requirements engineering and software design phases receive minimal attention at only 5\% and 3\%, respectively. % Additionally, anti-contamination strategies are largely absent from current benchmarks, leading to an increased risk of data leakage. Furthermore, current benchmarks lack effective anti-contamination strategies, posing significant risks of data leakage and potentially inflated performance assessments. Finally, we identify key open challenges in current research and outline future directions to narrow the gap between the theoretical capabilities of CodeLLMs and agents and their practical effectiveness in real-world scenarios.

I. INTRODUCTION

The paper asks whether CodeLLM benchmarks adequately assess real-world software engineering across the SDLC, a question left open by surveys with narrower scopes. It reviews benchmarks through an SDLC-oriented tiered framework and identifies mismatches with practical engineering needs.

A. LLMs for Software Engineering

  • The central open question is whether benchmarks cover all SDLC phases and faithfully measure capabilities needed in realistic settings.The paper situates this question amid CodeLLMs’ expanding capabilities, from code generation to broader SDLC activities.Source: §I · p. 1, A. LLMs for Software Engineering · p. 2

I. INTRODUCTION

  • Existing surveys either focus on traditional machine learning, lack comprehensive benchmark-centered analysis, or largely restrict CodeLLM coverage to implementation and testing.Concurrent studies emphasize task clustering and benchmark meta-properties rather than systematic analysis of coverage scope and assessment depth.Source: §I · p. 1
  • The paper reviews 178 benchmarks and maps them across five SDLC phases to reveal structural imbalances.It proposes a tiered framework to analyze alignment with real-world software engineering needs.Source: §I · pp. 1–2
  • The paper identifies benchmark deficiencies and outlines targeted directions to better align evaluation with realistic engineering scenarios.It also curates and open-sources detailed metadata for all 178 benchmarks.Source: §I · p. 2

B. Software Development Life Cycle

The paper frames the SDLC as a sequence from requirements engineering through maintenance and identifies a lack of comprehensive benchmark coverage across that cycle. It poses three review questions and uses a multi-stage literature search and screening process to assemble its study corpus.

B. Software Development Life Cycle

  • The SDLC spans requirements engineering, design, implementation, testing, and maintenance, but comprehensive benchmark surveys across the full cycle are lacking.These phases cover specification, architectural planning, system construction, validation, and long-term updates.Source: B. Software Development Life Cycle · p. 2

A. Review Questions

  • The review asks whether benchmarks cover real-world capabilities across the SDLC, what distributional imbalances exist, and what future directions are needed..Source: A. Review Questions · p. 2

B. Collection Strategy

  • The review derived 20 keyword combinations, searched four academic databases and retrieved 1,347 relevant papers; manual selection using stated criteria reduced the set to 321 articles.Literature published since 2022 was included [29].Source: B. Collection Strategy · p. 3
  • Forward and backward snowball sampling added 140 papers, bringing the final collection to 461 studies reviewed by two co-authors, with disagreements resolved by a third.Figure 1 maps 178 benchmarks by SDLC phase or task and omits six cross-stage benchmarks.Source: B. Collection Strategy · p. 3, Figure 1 · p. 3

C. Tiered Analysis Framework for CodeLLM Benchmarks

The paper proposes a tiered framework combining shared evaluation dimensions with criteria tailored to each SDLC phase. Its dimensions address contamination, authenticity, interaction and evaluation, alongside phase-specific demands such as domain diversity and repository context.

C. Tiered Analysis Framework for CodeLLM Benchmarks

1) General Dimensions:

  • General dimensions cover contamination defenses, data authenticity, interaction mode, and evaluation paradigm.Interaction ranges from single-turn prompts to iterative feedback and agentic planning; evaluation distinguishes human judgment from automated metrics and dynamic execution from text matching.Source: 1) General Dimensions · pp. 3–4

2) Stage-Specific Dimensions:

  • Requirements and design dimensions include application domain, input format, requirement evolution, and retrieval-based versus generative task paradigms.The paper notes that design benchmarks focused on well-defined tasks such as classification may miss the creative nature of software design.Source: 2) Stage-Specific Dimensions · p. 4
  • For implementation, testing, and maintenance, the framework emphasizes context scope and maintenance’s evolutionary character.It calls for repository-level understanding and distinguishes evolving an existing system with code history from building from scratch.Source: 2) Stage-Specific Dimensions · p. 4

IV. BENCHMARK TAXONOMY BY THE SDLC

The paper groups benchmarks by their primary SDLC evaluation objective, assigning overlapping tasks to their main phase and separating inherently multi-stage benchmarks. Its requirements-engineering review finds limitations in benchmark availability, artifact formats, evolution support, and evaluation practices.

IV. BENCHMARK TAXONOMY BY THE SDLC

  • Benchmarks are assigned to the SDLC phase matching their primary evaluation objective, with inherently multi-stage benchmarks placed in a cross-phase category.Each phase is further subdivided by task.Source: §IV · p. 4

A. Requirement Engineering

1) Requirements Elicitation:

  • PURE is a widely used requirements-acquisition benchmark with 79 real-world natural-language documents across domains, but it supports only plain text and not requirement evolution.Later benchmarks target specialized domains, including software contracts and cyber-physical systems.Source: 1) Requirements Elicitation · p. 4

2) Requirements classification:

  • PROMISE evaluates requirements classification using data from 15 projects, but is limited to plain text and does not evaluate requirement evolution..Source: 2) Requirements classification · p. 4

3) Requirements Modeling:

  • Ferrari’s requirements-modeling benchmark uses 28 industrial documents and structured artifacts, but relies on subjective human scoring.That scoring limits objectivity and reproducibility.Source: 3) Requirements Modeling · p. 4

4) Requirements Specification:

  • Krishna’s requirements-specification benchmark uses manually written documents aligned with IEEE standards, but is narrowly scoped to university club management systems.Its subjective scoring also limits generalizability, objectivity, and reproducibility.Source: 4) Requirements Specification · p. 4

5) Requirements verification:

  • Requirements-verification benchmarks use differing approaches, including code-centric tests, knowledge QA, and structured artifacts, but none evaluate requirement evolution.The review also reports no public requirements-management benchmarks, limited support for real-world artifacts, mostly single-turn interaction, and a lack of anti-contamination measures.Source: 5) Requirements verification · pp. 4–5, Table II · p. 5

B. Software Design

Software design spans UI, algorithm, architecture, and database design, but its benchmarks remain limited in scope and often miss creative or iterative work.

1) UI Design:

  • UI benchmarks emphasize GUI understanding, retrieval, description, or summarization rather than generative design, underrepresenting creativity.Fixed references make these auxiliary tasks easier to score.Source: 1) UI Design · p. 5

2) Algorithm Design:

  • CLRS-30 assesses understanding and generalization over classical algorithms, while BLADE explores algorithm synthesis; differing metrics prevent direct comparison.The review identifies a need for standardized evaluation.Source: 2) Algorithm Design · p. 5

3) Architectural & Database Design:

  • No public benchmarks cover architectural or database design; open-ended creativity is difficult to standardize, and static snapshots miss iterative optimization.The review also notes that current UI benchmarks often emphasize retrieval or summarization rather than generative creativity.Source: 3) Architectural & Database Design · p. 5, Table III · p. 6

C. Software Implementation

  • Software implementation translates design into executable code; the review catalogs its code-generation and other implementation tasks in Tables IV and V.The passage identifies code generation and other implementation-related tasks as separate table topics.Source: C. Software Implementation · p. 6

1) Code Generation and Completion:

Code-generation benchmarks span general software engineering, repository work, competitive programming, multilingual and multimodal tasks, specialized domains, and other evaluation goals.

General SE

  • Function-level benchmarks such as HumanEval and MBPP have limited test coverage and realism; HumanEval+ and MBPP+ expand test suites roughly 80×, while BigCodeBench adds multi-library calls.ClassEval provides 100 class-level examples; repository-level work broadens evaluation toward realistic development settings.Source: 1) Code Generation and Completion · p. 6

Repository-level benchmarks

  • Repository benchmarks progress from limited early suites toward expanded context and realistic workflows, including cross-file dependencies, repository environments, and project evolution.CoderEval and PragmaticCode expand context; CodeAgentBench targets end-to-end repository problem solving, while HumanEvo, FEA-Bench, and EvoCodeBench address evolving projects or incremental development.Source: 1) Code Generation and Completion · p. 6

Competitive Code Generation benchmarks

  • Static competitive-programming datasets face leakage risks; later benchmarks use temporal splits, post-cutoff problems, continuous evaluation, or hidden tests.LiveCodeBench collects problems released after the training cutoff; CODEELO submits solutions to Codeforces using its execution environment and hidden tests.Source: 1) Code Generation and Completion · pp. 7–8

Multilingual benchmarks

  • Multilingual code-generation benchmarks expand from single-language tasks to 40 programming languages, using manual translation, compiler translation, or broader language coverage.HumanEval-X covers four languages and MultiPL-E covers 18.Source: 1) Code Generation and Completion · p. 8

Multimodal benchmarks

  • Multimodal benchmarks assess code generation from mixed text, images, and structured tables, including visualization, front-end design, and agent workflows.Spider2-v adds multimodal interaction and real-world operations for an end-to-end data-engineering workflow.Source: 1) Code Generation and Completion · p. 8

Domain-specific benchmarks

  • Domain-specific benchmarks target fields including data science, bioinformatics, machine learning, education, robotics, and specialized programming languages.StudentEval provides 1,749 basic programming prompts for educational settings.Source: 1) Code Generation and Completion · p. 8

Other benchmarks

  • Other code-generation benchmarks cover interactive generation, automated benchmark construction, cross-domain generalization, and code quality beyond functional correctness.DomainEval spans 2,454 topics across six application areas; Race evaluates readability, maintainability, correctness, and runtime efficiency.Source: 1) Code Generation and Completion · p. 8

2) Text-to-SQL:

The surveyed implementation benchmarks cover text-to-SQL, code understanding and reasoning, translation, summarization, and type inference, with varying realism and task scope.

2) Text-to-SQL:

  • Text-to-SQL benchmarks range from single-table queries to complex, multi-domain databases, with variants addressing linguistic diversity, domain knowledge, schema cues, dialogue, and real-world data.Spider contains 10,181 questions and 5,693 SQL queries across 138 domains and 200 databases; the review notes that some queries rely on overly specific schema references.Source: 2) Text-to-SQL · p. 8

3) Code Understanding and Reasoning:

  • Code-understanding and reasoning benchmarks assess question answering, input/output prediction, intermediate runtime behavior, specifications, and fine-grained problem-solving tasks.CRUXEval-X extends input/output reasoning across languages; FAUN-Eval separates code QA, fault localization, and code editing.Source: 3) Code Understanding and Reasoning · pp. 8–9

4) Code Translation:

  • Code-translation benchmarks range from manually curated equivalent solutions in 14 languages to datasets covering over 40 languages with execution-based DSR@K evaluation.Source: 4) Code Translation · p. 9

5) Code Summarization:

  • Code-summarization benchmarks pair code with natural-language descriptions, with early work on C# and SQL and later datasets focused on Java repositories.Source: 5) Code Summarization · p. 9

6) Type Inference:

  • Type-inference benchmarks include human-rated identifier-embedding similarity and a dataset of 300 TypeScript codebases containing user-defined type annotations.Source: 6) Type Inference · p. 9

7) Code Retrieval:

Code retrieval benchmarks have expanded query realism, while non-functional evaluation addresses robustness, security, copyright, efficiency, bias, explainability, and human preferences.

7) Code Retrieval:

  • CodeSearchNet covers six languages and about six million paired examples, while CoSQA uses real Bing queries and CoSQA+ evaluates one-to-many query–code matching.DeepCS was limited to Java, and CodeSearchNet queries are described as not fully authentic.Source: 7) Code Retrieval · p. 10

8) Non-functional Benchmarks:

  • Robustness benchmarks test input perturbations, while security work ranges from static malicious-code assessment to dynamic execution for agents; copyright suites examine AI-code detection and reuse.ReCode covers over 30 input transformations; RMCBench covers 11 malicious-code types and nine languages, while RedCode evaluates dynamic interaction.Source: 8) Non-functional Benchmarks · p. 10
  • Efficiency benchmarks measure runtime and memory, but environment dependence complicates comparisons; proposed metrics address joint correctness-efficiency, hardware variation, and timed-out samples.Mercury introduces the Beyond metric, EvalPerf uses hardware performance counters, and ENAMEL proposes eff@k.Source: 8) Non-functional Benchmarks · p. 10
  • The review also covers explainability, social and recommendation bias, and alignment with human preferences as non-functional evaluation targets.CodeArena includes 397 samples across 40 task categories and 44 programming languages.Source: 8) Non-functional Benchmarks · pp. 10–11
  • Benchmarks often lack effective anti-contamination measures, realistic text-to-SQL queries, broad language coverage, enterprise privacy and collaboration dimensions, and multi-turn or environment interaction.Static defenses such as cutoffs and code perturbation have limited long-term effectiveness; dynamic updates and online evaluation reduce the chance of pre-training inclusion at the source.Source: 8) Non-functional Benchmarks · pp. 10–11

D. Software Testing

Software testing benchmarks cover vulnerability detection, test generation, and defect detection; the discussion also introduces software maintenance as modifying deployed software. The review highlights limitations in benchmark context, language coverage, and contamination safeguards.

1) Vulnerability Detection:

  • Vulnerability datasets have shifted from manually designed synthetic examples toward real-project mining, while FormAI generates independently compilable samples for LLM vulnerability analysis.Synthetic datasets can have accurate labels but limited error-type coverage; FormAI provides 112,000 (v1) and 265,000 (v2) samples.Source: 1) Vulnerability Detection · p. 11

2) Test Generation:

  • Test-generation benchmarks range from small-scale classes to over 780,000 real-world test-method mappings, while SWTBench evaluates whether generated tests reproduce real issues.SWTBench contains more than 1,900 samples built from real GitHub issues and fixes.Source: 2) Test Generation · p. 11

3) Defect Detection:

  • Defect-detection benchmarks include manually annotated JIT-Defects4J records and Java datasets spanning defects from projects of varied scale and diversity.JIT-Defects4J has 27,391 change records from 21 projects; Bugs.jar has 1,158 defects from eight projects, and Bears has 251 defects from 72 projects.Source: 3) Defect Detection · p. 12
  • Testing benchmarks often lack effective anti-contamination mechanisms, use function-level contexts and static verification, and provide limited agent interaction and modern-language coverage.The review also notes a heavy skew toward C/C++ and Java, with limited Rust and Go support.Source: 3) Defect Detection · p. 12

E. Software Maintenance

  • Software maintenance modifies deployed software to fix defects, improve performance, and adapt to changing requirements or operating environments.Source: E. Software Maintenance · p. 12

1) Program Repair:

The surveyed maintenance benchmarks span repair, editing, log parsing, vulnerability repair, decompilation, and API misuse detection. Their examples range from single-file fixes to repository-level tasks and binary-level evaluation.

1) Program Repair:

  • Repair benchmarks progress from real defects in single-language datasets to cross-language and repository-level tasks, although SWE-bench’s manual construction constrains scale and diversity.SWA-Bench covers 535 tasks from 44 projects and SWEE-Bench 885 tasks from 366 projects; both require modifying 3.26 files on average, versus 1.66 for SWE-bench.Source: 1) Program Repair · pp. 12–13

2) Code Editing:

  • PyCommits uses commit histories from 1,650 Python repositories, while CoEdPilot covers JavaScript, Java, Go, Python, and TypeScript.Source: 2) Code Editing · p. 13

3) Log Parsing:

  • Log-parsing benchmarks add template-level metrics to address limitations in Loghub’s scale, annotation quality, and focus on grouping accuracy.LogPM has over ten million entries; Loghub-2.0 contains 14 datasets averaging roughly 3.6 million entries each.Source: 3) Log Parsing · p. 13

4) Vulnerability Repair:

  • VJBench expands Vul4J’s vulnerability coverage, and VJBench-trans uses identifier renaming and program restructuring to mitigate data leakage.ManyVuls4J extends evaluation to Java binaries, covering 103 vulnerabilities across 55 projects.Source: 4) Vulnerability Repair · p. 13

5) Decompilation:

  • ExeBench evaluates decompiled-code functional correctness with automatically generated input/output test pairs, while WaDec supplies compiled C programs.WaDec includes 51,768 programs from Hugging Face and 377 from GitHub, compiled into WebAssembly.Source: 5) Decompilation · p. 14

6) API Misuse Detection:

  • API-misuse benchmarks cover Java through real-world Stack Overflow questions and C through handcrafted tests and real-world misuse cases.ROBUSTAPI covers 18 Java APIs; APIMU4C provides 2,172 handcrafted tests and 100 real-world cases.Source: 6) API Misuse Detection · p. 14

7) Code Clone Detection:

The review describes code-clone and cross-phase benchmarks, then analyzes benchmark usage and language distributions. It finds an implementation-heavy SDLC landscape and reports language differences across phases.

7) Code Clone Detection:

  • Code-clone benchmarks range from manually validated Java projects and C/C++ problem datasets to CodeNet’s 13.9 million snippets across 55 languages.Source: 7) Code Clone Detection · p. 14

F. Cross-Phase Benchmarks

  • Cross-phase benchmarks include multi-task and multilingual evaluations, while DevBench is the only benchmark described as covering every SDLC phase.CodeXGLUE covers tasks including clone detection, defect detection, generation, and completion; XCodeEval and CodeScope evaluate execution correctness across multiple languages.Source: F. Cross-Phase Benchmarks · p. 14
  • Current benchmarks are heavily skewed toward implementation, while requirements engineering and design remain underexplored; many also lack contamination safeguards and single-turn evaluation misses dynamic agent workflows.The review notes a shift toward repository-level implementation tasks, but maintenance benchmarks often overlook software evolution.Source: F. Cross-Phase Benchmarks · p. 14

V. STATISTICS AND ANALYSIS OF STUDIED BENCHMARKS

  • HumanEval leads benchmark usage with 62 papers, followed by MBPP (46) and APPS (23); CodeXGLUE has been adopted by 11 papers.Spider is used by 26 papers, BIRD by 13, and Spider-Realistic by seven.Source: §V · pp. 14–15, Figure 3 · p. 15
  • Python appears in 103 benchmarks, ahead of Java (65) and C/C++ (60), while most other languages appear in fewer than 10 benchmarks.The overall benchmark language distribution closely aligns with language popularity.Source: §V · p. 15, Figure 4 · p. 15
  • Across SDLC phases, C/C++ is most prevalent in testing, Java in maintenance, and Python in the other phases.The paper relates testing-phase C/C++ use to efficiency needs and maintenance-phase Java use to syntax and exception-handling characteristics.Source: §V · p. 15, Figure 5 · p. 15

VI. FUTURE DIRECTIONS

The paper proposes benchmark-development directions for agent evaluation, SDLC coverage, languages, and other evaluation dimensions. It concludes that benchmark coverage is imbalanced and notes limits to the study’s generalizability.

VI. FUTURE DIRECTIONS

  • The paper calls for standardized requirements benchmarks and broader design-phase tasks, including stakeholder needs, conflict resolution, architecture design, and objective creativity metrics.Design benchmarks should support inputs such as diagrams and natural-language specifications.Source: §VI · p. 15
  • Testing benchmarks should move to repository-level contexts, cross-file dependencies, dynamic execution, and native corpora for languages such as Rust and Go.Source: §VI · p. 15
  • Maintenance benchmarks should cover evolution-aware repair, dependency and configuration breakages, and log-driven troubleshooting beyond simple bug fixing.Source: §VI · p. 15
  • Future evaluations should measure human-machine collaboration efficiency and privacy-preservation risks alongside autonomous model capabilities.Source: §VI · p. 15
  • Future evaluations should replace single-turn question-answering with agent-based assessments of environmental awareness, tool use, execution, feedback interpretation, and iterative debugging.Source: §VI · p. 16
  • Benchmarks should expand native support for modern languages such as Rust and Go rather than relying only on translated tasks from Python-derived benchmarks.Source: §VI · p. 16

VII. THREATS TO VALIDITY

  • The study identifies rapid model evolution and its SDLC-only scope as limits on the durability and generalizability of its findings and taxonomies.Its review used 20 keyword combinations across four databases and Google Scholar, plus forward and backward snowball sampling.Source: §VII · p. 16

VIII. CONCLUSION

  • Approximately 61% of benchmarks focus on implementation, while requirements engineering and design together account for less than 8%.The review proposes research directions to address these deficiencies and align evaluation with real-world software engineering.Source: §VIII · p. 16
Loading 2505.05283v3…