PaperlayerView exact revision on arXiv

Source-linked AI summary

FeatBench: Towards More Realistic Evaluation of Feature-level Code Generation

Haorui Chen, Chengze Li, Jia Li

arXiv:2509.22237v2cs.CLcs.AIcs.SE

TL;DR

Existing feature-level benchmarks do not fully reflect realistic development, and evaluating feature implementation from natural-language requirements remains challenging. FeatBench evaluates this setting across diverse repositories; state-of-the-art agents resolve only 29.94% of tasks, and the analysis identifies prevalent scope creep. The benchmark currently focuses primarily on Python repositories.

  • Problem

    Existing feature-level benchmarks often fall short of reflecting realistic software-development scenarios.

    Source: §1 · p. 2

  • Method

    FeatBench evaluates feature implementation from natural-language requirements across diverse repositories.

    Source: §7 · p. 17

  • Results

    Across 27 diverse repositories, state-of-the-art agents achieve a resolved rate of only 29.94%, and the analysis finds prevalent scope creep.

    Source: §7 · p. 17

  • Takeaways & Limitations

    The analysis identifies aggressive implementation and scope creep as prevalent behaviors in feature-coding agents.

    Source: §7 · p. 17

    FeatBench currently focuses primarily on Python repositories.

    Source: §6.2 · p. 17

Abstract

from arXiv · show

Evaluating Large Language Models (LLMs) on repository-level feature implementation is a critical frontier in software engineering. However, establishing a benchmark that faithfully mirrors realistic development scenarios remains a significant challenge. Existing feature-level benchmarks generally suffer from two primary limitations: unrealistic task inputs enriched with code hints and significant data leakage risks due to their static nature. To address these limitations, we propose a new benchmark - FeatBench, which introduces the following advances: (1) Realistic Task Inputs. Task inputs consist solely of natural language requirements, strictly devoid of code hints (e.g., function signatures). This format mirrors realistic software development by requiring agents to independently bridge the gap between abstract user intent and concrete code changes. (2) Evolving Data. FeatBench employs a fully automated pipeline to construct new benchmark versions from the latest repositories, effectively mitigating data contamination. The initial release comprises 157 tasks sourced from 27 actively maintained repositories. We evaluate two state-of-the-art agent frameworks with four leading LLMs on FeatBench. The results reveal that FeatBench poses a significant challenge, with the highest resolved rate reaching only 29.94%. Crucially, our analysis uncovers a prevalent behavioral pattern of aggressive implementation, which leads to "scope creep" and widespread regressions where agents break existing features by diverging from the user's explicit intent. We release FeatBench, our automated pipeline, and all experimental results to facilitate further community research.

1 Introduction

Existing feature-level benchmarks often diverge from realistic development, while FeatBench uses natural-language requirements and evolving repository data. Experiments find low resolution rates and frequent regressions associated with agents extending work beyond user intent.

1 Introduction

  • FeatBench tasks use only natural-language requirements, requiring agents to interpret repository context and plan implementation without code hints.The benchmark is designed to update periodically using an automated pipeline that constructs versions from the latest repositories.Source: §1 · p. 2
  • The initial release contains 157 tasks from 27 actively maintained open-source repositories and spans domains including AI/ML, DevOps, and Web.Human evaluation found 93.3% of sampled tasks had comprehensive and unambiguous requirements.Source: §1 · pp. 2–3
  • Repository selection requires at least three formal releases and identifiable test suites, while eligible pull requests modify existing functions without adding or deleting functions.Tasks must include at least one Fail-to-Pass test, and the PR constraint is intended to keep developer-written tests valid for evaluation.Source: §1 · p. 2
  • Each task averages 1,694.6 tests: 2.2 Fail-to-Pass tests and 1,692.4 Pass-to-Pass tests derived from existing repository suites.The test suite checks both new-feature correctness and backward compatibility.Source: §1 · p. 3
  • 29.94% was the resolved rate of the top-performing configuration, Trae-agent + GPT-5; regressive implementation was the predominant failure reason, linked to aggressive scope creep.The scope creep involved proactive refactoring or feature extensions beyond explicit user intent.Source: §1 · p. 3
  • Autonomous planning-based agents substantially outperform rigid pipeline-based counterparts on FeatBench.The evaluation covers two agent frameworks and four leading LLMs.Source: §1 · p. 3
  • Resolved rates reach 60–70% for smaller repositories but fall to 10–30% above 800 files or 300k LOC; rates also collapse for modifications spanning over 5 files or 50 LOC.The authors report consistent performance across task-creation periods as evidence against data leakage.Source: §1 · p. 3

2 Related Work

Code-generation benchmarks have expanded from function- and class-level tasks to repository and feature implementation, but prior task setups often provide code hints. FeatBench targets feature implementation from natural-language requirements, while the behavior of autonomous agents on such tasks remains underexplored.

Code Generation Benchmarks

  • Research benchmarks progressed from isolated code snippets, including HumanEval [7] and MBPP [4], to class-level benchmarks such as ClassEval [11] and CoderEval [31], and repository-level generation with DevEval [16] and EvoCodeBench [15].Source: §2 · p. 4
  • SWE-bench [13] targets issue resolution and includes approximately 18–22% feature-implementation scenarios, but lacks a dedicated focus on implementing new features from scratch; FEA-Bench [17] and NoCodeBench [10] evaluate feature implementation.FEA-Bench explicitly provides signatures and documentation of new components.Source: §2 · p. 4
  • FeatBench distinguishes itself by evaluating feature implementation from realistic natural-language requirements without code hints, requiring agents to identify injection points autonomously.This formulation contrasts with feature benchmarks that provide component signatures or use documentation updates as specifications.Source: §2 · p. 4

Base Models and Software Engineering Agents

  • Prior comparisons of pipeline-based and autonomous agents primarily address bug-fixing rates; their behavior on feature tasks, especially scope creep, has received insufficient study.Pipeline agents use fixed localization-and-repair workflows, whereas autonomous agents plan dynamically and use tools.Source: §2 · pp. 4–5

3 FeatBench

This section describes FeatBench’s evaluation of repository-level feature implementation from natural-language requirements without code hints, alongside its automated task-construction pipeline. The pipeline configures PR-era test environments and validates feature and regression tests across repositories, including multi-file tasks with extensive test suites.

3.1 Task Definition

  • Tasks use abstract natural-language requirements without code hints, while agents receive a repository snapshot and runtime environment but not the F2P and P2P evaluation tests.F2P tests verify the feature; P2P tests check backward compatibility.Source: §3.1 · pp. 5–6

3.2 Features of FeatBench

  • FeatBench’s requirements average 1,848 characters, and its automated pipeline refreshes the benchmark every six months from recent repositories to mitigate data contamination.The paper reports a processing cost of approximately $0.28 per sample using DeepSeek V3.1.Source: §3.2 · p. 6
  • Curation filters repositories for active maintenance, formal releases, test suites, and production relevance, then filters pull requests for Python changes, tests, and modifications to existing functions.The initial release used a June 2024 cutoff; tasks that add or delete functions are excluded to preserve testability.Source: §3.2 · p. 6
  • Each task averages 1,694.6 test cases spanning feature verification and system stability, and the benchmark covers diverse application domains.The test coverage is intended to detect regressions as well as check intended functionality.Source: §3.2 · p. 6

3.3 Benchmark Construction Pipeline

  • The construction pipeline has data-curation, environment-configuration, and test-validation stages; it reverse-engineers natural-language requirements from PRs and code changes, then uses an agent to set up the base environment.Human solvability and information-completeness checks are described as quality assurance for synthesized requirements.Source: §3.3 · pp. 6–8
  • The initial pipeline release yielded 157 tasks after selecting 27 repositories from 44 candidates and validating feature samples with at least one F2P test.The curation process produced 297 candidate samples before final validation.Source: §3.3 · pp. 7–8
  • The environment pipeline detects the PR’s Python version, configures a matching container, and constrains package versions to those available when the PR was created.The configuration agent must install dependencies and make the PR’s test suite executable.Source: §3.3 · pp. 8–9
  • For F2P validation, AST analysis identifies tests added or modified by the test patch, which must fail before and pass after the feature patch.A PR is valid only if at least one identified test transitions from FAILED or ERROR to passing.Source: §3.3 · p. 9
  • For P2P validation, tests that pass on the pre-patch repository are rerun after the feature patch to check that they still pass.Source: §3.3 · p. 9

3.4 Dataset Statistics

  • 157 task instances span 27 real-world open-source repositories averaging 209k lines of code and 864 files.These repository sizes demand repository-level context understanding and file localization.Source: Table 3 · p. 9, §3.4 · p. 9
  • Ground-truth patches modify an average of 2.4 files and 5.1 code hunks; each task averages 2.2 F2P and 1,692.4 P2P tests.F2P tests verify new functionality, while P2P tests detect potential regressions.Source: §3.4 · p. 9

4 Experimental Setup

The experiments examine agent frameworks, factors influencing resolution, and failure patterns on realistic feature tasks. They compare four LLMs across two agent frameworks using resolution, patch, localization, test, and cost measures.

4.1 Research Questions

  • RQ1–RQ3 ask how agent frameworks compare, which factors influence resolution, and how failure patterns affect implementation quality and reliability.Source: §4.1 · p. 10

4.2 Model Selection

  • Four state-of-the-art LLMs, including DeepSeek V3.1, GPT-5, Doubao-Seed-1.6, and Qwen3-Coder-Flash, were selected to evaluate current capability limits.The models include one open-source and three proprietary systems; all use temperature 0.0 and top-p 1.0.Source: §4.2 · p. 10

4.3 Agent Frameworks

  • Agentless uses a two-stage localization-and-patching pipeline, while Trae-agent plans dynamically with a ReAct loop; Agentless produces one candidate per task.The evaluation omits Agentless regression-test reranking because it would require substantial infrastructure adaptation, and tasks are capped at 150 steps with supplementary tools.Source: §4.3 · p. 10

4.4 Evaluation Metrics

  • Resolved Rate is the primary metric, alongside patch-apply and file-localization success, feature-validation and regression-test pass rates, and token cost.Source: §4.4 · pp. 10–11

5 Results

The reported evaluation characterizes FeatBench as a significant hurdle for the state-of-the-art configuration.

5.1 RQ1: Performance Evaluation on FeatBench

  • Table 4 reports performance results for all agent-model configurations evaluated on FeatBench.Source: §5.1 · p. 11
  • FeatBench poses a significant hurdle even for the state-of-the-art configuration evaluated.Source: §5.1 · p. 11

the advanced GPT-5 model, the autonomous Trae-agent

This section examines agent performance on FeatBench and how performance varies with planning approach, token use, and repository and patch complexity. It also analyzes aggressive implementation as a source of regressions and, at times, more robust and general designs, motivating control that limits scope creep while preserving its benefits.

the advanced GPT-5 model, the autonomous Trae-agent

  • Trae-agent’s resolved rate rises with token consumption, whereas Agentless remains below 0.1M tokens but resolves only 7.00%–16.56%.The reported Trae-agent token range is 1.07M to 2.90M.Source: Figure 7 · p. 11, the advanced GPT-5 model, the… · p. 11
  • Trae-agent averages 41.72% FV% and 46.18% RT%, versus Agentless at 21.66% FV% and 29.94% RT%, and reaches 76.42% File% versus 48.90%.The authors attribute Trae-agent’s stronger localization to dynamic planning and its ability to identify modification targets in large repositories.Source: the advanced GPT-5 model, the… · p. 11
  • All evaluated agents show low RT% and a propensity to introduce regressions, posing a reliability bottleneck for production deployment.The paper says these regressions undermine the stability required in production environments.Source: the advanced GPT-5 model, the… · p. 12
  • 29.94% was the maximum Resolved% across evaluated configurations, underscoring FeatBench’s difficulty; autonomous planning agents outperform rigid pipeline-based agents but consume more tokens.Trae-agent’s higher resolved rates require 1.07M–2.90M tokens, while Agentless stays under 0.1M tokens and resolves 7.00%–16.56%.Source: the advanced GPT-5 model, the… · pp. 11–12

5.2 RQ2: Factors Influencing Resolved Rate

  • Trae-agent with Doubao-Seed-1.6 maintained a consistent resolved rate across five task-creation periods, which the authors interpret as evidence against data leakage.Patch volumes fluctuated across periods 2308 to 2509, but the resolved rate remained stable.Source: Figure 9 · p. 12, §5.2 · p. 13
  • 36% was the top resolved rate for single-file patches of 1–30 LOC, while success fell nearly to zero for patches exceeding 50 LOC or spanning five or more files.Performance is strongest for small, localized changes and degrades as patch size and distribution increase.Source: Figure 10 · p. 13, §5.2 · p. 13

5.3 RQ3: Case Study

  • Most analyzed Trae-agent failures were regressions, often driven by scope creep beyond explicit requirements; agents can also produce more reusable, scalable designs than human-written patches.The study groups failures as misunderstood requirements, incomplete implementation, or regressive implementation; its Arrow-loading example contrasts a localized gold-patch fix with a reusable agent helper.Source: Figure 11 · p. 14, §5.3 · p. 14
  • In failure cases, implementations extend beyond the original scope and introduce errors, while success cases show the same behavior enabling high-level architectural optimization.Source: §5.3 · p. 14
  • Because agents infer broader architectural needs beyond stated requirements, controlling implementation aggressiveness is needed to enable robust improvements while preventing defect-inducing scope creep.Source: §5.3 · p. 14
  • Aggressive implementation drives defects through unnecessary modifications but can enable architectural designs more robust and general than human-written patches.The RQ3 summary identifies regressive implementation, often driven by this behavior, as the predominant cause of failure.Source: §5.3 · p. 15

6 Discussion

Section 6 reports a human review of synthesized requirements, then discusses threats involving requirement fidelity, test-based evaluation, and FeatBench’s scope.

6.1 Solvability Validation of Synthesized Natural Language Requirements

  • Reviewers randomly sampled 30 tasks (about 19%) and assessed whether prompts supplied enough context to reproduce the gold functionality.The two evaluators were computer-science researchers from the author team with over two years of Python development experience.Source: §6.1 · p. 15
  • 28 (93.3%) of 30 sampled tasks scored 2; the other two scored 1, none scored 0, and mean completeness was 1.93/2.Source: §6.1 · p. 16

6.2 Threats to Validity

  • LLM-reversed requirements risk hallucination or ambiguity; reconstructive prompting, bidirectional checks, and human verification are used to align them with the ground-truth implementation.Source: §6.2 · p. 16
  • Repository-authored tests may still permit false positives when sparse; FeatBench also requires candidates to pass functional tests and a large regression suite.Source: §6.2 · p. 16
  • FeatBench currently focuses primarily on Python, so findings may not fully generalize to statically typed languages such as Java or C++; its construction method is language-agnostic.Source: §6.2 · p. 17
  • To mitigate data contamination, FeatBench continuously sources tasks from the latest repository updates so evaluation data remains unseen during model training.Source: §6.2 · p. 17

7 Conclusion and Future Work

FeatBench targets realism gaps in existing feature-level benchmarks with natural-language-only tasks and an evolving repository-based pipeline. Across its evaluation, agents resolve at most 29.94% of tasks, while aggressive implementation is associated with scope creep and regressions.

7 Conclusion and Future Work

  • Existing feature-level benchmarks use unrealistic task inputs and static datasets, limiting alignment with realistic development scenarios.The cited limitations include implementation hints such as function signatures and data-leakage risks associated with static benchmarks.Source: §7 · p. 17
  • FeatBench evaluates feature implementation from natural-language requirements using an automated pipeline that draws tasks from the latest open-source repositories.The evaluation covered 157 tasks across 27 diverse repositories.Source: §7 · p. 17
  • 29.94% was the maximum resolved rate reported for the evaluated agents, and the analysis found scope creep and regressions associated with aggressive implementation.Agents sometimes refactored code or extended features beyond explicit user intent.Source: §7 · p. 17
  • The authors plan to extend FeatBench beyond Python to languages such as Java and Go and explore ways to align agent behavior with realistic engineering practices.They aim to balance specification adherence with robust software design.Source: §7 · p. 17

A.1 Full Repositories List

The appendix lists repositories used in FeatBench and records each repository’s license, instance count, file count, and lines of code.

A.1 Full Repositories List

  • The repository inventory records each project’s license, number of instances, number of files, and lines of code.For example, conan-io/conan is listed with 56 instances, 1,056 files, and 162.5k LoC.Source: §A.1 · p. 19
  • The inventory includes projects such as huggingface/datasets, python-attrs/attrs, and datamodel-code-generator.Their listed instance counts are 6, 2, and 8, respectively.Source: §A.1 · p. 19
  • It also includes repositories identified in the table as Database, Science, and Cloud projects: pydata/xarray, PyBaMM, and cfn-lint.The listed repositories are pydata/xarray, pybamm-team/PyBaMM, and aws-cloudformation/cfn-lint.Source: §A.1 · p. 19

A.2 Prompts in FeatBench

This appendix documents prompts for identifying and rewriting feature-related changes as user requirements, determining repository setup needs, and configuring the runtime environment. It also covers installing dependencies, running specified tests, and distinguishing dependency failures from pytest framework problems.

Prompt for Feature-implementation PRs Identification

  • The PR-identification prompt classifies release-note changes as features, improvements, bug fixes, or other changes, extracts explicitly mentioned PR numbers, uses README context, and returns JSON.It also directs the prompt to ignore trivial version bumps unless they are part of larger features.Source: §A.2 · p. 20

Prompt for Uesr Requirement Synthesization

  • The synthesis prompt turns PR information and code changes into a complete natural-language “I want to” request that preserves requirements and workflow while excluding implementation code.It asks for the user’s perspective, including parameter changes as input or configuration needs, and focuses on desired behavior rather than implementation steps.Source: §A.2 · pp. 22–23

Prompt for Relevant Files Identification & Python Version Detection

  • The repository-analysis prompt identifies up to 20 development-relevant files and determines the project’s Python version, saving each result to its prescribed root-level file.The version file must contain only a version number, with a specified default used if no version requirement is found.Source: §A.2 · p. 24

Prompt for Configuration Agent

  • The configuration prompt requires reading the setup-file list before analyzing project dependencies and verifying tests with itemized pass/fail output, while allowing a few failures.It also instructs the agent to use --tb=short to limit traceback output.Source: §A.2 · p. 26

A.2 Prompts in FeatBench

  • Agents must attempt to install pytest-timeout and pytest-xdist, run pytest with --timeout=5 and -n auto, and test the specified files.For the specified test files, the prompt says to make every effort to install optional dependencies when imports fail; other test files may tolerate ImportError or ModuleNotFound.Source: §A.2 · p. 26
  • Agents should use the preconfigured Python version, avoid changing versions, check the target environment, and install dependencies into system Python rather than a virtual environment.The prompt also advises checking which packages are already installed to avoid conflicts.Source: §A.2 · pp. 26–28
  • Prefer UV or poetry for dependency setup, add --exclude-newer {created_time} when using UV, and fall back to system pip3 if UV fails.If pip leads to dependency version conflicts, the prompt says to resolve them manually according to the project timeline.Source: §A.2 · p. 27
  • When pytest reports collection, missing-test, skip, or internal framework errors, agents should stop trying to fix those test files and continue environment configuration.The prompt says such errors may mean tests are absent or invalid, or reflect pytest framework issues.Source: §A.2 · p. 28

A.3 Detailed Data Instance Example

The example instance shows a user-centric feature request alongside its repository environment and hidden evaluation ground truth. The ground truth includes fail-to-pass and pass-to-pass tests and a gold patch.

A.3 Detailed Data Instance Example

  • The example instance is titled as a detailed FeatBench task breakdown, identified as instructlab__instructlab-2428.The task table is the example for that instance.Source: Table 6 · p. 29

1. Task Input (User-Centric)

  • The user requests GGUF-model validation for MMLU and MMLU Branch, with a clear error and immediate exit while preserving support for other evaluation types.The request specifies checking both the main model and base model parameters for MMLU Branch evaluation.Source: §1 · p. 29

2. Running Environment

  • The running environment identifies the instructlab/instructlab repository, base commit ce87fdeb51cd3637121bde1ae0f9926fcd183c7f, and a Docker image.The image is featbench:instructlab__instructlab-2428.Source: §2 · p. 29, §1 · p. 29

3. Evaluation Ground Truth (Hidden from Agent)

  • Hidden evaluation ground truth includes a fail-to-pass test, existing pass-to-pass tests, and a gold patch that calls model validation with allow_gguf=False.The named fail-to-pass test is tests/test_lab_evaluate.py::test_invalid_gguf_model_mmlu; the pass-to-pass list includes 68 more existing tests.Source: §3 · p. 29

Related papers

Loading 2509.22237v2…