PaperlayerView exact revision on arXiv

Source-linked AI summary

FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation

Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, Scarlett Li

arXiv:2503.06680v2cs.SEcs.CL

TL;DR

Existing benchmarks lack a dedicated framework for evaluating new-feature implementation across repository-level codebases. FEA-Bench constructs such a benchmark from filtered pull requests and paired unit tests, and results show that current LLMs face substantial difficulty on this task.

  • Problem

    Existing benchmarks do not adequately evaluate repository-level incremental development, where implementing features requires changes across interconnected code.

  • Method

    FEA-Bench uses filtered pull requests from 83 repositories to create feature-implementation tasks paired with relevant unit tests.

  • Results

    The best-performing LLM resolves only about 10% of FEA-Bench task instances, demonstrating poor performance on repository-level incremental development.

  • Takeaways & Limitations

    FEA-Bench provides an execution-based resource for studying LLM reasoning and generation challenges in real-world repository-level feature implementation.

  • Takeaways & Limitations

    The benchmark includes only Python repositories, and experiments use a single generation round similar to Pass@1.

Abstract

from arXiv · show

Implementing new features in repository-level codebases is a crucial application of code generation models. However, current benchmarks lack a dedicated evaluation framework for this capability. To fill this gap, we introduce FEA-Bench, a benchmark designed to assess the ability of large language models (LLMs) to perform incremental development within code repositories. We collect pull requests from 83 GitHub repositories and use rule-based and intent-based filtering to construct task instances focused on new feature development. Each task instance containing code changes is paired with relevant unit test files to ensure that the solution can be verified. The feature implementation requires LLMs to simultaneously possess code completion capabilities for new components and code editing abilities for other relevant parts in the code repository, providing a more comprehensive evaluation method of LLMs' automated software engineering capabilities. Experimental results show that LLMs perform significantly worse in the FEA-Bench, highlighting considerable challenges in such repository-level incremental code development.

1 Introduction

Existing benchmarks largely overlook the interconnected edits and new-feature work required in repository-level development. FEA-Bench addresses this gap with feature-focused tasks from real repositories and finds that current LLMs perform poorly.

  • Benchmark gap: Standalone programming and localized code-completion benchmarks provide limited insight into development across interconnected repository files.Repository-level changes often require corresponding edits beyond the initially targeted code location.
  • Task motivation: New feature implementation can require adding new functions or entire files, making incremental repository development a central software-engineering task.The paper defines this ongoing addition of components as repository-level incremental code development.
  • Benchmark construction: FEA-Bench contains 1,401 feature-implementation tasks paired with unit tests, sourced from 83 diverse GitHub repositories.The dataset is derived from pull requests focused on adding new components.
  • Results: About 10% of task instances are successfully resolved by the best-performing LLM, DeepSeek-R1, indicating substantial difficulty.The result is reported using execution-based metrics.
  • Benchmark construction: The benchmark evaluates repository-level incremental development through execution-based assessment of feature implementation tasks.Its construction uses parsing and other filtering methods to identify suitable tasks.

2 Related Work

Prior code-generation benchmarks mainly evaluate standalone programs or localized completion, leaving broader repository-level development insufficiently covered. Existing repository benchmarks include code completion and issue repair, while FEA-Bench targets incremental feature development.

  • Standalone and bounded code generation: HumanEval, MBPP, and related benchmarks primarily evaluate standalone functions, scripts, or increasingly complex but bounded programming scenarios.These evaluations do not fully represent real-world repository-level development challenges.
  • Repository-level completion: Repository-level code-completion benchmarks evaluate generating snippets at specified locations but can overlook broader edits and suffer from future-context leakage.Removing a single line or function body can expose context that would not be available during genuine development.
  • Issue repair versus feature development: SWE-bench focuses on repairing repository issues, whereas FEA-Bench evaluates incremental development that modifies and integrates new features.The distinction separates bug fixing from adding new components across a repository.

3 Benchmark Construction

FEA-Bench is constructed from GitHub pull requests to evaluate repository-level new-feature implementation with executable unit-test verification. Filtering identifies feature-oriented changes, producing 1,401 tasks from 83 repositories whose solutions require both new-component generation and complementary repository edits.

  • 3.1 Overview: Each task instance provides feature-request content, new-component definitions, environment setup, patches, and unit tests for repository-level evaluation.New components include signatures and documentation; patches separate test changes from software changes, and tests verify correctness after applying the gold patch.
  • 3.2 Repository Collection: The collection pipeline selects GitHub repositories, gathers pull requests, filters feature-oriented candidates, and retains instances with reproducible test results.The pipeline is designed to support diverse, executable evaluation and public updating of the benchmark.
  • 3.3 Task Collection and Filtering: 83 repositories yield 1,401 task instances after collection and filtering for new-feature implementation.The filtering process retains pull requests with new components, sufficient new-component edits, and a GPT-4o new-feature classification.
  • 3.4 Benchmark Characteristics: FEA-Bench tasks average 128.5 modified lines, including 87.1 lines in new components and about 41.4 lines elsewhere in the repository.This structure evaluates complete feature implementation rather than isolated generation of specified components.
  • 3.4 Benchmark Characteristics: New-component lines average more than 8× those in SWE-bench and comprise approximately 67.8% of FEA-Bench edits versus 28.9% in SWE-bench.FEA-Bench therefore emphasizes new-component-driven development, whereas SWE-bench mainly involves editing existing code.
  • 3.4 Benchmark Characteristics: FEA-Bench solutions are more complex than SWE-bench solutions when measured by edited lines or edited files.The comparison characterizes FEA-Bench as a more complex repository-level feature-development benchmark.

4 Experimental Design

The experiments vary model types, prompt context, new-component hints, retrieval strategies, and edit-output formats to examine LLM feature implementation under repository-level conditions. Evaluation uses single-round generation and execution-based resolution, with prompts designed around known feature and component information.

  • Model Evaluation: The evaluation includes code-specialized and general-purpose LLMs to assess capabilities in repository-level incremental development.The tested code LLMs include CodeLlama, Codestral4, Qwen2.5-Coder, and DeepSeek-Coder-V2, while general-purpose models include GPT-4 and GPT-4o.
  • New Component Hints: New-component hints have Brief and Detailed settings, providing either signatures alone or signatures plus documentation and non-Python patch changes.These hints are derived from newly extracted functions and classes and from non-Python files changed in the patch.
  • Evaluation Protocol: Task instances are evaluated using single-round outputs, and a task is resolved only when all unit tests pass.The lite benchmark’s BM25 prompts are limited to 27K tokens so generation remains within a 32K context window for most tested models.
  • Retrieval Method: Retrieval compares Oracle context containing patch-involved files with BM25 retrieval of relevant repository files, while always including the README and new-component files.BM25 ranks files by relevance to the pull-request content until the context limit is reached.
  • Output Format: The output-format comparison uses Natural before-and-after edit snippets or direct Patch-format edits.Natural edits can be converted into repository patches through post-processing, whereas Patch generation includes line numbers for fragment locations.

5 Evaluation Results

LLMs perform poorly on FEA-Bench’s repository-level feature-development tasks, while context design affects results. Detailed hints generally help, Oracle retrieval usually outperforms BM25, and Natural edit generation substantially outperforms Patch generation.

  • Overall Performance: The best resolved ratio is 9.92% under Oracle and Detailed prompt settings, indicating poor performance on repository-level incremental development.The evaluation reports this result across single-round execution-based task resolution.
  • Overall Performance: Larger models generally perform better, while Qwen2.5-Coder is comparable to GPT-4 and newer DeepSeek models outperform GPT-4 and o1 series models.The results also indicate that strong foundational capabilities can bring general-purpose models close to specialized code LLMs.
  • Context Settings: Detailed new-component hints generally improve performance, but Brief signature-only hints perform better on FEA-Bench lite.The discrepancy is associated with how component information and documentation are structured in the prompt.
  • Context Settings: Oracle retrieval generally outperforms BM25, although the difference is not substantial.For single-file instances, BM25 can outperform Oracle because retrieval adds information beyond the known new-component file.

6 Discussion

The discussion shows that retrieval, edit format, model choice, repository category, and feature complexity all affect repository-level feature implementation. Longer contexts improve recall slightly but reduce performance, while simpler edit formats and agentic editing help expose remaining challenges.

  • 6.1 Retrieved Files in Context: Increasing the prompt limit from 27K to 40K tokens slightly improves recall but decreases model performance.The results suggest that simply adding more retrieved files does not help models use repository context effectively.
  • 6.2 Output Format of Edits: Natural generation substantially outperforms direct Patch generation because patch formatting is harder for current LLMs to follow accurately.Natural-mode edits also achieve higher application success rates, which correlates positively with resolved ratios.
  • 6.3 Evaluation of Agent Frameworks: Agentless improves resolved ratios over BM25 for most models, but Agentless and Agentless-lite show no significant advantage over Oracle retrieval.The gains correlate strongly with higher code-edit application success rates.
  • 6.4 Repository Categories: Testing repositories have the highest resolved ratios, while DeepSeek-R1 underperforms GPT-4o and o1 there but outperforms both in other categories.GPT-4o is slightly weaker overall, with a trend largely consistent with o1.
  • 6.5 Performances under Different Complexity of New Components: 18.96% resolved ratio occurs with one added function, falling to 8.24% with two and 5.47% with at least three.The authors associate more added functions with greater new-component complexity and lower implementation success.

7 Conclusion

FEA-Bench evaluates repository-level incremental code development for new-feature implementation and shows that current LLMs face significant challenges. The analysis identifies retrieval, output format, repository characteristics, and new-component complexity as factors affecting performance.

  • 7 Conclusion: FEA-Bench evaluates LLMs’ ability to implement new features by adding components to existing repositories.The benchmark targets repository-level incremental code development.
  • 7 Conclusion: Retrieval method, output format, repository characteristics, and new-component complexity all affect new-feature implementation performance.The authors connect these factors to the need for improved reasoning and generation capabilities.

Limitations

The benchmark and experiments have scope and evaluation limitations. FEA-Bench contains only Python repositories, relies on scarce high-quality pull requests, and uses single-round generation with some missing model results.

  • Dataset limitations: FEA-Bench includes only Python repositories because they are easier to execute and generally use consistent testing frameworks.This creates a scope boundary for the benchmark’s scenarios.
  • Dataset limitations: High-quality pull requests for repository-level new-feature development are scarce, limiting the dataset’s usable coverage.Early repository developments often lack rigorous review or GitHub-based records, producing scenario limitations after filtering.
  • Evaluation limitations: The experiments use single-round generation because long-context evaluation is costly, which may bias the results.Some main-experiment results are also missing because API resources for DeepSeek-V3 and DeepSeek-R1 were limited.

Ethics Statement

The ethics statement describes publicly sourced repositories, research-use licensing, and limited use of LLMs during dataset construction. It also notes that benchmark inference outputs may contain potentially harmful code.

  • Data sources: The data comes from publicly available GitHub repositories whose licenses allow free software use.The collection was conducted for research purposes.
  • LLM use: LLMs are used only for classification during dataset construction, so the dataset itself is not intended to create harmful information.The statement distinguishes dataset construction from benchmark inference outputs.
  • Potential risks: Benchmark inference results may contain code that is harmful to computer systems.This risk concerns generated task-instance outputs rather than the dataset-construction process.
  • AI assistance: ChatGPT was used as an AI assistant for refining the paper’s writing.The stated uses include writing refinement, debugging, and translation capabilities.

A Dataset Details

FEA-Bench task instances are collected from GitHub repositories through staged filtering, validation, and prompt construction. The dataset emphasizes new-feature implementation while constraining task complexity and supporting executable evaluation.

  • Task construction: The collection pipeline filters merged pull requests for test files, new components, and other conditions required for valid task instances.Unit-test files are identified as pytest-executable files, and the resulting test patch is applied to the base commit to verify evaluation.
  • Repository collection: FEA-Bench includes task instances from 83 Python packages corresponding to GitHub repositories.
  • Subset design: The lite subset further excludes changes exceeding 4,096 tokens, containing new classes, or adding more than ten functions, while a verified subset is planned with professional software engineers.
  • Filtering criteria: Low-quality instances are excluded when feature descriptions are under 40 words, contain cascading issues, commit SHA-256 references, or images.
  • Filtering criteria: Task difficulty is limited by excluding instances that delete code files, involve more than three code files, or contain over 10 code-change hunks.
  • Prompt construction: Each prompt combines pull-request and issue content with repository context, and models can generate edits in Natural or Patch format using explicit instructions and examples.

B.2 Generation Configurations

Generation configurations vary by model size and access method, while inference uses one generation per task and converts outputs into patches for Docker-based evaluation.

  • Inference hardware: Models with fewer than 32 billion parameters use vLLM on an 8-GPU NVIDIA A100 workstation with tensor parallelism and a 4,096-token generation limit.
  • Model access: Larger open-source and closed-source models, including DeepSeek-V3, R1, and OpenAI models, are accessed through APIs.
  • Model versions: The OpenAI configurations include gpt-4-turbo-2024-04-09, gpt-4o-2024-05-13, and o1-mini-2024-09-12.
  • Sampling settings: Temperature and top-p are fixed at 0.2 and 0.95 when possible, while output-token limits vary by model family.DeepSeek-V3 and R1 use 8,192 tokens; o1 and o1-mini use 100,000 and 65,536 tokens, respectively.
  • Evaluation: Each task receives one model generation, whose output is converted into a patch and evaluated with Docker-adapted SWE-bench scripts.
Loading 2503.06680v2…