Source-linked AI summary
FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
Qixing Zhou, Jiacheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, Dandan Tu, Zhaoxiang Zhang
TL;DR
Existing agentic coding benchmarks have limited task scope and often lack executable, continually updateable evaluation. FeatureBench introduces a test-driven, execution-based benchmark for feature-oriented development, and its agents solve only a small fraction of the resulting tasks, while failures expose persistent cross-file reasoning limitations.
Problem
Existing agentic coding benchmarks often focus on bug-level scenarios and lack automated, executable evaluation that can scale to feature-level development.
Method
FeatureBench traces F2P and P2P tests through an object dependency graph to extract feature tasks and generate verified executable environments automatically.
Results
12.5% of task cases are successfully completed by GPT-5.1-Codex with medium reasoning using the Codex agent framework.
Takeaways & Limitations
FeatureBench provides a scalable and verifiable framework for evaluating and developing next-generation agentic coding models.
Takeaways & Limitations
Current LLMs struggle with cross-file dependency resolution, often leaving unresolved symbols when features span multiple files.
Abstract
from arXiv · showhide
Agents powered by large language models (LLMs) are increasingly adopted in the software industry, contributing code as collaborators or even autonomous developers. As their presence grows, it becomes important to assess the current boundaries of their coding abilities. Existing agentic coding benchmarks, however, cover a limited task scope, e.g., bug fixing within a single pull request (PR), and often rely on non-executable evaluations or lack an automated approach for continually updating the evaluation coverage. To address such issues, we propose FeatureBench, a benchmark designed to evaluate agentic coding performance in end-to-end, feature-oriented software development. FeatureBench incorporates an execution-based evaluation protocol and a scalable test-driven method that automatically derives tasks from code repositories with minimal human effort. By tracing from unit tests along a dependency graph, our approach can identify feature-level coding tasks spanning multiple commits and PRs scattered across the development timeline, while ensuring the proper functioning of other features after the separation. Using this framework, we curated 200 challenging evaluation tasks and 3825 executable environments from 24 open-source repositories in the first version of our benchmark. Empirical evaluation reveals that the state-of-the-art agentic model, such as Claude 4.5 Opus, which achieves a 74.4% resolved rate on SWE-bench, succeeds on only 11.0% of tasks, opening new opportunities for advancing agentic coding. Moreover, benefiting from our automated task collection toolkit, FeatureBench can be easily scaled and updated over time to mitigate data leakage. The inherent verifiability of constructed environments also makes our method potentially valuable for agent training.
1 INTRODUCTION
FeatureBench addresses the limited scope and weak scalability of existing agentic coding benchmarks by evaluating realistic feature-level development with executable tests and automated collection. Its first version contains 200 tasks and 3825 executable environments, while state-of-the-art agents solve only a small fraction of tasks.
- Motivation: Existing benchmarks emphasize bug-level scenarios or handcrafted generation, leaving feature-level agentic coding insufficiently evaluated.These limitations motivate broader, automatically maintainable evaluation coverage.
- Results: 11.0% solution rate is achieved by Claude Opus 4.5 on FeatureBench.The benchmark evaluates whether agents can complete complex feature-oriented development tasks rather than only isolated fixes.
- Contributions: The benchmark combines feature-oriented tasks, reliable execution-based evaluation, scalable test-driven collection, and continual updatability.These contributions are intended to support broader assessment of current agentic coding capabilities.
- Scale: 200 evaluation tasks and 3825 executable environments were constructed from 24 open-source GitHub repositories.The collection pipeline supports continued updates and reduces dependence on human-written PR or commit trajectories.
- Benchmark: FeatureBench evaluates real-world feature development through an automated, execution-based evaluation pipeline.Tasks provide clear requirements and interfaces for implementing capabilities in existing repositories or standalone modules.
2 RELATED WORK
Prior agentic coding benchmarks improve task-oriented evaluation but remain constrained by bug-focused tasks, PR or commit-based collection, and incomplete guarantees of evaluation quality. FeatureBench targets feature-oriented tasks while retaining executable, verifiable environments.
- Existing benchmarks: SWE-bench is widely used but primarily focuses on bug fixing, with limited coverage of feature requests.Its verified subset has become a standard for assessing LLM-based coding agents.
- Collection pipelines: PR-based and commit-based benchmarks improve scalability but may provide limited guarantees of evaluation quality.Their task construction relies on pull requests, commits, synthesized tests, or back-translated code changes.
- FeatureBench: FeatureBench combines feature-oriented development with execution-based evaluation and a test-driven toolkit for automatic task generation.Its pipeline is designed to produce tasks that are both feature-oriented and reflective of real-world development.
- Collection pipelines: SWE-Flow uses fail-to-pass tests but neglects pass-to-pass tests and does not ensure other features remain functional.This can create discrepancies from actual development settings.
3 FEATUREBENCH
FeatureBench formulates callable feature-development tasks across incremental and from-scratch settings, then evaluates them with executable tests. Its test-driven pipeline traces dependencies to separate target functionality while preserving other features and generates scalable benchmark instances.
- Task formulation: Each task supplies a high-level description, functional interface, execution environment, and a requirement to produce a directly callable module.The interface includes invocation path, signatures, inputs, outputs, and annotations.
- Difficulty: FeatureBench defines Level 1 as extending an existing repository and Level 2 as implementing the same functionality from scratch.The two levels represent common software-development settings.
- Evaluation: Solutions are resolved only when they pass all fail-to-pass and pass-to-pass tests.The protocol also reports resolved rate and passed rate among its primary metrics.
- Test-driven extraction: Dynamic tracing constructs an object dependency graph from F2P and P2P test executions to identify target and supporting functions.The graph records function identities, source locations, dependencies, and whether functions were triggered by P2P tests.
- Verification and generation: The pipeline verifies that the undeveloped codebase passes P2P tests and fails F2P tests before generating task artifacts.It automatically produces a problem statement, undeveloped codebase, verified patch, and unit-test suite.
- Benchmark collection: 3825 coding environments from 24 Python repositories form the full set, while a randomly selected 30-instance lite set supports wider adoption.Full-set tasks exceed 100 lines of pending implementation and include at least 10 F2P test points.
4 EXPERIMENTS
Experiments show that current agents struggle with FeatureBench’s complex, feature-oriented tasks, while explicit interfaces, visible tests, and longer execution budgets improve performance. Failure analysis points to cross-file reasoning and incomplete information retrieval as key weaknesses.
- Baseline Performance: 12.5% of task cases are successfully completed by GPT-5.1-Codex with medium reasoning, while Claude Opus 4.5 configurations resolve at most 11.0% on the Full set.The benchmark evaluates seven scaffold+model configurations and finds that even the strongest settings solve only a small fraction of tasks.
- Baseline Performance: Below 50% passed rates remain substantially higher than resolved rates, while all evaluated LLMs consume over one million input tokens.The results indicate many solutions are partially plausible but remain far from fully solving the tasks; Lite and Full sets yield consistent model rankings.
- Comparison with SWE-bench: More complex FeatureBench tasks require editing more lines, files, and functions and passing more tests than SWE-bench tasks.FeatureBench also includes 16 repositories beyond the eight originally covered by SWE-bench.
- Comparison with SWE-bench: 5.2% of FeatureBench subset tasks are resolved by Claude Opus 4.5, compared with 74.40% on SWE-bench.The comparison uses repositories shared between the two benchmarks to provide a more aligned evaluation.
- Failure Cases Analysis: NameError failures dominate, reflecting difficulty resolving cross-file dependencies and maintaining coherent program context beyond one file.Models often make local edits without restoring required references, producing unresolved symbols and name-related failures.
- Ablation Study: Removing explicit interfaces lowers task success, whereas visible unit tests significantly increase task success and pass rates.These ablations support the importance of clear functional specifications and high-quality unit tests for agentic coding.
- Ablation Study: Task pass rates negatively correlate with code length but show minimal dependence on initial commit time.Normalized code-length and pass-rate trends fluctuate similarly, while future benchmark design should monitor increasing data-leakage risk.
- Ablation Study: Increasing maximum execution steps from 50 to 100 improves Gemini-3-Pro-Preview and Qwen3-Coder-480B-A35B-Instruct performance, with marginal gains beyond 100.The result comes from the Lite Set ablation on OpenHands.
5 CONCLUSION
FeatureBench evaluates LLM agents on realistic feature-oriented software development using test-driven task extraction and execution-based evaluation. Results show persistent challenges in planning, reasoning, and long-horizon task management, while its automated design supports diverse, scalable, verifiable evaluation.
- Conclusion: FeatureBench evaluates LLM-powered agents in realistic, feature-oriented software development scenarios.It uses test-driven task extraction and execution-based evaluation to expand task diversity, scalability, and verifiability.
- Conclusion: Current agentic systems face persistent challenges in planning, reasoning, and managing long-horizon tasks.The conclusion identifies these challenges as the central empirical finding.
- Conclusion: FeatureBench’s extensible and automated design provides a rigorous evaluation framework and a foundation for developing next-generation agentic coding models.This conclusion stays within the paper’s stated scope for the benchmark’s future value.
A DETAILED BENCHMARK COLLECTION
FeatureBench’s collection pipeline extracts feature-level tasks from repository tests and dependency graphs, then verifies that retained functionality still works. Most stages are automated after a brief environment-configuration step, and generated instances include standardized prompts and metadata.
- Environment Setup: After a manually prepared environment configuration file, the pipeline builds a Docker image and automates all subsequent operations.The configuration step is reported to take about three minutes and is the sole human-intervention stage.
- Patch Extraction: The pipeline constructs function-level dependency graphs from repository test files as the foundation for extracting feature-level patches.It collects viable pytest cases and executes them in a sandbox before selecting cases for extraction.
- Patch Extraction: An LLM classifies imported objects in fail-to-pass test files into test targets, dependencies, and utilities.The classified test targets become top-level objects and entry points for later extraction.
- Patch Extraction: Breadth-first traversal from top-level objects separates extracted code from remained nodes, with extraction capped at a randomly selected 3000–5000 lines.Pass-to-pass cases identify functionality that must remain available after the target feature is removed.
- Validation: Post-verification executes fail-to-pass and pass-to-pass tests on each modified codebase to check target behavior and retained functionality.The procedure expects fail-to-pass behavior below a predetermined parameter and complete passage of selected pass-to-pass cases.
- Instance Generation: Generated instances contain a task prompt, gold patch, test patch, and metadata file, while prompts combine task and interface descriptions through scripting.Missing interface documentation can be supplemented with LLM-generated docstrings.
B DETAILED BENCHMARKING PROCESS
FeatureBench evaluates agents through sequential inference and automated testing in resettable environments. It supports repository-visible and from-scratch settings, applies generated patches, runs pytest, and derives pass-rate and solved metrics while checking for unauthorized source access.
- Pipeline: The benchmarking pipeline consists of agent inference followed by automated evaluation.The two stages use initialized task environments and later reset the repository before testing the generated solution.
- Inference: In L1, the target feature and its fail-to-pass test file are removed; in L2, the entire repository is removed before agent execution.These setups distinguish repository-based implementation from from-scratch implementation.
- Inference: After receiving problem statement.md, the agent’s modifications are extracted as a patch for subsequent evaluation.The agent operates inside the initialized environment during inference.
- Evaluation: Evaluation reapplies the agent patch in the appropriate reset environment, restores the relevant tests, and runs pytest to produce a test report.L1 restores the fail-to-pass test file, whereas L2 installs the generated code before restoring the original repository and applying the test patch.
- Metrics: Pass rate is the fraction of executed fail-to-pass tests passed, while solved equals 1 only when pytest exits with 0 for both fail-to-pass and pass-to-pass tests.The solved metric is binary; otherwise it equals 0.
- Integrity Checks: The framework combines defensive prompting with automated execution-log inspection to detect attempts to access installed package source code.Regular-expression patterns flag commands that directly read files in the Python library directory.
C ANALYSIS OF FALURES OF GEMINI 3 PRO MODEL
Gemini 3 Pro’s poor benchmark performance is associated with repeated violations of tool-call JSON schemas. The observed errors include hallucinated parameter keys and are reported as evidence of difficulty with large-scale code editing.
- Observed Failure: Gemini 3 Pro frequently substituted semantically similar but syntactically invalid parameter keys during tool invocation.The analysis attributes this recurrent pattern to insufficient adherence to the required JSON schemas.
- Observed Failure: In one astropy case, Gemini 3 Pro passed path instead of the required file path argument to read file.The example indicates that the model understood the tool’s intent but failed to follow its API constraint.
- Interpretation: The reported tool-invocation issue occurred in the majority of evaluated cases and was interpreted as indicating deficiencies in large-scale code editing.The passage uses qualified language, stating that Gemini 3 Pro seems to have these deficiencies.
D COMPARISON WITH EXISTING BENCHMARKS
FeatureBench differs from existing benchmarks in task source, construction, validation, and implementation scope. It emphasizes realistic feature development, interface-driven specifications, preservation of existing behavior, and scalable generation across repositories.
- Scope of Comparison: FeatureBench compares with SWE-Dev and commit0 across task sources, construction pipelines, evaluation settings, and scalability.The comparisons are intended to clarify distinctions beyond the high-level related-work discussion.
- SWE-Dev: Unlike SWE-Dev’s unit-test and LLM-generated PRD formulation, FeatureBench uses a different approach to task specification, validation, and filtering.The passage identifies these pipeline differences as substantial.
- SWE-Dev: FeatureBench preserves existing repository features while leaving only the target feature unimplemented, using P2P filtering and post-verification to protect behavior.The comparison states that SWE-Dev lacks P2P verification and post-verification, permitting patches that may break existing behavior.
- commit0: Unlike commit0’s partial function-body completion, FeatureBench removes definitions, imports, and associated logic so the target feature is implemented from scratch.This design is presented as closer to real development workflows.
- commit0: FeatureBench is framed as supporting lower-cost scaling to new repositories than commit0.The supplied comparison identifies scalability as a distinction between the benchmarks.
E DATASET OVERVIEW AND EXPERIMENTAL RESULTS
FeatureBench provides a diverse, executable dataset for evaluating coding agents across feature-oriented tasks and reports repository-level performance using multiple metrics. The dataset includes 200 evaluation tasks from 3825 candidate environments across 24 Python repositories.
- Dataset Composition: 200 evaluation tasks were derived from 3825 candidate coding environments across 24 Python repositories.The repositories span machine learning, scientific computing, visualization, web frameworks, and software engineering utilities.
- Dataset Composition: The dataset covers diverse real-world coding scenarios across machine learning, scientific computing, visualization, web frameworks, and software engineering utilities.
- Dataset Structure: Repository statistics include extracted code-line averages and test-point counts, while a separate table summarizes repository information and licensing.
- Dataset Structure: Each task instance records an instance id, patches, test transitions, image name, repository metadata, base commit, problem statement, and repository settings.The field descriptions are provided in a dedicated dataset table.
- Experimental Results: Seven tables evaluate multiple language models using repository-level Passed, Resolved, and Token IO metrics.The tables cover Codex, Claude Code, OpenHands, Gemini CLI, and Qwen-based systems across the benchmark repositories.
Human Evaluation Guideline for Top-Level Tested Object Classification
The human evaluation guideline determines whether imported or referenced objects in Python tests are top-level tested objects rather than auxiliary components. It evaluates imports independently and begins by identifying the test file’s purpose.
- Classification Procedure: Each import statement is evaluated independently to determine whether it represents a top-level tested object or an auxiliary component.
- Classification Procedure: The classification procedure first requires understanding the purpose of the test file.
Expert Verification Guideline for Feature-Level Tasks (part 1 of 2)
Feature-level tasks undergo manual expert verification for structural correctness and implementability from the task prompt and remaining codebase. Reviewers assess whether the task description makes the required implementation clear without external documentation.
- Expert Verification: Each feature-level task is manually verified for structural correctness, including objects, imports, and masking.
- Expert Verification: Reviewers assess whether a competent engineer can implement the requested functionality using only the prompt and remaining codebase.
- Expert Verification: The task description is checked by asking whether it identifies what to implement given the codebase alone.