PaperlayerView exact revision on arXiv

Source-linked AI summary

Doc2Feat-Bench: Evaluating Documentation-Driven Feature Addition

Zhonghao Jiang, Le Deng, Jialun Cao, Michael Pradel, Zhongxin Liu

arXiv:2507.18130v4cs.SE

TL;DR

Existing benchmarks largely use issue reports, while documentation changes offer public-facing feature specifications for agent evaluation. Doc2Feat-Bench builds and validates tasks from those changes, but the best reported success rate is 37.72%, and analysis identifies several recurring agent challenges.

  • Problem

    Existing agent benchmarks rely primarily on issue reports, while documentation changes offer a public-facing source for evaluating implementation of intended feature behavior.

    Source: §1 · pp. 1–2

  • Method

    Doc2Feat-Bench pairs documentation changes with corresponding implementation and developer-written tests, and provides a manually validated subset for evaluation.

    Source: §1 · p. 2

  • Results

    37.72% is the best success rate, achieved by Qwen3-Coder-480B with OpenHands, compared with 69.60% for the same setup on SWE-bench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Documentation-driven feature addition remains a challenging, complementary benchmark setting, with cross-file editing, code-module understanding, and documentation interpretation among the identified challenges.

    Source: §1 · p. 2

    The benchmark focuses on well-maintained open-source Python projects with structured release notes, and evaluates only Agentless and OpenHands scaffolds.

    Source: §6.2 · p. 11

Abstract

from arXiv · show

Documentation changes in mature software projects often describe newly introduced or modified behavior. This makes them a natural basis for documentation-driven feature addition, where software engineering agents implement features from public-facing documentation updates. However, existing benchmarks for agentic software development primarily rely on issue reports written for maintainer coordination and often contain implementation-level details. Motivated by this observation, this work introduces Doc2Feat-bench, a benchmark for documentation-driven feature addition. Doc2Feat-bench consists of 634 tasks across 10 mature open-source software projects, involving about 114k code changes in total. Each task pairs a documentation change with the corresponding implementation and developer-written tests, enabling evaluation of whether a software engineering agent can implement the feature described by the documentation diff. The benchmark is constructed through a five-phase pipeline, starting from release notes, and offers broad coverage across libraries, developer tools, and frameworks while preserving real-world development settings. To facilitate lightweight and reliable evaluation under limited resources, we further curate a human-validated subset named Doc2Feat-bench Verified. It contains 114 high-quality instances whose task clarity and evaluation validity are manually verified. We use Doc2Feat-bench to assess a range of state-of-the-art software engineering agents with two widely used scaffolds. Experimental results show that despite significant token consumption, the best task success rate remains as low as 37.72% (OpenHands with Qwen3-Coder-480B). Our analysis reveals that software engineering agents face key challenges in performing cross-file edits, understanding existing code modules, and correctly interpreting documentation semantics.

1 INTRODUCTION

Issue-based benchmarks do not fully represent documentation-driven feature addition, where agents implement behavior from public-facing documentation. Doc2Feat-Bench evaluates this setting and finds low success rates alongside challenges in code editing, module understanding, and documentation interpretation.

1 INTRODUCTION

  • Existing agent benchmarks primarily use maintainer-facing issue reports, which often include implementation details and focus on bug fixing rather than feature addition.These inputs do not fully capture evaluation of documentation-driven feature addition.Source: §1 · pp. 1–2
  • Public-facing documentation changes describe intended behavior and provide a natural source of feature requirements for agent evaluation.Documentation is written for external users and integrators and is often updated as software features change.Source: §1 · p. 2
  • Doc2Feat-Bench pairs documentation changes with implementation tasks across 634 tasks from 10 mature open-source projects, with a manually validated 114-instance subset.Developer-written tests validate implementations; the subset’s task clarity and evaluation accuracy were manually verified.Source: §1 · p. 2
  • 37.72% is the best success rate, achieved by Qwen3-Coder-480B with OpenHands, versus 69.60% for the same setup on SWE-bench.The comparison highlights documentation-driven feature addition as a challenging, complementary benchmark setting.Source: §1 · p. 2
  • The analysis identifies cross-file editing, understanding existing code modules, and correctly interpreting documentation semantics as key agent challenges.The paper evaluates agents with Agentless and OpenHands scaffolds.Source: §1 · p. 2

2 Doc2Feat-Bench

Doc2Feat-Bench evaluates whether agents can turn real-world documentation changes into feature patches that pass relevant tests.

2 Doc2Feat-Bench

  • Each of the 634 tasks corresponds to a real-world feature addition documented in official release notes and includes documentation, code, and test changes.The tasks were collected from 10 popular open-source projects.Source: §2 · p. 2
  • The task input combines documentation changes with the complete codebase, and the goal is a patch implementing the documented feature that passes relevant tests.Each task reflects a coordinated documentation, implementation, and test change.Source: §2 · p. 2
  • The benchmark also provides Doc2Feat-Bench Verified, a subset constructed through manual validation and refinement for more reliable evaluation.The passage introduces the subset but does not specify its size here.Source: §2 · p. 2

2.1 Benchmark Construction

The benchmark is built through five phases, from selecting projects and collecting feature additions to preparing environments, filtering instances, and refining task inputs.

2.1.1 Phase 1: Project Selection.

  • 10 of the 12 SWE-bench projects meet the selection criteria, including feature-labeled release-note entries linked to GitHub PRs.Projects must also have available release notes; Flask and SymPy are excluded because their annotations lack feature labels.Source: §2.1.1 · pp. 2–3

2.1.2 Phase 2: Instance Collection.

  • 3,263 feature-addition PRs are filtered for source-code, test-file, and documentation changes, yielding 1,571 candidate instances.The documentation diff serves as the task input, while the code and test changes provide the corresponding implementation and validation material.Source: §2.1.2 · pp. 3–4

2.1.3 Phase 3: Environment Construction.

  • Instances are grouped by minor version and configured from project dependency files; failed builds may receive recorded manual dependency repairs.This phase aims to provide each instance with an isolated, runnable environment.Source: §2.1.3 · p. 4

2.1.4 Phase 4: Instance Filtering.

  • 634 instances remain after filtering for FAILED → PASSED feature-validation tests, while PASSED → PASSED tests serve as regression checks.Instances without a feature-validation test are excluded; errors before applying the feature patch are not automatically grounds for removal.Source: §2.1.4 · p. 4

2.1.5 Phase 5: Input Refinement.

  • Static analysis identifies new entity names shared by feature and test patches but absent from documentation, which can be supplied as optional hints; scripts also mask PR numbers, URLs, and issue references.The hints allow evaluation with or without this auxiliary information, depending on tolerance for potential data leakage.Source: §2.1.5 · p. 5

2.2 Doc2Feat-Bench Verified

Doc2Feat-Bench Verified is a manually validated subset intended to make evaluation more dependable and manageable under limited resources. The section describes its sampling and screening process, reports the retained set’s quality measures, and analyzes recurring documentation and test shortcomings.

2.2 Doc2Feat-Bench Verified

  • The verified subset offers a lighter, manually checked alternative to the 634-task full set for evaluation under limited resources.The full and verified sets are intended to balance broad coverage with lightweight evaluation.Source: §2.2 · p. 5

2.2.1 Construction Process.

  • The authors stratified a 238-task candidate sample by project and had two qualified annotators independently assess each task.The sample-size target was at least 84; annotators had at least three years of Python experience and a relevant bachelor’s degree or higher.Source: §2.2.1 · p. 5

2.2.2 Annotation Criteria.

  • Annotation assessed whether documentation clearly specified intended functionality, whether tests covered that functionality, and whether other major problems warranted exclusion.Lower scores indicate clearer task descriptions and more accurate, targeted test coverage.Source: §2.2.2 · p. 5

2.2.3 Annotation Results.

  • 114 instances remain in Doc2Feat-Bench Verified after filtering 106 for unclear tasks or inaccurate tests and 18 for other major problems.The 106 include 77 task-clarity flags and 73 test-accuracy flags, overlapping in 44 cases; retained tests have 94.60% median and 84.90% mean line coverage, with Cohen’s kappa of 0.4193.Source: §2.2.3 · pp. 5–6

2.2.4 Analysis of Filtered Instances.

  • Filtered examples expose documentation that omits required implementation behavior or understates scope, and tests that miss intended behavior or integration points.Examples include Birch’s undocumented clustering behavior, a LogisticRegression internal optimization, and tests that omit label interactions or real integration paths.Source: §2.2.4 · p. 6

3 Benchmark Characteristics

Doc2Feat-Bench tasks are more demanding than SWE-bench along input complexity, localization, and editing dimensions. Its documentation changes are longer, patches touch more files and hunks, and code edits are larger.

Input complexity

  • 739.06 words in Doc2Feat-Bench Full and 820.65 in Verified documentation changes exceed SWE-bench issue descriptions, averaging 480.37 and 447.72 words, respectively.The comparison indicates that agents must process longer inputs and identify fine-grained requirements.Source: §3 · p. 7, Figure 4 · p. 7

Localization Difficulty

  • 2.65 files and 10.32 hunks per Doc2Feat-Bench Full patch exceed SWE-bench Full’s 1.66 files and 4.12 hunks; new files are needed in 13.56% versus 1.70% of tasks.In the Verified subsets, patches involve 2.39 versus 1.24 files and 8.96 versus 2.43 hunks.Source: §3 · p. 7, Figure 4 · p. 7

Editing Difficulty

  • 179.12 lines edited per Doc2Feat-Bench Full patch and 179.22 in Verified far exceed SWE-bench Full’s 37.71 and Verified’s 14.32; nearly 20% of Doc2Feat-Bench patches exceed 200 lines.The line counts exclude test and documentation changes.Source: §3 · p. 7, Figure 4 · p. 7

4 Experimental Setup

The experiments ask how state-of-the-art agents perform on the Verified and Full benchmarks and why they fail. They evaluate selected models with Agentless and OpenHands, using task-success and supplementary patch, regression, cost, and feature-validation metrics.

4.1 Research Questions

  • The experiments examine agent performance on Doc2Feat-Bench Verified and Full and investigate the causes of failures.These are the three research questions guiding the evaluation.Source: §4.1 · p. 7

4.2 Model Selection

  • Eight state-of-the-art models were selected for evaluation; four were open-source and four closed-source, with four models reserved for Verified because Full evaluation was costly.Full contains 5.6 times as many instances as Verified; estimated cost for four models on Full was USD 5,124.Source: §4.2 · p. 7

4.3 Scaffolds

  • Agentless localizes files, classes or methods, and code lines hierarchically before patch generation, while OpenHands uses a CodeAct agent in a Docker sandbox.The study uses Agentless-1.0 and adapts documentation changes as OpenHands inputs.Source: §4.3 · pp. 7–8

4.4 Evaluation Metrics

  • Success%, Applied%, and token cost measure task success, patch applicability, and resource use; File% and RT% measure file alignment and regression-test preservation.Feature Validation Rate measures F→P test pass rates at micro and macro levels, reflecting overall test-level performance and instance-level robustness, respectively.Source: §4.4 · p. 8

5 Results

Agents achieve limited success on Verified, and average success declines further on Full. The analysis attributes failures to cross-file editing, understanding existing code modules, and interpreting documentation semantics.

5.1 RQ1: Performance on Doc2Feat-Bench Verified

  • 37.72% is the highest Verified success rate, achieved by Qwen3-Coder-480B with OpenHands; Claude-4-Sonnet with OpenHands scores 25.44% on Doc2Feat-Bench Verified versus 70.4% on SWE-bench Verified.GPT-4o and Gemini-2.5-Pro with OpenHands score 5.26% and 0%, respectively, on Verified.Source: §5.1 · pp. 8–9
  • Agentless outperforms OpenHands on Verified, with 98.80% versus 67.98% average patch application and 8.74% versus 4.92% FV-Micro.The paper attributes the difference partly to OpenHands context overflow and Agentless hierarchical localization and patch-completeness checks.Source: §5.1 · p. 8

5.2 RQ2: Performance on Doc2Feat-Bench Full

  • 14.83% is DeepSeek-R1’s best success rate on Full, where average success falls from 18.20% on Verified to 10.65%.The authors cite increased scale and complexity and possible noise in Full as factors that may contribute to the decline.Source: §5.2 · p. 9, Table 5 · p. 9
  • GPT-4o outperforms Qwen3-235B on Full despite similar Verified performance, indicating greater robustness as evaluation shifts to the larger set.The text identifies GPT-4o as an exception to otherwise broadly preserved model rankings.Source: §5.2 · p. 9

5.3 RQ3: Failure Analysis

  • Only 20.75% of Verified and 6.40% of Full multi-file tasks are solved by the best-performing model; single-file tasks account for 53.5% and 45.7% of instances, respectively.The analysis identifies limited cross-file editing as a main source of failures.Source: §5.3 · p. 9, Table 6 · p. 9
  • In 39.38% of examined cases, agents modify existing feature code, which can override prior behavior and cause regression-test failures.For pylint-7869, DeepSeek-v3 modified ColorizedTextReporter instead of introducing a separate NoHeaderReporter class.Source: §5.3 · pp. 9–10, Figure 5 · p. 10
  • Frequent documentation misreadings concern exception behavior (33.13%), edge conditions or thresholds (20.00%), and unspecified invalid-input handling (7.88%).Examples include missing AttributeError behavior, incomplete parameter constraints, and changing a documented default behavior.Source: §5.3 · p. 10, Figure 6 · p. 10

6 Discussion

Doc2Feat-Bench Verified supports controlled, higher-quality comparisons, while Full preserves the noisy distribution of real development data. The discussion also identifies task-quality, data-leakage, project-diversity, and scaffold-selection boundaries.

6.1 Doc2Feat-Bench Verified vs. Full

  • 124 of 238 sampled candidates (52.1%) were filtered out, and Verified is suited to controlled comparison while Full supports studying real-world noise and robustness.Model rankings remain largely consistent between the sets.Source: §6.1 · p. 10

6.2 Threats to Validity

  • Incomplete documentation and test suites constrain evaluation, while historical GitHub data creates leakage risk that PR-number masking and patch-similarity checks help assess.Successful DeepSeek-R1 patches on Full had average Levenshtein similarity of 0.48 to ground-truth patches.Source: §6.2 · p. 11
  • The benchmark covers well-maintained Python projects with structured release notes and evaluates systems only with Agentless and OpenHands.Performance on other scaffolds remains unknown; the construction method is described as adaptable to other ecosystems.Source: §6.2 · p. 11

7 Related Work

Prior agentic software engineering benchmarks largely use maintainer-facing inputs and emphasize issue solving. Doc2Feat-Bench instead evaluates feature addition from public documentation changes, a setting that also presents new challenges for existing agentic methods.

7.1 Benchmarks for Agentic SE Systems

  • Existing benchmarks such as SWE-bench [23] largely use maintainer-facing inputs and emphasize bug fixing or issue solving, leaving public-facing feature specifications less represented.Feature addition accounts for about 60% of software maintenance activities.Source: §7.1 · p. 11
  • Doc2Feat-Bench uses public-facing documentation changes, includes additions that extend existing code, and identifies feature-adding PRs through developer-maintained release notes.A manually constructed verified subset supports lightweight, reliable evaluation.Source: §7.1 · p. 11

7.2 Automating SE Tasks with Agentic Methods

  • Pipeline- and agent-based methods are typically evaluated with maintainer-facing, implementation-level inputs, whereas Doc2Feat-Bench specifies features through public documentation changes.This mismatch introduces new challenges for applying prior methods to the benchmark.Source: §7.2 · p. 11

8 CONCLUSION

The paper introduces Doc2Feat-Bench, a benchmark for documentation-driven feature addition, built from release-note-identified tasks. Its evaluation finds the setting remains difficult and points to concrete areas for improving agents.

8 CONCLUSION

  • 634 real-world feature-addition tasks were identified from developer-maintained release notes through a systematic five-phase construction pipeline.Systems receive repository documentation changes and must generate a feature-adding patch.Source: §8 · p. 11
  • The authors evaluate eight advanced models using the Agentless and OpenHands scaffolds.Source: §8 · p. 11
  • The authors characterize documentation-driven feature addition as uniquely difficult and unsolved, with challenges in cross-file editing, code-module understanding, and documentation-semantic interpretation.Source: §8 · p. 11

Data Availability

The replication package, online appendix, official repository, and leaderboard are available online; an earlier benchmark name was NoCode-bench.

Data Availability

Related papers

Loading 2507.18130v4…