PaperlayerView exact revision on arXiv

Source-linked AI summary

FeatureArc: Benchmarking Coding Agents on Evolving Features

Yunlong Wang, Alyssa Wu, Cuiyun Gao, Jia Feng, Songyuanyi Lu, Zhichao Hu

arXiv:2610.37434v1cs.SEcs.AI

TL;DR

Most feature benchmarks evaluate a single request on a fixed codebase, rather than successive extensions on an agent’s own earlier code. FeatureArc constructs repository-history-based tasks and evaluates coding agents as they extend a feature without external test feedback or rollback. Pooled success falls from 62.7% at the first stage to 28.5% on the whole feature, and inherited errors often go unnoticed.

  • Problem

    Most feature benchmarks evaluate a single request on a fixed codebase, leaving successive extensions on an agent’s own earlier code unevaluated.

    Source: §I · p. 1

  • Method

    FeatureArc constructs tasks from repository histories and uses sequential specifications, requirements and interface contracts, and executable checks as agents extend their own code.

    Source: §I · p. 2, B. Evaluation Process · p. 3

  • Results

    62.7% first-stage pooled success falls to 28.5% on the whole feature, and agents fail to notice earlier errors in 94.9% of subsequent stages.

    Source: §I · p. 2

  • Takeaways & Limitations

    Evolving a feature without human checks between stages remains a substantial challenge for current coding agents.

    Source: §I · p. 2

    FeatureArc may not cover the full range of workflows encountered in real-world feature development.

    Source: B. Threats to Validity · p. 10

Abstract

from arXiv · show

A software feature usually evolves over time: developers extend its first version as new use cases and requirements arrive, and each extension builds on code written earlier. Most feature benchmarks evaluate a single request on a fixed codebase. Feature evolution requires evaluating successive extensions on an agent's own earlier code and checking whether the feature remains correct. We introduce FeatureArc to evaluate coding agents on evolving features, together with a pipeline that constructs its tasks from repository histories. The benchmark combines dependencies between successive stages of the same feature, explicit requirements and interface contracts, and executable checks across four programming languages. The pipeline validates fail-to-pass (F2P) tests for new behavior and pass-to-pass (P2P) tests for existing behavior on the reference code before and after each stage. We evaluated 10 coding agents on 73 manually reviewed tasks of 2 to 10 stages. Each agent works in one persistent session and workspace without external test feedback or rollback. The evaluation reveals a gap between first-stage and whole-feature success: pooled success falls from 62.7% at the first stage to 28.5% when the whole feature is tested on the final code. Trajectory analysis identifies inherited defects as a common failure mechanism, present in 72.9% of failed later stages. These earlier errors often remain undetected: in stages following a failure, agents leave the earlier error unnoticed about 95% of the time, and their self-tests rarely revisit earlier stages. Evolving a feature autonomously, without human checks between stages, therefore remains a substantial challenge for current coding agents. At Tencent, the pipeline extends internal evaluations to this setting and guides the development of Hunyuan, Tencent's family of large language models.

I. INTRODUCTION

FeatureArc evaluates whether coding agents can extend their own code while keeping a feature correct as requirements evolve. Its repository-history-based tasks expose a gap between first-stage and whole-feature success, with inherited errors often persisting unnoticed.

I. INTRODUCTION

  • At L4, agents receive only the next specification and must find earlier errors themselves, unlike L1–L3 settings where users report failed stages.Tencent’s internal multi-stage evaluations covered only L1–L3.Source: §I · p. 1
  • Most existing feature benchmarks use single requests on fixed codebases or multi-stage settings that lack historical same-feature dependencies and L4 evaluation.FeatureArc combines historical dependencies, sequential requests, and continued evaluation on the agent’s own code without hidden-test feedback.Source: §I · p. 2
  • FeatureArc combines historical same-feature dependencies, explicit requirements and interface contracts, executable checks, and four programming languages.F2P tests check new behavior, P2P tests check existing behavior, and a final check evaluates the complete feature on the agent’s code.Source: §I · p. 2
  • FeatureArc’s agents receive specifications sequentially in a persistent session and workspace, with hidden-test results withheld and no rollback.Each agent carries its own code forward after every stage.Source: §I · p. 2
  • 62.7% first-stage pooled success falls to 28.5% for whole-feature success, while inherited errors often persist and go unnoticed.After a failed stage, agents never notice the error in 94.9% of subsequent stages.Source: §I · p. 2
  • At Tencent, the pipeline extends internal evaluations to L4 and assesses Hunyuan 4 Preview alongside external agents.The evaluation distinguishes implementing a new request from detecting errors in the code it depends on.Source: §I · p. 2

II. TASK FORMULATION

A FeatureArc task follows one user-visible feature through successive stages in one repository. Under L4, agents implement each request on their own earlier code without receiving hidden-test results or failed-stage feedback.

A. Task Definition

  • A task follows one user-visible feature through an ordered sequence of stages in one repository, with each later stage extending the same feature.The paper calls this sequence a same-feature chain.Source: A. Task Definition · p. 2
  • The illustrated chain adds app.run(), then app.runAndWait(), then file input and files.upload(), with each stage building on the preceding implementation.The second stage calls app.run() and waits for completion.Source: A. Task Definition · pp. 2–3
  • The agent receives an initial repository state and stage specifications one at a time, implementing each request on top of its earlier work.Each specification includes a requirements document and an interface contract covering behavior, names, signatures, constants, and error messages.Source: A. Task Definition · p. 3
  • Stage-level hidden F2P tests check new behavior, while P2P tests check existing behavior; separate chain-level suites check the complete feature.The tests are validated against reference code.Source: A. Task Definition · p. 3

B. Evaluation Process

  • One session and workspace persist throughout the task, while reference patches, hidden tests, and future specifications remain private to the agent.Agents may inspect their code and run their own checks.Source: B. Evaluation Process · p. 3
  • A stage gate runs F2P and P2P suites after each request, and both suites must pass for that stage to pass.Failed stages are neither rolled back nor reported; the next stage starts from the agent’s code.Source: B. Evaluation Process · p. 3
  • After the final stage, a chain gate runs chain-level F2P and P2P suites on the final workspace to check the complete feature.The final check evaluates the agent’s accumulated code.Source: B. Evaluation Process · p. 3, A. Task Definition · p. 3

III. BENCHMARK CONSTRUCTION

FEATUREARC turns repository histories into tasks where agents extend their own earlier code through successive stages of one feature. Its pipeline mines related changes and retains chains whose stages add observable, testable behavior.

III. BENCHMARK CONSTRUCTION

  • The five-step pipeline converts repository histories into dependent feature tasks, then selects and manually reviews an evaluation set.Its steps cover repository collection and history analysis, evolution-chain mining, reference-state construction, environment and test validation, and specification generation.Source: §III · p. 3, A. Step 1: Repository Collection and… · p. 3

A. Step 1: Repository Collection and History Analysis

  • Repositories must have at least 30 stars, permissive licenses, offline Linux builds, and code mainly in Python, TypeScript, Rust, or Go.The star threshold indicates basic community interest; the other conditions support redistribution and repeatable execution.Source: A. Step 1: Repository Collection and… · p. 3

B. Step 2: Evolution-Chain Mining

  • An evidence graph links commits through shared code, file history, references, or imports, helping trace related changes despite intervening development.A module map and topology and semantic agents provide structural and commit-intent evidence for historical relationships.Source: B. Step 2: Evolution-Chain Mining · p. 3, A. Step 1: Repository Collection and… · p. 3
  • Four complementary graph searches propose chains, which an LLM review agent checks against source code, tests, and diffs.Chains are retained only when every stage extends the same user-visible feature with observable, testable behavior; review can add missed extensions, remove unrelated work, and adjust stage boundaries.Source: B. Step 2: Evolution-Chain Mining · p. 3

C. Step 3: Reference-State Construction

The pipeline constructs paired reference states, validates tests against them, and generates behavior specifications and interface contracts. Curation and manual review produce the final task set.

C. Step 3: Reference-State Construction

  • Paired reference states isolate each selected extension while retaining surrounding repository context, enabling stage-by-stage validation.The pipeline removes selected patches from the final checkpoint to form S0, then reapplies them in order to create S1 through Sn.Source: C. Step 3: Reference-State Construction · p. 4

D. Step 4: Environment and Test Validation

  • F2P tests must fail before a stage and pass afterward, while P2P tests must pass in both states; chain-level suites validate S0 against Sn.An offline container is repaired until reference states build and tests can be discovered; historical tests are packaged with required fixtures and helpers.Source: D. Step 4: Environment and Test… · p. 4

E. Step 5: Specification Generation

  • A specification agent uses the stage patch and F2P tests to create behavior requirements and an interface contract without revealing implementation.Requirements state behavior and acceptance criteria; contracts specify names, signatures, constants, and error messages. The pipeline removes agent-generated documents and restores documents that reveal later implementations.Source: E. Step 5: Specification Generation · p. 4

F. Evaluation-Set Curation

  • The manually reviewed evaluation set contains 73 tasks and 263 stages from 53 repositories, and all reported agent comparisons use this same set.Curation samples 100 candidates across four languages and chain lengths, excludes seven identified task defects, and manually checks specification quality, contract-test alignment, and same-feature evolution.Source: F. Evaluation-Set Curation · p. 4

G. Benchmark Characteristics

FEATUREARC combines dependent stages, requirements-grounded checks, and varied repository contexts. Its evaluation setup examines completion, inherited errors, and agents’ detection and verification of their own errors.

G. Benchmark Characteristics

  • Later stages share an average of 54.3% of changed files with earlier stages, and tasks span 2 to 10 stages, averaging 3.60.The median file overlap across tasks is 59.1%.Source: G. Benchmark Characteristics · p. 5
  • Stages have median reference changes of 426 source lines and 416 test lines, with an average of 7.71 changed source files.The median whole-feature change is 1,453 source lines; paired-state validation and manual checks establish alignment between executable evaluation and requested behavior.Source: G. Benchmark Characteristics · p. 5
  • Tasks cover Python, Go, TypeScript, and Rust, as well as seven application domains; AI agents and LLM tooling account for 17 tasks and cloud and infrastructure for 14.The language counts are 21, 20, 18, and 14 tasks, respectively.Source: G. Benchmark Characteristics · p. 5

IV. EXPERIMENTAL SETUP

  • The study asks how reliably agents complete evolving features, how earlier errors affect later stages, and how effectively agents detect and verify their own errors.Its measures include stage and chain gates, conditional pass rates and inheritance labels, and awareness labels, self-test outcomes, and specification checks.Source: §IV · p. 5

A. Evaluated Agents

The study compares configured coding agents built on models from ten providers. Trajectory analysis labels observable verification behavior and uses hidden-test evidence for failure diagnoses.

A. Evaluated Agents

  • The evaluated set includes configured agents built on Claude Opus 5.5, GPT-6 Astra, Hunyuan 4 Preview, GLM-5.3, Grok 4.7, Qwen 3.8, Gemini 3.8 Flash, Doubao 2.1 Pro, Kimi K3, and DeepSeek V4 Pro.The comparison unit is the configured coding agent, including its model and harness; Hunyuan 4 Preview is the internal pre-release model evaluated in the study.Source: A. Evaluated Agents · p. 5

B. Trajectory Analysis

  • Trajectory labels track the strongest check, last test result, acceptance-criterion targeting, stop reason, and whether dismissal of a failure is supported.These process labels describe behavior visible in specifications, trajectories, and agent code; diagnosis labels also use hidden-test evidence.Source: B. Trajectory Analysis · p. 5
  • For stages after a failure, prior-awareness labels record whether agents missed, noticed without repairing, or repaired it; inheritance labels attribute later failures to earlier defects.F2P failure causes are recorded at fine-grained level and grouped into interface, other build, and behavioral failures, with timeouts, unrelated failures, and unknown causes separate.Source: B. Trajectory Analysis · pp. 5–6

C. Implementation Details

The implementation details describe task construction, persistent agent execution, trajectory analysis, and specification checks. The metrics section defines stage- and chain-level outcomes and the procedures used for comparisons and uncertainty intervals.

C. Implementation Details

  • Task acceptance requires executable reference-state validation and review of task materials; separate agents handle history review, chain review, patch classification, environments, test extraction, and specification writing.GPT-5.5, DeepSeek V4.1 Flash, and Claude Sonnet 5 handle Steps 1–3, 4, and 5, respectively.Source: C. Implementation Details · p. 6
  • Agents retain one session across stages under a four-hour stage budget, without external failure feedback or rollback; cross-agent scores combine model and harness effects.Nine agents run in Claude Code and GPT-6 Astra uses the native Codex CLI; all receive the same initial code, specifications, and hidden checks.Source: C. Implementation Details · p. 6
  • Three model judges annotate trajectories before viewing hidden-test evidence, with majority labels and first-author adjudication when all judges disagree.A blind human review samples 180 stages across passed, failed, and post-failure groups; Table II reports its validation.Source: C. Implementation Details · p. 6, Table II · p. 6
  • For specification checks, the first author examines sampled green-self-test cases that fail hidden F2P and classifies each failing expectation by its basis.Eligible stages start at m0 or after a passed stage, and checks must be labeled as targeting every acceptance criterion.Source: C. Implementation Details · p. 6

D. Metrics and Statistics

  • The metrics distinguish first-stage pass rate, final chain-gate pass rate, later-stage continuation, stage F2P, prefix depth, survival, and chain-level P2P failure.In-chain regression and pure regression use different conditions: the former concerns final chain-level F2P failure after every stage passed F2P; the latter concerns final P2P failure despite chain-level F2P passing.Source: D. Metrics and Statistics · p. 6
  • Within-task-stage comparisons retain groups containing both conditions and average their F2P pass rates equally; agent-performance intervals use 4,000 task-bootstrap resamples.The compared agents retain their own code and conversation histories while facing the same requested stage.Source: D. Metrics and Statistics · pp. 6–7

V. EXPERIMENTAL RESULTS

The results report FEATUREARC performance under the L4 evaluation process and organize the analysis around completion, earlier errors, and error detection and verification.

V. EXPERIMENTAL RESULTS

  • The evaluation reports FEATUREARC results under L4; Table III presents performance percentages and is ordered by chain-gate pass rate.The analysis proceeds from whole-feature completion (RQ1) to earlier errors (RQ2) and then error detection and verification (RQ3).Source: §V · p. 7, Table III · p. 7
  • Blind review of 180 stages yields 89.4%–97.2% agreement for process labels and 91.7%–100.0% for diagnosis labels.Agreement is 93.3% for prior awareness and 100.0% for inheritance; Fleiss’ κ across six labels ranges from 0.80 to 0.94.Source: §V · p. 7
  • The automatic tool classifier agrees with trajectory labels on 94.8% of stages where both identify a final red or green test result.Source: §V · p. 7

A. RQ1: Completing Evolving Features

At L4, success on a first stage substantially exceeds whole-feature success. Later-stage failures are frequently associated with inherited defects, while measured regressions after successful implementation are less common.

A. RQ1: Completing Evolving Features

  • 62.7% first-stage success falls to 28.5% at the chain gate, and every agent loses 26.0–38.4 percentage points.Even Claude Opus 5.5, which leads every metric, fails more than half of tasks.Source: A. RQ1: Completing Evolving Features · p. 7
  • 25.3% of runs passing the last-stage gate fail the chain gate, and only 2.7% of runs with a failed stage eventually pass the chain gate.The last-stage gate checks the final extension, whereas the chain gate checks the complete feature on the final workspace.Source: A. RQ1: Completing Evolving Features · p. 7
  • 36.2% of runs first fail at m0 and 20.7% at m1; chain-gate success drops from 44.4% on two-stage tasks to 1.5% on tasks with five or more stages.Only two runs pass tasks with five or more stages, both by Grok 4.7; Figure 5 relates performance to stage position and task length.Source: A. RQ1: Completing Evolving Features · p. 7, Figure 5 · p. 8

B. RQ2: The Effect of Earlier Errors

  • 72.9% of failed later stages involve errors left in earlier stages, and next-stage F2P success is 70.2% after a passed stage versus 22.6% after a failed stage.Inherited defects account wholly for 47.1% and partly for 25.9% of those failures.Source: B. RQ2: The Effect of Earlier Errors · pp. 7–8, Table IV · p. 8
  • The previous stage’s outcome is associated with a 47.6-point F2P gap, larger than the 5.2–6.4-point gaps reported for context compaction and cumulative specification length.Among later stages with fully passed prefixes, F2P success is 75.0% with 1,500–2,999 specification words and 68.6% with at least 8,000 words.Source: B. RQ2: The Effect of Earlier Errors · p. 8
  • In-chain regression occurs in 7.3% of runs where every stage passed F2P, while pure regression occurs in 3.3% of runs passing final chain-level F2P.The broader 5.5%–20.5% chain-level P2P failure rates also include runs where new feature behavior fails.Source: B. RQ2: The Effect of Earlier Errors · p. 8

C. RQ3: Detecting and Verifying Errors

Agents often fail to notice earlier errors, and their self-tests provide limited evidence that hidden acceptance checks will pass. Even checks targeting every criterion can miss specified behavior or encode incorrect expectations.

C. RQ3: Detecting and Verifying Errors

  • 94.9% of stages following a failure show no agent awareness of the earlier error; the agent repairs 34 of the 50 errors it does notice.Blind review agrees with the prior-awareness labels on 56 of 60 sampled stages.Source: C. RQ3: Detecting and Verifying Errors · p. 8
  • After failed stages, tool-call counts rise only 4%–18% for nine Claude Code agents, while Claude Opus 5.5’s testing share barely changes, from 15.0% to 15.3%.Source: C. RQ3: Detecting and Verifying Errors · p. 8
  • Only 52.7% of stages ending with a green self-test pass F2P, and after a failed stage that share is 24.5%.Across all stages, 73.8% end green; the overall F2P pass rate is 50.4%.Source: C. RQ3: Detecting and Verifying Errors · pp. 8–9
  • Of 367 red-ending stages, agents dismiss the failure as unrelated or pre-existing in 321 cases (87.5%); only 23 of those cases first rerun the test on starting code.Source: C. RQ3: Detecting and Verifying Errors · p. 9
  • 45% of stages with checks targeting every criterion still fail F2P, and 62% of those failures follow an earlier failed stage the agent did not notice.Checks target all acceptance criteria in 68.4% of stages, but criterion coverage offers limited assurance.Source: C. RQ3: Detecting and Verifying Errors · p. 9
  • Among 239 green-self-test stages targeting every criterion that still fail F2P, most failures involve wrong logic or a misread specification (54.4%) or missed cases or partial implementation (24.3%).In a 60-stage specification check, first failing expectations were explicit in a current or earlier specification in 34 cases, implied by specifications in 12, and implied by the starting repository in 14.Source: C. RQ3: Detecting and Verifying Errors · p. 9
  • Of 1,304 F2P-failing stages, 236 (18.1%) fail because the hidden suite does not compile; agent-caused interface mismatches account for 61.4% of the 228 agent-caused compile failures.Mismatches include missing symbols or exports, wrong parameter or return types, and duplicate declarations.Source: C. RQ3: Detecting and Verifying Errors · p. 9
  • Explicit contracts and separate acceptance checks make observable the gap between targeting a requirement and actually checking its specified behavior.Source: C. RQ3: Detecting and Verifying Errors · p. 10

VI. DISCUSSION

At Tencent, follow-up interactions motivate evaluating continued feature development without intermediate user checking. Hunyuan often continues correctly when prior stages pass, but frequently misses earlier failures and rarely verifies its explanations against starting code.

A. Industry Application at Tencent

  • In Tencent’s 8,881 IDE episodes, 43.5% of episodes with valid user turns contained follow-up interaction, including constraint additions, clarifications, and functionality extensions.The first follow-up annotations identified constraint additions (30.1%), requirement clarifications (19.7%), and functionality extensions (17.9%).Source: A. Industry Application at Tencent · p. 10
  • 90.2% of Hunyuan’s earlier failures go unnoticed, while next-stage F2P success falls from 71.6% after a passed stage to 22.5% after a failed one.Its 75.0% continuation rate indicates it can often extend a feature when earlier stages are correct.Source: A. Industry Application at Tencent · p. 10
  • Hunyuan attributed a final failing self-test to unrelated or pre-existing causes in 44 of 46 stages, but checked that explanation against starting code in only 5 cases.The resulting failure cases help the Hunyuan team target checks of earlier requirements and verification of failure explanations.Source: A. Industry Application at Tencent · p. 10

B. Threats to Validity

FEATUREARC acknowledges that its workflow coverage may be incomplete, situates its design among prior feature-evolution and verification benchmarks, and concludes that sustained feature correctness merits further study.

B. Threats to Validity

  • FEATUREARC spans Python, TypeScript, Rust, and Go across seven domains, but may not cover the full range of real-world feature-development workflows.Tasks are stratified by language and chain length and mined from repository histories.Source: B. Threats to Validity · p. 10

VII. RELATED WORK

  • Prior benchmarks vary in stage sources, feature scope, and feedback; FEATUREARC combines historical same-feature dependencies with continued evaluation on agents’ own code without intermediate failure feedback.Its validated stage and whole-feature checks measure correctness across extensions, while trajectory analysis examines inherited-error persistence and detection; related work also covers task construction and self-verification.Source: §VII · p. 10

VIII. CONCLUSION

  • Pooled success falls from 62.7% at the first stage to 28.5% for whole-feature completion, and inherited defects commonly persist undetected.FEATUREARC combines historical same-feature dependencies, requirements-grounded verification, and diverse repositories to evaluate agents extending their own code without external acceptance feedback.Source: §VIII · p. 10

Related papers

Loading 2610.37434v1…