PaperlayerView exact revision on arXiv

Source-linked AI summary

SpecForge: Recursive Self-Improvement for Coding Agents via Behavioral Specifications

Yutong Cheng, Haifeng Chen, Peng Gao, Wei Cheng

arXiv:2610.15708v1cs.AIcs.SE

TL;DR

Coding agents can refine behavioral specifications using execution feedback, but output coupling between program entities can make iterative improvements unstable. SPECFORGE searches multiple specification states while coordinating entity selection and failure-based refinements; it outperforms strong baselines and transfers to downstream coding tasks.

  • Problem

    Specification refinement is difficult because dependency and generation coupling let a change to one entity affect the correctness of others, making progress unstable.

    Source: §1 · p. 2

  • Method

    SPECFORGE uses multi-frontier tree search with Pareto-guided state selection, callee-before-caller refinement, and error-diversified failure batches.

    Source: §1 · p. 2

  • Results

    SPECFORGE outperforms strong baselines across five backbones, achieving a 94.9% solve rate with Claude-4.5-Sonnet and a 5.7-percentage-point average gain over the strongest baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The optimized specifications provide reusable behavioral knowledge that improves cross-language translation and new-feature implementation.

    Source: §1 · p. 2

    The loop requires a reference implementation and was evaluated on 20 Python modules; other languages were included only as downstream targets in the cross-language transfer experiment.

    Source: N Limitations · p. 38

Abstract

from arXiv · show

Coding agents based on frozen language models can improve their performance by refining the external context they rely on, without updating model weights. We study recursive self-improvement through behavioral specifications: natural-language code documentation that describes the behavior of program entities and guides agents in generating correct code. We formulate a self-improvement loop in which an agent generates code from its current specifications, evaluates it with execution-verified tests, and refines the specifications based on observed failures. The improved specifications then guide subsequent generation, progressively improving the agent's understanding of the codebase. However, recursive refinement is challenging because program entities are interdependent. Improving the specification of one entity can inadvertently degrade the correctness of others, a phenomenon we call output coupling. This creates a whack-a-mole effect in which local improvements cause regressions elsewhere, preventing consistent progress. We introduce SpecForge, a multi-frontier tree search framework that addresses this challenge with three mechanisms: (i) Pareto-guided state selection preserves complementary specification states and prevents premature commitment; (ii) dependency-constrained entity selection prioritizes callee-before-caller refinement; and (iii) error-diversified expansion explores multiple refinements using different subsets of test failures. On DevEval+, SpecForge achieves the best performance across five backbone models, including a 94.9% solve rate with Claude-4.5-Sonnet and a 5.7-percentage-point average improvement over the strongest baseline. The improved specifications also make the agent better at tasks outside the loop: they transfer to cross-language translation and to new-feature implementation, where they outperform the benchmark-provided requirement documents.

1 Introduction

SPECFORGE tackles output coupling in specification-mediated self-improvement by searching multiple specification states and coordinating which entities and failures guide refinement. It outperforms strong baselines across five backbone models, and its specifications also improve performance on downstream tasks.

1 Introduction

  • The recursive loop improves behavioral specifications from code-generation and execution feedback without changing model weights, and the resulting specifications also help with cross-language translation and new-feature implementation.Tests are automatically generated from the reference implementation and validated through execution; the goal is behavioral correctness rather than documentation readability.Source: §1 · p. 2
  • Output coupling makes iterative refinement unstable: changing one entity can fix it while breaking others, so progress may oscillate rather than accumulate.The paper distinguishes caller-callee dependency coupling from generation coupling, where changing one specification affects code generated for other entities.Source: §1 · p. 2
  • SPECFORGE uses Pareto-guided state selection, callee-before-caller entity selection, and error-diversified expansion to mitigate refinement interference.It preserves complementary specification states, restricts refinement to entities whose callees are resolved, and explores candidates from different subsets of failing tests.Source: §1 · p. 2
  • SPECFORGE improves over strong baselines across five backbones, with a 94.9% solve rate on Claude-4.5-Sonnet and a 5.7-percentage-point average gain over the strongest baseline.The evaluation measures specification quality by the correctness of code reconstructed from the specifications.Source: §1 · p. 2

2 Problem Formulation

Behavioral specification optimization seeks natural-language specifications from which a code-generation model can reconstruct correct implementations. Because specification quality is observable only through generated code and execution, the task is a black-box search whose entities can interfere with one another.

2 Problem Formulation

  • A behavioral specification is judged by whether a model can reconstruct correct entity implementations, rather than by human readability.The optimization goal is to maximize the number of resolved entities, where an entity is resolved when its pass rate is 1.Source: §2 · p. 3
  • Specification bundles are evaluated through generated code and execution outcomes, with test failures providing structured feedback about omitted behavioral information.The formulation also tracks per-entity pass rates and a module-level pass rate.Source: §2 · p. 3
  • Output coupling arises because callers depend on callees’ behavior and because changing one specification can alter code generation for other entities.These interactions can make a local refinement improve one entity while destabilizing others.Source: §2 · p. 3
  • The search is structured around specification states because different partial solutions resolve different entity subsets and thus enable different dependency-constrained refinements.A state records the specification bundle, per-entity pass rates, and resolved entities.Source: §2 · p. 3

3 Proposed Method

SPECFORGE casts behavioral specification optimization as multi-frontier search over states evaluated by executing generated code. It preserves complementary search paths, refines entities in dependency order, and diversifies the failure feedback used to propose revisions.

3 Proposed Method

  • Each search state is evaluated by executing generated code, and each edge represents a self-revision driven by that state’s failures.A single-entity transition keeps refinements localized and allows competing partial solutions to be explored in parallel.Source: §3 · pp. 3–4

3.1 Pareto-Guided State Selection

  • The Pareto archive retains non-dominated states with complementary per-entity progress, avoiding premature commitment to one unstable trajectory.Frequency-weighted sampling favors archive nodes that are frontier-best across more entity-wise dimensions.Source: §3.1 · pp. 4–5

3.2 Dependency-Constrained Entity Selection

  • Dependency-constrained selection refines only unsolved entities whose callees are resolved, then adaptively allocates budget among eligible entities using discounted UCB.Its normalized reward measures improvement against the entity’s remaining pass-rate gap, and the shared bandit balances uncertainty, past success, and remaining potential.Source: §3.2 · p. 5

3.3 Error-Diversified Expansion

  • Error-diversified expansion generates refinement candidates from distinct subsets of an entity’s failing tests to increase the chance of using a cleaner diagnostic signal.When at least one sampled batch is cleaner than the full failure set, the best one-step branch can have higher quality than repeatedly conditioning on all failures.Source: §3.3 · p. 6

4 Experiments

The experiments evaluate SPECFORGE on DEVEVAL+, compare its search behavior and components, and test whether optimized specifications transfer to translation and new-feature tasks. Results show gains over baselines in the main benchmark and both downstream settings.

4.1 Setup

  • DEVEVAL+ covers 20 Python modules, 247 entities, and 760 test cases; methods optimize with LLM-generated tests and are evaluated on hidden ground-truth tests.Solve Rate measures entities passing all tests, while Pass Rate measures the fraction of test cases passed.Source: §4.1 · pp. 6–7
  • Comparisons include documentation generators, monolithic prompt optimizers, commercial agents, iterative refinement, and DOCSEARCH, the only baseline optimizing the same objective.The commercial-agent variants include topological-order and test-feedback augmentations; DOCSEARCH uses callee-before-caller refinement and diversified error context.Source: §4.1 · pp. 6–7

4.2 Main Results

  • 94.9% solve rate with Claude-4.5-Sonnet; SPECFORGE leads across five backbones and exceeds the strongest baseline by 3.8–7.4 solve-rate points, averaging 5.7.The main results also report 88.9% with Qwen3-235B-A22B, 89.6% with Gemini-2.5-Flash, 90.9% with GPT-4o, and 91.0% with Gemini-3.1-Pro.Source: Table 1 · p. 7, §4.2 · pp. 7–8

4.3 Token Budget Efficiency Analysis

  • 41% to 93%: SPECFORGE’s cross-module average pass rate rises steadily, while iterative baselines oscillate and monolithic methods show instability.The steepest gains occur in the first 10 rollouts (+29%), as topological constraints and D-UCB exploration rapidly solve leaf entities.Source: Figure 3 · p. 8, §4.3 · p. 8

4.4 Component Contribution Analysis

  • −18.6%: removing reverse topological ordering causes the largest ablation loss; D-UCB, error diversity, and the Pareto archive contribute −10.4%, −9.5%, and −5.2%.The reported mechanisms are dependency ordering, adaptive budget allocation, diversified error sampling, and preserving frontier diversity.Source: Figure 3 · p. 8, §4.4 · p. 9

4.5.1 Cross-Language Translation

  • 65.7% solve rate and 71.3% pass rate transfer to cross-language translation, improving solve rate by 26.0% over Source-only; Spec+Source reaches 76.2% / 80.4%.The evaluation uses a 16-module DEVEVAL+-JAVA subset with 208 entities and 680 test cases.Source: Figure 4 · p. 8, §4.5.1 · p. 9

4.5.2 New Feature Implementation

  • 74.5% easy and 51.7% hard Pass@1: SPECFORGE specifications outperform SWE-DEV’s PRD by 18.8 and 7.1 percentage points, respectively.The study tests 27 feature-implementation instances across 10 packages; SPECFORGE-generated specifications are produced without access to the PRD or target tests.Source: Figure 4 · p. 8, §4.5.2 · pp. 9–10

5 Related Work

Related work spans recursive self-improvement, coding agents, and code documentation. SPECFORGE is positioned as a context-space approach using execution feedback, with specifications evaluated by the code they induce.

Recursive Self-Improvement

  • Recursive self-improvement methods differ by whether improvements are written to model weights or external context, and by whether feedback comes from model judgment or external verifiers.SPECFORGE is a context-space method with an execution-based signal and a fixed update procedure.Source: §5 · p. 10

Coding Agents

  • Coding agents already work across repositories, while SPECFORGE complements them by supplying behavioral specifications that improve translation and new-feature implementation.The passage situates these agents among applications extending beyond conventional software, including games, animation, 3D scenes, and symbolic music.Source: §5 · p. 10

Code Documentation

  • Repository documentation systems target human readability, whereas SPECFORGE judges specifications by downstream code-generation correctness.The described systems include structural API-reference generation and retrieval-grounded explanations, assessed with human or proxy quality measures.Source: §5 · p. 10

6 Conclusion

The conclusion presents SPECFORGE as a way to make specification-mediated self-improvement stable under output coupling. The appendix formalizes why preserving multiple non-dominated search frontiers can retain improving moves that a single-path search misses.

6 Conclusion

  • SPECFORGE uses Pareto-guided frontier preservation, dependency-constrained entity selection, and error-diversified expansion to support stable progress despite coupled refinements.The conclusion reports consistent gains on DEVEVAL+ and downstream improvements in cross-language translation and new-feature implementation.Source: §6 · p. 10

A Output Coupling and Refinement Interference

  • Output coupling means a local specification refinement can improve one entity while triggering regressions elsewhere because generated entities are interdependent.This instability can make ordinary sequential or monolithic refinement oscillate rather than accumulate progress.Source: §6 · p. 10, A Output Coupling and Refinement… · p. 17

B.1 Why Multi-Frontier Search Escapes Certain Frontier-Local Optima

  • Under the stated complementary-frontier assumption, a single-incumbent strategy cannot improve the solved-set objective, while retaining two non-dominated states gives multi-frontier search a nonzero chance to do so.The result is conditional on one state being locally saturated while another has an eligible improving refinement.Source: §B.1 · p. 18
  • Archive diversity preserves different eligibility frontiers, so a state with a lower current score can still expose a productive next refinement after another frontier is exhausted.A state’s eligibility depends on its solved subset, making scalar best-state retention potentially brittle.Source: §B.1 · p. 18

B.2 Proof of Signal Corruption

The proof formalizes why noisy failures can undermine refinement and when diversified sampling can improve the best candidate. The algorithm details specify dependency-aware search, cyclic-call handling, and adaptive entity selection.

B.2 Proof of Signal Corruption

  • Under Assumptions 2–3, conditioning on the full failure set is suboptimal when it includes noisy failures alongside informative ones.The theorem compares full-set conditioning with conditioning on the informative part alone.Source: §B.2 · pp. 18–19

B.3 Best-Branch Advantage of Diversified Expansion

  • If a sampled batch is cleaner than the full failure set, diversified expansion strictly improves the best one-step branch over repeating full-set conditioning.The motivating noise sources include cascading failures, redundant signals, and misleading co-occurrences.Source: §B.3 · pp. 19–20

C Algorithm Details

  • SPECFORGE selects a Pareto-archive state, refines an eligible entity using distinct error batches, evaluates child states, and updates the archive and best state.The algorithm counts each generated child as one specification-to-code-to-test evaluation.Source: C Algorithm Details · p. 21

C.1 Recursive and Cyclic Call Graphs

  • For cyclic call graphs, SPECFORGE contracts mutually recursive entities into meta-entities and jointly refines their specifications.A meta-entity becomes resolved once all its members are solved.Source: §C.1 · pp. 22–23

C.2 D-UCB Bandit Details

  • The D-UCB bandit normalizes improvement by the remaining pass-rate gap, estimating gap-closing rate rather than rewarding large gaps alone.Its shared statistics are discounted between updates.Source: §C.2 · p. 23

Hyperparameters.

  • A discount factor of 0.95 gives an effective observation window of about 20 steps, allowing adaptation as entity difficulty shifts.Untried entities receive gap-proportional priority.Source: Hyperparameters · p. 23

C.3 Expansion Implementation Details

The case study contrasts the refinement traces and entity-level pass rates of three methods. It illustrates iterative refinement’s inability to redirect effort to a blocking callee and ACE’s regressions after global rewrites.

C.3 Expansion Implementation Details

  • Each expansion uses distinct failure batches, with roughly half the failures per batch when possible, then performs diagnosis followed by a targeted specification update.When too few failures exist for W distinct same-sized batches, the method uses distinct nonempty subsets of varying sizes.Source: §C.3 · pp. 23–24

D Case Study: Module-Level Search Dynamics

D.1 Iterative Refinement: Diagnosis Without Action

  • Iterative refinement correctly diagnosed a broken callee but could not redirect effort to it, leaving the caller unsolved and the module in a plateau.At B=20, LineBreak.get ohlc data improved from 25% to 100%, but subsequent caller attempts remained blocked by the unsolved PnF.get ohlc data.Source: §D.1 · p. 25

D.2 ACE: Regression and Recovery Tax

  • ACE’s global rewrite at B=30 lowered the module score from 0.670 to 0.604 by regressing two previously solved entities.Two later budget steps were spent recovering; the score reached 0.717 at B=50, while both Renko entities remained near zero.Source: §D.2 · p. 25

D.3 SpecForge: Callee-Before-Caller Unlocking

SpecForge’s dependency-guided search turns callee refinements into caller progress, while the trace contrasts this with dependency blindness and collateral damage in other approaches.

D.3 SpecForge: Callee-Before-Caller Unlocking

  • At B=10, SpecForge solved the leaf LineBreak.get ohlc data entity in one step, but at B=20 the module score held at 0.679 while the archive explored alternatives.The selected leaf had no unsolved callees; the archive retained its state when no eligible entity improved the frontier.Source: §D.3 · p. 25
  • At B=30, a Renko.price movement bricks failure exposed a brick-count mismatch, and targeted refinement raised that entity’s score from 0% to 50%.The diagnosis identified that the first brick must align the close price with brick size to compute the first open.Source: §D.3 · pp. 25–26
  • At B=40, solving PnF.get ohlc data raised its score from 0% to 100% and simultaneously raised caller PnF.get bar ohlc data from 33% to 100%.The caller improved without a change to its specification because its callee dependencies were resolved.Source: §D.3 · p. 26
  • By B=50, refining Renko.period close bricks from 20% to 100% brought the module score to 0.906.The radar chart shows SpecForge nearly filling the chart while Iterative remains at its B=20 shape.Source: §D.3 · p. 26

D.4 Key Takeaways

  • Iterative refinement repeatedly targeted the same entity with the same diagnosis and no score gain, illustrating dependency blindness.Its reflection identified cross-entity dependencies, but the method lacked a mechanism to act on that structure.Source: §D.4 · p. 26
  • ACE’s global rewriting regressed at B=30 and required two steps to recover, reaching 0.717 at B=50 while entity-specific details remained beyond global rules.This illustrates the collateral-damage pattern of treating the specification bundle as a monolithic artifact.Source: §D.4 · p. 26
  • SpecForge’s callee-before-caller constraint lets resolved callee improvements unlock caller gains, turning dependency coupling into compounding progress.The B=40 PnF cascade demonstrates the structural unlocking effect.Source: §D.4 · p. 26, §D.3 · p. 26

E Fidelity of the Self-Generated Improvement Signal

The signal experiments find that self-generated execution tests are close to ground-truth tests as an optimization target, whereas LLM judgment performs substantially worse. The appendix also describes the refinement and evaluation prompts used in the experiments and baselines.

E Fidelity of the Self-Generated Improvement Signal

  • Generated test oracles use outputs from the reference implementation, so their residual imperfection concerns input coverage rather than output correctness.Different failing-test subsets in error-diversified expansion limit the influence of any one weak or redundant test.Source: E Fidelity of the Self-Generated… · p. 26
  • Replacing self-generated tests with ground-truth tests raised the mean score only from 93.2 to 94.8, a gain of 1.6 percentage points.The authors interpret this near-equivalence as evidence that the loop does not exploit its own tests.Source: E Fidelity of the Self-Generated… · p. 27, Table 2 · p. 27
  • Replacing execution feedback with LLM-as-a-judge scoring reduced performance from 93.2 to 73.6, a decrease of 19.6 percentage points.The authors attribute robustness to execution grounding rather than tolerance to arbitrary noise.Source: E Fidelity of the Self-Generated… · p. 27, Table 2 · p. 27
  • Lowering test coverage from τcov=0.9 to 0.7 or 0.5 kept the mean score above 92, while τcov=0.3 yielded 87.5, 5.7 points below default.At τcov=0.3, the execution-grounded signal still outperformed LLM judgment’s 73.6 by 13.9 points.Source: E Fidelity of the Self-Generated… · p. 27, Table 2 · p. 27

F.1 Stage 1: Diagnosis Prompt

  • The refinement prompts separate failure diagnosis from prescription, asking the agent to identify errors and missing information before revising an entity’s documentation.The prescription is instructed to address the diagnosis while preserving correct information.Source: §F.1 · p. 28, §F.2 · p. 28

F.3 Code Generation Prompt

  • The code-generation and test-generation prompts request implementations for all module entities and diverse tests covering normal, edge, corner, and branch cases.The test-generation prompt supplies entity source code and called dependencies as context.Source: §F.3 · p. 29, §F.4 · p. 29

F.5 Iterative Refinement Prompt (Baseline)

  • The Iterative baseline refines documentation in one step using all error messages simultaneously, unlike SpecForge’s two-stage diagnosis-prescription process.The appendix notes that conditioning on all errors at once can lead to signal corruption.Source: §F.5 · p. 30

F.6 Commercial Agent Prompts

  • Commercial-agent configurations compare one-shot documentation, dependency-graph context, test-driven refinement, and a combination of dependency context and testing.The +Topo prompt gives the dependency graph; +Test prompts agents to generate tests and iteratively refine until they pass.Source: §F.6.1 · p. 30, §F.6.2 · p. 30, §F.6.3 · p. 31

G Baseline Implementation Details

The baseline implementations range from one-pass and iterative documentation systems to monolithic prompt optimizers and commercial-agent variants. DevEval+ provides a 20-module benchmark with diverse sources and nontrivial dependency structure.

G Baseline Implementation Details

  • RepoAgent generates documentation in one pass, DocAgent iterates through multi-agent collaboration, and commercial-agent configurations add dependency context, test feedback, or both.The iterative baseline selects entities randomly or in reverse topological order; GEPA and ACE optimize the full specification bundle monolithically.Source: G Baseline Implementation Details · p. 32

H DevEval+ Benchmark

  • DevEval+ evaluates module-level code generation across 20 Python modules, 247 entities, and 760 tests drawn from DevEval and additional GitHub repositories.The benchmark is designed around complex dependency structures.Source: §H.1 · p. 33

H.1 Overview

  • DevEval+ has an average dependency depth of 4.2 levels, a maximum depth of 10, 12.4 entities per module, and 3.1 tests per entity.Source: §H.1 · p. 33

H.2 Module Composition

  • Table 3 lists each benchmark module’s source, domain, and structural statistics, with a dagger marking modules included in DevEval+-Java.Source: Table 3 · p. 33

H.3 Data Collection and Curation

The benchmark combines curated DevEval and GitHub modules chosen for dependency depth, testability, self-containment, and domain diversity. Its tests are checked against intended behavior, and a 16-module subset supports cross-language evaluation.

H.3 Data Collection and Curation

  • Modules were selected for at least three levels of call depth, testable input-output behavior, few external dependencies, and domain diversity.Ten modules came from DevEval and ten additional modules from open-source GitHub repositories; the latter required existing ground-truth tests and unanimous author agreement on test quality.Source: §H.3 · pp. 33–34

H.4 Dependency Structure

  • Across 20 modules, the average dependency depth is 4.2 and the maximum is 10; 39% of entities are leaves and 30% are roots.This structure allows callee-specification refinements to propagate across multiple levels.Source: §H.4 · p. 34

H.5 Test Suite Construction

  • Ground-truth test suites were checked to pass on reference implementations, reviewed for intended behavior, and checked for dependence only on an entity’s direct callees.Two authors independently reviewed test correctness.Source: §H.5 · p. 34

I New Feature Implementation Setup (SWE-DEV)

This section describes how SWE-DEV instances and input conditions are constructed, then details the test-generation process that guides SPECFORGE’s search. It also discusses generated-test quality and distinguishes SPECFORGE’s recursively improved specifications from related knowledge artifacts.

I New Feature Implementation Setup (SWE-DEV)

  • SWE-DEV instances use traced tests to select and remove target functions from PyPI packages, and conditions are scored by Pass@1 on the instance test file.Easy instances have call-tree depth 1–2; hard instances have depth ≥3 and typically require at least five interdependent functions.Source: I New Feature Implementation Setup… · p. 35
  • PRD+Source gives detailed target-function descriptions, while Spec+Source adds existing-code specifications to minimal target requirements.The PRD can omit behavior needed from existing dependencies, such as DEFAULTS keys and how InputError is constructed; this makes the comparison conservative for Spec+Source.Source: I New Feature Implementation Setup… · pp. 35–36

J Test Case Generation

  • SPECFORGE’s search uses LLM-generated tests rather than ground-truth tests, reflecting settings where comprehensive test suites may be unavailable.Even imperfect tests can expose specification knowledge gaps through their failures.Source: §J.1 · p. 36, J Test Case Generation · p. 36

J.1 Overview

  • Generated tests are run on the reference implementation; only passing tests that directly exercise the target and are not duplicates are retained.This validation makes retained tests consistent with the reference implementation and gives each a distinct diagnostic signal.Source: §J.1 · p. 36

J.2 Coverage-Driven Test Generation

  • For each entity, test generation iterates across normal, edge, corner, and uncovered-branch cases until coverage reaches τcov or progress stalls.The early stop occurs after P consecutive iterations with coverage gain below ε.Source: §J.2 · p. 36

J.3 Quality Considerations

  • Generated-test diagnostics vary: assertion messages may be informative or bare, and inputs may miss edge cases or repeat paths.The authors use diversified error-batch sampling to increase the chance of finding informative failure signals.Source: §J.3 · p. 37

J.4 Hyperparameters

  • Test generation uses GPT-4o, with no-progress patience P = 2 and ε = 0.01 coverage gain.Source: §J.4 · p. 37

K Additional Related Work

  • Unlike related knowledge artifacts typically built once and judged by extraction or answer accuracy, SPECFORGE recursively improves its artifact based on verified behavior induced in the consuming agent.Source: K Additional Related Work · p. 37

L Compute Resources Used

The appendix describes compute setups for proprietary and open-source models and reports that the approach has modest computational demands. It also states how LLMs were used for experiments and manuscript preparation.

L Compute Resources Used

  • The reported setup uses hosted APIs for proprietary models and one eight-H100-GPU node for open-source models, which the authors describe as modest in computational demand.Experiments on the open-source node ran sequentially for consistent runtime conditions.Source: L Compute Resources Used · p. 38

M Use of LLMs

  • LLMs generated responses for specification experiments and assisted with manuscript editing, while authors determined and verified the substantive technical content.Source: M Use of LLMs · p. 38

N Limitations

The paper bounds SPECFORGE’s recursion to specification search with a fixed procedure and reference-validated tests, and reports that its loop evaluation covers 20 Python modules. It also discusses potential societal benefits and risks and records reproducibility details in the checklist.

N Limitations

  • The recursion changes specifications, not the improver: the search procedure is fixed, requires a reference implementation, and was evaluated on 20 Python modules.Other languages appear only as downstream targets in the cross-language transfer experiment.Source: N Limitations · p. 38

O Broader Impact

  • The authors identify improved reliability and utility for coding agents as a potential benefit, while noting that more capable coding systems may lower barriers to harmful software production.They state that the method does not remove the need for verification or human oversight.Source: O Broader Impact · p. 38

NeurIPS Paper Checklist

  • The checklist reports that the authors release code and data and provide experimental details in the appendix to support reproducing the results.Source: §2 · pp. 40–41

2. Limitations

  • The checklist says experiments were repeated across random seeds and that mean and standard deviation are reported in the appendix.The authors also state that limitations are discussed in Appendix N and in sensitivity analysis.Source: §2 · pp. 39, 41

Related papers

Loading 2610.15708v1…