PaperlayerView exact revision on arXiv

Source-linked AI summary

TDD-Agent: Test-Driven Reasoning for Code Generation

Hongyue Yu, Kefan Li, Jiakun Li, Hongzheng Chai, Yuan Yuan, Rui He, Junyi Wei

arXiv:2608.16742v1cs.SEcs.AI

TL;DR

Repository-level code generation still struggles with correctness, while generated tests are often fixed validators that can provide misleading feedback. TDD-Agent generates tests before implementation and iteratively refines both tests and code, consistently outperforming baselines while improving test quality.

  • Problem

    Repository-level self-testing remains open because generated tests can bias feedback and dependencies make test generation substantially harder than at the function level.

  • Method

    TDD-Agent first generates executable tests to clarify behavioral requirements, then iteratively refines both tests and implementation using execution feedback.

  • Results

    TDD-Agent consistently outperforms baselines, while dual-track refinement improves the quality of generated tests alongside code implementation.

  • Takeaways & Limitations

    Test-first generation provides an executable intermediate representation of task intent, and tests can improve concurrently with code through iterative refinement.

  • Takeaways & Limitations

    Evaluation is confined to Python, and the lightweight tool set provides limited semantic access to repository-level information.

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved remarkable progress in code generation, yet ensuring correctness in complex, repository-level tasks remains challenging. Existing approaches often use generated tests as static post-hoc validators, which limits their ability to guide implementation and may introduce misleading feedback when the tests themselves are incomplete or incorrect. In this paper, we introduce TDD-Agent, which operationalizes the test-driven development paradigm for code generation. TDD-Agent first prompts the model to generate executable tests, encouraging it to clarify expected behaviors before implementation, and then performs iterative dual-track refinement over both the generated code and tests using execution feedback. We first isolate the effect of test-first reasoning through a prompt variant TDD-prompt on LiveCodeBench, where it consistently improves upon reasoning-based prompting baselines. Building on this finding, we evaluate the full TDD-Agent framework on RepoEval, a repository-level benchmark, and show that it consistently outperforms retrieval-based and agent-based baselines. Additional analyses show that iterative refinement improves not only code correctness but also the effectiveness of the generated tests, yielding higher pass rates, coverage, and mutation scores, suggesting that tests can serve as evolving reasoning artifacts rather than fixed validators. Our source code is available at https://anonymous.4open.science/r/TDD-Agent-Framework-6370/.

1 Introduction

TDD-Agent adapts test-driven development for code generation by requiring executable tests before implementation and iteratively refining both tests and code. The framework treats test-first generation as a structured reasoning aid and evaluates it on function-level and repository-level benchmarks.

  • Motivation: Existing test-centric approaches often treat self-generated tests as fixed validators, risking test bias and misleading feedback during implementation debugging.Prior work identifies test bias in post-execution debugging and notes that models typically rectify implementation code without verifying test validity.
  • Method: TDD-Agent requires the model to formulate executable test cases before implementation, providing an executable intermediate representation of task intent.Test-first formulation encourages the model to make assumptions about inputs and expected behavior explicit before coding.
  • Method: TDD-Agent is a single-agent framework that retains conversation history and dynamically modifies both code and tests to simulate a coherent TDD process.The design enables iterative alignment between implementation and tests without coordinating multiple agents.
  • Evaluation: The study evaluates test-first reasoning on LiveCodeBench and the full TDD-Agent framework on RepoEval, covering function-level and repository-level generation.RepoEval tasks require repository navigation and dependency understanding.

2 Methodology

TDD-Agent uses a two-phase workflow that first generates executable tests to specify behavior, then iteratively co-refines implementation code and tests using execution feedback. Failed executions trigger joint analysis of code and tests, while successful executions may prompt test strengthening or early termination.

  • Phase 1: Test-First Specification Setup: The framework begins by inspecting repository context and generating an initial unit-test suite U0 before writing implementation logic.This test-first setup requires the agent to disambiguate requirements and define precise behavioral boundaries.
  • Phase 2: Dual-Track Test-Code Co-Refinement: Using U0, the agent generates implementation C0, executes it against the tests, and obtains an execution report E0.This starts the second phase’s iterative execution-and-refinement loop.
  • Phase 2: Dual-Track Test-Code Co-Refinement: When execution succeeds, the agent may strengthen or expand the test suite, or invoke the Finish tool if confident in the implementation’s correctness and robustness.Successful iterations therefore support either further test development or early completion.
  • Phase 2: Dual-Track Test-Code Co-Refinement: When execution fails, the agent analyzes the cause and refines both code and tests rather than treating tests as immutable constraints.This dual refinement distinguishes TDD-Agent from approaches that use fixed unit tests as validators.
  • Phase 2: Dual-Track Test-Code Co-Refinement: The iterative process runs for at most 10 rounds unless the agent invokes the Early Terminator tool.Generated tests are saved temporarily and run with pytest, while implementation code modifies the target file in place.

3 Experiments

Experiments evaluate TDD-based prompting and TDD-Agent on function-level LiveCodeBench and repository-level RepoEval tasks using GPT-5-mini, DeepSeek-V3.2, and Qwen3-Coder-30B-A3B-Instruct. TDD-Agent outperforms baselines, surpasses mini-SWE-agent by its fifth iteration, and achieves superior performance with comparable token consumption.

  • Experimental Setup: Experiments use GPT-5-mini, DeepSeek-V3.2, and Qwen3-Coder-30B-A3B-Instruct across function-level and repository-level code-generation tasks.GPT and DeepSeek are accessed through APIs, while Qwen runs locally on one NVIDIA H20 GPU using vLLM.
  • LiveCodeBench: TDD-prompt asks models to formulate tests before producing implementations, isolating the effect of test-first reasoning on LiveCodeBench.It is compared with CoT, SCoT, Self-Planning, and ICoT prompting baselines.
  • RepoEval: RepoEval evaluation focuses on 455 function-completion problems from eight Python repositories and judges success by execution of the repositories’ existing unit tests.Generated code is integrated into the original repository, and a solution passes only when it passes all associated test cases.
  • Results: TDD-Agent outperforms all baselines on RepoEval, while retrieval-based methods provide relatively modest improvements and agent-based methods achieve stronger gains.The comparison includes RAG, RepoCoder, mini-SWE-agent, and TDD-Agent.
  • Results: By its fifth iteration, TDD-Agent surpasses mini-SWE-agent and exhibits comparable token consumption while consistently achieving superior performance.Increasing iterations yields further performance gains; average token usage and LLM calls are reported for both agents.

4 Analysis

TDD-Agent improves code and test quality through iterative execution-feedback refinement, but model stopping behaviors differ and generated tests can still produce internally consistent yet incomplete solutions. DeepSeek benefits most from continued refinement, while performance gains plateau around iterations 6–7.

  • Stopping behavior: DeepSeek rarely terminates after the first successful self-test, whereas GPT requires self-test success more consistently and Qwen sometimes terminates before passing its generated tests.These stopping patterns help explain DeepSeek’s larger cumulative improvement and Qwen’s smaller but positive gain.
  • Iteration-wise performance: All three models improve with iterative refinement, with DeepSeek achieving the largest overall gain and GPT showing rapid early improvement before stabilizing.Qwen also improves, but its gain is smaller; across models, curves approach a near-plateau around iterations 6–7.
  • Generated-test quality: Pass rate, coverage rate, and mutation score generally improve across iterations, as refinement removes invalid assertions, corrects expectations, and expands execution-path coverage.The findings indicate that iterative refinement improves generated tests alongside generated code.
  • Dual-track refinement: Generated tests are most effective as evolving reasoning artifacts rather than fixed validators because revising tests and code jointly improves their alignment with intended behavior.Dual-track refinement can correct flawed or insufficient tests, whereas single-track refinement cannot reliably address erroneous initial tests.
  • Limitations: Less than 10% of failed cases involve credit-assignment errors in which the agent fixes correct code for hallucinated tests or discards valid tests for erroneous code.This quantifies a recognized limitation of execution-feedback-based refinement.
  • Limitations: Most failures are matched failures: implementations pass generated tests but fail held-out repository tests because the tests incompletely or misaligningly specify required behavior.Code and tests can therefore co-evolve into an internally consistent but incomplete state, leaving untested defects invisible.

5 Related Work

Prior work explores test-guided code generation through interactive test clarification, test-intention-based coverage improvement, and program–test co-evolution. Other approaches emphasize repository navigation, specialized agent roles, or simpler non-agentic workflows.

  • Test-Guided Code Generation: Test-guided methods clarify intent, improve branch coverage, or co-evolve programs and tests through genetic search.TICODER generates candidate tests interactively, INTUT uses test intentions for unit-test branch coverage, and COCOEVO co-evolves programs with test cases.
  • Test-Guided Code Generation: SWE-AGENT uses a custom Agent-Computer Interface to reduce overload from verbose shell outputs.The interface is designed to prevent the model from being overwhelmed by lengthy shell-command outputs.
  • Repository-Level Code Generation: Repository-level agentic approaches integrate precise code navigation, role specialization, or simpler workflows that avoid agentic loops.CODEAGENT combines symbol navigation with testing tools; MAGIS assigns distinct agent roles; AGENTLESS argues for a simpler workflow without agentic loops.

6 Conclusion and Future Work

TDD-Agent operationalizes test-driven development by treating test generation as reasoning before implementation and refining code and tests together. Future work targets stronger test oracles, mutation-guided expansion, and uncertainty-aware termination to reduce false-positive verification.

  • Conclusion: TDD-Agent treats test generation as reasoning that clarifies requirements and defines executable boundaries before implementation.This operationalizes the Test-Driven Development paradigm for code generation.
  • Conclusion: The framework iteratively refines both generated code and tests through a dual-refinement process.Code and tests are refined together rather than treating tests as fixed artifacts.
  • Future Work: Future work may explore stronger test-oracle construction, mutation-guided test expansion, and uncertainty-aware termination criteria.These directions aim to reduce false-positive verification.

Limitations

TDD-Agent’s evaluation is limited to Python, and its lightweight tool set provides only limited semantic access for repository understanding. Extending the framework to other languages requires replacing environment-specific components while retaining its high-level reasoning and iteration logic.

  • Scope of Evaluation Languages: Evaluation is currently confined to Python, limiting validation across programming languages.Adapting the framework requires replacing environment-specific components, such as JUnit for Java or Jest for JavaScript.
  • Limited Repository Understanding: TDD-Agent’s lightweight tool set supports iterative refinement but provides limited semantic access for repository understanding.

A Generation Parameter Settings … Tool Usage Statistics

The experiments specify model-specific generation settings, evaluate test-first reasoning on LiveCodeBench and RepoEval, and analyze TDD-Agent’s resource usage and tool activity. These evaluations cover correctness, refinement behavior, token consumption, wall-clock overhead, and tool usage across models.

  • A Generation Parameter Settings: GPT-5-mini uses gpt-5-mini-2025-08-07, Qwen3-Coder-30B-A3B-Instruct follows official optimal parameters, and DeepSeek-V3.2 uses temperature 1.0.
  • Dataset and Evaluation Metric: LiveCodeBench uses 224 LeetCode-sourced problems released between May 1, 2024 and May 1, 2025, evaluated with execution-based pass@k.
  • Dataset and Evaluation Metric: Sampling generates 10 candidates per problem; multi-stage methods generate 10 reasoning chains before deterministic code generation at temperature 0.
  • Dataset and Evaluation Metric: Removing test-generation instructions from TDD-prompt decreases LiveCodeBench performance, identifying test-first generation as a critical factor.
  • C Detailed information on RepoEval: RepoEval reports detailed results for eight repositories, with repository information provided separately from the performance comparison.
  • Token Usage and Wall-Time: Table 9 reports TDD-Agent’s token usage across iterations for three RepoEval models, with cached tokens measured only for locally deployed Qwen.GPT and DeepSeek are accessed through APIs, preventing precise cached-token counts; caching would further reduce TDD-Agent’s consumption.
  • Token Usage and Wall-Time: Although iterative execution refinement adds time, the resulting wall-clock overhead remains acceptable given the reported performance improvements.Wall-clock comparisons between mini-SWE-agent and TDD-Agent are reported only for locally deployed Qwen to eliminate network latency and external service load.
  • Tool Usage Statistics: Figure 7 summarizes TDD-Agent’s tool usage across three models while excluding Early Terminator because it can be invoked at most once per task.

Failure Case Analysis

Failure analysis identifies premature termination and limited test discrimination as important failure modes, while credit-assignment errors remain below 10%. Generated tests can also expose flawed test oracles when they fail correct implementations.

  • Unmatched failures: Qwen has a relatively larger proportion of unmatched failures and sometimes terminates despite implementations failing generated tests.This behavior is consistent with the Section 4.1 analysis.
  • Credit assignment: 8.10% is the highest credit assignment problem rate across the three models, remaining below 10%.The rate is reported specifically for failure cases.
  • Generated-test diagnosis: In matched failures, tests that pass ground-truth implementations are insufficiently discriminative, whereas tests that fail them indicate potentially flawed oracles.Passing correct code means the tests cannot distinguish incorrect implementations from expected behavior.

D Tool Implementation Details · E All Prompts Used in Experiments

TDD-Agent is implemented as a function-calling framework that compiles tool invocations into command-based actions and iteratively refines tests and implementation using pytest feedback. The paper also documents distinct prompts for function-level, ablation, and repository-level experiments.

  • D Tool Implementation Details: TDD-Agent parses function-calling tool invocations and compiles them into a small set of command-based actions.The framework is implemented as a function-calling agent.
  • D Tool Implementation Details: Each tool is exposed through a JSON schema defining its name, description, argument types, and required fields.Every model response must contain at least one valid tool call.
  • D Tool Implementation Details: Directory Viewer deterministically limits repository context by excluding .git, hiding hidden files by default, sorting names, and capping returned entries.The cap is controlled by the configurable max_results parameter.
  • D Tool Implementation Details: File Reader supports full-file prefixes or inclusive line ranges with 1-based numbering and rejects invalid ranges.These controls constrain the context exposed to the model.
  • D Tool Implementation Details: Artifact Submitter writes generated tests to test_by_agent.py and replaces the corresponding target code in the original repository file.Test Runner executes only the most recently submitted generated test file with pytest and takes no arguments.
  • D Tool Implementation Details: After each Test Runner call, the pytest report is appended to conversation context as execution feedback for revising tests, implementation, context inspection, or termination.Hidden evaluation tests remain unavailable during prediction and are used only for final evaluation.
  • E All Prompts Used in Experiments: Tables 12 and 13 provide prompts for function-level experiments and ablation studies.These tables document the prompts used in those experimental settings.
  • E All Prompts Used in Experiments: Tables 14–17 provide repository-level prompts for the full TDD-Agent, Vanilla, Reflect, and Single-track settings, respectively.The variants are documented separately across the four tables.

F Case Study

The case study illustrates test-first reasoning on a LiveCodeBench problem and generated-test-plus-implementation refinement on a repository-level task. The example solution derives an O(n)-time, O(1)-space strategy by attacking the cheapest enemy, marking the rest, and repeatedly attacking the cheapest enemy.

  • TDD-Agent case study: The repository-level case study presents examples of tests and an implementation generated by TDD-Agent.These examples are provided in Table 20 and Table 21.
  • TDD-prompt case study: The generated tests cover ordinary, single-enemy, multi-enemy, and cannot-start cases, with expected outputs of 3, 5, 6, and 0.The tests include step-by-step energy and point calculations for each scenario.
  • Algorithmic strategy: The algorithm returns 0 when currentEnergy < min_e; otherwise, it attacks the cheapest enemy once, marks all others, and reuses the resulting energy.The resulting total is 1 + floor(net_energy / min_e), simplified to (currentEnergy + sum_all - min_e) // min_e.
  • Algorithmic strategy: The final strategy uses O(n) time and O(1) extra space.The implementation is presented for Solution.maximumPoints(enemyEnergies, currentEnergy).
Loading 2608.16742v1…