Source-linked AI summary
TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis
Pepe Alonso, Sergio Yovine, Victor A. Braberman
TL;DR
Regression behavior remains under-studied in coding-agent evaluation, which has focused largely on issue resolution. TDAD uses a source-code-to-test dependency map to guide pre-commit verification. Across the reported evaluations, it reduced test-level regressions and improved resolution as an agent skill, while procedural TDD prompting increased regressions in the tested setup.
Problem
Coding-agent evaluations focus mainly on issue resolution, while the frequency of previously passing tests broken by agent patches receives far less attention.
Source: §1 · p. 1Method
TDAD builds a source-code-to-test dependency map so agents can identify and verify at-risk tests before committing, with the map provided as a lightweight agent skill.
Source: §1 · p. 1Results
TDAD achieved a 70% reduction in test-level regressions in Phase 1 and a +8pp resolution improvement as an agent skill in Phase 2.
Source: §7 · p. 7Takeaways & Limitations
Resolution alone is insufficient for evaluating coding agents because regressions and CI/CD failures are among common reasons agent-authored pull requests are rejected.
Source: §2.1 · p. 2Evaluation uses smaller locally run models and SWE-bench Verified’s well-maintained, heavily tested Python repositories, so results may differ for frontier models or other codebases.
Source: §6.4 · p. 7
Abstract
from arXiv · showhide
AI coding agents can resolve real-world software issues, yet they frequently introduce regressions -- breaking tests that previously passed. Current benchmarks focus almost exclusively on resolution rate, leaving regression behavior under-studied. This paper presents TDAD (Test-Driven Agentic Development), an open-source tool that performs pre-change impact analysis for AI coding agents. TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct. The map is delivered as a lightweight agent skill -- a static text file the agent queries at runtime. Evaluated on SWE-bench Verified with two open-weight models running on consumer hardware (Qwen3-Coder 30B, 100 instances; Qwen3.5-35B-A3B, 25 instances), TDAD reduced regressions by 70% (6.08% to 1.82%) compared to a vanilla baseline. In contrast, adding TDD procedural instructions without targeted test context increased regressions to 9.94% -- worse than no intervention at all. When deployed as an agent skill with a different model and framework, TDAD improved issue-resolution rate from 24% to 32%, confirming that surfacing contextual information outperforms prescribing procedural workflows. All code, data, and logs are publicly available at https://github.com/pepealonso95/TDAD.
1 Introduction
The introduction identifies regression behavior as an under-studied complement to issue resolution and presents TDAD, which gives coding agents targeted test-impact information before committing. Its experiments report fewer regressions with TDAD, alongside a contrasting result for procedural TDD prompting and an improvement in resolution when deployed as an agent skill.
1 Introduction
- Regression behavior receives little attention in evaluation despite evidence that regressions and CI/CD failures contribute to rejected agent-authored pull requests.The SWE-bench harness records PASS_TO_PASS tests, but leaderboard rankings do not surface those results.Source: §1 · p. 1
- Agents risk either slow full-suite runs or missing indirect dependencies when selecting tests only near changed files.The vanilla baseline produced 562 pass-to-pass test failures across 100 instances, averaging 6.5 broken tests per patch.Source: §1 · p. 1
- TDAD maps source-code dependencies to tests so agents can identify at-risk tests before committing and self-correct if regressions appear.The map is delivered as a lightweight agent skill queried at runtime.Source: §1 · p. 1
- Test-level regression rate fell from 6.08% to 1.82% with TDAD in Phase 1, with P2P failures dropping from 562 to 155.The controlled comparison used 100 instances with Qwen3-Coder 30B.Source: §1 · p. 1
- Adding TDD procedural instructions to write tests first and then implement, without specifying which tests to check, increased regressions to 9.94%, performing worse than vanilla; procedural instructions without targeted context were counterproductive.Source: §1 · p. 1
- With Qwen3.5-35B-A3B and the OpenCode agent, TDAD improved resolution from 24% to 32% and generation from 40% to 68%, confirming generalization beyond the original setup.Source: §1 · p. 1
- On a 10-instance subset, iterative refinement of TDAD’s configuration raised resolution from 12% to 60% with 0% regression.The paper describes this as an auto-improvement loop.Source: §1 · p. 1
2 Related Work
Related work spans coding-agent benchmarks, regression test selection, graph-based code analysis, and TDD for agents. TDAD adapts graph-based test selection to pre-commit agent guidance, while the paper distinguishes targeted test context from procedural TDD instructions.
2.1 AI Coding Agents and Benchmarks
- Coding-agent benchmarks and systems primarily emphasize resolution, while regression results are often incidental or absent even though the SWE-bench harness runs P2P tests.Ehsani et al. [15] report CI/CD failures and regression among common reasons for rejecting agent-authored pull requests.Source: §2.1 · p. 2
2.2 Regression Testing
- TDAD adapts classical regression test selection to agents: it surfaces tests before commit for self-correction, through a static text file, rather than optimizing post-change CI execution time.Figure 1 depicts the agent querying the test map, verifying at-risk tests, and self-correcting when regressions are detected.Source: §2.2 · p. 2, Figure 1 · p. 2
2.3 Graph-Based Code Analysis
- Unlike graph-based code completion work, TDAD traverses an explicit code–test dependency graph to identify impacted tests.The related work describes code property graphs, GraphRAG, and GRACE as graph-based approaches addressing other code-analysis tasks.Source: §2.3 · p. 2
2.4 TDD and AI Agents
- TDAD retains TDD’s verify-before-commit principle but replaces its procedure with guidance on which tests are at risk; Rehan [11] instead targets agent behavior compliance.Cui [18] reports that long instructions can degrade TDD performance, consistent with the paper’s finding that smaller models benefit more from concise targeted context.Source: §2.4 · p. 2
3 System Design
TDAD indexes repositories into a code–test graph, identifies tests affected by changed files, and exports a static map for an agent to query. Its impact scores combine strategy weights and link confidence, while runtime integration uses lightweight text artifacts.
3.1 Architecture Overview
- TDAD builds a code–test dependency graph, identifies tests affected by changed files, and exports them in a static map for agent use.The agent uses grep on the map to find tests to verify.Source: §3.1 · p. 2, Figure 2 · p. 3
3.2 Graph Schema
- The graph represents repository structure with four node types and five edge types derived from static AST relationships.Source: §3.2 · p. 3
3.3 Indexing Pipeline
- Indexing parses source files, builds graph nodes and edges, and links tests to exercised code using naming, prefix, and directory-proximity strategies.The AST parser extracts definitions, imports, and call targets; graph construction records containment, calls, imports, inheritance, and test links.Source: §3.3 · p. 3
3.4 Impact Analysis
- Four impact strategies score candidate tests using strategy weights and link confidence; duplicate tests keep their highest score before tiering and a configurable cap.The default cap is 50; three weight profiles trade precision for recall in different ways.Source: §3.4 · p. 3
3.5 Agent Integration: TDAD as a Skill
- Agents use a grep-able source-to-test map and a 20-line skill that directs them to fix the bug, find related tests, run them, and address failures.The authors report that simplifying SKILL.md from 107 to 20 lines alone quadrupled resolution from 12% to 50%.Source: §3.5 · p. 4
3.6 Backend Architecture
- NetworkX is the default in-memory backend, removing the Docker dependency; Neo4j remains available for large-scale deployments.Both backends use the same GraphDB interface.Source: §3.6 · p. 4
4 Experimental Setup
The evaluation uses SWE-bench Verified in two phases, with different models and agent setups. The protocol measures issue resolution with fail-to-pass tests and regressions with pass-to-pass tests.
4.1 Benchmark
- SWE-bench Verified is a human-validated subset of 500 GitHub issues from 12 Python repositories; Phase 1 uses 100 canonical instances and Phase 2 uses 25 selected for diversity.Source: §4.1 · p. 4
4.2 Models and Agents
- Phase 1 tested quantized Qwen3-Coder 30B locally on 100 instances; Phase 2 used quantized Qwen3.5-35B-A3B with OpenCode on 25 instances.Phase 1 ran on consumer hardware with a 32K-token context window and up to 15 minutes per instance; Phase 2 changed the model, framework, and agent.Source: §4.2 · p. 4
4.3 Evaluation Protocol
- The Docker harness runs fail-to-pass tests for resolution and pass-to-pass tests for regression, measured at both test and instance levels.Empty patches are excluded from pass-to-pass evaluation; test-level regression rate is total P2P failures divided by total P2P tests.Source: §4.3 · p. 4
5 Results and Analysis
Across two evaluation phases, TDAD paired lower regression rates with contextual test guidance, while procedural TDD prompting alone increased regressions. The results also show a resolution trade-off, gains from deploying TDAD as an agent skill, and improvements from an iterative auto-improvement loop.
5.1 Phase 1: Regression Reduction (100 Instances)
- 70% reduction in test-level regression rate (6.08% →1.82%) accompanied a 72% drop in P2P failures (562 →155) for GraphRAG+TDD versus vanilla.GraphRAG+TDD had 81% fewer failures than TDD-only (799 →155); catastrophic regressions fell to 1, from 3 with vanilla and 5 with TDD-only. On astropy-13977, vanilla failed all 322 P2P tests while GraphRAG failed 12; instance-level rates differed partly because fewer patches were generated.Source: §5.1 · pp. 4–5, Figure 3 · p. 5
5.2 The TDD Prompting Paradox
- TDD prompting without graph context raised regressions from 6.08% to 9.94% and produced 42% more P2P failures than vanilla.The authors attribute this to verbose instructions displacing repository context and more ambitious, unlocalized fixes causing collateral damage; shortening prompts alone did not explain GraphRAG’s improvement. Their practical implication is that, for smaller models, test context outperforms procedure.Source: §5.2 · p. 5
5.3 Resolution vs. Regression Trade-off
- Resolution fell modestly as GraphRAG agents generated more empty patches when the test map indicated high regression risk.The reported resolution change was 31% →29%, alongside an empty-patch rate of 26% versus 14%; no approach produced patches that were both resolved and regressed.Source: §5.3 · p. 5
5.4 Phase 2: TDAD as an Agent Skill (25 Instances)
- With a different model and agent framework on 25 instances, TDAD as a skill raised resolution from 24% to 32% and generation from 40% to 68%.Both approaches had 0% regression on this smaller set; the authors say the test map also helped with codebase navigation and patch generation.Source: §5.4 · pp. 5–6, Table 6 · p. 5
5.5 Auto-Improvement Loop
- Across 15 auto-improvement iterations, generation rose from 28% to 80% and resolution from 12% to 60%, with 0% regression throughout.Four changes were accepted; simplifying SKILL.md from 107 to 20 lines quadrupled resolution from 12% to 50%. Later iterations refined test mapping, while rejected changes and diminishing returns followed convergence by iteration 5.Source: §5.5 · p. 6, Table 7 · p. 6
6 Discussion
The discussion argues that targeted context can matter more than procedural instructions and that regression rate should be reported alongside resolution. It also bounds the evidence by evaluation scale, benchmark and model scope, and measurement and validity concerns.
6.1 Context Over Procedure
- For smaller models, surfacing which tests and files are relevant may matter more than prescribing a workflow.The authors connect this to context quality and recommend prioritizing information density over procedural completeness for agent tools.Source: §6.1 · p. 6
6.2 Regression as a First-Class Metric
- Because resolution alone can reward patches with collateral damage, the authors advocate reporting regression rate alongside resolution rate.They propose net score = resolution rate − α· regression rate, with α > 1 reflecting the asymmetric cost of regressions versus missed fixes.Source: §6.2 · p. 6
6.3 Limitations
- The evaluation covers 100 Phase 1 and 25 Phase 2 instances without formal significance tests; results should be confirmed on the full 500-instance set.Experiments used consumer hardware and locally hosted models; the paper also notes that only smaller local models were tested, Python was the only language, and static analysis cannot capture dynamic dispatch, monkeypatching, or runtime-generated code.Source: §6.3 · pp. 6–7
6.4 Threats to Validity
- Sequential runs could favor earlier configurations, and the fixed 10-instance auto-improvement set risks overfitting.The evidence comes from well-maintained Python repositories and smaller local models; sparse-test projects, other languages, monorepos, and frontier models may behave differently. Test-level regression rate also weights failures equally, and the benchmark’s P2P set may miss affected code paths.Source: §6.4 · p. 7
7 Conclusion
TDAD combines graph-based test impact analysis with a methodology for reducing regressions in AI coding agents. The conclusion reports improvements across models and frameworks, identifies broader evaluation and tool support as future work, and advocates metrics that jointly capture resolution and regression.
7 Conclusion
- Across two models and agent frameworks on SWE-bench Verified, TDAD achieved a 70% reduction in test-level regressions (Phase 1) and +8pp resolution improvement (Phase 2).An auto-improvement loop raised resolution from 12% to 60% with 0% regression.Source: §7 · p. 7
- The auto-improvement loop raised resolution from 12% to 60% with 0% regression.The paper presents this as a demonstration of self-improving tool design.Source: §7 · p. 7
- Future work includes adding multiple-language support, dynamic coverage data for impact precision, and evaluation with frontier models.The proposed model evaluations would test whether the TDD prompting paradox persists at larger scale.Source: §7 · p. 7
- The authors advocate composite metrics that jointly capture resolution and regression so agents are evaluated on net contribution.Source: §7 · p. 7
8 Data Availability
The paper makes its code, patches, evaluation logs, and experiment records publicly available under an MIT license. It also discloses how generative AI tools contributed to the work.
8 Data Availability
- Code, patches, evaluation logs, and 28 experiment records are publicly available under an MIT license.The tool can be installed with pip install tdad and has no dependencies beyond NetworkX.Source: §8 · p. 7
- Generative AI tools assisted with implementation code, experimental-log organization, implementation summaries, and manuscript wording; the author designed and conducted the experiments, interpreted results, and verified the final content.Source: §8 · p. 7