Source-linked AI summary
DPIAgent: Divide, Protocol, Isolate for Agentic Reproduction Test Generation
Hao Liu, Steven Liu, Xin Zhang, Jane Luo, Yu Kang, Jie Wu, Fangkai Yang, Yangyu Huang, Pengfei Gao, Scarlett Li, Yan Lu
TL;DR
Reproduction test generation must diagnose a bug and write a fail-to-pass test, but existing agents often treat these as one monolithic objective. DPIAgent separates the phases with structured handoffs and isolated tools, and outperforms seven baselines across three backbones, reaching 81.76% with DPI alone on GPT-5 and 86.17% with test selection.
Problem
Existing agentic reproduction-test methods mix root-cause exploration and test writing in one loop despite their distinct objectives.
Method
DPIAgent divides exploration from generation, requires a structured diagnosis and test plan at handoff, and isolates phase-specific tools.
Results
DPIAgent outperforms seven baselines across three backbones, reaching 81.76% success rate on GPT-5 with DPI alone and 86.17% with test selection.
Takeaways & Limitations
Architectural structure and backbone capability are complementary axes, with DPI producing its largest absolute gain on GPT-5-Mini.
Takeaways & Limitations
Evaluation focuses on Python repositories, so validation on additional programming languages remains future work.
Abstract
from arXiv · showhide
Reproduction test generation, producing a failing-then-passing test that captures a reported bug, is a critical step in automated software engineering. Existing agentic methods treat this as a monolithic loop, despite the task inherently comprising two subtasks of distinct nature: diagnosing the root cause and writing a fail-to-pass test. Without explicit separation, the agent faces a compound objective with underspecified intermediate goals, leading to goal drift. We propose DPIAgent, a structured agentic framework built on three principles, Divide, Protocol, Isolate (DPI), that mitigates compound-objective ambiguity and goal drift: it Divides the task into single-objective phases of defect exploration and test generation; enforces a handoff Protocol that records the diagnosis and test plan, preventing context loss; and Isolates each phase's action space by tailoring the toolset to its task, preventing irrelevant tools from misleading execution. On SWT-Bench Verified, DPIAgent outperforms seven baselines across three backbone LLMs. With DPI alone it reaches 81.76% success rate on GPT-5, the highest reported among open-source methods, gaining up to 11.88 points over the strongest baseline on GPT-5-Mini; adding test selection further raises it to 86.17%. Our analysis shows that architectural structure and backbone capability are complementary axes rather than substitutes, demonstrating DPI's generalizability across model classes.
1 Introduction
Reproduction test generation requires both root-cause exploration and fail-to-pass test writing, but existing agents combine them in a monolithic loop. DPIAgent separates these objectives, structures their handoff, and improves reproduction performance across backbones.
- Motivation: Reproduction tests must fail on buggy code and pass after the fix, yet they are often unavailable when issues are first reported.Developers also view writing tests as tedious and time-consuming.
- Motivation: Existing agents mix root-cause exploration and test writing in one loop, creating a compound objective with underspecified intermediate goals.This setup is associated with goal drift, while developers naturally localize the cause before writing the test.
- DPIAgent: DPIAgent divides reproduction into defect exploration and test-generation phases with distinct objectives.The framework imposes structure at phase boundaries while preserving an agent loop within each phase.
- DPIAgent: Its Protocol requires an explicit diagnosis and test plan before transition, while Isolate tailors tools to each phase to prevent irrelevant actions.These mechanisms preserve context and reduce action-space distraction.
- Results: 81.76% success rate is achieved on GPT-5 with DPI alone, while test selection raises performance to 86.17%.DPIAgent outperforms seven baselines across three backbones.
- Results: 11.88 points is the largest absolute gain on GPT-5-Mini, supporting complementary roles for architectural structure and backbone capability.The analysis also reports peak pass rates in three times fewer steps and improved localization.
2 Related Work
Related work divides reproduction test generation into fixed workflow pipelines and dynamically controlled agents. Existing agent systems generally retain a monolithic exploration-and-writing loop, while multi-agent coordination can lose information across boundaries.
- Workflow-Based Test Generation: Workflow-based methods decompose test generation into fixed stages such as localization, planning, and generation.Examples progressively add localization, retrieval, reflection, execution-augmented repair, and surrogate-patch selection.
- Agent-Based Test Generation: Agent-based methods dynamically choose tools and termination actions from runtime observations rather than following hardcoded stage transitions.Representative systems include SWE-Agent+, OpenHands, and TraeAgent.
- Agent-Based Test Generation: SWE-Agent+, OpenHands, and TraeAgent all mix exploration and test writing in a single undifferentiated loop.AEGIS separates search and reproduction into two agents but risks information loss during coordination.
3 Methodology
DPIAgent structures reproduction through phased exploration, an explicit handoff, phase-specific tools, and surrogate-patch-based candidate selection. The method generates diverse candidates and ranks those showing fail-to-pass behavior across surrogate fixes.
- Problem Setup: DPIAgent frames reproduction as generating a test suite that fails on buggy code and passes on the golden-patched code.A suite is correct when one test reproduces the issue and other tests show pass-to-pass behavior.
- Defect Exploration: The Defect Exploration phase builds a complete diagnosis before any test is written, using structural code-graph search rather than text matching.The graph captures code dependencies and semantic relations for bug-relevant retrieval.
- Protocol: Protocol requires a structured handoff containing bug location, root cause, behavior differences, reproduction strategy, reference tests, and a test plan.The handoff is produced before entering test generation and is intended to prevent context loss.
- Isolate: Isolate removes exploration tools after transition and replaces them with editing and test-execution tools dedicated to test generation.The system prompt is also reoriented toward the test-writing task.
- Test Selection: Test selection generates six candidate patches from issue variants and uses three surrogate patches to evaluate fail-to-pass behavior.Candidates are filtered by fail-to-pass evidence and ranked by the number of surrogate patches supporting that behavior, with coverage as a tiebreak.
4 Experiments Setup
Experiments evaluate DPIAgent on SWT-Bench Verified against diverse baselines using success, coverage, and change-coverage metrics. The setup spans three backbones and compares candidate-selection behavior against an oracle upper bound.
- Benchmark: SWT-Bench Verified contains 433 validated instances and is used as the primary evaluation dataset.The benchmark is constructed from SWE-Bench issues, fixes, and test cases.
- Baselines: The baseline set spans specialized, lightweight, generalist, ensemble-based, indexed, and terminal-oriented agent architectures.Examples include SWE-Agent, Mini-SWE-Agent, OpenHands, TraeAgent, Aider, Terminus-2, and Claude Code.
- Test Selection: Figure 3 compares surrogate-patch selection with the pass@k oracle as the number of test candidates k varies.N denotes the number of code patches used to rank candidates, and legend percentages report resolve rate at k = 6.
- Evaluation Metrics: Success rate measures issue reproduction, mean coverage measures golden-patch executable-line coverage, and change coverage isolates incremental coverage beyond existing tests.The metrics distinguish touching modified code from actually triggering the bug.
- Implementation Details: Three backbone LLMs are evaluated: GPT-5-Mini, GPT-5, and Claude-Opus-4.5.Results use the official SWT-Bench Docker environment and average three runs.
5 Main Results
DPIAgent improves issue reproduction across backbone models, with gains concentrated in pre-feedback test design and further amplified by candidate selection.
- DPIAgent consistently achieves the highest success rate across all three backbone models.
- 81.76% success rate on GPT-5 is achieved with DPI alone, while test selection raises it to 86.17%.
- +14.16 S on GPT-5-Mini, +9.24 on GPT-5, and +2.07 on Claude-Opus-4.5 are the direct gains over SWE-Agent.
- DPI improves pre-feedback test readiness, with Ready cases converting to fail-to-pass tests at over 90% across configurations.
- Weaker models benefit more because they rarely recover from Not-Ready first edits, whereas stronger models self-correct more often.
- Adding test selection with k=6 and N=3 improves success rate by 8.04 points on GPT-5-Mini, 4.41 on GPT-5, and 2.40 on Claude-Opus-4.5.
6 Ablation Study
Ablations show that Protocol, Isolate, and Divide make distinct contributions, while the executable phase structure is the primary source of DPI’s gain.
- Protocol: Removing Protocol lowers S from 78.98% to 75.51% on Claude-Opus-4.5.
- Isolate: Exposing all tools across phases reduces ∆C by 6.11 points and Cov. by 8.17 points on GPT-5-Mini.
- Isolate: Exposing all tools across phases reduces ∆C by 7.71 points and Cov. by 10.02 points on Claude-Opus-4.5.
- Divide: Collapsing the two-stage design into a single reactive loop reduces S by 9.93 points and Cov. by 13.56 points on Claude-Opus-4.5.
- Attribution: Applying DPI phase structure with standard SWE-Agent tools achieves substantially larger improvement than adding individual tools or prompts.
- Divide: Without the two-phase design, the agent invokes edit tools from the first steps, interleaving test writing with repository exploration.
7 Analysis
The analysis examines how exploration tools affect agent efficiency and how quickly DPIAgent reaches high pass rates relative to baselines.
- Exploration efficiency: 29% fewer total steps result from structured exploration tools across all backbones.Replacing graph-based exploration with str_replace_editor and bash also reduces performance across the reported metrics.
- Exploration tools: On Claude-Opus-4.5, removing efficient exploration tools lowers S from 78.98% to 73.67% and reduces Cov. by 8.54 points.The ablation replaces structured graph navigation with str_replace_editor and bash.
- Pass-rate efficiency: DPIAgent reaches nearly 80% pass rate within 50 steps on GPT-5, while Trae Agent and MiniSWE Agent require over 150 steps and converge lower.The comparison is shown as cumulative pass rate against the step budget.
- Pass-rate efficiency: On GPT-5-Mini, DPIAgent exceeds 70% pass rate within 60 steps, surpassing baselines that need twice as many steps yet converge lower.This indicates faster pass-rate convergence under the reported step budget.
8 Conclusion
DPIAgent structures reproduction test generation through phase boundaries that separate exploration and test writing. The reported evaluations show consistent improvements across backbones and baselines.
- Conclusion: DPIAgent divides reproduction test generation into exploration and generation, uses a structured handoff, and removes wrong-phase tools.These mechanisms implement Divide, Protocol, and Isolate.
- Conclusion: DPI consistently improves reproduction success across diverse backbones and baselines.The conclusion presents this as the central evaluation finding.
- Conclusion: Minimal structural constraints at phase boundaries can yield greater gains than scaling backbone capability alone.The paper presents this as a design blueprint for structured agentic reasoning in software engineering.
A Detailed Methodology of DPIAgent
The detailed methodology instantiates DPIAgent with graph-based repository exploration, a mandatory reasoning protocol, and a controlled transition into test writing. The workflow preserves a full agent loop within each phase while enforcing phase-specific boundaries.
- Exploration Phase: DPIAgent’s exploration phase builds repository understanding before any test is written.It uses SYSPROMPTexp and an exploration tool set combining graph search, file viewing, and shell utilities.
- Exploration Phase: The Repository Unified Graph provides functionality and dependency subgraphs for navigating domain behavior, modules, calls, imports, inheritance, routes, and entry points.The functionality graph represents what the repository does, while the dependency graph represents how components connect.
- Mandatory Thinking Protocol: Every tool call must include structured reasoning covering progress, the current hypothesis, tool-choice justification, and the expected outcome and next step.The protocol is described as the only opportunity to reason explicitly during tool use.
- Exploration Tools: Exploration tools include search_code, str_replace_editor for viewing, and bash for repository inspection and diagnostics.search_code retrieves source code but cannot access test files; viewing test files requires str_replace_editor or bash.
- Phase Transition: The agent must call ready_to_write_test before creating or editing test files, which switches the available tools into test-writing mode.The handoff summary must cover the bug location, root cause, expected and actual behavior, reproduction steps, reference tests, and test plan.
- Task Workflow: The exploration template requires tests that fail on the buggy code and pass after the issue is fixed, without modifying the bug itself.The workflow separates exploration from test writing and explicitly forbids fixing the source code.
B Experiment Setup
The experiment setup adapts multiple agent baselines to reproduction-test generation under shared evaluation expectations. It specifies repository exploration, targeted test construction, failure verification, and restrictions against modifying source code.
- Experiment Protocol: The experiments aim to facilitate reproducibility and fair comparisons through shared datasets, metrics, execution environments, and fail-to-pass criteria.The common protocol uses the Docker-based evaluation harness described in Appendix B.2.
- Baselines: Leaderboard logs provide results for LIBRO, AssertFlip, Otter, Otter++, e-Otter++, and echo, while OpenHands patches are re-evaluated under the authors’ harness.The sources distinguish complete leaderboard logs from released patches requiring independent evaluation.
- Baselines: TraeAgent generates multiple candidate test patches, prunes invalid or duplicated candidates, and selects a final patch through repeated selector-agent majority voting.The adapted baseline combines static review with optional dynamic test execution.
- Test Construction: The reproduction test must use the project’s test framework, assert expected behavior, target the reported scenario, and remain self-contained.Closely related edge cases should also be included when the issue mentions them.
- Test Verification: The generated test must fail on the current code for the bug-related reason and pass after the bug is fixed.Wrong failures such as import or setup errors require revising the test.
- Baseline Configurations: Baseline adaptations include a bash-only Mini-SWE-Agent interface and prompts directing agents to inspect repositories, modify only tests, and verify generated tests.The supplied setup also specifies absolute file paths for tool arguments.
B.2 Evaluation Setup and Metrics
The evaluation uses SWT-Bench’s fail-to-pass reproduction criterion and coverage metrics that distinguish total from incremental patch coverage. Coverage analysis excludes instances with no executable changed lines.
- Success Rate: A test suite succeeds when it contains at least one fail-to-pass test and no test that becomes failing after the fix.Fail-to-pass means failure on the original codebase and passage on the fixed codebase.
- Coverage Metrics: Mean coverage measures the fraction of executable removed and added patch lines covered by generated tests.Executable lines are those run by the original or golden tests on both code versions.
- Coverage Metrics: Instances with no executable removed or added patch lines are excluded from coverage analysis.
- Coverage Metrics: Change coverage restricts coverage to executable patch lines not already covered by the repository’s original test suite.This isolates incremental coverage contributed by generated tests.
- Coverage Metrics: TDD coverage credits coverage only from successful instances, penalizing systems that cover code without reproducing the issue.The metric is normalized by the total number of instances.
C More Analysis
Additional analyses examine localization, repository and issue coverage, tool usage, step efficiency, and test-selection signals. Together, they show where DPIAgent gains arise and how its multi-configuration selection pipeline operates.
- Test Localization: DPIAgent improves file- and function-level localization across all three backbones, with the largest reported gain reaching 24.75 points in function-level Acc@5 on Claude-Opus-4.5.
- Issue Coverage: DPIAgent uniquely resolves 18 GPT-5-Mini, 16 GPT-5, and 8 Claude-Opus-4.5 instances missed by all baselines.These cases typically require understanding code across multiple files.
- Repository-Level Analysis: The pipeline achieves the highest accuracy on most repositories, while Sphinx remains difficult and reaches only 53.6% for GPT-5 in the best configuration.Scikit-learn exceeds 94% across all configurations.
- Tool Usage and Efficiency: Search_code dominates early turns and disappears after ready_to_write_test, reflecting an explicit switch from exploration tools to generation tools.
- Tool Usage and Efficiency: Exploration tools reduce median agent steps by approximately 29% across backbones compared with removing exploration.The reductions are 29.1% for GPT-5-Mini, 29.7% for GPT-5, and 28.4% for Claude-Opus-4.5.
- Test Selection: The resolved count is the dominant selection signal, while F2P filtering provides moderate gains and coverage mainly breaks ties.Resolved-only ranking remains within 0.3–0.4% of the full pipeline across backbones; F2P alone gains 10.3 points over random on GPT-5-Mini.
- Test Selection: DropCode is the most frequently selected source for GPT-5-Mini and GPT-5, whereas the full pipeline dominates for Claude-Opus-4.5.No source exceeds 30% of selections across any backbone, supporting complementary generation strategies.
D Limitations
The evaluation is limited to Python repositories, and the authors identify additional deployment considerations involving inference cost and misuse risks. Generated tests are executed in isolated Docker containers as a mitigation.
- Scope: The evaluation covers Python repositories from SWT-Bench, so validation on additional programming languages remains future work.
- Efficiency: Inference cost scales with repository size, leaving room for more selective context-management strategies.
- Risks and Mitigations: LM-generated tests may perform harmful operations during iterative verification, so the pipeline runs them in isolated Docker containers.
- Risks and Mitigations: Improved test generation could be misused to probe software for vulnerabilities, although requiring known bug reports limits offensive use.