Source-linked AI summary
Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
Zhi Chen, Zhensu Sun, Yuling Shi, Chao Peng, Xiaodong Gu, David Lo, Lingxiao Jiang
TL;DR
The value of tests written by LLM software-engineering agents remains unclear, particularly whether they improve issue resolution or consume interaction budget. The paper analyzes agent trajectories and prompt interventions, finding that testing is mainly a model-dependent process style whose volume has little effect on outcomes in this setting.
Problem
The paper asks whether agent-written tests improve issue resolution or mainly add interaction overhead when their value remains unclear.
Method
The study analyzes six models’ SWE-bench Verified trajectories across testing behavior, feedback signals, and prompt-induced changes in testing.
Results
Agent-written testing is model-dependent, mainly provides observational feedback through value-revealing prints, and prompt-induced changes usually have little effect on task outcomes.
Takeaways & Limitations
In this setting, agent-written tests appear more like a process style or habitual routine than a dependable driver of success.
Takeaways & Limitations
Findings are scoped to SWE-bench under a light scaffold and specific models, while stochastic runs, task differences, and extraction rules may affect results.
Abstract
from arXiv · showhide
Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. For example, GPT-5.2 writes almost no new tests yet achieves performance comparable to top-ranking agents.This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget? To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.
1 Introduction
This study examines how LLM-based code agents write and use tests during repository-level issue resolution, and whether test-writing behavior improves outcomes. Across three research questions, it finds that testing is widespread and model-dependent, primarily observational, and only weakly related to success or prompt-induced outcome changes.
- Motivation: Code agents combine LLMs, external tools, and iterative action–observation loops to edit repositories and attempt end-to-end issue resolution.The paper defines the scaffold as the surrounding tool interface and interaction protocol governing agent actions.
- Motivation: Repository-level agents use existing human-written tests and agent-written tests as validation interfaces during problem solving.Agent-written tests are new artifacts absent from the original codebase, unlike curated human-written suites.
- Motivation: Test generation and execution consume API calls, tokens, and context, potentially reducing budget for debugging and patching when feedback is low-value.The paper identifies this overhead as a possible source of diluted focus and net detriment.
- Study design: The study analyzes optional agent-written testing in SWE-bench Verified trajectories through three questions covering behavior, feedback signals, and intervention effects.RQ1 examines when and how intensively agents test; RQ2 examines test feedback content; RQ3 examines outcome and efficiency changes.
- Findings: Agent-written testing is widespread but model-dependent, mainly observational, and only weakly aligned with task success or prompt-induced outcome changes.The reported findings characterize testing as a process style rather than a dependable driver of success, with value-revealing prints dominating assertion-based checks.
2 Methodology
The study examines agent-written testing through three research questions using SWE-bench Verified and a lightweight bash-only agent scaffold. It compares diverse strong LLMs, extracts test artifacts from trajectories, and evaluates patches with the official benchmark harness.
- Research questions: The methodology is organized around three questions concerning emergent testing behaviors, feedback signals from agent-written tests, and prompting effects on outcomes and costs.These questions are labeled RQ1, RQ2, and RQ3 in the study design.
- Benchmark: The study uses SWE-bench Verified, a human-screened 500-instance subset in which each task includes an issue, repository snapshot, and official evaluation harness.The original benchmark draws resolved GitHub issues from 12 open-source Python repositories.
- Agent scaffold: To isolate base-model behavior, the experiments use mini-SWE-agent with a lightweight bash-only work loop and no dedicated testing function or tool.Agents can create and execute Python test files through bash, while the default testing recommendation is advisory only.
- Models: The model sample comprises representatives from the top six model families on the SWE-bench Bash Only leaderboard, selected using the highest-ranked model from each family.The selection uses a cutoff date of 2025-12-11 and includes claude-opus-4.5 at 74.4% and gemini-3-pro-preview at 74.2%.
- Data extraction and evaluation: Agent-written tests are identified by scanning time-ordered task trajectories for bash file-writing operations, and submitted patches are scored with the official SWE-bench sb-cli tool.The experiments run on the official mini-SWE-agent codebase and use provider or OpenRouter APIs for inference.
3 RQ1: What Testing Behaviors Emerge Under a Light Agent Scaffold?
Under a light scaffold, agent-written testing is common for most models but varies strongly by model and does not simply distinguish resolved from unresolved tasks. Tests typically finish late, unresolved tasks execute them more intensively, and execution failures vary mainly by model.
- Frequency: Test writing is common for most models, but GPT-5.2 writes tests in only 3/500 tasks; resolved and unresolved tasks usually have similar rates.When tests are written, unresolved tasks often produce as many or more distinct test artifacts than resolved tasks.
- Timing: Across models, first test-writing positions average 0.40 for resolved tasks and 0.37 for unresolved tasks, while last positions average 0.78 and 0.80.Gemini 3 Pro starts later at 0.53–0.55, whereas MiniMax M2 and Claude Opus 4.5 start earlier at 0.29–0.35.
- Timing: Unresolved tasks have a slightly larger average test-writing span than resolved tasks, 0.43 vs. 0.38, indicating only modestly more dispersed writing.Test writing typically finishes late, while start time and span are mainly model-dependent.
- Execution: Unresolved tasks average 6.52 test executions versus 4.89 for resolved tasks and 1.70 executions per written test versus 1.46.These aggregate resolved-versus-unresolved differences are statistically significant for ExecCount and ExecPerTest.
- Execution: FailRate varies mainly by model: Claude Opus 4.5 and Gemini 3 Pro are about 7–12%, while DeepSeek v3.2 Reasoner, Kimi K2 Thinking, and MiniMax M2 are about 21–30%.The execution-failure measure captures process-level friction rather than patch correctness.
- Overall pattern: Overall, agent-written testing is better understood as a model-dependent process behavior than as a simple marker of eventual success.This descriptive conclusion motivates examining what feedback the tests provide.
4 RQ2: What Feedback Signals Do Agent-Written Tests Provide?
Agent-written tests primarily provide observational feedback rather than explicit correctness checks: value-revealing prints consistently outnumber assertions, especially for inspecting runtime values and execution outcomes. The assertions that do appear are concentrated in property and exact-value checks, while relational constraints are uncommon.
- RQ2.1 Task-level feedback signal amount: Value-revealing prints consistently outnumber assertions across models, although total feedback volume varies markedly by model.Gemini 3 Pro has lower totals, while MiniMax M2 has much higher totals.
- RQ2.2 Assertion categorization: Across models, assertions are dominated by property checks and exact-value checks, whereas relational or range-style constraints remain uncommon.For Claude Opus 4.5, Gemini 3 Pro, Kimi K2 Thinking, and DeepSeek v3.2 Reasoner, exact checks comprise 40.85–42.91% and property checks comprise 33.62–41.25%.
- RQ2.3 Print types: 69.67–77.48% of value-revealing prints inspect concrete runtime values, intermediate values, or object contents.Exception or execution-status signals account for 19.06–25.00%, while structural-summary inspection accounts for 3.46–6.56%.
- RQ2 synthesis: Overall, agent-written tests function primarily as observational debugging probes rather than strong correctness oracles.The findings describe feedback behavior and do not establish correctness or impact on task resolution.
5 RQ3: How Does Prompting Test Writing Change Observed Outcomes and Costs?
Prompt interventions substantially changed agents’ test-writing behavior and resource usage, but produced little change in final task resolution. Encouraging tests increased overhead for GPT-5.2 without improving resolution, while discouraging tests reduced resource use for high test-writing models.
- 5 RQ3: How Does Prompting Test Writing Change Observed Outcomes and Costs?: RQ3 compares prompts that encourage or discourage new test files, measuring changes in test creation, task resolution, API calls, and token usage.The study uses baseline and intervention runs under the standard and revised mini-SWE-agent prompts.
- 5 RQ3: How Does Prompting Test Writing Change Observed Outcomes and Costs?: The encourage setup targets GPT-5.2 (0.6%) and Gemini 3 Pro (61.1%), while the discourage setup targets Kimi K2 Thinking (97.4%) and DeepSeek v3.2 Reasoner (89.2%).Models were selected according to baseline test-writing rates, with low- and medium-rate models used for encouragement and high-rate models for discouragement.
- 5.1 RQ3.1 How does encouraging or discouraging test writing change observed task outcomes?: 64.4% of GPT-5.2 tasks and 37.0% of Gemini 3 Pro tasks transition from No test to Has test under encouragement.The intervention substantially increases test creation for both models.
- 5.1 RQ3.1 How does encouraging or discouraging test writing change observed task outcomes?: 68.4% of Kimi K2 Thinking tasks and 75.2% of DeepSeek v3.2 Reasoner tasks move from Has test to No test under discouragement.The discourage prompt removes tests at scale for both high test-writing models.
- 5.1 RQ3.1 How does encouraging or discouraging test writing change observed task outcomes?: 83.2% of tasks retain the same final resolution result after intervention, and none of the four models shows a statistically significant outcome shift.All exact McNemar tests report p> 0.05, despite large changes in test-writing status.
- 5.2 RQ3.2 How do API calls and token usage change?: +5.5% API calls and +19.8% output tokens occur for GPT-5.2 under encouragement, without any resolution gain.Paired analyses also find API calls rise by +1.08 per task and input tokens by +21,907, both with 𝑝< 0.001.
- 5.2 RQ3.2 How do API calls and token usage change?: 49.0% and 32.9% reductions in input tokens occur for Kimi K2 Thinking and DeepSeek v3.2 Reasoner, respectively, while Kimi also reduces API calls by 35.4%.These discourage-setting reductions are directionally robust in paired analyses.
- 5.2 RQ3.2 How do API calls and token usage change?: The intervention’s effects are larger on efficiency than resolution: more agent-written tests impose resource costs but do not reliably produce more solves.Effects are robust for GPT-5.2, large and supported for discourage-setting savings, and small or statistically unclear for Gemini 3 Pro.
6 Discussion and Future Work
The discussion argues for targeted, budget-aware testing rather than simply increasing agent-written tests, while cautioning that benchmark scores require narrower interpretation. It identifies non-stationary test quality, validation-budget tracing, and validity threats as priorities for future evaluation and replication.
- Practical implications: Practitioners should make testing targeted and budget-aware instead of simply increasing the amount of agent-written tests.Suggested behaviors include generating one minimal regression test, converting prints into assertions, and running the smallest relevant test slice.
- Future work: Future studies should trace validation budget across test writing, execution, failure inspection, and patch revision.Observability tooling such as LangSmith could help identify when testing helps, which feedback is useful, and when cheaper validation suffices.
- Benchmark interpretation: SWE-bench Verified scores warrant narrower interpretation because contamination and benchmark-design concerns may weaken their meaning.The study does not directly evaluate those benchmark-level claims or test contamination and training-data effects.
- Future work: Future work should evaluate on-the-fly test quality for intermediate repository versions that may later be overwritten.Traditional metrics such as coverage, mutation score, and fault revelation assume a fixed system snapshot and reproducible executions, complicating attribution here.
- Validity and replication: Internal validity is threatened by stochastic decoding, tool and environment nondeterminism, and confounding from task difficulty or interaction length.The study supports transfer through replication-oriented measurement definitions and intervention prompts, while relying on explicit operational definitions and deterministic AST-based extraction.
7 Related Work
Prior work evaluates LLM-generated tests against predefined objectives, while related studies analyze software-agent trajectories beyond final patch success. This literature covers test quality, oracle generation, and intermediate action–observation patterns, trajectory characteristics, and workflow stages.
- Evaluation for LLM-Generated Tests: Prior work evaluates LLM-generated testing artifacts, especially unit tests and assertions, using test-suite quality and model- or prompt-improvement objectives.This includes systems and empirical studies of generated-test quality.
- Evaluation for LLM-Generated Tests: Targeted oracle-generation research focuses on assertions, while surveys organize how requirements artifacts become tests and define criteria for judging test quality.
- Trajectory Analysis of Software Agents: Trajectory-analysis studies examine action–observation patterns, trajectory length, fault-localization accuracy, and workflow stages including localization, patching, and testing.
8 Conclusion
Agent-written testing is better understood as a model-dependent process style than as a dependable driver of success. Test-writing propensity varies sharply across models, and test feedback is dominated by value-revealing prints rather than assertions.
- 8 Conclusion: Agent-written testing is better understood as a model-dependent process style than as a dependable driver of success.This conclusion revisits the intuition that testing helps in high-autonomy settings where writing and running tests is not specified in the prompt.
- 8 Conclusion: Test-writing propensity varies sharply across models.
- 8 Conclusion: Test feedback is dominated by value-revealing prints rather than assertions.