PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skills

Yunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui, Shiwen Ni

arXiv:2609.12012v1cs.SE

TL;DR

Tests shape construction, repair, selection, analysis, and evaluation, but their order, oracle, editable artifact, and execution role differ. This survey organizes 87 research and supporting records around the decision changed by a test and compares mechanisms, tasks, workflows, and evaluation evidence. Its synthesis finds that test passing or skill presence alone does not establish behavioral equivalence, effective feedback, or process adherence.

  • Problem

    Test-driven uses of tests vary in availability, validity, feedback use, and evaluation independence, making their reported evidence difficult to interpret as one uniform process.

    Source: C. Test Availability, Validity, and Use · p. 12, §VIII · p. 16

  • Method

    The survey synthesizes 87 records by test-to-decision mechanism and task, examines agent workflows and skills, and analyzes evaluation protocols without pooling heterogeneous pass rates.

    Source: D. Extraction and Synthesis · p. 4, §VIII · p. 16

  • Results

    Evidence is configuration-dependent: supplying tests does not ensure adherence, lower test-level regression need not mean higher issue resolution, and a TDD skill's presence does not establish utility.

    Source: §VIII · p. 16

  • Takeaways & Limitations

    Comparisons of test-driven capabilities should track intermediate mechanisms and use trustworthy oracles, independent outcomes, and explicit denominators.

    Source: §VIII · p. 16

    The review's evidence remains configuration-dependent, and skill presence alone does not establish adherence or utility.

    Source: §VIII · p. 16

Abstract

from arXiv · show

Tests increasingly participate in the decisions made by large language models and software engineering agents. They specify intended behavior, guide program construction and repair, select candidates, constrain transformations, and provide execution evidence for software analysis. These uses draw on test-driven development, yet differ substantially in test order, oracle availability, editable artifacts, and the role of execution. We present a structured scoping survey organized around the question of what decision a test changes. The review integrates 87 research and supporting records, with method- or protocol-level extraction for 83 records, alongside a separate collection of five practice resources. We distinguish the Red--Green--Refactor cycle from test-conditioned generation, execution-guided refinement, test-mediated analysis, and evaluation-only testing. We then compare code generation, repair, translation, refactoring, clone detection, code search, localization, training-data construction, and formal-specification validation. A dedicated analysis examines how agent workflows and reusable skills encode testing procedures and how their effects are evaluated. Across these tasks, the evidence supports treating test availability, test validity, feedback use, and evaluation independence as separate properties. Test passing alone does not establish behavioral equivalence, effective feedback, or process adherence; aggregate improvements can also conceal different outcomes across models, tasks, and denominators. We synthesize these distinctions into a mechanism taxonomy, a cross-task comparison, and a protocol-sensitive evidence analysis, and identify research directions in oracle validation, causal evaluation, long-horizon maintenance, and reusable test-driven agent capabilities

I. INTRODUCTION

Tests can shape software decisions before, during, or after development, but those uses do not all constitute the Red–Green–Refactor process. The survey organizes test-driven approaches by the decisions tests change and distinguishes full TDD from neighboring mechanisms.

I. INTRODUCTION

  • Tests can clarify requirements, guide implementation changes, reject damaging transformations, or determine evaluation scores, depending on when they enter the process.These placements correspond to different decisions rather than one uniform test-driven workflow.Source: §I · p. 1
  • Executable evidence also supports translation, clone classification, and issue localization, where outputs may be labels or ranked locations rather than revised programs.Restricting the survey to whether tests precede code would miss these analysis roles.Source: §I · p. 1
  • Its synthesis covers test mechanisms, software tasks, agent workflows and skills, and evaluation evidence, then brings these dimensions together in a research agenda.Individual studies can span multiple dimensions in the taxonomy.Source: Figure 1 · p. 2, §I · p. 2
  • The survey compares approaches by the decision a test changes, the artifact changed, expected-behavior authority, and model permissions.This common extraction unit permits comparison across tasks without assigning every test the same development phase.Source: §I · p. 2

A. Operational Meaning of Test-Driven

  • Full TDD requires a relevant failing test before implementation, followed by passing behavior and structural improvement that preserves behavior.Test-first preparation alone does not establish an observed Red step, and refinement alone does not establish refactoring.Source: A. Operational Meaning of Test-Driven · pp. 2–3
  • Test-conditioned generation, feedback-guided repair, candidate selection, test-mediated analysis, and evaluation-only testing are distinct mechanisms classified by experimental setting.A paper may include several configurations, so its title alone does not determine the classification.Source: A. Operational Meaning of Test-Driven · p. 3

B. Research Questions and Coding Dimensions

The review asks how tests affect software-engineering decisions and what evidence supports claims about outcomes and agent procedures. It combines a structured scoping search with operational coding and protocol-sensitive synthesis, while treating its collection as scoped rather than exhaustive.

B. Research Questions and Coding Dimensions

  • Five research questions cover mechanisms, test and oracle responsibilities, test-mediated decisions, agent workflows and skills, and protocol-supported conclusions about outcomes and cost.The 33-field extraction schema records task, artifact granularity, test source, oracle origin, mutability, ordering, phase evidence, evaluation isolation, outcomes, and other dimensions.Source: B. Research Questions and Coding… · p. 3

C. Search and Source Selection

  • Search combined topic and named-method queries, author leads, and reference tracing, but the resulting totals describe the collection rather than database recall.The retrieval process inspected primary sources but did not produce a complete bibliographic-database export.Source: C. Search and Source Selection · pp. 3–4
  • The review retained 87 research and supporting records alongside five separate practice resources, using eligibility based on a test-to-decision relationship.The collection uses English-readable primary sources and includes relevant earlier code models with their model class identified.Source: C. Search and Source Selection · p. 4

D. Extraction and Synthesis

  • The collection assigns each record one primary corpus role: 43 concern code construction or modification, six analysis or data construction, nine phase or test-asset support, ten evaluation support, 11 skills, and eight surveys.These categories are not counts of complete TDD implementations.Source: D. Extraction and Synthesis · p. 4
  • Method- or protocol-level extraction covers 83 records, while the coverage map recodes 56 operational or phase-specific records and excludes several support-only categories.Blank map cells mean unassigned coverage, not demonstrated absence; the paper-level ledger records inclusion and role assignments.Source: D. Extraction and Synthesis · p. 4
  • Synthesis identifies each test-to-decision relationship, compares it across tasks, and inspects empirical contrasts by configuration without pooling heterogeneous pass rates.The review has no independent duplicate screening or coder-agreement measurement, so retrieval coverage and prevalence claims remain scoped.Source: D. Extraction and Synthesis · p. 4

III. TDD PHASES AND TEST-DRIVEN MECHANISMS

The mechanisms span test design and failure validation, conditioning and feedback after candidate generation, candidate selection, and refactoring under behavioral constraints. Their evidence depends on test validity, feedback content, execution order, and the scope of observed behavior.

A. Test Design and Red: Establishing the Behavioral Target

  • A useful Red step requires distinguishing a failure that exposes the intended defect from uncompilable tests or environmental failures.Bug reproduction methods generate or adapt tests, but their outcomes differ in whether they reveal the reported behavior.Source: A. Test Design and Red: Establishing… · p. 5
  • Self-generated tests depend on expected-answer reliability, so input proposer, output authority, and validation should be recorded separately.Agreement among generated candidates can help select tests, but correlated candidates may share a misconception.Source: A. Test Design and Red: Establishing… · p. 5

B. Green: Conditioning, Feedback, and Selection

  • Test-conditioned generation supplies examples or assertions before a candidate exists, whereas execution-guided revision returns feedback after generation.These configurations alter available information but do not alone show that a test-first process caused an improvement.Source: B. Green: Conditioning, Feedback, and… · p. 5
  • Feedback can range from Boolean failures and assertion messages to stack traces or execution-state traces, with richer signals also potentially longer, noisier, or more expensive.Its value should be judged by the action it enables rather than by information volume alone.Source: B. Green: Conditioning, Feedback, and… · p. 5
  • Candidate selection uses tests to rank or reject candidates without necessarily editing them, and fair comparisons must account for search resources and recovery from poor directions.CodeT selects through agreement on generated tests; other systems use validation within larger pipelines.Source: B. Green: Conditioning, Feedback, and… · p. 5
  • Training-time execution signals must be distinguished from information available to a deployed agent solving a new task.A model benefiting from reference-based training does not imply that the same oracle is available at inference.Source: B. Green: Conditioning, Feedback, and… · p. 5

C. Refactor: Preserving Behavior While Changing Structure

  • Refactoring requires both structural improvement and preservation of relevant behavior; passing tests alone provide evidence only for tested behavior.Differential fuzzing can expose discrepancies missed by existing suites, while no discrepancy in a finite run does not establish universal equivalence.Source: C. Refactor: Preserving Behavior While… · p. 5
  • Test-code and testing-framework refactoring change verification assets, not necessarily production code through the Refactor phase.Their invariants may concern assertion meaning, setup behavior, or framework semantics.Source: C. Refactor: Preserving Behavior While… · pp. 5–6

D. Cross-Cycle Organization and Coevolution

Cross-cycle workflows differ in how they retain state, sequence actions, and let tests or patches evolve. The survey also treats execution as evidence for analysis and data construction, while code-generation settings vary in specification, test access, and success criteria.

D. Cross-Cycle Organization and Coevolution

  • Cross-cycle workflows vary in increment size, retained state, and action order, so tool access alone does not establish when or whether tests were used.When tests and patches coevolve, success after test edits may reflect either an improved specification or a weakened obligation.Source: D. Cross-Cycle Organization and… · p. 6
  • Long-horizon maintenance benchmarks motivate evaluating growing behavioral obligations because locally plausible changes can accumulate broader degradation.They extend evaluation across releases and repeated requirements rather than treating episodes as independent.Source: D. Cross-Cycle Organization and… · p. 6

E. Executable Evidence for Analysis and Data Construction

  • Execution supports analysis judgments and data construction through outputs, coverage, traces, and dependency relations, not only pass/fail assertions.Examples include clone classification, code localization, compiler-bug localization, and constructing development tasks or training examples.Source: E. Executable Evidence for Analysis… · p. 6
  • Behavior-informed learning can use execution signals without a missing implementation or failing requirement test, keeping it distinct from the Red–Green–Refactor process.Fuzzed input/output examples can enrich code-model inputs for clone retrieval and classification.Source: E. Executable Evidence for Analysis… · p. 6

A. Code Generation, Requirements, and Security

  • Generation settings vary with artifact granularity and dependencies, while interactive or role-based workflows incur costs for trustworthy specifications and coordination.A paired-intervention study tests specification use by changing expected outputs between compatible specifications and checking behavior on unseen inputs.Source: A. Code Generation, Requirements, and… · p. 6
  • Generation success includes unseen-input behavior, joint functional and security requirements, and sensitivity to intended specification changes, not just visible pass rate.Passing supplied examples may not implement the general rule, while incomplete execution environments may cause failures despite correct behavior.Source: A. Code Generation, Requirements, and… · p. 6

B. Bug Reproduction, Repair, and Performance Optimization

Tests guide repair and other software changes, but their role varies across tasks: they can provide repair feedback, validate translations, or assess preservation and maintenance. The section distinguishes those uses by the behavior or property being checked.

B. Bug Reproduction, Repair, and Performance Optimization

  • Reproductions support repair by establishing useful failures, while conversational and agentic workflows differ in how they inspect, edit, and validate patches [33], [35], [36], [40], [63], [64], [42], [45].Comparisons should separate defect localization from patch editing or selection.Source: B. Bug Reproduction, Repair, and… · p. 7
  • Optimization adds an explicit performance objective to functional preservation: PerfRL ranks optimized code using execution-based correctness and timing signals [52].A faster candidate that violates required behavior is unacceptable, while an unchanged correct candidate may make no optimization progress.Source: B. Bug Reproduction, Repair, and… · p. 7
  • Regression checks matter because a patch can fix a new failure while breaking unrelated behavior; TDAD provides relevant test context [29].The review calls for measuring severity, incidence, and issue resolution.Source: B. Bug Reproduction, Repair, and… · p. 7

C. Translation, Migration, Refactoring, and Maintenance

  • Executable source behavior supports translation checks, while SpecTra first validates specifications against source execution or existing tests before using them to condition translation [10], [11], [77].UniTrans generates test inputs and uses source execution as the expected behavior; TransAgent adds fine-grained execution alignment.Source: C. Translation, Migration,… · p. 7
  • DepWareTrans orders repository-migration batches by dependencies and checks the mixed-language repository after each batch [78].Its setting relies on languages that can coexist during migration.Source: C. Translation, Migration,… · p. 7
  • Refactoring evaluation needs both behavioral preservation and evidence that the requested structural change occurred [54], [55], [56].Differential fuzzing can add an observation channel when existing suites miss a semantic discrepancy.Source: C. Translation, Migration,… · pp. 7–8
  • Long-horizon maintenance treats tests as both accumulated specifications and assets to maintain; RepoMod-Bench’s hidden tests measure modernization outcomes, not whether tests guided agents [67]–[69], [73].Redundant tests, fragile fixtures, or changes that hinder later tasks can add costs; SWE-CI, SWE-EVO, and SlopCodeBench examine extended settings.Source: C. Translation, Migration,… · p. 8

D. Semantic Code Clone Detection

The section examines tests as behavioral evidence for clone detection, code-search annotation, localization, and task construction. Across these settings, the decision and the scope of the evidence differ, so evaluation must match the task.

D. Semantic Code Clone Detection

  • On POJ104, FuzzTuning’s clone-retrieval MAP@R changes from 84.29 to 92.01 for CodeBERT and from 90.52 to 93.40 for UniXcoder; its retrieval protocol differs from HyClone’s pairwise classification [25], [12].FuzzTuning adds fuzz-generated input/output examples to source code during adaptation; the scores do not establish which approach is more effective without a shared task and protocol.Source: D. Semantic Code Clone Detection · p. 8

E. Code Search and Relevance Annotation

  • Code-search tests help produce a relevance label, but the test must operationalize the query correctly before it can provide evidence about candidate code [70].Annotation accuracy, agreement with human verification, and downstream retrieval quality are separate evaluation targets.Source: E. Code Search and Relevance Annotation · p. 8

F. Issue and Compiler-Bug Localization

  • IssueExec uses passing-test traces to locate issue-relevant code, whereas LLM4CBI validates generated witness-program variants and uses passing/failing executions for compiler-bug isolation [13], [71].Localization rankings should be evaluated at their reported granularity and separately from downstream repair.Source: F. Issue and Compiler-Bug Localization · p. 8

G. Training-Data and Development-Task Construction

  • SWE-Flow uses execution to recover dependencies and construct development tasks, so its synthesized sequence is not evidence that original developers followed TDD [72].Evaluation should examine both task fidelity and model performance on the constructed tasks.Source: G. Training-Data and Development-Task… · p. 8

H. Formal-Specification Validation

Formal-specification validation treats generated tests as verification assets, while cross-task comparisons show that test roles depend on the oracle and decision involved. Agent testing procedures also differ in who controls them, what they prescribe, and whether they are executed or enforced.

H. Formal-Specification Validation

  • Formal-specification validation asks whether tests are valid, represent requirements, and distinguish correct from incorrect specifications, without establishing end-to-end model repair.Its immediate contribution is a verification asset for test-driven modeling.Source: H. Formal-Specification Validation · pp. 8–9

I. What Transfers Across Tasks?

  • A test-driven workflow transfers across tasks only when it preserves the test’s meaning and the decision it informs; an unchanged prompt sequence can alter its substantive guarantee.The oracle’s behavioral authority varies across generation, translation, clone detection, localization, and formal modeling.Source: §I · p. 9

A. Who Controls the Procedure?

  • Prompts, fixed workflows, autonomous agents, multi-agent systems, and human collaboration place control of testing at different points in the procedure.Evaluation must establish that agents selected and used testing capabilities; separate role names do not establish independent reasoning.Source: A. Who Controls the Procedure? · p. 9

B. Content, Invocation, and Enforcement

  • A TDD skill can be assessed both as reusable procedural knowledge and as a component that an agent selects and executes.Its description should specify triggers, targets, ordering, allowed edits, test commands, refactoring objectives, and stopping conditions.Source: A. Who Controls the Procedure? · p. 9, B. Content, Invocation, and Enforcement · p. 9
  • Written TDD skills do not prescribe identical phase coverage: Superpowers and Addy Osmani specify Red–Green–Refactor, while Matt Pocock separates refactoring into review.Probity describes operation-level checks that can block actions, but the inspected resources are mutable snapshots and their design descriptions are not reproduced effectiveness results.Source: B. Content, Invocation, and Enforcement · p. 9
  • Repository instructions can increase test execution and tool use without a corresponding improvement in issue resolution, so presence, invocation, adherence, and benefit require separate observation.The AGENTS.md study compared no context file with model-generated and developer-provided context.Source: B. Content, Invocation, and Enforcement · p. 9
  • Graph-based TDAD combines TDD guidance with impacted-test and code-relationship information, making test context another intervention beyond instruction text.The proposed contrast is text-only guidance versus guidance augmented with retrieved execution context.Source: B. Content, Invocation, and Enforcement · pp. 9–10

I. What Transfers Across Tasks?

  • 24 of 56 mapped records primarily address generation and 15 address repair or reproduction, where feedback-guided revision is prominent.Analysis rows concentrate on execution-supported judgments; these collection counts do not establish exhaustive field coverage or effectiveness.Source: §I · p. 9, Figure 2 · p. 10

C. Effectiveness, Cost, and Optimization

Evaluations of TDD-related skills report differing outcomes across settings, representations, and task populations. Testing also shapes agent and skill artifacts themselves, but artifact-level results do not by themselves establish task-level benefit or generalization.

C. Effectiveness, Cost, and Optimization

  • With Claude Code and Haiku 4.5, tdd-workflow pass rate rose from 21.4% to 28.6% on 14 tasks as average token cost rose from 83K to 148K; springboot-tdd pass rate fell from 80.0% to 70.0% on ten tasks as tokens fell from 374K to 236K.These small, setting-specific contrasts establish neither a universal benefit nor a universal cost penalty for TDD skills.Source: C. Effectiveness, Cost, and Optimization · p. 10
  • Broader skill benchmarks and optimization studies support measuring multiple outcomes, but do not substitute for TDD-specific treatment comparisons or establish effectiveness for every observed skill.Skill-corpus prevalence, benchmark utility, and optimization results are distinct evidence categories.Source: C. Effectiveness, Cost, and Optimization · p. 10
  • For tdd-workflow with Qwen3.6-27B/OpenCode, solved tasks out of ten were one with no skill, one with full skill text, and four with the adapter.The runtime cost measure is per-step tokens, excluding offline synthesis and training; the experiment supplied the skill identity and did not establish retrieval accuracy or TDD adherence.Source: C. Effectiveness, Cost, and Optimization · p. 10

D. Developing Agent and Skill Artifacts with Tests

  • Test-Driven AI Agent Definition uses behavioral tests to revise system prompts and assess them through mutations, rather than revising production-code implementations.A test-driven procedure can improve an agent definition, but transfer to a specific SKILL.md package requires evaluating invocation and use; artifact tests and repository repair are different outcome levels.Source: D. Developing Agent and Skill… · pp. 10–11
  • MUSE-Autoskill reports tests/ directories in 9% of generated packages, a 87.94% conditional score on 35 skill-yielding tasks, and a 60.35% score across all 51 tasks.Skills were created from successful runs and reevaluated on the same tasks, so new-task generalization requires a different evaluation.Source: D. Developing Agent and Skill… · p. 11
  • SKILLER revises textual skill bundles using verifier-grounded feedback, but its springboot-tdd case is qualitative and supplies no separate average effect for that skill.The optimizer can access a reference trajectory and benchmark feedback; held-out GAIA and EarthBench splits address a different transfer question from skill-benchmark evaluation.Source: D. Developing Agent and Skill… · p. 11

VI. EVIDENCE, EVALUATION, AND VALIDITY

Benchmarks establish different things depending on test access, feedback, task horizon, and measured outcome. Configuration-specific comparisons show that results can vary across benchmarks and denominators, so they require protocol-sensitive interpretation.

A. What Does Each Benchmark Establish?

  • Issue-test benchmarks can establish whether a generated test distinguishes an unresolved issue from a repaired version, but not whether an agent then repairs it or first constructed a failing test.Supplied tests assess satisfying an available executable specification, while hidden tests assess generalization beyond construction checks; RepoMod-Bench has no iterative feedback channel.Source: A. What Does Each Benchmark Establish? · pp. 11–12
  • Interactive execution benchmarks expose agent-feedback exchanges, while long-horizon benchmarks examine accumulated regressions and maintenance across sequences of changes.Success on one isolated change does not determine how test assets, dependencies, or earlier design choices affect later changes.Source: A. What Does Each Benchmark Establish? · p. 12

B. Comparisons Depend on the Experimental Configuration

  • Pass-rate changes, affected-test failure fractions, and rule-following rates have different denominators; cross-study comparison should preserve each configuration’s test source, authority, and editable object.The common schema leaves unestablished properties open rather than assigning identical phase labels to test selection, code repair, and prompt revision.Source: B. Comparisons Depend on the… · p. 12
  • On Graph-based TDAD’s 100-task Phase 1 subset, test-level regression rates were 6.08% for baseline, 9.94% for TDD guidance, and 1.82% for graph-augmented TDD, while issue resolution was 31%, 31%, and 29%.The evaluated test counts differed, and instance-level regression rates did not show the same reduction; the evidence does not establish improved issue resolution or fewer regressing instances.Source: B. Comparisons Depend on the… · p. 12
  • In SpecTra’s 24-function GNU coreutils cat case study, accepted first-candidate translations numbered 10 for baseline, eight for simultaneous specifications, and 11 for staged SpecTra.The counts come from one project case with bounded compiler repair and provide limited grounds for predicting effects across repositories or language pairs.Source: B. Comparisons Depend on the… · p. 12
  • With all tests provided, hidden joint correctness changed from 45.5% to 41.8% on SALLM but from 15.6% to 40.0% on CWEval.The conditions covered 11 and nine tasks, respectively, each repeated with five seeds; joint correctness is distinct from functional correctness and a security-only score.Source: B. Comparisons Depend on the… · p. 12

C. Test Availability, Validity, and Use

The evidence depends on whether tests are available, valid, and used, as well as on the task, oracle, and evaluation boundary. Outcomes and denominators must remain distinct when comparing studies.

C. Test Availability, Validity, and Use

  • 100% control-condition rule following contrasts with a 60.6% test-driven switching rate for Qwen3.6-27B, across 20 task families with six realizations each.The family structure matters: repeated realizations are not independent evidence for hundreds of unrelated task families.Source: C. Test Availability, Validity, and Use · p. 12
  • Generated tests and oracles can share interpretation errors, so reference execution, human confirmation, behavioral mutation, or independent evaluation provide distinct validation checks.More generated assertions can repeat the same missing assumption.Source: C. Test Availability, Validity, and Use · pp. 12–13

D. Behavioral Preservation and Human Outcomes

  • 3,538 filtered refactorings from 4,368 generated candidates include non-equivalent transformations missed by existing tests, limiting the finding to the filtered population.The conditional fraction missed by tests is not a failure rate over all generated transformations.Source: D. Behavioral Preservation and Human… · p. 13
  • A reference-based user proxy supports controlled answers but does not model all developer uncertainty, effort, or misunderstanding; small-group interaction findings do not establish populationwide productivity effects.The studies report correctness, effort, and experience as separate dimensions.Source: D. Behavioral Preservation and Human… · p. 13

E. Cross-Study Findings and Their Implications

  • Test order, information, and feedback are distinct interventions: supplying tests, selecting by execution, and changing behavior in response to tests measure different mechanisms.A test/no-test score difference may reflect clearer specification, candidate selection, or feedback-guided revision; useful execution evidence can also occur without observing Red.Source: E. Cross-Study Findings and Their… · p. 13
  • Oracle authority varies by task: source execution can guide translation, user input can resolve intent, and generated-test agreement can estimate candidate quality without independently establishing requirements.Behavior can support clone classification or representation learning without proving program equivalence.Source: E. Cross-Study Findings and Their… · p. 14
  • Lower test-level regression does not necessarily coincide with higher issue resolution, and joint-correctness contrasts can reverse across benchmarks.Skill-conditioned performance also has a different denominator from performance across all assigned tasks, so populations and units must be retained.Source: E. Cross-Study Findings and Their… · pp. 14–15
  • Skill utility depends on representation and evaluation boundaries, so construction history, repeated runs, cross-agent reuse, and held-out transfer should be reported separately.Studies examine written skills, skill-to-LoRA, verifier-guided revision, and generated packages; their access to skill identity and task feedback differs.Source: E. Cross-Study Findings and Their… · p. 15

VII. RESEARCH GAPS AND AGENDA

The agenda calls for experiments that isolate test-related mechanisms, build task-appropriate oracles, and track test assets as requirements evolve.

A. Separate the Effects of Tests, Ordering, and Feedback

  • Matched studies should vary test availability, observed failure, returned execution feedback, and refactoring objectives while holding model, task, candidate, and budget fixed.Trajectory records can distinguish prescribed TDD, observed adherence, and successful completion by tracking test availability, failures, artifact edits, and suite reruns.Source: A. Separate the Effects of Tests,… · p. 15

B. Develop Oracles That Can Discriminate Competing Interpretations

  • Test suites should distinguish plausible requirement interpretations, with evaluation measuring alternatives eliminated and the human or execution cost of obtaining expected results.Interactive selection, paired-rule evaluation, and mutation-based validation offer complementary approaches.Source: B. Develop Oracles That Can… · p. 15
  • Oracle construction should account for task-specific asymmetries, such as executable source behavior in translation versus partial tests and defect reports in repair.Hybrid designs can use reference execution, ask humans about unresolved requirements, and reserve independent checks for evaluation.Source: B. Develop Oracles That Can… · p. 15

C. Treat Test Assets as Maintained Software

  • Longitudinal evaluations should trace test edits and independent behavioral outcomes as requirements change, measuring retained behavior, test strength, maintenance effort, execution cost, and uncertainty resolution.The agenda frames test quality and agent behavior as evolving together, rather than counting tests generated for a fixed snapshot.Source: C. Treat Test Assets as Maintained… · p. 15

D. Connect Construction and Analysis Through Execution Evidence

The survey connects execution evidence across construction and analysis, then proposes more rigorous evaluation of reusable skills, human collaboration, and transfer. Its conclusion emphasizes protocol-sensitive evidence rather than a universal TDD effect.

D. Connect Construction and Analysis Through Execution Evidence

  • Execution evidence supports clone detection, code search, localization, and training-data construction, but reuse across decisions can reduce reliability and requires task-specific correctness criteria.An observation sufficient for retrieval may be too weak to establish clone equivalence; a task-spanning benchmark could retain distinct criteria for retrieval, classification, localization, and modification.Source: D. Connect Construction and Analysis… · p. 15

E. Evaluate Skills as Executed Procedures

  • Skill evaluation should separately test procedure content, task selection, adherence to order and permissions, and behavior and cost on independently assessed tasks.Matched comparisons can hold procedural content and executor fixed, separate construction from evaluation tasks, and charge creation or training costs to the intended reuse horizon.Source: E. Evaluate Skills as Executed… · pp. 15–16
  • Comparisons of prompts, reusable skills, and externally enforced workflows should jointly consider correctness, regression behavior, and cost, while keeping evaluation tasks separate from skill revision.Studies also vary relevant test context and the agent’s authority to edit tests.Source: E. Evaluate Skills as Executed… · p. 16

F. Study Collaboration and Transfer Beyond Short Tasks

  • Existing human-interaction studies leave sustained adoption open, while transfer across models, languages, and repositories must account for differing frameworks, conventions, and oracle availability.Longer studies could examine test selection, challenges to incorrect expected outputs, review of generated changes, and decisions about further validation.Source: F. Study Collaboration and Transfer… · p. 16

VIII. CONCLUSION

  • Across 87 records, the survey distinguishes full Red–Green–Refactor from other test uses and finds that order, oracle origin, edit permissions, and evaluation access shape what methods establish.The evidence is configuration-dependent: test availability does not ensure use, lower test-level regression need not mean higher issue resolution, and a TDD skill does not establish adherence or utility.Source: §VIII · p. 16
Loading 2609.12012v1…