Toward Evidence-Based Agentic Software Engineering

Sebastian Baltes; Christoph Treude

Source-linked AI summary

Toward Evidence-Based Agentic Software Engineering

Sebastian Baltes, Christoph Treude

arXiv:2610.35348v1Oct 26, 2026cs.SE

TL;DR

Agentic coding outcomes depend on both models and harnesses, but studies that change them together cannot isolate harness effects. The authors annotated reported configurations and comparisons in 325 experiments. Harness versions, dates, and complete traces were seldom reported, and 35 of 226 harness-comparing experiments never varied the harness alone.

Problem
Because coding-tool outcomes depend on both model and harness, comparisons changing them together cannot isolate harness effects.
Source: §I · p. 1
Method
The authors qualitatively annotated 325 experiments for reported configurations and whether comparisons isolated harness effects across five degrees of freedom.
Source: §I · p. 1, §II · p. 1
Results
Only 15 experiments reported exact versions of every harness, five reported experiment dates, 31 shared traces for every task attempt, and 35 of 226 harness comparisons never varied the harness alone.
Source: §III · pp. 3–4
Takeaways & Limitations
The authors call for reporting all experimental degrees of freedom and for venue tracks using a specified reference-harness release with separately versioned extensions.
Source: §I · p. 1, §V · p. 4
The search could miss eligible papers without an arXiv record or query terms in their titles, and the codebook was not tested on an independent set.
Source: §III · p. 4

Abstract

from arXiv · show

Task success rates and execution costs of agentic coding tools depend on both model and harness. The harness manages context, provides tools, and runs the agent loop. In empirical software engineering (SE) studies, comparisons that change model and harness together cannot isolate their effects. We qualitatively annotated 325 experiments reported in 146 peer-reviewed full papers published between 2025 and 2026. We examined the reported configurations and whether comparisons isolated harness effects, including harness configuration and version, model, tasks, and user interaction. The exact version of every harness used was reported for only 15 experiments. For five, the paper reported when the experiments were run, and for 31, authors shared execution traces for every session. Of the 226 experiments comparing harnesses, 35 never varied the harness in isolation. Based on this analysis, we derive requirements for a reference harness that the SE community can use for experiments. We call on the community to report all degrees of freedom of experiments with agentic coding tools and on SE venues to pilot dedicated tracks or special issues around a specified reference harness release. In these tracks, papers would hold the harness version fixed and vary features through separately versioned extensions to isolate feature effects and test their interactions.

I. INTRODUCTION

Task success and execution costs depend on both the model and the harness, so comparisons that change both cannot isolate harness effects. The paper reviews reporting and comparison practices, then proposes a reference-harness approach for building comparable evidence.

Task success and execution costs depend on both model and harness, and reported studies found that changing harnesses can reverse model rankings [2] [3].Source: §I · p. 1
Existing reporting guidance covers model and harness information only partially, while prior reviews did not report the role of harnesses [2] [4] [5] [6].Source: §I · p. 1
The authors annotated 325 experiments, assessed 11 open-source harnesses, and propose a shared reference release with separately versioned feature extensions for future comparisons.They also propose dedicated venue tracks or special issues using that release.Source: §I · p. 1

II. DEGREES OF FREEDOM OF AN EXPERIMENT

The paper treats harness-effect isolation as a comparison across five degrees of freedom and codes how each changes between conditions. Its framework also records configuration details and distinguishes isolated changes from confounded or unclear comparisons.

The five degrees of freedom are outer harness configuration, inner harness, model, tasks, and who answers agent requests during a session.Source: §II · p. 1
Outer harness settings include mechanisms such as context files, hooks, and MCP servers; inner harness releases can alter outcomes even with the model fixed.The paper notes that exhaustive outer-configuration comparisons are impractical and that isolated component changes do not establish combined-component performance.Source: §II · p. 2
Model records include action-generating and auxiliary models with versions and settings; task records include sets, instances, prompts, and environments, while interaction records who answers requests and when.Source: §II · p. 2
Contrasts are counted by patterns of changed and unclear degrees of freedom, not pairwise comparisons; Figure 1 depicts the coding of 782 contrasts.An experiment may span comparisons in multiple tables or figures, including a benchmark extension using the same harness.Source: §II · p. 2, Figure 1 · p. 2
A changed degree of freedom is the treatment; other simultaneous changes are confounded unless authors attribute the outcome difference, while whole-system attribution does not establish isolation.Source: §II · p. 2
A degree of freedom is coded held fixed when the paper reports no change, even if values are unreported; it is unclear when the comparison leaves a possible change unresolved.Comparing harnesses without identifying the models also makes the model degree of freedom unclear.Source: §II · pp. 2–3

III. REPORTING OF AGENTIC CODING TOOL STUDIES

The study annotated 325 experiments from 146 peer-reviewed papers to assess configuration reporting and whether comparisons isolated harness effects. Reporting was sparse for harness versions, dates, and traces, and many comparisons changed the harness alongside other factors.

The review covered 146 peer-reviewed papers and 325 experiments, using 21 disclosure questions and a codebook to annotate configurations and contrasts.Source: §III · p. 3
Annotation disagreement was 8.5% overall and 12.7% for codes assigned to the five degrees of freedom.Source: §III · p. 3
Only 15 experiments reported exact versions for every harness; five reported experiment dates, and 31 shared traces for every task attempt.Source: §III · p. 3
Of 226 experiments comparing harnesses, 35 never varied the harness alone, while 141 also changed the model or tasks with the harness.Source: §III · p. 4
The search could miss eligible papers without an arXiv record and without a query term in their title.Source: §III · p. 4

IV. REQUIREMENTS FOR A REFERENCE HARNESS

The paper proposes four requirements for a reference harness so researchers can compare feature effects and interactions across studies. These are a pinnable release, separately versioned extensions, support for different models, and open-source availability.

A reference harness should provide a pinnable release that researchers can keep fixed and identify exactly across experiments.Source: §IV · p. 4
Extensions should have separate version identifiers so a feature can change while the core harness release stays fixed.Source: §IV · p. 4
The harness should support open-weight and commercial models, allowing feature comparisons to be repeated with another model.Source: §IV · p. 4
The reference harness should be open-source so researchers can inspect and modify its code.Source: §IV · p. 4

V. FUTURE PLANS

The authors plan an experiment using separately versioned extensions to estimate feature effects and interactions with the pi harness. They also propose SE venue tracks that compare extensions against a shared baseline while holding the harness release and model fixed within each comparison.

The planned pi-harness experiment will test five separately versioned extensions across 16 of 32 combinations in a resolution V fractional factorial design.It assumes interactions involving three or more extensions are negligible.Source: §V · p. 4
The planned experiment will run each selected combination with one open-weight and one commercial model on the same tasks, using executable oracles and a fixed execution environment.Source: §V · p. 4
The authors propose SE venue tracks in which studies compare extensions against a published baseline on selected benchmark tasks, holding the harness release and model fixed within each feature comparison.Source: §V · pp. 4–5