Source-linked AI summary
How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation
Christoph Treude, Sebastian Baltes
TL;DR
The paper asks what coding-agent retrospectives change in a harness and whether those adaptations prevent targeted actions on later same-repository issues. After coding proposals and evaluating individual adaptations, the authors found mixed results: among seven informative events, two adaptations eliminated targeted actions, one reduced recurrence, and four did not.
Problem
The study asks what harness changes retrospectives propose and when implemented adaptations prevent targeted actions on later issues from the same repository.
Source: §I · p. 1Method
The authors coded retrospective proposals and transferred individual adaptations to later issues in the same repositories, tracking whether the problem arose and whether behavior changed.
Source: §I · p. 1, §III · p. 2Results
Among seven informative events, two adaptations eliminated targeted actions, one reduced recurrence, and four did not; 18 of 43 lint proposals recommended practices regardless of session events.
Source: §IV · pp. 3–4, §V · p. 4Takeaways & Limitations
Evaluating an adaptation requires checking whether its target problem arose, whether the agent consulted it, and whether actions changed; retained changes need continued evaluation.
Source: §VI · p. 4Targeted issue selection may overestimate usefulness on unselected tasks; five sessions across seven informative events do not establish why adaptations differed.
Source: §V · p. 4
Abstract
from arXiv · showhide
Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on later tasks remains unclear. We study \retro, a skill that asks an agent to review a completed session and propose harness changes, and ask what changes it proposes and whether implemented changes reduce repeated work on later tasks. We ran Claude Code on 30 SWE-bench Verified issues, qualitatively coded the 125 proposals from \retro, implemented 24, and evaluated 12 harness changes on later issues from the same repositories in fresh sessions with and without each change. Most proposals referred to an event in the session, but 18 of 43 lint-related proposals recommended a practice regardless of what had happened. Among the seven evaluations in which sessions without the change encountered the targeted problem, two changes eliminated the targeted action, one reduced its occurrence, four showed no reduction, and some changes added work on unrelated issues. Harness self-evolution requires more than generating and implementing changes: it needs ways to establish whether a change helps on later tasks and to revise or remove changes that do not.
I. INTRODUCTION
Harness self-evolution retains changes from completed sessions for later tasks, while this study examines what retrospectives propose and whether selected adaptations affect targeted actions on later issues.
I. INTRODUCTION
- Harness self-evolution retains changes from completed sessions for later tasks without retraining, but retention alone does not establish that a change helps.A retained change may describe one task, stay outside a later session’s context, or fail to change agent behavior.Source: §I · p. 1
- The /retro skill reviews a completed session and proposes numbered harness changes for human approval.Claude Code performed sessions, retrospectives, and approved implementations; humans selected proposals, later issues, and outcomes, so the process was not an autonomous loop.Source: §I · p. 1
- The study asks what harness changes /retro proposes and when implemented adaptations prevent targeted actions on later issues from the same repository.The authors examine whether adaptations change agent behavior when the targeted problem arises.Source: §I · p. 1
- Repeated work is operationalized as a later session performing an action an adaptation was intended to prevent, not as proof that the action was unnecessary or task outcomes improved.The study tracks whether later sessions encounter the problem, consult the adaptation, and repeat the targeted action.Source: §I · p. 1
II. BACKGROUND AND RELATED WORK
Earlier work examined pre-session configuration and retained experience, while this study evaluates individual adaptations on other issues from the same repository.
II. BACKGROUND AND RELATED WORK
- Prior configuration studies examined context-file prevalence [3], measured lower median wall-clock time with AGENTS.md [4], and found equal pass rates for random and curated rule files [5].These studies supplied instructions written or generated before the sessions.Source: §II · p. 1
- Experience-retention studies proposed harness edits from traces [6], traced combined changes across later attempts [7], and examined skill loading and instruction-following [8].In a text-adventure experiment, all six memory-update methods achieved less progress than the no-update baseline [9].Source: §II · pp. 1–2
- This study complements prior work by evaluating individual adaptations, one at a time, on other issues from the same repository.Earlier studies also examined whether updated harnesses help and whether agents load and follow instructions.Source: §II · p. 2
III. STUDY DESIGN
The study sampled coding sessions, coded retrospective proposals, selected persistent changes for implementation, and tested adaptations on later same-repository issues under controlled harness conditions.
III. STUDY DESIGN
- The study ran 30 Claude Code sessions, coded 125 /retro proposals, implemented 24, and evaluated 12 adaptations on later issues from the same repositories.Task sessions used Claude Code 2.1.295 with Opus 5.5, a common prompt template, and /retro from source commit b0618bc.Source: §III · p. 2
- The 10-issue exploratory sample was random, but 20 later issues were selected for prior environment problems and a later same-repository issue, limiting generalization of proposal counts.Issues ran in fresh Docker sessions with repository snapshots and dependencies; the prompt asked the agent to modify code and verify changes with existing tests.Source: §III · p. 2
- Two LLM coders classified proposal triggers and mechanisms, agreeing on 109 of 125 for each code type (κ = 0.67 and κ = 0.84, respectively).Eligibility required an experience-driven, persistent, safe change that did not merely fix the original issue and had a criterion for identifying repeated work later; 11 of 24 implementations were evaluable, alongside one author-written comparison adaptation.Source: §III · p. 2
- Each transfer tested an adaptation on another same-repository issue verified to reproduce its targeted problem, with five sessions per harness across 13 events for 12 adaptations.Three adaptations also had controls on unrelated issues to check for added verification work.Source: §III · pp. 2–3
- Recurrence meant performing the targeted action; an event was informative when at least three of five original-harness sessions encountered the problem.The study separately tracked whether an adaptation was loaded or consulted, since reading a file did not establish that behavior changed.Source: §III · p. 3
IV. RESULTS
The results show that most /retro proposals drew on session events, but lint recommendations often did not. Of seven informative transfer events, two adaptations eliminated the targeted action, one reduced it, and four showed no reduction; some adaptations also added work.
Sessions and proposals (RQ1)
- Only nine of 38 executable checks could report failure in the task environment as written.Of the other 29, 28 needed missing tools or were commit hooks that never fired because sessions made no commits; one was a CLAUDE.md pointer instead of installing pytest.Source: §IV · p. 3
- Of 125 proposals, 101 were event-based, while 18 of 19 prior-driven proposals recommended lint checks regardless of session events.Claude Code produced 43 lint-related proposals across 29 of 30 sessions.Source: §IV · p. 3
Transfer (RQ2)
- In five transfer events, the issue environment reproduced the target, but original-harness sessions encountered it in only one event.In the other four, none of the five original-harness sessions encountered the targeted problem.Source: §IV · pp. 3–4
- In four transfer events, recurrence did not decrease because sessions encountered failures absent from the transferred known-failures files.A separate Astropy adaptation reduced recurrence from 5 of 5 to 2 of 5 sessions, but no session encountered the leap-second failure it targeted.Source: §IV · p. 4
- A Sphinx known-failures file was followed by baseline comparisons in four of five adapted control sessions, versus none under the original harness.The file instructed Claude Code to treat failures absent from its list as caused by its own edits.Source: §IV · p. 4
- Task success differed by at most one session between harnesses in every event, and lower recurrence did not consistently shorten sessions.The SymPy median elapsed time rose from 88 to 156 seconds after adding CI lint checks; it fell by 5 to 28 seconds in the other four informative events.Source: §IV · p. 4
- Two adaptations eliminated repeated work, one reduced it, and four showed no reduction among seven informative transfer events; one unchanged case added unrelated work.The two eliminating adaptations reduced recurrence from 5 of 5 to 0 of 5 and from 3 of 5 to 0 of 5 sessions.Source: §IV · p. 4, Table II · p. 4
V. THREATS TO VALIDITY
The evaluation has several limits: targeted issue selection may overstate how often adaptations help on unselected tasks, and small, non-blind evaluations do not establish which adaptation properties explain differences.
V. THREATS TO VALIDITY
- Targeted issue selection may overestimate adaptation benefits on unselected tasks, while five sessions per harness and seven informative events cannot establish why adaptations differed.Implementation could add information beyond proposals, labeling was not fully blind, and adaptations varied in several properties at once.Source: §V · p. 4
VI. CONCLUSION
Most retrospective proposals referred to session events, but lint proposals often recommended practices regardless of what had happened, and implemented adaptations did not always reduce repeated work. Evaluating harness changes requires checking whether the targeted problem arose, whether the agent consulted the change, and whether its behavior changed; the evidence remains limited to one agent, one /retro version, and environment problems.
VI. CONCLUSION
- Most proposals referred to session events, but lint proposals often recommended practices regardless of what had happened; implemented adaptations did not guarantee less repeated work.Source: §VI · p. 4
- Evaluations should establish whether the targeted problem arose, whether the agent consulted the adaptation, and whether its actions changed.Retained changes also need continued evaluation because outdated or incomplete instructions can remain ineffective or add work.Source: §VI · p. 4
- The evidence covers one agent, one /retro version, and environment problems; future work should test other agents and adaptation types, isolate bundled components, and use later issues not selected for the targeted problem.Source: §VI · pp. 4–5