PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfSearch: Reward-Free Search for Self-Improving Agents

Jungwoo Yang, Injin Kong, Yohan Jo

arXiv:2609.37968v2cs.AIcs.CLcs.LG

TL;DR

Can agents use the experience of self-improvement to become better at improving themselves? SelfSearch uses records of prior self-improvement episodes to guide revisions without downstream evaluation during search, and its final populations improve mean task success across all six model–benchmark settings.

  • Problem

    Whether agents can use experience gained through self-improvement to become better at improving themselves remains an open question.

    Source: §1 · p. 1

  • Method

    SelfSearch guides agent revisions with records of previous self-improvement episodes, without using downstream tasks or evaluation results during search.

    Source: §1 · pp. 1–2

  • Results

    Across all six model–benchmark settings, SelfSearch populations improve mean task success over their initial agents without downstream evaluation during search.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The process of editing an agent can itself provide experience for further agent improvement.

    Source: §7 · p. 9

    Whether combining SelfSearch with evaluation-guided selection improves agents under the same total search budget remains for future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Advances in the coding capabilities of LLM agents allow them to inspect and modify their own instructions, tools, and execution procedures. Existing approaches use this ability to search for improved agents through repeated downstream evaluation, which incurs substantial costs and ties the search to the evaluated tasks. We introduce \textbf{SelfSearch}, a reward-free search procedure in which agents modify themselves using records of previous self-improvement episodes. These records capture the reasoning, tool actions, and outcomes of earlier modification attempts, providing concrete experience for improving both task solving and self-modification. Without downstream reward signals during search, SelfSearch improves population-mean success over the initial agent in all six model--benchmark settings, with individual agents gaining up to 11.2 percentage points on Terminal-Bench 2.1. On SWE-bench Multilingual, an agent improves success by \textbf{5.0} percentage points while reducing execution cost by \textbf{38.5}\% on tasks solved by both the initial and evolved agents. SelfSearch achieves competitive task success with evaluation-guided search baselines at lower search cost. With only \textbf{\$4.03} in search cost, it produces a harness that solves \textbf{82.0}\% of Terminal-Bench 2.1 tasks with DeepSeek V4 Flash under the settings of a public nine-harness comparison, matching the top-scoring harness, Codex. These results suggest that experience gained through self-modification can improve agents' downstream capabilities and efficiency.

1 INTRODUCTION

SelfSearch asks whether agents can learn to improve themselves from the experience of modifying their own implementations, rather than relying on repeated downstream evaluation. It uses records of prior self-improvement episodes to guide revisions and improves mean task success across the evaluated model–benchmark settings.

1 INTRODUCTION

  • Existing self-improvement searches rely on repeated downstream evaluation, tying improvement signals to evaluated tasks and incurring recurring evaluation costs.The cost grows with the number of tasks evaluated.Source: §1 · p. 1
  • SelfSearch is a reward-free procedure in which an agent modifies its implementation using records of previous self-improvement episodes.Its repository can be revised, and the records capture how agents gather information, make changes, and respond to failures.Source: §1 · pp. 1–2
  • The population mean exceeds the base in all six model–benchmark settings, with individual gains up to 11.2 percentage points on Terminal-Bench 2.1.On SWE-bench Multilingual, an agent gains 5.0 percentage points while reducing execution cost by 38.5% on tasks solved by both initial and evolved agents; harnesses also transfer across model families.Source: §1 · p. 2
  • For $4.03 in search cost, a SelfSearch harness achieves 82.0% on Terminal-Bench 2.1 with DeepSeek V4 Flash, matching Codex in a public nine-harness comparison.The comparison uses the same settings.Source: §1 · p. 2
  • Ablations suggest that both previous episode records and an evolving improver contribute to population-mean success.Code changes and execution traces show agents developing reusable tools and repairing failures in tool interactions.Source: §1 · p. 2

2 SELFSEARCH

SelfSearch iterates through self-improvement episodes: each agent revises an editable repository using prior episode records, then becomes the next improver. Search uses qualitative directions and local tool feedback, not downstream benchmark tasks or evaluation results.

2 SELFSEARCH

  • Each episode uses an editable repository copy and read-only records from earlier episodes; edits define the successor, while the running agent remains unchanged during modification.The underlying model weights remain fixed, and any part of the repository can be revised.Source: §2 · p. 3
  • Episode records combine the agent’s reasoning, tool actions, outcomes, and code changes, and successors use them to guide later revisions.Changes to tools and procedures can affect how the agent performs subsequent self-modification.Source: §2 · p. 3
  • Two lineages follow distinct qualitative directions: capability targets limitations in abilities, while adaptive targets how the agent revises its approach when actions fail or evidence changes.The directions guide investigation without scoring resulting changes.Source: §2 · p. 3
  • Both lineages begin from the same base agent, retain initial episode records, and share preceding-generation records while keeping separate implementations.All generated agents are retained in an archive for later evaluation.Source: §2 · p. 3
  • During search, agents use prior episodes and current tool interactions or local checks, but receive no downstream benchmark tasks or evaluation results.The runtime keeps the model–tool interaction loop outside editable agent code and controls the model, reasoning effort, output limits, and execution budgets.Source: §2 · pp. 3–4

3 EXPERIMENTS

Across two model configurations and three benchmarks, SelfSearch improves population-mean success in all six settings. It also achieves competitive success with evaluation-guided search, transfers across model families, and can match a leading harness at low search cost.

3.1 EXPERIMENTAL SETUP

  • Search runs for ten generations in isolated containers, and final lineage agents are evaluated without checkpoint selection based on downstream performance.The study compares initial, capability, and adaptive agents on SWE-bench Verified, SWE-bench Multilingual, and Terminal-Bench 2.1; average execution cost includes unsuccessful attempts.Source: §3.1 · p. 4

3.2 DOWNSTREAM PERFORMANCE AND EFFICIENCY

  • Population-mean success improves in all six model–benchmark settings, while the stronger lineage varies across benchmarks and model settings.On Terminal-Bench 2.1, capability-agent success rises from 43.8% to 55.1% with GPT and from 65.2% to 73.0% with DeepSeek.Source: §3.2 · p. 4, Table 1 · p. 5
  • Both DeepSeek lineages improve Terminal-Bench 2.1 success while reducing average cost by 12.1% and 16.9%, respectively.GPT capability success rises from 43.8% to 55.1% with a 3.1% average-cost increase, while the adaptive agent reaches 44.9% and reduces cost by 6.2%; on shared SWE-bench Multilingual successes, DeepSeek adaptive reduces cost by 38.5%.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3 COMPARISON WITH EVALUATION-GUIDED SEARCH

  • SelfSearch matches or exceeds evaluation-guided baselines in several benchmark and model settings, with search costs 13.4–47.2% lower for GPT and 49.0–53.2% lower for DeepSeek.Success comparisons use tasks outside the baselines’ development set; baseline costs include development-set evaluation.Source: §3.3 · pp. 5–6, Table 2 · p. 6

3.4 CROSS-MODEL TRANSFER

  • Both lineages improve over the initial agent in both cross-model transfer directions on SWE-bench Verified.GPT-searched agents reach 83.3% and 85.0% with DeepSeek V4 Flash, versus 81.7% initially; DeepSeek-searched agents reach 79.2% and 76.7% with GPT-5.6 Luna, versus 75.0% initially.Source: §3.4 · p. 6, Table 3 · p. 6

3.5 COMPARISON WITH OTHER HARNESSES

  • For the public nine-harness comparison settings, the DeepSeek V4 Flash agent solves 73 of 89 Terminal-Bench 2.1 tasks (82.0%), tying Codex at $4.03 search cost.The comparison uses xhigh reasoning and expanded execution limits rather than the Table 1 settings.Source: §3.5 · p. 6

4 ANALYSIS

The analysis tests the contributions of episode records and an evolving improver, traces changes across generations and lineages, and checks whether evolved tools are used on downstream tasks.

4.1 EPISODE RECORDS AND THE EVOLVING IMPROVER

  • Full SelfSearch has the highest population-mean success in both model settings; removing records lowers mean success by 2.1 points with GPT and 2.9 with DeepSeek.Keeping the improver fixed lowers mean success by 1.7 and 2.9 points, respectively.Source: §4.1 · p. 7, Table 4 · p. 7

4.2 HOW IMPROVEMENTS DEVELOP

  • Episode records let lineages reuse and refine each other’s tools, including bounded text search, trajectory reading, and later output limits.DeepSeek agents also used records to repair search excerpts that hid matching text in long lines.Source: §4.2 · pp. 7–8, Table 5 · p. 7

4.3 DOWNSTREAM USE OF EVOLVED TOOLS

  • Across 269 downstream tasks, GPT and DeepSeek agents use text search on 8.6% and 23.6% of tasks, and line-range viewing on 40.3% and 33.8%.These tools are used across downstream benchmarks, while GPT’s trajectory reader is not invoked during downstream evaluation.Source: §4.3 · p. 8
  • In a Sphinx task, a GPT capability agent uses its episode-record text-search tool to locate related tests and calling code, then runs the tests.The tool supports downstream verification after a function repair.Source: §4.3 · p. 8, Figure 3 · p. 8

5 RELATED WORK

Related work covers evaluation-guided search, self-modifying agents, and the use of experience from improvement attempts. SelfSearch differs by using self-modification records to guide revisions without downstream benchmark evaluation during search.

Evaluation-guided agent search

  • Evaluation-guided search uses candidate evaluations, archives, and selection to guide revisions, while shared experience can combine discoveries across agents, tasks, or lineages.SelfSearch instead draws its revision guidance from previous self-improvement episode records, without downstream benchmark evaluation during search.Source: §5 · p. 8

Self-modifying agents

  • Self-modifying-agent approaches vary in whether one agent solves tasks and improves itself, which parts can change, and whether changes occur across generations or within a task.SelfSearch makes the entire repository editable and carries revised implementations and episode records forward together.Source: §5 · p. 9

Experience from self-improvement

  • Prior work uses histories of candidate changes and evaluation outcomes, whereas SelfSearch records reasoning, tool calls, and intermediate outcomes from the editing process.These records can capture failed editing actions, incomplete observations, and difficulties inspecting earlier records.Source: §5 · p. 9

6 DISCUSSION

SelfSearch uses self-improvement experience to generate competitive agents without downstream evaluation guiding revisions, while downstream assessment and selection remain separate costs. Episode records also expose tool and procedure difficulties that can inform reusable revisions; combining this process with evaluation-guided selection remains future work.

6 DISCUSSION

  • SelfSearch can produce competitive agents from self-improvement experience without repeated downstream evaluation guiding revisions.Candidate assessment and selection still incur separate costs.Source: §6 · p. 9
  • Episode records capture tool and procedure difficulties that can be addressed through reusable tools for downstream tasks or tools supporting further self-improvement.The analysis reports a trajectory reader as an example of the latter.Source: §6 · p. 9
  • Combining several episode-record-informed revisions with evaluation-guided candidate selection is proposed as a future direction, not an evaluated result here.The paper proposes testing whether this combination improves agents under the same total search budget.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents SelfSearch as reward-free search using self-improvement episode records, with final populations improving mean task success across three benchmarks and gains transferring across execution models. The appendices describe the agent environment, search prompts, and model configurations.

7 CONCLUSION

  • SelfSearch uses self-improvement episode records to guide revisions, and final populations improve mean task success over initial agents across three benchmarks, with gains transferring across execution models.The authors conclude that editing an agent can itself provide experience for further improvement.Source: §7 · p. 9

A INITIAL AGENT AND EXECUTION ENVIRONMENT

  • The initial agent adapts DGM’s coding-agent design, while the runtime keeps model settings and execution limits outside the editable repository and records trajectories for later episodes.The same agent implementation handles downstream tasks and self-modification; the frozen implementation is evaluated across benchmarks.Source: A INITIAL AGENT AND EXECUTION… · pp. 11–12

B SEARCH PROMPTS

C.1 MODEL CONFIGURATIONS

  • Search uses gpt-5.6-sol for the GPT configuration and deepseek-v4-pro-0813 for DeepSeek, while downstream evaluation uses gpt-5.6-luna and deepseek-v4-flash-0731.Both configurations use medium reasoning effort and temperature 1.Source: §C.1 · p. 14

C.2 EVALUATION SETS AND PROTOCOL

The evaluation uses sampled SWE-bench sets and all Terminal-Bench 2.1 tasks.

C.2 EVALUATION SETS AND PROTOCOL

  • Evaluation combines 120 SWE-bench Verified tasks, 60 sampled SWE-bench Multilingual tasks, and all 89 Terminal-Bench 2.1 tasks.The Verified set combines 60 DGM small- and medium-subset tasks with 60 sampled from its large subset; the Multilingual sample uses seed 42.Source: §C.2 · p. 14

C.3 EVALUATION-GUIDED SEARCH COMPARISON

The comparison uses ten-revision evaluation-guided baselines whose candidate selection depends on development scores, while SelfSearch reports generation-10 lineages without that selection. The accompanying cost definitions distinguish average cost across attempts from costs on tasks both agents solve.

C.3 EVALUATION-GUIDED SEARCH COMPARISON

  • Evaluation-guided baselines make ten revisions and select the highest development-score candidate, whereas SelfSearch reports both generation-10 lineages without development-score selection, so the comparison is not budget-matched.The baselines receive prior execution records and may use local checks during editing before development-set evaluation.Source: §C.3 · pp. 14–15

C.4 COST METRICS AND INFERENCE PRICING

  • Average execution cost per task includes unsuccessful attempts, while shared-task comparisons use tasks solved by both agents; positive values indicate cost reductions, and execution-cost metrics exclude search cost.The appendix also specifies token prices for GPT-5.6 Luna and DeepSeek V4 Flash.Source: §C.4 · pp. 15–16

D ADDITIONAL RESULTS

The additional results report SelfSearch’s search resources, downstream execution efficiency, and a comparison with other harnesses.

D.1 SEARCH COST

  • Table 8 reports model calls, token usage, and inference costs for initialization and ten generations; costs exclude downstream evaluation.Input tokens include cached input; tokens are reported in millions or thousands as indicated by the table caption.Source: §D.1 · p. 17, Table 8 · p. 17

D.2 DOWNSTREAM EFFICIENCY

  • Figure 4 plots accuracy–cost tradeoffs; Table 9 averages execution resources across all tasks, while Table 10 reports cost reductions versus B0 on shared successes.Table 9 includes unsuccessful tasks; Figure 4 uses different axis ranges between benchmarks.Source: §D.2 · p. 17, Figure 4 · p. 17, Table 9 · p. 18

D.3 COMPARISON WITH OTHER HARNESSES

  • SelfSearch and Codex each solve 73 of 89 Terminal-Bench 2.1 tasks, with 66 shared successes and seven unique successes apiece.Evaluation uses DeepSeek V4 Flash with xhigh reasoning and the predefined task time limits.Source: §D.3 · p. 18

E.1 CHANGES ACROSS GENERATIONS

Across generations, SelfSearch lineages refine output-handling and trajectory-inspection tools; the section also describes downstream tool-use accounting and contrasts agent-search designs.

E.1 CHANGES ACROSS GENERATIONS

  • The GPT trajectory reader gained response limits, continuation positions, filtering, and links from tool results to originating calls and arguments.The adaptive lineage introduced trajectory inspection in generation 6 and the capability lineage in generation 7; later generations refined the inherited interface.Source: §E.1 · p. 18
  • DeepSeek lineages added output-inspection tools and repaired file-handling inconsistencies, while the adaptive lineage reused records to preserve matching text when shortening search results.Changes included head-and-tail shell excerpts, a temporary-file collision fix, and later handling for file views, search results, and directory listings.Source: §E.1 · pp. 18–19, Table 11 · p. 19

E.2 DOWNSTREAM TOOL USE

  • Table 12 counts tasks with at least one introduced operation invocation, pooling benchmarks with equal task weighting and grouping related refinements.Shell commands such as grep are excluded as uses of the introduced search tool; character-range viewing and the trajectory reader are omitted for DeepSeek agents.Source: §E.2 · p. 19, Table 12 · p. 19

F COMPARISON OF AGENT SEARCH DESIGNS

  • The design comparison distinguishes evolving task agents from meta-agents, with unified roles when one agent both solves tasks and modifies itself.DGM and HGM use separate diagnosis procedures; Hyperagents defines distinct task and meta-agents, and reward-free search does not use downstream rewards to guide revisions or selection.Source: F COMPARISON OF AGENT SEARCH DESIGNS · p. 20, Table 13 · p. 20

Related papers

Loading 2609.37968v2…