PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoIn: Bridging Evolution and Internalization for Agent Fine-Tuning

Shihan Dou, Shaofan Liu, Zhonghang Lu, Jiahang Lin, Shichun Liu, Binghai Wang, Jiajie Jin, Guanting Dong, Tao Gui, Qi Zhang, Xuanjing Huang

arXiv:2609.35290v1cs.AIcs.CL

TL;DR

Existing co-evolution approaches can bundle new tools, decision procedures, and model adaptation, making it difficult to tell whether decision-making itself improved. EvoIn discovers and validates procedures in an evolved harness, rewrites resulting traces, and internalizes the procedures through fine-tuning under the original harness. It reports accuracy gains in-domain and out-of-domain, with transfer to unseen tasks and improvements on another model family.

  • Problem

    Jointly evolving harnesses and models can bundle new tools, decision procedures, and harness adaptation, obscuring whether stronger decision-making transfers across harnesses.

    Source: §1 · p. 2

  • Method

    EvoIn evolves and evaluates procedures in a temporary harness, rewrites procedure-guided traces as self-contained reasoning, and fine-tunes the model under the original harness.

    Source: §1 · pp. 2–3, §3 · p. 5

  • Results

    EvoIn raises accuracy by 10.9 points in-domain and 9.2 points out-of-domain, with gains transferring to unseen tasks and another model family.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The internalized procedures can generalize to unseen tasks, where agents generate task-appropriate decision procedures rather than replaying those encountered during evolution.

    Source: §1 · p. 3

    The current evolution process is guided by a stronger agent rather than being carried out entirely by the target agent.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent work has explored improving agents by jointly evolving their harnesses and models, but often takes a ''potpourri'' approach that bundles together new tools, new decision-making procedures, and model adaptation to the evolved harness under a single notion of agent improvement. In this paper, we instead investigate how agents can improve their decision-making procedures. In particular, we propose EvoIn, an agent fine-tuning framework that bridges evolution and internalization. EvoIn first analyzes agent execution traces to evolve and validate new decision-making procedures by temporarily instantiating them in the harness. The validated procedures guide the agent to generate improved reasoning traces. These traces are then rewritten into self-contained reasoning traces, removing explicit references to harness instructions while expressing the induced decision logic as the model's own reasoning. Finally, EvoIn fine-tunes the model on the rewritten traces, internalizing these procedures so that the improved decision-making persists without the evolved harness at inference time. We evaluate EvoIn on diverse benchmarks and find that it consistently enables agents to learn stronger decision-making procedures, raising the pass rate by 10.9 points in-domain and by 9.2 points out-of-domain. Results further show that the internalized decision procedures generalize to unseen tasks. Case studies show that agents can learn to decide how to solve a task before solving it, for example by checking a document's length to choose between reading it in full and searching it. EvoIn is also broadly applicable, showing consistent improvements on another model family.

1 INTRODUCTION

EvoIn addresses how to improve agents’ decision-making procedures and whether agents can help drive that improvement. It uses an evolved harness to discover and validate procedures, then internalizes them in the model, with reported gains on in-domain and out-of-domain benchmarks.

1 INTRODUCTION

  • Decision-making procedures shape how agents reason and act, including their planning, tool use, and verification choices.The paper asks how those procedures can be improved and how much of that improvement agents can drive or automate.Source: §1 · p. 1
  • Model fine-tuning leaves procedures to change as a byproduct, harness evolution keeps them external, and co-evolution can confound learned reasoning with harness-specific adaptation.The paper frames these trade-offs as motivating a route that separates procedure exploration from model internalization.Source: §1 · p. 2, Figure 1 · p. 2
  • EvoIn uses harness evolution to explore and validate decision procedures, rewrites the resulting traces as self-contained reasoning, and fine-tunes the model under the original harness.The harness is a temporary space for discovering and validating procedures that are ultimately learned by the model.Source: §1 · pp. 2–3
  • EvoIn raises accuracy by 10.9 points in-domain and 9.2 points out-of-domain, with gains of up to 19.7 points on a single benchmark.The framework rewrites procedure-guided traces before fine-tuning so the decision logic can be learned without evolved-harness instructions.Source: §1 · pp. 2–3

2 RELATED WORK

The related-work discussion distinguishes approaches by whether they update the model, the harness, or both. EvoIn instead uses the evolved harness as a temporary procedure-discovery space and internalizes validated, rewritten procedures under the original harness.

2 RELATED WORK

  • Model fine-tuning updates weights while leaving the harness fixed, so decision procedures change only as a byproduct of training rather than as explicit candidates for comparison.The paper describes the improvement loop as designed by humans, with agents at most producing data and experience.Source: §2 · p. 3
  • Harness evolution makes procedures explicit and executable for proposing, testing, keeping, or discarding, but leaves the resulting procedures external to the model.The paper also notes reported limits in lifecycle management, held-out generalization, and the ability to produce harness updates.Source: §2 · p. 3
  • Model–harness co-evolution updates both components, but training and evaluation under the evolved harness can make learned decision-making hard to distinguish from harness-specific adaptation.The paper notes that harness choice alone can move measured accuracy by as much as 28 points within a model.Source: §2 · p. 4
  • Unlike prior internalization approaches that typically specify the procedure in advance, EvoIn has an agent propose and validate it, rewrite dependencies away, and train and evaluate under the original harness.This design aims to separate procedure learning from adaptation to the evolved harness.Source: §2 · p. 4
  • Figure 2 illustrates the transition from harness updates and instruction-dependent reasoning to the same decision logic expressed as the agent’s own reasoning.The figure depicts iterative procedure proposal and evaluation followed by trajectory sampling and rewriting for fine-tuning.Source: Figure 2 · p. 4

3 EVOIN: BRIDGING EVOLUTION AND INTERNALIZATION

EvoIn iteratively proposes and evaluates harness-based decision procedures, uses the selected procedure to generate successful trajectories, rewrites them to remove evolved-harness dependencies, and fine-tunes the model under the original harness.

3 EVOIN: BRIDGING EVOLUTION AND INTERNALIZATION

  • EvoIn analyzes training-data execution trajectories while reserving held-out development data for evaluating candidate harnesses.A trajectory includes reasoning, tool calls and observations when applicable, and the final answer; its reasoning trace is denoted z.Source: §3 · p. 5
  • A proposal agent uses the current harness, feedback, and prior trajectories to suggest procedure updates, which undergo static and runtime validation before evaluation.The process repeats across rounds, retaining the candidate harness with the highest development-set score.Source: §3 · p. 5
  • The selected evolved harness guides training-data sampling, and only trajectories with a full task score and valid final answer are retained for internalization.The sampled trajectories’ reasoning and actions reflect the improved decision procedures.Source: §3 · p. 5
  • EvoIn rewrites trajectories to remove evolved-harness instructions, tools, or runtime assumptions while preserving useful induced decision logic.The TAILOR pipeline edits trajectories, replays them under the original harness, checks for residual dependencies, and retains only trajectories that pass its checks.Source: §3 · p. 5
  • With the original harness restored, supervised fine-tuning on rewritten trajectories internalizes the procedures so inference no longer depends on the evolved harness.The loss is applied to model-generated turns.Source: §3 · p. 5

4 EXPERIMENTS

The experiments test whether task-specific harness evolution yields better training demonstrations and whether fine-tuning internalizes the resulting procedures. EvoIn improves performance under the seed harness on both in-domain and out-of-domain evaluations.

4.1 SETUP

  • The 23 ID task categories are organized into seven groups, and the six OOD benchmarks are absent from the evolution and SFT data.The evolution-test split is held out from the SFT corpus and later serves as the ID test set.Source: §4.1 · pp. 5–6
  • Every category starts from a task-agnostic seed harness, which is the only harness used at evaluation; the main run fine-tunes on 12,035 demonstrations for three epochs.Results use one sampled trajectory per example under a fixed protocol; OOD results pool examples across all six benchmarks.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • Pass rises from 35.96% to 46.83% on ID examples and from 18.99% to 28.20% on OOD examples after EvoIn fine-tuning.These are gains of 10.87 and 9.21 points, respectively; both models use the same seed harness and evaluation protocol.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • All seven ID task groups improve, with ID Score up 11.82 points overall; log and dialogue tracking gain +21.64 and structured-data reasoning +17.33.Score increases in 21 of 23 categories, while complex instruction following gains the least (+5.66) and remains below 10% Pass for both models.Source: §4.2 · p. 6, Table 1 · p. 7
  • All six OOD benchmarks improve in Pass and Score; Score gains range from +1.02 on BrowseComp-LongContext to +43.76 on MRCR.Overall OOD Score rises by 23.47 points, much more than Pass, largely because MRCR awards partial credit.Source: §4.2 · p. 6, Table 2 · p. 7

5 FURTHER ANALYSIS

Further analyses test the roles of harness evolution and rewriting, whether the target model can drive the pipeline, and how rollout-model and model-family choices affect results.

5 FURTHER ANALYSIS

  • Seed-harness SFT improves ID Pass by only 1.87 points and does not improve OOD Pass, while EvoIn gains 10.87 and 9.21 points.The comparison uses the same number of successful seed-harness trajectories as the categories used by EvoIn.Source: §5 · p. 7, Table 3 · p. 8
  • Without rewriting, OOD Pass falls to 19.67 from EvoIn’s 28.20, near Base’s 18.99, while ID Pass changes little at 45.00 versus 46.83.Raw trajectories can carry evolved-harness instructions and tools into unseen tasks; the model exhausts the seed harness’s 30-turn budget on 19.4% of OOD examples versus 6.1% for EvoIn.Source: §5 · pp. 7–8, Table 3 · p. 8
  • The All-Qwen pipeline raises ID Pass from 35.96% to 42.04%, but OOD Pass remains at Base level, 18.90 versus 18.99.It produces 6,123 demonstrations and achieves 56% of the main system’s ID gain.Source: §5 · p. 8, Table 3 · p. 8
  • On the same 1,752 tasks, GLM-5.3 demonstrations raise ID Pass from 46.52 to 52.43 and OOD Pass from 22.04 to 25.68, while OOD Score stays flat.The comparison holds the best harnesses and Tailor pipeline constant.Source: §5 · p. 8, Table 3 · p. 8
  • With Gemma-4-31B-it as both rollout target and student, EvoIn improves ID Pass from 49.61% to 53.04% and OOD Pass from 49.08% to 50.10%.This tests the method beyond the Qwen model family.Source: §5 · p. 9

6 DISCUSSION

EvoIn focuses on internalizing reusable decision procedures, while tools and other runtime components may remain in the harness. The discussion frames this as complementary to model–harness co-evolution and identifies current limits and directions for expansion.

6 DISCUSSION

  • EvoIn targets reusable decision procedures for internalization, while new tools, workflows, and other runtime components cannot be fully absorbed and should remain in the harness.Model–harness co-evolution can help models use those external components.Source: §6 · p. 9
  • EvoIn’s internalization recipe may extend to tool-use knowledge, recurring experience, and expectations about action consequences, beyond decision procedures.The discussion proposes learning which improvements belong in the model and which should remain external.Source: §6 · p. 9
  • Current limits include stronger-agent guidance, only one complete evolution–internalization cycle, and evaluation concentrated on context reasoning; future work targets self-improvement and other settings and capabilities.Proposed settings include software engineering, web interaction, and other long-horizon environments.Source: §6 · p. 9
  • The conclusion reports improvements across diverse in-domain tasks, transfer to unseen benchmarks under the original harness, and gains on another model family.Ablations and case studies indicate that gains require both harness evolution and rewriting.Source: §6 · p. 9
  • Combining EvoIn with model–harness co-evolution could support improving both external components and capabilities that can be internalized.The paper presents this combination as a route toward stronger and more general agents.Source: §6 · p. 10

A.1 ID TASK CATEGORIES

The in-domain data cover 23 task categories across seven task groups, with separate evolution and evolution-test splits for each category.

A.1 ID TASK CATEGORIES

  • The 23 in-domain categories are grouped into seven task groups, with 100 evolution examples and a disjoint 100-example evolution-test split per category.The training data cannot be released publicly; Table 4 describes the categories and groups, and Table 5 reports category results.Source: §A.1 · p. 14

A.2 OOD BENCHMARKS

The out-of-domain test set contains 2,064 examples from six benchmarks spanning long-context document, web, dialogue, code, and aggregate-reasoning tasks. Five benchmarks are public; Table-Longer was constructed by the authors and cannot be released.

A.2 OOD BENCHMARKS

  • The OOD test set contains 2,064 examples from six benchmarks, including five public benchmarks and the authors’ non-releasable Table-Longer.The passage says Table 2 lists benchmark-level example counts, but those values are not supplied here.Source: §A.2 · p. 14
  • AA-LCR requires reasoning across real-world documents, while BrowseComp-LongContext adapts BrowseComp to questions paired with more than 100 mostly distracting web pages.Source: §A.2 · p. 14
  • LongBench v2 covers long-context QA, in-context learning, dialogue histories, code repositories, and structured data; MRCR tests retrieval of specified responses from long conversations.MRCR scores string similarity and gives zero when the required prefix is missing.Source: §A.2 · p. 14
  • Oolong asks models to label many short texts before answering aggregate questions, and the evaluation uses 300 examples from its validation split.Source: §A.2 · p. 14

A.3 HARNESS EVOLUTION AND TAILOR DETAILS

Harness evolution is constrained to whole-file changes and must pass static and runtime checks. Tailor rewrites trajectories for the seed harness, then checks and repairs them before scaling the pipeline.

A.3 HARNESS EVOLUTION AND TAILOR DETAILS

  • The seed harness scans inputs, searches for relevant evidence, reads source lines, and can use bash and notepad before submitting answers.Source: §A.3 · p. 16
  • Candidate harnesses may add or replace whole files only within specified harness components, and must pass static checks plus an end-to-end run on one evolution example.The runtime answer need not be correct, but the run must finish without errors; failed candidates receive zero and skip evolution-test evaluation.Source: §A.3 · p. 16
  • Tailor replays trajectories under the seed harness and rewrites reasoning and tool use into forms that harness supports, while preserving answer meaning and avoiding fabricated capabilities.It can rephrase reasoning, remap tool calls, re-execute changed calls, or drop unsuitable steps; trajectories that cannot be safely converted are marked infeasible.Source: §A.3 · pp. 16–17
  • Rewriting also addresses training defects such as failed searches, unsupported evidence, contradictions, wrong line citations, and claims based on truncated reads.The reference answer is provided only to check meaning preservation, not as evidence for the rewritten trajectory.Source: §A.3 · p. 17
  • Deterministic checks reject residual evolved-harness wording, unsupported or contradictory reasoning, invalid citations, and unreplayed or failed actions; repair targets detected failures once before full replay and rechecking.Source: §A.3 · p. 17
  • Iterative development on ten trajectories added replay, evidence checks, an independent checker, and targeted repair after early outputs proved largely unusable for training.Targeted repair yielded eight directly usable trajectories out of ten; two were flagged for resampling before the frozen pipeline was scaled.Source: §A.3 · p. 17

A.4 PROPOSAL PROMPT

The proposal prompt directs an agent to use rollout evidence to develop general harness changes, with the workflow and multiple harness components open to revision. It constrains proposals to be evidence-grounded, orthogonal, and safe to implement.

A.4 PROPOSAL PROMPT

  • Proposals must follow failure evidence, target general mechanisms rather than task-specific hacks, and preserve useful rules unless evidence links them to regressions.Each candidate also supplies a causal account covering the failure pattern, root cause, predicted fix, rationale, and component choice.Source: §A.4 · pp. 18–19
  • The evolvable space includes the solving workflow and harness files, while new tools are reserved for strict last-resort cases.A lightweight preliminary context check may let the agent adapt how it gathers evidence.Source: §A.4 · p. 18
  • Candidate changes must respect runtime and component constraints, including valid tool schemas and signatures, relative paths, and middleware interface semantics.Tool or middleware edits require justified component choices; middleware is a fallback after prompt- and tool-level measures fail repeatedly.Source: §A.4 · p. 18
  • The query asks the agent to analyze failures, form general mechanisms, and propose three orthogonal harness candidates aimed at improving pass rate.It summarizes failure clusters, passing strategies, robustness, prior change attribution, and the best harness so far.Source: §A.4 · pp. 17, 20

A.5 TAILOR PROMPTS

A.5 specifies prompts for rewriting, repairing, and reviewing trajectories as seed-harness-compatible fine-tuning examples. Across these stages, the prompts prioritize visible evidence, causal coherence, and user scope, rejecting changes that require invented evidence or do not meet the stated requirements.

A.5 TAILOR PROMPTS

  • The rewriter must make retained reasoning and actions plausible under the seed prompt, tools, and budget while ensuring coherent, evidence-supported SFT demonstrations.It preserves useful solving behavior, removes evolved-only dependencies, and avoids stylistic polishing; edits are limited to seed-compatibility and high-confidence SFT defects.Source: §A.5 · pp. 20–21
  • The evidence-visibility rules reject truncated text, errors, navigation previews, unmatched calls, and gold answers as evidence for trajectory claims.Claims of full-document coverage require serialized observations that visibly show complete, untruncated coverage.Source: §A.5 · p. 22
  • The prompt requires citations, quotations, line ranges, and section-boundary claims to match visible source content without strengthening what it supports.It also recursively removes evolved-only labels and templates from every training-visible field while preserving underlying actions the seed harness supports.Source: §A.5 · p. 22
  • The rewriter preserves seed-plausible adaptive routing based on document profile and task instead of imposing a fixed retrieval sequence.A short-document full read is retained only when seed metadata and tools support it and the successful observation is untruncated.Source: §A.5 · p. 23
  • High-confidence defects may receive minimal repairs, but failed or duplicate steps are dropped only when non-load-bearing and later reasoning remains coherent.The prompt forbids adding answer facts from gold, requires infeasibility when material errors cannot be repaired from visible evidence, and preserves supported facts while changing representation only to meet the seed output contract.Source: §A.5 · pp. 23–24
  • Before editing, the rewriter must derive seven fields covering ungrounded terms, tool mappings, unsupported tools, budgets, output contracts, evidence gaps, and trajectory defects.For mapped calls, it checks seed-schema validity and whether the seed call would return equivalent content; edits use one action, while unlisted steps remain unchanged.Source: §A.5 · p. 24
  • Repairs must preserve evidence and causal coherence, remove stale references after dropped or remapped calls, and report infeasibility if the trajectory cannot fit the seed budget.When over the iteration limit, compatibility and evidence edits come first; empty, failed, or duplicate probes may be dropped only while preserving necessary raw evidence and coherence.Source: §A.5 · p. 25
  • Set sft_ready_after_edits=true only when issues are resolved, no regrounding remains, and the result is seed-reproducible and evidence-closed.Before output, check seed-plausible actions and arguments, visible support for retained facts, absence of unsupported coverage claims, budget fit, and a valid closing step or reported infeasibility.Source: §A.5 · pp. 26–27
  • The repair editor may make only sparse edits to existing agent steps, using rephrase, tool_remap, reground_flag, or drop.It returns the smallest edit list needed to resolve supported blockers and cannot add tool calls, change call IDs, or edit observations or system/user steps.Source: §A.5 · pp. 27–28
  • Only earlier successful, non-truncated replay observations count as evidence; if a repair needs new evidence or a new call, return no edits.The editor must not use gold, fabricate observations, or treat omitted or truncated text as visible support.Source: §A.5 · pp. 25, 27, 28
  • The independent SFT judge accepts only trajectories with seed-plausible workflows, visible support for factual claims, complete requested scope, and no remaining evolved-harness artifacts or contradictions.It reviews retained reasoning and the answer, not the final answer alone, and requires a compact, legible evidence-gathering path that is pedagogically useful.Source: §A.5 · p. 29
  • Section boundaries must match the preceding replayed lines, and reviewers reject unsupported semantic amplification, precision violations, or load-bearing claims unverifiable from the serialized trajectory.Reviewers should not reject merely for natural-language differences or a seed-plausible short-context read.Source: §A.5 · p. 30
  • Reject rewrites that retain false claims or contradictions about replayed observations, even when the final answer happens to be correct.A later correction does not erase a false retained claim.Source: §A.5 · p. 30
  • The mandatory review includes trace and scope passes comparing retained claims with replayed observations and the answer with the user request and visible outline or checklist.It also requires edit-integrity and citation-and-entailment checks, including verifying intended repairs and checking citations or quotations against replayed lines.Source: §A.5 · p. 30
  • Reviewers must verify the final serialized trajectory itself rather than trust a rewrite_audit statement that a repair occurred.Reject if a repair is absent from the training-visible result or a residual template or failed probe remains.Source: §A.5 · p. 30
  • The reviewer must return exactly one JSON object and list every rewritten-trajectory agent step ID exactly once, with consistency findings capturing unresolved false claims or cross-step contradictions.The required object also specifies quality_risk, blockers, warnings, and rationale fields.Source: §A.5 · pp. 30–31
  • An example is SFT-ready only when acceptance has empty blockers, warnings, and consistency findings; known quality caveats preclude acceptance.Reviewers must not suggest edits, invent missing evidence, or use gold to fill gaps.Source: §A.5 · p. 31

A.6 SFT CONFIGURATION

Appendix A.6–A.8 records the training and evaluation setup and its comparability boundaries; A.9 illustrates harness evolution, rewriting, and length-conditioned retrieval.

A.6 SFT CONFIGURATION

  • 12,035 SFT demonstrations remain after removing duplicates, test overlaps, and examples over 24,576 tokens; Table 6 lists the shared Qwen configuration.Source: §A.6 · p. 31, Table 6 · p. 31

A.7 EVALUATION SETTINGS

  • ID categories use Qwen3.5-35B-A3B or GPT-OSS-120B judges; OOD scoring uses official rules for three benchmarks and GPT-OSS-120B for the other three, with missing scores counted as zero.Table 7 lists decoding settings; a dash means a setting is not explicit.Source: §A.7 · p. 31, Table 7 · p. 32

A.8 ABLATION DETAILS

  • Tables 8–9 report per-category ID and per-benchmark OOD results, but Gemma results are not directly comparable because its evaluation setup differs.Gemma uses its own tool parser, training stack, decoding policy, and OOD judge version.Source: §A.8 · p. 32, Table 8 · p. 32, Table 9 · p. 33

A.9 CASE STUDIES

  • In Table QA, the base model solves 34 of 100 evolution-test questions under the seed harness and 61 under the evolved harness; about 25 of 32 newly solved cases fail only on answer format under the seed harness.Source: §A.9 · p. 33
  • In an accepted Multi-Doc Key Lookup rewrite, Tailor edits three of five steps, removing evolved-harness references while retaining the scan and adapting a call to the seed harness.Source: §A.9 · pp. 34–35
  • In 11 categories, a document-length check enters the selected harness after raising evolution-test Pass over its parent by 1 to 17 points; short and long examples follow different retrieval routes and both answers are correct.Thresholds vary by category; Faithfulness Verification uses about 120 lines as its short-document boundary, while Evidence-Located QA uses 300 lines.Source: §A.9 · pp. 35–36
  • 88.3% versus 66.4% of short-document examples are read in full before grep by EvoIn and the base model, respectively; on remaining examples, the rates are 45.1% versus 24.3%.Source: §A.9 · p. 36
Loading 2609.35290v1…