PaperlayerView exact revision on arXiv

Source-linked AI summary

LiteEvo: Automated, Cost-Efficient Harness Evolution for Generalization to Unseen Tasks

Euntae Choi, Sumin Song, Sungjoo Yoo

arXiv:2609.33146v1cs.AI

TL;DR

Harness evolution has relied on handcrafted starting harnesses and has lacked evaluation of whether gains extend to unseen tasks. LiteEvo evolves harnesses from a shared neutral start with benchmark-blind, tool-free meta-agents; across five benchmarks, it matches or exceeds a HarnessX reproduction at substantially lower API cost.

  • Problem

    Harness evolution lacks evidence that a benchmark-blind, neutral-start loop can generalize to unseen tasks across diverse benchmarks, while HarnessX uses handcrafted seeds and evaluates on evolved tasks.

    Source: §1 · p. 2

  • Method

    LiteEvo uses tool-free meta-agents to mine trajectories, curate a versioned component library, and compose harnesses from a shared neutral start without naming the benchmark.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    71.0 versus 67.3 mean pass@2 for LiteEvo and HarnessX, respectively, at 13.0× lower mean API cost across five benchmarks.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Train-evolved harnesses keep gains on unseen test tasks across four benchmarks, including AppWorld tasks requiring apps unseen in training.

    Source: §1 · p. 3

    A library is benchmark-specific: it transferred to unseen tasks and apps within AppWorld, but a library evolved elsewhere did not help an unseen benchmark.

    Source: E LIMITATIONS · p. 44

Abstract

from arXiv · show

An LLM agent is defined by two things: the weights inside its model and the harness of components assembled around it. Harnesses are still handcrafted, and HarnessX, which evolves them automatically, starts each benchmark from a handcrafted harness, reports gains on the tasks it evolved on, and budgets 100 to 175 million meta-agent tokens per benchmark. We propose LiteEvo, a lightweight harness-evolution algorithm whose tool-free meta-agents mine agent trajectories for reusable components, curate them into a versioned library, and compose each round's harness from it, starting every benchmark from the same neutral harness and never naming the benchmark. Evolving on the graded tasks of five agentic benchmarks with a frozen Qwen3.5-9B, LiteEvo lifts pass@2 by 10.5 to 67.7pp and reaches comparable or higher pass@2 than a reproduction of HarnessX (71.0 against 67.3 on average) at 13.0 lower mean API cost. Harnesses evolved on train tasks keep their gains on unseen test tasks of four benchmarks, and LiteEvo also lifts Claude Code with Sonnet 4.6 by 1.2 to 71.4pp.

1 INTRODUCTION

LiteEvo targets low-cost harness evolution from a neutral, benchmark-blind start, addressing handcrafted seeds and limited evaluation on unseen tasks. It reports comparable or higher pass@2 than a HarnessX reproduction at lower API cost, with gains on unseen test tasks.

Motivation

  • HarnessX starts each benchmark from a handcrafted harness and reports gains on evolved tasks, leaving unseen-task generalization untested.Its budget is 100 to 175 million meta-agent tokens per benchmark.Source: §1 · p. 1

Results

  • 71.0 mean pass@2 versus 67.3 for HarnessX at 13.0× lower mean API cost across five benchmarks.Both methods evolve on the graded tasks; the comparison is transductive.Source: Figure 1 · p. 2, §1 · p. 2

Approach

  • LiteEvo starts from the same neutral harness for every benchmark and uses four tool-free meta-agents that are never told the benchmark.They mine failure-oriented trajectory samples, curate a versioned component library, compose candidates, and use a pass@2 gate.Source: §1 · p. 2

Generalization

  • On unseen test tasks, inductive harnesses stay within 2.2 pp of transductive results on two benchmarks and exceed them on WebShop and SWE-bench Verified.AppWorld pass@2 is 48.6 on tasks requiring apps unseen in training, where the neutral harness solves none.Source: §1 · p. 3

2 PRELIMINARIES

The preliminaries define the harness as the adjustable layer between a frozen model and its environment, then specify the success metric and evaluation regimes. They distinguish inductive evaluation on unseen test tasks from transductive evaluation on tasks used during evolution.

Tasks and the frozen agent

  • The frozen agent uses fixed decoding and a runner that passes each reply’s action to the environment until the episode ends or reaches its turn cap.Each task supplies observations and the benchmark evaluator returns a success flag and partial credit.Source: §2 · p. 3

Harness

  • A harness comprises a system prompt, turn-time processors, and runner settings; the model, environment, decoding parameters, and turn cap remain fixed.Processors may add messages, rewrite extracted actions, or rewrite observations.Source: §2 · p. 3

Metric and regimes

  • pass@k is the probability that at least one of k independent attempts succeeds, averaged over tasks and estimated with an unbiased estimator.The task sets are train, validation, and test; inductive evaluation keeps them disjoint, while transductive evaluation reuses one set.Source: §2 · p. 4

3 LITEEVO

LiteEvo iteratively improves a harness from a shared neutral start using trajectory-reading meta-agents, a versioned component library, and a validation pass@2 gate. Its design combines failure-oriented sampling with tool-free calls and persistent components to limit API cost.

3 LITEEVO

  • LiteEvo runs a frozen agent with the current harness, turns train-task trajectories into candidates through four meta-agents, and uses validation pass@2 to select the next harness.In inductive runs, train and validation tasks are disjoint; in transductive runs, both use the test tasks and the gate reuses train rollouts.Source: §3 · p. 4

3.1 A NEUTRAL START AND BENCHMARK-BLIND META-AGENTS

  • Every benchmark starts from the same neutral harness, and filters and instructions keep meta-agents benchmark-blind.The shared 788-character prompt states the episode protocol; benchmark-specific harness content is learned through evolution.Source: §3.1 · pp. 4–5

3.2 FAILURE-ORIENTED GROUP SAMPLING

  • The analyst reads ten mixed-outcome groups with the highest mean partial credit and three all-fail groups with the lowest, supporting reliability fixes and exploration.Mixed groups contrast successes and failures on the same task; all-fail groups identify tasks not yet made solvable.Source: §3.2 · p. 5

3.3 THE META-AGENT STAGE

  • The analyst proposes typed components, the editor checks or drops candidates, the librarian versions library changes, and the composer assembles a fresh harness from selected components.Components can be prompt fragments, trajectory-derived knowledge, processors, knob settings, or strategies that advise the composer but do not reach the agent.Source: §3.3 · pp. 5–6

3.4 THE PASS@2 GATE

  • The pass@2 gate accepts candidates above the best validation score, reverts candidates below its margin, and otherwise holds them for continued search.Candidates with parse failures above 5% are never accepted; a revert discards the harness composition but keeps library edits.Source: §3.4 · p. 6

3.5 WHY LITEEVO IS CHEAP

  • About five tool-free API calls per round, fixed-size trajectory sampling, cheaper-model analysis, and a persistent component library keep API costs low.Only the analyst reads full trajectories, and it reads only a fixed-size sample.Source: §3.5 · p. 6

4 EVALUATION

LiteEvo is evaluated with a frozen Qwen3.5-9B on five agentic benchmarks, against a reproduced HarnessX baseline and through inductive-transfer and harness-component studies. It achieves higher average pass@2 at lower API cost, while ablations examine component contributions, library growth, and initialization.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses frozen Qwen3.5-9B on ALFWorld, WebShop, AppWorld, GAIA, and SWE-bench Verified, with three test runs of two attempts per task unless noted.HarnessX is reproduced with the same agent and tasks, using Opus 4.6 for eight rounds; its best round is reported.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • 71.0 mean pass@2 versus HarnessX’s 67.3, at 13.0× lower mean API cost across five benchmarks.LiteEvo costs $8.24 to $24.30 per run versus $118.47 to $190.32 for HarnessX.Source: §4.2 · p. 7, Table 1 · p. 7
  • On unseen test tasks, inductive harnesses stay within 2.2 pp of transductive runs on ALFWorld and AppWorld, and score 70.0 versus 62.7 on WebShop and 66.7 versus 63.2 on SWE-bench Verified.GAIA has no inductive evaluation because its 103 public questions are too few to split three ways; transfer to an unseen benchmark does not yet help.Source: §4.2 · p. 8
  • LiteEvo raises pass@2 in all nine Claude Code with Sonnet 4.6 benchmark-regime cells, by 1.2 to 7.5 pp where the bare agent is already strong and from 0.0 to 71.4 on inductive AppWorld.The evolved harness also adds 6.7 to 49.8 pp over the neutral start on Qwen3.5-4B.Source: §4.2 · p. 9

4.3 ABLATION STUDIES

  • The full harness has the best pass@2 on all five benchmarks; prompts carry most gain on ALFWorld and AppWorld, while processors do on GAIA and SWE-bench Verified.Because prompt and processors evolved together, this split does not isolate their contributions.Source: Table 2 · p. 8, §4.3 · p. 9
  • Across five transductive runs, the library grows from 3.8 to 16.4 components on average, while the composer selects 75.6% in the last round.The librarian records 85 additions, 25 updates, 7 merges, and 2 drops; forcing explicit cross-episode memory into every harness lowered pass@2.Source: §4.3 · p. 9, Table 4 · p. 9
  • Handcrafted starts end below neutral starts on AppWorld, 64.9 versus 67.7, and GAIA, 59.9 versus 65.2.These are one run per arm, so the comparison does not establish that a seed hurts.Source: §4.3 · p. 9, Table 5 · p. 9

5 CONCLUSION

LiteEvo uses a shared neutral start and tool-free meta-agents to evolve harnesses, reaching comparable or higher average pass@2 than a HarnessX reproduction at lower API cost. The paper also situates this approach among harness, agent, and memory optimization methods, and documents its evaluation and reproducibility practices.

5 CONCLUSION

  • LiteEvo reaches average pass@2 of 71.0 versus 67.3 for the HarnessX reproduction at 13.0× lower mean API cost, from one run per benchmark.Its inductive harnesses retain gains on unseen test tasks, while a handcrafted seed did not improve on the neutral start.Source: §5 · p. 10

AI USE STATEMENT

  • The authors report using generative-AI tools to polish manuscript language and draft appendix text, while verifying sensitive content, reported results, and evaluation code themselves.AI agents also ran and monitored experiments and helped write supplementary-code instructions; the authors report testing the code on their server.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The evaluation uses public benchmarks in sandboxed environments and involves no human subjects; web-searching agents can still reach published benchmark answers.Appendix C.4 describes blocking known copies on GAIA and the remaining exposure.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper specifies the loop, meta-agents, gate, settings, neutral harness, task sets, grading protocol, cost accounting, and HarnessX reproduction, with a minimal experiment code release attached.The full code is to be released on GitHub upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A RELATED WORK

  • Unlike related harness optimizers that use benchmark-specific seeds or optimizer knowledge, LiteEvo starts all five benchmarks from one neutral harness and never names the benchmark to its meta-agents.Its meta-agents are tool-free and read failure-oriented trajectory samples rather than full traces.Source: A RELATED WORK · p. 17
  • The related-work discussion distinguishes harness evolution from weight training, multi-agent orchestration, prompt and memory optimization, and whole-agent design search.LiteEvo changes no model weights, uses multiple LLM roles offline to write a harness for one test-time agent, and freezes its library at test time.Source: A RELATED WORK · pp. 17–18

B METHOD DETAILS

The appendix specifies LiteEvo’s neutral starting harness, shared runner, component assembly, meta-agent workflow, safeguards, and acceptance gate. It also records the human-written inputs and run settings that bound the procedure.

B.1 THE NEUTRAL HARNESS

  • Every run starts from a 788-character neutral prompt; code-action benchmarks adjust its action format, while the harness has no processors and uses default knob values.The neutral harness is the only text written for the agent, together with the runner’s reply-format block, labels, and reminder.Source: §B.1 · p. 19, Figure 3 · p. 19

B.2 THE RUNNER

  • The shared runner keeps a multi-turn chat history, executes the first action-tag contents, and applies benchmark-specific fallbacks and turn handling.It preserves the system message and first observation, drops older reply–observation pairs past the prompt budget, and always keeps the four most recent.Source: §B.2 · p. 19

B.3 PROCESSORS AND KNOBS

  • Processors can inject a message, rewrite an action, or rewrite an observation; five runner knobs configure protocol and display settings.The remaining-turns knob supplies information withheld by the neutral harness, so some gains may reflect information granted by the composer rather than learning by the loop.Source: §B.3 · p. 20

B.4 COMPONENTS AND ASSEMBLY

  • The versioned library stores typed prompt, knowledge, processor, knob, and strategy components, which the fixed assembly procedure validates and places into the harness.Only component bodies reach the harness; strategies advise the composer, and knob components inform it without directly setting knob values.Source: §B.4 · p. 21, Table 8 · p. 21

B.5 THE META-AGENTS

  • Four tool-free meta-agents analyze trajectories, edit and check proposals, maintain the library, and compose a candidate harness under generic composition rules.Their calls return structured output; the analyst and librarian have 32,000-token output budgets, while the editor and composer have 16,000-token budgets.Source: §B.5 · p. 22

B.6 SAMPLING, CHECKS AND FILTERS

  • The analyst samples high-partial-credit mixed groups and low-credit all-fail groups; processor checks, identity filtering, and cited-turn checks constrain proposed components and facts.SWE-bench Verified uses five mixed and two all-fail groups; processors are replayed, while other component types do not receive that replay check.Source: §B.6 · p. 23, Table 10 · p. 33

B.7 THE META-AGENTS’ INSTRUCTIONS

  • The shared meta-agent instructions prohibit recalled benchmark knowledge, and the documented human-written inputs include the workflow, strategies, instructions, and benchmark interfaces.The listed inputs do not name a benchmark; the meta-agent instructions also include a few development-run measurements stated without naming an environment.Source: §B.7 · p. 23, §B.9 · p. 32

B.8 THE GATE

  • The gate accepts a candidate when its score exceeds the incumbent’s measured score, but holds after excessive unparsed replies or when the score does not clear the rule.The final-round composition is never scored, and the output is the last accepted best harness rebuilt byte for byte from its library snapshot.Source: §B.8 · p. 32

C.1 BENCHMARKS

These appendices describe the five benchmarks and grading procedure, explain how API costs are counted, and document answer-copy and reproduction caveats. The runs use disjoint task-level splits for inductive evaluation, with stated exceptions for GAIA and the HarnessX comparison.

C.1 BENCHMARKS

  • The five benchmarks span text-game household tasks, simulated shopping, app use, web-and-file research, and repository repair, with benchmark-specific task and action settings.ALFWorld test games use rooms absent from training; WebShop splits share no products; and SWE-bench uses three disjoint 55-instance sets.Source: §C.1 · p. 34

C.2 GRADING

  • The returned harness receives two fresh attempts per test task; pass@2 counts tasks solved by either attempt, and reported tables average three test runs.GAIA is transductive only because its answers are public and its 103 questions are too few for three-way splitting.Source: §C.1 · p. 34, §C.2 · p. 35

C.3 API COST

  • API cost counts meta-agent tokens used to produce harnesses h1 through h7, excluding the final unrun composition and local model rollouts.HarnessX used 65–130 Opus calls per round versus about five LiteEvo calls on four benchmarks; repeated context accounts for 25%–36% of HarnessX input tokens.Source: §C.3 · p. 35

C.4 ANSWER COPIES ON GAIA

  • GAIA results are not exactly reproducible because runs see changing web pages, and missing mirror sites can still expose answers despite blocks on known copies.A scan found 30 such answer-copy cases, 10 in winning episodes; scoring those episodes as failures would leave pass@2 no lower than 62.9, versus 64.6 measured.Source: §C.4 · p. 36

C.5 THE HARNESSX REPRODUCTION

  • The HarnessX comparison runs the released single-agent meta-agent because the released code lacks the paper’s four-stage evolution engine; three setup differences favor one side or the other.HarnessX names required tests in its SWE-bench task description, its ALFWorld seeds cover only 88 of 134 test games, and the AppWorld seed was written by the authors.Source: §C.5 · p. 36, Table 12 · p. 36

D ADDITIONAL RESULTS

Additional experiments show gains with a smaller backbone and on AppWorld tasks requiring unseen apps; the AppWorld evolution trace starts from zero and improves over six rounds.

D.1 A SMALLER BACKBONE: QWEN3.5-4B

  • 6.7 to 49.8 pp: LiteEvo raises pass@2 in all nine Qwen3.5-4B benchmark and regime cells, with inductive results within 2.7 pp of transductive results.Runs cost $7.78 to $22.45; gains are largest on WebShop and ALFWorld and smallest on SWE-bench Verified.Source: §D.1 · p. 37, Table 13 · p. 37

D.2 APPWORLD TEST-CHALLENGE SPLIT

  • 48.6 pass@2 inductively and 49.2 transductively on AppWorld test-challenge, versus 50.5 for HarnessX; LiteEvo runs cost $14.56 and $12.51 versus $190.32.The split has 417 tasks requiring apps unseen in training, and the bare agent and neutral harness solve none.Source: §D.2 · p. 37, Table 14 · p. 37

D.3 EVOLUTION TRACE ON APPWORLD

  • 15.2× lower API cost: LiteEvo reaches its best AppWorld harness from a zero-scoring start in six rounds, returning a harness with graded pass@2 of 67.7.Each round adds at most four components and updates or merges at most two, targeting failures in the previous round’s trajectories.Source: Figure 13 · p. 38, §D.3 · p. 38

D.4 TRANSFER ACROSS BENCHMARKS

The transfer experiments distinguish reusable procedures from environment-specific library components, while the analyst comparison weighs pass@2 against API cost.

D.4 TRANSFER ACROSS BENCHMARKS

  • 1 of 2,502 episodes were won when AppWorld harnesses used a library built on the other four benchmarks, with no AppWorld component.The composer rarely selected foreign components; the only kind that transferred was a loop-breaking processor, independently invented in every benchmark library.Source: §D.4 · p. 38
  • 12.6 to 16.3 pp: worked examples written from 90 new-benchmark train transcripts improved over a harness without them, reaching 55% to 71% of the gain from a fully evolved library.On AppWorld’s 417-task test-challenge split, foreign-library harnesses won 1 episode, compared with 48.6 and 49.2 pass@2 for harnesses evolved on AppWorld itself.Source: §D.4 · p. 38

D.5 ANALYST SETUP

  • 36% to 75% of per-worker cost: failure-oriented group sampling cost less in all nine cells, with higher pass@2 in five, within 0.2 pp in two, and lower in two.The two lower cells were ALFWorld, by 1.7 and 1.5 pp, where both setups exceeded 94.Source: §D.5 · p. 39, Table 15 · p. 39

D.6 TASK SELECTION FOR THE ANALYST

The analyst’s default selection uses mixed-outcome groups alongside all-fail groups, and it produced the best pass@2 across the five benchmarks.

D.6 TASK SELECTION FOR THE ANALYST

  • The default selection gives the best pass@2 on all five benchmarks, though on SWE-bench Verified it leads all-success groups by only 0.2 pp.It reads 10 mixed-outcome groups and 3 all-fail groups, or 5 and 2 on SWE-bench Verified.Source: §D.6 · p. 40, Table 16 · p. 40
  • 16.9 and 12.3 pass@2: on AppWorld, all-success and all-fail groups trail the default’s 67.7.The number of groups read remains 13, or 7 on SWE-bench Verified.Source: §D.6 · p. 40, Table 16 · p. 40
  • A failure is plausibly easiest to diagnose beside a success on the same task, which only mixed groups provide.The default combines 10 mixed groups with 3 all-fail groups.Source: §D.6 · p. 40, Table 16 · p. 40

D.7 LIBRARY OPERATIONS PER BENCHMARK

These additional experiments examine how the library is selected and composed, and test persistent memory and merging the editor with the librarian.

D.7 LIBRARY OPERATIONS PER BENCHMARK

  • The composer adds components in most rounds and rarely removes them; in the last round it selects 8 of 16 components on SWE-bench Verified and all 13 of 13 on AppWorld.The table reports components picked over the library size available to the composer.Source: §D.7 · p. 41, Table 17 · p. 41

D.8 MEMORY IN EVOLVED HARNESSES

  • Every pinned cross-episode memory variant scored below the default: GAIA scored 65.2 against 55.8 and 58.4, and AppWorld scored 67.7 against 15.4.The comparison was uncontrolled; each variant was measured once, and AppWorld variants used a different code version and load from the default.Source: §D.8 · p. 42, Table 18 · p. 42

D.9 MERGING THE EDITOR AND THE LIBRARIAN

  • Merged editor-librarian runs scored lower on all four benchmarks, but the runs differed from defaults in other ways; the only controlled WebShop pilot found no difference.On AppWorld, inductive pass@2 fell from 65.3 to 14.0; no split control arm was run beside the merged runs.Source: §D.9 · p. 43, Table 19 · p. 43

D.10 REFINE OR COMPOSE

  • 68.1 against 64.3 pass@2 transductively and 72.4 against 65.8 inductively: Compose scored higher on average than Refine in both regimes.Refine won on WebShop transductively and SWE-bench Verified inductively; Compose builds afresh from the library, while Refine edits a carried harness.Source: §D.10 · p. 43, Table 20 · p. 43

E LIMITATIONS

LiteEvo’s evolved libraries transfer to unseen tasks within a benchmark, but a library evolved on other benchmarks did not help an unseen benchmark. The evaluation also has single-run, benchmark-exposure, resource, and scope limitations.

E LIMITATIONS

  • A library evolved on other benchmarks did not help an unseen benchmark, although LiteEvo transfers to unseen tasks within a benchmark and to unseen AppWorld apps.On AppWorld’s test-challenge split, HarnessX scores higher than LiteEvo.Source: E LIMITATIONS · p. 44
  • Each result comes from one evolution run, and GAIA episodes could reach benchmark copies or dataset mirror pages before answering.Scoring 10 such winning episodes as failures would leave round 7’s pass@2 no lower than 62.9, versus 64.6 measured; this bound covers only that round’s episodes.Source: E LIMITATIONS · p. 44
  • The study did not test a flagship agent or the hardest agentic benchmarks, and tasks outside agentic benchmarks are out of scope.The meta-agent instructions were written once and not optimized; more active library management and combining LiteEvo with fine-tuning or reinforcement learning remain future work.Source: E LIMITATIONS · p. 44
Loading 2609.33146v1…