PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Harness: End-to-End Optimization of Model Harnesses

Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn

arXiv:2603.28052v1cs.AI

TL;DR

LLM harnesses shape system performance but remain largely hand-designed. Meta-Harness searches over harness code using a coding agent that selectively inspects prior candidates and execution traces. It improves performance across classification, retrieval-augmented math reasoning, and agentic coding.

  • Problem

    Harness code shapes LLM system performance, but harness engineering remains largely manual.

    Source: §1 · p. 1

  • Method

    Meta-Harness searches end-to-end over harness code using a coding-agent proposer with selective access to prior candidates’ code, scores, and execution traces.

    Source: §1 · p. 2

  • Results

    Meta-Harness improves performance across online text classification, retrieval-augmented math reasoning, and agentic coding, including a 7.7-point classification gain over ACE (Zhang et al. [59]).

    Source: §1 · p. 3

  • Takeaways & Limitations

    Discovered harnesses generalize to out-of-distribution classification datasets and unseen base models in math, providing readable strategies reusable across models.

    Source: §5 · p. 10

    The experiments use one strong coding-agent proposer, Claude Code, leaving variation across proposer agents untested.

    Source: §5 · p. 10

Abstract

from arXiv · show

The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.

1 Introduction

Meta-Harness automates harness engineering through a coding agent that selectively inspects full candidate histories stored in a filesystem. Its discovered harnesses improve performance on online text classification, retrieval-augmented mathematics, and agentic coding.

1 Introduction

  • 6× performance gaps can result from changing a fixed LLM’s harness, motivating automation of today’s largely manual harness engineering.The harness is the code determining what information to store, retrieve, and show to the model.Source: §1 · pp. 1–2
  • Meta-Harness uses a coding agent to selectively inspect prior harness code, scores, and execution traces through a filesystem rather than compressed summaries.Each proposed harness is evaluated, and its logs are stored for subsequent iterations. In the most demanding setting, the proposer reads a median of 82 files per iteration, referencing over 20 prior candidates per step.Source: §1 · p. 2, Figure 2 · p. 2
  • 10,000,000 diagnostic tokens can arise from one harness evaluation, versus only 100 to 30,000 context tokens per optimization step in representative text optimization settings.Short-horizon or compressed feedback can obscure how early harness decisions cause downstream failures, motivating selective access to raw historical experience.Source: §1 · pp. 2–3
  • 7.7 points of improvement over Agentic Context Engineering (ACE, Zhang et al. [59]) accompany 4× fewer context tokens on online text classification.Meta-Harness also matches the next-best text optimizer’s final performance after only four proposals versus 60.Source: §1 · p. 3
  • 4.7 points of average accuracy improvement across five held-out models are achieved by a single discovered harness on 200 IMO-level retrieval-augmented math problems.Source: §1 · p. 3
  • #1 among all Haiku 4.5 agents on TerminalBench-2, the discovered harness also surpasses Terminus-KIRA.Source: §1 · p. 3

2 Related Work

Meta-Harness connects adaptive access to external context, executable code search, and text optimization. It uses prior rollout experience to diagnose failures and rewrite harness code rather than update model weights.

2 Related Work

  • Meta-Harness applies credit assignment and meta-learning [3] [17] [44] [2] at the harness level, diagnosing rollout failures and rewriting external code rather than model weights.It reasons about which steps and components caused failures to guide changes governing future behavior.Source: §2 · p. 3

External memory and adaptive access

  • Meta-Harness adapts external-context access to harness engineering, selectively inspecting historical code, scores, and execution traces to improve context-management procedures themselves.Related mechanisms include retrieval-augmented generation [28], interleaved retrieval and reasoning [48], memory-based agents [37], and recursive language models [56].Source: §2 · p. 3

Executable code search

  • Meta-Harness searches full harness implementations, including prompt construction, retrieval, and task-resetting state updates, rather than a predefined space of context-management procedures.Its minimal outer loop gives the proposer unrestricted filesystem access to prior experience, letting it decide what information to inspect.Source: §2 · p. 3

Text optimization methods

  • Unlike feedback-based text optimizers [38] [31] [53] [51] [1] [35] [26], Meta-Harness targets executable procedures and examines failed examples and execution traces to propose targeted edits.Relevant feedback is distributed across code, scores, and traces, making it difficult to summarize up front.Source: §2 · p. 3

3 Meta-Harness: A Harness for Optimizing Harnesses

Meta-Harness searches for task-specific harness programs by repeatedly proposing, evaluating, and logging candidates. Its coding-agent proposer selectively inspects prior code, scores, and execution traces through a filesystem rather than relying on lossy summaries.

Objective

  • Harness optimization seeks the stateful program that maximizes expected final task reward for a fixed language model over the target task distribution.The harness constructs model prompts and updates its state after each interaction; a task-specific reward function scores the resulting trajectory.Source: §3 · p. 4

Meta-Harness search loop

  • A single coding-agent proposer controls diagnosis and harness modification, selectively retrieving prior code, scores, and execution traces from a growing filesystem.Each evaluated harness contributes an artifact directory, which the proposer queries through terminal tools rather than ingesting everything into context. Initial harnesses are evaluated and logged; subsequently proposed candidates undergo interface validation before evaluation and logging.Source: §3 · pp. 4–5
  • Meta-Harness maintains a population and Pareto frontier without imposing parent selection, and evaluates the frontier on the test set after a fixed search budget.Multiple objectives, such as accuracy and context cost, are compared using Pareto dominance. The proposer can inspect any prior harness, but receives only search-set feedback and traces, never test-set results.Source: §3 · pp. 4–5

Advantages of code-space search

  • Execution-trace inspection lets the proposer diagnose failures and change algorithmic structure, from retrieval, memory, and prompt construction to full program rewrites.Reported search trajectories show diagnosis of confounded edits and safer modifications after regressions, without predefined mutation operators. Representing harnesses as programs supplies a regularization bias toward coherent, reusable algorithms rather than brittle, hard-coded solutions.Source: §3 · p. 5

Practical implementation

  • Roughly 60 harnesses are evaluated over 20 iterations in a typical run, using Claude Code with Opus-4.6 as proposer.Each harness is a single-file Python program modifying task-specific prompting, retrieval, memory, and orchestration; the underlying model varies by domain but remains frozen. A minimal domain-specific skill specifies where to write harnesses, how to inspect artifacts, and which files may be modified.Source: §3 · p. 5

4 Experiments

Meta-Harness improves domain-specific results in online text classification, retrieval-augmented math reasoning, and agentic coding. Experiments compare discovered harnesses with hand-designed strategies and program-search methods, with additional tests of generalization, context efficiency, and proposer information access.

4.1 Online Text Classification

  • 48.6% accuracy surpasses ACE [59] by 7.7 points and MCE [52] by 8.6 points in online text classification.The selected harness uses 11.4K context tokens, versus 50.8K for ACE and 28.5K for MCE.Source: Table 2 · p. 6, §4.1 · p. 7
  • Meta-Harness discovers a stronger accuracy–context Pareto frontier than comparison methods, enabling controlled exchanges of additional context for higher test accuracy.Free-form code optimization lets the proposer respond to current metrics and a desired accuracy–cost trade-off without fixing a single scalar objective beforehand.Source: Figure 3 · p. 6, §4.1 · p. 7
  • 50.0 median and 56.7 best accuracy with full traces exceed scores-only (34.6 median, 41.3 best) and scores-plus-summary (34.9 median, 38.7 best).Even the full interface’s median candidate exceeds either ablation’s best candidate. Summaries without raw traces fail to recover the missing signal, supporting the importance of detailed execution access.Source: §4.1 · pp. 6–7, Table 3 · p. 7
  • 10× fewer evaluations suffice to match OpenEvolve and TTT-Discover [55], while Meta-Harness’s final search-set accuracy exceeds theirs by more than 10 points.Comparisons use the same Opus-4.6 proposer configuration and evaluation budget, selecting candidates on search-set performance while withholding test sets. The authors attribute the advantage to filesystem access to full experience histories rather than structured, restricted proposer inputs.Source: §4.1 · p. 6, Table 4 · p. 7
  • 73.1% average accuracy across nine datasets unseen during search exceeds ACE’s 70.2% and every few-shot baseline.Meta-Harness performs best on six of nine datasets. Adding more than 32 few-shot examples instead hurts performance on seven of nine tasks.Source: §4.1 · p. 7, Table 5 · p. 8

4.2 Harnesses for Retrieval-Augmented Reasoning

  • 4.7 points average improvement over no retrieval across five models gives Meta-Harness the strongest overall average on 200 previously unseen IMO-level math problems.It exceeds BM25 retrieval by 1.3 points using the same lexical retrieval stack, without an additional dense encoder. One harness, selected using GPT-OSS-20B from search over a decontaminated corpus of ≥500,000 solved problems, is also evaluated on four models unseen during search.Source: Table 6 · p. 8, §4.2 · p. 8

4.3 Evaluating Agentic Coding Harnesses on TerminalBench-2

  • The coding proposer uses prior failures to hypothesize confounding, isolate structural changes from prompt rewrites, and discover a safer additive modification.That modification becomes the run’s best candidate. The trajectory provides qualitative evidence that filesystem access supports inspecting past experience, forming causal hypotheses, and revising the harness.Source: §4.3 · p. 9
  • 76.4% pass rate on Opus 4.6 exceeds Terminus-KIRA’s 74.7%; 37.6% on Haiku 4.5 exceeds Goose’s 35.5% on TerminalBench-2.Meta-Harness ranks second among Opus 4.6 agents, behind ForgeCode’s 81.8%, and first among Haiku 4.5 agents. Search and final evaluation use the same 89-task benchmark, so these results measure benchmark-specific discovery rather than held-out generalization.Source: Table 7 · p. 9, §4.3 · p. 9

5 Discussion · A Qualitative Proposer Behavior

Meta-Harness discovers transferable harnesses through search that can selectively inspect prior code, execution traces, and failures. The discussion highlights practical advantages and limitations, while the qualitative section introduces an examination of filesystem use during TerminalBench-2 search.

5 Discussion

  • Discovered harnesses generalize to out-of-distribution classification datasets and unseen base models in math.Search completes in a few hours and produces readable, transferable strategies reusable across models, including future, stronger ones.Source: §5 · p. 10
  • Selective access to prior diagnostic experience lets the proposer inspect raw code, execution traces, and failures to form and test hypotheses about changes.The proposed advantage extends beyond searching code: feedback is not restricted to scalar rewards or fixed summaries.Source: §5 · p. 10
  • Code-space overfitting is more inspectable than weight-space overfitting because brittle if-chains and hard-coded class mappings are visible in the harness.Inspectability does not establish that overfitting is absent.Source: §5 · p. 10
  • Experiments span three domains but use one particularly strong proposer, Claude Code; whether the effect varies across proposer agents remains untested.A broader study across proposer agents is left to future work.Source: §5 · p. 10
  • Future work could co-evolve harnesses and model weights, allowing harness strategies and model learning to shape each other.Source: §5 · p. 10
  • The authors connect their findings to a recurring machine-learning pattern: accessible search spaces allow stronger general-purpose agents to outperform hand-engineered solutions.Source: §5 · p. 10

A Qualitative Proposer Behavior

A.1 File Access Statistics

The proposer extensively reads prior harness code and execution traces, routinely consulting most available history rather than only the latest parent.

A.1 File Access Statistics

  • 41% of file reads targeted prior harness source code and 40% targeted execution traces, versus 6% for score summaries and 13% for other files.Source: §A.1 · p. 15
  • The proposer’s access pattern is non-Markovian, routinely inspecting the majority of available history rather than only the most recent parent.Source: §A.1 · p. 15
  • 82 files per iteration was the median read count in the TerminalBench-2 search run, with a range of 69–99.Statistics cover 10 iterations using Claude Opus 4.6.Source: Table 8 · p. 15, §A.1 · p. 15

A.2 Qualitative Behavior: Causal Reasoning Over Prior Failures · B Discovered Harnesses

The TerminalBench-2 proposer diagnosed regressions from prior failures and pivoted to an additive environment snapshot that became the run’s best candidate. The discovered harnesses are domain-specific executable policies selected by search-set performance.

A.2 Qualitative Behavior: Causal Reasoning Over Prior Failures

  • 64.4% was the Terminus-KIRA baseline that both initial candidates underperformed after bundling structural fixes with prompt-template changes.The first combined marker stripping and a loop breaker with a cleanup-oriented prompt and verification checklist. The second removed pending completion while retaining marker stripping and the rewritten prompt.Source: §A.2 · p. 16
  • 63.3% accuracy (-1.1pp) after restoring the original prompt supported the proposer’s diagnosis that cleanup-heavy prompt changes confounded earlier structural fixes.The proposer attributed the earlier regressions to cleanup directives that caused necessary state to be deleted before task completion; the isolated candidate retained marker stripping and a loop breaker.Source: §A.2 · p. 16
  • Further completion-state, cleanup-prompt, and smart-waiting changes regressed, reinforcing the risk of modifying prompts and completion flow.Verification commands could reset completion flags, with observed failures spending 30–60 steps re-verifying effectively solved tasks. Smart-waiting instead polled unchanged terminal output before another model call, but also regressed.Source: §A.2 · pp. 16–17
  • Iteration 7’s additive environment snapshot became the run’s best candidate after six regressions, without changing existing harness methods.A single shell command gathers environment information before the first model call and appends it to the initial prompt, avoiding the fragile completion machinery.Source: §A.2 · p. 17
  • The proposer next attempted to combine the environment snapshot with marker stripping and a no-tool-call loop breaker to address independent failure modes.This composition deliberately avoided prompt and confirmation-flow changes; the supplied passages report its rationale, not its outcome.Source: §A.2 · p. 17
  • +18pp was the proposer-reported benefit of “don’t cleanup service artifacts” from earlier history, later reused across runs.Iteration 9 targeted the same idea but crashed before evaluation. The trajectory illustrates composition and cross-run transfer alongside diagnosis of earlier failures.Source: §A.2 · p. 17

B Discovered Harnesses

  • Discovered harnesses are domain-specific executable policies with routing, filtering, and conditional context construction, selected solely for improved search-set performance.Full implementations span roughly 100–1000 lines of code. The draft-verification classification example first predicts from short retrieved context, then retrieves evidence for and against that draft before predicting again.Source: B Discovered Harnesses · p. 17, Figure 5 · p. 18

B.1 Text Classification Harness · B.2 Math Retrieval Harness

Meta-Harness discovers text-classification harnesses spanning an accuracy–context frontier and a four-route BM25 retrieval program for mathematics. The classification endpoints interrogate labeled memory differently, while the math harness uses subject-specific retrieval policies.

B.1 Text Classification Harness

  • Meta-Harness (Draft Verification) drafts from five nearest labeled examples, then retrieves five same-label confirmers and five different-label challengers to maintain or revise its prediction.With fewer than 5 labeled examples, it falls back to a standard single-call few-shot prompt. Short retrieved contexts keep context cost near the frontier’s low end despite two model calls.Source: §B.1 · p. 18
  • Meta-Harness (Label-Primed Query) builds one larger prompt containing valid labels, one query-relevant example per label, and local contrastive pairs.TF-IDF retrieval and query-anchored pairing select similar examples with different labels, exposing the full label space and sharpening local decision boundaries.Source: §B.1 · pp. 18–19, Figure 6 · p. 19

B.2 Math Retrieval Harness

  • 40 iterations of evolution selected the math harness’s routing predicates, reranking terms, deduplication thresholds, and per-route example counts.The proposer autonomously merged two successful lineages: one supplied stronger raw-BM25 geometry retrieval, while the other supplied stronger combinatorics retrieval using deduplication and difficulty reranking.Source: §B.2 · pp. 19–20

B.1 Text Classification Harness

  • Meta-Harness (Label-Primed Query) achieves the highest accuracy on the discovered frontier, while Meta-Harness (Draft Verification) uses the least context; selection relies solely on search-set performance.Table 9 reports non-dominated variants; Ctx denotes average additional characters in input context, in thousands.Source: §B.1 · p. 18, Table 9 · p. 20

B.2 Math Retrieval Harness

  • The math harness assigns each problem to exactly one BM25 route: combinatorics, geometry, number theory, or a default for algebra and other problems.Lightweight keyword predicates and geometry-notation regex features select the route without aggregating outputs across routes. A math-aware tokenizer preserves LaTeX tokens as atomic units.Source: §B.2 · p. 20
  • Subject-specific policies emphasize combinatorics diversity, raw geometry matches, explicit number-theory proof strategies, and adaptive default example counts.Combinatorics fetches 20 BM25 candidates, deduplicates to 8, and returns the top 3 after lexical-score and difficulty reranking; geometry returns 1 hard NuminaMath reference plus 2 raw BM25 neighbors. Number theory reranks 12 BM25 candidates using lexical score, difficulty, and an early-technique bonus; default reranks 10 BM25 candidates using lexical score and difficulty, adapting example count to top-score concentration.Source: §B.2 · p. 20

B.3 TerminalBench-2 Harness

The discovered TerminalBench-2 harness adds an initial sandbox-environment snapshot to Terminus-KIRA, gaining on 7 of 89 tasks compared with that baseline.

B.3 TerminalBench-2 Harness

  • 30KB output capping, native tool calling, and a multi-perspective completion checklist are inherited from Terminus-KIRA.Native tool calling replaces Terminus 2’s ICL-based JSON parsing.Source: §B.3 · p. 21
  • 2–4 exploratory turns are eliminated by gathering a sandbox snapshot before the agent loop and injecting it into the initial prompt.A single compound shell command collects the working directory, /app contents capped at 20 entries, programming languages and versions, package managers, and available memory.Source: §B.3 · p. 21, Figure 9 · p. 22
  • 15-second timeout protection and silent failure keep the bootstrap from breaking the agent in unusual environments.The full implementation adds roughly 80 lines on top of Terminus-KIRA.Source: §B.3 · p. 21

Per-task analysis

  • 7 of 89 tasks improve over Terminus-KIRA, with the largest gains on protein-assembly and path-tracing.Gaining tasks require domain-specific tooling whose availability cannot be assumed. The analysis suggests bootstrapping is most valuable in non-obvious environments, especially when turn budgets are tight or early wrong assumptions cascade.Source: §B.3 · pp. 21–22

C Dataset Details … C.3 Math IMO-level Test Set

The dataset details cover out-of-distribution text classification benchmarks, a filtered and decontaminated math retrieval corpus, and a 200-problem IMO-level evaluation set.

C.1 OOD Text Classification Datasets

  • SciCite tests 3-way citation-intent classification, while FiNER-139 [29] tests word-level financial numeric entity recognition across 139 fine-grained XBRL types.SciCite identifies a citation’s rhetorical role from scientific context; FiNER-139 uses financial filings and is more fine-grained than standard sentence-level classification.Source: §C.1 · p. 22
  • Amazon Reviews [22] uses English reviews for 5-way star-rating prediction, while Financial PhraseBank [32] tests 3-way financial sentiment classification.Financial PhraseBank labels financial news and related economic sentences as positive, neutral, or negative with respect to market sentiment.Source: §C.1 · pp. 22–23
  • GoEmotions [15] tests 28-way emotion classification on English Reddit comments, while Banking77 tests intent detection across 77 online-banking customer-service intents.GoEmotions includes 27 emotion categories plus neutral, assessing nuanced affect beyond coarse positive-negative sentiment.Source: §C.1 · p. 23
  • AG News [60] tests 4-way news topic classification, SciTail tests science-domain textual entailment, and TweetEval (Hate) [7] tests binary tweet hate-speech classification.These benchmarks cover general-domain news, science-focused premise–hypothesis inference, and noisy short-form social-media text.Source: §C.1 · p. 23

C.2 Math Retrieval Corpus

  • Corpus decontamination removes matches against every evaluation benchmark and the harness-search set using exact prefix matching followed by fuzzy Jaccard similarity at threshold 0.8.A match under either criterion causes the corpus problem to be discarded.Source: §C.2 · p. 23
  • The selected harness retrieves only non-empty solutions shorter than 4,000 characters and truncates retrieved solutions to 3,000 characters when inserting them into prompts.During corpus preparation, OpenMathReasoning and DeepMath solutions are truncated to 5,000 characters. The geometry route also builds a separate hard-reference index from NuminaMath problems with difficulty greater than 6.Source: §C.2 · p. 23
  • 535K problems compose the math retrieval corpus, with competition-focused NuminaMath-1.5 filtering and OpenMathReasoning deduplication retaining the highest-pass-rate solution per problem.NuminaMath-1.5 filtering discards lower-quality web-scraped entries. Before deduplication, OpenMathReasoning excludes problems whose sources match IMO, AIME, HMMT, SMT, USAMO, or Putnam evaluation families.Source: §C.2 · p. 23, Table 10 · p. 24

C.3 Math IMO-level Test Set

  • 200 IMO-level evaluation problems come from IMO-AnswerBench, IMO-ProofBench, ArXivMath December 2025, and ArXivMath January 2026.The set combines a stratified 100-problem IMO-AnswerBench subset with all problems from the other three benchmarks, mixing answer-style, proof, and research-style problems. The passage calls for separate Base and Meta-Harness results by benchmark across five held-out models.Source: §C.3 · p. 23, Table 11 · p. 24

D Practical Implementation Tips · E Extended Related Work

The authors offer engineering lessons from three domains, rather than scientific claims about implementation choices. They distinguish Meta-Harness from neighboring approaches through executable harness optimization and selective filesystem access to prior code, scores, and execution traces.

D Practical Implementation Tips

  • The authors found skill refinement more influential than iteration count or population size, recommending constraints on outputs and safety rather than diagnostic procedures.The natural-language skill specifies roles, directory layout, commands, and outputs; short runs of 3–5 iterations help refine it. These recommendations are engineering lessons from three domains, not scientific claims.Source: D Practical Implementation Tips · p. 24
  • Roughly 50 full evaluations per run should remain feasible with a small, difficult search set that exposes failures of a simple baseline harness.The authors used 50–100 classification examples and 88 math-retrieval problems, favoring fast, discriminative evaluation over larger datasets.Source: D Practical Implementation Tips · pp. 24–25
  • Log source code, scores, and execution traces in machine-readable, hierarchically organized artifacts with consistent names that support reliable queries and regex searches.JSON is suggested as a suitable format.Source: D Practical Implementation Tips · p. 25
  • An optional CLI can simplify growing histories by exposing Pareto frontiers, top-k harnesses, and pairwise code and result differences.Relevant offline experience can share the same directory structure to warm-start exploration. This interface reduces tokens spent navigating artifacts.Source: D Practical Implementation Tips · p. 25
  • Validate candidates before expensive benchmarks, and delegate evaluation to a separate harness rather than the proposer.A tiny test should import the module, instantiate the class, and call both methods to catch malformed candidates in seconds. The external evaluator scores candidates and writes results to the filesystem.Source: D Practical Implementation Tips · p. 25

E Extended Related Work

  • Unlike AlphaEvolve [35] and OpenEvolve, Meta-Harness targets stateful harnesses with cascading effects and lets an unconstrained coding agent inspect prior artifacts.The comparison systems use structured program databases, tournament-selected parents, and fixed mutation strategies for algorithm optimization.Source: E Extended Related Work · p. 25
  • GEPA [1] uses candidate-specific reflection, whereas Meta-Harness lets its proposer selectively compare code, scores, and traces across all prior candidates.The distinction matters for harnesses whose behavior must be understood across many examples and candidates, rather than through short, isolated feedback loops.Source: E Extended Related Work · p. 25
  • LMQL, LangChain, and DSPy [23] organize LLM programs; Meta-Harness instead searches executable implementations of retrieval, memory-update, and orchestration policies.Those frameworks typically leave these policy choices to manual design, whereas Meta-Harness makes the harness implementation itself the optimization target.Source: E Extended Related Work · pp. 25–26

Related papers

Loading 2603.28052v1…