Source-linked AI summary
Meta-Harness: End-to-End Optimization of Model Harnesses
Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn
TL;DR
LLM performance depends heavily on harness code, yet harness engineering remains manual and existing text optimizers compress feedback that can span long execution horizons. Meta-Harness searches over harness code with a coding-agent proposer that selectively accesses prior candidates’ code, scores, and traces through a filesystem, and it improves results across online classification, math reasoning, and agentic coding. The findings indicate that richer access to prior diagnostic experience can support automated harness engineering.
Problem
Harnesses can substantially affect LLM performance, but their design remains largely manual and existing text optimizers are poorly matched to long-horizon, distributed harness feedback.
Method
Meta-Harness uses an agentic proposer to search over executable harness code while selectively inspecting prior candidates’ source code, scores, and execution traces through a filesystem.
Results
Meta-Harness improves results across online text classification, retrieval-augmented math reasoning, and agentic coding, including 7.7 points over ACE with 4× fewer context tokens on classification.
Takeaways & Limitations
Richer access to prior diagnostic experience can enable automated harness engineering across diverse LLM application settings.
Takeaways & Limitations
The experiments use one particularly strong coding-agent proposer, Claude Code, so variation across proposer agents remains for future study.
Abstract
from arXiv · showhide
The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.
1 Introduction
Meta-Harness targets the manual design of LLM harnesses, which can substantially affect performance and produce long-horizon diagnostic feedback. It uses an agentic proposer with filesystem access to prior code, traces, and scores, and improves performance across three evaluated domains.
- 6× performance gaps on the same benchmark show that harness code can matter substantially around a fixed LLM.
- Existing text optimizers are poorly matched to harness engineering because they compress feedback that may be needed to trace downstream failures to earlier decisions.Their per-step context budgets range from 100 to 30,000 tokens, while harness evaluations can produce up to 10,000,000 diagnostic tokens.
- Meta-Harness uses a coding-agent proposer that selectively inspects prior candidates’ source code, evaluation scores, and execution traces through a filesystem.The proposer reads a median of 82 files per iteration in the most demanding setting and references over 20 prior candidates per step.
- 7.7 points higher accuracy than ACE with 4× fewer context tokens was achieved on online text classification.Meta-Harness also matched the next-best text optimizer’s final performance after 60 proposals with only four.
2 Related Work
Meta-Harness connects harness optimization to credit assignment, adaptive external memory, executable code search, and text optimization. Its distinguishing focus is end-to-end search over executable harness procedures using distributed diagnostic feedback.
- Meta-Harness assigns credit at the harness level and rewrites external code governing future model behavior rather than updating model weights.
- External memory and adaptive access: Like retrieval-augmented generation and memory-based agents, Meta-Harness lets its proposer adaptively access external context instead of consuming all information in one pass.
- Executable code search: Unlike executable-code search methods that use fixed scaffolds or predefined operators, Meta-Harness searches over complete harness procedures.
- Text optimization methods: Compared with text optimizers, Meta-Harness handles executable procedures whose feedback is distributed across code, scores, and execution traces.
3 Meta-Harness: A Harness for Optimizing Harnesses
Meta-Harness repeatedly searches over task-specific harness programs by letting a coding-agent proposer inspect a growing filesystem of prior artifacts and evaluation results. It supports unconstrained diagnosis and edits while evaluating candidates and retaining a Pareto frontier.
- Objective: Meta-Harness searches for a harness H that maximizes expected reward for a fixed model M over task distribution X.A harness wraps the model, constructs prompts, updates state, and receives a task-specific trajectory reward.
- Meta-Harness search loop: Each evaluated harness contributes source code, scores, and execution traces to a filesystem that the proposer queries with tools such as grep and cat.The filesystem is typically larger than the proposer’s context window, so information is retrieved selectively rather than assembled into one prompt.
- Meta-Harness search loop: The proposer may inspect any prior harness, diagnose failures, and choose either local edits or substantial rewrites without a parent-selection rule.This design leaves diagnosis and edit decisions to the coding agent instead of hard-coding search heuristics.
- Algorithm: The outer loop initializes valid harnesses, evaluates them, stores their results, proposes new candidates, validates interfaces, and returns the Pareto frontier.
- Practical implementation: Harnesses are single-file Python programs that modify task-specific prompting, retrieval, memory, and orchestration logic.The experiments use Claude Code with Opus-4.6 as the proposer, while the base model varies by domain.
4 Experiments
Across online classification, retrieval-augmented math reasoning, and agentic coding, Meta-Harness searches over harness code and outperforms strong baselines while enabling controlled accuracy–context trade-offs.
- Experimental setup: Meta-Harness is evaluated on online text classification, math reasoning, and agentic coding against human-designed and program-search baselines.The experiments use standard domain-specific evaluation metrics and compare against hand-crafted harnesses and smaller-scale program-search methods.
- Online Text Classification: Meta-Harness matches prior text optimizers with 10× fewer full evaluations and surpasses their final accuracy by more than 10 points.The authors attribute this to preserving full experience history in a filesystem and allowing the proposer to inspect needed information.
- Online Text Classification: 48.6% accuracy: Meta-Harness outperforms ACE by 7.7 points and MCE by 8.6 points while using fewer context tokens.It uses 11.4K context tokens versus 50.8K for ACE and 28.5K for MCE.
- Online Text Classification: Meta-Harness produces a smooth accuracy–context Pareto curve, allowing additional context to be traded for higher test accuracy rather than fixing one operating point.The proposer can discover harnesses across a broad range of accuracy and context costs.
- Online Text Classification: 73.1% average accuracy: the selected harness outperforms ACE and all few-shot baselines on nine unseen datasets.It achieves the highest performance on 6/9 datasets, while adding more than 32 few-shot examples hurts performance in 7/9 tasks.
- Harnesses for Retrieval-Augmented Reasoning: 4.7 points: a single discovered retrieval harness improves average accuracy across five held-out models on 200 IMO-level math problems.It outperforms no retrieval across all five models and improves over BM25 retrieval by 1.3 points overall.
- Agentic Coding: 76.4% pass rate: Meta-Harness surpasses Terminus-KIRA on Opus 4.6, while 37.6% on Haiku 4.5 exceeds Goose by 2.1 points.On Opus 4.6 it ranks #2 among reported agents; on Haiku 4.5 it ranks first among the compared agents.
5 Discussion
Meta-Harness offers practical benefits beyond benchmark improvements, including transfer across datasets and models, readable strategies, and inspectable code-space overfitting. Its evaluation spans three domains, but the experiments use one particularly strong coding-agent proposer.
- Practical advantages: Discovered harnesses generalize to out-of-distribution classification datasets and unseen base models in the math setting.Search runs complete in a few hours and produce readable, transferable strategies reusable across models.
- Practical advantages: Code-space overfitting is more inspectable than weight-space overfitting because brittle if-chains and hard-coded class mappings are visible on inspection.
- Scope: Meta-Harness is evaluated on three diverse domains but with one particularly strong coding-agent proposer, Claude Code.A broader study of how the effect varies across proposer agents remains future work.
A.1 File Access Statistics
During a TerminalBench-2 search run, the proposer extensively inspected the filesystem, drawing roughly equally on prior harness code and execution traces rather than only recent candidates.
- File access: The proposer reads a median of 82 files per iteration, ranging from 69 to 99.The statistics come from 10 iterations using Claude Opus 4.6.
- File access: 41% of reads concern prior harness source code and 40% concern execution traces.Score summaries account for 6% and other files for 13%.
- Interpretation: The access pattern is non-Markovian because the proposer routinely inspects most available history instead of only the most recent parent.
A.2 Qualitative Behavior: Causal Reasoning Over Prior Failures
The TerminalBench-2 trajectory shows the proposer diagnosing regressions from prior candidates, isolating harmful prompt changes, and pivoting toward additive modifications. It also composes fixes and transfers lessons across runs.
- Iterations 1–2: Iterations 1–2 regressed after structural fixes were bundled with cleanup-oriented prompt changes.The shared prompt intervention confounded the effects of the structural changes.
- Iteration 3: The proposer identified prompt-template changes as the common cause of regressions, while the structural bugfixes were not isolated.The earlier candidates scored 58.9% and 57.8%, while the isolated version scored 63.3%.
- Iterations 4–6: The diagnosed completion bug reset pending completion during verification, trapping agents in repeated checklist cycles.The proposer linked this behavior to verification spirals lasting 30–60 steps after tasks were effectively solved.
- Iterations 4–6: Prompt and completion-flow modifications remained high risk despite plausible hypotheses, and smart-waiting also regressed.
- Iteration 7: After six regressions, iteration 7 shifted to an additive environment snapshot appended before the first LLM call and became the best candidate.The change avoided modifying the fragile completion machinery and aimed to reduce wasted exploration on dependency-heavy tasks.
- Iterations 8–10: The proposer subsequently attempted composition, transferred a +18pp cleanup lesson across runs, and summarized the trajectory as hypothesis-driven rather than random mutation.
- Discovered harnesses: Meta-Harness discovers executable, domain-specific inference-time procedures that can include routing, filtering, and conditional context construction.
B.1 Text Classification Harness
Meta-Harness discovers multiple text-classification harnesses that trade accuracy against context cost, alongside a four-route retrieval harness for mathematical reasoning. The classification variants differ in how they interrogate memory and construct prompts.
- Text classification: The text-classification search yields a Pareto frontier of non-dominated variants trading average accuracy against context cost.Draft Verification is the lowest-context endpoint, while Label-Primed Query is the highest-accuracy endpoint used in the main text.
- Text classification: Both representative classification harnesses maintain growing memories of labeled examples but use different memory-interrogation control flows.Draft Verification uses two short calls, whereas Label-Primed Query uses one larger call exposing labels and local decision boundaries.
- Draft Verification: Draft Verification first retrieves five nearest labeled examples for a draft, then retrieves five confirmers and five challengers conditioned on that draft label.With fewer than five labeled examples, it falls back to a standard single-call few-shot prompt.
- Draft Verification: Draft Verification stays near the low end of the context-cost frontier despite using two model invocations.
- Label-Primed Query: Label-Primed Query lists valid labels, adds one query-relevant example per label, and includes contrastive pairs with different labels.It uses TF-IDF similarity and query-anchored partner selection.
- Mathematical reasoning: The mathematical retrieval harness assigns each problem to exactly one of four routes: combinatorics, geometry, number theory, or a default route.Lightweight lexical predicates, including keyword sets and geometry regex features, select the route; only that route retrieves final-prompt examples.
- Mathematical reasoning: The four routes use distinct BM25 retrieval and reranking policies, including deduplication, difficulty, technique-early bonuses, and adaptive example counts.
B.3 TerminalBench-2 Harness
The discovered TerminalBench-2 harness preserves Terminus-KIRA’s core components while adding environment bootstrapping before the agent loop. This snapshot reduces exploratory turns and helps most on tasks requiring non-obvious domain-specific tools.
- Discovered harness: Environment bootstrapping is Meta-Harness’s main modification to Terminus-KIRA, injecting a sandbox snapshot into the initial prompt before the agent loop.The snapshot gathers available files, languages, package managers, and memory.
- Discovered harness: The harness inherits native tool calling, a 30KB output cap, and a multi-perspective completion checklist from Terminus-KIRA.
- Environment bootstrap: The snapshot includes the working directory, /app contents, programming-language versions, package managers, and available memory.Large /app directories are truncated to 20 entries, and the bootstrap command has a 15-second timeout.
- Results: Compared with Terminus-KIRA, the discovered harness gains on 7 of 89 tasks, with the largest improvements on protein-assembly and path-tracing.
- Results: The gains concentrate on tasks requiring domain-specific tooling whose availability cannot be assumed in advance.Without bootstrapping, agents spend their first 2–4 turns probing the environment; tight turn budgets can make those losses decisive.
C.1 OOD Text Classification Datasets
The out-of-distribution text-classification evaluation uses benchmarks spanning citation intent, financial sentiment, emotion, banking intent, news topics, textual entailment, and hate speech. The math retrieval evaluation aggregates 200 IMO-level problems from four benchmarks, while the retrieval corpus contains 535K problems.
- Text classification datasets: SciCite evaluates 3-way citation-intent classification from scientific-paper contexts labeled by rhetorical roles such as background, method, or result.
- Text classification datasets: FiNER-139 evaluates domain-specific financial sentiment classification with positive, neutral, and negative labels.
- Text classification datasets: GoEmotions is a 28-way emotion classification benchmark with 27 emotion categories plus neutral, using annotated English Reddit comments.
- Text classification datasets: Banking77 uses online-banking utterances labeled with 77 intents to evaluate single-domain intent detection.
- Text classification datasets: AG News is a 4-way benchmark for broad news-topic classification, while SciTail tests textual entailment in science-focused premises and hypotheses.
- Text classification datasets: TweetEval (Hate) is a binary task for detecting hateful versus non-hateful content in noisy, short-form social-media text.
- Math retrieval evaluation: The four math evaluation datasets mix answer-style, proof, and research-style problems, while some source datasets were filtered or deduplicated before corpus construction.
- Math retrieval evaluation: The math retrieval corpus contains 535K problems, and the main evaluation aggregates 200 IMO-level problems from four benchmarks.The evaluation combines a stratified 100-problem IMO-AnswerBench subset with all problems from three other benchmarks.
D Practical Implementation Tips
The paper’s implementation guidance emphasizes a constrained, inspectable, and efficient outer loop for harness search. It recommends useful baselines and search sets, queryable artifacts, lightweight validation, and evaluation outside the proposer.
- Designing the search loop: Meta-Harness can apply across domains, but new applications require LLM-assisted coding over long-horizon histories whose effects may appear many steps later.
- Designing the search loop: A strong skill specification should define the proposer’s role, directory layout, commands, output format, forbidden behavior, artifacts, and objectives.The skill should constrain outputs and safety-relevant behavior rather than prescribe the proposer’s diagnosis procedure.
- Designing the search loop: Start with a simple baseline and a difficult or diverse search set, because saturated baselines leave little for optimization.The paper recommends keeping the search set small enough for roughly 50 full evaluations per run.
- Making experience usable: Store code, scores, and execution traces in machine-readable, hierarchically organized files with consistent names that support reliable querying.
- Making experience usable: A small CLI can make growing experience stores easier to navigate by exposing Pareto frontiers, top-k harnesses, and pairwise code-result diffs.
- Validation and evaluation: Run lightweight validation before expensive benchmarks, and automate evaluation in a separate harness that writes candidate results to the filesystem.Tiny import-and-call tests catch many malformed candidates quickly, while the proposer need not run evaluations itself.
- Related work: Unlike AlphaEvolve and OpenEvolve, which evolve stateless functions using structured scalar feedback, Meta-Harness optimizes executable harness implementations.
- Related work: Unlike GEPA’s one-candidate-at-a-time reflection, Meta-Harness lets its proposer inspect all prior candidates and choose which code, scores, and traces to examine.