PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Harness Self-Improvement

Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Matei Zaharia, Yujin Tang

arXiv:2607.15524v1cs.LGcs.AI

TL;DR

Because continually updating provider-built harnesses is costly, the paper tests whether task-specific harness optimization can improve execution-trace quality for a fixed foundation model. It introduces RHI, which iteratively refines a prompt-level agent-loop specification, and reports higher win rates than stronger test-time settings alongside up to 60% lower inference cost than ultracode.

  • Problem

    The paper asks whether task-specific optimization of user-constructed harnesses can improve execution-trace quality for a fixed foundation model, given the labor and cost of continually updating provider-built harnesses.

    Source: §1 · p. 3

  • Method

    RHI treats the harness as a prompt-level agent-loop specification and iteratively revises it using pairwise self-comparisons between the current version and its predecessor.

    Source: §1 · p. 3

  • Results

    Up to 60% lower inference cost than ultracode on opus-4.8 accompanies few-shot RHI’s higher win rates than xhigh, max, and ultracode across sonnet-4.6, opus-4.7, and opus-4.8.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    RHI is presented as a practical framework for users to continually specialize coding agents for diverse open-ended tasks.

    Source: §1 · p. 4

    Evidence for RHI’s implicit update objective is correlational and depends on embedding-based estimators, so it does not prove the optimizer LLM’s true latent objective.

    Source: §6.3.4 · p. 25

Abstract

from arXiv · show

Under model--harness co-evolution, harnesses are not merely inference-time scaffolds but data-generating components whose execution traces can shape future foundation models. This motivates harness-in-the-loop learning: optimizing harnesses for both immediate agent performance and the quality of traces used for future model training. However, continually updating provider-built scaffolds is costly and labor-intensive. We therefore investigate whether optimizing user-constructed harnesses in a task-specific manner can improve execution-trace quality while remaining computationally lightweight and requiring only a few update iterations. To this end, we introduce Recursive Harness Self-Improvement (RHI), which represents the harness as a prompt-level specification of the agent loop and iteratively refines it using pairwise feedback over its own revision history. Across 30 synthetic machine-learning research tasks spanning quantitative finance, robotics, and pharmacy, a few RHI iterations suffice to substantially raise the performance ceiling of low-reasoning-effort agents, exceeding the corresponding maximum-reasoning-effort setting while reducing inference cost by up to 60%. We show that these gains arise primarily from improved task-specific context management through more effective inter-agent information flow rather than longer reasoning traces. Finally, we formalize this behavior as an information-theoretic hypothesis for RHI's implicit optimization objective, suggesting RHI as a practical algorithm for continual learning within the paradigm of model--harness co-evolution.

1. Introduction

Recursive Harness Self-Improvement (RHI) targets execution-trace quality by iteratively optimizing user-constructed, task-specific harnesses for a fixed model. Across 30 research tasks, it improves agent performance over stronger test-time-scaling settings while offering evidence that gains come from better task-specific information flow, not longer outputs.

Motivation

  • The work targets the execution-trace bottleneck in model–harness co-evolution, where stronger harnesses can produce traces for training future foundation models.It focuses on improving harnesses for a fixed foundation model, rather than optimizing both sides of the recursive loop.Source: §1 · p. 3

Method

  • RHI treats the harness as a prompt-level agent-loop specification and refines it through pairwise self-comparison between each revision and its immediate predecessor.The specification covers agent roles and instructions, inter-agent information contracts, and workflow hops; trajectory-local feedback replaces population-level search.Source: §1 · p. 3

Motivation and method

  • A few lightweight RHI updates substantially raise low-reasoning-effort agents’ performance plateaus, often exceeding the corresponding maximum-reasoning-effort setting.The harness is designed for task-specific user optimization under limited per-task budgets.Source: §1 · p. 3

Evaluation

  • Few-shot RHI on high-reasoning agents achieves higher win rates than stronger test-time-scaling settings (xhigh, max, ultracode) across sonnet-4.6, opus-4.7, and opus-4.8, cutting opus-4.8 inference cost by up to 60% versus ultracode.Source: §1 · p. 3

Mechanism

  • RHI gains accompany nearly constant output-token usage and are associated instead with more task-specific contracts and workflow hops and less redundancy across harness components.Cache read/write usage and inference cost often decrease across iterations.Source: §1 · p. 4

Implicit objective

  • The authors formalize the observed updates as an information-theoretic hypothesis: increase mutual information between each harness component and the task while penalizing redundancy across components.They present this as a testable account of RHI’s implicit update objective.Source: §1 · p. 4

2. Preliminary

The section formulates task-specific harness optimization as maximizing pairwise preference against competing harnesses, then reviews finite-candidate search methods and their cost limitations. It motivates lightweight optimization that can improve performance in only a few iterations.

2.1. Problem definition

  • A harness H paired with a task prompt x induces an agent execution that produces a code repository y.The agent uses a fixed language model; repositories consist of file-path/content pairs.Source: §2.1 · p. 4
  • Because repositories require assessment across multiple criteria, the section uses an LLM evaluator to compare outputs pairwise, including judgments of preference or equivalence.The criteria include functional correctness, task alignment, reproducibility, and code quality; the evaluator may differ from the coding agent's model.Source: §2.1 · p. 4
  • The ideal task-specific harness maximizes its expected pairwise win rate against competing harnesses under an evaluation prompt.The objective is defined relative to a reference distribution over competing harnesses.Source: §2.1 · pp. 4–5

2.2. Existing methods

  • Prior methods make harness search tractable by restricting candidate representations and evaluating a finite population, with approaches spanning harnesses, agents, workflows, programs, prompts, and pipelines.They rank candidates with task-specific scores, evaluators, or pairwise judgments; order-consistent rankings approximate within-population win rates.Source: §2.2 · p. 5
  • Every additional candidate requires an expensive black-box execution and evaluation, making population-based optimization prohibitive for users specializing agents across open-ended tasks.The section therefore calls for an approach with far lower search overhead that converges in a few iterations while still improving performance.Source: §2.2 · p. 6

3. Recursive Harness Self-Improvement

RHI iteratively refines a prompt-level harness using pairwise feedback between consecutive outputs and accumulated revision history, while prioritizing task-specific workflow and information exchange.

3.2. RHI algorithm

  • At each iteration, an LLM evaluator compares current and previous outputs, records pairwise feedback, and an LLM optimizer revises the harness; evaluation prompts guide updates only indirectly.The optimizer does not directly observe the evaluation prompt, but accumulated comparisons are generated by an evaluator conditioned on it.Source: Figure 2 · p. 6, §3.2 · p. 9

3.1. RHI objective function

  • RHI replaces global comparison with a trajectory-local comparison against the previous harness; under a standard pairwise-preference model, both objectives target the same utility ordering when competitors are independent of candidates.The local objective is not an unbiased estimator of the original population objective; its pairwise signal is noisy local ascent.Source: §3.1 · pp. 6–8
  • RHI requires one new execution trace and one pairwise evaluation per iteration, versus ideal population-wide comparisons and finite-population search’s quadratic pairwise evaluations.The comparison concerns per-iteration computational cost; scaling to n independent tasks multiplies each method’s cost by n without changing their ordering.Source: §3.1 · p. 7, Table 1 · p. 7
  • Because pairwise comparisons are noisy and textual harnesses lack conventional gradients, RHI accumulates preference history to condition updates and guide future revisions as a momentum-semantic signal.The history supports robustness while retaining one new agent execution and one pairwise evaluation per iteration.Source: §3.1 · p. 8

3.3. RHI harness definition

  • RHI represents harnesses as prompts specifying candidate-agent roles and instructions, workflow contracts and hops, and task-dependent auxiliary rules such as acceptance gates, fallback, and recall triggers.Contracts specify information passed between agents; hops specify their interaction structure.Source: §3.3 · pp. 9–10, Figure 4 · p. 10

4. Benchmark & Evaluation

The benchmark comprises 30 synthetic, open-ended ML research tasks across quantitative finance, robotics, and pharmaceutical ML. Evaluation uses pairwise LLM judgments of repositories against task-specific deliverables and multiple quality criteria.

4.1. Synthetic open-ended machine learning research tasks

  • The benchmark contains 30 synthetic ML research tasks—10 each in quantitative finance, robotics, and pharmaceutical ML—generated from industry job postings and requiring complete code repositories.Tasks combine coding, domain reasoning, empirical analysis, and ML experimentation, with objectives such as data analysis, model training, benchmark construction, ablation design, and empirical reporting.Source: §4.1 · pp. 10–11
  • Tasks share standardized deliverables—research_report.md, PNG visualizations, metrics.json, and index.json—while some also require domain-specific artifacts.The prompt specifies a research objective and required output artifacts.Source: §4.1 · p. 11

4.2. LLM-as-a-judge evaluation

  • Repository pairs are compared through LLM pairwise preference judgments using task-specified deliverables and a common interface enabled by the standardized structure.The evaluator selects a preferred repository for the same task, following the paper’s pairwise evaluation formulation.Source: §4.2 · p. 11
  • The evaluator assesses deliverable coverage, empirical rigor, reproducibility, presentation, engineering quality, and task alignment.Only task-specified deliverables are supplied; long files are truncated identically for both candidates, and evaluator context is kept to approximately 30–40% of its maximum input length.Source: §4.2 · p. 11

5. Experiment

Across three base models, a few RHI iterations raise performance beyond same-family test-time-scaling baselines while reducing inference cost. The gains are associated with more efficient context management, not simply longer outputs.

5.1. Experiment settings

  • The experiments compare RHI-improved high-reasoning agents with higher-reasoning baselines, changing the prompt by adding the evolved textual harness.The study uses Claude Sonnet 4.6, Opus 4.7, and Opus 4.8 agents.Source: §5.1 · p. 12

5.3. Claim 1. Few-shot RHI lifts the performance ceiling of test-time scaling

  • After two iterations, sonnet-4.6-high+H [2] wins 20 of 30 comparisons against sonnet-4.6-max; one iteration beats Opus-4.7 xhigh/max, and two beat Opus-4.8 xhigh/ultracode/max.The results extend across progressively stronger base models.Source: Figure 5 · p. 12, Figure 6 · p. 13, Figure 7 · p. 14

5.5. Claim 3. Few-shot RHI improves cost efficiency through reduced cache read/write usage

  • RHI improves cost efficiency across all base models, with cost and cache-use reductions relative to test-time-scaling baselines.For Sonnet-4.6-high+H [2], normalized cost is 2.38 versus sonnet-4.6-max at 2.56 (7% lower), and cache usage falls 33% (4.91 → 3.31). For Opus-4.7-high+H [1], cost is 2.11 versus opus-4.7-max at 2.60 (18% lower), and cache usage falls 37% (3.37 → 2.11).Source: Figure 5 · p. 12, Figure 6 · p. 13, §5.5 · pp. 15–16

5.2. Evaluation metrics

  • Evaluation compares pairwise wins, losses, and ties across 30 tasks, alongside normalized cost, output-token count, and cache read/write usage.Resource measures are normalized to each model’s high-reasoning base agent; results are averaged over three seeds and evaluator configurations vary by model.Source: §5.2 · pp. 12–13

5.5. Claim 3. Few-shot RHI improves cost efficiency through reduced cache read/write usage

  • For opus-4.8, high+H [2] costs 1.69 versus max at 2.19 and ultracode at 4.15, reducing cost by 23% and 60%, respectively.Cache read/write usage falls 32% versus max (2.51 → 1.69) and 64% versus ultracode (4.74 → 1.69).Source: Figure 7 · p. 14, §5.5 · p. 16

Claim 1.1. Optimizing user-constructed harnesses at the prompt level can outperform a system-level provider-built harness

  • RHI’s prompt-level harness optimization showed an advantage over opus-4.8-ultracode, supporting task-specific harness design as an explicit optimization variable rather than a fixed backend workflow.Source: Claim 1.1. Optimizing user-constructed… · p. 15

5.4. Claim 2. RHI performance gains are not explained by increased output-token usage

  • For Sonnet-4.6 and Opus-4.8, performance improves across RHI iterations while normalized output-token usage remains nearly flat.Usage ranges from 1.71 to 1.86 for Sonnet-4.6 and 1.42 to 1.81 for Opus-4.8; evidence for Opus-4.7 is inconclusive because usage rises with performance across only two iterations.Source: §5.4 · p. 15

5.5. Claim 3. Few-shot RHI improves cost efficiency through reduced cache read/write usage

  • RHI reduces cache read/write usage by learning task-specific communication contracts that pass downstream-relevant information instead of full interaction histories.This limits redundant context propagation, which the authors connect to lower inference cost.Source: §5.5 · p. 16

6. Ablation Study

The ablation study examines how RHI changes harness structure and measured component dependence across configurations. It presents an information-theoretic account and an embedding-based, correlational hypothesis linking task-specific coordination and specialization with reduced redundancy.

6.1. Can RHI also raise the performance plateau of train-time scaling?

  • RHI substantially improves sonnet-4.6-high-high, with gains plateauing after 2–4 iterations, but does not consistently close the gap to opus-4.7-high or opus-4.7-xhigh.The comparison is shown in Figure 8.Source: §6.1 · pp. 16–17, Figure 8 · p. 17

6.2.1. Full-harness evolution

  • Full-harness embeddings separate the initial harness from RHI-updated harnesses across embedding models and projections, while cosine similarity falls from 0.82 to 0.78, 0.76, and 0.75 relative to the initial harness.Consecutive similarity is 0.82 for the first update and 0.97, 0.98, and 0.99 for later transitions, suggesting the largest semantic change occurs in the first update.Source: §6.2.1 · pp. 17–18, Figure 9 · p. 18
  • Within-domain similarity between evolved harnesses increases after the first RHI iteration, indicating shared domain-level structure even as harnesses diverge from their initial specification.The authors then turn to component-level analysis because whole-harness embeddings aggregate roles, instructions, contracts, and hops.Source: §6.2.1 · pp. 18–19

6.2.2. Harness-component evolution

  • Across domains, contracts show the clearest task-dependent clustering, with hops and instructions less distinct and roles least separable, consistent with gains from task-specific coordination rather than longer reasoning traces alone.The embedding analysis is correlational, not causal.Source: §6.2.2 · p. 19, Figure 10 · p. 20
  • Contract similarity across consecutive iterations rises from 0.48 to 0.66, 0.69, and 0.72, while role similarity changes more gradually from 0.28 to 0.31, 0.33, and 0.31.The authors suggest contracts specialize within a few iterations, possibly because pairwise evaluator feedback provides a stronger learning signal for contract refinement.Source: §6.2.2 · p. 21, Figure 11 · p. 21

6.3.1. Evidence for an implicit objective function

  • The observed trajectory is systematic and task-dependent rather than arbitrary or generic, but it does not prove that RHI optimizes a unique scalar objective.These observations motivate modeling RHI as guided by an implicit preference over task-specific harness components.Source: §6.3.1 · p. 22

Hypothesis: information-theoretic implicit objective of RHI

  • The proposed information-theoretic hypothesis combines prompt-controlled task information in contracts and hops with model-dependent functional specialization that favors components carrying non-overlapping information after conditioning on the task.The specialization term is intended to avoid degenerate harnesses whose components repeat the same task description or coordination rule.Source: Hypothesis: information-theoretic… · pp. 23–24

6.3.3. Evidence for increasing 𝑓ext

  • Contracts and hops increase task mutual information monotonically from RHI iterations 1 to 4 across all encoder and debiasing configurations, whereas roles decrease and instructions remain approximately flat or mildly decrease.This pattern is consistent with the optimizer prompt selectively increasing task information in the components it emphasizes.Source: §6.3.3 · p. 24, Table 2 · p. 24

6.3.4. Evidence for decreasing 𝑓int

  • TC|task decreased from 4.84 to 3.63 nats with text-embedding-3-large and from 3.51 to 2.62 nats with all-mpnet-base-v2; unconditional total correlation also decreased across configurations.TC|task decreased monotonically from iteration 1 to 4 in all four configurations.Source: Table 3 · p. 25, §6.3.4 · p. 25
  • RHI makes contracts and hops more task-specific while reducing residual redundancy among harness components.The proposed implicit update objective is task-specific coordination that encodes workflow-relevant information in contracts and hops while discouraging duplicated information across components; this evidence is correlational and depends on embedding-based estimators, not proof of the optimizer LLM’s latent objective.Source: §6.3.4 · pp. 25–26
  • Because hop is constant across agents for a fixed task and iteration, per-task centering removes it, so TC|task effectively measures TC(role, instr, cont | task).Hop contributes to unconditional total correlation but not to this per-agent task-conditional estimate.Source: §6.3.4 · p. 25

7. Related Work

RHI is positioned as optimization of a prompt-represented multi-agent harness around a fixed foundation model, at the intersection of recursive self-improvement, harness and prompt optimization, and multi-agent coordination. It changes the harness rather than training a stronger model or merely allocating more test-time reasoning.

Recursive self-improvement and harness-level RSI

  • RHI is a bounded harness-level RSI process that rewrites and reuses a prompt-represented harness from prior self-comparison while keeping the base model, evaluator, and optimizer fixed.Its editable specification includes agent roles, instructions, workflow hops, and communication contracts.Source: §7 · p. 26

Harness optimization and system scaling

  • Unlike TTHE (Nie et al., 2026), which evolves parallel branches and selects with execution-derived proxies, RHI uses one trajectory-local pairwise preference comparison per iteration, including for open-ended tasks.RHI and TTHE both adapt harnesses on the target task, but use different update signals and search procedures.Source: §7 · pp. 26–27

Agent design, workflow search, and program evolution

  • ADAS (Hu et al., 2025), GPTSwarm, and AFlow (Zhang et al., 2025c) optimize agents, graphs, or workflows; RHI edits only textual harnesses for black-box agents.Related approaches commonly use executable workflows, code-level control, or candidate populations, unlike RHI's single textual harness.Source: §7 · p. 27

Prompt and pipeline optimization

Multi-agent systems and test-time scaling

  • Because multi-agent gains can reflect added test-time computation rather than better coordination, the study compares scaling baselines and measures tokens, cost, and cache reads and writes.The comparisons include a default harness, stronger same-family test-time scaling baselines, and a built-in multi-agent coding harness.Source: §7 · p. 27

8. Conclusion

The conclusion frames execution-trace quality as a priority for harness–model co-evolution and presents task-specific RHI as one practical approach. It identifies internalizing the resulting traces into future foundation models as future work.

8. Conclusion

  • RHI offers a practical approach to improving post-training execution traces through task-specific optimization of user-constructed harnesses.The paper argues that these traces can serve as data for future foundation models.Source: §8 · p. 27
  • Future work will investigate how to internalize the resulting execution traces into future foundation models.Source: §8 · p. 28

A. QnAs

The Q&As explain why RHI can use local pairwise updates, why its harness revisions stabilize, and why explicit multi-agent coordination can improve cost–performance over default scaffolds. They also identify multi-agent contracts and hops as a focus of harness improvement.

A. QnAs

  • Cosine similarity between consecutive harnesses and their components converges toward 1 over RHI iterations, indicating revisions settle rather than oscillate.The paper defines learning stability as how much textual information H[i] differs from H[i+1].Source: A. QnAs · p. 31
  • Although each update compares against the previous harness, compressed pairwise comparisons across harness history remain in D(i)_x as a momentum signal.Thus, earlier history is retained in compressed form rather than discarded.Source: A. QnAs · p. 31
  • RHI prioritizes multi-agent contracts and hops because built-in workflows coordinate unreliably, while an explicit multi-agent harness outperforms a single-agent multi-persona harness.The results attribute multi-agent effectiveness to coordination quality, including specialized agents, independent execution contexts, and explicit orchestration.Source: A. QnAs · pp. 31–33
  • Built-in multi(default) had lower Elo than single(default) at substantially higher cost, while explicit H (0) improved multi-agent cost–performance across both base models.For opus-4.7-high, multi(ours) achieved higher Elo at lower normalized cost; for sonnet-4.6-high, it achieved substantially higher Elo at comparable or slightly lower cost.Source: Figure 13 · p. 32, A. QnAs · pp. 32–33
  • multi(ours) consistently achieved higher Elo than single(unionOur), showing that added roles and instructions alone do not explain the gains.single(unionOur) was competitive and sometimes beat single(default), but the best performance came from executing the specification as a multi-agent workflow.Source: Figure 13 · p. 32, A. QnAs · p. 33

B. Examples of RHI harness … Harness H [2]

This appendix traces prompt-level harness revisions for a pharmaceutical ML research task and describes how Harness H[2] structures specialist collaboration, validation, and review.

B. Examples of RHI harness

  • The example task requests a reproducible SE(3)-equivariant training recipe for side-chain χ-angle prediction using PDB-derived data, angular-error evaluation, and speed and memory profiling.The harness variants use sonnet-4.6-high and RHI iterations 0–4; their outputs contribute to Figure 5a evaluation.Source: B. Examples of RHI harness · p. 33

Initial Harness (H [0])

  • H[0] assigns a broad drug-discovery research team to orchestration, structural data, geometric ML, generative modeling, training, evaluation, publication, and quality review.The orchestrator is instructed to decompose work, integrate specialist outputs, and enforce rigor, reproducibility, and safety boundaries.Source: Initial Harness (H [0]) · pp. 34–35

Harness H [1]

  • H[1] replaces one-pass delegation with staged contract negotiation, structured handoffs, independent artifact reviews, and targeted specialist recalls for blocking issues.Its mandatory dependencies include joint dataset and model/ablation contracts before implementation, with evaluation and profiling contracts required before quantitative reporting.Source: Harness H [1] · pp. 36, 38, 39

Harness H [2]

  • H[2] makes exact deliverable paths, contract acknowledgements, provenance artifacts, and early smoke checks explicit, targeting prior path, data-evidence, and report–profiling inconsistencies.It distinguishes required src/profile.py from src/profiler.py and calls for patch-and-retest cycles until blockers are closed or waived.Source: Harness H [2] · p. 40
  • An independent quality-and-safety agent audits leakage, reproducibility, equivariance claims, profiling, metrics, provenance, deliverable paths, and overclaiming before approving the computational-only workflow.Final approval requires concrete evidence and no blocking leakage or uncorrected report-metric contradictions.Source: Harness H [2] · p. 44
  • Every agent response must use a machine-checkable HandoffV2 object specifying inputs, interface exports, artifacts, validation checks, risks, open questions, and recall recommendations.Mandatory handoffs establish contracts before data, model, training, evaluation, profiling, and reporting dependencies proceed.Source: Harness H [2] · pp. 44–45
  • The orchestrator follows ten hops from evidence-rubric fan-out and contract acknowledgment through implementation, frozen results, independent critique, targeted patches, and retesting.Blocking contract changes trigger reconciliation before implementation, while patch recalls narrow work to the issue owner and required retests.Source: Harness H [2] · p. 45
  • Recall triggers route missing provenance, leakage checks, invariance tests, ablations, metrics, profiling evidence, or report inconsistencies back to the responsible specialist.Missing src/profile.py is always blocking; a report speed ratio differing from profiling JSON by more than 10 percent also triggers recall.Source: Harness H [2] · p. 45
  • Final passage requires exact-path artifacts, reproducible PDB provenance, leakage-free inputs, required invariant and equivariant variants, robust profiling, consistent report metrics, five non-empty plots, and logged smoke checks.The gate also requires deliverables/index.json and logs.txt to record required artifacts and workflow evidence.Source: Harness H [2] · p. 46

Harness H [3]

This section specifies Harness H, a staged, contract-driven multi-agent workflow for building and validating a reproducible SE(3) side-chain χ1 training pipeline. It uses specialist handoffs, independent checks and artifact audits, targeted repairs, documented disagreement resolution, and a final evidence-based gate.

Orchestration and acceptance

  • The orchestrator runs contract negotiation, implementation, smoke and equivariance gates, full evaluation, evidence integration, and targeted patch–retest cycles against a tracked defect register.The workflow preserves v2 strengths while restoring formal equivariance tests, detailed runtime and memory profiling in results/metrics.json, stronger profiling evidence, and a non-stale entry point.Source: Harness H [3] · p. 46

Data and geometry

  • Data and geometry specialists must document RCSB acquisition and splits, define χ1 labels and residue handling, and verify side-chain coordinates never enter model inputs.Required provenance artifacts include query, response-ID, download, data-card, and split manifests; data and geometry jointly co-sign DatasetRecord v3 before downstream consumers use it.Source: Harness H [3] · pp. 47, 48, 53

Model and experiments

  • Model and ablation agents specify invariant χ1 outputs, equivariant internal features, one invariant and at least two equivariant variants, and independent transformation tests.The referee tests synthetic and real batches, records results/equivariance_tests.json, and blocks unsupported claims when testing fails or cannot run.Source: Harness H [3] · pp. 48, 49, 53

Profiling and metrics

  • Profiling and metrics agents must validate latency and interpretable memory evidence, compute throughput ratios, and merge runtime, profiling, and equivariance summaries into results/metrics.json.Independent checks recompute ratios and inspect memory anomalies; exact-path validation requires src/profile.py, and metrics completeness checks require runtime and profiling fields.Source: Harness H [3] · pp. 50–52

Reporting and final audits

  • Independent reviewers cross-check report claims, index entries, manifests, plots, logs, and artifacts against source evidence before final acceptance.The reporting agent drafts only from frozen contracts and number-source mappings; claims, quality, validation, and log audits check consistency, reproducibility, and unresolved risks.Source: Harness H [3] · pp. 51–53

Contracts and handoffs

  • Agents exchange machine-checkable HandoffV3 records, while frozen path and interface contracts, field-level acknowledgements, and retested PatchHandoffV3 updates govern implementation.RequiredPathManifest v2 and InterfaceContract v3 precede implementation; mandatory handoffs coordinate data, geometry, model, ablation, training, evaluation, profiling, and equivariance work.Source: Harness H [3] · pp. 47, 53

Harness H [3]

  • Independent reviewers audit frozen artifacts and may open recall tickets; after patches, the original reviewer and validator or quality retest the specific issues.The orchestrator narrows each repair to its owner, required changed files, issue IDs, and retest criteria; checks repeat until no blockers remain or a waiver is recorded.Source: Harness H [3] · p. 54
  • At least 17 orchestrator–subagent hops carry work from evidence fan-out and contract acknowledgement through implementation, review, targeted patches, and retesting.Contract conflicts trigger reconciliation before implementation; the final retest loop continues until blockers are cleared or explicitly waived with evidence and rationale.Source: Harness H [3] · p. 54
  • Early artifact validation checks paths, imports, provenance, dataset keys, and smoke commands before expensive training or profiling proceeds.An independent equivariance gate runs before reporting; later stages freeze evaluation, profiling, and integrated metrics before plots and report drafting.Source: Harness H [3] · p. 54
  • When agents disagree, the orchestrator records positions, evidence, affected artifacts, downstream fields, and its decision, favoring the scientifically conservative option.Contract changes after implementation require renewed acknowledgement from affected consumers and targeted retesting.Source: Harness H [3] · p. 55

Harness H [4] · C. Harness optimizer prompt · 1. Following query solved with claude coding

This section describes a harness for a reproducible SE(3) side-chain χ1 training pipeline, using contract-gated agent workflows, evidence checks, and independent audits. It also presents an optimizer prompt focused on artifact-grounded improvements to inter-agent communication and iterative delegation, with execution and workflow logs required.

Harness H [4]

  • The orchestrator assigns defect-driven rounds, freezes shared contracts before implementation, and requires evidence-backed acceptance gates and tracked retests.Required artifacts include a path manifest, interface contracts, metrics-pointer matrix, defect register, issue tracker, and final gate decision.Source: Harness H [4] · pp. 56–57
  • Data acquisition must record RCSB query attempts and verify each fallback PDB ID, while geometry checks define χ1 labels and prevent side-chain coordinates from entering model inputs.A failed query cannot be described as returning a predefined fallback set; provenance, metadata, and leakage checks are required.Source: Harness H [4] · pp. 57–58
  • Model and experiment agents coordinate invariant and equivariant variants, fair ablations, reproducible training histories, circular evaluation metrics, and independent transformation tests.The prompt requires an invariant baseline plus at least two equivariant variants, and tests on synthetic and real batches with command evidence.Source: Harness H [4] · pp. 58–60
  • Profiling must report stable timing statistics and memory measures, then integrate runtime, training-step latency, throughput, and provenance into metrics with cross-artifact consistency checks.The prompt flags high-variance or nonmonotonic timings and requires checking metric values against histories, checkpoints, and profile results.Source: Harness H [4] · pp. 60–61
  • Publication claims must trace to frozen evidence, while independent validators, quality reviewers, and log auditors check deliverables, provenance, reproducibility, and unresolved risks.The gates cover report-number and fallback wording, exact paths, stale entrypoints, required metrics, audit artifacts, commands, agent handoffs, and final decisions.Source: Harness H [4] · pp. 62–63
  • HandoffV4 requires machine-checkable records for consumed inputs, exports, artifacts, validation, metrics, evidence, risks, agent requests, and prior-defect audits.The schema also specifies blocking status and structured issue and recall recommendations.Source: Harness H [4] · p. 64
  • The minimum workflow spans 24 hops, from evidence fan-out and contract reconciliation through targeted patch recalls, retests, and a final gate.The orchestrator recalls only disputed parties or issue owners, with narrowed scope and retest criteria.Source: Harness H [4] · pp. 65–66
  • Recall triggers and final-gate checks enforce required artifacts, data provenance, leakage controls, metric consistency, profiling evidence, and report-to-JSON agreement.Missing or inconsistent evidence routes work back to the responsible specialist; the gate also checks plots, logs, and exact paths.Source: Harness H [4] · p. 66
  • When agents disagree, the orchestrator records evidence, affected artifacts, decisions, and re-acknowledgement needs, favoring conservative scientific claims.Decisions go in logs.txt, with limitations documented in research_report.md.Source: Harness H [4] · p. 66

C. Harness optimizer prompt

  • The optimizer prompt restricts revisions to the multi-agent design and prioritizes stronger communication contracts and iterative recall over one-pass execution.It requires changes to be grounded in submission artifacts and comparison history, with valid JSON output.Source: C. Harness optimizer prompt · p. 67

1. Following query solved with claude coding

  • The user prompt requires uv for Python workflows and logs documenting plans, execution, reflection, spawned agents, and inter-agent workflow.The prompt also includes placeholders for the task, current multi-agent design, and save rules.Source: §1 · p. 67

2. Current submission code repo from Claude Code (evidence only) · 3b. Pairwise history delta checklist (recurring issues to fix) · 4. Instructions

The v0 and v1 submissions were judged a tie: they shared the same workspace, artifacts, strengths, and weaknesses. The review identified recurring quantitative-quality defects and asked v2 to improve evidence-grounded multi-agent collaboration and verification.

2. Current submission code repo from Claude Code (evidence only)

  • The report calls 5-day GBM IC 0.039 significant despite p=0.122, and claims ElasticNet beats Ridge and GBM although its 5-day IC=-0.0458 versus 0.0896 and 0.0391.The report names GBM as primary, while metrics metadata names elasticnet and trading metrics names elasticnet_h5_wf.Source: §2 · p. 68
  • Both submissions lack substantive pre/post-2020 stability analysis: pre-2020 metrics are empty, stability scores are NaN/null, and model-comparison stability scores are null.The metrics JSON excerpt also contains non-standard NaN tokens, a reproducibility and manifest-quality defect.Source: §2 · p. 69
  • The excerpts do not establish that proxy-shock features are lagged safely: their construction uses post-event 3–5-day moves before overlapping 1–20-day return prediction.Both submissions use plausible time splits and costs, but the available evidence does not resolve this potential feature-leakage risk.Source: §2 · p. 69
  • Neither submission was better: v0 and v1 had identical workspaces, directory trees, and cited file excerpts, with indistinguishable strengths and weaknesses.Both materially covered the requested artifact checklist, but shared the same quantitative-quality issues.Source: §2 · pp. 68–69

3b. Pairwise history delta checklist (recurring issues to fix)

  • The recurring-issues checklist marks report-metrics inconsistency, stability/robustness gaps, null/NaN defects, reproducibility gaps, and validation/test weakness once each.The instructions say to prioritize issues with higher history counts.Source: 3b. Pairwise history delta checklist… · p. 69

4. Instructions

  • For v2, preserve the original intent and deliverables while creating genuine multi-agent advantages through specialist parallelism, cross-agent validation, communication, and conflict resolution.Strengthen structured outputs for downstream agents and let the orchestrator recall subagents for narrower follow-ups with updated acceptance criteria.Source: §4 · p. 69
  • The redesign must be specific and evidence-grounded, explain mechanisms and expected effects rather than making generic coordination claims, and leave non-design parts unchanged.It should refer to concrete files, metrics, or history signals.Source: §4 · pp. 69–70
  • The response must provide a replacement design, changes from the prior version, rationale, evidence used, expected impact, and verification checks.The design block must start with “Create an agent team with the following agent candidates to solve this problem:”.Source: §4 · p. 70

D. Task examples

The appendix presents representative examples from a 30-task evaluation benchmark spanning robotics, pharmacy, and quantitative research.

D. Task examples

  • The benchmark covers 30 tasks across three domains—robotics, pharmacy, and quantitative research—with 10 tasks in each domain.These are representative examples of the task domains used in the evaluation benchmark.Source: D. Task examples · p. 70

D.1. ML Research Task (Robotics)

The robotics task specifies a reproducible fine-tuning and evaluation experiment for a small vision-language model on robot-relevant instruction classification, with robustness testing and documented deliverables.

D.1. ML Research Task (Robotics)

  • The experiment fine-tunes a small vision-language model for robot-relevant instruction classification and evaluates accuracy, calibration, and robustness to paraphrases.Suggested data are EPIC-KITCHENS captions aligned with actions or a curated Something-Something V2 subset.Source: §D.1 · p. 71
  • Deliverables include a 6–9-page-equivalent conference-style report, plots, an index, and reproducible code for data preparation, fine-tuning, evaluation, and paraphrase generation.The code must include requirements.txt.Source: §D.1 · p. 71
  • The requested results report Top-1 accuracy, Macro F1, ECE, paraphrase robustness drop, per-class metrics, and comparisons of frozen, fine-tuned, and partially fine-tuned variants.The model comparison includes corresponding hyperparameters.Source: §D.1 · p. 71

D.2. ML Research Task (Quantitative)

This task tests whether Wikipedia pageviews improve short-term return prediction for large-cap equities, using a Dow 30 universe and SPY as the benchmark. It specifies attention features and leakage-safe predictive and trading evaluations.

D.2. ML Research Task (Quantitative)

  • The task investigates whether Wikipedia pageviews improve short-term return prediction for Dow 30 constituents, with SPY as the benchmark.The task uses daily pageviews, prices, and trading volume, with optional macroeconomic controls from FRED.Source: §D.2 · p. 72
  • Attention features include abnormal pageview surges, attention momentum, day-of-week adjustments, and interactions with earnings dates.Abnormal surges are z-scores relative to a trailing baseline; earnings dates come from a public calendar, with fallback procedures documented if scraping is unstable.Source: §D.2 · p. 72
  • Evaluation requires leakage-safe next-day and next-week return or volatility prediction, purged time-series cross-validation, and a market-neutral long/short backtest with transaction costs.A placebo test shifts pageviews forward by seven days and checks that the predictive signal disappears.Source: §D.2 · p. 72

D.3. ML Research Task (Pharmacy)

The pharmacy task evaluates whether combining ESM-2 protein-sequence embeddings with PDB-derived graph features improves enzyme commission number prediction over sequence-only and structure-only representations. It specifies public enzyme data and reproducible research deliverables for evaluating fusion methods.

D.3. ML Research Task (Pharmacy)

  • The task compares sequence-only, structure-only, and fused ESM-2 sequence embeddings and PDB-derived graph features for predicting enzyme commission numbers on a public dataset such as Swiss-Prot.The proposed data sources include UniProt/Swiss-Prot annotations and PDB mappings through SIFTS.Source: §D.3 · p. 73
  • Success is defined as determining whether fusion improves prediction for proteins with known structures, using macro and micro F1, EC-level hierarchical accuracy, and coverage statistics.The metrics are specified for results/metrics.json.Source: §D.3 · p. 73
  • Deliverables include an 8–10-page conference-style report, an index, evaluation plots, and reproducible Python code spanning data processing through evaluation.The report should describe fusion architectures and dataset split strategy.Source: §D.3 · p. 73
  • The task requires ablation results comparing concatenation, attention, and gating fusion variants with their configurations and evaluation metrics.These results are specified for deliverables/ablation_results.json.Source: §D.3 · p. 73

E. Evaluator Prompt · F. Distribution of normalized cost, output tokens, and cache read/write

Leval is specified as an evidence-driven pairwise evaluator for completed submissions, using explicit task requirements and a multi-part comparison rubric. Figure 14 summarizes normalized cost, output-token, and cache read/write distributions across 30 tasks, with trends consistent with means reported in Figures 5b, 6c, and 7e.

E. Evaluator Prompt

  • Leval is framed as a strict, evidence-driven senior quantitative researcher judging rigor, out-of-sample and leakage discipline, uncertainty, baselines, reproducibility, and consistency between narrative and numeric artifacts.Source: E. Evaluator Prompt · p. 73
  • The task block alone defines required outputs; workspace content is evidence, and reviewers should use explicit deliverables and concrete citations rather than assumptions or generic praise.Source: E. Evaluator Prompt · p. 73
  • The evaluator should choose A or B when one submission is clearly better overall, use a tie only for genuinely comparable submissions, and return only the specified JSON keys.Source: E. Evaluator Prompt · pp. 73, 87
  • The sample assignment asks for robot-plan success-probability calibration from simulated executions, measuring ECE, AUROC, and decision utility, with specified reports, plots, code, and metrics files.Source: E. Evaluator Prompt · p. 74
  • The comparison rubric weighs deliverable coverage, empirical rigor, reproducibility, presentation, engineering, and alignment with the task.Source: E. Evaluator Prompt · p. 87

F. Distribution of normalized cost, output tokens, and cache read/write

  • Figure 14 shows distributions of normalized cost, output tokens, and cache read/write across 30 tasks, with trends consistent with means in Figures 5b, 6c, and 7e.The caption does not provide distribution values.Source: Figure 14 · p. 88

Related papers

Loading 2607.15524v1…