Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents

1 Introduction

• Co-Harness alternates harness improvement from failures with model fine-tuning on high-quality trajectories produced by the improved harness.HarnessCritic proposes local diffs, which are accepted only when validation improves the targeted failure mode without regressing held-out behaviors.Source: §1 · p. 2
• HarnessCritic attributes failed trajectories to structured harness-level causes and proposes validated local repairs across prompts, tools, skills, and middleware.This makes harness changes part of the post-training loop rather than inference-time scaffolding around a fixed model.Source: §1 · p. 2
• +20.4 pp average accuracy improvement across Qwen3-8B and Qwen3-32B follows two Co-Harness rounds on AIME24, AIME25, and HMMT25.The largest gain is +27.2 pp on HMMT25 for Qwen3-32B.Source: §1 · pp. 2–3

Tool-Integrated Reasoning

• Tool-Integrated Reasoning interleaves chain-of-thought with executable code, and a tool-schema error or misconfigured retry hook can abort a multi-turn session.Prior work reports that TIR can unlock strategies such as iterative search, dynamic programming, and hypothesis-driven exploration [14].Source: §2 · p. 3

Automated Harness Design

• Meta-Harness, Agentic Harness Engineering [9], and AEVO [10] optimize harness components or procedures without changing the base model.Co-Harness instead uses accepted harness edits to change supervision trajectories and then updates the model with those trajectories.Source: §2 · p. 3

Automated design of agentic systems

• ADAS [15] searches a broader agent-design space, while HarnessCritic targets explainable repair of known failure patterns using failure trajectories as its signal.ADAS optimizes in-context task accuracy at inference time rather than the quality of training trajectories.Source: §2 · p. 3

3.1 Overview and Problem Formulation

• The joint objective optimizes model parameters and a five-part Harness, but full-rollout evaluation, sparse rewards, and changing edit relevance motivate alternating optimization.The Harness comprises prompts, tools, skills, middleware, and long-term memory policy.Source: §3.1 · p. 4

3.2 Dual-Loop Co-Evolution Architecture

• The Co-Harness Loop holds the model fixed while repairing scaffolding; the Model Alignment Loop holds the evolved Harness fixed and fine-tunes the model on its higher-quality trajectories.Model updates are fed into the next co-evolution round.Source: §3.2 · p. 4

3.3 HarnessCritic: Evolving the Harness from Failure Trajectories

• HarnessCritic turns failures into structured causes and local patch suggestions; it aggregates recurring edits, while agent-error cases are excluded from Harness patch generation.The taxonomy distinguishes actionable Harness failures from non-Harness agent errors; local diffs support interpretability and rollback.Source: §3.3 · p. 5, Table 4 · p. 5

3.4 Model Alignment Loop

• The Model Alignment Loop fine-tunes on trajectories that pass the task verifier or benchmark quality filters, distilling behavior enabled by the improved Harness into model parameters.A stronger model may expose failure regimes that were masked by weaker behavior.Source: §3.4 · p. 6

3.5 Patch Validation and Harness Registry

• A Harness patch is accepted only when it improves its targeted failure mode without held-out regressions; accepted patches enter a versioned registry that enables audit and rollback.The stated validation conditions are δin > 0 and δout ≥ 0.Source: §3.5 · p. 6

3.6 Why the Loop Can Compound

• Compounding requires better trajectory quality, model internalization, and stronger models that reveal useful new Harness opportunities; the paper tests this feedback-loop hypothesis.If any condition fails, the process reduces to single-loop optimization with diminishing returns.Source: §3.6 · p. 6

4.1 Setup

• The experiment evaluates Qwen3-8B and Qwen3-32B on 30-problem AIME24, AIME25, and HMMT25 pass@1 tasks using Python-interpreter Tool-Integrated Reasoning and two co-evolution/SFT rounds.Source: §4.1 · pp. 6–7

4.2 Dual-Loop Co-Evolution: Core Experiment

• Across both model scales and all three benchmarks, mean accuracy rises from 58.5% at Baseline to 78.9% at Round 2 (+20.4 pp), which exceeds the Human-designed Harness by +24.7 pp.Round 1 mean accuracy is 73.5% (+15.0 pp over Baseline).Source: §4.1 · p. 7, Table 6 · p. 8, §4.2 · p. 8
• On HMMT25, the largest gains are +20.1 pp for Qwen3-8B and +27.2 pp for Qwen3-32B; AIME24 shows the smallest gains for both models.Source: §4.2 · p. 8
• For Qwen3-32B, HMMT25 improves by +27.2 pp versus +10.6 pp on AIME24, indicating larger gains on the harder task.Source: §4.2 · p. 8

4.3 Autonomous Harness Evolution: Case Study on AIME24

• In the autonomous AIME24 run, HarnessCritic repaired an initial vLLM KV-cache crash and a zombie-thread bug; the first complete evaluation reached 59.6% at 3.78 h.Source: §4.3 · p. 8
• Global-batched inference cut runtime by 8.7×, from 3.78 h to 1.11 h, while restoring chain-of-thought on follow-up turns maintained accuracy at 61.5%.Source: §4.3 · p. 8
• Six-trajectory majority voting over two diverse seeds reached 63.3% accuracy in 2.29 h, within the 3.33 h budget.Source: §4.3 · p. 8, Figure 7 · p. 9
• The run also detected and rolled back regressions from domain-specific prompts that conflicted with Qwen3’s internal reasoning, and discovered ensemble strategies.Source: §4.3 · p. 9

• For Qwen3-8B on AIME24, Round 2 accuracy is 84.7% versus 84.0% in Round 1; the paper attributes the negligible gain to reasoning-capacity saturation and fewer remaining harness-attributable failures.Source: §5 · p. 9
• By contrast, Qwen3-32B continues improving on AIME24 from Round 1 to Round 2 by +2.0 pp, which the paper attributes to its stronger reasoning exploiting the evolved harness.Source: §5 · p. 9
• Co-evolution may stop compounding through SFT plateaus, harness over-patching that overwhelms weaker models, or attribution cascades; a version-controlled registry permits rollback to the last validated harness.Source: §5 · p. 9
• Qwen3-8B AIME25 accuracy rises from 56.9% at Baseline to 78.3% at Round 2 (+21.4 pp), and the paper reports higher absolute test-time accuracy each round as evidence of transferable skill acquisition.Source: §5 · p. 9

6 Conclusion

• Co-Harness alternates failure-driven Harness evolution with fine-tuning on high-quality trajectories generated by the evolved Harness.HarnessCritic uses semantically grounded failure attribution to evolve the scaffolding.Source: §6 · p. 10
• +20.4 pp average accuracy improvement across Qwen3-8B and Qwen3-32B follows two rounds; Qwen3-32B gains +27.2 pp on HMMT25.The authors describe the paired loops as producing better trajectories for training, followed by further Harness improvements.Source: §6 · p. 10
• The authors frame model and scaffolding as interdependent optimization axes, contrasting co-evolution with paradigms that freeze either the Harness or the model.They describe the two loops as a positive feedback spiral: better Harnesses produce training trajectories for stronger models, which unlock further Harness improvements.Source: §6 · p. 10
• The approach requires a capable critic LLM, enough failure trajectories for cold-start, and substantial compute for multi-round SFT; structural Harness redesign remains human-led.Attribution accuracy degrades for failure modes requiring counterfactual control-flow reasoning.Source: §6 · p. 10
• Proposed extensions include online attribution during rollouts, reward-driven Harness search, and co-evolution for multi-agent systems.These are described as future directions toward continuous, online scaffolding monitoring.Source: §6 · p. 10

A TIR Agent Setup: Multi-Turn Code Interpreter Loop

• Episodes alternate model responses and Python calls, with Middleware returning interpreter output; the case study increased the turn limit from 15 to 30 after Phase 2.The model may issue further calls or produce a final answer after receiving each tool result.Source: A TIR Agent Setup: Multi-Turn Code… · p. 12
• TIR failures include malformed tool calls, missing or misconfigured retry hooks, context overflow, and prompts that discourage iterative code use.HarnessCritic attributes failed trajectories to categories and generates targeted patches.Source: A TIR Agent Setup: Multi-Turn Code… · p. 12

B Attribution Prompt Template

• The attribution prompt asks an expert scaffolding engineer to diagnose a failed trajectory against the current Harness and return a structured JSON diagnosis and diff suggestion.The output schema includes a failure category, Harness dimension, severity, evidence, and suggested field change.Source: B Attribution Prompt Template · p. 13

C Harness Configuration Details

• The experiments use an initial Full-Harness configuration for TIR, with Python as the primary tool and Middleware managing the multi-turn code-execution loop.Source: C Harness Configuration Details · p. 13

D Human Annotation Protocol

• Two expert annotators independently labeled 200 failure trajectories; a third resolved disagreements, yielding κ = 0.77.Annotators had access to the Harness configuration and trajectory logs, but not LLM attributions.Source: D Human Annotation Protocol · p. 13

E Patch Validation Protocol and Harness Registry

• A patch is accepted only when target-failure examples improve and held-out behaviors do not regress; rejected patches are rolled back.Validation compares the proposed patch against the current Harness on held-in target-failure examples and held-out previously successful or orthogonal behaviors.Source: E Patch Validation Protocol and… · pp. 13–14
• The versioned Harness registry stores configurations and self-contained diffs with metadata, supporting rollback, reproducibility, and an audit trail of changes and rationale.Source: E Patch Validation Protocol and… · p. 14

F Extended Analysis

• In TIR, invalid tool schemas or missing retry hooks can terminate an episode without a result for the model; HarnessCritic targets these interface components.The passage contrasts this with pure-text agents, which can sometimes fall back to verbal computation when prompts are suboptimal.Source: F Extended Analysis · p. 15
• HMMT25 gains are +20.1 pp for 8B and +27.2 pp for 32B; AIME24 gains are smaller but still meaningful.The authors associate the larger gains on harder problems with more Harness deficiencies and more code-interpreter turns per episode.Source: F Extended Analysis · p. 15
• AIME25 gains are +21.4 pp for both models, while the 32B model gains more than the 8B model on HMMT25.The authors suggest Harness–model interaction depends on the failure distribution for each benchmark–model combination.Source: F Extended Analysis · p. 15

G Co-Evolution Trajectory Analysis

• Figure 9 plots per-benchmark accuracy across co-evolution rounds for Qwen3-8B and Qwen3-32B, with all curves trending upward.The steepest gains typically occur between Baseline and Round 1.Source: Figure 9 · p. 15, G Co-Evolution Trajectory Analysis · p. 15
• +15.0 pp average accuracy gain from Baseline to Round 1 is followed by a further +5.4 pp from Round 1 to Round 2.The gains are diminishing but persistent.Source: G Co-Evolution Trajectory Analysis · p. 15
• On AIME24, Qwen3-8B improves only from 84.0% in Round 1 to 84.7% in Round 2, while Qwen3-32B gains +2.0 pp.The passage suggests the 8B model approaches its reasoning ceiling on this benchmark, whereas the 32B model continues to exploit the evolved harness.Source: G Co-Evolution Trajectory Analysis · p. 16

H Case Study: Autonomous 22-Version Evolution on AIME24

• After an initial GPU-utilization configuration crash and a later zombie-thread OOM, HarnessCritic fixes enabled a complete 90-question evaluation at 59.6% in 3.78 hours.Later tested variants reached 63.3% but exceeded the 3.33-hour time budget.Source: H Case Study: Autonomous 22-Version… · p. 16
• Global-batched inference reduced runtime from 3.78 hours to under 1.1 hours, an 8.7× speedup; restoring chain-of-thought and extending max turns yielded 61.5% in 1.11 hours.The initial batched variant had a transient accuracy drop after chain-of-thought was inadvertently disabled on follow-up turns.Source: H Case Study: Autonomous 22-Version… · p. 16
• Six-trajectory majority voting combined with v15 reached 63.3% accuracy in 2.29 hours, the highest score within budget; conflicting prompt instructions degraded later variants by ∼16.5 pp and were rolled back.The independent seed alone scored 50.1%.Source: H Case Study: Autonomous 22-Version… · p. 16
• The case study illustrates correctness-first repair, accuracy–efficiency trade-off management under an explicit time constraint, and validation-based rollback of regressions.The final configurations avoided degradation from the later over-patched variants.Source: H Case Study: Autonomous 22-Version… · p. 16