CoEvo-RSI: When Harnesses Evolve, Which Models Should Follow?

1 Introduction

• CoEvo-RSI formalizes brain–body mismatch as divergence between harness-required behaviors and model capabilities, including harnesses that restrict capabilities the model already has.The problem is to maintain compatibility through successive rounds of harness–model co-adaptation.Source: §1 · p. 2
• A model may emit XML when its harness expects JSON, or poorly use a new context-management tool despite a historical checkpoint possessing relevant skills.Historical harness designs can also provide workflows suited to a model’s capabilities.Source: §1 · p. 2
• CoEvo-RSI uses an archive and execution feedback to guide harness editors’ historical design choices and model checkpoint selection or merging, followed by adaptation under the selected harness.The resulting designs, checkpoints, and outcomes are added to the archive.Source: §1 · pp. 2–3
• +14.16 agent accuracy on TraceElephant over DeepSeek-V4-Pro; co-evolved Qwen3.5-9B also exceeds Qwen3.8-Max on Who&When Pro by +4.40 agent accuracy.The evaluation spans multi-agent failure attribution, home care, and SQL tasks.Source: §1 · p. 3

2 Related Work

• Harness research progresses from hand-designed reasoning, feedback, and skill mechanisms to optimization of context, memory, planning, retrieval, and complete execution systems (Yao et al., 2023; Shinn et al., 2023; Wang et al., 2023).Some systems preserve earlier code, scores, and traces or archive alternative agent designs (Lou et al., 2026; Lee et al., 2026c; Zhang et al., 2025c).Source: §2 · p. 3
• Model-improvement work uses curricula, self-distillation, and experience reuse, while the harness determines the tools, observations, and constraints present in training trajectories (Zhao et al., 2025; Zhang et al., 2026b; Xia et al., 2026).SkillRL jointly evolves a skill library and policy (Xia et al., 2026).Source: §2 · pp. 3–4
• Co-evolution studies connect context with weights, executable harnesses with policies, or execution and editing in shared learning processes (Lu et al., 2026a; Chen et al., 2026b,c,d; Luo et al., 2026a).Examples include harness-conditioned alignment, fine-tuning on improved-scaffold trajectories, and rejection-sampling fine-tuning alternating with harness search (Chen et al., 2026b,d; Kim et al., 2026).Source: §2 · p. 4
• CoEvo-RSI makes bidirectional historical inheritance an explicit decision, selecting harness bases and independently selecting or merging archived checkpoints for their execution requirements.This can recover earlier capabilities without tying initialization to the latest generation or the harness’s parent branch.Source: §2 · p. 4

3 Method

• A meta-agent coordinates historical harness development and checkpoint selection or merging, evaluates candidate pairs, adapts the selected model under its harness, and promotes validated improvements into an archive.The archive records designs, checkpoints, and execution outcomes for later rounds.Source: §3 · p. 4, Figure 3 · p. 5

3.1 Model–Harness Co-Evolution

• Each co-evolution generation uses incumbent execution feedback and an archive of earlier harnesses, checkpoints, and histories to generate offspring harnesses and pair them with inherited models.TIMR measures observable interface violations per model turn; lower TIMR indicates fewer violations, but it does not fully characterize broader capability alignment.Source: §3.1 · pp. 5–6

3.2 Harness Inheritance

• The Curator retrieves relevant archive history and turns it into evidence and modification hypotheses; independent editors then choose historical harness bases and record adopted designs and ancestry.Linking designs to observed behavior gives editors evidence for deciding when a prior design is useful.Source: §3.2 · p. 6

3.3 Model Inheritance

• Model inheritance selects one archived checkpoint or merges several using meta-agent-chosen TIES configurations, guided by performance and TIMR evaluated under each offspring harness.The selected pair generates training trajectories; verifier-accepted executions support SFT with the harness fixed, and promotion retains the strongest eligible validated outcome or the incumbent if none qualifies.Source: §3.3 · pp. 6–7

4.1 Experimental Setup

• The evaluation targets affordable, reliable model–harness pairs for vertical applications, testing failure attribution, policy-constrained SQL, and structured device control against multiple baseline families.Baselines include direct LLMs, harness-enhanced systems, harness evolution, model training, and model–harness co-evolution.Source: §4.1 · pp. 7–8

4.2 Main Results

• 70.78% Who accuracy on TraceElephant is 14.16 points above DeepSeek-V4-Pro-0813 (56.62%), while CoEvo-RSI reaches 69.43% on Who&When Pro versus Qwen3.8-Max (65.03%).The failure-attribution benchmarks are held out from training and evolutionary selection.Source: Table 1 · p. 8, §4.2 · p. 9

4.3 Ablation Study

• On Who&When Algo., full CoEvo-RSI recovers from its round-4 dip by reusing round-4 weights with new harnesses, reaching 61.11% and 61.90% AA at rounds 6 and 8.Harness-only has lower SA in every round and TIMR above 1% in eight of ten rounds; full CoEvo-RSI keeps TIMR at or below 0.22%.Source: Figure 4 · p. 9, §4.3 · p. 11

4.2 Main Results

• On Spider, CoEvo-RSI achieves 80.85% Safe-EX, 96.64% Safe-Deny, and a 1.57% violation rate, surpassing Pi + MiniMax-M3; on BIRD its Safe-EX is 60.15% versus GLM-5.3 at 60.90%.On BIRD, it also has the highest Safe-Deny (91.26%) and lowest violation rate (5.65%).Source: §4.2 · p. 9, Table 2 · p. 10
• On HomeBench VM, CoEvo-RSI achieves 75.51% success and 89.23% F1, ahead of DeepSeek-V4-Pro-0813 (75.10% and 89.18%); on IM, it ties for best success at 84.54% and leads F1 at 91.80%.The table reports results for valid (VM) and invalid (IM) instructions.Source: §4.2 · pp. 9–10, Table 3 · p. 11

4.4 Framework Analysis

• Across three-round runs, API expenditure ranges from $2.06 to $3.69 and runs finish within 17.48–20.71 hours; default K=8 takes 17.48 hours at $2.72.Across ten rounds, CoEvo-RSI uses fewer rollout budgets than GRPO/GSPO and achieves higher accuracy on all four failure-attribution benchmarks.Source: Figure 6 · p. 11, §4.4 · p. 11
• On Who&When Hand, CoEvo-RSI reaches 68.39% Who accuracy at $0.16, versus DeepSeek-V4.1-Flash at 60.34% and $3.55; on RBAC Spider, it exceeds Pi + MiniMax-M3 by 0.59 Safe-EX points at 38× lower cost.Figure 7 compares task accuracy with the cost of a full evaluation pass.Source: Figure 7 · p. 12, §4.4 · pp. 12–13
• Across RBAC rounds 6–10, selected pairs have mean normalized TIMR of 0.13, and round 10 reuses the round-3 checkpoint to reach 80.85% Safe-EX under the final harness.The checkpoint has TIMR 0.54–1.00 under harnesses 4–9 but 0.02 under the final harness, illustrating pair-specific compatibility.Source: Figure 8 · p. 12, §4.4 · p. 13

5 Conclusion

• CoEvo-RSI uses a shared archive and execution feedback to guide historical harness development, checkpoint selection or merging, and adaptation under the selected harness.The framework is presented as a bi-level approach to model–harness co-evolution.Source: §5 · p. 13
• Using only SFT on its own rejection-sampled trajectories, CoEvo-RSI makes a compact task model competitive with larger models across three application areas.The applications are multi-agent failure attribution, role-aware text-to-SQL, and smart-home instruction following.Source: §5 · p. 13
• The findings highlight brain–body alignment as a central consideration for self-improving agents and bidirectional historical inheritance as a practical route to co-evolving execution scaffolds and intrinsic capabilities.Source: §5 · p. 13

A Experimental Details

• Training, validation, and test sets are disjoint at the level of sample IDs, tasks, and trajectories.Source: A Experimental Details · p. 21
• The same validation set guides harness and checkpoint selection and promotion during evolution, while test sets are reserved for performance reporting.Source: A Experimental Details · p. 21
• All 200 failure-attribution validation instances are held out from TracerTraj and removed from the training pool.Source: A Experimental Details · p. 21

A.1 Common Setup

• Each domain runs ten co-evolution rounds with K=8 harness candidates, except the candidate-sensitivity study, which uses three rounds.Source: §A.1 · p. 21
• DeepSeek-V4.1-Flash, running in Codex with high reasoning effort, receives validation rollouts; editors select archive bases and develop candidates evaluated on the full validation set.Candidates are compared using domain-specific objectives.Source: §A.1 · p. 21
• The task model uses thinking at high reasoning effort, with generation temperature 1.0, top-p 0.95, and top-k 20.Source: §A.1 · p. 21
• Qwen3.5-9B samples up to three trajectories per task under its selected paired harness, and verifier-accepted trajectories support one epoch of full-parameter SFT.Selected high-quality accepted trajectories from earlier rounds may also be replayed; training uses a global batch size of 8.Source: §A.1 · p. 21

A.2 Failure Attribution

• The task is to name the agent responsible for a failed multi-agent run and the decisive step where failure becomes unrecoverable.Source: §A.2 · p. 21
• The held-out evaluation includes Who&When Hand and Algo. trajectories, Who&When Pro multi-agent text-only trajectories, and 219 evaluable TraceElephant instances.Single-agent Who&When Pro trajectories are excluded because attribution is nearly saturated; records with missing or invalid decisive-step annotations are dropped from TraceElephant.Source: §A.2 · pp. 21–22, Table 5 · p. 22
• Training uses 3,400 filtered instances from TracerTraj, CORRECT-Error, AFTraj, and a self-constructed set, with 200 TracerTraj instances held out for validation.Training and evaluation were checked for separation at sample-ID and task levels, with no identical or highly similar trajectories found.Source: §A.2 · p. 22
• Agent accuracy measures exact responsible-agent matches, while step accuracy measures exact decisive-step matches; predictions are parsed from the structured final_answer tool without an LLM judge.Source: §A.2 · p. 22
• The initial harness shows per-step metadata first, requires inspect calls for full step content, and limits each episode to 20 turns.The model submits the agent and step through final_answer; ground-truth labels are not exposed.Source: §A.2 · p. 22

A.3 Role-Aware Text-to-SQL

• Role-aware SQL instances pair a database question with a user role, requiring executable SQL when authorized and refusal otherwise.A request is authorized only when its needed tables, columns, and operations are covered by the role policy.Source: §A.3 · p. 22
• SQL training and evolution use Spider, while full Spider and unseen BIRD test sets evaluate performance.The 8,000 training and 2,000 validation instances are disjoint samples from the official training split.Source: §A.3 · p. 22, Table 6 · p. 23
• SQL evaluation distinguishes correct and incorrect executions, over-refusals, policy violations, and proper refusals, and tracks LLM use and rule-based decision quality.SQL correctness is judged by execution equivalence on the sample database; the LLM rate is the fraction of instances reaching the LLM.Source: §A.3 · p. 23
• The initial SQL harness routes confidently handled requests with deterministic rules and sends uncertain cases to a multi-turn LLM agent with query-testing and answer-submission tools.The agent is limited to 10 turns.Source: §A.3 · p. 23

A.4 Smart-Home Instruction Following

• HomeBench queries combine 2 to 10 device sub-instructions, and the system must issue commands for feasible instructions and errors for infeasible ones.The study uses valid-only VM and invalid-only IM subsets; training and validation are held out from the official training split, and the official multi-device test set is reserved for final reporting.Source: §A.4 · p. 23, Table 7 · p. 24
• HomeBench success is exact match on command multisets, including errors, and the initial harness supplies a room/device index, four examples, read-only inspection tools, and a 20-turn limit.The official scorer also computes micro-averaged command-level F1.Source: §A.4 · pp. 23–24

• Meta-Harness and Self-Harness each evolve the same frozen Qwen3.5-9B model and initial harness for ten rounds, using the same meta-agent and 200 held-out validation instances.Each proposes eight candidates per round and evaluates each candidate twice; their proposer workflows differ.Source: §A.5 · p. 24
• The GRPO and GSPO baselines train Qwen3.5-9B under the initial harness on the same 3,400 instances, differing only in policy loss.Both use group-normalized advantages and no KL penalty.Source: §A.5 · p. 24
• CoEvo-RSI samples 8.5% fewer trajectories than one RL run and fine-tunes on 39,395 verifier-accepted trajectories, 48.3% of the RL training samples, across ten rounds.The CoEvo-RSI training count includes replayed trajectories, while the RL run updates on every rollout.Source: §A.5 · p. 24

• TIMR counts observable interface violations, but valid calls can still occur in unsuccessful trajectories when a harness underuses or constrains model capabilities.Assessing those cases requires task outcomes and examination of the information and actions the harness makes available.Source: B Scope and Interpretation of TIMR · pp. 24–25
• 0.22% is the maximum TIMR reported for full CoEvo-RSI, which also improves task performance; Model-only has near-zero TIMR but step accuracy at most 35%.Harness-only has TIMR above 1% in eight of ten rounds and lower step accuracy than full CoEvo-RSI in every round.Source: B Scope and Interpretation of TIMR · p. 25
• CoEvo-RSI uses TIMR alongside task scores for checkpoint proposals, while validation task scores determine pair selection and promotion.TIMR signals interface compatibility; task evaluation measures effectiveness on the intended problems.Source: B Scope and Interpretation of TIMR · p. 25

C Interpreting Comparisons with Frontier Models

• 95.51% Spider AC-F1 is close to Pi + MiniMax-M3’s 95.85%, while 75.51% HomeBench VM success exceeds DeepSeek-V4-Pro’s 75.10%.These comparisons contextualize how far co-evolution brings the compact model toward frontier-level performance on selected tasks.Source: C Interpreting Comparisons with… · p. 25
• The paper attributes the 9B model’s competitiveness to consolidating its own verifier-accepted execution experience through coordinated model–harness evolution.SFT supervision comes from Qwen3.5-9B rejection-sampled trajectories, including replay from earlier checkpoints.Source: C Interpreting Comparisons with… · p. 26
• CoEvo-RSI achieves 68.39% agent accuracy on Who&When Hand versus 60.34% for DeepSeek-V4.1-Flash; the reported evaluation passes cost $0.16 versus $3.55.The paper describes the latter cost as approximately 22× lower and frames specialization as useful when recurring cost and response-time budgets matter.Source: C Interpreting Comparisons with… · p. 26

D Comparison with Related Evolution Frameworks

• CoEvo-RSI independently chooses historical harness starting designs and selects or merges historical checkpoints, allowing components from different generations to form candidate pairs.Execution feedback guides their selection and subsequent adaptation.Source: D Comparison with Related Evolution… · p. 26

E Meta-Agent Prompts

• Meta-agent prompts combine a shared evidence-and-workflow skeleton with domain-specific metrics, interface contracts, and leakage constraints; the workflow requires selecting an archived harness base and validating its edited candidate.Run-specific placeholders are filled for each round.Source: E Meta-Agent Prompts · pp. 26–27
• Failure-attribution prompts rank candidates by weighted validation accuracy and direct diagnosis toward trajectory length, agent count, decisive evidence, context pressure, and attribution errors.The prompts also call for checking the same initialization before and after SFT and testing targeted changes with measurable validation checks.Source: E Meta-Agent Prompts · p. 28
• SQL prompts prioritize safe execution and access-control F1, prohibit regressions on those gates, and caution against broad rule-based filtering or validation memorization.They preserve a tool-based SQL agent loop and allow structural changes to routing, prompts, verification, and error recovery.Source: E Meta-Agent Prompts · pp. 28–29
• Smart-home prompts target exact-match success while retaining command F1, preserve the runtime interface, and restrict candidates to public home state and tool observations.Validation is frozen, held out from training, and disjoint from evaluation.Source: E Meta-Agent Prompts · p. 29