PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?

Kidus Seyoum, Ajay Mittur

arXiv:2609.28908v1cs.SEcs.LG

TL;DR

Design-verification debugging requires an agent to trace root causes across heterogeneous hardware artifacts, and it remains unclear whether harness evolution can reliably improve this work. The paper evaluates archive-aware evolution around fixed subject models and finds broader performance gains without consistent consolidation into one harness.

  • Problem

    Design-verification debugging requires tracing symptoms across heterogeneous artifacts to identify actionable root causes under bounded context and runtime.

    Source: §1 · p. 1

  • Method

    The study evaluates archive-aware harness evolution that proposes and tests candidates around a fixed subject model while retaining candidate records and lineage.

    Source: §1 · p. 2

  • Results

    Evolved harnesses increased completion and broader pass@5 coverage, but useful behaviors remained complementary and did not reliably consolidate into one harness.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The archive may be more valuable than the final incumbent, so systems should preserve candidates and task-level evidence and select or route among complementary harnesses.

    Source: §5.1 · p. 11

    The strongest repeated root-cause results cover only 12 proprietary optimization tasks, and the full-pool replay is not an independent test.

    Source: §5.2 · p. 11

Abstract

from arXiv · show

Agent behavior depends on the harness surrounding a language model, but it remains unclear whether language models can reliably improve such harnesses for hardware-design tasks. We study automatic harness evolution around a fixed subject model on 12 proprietary design-verification root-cause localization tasks. Across five trials per task, automatically evolved harnesses increased completed attempts by 71-76% and any-hit task coverage by 80-100%, while total correct attempts improved by only 18-24%. The strongest success reproducible at least twice result improved by one task, and later candidates exchanged gains across tasks rather than preserving them. An auxiliary candidate improved on a four-task validation set excluded from search but tied its baseline on a subsequent 12-task replay containing both search and validation tasks, so the selected gain did not persist across the full pool. Across the tested lineage, useful search, evidence, and finalization behaviors appeared in different candidates but did not consistently consolidate into a single harness that dominated across tasks and metrics. In a separate CVDP cross-benchmark case study, an automatically evolved defined-width repair harness produced 35.6% more functional passes than its 142-task reference baseline; the final functional verifier scored completed outputs but was not shown to the subject agent during repair. These results support archive-aware selection when evolution yields complementary specializations without consistent consolidation.

1 Introduction

The paper asks whether automatic harness evolution can improve a fixed model’s performance on hardware-design verification tasks. It presents an archive-aware framework and finds broader task coverage, but no reliable consolidation of specialized behaviors into one harness.

1 Introduction

  • Design-verification debugging requires tracing symptoms across logs, testbench code, RTL, configuration, and simulation infrastructure to identify an actionable root cause.Challenges include both generic agent failures and hardware-specific execution and state issues.Source: §1 · p. 1
  • The paper uses “lower-resource domain” for the relative scarcity of public hardware examples, debugging traces, tool-use demonstrations, and documentation.It does not refer to compute resources or model size; cited context includes ChipNeMo [1] and Revisiting VerilogEval [2].Source: §1 · p. 1
  • The study asks whether harness evolution can compensate for limited hardware-domain resources without changing model weights.The 12 proprietary tasks were used both during search and final repeated evaluation.Source: §1 · p. 2
  • Across the tested lineage, evolved harnesses completed more attempts and solved more tasks, but their gains were complementary rather than cumulative.No single harness preserved all observed benefits.Source: §1 · p. 2
  • The work contributes an archive-aware framework, a repeated optimization-set study, and an audit documenting incomplete consolidation of specialized behaviors.The audit reports complementary candidates, greater archive than incumbent coverage, and inherited mechanisms that do not reliably preserve predecessor behavior.Source: §1 · p. 2

2 Background and related work

The background distinguishes an agent’s external runtime harness from its model and scaffold, and situates harness evolution within work on program, agent-structure, and harness optimization. Prior evaluation findings motivate repeated, task-level assessment and activation audits.

2 Background and related work

  • An agent harness is the external runtime layer that runs the model–tool loop, manages context and state, mediates tools, enforces execution policies, and returns results.The paper distinguishes it from the model, environment, evaluator, and scaffold, though candidates package the harness with its scaffold configuration.Source: §2 · p. 2
  • Prompt and program optimizers such as DSPy [8] and GEPA [9], and agent-structure searches such as Automated Design of Agentic Systems [10] and AFlow [11], motivate optimizing beyond one instruction string.These approaches optimize declarative programs, prompts, agent programs, or code-represented workflows.Source: §2 · p. 2
  • Recent systems evolve harnesses directly, while this work evaluates mixed RTL, testbench, functional-model, and regression evidence for root-cause analysis.The cited systems include Meta-Harness [12], Self-Harness [13], Recursive Harness Self-Improvement [14], EvolveNet [15], and HarnessDev [17].Source: §2 · p. 2
  • Prior evaluations report robustness gains without consistently solving hard tasks, held-out regressions, run-to-run noise, and unreachable mechanisms, motivating repeated evaluation and activation audits.Harness materiality in agent comparisons is also argued in Stop Comparing LLM Agents Without Disclosing the Harness [18].Source: §2 · p. 2
  • Unlike CVDP generation and repair under deterministic checks, this study diagnoses failures across heterogeneous artifacts and matches a format-normalized root-cause function.CVDP is introduced in Comprehensive Verilog Design Problems [3]; the paper treats that regime as a separate cross-benchmark case study.Source: §2 · pp. 2–3

3 Harness-evolution framework

The framework separates a fixed subject model from an optimizer that proposes harness changes, evaluates isolated candidates, and retains an archive. Its search process checks activation, evaluates and confirms promising candidates, and explores bounded search and evidence behaviors.

3.1 System overview

  • A candidate harness and task form an isolated attempt around a fixed subject model, while an outer loop proposes branches, verifies activation, evaluates candidates, confirms promising results, and retains an incumbent and archive.The system records candidate manifests, lineage, outputs, traces, scores, and failure state.Source: Figure 1 · p. 3, §3.1 · p. 3

Algorithm 1: Harness evolution with archive-aware retention

  • The search procedure evaluates the baseline, summarizes evidence, generates candidates, checks activation, confirms promising candidates, promotes qualifying results, and retains candidate records in an archive.The output is a selected harness and candidate archive.Source: Algorithm 1: Harness evolution with… · p. 3
  • The three harness configurations had different editable boundaries and runtime conditions, so their results are interpreted within each experiment rather than as a matched framework comparison.Some runs mainly changed instructions and skills; the custom harness exposed additional runtime controls.Source: Algorithm 1: Harness evolution with… · p. 4

3.2 Search, activation, and retention

  • Search targets ungrounded queries, repetitive reads, weak evidence tracking, late commitment, and timeouts, while whole-harness changes can also alter controls and finalization.Domain documentation or retrieval is supplied to the proposer or candidate harness, not added to tasks or the evaluator.Source: §3.2 · p. 4
  • Branching evaluates sibling candidates independently and retains task-level outcomes, activation proofs, and lineage because candidates often solve different tasks.The final study separates inexpensive discovery from five-trial confirmation.Source: §3.2 · p. 4
  • Activation checks establish that a proposed rule reaches the rendered harness, but do not establish that the model follows it.Evaluation distinguishes activation failures from environment failures, timeouts, and completed incorrect or correct answers.Source: §3.2 · p. 4

4 Experimental evaluation

Across the repeated primary evaluation, evolved harnesses improved completion and task coverage, but later candidates traded gains rather than preserving every advantage. The experiments also show that archive coverage, domain-context treatments, and a separate CVDP repair case do not establish a single broadly dominant harness.

4.1 Benchmarks, scoring, and protocol

  • The primary benchmark covers 12 proprietary root-cause tasks across RTL, functional models, and testbench code, with exact-match scoring in the repeated Kimi K3 study.The auxiliary four-task validation result was a selection result, and its subsequent 12-task replay included both search and validation tasks.Source: §4.1 · pp. 4–5

4.2 Primary results

  • 10/12 pass@5 was reached by batched causal search, while first-trial success and Success≥2/5 were each 5/12, below the earlier candidate’s 6/12.Batched causal search completed 60/60 attempts; total correct attempts rose from 17/60 to 20/60 and 21/60 across the compared harnesses.Source: §4.2 · pp. 5–6, Table 2 · p. 6
  • The two evolved harnesses together solved 11/12 tasks, but each gained and lost task-level coverage, and T1 remained unsolved.Evidence-grounded search gained T3, T6, T7, T9, and T12 while losing T10; batched causal search added T4 but lost T3 and was less reliable on T9.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 What evolved, and what was executed

  • Audited traces showed a T9 localization succeeding despite omitted LITERALS formatting and a T5 attempt demonstrating causal-commit compliance, without isolating either behavior’s causal effect.T9 passed 3/5 versus 0/5 at baseline; T5 passed 4/5 versus 2/5.Source: §4.3 · p. 7

4.4 Archive coverage and incomplete consolidation

  • The large branching archive covered 10/12 tasks in union, although its promoted incumbent stayed at 3/12 and its strongest unpromoted individual reached 4/12.In a second run, best-observed mean rose from approximately 0.26 to 0.29 and then flattened; pass-level behavior remained distributed across branches.Source: §4.4 · p. 8, Figure 6 · p. 9
  • Later harnesses inherited earlier controls and added causal commitment, batching, and turn management, yet did not preserve all predecessor advantages.The artifacts do not identify whether this outcome-level failure to consolidate arose from optimizer choice, instruction interaction, model compliance, or sampling variation.Source: §4.3 · p. 7, Table 3 · p. 7, §4.4 · p. 8

4.5 Domain context and CVDP cross-benchmark case study

  • The tested domain-context treatments showed no demonstrated lift: treatment means were 10.3%–10.6% versus 14.6% for empty-policy baselines, with one trial per task.The authors limit the claim to these treatments; retrieval records were incomplete and the results do not establish that domain knowledge is ineffective.Source: §4.5 · p. 9, Table 4 · p. 9
  • On the 142-task CVDP follow-up, the evolved defined-width repair harness passed 61 tasks versus 45 for the reference baseline, a 35.6% increase in functional passes.The final functional verifier scored completed outputs and was not shown to the subject agent during repair; the case study is not a matched control for the proprietary study.Source: §4.5 · pp. 9–10, Table 5 · p. 10, Figure 7 · p. 10

5 Discussion and limitations

Across the repeated results, evolved harnesses improved operational robustness and task-specific outcomes, but did not reliably consolidate diagnostic behaviors. The authors recommend preserving and comparing complementary candidates while treating the primary benchmark’s size and evaluation conditions as limits on interpretation.

5.1 Interpretation and design implications

  • Evolved candidates completed more attempts and solved more tasks, but task-level regressions and coupled changes prevent a stronger causal interpretation.The authors distinguish operational robustness from a coherent expansion of diagnostic capability.Source: §5.1 · p. 10
  • The archive may be more valuable than its incumbent; preserve candidate evidence, route among complementary harnesses, and promote candidates through repeated clean trials and task-level comparisons.The authors also recommend executable checks for search evidence, read limits, activation, and finalization, with distinct failure categories.Source: §5.1 · p. 11

5.2 Limitations

  • The primary result covers only 12 proprietary tasks; the best candidate reached 10/12 pass@5, the best Success≥2/5 result was 6/12, and correct attempts peaked at 21/60.The validation-set gain did not persist in a replay containing search and validation tasks, which was not an independent test.Source: §5.2 · p. 11
  • The selected harnesses differ in multiple instructions and runtime controls, so score changes cannot be attributed to one edit.Completion and diagnostic behavior changed together; five trials reduce but do not eliminate sampling variation.Source: §5.2 · p. 11
  • The CVDP case study used a selected subset, task-specific routing at its 10/12 endpoint, and a direct-clean baseline with a different protocol.The same passage notes that earlier exploratory reruns were affected by provider and environment failures, while the primary Kimi K3 rows used a later, lower-concurrency protocol.Source: §5.2 · p. 11

6 Conclusion

Harness evolution improved how a fixed model used its capabilities on design-verification debugging, while specialized behaviors did not reliably consolidate into one dominant harness. The separate CVDP case study reported stronger functional outcomes, and the conclusion recommends treating the archive as an output and verifying execution and promotion carefully.

6 Conclusion

  • Evidence-grounded and batched causal harnesses increased completion, first-trial success, and pass@5 over baseline, and their union solved 11/12 tasks.The optimizer proposed search, evidence-tracking, causal-commitment, turn-management, and finalization controls that appeared in active harnesses and sometimes in successful traces.Source: §6 · p. 11
  • In the evaluated lineage, useful specialized behaviors did not reliably consolidate, and the archive covered more tasks than any incumbent.The separate GLM-5.2 CVDP case study reported a 35.6% increase in functional passes for the evolved defined-width repair harness over the 142-task reference verification-loop baseline.Source: §6 · p. 11
  • The authors describe automatic harness evolution as promising for discovering operational safeguards and specialized candidates, but not yet a substitute for domain competence or deliberate system design.They recommend treating the archive as an output, verifying activation separately from execution, confirming promotions with repeated task-level evaluation, and enforcing critical constraints in code.Source: §6 · pp. 11–12

A.1 Additional framework views

The appendix provides two additional framework views: one depicts stages within a subject attempt, and the other depicts candidate-branch processing and retention.

A.1 Additional framework views

  • Figure 8 depicts evidence acquisition, coverage tracking, causal analysis, and final commitment within one subject attempt.The caption identifies the stages represented, without specifying their implementation.Source: Figure 8 · p. 14
  • Figure 9 depicts candidate branches passing through materialization, static validation, activation checks, evaluation, and experiment-specific retention.The caption identifies the stages represented, without reporting their outcomes.Source: Figure 9 · p. 14

A.2 Benchmark detail and outcome labels

The appendix identifies provisional task subfamilies and clarifies that their grouping is retrospective, overlapping, and not shown to agents.

A.2 Benchmark detail and outcome labels

  • Table 6 presents provisional design-verification task subfamilies represented in the benchmark.The caption does not specify the subfamily names or their counts.Source: Table 6 · p. 14
  • The task grouping is retrospective, was not shown to agents, and is descriptive rather than a benchmark label; its subcategories overlap.The passage qualifies how the grouping should be interpreted.Source: §A.2 · p. 14

A.3 Domain-context and cross-model detail

Domain-context values are descriptive reconstructions, and the pooled mean for 58 evolved scaffolds could not be reported. Cross-model results used an older configuration and are not controlled comparisons with the final evolution results.

A.3 Domain-context and cross-model detail

  • The pooled mean for 58 evolved scaffolds is unreconstructed because candidate counts and unrounded scores were not retained; n.r. does not mean zero.The three treatment-level means are shown in the preceding table.Source: §A.3 · pp. 14–15
  • Domain-context values were reconstructed from experiment summaries rather than raw per-attempt artifacts and should be treated as descriptive.Source: §A.3 · p. 14
  • Tables 7–9 cover domain-context treatment scores, empty-policy baselines, and cross-model descriptive baseline results.Source: Table 7 · p. 15, Table 8 · p. 15, Table 9 · p. 15
  • Cross-model runs used an older configuration and are not controlled comparisons with the final Kimi K3 evolution results.Source: §A.3 · p. 15

A.4 CVDP inventory

The CVDP follow-up evaluated 142 selected tasks, not all directories in the preserved dataset. Figure 5 is descriptive because its raw numeric matrices were not retained.

A.4 CVDP inventory

  • The follow-up evaluated 142 selected CVDP v1.1.0 tasks: 49 cvdp_agentic and 93 cvdp_copilot, while the preserved directory contains 151 task directories.The 12-task subset corresponds to ordinals 1–12 of the selected tasks.Source: §A.4 · p. 15
  • Figure 5 is descriptive, generated from archived heatmap rasters because raw numeric matrices were not retained.Source: §A.4 · p. 15

Related papers

Loading 2609.28908v1…