Source-linked AI summary
ChipMEM: Verification-Grounded Memory for EDA Agents
Abdulrahman AlRabah, Joshua Mabry, Dilek Hakkani-Tür, Abdussalam Alawini, Hamid Shojaei, Kartik Hegde, Sandesh Adhikary
TL;DR
EDA agents can learn from tool feedback, but whether experience transfers beyond the tasks that produced it remains unresolved. ChipMEM combines verified procedural memory and Bayesian statistical guidance in a common adapter; on held-out CVDP tasks, its frozen procedural library achieves 20/20 accepted outcomes versus 18/20 without memory.
Problem
EDA memory methods are typically evaluated on the tasks that produced the experience, leaving transfer to unseen designs unresolved.
Source: §1 · p. 2Method
Results
20/20 accepted outcomes versus 18/20 without memory on held-out CVDP tasks using a frozen procedural library.
Source: §5 · p. 9Takeaways & Limitations
ChipMEM improves accepted outcomes without changing the underlying agent, and the held-out evaluation tests transfer without adding skills during evaluation.
Source: §1 · p. 3, §7 · p. 9The studies use a fixed top-two retrieval setting at threshold 0.6 and cover five RTL-OPT designs and ten unseen CVDP tasks per category.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Large language model (LLM)-based agents use Electronic Design Automation (EDA) tools to generate and revise register-transfer-level (RTL) designs under synthesis and verification feedback. Recent methods learn from this feedback by distilling reusable skills from execution traces or by training on rewards derived from EDA-tools. Both methods are typically evaluated on the tasks that produced the experience. Repeated access to benchmark feedback on the same task can reward task-specific revision rather than creating reusable knowledge that transfers. We introduce ChipMEM, a verification-grounded memory layer for EDA agents. It combines cross-task procedural memory with within-trajectory statistical guidance. Its procedural component distills and stores a skill only after it passes synthesis, simulation, or formal checks, rather than relying on model self-assessments. A Bayesian component maintains hierarchical Beta estimates over tool-call outcomes and ranks recovery strategies that succeeded under comparable errors. A common adapter applies the same memory interface to RTL optimization and testbench-generation agents while preserving each domain's tools and acceptance criteria. We measure performance on training tasks and evaluate whether learned skills transfer to unseen tasks. On RTLRewriter-Bench, under matched model and tool settings, ChipMEM produces equivalence-passing outputs on 39/54 scored designs versus 35/54 without memory; on the 49-design short suite, mean area improvement is 8.69% versus 5.66%. On held-out CVDP tasks, ChipMEM with a frozen procedural library achieves 20/20 accepted outcomes versus 18/20 without memory in a single evaluation per setting.
1 INTRODUCTION
The introduction argues that EDA agents need memory that transfers beyond the tasks that generated their experience. ChipMEM combines verified procedural skills with statistical tool-outcome guidance and evaluates memory across EDA tasks.
1 INTRODUCTION
- Automating RTL revision matters because early design choices affect power, performance, and area, while revised RTL must pass simulation and equivalence checks.Useful changes can require source restructuring rather than synthesis alone, and iterations are slow.Source: §1 · p. 2
- Prior EDA memory methods are often evaluated on the tasks that generated them, leaving transfer to unseen designs unresolved.Memories from dissimilar tasks may also be of limited use or counterproductive.Source: §1 · p. 2
- ChipMEM combines trajectory-derived procedural skills with Bayesian statistical memory that estimates tool-strategy success and advises when retries are unlikely to help.A successful strategy can create or refine a skill; a verified failure can be retained as guidance against repeating an ineffective action.Source: §1 · p. 2
- A common agent adapter records task context and execution artifacts while preserving each agent’s tools and domain-specific acceptance criteria.Procedural memory can evolve across tasks or remain frozen for evaluation on unseen designs.Source: §1 · pp. 2–3
- The evaluation spans RTL optimization benchmarks and CVDP testbench-generation tasks, including transfer to later tasks and unseen designs.The studies also examine whether accumulated tool outcomes improve decisions during execution.Source: §1 · p. 3
2 RELATED WORK
Related work explores reusable procedural memory, tool-guided RTL optimization, and benchmarks for RTL generation and verification. ChipMEM combines tool-grounded procedural and statistical memory and evaluates whether experience transfers across tasks and unseen designs.
Skill memory and design-aware retrieval
- Skill-memory approaches retain and retrieve prior experience, but RTL skill relevance depends on the target design, specification, tool context, and failure mode.ChipMEM integrates trajectory-distilled procedural memory with Bayesian statistical memory, grounding both in EDA-tool outcomes.Source: §2 · pp. 3–4
Tool-grounded RTL optimization
- RTL optimization methods use reward-driven weight updates, evolutionary or agentic search, reusable skills, or test-time adaptation, whereas ChipMEM stores verified outcomes for later tasks and agents.The passage contrasts these methods’ search or adaptation with ChipMEM’s procedural and statistical memory.Source: §2 · p. 4
LLMs for RTL design and evaluation
- VerilogEval, RTLLM, and CVDP benchmark RTL generation, completion, and verification; ChipMEM extends evaluation to cross-task memory transfer and tool-guided decisions.The passage identifies VerilogEval, RTLLM, and CVDP [Pinckney et al., 2025].Source: §2 · p. 4
3 ChipMEM
ChipMEM combines retrieved procedural skills with Bayesian tool-outcome guidance in an agent-agnostic memory layer. Procedural skills are grounded in verified task outcomes, while statistical memory supports retry and recovery decisions during execution.
Procedural memory
- Procedural entries pair a complete skill file with domain, task, provenance, and verifier metadata, indexed by an embedding of a predefined task artifact.The artifact is the top RTL file for RTL optimization or the task-definition PROMPT.md for CVDP.Source: §3 · p. 4
- Only verified PASS trajectories add skills; evolving mode carries additions forward, while frozen mode keeps its initial library read-only and skips storage.Skills are retrieved by task-embedding similarity, with only matches above τproc returned.Source: §3 · pp. 4, 6
Statistical memory
- Statistical memory tracks tool-call success and failure, using hierarchical Beta estimates to predict retries and provide recovery guidance at tool-step granularity.Retry context conditions progressively on command type, prior error, and capped retry count; recovery estimates inherit probabilities from broader levels.Source: §3 · p. 5
Algorithm
- The algorithm updates statistical memory after every tool call and distills a procedural skill from the trajectory only when the domain harness returns PASS.Frozen mode starts from a fixed library and skips the skill-storage step.Source: §3 · pp. 5–6
Agent adapter and verification
- The adapter records tool actions and results, returns artifacts to a deterministic domain harness, and applies distinct PPA and CVDP acceptance checks.PPA requires synthesis, functional equivalence, and positive audited improvement; CVDP requires hidden simulation, coverage, and mutation checks to pass.Source: §3 · p. 6
4 Experimental Setup
The evaluation spans RTL optimization, testbench generation, OpenTitan IP blocks, and custom designs, using domain-specific tools and acceptance criteria. Comparisons use matched memory-off baselines and report correctness, PPA, and interaction measures.
Task coverage
- The task suite includes 40 CVDP tasks split between training and held-out tasks, 38 RTL-OPT designs, 59 RTLRewriter-Bench cases, ten OpenTitan IP blocks, and 20 custom designs.CVDP divides its 20 held-out tasks evenly between CID012 stimulus generation and CID013 checker generation; OpenTitan uses five attempts per mode.Source: §4 · p. 6
Implementation
- GPT-5.5 serves the RTL-OPT, RTLRewriter-Bench, custom-design, and CVDP studies, while OpenTitan uses Qwen3.8-27B.Retrieval returns at most two skills at a similarity threshold of 0.6.Source: §4 · p. 6
Evaluation metrics
- PPA gains are measured after synthesis and equivalence; failed, invalid, unproven, unchanged, and nonimproving outputs receive zero gain.CVDP pass rate requires generated testbenches to pass every hidden Xcelium/IMC test.Source: §4 · pp. 6–7
Task coverage
- The CVDP evaluation keeps unseen tasks disjoint from training, with hidden tests and reference artifacts outside the agent workspace.CID012 and CID013 each contribute 10 training tasks and 10 unseen tasks.Source: §4 · p. 7
PPA scope
- RTL-OPT and RTLRewriter-Bench use first-pass mapped-area studies, whereas the Custom Design Set and OpenTitan report full PPA; Figure 3 presents both scopes.No matched characterized downstream power or timing flow was run for the two first-pass benchmarks.Source: §4 · p. 7, Figure 3 · p. 7
Baselines and ablations
- Memory OFF receives no retrieved or written skills and no Bayesian guidance; evolving ChipMEM updates skills after verified passes, while frozen mode reads a fixed library.The five-design RTL-OPT study compares Memory OFF, procedural-only, Bayesian-only, and combined memory configurations.Source: §4 · pp. 7–8, Table 1 · p. 8
5 Results and Discussion
ChipMEM improves mean area results across both public RTL optimization benchmarks and increases area and power improvement in the broader full-PPA studies. Results also cover interaction cost, memory ablations, held-out retrievals, and frozen CVDP transfer.
Cross-benchmark results
- Mean area improvement rises from 6.18% to 8.79% on RTL-OPT and from 5.95% to 8.88% on RTLRewriter-Bench with ChipMEM.RTLRewriter-Bench cell and wire reductions rise from 5.9%/6.0% to 8.3%/8.0%.Source: §5 · p. 8
- In full-PPA studies, area and power improvement rises from 0.14%/0.39% to 0.31%/0.52% on the Custom Design Set and from 0.05%/0.00% to 0.24%/2.54% on OpenTitan.OpenTitan timing remains stable; full-PPA gate passes increase from 1/10 to 4/10 while equivalence remains 6/10 in both modes.Source: §5 · p. 8
Interaction cost
- Mean LLM calls fall from 29.4 to 26.1, tool calls from 27.4 to 23.3, and tool time from 420.2 to 369.8 seconds with ChipMEM.Across 50 OpenTitan sessions, input tokens decrease by 6.8% and total tokens by 3.4%.Source: §5 · p. 8, Figure 4 · p. 8
Memory ablation
- On five matched RTL-OPT designs, procedural-only and Bayesian-only each produce four strict passes versus three for Memory OFF, while combined memory produces five.The combined state also has the highest mean audited gain and lowest tokens per accepted design.Source: §5 · p. 8
Held-out PPA skill retrieval
- Held-out PPA retrievals include 25 RTL Optimization, 21 Flow & Validation, and 4 Design Setup skill–design pairs across 36 designs.Cohort A has more retrievals and more FIFOs, arbiters, and peripheral controllers similar to designs represented in the skill library; cohort B is more heterogeneous.Source: §5 · pp. 8–9, Figure 5 · p. 9
Held-out CVDP transfer
- 20/20 held-out CVDP tasks are accepted with frozen memory versus 18/20 without memory, with CID013 rising from 8/10 to 10/10 and CID012 staying at 10/10.The library is read-only during this evaluation.Source: §5 · p. 9, Table 2 · p. 9
What the memory learns
- The held-out retrievals include cross-design RTL knowledge as well as EDA operating procedures, but individual skills have not been isolated through paired skill-level ablations.The paper identifies this isolation as future work for attributing flow completion and PPA improvement.Source: §5 · p. 9
6 Limitations
The reported evidence uses fixed retrieval settings and covers small component and transfer studies.
Scope
- ChipMEM uses a fixed top-two retrieval setting with a similarity threshold of 0.6.The paper calls for testing broader retrieval settings.Source: §6 · p. 9
- Component and frozen-transfer studies cover five RTL-OPT designs and ten unseen CVDP tasks per category.The paper identifies additional EDA tasks and controlled runtime environments as future evaluation targets.Source: §6 · p. 9
Future work
- Broader retrieval settings, additional EDA tasks, and controlled runtime environments remain future work.Source: §6 · p. 9
7 Conclusion
ChipMEM turns tool-verified EDA experience into procedural and statistical memory, improving accepted outcomes without changing the underlying agent. The conclusion reports lower average interaction costs in OpenTitan and calls for testing larger memories and broader cross-domain transfer.
7 Conclusion
- ChipMEM converts tool-verified EDA experience into reusable procedural and statistical memory.It improves accepted outcomes without changing the underlying agent.Source: §7 · p. 9
- In OpenTitan, ChipMEM used fewer LLM and tool calls, fewer input and total tokens, and less tool time on average.The conclusion also notes that output tokens and session wall time were not lower.Source: §7 · p. 9
- The SWE-bench Pro pilot extends the evaluation to software engineering and motivates testing ChipMEM across additional domains.Future work should test larger memories and broader cross-domain transfer.Source: §7 · p. 9
A Appendix
The appendix reports five-pass CVDP training aggregates.
A Appendix
- CVDP training aggregates cover 50 executions per setting.Source: Table 4 · p. 14
A.1 Per-Design PPA Results
The appendix documents held-out PPA cohorts, per-design outcomes, skill inventories, interaction costs, execution metrics, and a SWE-bench Pro pilot. It also includes prompt materials for the reported experiments.
A.2 Held-Out PPA Cohorts
- The held-out PPA evaluation covers two disjoint cohorts of previously unseen designs: cohort A has 20 designs and cohort B has 16.Each design is evaluated once in cohort order, with library updates enabled, so a reused skill may come from an earlier held-out run.Source: §A.2 · p. 15
A.2.1 Per-design outcomes
- Table 6 distinguishes successful synthesis with available PPA measurements from synthesis runs without sufficient metrics for PPA comparison.Its per-design held-out outcomes cover two cohorts and report full or partial synthesis completion.Source: §A.2.1 · p. 15, Table 6 · p. 15
A.2.2 Skill inventory
- The skill inventory groups guidance into Flow & Validation, Design Setup, and RTL Optimization.Retrieval frequency counts designs receiving a skill, not whether it was applied or caused the resulting PPA outcome.Source: §A.2.2 · p. 16
A.4 Memory ON vs OFF—Full Cost, Success, and Consistency Data
- The legacy memory-on run mined a skill after every session, including all 45 failed sessions, so its results are descriptive rather than a clean test of the current PASS-only learning rule.Source: §A.4 · p. 18
A.4.1—Per-design success
- 4/10 designs received passing, equivalence-verified edits with ChipMEM versus 1/10 with the EDA Agent.ChipMEM also passed on pattgen, spi host, and keymgr, where the EDA Agent failed every attempt.Source: Table 9 · p. 18
A.5 Execution and Resource Metrics
- Across 50 OpenTitan sessions per mode, ChipMEM used 14.834M versus 15.357M total tokens and 39.22 versus 34.14 cumulative session-hours.It also used 1303 versus 1468 LLM calls and 1166 versus 1372 tool calls; session-hours are cumulative duration, not parallel-run makespan.Source: Table 14 · p. 20
A.7 SWE-bench Pro Pilot
- 43/100 SWE-bench Pro tasks were resolved with SWE-agent + ChipMEM versus 32/100 with the SWE-agent baseline.The pilot summary lacks matched per-task outcomes, so no paired significance test is reported.Source: Figure 6 · p. 21, Table 15 · p. 21