Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification

1 Introduction

• Earlier Verilog benchmarks have narrow scope and high pass rates, leaving little headroom for measuring future improvements.RTL-Repo adds realistic repository context but focuses on code completion rather than specification-to-RTL generation, debugging, or verification.Source: §1 · p. 2
• CVDP includes 783 human-authored problems across 13 categories, covering hardware design, verification, debugging, and technical comprehension.It provides both single-turn Non-Agentic and multi-turn, tool-using Agentic formats.Source: §1 · p. 2
• CVDP is designed to evaluate Dockerized LLM-based agents on real-world Verilog problems with EDA tool use.The benchmark also supports a broader range of hardware design and verification tasks than prior work.Source: §1 · p. 2
• No more than 34% pass rate is achieved by state-of-the-art models on CVDP code-generation questions.The reported models include Claude 3.7 Sonnet, GPT-4.1, and LLaMA 3.1 405B.Source: §1 · p. 2
• CVDP analyzes model failures across categories to identify capabilities needed for reliable deployment in hardware design and verification.Source: §1 · p. 2
• The benchmark is motivated by limited public RTL code and design knowledge that remains proprietary within industry.The authors identify a need for a human-written, publicly available benchmark authored by design and verification experts.Source: §1 · p. 2

2 CVDP Dataset

• About 35 experienced hardware engineers authored problems, and subject-matter experts reviewed them for accuracy, task fit, and scope.Of 1,313 problems written, 783 were retained after quality filtering; sanity checks tested reference solutions and incomplete contexts.Source: §2 · p. 3
• Problems are multi-file repositories evaluated with task-specific harnesses; BLEU scores snippets expected verbatim, while technical answers receive LLM-based judging.The harness is typically a Cocotb simulation script.Source: §2 · p. 3

2.1 Task Categories

• Categories cover code generation and comprehension, including RTL tasks, verification, specification/code matching, and technical question answering.Non-Agentic tasks use fully provided single-turn context; Agentic tasks run in Docker containers where agents can inspect mini-repositories and invoke tools, using oracle contexts with minimal relevant information.Source: §2.1 · p. 3

2.2 Datapoint Author Guidelines

• Writers were instructed to assign human-tagged difficulty levels and prioritize clarity and coding practices over artificial complexity.Non-Agentic problems include easy and medium tasks; Agentic problems span easy, medium, and hard levels, across topics such as control logic, datapaths, interconnects, memory, and architecture.Source: §2.2 · pp. 3–4

3 Benchmark Infrastructure

• The Python infrastructure supports custom model and agent callbacks, self-checking harness validity, and Docker-isolated evaluation using open-source tools where possible.Tasks 12–14 currently require commercial Cadence Xcelium; the infrastructure also supports configurable timeouts and retry counts.Source: §3 · p. 4
• Figure 1 presents an overview of the benchmark evaluation flow.Source: Figure 1 · p. 4, §3 · p. 4
• The map feature supports custom-prompt querying and LLM-based quality filtering, and the infrastructure can convert between Agentic and Non-Agentic formats.Source: §3 · p. 4

4 LLM Benchmark Results

• 34% was the highest aggregate pass@1 rate on CVDP, achieved by Claude 3.7 Sonnet, followed by GPT-4.1 at 29% and LLaMA 3.1 405B at 23%.The results are reported after automated quality control; before filtering, average pass rates were approximately 3% lower for Non-Agentic and 1.5% lower for Agentic problems.Source: §4 · p. 5
• 21% was GPT-4.1’s Agentic single-turn pass@1, 8% below its Non-Agentic score; Claude 3.7 Sonnet and LLaMA 3.1 405B dropped 4% and 2%, respectively.The paper notes that LLaMA 3.1 405B’s small drop likely reflects difficulty with harder problems in both settings.Source: §4 · p. 5
• Testbench stimulus, checker, and assertion generation had substantially lower pass rates than other code-generation categories, despite using the same hardware description language as RTL generation.The authors suggest this gap may stem from testbench code’s procedural and imperative nature, compared with RTL’s typically declarative structure.Source: §4 · p. 5
• Claude 3.7 Sonnet performed well on most RTL generation and debugging categories but showed no significant advantage on Spec-to-RTL Component Reuse, which requires composing existing modules.Reuse tasks may include additional glue logic to meet specified behavior.Source: §4 · p. 6
• Question & Answer tasks had strong performance across LLMs, with minimal gains from newer models, but the technical reliability of the LLM-based scores needs further investigation.Correspondence tasks use BLEU scores, while Question & Answer responses are evaluated by an LLM judge.Source: §4 · p. 6

5 Failure Analysis and Insights

• The analysis is intended to identify critical shortcomings in LLMs across Verilog design and verification, with particular challenges in design verification.The authors frame these findings as insights for advancing hardware design and verification research.Source: §5 · pp. 6, 8
• The analysis embeds reflections on failed cases, clusters them with K-means using the maximum silhouette score, then uses a reasoning LLM to summarize category-level failures.The reflection vectors are produced using SentenceTransformer embeddings.Source: §5 · p. 6
• Models particularly struggled with testbench stimulus, checker, and assertion generation; verification and debugging had more average failure clusters than RTL coding across all three models.Failure entities included “Misplaced SVA” and “Insufficient Coverage,” in addition to syntax and functional errors.Source: §5 · p. 7
• Claude 3.7 Sonnet’s Testbench Checker Generation failures fell from 6 clusters to 2 after quality filtering, alongside reduced ambiguity and increased prompt consistency.The analysis covers checker-generation cases in cid13, a category with low pass rates and substantial filtering.Source: §5 · p. 7

6 Limitations

• CVDP is more challenging than prior benchmarks, particularly in design verification and module reuse.The benchmark remains limited to standard hardware design and verification tasks, not the full range of work from project inception through fabrication.Source: §6 · pp. 8–9
• Agentic datapoints use oracle contexts that omit files referencing additional units, while Code Comprehension Q&A datapoints do not sufficiently challenge LLMs.The authors suggest a separate task category creating specifications from RTL as a more informative and demanding way to address similar comprehension goals.Source: §6 · p. 9
• CVDP may not address requirements, tooling, or specialized needs of particular academic and industry organizations.Source: §6 · p. 9

7 Conclusions

• No more than 34% pass@1 is achieved by state-of-the-art LLMs on Code Generation, with notable gaps in SystemVerilog testbench generation.Source: §7 · p. 9
• CVDP comprises 783 human expert-authored problems spanning 13 hardware design and verification task categories and Non-Agentic Code Generation, Non-Agentic Code Comprehension, and Agentic Code Generation.Source: §7 · p. 9
• Dockerized agents and test harnesses support realistic tool interaction, and the benchmark is designed to accommodate future task categories and increased difficulty.Source: §7 · p. 9

A.1 Case Study 1:

• The brick-sort implementation mixes assignment semantics, omits bounds checks, and delays pass_cnt, adding N cycles and violating specified latency.The extra cycle occurs per pass, and brick sort performs N passes for an input of size N.Source: §A.1 · p. 12, Figure 4 · p. 12
• These errors illustrate that high-level algorithmic structure does not ensure cycle-accurate sequencing, boundary handling, or timing-contract fidelity in generated RTL.The code may appear syntactically correct while lacking the semantic fidelity expected in hardware design.Source: §A.1 · p. 12

A.2 Case Study 2:

• The ALU implementation assigns 4-bit operand expressions to an 8-bit output, implicitly zero-extending the upper bits instead of explicitly clearing them.Source: §A.2 · p. 12, Figure 5 · p. 13
• The golden solution explicitly drives the upper four bits to zero with concatenations such as {{4’b0}, ...}.Source: §A.2 · p. 13

• The initial sanity checks require each harness to pass with its reference solution and fail with the initial context; 78 of 1,313 datapoints failed these checks.Source: B Quality Filtering · p. 13
• The LLM judge scores ambiguity, consistency, category match, and behavioral match, assessing clarity, cross-field alignment, category fit, and prompt-to-implementation/test alignment.Source: B Quality Filtering · pp. 13, 15
• Post-processing combines the judge’s four metrics into an aggregate score and removes problems below the passing threshold of 8.0.Source: B Quality Filtering · p. 13
• Although the judge prompt supports prompt refinement, the authors report no results from that experiment because further vetting is needed to avoid overly descriptive or trivial prompts.Source: B Quality Filtering · p. 13
• Claude 3.7’s pass rate increased 10–14 percentage points for Code Improvement and Debugging; Spec-to-RTL and Code Modification also improved.Testbench and Assertion Generation showed little improvement despite aggressive filtering.Source: B Quality Filtering · p. 13

• Benchmark infrastructure and model evaluation used a virtual machine with 12 virtual CPUs and 24 GB of RAM running Rocky Linux 8.10.Source: C Compute Requirements · p. 15
• Model evaluation used 6.4–15 GB of disk, primarily from large simulation logs or Verilog Change Dumps generated by errant RTL or testbench outputs.Source: C Compute Requirements · p. 16
• A disk monitor checks each active datapoint run directory every second and aborts execution above 100 MB; API model endpoints are excluded from the compute figures.Source: C Compute Requirements · p. 16