Source-linked AI summary
ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design
Zhongkai Yu, Chenyang Zhou, Yichen Lin, Hejia Zhang, Haotian Ye, Junxia Cui, Zaifeng Pan, Jishen Zhao, Yufei Ding
TL;DR
Existing chip-design benchmarks are saturated and omit debugging and reference-model generation needed to reflect industrial workflows. ChipBench evaluates these tasks on realistic modules and finds substantial performance gaps, while providing a toolbox for verified reference-model training data. Its results indicate that current LLMs remain far from reliable industrial chip-design integration.
Problem
Existing chip-design benchmarks are saturated and limited in task diversity, using simplified modules that do not reflect industrial workflows.
Method
ChipBench evaluates Verilog generation, debugging, and reference-model generation across realistic modules, and includes an automated toolbox for reference-model verification and training-data generation.
Results
Current models show substantial performance gaps: MAGE reaches 37% on ChipBench versus 95% on VerilogEval, while Claude-4.5-opus reaches 15.93% on Python reference-model generation.
Takeaways & Limitations
ChipBench shows that current LLMs remain limited for realistic AI-aided chip design, while debugging and reference-model generation warrant further research.
Takeaways & Limitations
Complete hierarchical Verilog generation has a 0% success rate, and Python reference-model generation remains extremely low.
Abstract
from arXiv · showhide
While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs' performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, and reference model generation. Our benchmark features 44 realistic modules with complex hierarchical structures, 89 systematic debugging cases, and 132 reference model samples across Python, SystemC, and CXXRTL. Evaluation results reveal substantial performance gaps, with state-of-the-art Claude-4.5-opus achieving only 30.74\% on Verilog generation and 13.33\% on Python reference model generation, demonstrating significant challenges compared to existing saturated benchmarks where SOTA models achieve over 95\% pass rates. Additionally, to help enhance LLM reference model generation, we provide an automated toolbox for high-quality training data generation, facilitating future research in this underexplored domain. Our code is available at https://github.com/zhongkaiyu/ChipBench.git.
1. Introduction
Existing chip-design benchmarks have saturated on simple Verilog tasks and poorly represent industrial hardware complexity. ChipBench is motivated by the need for broader, more realistic evaluation of LLM capabilities.
- LLMs have shown promise in chip design after demonstrating strong capabilities in conversational AI and high-level code generation.
- Over 95% accuracy on existing Verilog benchmarks indicates saturation and motivates more challenging evaluations.These benchmarks use simple, self-contained modules and therefore provide limited evidence about industrial deployment capability.
- Chip-design benchmarks such as VerilogEval and RTLLM were introduced to evaluate LLM capabilities in hardware engineering.
- Existing benchmark modules are typically 10–76 lines and self-contained, unlike industrial designs exceeding 10,000 lines with hierarchical submodules.The source of many benchmark modules—coding contests rather than industrial projects—also creates functional and coding-style discrepancies.
2. The exclusive focus on Verilog code generation overlooks LLMs’ debugging capabilities, which are even
ChipBench broadens chip-design evaluation beyond Verilog generation to debugging and heterogeneous reference-model generation. Its benchmark results show lower performance on realistic generation tasks and stronger debugging performance.
- Reference models support hardware-behavior prediction and cross-comparison, making their evaluation important for industrial chip-design verification.Reference-model generation is described as more resource-intensive than Verilog design in industrial workflows.
- MAGE achieves a 37% pass rate on ChipBench versus 95% on VerilogEval, demonstrating the greater difficulty of realistic Verilog generation.ChipBench uses 44 modules from CPU IPs and competitive platforms, with code 3.8× longer and 13.9× more cells than VerilogEval.
- Debugging pass rates are generally 5%–20% higher than code-generation pass rates across 89 cases covering timing, arithmetic, assignment, and state-machine bugs.The cases are created by manually injecting faults into golden Verilog modules.
- Claude-4.5-opus achieves only 15.93% accuracy on Python reference-model generation across a 132-sample Python, SystemC, and CXXRTL benchmark.The result indicates a substantial capability gap relative to other Python tasks.
- An automated toolbox verifies generated reference models and produces 2,206 high-quality Python models from 10,000 Verilog training samples.
2. ChipBench
ChipBench evaluates Verilog generation, debugging, and heterogeneous reference-model generation across structurally complex cases. Its workflows combine expert-authored tests, fault injection, functional verification, and automated reference-model data generation.
- 2.1. Overview: ChipBench contains 44 Verilog-generation cases, 89 debugging cases, and 132 reference-model cases across Python, SystemC, and CXXRTL.The benchmark spans more than 264 test cases and can be extended with the toolbox.
- 2.2. Verilog Generation Tasks: The Verilog suite covers self-contained modules, hierarchical non-self-contained modules, and CPU IP modules sourced from open-source projects.Prompts include submodule descriptions and source code for hierarchical cases, while CPU cases target professional engineering functionality.
- 2.2. Verilog Generation Tasks: Each Verilog case includes golden code, an expert-written prompt, and a manually developed test file using directed and over 1,000 constrained-random tests.
- 2.3. Verilog Debugging Tasks: Debugging cases inject arithmetic, assignment, timing, and state-machine faults into golden Verilog modules and evaluate zero-shot or waveform-assisted one-shot repair.The one-shot setting supplies simulation waveform data, while zero-shot provides no fault localization.
- 2.4. Reference Model Generation Tasks: The heterogeneous test engine syntax-checks and functionally verifies generated Python, SystemC, and CXXRTL reference models against golden Verilog implementations.It extracts I/O signals and generates cross-language verification stimuli.
- 2.5. Toolbox: The toolbox automates reference-model testbench creation and training-data generation, producing 2,206 verified Python models from 10,000 CodeV-R1 samples.It extracts golden-Verilog I/O signals and injects random stimuli for comparison.
3. Evaluation
ChipBench evaluates LLMs on complex Verilog generation, debugging, and reference-model tasks, revealing low generation and modeling performance, stronger debugging, and cost–iteration trade-offs.
- Verilog generation: 37.41% is MAGE’s pass rate on ChipBench, versus over 95% on VerilogEval, with CPU IP design below 22.22% for every model.These results indicate that ChipBench is substantially more difficult than saturated generation benchmarks.
- Verilog debugging: 47.45% versus 30.74% shows Claude-opus performs better on debugging than generation for the same Verilog module.The authors suggest multi-turn self-correction as a possible way to improve generation accuracy.
- Reference model generation: Python reference-model generation is 5%–20% more accurate than Verilog generation on simple modules but falls below 11.11% pass@10 on complex modules and CPU IP designs.The results indicate stronger Python syntax knowledge than hardware-behavior modeling capability.
- One-shot debugging analysis: One-shot waveform debugging outperforms zero-shot debugging on 8 models, but the overall results are mixed.The comparison tests whether providing waveform files improves debugging over access to only the buggy implementation and module description.
- Reference model training dataset generation: 50% is the reported pass-rate plateau reached as maximum debugging iterations increase, while higher iteration counts also raise total and average cost per successful sample.The authors recommend fewer iterations when Verilog data is abundant and more iterations when data is limited.
- Reference model training dataset generation: 2,206 verified Python reference models were generated from 10,000 Verilog samples with a maximum iteration count of 1.DeepSeek V3.2-coder is also identified as the most cost-efficient model for Verilog generation, while Claude’s Cost/Pass@1 is over 275× higher.
4. Take-Aways
The paper identifies hierarchical generation, waveform-aware debugging, and reference-model generation as major unresolved challenges for practical AI-aided chip design.
- Hierarchical Verilog Generation: Complete hierarchical-design generation falls to 0%, from below 22.22% for top-module-only generation.The paper highlights hierarchical capabilities as necessary for alignment with industrial design practices.
- Waveform-Aware Debugging: Poor waveform interpretation limits the practical value of LLM debugging despite stronger debugging than generation performance.The paper identifies waveform analysis and waveform-aware training as promising directions.
- Reference Model Generation: Extremely low Python reference-model pass rates indicate a gap between Python syntax proficiency and hardware-behavior modeling knowledge.The paper frames this gap as an unexplored opportunity for chip design and verification workflows.
5. Related Work
Existing chip-design benchmarks differ from ChipBench in task coverage and difficulty: some focus narrowly on simple generation, while others are too difficult or lack comprehensive debugging and reference-model evaluation.
- Verilog Generation Benchmark: VerilogEval, VerilogEval V2, RTLLM, and RTLLM V2 primarily evaluate specification-to-RTL generation using low-complexity, self-contained modules.VerilogEval V2 contains only self-contained modules under 76 lines, and these benchmarks omit debugging and reference-model generation.
- Verilog Generation Benchmark: RealBench offers highly complex open-source IPs but focuses only on generation and is described as too difficult for current LLMs.CVDP covers 13 task categories but provides limited debugging cases and lacks comprehensive reference-model evaluation.
- LLM for testbench generation: AutoBench, CorrectBench, and PRO-V include reference-model generation within testbench generation but evaluate correctness at the whole-testbench level.Their use of 10-sampling and majority voting does not replace direct evaluation of reference-model correctness.
6. Conclusion
ChipBench provides a broader and more realistic evaluation of LLMs for chip design by combining complex generation, debugging, and reference-model tasks.
- 6. Conclusion: ChipBench expands beyond simple Verilog generation to evaluate complex modules, debugging, and reference-model generation under industrially relevant requirements.The paper reports that evaluations reveal significant limitations in current models for AI-aided chip design.
A.1. Detailed Model Version Information
This section identifies the evaluated model versions and directs readers to their release dates, sizes, and API names.
- Table 6 lists the model versions used for evaluation.
- The evaluation-model overview includes verified release dates and parameter sizes.
- Parameter sizes marked with an asterisk are industry estimates based on performance benchmarks.
A.2. Detailed Information of Modules
This section provides detailed descriptions of the Verilog modules used in the benchmark through Table 7.
- Table 7 summarizes the descriptions of the Verilog modules.
- Readers are directed to Table 7 for detailed module information.
- The module descriptions are presented in a dedicated table.