Source-linked AI summary
Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation
Nathaniel Pinckney, Christopher Batten, Mingjie Liu, Haoxing Ren, Brucek Khailany
TL;DR
Hardware-code generation has few benchmarks, and existing benchmarks lack in-context examples and detailed failure inspection. This work extends VerilogEval with those capabilities and specification-to-RTL tasks, then evaluates newer models. Results show substantial progress and performance variation across models, prompts, and task types.
Problem
Existing hardware-code benchmarks did not support in-context learning examples or detailed inspection of model failures, while VerilogEval v1 focused on code completion.
Source: §1 · p. 2Method
The authors extend VerilogEval with specification-to-RTL tasks, variable in-context examples, and failure classification, and evaluate recent language models.
Source: §1 · p. 2Results
Recent models push the state of the art, while performance varies with in-context examples and task type, including cases where code completion outperforms specification-to-RTL.
Source: §7 · p. 19Takeaways & Limitations
Failure classification provides more detailed insight into model errors, which the authors say can support targeted improvements in hardware-code generation.
Source: §7 · p. 19
Abstract
from arXiv · showhide
The application of large-language models (LLMs) to digital hardware code generation is an emerging field, with most LLMs primarily trained on natural language and software code. Hardware code like Verilog constitutes a small portion of training data, and few hardware benchmarks exist. The open-source VerilogEval benchmark, released in November 2023, provided a consistent evaluation framework for LLMs on code completion tasks. Since then, both commercial and open models have seen significant development. In this work, we evaluate new commercial and open models since VerilogEval's original release-including GPT-4o, GPT-4 Turbo, Llama3.1 (8B/70B/405B), Llama3 70B, Mistral Large, DeepSeek Coder (33B and 6.7B), CodeGemma 7B, and RTL-Coder-against an improved VerilogEval benchmark suite. We find measurable improvements in state-of-the-art models: GPT-4o achieves a 63% pass rate on specification-to-RTL tasks. The recently released and open Llama3.1 405B achieves a 58% pass rate, almost matching GPT-4o, while the smaller domain-specific RTL-Coder 6.7B models achieve an impressive 34% pass rate. Additionally, we enhance VerilogEval's infrastructure by automatically classifying failures, introducing in-context learning support, and extending the tasks to specification-to-RTL translation. We find that prompt engineering remains crucial for achieving good pass rates and varies widely with model and task. A benchmark infrastructure that allows for prompt engineering and failure analysis is essential for continued model development and deployment.
1 Introduction
Hardware-code generation benchmarks and evaluation capabilities lagged the rapid release of LLMs. VerilogEval v2 extends the benchmark with specification-to-RTL tasks, in-context examples, and failure classification, while the study finds that model responses vary with prompts and context.
1 Introduction
- The study evaluates newer commercial, open-source, and domain-specific models to assess the current frontier and prompt-tuning effects in Verilog generation.The authors report that recent open models are competitive with closed models and that prompt-tuning effects vary considerably across models.Source: §1 · p. 1
- VerilogEval v2 adds specification-to-RTL generation alongside the code-completion task supported by VerilogEval v1.The authors describe this extension as a way to evaluate models on instruction-style hardware generation.Source: §1 · p. 2
- The revised benchmark supports a variable number of in-context learning examples, which VerilogEval v1 did not support.The paper notes that prompt-tuning techniques such as in-context learning can improve LLM responses.Source: §1 · p. 2
- VerilogEval v2 adds failure classification and granular feedback where VerilogEval v1 reported only pass or fail.The authors present this as a means to inspect why a model fails a benchmark problem.Source: §1 · p. 2
- A Makefile-based evaluation approach replaces the original monolithic dataset, making evaluation easier to scale and inspect.The authors say this approach supports sweeping settings across more models and human inspection of the dataset.Source: §1 · p. 2
2 VerilogEval v1 Revisited
This section revisits VerilogEval v1 by evaluating recent models on its human-written code-completion prompts and explaining the benchmark’s measurement and dataset choices. Results show substantial progress across model releases, with Llama3.1 405B leading the reported pass rates.
2 VerilogEval v1 Revisited
- The study reports pass@1 to model a single-response deployment, using 20 samples at high temperature and one sample at low temperature.Pass@k means at least one sample passes among k samples; the reported settings are T=0.8 and top_p=0.95 for high temperature, and T=0.0 and top_p=0.01 for low temperature.Source: §2 · pp. 2–3
- The evaluation uses only VerilogEval-human, revising 14 problem descriptions or test benches for consistency or clarity while otherwise keeping the dataset and prompts similar to the earlier study.VerilogEval-human uses human-created problem descriptions from HDLBits and is described as more aligned with common code-generation deployments than VerilogEval-machine.Source: §2 · pp. 2–3
- 57.0% was Llama3.1 405B’s pass rate, ahead of GPT-4o (56.1%) and GPT-4 Turbo (49.8%) on VerilogEval-human.The figure compares pass@1 rates across recent models; the paper also reports RTL-Coder 6.7B at 31.5%.Source: Figure 1 · p. 4, §2 · p. 4
- Generated-code quality is not determined by the model alone: prompt tuning can have a large impact, and the benchmark evaluates in-context learning and changing task type from code completion to specification-to-RTL.Source: §2 · p. 4
3 VerilogEval v2 Improvements
VerilogEval v2 adds specification-to-RTL and in-context learning support alongside code completion, and expands evaluation with automated failure classification and a revised workflow. Its prompting, execution, and failure-analysis stages are configured to support varied evaluation runs and human inspection.
3 VerilogEval v2 Improvements
- VerilogEval v2 runs selected datasets through LLM prompting, Icarus Verilog evaluation, and automatic failure classification, saving failure summaries for human analysis.A Makefile selects the dataset; datasets provide prompts, reference solutions, and ICL examples, where applicable.Source: §3 · p. 5
3.1 Specification-to-RTL Task Support
- The benchmark converts all 156 VerilogEval problems to specification-to-RTL prompts while retaining the original code-completion task.Specification-to-RTL uses chatbot-style Question and Answer sections; code completion places the problem description in comments and appends the module interface.Source: §3.1 · p. 5
3.2 Support for In-Context Learning Examples
- ICL support parameterizes prompt examples from one to three shots, adding a sequential incrementer and then a finite-state machine to the initial combinational example.The shot count can be swept to test pass-rate sensitivity to added examples.Source: §3.2 · p. 6
3.3 Support for Failure Classification
- Automatic failure classification groups compile-time and simulation-time errors, including incorrect wire/register use, bit widths, and missing module interfaces.Its categories depend on Icarus Verilog or test-harness warnings and errors; human-inspected failure patterns can guide prompt tuning.Source: §3.3 · p. 9
3.4 Other Infrastructural Improvements
- The revised infrastructure splits problems into files and creates per-problem evaluation directories containing prompts, responses, generated Verilog, and Icarus logs.Make can run evaluations in parallel and resume interrupted runs; backwards-compatible JSONL support was planned.Source: §3.4 · pp. 9–10
4 VerilogEval v2 Evaluation
Across VerilogEval v2 tasks, larger models generally score higher, with GPT-4o and Llama3.1 405B among the leaders and open models reaching parity with closed models. Results also vary by task and model, including in response to in-context examples.
Code completion
- At 1-shot, GPT-4o reaches approximately 61% in code completion, up from 56% at 0-shot, establishing the new state-of-the-art frontier.GPT-4o also shows robust improvement over GPT-4 under both prompt configurations.Source: §4 · p. 10
In-context learning
- Llama3.1 405B reaches 57.0% at 0-shot and 57.9% at 1-shot, while the older Llama3 70B declines with 1-shot ICL.The paper describes Llama3.1 405B’s 0-shot result as a new Pareto frontier and notes little change with ICL.Source: §4 · p. 10
Specialized models
- RTL-Coder 6.7B achieves 32.6% in code completion with ICL and 33.5% in specification-to-RTL, leading DeepSeek Coder 6.7B in both reported comparisons.DeepSeek Coder 6.7B reaches 30.3% in code completion with ICL and 25.6% in specification-to-RTL.Source: §4 · p. 12
Specification-to-RTL
- Specification-to-RTL results are generally similar to code-completion results, but some models improve or degrade by task and prompt configuration.For example, Llama3.1 70B 1-shot scores 48.5% in code completion versus 39.0% in specification-to-RTL.Source: §4 · p. 12
Overall results
- GPT-4o and Llama3.1 405B lead the reported pass rates, open models reach parity with closed models, and smaller RTL-Coder models are competitive in some scenarios.The table reports results across models and sampling settings.Source: Table 2 · p. 12, §4 · p. 12
5 Impact of ICL on Pass Rates and Failures
The section shows that in-context examples can help or harm Verilog generation depending on the model, task, and examples chosen. Case studies and failure classification reveal how those changes arise, including recurring syntax and interface errors.
5.1 Increased In-Context Learning Examples
- With more ICL examples, Llama3 70B improves on specification-to-RTL but loses code-completion accuracy, while Llama3.1 70B and RTL-Coder are steadier across tasks.GPT-4o maintains a 55%–63% pass rate from 1-shot through 3-shot.Source: §5.1 · p. 13, Figure 4 · p. 13
5.2 Case Study: Problem 9 and Problem 34
- In Llama3 70B code completion, Problem 9 rises from 0% at 0-shot to 65% at 1-shot, while Problem 34 falls from 100% to 5%.Problem 9 reaches 100% at 2- and 3-shot, but Problem 34 falls to 50% at 2-shot and 0% at 3-shot; failures include missing endmodule and sequential/combinational confusion.Source: §5.2 · pp. 14–15
- For Llama3 70B, specification-to-RTL outperforms code completion on both case-study problems at 0-shot, recovers to 100% on both at 2-shot, and remains there at 3-shot.At 1-shot, Problem 9 drops to 0% after the model copies an input-port name from the example that is absent from the prompt.Source: §5.2 · p. 15
- Llama3.1 70B reaches 75% on Problem 9 and 100% on Problem 34 at 1-shot, but the added second example lowers Problem 9 to 60% and makes Problem 34 fail.At 3-shot, Problem 34 failures clear and Problem 9 returns to 75%; the 2-shot failures include natural-language explanations that cause syntax errors.Source: §5.2 · pp. 15–16
- Llama3.1 405B passes all 20 samples for both problems at every ICL shot count, while Llama2 70B shows no consistent benefit from added examples.Llama2 70B never passes Problem 9 at 2- or 3-shot, and Problem 34 regresses to 15% and 5%, respectively.Source: §5.2 · pp. 15–16
5.3 Aggregate Failure Analysis
- For Llama2 70B, specification-to-RTL reduces compile-time errors, including wire/register confusion, but runtime errors remain when generated logic is incorrect.In code completion, wire/register errors become especially pronounced at 3-shot.Source: §5.3 · p. 16
- Llama3 and Llama3.1 70B share some failure patterns, but only Llama3.1 improves substantially and consistently with added ICL examples; specification-to-RTL also reduces syntax errors.For Llama3 code completion, added examples tend to increase compile-time failures, unlike the relatively stable failure rate for Llama3.1.Source: §5.3 · p. 16
- Failure classification shows that ICL can fix some errors while introducing others, so inspecting failure types and individual problems can guide more targeted prompt tuning.Compiler errors mask runtime errors in Figure 5, so its stacked bars should be read from bottom to top.Source: §5.3 · pp. 16–17, Figure 5 · p. 17
6 Future Work
The paper argues that VerilogEval’s low-complexity problems and narrow RTL-generation scope are increasingly insufficient for evaluating hardware-design LLMs. Future benchmarks should address agent-based flows and a wider range of hardware tasks with realistic, flexible problems.
6 Future Work
- VerilogEval’s low-complexity problems limit its usefulness as LLM deployments move beyond single-turn exchanges, and hardware benchmarks need to cover tasks beyond specification-to-RTL.The paper notes that VerilogEval is already solved by VerilogCoder and MAGE.Source: §6 · p. 17, §6.1 · p. 18
6.1 Agent-based Code Generation
- Agent-based code generation divides work among planning and specialized agents, with some approaches iteratively checking or debugging generated RTL.VerilogCoder reports 94% pass@1 and MAGE 95% on VerilogEval, using testbenches, simulation, waveforms, and iterative candidate refinement.Source: §6.1 · pp. 17–18
- Future benchmarks for agentic flows should use realistic, complex specifications that allow design decisions within reasonable bounds rather than prescribe every wire or flip-flop.The paper calls for accurate behavioral specifications and room for microarchitectural decision-making.Source: §6.1 · p. 18
6.2 Related Hardware Design Tasks
- Hardware benchmarks should cover tasks beyond RTL generation, including verification, documentation, debugging, and specification analysis.The paper notes that frontend hardware benchmarks beyond code generation remain at an early stage.Source: §6.2 · pp. 18–19
- Evaluation must distinguish RTL from testbench generation: verification code calls for coverage metrics, while RTL quality centers on power, performance, and area.Testbench code is not constrained to synthesizable Verilog and may follow organizationally distinct conventions.Source: §6.2 · p. 19
- Complex hardware-design tasks require decomposition into manageable subtasks, with benchmarks tailored to identify model weaknesses in those subtasks.The paper identifies agent-based approaches with specialized LLMs as particularly suited to this decomposition.Source: §6.2 · p. 19
7 Conclusions
The paper concludes that hardware code-generation models have improved since VerilogEval v1 and presents VerilogEval v2 as a more robust evaluation framework. It highlights state-of-the-art model results, the value of failure analysis, and the need for benchmarks that expand beyond RTL generation as models advance.
7 Conclusions
- Llama3.1 405B and GPT-4o push the state of the art, while RTL-Coder 6.7B and DeepSeek Coder 6.7B achieve notable pass rates for their size.The paper reports that GPT-4o and Llama3.1 405B reach parity as ICL examples are added, although Llama3.1 405B leads in 0-shot code completion on VerilogEval-human.Source: §7 · p. 19
- VerilogEval v2’s failure classification provides deeper insight into model errors, including recurring missing-endmodule errors in Llama3 70B code completion.Problem-level inspection may help identify prompt-tuning or model-alignment changes to address such errors.Source: §7 · p. 19
- Future benchmarks should include tasks beyond RTL generation and increase in complexity as models and agents become more capable.VerilogEval v2 is presented as a step toward broader assessment on a common set of design problems.Source: §7 · pp. 19–20