Source-linked AI summary
Benchmarking Large Language Models for Automated Verilog RTL Code Generation
Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt, Siddharth Garg
TL;DR
The paper addresses whether LLMs can generate useful Verilog, a task requiring study despite success in other programming languages. It fine-tunes pre-trained models on GitHub and textbook corpora and evaluates generated code through compilation and functional tests. Fine-tuning improves both compilation and functional correctness, with CodeGen-16B the strongest evaluated model.
Problem
LLM-generated Verilog lacks comprehensive evidence on syntactic and functional correctness, while datasets and scalable evaluation methods are limited.
Method
The authors fine-tune five pre-trained LLMs on Verilog corpora from GitHub and textbooks, then evaluate completions across problems using compilation and functional tests.
Results
27.0% of completions were functionally correct after tuning versus 1.09% before tuning, and fine-tuned CodeGen-16B reached 41.9% versus 35.4% for non-fine-tuned code-davinci-002.
Takeaways & Limitations
Fine-tuned CodeGen-16B was the most successful evaluated model and could generate functioning code for challenging Verilog problems.
Takeaways & Limitations
The reported results use models fine-tuned on the GitHub corpus, while textbook data is assessed separately in an ablation study.
Abstract
from arXiv · showhide
Automating hardware design could obviate a significant amount of human error from the engineering process and lead to fewer errors. Verilog is a popular hardware description language to model and design digital systems, thus generating Verilog code is a critical first step. Emerging large language models (LLMs) are able to write high-quality code in other programming languages. In this paper, we characterize the ability of LLMs to generate useful Verilog. For this, we fine-tune pre-trained LLMs on Verilog datasets collected from GitHub and Verilog textbooks. We construct an evaluation framework comprising test-benches for functional analysis and a flow to test the syntax of Verilog code generated in response to problems of varying difficulty. Our findings show that across our problem scenarios, the fine-tuning results in LLMs more capable of producing syntactically correct code (25.9% overall). Further, when analyzing functional correctness, a fine-tuned open-source CodeGen LLM can outperform the state-of-the-art commercial Codex LLM (6.5% overall). Training/evaluation scripts and LLM checkpoints are available: https://github.com/shailja-thakur/VGen.
I. INTRODUCTION
The paper studies whether LLMs can generate syntactically and functionally correct synthesizable Verilog, addressing gaps in datasets and scalable evaluation. It fine-tunes multiple pre-trained models on Verilog data and evaluates completions across prompts and problem difficulties.
- LLMs can generate code from natural-language prompts, but their ability to generate HDL code requires dedicated study.
- The study evaluates syntactic and functional correctness of synthesizable Verilog generated by open-source and commercial LLMs.
- Baseline systems frequently fail syntax, synthesis, and functional checks, while Verilog fine-tuning is constrained by limited training data and evaluation methods.
- The authors consolidate GitHub code and Verilog textbooks into a large training corpus and fine-tune five pre-trained LLMs ranging from 345M to 16B parameters.
- The evaluation varies temperature, completion count, prompt detail, and model size, then compiles generated code and checks it against unit tests.
- Fine-tuned CodeGen-16B outperforms the other evaluated LLMs and generates functioning code for challenging problems.
A. Background
The background motivates adapting pre-trained LLMs to Verilog and situates the work among prior code-generation efforts. The paper uses curated GitHub and textbook corpora because open datasets for Verilog generation and evaluation were lacking.
- LLMs predict successive tokens and can complete prose or code from comments and initial code lines.
- Fine-tuning specializes pre-trained LLMs for a task with fewer epochs than training from scratch.
- Prior Verilog-generation work used template-based descriptions or small-scale assertion generation, limiting the range of explored functions and tasks.
- The study addresses the absence of an open dataset for training and evaluating LLMs on Verilog writing.
- The method curates Verilog data from public GitHub repositories and Verilog textbooks to support LLM fine-tuning.
- The GitHub corpus is deduplicated and filtered for Verilog module structure and file size, while textbook text is extracted, cleaned, and checked for high-level syntax.
B. Baseline LLM Architectures
The study compares five fine-tuned baseline LLM architectures with differing design parameters and data sources. Large models require substantial GPU memory and distributed training resources, while evaluation compiles and functionally tests their completions.
- Table I summarizes baseline architectures by layers, attention heads, embedding size, context length, and natural-language or code data source.
- CodeGen-16B parameters occupy 30 GB of GPU memory at 16-bit precision, while fine-tuning requires around 250 GB across multiple GPUs.
- CodeGen training uses model and data parallelism with optimizer-state sharding across GPUs.
- Training completes over two, four, or six days on specified RTX8000 and A100 configurations, while Megatron-LM uses one RTX8000 for 15 hours over nine epochs.
- Generated completions are truncated at end and endmodule, compiled with Icarus Verilog v11.0, and checked for functional correctness.
A. The Problem Set
The 17-problem Verilog set spans combinational and sequential designs across basic, intermediate, and advanced difficulty levels, with prompts varying in descriptive detail.
- Problem coverage: The problem set contains 17 Verilog problems spanning combinational and sequential logic.It includes finite-state machines, permutations, shifts, rotates, MUXes, RAM, LFSRs, adders, and counters.
- Input parameters: Each query combines a problem prompt with sampling temperature and a number of completions per prompt.The evaluation varies temperature and completion count alongside prompt detail.
- Prompt design: Prompts use low, medium, and high detail levels, progressing from module comments and headers to signal-specific comments and pseudocode-like specifications.The high-detail prompt adds or replaces comments with more detailed guidance.
- Test benches: Test benches check functional correctness, using exhaustive testing for basic and some intermediate problems and unit-test-like checks for remaining cases.Some specifications allow multiple correct responses because reset behavior, such as synchronous versus asynchronous reset, may be ambiguous.
V. LLM EVALUATION AND RESULTS
The study evaluates Verilog generation quality through four questions concerning base performance, fine-tuning, model size, and prompt variability.
- Research questions: The evaluation asks how well base LLMs perform on Verilog generation problems.
- Research questions: It examines whether fine-tuning improves Verilog-generation performance.
- Research questions: It tests whether larger LLMs with more parameters perform better.
- Research questions: It studies whether problem-description variability affects quality and the number of correct completions.
B. Results
The evaluation selects each model’s best temperature per scenario and measures compilation and functional success using Pass@k. Fine-tuned models outperform their pretrained counterparts, with CodeGen-16B leading the reported results.
- Evaluation metric: Pass@k measures the proportion of completions that compile or pass functional tests, with k equal to scenario problems multiplied by completions per problem.Best results use the temperature producing the most successful completions for each difficulty and description scenario, reported at n = 10.
- Generated examples: FSM examples illustrate generated Verilog containing state encoding, sequential state updates, next-state logic, and output logic.The example recognizes input sequences involving signals a and b and drives output z from the current state.
- Model comparison: Fine-tuned CodeGen-16B outperforms all evaluated LLMs, and every fine-tuned model outperforms its pretrained counterpart.These results address the study’s questions about baseline performance and fine-tuning.
1) Completions vs. Temperature (t):
Lower sampling temperatures produce higher Pass@(scenario*10), while increasing the number of completions improves the chance of finding test-bench-passing candidates; n = 10 is reported as broadly useful.
- Completions vs. Temperature (t): Pass@(scenario*10) is highest at t = 0.1 and degrades exponentially as temperature increases.The reported pattern associates low temperatures with accurate solutions and synthesizable code.
- Completions vs. Temperature (t): Pass@(scenario*1) is better than Pass@(scenario*10), and performance improves as the number of completions increases.The paper attributes this to more candidate solutions at low temperatures passing the test benches.
- Completions vs. Temperature (t): n = 10 is reported as suitable across all problem difficulty levels.
3) Completions vs. LLM Size:
LLMs with more parameters outperform smaller LLMs on generating Verilog completions that pass test benches.
- Larger LLMs, including CodeGen-16B and code-davinci-002, outperform smaller models such as Megatron-355M and CodeGen-2B.The larger models produce more completions that pass test benches and more correct completions.
4) Completions vs. Prompts:
Prompt difficulty and description level affect Verilog generation quality: harder problems and terser prompts yield fewer successful completions.
- Pass@(scenario*10) decreases as prompt difficulty increases, with simple AND problems easier than advanced LFSR problems.
- The number of correct solutions decreases when prompts become terse.
VI. DISCUSSION AND LIMITATIONS
Fine-tuning substantially improves compilation rates and supports syntactically correct design skeletons, but functional evaluation is constrained by test-bench coverage and prompt or corpus limitations.
- 64.6% of fine-tuned completions compiled, compared with 11.9% of pre-trained completions using the best Pass@(scenario*10) values.The authors suggest using fine-tuned models to generate syntactically correct design skeletons before functional refinement.
- As problem complexity increases, test benches cover only behaviors fully specified in the problem comments rather than all possible behaviors.
- Reset synchronization and other edge conditions are not tested when problem comments leave synchronous versus asynchronous reset unspecified.
- CodeGen-16B (FT) produced no passing completions for LFSR and Truth table problems and only one for Shift and Rotate among 540 completions per problem.The investigated failures involved bit concatenation, incomplete shift coverage or incorrect bit positions, and incorrect expressions between input bits.
- Fine-tuning CodeGen-16B with GitHub repositories and textbooks was marginally better by 1.4% than fine-tuning with GitHub repositories alone.The additional PDF corpus provides more examples, which helps the model generalize to Verilog.
VII. CONCLUSIONS
Fine-tuning improves functional correctness of Verilog generation, and fine-tuned CodeGen-16B is the strongest evaluated model, exceeding commercial code-davinci-002.
- 27.0% of completions were functionally correct after fine-tuning, compared with 1.09% for pretuned LLMs.
- Fine-tuned CodeGen-16B produced functionally correct code 41.9% of the time, versus 35.4% for non-fine-tuned code-davinci-002.