Source-linked AI summary
VerilogEval: Evaluating Large Language Models for Verilog Code Generation
Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, Haoxing Ren
TL;DR
VerilogEval addresses limited task alignment and costly labeled data for Verilog code generation by introducing an automated benchmark and synthetic-data supervised fine-tuning. The paper reports improved pretrained-model capabilities, while noting that the benchmark currently covers only small-scale standalone module generation.
Problem
LLM pretraining objectives may not align with specific tasks, while acquiring labeled data for supervised alignment is costly.
Method
The paper introduces VerilogEval, an open-source benchmark of 156 curated HDLBits problems, and evaluates supervised fine-tuning with LLM-generated synthetic problem-code pairs.
Results
Supervised fine-tuning with synthetic problem-code pairs enhances pretrained language models’ Verilog code generation capabilities and downstream performance.
Takeaways & Limitations
VerilogEval provides automated evaluation for Verilog generation and supports studying synthetic-data alignment through functional code testing.
Takeaways & Limitations
Evaluations are confined to boilerplate generation for relatively small-scale designs, and module instantiation is absent from the benchmark.
Abstract
from arXiv · showhide
The increasing popularity of large language models (LLMs) has paved the way for their application in diverse domains. This paper proposes a benchmarking framework tailored specifically for evaluating LLM performance in the context of Verilog code generation for hardware design and verification. We present a comprehensive evaluation dataset consisting of 156 problems from the Verilog instructional website HDLBits. The evaluation set consists of a diverse set of Verilog code generation tasks, ranging from simple combinational circuits to complex finite state machines. The Verilog code completions can be automatically tested for functional correctness by comparing the transient simulation outputs of the generated design with a golden solution. We also demonstrate that the Verilog code generation capability of pretrained language models could be improved with supervised fine-tuning by bootstrapping with LLM generated synthetic problem-code pairs.
I. INTRODUCTION
VerilogEval addresses limitations in existing Verilog benchmarks by combining a diverse, clearly specified dataset with automated functional testing. It also explores synthetic problem-code pairs for supervised fine-tuning.
- Existing Verilog benchmarks have limited comprehensiveness, quantity, and problem diversity, while effective evaluation also requires unambiguous descriptions and reliable automation.
- VerilogEval introduces an open-source benchmark with diverse questions, clear problem descriptions, and automated, reproducible testing procedures.
- 156 curated problems sourced from HDLBits form the evaluation dataset, with attention to clarity and diversity.
- The framework automatically tests Verilog code completions for functional correctness.
- LLMs generate synthetic problem descriptions paired with Verilog code for supervised fine-tuning experiments.
II. EVALUATION FRAMEWORK
The evaluation framework uses a sandbox to assess Verilog generation on HDLBits tasks, emphasizing reproducibility, functional correctness, and self-contained modules. Problem prompts combine natural-language specifications with explicit module interfaces and golden solutions.
- VerilogEval uses a sandbox environment for simple and reproducible evaluation of LLM-generated Verilog code.
- The evaluation set is drawn from HDLBits, whose tasks span combinational circuits, finite state machines, debugging, and testbench construction.
- The benchmark focuses on self-contained Verilog modules that do not require instantiation of other modules.
- Problem descriptions include natural-language requirements, module headers, and input/output definitions, while canonical solutions provide golden references for testing.
B. Problem Descriptions
Because many HDLBits descriptions depend on diagrams, tables, and other modalities, VerilogEval creates text-only alternatives using machine-generated or manually converted descriptions. The machine pipeline validates generated descriptions by testing sampled code solutions.
- HDLBits descriptions often use circuit schematics, state-transition graphs, Boolean tables, and Karnaugh maps that text-only models cannot directly consume.
- VerilogEval-machine: VerilogEval-machine uses gpt-3.5-turbo to generate descriptions, validates them through generated code, and discards descriptions whose 100 sampled completions all fail.
- VerilogEval-machine: The machine-description pipeline instructs the model to provide concise natural-language specifications without code.
- VerilogEval-machine: 143 valid machine-generated problem descriptions were produced after iterative sampling.
2) VerilogEval-human:
VerilogEval-human manually converts HDLBits problems into text-only descriptions, resolving hardware-specific ambiguities and representing non-text modalities in textual form. Machine descriptions remain useful but may be verbose, implementation-focused, ambiguous, or erroneous.
- VerilogEval-human: The conversion resolves ambiguities involving clock edges, reset and enable polarity, and synchronous versus asynchronous operation.
- VerilogEval-human: Boolean tables, Karnaugh maps, circuit schematics, and sequential waveforms were translated into textual descriptions.
- VerilogEval-human: Machine descriptions are often more verbose and may mirror implementation details rather than explain circuit functionality at a high level.
- VerilogEval-human: VerilogEval-machine cannot guarantee that generated descriptions are free of ambiguity and errors, despite validation for passing solutions.
C. Automated Testing Environment
VerilogEval evaluates generated modules by comparing simulation outputs with golden solutions, using edge-triggered checks for sequential logic and input-change checks for combinational logic.
- Generated Verilog is evaluated through simulation against golden reference solutions.
- Sequential circuits are checked at relevant clock transition edges, including posedge and/or negedge events.
- Combinational circuits are validated whenever any input signal changes.
- Testbenches combine manually crafted significant patterns with randomly generated patterns.Random tests range from hundreds of clock cycles for simple problems to several thousand for more complex ones.
- The environment runs untrusted programs in Docker with the open-source ICARUS Verilog simulator.Syntax coverage is limited by the simulator’s supported Verilog features.
D. Evaluation Metric
The evaluation uses functional correctness rather than textual similarity: pass@k counts whether any sampled completion passes unit tests, while sufficiently large sample counts reduce estimator variance.
- BLEU-score distributions for correct and wrong Verilog solutions are not clearly separable.This limits BLEU’s usefulness as a proxy for functional correctness.
- pass@k considers a problem solved when any of k generated samples passes the unit tests.
- The unbiased pass@k estimator generates n ≥ k samples, of which c ≤ n pass testing.
- The sample count n must be sufficiently large to obtain low-variance pass@k estimates.
III. SUPERVISED FINE-TUNING
The paper bootstraps synthetic supervised fine-tuning data by pairing LLM-generated descriptions with filtered, self-contained Verilog modules from GitHub.
- The paper reports extensive SFT experiments using the generated synthetic data to study model-performance improvement.
- The SFT approach uses LLMs to generate problem descriptions for self-contained Verilog modules sourced from GitHub.
- Pyverilog extracts abstract syntax trees to support filtering of candidate modules.
- Modules are filtered by syntax boundaries, size limits, required Verilog constructs, and absence of module instantiations.The size thresholds are 200 lines of code or 1024 tokens.
- Approximate deduplication uses MinHash with a Jaccard similarity threshold of 0.8.GPT-3.5-turbo generates descriptions using selected VerilogEval-human examples as few-shot demonstrations.
B. Results on Supervised Fine-tuning
Experiments fine-tune CodeGen-family models across several sizes and compare base, Verilog-trained, and synthetic-SFT variants; machine descriptions correlate well with human descriptions and downstream performance.
- Experiments cover CodeGen and Verilog-trained models at 350M, 2B, 6B, and 16B parameters.
- The -sft notation identifies models fine-tuned with the paper’s synthetic supervised fine-tuning data.
- The model comparisons include natural-language, multilingual-code, and Verilog-code training variants.
- Machine descriptions correlate well with human descriptions across pass rates measured over different SFT training epochs.The figure also includes dashed lines representing gpt-3.5 results.
1) Training Epochs:
SFT performance depends on training duration, model scale, and pretraining alignment: longer training improves pass@1 but can reduce solution diversity, while larger models generally perform better.
- Training Epochs: Pass@1 improves with additional SFT epochs, while pass@5 and pass@10 deteriorate in most cases.The authors interpret this as overfitting that increases confidence on simpler problems but limits diverse solutions for harder ones.
- Training Epochs: The study uses 10 SFT epochs for multi models and 5 epochs for Verilog models.
- Model Size and Base Model: Larger and more capable models generally achieve better Verilog coding performance.Figure 9 compares SFT models, corresponding pre-SFT base models, and gpt-3.5 across model sizes.
- Model Size and Base Model: Synthetic SFT data usually improves downstream performance, especially for multi models lacking substantial Verilog pretraining.For Verilog models, gains are pronounced on VerilogEval-machine but smaller on VerilogEval-human, whose descriptions are more diverse.
- Model Size and Base Model: Multi models show only approximately 3% enhancement on Verilog coding tasks, suggesting limited transfer from software-language pretraining.The results emphasize the value of substantial Verilog pretraining for Verilog-related performance.
3) SFT Data Quality:
The study tests whether SFT data quality matters by deliberately mismatching problem descriptions and Verilog solutions, finding that incorrect pairs degrade performance.
- SFT Data Quality: The experiment creates sft-error by shuffling problem descriptions with incongruous Verilog code solutions.It compares codegen-2B-verilog models fine-tuned on these erroneous pairs for the VerilogEval-machine task.
- SFT Data Quality: The comparison evaluates the effect of data quality through fine-tuning results on VerilogEval-machine.
- SFT Data Quality: Incorrect low-quality SFT data degrades model performance.
IV. LIMITATIONS AND FUTURE DIRECTIONS
VerilogEval evaluates self-contained, small-scale Verilog generation through functional correctness, leaving system-level design, synthesizability, and PPA outside its scope while motivating broader future uses.
- Limitations: VerilogEval generates self-contained Verilog modules from natural-language descriptions and tests functional correctness only.It does not assess synthesizable formatting or downstream circuit implementation performance.
- Limitations: The benchmark covers relatively small-scale boilerplate designs and excludes module instantiation, a capability important for complex system-level designs.
- Limitations: Boilerplate HDL generation occupies a limited scope within hardware design, which also requires multidisciplinary expertise, PPA optimization, and robust verification.
- Future Directions: Future research could pair LLMs with domain experts to formulate novel hardware problems and devise innovative solutions.
- Conclusion: The paper presents a 156-problem benchmark and reports that synthetic problem-code pairs can enhance pretrained models through SFT.