VerilogEval: Evaluating Large Language Models for Verilog Code Generation

I. INTRODUCTION

• Because pretrained models may lack task alignment and labeled data is costly, the study builds an LLM-generated synthetic dataset of problem descriptions paired with Verilog code for supervised fine-tuning experiments.The introduction identifies a gap in exploring supervised fine-tuning to improve Verilog coding.Source: §I · pp. 1–2
• Existing Verilog benchmarks have limitations in comprehensiveness, quantity, and problem diversity, motivating broader coverage and unambiguous descriptions.The introduction also emphasizes reliable automated testing as a requirement for evaluating generated code.Source: §I · p. 1
• The benchmark contains 156 curated HDLBits problems selected for clarity and diversity.The paper presents the dataset as an open-source contribution to Verilog code-generation evaluation.Source: §I · p. 2
• Verilog completions are evaluated through automatic functional-correctness testing in an easily reproducible framework.The introduction identifies reliable, automated testing as a benchmark need.Source: §I · pp. 1–2

II. EVALUATION FRAMEWORK

• The framework follows HumanEval [21] best practices and uses a sandbox for simple, reproducible evaluation of LLM Verilog code generation.Source: §II · p. 2

A. VerilogEval Evaluation Set

• Functional correctness is evaluated on selected problems from HDLBits, a digital-circuit design exercise collection and online judge.Source: A. VerilogEval Evaluation Set · p. 2
• The evaluation set spans simple combinational circuits, complex finite state machines, code debugging, and testbench construction.Source: A. VerilogEval Evaluation Set · p. 2
• The benchmark focuses on self-contained Verilog modules generated from natural-language descriptions; it excludes module instantiation, which is important for system-level designs.A module is self-contained when its implementation does not require instantiating another module.Source: A. VerilogEval Evaluation Set · p. 2
• Though most problems are concise, they require hardware-design comprehension and problem-solving across circuits, Boolean logic, and state transitions.Source: A. VerilogEval Evaluation Set · p. 2
• Each problem description combines natural language with a module header and I/O definition, while the canonical solution serves as the testing reference.The header removes ambiguity such as signal bit width; the description is concatenated with the question prompt for inference. Fig. 2 illustrates an example.Source: A. VerilogEval Evaluation Set · p. 2, Figure 2 · p. 2

B. Problem Descriptions

• HDLBits descriptions often include schematics, state-transition diagrams, Boolean logic tables, and Karnaugh maps, which are not readily compatible with text-only models.The paper explores two methods for generating text-only descriptions.Source: B. Problem Descriptions · p. 2

1) VerilogEval-machine:

• Zero-shot sampling initially produced successful solutions for 108 of 156 candidate problems.Descriptions were tested by asking the LLM to generate code solutions.Source: 1) VerilogEval-machine · p. 2
• A generated description was discarded as invalid if none of 100 code completions succeeded.The descriptions were validated by using the LLM to generate solutions.Source: 1) VerilogEval-machine · p. 2
• For unsolved problems, valid descriptions became 4-shot examples for iterative sampling with eight code completions per description.A description was labeled valid as soon as any completion passed testing.Source: 1) VerilogEval-machine · p. 2
• VerilogEval-machine disregards the website descriptions and uses gpt-3.5-turbo to generate descriptions with a prompt template.The method first generates descriptions using zero-shot sampling.Source: 1) VerilogEval-machine · p. 2
• The process produced 143 valid problem descriptions, including 35 additional solutions after sampling unsolved problems.Sampling stopped when each problem reached its allocated budget.Source: 1) VerilogEval-machine · p. 3

2) VerilogEval-human:

• The conversion process addressed ambiguities in clock edges, reset and enable polarity, and synchronous versus asynchronous operation.Initial explorations found that ChatGPT could generate meaningful code for simple problems from converted descriptions.Source: 2) VerilogEval-human · p. 3
• 156 HDLBits problem descriptions were manually converted into text-only formats, including edge-list representations of state-transition graphs.Boolean tables and Karnaugh maps became text tables, schematics became natural-language connection explanations, and sequential waveforms were tabulated by clock edge and time step.Source: 2) VerilogEval-human · p. 3
• Machine descriptions are often more verbose and implementation-focused than human descriptions, and may still contain ambiguity or errors.They remain useful for testing whether models can follow low-level instructions and generate syntactically and functionally accurate Verilog.Source: 2) VerilogEval-human · p. 3

C. Automated Testing Environment

• Generated code is tested by comparing its simulation outputs with golden solutions, checking sequential outputs at clock edges and combinational outputs when inputs change.The testbench asserts output correctness at posedge and/or negedge transitions for sequential circuits.Source: C. Automated Testing Environment · p. 4
• Each problem’s testbench combines manually crafted significant input patterns with random patterns spanning hundreds to several thousand clock cycles.The random-test duration varies with problem complexity.Source: C. Automated Testing Environment · p. 4
• The sandbox runs untrusted programs in Docker with ICARUS Verilog, whose syntax support may not cover every feature of IEEE-1364.Simulation and testing are handled under the hood and can be run with a single command.Source: C. Automated Testing Environment · p. 4

D. Evaluation Metric

• Correct and wrong Verilog solutions are not clearly separated by BLEU score probability densities, indicating that BLEU does not track functional correctness well.Fig. 6 shows the distributions for two VerilogEval-human tasks using codegen-16B-verilog [12].Source: D. Evaluation Metric · p. 4, Figure 6 · p. 4
• A problem counts as solved under pass@k when any of its k samples passes the unit tests.The paper follows prior work [21], [22], in directly measuring functional correctness.Source: D. Evaluation Metric · p. 4
• For pass@k, the paper suggests [21]’s unbiased estimator and reports that n must be sufficiently large for low-variance estimates.The estimator uses n ≥ k samples per task, of which c ≤ n pass testing.Source: D. Evaluation Metric · p. 4

Section III-B

• The SFT experiments showcase the potential for improving model performance.Source: §III · p. 4

Section III-A

• The synthetic SFT data is generated by using LLMs to create problem descriptions.Source: §III · p. 4

• The section presents findings on supervised fine-tuning LLMs for Verilog coding.Source: §III · p. 4

A. Synthetic SFT Data Generation

• Pyverilog was used to extract syntax trees and filter open-source GitHub code for self-contained Verilog modules.Modules had to begin with module and end with endmodule; be no more than 200 lines or 1024 tokens; contain at least one of always, assign, always_ff, always_comb, or always_latch; and have no module instantiation.Source: A. Synthetic SFT Data Generation · p. 5
• The pipeline generated 8,502 problem-description and code pairs from GitHub Verilog modules, using gpt-3.5-turbo to produce module descriptions.The descriptions were generated with the prompt template in Fig. 3.Source: A. Synthetic SFT Data Generation · p. 5
• Approximate deduplication used MinHash [29] with a Jaccard similarity threshold of 0.8, as in [30].Source: A. Synthetic SFT Data Generation · p. 5
• The gpt-3.5-turbo descriptions used four VerilogEval-human few-shot examples selected to span varied designs and natural-language descriptions, including tabular formats.The examples were shift18, rule110, lemmings1, and fsm3onehot.Source: A. Synthetic SFT Data Generation · p. 5

B. Results on Supervised Fine-tuning

• The synthetic SFT dataset is 11MB, compared with ∼700MB of Github Verilog data used in.Training used Adam with β1 = 0.9, β2 = 0.999, ϵ = 10−8, learning rate 2e−5, effective batch size 1M tokens, and no weight decay; sampling used n = 20, top p = 0.95, temperature 0.8, and context length 2048.Source: B. Results on Supervised Fine-tuning · p. 5
• Experiments focus on CodeGen models and Verilog-trained counterparts at 350M, 2B, 6B, and 16B parameters, using -sft for synthetically fine-tuned models.Comparisons also include gpt-3.5-turbo and gpt-4 through OpenAI APIs, using default 4k-context 0613 checkpoints.Source: B. Results on Supervised Fine-tuning · p. 5
• The model variants progress from codegen-nl, trained on ThePile English text, to codegen-multi, further trained on multilingual code, and codegen-verilog, further trained on Verilog and textbook data.The codegen-verilog training data comprises ∼300MB of Github Verilog and 400MB of textbook data.Source: B. Results on Supervised Fine-tuning · p. 5
• Machine-description results correlate well with human-description results, supporting synthetic-generated benchmarks as indicators of downstream-task performance.Fig. 8 plots VerilogEval pass rates across SFT epochs, with dashed lines showing gpt-3.5 results.Source: 1) Training Epochs · p. 5, Figure 8 · p. 6

1) Training Epochs:

• In most cases, more SFT epochs improve pass@1 but lower pass@5 and pass@10, consistent with overfitting that reduces solution diversity.Overfitting also increases confidence and success on simpler problems; the authors encourage reporting both pass@1 and pass@10, particularly for post-alignment models.Source: 1) Training Epochs · p. 5, Figure 8 · p. 6
• The study uses 10 SFT epochs for multi models and 5 epochs for Verilog models in the remainder of its experiments.Source: 1) Training Epochs · pp. 5–6

2) Model Size and Base Model:

• Larger, more capable models generally achieve better Verilog coding performance, while 350M-model results are omitted.The omitted results were unavailable or had pass rates insufficient to demonstrate statistical significance.Source: 2) Model Size and Base Model · p. 6, Figure 9 · p. 6
• Synthetic-data SFT usually improves downstream performance, especially for multi models, but Verilog models gain on VerilogEval-machine and improve less or sometimes decline on VerilogEval-human.The synthetic data comes from GitHub Verilog code and supplies problem-code pairs that can improve alignment for VerilogEval-machine.Source: 2) Model Size and Base Model · p. 6
• Low-level synthetic descriptions lack the diversity of human examples, potentially explaining slight Verilog-SFT declines; greater SFT and pretraining data diversity and quality may help.Human examples include state-transition graphs, waveforms, and Karnaugh maps.Source: 2) Model Size and Base Model · p. 6
• Multi-model pretraining yields only marginal enhancements of approximately 3% for Verilog coding, suggesting limited positive transfer and highlighting the importance of substantial Verilog pretraining.The codegen-nl tokenizer prevents some VerilogEval-human problems from fitting within its 2048-token context window, so the comparison reports only VerilogEval-machine results.Source: 2) Model Size and Base Model · pp. 6–7, Table III · p. 6

3) SFT Data Quality:

• Incorrect problem-code pairs degrade codegen-2B-verilog performance on VerilogEval-machine, highlighting the importance of high-quality SFT data.The reported results compare fine-tuning with erroneous pairs, called sft-error, against the comparison condition; no numeric values are provided in the supplied passages.Source: 3) SFT Data Quality · p. 7, Table IV · p. 7
• The experiment created erroneous sft-error examples by shuffling problem descriptions with incongruous Verilog code solutions.This manipulation was used to assess the significance of data quality in SFT.Source: 3) SFT Data Quality · p. 7

IV. LIMITATIONS AND FUTURE DIRECTIONS

• VerilogEval evaluates boilerplate generation for relatively small-scale, self-contained Verilog designs, leaving more complex system-level design outside its scope.The benchmark uses human-generated descriptions across a wide array of hardware topics.Source: §IV · p. 7
• Module instantiation, which is important for constructing complex system-level designs, is absent from the benchmark.Recent LLM coding benchmarks are beginning to explore pragmatic code generation beyond standalone functions.Source: §IV · p. 7
• Testing assesses functional correctness but not synthesizable formatting or downstream circuit performance, which [13] addresses.Source: §IV · p. 7
• Recent LLM-based coding benchmarks are beginning to explore pragmatic code generation beyond standalone functions.Source: §IV · p. 7
• Boilerplate HDL generation covers only a limited part of hardware design, which spans transistor devices, circuit design, and hardware system architecture.The paper characterizes successful design as requiring a multidisciplinary approach.Source: §IV · p. 7
• Hardware design also demands balancing power, performance, and area while investing extensively in verification across conditions and corner cases.The paper identifies these practices as important for reliability and avoiding costly errors.Source: §IV · p. 7
• The authors identify future potential for LLMs to collaborate with domain experts on novel problems and solutions that may improve design efficiency, reliability, and agility.Source: §IV · p. 7

V. CONCLUSION

• The paper introduces a 156-problem HDLBits benchmark and shows that supervised fine-tuning with LLM-generated synthetic problem-code pairs can enhance pretrained models’ Verilog generation.The authors frame these findings as contributions to Verilog code generation and hardware design research.Source: §V · p. 7