PaperlayerView exact revision on arXiv

Source-linked AI summary

VeriGen: A Large Language Model for Verilog Code Generation

Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, Siddharth Garg

arXiv:2308.00708v1cs.PLcs.LGcs.SE

TL;DR

The paper addresses failures in LLM-generated Verilog and investigates whether specialized fine-tuning can support hardware-design automation. It creates and validates Verilog-generation methods using specialized data and testing, reporting improved functional accuracy after tuning while noting that complex cases still require manual adjustment.

  • Problem

    Existing commercial LLMs may generate Verilog code that fails, while increasing design complexity creates a need to reduce hardware-design costs and developer effort.

  • Method

    The study consolidates open-source Verilog code, uses specialized fine-tuning, and introduces a framework for automatically creating and validating Verilog code with LLMs.

  • Results

    41.9% functional accuracy was achieved overall, while one scenario increased from 1.09% accurate completions before tuning to 27.0% after tuning.

  • Takeaways & Limitations

    The results indicate that fine-tuned LLMs can generate functionally correct Verilog, including in complex problems, although performance remains scenario-dependent.

  • Takeaways & Limitations

    Generated code still needs manual adjustments to handle edge cases and fully align with problem requirements, particularly as functional correctness remains challenging in complex problems.

Abstract

from arXiv · show

In this study, we explore the capability of Large Language Models (LLMs) to automate hardware design by generating high-quality Verilog code, a common language for designing and modeling digital systems. We fine-tune pre-existing LLMs on Verilog datasets compiled from GitHub and Verilog textbooks. We evaluate the functional correctness of the generated Verilog code using a specially designed test suite, featuring a custom problem set and testing benches. Here, our fine-tuned open-source CodeGen-16B model outperforms the commercial state-of-the-art GPT-3.5-turbo model with a 1.1% overall increase. Upon testing with a more diverse and complex problem set, we find that the fine-tuned model shows competitive performance against state-of-the-art gpt-3.5-turbo, excelling in certain scenarios. Notably, it demonstrates a 41% improvement in generating syntactically correct Verilog code across various problem categories compared to its pre-trained counterpart, highlighting the potential of smaller, in-house LLMs in hardware design automation.

1 INTRODUCTION

The study investigates LLM-generated Verilog by building an evaluation framework for syntactic and functional correctness, addressing limited datasets and testing methods. It fine-tunes open-source models on expanded Verilog data and compares them with commercial LLMs.

  • The paper evaluates whether LLMs can generate synthesizable Verilog from natural-language or technical specifications.The evaluation focuses on both syntactic and functional correctness.
  • Existing hardware-code LLMs can fail syntax, synthesis, and functional checks, while prior Verilog fine-tuning suffered from small datasets and poor generalization.The authors identify limited evaluation resources and scarce open Verilog datasets as central challenges.
  • The authors consolidate open-source Verilog code into a large training corpus and fine-tune five pre-trained LLMs ranging from 345M to 16B parameters.The resulting models are specialized for Verilog code generation.
  • The evaluation uses coding problems with varying difficulty and corresponding test benches to measure functional correctness.Generated code is compiled and checked against unit tests.
  • CodeGen-16B fine-tuned on the Verilog corpus demonstrates competitive performance over the evaluated LLMs, and the best fine-tuned models generate functioning code for complex problems.The study compares open-source fine-tuned models with GPT-3.5-turbo, GPT4, and PALM2.

2 BACKGROUND AND RELATED WORK

The background presents LLMs as autoregressive code generators and reviews their potential for hardware design. It motivates this study through reliability, cost, access, security, and dataset limitations in existing approaches.

  • LLMs generate completions autoregressively by predicting and appending successive tokens to a prompt.Code-trained models can infer a target language from prompts containing instructions, comments, or code.
  • Fine-tuning pre-trained LLMs on specialized datasets offers a more efficient alternative to training models from scratch.The paper notes that fine-tuning requires a limited number of training epochs.
  • Commercial LLMs introduce usage fees, availability risks, undisclosed parameters, security vulnerabilities, and possible latency through API access.The paper cites Code-Davinci-002 becoming inaccessible as an example of dependency risk.
  • The paper positions automated Verilog generation as a response to the difficulty of translating hardware specifications into programming structures.Prior studies explored NLP for hardware modeling, assertions, and limited Verilog snippets.
  • Hardware-oriented LLM research remains limited because no open dataset exists for training and evaluating Verilog generation.Earlier work used small or template-based settings, while this study expands evaluation across varied design tasks.

3 LLM TRAINING

The training pipeline combines GitHub Verilog repositories with textbook-derived material, preprocesses the data, and fine-tunes multiple pre-trained LLM architectures. The resulting corpus and model configurations support systematic Verilog generation experiments.

  • The primary training data combines open-source Verilog from GitHub with text and code extracted from 70 Verilog textbooks.The textbook corpus was created to test whether educational material further improves model performance.
  • Textbook extraction uses OCR and filtering to identify prose-associated Verilog blocks, which are segmented with overlapping sliding windows.The final combined textbook-extracted and GitHub corpus is 400 MB.
  • The study fine-tunes five LLMs whose architectures differ in layers, heads, embedding dimensions, context lengths, and data sources.The evaluated models include CodeGen, Megatron-LM, and J1-Large families.
  • Fine-tuning uses different hardware and schedules across models, including one epoch for CodeGen models and multi-GPU training for CodeGen-16B.CodeGen-16B requires approximately 250 GB for parameters, intermediate computations, and optimizer states at the described precision.

4 LLM EVALUATION SETUP

The evaluation uses two Verilog-generation harnesses: a transparent hand-designed set with custom test benches and a broader HDLBits-derived set. Prompts vary in detail, temperature, and completion count, and outputs are checked for compilation and functional correctness.

  • Evaluation harnesses: Two evaluation harnesses assess whether generated Verilog satisfies functional-correctness criteria.Set I uses hand-designed problems, test benches, and an end-to-end pipeline; Set II uses a broader HDLBits-derived problem set.
  • Problem sets: Problem Set I contains 17 classroom-rooted Verilog challenges spanning combinational and sequential logic, finite-state machines, shifts, RAM, LFSRs, adders, and counters.The set covers multiple design concepts and difficulty levels.
  • Problem sets: Problem Set II expands the dataset to 181 problems by integrating problems from HDLBits.The expanded set introduces broader problem coverage than the initial hand-designed set.
  • Evaluation procedure: Generated code is truncated at end or endmodule and passed to compilation and simulation-based evaluation harnesses.Set I uses Icarus Verilog v11.0; Set II uses Quartus synthesis and Modelsim simulations, returning Success or stage-specific failures.
  • Inference parameters: Sampling temperature ranges from 0.1 to 1, while completion counts per prompt are 1, 10, or 25, subject to model-specific limits.GPT4 is evaluated only with one completion per problem because of cost constraints.
  • Prompt construction: Problem Set I varies prompt detail across low, medium, and high versions, with high-detail prompts adding more explicit, pseudocode-like guidance.The low, medium, and high prompts progressively extend the supplied description.
  • Prompt construction: Prompts combine problem comments, a top_module header with typed inputs and outputs, and an insertion point for the generated code.This format is illustrated for the vibrate&ring problem.
  • Evaluation procedure: The HDLBits-based evaluation has limited visibility into its test benches, although returned system feedback indicates compilation and functional outcomes.The test benches are exhaustive for basic and some intermediate cases and analogous to unit tests for remaining cases.

5 LLM EVALUATION AND RESULTS

The study compares pretrained and fine-tuned models across Verilog problems, prompt settings, temperatures, and completion counts using Pass@k. Fine-tuning and larger models generally improve outcomes, but performance varies with temperature, prompt detail, difficulty, and model.

  • Research questions: The evaluation asks whether base models, fine-tuning, parameter scale, prompt variation, model comparisons, difficulty, and diverse training data affect Verilog generation.These questions span model capability, training, inputs, and problem coverage.
  • Metrics and protocol: The study evaluates generated-code quality with Pass@k, measuring compiling completions and completions that pass functional tests.Higher Pass@k indicates a relatively better result.
  • Metrics and protocol: The comparison uses best-performing temperature settings for each model and scenario, reporting compilation at n=10 alongside functional-test results.Table 4 covers compilation and Table 5 covers functional tests.
  • Overall results: Fine-tuned CodeGen-16B outperforms all evaluated LLMs, and every fine-tuned LLM outperforms its pretrained counterpart.These findings answer the first two research questions in the reported study.
  • Completions and temperature: Pass@(scenario*10) is highest at temperature 0.1 and degrades exponentially as temperature increases.The authors associate low temperatures with more accurate solutions and expect fewer candidates to be needed for synthesizable code.
  • Completions and temperature: Pass@(scenario*1) is better than Pass@(scenario*10), while performance improves with more completions and n=10 works well across difficulty levels.The reported explanation is that more low-temperature candidates increase the number passing test benches.
  • Model size: Larger models such as CodeGen-16B and code-davinci-002 outperform smaller models, while smaller models can produce more test-bench-passing completions in some settings.The findings indicate that parameter count is not uniformly aligned with every reported completion outcome.

Completions vs. LLM Size.

Model size is associated with higher overall performance, but smaller models can still yield more correct completions in the reported experiments.

  • Completions vs. LLM Size.: Smaller models including Megatron-355M and CodeGen-2B yield more test-bench-passing and correct completions in some settings.The result qualifies the broader association between parameter count and performance.

Completions vs. Prompts.

Prompt structure and model choice both affect Verilog-generation outcomes. Detailed system prompts improve GPT-3.5-turbo, while fine-tuned CodeGen-16B-FT remains competitive across difficulties and selected expanded-set categories.

  • Completions vs. Prompts.: Pass@(scenario*10) decreases as prompt difficulty increases, with simple AND problems easier to translate than advanced LFSR problems.The comparison uses prompt difficulty and problem complexity within Problem Set I.
  • Completions vs. Prompts.: Correct-solution counts decrease with terse prompts.This result complements the reported decline in Pass@(scenario*10) for more difficult problems.
  • System-prompt variation: A guided GPT-3.5-turbo system prompt directs the model to autocomplete a partially written Verilog module and format and complete it appropriately.The unguided alternative only instructs the model to behave as a programming assistant.
  • Comparison with emerging LLMs: CodeGen-16B-FT solves up to 74% of medium-complexity problems and remains reliable across problem difficulties.The authors contrast this with GPT4’s strength on advanced problems and suggest further fine-tuning with diverse Verilog problems.
  • Comparison with emerging LLMs: CodeGen-16B-FT achieves an average score of approximately 0.40, compared with approximately 0.53 for GPT4, 0.41 for GPT-3.5-turbo, and 0.30 for PALM2.GPT4 was evaluated with limited access and a single completion per problem.
  • Comparison with emerging LLMs: GPT4 scores approximately 0.6 across prompt-detail levels, while CodeGen-16B-FT reaches approximately 0.54 on intermediate-level problems.The comparison notes GPT4’s strength on advanced problem-solving and comparable performance with CodeGen-16B-FT in intermediate settings.

6 IMPACT OF TRAINING DATA ON VERILOG QUALITY

Combining GitHub Verilog code with textbook content improved CodeGen performance across problem difficulties and description levels. The mixed corpus also produced more functionally correct, structured counter implementations while highlighting a trade-off between inference speed and code quality.

  • Training corpus and overall performance: CodeGen-2B-FT++, fine-tuned on GitHub Verilog code and textbook content, consistently outperformed CodeGen-2B-FT and CodeGen-2B-FT* across problem difficulties and description levels.The evaluation used Pass@(scenario*10) with ten completions per problem on problem Set I.
  • Training corpus and overall performance: 10% improvement: CodeGen-2B-FT++ exceeded CodeGen-2B-FT on low-description problems, while CodeGen-2B-FT* scored 0.083 in that category.The trend continued for medium- and high-description problems.
  • Training corpus and overall performance: 0.548 Pass@(Scenario*10): CodeGen-2B-FT++ narrowly exceeded CodeGen-2B-FT at 0.547 on basic tasks, but substantially exceeded CodeGen-2B-FT* at 0.077.The same comparative trend continued across intermediate and advanced problems.
  • Training corpus and overall performance: Textbook content was associated with generation of Verilog that was not only correct but also idiomatic and well-structured.The paper attributes this to textbooks' examples, explanations, and broader context for Verilog use.
  • Counter implementation: In the 1-to-12 counter task, CodeGen-16B-FT++ generated functionally correct sequential Verilog with clocked logic and wrap-around from 12 to 1.CodeGen-16B-FT* used a non-synthesizable initial block and omitted reset behavior, while CodeGen-16B-FT omitted wrap-around.

7 DISCUSSION AND LIMITATIONS

The discussion finds that fine-tuning substantially improves compilability and functional correctness, but complex problems, underspecified behavior, and limited test coverage still require designer oversight. The authors conclude that smaller fine-tuned models can be useful, while acknowledging important performance and evaluation boundaries.

  • Performance gains: 64.6% of fine-tuned LLM completions compiled, compared with 11.9% of completions from pre-trained LLMs.The comparison uses Pass@(scenario*10) values.
  • Evaluation boundaries: Test benches are comprehensive for basic problems but cover only behaviors fully specified in comments as problem complexity increases.The authors note that similar completions and unspecified reset semantics can make test-bench implementation strongly affect passing counts.
  • Implications: CodeGen-16B-FT remained poor on some problem sets, and the authors identify prompt engineering, training-corpus diversity, and further fine-tuning as improvement directions.They also suggest hybrid models and domain-specific knowledge as possible future work.
  • Remaining challenges: Failures included incorrect bit handling, incomplete shift coverage, malformed expressions, race conditions, invalid signal declarations, and flawed Conway’s Game of Life logic.The examples indicate errors in both local Verilog details and problem-specific behavioral requirements.
  • Remaining challenges: Fine-tuned models generate syntactically correct design skeletons more reliably, but complex problems still frequently contain functional errors requiring manual adjustment.The authors emphasize that designers remain responsible for refining generated code to handle edge cases and satisfy functional requirements.
  • Implications: The framework combines LLM-based Verilog generation with automated validation, while model choice depends on complexity, computational resources, cost, and reliability.The paper presents the framework as a basis for using LLMs to generate syntactically correct Verilog that designers then adapt to functional requirements.
  • Performance gains: 27.0% of completions were functionally accurate after tuning, compared with 1.09% before tuning.These values are reported from Pass@(scenario*𝑛) results.
  • Performance gains: CodeGen-16B-FT produced functionally accurate code 41.9% of the time, versus 35.4% for code-davinci-002.The paper also reports that refined CodeGen-16B was the most proficient model for generating functionally correct completions.
Loading 2308.00708v1…