PaperlayerView exact revision on arXiv

Source-linked AI summary

VHDLBench: Benchmarking and Fine-Tuning Large Language Models for VHDL-Based Chip Design

Patrik Drazic, Benaoumeur Senouci, Boualem Benatallah

arXiv:2610.28676v1cs.ARcs.AI

TL;DR

VHDL generation remains relatively underexplored, and the paper addresses limitations in available data and evaluation beyond syntax. It curates a VHDL dataset, fine-tunes open-source models, and evaluates generated designs through syntax, functional, and hardware checks. Fine-tuning improves generation quality, with DeepSeek-Coder achieving the highest performance gains, and generated circuits are deployed on FPGA hardware.

  • Problem

    VHDL dataset scarcity and limited evaluation beyond syntax, alongside challenges with complex designs, leave evidence on fine-tuned LLM generation comparatively limited.

    Source: 1) Data Sources · p. 3, §V · p. 11

  • Method

    The paper curates instruction-response VHDL data, fine-tunes five open-source 7B-parameter LLMs using QLoRA, and evaluates outputs through syntax, functional simulation, and FPGA prototyping.

    Source: §VI · pp. 11–12

  • Results

    DeepSeek-Coder improved from 57% to 92% syntax correctness, and the paper reports successful FPGA deployment of LLM-generated circuits.

    Source: B. Perfomance Analyisis and Results · p. 9, §VI · p. 12

  • Takeaways & Limitations

    The results support the practicality of adapted LLMs for VHDL generation when evaluation includes functional behavior and hardware deployment, not syntax alone.

    Source: §VI · p. 12

    The work focuses on functional correctness rather than optimization or advanced verification, and its dataset structure leaves hierarchical design support as a future direction.

    Source: §V · p. 11

Abstract

from arXiv · show

The rapid evolution of digital circuit complexity demands innovative approaches to hardware design automation. While Large Language Models (LLMs) have influenced software development, their application to Hardware Description Language (HDL) generation remains largely unexplored, particularly for VHDL. This paper presents a comprehensive study for AI-assisted digital circuit design, addressing gaps in current Electronic Design Automation workflows. The work introduces a VHDL dataset of 18,500+ syntax-verified examples from open-source repositories and converted Verilog designs. Five 7B-parameter LLMs were fine-tuned using parameter-efficient QLoRA technique, enabling deployment in resource-constrained environments. To evaluate model efficacy, a rigorous three-level pipeline was established: (1) syntax and synthesis validation, (2) functional simulation with testbenches, and (3) FPGA prototyping on Zynq SoCs. Results show fine-tuned models achieve up to 92% syntax correctness and 73% functional accuracy on common design tasks, with DeepSeek-Coder demonstrating the highest performance gains. Generated designs were successfully deployed on real hardware, confirming not only syntactic but operational viability. This work demonstrates that LLMs, when tailored with domain-specific data and lightweight fine-tuning can contribute meaningfully to the future of generative EDA workflows.

I. INTRODUCTION

Digital design has moved toward higher abstractions, from RTL to HLS and AI-based HDL generation. The introduction frames LLM assistance as a response to continued manual RTL work and gaps in VHDL generation and evaluation.

A. Evolution of Digital Design Paradigms

B. RTL and EDA in Practice

C. AI oriented HDL Generation

  • Existing HDL-generation work has focused largely on Verilog, while generating complex modules that meet functional, synthesis, and deployment requirements remains challenging.The introduction identifies VHDL as relatively underexplored despite its use in aerospace, defense, and European industry.Source: C. AI oriented HDL Generation · p. 2
  • Prior evaluation gaps include limited large VHDL datasets and pipelines that extend beyond syntax to simulation and hardware validation.The introduction also notes variable model performance, particularly on complex design tasks.Source: C. AI oriented HDL Generation · p. 2

D. Contribution of This Work

The paper focuses on direct VHDL generation and addresses a literature gap in full fine-tuning of open-source models for this task. Its contributions combine a dataset, prompt formatting, multi-level evaluation, and comparative analysis.

D. Contribution of This Work

  • The study investigates direct VHDL generation and reports full fine-tuning of open-source LLMs specifically for VHDL, an area largely overlooked relative to Verilog.The introduction notes VHDL-Eval and Chain-of-Descriptions as recent work in this area.Source: D. Contribution of This Work · p. 2
  • The work contributes a diverse, verified dataset of over 18,500 VHDL examples and a prompt format tailored to instruction-tuned models.Source: D. Contribution of This Work · p. 2
  • Its evaluation pipeline covers syntax and synthesizability, functional verification with GHDL simulations and testbenches, and Zynq-based FPGA prototyping.Source: D. Contribution of This Work · p. 2
  • The comparative analysis spans basic, intermediate, and advanced VHDL tasks and examines model differences, strengths, limitations, and improvement opportunities.Source: D. Contribution of This Work · p. 2

A. LLM for software development

The section contrasts software-oriented code generation with hardware-specific requirements, then reviews HDL-generation work and the paper's VHDL contribution. It presents QLoRA fine-tuning and a public task-code dataset as extensions of prior efforts.

A. LLM for software development

B. LLMs for Electronics Design Automation

  • Prior HDL work includes fine-tuning CodeGen for Verilog, VerilogEval, and VHDL-Eval; the cited VerilogEval study reports overfitting as epochs increase.It observed improved pass@1 alongside declining pass@5 and pass@10.Source: B. LLMs for Electronics Design… · p. 3
  • This work contributes a public SFT task-code dataset of approximately 18,500 samples and demonstrates QLoRA for VHDL generation.Source: B. LLMs for Electronics Design… · p. 3
  • Five approximately 7-billion-parameter LLMs with varying architectures were selected to compare architecture-related performance differences.Source: B. LLMs for Electronics Design… · p. 3

III. METHODOLOGY

The methodology combines VHDL dataset curation, parameter-efficient LLM adaptation, and a three-level evaluation pipeline. Dataset construction draws on public code and converted Verilog, then filters examples for validity and diversity.

III. METHODOLOGY

  • The methodology addresses scarce fine-tuning data, limited evaluation beyond syntax, and the challenge of generating functional, deployable designs across complexity levels.It curates a dataset, fine-tunes models with parameter-efficient methods, and evaluates outputs through three checking levels.Source: §III · p. 3

A. Dataset

  • The instruction-based dataset contains over 18,500 VHDL instruction-response pairs, combining repository and FPGA-tool examples with translated Verilog designs and generated natural-language prompts.Source: A. Dataset · p. 3

1) Data Sources

  • The source corpus combines public GitHub repositories, installed FPGA development tools, a Hugging Face dataset, and RTL-Coder Verilog examples translated into VHDL.The RTL-Coder examples were translated using Icarus Verilog and validated for syntax with GHDL.Source: 1) Data Sources · p. 4

2) Dataset pre-processing

  • Because bulk-collected files could be incomplete or unsynthesizable, preprocessing used GHDL syntax validation and removed files with missing dependencies or syntax errors.Source: 2) Dataset pre-processing · p. 4
  • Final examples are self-contained circuits, which simplifies fine-tuning and evaluation but leaves hierarchical, multi-module designs as an open challenge.Filtering also removed converted examples with more than nine occurrences of tmp_ivl and applied 95% deduplication using MinHash and Jaccard similarity.Source: 2) Dataset pre-processing · pp. 4–5
  • VHDL Style Guide reformatting was an optional preprocessing step used only for examples from the Hugging Face dataset.Source: 2) Dataset pre-processing · p. 5

3) Prompt Processing

The authors pair VHDL circuits with natural-language design prompts generated by an advanced LLM. They specify prompt content and remove code comments before training.

3) Prompt Processing

  • Each VHDL example was sent to a commercial LLM to produce a short natural-language task description of the circuit’s behavior.The process initially used GPT-4omini and later Gemini 2.0 Flash, with few-shot examples and a structured system prompt.Source: 3) Prompt Processing · p. 5, Figure 3 · p. 5
  • Prompts had to name input and output ports and types, describe key behavior, and avoid code-specific terminology or implementation details.Source: 3) Prompt Processing · p. 5
  • After prompt generation, code comments were removed to reduce token overhead while retaining code syntax and semantics.Source: 3) Prompt Processing · p. 5

4) Dataset Statistics

The dataset is characterized by automated complexity labels and code-length statistics, with examples structured as natural-language instructions paired with VHDL responses.

4) Dataset Statistics

  • Complexity labels use entity port counts—fewer than 5 for basic, 5–9 for intermediate, and more than 9 for advanced—and also consider signal and port types.Gemini classified designs from extracted entity declarations, but the authors characterize the method as an inaccurate rough estimate.Source: 4) Dataset Statistics · pp. 5–6
  • 56.2% of examples are basic, 31.9% intermediate, and 11.9% advanced, as shown in Figure 4.The authors attribute the imbalance to simpler modules being more common in publicly available VHDL code.Source: 4) Dataset Statistics · p. 6, Figure 4 · p. 6
  • Most VHDL examples contain 200–1000 tokens; the mean is 723 and the median is 418, as shown in Figure 5.Although a small number exceed 2000 tokens, most remain within the 4096-token training limit.Source: 4) Dataset Statistics · p. 6, Figure 5 · p. 6
  • Each dataset entry pairs a natural-language circuit instruction with a complete VHDL implementation.The example describes a 5-input OR gate and provides its VHDL response.Source: 4) Dataset Statistics · p. 6

B. Large Languange Model Selection

The study selects code-oriented, instruction-tuned, open models with hardware availability in mind, then fine-tunes them using QLoRA. Experiments found dataset choice to be the most significant factor for generalization in this study.

B. Large Languange Model Selection

C. LLM Fine-Tuning

  • QLoRA freezes the base model and trains inserted low-rank adapter layers, reducing memory requirements while aiming for gains comparable to full-parameter fine-tuning.Source: C. LLM Fine-Tuning · p. 6
  • Training used 4-bit quantization to fine-tune 7B-parameter models on an NVIDIA A100 with 40 GB VRAM.Each model was trained for 5 epochs, averaging 6 hours.Source: C. LLM Fine-Tuning · pp. 6–7
  • Adding NEFTune noise at parameter 5 degraded Mistral accuracy by 3%, while masking prompt tokens made little difference in the reported tests.The authors note that further human evaluation would be needed to assess whether noise improves robustness across diverse inference prompts.Source: C. LLM Fine-Tuning · p. 7
  • The authors report that the dataset was the most significant, if not the only relevant, factor for improving LLM generalization in this study.Fine-tuning methods and parameters were therefore kept mostly consistent across models.Source: C. LLM Fine-Tuning · p. 7

1) EvaluationEvaluation Design-Task Set

The evaluation uses separately constructed, paraphrased design prompts based on HDLBits and FPGA4Student examples. A 100-task set provides broad coverage, while a 15-task subset supports deeper simulation and hardware testing.

1) EvaluationEvaluation Design-Task Set

  • The authors created evaluation prompts from HDLBits.net and FPGA4Student.com examples because VHDL-Eval was not publicly available.Prompts were generated with the training-set approach and paraphrased to make user inputs unique.Source: 1) EvaluationEvaluation Design-Task Set · p. 7
  • The 100-eval set contains 100 varied-complexity tasks for a broad performance overview.Source: 1) EvaluationEvaluation Design-Task Set · p. 7
  • The 15-eval set contains five tasks at each of three complexity levels and supports functional simulation and hardware prototyping.Its smaller size reflects the labor required for testing generated circuits and prototyping hardware.Source: 1) EvaluationEvaluation Design-Task Set · p. 7

2) Three-Level Validation Pipeline

The evaluation pipeline moves from syntax and synthesis checks to functional simulation and FPGA prototyping, because passing syntax alone does not establish practical utility. The pass@k metric estimates the probability of obtaining a correct output within the top k attempts.

2) Three-Level Validation Pipeline

  • The three validation stages assess syntax and synthesizability, functional behavior against testbenches, and hardware-level properties through FPGA prototyping.Hardware measurements included Vivado power estimates, longest-path propagation delay, and configurable-logic resources.Source: 2) Three-Level Validation Pipeline · pp. 7–8
  • Syntax checking used GHDL, and only files that compiled without errors advanced to functional verification.Generated responses were saved as VHDL files and checked automatically; syntax correctness was scored as a binary outcome.Source: 2) Three-Level Validation Pipeline · p. 8
  • Functional verification covered only the 15-problem set and relied on manually created testbenches, deduplicated designs, and simulation-output review.The manual verification process consumed approximately 90% of total evaluation time; designs had to pass all test scenarios without assertion violations.Source: 2) Three-Level Validation Pipeline · p. 8
  • FPGA prototyping implemented functionally correct designs on a Zynq SoC, with selected tasks mapped to physical switches and LEDs to confirm operation on hardware.Vivado reports supplied power, timing, and resource metrics; this was not a real PPA assessment.Source: 2) Three-Level Validation Pipeline · p. 8

3) Checking analyis and used metrics

  • pass@k estimates the probability of finding a correct output among the top k attempts, where n is outputs per prompt and c is correct outputs.The metric accounts for a correct solution appearing after the first generated output.Source: 3) Checking analyis and used metrics · pp. 8–9

IV. INFERENCE SETUP & RESULTS

Inference settings were controlled for reproducibility, with the number of completions adjusted between the two evaluation sets.

A. Inference setup

  • Temperature was fixed at 0.1; models generated 3 completions per task for the 100-eval set and 10 for the 15-eval set.The 3-completion setting was the minimum for pass@k assessment, while 10 completions were used for a more accurate probability estimate.Source: A. Inference setup · p. 9

B. Perfomance Analyisis and Results

Fine-tuning improved syntax correctness most for DeepSeek-Coder, while performance on the 15-design evaluation set varied with circuit complexity. Human review found near-valid syntax improvements, but advanced designs remained difficult and the dataset’s single-file, simple-example bias constrained generalization.

B. Perfomance Analyisis and Results

  • Baseline performance ranged from CodeLlama-7b-instruct at 68% to Falcon at 3%.The paper attributes CodeLlama’s higher baseline to code-focused pretraining and describes Falcon as poorly suited to HDL generation.Source: B. Perfomance Analyisis and Results · p. 9
  • Fine-tuning improved syntax correctness: DeepSeek-Coder rose from 57% to 92%, while CodeLlama rose from 68% to 81%, with DeepSeek-Coder showing the larger reported gain.Source: B. Perfomance Analyisis and Results · p. 9
  • Human review found that fine-tuned outputs that still failed syntax checks often needed only minimal edits to become valid VHDL.The review used GHDL error messages to examine a subset of outputs that failed before and after fine-tuning.Source: B. Perfomance Analyisis and Results · p. 9
  • On the 15-design set, basic circuits generally succeeded, while PWM and 8-bit comparator functional success fell to around 40–60% for most models.The 9-to-1 multiplexer achieved 100% syntax-checking success across all models except Falcon; intermediate designs had more functional-verification failures.Source: B. Perfomance Analyisis and Results · p. 9
  • All models failed to generate syntactically correct 8-bit FFT designs, and only DeepSeek produced working matrix-multiplier designs, at low accuracy.Hardware prototyping also confirmed selected designs on a Zedboard, while outputs for the same task varied in optimization across models.Source: B. Perfomance Analyisis and Results · p. 10
  • The models were limited to complete designs in a single VHDL file, while complex systems commonly use multiple interconnected files.The authors link weaker generalization on sophisticated designs to a dataset dominated by simpler examples and insufficient examples of complex patterns.Source: B. Perfomance Analyisis and Results · p. 10

V. DISCUSSION AND FUTURE WORK

The discussion identifies data scarcity, evaluation effort, and limited scope as constraints, while the conclusion reports improved generation quality after fine-tuning and successful FPGA deployment.

V. DISCUSSION AND FUTURE WORK

  • VHDL lacks the large, diverse public corpus available for mainstream programming languages, and proprietary advanced designs further constrain training-data availability.The paper identifies this scarcity as a barrier for researchers and students outside the hardware industry.Source: §V · p. 11
  • Manual testbench creation and simulation consumed approximately 90% of evaluation time, limiting the practicality of larger-scale functional assessment.The authors call for enhanced automated frameworks because syntax and synthesis checks can be automated more readily than functional evaluation.Source: §V · p. 11
  • The study focused on functionally correct VHDL rather than optimization or advanced verification, and future work includes hierarchical designs and broader dataset diversity.Other stated directions include area, power, and timing objectives, formal verification, and automated test generation with systematic coverage analysis.Source: §V · p. 11

VI. CONCLUSION

  • Five open-source 7B-parameter models were fine-tuned with QLoRA, and the paper reports improved generation quality, highest gains for DeepSeek-Coder, and successful FPGA deployment.The evaluation combined syntax correctness, functional behavior, and hardware prototyping; the authors conclude that syntax alone is insufficient to assess practical utility.Source: §VI · pp. 11–12

Related papers

Loading 2610.28676v1…