Source-linked AI summary
AutoChip: Automating HDL Generation Using LLM Feedback
Shailja Thakur, Jason Blocklove, Hammond Pearce, Benjamin Tan, Siddharth Garg, Ramesh Karri
TL;DR
Writing and debugging complex Verilog is demanding, while zero-shot LLM generation does not reflect iterative engineering workflows. AutoChip automates refinement by returning compiler and testbench feedback to LLMs, and its evaluation reports higher functional success, including 24.2% average improvement over baseline generation and up to 89.19% Pass@10 success with GPT-3.5 and GPT-4.
Problem
Zero-shot Verilog generation leaves developers to debug code manually, unlike iterative hardware workflows using simulation and synthesis feedback.
Method
AutoChip iteratively generates Verilog, checks it with compilation and testbench simulation, and feeds tool feedback back to an LLM without human feedback.
Results
Iterative feedback improved functional-test success by 24.2% on average versus baseline generation, with up to 89.19% Pass@10 success using GPT-3.5 and GPT-4.
Takeaways & Limitations
The results suggest AutoChip provides a pathway toward automatic hardware-circuit design through iterative LLM feedback.
Takeaways & Limitations
Some task classes consistently thwart AutoChip, and testbench-generation problems produced code that could not compile without human assistance.
Abstract
from arXiv · showhide
Traditionally, designs are written in Verilog hardware description language (HDL) and debugged by hardware engineers. While this approach is effective, it is time-consuming and error-prone for complex designs. Large language models (LLMs) are promising in automating HDL code generation. LLMs are trained on massive datasets of text and code, and they can learn to generate code that compiles and is functionally accurate. We aim to evaluate the ability of LLMs to generate functionally correct HDL models. We build AutoChip by combining the interactive capabilities of LLMs and the output from Verilog simulations to generate Verilog modules. We start with a design prompt for a module and the context from compilation errors and debugging messages, which highlight differences between the expected and actual outputs. This ensures that accurate Verilog code can be generated without human intervention. We evaluate AutoChip using problem sets from HDLBits. We conduct a comprehensive analysis of the AutoChip using several LLMs and problem categories. The results show that incorporating context from compiler tools, such as Icarus Verilog, improves the effectiveness, yielding 24.20% more accurate Verilog. We release our evaluation scripts and datasets as open-source contributions at the following link https://github.com/shailja-thakur/AutoChip.
1 INTRODUCTION
AutoChip addresses the limitations of zero-shot Verilog generation by using compiler and simulation feedback to iteratively refine designs without human intervention. The paper evaluates this approach across LLMs and HDLBits problems, reporting improved success rates and open-source resources.
- HDL coding requires substantial expertise and can produce buggy implementations, motivating more accessible generation techniques.
- Zero-shot generators require developers to debug incorrect code, unlike real-world workflows that iteratively use simulation and synthesis feedback.
- AutoChip automatically refines Verilog by feeding compilation and simulation errors back to an LLM across multiple interaction rounds.
- 27% improvement in success rate is demonstrated over the best baseline without feedback, alongside comparisons across four LLMs and multiple prompting methods.
- The implementation and a dataset of 120 benchmark prompts with corresponding Verilog testbenches are released as open-source resources.
2 BACKGROUND AND PRIOR WORK
Prior work established LLM-based Verilog generation and hardware-oriented conversational tools, but most generation remains zero-shot or human-mediated. AutoChip is situated among these efforts as an automated feedback-driven alternative.
- LLMs use transformer-based self-supervised training on vast language datasets to predict subsequent tokens across broad tasks.
- DAVE and VeriGen specialized LLMs for Verilog generation, while Chip-Chat explored conversational hardware design with ChatGPT-4.
- Commercial tools including RapidGPT, JedAI, ChipNeMo, and Synopsys.ai Copilot target hardware generation or EDA assistance with differing goals and trade-offs.
- VerilogEval evaluates Verilog generation using a zero-shot setup in which a single LLM receives a design prompt and optionally examples.
3 AUTOCHIP DESIGN FRAMEWORK
AutoChip converts a natural-language hardware specification and testbench into an iterative generate–compile–simulate–repair loop. It supports multiple LLMs, context strategies, and a targeted small-plus-large-model ensemble.
- AutoChip passes a design description, Verilog module definition, and testbench to an LLM, then feeds compilation or simulation failures back for repair.
- The loop exits when compilation and simulation pass or after a user-selected number of iterations.
- The system uses static system and design prompts, while the feedback prompt changes each iteration using the latest LLM response and tool output.
- Succinct feedback retains only the latest generated module and associated errors, reducing context while focusing repairs on the current issue.
- AutoChip evaluates GPT-4, GPT-3.5, Claude 2, and Code Llama, while VeriGen cannot participate in the feedback loop because of its non-conversational architecture.
- The ensemble strategy invokes GPT-4 only when GPT-3.5 cannot pass tests after n iterations, targeting higher accuracy at lower query cost.
4 EXPERIMENTAL SETUP
The evaluation uses HDLBits-derived Verilog problems, reconstructed testbenches, and repeated LLM trials under varying feedback settings and iteration counts. Text-only evaluation excludes problems requiring waveform or state-diagram interpretation.
- HDLBits supplies problems spanning foundational tutorials to advanced hierarchical systems and testbenches.
- The benchmark categorizes prompts by HDLBits topic order to assess which problems each LLM can solve.
- 120 of the original 178 HDLBits problems remain after excluding tasks requiring LLM interpretation of simulation waveforms and state diagrams.
- Researchers reconstructed HDLBits testbenches from provided waveforms so simulations could report mismatches and quantify failing cases.
- Failed test cases report inputs, outputs, and expected outputs, while passing cases report only a test identifier to reduce token use.
- Each prompt was run five times, with Pass@k using the best results; feedback iterations varied with a default of n = 10.
5 EXPERIMENTAL RESULTS
The experiments evaluate feedback-driven Verilog generation across LLMs, feedback settings, iteration counts, and ensembles. Feedback improves Pass@k, succinct context generally outperforms full context, and targeted GPT-4 use improves success while reducing token use.
- Feedback and iterations: Feedback substantially increases Pass@k across LLMs compared with the no-feedback baseline, even after one iteration.The evaluation uses Pass@k for n=0, 1, 5, and 10, where higher values indicate better performance.
- Feedback and iterations: Claude 2 Pass@1 rises from 37.50% at n=1 to 47.5% at n=10 with succinct feedback.The result illustrates continued gains from additional feedback iterations.
- Feedback context: Succinct feedback improves successes and reduces compilation errors as iterations increase, whereas full-context feedback does not produce the same consistent gains.Succinct feedback retains only the most recent relevant module and errors, reducing context length and usage cost.
- Feedback context: GPT-3.5-turbo Pass@5 successes increase from 27.27% to 39%, while compilation errors fall from 35.04% to 6.33% after 10 succinct-feedback iterations.With full context, GPT-3.5-turbo successes increase from 28% to 36.36% at 10 iterations, a less consistent gain.
- LLM ensembles: Selective GPT-4 use raises success from 63% with GPT-3.5-turbo alone to 79% with an ensemble while reducing token use by 60%.The ensemble applies GPT-4 to problems where GPT-3.5-turbo failed and achieved 20–80% success on 14 of 18 failing problems.
- LLM ensembles: GPT-4 inputs cost 20× more than GPT-3.5-turbo, so invoking GPT-4 once in the ensemble reduces total cost while retaining high success.The ensemble uses the larger model selectively rather than applying it to every problem.
6 DISCUSSION
The discussion uses case studies to show how iterative tool feedback helps repair generated Verilog, while also identifying context and task-class limitations. Recent-context feedback improves results and lowers token use, but some verification problems remain unsolved.
- Case studies: Iterative feedback repaired vector-concatenation code that initially failed most tests, producing valid Verilog within four rounds.The vector-concatenation case study demonstrates progressive improvement through feedback.
- Case studies: Compilation diagnostics guided correction of misdeclared variables and start/stop-bit handling in the serial-receiver FSM case study.The feedback supported rapid debugging and iterative improvement of the generated code.
- Trade-offs: Intermediate simulations improve code quality but increase cost.Simulation feedback is therefore both a debugging resource and an added expense in the iterative workflow.
- Limitations: Cellular automata, counters, and testbench-generation problems remain difficult; all generated code for the testbench-generation problems failed to compile.The authors report that some task types remain unsolved even with the GPT-3.5 and GPT-4 ensemble.
- Context choice: Most-recent-context feedback performs better than full-context feedback and uses fewer input tokens per iteration.The paper suggests that additional full-conversation context may confuse LLMs, while recent context reduces execution cost.
7 CONCLUSION
AutoChip improves iterative hardware development by using feedback to raise functional-test success over baseline generation. With GPT-3.5 and GPT-4, it reaches up to 89.19% Pass@10 success.
- 24.2% average improvement in functional-testbench success over baseline generation demonstrates the benefit of iterative feedback.
- AutoChip evaluates conversational LLMs through an iterative hardware-development workflow similar to human engineering practice.
- Up to 89.19% Pass@10 success with GPT-3.5 and GPT-4 suggests a pathway toward automatic hardware-circuit design.