Source-linked AI summary
RTLCoder: Outperforming GPT-3.5 in Design RTL Generation with Our Open-Source Dataset and Lightweight Solution
Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, Zhiyao Xie
TL;DR
RTL generation lacks strong open-source solutions because existing approaches rely heavily on commercial LLMs and high-quality training data is scarce. RTLCoder combines a 27K-plus labeled RTL dataset with code-quality-feedback training in a fully open-source 7B solution. It outperforms GPT-3.5 across representative benchmarks and exceeds GPT-4 on VerilogEval EvalMachine.
Problem
High-performance open-source RTL generation is limited by reliance on commercial LLMs and the scarcity of organized, high-quality circuit-design training data.
Method
RTLCoder combines an automated flow generating over 27 thousand labeled Verilog samples with a memory-efficient training scheme based on code quality feedback.
Results
RTLCoder outperforms GPT-3.5 on representative RTL-generation benchmarks and exceeds GPT-4 in VerilogEval EvalMachine, with RTLCoder-DeepSeek scoring 61.2.
Takeaways & Limitations
The fully open-sourced 7B solution offers a lightweight, reproducible alternative for RTL generation with a lower hardware barrier.
Abstract
from arXiv · showhide
The automatic generation of RTL code (e.g., Verilog) using natural language instructions and large language models (LLMs) has attracted significant research interest recently. However, most existing approaches heavily rely on commercial LLMs such as ChatGPT, while open-source LLMs tailored for this specific design generation task exhibit notably inferior performance. The absence of high-quality open-source solutions restricts the flexibility and data privacy of this emerging technique. In this study, we present a new customized LLM solution with a modest parameter count of only 7B, achieving better performance than GPT-3.5 on all representative benchmarks for RTL code generation. Especially, it outperforms GPT-4 in VerilogEval Machine benchmark. This remarkable balance between accuracy and efficiency is made possible by leveraging our new RTL code dataset and a customized LLM algorithm, both of which have been made fully open-source.
I. INTRODUCTION
RTL generation research increasingly uses LLMs, but high-performance open-source solutions remain limited by scarce, poorly organized training data and dependence on commercial tools. RTLCoder addresses this gap with an open-source dataset, quality-feedback training scheme, and lightweight model.
- High-performance open-source RTL generation models are unavailable because organized circuit data is proprietary while online data is messy or costly to clean.
- RTLCoder introduces an automated flow producing over 27 thousand diverse labeled Verilog design problems and answers.
- A memory-efficient training scheme based on code quality feedback further boosts performance, while the 7B model requires only four commercial GPU cards for training.
- The complete data-generation flow, dataset, training algorithm, and fine-tuned models are fully open-sourced.
II. AUTOMATIC DATESET GENERATION
RTLCoder constructs an automated dataset-generation flow that pairs natural-language design instructions with reference Verilog code. The flow uses GPT-3.5 across keyword preparation, instruction generation, and reference-code generation.
- The generated dataset contains over 27 thousand samples, each pairing a natural-language design instruction with expected Verilog code across varied difficulty levels.
- The automated flow has three stages: RTL-domain keyword preparation, instruction generation, and reference-code generation.
- GPT-3.5 is used for dataset generation through general prompt templates that control outputs at each stage.
A. Stage 1: Keywords Preparation
Stage 1 prepares a broad RTL keyword pool by prompting GPT with a tree-like structure of digital-design categories and subfields.
- GPT generates hundreds of digital-design keywords for subsequent dataset-generation stages.
- A tree-like querying structure expands from root-level RTL categories into related subfields and examples to broaden topic coverage.
B. Stage 2: Instruction Generation
Stage 2 generates and expands RTL design instructions from keyword combinations and existing Verilog code. Automated mutation and validation produce a larger, more complex instruction pool with reasoning steps.
- Keyword combinations and existing Verilog source code generate complementary RTL design instructions, increasing dataset diversity.
- Mutation operations and rule-based validation iteratively expand the instruction library to over 50,000 valid instructions.
- GPT-generated reasoning steps add detailed information to the instruction pool.
C. Stage 3: Reference Code Generation
Stage 3 generates five reference Verilog candidates per instruction, retains syntax-correct outputs, and discards instructions whose candidates all fail checking.
- C. Stage 3: Reference Code Generation: Five reference design-code candidates are generated for each instruction, then evaluated with an automated syntax checker.Only syntax-correct candidates are retained; instructions with five failed candidates are discarded.
- C. Stage 3: Reference Code Generation: The resulting dataset contains more than 27,000 samples after reference-code generation and filtering.The authors attribute RTLCoder’s performance partly to retaining largely correct GPT-3.5 outputs through syntax filtering.
- C. Stage 3: Reference Code Generation: Syntax filtering is presented as a refinement of GPT-3.5’s Verilog-generation capabilities for training RTLCoder.
III. NEW TRAINING SCHEME INCORPORATING CODE QUALITY FEEDBACK
The training scheme addresses exposure bias by scoring multiple generated code candidates and increasing the model’s probabilities for higher-quality answers.
- III. NEW TRAINING SCHEME INCORPORATING CODE QUALITY FEEDBACK: Code-quality feedback is incorporated into training to alleviate exposure bias and further improve RTL-generation performance.The issue arises because autoregressive decoding uses the model’s own previous tokens rather than reference tokens.
- III. NEW TRAINING SCHEME INCORPORATING CODE QUALITY FEEDBACK: Multiple candidates and the original reference code are grouped for each instruction, forming the training candidates to be scored.The candidate set is represented as yi = {yi,k}, with K generated codes for instruction xi.
- III. NEW TRAINING SCHEME INCORPORATING CODE QUALITY FEEDBACK: A syntax checker or unit test assigns quality scores, and training increases generation probabilities for higher-scoring answers.
- III. NEW TRAINING SCHEME INCORPORATING CODE QUALITY FEEDBACK: Gradient-accumulation-like computation reduces the scheme’s space complexity from O(K) to O(1).
IV. EXPERIMENTAL RESULTS
The evaluation uses VerilogEval and RTLLM, covering RTL-generation tasks at different scales and assessing syntax, functionality, and pass@k performance.
- IV. EXPERIMENTAL RESULTS: VerilogEval contains EvalMachine and EvalHuman, each with more than 100 RTL design tasks, and is evaluated using pass@k.
- IV. EXPERIMENTAL RESULTS: RTLLM V1.1 contains 29 larger-scale RTL design tasks and reports separate syntax and functionality scores computed with Synopsys VCS.
- IV. EXPERIMENTAL RESULTS: The benchmarks jointly evaluate RTL generation through pass@k, design syntax, and design functionality measures.
B. Model Training
The experiments remove training samples highly similar to benchmark tests, train Mistral-7B and DeepSeek-Coder models on the resulting data, and compare generators across two benchmarks.
- B. Model Training: Training samples highly similar to benchmark test cases are removed using Rouge-L before model training.
- B. Model Training: Mistral-7B-v0.1 and DeepSeek-Coder-6.7b are fine-tuned on 27K instruction-code pairs using four RTX 4090 GPUs.The reported setup uses Adam with learning rate 1e-5, context length 2048, and global batch size 256.
- B. Model Training: Some compared models cannot be directly evaluated on RTLLM because they are closed-source, so their RTLLM scores are unavailable.
- B. Model Training: RTLCoder outperforms GPT-4 on VerilogEval EvalMachine and ranks second to GPT-4 on the other reported benchmarks.
- B. Model Training: The quality-feedback scheme generates three candidates per instruction with beam search and scores them using Pyverilog.
C. Experiment Results Overview
RTLCoder’s experiments compare commercial, customized, and ablated RTL-generation systems across representative benchmarks. The results show gains from both the proposed training scheme and the full 27K-sample dataset.
- 61.2 and 41.6 are RTLCoder-DeepSeek’s VerilogEval EvalMachine and EvalHuman scores, respectively, outperforming GPT-3.5 and most non-commercial baselines.RTLCoder-DeepSeek is second only to GPT-4 on EvalHuman, while RTLCoder variants surpass GPT-4 on EvalMachine.
- The proposed training scheme further outperforms direct MLE training across all benchmarks.The ablation indicates that code-quality-feedback training improves performance beyond dataset effects alone.
- The full 27K-sample dataset clearly outperforms a randomly selected 10K subset on every metric.The comparison associates larger and more diverse training data with stronger benchmark performance.
V. CONCLUSION
RTLCoder is presented as a fully open-source, lightweight RTL-generation solution that outperforms GPT-3.5 and achieves state-of-the-art performance among non-commercial systems. Its dataset-generation flow, labeled data, and design-quality-based training scheme support replication and further improvement.
- RTLCoder achieves state-of-the-art performance among non-commercial RTL-generation solutions while outperforming GPT-3.5.
- The work contributes a complete dataset with over 27 thousand labeled samples and a data-generation flow for hardware-design tasks.
- A design-quality-scoring training scheme improves performance, while RTLCoder’s low hardware barrier supports replication and further improvement.