PaperlayerView exact revision on arXiv

Source-linked AI summary

RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model

Yao Lu, Shang Liu, Qijun Zhang, Zhiyao Xie

arXiv:2308.05345v3cs.LGcs.AR

TL;DR

Existing LLM-based RTL-generation studies lacked broad, unified, quality-aware evaluation. The paper introduces RTLLM, an open-source benchmark with automated syntax, functionality, and design-quality assessment, and proposes self-planning for prompting. GPT-3.5 with self-planning approaches GPT-4’s reported correctness and quality performance in the benchmark.

  • Problem

    Prior work used small, author-defined designs and often evaluated correctness without design quality, making scalability and fair comparison difficult.

  • Method

    RTLLM provides 30 varied designs, unified natural-language descriptions, automated evaluation of syntax, functionality, and PPA, human-crafted reference RTLs, and the self-planning prompt technique.

  • Results

    GPT-3.5 with self-planning reaches 73% correct syntax and 14/30 correct functionalities, compared with GPT-3.5 alone at 55% and 10/30, while GPT-4.0 achieves 81% and 15/30.

  • Takeaways & Limitations

    RTLLM enables quantitative comparison of LLM-based RTL-generation solutions across correctness and design quality using a common benchmark.

Abstract

from arXiv · show

Inspired by the recent success of large language models (LLMs) like ChatGPT, researchers start to explore the adoption of LLMs for agile hardware design, such as generating design RTL based on natural-language instructions. However, in existing works, their target designs are all relatively simple and in a small scale, and proposed by the authors themselves, making a fair comparison among different LLM solutions challenging. In addition, many prior works only focus on the design correctness, without evaluating the design qualities of generated design RTL. In this work, we propose an open-source benchmark named RTLLM, for generating design RTL with natural language instructions. To systematically evaluate the auto-generated design RTL, we summarized three progressive goals, named syntax goal, functionality goal, and design quality goal. This benchmark can automatically provide a quantitative evaluation of any given LLM-based solution. Furthermore, we propose an easy-to-use yet surprisingly effective prompt engineering technique named self-planning, which proves to significantly boost the performance of GPT-3.5 in our proposed benchmark.

I. INTRODUCTION

LLM-based RTL generation aims to make hardware design more agile, but prior evaluations used small, author-defined designs and often omitted design quality. RTLLM addresses these gaps with a broader benchmark, automated evaluation across syntax, functionality, and quality, and the self-planning technique for improving GPT-3.5.

  • Motivation: LLMs are being explored to generate design RTL from natural-language instructions for more agile hardware design.This approach is intended to reduce the barrier to hardware design and improve design productivity.
  • Research gap: Prior studies used relatively simple, small-scale, author-defined designs, limiting scalability analysis and fair comparison across LLM solutions.Different human-written natural-language descriptions can also vary substantially, motivating unified descriptions.
  • Benchmark: RTLLM provides 30 designs spanning varied complexities and scales for natural-language RTL generation.The benchmark is open-source and supports HDL formats including Verilog, VHDL, and Chisel when synthesis and simulation are supported.
  • Evaluation: RTLLM evaluates generated RTL against syntax, functionality, and design-quality goals using automated scripts and human-crafted reference RTLs.The reference designs provide a baseline for quantitatively evaluating design quality.
  • Prompt engineering: Self-planning is a prompt-engineering technique that requires no human interference and brings GPT-3.5 close to GPT-4 performance.The paper presents self-planning alongside the benchmark as a contribution for improving LLM-based RTL generation.

II. PROBLEM FORMULATION

The RTL generation task maps a natural-language functionality description to RTL using an LLM. Prompt engineering can revise the description before generation, with optional human revision afterward.

  • Task formulation: Given a natural-language description L, the task develops a model F that generates design RTL V = F(L).In the paper’s formulation, F is currently based on LLMs.
  • Prompt and human revision: Prompt engineering P transforms L into LP = P(L), which is then supplied to the LLM for RTL generation.The generated RTL may additionally be revised by human engineers as V = H(F(LP )).
  • Prompt and human revision: Human engineers can revise the LLM output when needed, producing the ultimate RTL through V = H(F(LP )).This formulation distinguishes automated prompting from optional manual post-processing.

III. RTLLM: AN RTL GENERATION BENCHMARK

RTLLM evaluates generated RTL through three progressive goals: syntax correctness, functionality correctness, and practical design quality. Functionality is tested with sampled cases, while quality concerns performance, power, and area.

  • Syntax goal: The syntax goal requires generated RTL to be syntactically correct and synthesizable into a netlist without syntax errors.Synthesis tools are used to verify this fundamental goal.
  • Functionality goal: The functionality goal requires generated RTL to match designer expectations by passing all test cases in a comprehensive testbench.RTLLM samples a reasonable number of cases, so passing them does not guarantee 100% functional correctness.
  • Success criteria: A design is considered successful when it satisfies both syntax and functionality goals.The paper then distinguishes practical usefulness through additional design-quality requirements.
  • Design quality goal: Design quality is assessed through performance, power, and area, which should also be desirable for practically useful RTL.RTLLM presents the goals as progressive evaluation criteria rather than correctness alone.

B. An Overview of the Design Generation Benchmark

RTLLM packages each design with a natural-language description, testbench, and human-crafted reference RTL, then automates generation, functionality testing, and PPA evaluation.

  • Benchmark contents: RTLLM collects 30 common designs with varied scales and complexities and provides three files for each design.The files support description-based generation, functionality verification, and quality comparison.
  • Benchmark contents: The description file L specifies functionality, module name, and input/output signals with names and widths.These predefined interfaces enable automatic functionality verification with the provided testbench.
  • Benchmark contents: The testbench T contains multiple input cases and correct output values for verifying generated design functionality.It corresponds to the module and signal definitions in L.
  • Benchmark contents: The reference RTL VH is hand-crafted by human designers and provides the baseline for quantitatively evaluating generated design quality.The reference designs have passed the proposed testbenches.
  • Automated workflow: The automated workflow generates RTL from each description, tests functionality with T, and synthesizes RTL to compare PPA against VH.The three stages are generation, functionality testing, and synthesis-based quality analysis.

C. Detailed Inspection of the Benchmark

RTLLM contains 30 diverse digital designs spanning functionalities, implementation requirements, complexities, and scales. Its arithmetic and logic designs increase coverage beyond prior released datasets.

  • Benchmark composition: RTLLM provides 30 common digital designs, including 11 arithmetic and 19 logic designs with varied functionalities.The benchmark includes human-crafted Verilog reference designs and reports HDL lines and synthesized netlist cell counts to characterize scale and complexity.
  • Arithmetic designs: The arithmetic subset covers accumulators, adders, multipliers, and dividers across bit widths from 4 to 64 bits.Adder examples range from basic and full-adder implementations to lookahead and pipelined ripple adders.
  • Logic designs: The logic subset ranges from counters and finite state machines to a simplified RISC CPU and a multiply–accumulate processing element.
  • Coverage: RTLLM combines diverse functionalities, implementation requirements, complexities, and scales, making its overall scope significantly larger than prior released datasets.

IV. SELF-PLANNING TECHNIQUE

Self-planning changes RTL generation from one direct query into two steps: first produce a natural-language plan and syntax-error advice, then generate RTL using that material. The approach requires no additional human effort or design data and reduces observed errors in examples.

  • Overview: Self-planning decomposes RTL generation into a two-step process without requiring extra human effort or existing design data.
  • Planning step: The first step asks the LLM for natural-language reasoning steps and advice about syntax errors to avoid during RTL generation.The output includes a detailed plan and syntax guidance, which are collected for the second step.
  • RTL-generation step: The second step feeds the original design description together with the collected plan and advice into the LLM to generate final RTL.
  • Example: multi 16bit: For multi 16bit, self-planning resolves GPT-3.5’s syntax errors caused by defining new variables inside the always block.

A. Experiment Setup

The evaluation synthesizes generated RTL with Synopsys Design Compiler and simulates it with Synopsys VCS, while testing five parallel outputs per design without human correction or additional LLM queries.

  • Evaluation flow: Design quality is evaluated through logic synthesis with Synopsys Design Compiler using the ‘compile ultra’ option and timing comparisons under uniformly negative slack.
  • Evaluation flow: Functionality is verified through RTL simulation with Synopsys VCS.
  • Evaluated systems: The experiment evaluates GPT-3.5, GPT-4.0, two academic LLMs, and GPT-3.5 plus self-planning.The listed academic systems are Thakur et al.’s 16-billion-parameter Verilog-finetuned model and StarCoder’s 15-billion-parameter general code model.
  • Sampling protocol: Each design is generated in five parallel sessions with the same description, and all five outputs are evaluated without human fixing or another LLM query.

B. RTL Generation Correctness

RTLLM evaluates both syntax and functionality correctness across generated RTL, with GPT-4.0 leading and self-planning substantially recovering GPT-3.5’s performance. The reported ranking places GPT-4 with self-planning first and academic models last.

  • Commercial models: 81% correct syntax and 15/30 correct functionalities make GPT-4.0 the strongest evaluated standalone model.
  • Self-planning effect: GPT-3.5 alone reaches 55% correct syntax and 10/30 correct functionalities, while GPT-3.5 with self-planning reaches 73% and 14/30.
  • Academic models: Thakur et al. achieves 40% syntax and 5/30 functionality correctness, while StarCoder achieves 27% syntax and 5/30 functionality correctness.
  • Overall ranking: The summarized performance ranking is GPT-4 + self-planning > GPT-4 > GPT-3.5 + self-planning > GPT-3.5 > Thakur et al. >= StarCoder.

C. RTL Generation Quality

RTLLM evaluates generated RTL beyond syntax and functionality by measuring synthesized design qualities, using eligibility rules that exclude incorrect designs. GPT-4 achieves the most best-quality results, while the aggregate comparison has an acknowledged rigor limitation.

  • Quality evaluation: Design-quality values are measured on post-synthesis netlists using power, timing, and area, with worst negative slack (WNS) as the timing metric.The table also includes designer-generated reference design VH, whose designs are functionally correct.
  • Quality evaluation: Only designs with correct syntax and functionality are eligible for best-quality comparisons across power, performance, and area.Syntax-correct but functionally wrong designs are marked separately, while unsynthesizable designs are left blank.
  • Results: GPT-4 and GPT-3.5 + self-planning outperform the designer-crafted reference designs VH in the reported quality comparison.The reported ordering is GPT-4.0 > GPT-3.5 + self-planning > GPT-3.5 > Thakur et al..
  • Caveat: Summing individual best-quality counts provides a straightforward but less rigorous comparison because design objectives have strong trade-offs.Table IV reports gate-level netlist qualities synthesized with Design Compiler.

VI. CONCLUSION

The paper concludes by presenting RTLLM as an open-source benchmark for natural-language RTL generation and introducing self-planning as a prompt-engineering technique. Future work will extend the benchmark, validate self-planning, and fine-tune open-source models.

  • VI. CONCLUSION: RTLLM is an open-source benchmark for design RTL generation with natural language instructions and includes more designs at higher scale and complexity than recent datasets.The authors plan to keep extending and maintaining the benchmark.
  • VI. CONCLUSION: The paper proposes self-planning as an effective prompt-engineering technique for natural-language RTL generation.Future work includes continued validation of the technique.
  • VI. CONCLUSION: Future work will fine-tune the authors’ own open-source models to improve performance on the RTLLM benchmark.
Loading 2308.05345v3…