PaperlayerView exact revision on arXiv

Source-linked AI summary

LintLLM: An Open-Source Verilog Linting Framework Based on Large Language Models

Zhigang Fang, Renzhi Chen, Zhijie Yang, Yang Guo, Huadong Dai, Lei Wang

arXiv:2502.10815v1cs.AR

TL;DR

Traditional Verilog linting can produce false positives and redundant reports. LintLLM combines LLM-based detection with Prompt of Logic-Tree and Defect Tracker and evaluates it on a mutation-based benchmark. On that benchmark, o1-mini outperformed the best-performing EDA tool on correct and false-positive rates, and LintLLM cost less than one-tenth as much as commercial EDA tools.

  • Problem

    Traditional Verilog linting tools can produce false positives and redundant defect reports.

    Source: §I · p. 1

  • Method

    LintLLM is an open-source framework that uses LLMs, Prompt of Logic-Tree, and Defect Tracker to detect Verilog defects, evaluated on a mutation-based benchmark.

    Source: §I · p. 1, B. Code Defect Injection · p. 3

  • Results

    18.89% higher correct rate and 15.56% lower false-positive rate: o1-mini outperformed the best-performing EDA tool, and LintLLM cost less than one-tenth as much as commercial EDA tools.

    Source: §I · p. 1, §VI · p. 4

  • Takeaways & Limitations

    The study presents LLMs as a potentially efficient and economical solution for hardware defect detection.

    Source: §VII · p. 6

Abstract

from arXiv · show

Code Linting tools are vital for detecting potential defects in Verilog code. However, the limitations of traditional Linting tools are evident in frequent false positives and redundant defect reports. Recent advancements in large language models (LLM) have introduced new possibilities in this area. In this paper, we propose LintLLM, an open-source Linting framework that utilizes LLMs to detect defects in Verilog code via Prompt of Logic-Tree and Defect Tracker. Furthermore, we create an open-source benchmark using the mutation-based defect injection technique to evaluate LLM's ability in detecting Verilog defects. Experimental results show that o1-mini improves the correct rate by 18.89\% and reduces the false-positive rate by 15.56\% compared with the best-performing EDA tool. Simultaneously, LintLLM operates at less than one-tenth of the cost of commercial EDA tools. This study demonstrates the potential of LLM as an efficient and cost-effective Linting tool for hardware design. The benchmark and experimental results are open-source at URL: https://github.com/fangzhigang32/Static-Verilog-Analysis

I. INTRODUCTION

LintLLM applies LLMs to Verilog linting to address traditional tools’ false positives and redundant reports, and introduces a benchmark for evaluating defect detection. The reported results show improved detection rates and lower cost than commercial EDA tools.

I. INTRODUCTION

  • Traditional rule-matching EDA linting can generate false positives, while customizing rules is difficult and commercial licensing fees burden smaller organizations and research institutions.Earlier open-source tools tended to have lower performance.Source: §I · p. 1
  • LintLLM is an open-source LLM-based Verilog linting framework, accompanied by an open-source benchmark of 90 designs and 11 common defect types.The framework uses Prompt of Logic-Tree and Defect Tracker.Source: §I · p. 1
  • 18.89% higher correct rate and 15.56% lower false-positive rate: o1-mini outperformed the best-performing EDA tool, while LintLLM cost less than one-tenth as much.The evaluation used a benchmark of 90 Verilog designs covering 11 common defect types.Source: §I · p. 1
  • The paper presents Prompt of Logic-Tree as a template that translates complex algorithms into tree-structured prompts intended to be easier for LLMs to understand.Source: §I · p. 1
  • Defect Tracker locates a main defect that causes multiple secondary defects, reducing redundant defect reports.Source: §I · p. 1

II. BACKGROUND AND RELATED WORKS

Prior chip-design research applies LLMs to generation, verification, and debugging, while this work examines LLMs as open-source Verilog linting tools. Traditional static-analysis linting relies on rules and can produce false positives.

A. LLMs for Chip Design

  • Unlike prior LLM hardware studies that largely focus on debugging, this work investigates LLM-native Verilog linting without relying on EDA tools.Related work spans RTL generation, EDA scripting, verification, and design assistance; existing datasets cited here lack mechanisms to evaluate defect detection.Source: A. LLMs for Chip Design · p. 2, B. Static Code Analysis · p. 2

B. Static Code Analysis

  • Static code analysis checks syntax, coding style, and design inconsistencies against predefined patterns or rules without running the code; traditional hardware linting can produce false positives.The discussion names SpyGlass, JasperGold, and Questa Lint as examples of traditional EDA tools.Source: B. Static Code Analysis · p. 2

III. METHODS

LintLLM’s workflow sends the design under test through an LLM-guided detection process and a defect-tracking feedback loop before producing a report. Its Prompt of Logic-Tree organizes task instructions and problem-solving algorithms into an ordered tree.

A. Overview

  • LintLLM’s flow inputs the DUT to an LLM, guides detection with a prompt, tracks detected defects through a feedback loop to locate the main defect, and outputs a report.Source: A. Overview · p. 2

B. Prompt of Logic-Tree

  • Prompt of Logic-Tree translates complex problem-solving algorithms into a tree with a root node for the role and task and child nodes for detailed solving steps.Source: B. Prompt of Logic-Tree · p. 2, Figure 2 · p. 2
  • The child nodes are ordered left to right: parent nodes represent main algorithm steps, and their children represent sequential sub-steps.Source: B. Prompt of Logic-Tree · p. 3

C. Defect Tracker

Verilog defects can be dependent: a main defect may cause multiple secondary defects that line-by-line linting struggles to trace. Defect Tracker identifies the main defect by fixing detected defects one at a time and checking the remaining defect count.

C. Defect Tracker

  • A main Verilog defect can cause multiple related secondary defects, which traditional line-by-line linting may not trace to their root cause.Fig. 3 illustrates an incorrect bit-width in Line 6 causing secondary defects in Lines 9 and 10.Source: C. Defect Tracker · p. 3, Figure 3 · p. 3
  • Defect Tracker identifies the main defect by fixing each detected defect individually and selecting the repair that leaves the fewest remaining defects.It records each defect’s location, type, and dependencies before this process.Source: C. Defect Tracker · p. 3

IV. BENCHMARK OF STATIC CODE ANALYSIS

The benchmark is designed to evaluate Verilog defect detection using seeded defects. It starts from syntax- and function-verified code, then applies mutation-based rules intended to imitate defects introduced during coding.

A. Correct Code Collection

  • The authors screened 150 GitHub Verilog files, removed comments, checked syntax and function, and retained 90 correct design files.Each source file was a complete module-endmodule block without an include keyword.Source: A. Correct Code Collection · p. 3

IV. BENCHMARK OF STATIC CODE ANALYSIS

  • The benchmark workflow uses mutation-based rules to imitate defects that hardware designers may unintentionally introduce during coding.The study motivates the benchmark by noting that prior benchmarks mainly assess syntax and function correctness of LLM-generated Verilog.Source: B. Code Defect Injection · p. 3, §IV · p. 3

B. Code Defect Injection

C. Benchmark Construction

The benchmark contains 11 defect categories across 90 design files, with defect code and corresponding line number. The evaluation also includes recent LLMs, uses temperature 0 for deterministic outputs where supported, and compares against commercial EDA and Verilator baselines.

C. Benchmark Construction

A. LLMs Selection

  • The study selects recent LLMs and sets temperature to 0 for deterministic outputs, although o1-mini does not support this property.Table III describes the selected models.Source: A. LLMs Selection · p. 4, Table III · p. 4
  • The best-performing commercial EDA tool and Verilator serve as the evaluation baselines.The supplied passage does not identify the commercial tool by name.Source: A. LLMs Selection · p. 4

B. Experimental Setting

The experiments compare four configurations, from direct LLM input to the combined LintLLM approach. They evaluate defect detection using correct rate and false-positive rate, with correctness determined by whether the reported line matches the injected defect.

B. Experimental Setting

  • The study evaluates LLM defect-detection capability through four experiment groups.The groups are the direct-input baseline and the three component or combined configurations.Source: B. Experimental Setting · p. 4
  • The four experiments compare direct DUT input, Prompt of Logic-Tree alone, Defect Tracker alone, and both components together as LintLLM.The Logic-Tree experiment examines correct rate, the Tracker experiment examines false-positive rate, and the combined approach explores best performance.Source: B. Experimental Setting · p. 4

C. Evaluation Metrics

  • Correct Rate measures the percentage of defects correctly detected in all DUTs, while False-Positive Rate measures false positives in all DUTs.A detection is correct when its reported defective line matches the injected defect line; reporting a defect-free line is a false positive.Source: C. Evaluation Metrics · p. 4

D. Research Questions

The paper evaluates LLM defect detection against EDA tools, examines the effects of its prompting methods, and compares detection performance across the proposed benchmark. On that benchmark, o1-mini +LintLLM achieves an 83.33% correct rate and a 12.22% false-positive rate, outperforming the best-performing commercial EDA tool.

D. Research Questions

  • The study asks how LLMs and EDA tools differ in detection performance, why the proposed methods improve LLM detection, and how LLM detection costs compare with commercial EDA tools.Source: D. Research Questions · p. 4

VI. RESULT

  • 83.33% correct rate and 12.22% false-positive rate are the best results for o1-mini +LintLLM; versus the best commercial EDA tool, its correct rate rises 18.89% and false-positive rate falls 15.56%.Compared with Verilator, its correct rate rises 21.11% and false-positive rate falls 20.00%.Source: Table IV · p. 4, §VI · p. 4
  • o1-mini +LintLLM is reported as the best-performing configuration, combining a high correct rate with a low false-positive rate.Source: §VI · p. 4, Table IV · p. 4

A. [RQ1]: What are the differences in detection performance between LLMs and EDA tools?

On the 90-DUT benchmark, LLMs outperform EDA tools in detection accuracy and false-positive control. Performance varies with benchmark complexity, and the LintLLM configuration improves all evaluated LLMs, although some GPT-series models produce slightly more false positives.

A. [RQ1]: What are the differences in detection performance between LLMs and EDA tools?

  • On simple DUTs, LLM and commercial EDA performance is comparable; on complex DUTs, o1-mini +LintLLM detects 18 defects with 9 false positives, versus 12 detected and 16 false positives for commercial EDA.Verilator detects 12 of 30 complex-DUT defects and produces 14 false positives.Source: A. [RQ1]: What are the differences in… · pp. 4–5
  • All LLMs improve with LintLLM, but false positives decrease for Llama-3.1 and DeepSeek V2.5 while increasing slightly for GPT-4, GPT-4o, and o1-mini.Llama-3.1 reduces false positives by 6 and DeepSeek V2.5 by 1.Source: A. [RQ1]: What are the differences in… · p. 5
  • Among 90 DUTs, o1-mini +LintLLM detects 75 defects and produces 11 false positives, while the commercial EDA tool detects 58 defects and produces 15 false positives.Source: A. [RQ1]: What are the differences in… · p. 5

B. [RQ2]: Why our methods can improve the detection performance of LLM?

Prompt of Logic-Tree and Defect Tracker address different detection challenges, and their combination improves overall performance, though GPT-series false-positive rates do not fall as expected. LLM cost advantages depend on code volume, while a costed CPU-control-unit example reports improved detection performance from integration with commercial EDA.

B. [RQ2]: Why our methods can improve the detection performance of LLM?

  • Prompt of Logic-Tree guides LLMs through defect detection but overlooks multiple defects, whereas Defect Tracker follows defect paths to locate main defects and reduce false positives.Except for GPT-4o, each model’s false-positive rate drops with Defect Tracker while detection correctness improves.Source: B. [RQ2]: Why our methods can improve… · p. 6
  • Combined methods improve overall performance, but false-positive rates for GPT-4, GPT-4o, and o1-mini do not drop as expected; the authors suggest possible prompt resistance or information conflicts.Source: B. [RQ2]: Why our methods can improve… · p. 6
  • DeepSeek V2.5 accuracy rises from 63.33% to 75.56% with Prompt of Logic-Tree, while Defect Tracker lowers o1-mini’s false-positive rate by 42.23%; combining both improves comprehensive performance.Source: B. [RQ2]: Why our methods can improve… · p. 6

C. [RQ3]: What are the advantages of LLMs in detection costs compared with commercial EDA tools?

  • Below 4,800 million lines of annual code, LLMs are more economical; above that threshold, commercial EDA has a clearer cost advantage.The paper assumes an annual commercial EDA license fee of about $1.2 million.Source: C. [RQ3]: What are the advantages of… · p. 6
  • For a roughly 1,000-line CPU control unit run 1,000 times daily, LLM detection costs about $0.1 million annually; integrating LLM with commercial EDA for that investment is reported to improve performance by approximately 20.00%.The reported integrated result is an 87.78% correct rate and a 7.78% false-positive rate.Source: C. [RQ3]: What are the advantages of… · p. 6

VII. CONCLUSION

The conclusion presents LintLLM as an open-source framework for detecting Verilog defects with LLMs. It characterizes LLMs as an efficient, economical hardware-defect detection solution and identifies automatic repair and IDE integration as future directions.

VII. CONCLUSION

  • LintLLM is an open-source framework that uses LLMs to detect defects in Verilog code.Source: §VII · p. 6
  • The paper concludes that LLMs offer an efficient and economical solution for hardware defect detection.Source: §VII · p. 6
  • Future work will explore automatic defect repair after detection and consider integrating LintLLM into IDEs.Source: §VII · p. 6

Related papers

Loading 2502.10815v1…