PaperlayerView exact revision on arXiv

Source-linked AI summary

UVLLM: An Automated Universal RTL Verification Framework using LLMs

Yuchen Hu, Junhao Ye, Ke Xu, Jialin Sun, Shiyue Zhang, Xinyao Jiao, Dingrong Pan, Jie Zhou, Ning Wang, Weiwei Shan, Xinwei Fang, Xi Wang, Nan Guan, Zhe Jiang

arXiv:2411.16238v1cs.AR

TL;DR

Hardware verification remains labor-intensive, and existing approaches can miss errors when test coverage is limited. UVLLM combines LLMs with UVM-based testing and repair; it reports higher fix rates across syntax and functional errors and faster execution than MEIC. The authors identify the lack of robust quantitative metrics for judging repair correctness as a limitation.

  • Problem

    Hardware verification relies heavily on human expertise, while limited testbench coverage can let errors escape detection and make repair results unreliable.

    Source: §I · p. 1

  • Method

    UVLLM combines UVM testbenches with LLM-based reference models, log-guided error localization, and iterative RTL repair.

    Source: B. UVM Processing · p. 3, C. Post-processing via Localization… · p. 3

  • Results

    UVLLM achieved fix rates of 86.99% for syntax errors and 71.92% for functional errors, and averaged 10.42x faster execution than MEIC.

    Source: B. Results and Discussions · p. 6

  • Takeaways & Limitations

    The authors conclude that integrating formal verification processes improves the practicality of hardware design verification against specification requirements.

    Source: B. Results and Discussions · p. 5

    The paper notes a lack of robust quantitative metrics for assessing repair correctness, which can make rollback inefficient and cause false positive rollbacks.

    Source: C. Post-processing via Localization… · p. 4

Abstract

from arXiv · show

Verifying hardware designs in embedded systems is crucial but often labor-intensive and time-consuming. While existing solutions have improved automation, they frequently rely on unrealistic assumptions. To address these challenges, we introduce a novel framework, UVLLM, which combines Large Language Models (LLMs) with the Universal Verification Methodology (UVM) to relax these assumptions. UVLLM significantly enhances the automation of testing and repairing error-prone Register Transfer Level (RTL) codes, a critical aspect of verification development. Unlike existing methods, UVLLM ensures that all errors are triggered during verification, achieving a syntax error fix rate of 86.99% and a functional error fix rate of 71.92% on our proposed benchmark. These results demonstrate a substantial improvement in verification efficiency. Additionally, our study highlights the current limitations of LLM applications, particularly their reliance on extensive training data. We emphasize the transformative potential of LLMs in hardware design verification and suggest promising directions for future research in AI-driven hardware design methodologies. The Repo. of dataset and code: https://anonymous.4open.science/r/UVLLM/.

I. INTRODUCTION

Hardware verification remains labor-intensive, while prior APR methods often use fixed templates and existing LLM-based approaches can miss errors because of testbench coverage limits. UVLLM combines UVM and LLMs for testing and repair, reporting higher repair rates and faster debugging.

I. INTRODUCTION

  • Existing APR methods such as Cirfix, Strider, and RTLrepair largely use fixed templates and focus on functional errors, limiting repair scope.Source: §I · p. 1
  • Prior solutions can let about 10% of benchmarks bypass the testbench without repair, while limited test coverage makes some repairs unreliable.Source: §I · p. 1
  • UVLLM integrates UVM with LLMs in an end-to-end verification framework intended to operate under practical assumptions and manage LLM uncertainty.Source: §I · p. 1
  • The paper presents flexible UVM testing with LLM-generated reference models, an open-source toolset, and an error dataset of 331 code instances.The dataset is derived from verified projects and contains realistic errors across various modules.Source: §I · p. 1
  • 86.99% syntax-error and 71.92% functional-error fix rates; UVLLM exceeded MEIC in repair rates and execution time and reached up to 48x debugging speedup over experienced engineers.The results are reported for realistic testing scenarios.Source: §I · p. 1

II. UVLLM: AN OVERVIEW

UVLLM verifies RTL against design specifications through an iterative process combining pre-processing, UVM testing, log analysis, and repair. It archives candidate code and pass rates, stopping when errors are cleared or the iteration limit is reached.

II. UVLLM: AN OVERVIEW

  • The framework takes design specifications and untested RTL as inputs, using the specifications to generate a reference model for verification.Source: §II · p. 2
  • Pre-processing removes syntax errors and focused timing-related issues before UVM testing, which checks behavioral discrepancies and produces logs for targeted repair.Source: §II · p. 2
  • Post-processing extracts error locations from UVM logs for debug agents to propose patches; repaired code is synthesized for further iterations.Source: §II · p. 2
  • The loop stops when no errors remain or the maximum iterations are reached, while code versions and pass rates are archived for future iterations.Source: §II · p. 2

III. UVLLM: THE FRAMEWORK PIPELINE

The paper outlines UVLLM’s pipeline stages, its LLM-agent microsystems, and the human-like error generation used for evaluation.

III. UVLLM: THE FRAMEWORK PIPELINE

  • The framework pipeline includes pre-processing, testing, post-processing for error location, and repair.Source: §III · p. 2
  • The paper discusses the microsystems integrated with the LLM agents as part of the framework operation.Source: §III · p. 2
  • The paper presents human-like error generation for evaluation.Source: §III · p. 2

A. Pre-processing via the Linter

UVLLM uses linting and a combined LLM-script approach to prepare RTL before verification, then applies UVM testing and coverage collection to identify discrepancies.

A. Pre-processing via the Linter

  • Verilator linting, static analysis, and slicing identify syntax and potential error sources before execution, while LLM-assisted preprocessing can reduce later debugging calls.Source: A. Pre-processing via the Linter · p. 2
  • LLMs address syntax errors, scripts handle selected timing-related warnings, and preprocessing iterates until syntax errors and focused warnings are resolved.One example replaces non-blocking assignments with blocking assignments in combinational logic when Verilator warns about the timing behavior.Source: A. Pre-processing via the Linter · p. 3

B. UVM Processing

  • UVM testing addresses finite-test coverage limitations by using flexible test modes, coverage collection, and a scoreboard that compares actual and expected outcomes.Its agents combine sequencers, drivers, and monitors to exercise the DUT.Source: B. UVM Processing · p. 3

10 End Function

The passage names ErrInfoFetch and identifies SL as suspicious code lines.

10 End Function

  • ErrInfoFetch is shown with inputs FD, L_UVM, WR, WS, and Iter.The excerpt does not explain how these inputs are used.Source: §10 · p. 3
  • SL denotes suspicious code lines.Source: §10 · p. 3

23 End Function

UVLLM enriches UVM logs with dynamically localized error information and uses rollback to manage unsuccessful repairs. Its localization process escalates from mismatch signals to suspicious execution paths when needed.

23 End Function

  • LLM-generated reference models can support simulations, and UVM can be extended with automated assertions for protocols such as APB and AHB.Source: §23 · p. 3

C. Post-processing via Localization Engine

  • UVLLM extracts higher-value error information from UVM logs using time-aware dynamic localization tailored to HDL.The method is described as more precise and temporally sensitive than static localization.Source: C. Post-processing via Localization… · p. 3
  • When mismatch signals alone do not resolve errors, UVLLM adds suspicious execution paths to the LLM analysis.The localization engine uses temporal signal discrepancies and dynamic slicing over data flow graphs to refine diagnosis.Source: C. Post-processing via Localization… · p. 3
  • The rollback process lacks robust quantitative metrics for judging repair correctness, risking inefficiency and false-positive rollbacks.Source: C. Post-processing via Localization… · p. 4
  • Rollback preserves code versions and scores, reverting to the highest-scoring version when a new iteration scores lower.The scoreboard's test pass rate is assumed to correlate with fewer errors and better functionality.Source: C. Post-processing via Localization… · p. 4

D. Repair Agent

The repair agent uses design specifications, RTL, and error information to propose code patches, with damage repairs helping avoid repeating unsuccessful corrections. The benchmark-generation passages describe sampling validated code and comparing repository versions to capture realistic Verilog errors.

D. Repair Agent

  • The repair agent receives the design specification, RTL code, and error information, with damage repairs added to discourage repeated unsuccessful corrections.Agent prompts are tailored to different debugging roles, and structured JSON outputs pair wrong code with corrected code.Source: D. Repair Agent · p. 4

E. Benchmark Generation

  • Table I is identified as a partial listing of common Verilog errors in real-world designs.Source: Table I · p. 4
  • Benchmark samples were selected from validated open-source datasets and deliberately infused with typical errors.The selection reflects the relationship between module errors, code length, and functional complexity.Source: E. Benchmark Generation · p. 4
  • Comparing code immediately before and after repository commits yielded error-modification pairs for LLM prompts and pattern matching.Examples include assignment misuse and port mismatches in instantiations.Source: E. Benchmark Generation · p. 4

IV. EVALUATION

The evaluation uses GPT-4-turbo by default, runs each instance five times, and limits iterations to five. It defines hit rate, fix rate, and execution time as complementary evaluation measures.

IV. EVALUATION

  • The evaluation uses GPT-4-turbo by default, five runs per instance, and a five-iteration threshold.The stated setup uses RTLLM [38] and VCS, Iverilog, Modelsim, Yosys, and Verilator.Source: §IV · p. 4

A. Evaluation Metrics

  • Hit Rate measures the proportion of instances resolved under test cases, while Fix Rate adds independent expert validation across broader scenarios.The excerpt describes test-case correctness across inputs for HR and expert-confirmed effectiveness for FR.Source: A. Evaluation Metrics · pp. 4–5
  • Execution time is measured from initial design input to completion of the final code output.Source: A. Evaluation Metrics · p. 5

B. Results and Discussions

UVLLM achieves higher fix rates across syntax and functional errors than the compared methods. Its results also show small hit-rate–fix-rate deviations and distinct strengths across repair stages and module types.

  • 1.4% HR–FR deviation for UVLLM on functional errors, versus deviations above 30% for other methods; Fig. 6 shades these differences.Source: Figure 6 · p. 5, B. Results and Discussions · p. 5
  • Across 27 modules and nine error types, counters reached 100% syntax and 95% functional FR, while FSMs reached 89% and 32%, respectively.The paper notes that repairing more complex designs remains challenging.Source: B. Results and Discussions · p. 5
  • Syntax-error FRs exceeded functional-error FRs across the same module types; the paper attributes this to HDL-code training and compiler/linter localization information.Source: B. Results and Discussions · pp. 5–6
  • Pre-processing resolves 74.72% of syntax cases, while Mismatch Signal mode corrects 41.46% of functional cases in the segmented repair process.Source: Table II · p. 6, B. Results and Discussions · p. 6
  • 86.99% syntax-error FR and 71.92% functional-error FR; UVLLM consistently achieves higher fix rates than the compared script-based and LLM-aided methods.Source: B. Results and Discussions · pp. 5–6
  • 25.96% of functional-error repairs introduced new syntax issues, which pre-processing then addressed; 11.29% of syntax-only errors advanced to Mismatch Signal repair.Source: B. Results and Discussions · p. 6
  • 88.50% miscellaneous-module syntax FR versus MEIC’s 66.67%, and 66.23% arithmetic-module functional FR versus MEIC’s 40.53%.Source: B. Results and Discussions · p. 6
  • 10.42x faster than MEIC on average; pre-processing typically takes less time than Mismatch Signal mode despite handling over half of benchmark repairs.Source: B. Results and Discussions · p. 6

C. Ablation Study

The ablation compares generating complete code snippets with generating new code from original-repair pairs. Complete-code generation slightly reduces repair accuracy and increases execution time overall, but can handle some minor errors more effectively.

C. Ablation Study

  • Complete-code generation slightly lowers repair accuracy and increases execution time compared with generating original-repair code pairs.This is the overall result of the ablation.Source: C. Ablation Study · p. 6
  • Direct generation can outperform the replacement strategy in specific scenarios, particularly when minor errors make searching for a code segment difficult.The authors attribute this advantage mainly to handling minor errors that challenge LLM search efficiency.Source: C. Ablation Study · p. 6
  • For a missing port definition, reproducing the entire module is more straightforward than replacement, which may overlook essential context.The example contrasts repairing “module a(A); ...(Missing Definition of Port A)...endmodule” by replacement versus reproduction.Source: C. Ablation Study · p. 6

V. CONCLUSION

UVLLM addresses testbench construction, test execution, result analysis, and repair for hardware design verification. On the proposed benchmark, it achieves high syntax and functional error fault rates, nearly 100% test coverage, and faster execution than MEIC.

V. CONCLUSION

  • UVLLM comprehensively addresses testbench construction, test execution, result analysis, and repair in hardware design verification.The conclusion presents this as an automated universal verification framework.Source: §V · pp. 6–7
  • 86.99% syntax and 71.92% functional error fault rates, nearly 100% test coverage, and 10.42 times faster performance than MEIC were reported for UVLLM on the proposed benchmark.The conclusion states that reasonable segmentation and feedback engineering improve the framework’s verification efficiency.Source: §V · p. 7
  • The framework demonstrates the feasibility of using LLMs to verify Verilog code regardless of its initial state.The reported feasibility is demonstrated on the proposed benchmark.Source: §V · p. 7

Related papers

Loading 2411.16238v1…