PaperlayerView exact revision on arXiv

Source-linked AI summary

From Concept to Practice: an Automated LLM-aided UVM Machine for RTL Verification

Junhao Ye, Yuchen Hu, Ke Xu, Dingrong Pan, Qichun Chen, Jie Zhou, Shuai Zhao, Xinwei Fang, Xi Wang, Nan Guan, Zhe Jiang

arXiv:2504.19959v4cs.ARcs.AI

TL;DR

IC verification consumes nearly 70% of development effort, and UVM testbench construction plus coverage-oriented stimulus generation remain difficult. UVM2 automates planning, UVM testbench generation, error handling, and coverage-feedback refinement with specialised LLM agents. Across realistic RTL benchmarks, it achieves higher coverage than prior LLM-based solutions and substantially reduces verification time.

  • Problem

    IC verification consumes nearly 70% of development effort, while manual UVM testbench construction and testcase supplementation remain costly and expertise-intensive.

    Source: §I · p. 1

  • Method

    UVM2 uses specialised LLM agents and templates to plan tests, generate UVM testbenches, repair errors, and iteratively supplement stimuli using coverage feedback.

    Source: §III · p. 2

  • Results

    UVM2 achieves 87.44% average code coverage and 89.58% function coverage, exceeding UVLLM by 20.96% and 23.51%, respectively.

    Source: C. Results and Discussions · p. 7

  • Takeaways & Limitations

    UVM2 provides an automated and scalable workflow that integrates with practical verification processes while reducing manual verification effort.

    Source: C. Results and Discussions · p. 7

Abstract

from arXiv · show

Verification presents a major bottleneck in Integrated Circuit (IC) development, consuming nearly 70% of the total development effort. While the Universal Verification Methodology (UVM) is widely used in industry to improve verification efficiency through structured and reusable testbenches, constructing these testbenches and generating sufficient stimuli remain challenging. These challenges arise from the considerable manual coding effort required, repetitive manual execution of multiple EDA tools, and the need for in-depth domain expertise to navigate complex designs.Here, we present UVM^2, an automated verification framework that leverages Large Language Models (LLMs) to generate UVM testbenches and iteratively refine them using coverage feedback, significantly reducing manual effort while maintaining rigorous verification standards.To evaluate UVM^2, we introduce a benchmark suite comprising Register Transfer Level (RTL) designs of up to 1.6K lines of code.The results show that UVM^2 reduces testbench setup time by up to UVM^2 compared to experienced engineers, and achieve average code and function coverage of 87.44% and 89.58%, outperforming state-of-the-art solutions by 20.96% and 23.51%, respectively.

I. INTRODUCTION

IC verification consumes nearly 70% of development effort, while UVM testbench construction and coverage-oriented testcase supplementation remain costly and expertise-intensive. UVM2 addresses these bottlenecks with automated LLM-guided testbench generation, iterative coverage refinement, and an open benchmark for realistic RTL designs.

  • Nearly 70% of IC frontend development effort is consumed by verification, with functional verification dominating the workflow.Source: §I · p. 1, Figure 1 · p. 1
  • UVM testbench construction requires extensive manual hierarchy, configuration, interface integration, and protocol handling, producing codebases typically four to five times larger than RTL.Source: §I · p. 1
  • Coverage-driven testcase supplementation requires manually generating boundary, rare-event, and state-transition scenarios, making refinement costly and error-prone as designs grow.Source: §I · p. 1
  • Prior LLM approaches either lacked UVM integration and realistic scale or retained manual construction and randomised stimuli without targeted coverage refinement.Source: §I · p. 1
  • UVM2 combines domain-guided LLM generation with iterative coverage feedback to reduce human involvement in scalable functional verification.Source: §I · pp. 1–2
  • UVM2 integrates into practical workflows and releases a benchmark of 10 RTL designs ranging from 400 to 1,600 lines for comparison and future research.Source: §I · p. 2

II. UVM2: AN OVERVIEW

UVM2 is an automated workflow that accepts standard verification inputs, uses specialised LLM agents for planning, generation, and optimisation, and returns error and coverage reports. Its iterative stages preserve compatibility with existing industrial verification practices.

  • UVM2 assumes a Markdown design specification, JSON configuration, and RTL DUT, all of which align with existing industrial verification inputs.Source: §II · p. 2
  • AgentA extracts functional points and creates a coverage-driven testcase plan from the design specification.Source: §II · p. 2
  • AgentG combines the test plan with pre-validated templates to generate a UVM-based verification testbench.Source: §II · p. 2
  • Simulation captures testbench errors for refinement while collecting coverage data for subsequent testcase supplementation.Source: §II · p. 2
  • AgentO adds targeted sequences when coverage is insufficient and iterates until coverage goals or user-defined iteration limits are reached.Source: §II · p. 2
  • Artifacts use well-defined exchange formats so designers can replace LLMs or EDA tools and insert processes without invalidating the workflow.Source: §II · p. 2

III. THE FRAMEWORK PIPELINE

The framework pipeline structures verification as sequential planning, generation, simulation analysis, and coverage-driven supplementation. Its test-planning stage turns specifications into consistent, inspectable functional objectives and test strategies.

III. THE FRAMEWORK PIPELINE

  • The pipeline proceeds from test planning and UVM testbench generation to simulation error handling, coverage supplementation, and benchmarking.Source: §III · p. 2

A. Test planning

  • AgentA interprets design specifications to identify signal dataflow, control dependencies, interactions, state transitions, and functional expectations.Source: A. Test planning · pp. 2–3
  • For each functional point, AgentA defines stimulus conditions, observability points, and coverage goals, such as arithmetic overflow and underflow cases.Source: A. Test planning · p. 3
  • Predefined output templates store functional points, test strategies, and draft testcases in a consistent format for UVM integration and manual inspection.Source: A. Test planning · p. 3
  • The three-step planning process enforces deterministic reasoning paths and aligns later testbench implementation with specification-driven objectives.Source: A. Test planning · p. 3

B. Verification Testbench Generation

UVM2 generates testbenches according to component dependencies and divides synthesis between templates for regular structure and LLMs for behavioural logic. Guided prompts then constrain behavioural generation using component context, responsibilities, and mistake mitigation.

B. Verification Testbench Generation

C. Simulation Analysis

UVM2 simulates the assembled testbench with VCS, collects coverage and runtime diagnostics, and repairs failed components through iterative LLM-guided regeneration.

C. Simulation Analysis

  • UVM2 uses VCS to compile and simulate the DUT and generated testbench, collecting function coverage, code coverage, and runtime logs.These reports evaluate testbench quality and identify verification gaps.Source: C. Simulation Analysis · p. 4
  • When simulation fails, UVM2 feeds structured VCS error logs to AgentG to regenerate faulty components and correct structural or behavioral issues.The repair loop repeats until simulation succeeds or a predefined maximum iteration count is reached.Source: C. Simulation Analysis · p. 4
  • The repair mechanism targets generative errors including syntax issues, interface mismatches, and improper sequencing caused by ambiguous context or insufficient dependency information.Source: C. Simulation Analysis · p. 4

D. Testcase Supplement

UVM2 supplements testcases by analyzing coverage gaps and refining protocol-level sequences with targeted prompts, while preserving the last valid testbench when optimization fails.

D. Testcase Supplement

  • Coverage-driven sequence refinement supplements missing protocol behaviors because low or plateaued coverage indicates that stimuli are not exercising key DUT regions.Source: D. Testcase Supplement · pp. 4–5
  • UVM2 analyzes coverage gaps by logic region, transaction stage, or stimulus dependency, then passes deficient categories to AgentO for targeted testcase generation.The loop continues until satisfactory coverage is achieved.Source: D. Testcase Supplement · p. 5
  • AgentO prompts combine role customization, coverage analysis, stimulus constraints, and mistake mitigation to guide missing-stimulus generation and avoid invalid sequences.Source: D. Testcase Supplement · p. 5
  • If an updated testbench fails simulation, UVM2 reverts to the last valid version and continues optimization until target coverage or a user-defined iteration limit is reached.This makes the coverage-refinement loop fault-tolerant.Source: D. Testcase Supplement · p. 5

E. Benchmark

The benchmark suite spans diverse RTL modules and deliberately varies design size, structure, and behavioral complexity to assess UVM2’s scalability and adaptability.

E. Benchmark

  • Smaller benchmarks such as RTLLM [44] and VerilogEval use designs under 200 lines and therefore do not represent the full complexity of industrial scenarios.The paper considers them unsuitable for evaluating UVM2’s full scope and scalability.Source: E. Benchmark · p. 5
  • The benchmark covers cryptographic cores, arithmetic units, memory controllers, compression, and peripheral communication interfaces.Examples include AES, SHA256, SM4, ALU, DDR3, SDRAM, HUF, SPI, and UART.Source: E. Benchmark · p. 5
  • Benchmark designs range from sequential logic and pipelined arithmetic to complex control flow, protocol compliance, asynchronous communication, and timing constraints.Source: E. Benchmark · p. 5
  • Modules were selected with varying line counts and complexities to assess the approach’s scalability and adaptability across realistic verification challenges.AES and HUF stress datapath and state-machine coverage, while DDR3 and SDRAM stress protocol compliance.Source: E. Benchmark · p. 5

IV. EVALUATION

The evaluation uses Deepseek-based LLM agents, VCS simulation, standardized coverage and completion metrics, and repeated generation across a diverse benchmark.

IV. EVALUATION

  • The experiment uses Deepseek-v3 agents at temperature 0.3, VCS simulation, two iterations for analysis and supplement, and five generations per module under pass@5.Source: §IV · p. 5

A. Evaluation Metrics

  • Success Rate of Generation measures whether a generated UVM testbench passes the full VCS simulation flow and matches reference-testbench outputs.A failed simulation or mismatch is categorized as a generation failure.Source: A. Evaluation Metrics · p. 5
  • Coverage measures test completeness after successful simulation through code coverage and functional coverage.Code coverage measures exercised RTL code, while functional coverage measures specified functional scenarios.Source: A. Evaluation Metrics · p. 5
  • Completion Time measures the time required to complete the entire functional verification process when UVM2 generates a correct testbench.Source: A. Evaluation Metrics · p. 5

B. Research Questions

The paper evaluates UVM2 through five research questions covering testbench correctness, verification completeness, efficiency, repair, and coverage optimization.

  • RQ1 examines whether UVM2 generates syntactically and semantically correct UVM testbenches that conform to standard structure and compile without errors.Source: B. Research Questions · pp. 5–6
  • UVM2 is evaluated on benchmark designs spanning commonly implemented hardware modules, including cryptographic, arithmetic, memory, and communication designs.Source: Table I · p. 6
  • RQ2 compares UVM2 with existing LLM-based verification approaches using code and functional coverage.Source: B. Research Questions · p. 6
  • RQ3 measures end-to-end verification efficiency against experienced engineers, while RQ4 evaluates automatic repair’s effect on generation success.Source: B. Research Questions · p. 6
  • RQ5 assesses whether UVM2’s optimization mechanism improves functional coverage during simulation.Source: B. Research Questions · p. 6

C. Results and Discussions

UVM2 improves UVM testbench generation, verification coverage, speed, repair success, and testcase optimization, while exposing domain-specific limitations in complex components and protocols.

  • UVM2 achieves an average 89.70% success rate across UVM components, with template-generated components reaching 100%.Driver and monitor components remain near 80% because they require detailed timing and handshake knowledge.Source: C. Results and Discussions · p. 6
  • UVM2 reaches a 2.59× higher component-generation success rate than Deepseek-v3’s 34.54% baseline and surpasses all tested baselines.The evaluation replaces corresponding modules in manually crafted UVM testbenches while keeping other components unchanged.Source: C. Results and Discussions · p. 6
  • LLM-generated UVM components exhibit protocol, naming, TLM, and responsibility errors that can make verification components invalid or non-functional.Structured guidance, templates, and iterative repair address these domain-knowledge gaps.Source: C. Results and Discussions · pp. 6–7, Figure 10 · p. 7
  • UVM2 achieves average code coverage of 87.44% and function coverage of 89.58%, exceeding UVLLM by 20.96% and 23.51%, respectively, across the evaluated modules.Coverage decreases as RTL code lines and submodule count increase but remains competitive; UVM2 coverage is higher than the other two works for each module.Source: C. Results and Discussions · p. 7
  • UVM2 achieves speedups of 14.68× on HUF and up to 38.82× on DFI compared with experienced engineers under similar coverage conditions.Testbench generation and testcase supplementation average 41.34 and 20.84 minutes, respectively.Source: C. Results and Discussions · p. 7, Table II · p. 8
  • Two repair iterations raise final testbench generation success to 86.67% from 69.13%, a 17.54% increase.The gain from the second to third round is only 0.50%, supporting a two-iteration setting.Source: C. Results and Discussions · p. 8, Table III · p. 8
  • Testcase supplementation increases code coverage by 7.6% and function coverage by 9.7%, although benefits vary by module.SPI gains 17.7% code coverage and ALU gains 29.5% function coverage, while SM4 shows no noticeable functional-coverage improvement after reaching 83.5%.Source: C. Results and Discussions · p. 8, Figure 12 · p. 8

V. CONCLUSION

UVM2 automates UVM testbench generation and verification workflow integration while iteratively improving stimuli through coverage feedback. On a benchmark of designs up to 2k lines, it achieves strong coverage and substantially improves development efficiency, while hardware-protocol understanding remains an open challenge.

V. CONCLUSION

  • UVM2 generates UVM testbenches with an 86.67% success rate, approximately 88% coverage, and an average runtime of 102.75 minutes.The framework achieves up to 38.82x speedup in development efficiency.Source: §V · p. 8
  • The benchmark includes designs of up to 2k lines, addressing the limited scale of prior benchmarks whose DUTs were mostly under 150 lines.This benchmark was developed to better reflect real-world verification demands.Source: §V · p. 8
  • 87.44% average code coverage and 89.58% average function coverage outperform state-of-the-art solutions by 20.96% and 23.51%, respectively.An optimisation mechanism supports the reported coverage results.Source: §V · p. 8
  • UVM2 integrates with industrial tools such as Synopsys VCS to automate the entire verification workflow.The framework is intended to narrow the gap between LLM capabilities and industrial-grade hardware verification requirements.Source: §V · p. 8
  • Future work must improve LLM understanding of hardware protocols and semantics and embed richer verification knowledge into generation.These directions are identified as necessary for advancing toward autonomous, expert-level verification assistance.Source: §V · p. 8
Loading 2504.19959v4…