PaperlayerView exact revision on arXiv

Source-linked AI summary

HAVEN: Hybrid Automated Verification ENgine for UVM Testbench Synthesis with LLMs

Chang-Chih Meng, Yu-Ren Lu, Guan-Yu Lin, Tsung Tai Yeh, Kai-Chiang Wu, I-Chen Wu

arXiv:2604.27643v1cs.ARcs.AI

TL;DR

LLMs struggle to generate correct HDL-based UVM testbenches because hardware-specific syntax and timing conventions are poorly represented in their training data. HAVEN uses LLMs for structured specification and coverage-gap analysis, while templates and rule-based DSL CodeGen produce UVM components and sequences; across 19 open-source IP designs, it achieves 100% compilation success, 90.6% code coverage, and 87.9% functional coverage.

  • Problem

    LLMs have difficulty generating correct UVM testbench code because HDL syntax and timing constructs are rare in their training data.

    Source: §I · p. 1

  • Method

    HAVEN uses LLMs for structured architectural and coverage-gap information, then applies protocol-specific templates and rule-based DSL CodeGen for UVM components and sequences.

    Source: §I · p. 2, A. DSL Sequence Generation · p. 5

  • Results

    Across 19 open-source IP designs, HAVEN achieves 100% compilation success, 90.6% average code coverage, and 87.9% average functional coverage.

    Source: §I · p. 2

  • Takeaways & Limitations

    Delegating SystemVerilog generation to rule-based systems lets HAVEN combine HDL syntax correctness with high coverage while reducing LLM calls and token usage.

    Source: §I · p. 2, §VIII · p. 8

    The evaluation uses open-source designs that may not reflect proprietary IP complexity.

    Source: §VII · p. 8

Abstract

from arXiv · show

Integrated Circuit (IC) verification consumes nearly 70% of the IC development cycle, and recent research leverages Large Language Models (LLMs) to automatically generate testbenches and reduce verification overhead. However, LLMs have difficulty generating testbenches correctly. Unlike high-level programming languages, Hardware Description Languages (HDLs) are extremely rare in LLMs training data, leading LLMs to produce incorrect code. To overcome challenges when using LLMs to generate Universal Verification Methodology (UVM) testbenches and sequences, wepropose HAVEN (Hybrid Automated Verification ENgine) to prevent LLMs from writing HDL directly. For UVM testbench generation, HAVEN utilizes LLM agents to analyze design specifications to produce a structured architectural plan. The HAVEN Template Engine then combines with predefined and protocol-specific templates to generate all UVM components with correct bus-handshake timing. For UVM sequence generation, HAVEN introduces a Protocol-Aware Sequence Domain-Specific Language (DSL) that decomposes sequences into fine-grained step types. A set of predefined DSL patterns first establishes sequences that achieve a high coverage rate without LLM involvement. HAVEN continues to improve the coverage rate by iteratively leveraging LLM agents to analyze coverage gap reports and compose additional targeted DSL sequences. Unlike previous works, HAVEN is the first system that utilizes pre-defined, protocol-specific Jinja2 templates to generate all UVM components and UVM sequences using our proposed Protocol-Aware DSL and rule-based code generator. Our experimental results on 19 open-source IP designs spanning three interface protocols (Direct, Wishbone, AXI4-Lite) show that HAVEN achieves 100% compilation success, 90.6% code coverage, and 87.9% functional coverage on average, and is SOTA among LLM-assisted testbench generation systems.

I. INTRODUCTION

HAVEN addresses the difficulty of generating correct UVM testbenches by restricting LLMs to structured analysis and delegating HDL generation to templates and rule-based tools. It combines protocol-aware component generation with iterative DSL-based sequence optimization.

  • HAVEN targets the overhead of IC verification, which consumes nearly 70% of the development cycle, by automating UVM testbench and sequence generation.Source: §I · p. 1
  • LLMs frequently generate incorrect HDL because hardware constructs such as blocking versus non-blocking assignments and clock synchronization are rare in their training data.Source: §I · p. 1
  • HAVEN prevents LLMs from writing SystemVerilog directly, using them instead to extract structured information while a rule-based generator guarantees syntactic correctness.Source: §I · p. 2
  • Protocol-specific Jinja2 templates combine architectural plans with design information to generate all UVM components, including drivers with correct bus-handshake timing.Source: §I · p. 2
  • A Protocol-Aware Sequence DSL decomposes stimuli into fine-grained steps, beginning with predefined patterns and then using iterative LLM analysis of coverage gaps to add targeted sequences.Source: §I · p. 2
  • Across 19 open-source IP designs, HAVEN achieves 100% compilation success, 90.6% average code coverage, and 87.9% average functional coverage, with an average of 6 LLM calls, 68k tokens, and $0.38 per design.Source: §I · p. 2

II. HAVEN: AN OVERVIEW

HAVEN separates UVM testbench generation into a template-based Stage 1 and an iterative DSL sequence-optimization Stage 2. Both stages use structured artifacts and rule-based code generation so LLMs select or organize information without writing SystemVerilog.

  • Stage 1 generates a complete, compile-passing UVM testbench with predefined sequences from structured UVM Blueprint and Protocol Flows artifacts.Source: §II · p. 3
  • Stage 2 feeds simulation coverage gaps to an LLM, which produces targeted DSL sequences that rule-based CodeGen converts into protocol-correct UVM sequences over repeated iterations.Source: §II · p. 3
  • The shared Protocol-Aware DSL represents operations such as register writes, polling loops, value sweeps, and BFM actions as JSON step types.Source: §II · p. 3
  • Rule-based CodeGen translates DSL JSON into protocol-correct SystemVerilog, while safety and timing rules prevent the LLM from generating HDL directly.Source: §II · p. 3

III. STAGE 1: TESTBENCH GENERATION

Stage 1 converts a design specification into a structured Blueprint, infers stimulus strategies without further LLM involvement, and uses templates plus a compile-fix loop to produce a complete UVM testbench.

III. STAGE 1: TESTBENCH GENERATION

  • Stage 1 takes the design specification and produces a compile-passing UVM testbench with predefined sequences before Stage 2 begins.Source: §III · p. 3
  • The pipeline combines structured artifacts with template rendering, rule-based stimulus inference, and compile-fix validation before proceeding to coverage optimization.Source: §III · p. 3

A. Blueprint Extraction

  • The UVM Blueprint records agent topology, transaction-field contracts, protocol type, and BFM requirements while excluding non-stimulus signals from sequence items.Source: A. Blueprint Extraction · p. 3

B. Strategy Inference

  • A rule-based strategy engine assigns exhaustive enumeration to fields with width w ≤4, constrained random verification to wider fields, and fixed values to configured defaults.Source: B. Strategy Inference · p. 3

C. Template Rendering and Predefined Sequence Generation

HAVEN renders protocol-correct UVM components and predefined sequences from templates and DSL strategies, including peripheral models and functional coverage infrastructure. Its examples contrast these generated artifacts with common LLM driver errors.

C. Template Rendering and Predefined Sequence Generation

D. Compile-Fix and Simulation

HAVEN protects template-generated UVM components during compile-fix and simulation, then uses coverage gaps to generate targeted DSL sequences for remaining behaviors.

D. Compile-Fix and Simulation

  • The compile-fix loop leaves template-generated UVM components unchanged while correcting LLM-generated sequence items from VCS errors before simulation.After successful compilation, simulation produces the initial coverage report used by Stage 2.Source: D. Compile-Fix and Simulation · pp. 4–5

IV. STAGE 2: SEQUENCE OPTIMIZATION

  • Stage 1’s predefined sequences reach 84.6% average code coverage and 79.8% average functional coverage, but complex protocols and deep state machines require targeted stimuli.Stage 2 runs for K=3 iterations in the evaluation, feeding each latest coverage report back to the LLM.Source: §IV · p. 5

A. DSL Sequence Generation

  • Stage 2 feeds typed coverage gaps and prior DSL sequences to an LLM constrained to output JSON, preventing free-form SystemVerilog generation.The input includes protocol flows, the UVM Blueprint, and parsed line, toggle, FSM, branch, and functional-coverage gaps.Source: A. DSL Sequence Generation · p. 5
  • A CAN example maps an uncovered transmission-status coverpoint to a register write followed by a bounded status poll, entirely represented as DSL JSON.The LLM writes no SystemVerilog in this targeted-sequence example.Source: A. DSL Sequence Generation · p. 5

B. Rule-Based Code Generation

HAVEN translates constrained DSL outputs into protocol-correct UVM sequences through rule-based generation, then accumulates targeted sequences across coverage-refinement iterations.

B. Rule-Based Code Generation

  • The CodeGen wraps register writes in UVM transaction calls, maps fields through Blueprint signal names, and inherits protocol timing from driver templates.An auto-fix pass repairs common DSL JSON mistakes before translation, avoiding an extra LLM round-trip.Source: B. Rule-Based Code Generation · p. 5
  • Fig. 4 contrasts error-prone direct SystemVerilog generation with JSON-only DSL generation whose CodeGen produces Blueprint-matched transactions and bounded timeouts.The direct approach uses a wrong API, lacks UVM transaction handling, and contains an unbounded wait.Source: Figure 4 · p. 5
  • 96.8% of LLM-emitted steps fall within the five most common DSL types, while 15.8% of designs reach the current DSL expressiveness ceiling.Those designs require control-flow primitives beyond the current linear DSL.Source: B. Rule-Based Code Generation · p. 5

C. Accumulation and Iterative Refinement

  • Table III reports benchmark coverage, functional coverage, token cost, and Stage 2 iteration count across the 19-design evaluation, with K=3 used throughout.The UVM2 column identifies the nine designs overlapping that benchmark, and its average is reported for those designs.Source: Table III · p. 6, C. Accumulation and Iterative Refinement · p. 6
  • Each refinement iteration adds sequences to Stage 1 rather than replacing them, preserving previously covered states while targeting updated coverage gaps.The loop terminates when improvement falls below 0.1 percentage points or when the maximum iteration count is reached; most designs converge within 2–3 iterations.Source: C. Accumulation and Iterative Refinement · pp. 5–6

V. EVALUATION

HAVEN evaluates compile reliability, coverage, and prior-work comparison on 19 open-source IP designs spanning Direct, Wishbone, and AXI4-Lite protocols.

V. EVALUATION

  • HAVEN is evaluated on compile reliability, effective coverage, and comparison with UVM2 and other LLM-based approaches.The evaluation reports whether each generated testbench compiles, how much code and functional coverage it achieves, and how it compares with prior work.Source: §V · p. 6

A. Benchmark Suite

  • The benchmark contains 19 open-source designs across Direct, Wishbone, and AXI4-Lite protocols, with complexity ranging from 180 to 11 k LOC.Nine designs overlap with the UVM2 benchmark, and the Wishbone subset tests driver-template reuse across SPI and GPIO controllers.Source: A. Benchmark Suite · p. 6

B. Methodology and Metrics

  • Each design uses one Stage 1-plus-Stage 2 pipeline execution without per-design tuning or manual intervention.This defines the evaluation setting for the reported results.Source: B. Methodology and Metrics · p. 7
  • Code coverage is the unweighted mean of line, condition, toggle, branch, and FSM metrics, excluding the group metric.The metric is computed from URG measurements.Source: B. Methodology and Metrics · p. 7
  • Functional coverage uses URG’s group metric over covergroups emitted by HAVEN’s subscriber template, while code coverage independently measures RTL source coverage.Blueprint cover_bins define the functional bins, and code coverage is unaffected by covergroup design.Source: B. Methodology and Metrics · p. 7
  • Compile success means VCS produces a simulation binary without errors after the compile-fix loop.This operational definition distinguishes compilation success from later coverage outcomes.Source: B. Methodology and Metrics · p. 7

C. Environment

The evaluation uses GPT-5.2 and Synopsys VCS, reporting perfect compile success from template-based generation and a failed no-template ablation.

C. Environment

  • HAVEN uses OpenAI GPT-5.2 for all pipeline stages without fine-tuning or domain-specific training.The environment specification identifies the model configuration used in the pipeline.Source: C. Environment · p. 7
  • Synopsys VCS 2024.09-SP2 performs compilation, simulation, and URG coverage analysis.These tools support the reported compile and coverage measurements.Source: C. Environment · p. 7

A. Compile Reliability

  • 100% compile success is achieved across all 19 IP designs, with the template-plus-fixup loop requiring 1–4 iterations and averaging 1.9.Every run produces a VCS-compilable testbench.Source: A. Compile Reliability · p. 7
  • Disabling all templates causes all three tested designs—ALU, AES, and SPI—to fail compilation, due to driver, scoreboard, and monitor errors.The ablation identifies template-generated UVM components as load-bearing for compile success.Source: A. Compile Reliability · p. 7

B. End-to-End Coverage

HAVEN combines predefined and targeted DSL sequences to improve coverage across 19 IP designs, while outperforming the closest prior system on shared designs. Its pipeline also reports low average LLM usage and cost.

B. End-to-End Coverage

  • 90.6% code coverage and 87.9% functional coverage are achieved on average after targeted DSL refinement across 19 IP designs.Stage 1 provides 84.6% code and 79.8% functional coverage; Stage 2 adds 6.0 and 8.1 percentage points, respectively.Source: B. End-to-End Coverage · p. 7
  • The largest code-coverage gains occur for DFI (+47.6 pp), UART (+24.3 pp), and UE SPI (+16.2 pp), while some designs require protocol-level state setup beyond register-level stimuli.The framework generalizes across all three bus protocols without architectural changes.Source: B. End-to-End Coverage · p. 7

C. Comparison with Prior Work

  • HAVEN templates every UVM component slot and uses structured DSL sequence generation, unlike systems relying on free-form LLM SystemVerilog output.Its benchmark contains 19 IP designs spanning three bus protocols.Source: C. Comparison with Prior Work · p. 7
  • HAVEN improves over UVM2 on nine shared designs by 3.6 percentage points in code coverage and 1.1 percentage points in functional coverage.UVM2 reports 87.44% code and 89.58% functional coverage, while HAVEN reaches 91.0% and 90.7%.Source: C. Comparison with Prior Work · p. 7

D. Cost and Token Usage

  • HAVEN averages 6 LLM calls and $0.37 per complete testbench across 19 designs, with template rendering contributing zero LLM tokens.Per-design cost ranges from $0.19 to $0.62.Source: D. Cost and Token Usage · p. 7

E. LLM Robustness

HAVEN remains usable across open-source models, but coverage and compilation reliability decline on designs requiring larger outputs or heavier protocol reasoning. Reported model averages exclude failed designs, limiting direct comparability.

E. LLM Robustness

  • Table IV reports coverage and average token usage per design for combined Stage 1 and Stage 2 runs with K=3 across the evaluated language models.The table summarizes the robustness comparison across models under the same two-stage pipeline.Source: Table IV · p. 8
  • Model averages exclude failed designs, and token averages for some Qwen models cover fewer than 19 designs because tracking began mid-experiment.Qwen3.5-9B fails to produce a compiling testbench on eight of 19 designs, so its average uses the remaining 11.Source: E. LLM Robustness · p. 8
  • Every evaluated open-source model runs the HAVEN pipeline and exceeds 79% code coverage on the designs it handles, although GPT-5.2 remains better on complex peripherals.Qwen3.5-27B reaches 85.8% code and 78.3% functional coverage, 4.6 and 7.4 percentage points below GPT-5.2.Source: E. LLM Robustness · pp. 7–8
  • Context length and model capacity are the main robustness bottlenecks, with ETHMAC exceeding the 16 K output-token limit for both gpt-oss models.DFI stresses nearly all open models, while smaller models may fail the compile-fix loop on complex designs.Source: E. LLM Robustness · p. 8

VII. DISCUSSION

HAVEN achieves strong verification results through rule-based SystemVerilog generation, while discussion identifies coverage, reproducibility, benchmark, and template-engineering boundaries.

VII. DISCUSSION

  • Six designs remain below 90% code coverage because current DSL expressiveness, multiagent coordination, or state-space exploration is insufficient for specific behaviors.SDRAM, ETHMAC, HUF, I2C, and CAN are identified as examples with distinct limiting factors.Source: §VII · p. 8
  • Coverage varies across runs even though compile success is deterministic, because sequence items and subscribers remain LLM-generated.The paper proposes characterizing run-to-run variance in future work.Source: §VII · p. 8
  • Protocol-specific templates require one-time engineering and validation, after which a single Wishbone driver template serves all eight evaluated Wishbone designs.Adding a new bus protocol requires authoring and validating one driver template and one monitor template.Source: §VII · p. 8
  • The evaluation uses open-source designs, which may not reflect the complexity of proprietary IP.The paper reports generalization across five LLMs with a 1–8 percentage-point coverage gap.Source: §VII · p. 8

VIII. CONCLUSION AND FUTURE WORK

  • 100% compile success, 90.6% code coverage, and 87.9% functional coverage are reported across 19 IP designs spanning three bus protocols.The pipeline averages 6 LLM calls and $0.38 per design.Source: §VIII · p. 8

Related papers

Loading 2604.27643v1…