PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentDV: Closed-Loop Agentic AI for Hardware Design Verification

Navya Goli, Junzhe Liu, Zhenge Jia, Umamaheswara Rao Tida

arXiv:2608.27148v1cs.SE

TL;DR

RTL verification remains a major, manual, iterative SoC-development bottleneck, while autonomous flows must produce executable, design-grounded, coverage-guided verification environments. AgentDV uses a two-phase closed-loop pipeline that prepares verification environments, runs tests, measures coverage, iterates on feedback, and reports results. 80.9% average pass rate across all nine DUTs was achieved by Claude, while effectiveness varied across LLMs and DUT structures.

  • Problem

    RTL verification remains a major, manual, iterative SoC-development bottleneck, while autonomous flows must produce executable, design-grounded, coverage-guided verification environments.

  • Method

    AgentDV uses a two-phase closed-loop pipeline that prepares verification environments, runs tests, measures coverage, iterates on feedback, and reports results.

  • Results

    80.9% average pass rate across all nine DUTs was achieved by Claude, while effectiveness varied across LLMs and DUT structures.

  • Takeaways & Limitations

    AgentDV is most effective for control-oriented designs with register-level interfaces, whereas full coverage for AES and HMAC requires protocol-aware stimulus and datapath-level checking.

  • Takeaways & Limitations

    The CSR-focused flow can identify uncovered AES datapath logic but cannot fully activate it without protocol-level stimulus generation.

Abstract

from arXiv · show

Register-transfer level (RTL) verification consumes a major part of modern system-on-chip (SoC) development effort. Yet, recent LLM-based verification-code generation often fails to produce runnable, design-consistent, and coverage-producing testbenches. We present AgentDV, a closed-loop agentic AI framework for automated RTL verification environment generation. AgentDV transforms single-shot LLM testbench generation into a tool-grounded verification pipeline by combining LLM-guided analysis, testbench construction, simulation, coverage measurement, and iterative refinement. The framework introduces three key ideas: 1) runnability filtering to reject invalid generated environments, 2) CSR-grounded checking to reduce hallucinated signals and incorrect expected behavior, and 3) coverage-guided iteration to regenerate tests based on measured verification gaps. We evaluate AgentDV using three LLMs on challenge DUTs and public OpenTitan peripheral and security IP blocks. From our analysis, we observed that direct single-shot prompting fails to produce a valid coverage-producing environment on benchmarks. AgentDV achieves 100% pass rate on four DUTs and an average of 80.9% pass rate on all DUTs using Claude Sonnet 4.6. Similarly, an average of 58.7% and 60.6% pass rate is achieved for Llama and Qwen models, respectively. In addition, an average of 74.5%, 69.1%, and 64.9% of line coverage and 88.4%, 82.3%, and 76.7% of branch coverage for the benchmarks under consideration for Claude Sonnet 4.6, Llama, and Qwen models, respectively.

I. INTRODUCTION

AgentDV addresses the manual, costly, and insufficiently practical nature of RTL verification with a closed-loop, tool-grounded framework for executable, design-grounded, coverage-guided environment generation. It combines CSR-grounded checking, runnability filtering, and coverage-guided iteration, achieving strong pass rates and coverage across three LLM backends.

  • Motivation: RTL verification consumes about 50–60 % of total ASIC project time and remains highly manual and iterative despite advances in simulation tools and methodologies.This motivates automated workflows that generate executable tests and measure verification progress.
  • Motivation: Existing LLM-based approaches improve test generation, self-correction, and reference-model checking –[4], but incorrect generated reference models can validate flawed checking logic and evaluations remain focused on academic benchmarks,.Practical IP-level DV automation therefore remains less explored.
  • Task Setting: The ISQED 2026 Agentic AI design verification challenge formalizes an autonomous setting from RTL, natural-language specifications, and a CSR map, but leaves executable, design-grounded, coverage-guided solutions open.AgentDV treats autonomous DV as a closed-loop verification flow rather than a one-shot code-generation task.
  • Framework: AgentDV integrates LLM-guided test generation, tool execution, and coverage feedback into a closed-loop framework for automated RTL verification environment generation.The implementation uses Verilator, cocotb, and pyUVM – with Claude Sonnet 4.6, Llama 3.3 70B, and Qwen2.5-Coder 32B –.
  • Framework: CSR-grounded checking reduces reliance on LLM-inferred register addresses, reset values, access policies, and expected behavior, while runnability filtering rejects invalid environments before execution.Coverage-guided iteration then uses measured verification gaps to filter, execute, measure, and refine generated artifacts.
  • Results: 100% pass rate on four DUTs and 80.9% average pass rate across all DUTs were achieved with Claude Sonnet 4.6, versus 58.7% for Llama and 60.6% for Qwen.Average line coverage was 74.5%, 69.1%, and 64.9%, while branch coverage was 88.4%, 82.3%, and 76.7% for Claude Sonnet 4.6, Llama, and Qwen, respectively.

II. BACKGROUND AND MOTIVATION · A. LLM-Assisted RTL Verification

Prior work increasingly connects LLM-based RTL verification generation to structured workflows and external feedback, but design-consistent checking and complete verification-environment generation remain unresolved. Existing efforts span testbench generation, self-correction, tool feedback, coverage-directed stimulus, assertion synthesis, and UVM automation, without fully closing this gap.

  • A. LLM-Assisted RTL Verification: Structured generation with external feedback improves LLM-generated verification artifacts over direct prompting alone, as shown by AutoBench, CorrectBench, and PRO-V-R1 [4].These approaches target automatic testbench generation, self-correction, and tool- or reference-model-based checking.
  • A. LLM-Assisted RTL Verification: Testbench-generation advances improve stimulus creation but do not guarantee design-consistent checking, while incorrect reference models can let invalid checking logic pass tests.Reference models therefore become part of the verification trust chain and introduce their own correctness risk.
  • A. LLM-Assisted RTL Verification: Coverage-directed stimulus generation and specification-derived assertion synthesis address complementary verification stages through LLM4DV and assertion-generation methods [18].These efforts focus respectively on coverage guidance and deriving properties from design specifications.
  • A. LLM-Assisted RTL Verification: UVM2 [7] advances environment-level automation by generating UVM components and refining tests with coverage feedback.Its scope moves beyond isolated stimulus creation toward verification-environment construction.
  • A. LLM-Assisted RTL Verification: Complete subscriber, scoreboard, and reference-model generation remain outside UVM2 [7]’s core scope, leaving design-consistent checking unresolved.This limitation persists despite UVM2’s coverage-feedback loop.
  • A. LLM-Assisted RTL Verification: HAVEN [19], UVMarvel [20], and ChatTest further advance LLM-assisted testbench and verification-environment generation.The supplied passage identifies these systems as recent UVM-oriented developments without reporting comparative results.

B. Autonomous Verification Task Setting · C. Register-Based Checking and RAL · III. PROBLEM FORMULATION

The paper formulates autonomous design verification as a closed-loop task that maps RTL, specifications, and CSR maps to executable, measurable verification packages. It emphasizes runnable tests, design-consistent checking, and coverage-guided iteration, with CSR-derived facts preventing hallucinated register behavior.

  • B. Autonomous Verification Task Setting: The AutoVerifier task requires an agent to transform RTL, a natural-language specification, and a CSR map into a verification environment that executes tests, measures coverage, iterates, and reports results.The workflow comprises ANALYZE, PLAN, BUILD, TEST, MEASURE, ITERATE, and REPORT stages.
  • B. Autonomous Verification Task Setting: The task differs from one-shot generation because environments must compile, run, check meaningful behavior, refine coverage under a bounded budget, and generalize to unseen TileLink-UL DUTs.These constraints motivate deterministic processing for design facts and LLM use for planning, reasoning, and test generation.
  • C. Register-Based Checking and RAL: CSR maps specify addresses, fields, reset values, and access policies, while UVM RAL mirrors expected register-field values and compares them during check-enabled reads.This provides a register-based checking foundation for bus-attached IP verification.
  • C. Register-Based Checking and RAL: Deriving the register model from CSR artifacts makes register facts explicit and deterministic, avoiding hallucinated addresses, encodings, reset values, or access policies in LLM-generated checkers.The approach grounds checking behavior in design artifacts rather than model inference.
  • III. PROBLEM FORMULATION: The problem input is I = (R, S, M), comprising RTL source R, natural-language specification S, and CSR register map M.For the UART example, these artifacts are mapped to an executable verification package.
  • III. PROBLEM FORMULATION: The objective is to generate O = (E, T, C, F), containing an executable environment E, generated tests T, coverage report C, and verification findings F.This output formulation captures the complete verification package rather than test code alone.
  • III. PROBLEM FORMULATION: The generated package must use valid interfaces and runnable simulation, derive design-consistent checks from artifacts, and produce coverage that guides subsequent test generation.These requirements define the measurable, iterative nature of the verification flow.
  • III. PROBLEM FORMULATION: LLM-generated artifacts may reference invalid signals, assume incorrect register behavior, fail simulation, or exercise no new RTL behavior, so the goal extends beyond code generation.The artifacts must be interface-valid, design-consistent, executable, and useful for coverage-driven refinement.

IV. AGENTDV FRAMEWORK · A. Overview

AgentDV implements an autonomous design-verification formulation that transforms input design artifacts I = (R, S, M) into a verification package O. Its pipeline combines environment setup with execution, coverage measurement, feedback-driven iteration, gap closure, and final reporting.

  • A. Overview: AgentDV implements the autonomous design-verification formulation by mapping input tuple I = (R, S, M) to verification package O.
  • A. Overview: The pipeline begins with a setup phase that prepares the verification environment from the input design artifacts.
  • A. Overview: The framework is organized around setup, execution and closure, and closed-loop coverage-directed iteration.
  • A. Overview: A second phase executes tests, measures coverage, iterates using feedback, and generates the final report.
  • A. Overview: The execution-and-closure phase runs tests against the prepared verification environment.
  • A. Overview: Coverage measurement supplies feedback for iterative refinement within the pipeline.
  • A. Overview: The framework’s closed-loop coverage-directed iteration drives verification-gap closure.

B. Setup Phase · C. Execution and Closure Phase

AgentDV transforms RTL, specification, and CSR-map inputs into a runnable verification environment through Analyze, Plan, and Build, then executes, measures, iterates, and reports coverage. LLM-driven generation is constrained by CSR information, retrieval support, and runnability checks within a closed loop.

  • B. Setup Phase: The setup phase converts RTL, specifications, and CSR maps into an executable verification environment through Analyze, Plan, and Build.Analyze extracts ports, the TL-UL interface, and register access policies using Verilator and OpenTitan reggen.
  • B. Setup Phase: Analyze constructs a structured view of the design and specification, including the port list, TL-UL interface, and register list with per-field access policies.The extraction uses Verilator’s front-end and the OpenTitan reggen library.
  • B. Setup Phase: Plan invokes the LLM once with the specification and register list to produce a YAML verification plan tagged by feature, priority, and target coverage.Build then uses this plan to create the executable environment and templated CSR suite.
  • C. Execution and Closure Phase: The execution and closure phase runs the RAL stack and templated CSR suite under cocotb and Verilator, producing simulation pass/fail records and coverage data.The framework organizes Test, Measure, Iterate, and Report after the setup stages.
  • C. Execution and Closure Phase: Test generates one feature test per verification-plan item, typically 10 to 20 items per DUT, while Iterate generates one additional test per coverage-guided iteration.Both stages are bound at inference time to retrieval-augmented documentation and a runnability check that validates tests before suite admission.
  • C. Execution and Closure Phase: Measure parses simulation results into line, branch, and toggle coverage plus annotated uncovered RTL regions, which guide subsequent gap-directed test generation.Iterate re-runs simulation after each generated test and uses updated coverage for the next iteration.

D. Closed-Loop Coverage-Directed Iteration · V. EVALUATION

AgentDV closes the verification loop by generating gap-directed tests, validating them through simulation, and admitting only runnable, coverage-improving artifacts. The evaluation examines this process on challenge and public OpenTitan-derived suites, including benchmark results and ablations.

  • D. Closed-Loop Coverage-Directed Iteration: AgentDV selects an uncovered RTL region and register-table context, generates a gap-directed test, and admits it only if it is runnable and improves coverage.Each candidate passes a runnability check, executes under cocotb with Verilator, and is added to the suite only when it contributes coverage.
  • D. Closed-Loop Coverage-Directed Iteration: The loop terminates after five iterations or two consecutive iterations without coverage improvement, capping per-DUT cost while preserving closed-loop refinement.The configured limits are max_iterations=5 and patience=2.
  • D. Closed-Loop Coverage-Directed Iteration: Selective retrieval adds protocol, cocotb, coverage, and pyUVM documentation only when the prompt references an undefined DUT symbol.This retrieval gating keeps documentation context targeted to the analyzed design.
  • D. Closed-Loop Coverage-Directed Iteration: The verify stage uses ast.parse and a single-test cocotb invocation under Verilator to filter generated files before suite admission.Python crashes are retried once with error feedback, while a RAL check_t.CHECK failure is retained as a verification finding.
  • V. EVALUATION: The evaluation first describes the experimental setup and metrics, then reports benchmark results and an ablation study.The benchmark suites include challenge designs and public OpenTitan-derived suites.
  • V. EVALUATION: Figure 3 depicts stimulus generation from a target feature or uncovered region, the register table, and retrieved documentation when needed.It shows the candidate-test path through verification, retry behavior, suite admission, and retained RAL findings.

A. Experimental Setup

AgentDV is evaluated across two RTL verification benchmark suites, three LLM backends, and a standardized tool flow, with controlled comparisons and pass@1 single-suite evaluation. The benchmarks combine compact challenge DUTs with larger OpenTitan integrations, while the closed loop permits up to five iterations.

  • A. Experimental Setup: The seven-stage pipeline uses Verilator 5.x, cocotb 2.0, pyUVM 4.0, and LangGraph for simulation, co-simulation, environment construction, and orchestration.These tools support the standardized AgentDV verification flow across evaluated models and DUTs.
  • A. Experimental Setup: Three LLM backends—Claude Sonnet 4.6, Llama 3.3 70B Instruct AWQ, and Qwen2.5-Coder 32B Instruct AWQ –—use the same verification process.Claude is accessed through an API, whereas Llama and Qwen run locally with vLLM on a two-GPU, 48 GB machine; the open-weight models use RAL-aware prompts.
  • A. Experimental Setup: The study compares direct single-shot prompting with AgentDV and compares three LLM backends under the same AgentDV flow because prior frameworks use misaligned artifacts, tools, and benchmarks.All results use pass@1: one generated suite per DUT and LLM, without retries or candidate selection; the loop allows Kmax = 5 iterations and stops after patience = 2 iterations without coverage gain.
  • A. Experimental Setup: The evaluation covers compact ISQED 2026 challenge DUTs and larger OpenTitan peripheral and security IP integrations with package, primitive, and shared dependency files.This split tests controlled challenge designs alongside open-source RTL integration settings.

B. Evaluation Metrics

AgentDV is evaluated using pass rate, line coverage, and branch coverage. Together, these metrics assess whether generated verification suites are runnable and whether they exercise meaningful RTL behavior and control-flow alternatives.

  • Metric definitions: Line coverage measures reached RTL code, while branch coverage measures exercised control-flow alternatives such as if/else and case outcomes.These coverage metrics distinguish meaningful behavioral activation from merely executing the design.
  • Metric definitions: Pass rate measures whether generated verification suites are runnable and valid on clean RTL, with zero reported when no generated test passes runnability checking.It is computed from the subset of generated tests that pass verification and simulate on clean RTL without assertion failure.
  • Coverage collection: Line and branch coverage are collected with Verilator’s built-in coverage support after executing the verification-test subset that passes runnability checking.Coverage is measured on Tverify rather than on all generated tests.

C. Results and Discussion

AgentDV produces runnable, coverage-effective verification suites, with strongest overall results for Claude and register-level control-oriented DUTs. Its effectiveness decreases on larger, datapath-heavy cryptographic designs because CSR-oriented stimulus cannot fully exercise transaction-dependent behavior.

  • Overall results: Claude with AgentDV achieves an 80.9% average pass rate across nine DUTs, compared with 58.7% for Llama and 60.6% for Qwen.Claude reaches at least 90% pass rate on six DUTs and 100% on four; the one-shot Claude baseline has a 0.0% pass rate.
  • Overall results: On challenge DUTs, Claude achieves 76.9% line and 95.7% branch coverage, versus 70.2%/87.1% for Llama and 69.6%/83.0% for Qwen.AgentDV’s results depend on both the selected LLM and the DUT structure.
  • DUT complexity: AgentDV performs best on compact control-oriented DUTs: bastion_gpio reaches 100% line and branch coverage, while warden_timer reaches 97.7% line and 100% branch coverage with Claude.Challenge designs expose most control behavior through small CSR interfaces, making uncovered behavior more reachable by the feedback loop.
  • DUT complexity: On public OpenTitan IPs, Claude reaches 100% pass rate on opentitan_uart and 91% on opentitan_rv_timer, but only 55% on opentitan_aes and opentitan_hmac.Larger dependency trees and deeper internal structures make public cryptographic IPs more difficult; the public timer reaches 100% branch coverage for all three LLMs, while AES and HMAC are lower.
  • Limitations: Aegis AES remains near 37% line coverage for all LLMs despite branch coverage above 93%, because full datapath exercise requires transaction sequences beyond isolated CSR accesses.AES and HMAC coverage saturates under CSR-oriented generation; full closure requires protocol-aware stimulus, transaction-level sequences, and datapath-level checking.

D. Ablation Study

The ablation shows that closed-loop coverage feedback improves beyond initial test generation, with gains varying by DUT and often diminishing as coverage saturates. Its benefits are strongest for reachable register-level behavior, while deeper datapath logic remains a limitation.

  • Measurement setup: Figures 4 and 5 report iteration-wise contributions to line and branch coverage, separating pre-feedback coverage from incremental gains through closed-loop iterations.Each stacked bar begins with coverage before feedback and continues with gains from subsequent iterations.
  • Iterate-stage ablation: Coverage feedback provides measurable improvement beyond the initial test-generation step, with the largest gain often occurring in the first iteration and smaller gains thereafter.Later iterations may encounter harder-to-reach uncovered logic or coverage saturation.
  • DUT-dependent impact: Iterate-stage gains vary across DUTs: bastion_gpio and warden_timer reach high coverage early, whereas nexus_uart, aegis_aes, and sentinel_hmac show clearer incremental benefits.The Iterate stage is most effective when remaining uncovered behavior is reachable through additional register-level scenarios.
  • Scope and limitation: AgentDV improves CSR-access, register-decode, and control/status coverage, but deeper datapath logic often requires complete protocol transactions and stronger expected-behavior modeling.Consequently, later iterations may saturate even when uncovered logic remains.

VI. CONCLUSION

AgentDV is a closed-loop framework for LLM-assisted RTL verification environment generation that combines automated test construction, simulation, and coverage-guided feedback. Simulation results indicate that it produces runnable and coverage-effective verification suites across compact and large RTL designs.

  • Framework: AgentDV combines LLM-based test construction with Verilator, cocotb, pyUVM, and coverage-guided feedback.This tool-grounded combination targets executable and interface-aware verification suites.
  • Evaluation: Simulation results show that AgentDV generates runnable and coverage-effective verification suites across compact and large RTL designs.
  • Evaluation: With Claude Sonnet 4.6, AgentDV reaches a 100 % pass rate on four DUTs.
Loading 2608.27148v1…