PaperlayerView exact revision on arXiv

Source-linked AI summary

VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool

Chia-Tung Ho, Haoxing Ren, Brucek Khailany

arXiv:2408.08927v2cs.AIcs.CL

TL;DR

Verilog design is time-consuming and bug-prone, and general-purpose agent frameworks do not directly meet hardware-design needs. VerilogCoder combines multi-agent planning and code implementation with TCRG retrieval and AST-based waveform tracing; it reports a 94.2% pass rate and 33.9% higher pass rate than state-of-the-art methods on VerilogEval-Human v2.

  • Problem

    Verilog design is time-consuming and prone to bugs, while hardware tasks require domain knowledge and circuit-specific tools that general-purpose agent frameworks lack.

    Source: Introduction · p. 1, Multi-AI Agents · p. 2

  • Method

    VerilogCoder coordinates multiple AI agents for code generation and error correction, using TCRG-based task planning, collaborative Verilog tools, and AST-based waveform tracing.

    Source: Introduction · p. 1, Functional Debug with Waveform · p. 3, Task Planning · p. 4

  • Results

    94.2% pass rate including syntax and functional correctness, and 33.9% higher pass rate than state-of-the-art methods on VerilogEval-Human v2.

    Source: Introduction · p. 1

  • Takeaways & Limitations

    The TCRG planner and AST-based waveform tracing tool each improve pass rate, and their combination achieves a 33.9% higher pass rate than the state-of-the-art method.

    Source: Conclusion and Future Work · p. 7

Abstract

from arXiv · show

Due to the growing complexity of modern Integrated Circuits (ICs), automating hardware design can prevent a significant amount of human error from the engineering process and result in less errors. Verilog is a popular hardware description language for designing and modeling digital systems; thus, Verilog generation is one of the emerging areas of research to facilitate the design process. In this work, we propose VerilogCoder, a system of multiple Artificial Intelligence (AI) agents for Verilog code generation, to autonomously write Verilog code and fix syntax and functional errors using collaborative Verilog tools (i.e., syntax checker, simulator, and waveform tracer). Firstly, we propose a task planner that utilizes a novel Task and Circuit Relation Graph retrieval method to construct a holistic plan based on module descriptions. To debug and fix functional errors, we develop a novel and efficient abstract syntax tree (AST)-based waveform tracing tool, which is integrated within the autonomous Verilog completion flow. The proposed methodology successfully generates 94.2% syntactically and functionally correct Verilog code, surpassing the state-of-the-art methods by 33.9% on the VerilogEval-Human v2 benchmark.

Introduction

Verilog design is time-consuming and bug-prone, while prior generation approaches lacked reliable functional correction. VerilogCoder combines multi-agent code generation, task planning, and AST-based waveform tracing, reporting a 94.2% pass rate and a 33.9% advantage over state-of-the-art methods on VerilogEval-Human v2.

  • Verilog and VHDL design is time-consuming and prone to bugs, requiring repeated functional debugging and motivating lower design costs and effort.Source: Introduction · p. 1
  • Prior Verilog-generation methods lacked mechanisms for correcting functional errors, and simulator-feedback agents failed to improve functional success rates.Source: Introduction · p. 1
  • VerilogCoder uses multiple AI agents and collaborative Verilog tools to write code and fix syntax and functional errors.Source: Introduction · p. 1
  • 94.2% pass rate, including syntax and functional correctness, and 33.9% higher pass rate than state-of-the-art methods on VerilogEval-Human v2 are reported for VerilogCoder.Source: Introduction · p. 1
  • Its contributions include a TCRG planner for sub-tasks and circuit information, plus an AST-based waveform tracing tool to assist functional correction.Source: Introduction · p. 1

Background

The background describes LLM-agent research and multi-agent frameworks, then identifies why general-purpose frameworks do not directly meet hardware-design needs.

Background

  • LLM-based autonomous agents are studied across academic and industrial settings, with recent interest driven by LLM capabilities.Source: Background · p. 2

AI Agent

  • Agent systems combine planning, memory, and action; software-engineering agents use planning and external tools on challenging benchmarks.Source: AI Agent · p. 2

Multi-AI Agents

  • Multi-agent frameworks support coordinated agents but mostly target general tasks; hardware design additionally requires domain knowledge and circuit-specific tools.Source: Multi-AI Agents · p. 2

Motivation and Preliminary Study

The paper motivates VerilogCoder by the difficulty of planning hardware tasks and debugging functional errors, then proposes graph-enriched plans and AST-based waveform tracing.

Motivation and Preliminary Study

  • Hardware-module implementation requires decomposition into sub-tasks, code generation, and iterative simulation, waveform debugging, and updates until outputs match expected behavior.Source: Motivation and Preliminary Study · p. 3

Planning

  • Traditional LLM plans can omit relevant signals and transitions, as in an FSM example where missing next-state transitions led to incorrect implementation; TCRG plans include such information.Source: Planning · p. 3

Functional Debug with Waveform

  • The proposed waveform-debugging tool follows a human designer’s signal backtracing process by tracing target-signal RVALUEs in the AST to help agents fix functional errors.Source: Functional Debug with Waveform · p. 3

VerilogCoder

VerilogCoder coordinates task planning and code implementation through multi-AI agents, TCRG retrieval, Verilog tools, and ReAct.

  • VerilogCoder consists of task planning and Verilog code implementation, with agents operating cohesively through TCRG retrieval and Verilog tools using ReAct.Source: VerilogCoder · p. 4
  • The agents use developed TCRG retrieval and Verilog tools as part of their coordinated operation.Source: VerilogCoder · p. 4
  • ReAct is the stated technique for coordinating the agents’ operation.Source: VerilogCoder · p. 4

Flow Overview

VerilogCoder plans from a module description, then assigns dependent subtasks to multi-LLM agents that write code and correct functionality using Verilog tools.

Flow Overview

  • The TCRG-based planner creates task plans, after which a task dependency graph orders subtasks assigned to agents for Verilog coding and functional correction.Agents use a syntax checker, simulator, and AST-based waveform tracing tool; the testbench is used only to check functional correctness.Source: Flow Overview · p. 4

Task Planning

The Task and Circuit Relation Graph (TCRG) planner enriches task plans with relevant circuit details, organizing tasks, signals, transitions, and examples for retrieval by subtask.

Task Planning

  • The TCRG-based planner creates high-quality plans that pair high-level objectives with relevant signal descriptions, transitions, and examples for each subtask.This information is intended to make the plan useful beyond a list of task goals.Source: Task Planning · p. 4

High-level planner agent

  • A planner decomposes the module specification into subtasks, and a verification assistant iterates until the plan is consistent with that specification.The assistant suggests revisions when it finds inconsistencies.Source: High-level planner agent · p. 4

Circuit signal, transition, example extraction agent

  • A Verilog Engineer agent extracts signals, transitions, and examples from the module description into JSON for subsequent TCRG construction.The extracted information is represented as nodes in the graph.Source: Circuit signal, transition, example… · p. 4

TCRG construction

  • The TCRG links task nodes to signal nodes, then signal nodes to transition and example nodes using IMPLEMENTS, SIGNALTRANSITION, and EXAMPLES relations.Source: TCRG construction · p. 5

Task-driven circuit relation graph retrieval agent

  • For each subtask, an agent retrieves related circuit information from the TCRG; it chooses the retrieval hop count and compiles relevant results while removing irrelevant information.The tool returns signals, transitions, and examples associated with the subtask.Source: Task-driven circuit relation graph… · p. 5

Verilog Code Implementation

VerilogCoder separates code writing from verification and debugging, using dependent subtasks and collaborative tools to iteratively produce code that meets syntax and functional checks.

Verilog Code Implementation

  • The task dependency graph blocks a child subtask until its parent tasks finish without errors, separating code-writing tasks from verification and debugging tasks.Source: Verilog Code Implementation · p. 5

Verilog Tools

  • The syntax checker compiles code, while the simulator reports output mismatches and their first time point and creates a waveform file for tracing.AST-WT traces mismatched signals backward through RVALUE signals to an agent-selected level and returns code references, a waveform table, and traced signals.Source: Verilog Tools · pp. 5–6

Code Agent

  • A Verilog Engineer writes code for each subtask, while a Verification Assistant checks syntax and consistency and suggests fixes until both checks pass.Source: Code Agent · p. 6

Debug Agent

  • The Debug Agent uses a simulator and AST-WT in a Thought-Action-Observation loop to modify generated code until it passes the provided testbench.Source: Debug Agent · p. 6

Experimental Results

On VerilogEval-Human v2, VerilogCoder (GPT-4 Turbo) achieves a 94.2% pass rate and exceeds recent LLM baselines; ablations and category analyses compare planner and waveform-tracing configurations.

Experimental Results

  • The evaluation runs VerilogCoder once on each of 156 VerilogEval-Human v2 problems and checks functional correctness with the provided golden testbench.Source: Experimental Results · p. 6

Main Results

  • 94.2% pass rate: VerilogCoder (GPT-4 Turbo) outperforms GPT-4 and GPT-4 Turbo by 43.6% and 33.9%, respectively, on VerilogEval-Human v2.The table compares VerilogCoder with recent non-agentic LLMs.Source: Table 1 · p. 6, Main Results · p. 6

Ablation Study

  • The taxonomy analysis groups the union of failed problems by module and query-prompt type, then compares category pass rates across four planner and AST-WT configurations.Source: Figure 5 · p. 7, Ablation Study · p. 7
  • 27.5% improvement over baseline: Planner2 with AST-WT improves pass rate, compared with 11.5% for Planner1 with AST-WT and 7.7% for Planner2 without AST-WT.Planner1 without AST-WT is the baseline, and Planner2 with AST-WT is the proposed VerilogCoder.Source: Ablation Study · p. 7, Table 2 · p. 7
  • Planner1 with AST-WT exceeds Planner2 without it by 10.5%, 39.1%, and 12.5% on Application (Descr.), Comb+Seq+FSM (Descr.), and Comb+Seq+FSM (Waveform).Source: Ablation Study · p. 7
  • Planner2 without AST-WT beats Planner1 with it by 33.3% on Comb (Kmap) and 44.5% on FSM (Trans. Table).Source: Ablation Study · p. 7
  • The authors attribute the latter category results to the planner capturing specified input-output mappings or state transitions without missing information, supporting step-by-step subtask solving.Source: Ablation Study · p. 7

Conclusion and Future Work

VerilogCoder combines a TCRG-based task planner with AST-WT to generate Verilog and fix syntax and functional errors, achieving a higher pass-rate than the state-of-the-art method. Future directions include improving Verilog code quality and agent learning and memory.

Loading 2408.08927v2…