PaperlayerView exact revision on arXiv

Source-linked AI summary

MAGE: A Multi-Agent Engine for Automated RTL Code Generation

Yujie Zhao, Hejia Zhang, Hanxian Huang, Zhongming Yu, Jishen Zhao

arXiv:2412.07822v1cs.ARcs.LG

TL;DR

RTL code generation remains challenging because single-agent systems must navigate task and language shifts while producing syntactically and functionally correct designs. MAGE uses specialized agents, high-temperature candidate sampling, and state-checkpoint debugging. It reports 95.7% Pass@1 on VerilogEval-V2, surpassing all baselines.

  • Problem

    Single-agent RTL generation faces substantial limitations in switching among programming languages and handling generation, verification, and modification tasks.

    Source: §I · p. 1

  • Method

    MAGE assigns RTL generation, testbench generation, judging, and debugging to specialized agents, using candidate sampling and Verilog-state checkpoint feedback.

    Source: §I · p. 1

  • Results

    95.7% Pass@1 on VerilogEval-V2, surpassing all baselines, with a +23.3% relative improvement over the highest-performing vanilla LLMs.

    Source: B. Key Results · p. 5

  • Takeaways & Limitations

    MAGE offers a robust methodology for AI-driven RTL design, with sampling and state-checkpoint debugging aimed at exploring candidates and providing precise fixes.

    Source: §V · p. 7

Abstract

from arXiv · show

The automatic generation of RTL code (e.g., Verilog) through natural language instructions has emerged as a promising direction with the advancement of large language models (LLMs). However, producing RTL code that is both syntactically and functionally correct remains a significant challenge. Existing single-LLM-agent approaches face substantial limitations because they must navigate between various programming languages and handle intricate generation, verification, and modification tasks. To address these challenges, this paper introduces MAGE, the first open-source multi-agent AI system designed for robust and accurate Verilog RTL code generation. We propose a novel high-temperature RTL candidate sampling and debugging system that effectively explores the space of code candidates and significantly improves the quality of the candidates. Furthermore, we design a novel Verilog-state checkpoint checking mechanism that enables early detection of functional errors and delivers precise feedback for targeted fixes, significantly enhancing the functional correctness of the generated RTL code. MAGE achieves a 95.7% rate of syntactic and functional correctness code generation on VerilogEval-Human 2 benchmark, surpassing the state-of-the-art Claude-3.5-sonnet by 23.3 %, demonstrating a robust and reliable approach for AI-driven RTL design workflows.

I. INTRODUCTION

RTL generation requires repeated design and debugging work, and single-agent systems struggle with the context shifts across languages and tasks. MAGE divides the workflow among specialized agents and combines candidate sampling with state-checkpoint feedback.

I. INTRODUCTION

  • HDL design is time-consuming and error-prone, while single-agent approaches struggle to switch among tasks, languages, and domains.A single Claude-3.5-sonnet agent is reported at a 75.0% functionality pass rate even on simple design tasks.Source: §I · p. 1
  • MAGE assigns RTL generation, testbench generation, judging, and debugging to four specialized agent types in a collaborative framework.The agents handle distinct aspects of the RTL development pipeline.Source: §I · p. 1
  • High-temperature candidate sampling and simulation-based scoring explore diverse RTL outputs and select promising candidates for optimization.The paper’s stated insight is that higher temperatures increase output diversity and the likelihood of including high-quality candidates.Source: §I · p. 1
  • Verilog-state checkpoints compare state values with expected outputs at each clock edge, enabling early functional-error detection and targeted fixes.This contrasts with feedback based only on a final output mismatch.Source: §I · pp. 1–2
  • 95.7% syntactic and functional correctness on VerilogEval-Human v2; MAGE surpasses existing methods.The paper presents this as a result of its LLM-based multi-agent system.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

RTL design involves iterative implementation, verification, and refinement, and single agents must manage complex contexts across these tasks. The paper identifies opportunities in task separation, open-source multi-agent tooling, and efficient candidate filtering.

A. Characteristics and challenges of LLM-based RTL design

  • Single-agent RTL methods must handle synthesizable code, verification code, and repeated context shifts, leading to sub-optimal results.Traditional RTL work iterates among Verilog implementation, testbench verification, simulation, waveform reasoning, and refinement.Source: A. Characteristics and challenges of… · p. 2
  • Existing multi-agent systems are described as closed-source or reliant on proprietary components, while high-temperature sampling can add noise and errors.The paper argues that single-agent mechanisms limit independent sampling and feedback for RTL.Source: A. Characteristics and challenges of… · p. 2

B. Opportunity on Effective Code Generation

  • Generating testbenches and RTL together in one conversation may reduce effectiveness and make test cases biased by the generated code.The passage cites prior work reporting this risk.Source: B. Opportunity on Effective Code… · p. 2
  • The paper identifies a need for an open-source multi-agent RTL system to improve extensibility and transparency.Its review found existing multi-agent systems closed-source and dependent on third-party proprietary tools.Source: B. Opportunity on Effective Code… · p. 2
  • Early selection of top-scoring RTL candidates is proposed to reduce exploration cost by filtering suboptimal results.The approach aims to retain the exploration benefits of high-temperature sampling while mitigating randomness.Source: B. Opportunity on Effective Code… · p. 3

C. Opportunity on Effective Code Debugging

Pass-rate-only feedback provides limited information for debugging RTL, while closed-source analysis tools and graphical outputs constrain flexibility. The paper motivates textual waveform-like testbench logs that LLMs can use directly.

C. Opportunity on Effective Code Debugging

  • Golden-testbench pass rates provide little detail about timing, signal interactions, variable values, or mismatch locations needed to refine RTL.The paper identifies these details as crucial to high-quality RTL design.Source: C. Opportunity on Effective Code… · p. 3
  • Closed-source AST analysis and graphical tool outputs constrain flexibility and complicate tasks for LLMs.The passage specifically describes VerilogCoder’s AST analysis as closed-source.Source: C. Opportunity on Effective Code… · p. 3
  • The paper calls for optimized testbenches that emit textual waveform-like logs, providing an open-source protocol directly adaptable by LLMs.The proposed logs are intended to improve RTL debugging effectiveness and analysis quality.Source: C. Opportunity on Effective Code… · p. 3

III. MAGE DESIGN

MAGE integrates an RTL-specific communication protocol and LLM-adapted tools in an orchestrated process. Its workflow separates testbench creation, RTL generation, judging, candidate sampling, and debugging.

III. MAGE DESIGN

  • MAGE combines an RTL-specific context communication protocol with LLM-adapted tools in an orchestrated multi-agent process.The design is presented as a multi-agent engine built specifically for RTL.Source: §III · p. 3

A. Multi-Agent System Overview

  • Four agent types create textual-waveform testbenches, generate syntax-checked RTL, simulate and score candidates, and iteratively refine failing code.The Judge Agent can decide whether to debug RTL or regenerate a testbench.Source: A. Multi-Agent System Overview · p. 3
  • The workflow generates testbenches and initial RTL, checks testbench validity when RTL fails, then samples candidates and debugs remaining functional errors.Agents perform at most five iterations to fix syntax errors.Source: A. Multi-Agent System Overview · p. 4

B. High-Temperature RTL Sampling and Scoring

With sufficiently extensive sampling, high-temperature RTL candidates tend to have lower mismatch counts; MAGE scores candidates and iteratively debugs selected ones.

B. High-Temperature RTL Sampling and Scoring

  • For each issue, MAGE samples c RTL candidates conditioned on temperature, the system prompt, the natural-language specification, and its testbench.Source: B. High-Temperature RTL Sampling and… · p. 4
  • High-temperature sampling (T=0.85, n=20) typically yields lower mean mismatch counts than low-temperature sampling (T=0, n=1) across most benchmark problems.The figure covers VerilogEval-v1-Human and VerilogEval-v2; the reported comparison concerns problems not already passing before Step 4 or with zero mean mismatches in both settings.Source: Figure 2 · p. 4, B. High-Temperature RTL Sampling and… · p. 4
  • Candidates are ranked by a normalized mismatch-based score using mismatch count and total checks, and the top K are selected.The score is s(r) = 1 − m(r)/tc(r), where m(r) is mismatch count and tc(r) is the total number of checks.Source: B. High-Temperature RTL Sampling and… · p. 4
  • A debug agent runs trials on selected candidates and updates the candidate set iteratively.The process repeats until its stated stopping condition.Source: B. High-Temperature RTL Sampling and… · pp. 4–5

C. RTL Debugging with State Checkpoint Mechanism

MAGE’s state-checkpoint debugging locates the earliest output mismatch, gives the debugging agent a textual waveform window, and reruns simulation to check its RTL fixes.

C. RTL Debugging with State Checkpoint Mechanism

A. Experimental Setup

Experiments evaluate Claude 3.5 Sonnet on two VerilogEval benchmarks using low- and high-temperature settings, with Pass@1 as the metric; the high-temperature setting performs better and is used afterward.

  • Experiments measure Pass@1 for Claude 3.5 Sonnet on VerilogEval-v1-Human and VerilogEval-v2 under low temperature (T=0, top p=0.01, n=1) and high temperature (T=0.85, top p=0.95).Source: A. Experimental Setup · p. 5
  • Pass@1 estimates the percentage of problems the system would solve correctly when run once per problem, accounting for multiple runs.Source: A. Experimental Setup · p. 5
  • The high-temperature setting achieves higher Pass@1 than the low-temperature setting and is adopted for subsequent experiments.Source: A. Experimental Setup · p. 5

B. Key Results

MAGE achieves the highest reported Pass@1 on both benchmarks, outperforming vanilla models and coding systems, including open-source solutions.

  • 94.8% Pass@1 on VerilogEval-Human and 95.7% on VerilogEval-V2 make MAGE the top-performing system on both benchmarks.Source: B. Key Results · p. 5
  • +19.8% relative improvement on VerilogEval-Human and +23.3% on VerilogEval-V2 versus the highest-performing vanilla LLMs, including Claude 3.5 Sonnet.Source: B. Key Results · p. 5
  • MAGE outperforms both closed-source systems and accessible alternatives, including vanilla models and open-source coding systems such as OriGen.OriGen scores 54.4% Pass@1 on VerilogEval-Human.Source: B. Key Results · p. 5

C. Ablation Study

The ablations examine task allocation, state checkpoints, and sampling with iterative debugging; reported evidence associates each with improved RTL generation or debugging outcomes.

  • The multi-agent setup reaches a 93.6% pass rate, above the single-agent setup (83.9%) and vanilla LLM (72.4%).Source: C. Ablation Study · p. 5, Table III · p. 6
  • State checkpoints let the Debug Agent pinpoint output mismatches and missing logic terms, unlike log-only debugging, which may only approximate the faulty location.Source: C. Ablation Study · p. 6, Figure 3 · p. 6
  • Sampling concentrates RTL scores near 1 rather than spreading them nearly uniformly across [0, 1], while debugging raises mean score from 0.669 to 0.890.The score distributions exclude problems fixed before the debug stage.Source: C. Ablation Study · p. 6, Figure 4 · p. 6

V. CONCLUSION

MAGE is an open-source multi-agent system for automated Verilog RTL generation. Its high-temperature sampling and scoring explore candidate designs, while state-checkpoint debugging provides precise feedback for further code refinement.

V. CONCLUSION

  • MAGE is an open-source multi-agent system designed for automated and accurate Verilog RTL code generation.It is presented as a step toward more robust AI-driven RTL design workflows.Source: §V · pp. 6–7
  • High-temperature RTL sampling and scoring explores potentially correct candidates and leads to higher pass rates than prior studies.The process is paired with RTL debugging using a state-checkpoint mechanism.Source: §V · p. 7
  • State-checkpoint debugging further optimizes generated code by providing more precise feedback.Source: §V · p. 7
Loading 2412.07822v1…