PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatEDA: A Large Language Model Powered Autonomous Agent for EDA

Zhuolun He, Haoyuan Wu, Xinyun Zhang, Xufeng Yao, Su Zheng, Haisheng Zheng, Bei Yu

arXiv:2308.10204v4cs.ARcs.AI

TL;DR

Complex EDA toolchains require tedious, error-prone scripting, motivating better interoperability for circuit designers. The paper introduces ChatEDA, which uses fine-tuned AutoMage controllers to decompose requirements, generate scripts, and execute EDA tasks. On ChatEDA-Bench, AutoMage2 earned Grade A on 82% of test cases versus 62% for GPT-4, while the framework remains constrained by tool-specific documentation and substantial model complexity.

  • Problem

    Complex EDA tools expose intricate procedures and APIs, while conventional scripting is tedious and prone to errors; generic LLMs also lack bespoke fine-tuning for EDA toolchain automation.

  • Method

    ChatEDA uses fine-tuned AutoMage controllers to interpret natural-language requirements, decompose tasks, generate Python scripts, and execute them through EDA tools.

  • Results

    82% Grade A on ChatEDA-Bench test cases was achieved by AutoMage2, compared with 62% for GPT-4; the AutoMage series outperformed other notable LLMs.

  • Takeaways & Limitations

    ChatEDA provides a conversational interface for designers to interact with and automate the RTL-to-GDSII design flow.

  • Takeaways & Limitations

    ChatEDA lacks universal applicability across diverse EDA APIs and requires specialized documentation, while AutoMage models contain a substantial number of parameters.

Abstract

from arXiv · show

The integration of a complex set of Electronic Design Automation (EDA) tools to enhance interoperability is a critical concern for circuit designers. Recent advancements in large language models (LLMs) have showcased their exceptional capabilities in natural language processing and comprehension, offering a novel approach to interfacing with EDA tools. This research paper introduces ChatEDA, an autonomous agent for EDA empowered by an LLM, AutoMage, complemented by EDA tools serving as executors. ChatEDA streamlines the design flow from the Register-Transfer Level (RTL) to the Graphic Data System Version II (GDSII) by effectively managing task decomposition, script generation, and task execution. Through comprehensive experimental evaluations, ChatEDA has demonstrated its proficiency in handling diverse requirements, and our fine-tuned AutoMage model has exhibited superior performance compared to GPT-4 and other similar LLMs.

I. INTRODUCTION

The paper presents ChatEDA as an LLM-powered framework for interfacing with complex EDA tools, using fine-tuned AutoMage models to automate RTL-to-GDSII workflows. It combines task decomposition, script generation, and execution while targeting improved reliability and specialized EDA performance.

  • Motivation: Conventional TCL-based interaction with complex EDA tools is tedious and prone to errors.RTL-to-GDSII platforms expose numerous procedures and adjustable parameters, requiring iterative scripting for circuit-design objectives.
  • Workflow: ChatEDA organizes automation into task decomposition, script generation, and task execution.AutoMage interprets user requirements, decomposes them into subtasks, generates Python scripts using external-tool specifications, and executes the scripts through EDA tools.
  • Proposed framework: ChatEDA is an LLM-powered EDA interfacing framework that generates code for manipulating EDA tools from natural-language instructions.The system is designed to provide autonomous RTL-to-GDSII capabilities without requiring users to write code.
  • Proposed framework: AutoMage and AutoMage2 are fine-tuned LLM controllers specialized for using EDA tools.They are purpose-built to enhance ChatEDA, addressing capabilities that are barely learned in open-source LLMs.
  • Evaluation: ChatEDA-Bench evaluations report that the AutoMage series surpasses GPT-4 and other well-known LLMs across various tasks.The paper positions comprehensive benchmarking as evidence for the superior performance of its specialized models.
  • Model and training: AutoMage is fine-tuned from Llama2 using LoRA, which preserves pretrained weights while training low-rank matrices.This approach substantially reduces trainable parameters and introduces no additional latency during regular inference after weights are merged.

C. In-Context Learning

In-context learning conditions LLM predictions on a prompt containing examples, allowing models to infer shared concepts without explicit fine-tuning. ChatEDA uses this capability for self-instruction and instruction-dataset collection, alongside quantized-model training components.

  • C. In-Context Learning: In-context learning conditions downstream predictions on a prompt containing a limited number of input-output examples without explicit fine-tuning.The model infers shared concepts within the prompt to produce predictions.
  • C. In-Context Learning: The posterior predictive formulation characterizes completions given an in-context prompt by marginalizing latent concepts.It represents the conditional distribution p(x_o|x_ic) under the pretrained distribution.
  • C. In-Context Learning: ChatEDA employs in-context learning for self-instruction and instruction-dataset collection before fine-tuning Llama2.This connects the general ICL mechanism to AutoMage’s training process.
  • D. Block-wise k-bit Quantization: Block-wise k-bit quantization divides tensors into contiguous blocks that are independently quantized with separate quantization constants.For an input tensor X ∈ R^{b×h}, the number of blocks is n = (b × h)/B.
  • D. Block-wise k-bit Quantization: Quantized pretrained LLM weights are stored in 4-bit format and dequantized to 16-bit for forward and backward computations.This supports efficient fine-tuning while retaining higher-precision computation during training.
  • A. Task Decomposition: ChatEDA applies AutoMage to interpret complex natural-language RTL-to-GDSII requests and decompose them into structured subtasks for EDA tools.A representative request sequences synthesis, floorplanning, placement, CTS, and post-routing power evaluation.

B. Script Generation

ChatEDA generates executable Python scripts by combining user requirements, decomposed subtasks, and EDA API specifications. AutoMage is instruction-tuned on constructed examples so it can map EDA requirements to tool-manipulating scripts that are executed through an EDA wrapper.

  • B. Script Generation: AutoMage receives API specifications, user requirements, and decomposed subtasks, then generates a Python script ready for direct execution.The script is intended to invoke corresponding EDA-tool APIs for the defined subtasks.
  • C. Task Execution: ChatEDA executes generated scripts with Python, using environment setup and subprocesses to run OpenROAD scripts through a wrapper.The wrapper invokes relevant Tcl scripts or commands implementing tool functions.
  • SELF-INSTRUCTION PROMPT: Self-instruction uses diverse in-context prompts to query GPT-3.5/4 for EDA-tool instructions, with restrictions guiding requirements, analyses, and executable Python scripts.The instance format includes requirement, decomposition, and script components.
  • SELF-INSTRUCTION PROMPT: The framework’s intended flow covers autonomous digital-layout generation across die sizes, process nodes, and foundry options.Examples include tuning synthesis clock period, floorplanning utilization, and placement density for power performance.
  • B. Instruction Tuning for AutoMage.: Instruction tuning trains AutoMage on approximately 1500 instances pairing requirements with detailed decompositions and scripts.The training uses an Alpaca-style prompt to teach EDA-tool utilization.

C. Efficient Fine-tuning of Quantized LLMs

QLoRA enables efficient fine-tuning of quantized LLMs by combining low-bit quantization, quantized quantization constants, and parameter-efficient factorized projections. The method uses dequantization during computation while training only LoRA parameters in BF16.

  • QLoRA combines NF4 quantization, double quantization, and paged optimizers to reduce fine-tuning memory demands.NF4 and double quantization support low-bit storage, while paged optimizers mitigate memory spikes during gradient checkpointing.
  • NF4 Quantization: NF4 quantization distributes values evenly across quantization bins through quantile-based quantization.The paper describes NormalFloat as an information-theoretically optimal extension of quantile quantization.
  • Double Quantization: Double quantization quantizes first-stage quantization constants again, producing lower-precision constants for additional memory savings.The second quantization uses 8-bit floats with a block size of 256 and centers positive constants around zero for symmetric quantization.
  • QLoRA: QLoRA enhances a transformer linear projection with an additional factorized projection while preserving the original computation structure.For a projection y = Wx, the method applies the QLoRA computation described by the paper’s equation.
  • Double Quantization: The matrix W is quantized to 4-bit precision, while its quantization constants use FP8 with a block size of 256.The paper specifies a block size of 64 for W and 256 for the second-level constants.
  • QLoRA: QLoRA stores weights in NF4 and computes with BF16, updating gradients only for LoRA parameters.The forward and backward passes dequantize the storage type into the computational type.

D. Auto-Regressive Decoding

AutoMage uses beam search during autoregressive decoding, while AutoMage2 addresses limitations associated with limited fine-tuning data and the underlying Llama2 model. Its upgrade emphasizes richer data and reasoning-oriented training techniques.

  • D. Auto-Regressive Decoding: Beam search is used in AutoMage autoregressive decoding with a beam width of 4.The selected width lets the model explore multiple token sequences before choosing an output.
  • AutoMage Limitations: AutoMage may overfit because it was fine-tuned on approximately 1500 EDA tool instructions.The paper identifies the dataset size as a limitation of the original model.
  • AutoMage Limitations: AutoMage also inherits deficiencies in coding and logical reasoning from its Llama2 base model.The paper links AutoMage’s capabilities largely to pretraining of the underlying LLMs.
  • AutoMage2: AutoMage2 is designed as a more stable and capable ChatEDA controller, especially for task decomposition and script-generation coding.The upgrade retains AutoMage’s architecture, fine-tuning process, and autoregressive decoding while adding targeted improvements.
  • AutoMage2: AutoMage2 augments training with an enriched corpus, instruction tuning with explanations, and chain-of-thought prompting.These techniques are intended to improve logical reasoning, task decomposition, and script generation.

B. Instruction Tuning with Explanation.

The instruction-tuning approach teaches AutoMage2 to generate executable EDA scripts while exposing it to explanations and stepwise reasoning. The prompts explicitly connect user requirements to available ChatEDA tools and functions.

  • B. Instruction Tuning with Explanation.: The training corpus includes teacher responses that explain reasoning beyond ordinary prompt-response pairs.These explanations provide additional learning signals for AutoMage2 instruction tuning.
  • B. Instruction Tuning with Explanation.: AutoMage2 follows an Orca-style instruction format that asks the model to think step-by-step and justify its steps.The format is used to encourage imitation of the teacher’s reasoning process.
  • INSTRUCTION FINE-TUNING PROMPT: The fine-tuning prompt assigns the assistant to generate a Python script using provided tools and functions.The prompt frames script generation as the response to a user-specified task.
  • INSTRUCTION FINE-TUNING PROMPT: The prompt describes ChatEDA as a tool for automating RTL-to-GDSII execution through functions and parameters.It also identifies tune as a function for parameter tuning.
  • C. Chain of Thoughts.: Chain-of-thought prompting elicits intermediate reasoning steps that can support more accurate answers.The paper applies zero-shot chain-of-thought prompting during inference for AutoMage models and other LLMs.
  • ZERO-SHOT COT PROMPT: The zero-shot prompt asks the model to describe the task, analyze tool use step by step, and then generate the Python script.This prompt combines task interpretation, procedural analysis, and executable output.
  • ZERO-SHOT COT PROMPT: The final zero-shot instruction orders AutoMage to produce the script after explaining what the task requests and how to complete it.The sequence explicitly places script generation after the reasoning steps.

A. Setup

The setup trains AutoMage2 with a specified optimization configuration and uses natural-language requirements to produce executable scripts. Evaluation compares ChatEDA controllers based on several LLM baselines using a simplified OpenROAD API.

  • Training Setup: AutoMage2 is fine-tuned with a constant learning rate schedule and a 0.03 warm-up ratio.The configuration uses paged AdamW 8-bit optimization, a learning rate of 1 × 10−4, no weight decay, batch size 128, and sequence length 4096.
  • Training Setup: The model is fine-tuned for 1 epoch on 16×A100 GPUs with 80G memory each.This hardware and duration define the reported fine-tuning setup.
  • Inference Setup: During inference, AutoMage2 receives natural-language requirements and outputs an executable script.Requirements may specify a simple EDA operation or request a broader design-flow script.
  • Evaluation Setup: Evaluation compares Claude2, GPT-3.5, and GPT-4 as baseline LLM controllers for ChatEDA.The target API is a simplified Python wrapper of OpenROAD.
  • Reproducibility: The authors provide ChatEDA-bench, example EDA instruction data, and an OpenROAD API implementation for reproduction.These materials are released in the project’s open-source repository.

B. ChatEDA-Bench

ChatEDA-Bench evaluates LLMs on simple, complex, and parameterized EDA-flow interactions, covering both basic API sequencing and more demanding reasoning over parameters.

  • 50 tasks span simple flow calls, complex flow calls, and parameter flow calls.The categories represent 30%, 30%, and 40% of the benchmark, respectively.
  • Simple Flow Calls: Simple flow calls test complete-flow execution and correct sequencing of API interfaces.
  • Complex Flow Calls: Complex flow calls test logical reasoning over parameters and understanding of individual API arguments.
  • Parameter Tuner Calls: Parameter tuner calls require solutions that thoroughly test the LLM’s logic and use of EDA tools.

C. Evaluation of LLMs

The evaluation grades generated scripts by executability and user- requirement satisfaction, with blinded judges assigning Grades A through C. AutoMage2 achieved the strongest benchmark performance.

  • The evaluation combines EDA-interface execution testing with blinded manual assessment of whether responses meet user requirements.Multiple judges use a three-tier grading scheme: Grade A, B, or C.
  • 82% of test cases received Grade A from AutoMage2, compared with 62% for GPT-4.Claude2 achieved 46% Grade A and GPT-3.5 achieved 28%.
  • AutoMage2 achieved the best performance among the evaluated LLMs in task planning and script generation.

D. Case Studies

Case studies examine conversational EDA-tool use, emphasizing parameter search, tuning, and script generation. AutoMage2 is reported to handle these tasks more effectively than GPT-4 and to correct errors.

  • Five conversational tasks illustrate user requirements, task decomposition, and scripts generated by AutoMage2, with GPT-4 included for comparison.The study is qualitative and uses selected representative samples rather than controlling all possible LLM responses.
  • Task 1: Parameter Grid Search: AutoMage2 demonstrated stronger parameter-combination optimization and API-argument comprehension than GPT-4 in a grid-search task.The task searched floorplan, CTS, and placement parameters for a balance of chip area, power consumption, and performance.
  • Task 1: Parameter Grid Search: The grid-search flow initializes the tool, runs synthesis, iterates through floorplan, placement, and CTS parameters, then collects area, power, and performance metrics.
  • Task 2: Parameter Tuning Process: In parameter tuning, both models generated complex search spaces, while GPT-4 omitted an intermediate synthesis step that AutoMage2 included.
  • Both AutoMage2 and GPT-4 successfully completed the provided test case.
  • AutoMage2 and GPT-4 accurately identified and fixed errors in the evaluated script, demonstrating AutoMage2’s self-correction skills and EDA-tool understanding.

VII. DISCUSSION

The discussion identifies limitations in ChatEDA’s applicability, decoding efficiency, and benchmark evaluation support. It proposes broader documentation generalization, faster decoding, and stronger dialogue capabilities as future directions.

  • ChatEDA requires specialized API documentation and bespoke integration for different EDA tools, limiting scalability across platforms.
  • AutoMage’s substantial parameter count slows a single decoding step.
  • ChatEDA-Bench lacks basic evaluation flows and semi-automated scoring tools for straightforward comparison of external solutions.
  • Future work targets decoding-speed optimization, zero-shot or few-shot handling of unfamiliar EDA documentation, and multi-turn error correction.

VIII. CONCLUSION

ChatEDA integrates a fine-tuned AutoMage with EDA tools to provide a conversational interface for interacting with the design flow. It handles diverse user requirements and outperforms other LLM models such as GPT-4, while motivating future EDA tool evolution.

  • ChatEDA integrates a fine-tuned AutoMage that orchestrates the EDA design flow through task decomposition, script generation, and task execution.
  • ChatEDA enables designers to interact conversationally with the EDA design flow.
  • ChatEDA handles diverse user requirements well and outperforms other LLM models such as GPT-4.
  • The work aims to inspire next-generation EDA tool evolution.
Loading 2308.10204v4…