Source-linked AI summary
ChipNeMo: Domain-Adapted LLMs for Chip Design
Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Ankit Jindal, Brucek Khailany, George Kokai, Kishor Kunal, Xiaowei Li, Charley Lind, Hao Liu, Stuart Oberman, Sujeet Omar, Ghasem Pasandi, Sreedhar Pratty, Jonathan Raiman, Ambar Sarkar, Zhengjiang Shao, Hanfei Sun, Pratik P Suthar, Varun Tej, Walker Turner, Kaizhe Xu, Haoxing Ren
TL;DR
ChipNeMo addresses how LLMs can support industrial chip-design tasks without relying directly on general-purpose models. It combines tokenizer adaptation, continued domain pretraining, alignment, and domain-adapted retrieval, and reports stronger domain-task performance, including ChipNeMo-70B surpassing GPT-4 on two use cases.
Problem
General-purpose LLMs offer broad capabilities, but chip-design tasks involving natural language and programming remain insufficiently automated, motivating cost-effective domain adaptation.
Method
The paper combines domain-adapted tokenization, domain-adaptive continued pretraining, instruction-based alignment, and retrieval-augmented generation with a fine-tuned retrieval model.
Results
ChipNeMo models achieve similar or better results than base LLaMA2 models at 1.5% additional pretraining compute, while ChipNeMo-70B surpasses GPT-4 on two use cases.
Takeaways & Limitations
Domain-specific customization can improve LLM effectiveness for specialized chip-design applications without reported degradation in generic capabilities.
Takeaways & Limitations
Retrieval remains limited for queries that do not map directly to corpus passages or require context absent from those passages.
Abstract
from arXiv · showhide
ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models. We evaluate these methods on three selected LLM applications for chip design: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Our evaluations demonstrate that domain-adaptive pretraining of language models, can lead to superior performance in domain related downstream tasks compared to their base LLaMA2 counterparts, without degradations in generic capabilities. In particular, our largest model, ChipNeMo-70B, outperforms the highly capable GPT-4 on two of our use cases, namely engineering assistant chatbot and EDA scripts generation, while exhibiting competitive performance on bug summarization and analysis. These results underscore the potential of domain-specific customization for enhancing the effectiveness of large language models in specialized applications.
1. Introduction
ChipNeMo investigates domain-adapted LLMs for three chip-design applications, combining continued pretraining, tokenizer adaptation, alignment, and retrieval augmentation. The approach targets language- and code-related engineering tasks while reporting strong application results.
- Applications: ChipNeMo targets engineering assistance, EDA script generation, and bug summarization and analysis for chip design.
- Approach: The proposed adaptation combines domain-adapted tokenization, domain-adaptive pretraining, instruction-based alignment, and a domain-adapted retrieval model.
- Findings: 6.0/7 expert evaluation marks the engineering assistant, over 70% correctness covers simple EDA scripts, and summarization and assignment identification ratings exceed 5/7.
- Findings: 30% higher retriever hit rate from domain-specific fine-tuning further improves the quality of retrieval-augmented responses.
2. ChipNeMo Domain Adaptation Methods
ChipNeMo adapts pretrained LLaMA2 models through tokenizer augmentation, domain-adaptive pretraining, alignment, and retrieval augmentation. The methods improve domain task performance, but training-loss anomalies and retrieval coverage remain practical boundaries.
- Domain-Adaptive Tokenization: Tokenizer augmentation selectively adds domain-specific tokens while preserving the pretrained tokenizer and model validity.New-token embeddings are initialized from the original tokenizer’s embeddings, reducing retraining effort.
- Domain-Adaptive Pretraining: Domain-adaptive pretraining continues autoregressive training on proprietary hardware code and natural-language data using LLaMA2-initialized foundation models.The reported regimen uses roughly 1.5% of the original pretraining compute.
- Model Alignment: Alignment follows pretraining and uses supervised fine-tuning or SteerLM with general chat data combined with approximately 1.4K domain-specific instructions.The domain alignment data is blended with larger general chat datasets, while additional in-domain epochs can cause overfitting.
- Model Alignment: DAPT applied directly to a chat-aligned LLaMA2 model significantly degrades alignment and makes the resulting model useless for downstream tasks.
- Retrieval-Augmented Generation: RAG retrieves relevant database passages to ground responses, and domain-adapted language and retrieval models improve answer quality and retrieval accuracy.The retriever is fine-tuned from e5-small with 3,000 domain-specific auto-generated samples.
- Retrieval-Augmented Generation: Retrieval still struggles with queries that do not map directly to corpus passages or require context absent from those passages.Such queries are described as more representative of engineers’ real-world questions.
3. Evaluations
ChipNeMo evaluations examine domain adaptation across tokenizer efficiency, pretraining, retrieval, chatbot assistance, EDA script generation, and bug analysis. Domain-adapted models generally improve chip-design performance, with gains linked to domain data, model alignment, retrieval quality, and model scale.
- Tokenizer Evaluation: 1.6% to 3.3% tokenizer-efficiency gains were achieved across chip-design datasets by adding approximately 9K domain-specific tokens.Public-data tokenizer efficiency showed no obvious change, and no significant public-benchmark accuracy decline was observed before DAPT.
- Domain-Adaptive Pretraining: DAPT substantially improved internal design and general circuit-design knowledge, while slightly degrading open-domain academic benchmark performance.Larger foundational models produced better zero-shot domain results, and DAPT gains positively correlated with model size.
- Retrieval-Augmented Generation: The domain-adapted retrieval model performed 2x better than the original e5 small unsupervised model and 30% better than sentence transformer.Retrieval quality differed by category because Specs answers were directly contained in concise passages, whereas Testbench and Build queries required more context.
- Engineering Assistant Chatbot: ChipNeMo-70B-Steer outperformed GPT-4 in all chatbot evaluation categories, including retrieval hits and misses.It also exceeded LLaMA2-70b-Chat by 3.31 points without RAG and 1.81 points with RAG on the reported human evaluations.
- Engineering Assistant Chatbot: RAG improved human scores by 0.56 for ChipNeMo-70B-Steer, 1.68 for GPT-4, and 2.05 for LLaMA2-70b-Chat.Scores were generally higher than without retrieval even when RAG missed the ideal context.
- EDA Script Generation: ChipNeMo-70B-Steer performed significantly better than GPT-4 and LLaMA2-70B-Chat on automated and human-evaluated EDA script benchmarks.The benchmarks covered easy and medium tasks checked for fully accurate code, plus hard tasks scored by human engineers on a 0-10 scale.
- EDA Script Generation: GPT-4 and LLaMA2-70B-Chat achieved close to zero accuracy on the Python-tool benchmark, while DAPT supplied necessary domain API knowledge and alignment further improved scripts.The comparison highlights the importance of DAPT and domain instructional data for proprietary-tool script generation.
4. Related Works
Prior work applies LLMs to chip design, but language- and programming-intensive tasks remain incompletely automated. Research has explored domain-specific adaptation, retrieval augmentation, model alignment, and LLM-based Verilog generation.
- Domain adaptation: Domain-specific foundation models can outperform general-purpose models on specialized tasks, but training them from scratch often requires more than 100B domain tokens.DAPT offers an alternative to training a new foundation model from scratch.
- Retrieval-augmented generation: RAG grounds generation in retrieved information and can let smaller models outperform larger models without retrieval.Retrieval methods include sparse approaches such as TF-IDF and BM25.
- Model alignment: Foundation models have limited chat and instruction-following capabilities, motivating alignment through instruction fine-tuning or reinforcement learning from human feedback.Alignment converts a foundation model into a corresponding chat model.
- LLMs for chip design: Chip-design LLM research has included generating Verilog from English and fine-tuning open-source models on Verilog datasets.One reported study found fine-tuned CodeGen models outperformed code-davinci-002 on 17 Verilog questions.
5. Conclusions
ChipNeMo evaluates domain-adapted LLMs for industrial chip design and reports that its models match or exceed base LLaMA2 results with limited additional compute. ChipNeMo-70B surpasses GPT-4 on two use cases and remains competitive on a third.
- Conclusions: ChipNeMo's 7B, 13B, and 70B models achieve similar or better results than base LLaMA2 models with only 1.5% additional pretraining compute cost.The comparison covers domain-adapted pretrained models across the reported variants.
- Conclusions: ChipNeMo-70B surpasses GPT-4 on engineering assistant chatbot and EDA scripts generation, while showing competitive performance on bug summarization and analysis.These are the paper's three evaluated chip-design use cases.
A.1. Contributions
The appendix credits contributors across research leadership, model training, infrastructure, applications, benchmarks, data preparation, and technical components. Contributions span both the engineering assistant and other chip-design applications.
- Model development: Mingjie Liu conducted domain-adaptive pretraining and model alignment.These are core model-development activities credited in the contributions.
- Infrastructure and retrieval: Teodor-Dumitru Ene and Robert Kirby developed inference and application evaluation infrastructure, while Chris Cheng developed the RAG framework.The contributions cover both evaluation infrastructure and retrieval-augmented generation.
- Data and tokenization: Nathaniel Pinckney collected and prepared training datasets, and Rongjian Liang developed custom tokenizers.These contributions support data preparation and domain-adapted tokenization.
- Benchmarks: Walker Turner, Charley Lind, and George Kokai developed a general circuit design knowledge benchmark.The benchmark is listed separately from the application-specific evaluation work.
- Engineering assistant chatbot: Siddhanth Dhodhi, Ismet Bayraktaroglu, Himyanshu Anand, and Eric Hill designed the engineering assistant chatbot and conducted its evaluation.They also provided domain instruction datasets and evaluation benchmarks.
- Applications: Parikshit Deshpande and collaborators developed the engineering assistant chatbot application, while Sun developed EDA scripts generation and its supporting datasets and benchmarks.The contributions also credit Bonita Bhaskaran, Arjun Chaudhuri, Sanmitra Banerjee, and Ghasem Pasandi for bug summarization and analysis.
- Additional contributions: Sreedhar Pratty and collaborators are credited in the contributions list for additional work.The supplied passage truncates the remainder of this contribution entry.
- Research leadership: Haoxing Ren designed and led the research, while several advisors contributed AI research and hardware engineering perspectives.The listed advisors included Brucek Khailany, Stuart Oberman, Sharon Clay, Sameer Halepete, Jonathan Raiman, Bryan Catanzaro, Jonah Alben, and Bill Dally.
A.2. Data Collection Process
ChipNeMo's data collection used scripts and internal sources to gather, normalize, filter, deduplicate, categorize, and compress hardware-related data. The collected files covered diverse design, verification, programming, and build formats.
- Collection workflow: Shell and Python scripts identified relevant design data and documentation, converted applicable sources to plain text, filtered them, deduplicated files with checksums, and compressed them.The collection flow emphasized in-situ collection from internal data sources.
- Data sources: Collected design and verification data included Verilog, VHDL, C++, Spice, Tcl, scripting languages, and build-related configuration files.Internal web services were accessed through REST APIs and conventional crawling, with HTML removed using BeautifulSoup.
- Filtering and categorization: Because most internal data was considered high quality, filtering was minimal and used line-count thresholds plus broad manually written versus tool-generated categories.The procedure excluded exceedingly large or small files.
A.3. Training Data
DAPT combines proprietary chip-design data with publicly available datasets, filtering internal sources and balancing code, documentation, and application-relevant scripts.
- DAPT combines proprietary chip-design-specific sources with publicly available datasets.Internal sources span design, verification, infrastructure, and documentation, while public data supplements the domain corpus.
- Internal data is collected and filtered by file type and by whether content is machine-generated or human-written.
- Public data includes Wikipedia for natural language and GitHub code in C++, Python, and Verilog, with approximately 9.2% of training tokens sampled.
- The data blend downsamples code, upsamples design documentation, increases human-written EDA scripts, and adds public domain data for one epoch.
A.4. Alignment Data
Alignment uses general chat instruction data together with a smaller domain-specific dataset crafted by subject matter experts for downstream chip-design use cases.
- Supervised fine-tuning uses commercially accessible general instruction-following data from OASST, FLAN, P3, and a broad proprietary dataset.
- SteerLM attribute training uses publicly available HelpSteer and OASST data, with 56,000 OASST samples for attribute-conditioned fine-tuning.
- Subject matter experts created a domain-specific instruction dataset of single-turn questions and answers for downstream-use-case alignment.
A.5. Domain Evaluation Benchmarks
The evaluation suite combines expert-designed multiple-choice domain benchmarks with public academic and code-generation benchmarks, averaging five runs to reduce testing variance and noise.
- Domain-specific evaluations use expert-designed multiple-choice questions aligned with established benchmarks such as MMLU.
- Reported benchmark results average five distinct runs, each using varied 5-shot examples, to mitigate testing variance and noise.
- Public evaluations include academic LLM benchmarks plus HumanEval for Python and VerilogEval for Verilog code generation.
A.6. Domain Adaptive Pretraining (DAPT)
DAPT ablations examine tokenizer choice, public-data mix-in, learning rate, and parameter-efficient fine-tuning across aggregated chip, academic, reasoning, and coding benchmarks. Domain-adapted tokenization improves domain benchmarks without degrading general language performance, whereas larger learning rates and LoRA impose trade-offs.
- DAPT Evaluation Setup: DAPT ablations aggregate results across chip-design, MMLU, reasoning, and coding benchmarks.The chip aggregate covers Design, Scripting, Bugs, and Circuits benchmarks.
- Domain-Adaptive Tokenization: Domain-adapted tokenization significantly improves domain benchmarks with either tokenizer, including Verilog coding, while leaving general academic performance only slightly affected.HumanEval shows no major difference, and tokenizer adaptation is reported to improve tokenization and training efficiency without general-language degradation.
- Public Datasets Mix-in: Public-data mix-in slightly improves ChipNeMo-13B results.
- Learning Rate: A larger learning rate significantly degrades language-related tasks but slightly improves coding on ChipNeMo-7B.The larger-learning-rate setup uses a cosine schedule with 200 warm-up steps and a final learning rate of 1/30th of 3 · 10^-4.
- Parameter Efficient Fine-Tuning: LoRA models underperform full-parameter training on in-domain chip-design tasks, although both improve over non-DAPT counterparts.The experiments add 26.4 million or 211.2 million trainable parameters.
A.7. Model Alignment
Model alignment combines supervised fine-tuning and SteerLM, with results showing that domain instructional data and domain adaptation affect applications differently. SteerLM helps on harder script-generation cases, while domain data can degrade chatbot responses when its labeling is insufficiently compatible.
- Alignment methods: Model alignment uses standard supervised fine-tuning and SteerLM after domain adaptation.The training masks system and user prompt losses during autoregressive optimization.
- Engineering assistant chatbot: Domain-specific instruction data produces only marginal improvement for SFT on the engineering assistant task.
- Engineering assistant chatbot: Including domain-specific instruction data in SteerLM degrades chatbot response quality, likely because the data is insufficiently adapted to SteerLM labeling.
- EDA script generation: SteerLM training helps on the “hard” EDA script-generation benchmark, indicating improved performance for relevant real-world cases.
- Additional evaluation results: DAPT substantially improves domain-task performance, while adding domain instructional data further improves chat-model accuracy; RAG helps easier script benchmarks but degrades the hard benchmark.Bug summarization shows only minor DAPT gains, slight summarization gains from SteerLM, and no significant task-assignment difference.