PaperlayerView exact revision on arXiv

Source-linked AI summary

VeriCoder: Enhancing LLM-Based RTL Code Generation through Functional Correctness Validation

Anjiang Wei, Huanmi Tan, Tarun Suresh, Daniel Mendoza, Thiago S. F. X. Teixeira, Ke Wang, Caroline Trippel, Alex Aiken

arXiv:2504.15659v2cs.ARcs.AIcs.CLcs.LGcs.SE

TL;DR

Existing RTL datasets often emphasize syntax over functional correctness, leaving whether designs meet their specifications unverified. VeriCoder uses teacher-guided test generation and feedback-directed refinement to build a functionally validated training set, achieving relative pass@k gains of up to 71.7% on VerilogEval and 27.4% on RTLLM over prior best fine-tuned models.

  • Problem

    Most existing RTL datasets emphasize syntactic validity without evidence of functional correctness, so code may compile yet fail to meet its natural-language specification.

    Source: B. Related Work on RTL Code Generation · p. 3

  • Method

    A teacher LLM generates unit tests and iteratively refines RTL designs, updating tests when needed, to produce 125,777 validated specification-design-test triples.

    Source: §I · p. 2

  • Results

    71.7% on VerilogEval and 27.4% on RTLLM: VeriCoder's relative pass@k gains over the previous best fine-tuned model.

    Source: §I · p. 2

  • Takeaways & Limitations

    An ablation study confirms the impact of functional validation on model performance, underscoring the importance of high-quality training data.

    Source: §VII · p. 9

    Synthetic tests may miss edge cases, so the pipeline does not fully guarantee correctness; the authors identify formal verification as future work.

    Source: §VI · p. 8

Abstract

from arXiv · show

Recent advances in Large Language Models (LLMs) have sparked growing interest in applying them to Electronic Design Automation (EDA) tasks, particularly Register Transfer Level (RTL) code generation. While several RTL datasets have been introduced, most focus on syntactic validity rather than functional validation with tests, leading to training examples that compile but may not implement the intended behavior. We present VERICODER, a model for RTL code generation fine-tuned on a dataset validated for functional correctness. This fine-tuning dataset is constructed using a novel methodology that combines unit test generation with feedback-directed refinement. Given a natural language specification and an initial RTL design, we prompt a teacher model (GPT-4o-mini) to generate unit tests and iteratively revise the RTL design based on its simulation results using the generated tests. If necessary, the teacher model also updates the tests to ensure they comply with the natural language specification. As a result of this process, every example in our dataset is functionally validated, consisting of a natural language description, an RTL implementation, and passing tests. Fine-tuned on this dataset of 125,777 examples, VERICODER achieves state-of-the-art metrics in functional correctness on VerilogEval and RTLLM, with relative gains of up to 71.7% and 27.4%, respectively. An ablation study further shows that models trained on our functionally validated dataset outperform those trained on functionally non-validated datasets, underscoring the importance of high-quality datasets in RTL code generation. Our code, data, and models are publicly available at https://github.com/Anjiang-Wei/VeriCoder

I. INTRODUCTION

The paper targets RTL code generation amid a gap between syntactic validity and functional correctness in existing datasets. It introduces VeriCoder, trained on functionally validated examples, and reports state-of-the-art benchmark performance among open-source fine-tuned models.

I. INTRODUCTION

  • Existing RTL datasets often emphasize syntax, although code that compiles may still fail to implement its natural-language specification.The paper notes that RTLLM reports GPT-4o at 100.0% syntax accuracy but 69.0% functional correctness.Source: §I · p. 1
  • A teacher LLM generates unit tests, revises RTL from simulation feedback, and updates tests when needed to match the specification.The pipeline uses GPT-4o-mini and continues until simulation passes or a retry limit is reached.Source: §I · p. 2
  • VeriCoder achieves state-of-the-art performance among open-source fine-tuned models, with relative pass@k gains of up to 71.7% on VerilogEval and 27.4% on RTLLM over the prior best.The evaluation compares VeriCoder with the previous best fine-tuned model, OriGen.Source: §I · p. 2
  • An ablation found that, under the same base model and training setup, models trained on functionally validated data outperform those trained on non-validated data.The authors cite this comparison as evidence for the importance of high-quality, functionally validated datasets.Source: §I · p. 2
  • The dataset contains 125,777 validated triples pairing natural-language specifications, RTL designs, and passing tests.The contribution describes these examples as validated for functional correctness.Source: §I · p. 2
  • Manual review of 100 randomly sampled examples found that 92% of generated RTL correctly matched the corresponding natural-language descriptions.The paper states that functional correctness is not fully guaranteed.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

LLMs are trained to predict next tokens, then adapted to specific tasks through post-training. LoRA supports resource-efficient adaptation by training low-rank updates instead of changing the full model weights.

A. Language Modeling and Fine-Tuning

  • Language modeling trains an LLM to predict the next token by maximizing the log-likelihood of each token given the preceding context.The context tokens are denoted x<t, and θ denotes the model parameters.Source: A. Language Modeling and Fine-Tuning · p. 2
  • Pre-training uses massive, diverse corpora to acquire broad knowledge, while post-training adapts a pretrained model using smaller task-specific datasets.Supervised fine-tuning is a post-training approach based on task-specific input-output pairs.Source: A. Language Modeling and Fine-Tuning · p. 2
  • LoRA freezes the original weight matrices and trains low-rank updates, reducing memory and compute overhead during domain-specific adaptation.The update is W′ = W + AB, with only A and B updated.Source: A. Language Modeling and Fine-Tuning · pp. 2–3

B. Related Work on RTL Code Generation

Prior RTL dataset construction commonly relies on mining or LLM-generated code with syntax checks, leaving functional correctness unverified. VeriCoder instead uses unit tests for functional validation and reports a larger validated dataset than prior work described here.

B. Related Work on RTL Code Generation

  • Existing dataset strategies largely establish syntax validity, but lack evidence of functional correctness and may retain designs that fail their intended specifications.Strategies include mining open-source RTL and generating or rewriting code with commercial LLMs.Source: B. Related Work on RTL Code Generation · p. 3
  • VeriCoder uses generated unit tests rather than assertions for validation, and its dataset contains over 125,777 examples versus OpenLLM-RTL’s 7k.The authors describe their dataset as the largest functionally validated RTL dataset to date.Source: B. Related Work on RTL Code Generation · p. 4
  • VeriCoder uses standard supervised fine-tuning and is complementary to approaches based on reasoning, curriculum learning, reinforcement learning, retrieval, or fill-in-the-middle generation.The related methods include ScaleRTL, DeepRTL, VeriSeek, AutoVCoder, and CodeV.Source: B. Related Work on RTL Code Generation · p. 4

A. Overview

VeriCoder augments unvalidated RTL datasets by generating tests and iteratively refining designs against simulation feedback. A motivating example shows how this process detects a syntactically valid but functionally incorrect design and produces a corrected implementation.

A. Overview

  • The augmentation pipeline starts from specification–design pairs, generates a test, and iteratively revises the design or test using simulation errors until success or the attempt limit.The output is a dataset of specification, design, and test triplets; the approach is described as applicable beyond Origen.Source: A. Overview · p. 4

B. Motivating Example

  • The example illustrates that syntax-valid RTL can still fail its intended behavior, a problem that test-based validation and refinement can detect and correct.The original dataset design was syntactically valid but contained semantic errors.Source: B. Motivating Example · pp. 4–5
  • In the and3 example, the original design times out on a generated test that sets all three inputs to 1, while the corrected design passes simulation.The design’s feedback loop retriggers because y is both read and written in the combinational block; the correction uses a single blocking assignment.Source: B. Motivating Example · p. 5

C. Algorithm and Prompts

The algorithm applies teacher-generated tests to specification–design pairs, uses simulation errors to guide revisions, and adds passing triples to the augmented dataset. The authors caution that unit tests do not guarantee correctness for every possible input.

C. Algorithm and Prompts

  • GPT-4o-mini powers test generation and refinement; the authors chose it over stronger models because of dataset scale and repeated-query costs.Origen contains 217,462 examples.Source: C. Algorithm and Prompts · p. 6
  • Each specification–design pair receives a teacher-generated unit test, and simulation failures trigger edits to the design, test, or both.The refinement cycle ends when the design passes or reaches the maximum of five attempts.Source: C. Algorithm and Prompts · p. 6
  • Passing specification–design–test triples are added to the output dataset, which the pipeline validates through simulation rather than syntax checks alone.The process is presented as a way to detect and correct subtle RTL bugs.Source: C. Algorithm and Prompts · p. 6
  • Unit-test validation cannot guarantee correctness for all possible inputs, and manual review found that 92% of sampled RTL code matched its natural-language description.The review covered 100 randomly sampled examples.Source: C. Algorithm and Prompts · p. 6

A. Dataset

The dataset pairs specifications, RTL designs, and unit tests; VeriCoder is fine-tuned from Qwen2.5-14B-Instruct using LoRA.

A. Dataset

  • The dataset contains 125,777 examples, each pairing a natural language specification with an RTL design and associated unit tests.Specifications average 247 words, RTL implementations 35 lines, and unit tests 55 lines.Source: A. Dataset · p. 6
  • Specifications range from 116 to 549 words, RTL implementations from 5 to 225 lines, and unit tests from 6 to 197 lines.Source: A. Dataset · p. 6

B. LoRA Fine-Tuning Setup

  • Fine-tuning uses Qwen2.5-14B-Instruct with LoRA rank 16 and scaling factor 32 across all transformer linear projection layers.Training runs for three epochs with batch size 40, a constant 1×10−5 learning rate, linear decay, and 0.05 warm-up ratio.Source: B. LoRA Fine-Tuning Setup · p. 6

C. Benchmarks and Metrics

Evaluation uses VerilogEval and RTLLM with benchmark-specific correctness metrics; model comparisons standardize prompts and post-processing.

C. Benchmarks and Metrics

  • VerilogEval reports Pass@k for k=1, 5, and 10, estimating the probability that at least one of the top-k programs passes all tests.For RTLLM, the reported measures are syntax correctness and functional correctness using Pass@5.Source: C. Benchmarks and Metrics · p. 6

D. Models for Evaluation

  • The evaluated models include pretrained base models and released fine-tuned models such as OriGen, RTLCoder, and ChipGPT.The base-model group includes Qwen2.5-14B-Instruct, which is used for VeriCoder fine-tuning.Source: D. Models for Evaluation · p. 6
  • All models receive identical prompts and post-processing under a uniform evaluation script, controlling comparison inputs and inference handling.The authors state that this standardization helps isolate model capability.Source: D. Models for Evaluation · pp. 6–7

A. Main Evaluation Results

VeriCoder outperforms prior open-source fine-tuned models on both benchmarks, while the dataset ablation favors functionally validated training data over unvalidated data.

A. Main Evaluation Results

  • VeriCoder leads open-source fine-tuned models on both benchmarks, with relative pass@k gains up to 71.7% on VerilogEval and 27.4% on RTLLM.Source: A. Main Evaluation Results · p. 7
  • VerilogEval-Machine pass@1 is 55.7%, 19.8 percentage points above OriGen; VerilogEval-Human reaches 38.3%, 16.0 points above OriGen.Source: A. Main Evaluation Results · p. 7
  • On RTLLM, VeriCoder records 79.3% syntax correctness and 48.3% functional correctness, versus OriGen’s 51.7% and 37.9%, respectively.Source: A. Main Evaluation Results · p. 7
  • A performance gap remains against larger models: o3-mini scores 66.4% on VerilogEval Pass@1 versus VeriCoder’s 55.7%.GPT-4o reaches 69.0% functional correctness compared with VeriCoder’s 48.3%.Source: A. Main Evaluation Results · p. 7

B. Ablation Study of Dataset

  • With the same base model and training setup, validated data raises VerilogEval Pass@5 to 55.8% from 53.5% with unvalidated data and 46.8% for the base model.On RTLLM, validated-data training reaches 79.3% syntax correctness and 48.3% functional correctness, versus 75.9% and 44.8% with unvalidated data.Source: B. Ablation Study of Dataset · p. 8, Table IV · p. 8

C. Test Passing Rates of Non-Validated Datasets

VeriCoder reports stronger benchmark results and finds that prior datasets differ in how often their sampled designs pass synthetic functional tests. The paper also identifies limits in test coverage and small-scale RTL scope.

C. Test Passing Rates of Non-Validated Datasets

VI. DISCUSSION AND FUTURE WORK

  • Synthetic tests may miss edge cases, so passing them does not fully guarantee RTL correctness.The discussion proposes integrating formal verification into dataset construction.Source: §VI · p. 8
  • Most approaches, including VeriCoder, target small-scale RTL, while practical hardware development often involves repository-level codebases with cross-file dependencies and long-range context.The authors identify extending VeriCoder’s test-generation and refinement components to repository scale as future work.Source: §VI · p. 8
  • The authors suggest using reinforcement learning with VeriCoder’s accompanying tests as feedback to further improve RTL generation.This is proposed as future work, building on studies that use test outcomes, compiler diagnostics, or formal verification results as feedback.Source: §VI · p. 8

VII. CONCLUSION

  • VeriCoder’s dataset contains 125,000 functionally validated triples of specifications, RTL implementations, and passing tests, constructed through teacher-guided refinement.The conclusion says the teacher iteratively updates RTL designs and unit tests until designs pass simulation.Source: §VII · p. 8
  • Up to 71.7% relative improvement on VerilogEval and 27.4% on RTLLM; an ablation also supports the impact of functional validation.The conclusion reports state-of-the-art results on both benchmarks and points to formal verification and reinforcement learning as future directions.Source: §VII · p. 9
Loading 2504.15659v2…