PaperlayerView exact revision on arXiv

Source-linked AI summary

Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation

Jinuk Kim, Junsoo Byun, Donghwi Hwang, Seong-Jin Park, Hyun Oh Song

arXiv:2605.15669v1cs.LG

TL;DR

DRC script synthesis from natural-language rules is labor-intensive, while existing benchmarks often lack scale and execution-based evaluation. Rule2DRC introduces a large benchmark and SplitTester, which uses execution feedback to improve Best-of-N selection by distinguishing candidate scripts.

  • Problem

    Existing DRC script benchmarks are small and often assess surface-level code similarity rather than functional correctness under execution.

  • Method

    Rule2DRC provides execution-based evaluation, while SplitTester generates discriminative layouts by targeting clusters of candidates with identical observed behavior.

  • Results

    SplitTester outperforms prior tester agents under Best-of-N selection by more reliably identifying correct scripts among candidates.

  • Takeaways & Limitations

    Rule2DRC and SplitTester support reproducible progress in execution-guided DRC script synthesis and program selection.

  • Takeaways & Limitations

    Automated DRC generation may be unsafe when scripts are inaccurate, omit critical rules, or misimplement constraints, and executable environments require access controls.

Abstract

from arXiv · show

Manufacturable chip layouts must satisfy thousands of geometry-based design rules, and design rule checking (DRC) enforces them by running executable DRC scripts on layouts. Translating natural language rules into correct DRC scripts is labor-intensive and requires specialized expertise, motivating LLM agents for DRC script synthesis and debugging. However, existing benchmarks have small evaluation sets and often evaluate scripts by code similarity rather than execution correctness, and prior machine learning-based methods either ignore execution feedback or require labeled test layouts as agent's input. To this end, we introduce Rule2DRC, a large-scale benchmark for DRC script coding agents with 1,000 rule-to-script tasks and 13,921 evaluation chip layouts for execution-based scoring. Rule2DRC provides an evaluation pipeline that measures functional correctness via DRC execution outcomes without requiring evaluation layouts as input to the agent. We also propose SplitTester, a tester agent for program selection that uses execution feedback to generate discriminative test cases and separate previously indistinguishable candidate scripts, substantially improving Best-of-N selection performance in this domain. We release the code at https://github.com/snu-mllab/Rule2DRC.

1. Introduction

Rule2DRC addresses the labor-intensive translation of increasingly complex natural-language design rules into executable DRC scripts and limitations in existing benchmarks and methods. It provides large-scale execution-based evaluation and SplitTester, which uses DRC feedback to distinguish candidate scripts and improve selection.

  • Motivation: Translating high-level natural-language design rules into correct executable DRC scripts is labor-intensive, and increasing rule complexity amplifies validation effort, turnaround time, and maintenance costs.The burden grows as technology scales to advanced process nodes, including 7 nm and below.
  • Limitations: Existing benchmarks often contain fewer than 200 rule-script pairs and evaluate scripts by surface-level code similarity rather than functional execution correctness.These limitations make benchmark scale and evaluation fidelity inadequate for DRC script synthesis.
  • Limitations: Existing methods either omit DRC execution feedback or require evaluation layouts and ground-truth verification outcomes as agent inputs.The latter assumption conflicts with evaluation settings where layouts are not available to the agent.
  • Rule2DRC: Rule2DRC provides 1,000 rule-to-script tasks and 13,921 evaluation layouts for execution-based scoring without requiring test layouts as agent input.Its dataset combines 310 SkyWater130-derived rules with 690 synthetic rules covering more complex multi-constraint scenarios and broader DRC grammar operators.
  • SplitTester: SplitTester clusters candidate scripts by observed execution outputs, generates tests targeting indistinguishable clusters, and selects high-quality scripts using feedback, outperforming other agents in Best-of-N selection.It repeatedly adds discriminative layout tests to split large groups of candidates that remain equivalent on current tests.

2. Related Work

Prior DRC script synthesis methods often rely on code-similarity evaluation or labeled evaluation layouts, limiting assessment of functional correctness and full natural-language-to-code translation. Rule2DRC instead uses execution-based evaluation without requiring test layouts as model input, while SplitTester generates tests from execution feedback to improve candidate selection.

  • DRC Script Synthesis: Existing learning-based DRC script synthesis methods translate natural-language rules into code but primarily evaluate against reference scripts using code similarity.Code similarity is unreliable because one rule can have multiple syntactically different implementations.
  • DRC Script Synthesis: DRC-Coder combines vision-language and language models with labeled violation layouts, but assumes evaluation layouts and annotations that often require expert effort or trusted reference scripts.It therefore primarily targets distillation and acceleration of existing DRC flows rather than full translation.
  • LLM Agents with Test Generation: Prior coding-agent methods use execution feedback, tester agents, and Best-of-N selection, while newer approaches generate tests systematically from execution signals.The paper’s setting assumes no ground-truth outputs or public tests and focuses on selecting among candidate scripts.
  • DRC Script Synthesis: Rule2DRC is an order of magnitude larger than prior DRC evaluation sets and supports execution-based scoring without requiring test layouts as model input.It provides evaluation layouts with ground-truth violation labels and is open-sourced with the benchmark code.
  • LLM Agents with Test Generation: SplitTester targets indistinguishable candidate clusters, continuously reclusters candidates after new execution outcomes, and avoids premature pruning during Best-of-N script selection.Its tester agents are evaluated by generating test chip layouts and selecting the best script among candidates.

3. Rule2DRC Benchmark

Rule2DRC is a large-scale benchmark for generating DRC scripts from natural-language design rules, evaluated by execution outcomes on private chip layouts. It combines real PDK-derived rules with synthetic multi-layer and grammar-coverage tasks, using KLayout and execution-based success and error metrics.

  • Benchmark Overview: Rule2DRC contains 1,000 rule-to-script tasks and 13,921 evaluation chip layouts for execution-based scoring.Each task pairs a natural-language design rule with its corresponding executable DRC script.
  • Problem Formulation: Agents receive only a natural-language rule and output a DRC script, while private evaluation layouts assess functional correctness.Correctness is based on agreement between generated and ground-truth violation outputs across the evaluation layouts.
  • Implementation: The benchmark uses KLayout’s built-in DRC language and represents chip layouts in the standard GDSII format.KLayout supports both DRC execution and programmatic layout drawing, enabling agents to generate and inspect layouts through code.
  • Benchmark Composition: Tasks include SkyWater130 PDK rules, hard negatives, corner cases, 490 synthetic multi-layer rules, and 200 grammar-coverage tasks.The construction process ensures passing and failing examples while expanding coverage of chained geometric operations and underrepresented grammar constructs.
  • Evaluation Protocol: Evaluation reports success rate and error rate, measuring correct task solutions and compile-time or runtime failures, respectively.Success requires exact agreement with the ground-truth script’s violation outputs on every evaluation layout.

4. SplitTester

SplitTester improves candidate-script selection by generating execution-guided tests that split large, high-scoring clusters of scripts indistinguishable under existing tests. It iteratively reclusters candidates under new tests, then uses a judge LLM to select among the top three candidates using tests where their outputs differ.

  • Initial clustering: SplitTester initially executes all candidate scripts on generated tests and clusters candidates with identical outputs under the current test set.The same underlying LLM serves as test generator and judge through different prompts and inputs.
  • Targeted splitting: It targets the non-singleton cluster maximizing s_i|C_i|, prioritizing clusters that are both high-scoring and large.Here, s_i is the cluster score and |C_i| is its size, reflecting the likelihood that the cluster contains correct candidates and correctness differences.
  • Targeted splitting: For each target cluster, SplitTester conditions test generation on K randomly sampled representative candidates, evaluates all candidates, and reclusters them under the expanded tests.Sampling representatives reduces distraction and prompt length when the target cluster is large.
  • Early stopping: The splitting phase stops at the test-budget limit or after P consecutive failed split attempts, avoiding further tests when generated layouts remain nondiscriminative.The main experiments set P = 1 as a Pareto-efficient trade-off.
  • Final judging: After splitting, a judge LLM selects the best of the top three candidates using the subset of tests on which their outputs differ.This final phase addresses potentially incorrect labels from self-generated tests by asking the judge to choose the correct solution.

5. Experiments

Experiments evaluate three language models and measure Rule2DRC performance using success rate, error rate, and Oracle@N. They show that API documentation substantially improves generation, SplitTester improves Best-of-N selection across models, lies on the runtime Pareto frontier, and benefits from its final judge and expected-label components.

  • Experimental setup: Three models are evaluated: Qwen3-30B-A3B-Instruct-2507, GPT-OSS-20B, and GPT-OSS-120B, with medium reasoning effort for both GPT-OSS models.The models are selected as state-of-the-art models of comparable size and type.
  • Evaluation metrics: Success rate, error rate, and Oracle@N are the primary Rule2DRC metrics, with Oracle@N defined as pass@N for success and the all-error fraction for error.Oracle@N is an upper bound on Best-of-N selection performance for a given number of samples.
  • API documentation: Including the 60K-token API document raises GPT-OSS-120B pass@1 by over 20%p and pass@20 by over 40%p versus omitting it.The results motivate including API documentation in subsequent prompts because the DRC domain-specific language benefits strongly from in-context documentation.
  • Tester agents: 44.4% success and 12.6% error are achieved by SplitTester on GPT-OSS-20B BoN-20, versus 41.1% and 14.1% for CodeMonkey.On Qwen3-30B-A3B-Instruct-2507 BoN-20, SplitTester reaches 18.0% success and 35.1% error versus CodeMonkey’s 17.2% and 38.6%.
  • Cost-performance trade-off: SplitTester lies on the Pareto frontier across all three models and achieves higher success than every baseline at comparable runtime.Runtime is measured over all 1,000 Rule2DRC tasks using 2 H100 GPUs for LLM serving and an Intel Xeon Gold 5218R CPU for DRC evaluation.
  • Ablations: 58.0% full-system success falls to 55.5% without the final judge LLM and to 57.1% without expected labels.The ablations indicate that generated expected-label scoring and final judging provide complementary selection signals.

6. Conclusion

The work advances DRC script synthesis from natural-language rules by using execution feedback with an LLM-based tester agent. It introduces Rule2DRC for execution-based scoring and SplitTester for generating discriminative layout tests.

  • DRC script synthesis is labor-intensive because it requires domain expertise and familiarity with domain-specific languages.
  • Rule2DRC provides 1,000 rule translation tasks and 13,921 evaluation layouts for execution-based scoring.
  • SplitTester clusters scripts by identical observed behavior and generates discriminative layout test cases to split previously indistinguishable scripts.

Impact Statement · A. Implementation Details

Rule2DRC advances LLM-agent automation for generating executable DRC scripts through an open-source benchmark, while SplitTester uses execution-guided testing to improve Best-of-N selection. The tools may reduce manual DRC-deck effort, but executable environments and inaccurate scripts create important safety risks.

  • Impact Statement: Rule2DRC is an open-source benchmark for translating natural-language design rules into executable DRC scripts.It targets automation of the labor-intensive task of generating DRC scripts with LLM agents.
  • Impact Statement: SplitTester is an execution-guided tester agent that improves Best-of-N selection and increases success rates.Its role is to use execution feedback to distinguish and select candidate DRC scripts.
  • Impact Statement: Together, Rule2DRC and SplitTester can reduce manual effort required to implement and validate DRC decks.The proposed tools are intended to automate both script generation and validation work.
  • Impact Statement: The tools can shorten turnaround time when migrating DRC decks to new process nodes.This impact follows from reducing manual implementation and validation effort.
  • Impact Statement: The released resources can lower barriers for researchers building execution-based LLM agents for electronic design automation.The benchmark and tester agent provide open-source infrastructure for this research direction.
  • Impact Statement: LLM interaction with executable environments can produce unintended side effects unless permissions are properly constrained.Sandboxing and strict access control can mitigate these risks, but they remain important considerations.
  • Impact Statement: Over-reliance on automated DRC generation is unsafe when scripts are inaccurate, omit critical rules, or misimplement constraints.The risk concerns trusting generated scripts without sufficient validation of their correctness and coverage.

A.1. Benchmark Construction · A.2. Tester Agents · B. Additional Experiments

The benchmark combines manually constructed SkyWater-derived tasks with synthetic rule tasks, using corner-case test layouts and quality assurance to support execution-based evaluation. Tester agents generate or adapt GDSII layouts and select among candidate scripts using execution outcomes, while the provided passages contain no findings from B. Additional Experiments.

  • A.1. Benchmark Construction: SkyWater-derived tasks required two months of manual effort to extract rules, implement KLayout DRC scripts, and construct GDS test layouts.Corner cases targeted threshold behavior at the rule’s minimum 1 nm resolution and difficult cases distinguishing correct from near-correct scripts.
  • A.1. Benchmark Construction: Synthetic tasks added one month of effort by using GPT-5.2-high to draft rules, scripts, and tests grounded in verified SkyWater-derived tasks.The process introduced multi-constraint or niche-grammar rules and threshold and topological corner cases, followed by expert manual review.
  • A.2. Tester Agents: The test generator prompts an LLM to produce a GDSII layout through KLayout’s programmable API and its expected violation output.Failed generations receive error traces and up to five retries; tasks still failing after five attempts are skipped.
  • A.2. Tester Agents: SplitTester uniformly samples K = 3 candidates within the target cluster, uses all candidates when fewer are available, and applies early-stopping patience P = 1 by default.The passages note that larger patience values are analyzed in Section B.2.
  • B. Additional Experiments: No additional-experiment findings from B. Additional Experiments are included in the supplied passages.The provided passages cover benchmark construction and tester-agent implementations only.
  • A.2. Tester Agents: CodeMonkey is adapted to iteratively edit programmatically generated GDSII layouts to distinguish the top three scripts.It permits 8 additional layout edits after 8 initial tests, for a total test-layout budget of 16.
  • A.2. Tester Agents: S∗ clusters candidates by test outcomes, generates additional inputs, and uses a debiased LLM judge queried twice with swapped candidate order.This follows the procedure originally proposed for code-competition benchmarks.
  • A.2. Tester Agents: The LLM-as-a-Judge method uses tournament-style k-way selection with k = 4, repeatedly advancing one winner until a single script remains.This avoids overwhelming the judge or exceeding its context budget when evaluating all candidates simultaneously.

B.1. Cost Analysis · B.2. Varying Early Stop Parameters · B.3. Reduced Test Budget

The appendix analyzes SplitTester’s cost, early-stopping patience, and performance under a reduced test budget. SplitTester remains Pareto-efficient while achieving the strongest reported success rates across models and Best-of-N settings.

  • B.1. Cost Analysis: Tables 4–5 break down each tester agent’s runtime, generated tokens, and DRC evaluations across all 1,000 Rule2DRC tasks.Prefix caching is applied uniformly across methods to avoid redundant prefix recomputation.
  • B.1. Cost Analysis: Figure 7 compares Pareto curves including LLM-as-a-Judge for Best-of-N values of 10, 15, and 20.Runtime covers all 1,000 tasks and uses 2 × H100 GPUs for LLM serving plus an Intel Xeon Gold 5218R CPU for DRC evaluation.
  • B.1. Cost Analysis: SplitTester with P = 1 matches CodeMonkey’s runtime, costs less than S∗, and reaches the highest success rate across models and Best-of-N settings.On GPT-OSS-20B with BoN-20, runtimes are 227.40 versus 226.43 minutes, while success increases from 41.1% to 44.4%.
  • B.2. Varying Early Stop Parameters: P controls how many consecutive failed split attempts SplitTester tolerates before stopping test generation, and the study evaluates P ∈{1, 2, 3}.Figure 8, Table 4, and Table 5 report the resulting Pareto curves and numerical results.
  • B.2. Varying Early Stop Parameters: Increasing patience P trades runtime for success, with P = 3 outperforming P = 1 across all three models and Best-of-N settings.On GPT-OSS-120B with BoN-20, success rises from 63.8% at P = 1 to 64.6% at P = 3.
  • B.2. Varying Early Stop Parameters: 187.99 to 226.62 minutes: GPT-OSS-120B with BoN-20 incurs this runtime increase when patience rises from P = 1 to P = 3.All three variants achieve higher success rates than the baselines; P = 1 is retained as the default for Pareto efficiency.

B.4. Benchmark Statistics

Rule2DRC spans 1,000 tasks with substantial execution-oriented evaluation coverage and diverse DRC method usage. Its rule categories complement one another, ranging from simpler SkyWater-derived rules to more complex synthetic multi-constraint rules and broader syntax coverage.

  • Overall benchmark: Across 1,000 tasks, each task averages 13.9 evaluation layouts and 6.4 DRC method invocations from 5.1 unique methods.The full benchmark covers 184 unique KLayout DRC methods.
  • Category roles: Category 1 uses 2.6 method invocations and 5.3 evaluation layouts per task on average, making it the simplest category in per-task structure.These rules are extracted from the public SkyWater130 PDK used in real chip manufacturing.
  • Category roles: Category 2 is the most complex per task, averaging 9.2 method invocations and 20.7 evaluation layouts.It consists of synthetic multi-constraint rules.
  • Category roles: Category 3 emphasizes breadth through syntax coverage and uses 170 unique DRC methods.The passage characterizes this category by breadth rather than per-task structural simplicity or complexity.

B.5. Expected Label Error Rates … B.8. Per-Category Breakdown

The analyses show that SplitTester remains effective despite noisy LLM-generated labels, is robust to cluster-scoring variants, benefits from sequential revision, and performs strongly across rule categories. Its final judge complements expected labels, while cluster-targeted tests support both selection and refinement.

  • B.5. Expected Label Error Rates: Expected-label error rates range from 15.71% for GPT-OSS-120B to 37.60% for Qwen3-30B-A3B-Instruct.Errors are measured by comparing LLM-predicted labels with outputs from the ground-truth DRC script on generated test layouts.
  • B.5. Expected Label Error Rates: Removing expected labels reduces success rate from 58.0% to 57.1%, while removing the final judge LLM reduces it to 55.5%.The final judge mitigates label noise by comparing candidate behavior on discriminative tests, and the two components are complementary.
  • B.6. Alternative Cluster Scoring Rules: SplitTester’s default si|Ci| cluster-scoring rule combines cluster score and size, while alternatives ignore either size or score.The alternatives are Largest cluster and Highest-score cluster, evaluated on GPT-OSS-120B.
  • B.6. Alternative Cluster Scoring Rules: All three cluster-scoring variants achieve success rates within one standard deviation of one another and identical error rates.This indicates robustness to the specific cluster-scoring rule and shows gains are not driven by si|Ci|.
  • B.7. Sequential Revision: Sequential revision raises SplitTester’s success rate from 62.3% at M = 0 to 63.4% at M = 3.A revision is accepted only when it strictly improves the candidate’s score on collected tests.
  • B.7. Sequential Revision: SplitTester’s error rate falls from 4.2% to 2.7% as revision rounds increase, while maintaining a roughly 6 percentage-point success-rate lead over Generated Tests.The comparison is restricted to Generated Tests because it is the only baseline tester agent producing expected labels.
  • B.8. Per-Category Breakdown: Across GPT-OSS-20B rule categories, one-shot success ranges from 39.2% for SkyWater-derived rules to 5.0% for synthetic multi-constraint rules, with syntax coverage at 11.7%.Category difficulty therefore varies substantially.
  • B.8. Per-Category Breakdown: SplitTester achieves the highest success and lowest error rates across all three categories and Best-of-N settings.The reported breakdown is for GPT-OSS-20B.

B.9. F1 Score Evaluation · C. An Additional Qualitative Example · D. Prompts

The paper evaluates partial DRC-script correctness with per-task F1 and shows that SplitTester leads across models and Best-of-N settings. It also provides a qualitative benchmark example and specifies prompts for deterministic testcase generation, candidate selection, retries, and interactive judging.

  • B.9. F1 Score Evaluation: F1 captures partial correctness by measuring violation-class prediction over ground-truth test cases, assigning compile failures an F1 of 0.Scores are averaged per task across all 1,000 Rule2DRC tasks.
  • B.9. F1 Score Evaluation: 67.7 F1 is achieved by SplitTester on GPT-OSS-20B at BoN-20, compared with 66.6 for CodeMonkey.On GPT-OSS-120B, SplitTester reaches 85.6 versus the Oracle@20 upper bound of 87.4.
  • B.9. F1 Score Evaluation: SplitTester achieves the highest F1 across all three models and Best-of-N settings, extending its success-rate advantage to partial-correctness scoring.The reported gains are attributed to cluster-targeted test generation.
  • C. An Additional Qualitative Example: Figure 9 presents a qualitative Rule2DRC task with its natural-language rule, ground-truth DRC script, and corresponding test layouts.Violating layouts are highlighted in red, while non-violating layouts are highlighted in green.
  • D. Prompts: The shared prompt identifies the model as a senior physical verification engineer and treats documentation inside <doc>...</doc> as reference material rather than instructions.The reference block contains KLayout documentation.
  • D. Prompts: SplitTester’s generator prompt requires one deterministic runnable Python testcase script using pya, writing GDS output to OUT DIR and placing the expected result in the first-line comment.The prompt instructs the generator to expose differences between candidate DRC decks and supports a retry using the previous script and error output.
  • D. Prompts: Interactive CodeMonkey prompts let the model either decide the best candidate or refine a testcase generator for the next testcase after observing candidate execution results.The generator must produce compact, fast GDS layouts with deterministic randomness and bounded testcase counts.
Loading 2605.15669v1…