Validation and Simulation Catch Different Errors: Four Levels of Evaluation for LLM-Generated Circuits

1 Introduction

• The paper measures schema validity, topological validity, backend executability, and component-set agreement as distinct evaluation levels, not a hierarchy of correctness.Topological validity guarantees neither physical validity nor simulation success.Source: §1 · p. 2
• The study measures all four levels on the same generated circuits and quantifies disagreement between topology checks and simulation in both directions.The contribution also includes a silent-failure class and a minimal voltage-divider experiment demonstrating its consequence.Source: §1 · p. 2
• A paired 45-circuit replay evaluates the same model samples at successive pipeline stages to separate repair effects from fresh-generation sampling noise.The replay isolates deterministic ground canonicalisation and model-based repair.Source: §1 · p. 2
• The paper compares the full pipeline with direct netlist generation and conservatively accounts for failures that cannot confidently be attributed to the generated netlist.This is presented as a comparison with a direct-netlist baseline.Source: §1 · p. 2

2.1 LLMs for analog and mixed-signal circuits

• Analog-circuit systems generate SPICE or explore topology, while adjacent digital systems target HDL rather than analog netlists.Examples include AnalogCoder [1], LaMAGIC [2] and LaMAGIC2 [3], AnalogGenie [4], CktGen [5], AnalogAgent [6], AnalogMaster [7], Chip-Chat [10], ChipNeMo [11], VerilogEval [12], and RTLLM [13].Source: §2.1 · pp. 2–3
• The paper asks what each checking stage contributes and misses on the same artifacts, without claiming priority for generating executable SPICE from natural language.The authors say cited work reports executability or task success rather than disagreement between structural checks and simulation on the same artifacts.Source: §2.1 · p. 3

2.2 Self-correction and constrained generation

• Schema-constrained decoding can guarantee schema adherence but not graph properties; exactly three circuits (2.0%) failed at schema level, while it would not address 26 higher-level failures.The paper treats constrained decoding and validation as complementary because they address different levels.Source: §2.2 · p. 3

2.3 Intermediate representations

• CIR is a typed, validated interchange representation with a closed type system, validator, and canonicalisation pass, but no optimisation passes or lowering hierarchy.The authors distinguish it from a compiler IR.Source: §2.3 · p. 3

2.4 Interactive and educational simulators

• CircuitLab, EasyEDA, Tinkercad Circuits, and Falstad offer schematic capture and simulation without natural-language input; Wokwi is closest to the firmware backend.Wokwi uses the same avr8js instruction-set simulator, and the authors claim no novelty in browser-based AVR emulation.Source: §2.4 · p. 3

3.1 CIR

• CIR stores circuits as flat component lists using a closed enum of 38 types, with ordered pin semantics shared by consumers such as the netlist emitter and schematic renderer.Five producers and four consumers meet at CIR.Source: §3.1 · p. 4

3.2 The ladder, and what it refuses

• The emitter refuses netlists referencing undefined .model or .subckt entries; 23 of 79 baseline failures were unknown-subcircuit cases that the pipeline cannot produce by construction.The refusals also include a requested relay and zener whose unsupported types surfaced as explicit errors rather than being mapped to similar components.Source: §3.1 · p. 4, §3.2 · p. 4
• The pipeline canonicalises types, validates the schema, infers ground, runs topological validation, applies bounded model repair, and emits to a backend.Mechanically recoverable type and ground errors are handled before a model round-trip; type canonicalisation refuses mappings that could simulate the wrong circuit.Source: §3.2 · p. 4

3.3 Backends

• The SPICE backend routes CIR through a netlist emitter to ngspice, while circuits containing microcontrollers use a compiled firmware backend and cycle-accurate instruction-set simulator.The firmware and SPICE backends are mutually exclusive; components without an electrical model are tied to ground at 1012 Ω rather than omitted.Source: §3.3 · p. 4

4 Benchmark and protocol

• The benchmark contains 150 circuits across 20 categories and three difficulty levels, with 78 Persian, 51 English, and 21 Spanish prompts.It exercises 34 of 38 CIR types; 21 parallel triples hold the circuit constant across all three languages, and expected components and node counts provide a weak machine-checkable notion of component agreement.Source: §4 · p. 5
• The main comparison runs the full pipeline and a gpt-4o-mini direct-netlist baseline on all 150 circuits, with failover disabled.A second model arm is reported as a non-primary contrast.Source: §4 · p. 5
• A stratified 45-circuit subsample records each pipeline rung from the same model sample and evaluates snapshots through the same CIR-to-ngspice path.It is proportional by difficulty, covers all 20 categories, and reproduces the full-pipeline results closely; type canonicalisation is not ablated because it precedes schema validation, while ground inference is the ablatable deterministic rung.Source: §4 · p. 5

5.1 Four levels, two arms

• 88.7% of pipeline circuits were backend-executable, compared with 47.3% for the direct-netlist baseline; the four levels also distinguish schema validity and component-set agreement.The pipeline recorded 82.7% topological validity and 74.0% component-set agreement; those levels are undefined for the baseline because it has no typed representation.Source: §5.1 · pp. 5–6

5.2 The levels are not nested

• 16 circuits (10.7%) failed topological validation but ran in ngspice, while 7 (4.7%) passed validation but ngspice refused them.Twelve of the 16 contained exactly the requested components, with one terminal disconnected; the validator’s pin-count check misses nodes lacking DC paths.Source: §5.2 · p. 6

5.3 What the silent failures cost, measured

• A dangling resistor produced 5.00 V instead of 2.50 V, although ngspice reported no error.The comparison used a 5 V divider of two equal resistors; the open terminal carried no current, leaving the measured node at the supply rail.Source: §5.3 · p. 7

5.4 What the ladder did

• On the paired 45-circuit subsample, model repair raised topological validity from 40.0% to 84.4%, while executability changed by a net 6 circuits and component agreement by 2.There were 20 topological-validity gains and no regressions; the executability shift (+7, -1) was not resolved by this sample size. Ground canonicalisation changed one circuit and left executability unchanged.Source: §5.4 · pp. 7–8

5.5 The no-IR baseline

• The pipeline executed 88.7% of circuits versus 47.3% for direct netlists; even crediting the baseline with all 23 uncertain failures raised it only to 62.7%.Paired under the conservative bound, 46 circuits succeeded only with the pipeline and 7 only without it.Source: §5.5 · pp. 8–9

5.6 Language

• The paired comparison of 21 parallel circuit triples detected no reliable language effect; executability was identical across all three languages.The largest gap was two circuits out of 21, within what the sample could resolve.Source: §5.6 · p. 9

5.7 Cost

• The comparison arm used 1,594,489 tokens, approximately $0.42, with most usage attributed to repair traffic.Repair averaged 1.62 calls per circuit rather than one.Source: §5.7 · p. 9

6 A methodological caution

• The initial level-3 measurement was 94.0%, five points below the corrected figure, because the harness unconditionally requested an operating point.The deployed pipeline instead simulated relaxation oscillators in the time domain with initial conditions.Source: §6 · p. 9
• The harness and deployed pipeline now share one definition of oscillator component types; a duplicated definition caused their earlier divergence.The correction was larger than several effects measured in the paper.Source: §6 · p. 9
• The authors also use paired ablation, a measured divider experiment, and a conservative baseline bound to keep evaluation aligned with what each comparison supports.These choices are presented as part of the same measurement discipline.Source: §6 · p. 9

• Component-set agreement checks expected parts, not whether the circuit functions; the benchmark has no functional ground truth.It cannot distinguish a working amplifier from a broken one.Source: §7 · p. 10
• The 10.7% silent-error rate is specific to gpt-4o-mini, and that class empties out on a stronger model.The result is that simulation misses these errors when they occur, not that they occur at a fixed rate.Source: §7 · p. 10
• The deployed validator misses an identified DC-path class; testing DC path rather than pin count would address that limit.The figures describe the validator as deployed, before that fix.Source: §7 · p. 10
• The baseline is one direct-netlist protocol, not the strongest non-CIR alternative; longer prompts, few-shot examples, or a linter could raise it.The paper reports an adversarial bound for that reason.Source: §7 · p. 10
• Rates come from an author-constructed 150-circuit benchmark, while its named failure classes and diagnostics are presented as transferable.Source: §7 · p. 10
• In the 45-circuit paired ablation, topology gains (+20, -0) were well resolved, but the executability change (+7, -1) was not.Pairing was chosen to remove sampling noise, even at the cost of fewer circuits.Source: §7 · p. 10

• On gpt-4o-mini, 16/150 (10.7%) topologically invalid circuits ran cleanly, while 7/150 (4.7%) validator-passing circuits were rejected by ngspice.Twelve silent failures had the requested components but a disconnected terminal; a divider reported 5.00 V instead of 2.50 V.Source: §8 · p. 10
• On 45 paired circuits, repair raised topology validity from 40.0% to 84.4% (+20, no regressions), while execution's +7, -1 effect was unresolved.Component-set agreement changed by two circuits, and one circuit moved in opposite directions at two levels.Source: §8 · p. 10
• The pipeline executed 88.7%, versus 47.3% for direct netlists or 62.7% when unattrib­utable failures were credited to the baseline.Source: §8 · p. 10
• The paper recommends reporting structural validation and simulation separately, because a final executable percentage cannot show checker misses or intervention effects at each level.Source: §8 · pp. 10–11

• The silent-failure result is shown as two four-line netlists that any SPICE implementation can reproduce without the authors' code, models, or benchmark.Source: §9 · p. 11
• Findings in §§5.2–5.5 use gpt-4o-mini through its documented API, with the temperature and prompt structure described in the paper.The comparison model and interface are public.Source: §9 · p. 11
• Failure classes are paired with diagnostic messages so readers can look for the same classes in their own pipelines.The dataset, result files, and evaluation scripts are available from the author on request.Source: §9 · p. 11

• On the stronger model, repair ran on 3/150 circuits versus 67/150 for gpt-4o-mini, and ground canonicalisation ran on 1 versus 7.The stronger-model arm is not independently reproducible, and no paper claim depends on it.Source: A A stronger model, for contrast · p. 11
• The structurally broken-but-cleanly-simulating class was empty on the stronger model, but the validator's DC-path blind spot persisted on AC001.Both models produced a byte-identical diagnostic for that case.Source: A A stronger model, for contrast · p. 11
• Whether validation helps has no model-independent answer; the relevant question is which model and which evaluation level are being considered.Source: A A stronger model, for contrast · p. 11