SGAnalog: An End-to-End Circuit Benchmark from Open-Source Silicon Tapeouts

1 Introduction

• Existing benchmarks make it difficult to assess both possible training overlap and circuit function under original test conditions.The field lacks a broad set of human designs supporting structural comparison and execution under original conditions.Source: §1 · pp. 1–2
• Tiny Tapeout submissions provide pinned revisions, transistor-level artifacts, PDK context, and author testbenches across varied circuit families.Shuttle dates support model-specific cutoff analysis, and new shuttles provide a refresh path.Source: §1 · p. 2
• The benchmark targets schematic-level reasoning rather than layout, subjective design style, or topology generation.Topology generation remains future work because it requires a uniform design brief and automatic testbench alignment.Source: §1 · p. 2
• SGAnalog pairs graph-scored schematic transcription with simulator-scored device sizing against human references under the same PDK and author testbench.The benchmark uses a deterministic pipeline and aligned schematic images and netlists generated from the same source.Source: §1 · p. 2
• Its exposure-aware protocol combines commit dates and model-specific cutoffs with closed-book evaluation, blinded renders, release canaries, and shuttle refreshes.The protocol is described in Section 5.Source: §1 · p. 3
• The seven-model baseline shows a broad small-to-medium transcription drop and different rankings across transcription and sizing.Six of seven models drop sharply between the small and medium tiers.Source: §1 · p. 3

2 Related work

• Table 1 defines coverage indicators for numeric device sizes, author testbenches, open PDK flows, partial coverage, and contamination audits.Reported scale follows each source’s own unit and is therefore not directly comparable.Source: Table 1 · p. 3
• Prior image-based sets AMSnet and Masala-CHAI [13] pair scanned textbook figures with manually or model-assisted extracted netlists.This benchmark instead generates each eligible image–netlist pair from one schematic source, avoiding a separate annotation step.Source: §2 · p. 3
• AnalogGym [3] is a close simulator-backed comparator, but its amplifier topologies come from published literature rather than submission-author designs.SGAnalog evaluates the author’s sizing under the same PDK and testbench as the model; NotSoTiny [14] targets digital RTL from Tiny Tapeout.Source: §2 · p. 3

3 Dataset construction

• Design sources are fetched at the shuttle-indexed commit, with licensing checked from the license text at that revision.Blobless triage uses file trees and licenses before design data are downloaded.Source: §3 · p. 3
• A top-level design is a circuit that no other non-testbench schematic instantiates.The classification distinguishes testbenches, circuits, and top-level designs by role rather than counting schematic files or hierarchy tops alone.Source: §3 · p. 3
• Strict Weisfeiler–Lehman hashes of top-level device–net graphs, computed after removing net names, identify candidate duplicate groups for split control.Task 1 scoring uses explicit graph isomorphism rather than hash equality.Source: §3 · p. 3
• Table 2 traces the dataset spine from the shuttle index to the fixed task sets, with Appendix B providing the broader inventory.The appendix covers triage, licensing, composition, and redundancy.Source: Table 2 · p. 4
• All stages run in one pinned container image; classification tables regenerate byte-identically across complete runs.Simulation tables also regenerate identically apart from per-run timing columns.Source: §3 · p. 4
• The fixed task sets contain 66 eligible transcription tasks and 17 sizing tasks selected from 190 eligible testbench runs.Sizing eligibility follows a 600-second completion gate, real device cards, hierarchy resolution, end-to-end DUT mapping, and a primitive-only hierarchy requirement.Source: §3 · p. 4

Task 1: schematic-to-netlist transcription

• Transcription scores image-derived netlists by graph isomorphism, while sizing holds topology fixed and tests proposed W/L values against human sizing in the same author testbench.Transcription also provides a common evaluation across circuits regardless of testbench state.Source: §4 · pp. 4–5
• 51.5% to 36.4% exact isomorphism on claude-opus-5 after blinding, with structural F1 unchanged and parameter accuracy improved.The scorer ignores names, so the result is consistent with labels serving as visual anchors for connectivity rather than as part of the answer.Source: §4 · p. 4

Task 2: device sizing

• Every Claude or Gemini proposal that supplied sizes converged; on answered tasks claude-opus-5 scored 97.2 and claude-fable-5 90.2, while refusals numbered 4 and 11 of 17.A proposal without sizes scores zero; sizing scores compare eligible metrics against the human reference under the author testbench.Source: §4 · p. 5, Table 3 · p. 6
• Sizing scores remain author-testbench-relative; human sizes are public, and per-class measurement templates are a next refinement.A proposal without sizes receives zero because the unmodified deck retains the human sizes and would otherwise grade the reference against itself.Source: §4 · pp. 5–6

Evaluation protocol

• The harness evaluates API requests closed-book and uses deterministic scoring; scorer corrections address symmetric device pairing and unoriented resistor pins.The corrected scorer still rejects genuinely misread connections, including explicit bulk ties replaced by rail defaults.Source: §4 · p. 6

Baseline

• Six of seven models drop sharply from small to medium; claude-fable-5 leads exact matches and structural F1, while claude-opus-5's failures are more severe than gpt-5.6-sol's.The sol model has the second-best structural F1 despite a lower exact-match rate, making the preferred model dependent on whether a downstream system can repair netlists.Source: §4 · p. 6

Exposure protocol

• The exposure protocol combines pinned commit dates, blinded renders, release canaries, and closed-book evaluation, while public repositories prevent assuming training-data independence.The canary marks generated artifacts but cannot protect already-public upstream sources; commit-date splits and blinded renders address distinct channels.Source: §5 · pp. 6–8

Commit-date analysis

• Commit dates can be compared with reported model cutoffs, but the fixed-set comparison shows no consistent pre-cutoff exact-match advantage across two usable older-cutoff comparisons.Claude-opus-5 has no post-cutoff tasks in the fixed set; difficulty differs across date boundaries, graph-hash groups cross them, and earlier design versions may predate pinned revisions.Source: §5 · pp. 6–7

Training-time and inference-time channels

• Closed-book baselines remove retrieval by declaring no tools, while search- or repository-enabled systems belong in a separately labeled open-book track.Blinded renders remove author-selected labels; reproducing an invisible name can indicate memorization or external retrieval.Source: §5 · pp. 6–8

Answer-key exposure

• Transcription receives an image without its netlist, whereas sizing repositories contain human sizes, so sizing uses perturbed operating points and regurgitation diagnostics.Figure 3 presents the two-task results and shows that transcription and sizing model rankings differ.Source: Figure 3 · p. 7, §5 · p. 8

Scope

• Most entries target Sky130; broader GF180 and IHP SG13G2 coverage, including SiGe, is planned.Schematics with unresolved host paths are excluded from rendering-sensitive splits, and top-level designs without parseable netlists remain outside graph-hash analysis.Source: §6 · p. 8
• Circuit-class labels are keyword-derived and are slated for reclassification from netlist graphs.The current classification is therefore not yet graph-derived.Source: §6 · p. 8

Roadmap

• The archival roadmap adds per-class measurement templates and per-circuit memorization probes; topology generation is deferred because uniform design briefs and automatic pin-matching remain unsolved.Initial amplifier templates will cover gain, gain-bandwidth product, and phase margin, widening sizing coverage beyond the leaf-subcircuit set.Source: §6 · p. 8

Licensing

• Redistributed designs are Apache-2.0, with licenses verified from texts archived at pinned commits.The Tiny Tapeout index is CC0, and the PDKs are Apache-2.0.Source: Ethics, licensing, and attribution · p. 8

Attribution and release

• The attribution file credits repositories and authors, while per-circuit manifests record source URL, commit, license, and extraction method.Benchmark code and released data use Apache-2.0, matching the source licenses.Source: Ethics, licensing, and attribution · p. 8

LLM policy

• An LLM assistant helped build the harvest and evaluation harness and draft the manuscript; authors verified released numbers against committed tables.The release includes a consistency checker enforcing the match.Source: Ethics, licensing, and attribution · p. 8

• Figure 4 plots exact isomorphism for seven models across difficulty tiers, supplying the per-tier view behind Section 4’s tier claims.Source: A Difficulty-tier breakdown of Task 1 · p. 9

• Table 4 expands Table 2’s inventory to include the complete harvest, licensing, composition, and redundancy record.Source: B Full dataset inventory · p. 9, Table 4 · p. 10
• Figure 4 plots exact isomorphism by difficulty tier (n = 9, 38, 19): five models drop from small to medium, while Haiku is already at floor.Source: Figure 4 · p. 10
• The inventory’s composition covers role, transistor-count tier, era, and device–net graph hash.Source: Table 4 · p. 10

• For each verified pair, the browser shows circuit and testbench schematics, extracted netlists, the pinned upstream repository, and published specifications.Source: C Simulation browser · p. 10
• Browser-started runs use the pipeline’s pinned-container entry point, making results bit-identical to the batch simulatability check.Source: C Simulation browser · p. 10
• The static companion page provides precomputed runs for ten representative pairs, one per circuit class.Source: C Simulation browser · p. 10
• Figure 5 illustrates a verified differential-amplifier pair with complementary output sines around the input common mode.Source: Figure 5 · p. 11