Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks

Motivation

• Analog design requires expert iteration because a syntactically valid netlist can still implement the wrong function or miss a performance limit.The ability of general-purpose coding agents to complete the analog design workflow end to end remains underexplored.Source: §1 · p. 2

Research gap

• Prior LLM-based analog systems show feasibility, but do not jointly provide original expert tasks, multi-hour autonomous simulator use, and isolated specification-based verification.Analog Design Bench targets that missing combination.Source: §1 · p. 2

Evaluation findings

• Claude Fable 5 leads at 78.0%, while the remaining eleven of 15 configurations score below 50%.All four configurations above 50% use proprietary frontier models.Source: §1 · p. 2
• Coding and analog rankings correlate, but coding scores leave much of the analog-performance spread unexplained; 98.3% of non-passing attempts lack a recorded legality rejection and fail electrical acceptance.Longer runs and greater reasoning effort improve pass rates, while general-purpose skills have small or inconsistent effects; task-matched topology helps a weaker model substantially and mainly accelerates a stronger one.Source: §1 · p. 2

Contributions

• The paper introduces 50 expert-contributed transistor-level tasks, an open-source toolchain, isolated verification, and specification-based electrical tests.It evaluates 15 configurations across 2,250 two-hour attempts and examines factors shaping performance.Source: §1 · p. 2

Agentic and hardware benchmarks

• Other benchmarks cover software patches, terminal tasks, RTL generation, and agentic or non-agentic RTL design, verification, and comprehension.The cited examples include SWE-bench, DeepSWE, Terminal-Bench, VerilogEval, RTLLM, and CVDP.Source: §2 · p. 2

Analog design automation

• Analog Design Bench uniquely combines original expert-contributed tasks, two-to-six-hour tool use, autonomous simulation, and an isolated verifier among the listed analog benchmarks.Table 1 positions these properties against other analog benchmarks.Source: Table 1 · p. 3
• Prior analog automation includes reinforcement-learning, Bayesian-optimization, and graph-based sizing, as well as learned topology generation and simulator-iterating agent systems.Examples include AutoCkt, AnalogGym, AnalogCoder, AnalogCoder-Pro, AmpAgent, Atelier, and AnalogAgent.Source: §2 · p. 3

3.1 Task sourcing and coverage

• More than twenty designers proposed over 100 tasks; multi-stage review and trial runs retained 50 tasks from 17 experts.Each retained task has an independently verified reference result showing its contract is achievable without prescribing the agent’s topology.Source: §3.1 · p. 3
• The suite spans six circuit families, from power management and references to interfaces and drivers, with tasks ranging from a five-transistor OTA to asynchronous SAR ADCs.Data-conversion and sampling tasks also exercise switching, timing, and quantization.Source: §3.1 · p. 3

3.2 Task format and agent environment

• Tasks use the open-source SKY130 PDK and ngspice, and agents edit the DUT and diagnostic testbenches, simulate, and refine designs from measured results.The benchmark is schematic-level; layout and physical implementation are outside its scope, and each simulation takes at most roughly three minutes.Source: §3.2 · pp. 3–4

3.3 Verification and scoring

• Only the declared circuit crosses from the agent sandbox to an independent verifier, where legality checks precede specification-based electrical grading.The grading benches are hidden while their specifications are disclosed; every acceptance check must pass for a run to pass, with 11 to 2,083 checks per task.Source: Figure 2 · p. 4, §3.3 · p. 4

Main evaluation

• Claude Fable 5 [max] passes 117/150 attempts (78.0%), ahead of Claude Opus 5 [max] at 69.3% and GPT-5.6 Sol [max] at 68.0%.Across configurations, full-specification pass rates range from 8.0% to 78.0%.Source: §4 · p. 5, Table 2 · p. 5

Resource use

• Pass@1 has weak rank associations with iterations, tool calls, and recovered candidates, while recorded tool wait rises and mean runtime falls with pass@1.Passing attempts last a median of 72.0 minutes versus 118.9 minutes for failures; this does not mean stopping early causes success.Source: §4 · p. 5

Failure analysis

• Only 24 of 2,250 attempts (1.1%) receive a recorded legality rejection, while 1,364 of 1,388 non-passing attempts (98.3%) fail electrical acceptance without one.This category includes simulation failures, functional failures, and missed performance limits; it does not classify errors that agents encounter and repair during search.Source: §4 · p. 6, Figure 3 · p. 6

Coding benchmark comparison

• Coding and analog rankings are positively associated (Spearman ρ = 0.88), but six models scoring 67–70 on DeepSWE span analog pass rates from 31% to 78%.The benchmarks differ in protocol, harness, and budget, so the gap is not a calibrated transfer loss.Source: §4 · p. 6, Figure 3 · p. 6

5.1 Test-time scaling

• 14.0–20.0 percentage points: extending runs from two to six hours improves all five shown configurations, although some plateau while others continue improving.Figure 4a reports checkpoint-level pass rates from each run’s latest scored circuit.Source: §5.1 · p. 7, Figure 4 · p. 8
• At six hours, pass rates across five reasoning-effort settings range from 8.0–82.0%, and only Max and XHigh continue gaining after two hours.Under Low effort, unsuccessful runs make their final circuit revision within the first hour.Source: §5.1 · p. 7, Figure 4 · p. 8
• Harness pass rates narrow from 56.0–72.0% at two hours to 80.0–86.0% at six hours, a maximum difference of three tasks.The five model configurations span 54 percentage points at six hours.Source: §5.1 · p. 7, Figure 4 · p. 8

5.2 Design skills

• At six hours, the three general-guidance formats finish within 3.3 percentage points of baseline, while full-suite skill endpoints range from 53.3% to 62.0% versus 59.3% without a skill.The held-out split changes task-family mix; full-suite skills reuse information from evaluated tasks.Source: §5.2 · p. 7, Figure 4 · p. 8
• 18.7 percentage points: a task-matched reference topology raises DeepSeek V4 Pro [max]’s six-hour pass rate, while GPT-5.6 Sol [max] gains only 0.7 percentage points at six hours.The blueprint preserves topology and connectivity but leaves numerical sizing and electrical closure to the agent.Source: §5.2 · p. 7, Figure 4 · p. 8

5 Performance Factors

• Figure 5 traces four six-hour outcomes on one sampling-feedback OTA, including early pass, pass-then-regress, delayed pass, and no pass.Its curves show normalized margins for ten metrics drawn from the verifier’s 15 scoring gates.Source: Figure 5 · p. 9

Case trajectories

• GPT-5.6 Sol [max] in Codex passed every scoring gate at 98.2 minutes and retained the pass in its 99.4-minute final submission.The task couples settling, accuracy, noise, and common-mode control across process, supply, temperature, and mismatch.Source: §6 · p. 9
• GPT-5.6 Sol [max] in Kimi Code passed at 181.9 minutes but regressed, with its 265.0-minute submission failing output noise at 317.7 µV against a 300 µV limit.The verifier’s verdict is hidden, so the run cannot know it had passed.Source: §6 · p. 9
• GPT-5.6 Terra [max] in Codex first passed at 230.1 minutes and its 244.6-minute final circuit satisfied every check.It was still failing at the two-hour mark.Source: §6 · p. 9
• GPT-5.6 Luna [max] never passed: its last runnable circuit met only the power limit, and its final revision pushed the transistor model outside its valid range.The simulator returned no operating point, blocking every check; the last runnable circuit had 11.0% static error against 0.01%.Source: §6 · p. 9
• These cases illustrate that coupled specifications make search sensitive to topology, sizing, operating point, and measurement coverage, motivating physical diagnosis and regression testing across conditions.The verifier’s hidden verdict means a run that passes and then regresses cannot know it had passed.Source: §6 · p. 9

Discussion and conclusions

• DeepSWE scores of 67–70 accompany analog pass rates from 31% to 78%, while recorded legality rejections account for 1.1% of attempts.The paper concludes that coding scores are an incomplete proxy for analog design and that difficulty lies in electrical closure.Source: §7 · p. 9
• From two to six hours, pass rates rise by 14.0–20.0 points; reference topology raises DeepSeek V4 Pro [max] by 18.7 points and mainly accelerates GPT-5.6 Sol [max].General and trajectory-distilled skills move endpoints by −6.0 to +3.3 points; pass@1 has weak rank associations with iterations, tool calls, and recovered candidates.Source: §7 · p. 10
• The authors propose retrieving and adapting real circuit-library designs and improving closure through condition-margin checks, device-model validity, and retaining circuits that pass checks.The released benchmark and trajectories make both directions measurable.Source: §7 · p. 10
• The benchmark uses SKY130 at schematic level, excluding layout, parasitics, and silicon measurements; passing does not imply a commercially competitive chip.Its absolute circuit performance is not comparable to advanced commercial processes.Source: §7 · p. 10

Reproducibility Statement

• The planned benchmark release includes 50 task packages, electrical contracts, starter files, reference results, a verifier container, and eight skill packages.The verifier container pins ngspice and SKY130 versions, with electrical benches hidden during reported experiments.Source: Reproducibility Statement · p. 10
• Archived trajectories, circuit revisions, and verifier verdicts for every main-experiment and six-hour attempt are slated for online release.Analysis scripts regenerate result tables and quantitative plots from the frozen data snapshot.Source: Reproducibility Statement · p. 10
• Stochastic model sampling means reruns will not reproduce individual trajectories, though every reported number can be recomputed from archived runs.Future benchmark versions are intended to introduce held-out tests inaccessible to evaluated agents.Source: Reproducibility Statement · p. 10

• The shared tasks, open-source toolchain, and specification-based verification improve reproducibility and lower the barrier to studying agentic analog design.The toolchain avoids requiring commercial EDA licenses.Source: Broader Impact and Ethics Statement · p. 10
• A passing circuit meets the listed electrical specifications, but unconstrained component values, transistor multiplicity, or area may remain impractical.Source: Broader Impact and Ethics Statement · p. 10
• The released tasks use the open-source SKY130 PDK and contain no personal or proprietary information.Source: Broader Impact and Ethics Statement · p. 10

AI Use Statement

• AI assistance supported repository inspection, rollout-table validation and summarization, plotting and LaTeX scaffolding, and prose editing; authors verified the final work.Source: AI Use Statement · p. 10

A Extended related work

• Earlier learned analog-automation methods perform simulator-guided sizing or task-specific topology generation, but do not measure general-purpose agents completing unseen design tasks.The surveyed methods include Bayesian optimization, reinforcement learning, graph-based approaches, and topology generators.Source: A Extended related work · p. 15
• LLM-based systems add circuit knowledge, simulator feedback, specialized agents, or automated verification, but evaluate different task selections, budgets, technologies, and verification boundaries.Source: A Extended related work · p. 15
• Existing circuit datasets and benchmarks assess outcomes such as prediction, perception, code generation, or framework-specific specification satisfaction rather than the same general-purpose agents on a fixed suite with isolated re-simulation.Analog Design Bench admits any legal transistor topology and keeps verifier implementation and feedback inaccessible during attempts.Source: A Extended related work · p. 15

• Table A1 lists 50 scored design tasks, their one-line objectives, and condition-level electrical-check counts; Figure A1 also summarizes author contributions and measured electrical dimensions.Source: B Task inventory · p. 15, Table A1 · p. 16, Figure A1 · p. 16
• The suite spans signal-chain amplifiers and filters, from source-follower buffers and gain-stable differential amplifiers to Miller op amps and OTA-C biquads.Source: B Task inventory · pp. 16–17
• Power-management tasks include bandgap and current references, charge pumps, LDOs, and switched-capacitor conversion.Source: B Task inventory · pp. 16–17
• Conversion and sampling tasks include switched-resistor DACs, flash and SAR ADCs, delta-sigma conversion, and sample-and-hold circuits.Source: B Task inventory · pp. 17–18
• Other objectives cover RF and high-speed circuits, including a 2.4 GHz mixer, LNA, VCOs, CML driver, and frequency divider.Source: B Task inventory · pp. 16–18
• Table A2 summarizes performance by functional family, but its caption cautions that outcome-conditioned task selection prevents interpreting those values as intrinsic family difficulty.Source: Table A2 · p. 18

C Verification and data quality

• The verifier accepts a submission only when it is legal and every scoring gate passes; its contracts expand to 11–2,083 discrete electrical checks per task.Protected benches and PDK are isolated from agents, and failed or missing required analyses cannot count as passing.Source: C Verification and data quality · pp. 18–19
• The retained suite is outcome-conditioned, offers limited stochastic characterization, excludes industrial simulations lasting hours or days, and does not establish fabrication readiness when implementation costs are unconstrained.The case studies illustrate trajectories but do not estimate how prevalent those outcomes are.Source: C Verification and data quality · p. 19
• The cohort has complete task–configuration–rollout records, but scoring-gate counts vary from the task’s modal count in six rows, none of them full passes.The main experiment uses two harness arrangements, and seven configurations derive two-hour scores from six-hour runs at the same cutoff as standalone runs.Source: C Verification and data quality · pp. 19–20

D Resource accounting

• Tool wait includes simulator execution, dispatch, and harness overhead, but is a lower bound; recovered candidate circuits also remain incomplete under the capture methods.Candidate recovery combines trajectory reconstruction and retained snapshots, which can miss transient or scratch-file contents.Source: D Resource accounting · p. 21

E Task-level outcomes

• Figure A4 orders task columns by how many configurations solve them and configuration rows by leaderboard rank, with shading for mean gate reward and dots for any full pass.Source: E Task-level outcomes · p. 22, Figure A4 · p. 22

F Design skills

• The held-out guidance study distills general observations from the first 30 tasks for evaluation on the final 20, while broader skill packages reuse evaluated tasks and exclude reference topologies.General guidance ends within 3.3 percentage points of baseline at six hours; skill 7 yields 60.0% versus 59.3% without a skill, one additional pass in 150 attempts.Source: F Design skills · pp. 22–23
• The reference-topology library supplies 50 task-matched blueprints with tunable values replaced by placeholders, testing topology reuse under idealized retrieval rather than held-out transfer.Source: F Design skills · p. 23

G Case-study evidence

• In the Flash ADC case, GPT-5.6 Sol’s last submission passed transfer, SNDR, SFDR, and power, but non-monotonic fine-ramp behavior failed INL and DNL; the selected cases do not estimate prevalence.GLM first passed at 242.0 minutes and its final submission still passed; the figure displays direction-aligned ratios to specification.Source: G Case-study evidence · pp. 24–25, Figure A5 · p. 24