Can Agents Design Libraries for Agents?

• LibraryDesignBench evaluates libraries through downstream programs across 242 expert-validated problems, 15 design tasks, and four languages.Designers implement libraries from capability specifications that leave interfaces and abstractions open; user agents’ programs are scored for correctness and simplicity against production-library references.Source: §1 · pp. 1–2
• Opus 5.5 scored 48.9, 2.3 points above the production library, while designers reproduced production-library abstractions in 11 of 15 tasks.Source: §1 · p. 2
• Downstream agents underuse libraries; 64% of sampled excess-code cases involved rigid or hard-to-use interfaces, versus 14% involving missing capabilities.Source: §1 · p. 2
• With the production library, implementers reached only 61.5 simplicity; in LibraryUseBench, Opus 5.5 scored best at 66.9.The paper evaluates eight models as implementers to separate library-use ability from library design.Source: §1 · p. 2
• Consumer-first sketches, runnable examples, and subagent tests raised GPT-6 Astra’s score by 2.3 points, mainly through 6.8% higher simplicity, but remained below production.Source: §1 · p. 2

2 LIBRARYDESIGNBENCH

• LibraryDesignBench has one agent build a library and downstream agents solve tasks with it; scores combine correctness and simplicity against production-library references.The benchmark evaluates the designed library through downstream use rather than correctness tests alone.Source: §2 · p. 3, §2.1 · p. 3, §2.2 · p. 4

2.1 EVALUATING LIBRARIES THROUGH REAL OBSERVATION

• Design instructions leave abstractions open while specifying required functionality and giving visible usage examples; implementers are told to read documentation and build thin adapters.The visible problems remain in the scored evaluation set.Source: §2.1 · p. 3

2.2 SCORING THE QUALITY OF A LIBRARY

• The library score combines squared test-pass fraction with simplicity, prioritizing correctness while retaining graded credit for partially correct programs.A library that cannot be installed receives a score of zero across all problems.Source: §2.2 · pp. 3–4
• Simplicity averages capped reference-to-program ratios across cyclomatic complexity, cognitive complexity, Halstead volume, and formatted source lines.The fixed references use idiomatic production-library abstractions rather than code-golfed solutions.Source: §2.2 · p. 4

2.3 BENCHMARK AGGREGATION AND REPORTING

• The benchmark averages three runs, weights design tasks equally, and estimates rerun uncertainty by task while retaining each complete library evaluation as an observation.Keeping outcomes from the same library evaluation together accounts for shared effects across problems or implementers.Source: §2.3 · pp. 4–5
• Approximate 95% Student-t intervals describe rerun variation on this fixed benchmark, but three generations per task do not guarantee nominal coverage.Descriptive standard errors for pass rate, simplicity, cost, and tokens instead summarize variation across problems and runs.Source: §2.3 · p. 5

2.4 BENCHMARK CONSTRUCTION

• The benchmark contains fifteen tasks across four languages, with realistic, library-optional problems built from repository usages and reviewed by library experts.A multi-stage pipeline extracts library-focused programs, rewrites them without the library, tests behavioral agreement, and includes expert review and a second audit.Source: §2.4 · p. 6

3.1 AGENT DESIGN QUALITY

• The strongest designer exceeds the production-library baseline by 2.3 points (4.9% relative), and designers reproduce production-library abstractions on eleven of fifteen tasks.Source: §3 · p. 6
• Production libraries add 12.3 ± 0.4 points over no library at $0.14 ± $0.02 more per problem; no library reaches 86.4% test pass, within 0.2 percentage points of the highest mean.Source: §3.1 · p. 6
• Designers converge on production-library abstractions in eleven of fifteen tasks; in clirs, Astra and Fable use clap’s builder design rather than its shorter derive macro.Source: §3.1 · p. 6

3.2 FAILURE TAXONOMY

• Rigidity and verbosity account for 64% of sampled excess-code classifications, compared with 14% for missing capabilities.In clirs, 24 of 33 agent-written libraries do not easily implement long-option prefix inference; on tasks needing it, libraries lacking it score 16.7 points lower.Source: §3.2 · p. 7

3.3 HOW DO AGENTS USE LIBRARIES?

• Raising GPT-5.6 Luna’s reasoning effort increases pass rate from 51.4% to 84.5%, while the default prompt raises simplicity from 48.7 to 60.4 over the minimal prompt.With the minimal prompt, 24% of trials never open the library and 36% of reference library symbols go unseen, versus 5% under the default prompt.Source: §3.3 · pp. 7–8

3.4 DO AGENTIC DESIGN PATTERNS WORK?

• Agent-first guidance raises the score from 44.1 to 46.4, just below production at 46.6, and improves scores for all three implementers and three of four languages.The combined intervention nearly doubles design cost ($8.24 vs. $4.51) while per-problem cost is unchanged (−$0.01 ± $0.01).Source: §3.4 · p. 9

• Prior agent-code work covers repository generation, library refactoring, long-horizon work, and tools for agents, but tests or task accuracy can miss whether a requested library is actually used.ReGAL and LATM pass one model’s code to another but build small functions from solved tasks rather than libraries designed from open specifications.Source: §4 · p. 10
• Software-engineering studies evaluate APIs through developers’ performance or API-surface and complexity measures, while LLM benchmarks often fix the library and score model use.LibraryDesignBench instead evaluates agents as library users through correctness and code size against an expert reference; LibraryUseBench applies that measure to production libraries.Source: §4 · pp. 10–11
• Prior agent-oriented evaluations vary consumers, frameworks, documentation, or tools, whereas LibraryDesignBench varies the designer to measure library design for other agents.The authors describe it as, to their knowledge, the first benchmark to measure how well an agent designs a library for other agents.Source: §4 · p. 11

• Scores describe library value under specified tasks, consumer configurations, and execution budgets, not a consumer-independent ranking.Source: §5 · p. 11
• The benchmark measures tested correctness and static consumer-program size and complexity, not full library correctness, security, runtime efficiency, or maintainability.Source: §5 · p. 11
• Reference programs normalize scores but are not uniquely optimal, and confidence intervals cover reruns of the fixed task set rather than generalization to all library domains.Source: §5 · p. 11

6 CONCLUSION

• Opus 5.5 scores 4.9% above the production library across fifteen tasks, while designers reproduce production-library abstractions in eleven tasks.Downstream agents underuse these libraries; most sampled excess-code cases stem from rigid or hard-to-use interfaces rather than missing capabilities.Source: §6 · p. 11
• Most sampled excess-code cases reflect rigid or hard-to-use interfaces, while only 14% are classified as missing capabilities.Downstream agents underuse the libraries despite reproducing production-library abstractions in eleven of fifteen tasks.Source: §6 · p. 11
• Agent-first guidance and subagent testing yield libraries that copy fewer production designs and shrink downstream programs, yet still fall just short of the production library.The conclusion identifies designing libraries for agents as an open problem.Source: §6 · p. 11
• LibraryDesignBench scores agent-designed libraries by the correctness and simplicity of downstream programs against expert references built on production libraries.The authors propose it as a testbed for studying interfaces, abstractions, and documentation that help agents build on one another’s code.Source: §6 · p. 11

• Code is formatted to 80 columns before measurement; formatting failures score zero, and parse failures leave the readable portions measurable.A single tree-sitter pass counts metrics over source files, skipping build output, dependencies, and symlinks.Source: A STATIC MEASUREMENT DETAILS · p. 16
• SLOC excludes comments and Python docstrings; complexity metrics count functions, branches, nesting, and logical operators, while Halstead volume uses operator and operand counts.Cyclomatic complexity adds one per function, branch, and && or ||; cognitive complexity adds for branches and nesting; Halstead volume is N log2 η.Source: A STATIC MEASUREMENT DETAILS · p. 17
• Python rules count constructs such as comprehensions and asserts as branches or complexity increments, while lambdas do not count as functions.Match affects cognitive complexity; elif adds one to each complexity without nesting cost.Source: A STATIC MEASUREMENT DETAILS · p. 17
• JavaScript and TypeScript share counting rules: functions include arrows and methods, while ?? adds only to cyclomatic complexity.An else-if chain is counted as nested; TypeScript submissions also measure JavaScript files they use.Source: A STATIC MEASUREMENT DETAILS · p. 17
• Rust counts match once without adding for its arms; labeled breaks or continues and recursive calls add cognitive complexity.Code inside macros and the ? operator counts as zero.Source: A STATIC MEASUREMENT DETAILS · p. 17
• Haskell counts each function equation toward cyclomatic complexity, while guards and case alternatives follow specified branch and nesting rules.The first guard adds one plus nesting depth to cognitive complexity; later guards add one each.Source: A STATIC MEASUREMENT DETAILS · p. 17

B SETUP DETAILS

• Designers and implementers run in mini-SWE-agent unless specified otherwise, with shared Docker environments, task limits, and pinned measurement normalizers.LibraryUseBench evaluates eight models on the production library; three implementer models also solve the benchmark tasks.Source: B SETUP DETAILS · p. 17, Table 4 · p. 18, Table 5 · p. 18

C LIBRARY PACKAGING INSTRUCTIONS

• Packaging instructions request expected library functionality and specify language-specific package layouts, while leaving interfaces, module decomposition, and abstractions unconstrained.The design freedom is what LibraryDesignBench measures.Source: C LIBRARY PACKAGING INSTRUCTIONS · pp. 17–18

D MAIN PROMPTS USED

• The library-use prompt asks for a thin adapter: map task requirements to library entry points, prefer whole-operation primitives, and audit remaining hand-written code.Reimplementation is a last resort after searching for missing functionality; the prompt also directs implementers to read documentation and examples.Source: D MAIN PROMPTS USED · p. 19
• The general implementer prompt asks for a small, direct solution, available dependencies, exact task validation, and successful sample runs.Sample runs are sufficient; the prompt says to skip test suites.Source: D MAIN PROMPTS USED · p. 20

• Table 7 lists each of the fifteen library-design problems, its production-library reference, and the number of Evaluation Phase problems.The library name is also the name under which the designer packages it and the name used in the paper.Source: E BENCHMARK PROBLEMS · p. 20, Table 7 · p. 20

F TAXONOMY

• Each generated library is audited on three downstream cells that both fail at least one test and exceed reference code; fully passing cells are excluded.The same cells yield 810 excess-code and 810 failed-test classifications.Source: F TAXONOMY · p. 20
• Each cell receives one same-family model audit without human labels or an agreement check, and the auditor reasons about the best as-shipped path rather than running it.The audit attributes failures to library changes that would have prevented them, including implementer task-logic errors; reported shares describe partially passing cells, not every downstream cell.Source: F TAXONOMY · pp. 20–21

G LIBRARY USAGE PROMPTS

• The low-prescription prompt requires library use and minimal code, while the medium-prescription prompt additionally directs implementers to consult examples before coding and run examples afterward.Both prompts specify available dependencies and a one-hour limit.Source: G LIBRARY USAGE PROMPTS · p. 22
• The prompts tell implementers to use the highest-level fitting interface, search the library before hand-writing operations, and reread their code to remove work the library can do.They also prescribe reading the library’s examples and documentation before coding.Source: G LIBRARY USAGE PROMPTS · p. 23

H HOW AGENTS SEARCH LIBRARIES

• The interaction analysis covers 14 runs, 10,164 trials, and 412,802 shell commands; spot checks match manual labels on 59 of 60 commands and import extraction on 30 of 30.Seen-symbol rates are upper bounds because a symbol counts as seen when it appears in library-read output.Source: H HOW AGENTS SEARCH LIBRARIES · p. 23
• Under the minimal prompt, 24% of Luna trials never touch a library and another 24% only list it or print its version; 92% of skips still import it from memory.Identifier-shaped searches dominate, and the grep→source→grep loop appears in 38–80% of trials in every run.Source: H HOW AGENTS SEARCH LIBRARIES · p. 23
• More prescriptive prompts prompt documentation reading before the first write; 56% mention a post-run audit, but only 40% reread the library and fewer than 11% do so soon after a failing run.Symbols agents had not seen become symbols they use, while seen-but-unused symbols remain at 17–22%.Source: H HOW AGENTS SEARCH LIBRARIES · p. 23
• Low-effort agents make 88–91% of their first “library lacks X” claims before writing, and only 22–38% of sampled claims follow a capability grep.In one tantivy problem, a truncated search missed tokenizer registrations; the low-effort attempt scored 0, versus 79–96 for six medium- and high-effort attempts.Source: H HOW AGENTS SEARCH LIBRARIES · p. 24
• Agents mostly copy documented import idioms rather than search for short names; deeper reading raises deep Rust imports from 4% to 21% between low and high effort.Root or prelude imports appear in 91% of Python, 70% of Rust, and 47% of Haskell cases.Source: H HOW AGENTS SEARCH LIBRARIES · p. 24

I EXPLICIT GUIDANCE PROMPT

• The agent-first style, called neuralese, favors one intent-matching verb, task-shaped arguments and results, useful defaults, and edge-case handling inside the call.It favors dense examples and broad flat interfaces when they make the agent’s program shorter than layered abstractions.Source: I EXPLICIT GUIDANCE PROMPT · pp. 24–25
• The guidance prompt gains 2.5 points from simplicity and loses 0.9 from correctness.The contributions use a symmetric split of paired score changes, averaged over tasks and then languages with equal weight.Source: I EXPLICIT GUIDANCE PROMPT · p. 24
• The design workflow starts with downstream program sketches, derives public API from their calls, tests designs with fresh subagents, then freezes runnable sketches as examples.The README puts examples first, followed by reference material and packaging instructions.Source: I EXPLICIT GUIDANCE PROMPT · p. 25

J TIMEOUTS

• 2.3% of Evaluation Phase trials end through budget exhaustion or library-installation failure.Agents run out of time or budget in 1.7% of trials, and authored libraries fail to install in 0.6%.Source: J TIMEOUTS · p. 26
• Dropping timeout and installation-failure trials raises the mean score by at most 1.2 points per arm.The benchmark keeps these trials and grades whatever the agent produced when it runs out of time or budget.Source: J TIMEOUTS · p. 26
• Excluding those trials does not change the ordering of the no-library, agent-authored, and production-library arms.Source: J TIMEOUTS · p. 26