A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

1 INTRODUCTION

• On AgenticBBO-Bench’s five-task frontier challenge, current frontier LLMs differ substantially in overall optimization performance, while models with similar performance can incur greatly different official list-price token costs per task.Source: Figure 1 · p. 2, §1 · p. 3
• Agentic BBO lets the LLM inspect the search state, use code or domain tools, and choose intermediate actions online before submitting a candidate.This contrasts with methods that use the LLM as a fixed component performing a predefined operation.Source: §1 · p. 2
• Existing studies use different tasks and system configurations, making it difficult to compare results or tell general findings from task- or design-specific ones.Agent behavior also depends on observed information, available tools, and the interaction harness.Source: §1 · p. 2
• AgenticBBO-Bench evaluates LLM agents across five optimization domains under a common finite-budget protocol.The domains are synthetic numerical optimization, hyperparameter optimization, database tuning, chip design, and molecular design.Source: §1 · pp. 2–3
• Task semantics are broadly beneficial, but additional numerical tools do not consistently help and specific priors depend strongly on the problem.The studies also find that numerical methods can sometimes continue effectively from agent-guided search trajectories.Source: §1 · pp. 2–3
• The paper also establishes a five-task frontier challenge as a standardized leaderboard for general-purpose optimization agents.It benchmarks frontier LLMs under a common setting and supports future comparisons of models and agent systems.Source: §1 · p. 3

• Each round selects a candidate using available information, evaluates it, appends the observation to history, and repeats until the finite budget is exhausted.Source: §2 · p. 3
• BBO methods choose candidates from observed history in different ways, ranging from predefined procedures to learned decision rules and LLM-conditioned search.Classical methods include random search, evolutionary algorithms, and Bayesian optimization; learning-to-optimize methods learn part or all of the decision rule from related or synthetic tasks.Source: §2 · p. 3
• Agentic BBO lets an LLM organize intermediate decisions before each expensive evaluation, unlike traditional Bayesian optimization’s fixed optimizer-driven procedure.The actions shown for the agent are representative rather than exhaustive; the agent may choose, skip, repeat, or combine them.Source: Figure 2 · p. 4

3 AGENTIC BLACK-BOX OPTIMIZATION

• In Agentic BBO, the agent chooses and combines a variable sequence of intermediate actions before submitting each candidate for evaluation.Actions can include reading task information, analyzing prior evaluations, using tools, and planning the next candidate.Source: §3 · p. 4
• Unlike methods that assign an LLM a predefined role, Agentic BBO lets the agent determine the intermediate procedure online and use, modify, or ignore optimizer outputs.The agent can also write code to build task-specific analyses or optimization tools during search.Source: §3 · p. 4

3.1 EVALUATING AGENTIC BBO SYSTEMS

• Agentic BBO results depend on the full system configuration, and differing tasks and evaluation settings make existing studies difficult to compare directly.The benchmark fixes the outer interaction and evaluation protocol while varying task information, tools, and the degree of LLM control to support comparison and controlled analysis.Source: §3.1 · p. 5

4.1 TASK SUITE

• The suite spans BBOB, Bayesmark, DBTune, BBOPlace-Bench, and GuacaMol across synthetic optimization, hyperparameter optimization, database tuning, chip design, and molecular design.Together the tasks cover continuous, mixed, and structured search spaces, with varying semantics and feasibility constraints; Table 1 summarizes task-family characteristics.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 COMMON OPTIMIZATION PROTOCOL

• All tasks and methods use a common sequential protocol with the same initial observations and objective-evaluation budget for each task and seed.Only objective evaluations count toward the budget; model inference, code execution, tool calls, and candidate validation do not.Source: §4.2 · p. 5

4.3 EVALUATION

• The benchmark evaluates best-so-far performance across the full optimization trajectory rather than only the final solution.The trajectory average includes the final best-so-far value, which also receives an additional 30% weight in the overall score.Source: §4.3 · pp. 5–6
• The benchmark score weights average normalized best-so-far quality over the trajectory at 70% and final best-solution quality at 30%.Objective values are normalized because task scales differ, and the same scoring rule is used across performance experiments.Source: §4.3 · p. 6

5.1 EXPERIMENTAL SETTING

• Matched initialization and evaluation budgets support comparisons between Direct and Agentic, using DeepSeek V4.1 Flash with the Codex harness.Benchmark-task ablations drew subsets from a common eight-task pool.Source: §5.1 · p. 6

5.2 MAIN RESULTS

• Agentic improved over Direct on all five families by 30.3% on average and outscored numerical baselines on four; TuRBO led on DBTune.Agentic achieved the best score on BBOB, BBOPlace, HPO, and GuacaMol.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3 DO OPTIMIZATION TOOLS IMPROVE AGENTIC BBO?

• Base Agentic scored 0.589, versus 0.574 for + GP Suggest and 0.538 for + GP Tools; on Bigblue1, proxy-guided Python refinement raised the score from 0.551 to 0.676.The agent fit a linear proxy from observed layouts and used the pattern to construct a new placement.Source: §5.3 · p. 7

TAKEAWAY

• Across the studies, task semantics were broadly beneficial, additional numerical tools were inconsistent, and numerical optimizers could continue effectively from agent-guided search.Agents used trajectory-derived, task-specific analyses rather than simply handing search to a fixed optimizer.Source: TAKEAWAY · pp. 7–9

5.4 WHAT TASK INFORMATION CAN AGENTS EXPLOIT?

• On six real-world tasks, semantics raised the mean score from 0.465 to 0.595, while adding priors reduced it to 0.569 and helped some tasks but hurt others.The additional priors improved Bigblue1 and Sysbench-5 but hurt PostgreSQL-5 and Adaptec1.Source: §5.4 · pp. 7–8
• On controlled objectives, correct support plus geometry raised the mean from 0.685 to 0.872; incorrect support reduced it to 0.682, while either correct prior alone gave smaller gains.The study varied information about support and local geometry on three anonymous 12-dimensional objectives with four active variables.Source: §5.4 · p. 8, TAKEAWAY · p. 8

5.5 WHERE SHOULD THE LLM ACT IN THE OPTIMIZATION LOOP?

• GP-policy control scored 0.516 versus 0.569 for Agentic; handing search to TuRBO after agent warm-up scored 0.570, above TuRBO from initial observations at 0.475.The offline-developed optimizer scored 0.501 on held-out tasks, above Fixed GP at 0.451 but below online Agentic, with substantial variation across programs.Source: §5.5 · pp. 8–9

5.6 FIVE-TASK FRONTIER CHALLENGE

• Across seven models and five representative tasks, GPT-6 Astra scored 0.591 and DeepSeek V4.1 Flash 0.584, but no model led every domain.Performance was only weakly aligned with official token cost; the compact challenge retains diversity across optimization domains.Source: §5.6 · p. 9

6 CONCLUSION

• Across five optimization domains, Agentic configuration consistently improved over Direct and was competitive with established numerical optimizers.The controlled studies found that additional tools do not always help, semantics are broadly useful, specific priors are less reliable, and numerical optimizers can sometimes continue from agent-established trajectories.Source: §6 · pp. 9–10

AI USE STATEMENT

• Generative AI tools assisted with benchmark-code development and editing the manuscript; the authors reviewed and verified all assisted outputs.The authors take responsibility for the final content, code, and artifacts.Source: AI USE STATEMENT · p. 10

BBOB

• BBOB uses noiseless COCO functions that span landscape properties including separability, conditioning, variable interactions, and multimodality.The objective is evaluated directly with the corresponding analytic benchmark function.Source: §A.1 · p. 15

Hyperparameter optimization

• Hyperparameter tasks specify a model, dataset, and mixed-type search space; each candidate is a complete configuration scored by predictive performance.The resulting model is evaluated on the fixed task.Source: §A.1 · p. 15

Database tuning

• Database tuning searches mixed-type configuration knobs using fixed learned surrogate models built from measurements of the underlying system.The surrogate benchmark supports repeated, reproducible evaluations.Source: §A.1 · p. 15

Macro placement

• Macro-placement candidates specify movable-macro positions, decoded into layouts that must satisfy physical constraints before objective evaluation.Experiments use the compact 32-macro setting.Source: §A.1 · p. 15

Molecular optimization

• Molecular optimization searches valid SMILES strings using task-specific scoring functions over structured discrete objects.This differs from the vector-valued search spaces used in the other benchmark families.Source: §A.1 · p. 15

A.2 BENCHMARK INTERACTION PROTOCOL

• Agents use a shared containerized workspace to inspect benchmark-visible files, run local analysis, and access benchmark operations, while the evaluator and hidden task state remain host-managed.The shared interface exposes task context, search spaces, trial history, incumbents, and candidate submission; the information available can vary by experimental condition.Source: §A.2 · pp. 15–17
• Only accepted candidates evaluated by the host consume objective budget; invalid submissions may be revised within the same round, while local computation and queries do not use evaluations.The host returns each accepted candidate’s result to the persistent session for the next round.Source: §A.2 · p. 16

A.3 SCORE NORMALIZATION AND AGGREGATION

• For tasks without a known finite optimum, normalization uses shared initialization and a matched GP endpoint, assigning quality 0 to the initial best and 0.6 to the GP result when it improves.Beyond the GP reference, quality approaches 1 smoothly; if GP does not improve, its midpoint is omitted and the initial-observation interquartile range is used.Source: §A.3 · pp. 18–19
• For tasks with a known finite optimum, normalization uses that optimum as an upper reference and omits the GP midpoint when it is not a distinct intermediate anchor.Molecular tasks use a task-specific upper reference without a GP midpoint.Source: §A.3 · p. 19
• Run scores are averaged across matched seeds within each task, then tasks receive equal weight within a cohort; reported ranks are derived from seed-averaged task scores.Incomplete runs carry their last observed incumbent through remaining scoring checkpoints without adding evaluations.Source: §A.3 · p. 19

B.1 AGENT CONFIGURATIONS

• Direct returns candidates from task descriptions and observed history, whereas Agentic also has a persistent Codex workspace with Bash/Python and structured optimization interfaces.Both configurations use the same underlying model and harness.Source: §B.1 · p. 20

B.2 TASKS AND EVALUATION BUDGETS

• 77 tasks span five families: 24 BBOB functions, 25 HPO tasks, six database-tuning tasks, 12 macro-placement tasks, and ten molecular-design objectives.The task panels in controlled diagnostics can overlap, so those results are complementary diagnostic views rather than independent enlargements of the broad benchmark.Source: §B.2 · p. 20

B.3 NUMERICAL BASELINES

• Baselines include GP-BO, TuRBO, CMA-ES, TPE, GIT-BO, Sobol, and Random Search, while molecular optimization uses GPBO and Graph-GA.The numerical methods span Bayesian optimization, evolutionary search, and non-adaptive search; the molecular baselines address candidates represented as complete SMILES strings.Source: §B.3 · pp. 20–22
• Within each matched comparison, methods receive the same evaluated initialization and number of objective evaluations, with internal computation and tool use excluded from that budget.Numerical methods use the same task-specific search spaces and host evaluation interface as agent methods.Source: §B.2 · p. 20, §B.3 · p. 21

Frontier task selection

• The five-task frontier panel selects one representative task per family, including an eight-hyperparameter Optical Digits MLP task, a 196-parameter MySQL surrogate, Median1 molecular design, BBOB f15, and Bigblue1 placement.Task cards expose objective, budget, candidate representation, and an index of additional information retrievable on demand.Source: §B.4 · p. 22

HPO

• The HPO card asks agents to tune eight MLPClassifier hyperparameters on Optical Digits to reduce cross-validation classification error.The task details specify five-fold stratified validation on a fixed 1,437-example training split and describe interactions among several hyperparameters.Source: §B.4 · pp. 22–23

Database tuning

• The DBTune task optimizes 196 MySQL settings for throughput predicted by a fixed surrogate rather than a live database workload.Parameter descriptions provide semantics, but do not establish importance under the surrogate; surrogate weights and its construction configuration remain hidden.Source: §B.4 · pp. 23–24

Molecular design

• The molecular task seeks a molecule simultaneously similar to camphor and menthol, proposing one SMILES string per evaluation.Its geometric-mean score rewards similarity to both references, while only the combined loss is returned to the agent.Source: §B.4 · p. 24

BBOB

• The anonymous BBOB task minimizes an unknown scalar objective over ten bounded numerical parameters without function-specific information.The agent must infer useful search structure from observations and general numerical reasoning.Source: §B.4 · p. 25

Macro placement

• The Bigblue1 task minimizes HPWL for a fixed 32-macro placement, with coordinates legalized before scoring and possible displacement of macros during repair.The repair is deterministic and geometry-only; it does not inspect the hidden netlist or wire cost.Source: §B.4 · p. 25

B.5 ABLATION SETTINGS

• Ablations use matched tasks, initializations, evaluation budgets, models, runtime configurations, and reporting protocols unless specified otherwise.The controlled studies select representative tasks from non-molecular benchmark families; molecular tasks are excluded.Source: §B.5 · p. 26
• The tool ablation compares base Agentic, which has no benchmark optimizer interface, with + GP Suggest and + GP Tools, while the agent retains final candidate authority.The augmented conditions share a GP backend; optimizer queries do not consume objective-evaluation budget, and the agent may accept, alter, or ignore suggestions.Source: §B.5 · pp. 26–27
• The task-information study contrasts anonymous coordinates (I0), task semantics without optimization-oriented priors (I1), and I1 plus mechanism and domain-knowledge context (I2).I0 changes representation and semantic information, whereas I1–I2 more directly isolate added domain knowledge at fixed representation.Source: §B.5 · pp. 27–28
• The controlled-prior study varies prior content across anonymous 12-dimensional objectives with four active variables, holding task files, tools, observations, and budgets fixed.Conditions include active-variable count, geometry, correct or incorrect support, and combinations; formulas, optima, and support mappings remain host-only.Source: §B.5 · pp. 28–29
• The LLM-role study compares full-budget agent selection with handoffs to numerical optimizers after a shared agent-generated prefix, and separately tests LLM control of GP policy.Handoff branches inherit the same prefix, while full-budget numerical controls start from shared initialization; in GP-policy control, the LLM chooses policy and the optimizer chooses candidates.Source: §B.5 · pp. 29–30
• The optimizer-development experiment asks agents to turn development-set experience into a reusable Python optimizer, starting from GP-EI and targeting a weighted development score.Development and held-out tasks are separated for BBOB and HPO; the score weights normalized anytime improvement at 70% and normalized final improvement at 30%.Source: §B.5 · p. 30
• Diagnostic ablations use task-specific evaluation budgets and seeds, while optimizer development selects among revisions on development tasks before held-out evaluation without LLM calls or source changes.Budgets are 20 + 50 for BBOB, 5 + 25 for HPO, and 50 + 50 for DBTune and BBOPlace; controlled-prior experiments use 16+48 evaluations. Development uses seeds 0–1, held-out evaluation uses seeds 2–5, and budgets count host objective evaluations.Source: §B.5 · p. 31

C FUTURE DIRECTIONS

• Future agents could select or combine complementary optimizers according to task fit and the current search state rather than rely on fixed tools.The value of an optimization tool depends more on its match to the task than on its isolated power.Source: C FUTURE DIRECTIONS · p. 31
• Future agents could actively discover useful priors—such as important variables, interactions, and promising regions—and acquire knowledge that guides concrete search decisions.Richer prior knowledge is useful only when it can guide concrete optimization choices.Source: C FUTURE DIRECTIONS · p. 32
• Cost-aware routing could decide when continued LLM reasoning is worth its computational cost after an agent has established a useful search trajectory.The handoff experiments indicate continued LLM control is not always necessary once such a trajectory exists.Source: C FUTURE DIRECTIONS · p. 32
• LLM-developed optimizers should generalize across tasks through representative development sets and reusable strategies rather than heuristics overfit to few tasks.This shifts the challenge from online search toward cross-task generalization and algorithm discovery.Source: C FUTURE DIRECTIONS · p. 32
• Future systems could jointly adapt tools, optimizer interfaces, and control strategies instead of relying on fixed agent harnesses.The paper notes that these design choices can change optimization behavior even when the backbone model stays the same.Source: C FUTURE DIRECTIONS · p. 32
• Optimization agents could retrieve relevant prior strategies, tools, and priors from reusable experience while guarding against negative transfer.An experience library could draw on previous optimization trajectories or agent executions so new problems do not always start from scratch.Source: C FUTURE DIRECTIONS · p. 32

• Three sessions per family revised a shared GP-EI implementation through six versions on four development tasks, then froze selected programs for disjoint held-out deployment without LLM calls.The evaluation comprised 48 held-out deployments, with all six selected programs included in the reported mean and held-out performance excluded from selection.Source: §D.1 · p. 32
• 0.501 mean held-out score exceeds Fixed GP (0.451) but trails online Agentic optimization (0.569).Source: §D.1 · p. 32, Table 5 · p. 33
• Each selected program exports create optimizer() under a common setup/replay/ask/tell interface, with fresh optimizer state for every task–seed pair.Source: §D.1 · p. 32, Table 6 · p. 33
• BBOB programs ranged from 0.484 to 0.630 and HPO programs from 0.279 to 0.532; HPO session 3 fell from development S = 0.607 to held-out 0.279.Sessions 1 and 2 scored 0.532 and 0.531 on the same held-out tasks and seeds; with three sessions per family, the authors treat transfer evidence as descriptive rather than causal.Source: §D.1 · p. 33