LLM4AD: A Platform for Algorithm Design with Large Language Model

• LLM4AD addresses missing tooling, generic pipelines, and standardized benchmarks for LLM-assisted algorithm design.Existing studies use inconsistent implementations, runtimes, queries, prompts, and task comparisons.Source: §1 · p. 2
• LLM4AD integrates key methods and supports algorithm design tasks across optimization, machine learning, and scientific discovery.Source: §1 · p. 2
• The platform provides a unified evaluation sandbox for secure and robust algorithm assessment.Source: §1 · p. 2
• LLM4AD includes tutorials, examples, a user manual, online resources, and a GUI to improve usability.Source: §1 · p. 2

2.1 Framework

• The framework consists of search methods, an LLM interface, and a task evaluation interface.Source: §2.1 · p. 2
• Search uses an iterative framework that maintains a population and preserves elite algorithms.The framework supports single- and multi-objective search, while some methods can use a population size of one.Source: §2.1 · p. 2
• Prompts guide local or remote LLMs to create algorithms, with a unified interface and parallelized sampling.Source: §2.1 · p. 3
• The evaluation interface scores algorithms on task instances inside a secure, effective, and manageable sandbox.Source: §2.1 · p. 3

2.2.1 Script Usage

• Script usage initializes the LLM, evaluation, and search interfaces before running the search process with profiler-recorded logs.The example imports an online bin-packing evaluation, HTTPS LLM API, and EoH search method.Source: §2.2.1 · pp. 3–4

2.2.2 GUI Usage

• The GUI enables no-code configuration, experiment execution, result monitoring, and access to platform resources.Its main window includes a menu bar, configuration panel, results dashboard, run and stop controls, and log-file access.Source: §2.2.2 · p. 4, Figure 2 · p. 5
• Users configure the LLM, select search methods and tasks, and set parameters such as the maximum number of LLM invocations.Source: §2.2.2 · p. 4
• The results dashboard displays convergence curves and the current best algorithm with its objective value, while experiments can be stopped or inspected through logs.Source: §2.2.2 · p. 5
• The current GUI supports only one method under one LLM configuration per experiment; batch experiments are planned for future work.Source: §2.2.2 · p. 5

2.3 Search Methods

• LLM4AD integrates sampling, neighborhood, single-objective evolutionary, and multi-objective evolutionary search methods.Source: §2.3 · pp. 5–6
• The platform includes EoH, FunSearch, and (1+1)-EPS for evolutionary search, plus MEoH, NSGA-II, and MOEA/D for multi-objective search.Source: §2.3 · p. 6
• An abstract base method modularizes essential functions so users can extend the platform with custom search methods.Source: §2.3 · p. 6
• Each search method provides base, TensorBoard, and Weights & Biases profilers for different user requirements.Source: §2.3 · p. 6

2.4.1 Tasks

• Over 20 algorithm-design tasks span optimization, machine learning, scientific discovery, and other domains, with 160+ tasks planned.The tasks have clearly defined formulations for easy comparison.Source: §2.4.1 · p. 6
• The platform’s task collection is organized for quick evaluation and direct comparison across diverse algorithm-design formulations.Table 1 catalogs the algorithm-design tasks included in LLM4AD.Source: §2.4.1 · p. 6, Table 1 · p. 7

2.4.2 Examples

• Examples demonstrate different settings and showcase more complex local algorithm-design tasks.Source: §2.4.2 · p. 6

2.4.3 Evaluation Sandbox

• The evaluation sandbox safely and configurably assesses generated code using timeout handling and protected division.Optional optimizations and safety features support robust evaluation.Source: §2.4.3 · p. 6

• The general LLM interface is tailored for iterative algorithm search and supports remote API interaction and local deployment.The remote option uses an OpenAI-format API, while local deployment includes a guide and template.Source: §2.5 · p. 7
• Both interface types are modularized for efficiency and control through parallel processing, time control, and failure detection.Source: §2.5 · p. 7

3.1 Settings

• The benchmark compares four search methods initialized with template algorithms across nine tasks spanning machine learning, combinatorial optimization, and scientific discovery.The evaluation uses consistent benchmark settings and includes eight open- and closed-source general-purpose LLMs.Source: §3.1 · p. 7

3.2 Results on Different Tasks

• Search-enhanced methods, including EoH, (1+1)-EPS, and FunSearch, significantly outperform random sampling on most tasks.This underscores the importance of combining LLMs with search for algorithm design.Source: §3.2 · p. 8
• Performance gaps differ by task: methods vary little on Mountain Car but differ noticeably on OSC, SET, TSP, and VRPTW.These results indicate that the platform includes tasks with different difficulty levels.Source: §3.2 · p. 8
• EoH and FunSearch perform better on most tasks through diversity control, whereas (1+1)-EPS variants vary substantially because of their greedy nature.Source: §3.2 · p. 8

3.3 Results with Different LLMs

• LLM4AD integrates open- and closed-source LLMs, but no single model is notably superior because performance standard deviations overlap.Source: §3.3 · p. 9
• LLM choice produces significant performance variance on three tasks, except CAR, and stronger HumanEval coding capability does not necessarily improve algorithm design performance.Source: §3.3 · p. 9, Table 4 · p. 9

4.1 Add New Methods

• LLM4AD provides open-source evolutionary search implementations, diverse test cases, and unified LLM interfaces for developing and debugging new algorithm-design methods.Supported population strategies include genetic algorithms, island models, and 1+1 search.Source: §4.1 · p. 9
• The convergence figures report repeated-run mean performance with shaded standard deviations, using lower fitness or score values as better outcomes.Figure 3 covers GPT-4o-Mini-generated top-1 algorithms, while Figure 4 compares top-1 heuristics generated by different LLMs.Source: Figure 3 · p. 10, Figure 4 · p. 11, Figure 5 · p. 11
• LLM4AD also provides tools and APIs for code manipulation, secure evaluation, task definition, documentation, and notebooks that explain and visualize its modules.These resources are intended to support understanding of the search and code-manipulation pipeline.Source: §4.1 · p. 10

4.2 Add New Tasks

• Users can add tasks by defining objective evaluation and an executable template whose precise documentation is assembled into later search prompts.The evaluation interface also supports timeouts and numba.jit acceleration.Source: §4.2 · pp. 11–12
• LLM4AD automatically invokes task evaluations and securely assesses generated code to prevent harmful behavior such as aborting the search or endless loops.The sandbox is applied after a modified evaluation instance is passed to the search method.Source: §4.2 · p. 12

4.3 Add New LLM Sampler

• Users can customize LLM access through samplers that query remote models or run locally deployed models via an overridable interface.The sampler interface leaves draw_sample() unimplemented so user-defined samplers can be passed to search methods.Source: §4.3 · p. 12

5 Conclusion

• LLM4AD combines modular search methods, algorithm-design tasks, and LLM interfaces across optimization, machine learning, and scientific discovery.The platform is designed as a comprehensive Python framework for algorithm design with large language models.Source: §5 · p. 12
• Its secure evaluation sandbox and support resources, including tutorials, examples, documentation, online resources, and a GUI, enhance platform utility.The conclusion describes these elements as improving user experience and algorithm assessment.Source: §5 · p. 12
• The platform aims to promote extensive usage, comparative research, standardization, and further innovation in LLM-based algorithm design.These intended contributions are stated as goals of the platform rather than reported empirical findings.Source: §5 · p. 12