PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Survey on Large Language Models for Algorithm Design

Fei Liu, Yiming Yao, Ping Guo, Zhiyuan Yang, Zhe Zhao, Xi Lin, Xialiang Tong, Kun Mao, Zhichao Lu, Zhenkun Wang, Mingxuan Yuan, Qingfu Zhang

arXiv:2410.14716v5cs.LGcs.AIcs.CL

TL;DR

Existing surveys of LLMs and algorithms cover narrower subfields or adjacent objectives, leaving no systematic review of LLM-assisted algorithm design. This survey organizes the literature by LLM role and design stage, synthesizes its progress and limitations, and identifies open challenges for future research.

  • Problem

    Existing surveys address narrow subfields or adjacent objectives rather than providing a systematic review of LLM-assisted algorithm design.

    Source: §1 · p. 2

  • Method

    The paper synthesizes LLM4AD research using a four-role taxonomy and maps works across ideation, implementation, and evaluation.

    Source: §4 · p. 13, §7 · pp. 24–25

  • Results

    The synthesis organizes the field into optimizer, predictor, extractor, and designer paradigms and identifies challenges including scalability, generalization, interpretability, efficiency, and benchmarking.

    Source: §7 · p. 25

  • Takeaways & Limitations

    The survey’s synthesis of progress, limitations, and open challenges is intended to guide future research in LLM-assisted algorithm design.

    Source: §7 · p. 25

    LLM-based algorithm analysis often remains descriptive rather than causal and is highly sensitive to prompt engineering.

    Source: §4.3 · p. 16

Abstract

from arXiv · show

Algorithm design is crucial for effective problem-solving across various domains. The advent of Large Language Models (LLMs) has notably enhanced the automation and innovation within this field, offering new perspectives and promising solutions. In just a few years, this integration has yielded remarkable progress in areas ranging from combinatorial optimization to scientific discovery. Despite this rapid expansion, a holistic understanding of the field is hindered by the lack of a systematic review, as existing surveys either remain limited to narrow sub-fields or with different objectives. This paper seeks to provide a systematic review of algorithm design with LLMs. We introduce a taxonomy that categorises the roles of LLMs as optimizers, predictors, extractors and designers, analyzing the progress, advantages, and limitations within each category. We further synthesize literature across the three phases of the algorithm design pipeline and across diverse algorithmic applications that define the current landscape. Finally, we outline key open challenges and opportunities to guide future research. To support future research and collaboration, we provide an accompanying repository at: https://github.com/FeiLiu36/LLM4AlgorithmDesign.

1 Introduction

LLMs have prompted work on algorithm design, but existing reviews cover narrower subfields or adjacent objectives. This survey organizes LLM4AD with a role-based taxonomy and synthesis across design stages and applications.

1 Introduction

  • Algorithm design has traditionally required substantial labor and expertise, motivating efforts to automate and enhance algorithm development.The paper situates this work across domains including industry, economics, healthcare, and technology.Source: §1 · p. 2
  • LLMs have shown potential from algorithm ideation to implementation, with reported reductions in design effort and gains in solution creativity and efficiency.The introduction cites examples across these stages and outcomes.Source: §1 · p. 2
  • Existing surveys focus on code generation, specific optimization subfields, or LLM agents and planning, leaving partial coverage of algorithm design.The paper distinguishes implementation-focused code generation from upstream algorithm ideation and strategic design.Source: §1 · p. 2
  • The survey synthesizes more than 180 papers through four LLM roles, three design stages, application domains, and open challenges.Its taxonomy names optimizers, predictors, extractors, and designers.Source: §1 · p. 2

2 Methodology

The survey defines its scope around LLMs that substantively contribute to algorithm conception, synthesis, or refinement, distinguishing design from general-purpose code generation. Its collection and screening pipeline combines database searches, full-text review, and snowballing to assemble a representative literature corpus.

2.1 Definition and Scope

  • Algorithm design covers ideation, implementation, and evaluation, while the survey excludes studies that merely translate algorithms into code.An algorithm is defined as a finite computational procedure transforming inputs into outputs, including deterministic or stochastic methods, exact methods, approximations, heuristics, and metaheuristics.Source: §2.1 · p. 4
  • The survey focuses on large-scale language models, typically with billions of parameters, including text-only and multimodal models where language is central.Smaller models and traditional machine-learning approaches are outside its scope.Source: §2.1 · p. 4

2.2 Paper Collection and Scanning

  • Database searches yielded about 3,000 papers, title-and-abstract screening narrowed them to about 500, and full-text review produced 150 relevant papers.The search used Google Scholar, Web of Science, and Scopus, with title terms combining LLM-related and algorithm-related keywords.Source: §2.2 · p. 4
  • Backward snowballing and expert additions brought the final corpus above 180 papers, while the authors describe it as representative rather than exhaustive.The surveyed literature shows a marked publication rise, with most related studies conducted in the last two years.Source: §2.2 · pp. 5–6

3 Taxonomy of LLM Roles in Algorithm Design

The survey organizes LLMs in algorithm design into four roles—optimizers, predictors, extractors, and designers—and reviews the progress, advantages, and limitations of each.

3 Taxonomy of LLM Roles in Algorithm Design

  • The taxonomy distinguishes LLMs as optimizers (LLMaO), predictors (LLMaP), extractors (LLMaE), and designers (LLMaD).The survey discusses the progress, advantages, and limitations of each role.Source: §3 · p. 6

3.1 LLM as Optimizer (LLMaO)

  • LLMaO proposes and refines solutions using problem context and past trajectories, but black-box behavior, prompt sensitivity, and scaling costs remain limitations.The survey contrasts this language-conditioned process with fixed mathematical update rules.Source: §3.1 · pp. 7–8

3.2 LLM as Predictor (LLMaP)

  • LLMaP uses LLMs as surrogate models to predict solution outcomes, but predictions can be less accurate than target-trained Gaussian processes and sensitive to prompt phrasing.It is suited to settings where evaluation is expensive or difficult to obtain.Source: §3.2 · pp. 8–10

3.3 LLM as Extractor (LLMaE)

  • LLMaE extracts semantic features and design knowledge from text, code, and other sources, but extracted features may be misinterpreted and embeddings remain difficult to validate.Its representations can combine textual, spatial, and numeric information.Source: §3.3 · p. 11

3.4 LLM as Designer (LLMaD)

  • LLMaD creates algorithm components or whole algorithms, with work spanning reward functions, heuristics, and agent designs; complete solvers and correctness or runtime guarantees remain challenging.Many systems iteratively test, execute, and refine designs through search frameworks.Source: §3.4 · pp. 11–13

4 Stages of LLM-Assisted Algorithm Design

The survey organizes LLM assistance across algorithm ideation, implementation, and evaluation, covering both the artifacts produced at each stage and their challenges. [p09]

4.1 Stage I: Algorithm Ideation

  • At ideation, LLMs formulate optimization models from natural language and generate algorithm concepts, including hybrid designs and concept-code co-design.Reasoning over ideas and diversity control are also explored; measuring idea quality and novelty, and connecting abstract concepts to implementations, remain open challenges.Source: §4.1 · p. 14

4.2 Stage II: Algorithm Implementation

  • Code generation ranges from individual components to entire algorithms, while complex programs remain difficult to generate reliably and faithfully implement intended ideas.Diff blocks support partial revisions, and LLaMoCo tunes models for optimization code generation; multi-agent frameworks are among proposed ways to manage complexity.Source: §4.2 · pp. 14–15
  • Algorithm optimization commonly uses execution-feedback refinement and evolutionary search, with non-evolutionary search and task-specific fine-tuning also explored.Optimized algorithms may overfit benchmark instances or conditions and perform poorly on unseen instances.Source: §4.2 · p. 15

4.3 Stage III: Algorithm Evaluation

  • Performance testing uses domain-specific and broader benchmarks, while some methods co-evolve algorithms with test instances or adversarially increase instance difficulty.Evaluation can also examine search trajectories, robustness, convergence, and computational efficiency rather than relying on a single score.Source: §4.3 · p. 15
  • LLMs can produce algorithm analyses and reports, but current explanations may remain descriptive, prompt-sensitive, and short of causal, principled understanding.One comparison of quantum-algorithm explanations found consistency across iterations but quality gaps and sensitivity to prompts; deeper analysis would explain properties such as convergence or complexity.Source: §4.3 · p. 16

5.1 Optimization Algorithms

The survey organizes LLM applications in optimization across combinatorial and continuous problems, comparing the roles models play and the tasks addressed. [p29]

5.1.1 Combinatorial Optimization.

  • Combinatorial optimization work uses LLMs to evolve algorithms, optimize solutions from score feedback, generate offspring, and support multimodal routing approaches.Applications include TSP, CVRP, cap set, bin packing, and scheduling problems.Source: §5.1.1 · p. 16

5.1.2 Continuous Optimization.

  • Continuous optimization spans algorithm evolution, solution generation, surrogate modeling, prompt-based optimization, and LLM-enhanced Bayesian optimization.The surveyed tasks include single- and multi-objective optimization, hyperparameter tuning, and acquisition-function design; Table 2 summarizes methods across domains and tasks.Source: §5.1.2 · p. 17, Table 2 · p. 18

5.2 Machine Learning Systems and Data Science

The surveyed machine-learning applications span reinforcement learning, architecture search, prompt tuning, graph learning, dataset labeling, and other tasks; Table 3 summarizes this landscape. [p35]

5.2.1 Reinforcement Learning.

  • In reinforcement learning, LLMs design reward functions, guide heuristic planning, regularize policies, and support multi-agent systems.The applications include using a mixture-of-experts system to direct RL models for intelligent network solutions.Source: §5.2.1 · p. 18

5.2.2 Neural Architecture Search.

  • For neural architecture search, LLMs generate code and architectures, predict performance, or transfer design principles to guide the search space.Reported work combines LLMs with evolutionary or quality-diversity search on tasks including CIFAR-10 and NAS-bench-201.Source: §5.2.2 · pp. 18–19

5.2.3 Prompt Tuning.

  • Prompt-tuning methods use LLMs to propose or iteratively optimize prompts, including approaches inspired by gradients, evolutionary operators, and collaborating agents.The surveyed work reports human-level performance for APE on selected tasks and improvements over human-designed prompts for OPRO on BBH and GSM8K.Source: §5.2.3 · p. 19

5.2.4 Graph Learning.

  • Graph-learning applications use LLMs for graph labeling, evolving functions for node identification or interpretability, and aligning or searching knowledge graphs.The cited examples include semantic entity alignment across knowledge graphs and a language for knowledge-graph search.Source: §5.2.4 · p. 19

5.2.5 Dataset Labeling.

  • For dataset labeling, LLMs train interpretable image-attribute classifiers and generate label functions for weakly supervised learning.Source: §5.2.5 · p. 20

5.2.6 Other Applications.

  • Other machine-learning applications include iterative prompt refinement, tensor-network heuristic design, and generating decision-based adversarial attack algorithms.Source: §5.2.6 · p. 20

5.3 Scientific and Symbolic Domains

The survey covers scientific and symbolic applications in which LLMs help create search strategies, predict performance, or generate symbolic expressions. [p43, p44]

5.3.1 Symbolic Regression.

  • In symbolic regression, LLM-based systems generate and refine equations using evolutionary search, physical knowledge, or simulations with experimental feedback.The surveyed approaches target physical-law discovery and optimization of continuous variables.Source: §5.3.1 · p. 21

5.3.2 Chemistry, Biology & Physics.

  • Chemistry, biology, and physics applications include molecular and catalyst design, biological sequence tasks, and analytical calculations in theoretical physics.The survey also notes materials discovery, synthesis planning, green chemistry, protein engineering, and enzyme design.Source: §5.3.2 · p. 21

5.3.3 Mechanics.

  • Mechanics applications include LLM agents that write, execute, and self-correct finite-element code, alongside equation discovery and turbulence-model design.One equation-discovery approach reports better generalization on unseen data than other models.Source: §5.3.3 · pp. 21–22

6 Challenges and Future Directions

LLM-driven algorithm design faces constraints in scalability, generalization, interpretability, efficiency, and benchmarking. The section describes current mitigation efforts while identifying unresolved boundaries and research needs.

6.1 Scalability

  • LLMs often struggle with novel or large-scale algorithm design: optimizers perform well mainly on low-dimensional problems, while LLMaD may produce components rather than complete frameworks.Fixed context windows constrain processing detailed specifications or large inputs, and long-sequence reasoning and solution generation remain difficult [19].Source: §6.1 · p. 22

6.2 Generalization

  • Methods often fail to generalize beyond training distributions, and algorithms designed for one problem type rarely transfer to other problems.Examples include synthetic-to-real bin packing and fixed-size-to-other-size TSP transfer; proposed approaches include meta-learning, instance-feature subclasses, and portfolios of complementary heuristics [134], [130], [187], [89].Source: §6.2 · p. 23

6.3 Interpretability

  • LLM black-box behavior and poorly understood iterative design make input-output effects and design dynamics difficult to interpret.Proposals include interpretable surrogate models, Code Evolution Graphs [147], graph-based algorithm relationships [92], and Markov-chain analysis [79].Source: §6.3 · p. 23

6.4 Efficiency

  • Algorithm-design efficiency remains constrained by costly search and evaluation, including many test instances and time-consuming individual evaluations.Search strategies and task-specific fine-tuning have been explored [178], [194], [116], [93], [66], [139]; quality prediction can end unpromising evaluations early [69], but balancing exploration and reliability remains open.Source: §6.4 · p. 24

6.5 Benchmarking

  • LLM-driven algorithm design lacks standardized benchmarks and evaluation pipelines, limiting fair, reproducible, and standardized comparisons.Widely accepted settings still require datasets, unified protocols, and transparent reporting; existing specialized benchmark efforts include combinatorial optimization [138], [17], black-box optimization [148], and broader domains [94].Source: §6.5 · p. 24

7 Conclusion

The survey organizes LLM-driven algorithm design into four roles and maps research across the design pipeline, application domains, and open challenges. It identifies issues for future research.

7 Conclusion

  • The survey categorizes LLM roles as Optimizer, Predictor, Extractor, and Designer, distinguishing solution generation, prediction, semantic-feature extraction, and algorithm design.LLMaD can generate components or design an entire algorithm.Source: §7 · pp. 24–25
  • The review maps research across ideation, implementation, and evaluation, summarizes application domains, and identifies scalability, generalization, interpretability, efficiency, and benchmarking as open challenges.These challenges are presented as directions to guide future research.Source: §7 · p. 25

Related papers

Loading 2410.14716v5…