Source-linked AI summary
Is manual software optimization a thing of the past?
Pavlin G. Poličar, Martin Špendl, Tomaž Hočevar
TL;DR
As scientific datasets grow, software efficiency affects which analyses are feasible, while optimization requires substantial specialist expertise and effort. An LLM agent worked without further intervention; resulting implementations were faster in every case study, by up to two orders of magnitude over the fastest existing tools. The implementations were faster in every case study, by up to two orders of magnitude over the fastest existing tools.
Problem
As scientific datasets grow, software efficiency affects which analyses are feasible, while optimization requires substantial specialist expertise and effort.
Source: §1 · p. 1Method
An LLM agent worked without further intervention; resulting implementations were faster in every case study, by up to two orders of magnitude over the fastest existing tools.
Source: §1 · p. 2Results
The implementations were faster in every case study, by up to two orders of magnitude over the fastest existing tools.
Source: §1 · p. 2Takeaways & Limitations
For well-scoped, verifiable problems, optimization can increasingly be delegated to agents while humans define objectives and ensure correctness.
Source: §4 · p. 4Open-ended objectives and problems with hard-to-verify correctness remain untested.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Scientific software is increasingly required to process larger datasets while maintaining acceptable execution times. Software optimization traditionally requires substantial expertise in programming, algorithms, and numerical methods. Recent advances in large language models (LLMs) offer the possibility of automating much of this process. We investigate whether LLM-based agents can autonomously achieve substantial performance improvements in scientific software, including mature implementations that have already been extensively optimized by human developers. We tasked an LLM-based agent with optimizing software for three computational problems: t-SNE, single-sample gene set enrichment analysis (ssGSEA), and graphlet counting. Humans defined the scope, correctness criteria, and a verification mechanism, after which the agent worked autonomously, in some cases for several hours. Code maintainers reviewed each resulting implementation and verified its correctness. The optimized implementations were faster in all tested configurations, by up to two orders of magnitude over the fastest existing tools. The improvements included low-level code optimizations, mathematical reformulations, and an entirely new algorithm for graphlet counting. Software optimization can increasingly be delegated to autonomous agents, with the human role shifting from implementing optimizations to deciding which software to optimize, defining objectives, providing verification mechanisms, and ensuring the correctness of the final software. For well-scoped, verifiable problems, we argue that manual software optimization may be a thing of the past.
1 Introduction
Scientific software must handle growing datasets efficiently, but optimizing non-trivial software takes substantial specialist expertise and effort. The paper asks whether LLM agents can autonomously find substantial improvements beyond low-level changes, and reports faster implementations across its three case studies.
1 Introduction
- Growing datasets make software efficiency a determinant of which scientific analyses are feasible, while optimization requires substantial development effort and expertise.The required expertise spans software engineering, numerical computing, and algorithm design.Source: §1 · p. 1
- Whether LLM agents can autonomously find improvements beyond low-level changes for well-scoped, verifiable problems remains unclear.Existing evaluations had reported mostly surface-level gains.Source: §1 · p. 1
- The study investigates t-SNE, ssGSEA, and graphlet counting as three computational optimization problems.Figure 1 compares runtimes for these workloads on a consumer-grade M1 MacBook Pro.Source: §1 · p. 1, Figure 1 · p. 2
- The agent’s implementations were faster in every case study, by up to two orders of magnitude over the fastest existing tools.The improvements spanned low-level code optimization, mathematical reformulation, and algorithm replacement.Source: §1 · p. 2
2 Related Work
Earlier LLM optimization systems used fixed pipelines, specialized training, or automated evaluators, while general-purpose agents had been tested on scientific software. Their effectiveness remained uncertain, with prior benchmarks reporting limited gains and failures to find algorithmic improvements.
2 Related Work
- Earlier approaches included hotspot rewriting from curated patterns, training on slower–faster code pairs, and LLMs paired with automated evaluators.General-purpose coding agents had also been applied to scientific software, with their largest gains coming from faster languages or GPUs.Source: §2 · p. 2
- Prior evaluations found modest speedups over mature numerical libraries, less than a quarter of expert speedup on scientific Python commits, or negligible gains.On AlgoTune, current models also failed to discover algorithmic improvements.Source: §2 · p. 2
3 Case studies
Across t-SNE, ssGSEA, and graphlet counting, the agent improved runtimes using code optimizations, a mathematical simplification, and a new algorithm. The work used correctness checks, including expert review and exact score or count matching.
3 Case studies
- Domain experts, including original developers of openTSNE and Orca, reviewed the implementations and verified their correctness.The study tasked the agent with improving an existing implementation, producing a faster implementation of an existing algorithm, or developing a novel algorithm.Source: §3 · p. 2
3.1 openTSNE
- The agent optimized openTSNE’s optimization phase while preserving its API, defaults, procedure, and accuracy, and ran 174 unit tests after each modification.It retained only changes that improved measured runtime.Source: §3.1 · pp. 2–3
- At 500,000 points, optimized openTSNE achieved 4.4–7.7× speedups across tested configurations.The work retained 34 of approximately 60 evaluated ideas, including code-level changes and a mathematical simplification that reduced one FIt-SNE stage’s runtime by about a quarter; overall asymptotic complexity was unchanged.Source: §3.1 · p. 3
3.2 Single-sample GSEA
- 136× faster, the new ssGSEA implementation scored 2,000 gene sets across 1,000 samples in 2.2 seconds versus 5 minutes for GSEApy.The runtime gap widened as the number of gene sets increased.Source: §3.2 · p. 3
3.3 Graphlet counting
- About 160× faster than Orca, the new graphlet algorithm counted all orbits on a human protein–protein interaction network in 13 seconds versus 35 minutes.On random 10k-node graphs, runtime grew roughly quadratically with edge count for the agent’s implementation, versus cubically for Orca and roughly quartically for brute-force enumeration.Source: §3.3 · p. 3
4 Conclusion
The case studies show that autonomous LLM agents can accelerate scientific software through code-level changes, mathematical reformulations, and new algorithms. The authors argue that for well-scoped, verifiable problems, optimization can increasingly be delegated to agents, while humans define objectives and verify correctness.
4 Conclusion
- Across three case studies, agents improved scientific software through low-level optimizations, mathematical reformulations, and an entirely new graphlet-counting algorithm.The resulting implementations were faster in every case, in some cases by orders of magnitude.Source: §4 · p. 4
- For well-scoped, verifiable problems, the authors argue that optimization can increasingly be delegated to agents, shifting human work toward selecting software, defining scope and objectives, providing verification, and ensuring correctness.Open-ended objectives and problems with hard-to-verify correctness remain untested.Source: §4 · p. 4