Source-linked AI summary
Barbarians at the Gate: How AI is Upending Systems Research
Audrey Cheng, Shu Liu, Melissa Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, Jeff Chen, Lakshya Agrawal, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, Ion Stoica
TL;DR
Systems research offers a promising setting for AI-driven solution discovery because performance-oriented algorithms can often be verified empirically. The paper presents ADRS, an iterative generate-evaluate-refine approach, and reports that it can outperform human baselines across systems problems. It argues that researchers will increasingly guide AI systems through problem formulation and strategic evaluation, while noting that ADRS is best suited to isolated problems with reliable evaluations.
Problem
Systems performance research needs automated solution discovery, but AI-driven approaches require reliable verification of whether generated solutions solve the problem.
Method
ADRS iteratively uses LLMs to generate or refine solutions and evaluates them in real systems or simulators, with human guidance optionally shaping future prompts.
Results
ADRS case studies show that generated algorithms can match or surpass state-of-the-art human-designed solutions, including a 5.0× runtime improvement and roughly 30% greater savings.
Takeaways & Limitations
Human researchers will increasingly define problems, steer research, and critically evaluate results as AI takes on algorithm discovery and optimization.
Takeaways & Limitations
ADRS is best suited to isolated algorithm changes with reliable evaluations and does not work well for policies or protocols distributed across large systems.
Abstract
from arXiv · showhide
Artificial Intelligence (AI) is starting to transform the research process as we know it by automating the discovery of new solutions. Given a task, the typical AI-driven approach is (i) to generate a set of diverse solutions, and then (ii) to verify these solutions and select one that solves the problem. Crucially, this approach assumes the existence of a reliable verifier, i.e., one that can accurately determine whether a solution solves the given problem. We argue that systems research, long focused on designing and evaluating new performance-oriented algorithms, is particularly well-suited for AI-driven solution discovery. This is because system performance problems naturally admit reliable verifiers: solutions are typically implemented in real systems or simulators, and verification reduces to running these software artifacts against predefined workloads and measuring performance. We term this approach as AI-Driven Research for Systems (ADRS), which iteratively generates, evaluates, and refines solutions. Using penEvolve, an existing open-source ADRS instance, we present case studies across diverse domains, including load balancing for multi-region cloud scheduling, Mixture-of-Experts inference, LLM-based SQL queries, and transaction scheduling. In multiple instances, ADRS discovers algorithms that outperform state-of-the-art human designs (e.g., achieving up to 5.0x runtime improvements or 50% cost reductions). We distill best practices for guiding algorithm evolution, from prompt design to evaluator construction, for existing frameworks. We then discuss the broader implications for the systems community: as AI assumes a central role in algorithm design, we argue that human researchers will increasingly focus on problem formulation and strategic guidance. Our results highlight both the disruptive potential and the urgent need to adapt systems research practices in the age of AI.
1 INTRODUCTION
AI-driven approaches are beginning to automate algorithm discovery in systems research, challenging traditional researcher roles. ADRS case studies show that generated solutions can match or exceed human-designed systems while shifting researchers toward strategic guidance.
- ADRS is presented as a new class of AI-driven approaches for automated algorithm discovery in systems research.
- 5.0× faster than the best-known baseline, an OpenEvolve algorithm rebalanced Mixture-of-Experts experts across GPUs.
- Roughly 30% greater savings than an expert-developed baseline were achieved for deadline-driven spot-instance scheduling across cloud regions.
- Systems performance problems are well suited to AI-driven research because solutions can be generated and verified through empirical system evaluation.
- Human researchers will likely emphasize problem formulation, high-level ideation, strategic direction, and interpretation of generated solutions.
2 RELATED WORK
ADRS builds on earlier systems optimization, automated discovery, coding-assistant, and LLM-driven research efforts. Its distinguishing focus is automated algorithm discovery for systems problems with strong evaluators.
- ADRS extends prior work combining large-scale search and machine learning for complex systems problems.
- Earlier automated-discovery systems demonstrated search- and learning-based advances in games, protein prediction, and computational algorithm design.
- LLM coding assistants accelerate research by translating algorithmic ideas into prototypes, simulators, baselines, and code modifications.
- Recent frameworks use evolutionary search, reflective prompt evolution, or unified search-generation-evaluation interfaces to automate algorithm discovery.
- ADRS focuses on systems-domain algorithm discovery where strong evaluators enable reliable verification.
3 WHY AI-DRIVEN RESEARCH FOR SYSTEMS?
Systems performance problems are a strong fit for AI-driven research because their implementations, correctness, and performance can often be evaluated empirically. Simulators make broad candidate exploration practical, although faithful simulation remains difficult for complex systems.
- Over one-third of papers in surveyed top systems venues feature performance-optimization algorithms as their core contribution.
- Performance improvements can be verified by running implementations under representative workloads and comparing relevant metrics with baselines.
- Correctness is often preserved or relatively easy to check, such as verifying that schedulers assign tasks and routers forward packets.
- Small changes to isolated scheduler, load-balancer, or allocator logic make generated solutions easier for humans to interpret and verify.
- Simulator-based verification is inexpensive and practical for broad searches, but building faithful, low-cost simulators for complex systems remains difficult.
4 USING AI TO ACCELERATE SYSTEMS RESEARCH
ADRS accelerates systems research by automating iterative solution generation and evaluation while leaving problem formulation and other research stages to humans. Its architecture combines LLM generation, evaluators, storage, and selection, often around fast simulators.
- Systems performance research process: The systems research process includes problem formulation, evaluation-framework development, solution development, evaluation, and paper write-up.
- Systems performance research process: Algorithm Design (21.5%) and Evaluation (20.1%) together account for over 40% of surveyed systems-research effort.
- AI-Driven Research for Systems: ADRS automates the iterative Solution and Evaluation stages, which account for about 40% of the time spent producing new results.
- AI-Driven Research for Systems: Each ADRS iteration prompts LLMs to create or refine solutions, evaluates them in a system or simulator, and continues until success, budget exhaustion, or researcher termination.
- AI-Driven Research for Systems: ADRS commonly uses simulators because real-system code may exceed LLM context windows and simulator evaluations can be orders of magnitude faster.
- AI-Driven Research for Systems: ADRS does not automate researchers’ decisions about problems, evaluation frameworks, documentation, or other stages outside solution and evaluation.
- ADRS examples: OpenEvolve, AlphaEvolve, GEPA, LLM4AD, and coding assistants exemplify ADRS-like systems using evolutionary search, prompt reflection, modular evaluation, or codebase-level feedback.
5 EVALUATION AND CASE STUDIES
Across cloud scheduling and Mixture-of-Experts inference, ADRS evolves policies that improve cost or runtime while preserving deadline or load-balancing requirements. The case studies show iterative discovery of adaptive scheduling logic and efficient expert placement, alongside configuration differences that limit direct comparisons.
- Evaluation and Limitations: The evaluation requires valid algorithms to meet all deadlines, and the case studies use differing configurations that make direct comparison difficult.The paper identifies systematic ablation studies as a future need for determining which configurations work best across problems.
- Spot Instance Savings Under Deadlines: ADRS improves spot-instance savings by 7% over Uniform Progress in one region and by 26% over a hand-tuned baseline across multiple regions.The single-region policy meets all deadlines, while the multi-region policy explores cheaper capacity under slack and prioritizes progress when deadlines are at risk.
- Single Region: Can’t Be Late: In the single-region setting, the evolved policy adapts spot usage to recent availability and deadline slack instead of following Uniform Progress’s fixed formula.It becomes more aggressive when spot capacity is stable and more conservative when availability is unstable, avoiding repeated switches caused by brief allocations.
- Multi-Region: Can’t Be Late: The multi-region policy combines urgency detection with regional exploration, migrating jobs when capacity conditions are favorable while falling back to on-demand resources under deadline pressure.Its evolution separates deadline assessment from resource provisioning and adds region caching and urgency calculation.
- Expert Placement in MoE Inference: 5.0× faster runtime is achieved for MoE expert rebalancing while matching baseline load-imbalance factors.OpenEvolve discovers staggered expert placement using tensor reshaping and reversal, with additional ordering and reshaping enhancements.
- Expert Placement in MoE Inference: The MoE task formulates expert placement as choosing replica counts and GPU mappings that minimize load imbalance for a workload and model.The baseline process distributes expert groups, replicates popular experts, and assigns replicas to GPUs.
6 EARLY BEST PRACTICES
The paper distills early best practices for avoiding ADRS failures and guiding effective algorithm evolution, from problem and prompt design to search control. Key recommendations include strong baselines, calibrated hints, suitable abstraction, diverse evaluation, reward-hacking defenses, and balanced exploration.
- Failure modes: ADRS pipelines encounter runtime, search, and algorithm failures, including invalid code, stalled evolution, constraint violations, overfitting, and evaluator exploitation.Table 3 categorizes these recurring failure patterns from 420 LLM-judged traces.
- Problem and prompt design: A clear problem formulation and structured prompt should specify the task, evaluation criteria, correctness constraints, and necessary context.The authors recommend defining required APIs and other information needed for execution and algorithm design.
- Base programs: Clean, minimal, high-quality base programs accelerate meaningful evolution, whereas buggy or overly strong baselines waste iterations or restrict exploration.The paper contrasts simple greedy initialization with stronger baselines that can limit discovery to shallow optimizations.
- Solution hints: Solution hints should vary in specificity: excessive guidance risks premature convergence, while intermediate human feedback can help when search becomes inefficient or stuck.The authors recommend trying multiple prompt configurations and injecting hints as evolution progresses.
- Evaluator design: Broad workloads, adversarial tests, and multiple evaluation signals help prevent overfitting and reward hacking by testing generalization, correctness, efficiency, and robustness.Narrow traces can produce solutions that fail on unseen patterns, while evaluator loopholes can reward bypassing the intended task.
- Solution selector: Selectors should preserve diversity while advancing high-quality candidates, because greedy search can converge prematurely and random search can waste resources.The exploration–exploitation ratio is tunable in OpenEvolve and requires careful adjustment.
7 LIMITATIONS AND OPEN CHALLENGES
The paper identifies where ADRS is effective and outlines challenges for extending it. ADRS is best suited to isolated, quickly evaluated problems with reliable verifiers, while broader system changes, weak verification, costly evaluation, and unresolved framework controls remain boundaries.
- Scope boundaries: ADRS works best for localized changes, fast evaluation, and reliable verification, but is less effective across many systems, with weak verifiers, or under costly evaluation.These boundaries motivate improvements to the approach and its supporting infrastructure.
- Suitable problems: Isolated policies and algorithms such as schedulers, cache managers, load balancers, and resource allocators are better targets than distributed protocols spanning complex system components.Existing LLMs are described as more reliable when modifying small amounts of code.
- Evaluator design: Evaluators should provide high fidelity, broad workload coverage, and fast, reliable feedback; simulators and cascading evaluation can improve cost-effectiveness.Cascades can progress from coarse cost models to simulators, emulators, and real-system tests.
- Search challenges: Current evolutionary searches are monolithic and inefficient, often mixing good ideas with poor implementations or repeating failed heuristics.More flexible selectors could separate ideation from code generation, lock in working code, increase diversity, or roll back stalled evolution.
- Framework improvements: Open challenges include automated hyperparameter tuning, richer retrieval-based prompts, autonomous codebase navigation, specialized agent ensembles, and learned user preferences.These directions aim to broaden ADRS beyond current solution-generation capabilities.
- Human interaction: The optimal division between synchronous human assistance and asynchronous autonomous ADRS remains unresolved.A central open question is when human guidance adds value versus when ADRS should operate autonomously.
8 HOW CAN ADRS IMPACT THE RESEARCH PROCESS?
ADRS can automate implementation and exploration while helping researchers pursue higher-leverage questions. The paper therefore envisions researchers increasingly selecting and formulating problems, guiding AI assistants, and interpreting generated solutions.
- Research productivity: ADRS accelerates discovery by automating implementation and some debugging, freeing researchers to focus on problem selection, system architecture, and design.Even imperfect generated solutions can reveal promising directions for human-designed algorithms.
- Research productivity: ADRS can achieve better-than-human results by exploring the solution space more thoroughly than humans.The paper presents this as one of two key ways ADRS can already help researchers.
- Research roles: ADRS is framed as providing researchers with many reliable assistants that work from clear problem specifications and well-defined goals.The tools add another layer to existing academic mentoring hierarchies.
- Research roles: Researchers are expected to spend more time selecting and precisely formulating high-leverage problems as ADRS solves more tasks autonomously.The paper argues this could elevate productivity across faculty and students.
- Research community: The authors believe ADRS will expand the research community by enabling non-expert problem solvers to contribute meaningfully.They connect this expansion to solving more problems faster and better.
9 CONCLUSION
The conclusion presents ADRS as a promising approach for systems research and argues that AI-driven discovery will reshape human researchers’ responsibilities. Researchers will increasingly define problems, steer the process, and evaluate results while ADRS may improve itself.
- Human researchers will increasingly define problems, steer research, and critically evaluate AI-generated results.
- ADRS may form a virtuous cycle by refining its reasoning, debugging code, and discovering more effective strategies.
- ADRS can already outperform human baselines on key systems performance problems.
- The authors argue that improving ADRS’s efficiency, scalability, and reliability is itself a systems challenge.
A ABLATION STUDIES
The paper reports ablation studies for the Can’t Be Late case study and presents their results.
- The Can’t Be Late case study is evaluated through several ablation studies.
A.1 GEPA RESULTS
The GEPA evaluation uses a two-stage process and improves a greedy baseline during iterative refinement. The final result is measured on downstream single-region traces.
- GEPA validates syntax and simulator compliance before running full simulations to compute cost savings.
- 4% improvement over the greedy baseline was achieved at iteration 68, with the search capped at 200 iterations.
A.2 CHANGING TRAIN SET COVERAGE
The training-coverage ablation compares policy evolution using 3% of the available training data with the full training set, while holding out 70% for testing.
- The 3% training-coverage condition uses only 3% of the available training set.
- The full-coverage condition uses the entire available training set.
- Figure 8 splits the data into 30% training and 70% testing while comparing the two training-coverage conditions.
B FAILURE TAXONOMY
ADRS failures fall into execution errors, search failures, and algorithm failures, motivating careful setup and failure-specific guidance. The taxonomy distinguishes immediate code or budget problems from stalled evolution and candidates that run but do not advance the algorithm.
- The failure taxonomy motivates best practices and guidelines intended to address ADRS limitations.The paper presents these practices after characterizing common ADRS failure modes.
- ADRS failures are grouped into execution errors, search failures, and algorithm failures.Their frequencies are reported in Table 3, with detailed descriptions provided in Section 6.
- Execution Errors: About one third of failures are immediate execution failures involving syntax, interfaces, or exhausted computational and model budgets.Examples include missing imports, type mismatches, excessive memory, timeouts, and API-quota exhaustion.
- Search Failures: Roughly half of failures occur when executable searches make no progress through premature convergence, duplicate generation, or contradictory mutations.Examples include settling on a Steiner tree, repeatedly renaming variables, or oscillating between BFS-tree and graph-based methods.
- Algorithm Failures: Algorithm failures occur when candidates run but ignore constraints, make shallow API changes, or overfit evaluation traces.A cited example is improving throughput while violating latency SLOs.
C.1 ADAPTIVE WEIGHT COMPRESSION
The case studies apply ADRS to diverse systems problems, using evaluators that measure task-specific objectives and correctness. Evolution rediscovers or improves several established strategies, while some tasks yield only marginal or incomplete gains.
- C.1 ADAPTIVE WEIGHT COMPRESSION: Adaptive weight quantization targets lower average bits per element while preserving Wikitext-2 perplexity; the hand-tuned baseline reached 2.64 bits/elem and PPL 22.9.The evaluator combines Wikitext, PTB, and bitrate scores, with zero score above 2.5 bits/elem and PPL 30.
- C.1 ADAPTIVE WEIGHT COMPRESSION: The quantization evolution produced marginal gains by tuning the hand-crafted importance-to-bitrate mapping rather than changing the algorithm.The result therefore functioned more as automated hyperparameter tuning than as a novel quantization strategy.
- C.2 TELEMETRY REPAIR: 95% repair score and 95% confidence calibration were achieved by an evolved strategy for faulty router telemetry.Its structured logic averages nearby counters and separates repair from confidence estimation.
- C.3 COST-AWARE MULTICAST: 31.1% lower average egress cost was achieved by rediscovering a Steiner tree for multi-region, multi-cloud multicast.Intermediate waypoints allow data to be replicated once and forwarded to multiple destinations.
- C.4 GLOBAL MODEL PLACEMENT: For multi-LLM serving, ADRS evolves placement strategies around maximum KVPR, adding local moves or swaps after initial model placement.The objective is to reduce GPU contention by lowering the maximum SLO-weighted request rate relative to available KV cache memory.
- In sparse attention, multi-agent systems, and GPU cache optimization, the studies address accuracy-compute trade-offs, coordination failures, and GPU overheads, with mixed outcomes.Multi-agent evolution introduced context-management and verification flows, while GPU evolution improved speed but retained edge-case failures and undesirable final performance.