Source-linked AI summary
$Φ$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?
Leilei Ding, Shumin Wang, Yuting Huang, Fanqi Wan, Yinmin Zhang, Qi Han, Yiming Xu, Feiyuan Zhang, Xiaomeng Chu, Guoliang You, Wuyang Zhang, Daxin Jiang, Yanyong Zhang
TL;DR
Existing benchmarks do not capture the open-ended, long-horizon work required to engineer real-world LLM infrastructure. Φ-Bench evaluates frontier models on 85 tasks spanning infrastructure domains and increasing task scope, finding substantial but uneven capabilities, with Claude Opus 5 achieving 36.53% and no model performing consistently well.
Problem
Existing benchmarks focus on isolated kernels or components and do not capture the open-ended, long-horizon engineering of real-world LLM infrastructure.
Method
Φ-Bench uses a taxonomy-guided, agent-assisted pipeline to synthesize 85 real-world tasks across nine infrastructure domains and three increasingly open-ended formats.
Results
36.53%: Claude Opus 5 achieves the highest overall score, but frontier-model capabilities are substantial yet uneven and no model performs consistently well.
Takeaways & Limitations
Φ-Bench exposes substantial remaining challenges in reliably applying frontier LLMs to real-world LLM infrastructure engineering.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated remarkable capabilities in reasoning and code generation, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. However, existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore fail to evaluate the ability of LLMs to perform open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present $Φ$-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack. Derived from optimization problems studied in frontier research and grounded in real-world code repositories, $Φ$-Bench provides broad coverage of the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Extensive experiments on frontier LLMs reveal their current capabilities and limitations in engineering complex LLM infrastructure, offering insights into the challenges that remain on the path toward autonomous optimization of future AI infrastructure.
Introduction
Φ-Bench evaluates whether frontier LLMs can engineer real-world LLM infrastructure through long-horizon, open-ended tasks spanning kernels, repositories, and end-to-end optimization. Its scalable, taxonomy-guided benchmark contains 85 tasks, and Claude Opus 5 achieves 36.53%, indicating substantial room for improvement.
- Motivation: Existing benchmarks typically restrict LLM evaluation to individual GPU kernels or isolated components, missing the complexity of real-world infrastructure engineering.The paper characterizes infrastructure engineering as inherently long-horizon and open-ended.
- Benchmark design: Φ-Bench systematically evaluates frontier LLMs on real-world workloads derived from top-tier systems papers and public LLM infrastructure repositories.Its design emphasizes long-horizon, open-ended problem solving, comprehensive infrastructure coverage, and scalable task synthesis.
- Benchmark construction: A taxonomy built from 2,260 papers and 1,852 repository artifacts guides task synthesis, while an agent-loop pipeline mines challenges and iteratively generates test cases.These methods support broad, realistic, and scalable evaluation of infrastructure engineering.
- Task formats: The benchmark contains 85 tasks across Kernel Function Completion, Long-Horizon Implementation, and End-to-End Optimization, with progressively broader scope and open-endedness.Together, the formats span local kernel implementation, repository-scale development, and end-to-end system optimization.
- Results: 36.53%: Claude Opus 5 achieves the highest reported Φ-Bench score, while experiments examine refinement iterations, reasoning budgets, and solution trajectories.The results leave substantial room for improvement and expose distinct model strengths and weaknesses.
Related Works
Prior work spans low-level GPU kernels, attention runtimes, training frameworks, and inference systems for improving LLM infrastructure performance, efficiency, and scalability. Existing benchmarks largely evaluate isolated operators or fused compositions under predefined interfaces, specifications, and objectives rather than complete infrastructure repositories.
- LLM Infrastructure Systems: LLM infrastructure optimization targets performance, efficiency, and scalability across multiple layers, including GPU kernels, programming systems, and attention runtimes.CUTLASS and Triton support optimized GPU programming, while FlashAttention and FlashInfer provide complementary attention-oriented systems.
- LLM Infrastructure Systems: Training frameworks provide parallel execution, memory partitioning, and communication optimization, while inference systems support KV-cache management, continuous batching, scheduling, and distributed serving.The cited examples include Megatron-LM and DeepSpeed for training, and Orca, vLLM, and SGLang for inference.
- Infrastructure Engineering Benchmarks: KernelBench, TritonBench, and FlashInfer-Bench primarily evaluate individual GPU operators or fused compositions with predefined interfaces, input-output specifications, and optimization objectives.These benchmark designs do not require LLMs to navigate complete infrastructure repositories.
Benchmark Design and Construction
Φ-Bench combines three increasingly open-ended task formats with real-world LLM infrastructure repositories, research artifacts, workloads, and hidden evaluation harnesses. Its construction uses a coverage taxonomy and complementary synthesis processes, followed by quality controls, yielding 85 tasks across nine infrastructure categories.
- Task Formats: Each task provides a natural-language specification, complete infrastructure repository, executable workloads or tests, and visible materials except the evaluation harness.The specification defines required functionality or a performance objective.
- Task Formats: KFC targets a specified kernel interface, LHI requires repository-level feature implementation, and E2EO leaves system optimization strategies open-ended.Scope expands from single-file implementation to multi-file repository changes and potentially repository-wide optimization involving profiling and bottleneck discovery.
- Source Collection and Taxonomy: Tasks derive from research papers and public repositories, organized through a three-level taxonomy spanning 4,112 sources, 410 fine-grained tags, 62 middle-level topics, and nine top-level categories.Middle-level topics index task synthesis, while source provenance and characteristics determine the task format.
- Task Synthesis: Three complementary approaches construct tasks: PR- and Issue-Grounded Synthesis, Agent-Assisted Synthesis, and Expert-Curated Synthesis.They reconstruct repository changes, identify implementation sites and generate tests with agents, or curate challenging problems requiring substantial infrastructure reasoning.
- Quality Control and Coverage: The final benchmark contains 85 tasks—55 KFC, 20 LHI, and 10 E2EO—covering all nine major infrastructure topic categories.Quality control requires stable reference performance of at least 1.15 for performance tasks, while implementation tasks include at least five behavioral test cases.
Evaluation
Evaluation shows substantial limitations in frontier models’ LLM infrastructure engineering, with the best model reaching only 36.53% overall and no model performing uniformly across domains. Iterative refinement generally helps, but reasoning-budget increases are inconsistent and can be critical for some models.
- Overall performance: 36.53%: Claude Opus 5 achieves the highest overall score, followed by Kimi K3 at 28.12% and Qwen3.8 Max at 27.73%.Even with complete repositories and test cases, the strongest model reaches only slightly more than one-third of the maximum score.
- Domain performance: Claude Opus 5 leads five of nine categories, while Kimi K3, Qwen3.7 Max, and GLM 5.2 each lead specialized infrastructure domains.No model is uniformly strong across all categories, and Hardware & Edge is particularly challenging.
- Performance over iterations: Most models can iteratively refine solutions, with Claude Opus 5 continuously improving after a very low first-submission BPB score.Qwen3.8-Max and Kimi K3 perform poorly early but rapidly refine their approaches.
- Performance over reasoning budget: All three evaluated models perform best at maximum reasoning effort, but scores do not increase consistently with reasoning budget.GPT-5.6 Sol shows notable instability at intermediate effort levels.
- Performance over reasoning budget: 45%: Kimi K3 loses about 45% of its score in the low setting compared with the max setting, indicating strong dependence on substantial test-time reasoning compute.The performance gap between Claude Opus 5 and GPT-5.6 Sol is not particularly significant across budget settings.
Analysis
The analysis examines model trajectories, error patterns, task-format performance, and potential hacking during evaluation. Higher-scoring models generally attempt more corrections, Claude Opus 5 leads all task formats, and anti-hacking safeguards prevented confirmed cheating.
- Error modes: The evaluation categorizes trajectory errors into Python runtime, CUDA execution, Triton / MLIR / CUDA compile, and tensor shape mismatch classes.The analysis collects error messages and reports both error counts and type distributions across models.
- Error modes: Higher-scoring models—Claude Opus 5, Kimi K3, and Qwen3.8 Max—produce more errors, consistent with more trial, diagnosis, and correction rounds.Weaker models produced fewer errors, suggesting limited attempts or simpler solutions.
- Error modes: Python runtime errors comprise more than half of errors for most models, while Claude Opus 5 has fewer of them and a larger CUDA execution-error share.This indicates stronger first-attempt Python code generation within the repository and greater focus on challenging errors.
- Task-format performance: Claude Opus 5 ranks first across all formats, scoring 37.16% on KFC, 21.60% on LHI, and 62.94% on E2EO.Kimi K3 ranks second overall, while Qwen3.8 Max ranks third with consistently strong performance.
- Anti-hacking evaluation: Three DeepSeek V4 Pro detector triggers involved requests to retrieve PyTorch code, but the Proctor Agent confirmed no actual hacking behavior.Soft network disconnection and explicit prompting were used to minimize hacking behavior.
Case Study
The case study shows that effective long-horizon infrastructure optimization depends on low-cost validation, informative controlled experiments, and careful checks for confounding factors. Claude Opus 5 maintains a coherent strategy across rounds, whereas weaker models more often rely on costly or local decisions that impede cumulative learning.
- Hypothesis screening: Claude Opus 5 uses lightweight local validation to screen candidate hypotheses before formal submission, reducing the cost of unsuccessful optimization iterations.Its local experiments reproduce only the relevant workload and compare short runs against development measurements.
- Search strategy: Opus explores candidate directions broadly before focusing on the most promising ones, forming a coherent search strategy across optimization rounds.By contrast, Qwen3.7 Max and DeepSeek V4Pro largely rely on implementation–submission–observation loops without preliminary validation.
- Experimental design: Opus accounts for measurement variation, completed optimization steps, and effect size when interpreting experiments, rather than relying on variable isolation alone.A learning-rate schedule change worsened BPB from 1.3646 to 1.3932 under matched seed, step count, and nearly identical runtime, making the degradation reliable evidence.
- Causal interpretation: Opus checks alternative explanations and confounding factors before attributing performance changes to code modifications.It detected Inductor compilation-cache invalidation, repeated the comparison under matched cache conditions, and found unchanged throughput while retaining a BPB improvement.
- Overall trajectory: Across rounds, Opus updates its plan using reliable evidence, while weaker models more often make local, greedy decisions that prevent useful knowledge from accumulating.The trajectories indicate that long-horizon infrastructure optimization requires more than reliable local execution.
Conclusion
Φ-Bench evaluates LLM infrastructure engineering across 85 tasks spanning nine domains and three increasingly open-ended formats, revealing substantial but uneven model capabilities. Claude Opus 5 leads at 36.53%, yet no model performs consistently well.
- Benchmark scope and construction: Φ-Bench comprises 85 tasks spanning nine LLM infrastructure domains and three increasingly open-ended task formats.Its taxonomy-guided, agent-assisted construction pipeline grounds task synthesis in systems research and public repositories while enabling coverage and scalability.
- Evaluation findings: Claude Opus 5 leads the evaluation with 36.53%, but no model performs consistently well.The results reveal substantial but uneven capabilities across eight frontier models.