Source-linked AI summary
On Evaluating the Efficiency of Source Code Generated by LLMs
Changan Niu, Ting Zhang, Chuanyi Li, Bin Luo, Vincent Ng
TL;DR
The efficiency of LLM-generated code has received less attention than its correctness, although efficient code may improve software performance and developer acceptance. The paper evaluates runtime on entry-level and more complex benchmarks and explores prompts for improving efficiency. It reports that efficiency is independent of correctness-generation performance and model size, while step-by-step prompting could yield more efficient code, especially on complex problems.
Problem
The efficiency of LLM-generated code is overlooked despite its potential relevance to software performance and developer acceptance.
Source: §1 · p. 1Method
The study measures runtime on HumanEval and MBPP and a LeetCode-based complex-problem benchmark, and explores prompts for more efficient code.
Source: §1 · p. 2, §2.1.1 · p. 2Results
LLM-generated code efficiency is independent of correctness-generation performance and model size, while step-by-step prompting could improve efficiency, especially on complex problems.
Source: §5 · p. 4Takeaways & Limitations
The study suggests code-efficiency research as an avenue for improvement and offers practical insights for model selection.
Source: §5 · pp. 4–5The evaluation cannot determine whether other models' training data included the evaluated problems because their data cutoffs are unknown.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Recent years have seen the remarkable capabilities of large language models (LLMs) for code generation. Different from existing work that evaluate the correctness of the code generated by LLMs, we propose to further evaluate its efficiency. More efficient code can lead to higher performance and execution efficiency of programs and software completed by LLM-assisted programming. First, we evaluate the efficiency of the code generated by LLMs on two benchmarks, HumanEval and MBPP. Then, we choose a set of programming problems from the online judge platform LeetCode to conduct a more difficult evaluation. Finally, we explore several prompts that would enable LLMs to generate more efficient code.
1 INTRODUCTION
LLMs can generate correct code, but the efficiency of that code remains overlooked despite its potential relevance to software performance and developer acceptance. This paper studies code efficiency and prompting for more efficient code.
1 INTRODUCTION
- Efficient code may improve program performance, increase developer acceptance, and reduce further optimization needs, as illustrated by differing prime-checking complexities in two correct HumanEval solutions.GPT-3.5's solution is O(√n), while Code Llama's is O(n).Source: §1 · p. 1
- The study asks how efficient LLM-generated code is and how prompting can make it more efficient.Source: §1 · p. 2
- The authors evaluate runtime on HumanEval, MBPP, and a more complex benchmark, then test prompts for efficiency.They report that simple prompts help on basic problems, while chain-of-thought prompting benefits complex problems.Source: §1 · p. 2
- The paper's contributions include evaluating generated-code efficiency, proposing a LeetCode-based benchmark for complex code, and investigating efficiency-focused prompts.The authors state these results may guide model selection and benefit developers and organizations using the models.Source: §1 · p. 2
2 APPROACH AND EXPERIMENTS
The study evaluates LLM-generated code efficiency across HumanEval, MBPP, and a more complex LeetCode benchmark, then tests prompts intended to improve efficiency. It finds that efficiency and correctness do not consistently align, and prompt effectiveness varies with problem complexity.
2.1.1 Datasets.
- LeetCodeEval uses recent LeetCode problems, filtered to avoid data leakage and image-based descriptions, and divides them into easy, medium, and hard subsets.The dataset contains 44 easy, 85 medium, and 33 hard problems; LeetCode supplies test cases, runtime, and percentage-beats information for accepted code.Source: §2.1.1 · p. 2, Table 1 · p. 2
2.1.2 Study Design.
- For HumanEval and MBPP, the study selects the first passing code among k generated samples and averages runtime across ten simulator executions.The study uses gem5 to reduce noise from running code on real hardware; LeetCodeEval instead evaluates three generated C++ solutions per problem on the platform, repeating submissions three times.Source: §2.1.2 · pp. 2–3
2.1.3 Models.
- The evaluation selects commercial and open-source LLMs achieving SOTA performance on HumanEval and MBPP, including GPT-3.5, GPT-4, Phi-2, Code Llama, WizardCoder, and DeepSeek Coder 33B base and instruct.Source: §2.1.3 · p. 3
2.1.4 Metrics.
- Runtime comparisons use average normalized runtime on problems passed by all models, with each problem’s maximum test-case runtime as its runtime measure.The study also reports Pass@10 and, for LeetCodeEval, average percentage beaten; no hard-subset comparison is possible because no problem is passed by all models.Source: §2.1.4 · p. 3
2.1.5 Results.
- Correctness does not consistently predict efficiency: GPT-4 is less efficient than GPT-3.5 on HumanEval and MBPP despite higher Pass@10, while the lowest-runtime model varies by benchmark.DeepSeek Coder 33B Instruct has the lowest runtime on HumanEval, WizardCoder on MBPP, and GPT-4 has the highest average efficiency on LeetCodeEval.Source: §2.1.4 · p. 3, §2.1.5 · p. 3, Table 2 · p. 3
2.2 RQ2: Prompting for More Efficient Code
- Prompting generally works better on LeetCodeEval than on HumanEval and MBPP, and chain-of-thought optimization improves medium-subset results where direct efficiency prompting struggles.The three methods directly request efficient code, optimize initially generated code, or analyze optimization strategies before generating optimized code; Table 4 reports their speedups.Source: §2.2 · p. 4, Table 4 · p. 4
3 THREATS TO VALIDITY
The evaluation faces construct-validity risk from possible training-data leakage and internal-validity risk from unstable runtime measurements.
3 THREATS TO VALIDITY
- Training-data leakage threatens construct validity because the authors cannot know whether evaluation problems appeared in models’ training data.They select LeetCode problems after April 2023, but lack data cutoffs for other models.Source: §3 · p. 4
- Runtime instability threatens internal validity, so evaluations are run multiple times using the gem5 CPU simulator.Source: §3 · p. 4
4 RELATED WORK
Prior work addresses software performance optimization and code quality, while this paper focuses on the efficiency of LLM-generated code.
4 RELATED WORK
- DeepDev-PERF generates the same performance-improvement suggestions as developer patches in 53% of cases.Source: §4 · p. 4
- Madaan et al. introduce PIE, a runtime-annotated C++ program-pair dataset, and evaluate prompting and fine-tuning for optimization.Self-Refine [20] uses iterative self-feedback and refinement to improve LLM performance on PIE.Source: §4 · p. 4
- Other studies assess LLM-generated code quality through coding-standard adherence, code smells, security smells, validity, and maintainability [24] [26].Source: §4 · p. 4
5 CONCLUSION AND FUTURE WORK
The study concludes that generated-code efficiency is independent of correctness performance and model size, and that step-by-step prompting could improve efficiency, especially on complex problems. It offers model-selection insights and identifies a novel prompt method as future work.
5 CONCLUSION AND FUTURE WORK
- Generated-code efficiency is independent of models’ performance on correctness and their size.Source: §5 · p. 4
- Step-by-step prompting could generate more efficient code, especially on complex problems.Source: §5 · p. 4
- The study offers practical insights for model selection, while future work will propose a novel prompt method to enhance code efficiency.Source: §5 · p. 5