PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback

1 Introduction

• Code-LM evaluations have focused almost exclusively on functional correctness, often overlooking runtime efficiency and other code-quality aspects.Runtime efficiency affects user experience, serving costs, and carbon footprint.Source: §1 · p. 1
• Prior performance-focused fine-tuning methods require parallel training data and a functionally correct input, and do not use unit-test execution feedback.Prompting and tool-feedback approaches also often fail to assess or improve code quality beyond functionality.Source: §1 · p. 2
• PERFCODEGEN first uses unit-test feedback to refine correctness, then targets the most expensive test with runtime feedback for performance refinement.It is evaluated on HumanEval, MBPP, and APPS using open and closed models of varying sizes.Source: §1 · p. 2

2 PERFCODEGEN Methodology

• PERFCODEGEN samples K candidate programs and uses an execution environment to collect correctness feedback before its correctness and performance phases.The framework is illustrated in Figure 1.Source: §2 · p. 3

2.1 Generating Correct Code

• Incorrect candidates are iteratively refined using a failing unit test and verbalized execution feedback, then checked again for correctness.The method performs one correctness-refinement iteration to limit computational costs, as gains diminish after the first iteration.Source: §2.1 · p. 3

2.2 Optimising Correct Code using Execution Feedback

• For each correct candidate, the method measures execution time across unit tests, identifies the slowest test, and feeds it back to guide performance refinement.Execution-time estimates exclude the smallest and largest of E observations to reduce measurement noise; if a refinement fails correctness, the fastest correct candidate is returned.Source: §2.2 · pp. 3–4
• The final solution is the fastest correct refined output, or the fastest correct base candidate when no refinement passes.Performance refinement uses one greedy-decoded refinement per correct candidate to reduce inference costs.Source: §2.2 · p. 4

3.1 Setup: Metrics, Datasets and Models

• The experiments compare the fastest correct program among k samples using %Opt, %Correct, and speedup, estimating execution time with 12 executions per unit test.%Opt counts problems where a generated program is at least 10% faster than the ground truth; APPS uses its fastest ground-truth solution as reference.Source: §3.1 · pp. 4–5

3.3 Evaluating Alternatives to PERFCODEGEN’s Execution Feedback

• With GPT-3.5 at k = 1, PERFCODEGEN gains 6.17% in %Opt on HumanEval and 3.06% on MBPP, outperforming the alternative prompting approaches.Other performance-improvement prompting, multi-round, and multi-agent approaches produce insignificant or often negative gains; direct execution feedback yields a 5.55% gain on HumanEval and a 1.31% drop on MBPP.Source: §3.3 · pp. 7–8, Table 3 · p. 8

3.4 Role of Planning in Correctness Refinement

• Across HumanEval and MBPP with GPT-3.5 and Llama 3 70B, planning generally yields higher correctness than the direct approach at k = 8 and 20, and PERFCODEGEN achieves the highest correctness rates on both benchmarks.At k = 1, PERFCODEGEN’s additional planning step often leads to slightly lower correctness gain than the direct approach.Source: §3.4 · p. 9

4 Related Work

• Earlier code-to-code editing work targets bug fixing, performance, maintainability, and security, whereas this work frames runtime improvement as text-to-code prompting.The authors argue that code-quality research should study natural-language inputs expressing developer intent or program specifications.Source: §4 · p. 9
• Preference-based methods such as RLHF and DPO rely on human preference data, while RL also entails costly fine-tuning and training instability.The passage notes that large-scale code preference data may be more difficult and expensive to collect than natural-language response preferences.Source: §4 · p. 9
• PERFCODEGEN extends LLM self-refinement and tool-feedback approaches from functional correctness to runtime efficiency in code generation.Prior approaches add planning or analysis stages, or use environment feedback; Reflexion focuses on functional correctness.Source: §4 · p. 9

5 Conclusion

• PERFCODEGEN achieves state-of-the-art runtime efficiency on a sizeable fraction of HumanEval and MBPP problems, sometimes far exceeding ground-truth solutions with GPT-4.The authors also report consistent and significant gains in correctness and runtime efficiency across three Python benchmarks and open and closed models of varying sizes.Source: §5 · p. 9
• PERFCODEGEN incorporates code execution feedback into iterative LLM self-refinement to improve the runtime efficiency of generated code.The authors report that Phi-3-mini can achieve code quality comparable to GPT-4 with this approach.Source: §5 · p. 9
• The authors identify integrating execution feedback into code generation as a path toward more robust and reliable AI-driven software development.Source: §5 · p. 10

A Limitations

• The evaluation focuses on performant modules or Python functions, not architectural patterns across entire project repositories.The authors identify minimal redundancy and wasteful computation across a project as broader concerns outside the current scope.Source: A Limitations · p. 14
• PERFCODEGEN measures runtime performance but not memory consumption, readability, maintainability, security, or harmlessness.The authors say balancing optimization across these dimensions is non-trivial.Source: A Limitations · p. 14
• Runtime measurements can vary across compute environments, and averaging 10 independent runs adds execution cost.The study controls environments and runs one program at a time; future work could explore deterministic instruction counts.Source: A Limitations · p. 14

B.1 Sanitized Benchmarks

• Table 5 reports each sanitized benchmark’s problem count and average ground-truth-program and testcase counts per problem.The table distinguishes benchmarks used for correctness evaluation from common subsets used for time-efficiency evaluation.Source: Table 5 · p. 14
• The evaluation retains 3,249 instances for correctness after excluding APPS instances whose ground-truth solutions fail the exact-match tests.The authors identify 1,511 APPS instances with ground-truth failures, attributing the discrepancy to exact-match evaluation versus APPS similar-match criteria and possible incorrect references.Source: §B.1 · p. 14
• Time-efficiency data come only from benchmark subsets where every baseline method generates at least one functionally correct program.Source: §B.1 · p. 15

• Inference ran on 16 Nvidia A100 GPUs with vLLM for approximately three weeks.The experiments used gpt-4-0613 and gpt-3.5-turbo-0125 endpoints from OpenAI.Source: §B.2 · p. 15
• The OpenAI API experiments required nearly 411k GPT-4 and 1.5M GPT-3 requests, contributing to the study’s major costs.Source: §B.2 · p. 15
• Executing generated programs used 40 16-CPU, 104-GB-RAM GCP instances for roughly four weeks and cost more than LLM inference.The setup also implemented safeguards prescribed in Section 2.3 of Chen et al. [2021] to mitigate risks from untrusted programs.Source: §B.2 · p. 15

• PERFCODEGEN yields significant gains in %Opt compared with the base LLM.Source: §B.3 · p. 15
• The improvement is statistically significant at α < 0.05 on MBPP and APPS, but reaches only α < 0.2 on HumanEval.The authors attribute the lower confidence on HumanEval to its smaller problem count of 164.Source: §B.3 · p. 15
• For GPT-4, Z-scores at k = 1 and k = 8 are −0.878 and −1.34 on HumanEval, −2.588 and −2.947 on MBPP, and −1.8 and −2.675 on APPS.Source: §B.3 · p. 15

• The authors suggest more efficient code could accelerate software development, reduce development costs and execution energy, and make open LLMs more competitive with closed models.They also propose that it could widen access to basic programming and offer a route to discovering more efficient algorithms for longstanding problems (Romera-Paredes et al. [2024]).Source: §B.4 · p. 15
• Potential risks include malicious-code misuse, vulnerabilities in optimized code, and debugging or maintenance challenges if generated code is less readable.The authors recommend filtering, malicious-code detection, testing, validation, and code reviews to help reduce these risks.Source: §B.4 · p. 15
• The authors note that further automation in the software industry could displace some programmers and call for discussion of re-skilling initiatives.Source: §B.4 · p. 15
• Table 6 reports PERFCODEGEN results on HumanEval, MBPP, and APPS with k = 20.Source: Table 6 · p. 16

B.6 Examples of Optimal Solutions Generated by LLMs

• The FibFib example asks for an efficient n-th sequence element and labels the displayed PERFCODEGEN solution as optimal.Source: §B.6 · p. 17
• The generated FibFib code updates three running values in a loop, whereas the cited ground-truth solution uses recursive calls.Source: §B.6 · p. 17

B.7 Prompts

• The reflection-and-test-case-feedback prompt presents a failed test case and error, asks for a failure analysis and strategy, then requests a corrected solution.Source: §B.7 · p. 19, Figure 3 · p. 19
• The runtime-optimization prompts ask for a faster solution while preserving functional equivalence and the input-output format; one variant first asks for an optimization strategy.Source: §B.7 · pp. 20–23, Figure 4 · p. 20, Figure 5 · p. 21
• The prompt appendix also includes a Coder–Reviewer variant and a Leader–Coder–Reviewer team prompt.Source: Figure 8 · p. 24, Figure 9 · p. 25, §B.7 · pp. 25–26