EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization

1 Introduction

• Efficiency remains underexplored despite its importance for faster execution and lower resource consumption, and LLM code can be less efficient than canonical solutions.On EffiBench, GPT-4-Turbo's average and worst-case execution times are reported as 1.69 and 45.49 times those of canonical solutions.Source: §1 · p. 2
• EffiLearner executes LLM-generated code locally, feeds its execution-time and memory profiles back to the LLM, and iteratively revises code to reduce overhead.The authors note that iteration takes extra time, while optimized code can improve efficiency after deployment.Source: §1 · p. 2
• The evaluation compares code before and after optimization on EffiBench, HumanEval, and MBPP using 16 open-source and 6 closed-source models.The reported outcomes include improvements in execution time and memory-use metrics.Source: §1 · pp. 2–3
• ET for StarCoder2-15B fell from 0.93 (s) to 0.12 (s), while TMU fell from 22.02 (Mb*s) to 2.03 (Mb*s).These correspond to reported reductions of 87.1% in execution time and 90.8% in total memory consumption.Source: §1 · p. 3

2.1 LLMs for Code Generation

• Code LLMs support generation and other programming tasks, but their execution-time and memory efficiency has received less attention than their capabilities and correctness.Prior studies report lower efficiency than canonical solutions [57] [27] [49].Source: §2.1 · p. 3

2.2 Learning From Feedback

• Feedback-driven code refinement commonly uses execution results to improve correctness, whereas EffiLearner uses execution overhead profiles to improve efficiency.Automated feedback in prior work can also come from LLMs, tools, verifiers, external knowledge, or generation logits.Source: §2.2 · p. 3

3 EFFI-LEARNER

• EffiLearner's three components are Code Generation, Overhead Profiling, and Code Refinement, arranged as a self-optimization loop.The LLM-generated code is executed locally, and its profiles guide successive revisions.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Code Generation

• For a task description or code-generation requirement, the LLM produces an initial code version intended to solve the task.The task description is the input and the initial code is the output.Source: §3.1 · p. 4

3.2 Overhead Profiling

• Overhead profiling runs generated code on open test cases and records line-level execution-time and memory-use information.Execution-time profiling uses line_profiler and memory profiling uses memory_profiler; both report total consumption across the open test cases.Source: §3.2 · p. 4

3.3 Code Refinement

• The LLM uses overhead profiles to identify costly code segments, apply optimizations, and refine code until a target efficiency or iteration limit is reached.Possible optimizations include loop unrolling, memorization, data-structure optimization, algorithm substitution, and code simplification; optimized code is checked against open test cases.Source: §3.3 · pp. 4–5

3.4 Prompt Construction

• The optimization prompt combines the task description, test case, initial code, overhead analysis, and rules for producing an efficient answer.The prompt is designed to guide optimization while ensuring the optimized code passes predefined test cases.Source: §3.4 · p. 5

4.1 Dataset and Metrics

• EffiBench uses ET, NET, MU, NMU, TMU, and NTMU; final efficiency metrics are calculated only when both initial and optimized code pass private tests.HumanEval and MBPP use their supplied tests as open tests and EvalPlus tests as private tests.Source: §4.1 · p. 5

4.3 Main Results

• Efficiency metrics improved for most closed-source models, with improvements reported across open- and closed-source LLMs; GPT-3.5-Turbo-0301 TMU fell from 157.50 (Mb*s) to 12.43 (Mb*s).The table reports code-efficiency results for LLMs with EFFI-LEARNER on EffiBench.Source: Table 1 · p. 6, §4.3 · p. 6

4.2 Implementation Details

• The evaluation uses open- and closed-source models and retains initially correct code for efficiency evaluation across optimization iterations.Experiments were conducted on an edge server with an Intel Xeon Platinum 8336C CPU and eight NVIDIA A100-SXM GPUs.Source: §4.2 · pp. 5–6

4.4 Impact of Self-Optimization Steps

• The first self-optimization step yields the largest efficiency gains for CodeLlama-70B and GPT-3.5-Turbo-0301, with later steps bringing diminishing returns.By the fifth step, ET reaches 0.47 (s) for CodeLlama-70B and 0.28 (s) for GPT-3.5-Turbo-0301.Source: §4.4 · p. 7

4.5 Feedback of Overhead Profile

• Profile feedback outperformed prompt-only and aggregate-metric refinements: EFFI-LEARNER reduced ET, MU, and TMU, while alternatives increased TMU for GPT-3.5-Turbo-0301.The authors attribute the gains to overhead profiles providing resource-use information for targeted optimization.Source: §4.5 · pp. 7–8

4.6 Discussion

• Efficiency gains extend to HumanEval, MBPP, and additional LLMs, while pass@1 may fall by about 0%–0.5% when public tests miss private edge cases.CodeLlama-70B ET on HumanEval fell from 0.21 (s) to 0.18 (s).Source: §4.6 · pp. 8–9
• A case study changed unidirectional BFS to bidirectional BFS, reducing ET from 23.59 (s) to 3.36 (s).The passage explains that bidirectional search reduces the number of states explored by meeting in the middle.Source: §4.6 · p. 10
• When initial and optimized code share O(log(min(m, n))) complexity, small inputs and runtime overhead leave little measurable efficiency headroom.In the example, ET changes from 0.0012 (s) to 0.0011 (s).Source: §4.6 · p. 10

5 Conclusion

• EFFI-LEARNER uses execution overhead profiles to guide LLMs in improving generated-code efficiency.The paper identifies efficiency as its focus, alongside slower execution and higher resource consumption in generated code.Source: §5 · p. 10
• Experiments and analysis report substantial reductions in execution time and memory usage for LLM-generated code.Source: §5 · p. 10
• Future work will examine other programming tasks and languages, and the potential benefits of domain-specific knowledge.Source: §5 · p. 10

A.1 Limitations

• Multi-iteration self-optimization can be time-consuming, particularly for intricate programming tasks.Source: §A.1 · p. 17
• Long overhead profiles may consume additional tokens when fed to LLMs.Source: §A.1 · p. 17
• EFFI-LEARNER has been evaluated primarily on Python, so performance in other languages or environments may vary.The authors call for further testing and validation in diverse contexts.Source: §A.1 · p. 17

A.2 Broader Impacts

• Lower resource consumption from optimized code could lead to reduced operational costs.Source: §A.2 · p. 17
• Faster execution and lower resource use could benefit resource-constrained environments such as mobile devices and embedded systems.Source: §A.2 · p. 17
• Greater reliance on LLM-based tools could reduce human coding skills and understanding of underlying code.The authors also identify possible job displacement, particularly for routine coding tasks.Source: §A.2 · p. 17
• EFFI-LEARNER does not necessarily address security vulnerabilities or privacy issues in generated code.Source: §A.2 · p. 17

Prompt Template

• Figure 3 presents the prompt template used by EFFI-LEARNER during self-optimization.Source: Figure 3 · p. 17
• Table 7 reports evaluation results for EFFI-LEARNER and baselines.Source: Table 7 · p. 18
• Table 7 uses fine-tuned CodeLlama 7B because the GPT-3.5-Turbo PIE fine-tuned link was unavailable, and omits initial results because task sets differed.Source: Table 7 · p. 18

• ET measures average execution time; Tcode is the code’s execution time across all test cases, and N is the number of generated code instances.Source: §A.5 · p. 19
• NET compares generated-code runtime with the canonical solution; values above 1 indicate slower code, while values below 1 indicate faster code.Source: §A.5 · p. 19
• MU measures average maximum memory consumption during execution, using each code instance’s maximum across test cases.Source: §A.5 · p. 19
• NMU compares generated-code peak memory with the canonical solution; values below 1 indicate greater memory efficiency, while values above 1 indicate less.Source: §A.5 · p. 20
• TMU integrates the memory-usage profile over total execution time, capturing both memory magnitude and duration; lower values indicate greater memory efficiency.The profile is sampled at discrete time intervals, and numerical integration methods such as the trapezoidal rule estimate its area.Source: §A.5 · p. 20
• NTMU compares generated-code TMU with canonical-solution TMU, with values below 1 indicating more efficient dynamic-memory use and values above 1 indicating less.Source: §A.5 · p. 20

A.6 Additional Related Work

• Code instruction-tuning work includes synthetic instruction data, Self-Instruct, Code Alpaca, WizardCoder, OSS-Instruct, and Octopack’s filtered commit messages.The discussion describes these as approaches to producing or collecting instruction data for code-related tasks.Source: §A.6 · p. 20

1. Claims; 2. Limitations

• The authors affirm that EFFI-LEARNER improves LLM-generated code efficiency and state that its limitations are discussed in the appendix.Source: §1 · p. 35, §2 · p. 35

3. Theory Assumptions and Proofs

• The checklist marks theoretical assumptions and proofs as not applicable because the paper has no theoretical results.Source: §3 · pp. 35–36

4. Experimental Result Reproducibility; 5. Open access to data and code

• The authors report that source code is supplied in the supplementary file and that dataset access instructions are provided.Source: §4 · p. 36, §5 · p. 37

6. Experimental Setting/Details; 7. Experiment Statistical Significance

• The authors say experimental settings appear in the paper and appendix, and greedy decoding makes results consistent across sessions.Source: §6 · p. 37, §7 · p. 37

8. Experiments Compute Resources; 9. Code Of Ethics; 10. Broader Impacts

• The authors say compute-resource information is in the appendix, report following the NeurIPS Code of Ethics, and say societal impacts are discussed there.Source: §8 · p. 38, §9 · p. 38, §10 · p. 38

11. Safeguards; 12. Licenses for existing assets; 13. New Assets

• The authors state that the paper poses no safeguard-related risks, cite evaluated models and datasets, and report releasing no new assets.Source: §11 · p. 39, §12 · p. 39, §13 · p. 40

14. Crowdsourcing and Research with Human Subjects

• The checklist marks crowdsourcing and research with human subjects as not applicable because the paper involves neither.Source: §14 · p. 40