PaperlayerView exact revision on arXiv

Source-linked AI summary

EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning

Dong Huang, Guangtao Zeng, Jianbo Dai, Meng Luo, Han Weng, Yuhao Qing, Heming Cui, Zhijiang Guo, Jie M. Zhang

arXiv:2410.10209v4cs.CLcs.SE

TL;DR

Existing LLM code-generation methods often prioritize correctness while neglecting efficiency. EffiCoder fine-tunes models on code selected for low runtime and memory use; experiments report a 48.4% reduction in execution-time overhead on EffiBench alongside improved correctness.

  • Problem

    Existing work primarily focuses on correctness, while LLM-generated code can be less efficient than human-written canonical solutions in execution time and memory usage.

    Source: §1 · p. 1

  • Method

    EffiCoder builds EFFIINSTRUCT by using multiple LLMs to generate candidate programs, measuring their local execution time and memory use, selecting efficient solutions, and fine-tuning LLMs on them.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    48.4% reduction in execution-time overhead on EffiBench accompanied reported improvements in correctness and efficiency across experiments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments support EFFIINSTRUCT fine-tuning as an approach that improves correctness and efficiency in the tested LLMs.

    Source: §1 · p. 2, §5 · p. 9

    Source-code biases, including language-specific optimizations or coding norms, may propagate through the framework, requiring careful dataset design and validation.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

As large language models (LLMs) play an increasingly important role in code generation, enhancing both correctness and efficiency has become crucial. Current methods primarily focus on correctness, often overlooking efficiency. To address this gap, we introduce EffiCoder to improve both aspects by fine-tuning LLMs on a high-quality dataset comprising correct and efficient code samples. Our methodology involves leveraging multiple LLMs to generate diverse candidate code solutions for various tasks across different programming languages. We then evaluate these solutions by measuring their execution time and memory usage through local execution. The code solution with the lowest execution time and memory consumption is selected as the final output for each task. Experimental results demonstrate significant improvements when fine-tuning with Effi-Instruct. For instance, Qwen2.5-Coder-7B-Instruct's pass@1 score increases from 44.8\% to 57.7\%, while the average execution time for correct tasks decreases by 48.4\%. EffiCoder offers a scalable and effective solution for advancing AI-driven code generation, benefiting software development and computational problem-solving. The source code of Effi-Code was released at https://github.com/huangd1999/EffiCoder.

1. Introduction

EffiCoder addresses the neglect of code efficiency in LLM training by selecting fast, memory-efficient solutions for fine-tuning data. The paper reports gains in code correctness and execution time after fine-tuning with EFFIINSTRUCT.

  • GPT-4-Turbo code averaged 1.69 times and worst-case 45.49 times longer execution times than canonical solutions, motivating efficiency-aware training for energy, cost, and resource-constrained-device concerns.Source: §1 · pp. 1–2
  • EffiCoder aggregates open-source code, preprocesses tasks and creates test cases, then locally measures execution time and memory for diverse LLM-generated candidates across programming languages and selects the lowest-resource solutions.The selected code and its metadata form EFFIINSTRUCT.Source: §1 · p. 2
  • After EFFIINSTRUCT fine-tuning, Qwen2.5-Coder-7B-Instruct (Hui et al., 2024) raises pass@1 from 44.8% to 57.7% on EffiBench and from 76.2% to 78.0% on HumanEvalPlus, while average EffiBench execution time drops 48.4%, from 0.31 to 0.16 seconds.Source: §1 · p. 2
  • On HumanEvalPlus, EFFIINSTRUCT-tuned CodeLlama-7B (Rozière et al., 2023) reaches 31.1% pass@1 versus PIE’s 19.5% (from 12.2%), and cuts execution time 46.2% versus PIE’s 23.1% (Shypula et al., 2024).Source: §1 · p. 2
  • EffiCoder offers a versatile framework across languages and sources, builds EFFIINSTRUCT as an instruction-tuning dataset targeting code efficiency, and fine-tunes widely used LLMs to improve correctness and efficiency.The paper describes EFFIINSTRUCT as, to its knowledge, the first instruction-tuning dataset designed to improve efficiency of LLM-generated code.Source: §1 · p. 2

2. Related Works

The related work traces code LLMs and synthetic instruction-tuning methods, noting that prior coding approaches primarily emphasize correctness while EffiCoder targets execution efficiency through fine-tuning.

2.1. LLMs for Code

  • Code LLMs evolved from coding-specific models to foundation-model fine-tunes and support generation, repair, testing, translation, type prediction, and summarization.Examples include CodeT5, AlphaCode, CodeGen, InCoder, StarCoder, SantaCoder, DeepSeek-Coder, Codex, and CodeLlama.Source: §2.1 · p. 2
  • Prior studies find LLM-generated code less efficient than canonical solutions in execution time and memory, while this work proposes fine-tuning to improve efficiency and correctness.The passage describes this as the first fine-tuning method to significantly improve both aspects across various LLMs.Source: §2.1 · p. 2

2.2. Instruction Tuning for Code

  • Synthetic instruction-tuning approaches include Self-Instruct, Code Alpaca, and WizardCoder, which generate instruction-response data or more complex coding instructions.Self-Instruct uses LLM-generated instruction-response pairs; Code Alpaca adapts the approach to coding tasks, and WizardCoder applies Evol-Instruct with heuristic prompts.Source: §2.2 · p. 2
  • OSS-Instruct generates coding problems inspired by open-source code snippets, whereas Octopack collects and filters high-quality Git commit messages as natural-language instructions.EffiCoder focuses on improving code efficiency through fine-tuning, an approach described as orthogonal to existing synthetic techniques and potentially combinable with them.Source: §2.2 · p. 3

3. EFFICODER: Fine-Tuning For Efficiency

EffiCoder tests whether efficient training code improves generated-code efficiency, then constructs EFFIINSTRUCT by selecting efficient solutions for tasks drawn from open-source datasets. The dataset spans five programming languages and contains solutions with lower execution-time and memory-use means than the original solutions.

3.1. Preliminary Study

  • Training-data efficiency correlates strongly with generated-code efficiency: 0.972 for ET, 0.950 for MU, and 0.986 for TMU.The study measures Execution Time (ET), Max Memory Usage (MU), and Total Memory Usage (TMU).Source: §3.1 · p. 3, Figure 3 · p. 4

3.2. Dataset Construction

  • EFFIINSTRUCT selects the candidate with the lowest execution time and memory usage after multiple LLMs generate solutions that are run locally.Candidate tasks come from nine open-source datasets; filtering reduced an initial pool of about 790k tasks to 65k.Source: §3.2 · pp. 3–4

3.3. Dataset Statistics

  • EFFIINSTRUCT contains 65,710 tasks across Python, C++, Java, Rust, and Go.The dataset includes 33,489 Python, 11,547 C++, 14,726 Java, 4,270 Rust, and 1,678 Go tasks; analysis found no exact train-evaluation duplicates and only 0.20% minimal vocabulary overlap.Source: §3.3 · p. 4
  • The dataset combines a diverse range of coding challenges with optimized solutions produced through curation and SOAP optimization.The authors present it as a resource for training models to generate efficient code.Source: §3.3 · pp. 4–5
  • Efficient solutions average 0.31s execution time and 6.03 MBs memory usage, versus 1.14s and 26.50 MBs for inefficient solutions.The figure also compares max memory peak and reports higher overheads for inefficient solutions across all three metrics.Source: §3.3 · p. 4, Figure 4 · p. 5

4. Experiment

Across Python, data-science, and multilingual code-generation benchmarks, fine-tuning with EFFIINSTRUCT generally improved code efficiency and, often, correctness. The experiments also compare the fine-tuned model with efficiency-focused baselines and describe the evaluation and training setup.

4. Experiment

  • The experiments covered five code-generation benchmarks and fine-tuned eight open-source LLMs with fully supervised training using Llama-factory.Training used a maximum sequence length of 2048 tokens, batch size 128, learning rate 5e-6, cosine scheduling, 0.03 warmup ratio, and four bf16 epochs.Source: §4 · p. 5

4.1. Evaluation of Python Code

  • On EffiBench, Qwen2.5-Coder-7B-Instruct’s correct-code average ET fell 48.4% (0.31 to 0.16 s), while HumanEvalPlus pass@1 for Qwen2.5-Coder-7B rose from 40.2% to 78.7%.On EffiBench, DeepSeek-Coder-6.7B-Instruct TMU fell from 30.05 to 9.48 Mb*s, and its ET fell 35.3%; its MU fell from 47.26 to 36.31 Mb, a 23.2% reduction. HumanEvalPlus average ET for Qwen2.5-Coder-7B-Instruct fell from 0.52 to 0.32 s (38.5%); on MBPPPlus, deepseek-coder-6.7b-base ET and TMU fell 36.7% and 40.5%.Source: §4.1 · pp. 5–6, Table 2 · p. 6, Table 3 · p. 6
  • On EvoEval’s Tool_Use, Subtle, and Creative benchmarks, fine-tuning reduced execution time and memory in most reported model-benchmark pairs, while pass@1 improved or remained unchanged.On Tool_Use, Qwen2.5-Coder-7B-Instruct ET and TMU fell 42.9% and 41.6%, and deepseek-coder-6.7b-instruct ET and TMU fell 41.5% and 15.9%. On Subtle, pass@1 rose from 55.0% to 72.0% and from 56.0% to 69.0%; on Creative, Qwen2.5-Coder-7B-Instruct pass@1 stayed at 44.0%, while deepseek-coder-6.7b-instruct rose from 31.0% to 41.0%.Source: §4.1 · pp. 6–7

4.2. Data Science Programming

  • On DS-1000, Qwen2.5-Coder-7B-Instruct pass@1 rose from 17.00% to 35.70%, while its ET and TMU fell 0.8% and 0.9%.For deepseek-coder-6.7b-instruct, ET and TMU fell 4.5% and 5.6%, and MU and NMU fell 1.9% and 2.0%; pass@1 remained 37.50%.Source: §4.2 · p. 7

4.3. Different Programming Language

  • On HumanEval-X, fine-tuning improved efficiency and pass@1 for C++ tasks, and deepseek-coder-6.7b-instruct’s Java pass@1 rose from 14.63% to 57.93%.For C++, Qwen2.5-Coder-7B-Instruct pass@1 rose from 7.32% to 48.17%, while deepseek-coder-6.7b-instruct rose from 14.63% to 40.24%. For Java, deepseek-coder-6.7b-instruct ET, MU, and TMU fell 36.5%, 50.5%, and 30.4%, respectively.Source: §4.3 · pp. 7–8

4.4. Comparison with Baselines

  • On HumanEvalPlus, fine-tuned CodeLlama-7b-hf reduced average execution time by 46.2% versus PIE and Mercury, while pass@1 rose from 19.5% and 9.1% to 31.1%.Compared with PIE, it also reduced NET by 46.9%, TMU by 43.9%, and NTMU by 43.2%; compared with Mercury, the reductions were 47.9%, 45.6%, and 46.4%, respectively. Both methods overlapped with the fine-tuned model on 4.3% of tasks.Source: Table 7 · p. 8, §4.4 · p. 8

5. Conclusion

EffiCoder introduces Effi-Instruct to improve both the correctness and execution efficiency of LLM-generated code through fine-tuning. The authors report performance gains for open-source models and release the model weights, training data, and source code to support further research.

5. Conclusion

  • Effi-Instruct is a dataset designed to improve the correctness and execution efficiency of LLM-generated code through fine-tuning.The authors describe it as a resource built through aggregation, preprocessing, and iterative optimization.Source: §5 · p. 9
  • Experiments show increased pass rates and decreased execution time for open-source models including DeepSeek-Coder and Qwen.The conclusion presents these gains as evidence of EffiCoder's potential in resource-constrained environments.Source: §5 · p. 9
  • The authors released the model weights, training data, and source code to foster further research and innovation.Source: §5 · p. 9

Impact Statement

EffiCoder aims to improve the efficiency and correctness of LLM-generated code, with potential benefits for sustainability and resource-constrained computing. The paper also notes risks involving data provenance, bias, developer skills, and trade-offs among efficiency, maintainability, and fairness.

Societal Benefits

  • Reducing computational resource use could lower energy demands and operational costs while enabling LLM-generated code in mobile, embedded, and edge systems.The paper frames these as potential societal benefits of more efficient code generation.Source: Impact Statement · p. 9
  • Open-sourcing the dataset and models is intended to foster transparency and accelerate research into sustainable AI and code optimization.Source: Impact Statement · p. 9

Ethical Considerations

  • The dataset comes from publicly available Hugging Face open-source repositories, but future curation should continue to prioritize responsible practices.The paper states that this sourcing ensures compliance with licensing terms.Source: Impact Statement · p. 9
  • Biases in source code, including language-specific optimizations and cultural coding norms, may propagate despite the framework’s multilingual adaptability.The paper calls for careful dataset design and validation to mitigate these biases.Source: Impact Statement · p. 9
  • Widespread use of optimized LLM-generated code could influence coding practices, so developers should retain critical problem-solving skills.Source: Impact Statement · p. 9
  • The paper encourages future research on trade-offs among efficiency, maintainability, and fairness in automated code generation.Source: Impact Statement · p. 9

A. Appendix · A.1. Prompt Template

The prompt template asks the model to complete the provided function without modifying the given code and return all completed functions in a code block.

A.1. Prompt Template

  • The template instructs the model to complete the function without modifying the provided code and return all completed functions in a code block.It inserts the task code at the {{Prompt}} placeholder.Source: §A.1 · p. 14

A.2. Efficiency Metrics

The section defines absolute and canonical-normalized measures of runtime and memory efficiency, including peak and duration-weighted memory usage.

A.2. Efficiency Metrics

  • Execution time (ET) averages runtime across generated code, while normalized execution time (NET) compares runtime with the canonical solution; NET below 1 means faster execution.ET covers execution across all test cases; NET values above 1 indicate slower generated code.Source: §A.2 · p. 14
  • Max memory usage (MU) averages peak memory across generated code; normalized max memory usage (NMU) compares it with canonical peak memory, with values below 1 indicating greater efficiency.MU is relevant where maximum memory capacity is limited.Source: §A.2 · p. 14
  • Total memory usage (TMU) integrates memory use over execution time; normalized total memory usage (NTMU) compares generated code with the canonical solution, and values below 1 indicate greater dynamic-memory efficiency.TMU is calculated as the area under the memory-usage profile over total execution time; lower TMU indicates higher memory efficiency.Source: §A.2 · pp. 14–15

A.3. Robustness of Overhead Results

Because local environments can affect overhead results, the authors report cross-setup comparisons, repeated-run consistency, and an evaluation setup aligned with established code-efficiency benchmarks.

A.3. Robustness of Overhead Results

  • 32%: In Python 3.11.10 on an Intel Xeon Platinum 8336C, Qwen2.5-Coder-7B’s average execution time fell from 0.59s to 0.40s after Effi-Code fine-tuning, with higher efficiency across seven setups.The authors report that the fine-tuned models were more efficient than the original models across all seven software-hardware setups.Source: §A.3 · p. 15
  • 0.00548 s: Execution-time standard deviation across five runs indicates consistent, reliable efficiency results within the same setup.The authors report repeated evaluations of generated code under an unchanged setup.Source: §A.3 · p. 15
  • The evaluation setup follows code-efficiency benchmarking practices used by Mercury, Effibench (Huang et al., 2024b), and SOAP.The authors state that this alignment keeps their evaluation in line with current field standards.Source: §A.3 · p. 15

A.4. Test case augmentation

EffiCoder augments tasks lacking tests by generating test cases with GPT-3.5-turbo and retaining only cases that pass the task’s initial solution.

A.4. Test case augmentation

  • GPT-3.5-turbo generates tests from each task description and source code; tests that fail on the initial solution, and tasks without passing tests, are filtered out.Each generated test is executed individually, and any test that raises an error is treated as incorrect.Source: §A.4 · p. 15

A.5. Comparison with PIE strategies … A.10. Randomness

Across PIE, EffiLearner, and ENAMEL comparisons, EFFIINSTRUCT improves code efficiency and, on ENAMEL, correctness; the discussion attributes its scope to automated, multilingual optimization at larger scale. A case study illustrates faster execution, and repeated measurements show low variability.

A.5. Comparison with PIE strategies

  • EFFIINSTRUCT reduced execution time by 22.5% versus the base model, exceeding PIE’s “All” strategy at 19.6%, and reduced total memory usage by 75.9%.The comparison reports that EFFIINSTRUCT outperforms all PIE variants on ET and TMU in EffiBench.Source: §A.5 · pp. 15, 17, Table 9 · p. 17

A.6. Comparison with EffiLearner

  • For shared tasks, average ET fell from 0.56 (s) to 0.46 (s) with EffiLearner and 0.39 (s) with EFFIINSTRUCT on DeepSeek-Coder-6.7B-Instruct.The reported execution-time reductions were 17.9% for EffiLearner and 30.4% for EFFIINSTRUCT.Source: Table 10 · p. 17, §A.6 · p. 17

A.8. Further discussion with baselines

  • EFFIINSTRUCT automates optimization from task descriptions without human intervention, covers C++, Python, Java, Rust, and Go, and includes 65,710 tasks versus Mercury’s 1,889 and PIE’s 1,474.The discussion contrasts PIE’s human-written solutions and C++ focus, and Mercury’s selection from pre-existing human-written Python code; it says EFFIINSTRUCT models show superior pass@1 and efficiency in Table 7. The cited works are Du et al. (2024) and Shypula et al. (2024).Source: §A.8 · pp. 17–18

A.7. Evaluation results on ENAMEL

  • effi@1 rose from 0.373 to 0.458 and pass@1 from 0.589 to 0.739 for Qwen2.5-Coder-Instruct-7B on ENAMEL.The passage reports improvements in both efficiency and correctness metrics.Source: Table 11 · p. 17, §A.7 · p. 17

A.9. Case Study

  • Qwen2.5-Coder-7B’s generated code took 9.89 (s) to run all unit tests, versus 0.14 (s) for the EFFIINSTRUCT fine-tuned model using SFT.The passage attributes the difference to the baseline’s extra recursive calls and lack of pruning, while the fine-tuned code terminates recursion early under certain conditions.Source: Figure 5 · p. 18, §A.9 · p. 18

A.10. Randomness

  • Across five executions of EFFIINSTRUCT fine-tuned Qwen2.5-Coder-7B, ET, MU, and TMU results were described as robust, with ET standard deviation of 0.00.The passage notes that execution time and memory measurements can fluctuate with server workload and hardware availability.Source: Table 12 · p. 19, §A.10 · p. 19
Loading 2410.10209v4…