Source-linked AI summary
A Performance Study of LLM-Generated Code on Leetcode
Tristan Coignion, Clément Quinton, Romain Rouvoy
TL;DR
This paper addresses the lack of research evaluating the performance of LLM-generated code. It compares 18 models on Leetcode using a method for comparing runtime, and reports broadly similar performance across models, while noting limits to generalization beyond algorithmic problems.
Problem
The authors identify a lack of research evaluating LLM-generated code performance, a quality relevant to scarce-resource and large-scale software settings.
Source: §1 · p. 1Method
The study evaluates 18 LLMs on Leetcode problems using a novel method to measure and compare generated-code runtime.
Source: §1 · p. 1, §2.3 · p. 3, §6.1 · p. 9Results
Generated-code performance was largely similar across models regardless of size or training data; higher temperature increased performance variance, but not consistently in either direction.
Source: §8 · p. 10Takeaways & Limitations
For fast code generation, developers need not consistently choose the largest model, as smaller models may suffice.
Source: §6.1 · p. 9Because the evaluation uses algorithmic problems, its performance findings are hard to generalize across all programming fields.
Source: §6.2 · p. 9
Abstract
from arXiv · showhide
This study evaluates the efficiency of code generation by Large Language Models (LLMs) and measures their performance against human-crafted solutions using a dataset from Leetcode. We compare 18 LLMs, considering factors such as model temperature and success rate, and their impact on code performance. This research introduces a novel method for measuring and comparing the speed of LLM-generated code, revealing that LLMs produce code with comparable performance, irrespective of the adopted LLM. We also find that LLMs are capable of generating code that is, on average, more efficient than the code written by humans. The paper further discusses the use of Leetcode as a benchmarking dataset, the limitations imposed by potential data contamination, and the platform's measurement reliability. We believe that our findings contribute to a better understanding of LLM capabilities in code generation and set the stage for future optimizations in the field.
1 INTRODUCTION
Although research has examined LLM code correctness and other qualities, the authors identify a gap in evaluating the performance of LLM-generated code. They study code from 18 LLMs, compare it with human-written code, and assess Leetcode as a dataset.
1 INTRODUCTION
- Faster code is sought after, particularly where resources are scarce or software runs at large scale; performance improvements can also reduce energy consumption.Source: §1 · p. 1
- The authors identify a gap in research evaluating LLM-generated code performance, despite existing work on security, bugs, developer interaction, benchmarks, and success rates [29] [30] [33] [20] [7] [5] [10] [17] [37] [43].Source: §1 · p. 1
- The paper studies code performance from 18 LLMs on 204 problems, compares generated code with human-written code, and evaluates Leetcode as a dataset.It introduces a method for measuring and comparing the performance of generated code.Source: §1 · p. 1
2 METHODOLOGY
The methodology frames four research questions about Leetcode’s suitability, differences in generated-code efficiency, temperature and success-rate effects, and comparison with human solutions. It uses recent Leetcode problems, defined prompts and test inputs, and a set of 18 code-focused LLMs.
2.1 Research questions
- The four questions address Leetcode as a benchmark, efficiency differences among LLMs, temperature and success-rate effects, and comparisons with human solutions.Source: §2.1 · p. 2
2.2 Tasks & Dataset
- The study uses algorithmic Leetcode problems with large test inputs, selecting 204 problems published after January 1, 2023 to reduce contamination risk for most evaluated models.The authors also reran the experiment on 300 older Leetcode questions and used Python.Source: §2.2 · pp. 2–3
- Prompts included the problem description, while each problem had a human-written canonical solution used as a benchmarking baseline.The authors extracted three large inputs per problem using modified canonical solutions that failed above manually chosen input-size thresholds.Source: §2.2 · p. 3
2.3 LLMs Under Study
- The study evaluated 18 popular code-focused LLMs, including open-source models and GitHub Copilot, across model variants and two experiment rounds.Source: §2.3 · p. 3, Table 1 · p. 3
3 EXPERIMENT SETUP
The experiment generated, validated, and benchmarked model solutions against large inputs, with repeated canonical-solution measurements used to check runtime stability. Study artifacts, including results, code, and datasets, are publicly available.
3.1 Code Generation
- Each problem received 10 generations across six temperatures, except Copilot, whose temperature could not be configured; open-source models used nucleus sampling.Copilot was generated at its default temperature, and the study used inline generations for Copilot.Source: §3.1 · pp. 3–4
3.2 Validation
- A three-stage validation process reduced 210,120 generations to 7,481 valid solutions, excluding errors and runs exceeding 10 seconds.The stages were local checks, Leetcode judge validation, and checks using the study’s large-input benchmark setup.Source: §3.2 · p. 4
3.3 Measuring run time
- Runtime was measured with repeated pytest-benchmark runs alongside canonical solutions; 196 of 204 canonical solutions had standard deviations below one-tenth of their mean runtime.Each solution ran at least 10 times and for at least one second total.Source: §3.3 · p. 4
3.4 Replication package
- The study’s results, code, and datasets are available in a public replication package.Source: §3.4 · p. 4
4 DATA ANALYSIS
The analysis measures functional correctness with pass@k and code performance with memory use, benchmark runtime, and Leetcode rank. It also compares model runtimes pairwise on problems where both models produced enough valid solutions.
4.1 Functional Correctness
- Functional correctness is measured with pass@1 and pass@10 using Chen et al.’s unbiased estimator, selecting each model’s best pass@k across temperatures [10].A test counts as successful if at least one of k generated samples passes all tests.Source: §4.1 · p. 5
4.2 Code Performance
- Code performance is measured by Leetcode memory use, median pytest-benchmark runtime, and Leetcode rank relative to slower submitted solutions.A rank of 90 means the solution is faster than 90% of Leetcode submissions.Source: §4.2 · p. 5
- For pairwise model comparisons, the authors tested mean runtimes on problems where both models had more than five valid solutions, then compared each model’s share of significantly faster results.Source: §4.2 · p. 5
5 RESULTS
Across Leetcode’s evaluation and measurement checks, functional validity shifts sharply across datasets, while its timing and rank measures have reliability constraints. Mean speed differences among LLMs are negligible, and generated solutions rank faster than most prior submissions.
5.1.1 Can Leetcode problems be adopted as a dataset for LLM gen-
- Pass@k fell tenfold for every tested LLM between the two datasets, a shift the authors believe may reflect contamination in the older dataset.The authors note that a genuine difference in question difficulty could also explain the shift, but say quantifying that possibility is challenging.Source: §5.1.1 · pp. 5–6, Figure 2 · p. 5
5.1.2 Are Leetcode measurements reliable?
- Leetcode’s timing coefficient of variation was 0.089 versus 0.035 locally, and average local–Leetcode time correlation was only 0.28.For some problems the measures correlated above 0.8, while for others they were below 0.2; Leetcode’s higher variance limits its ability to distinguish runtime differences between implementations.Source: §5.1.2 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6
- Leetcode’s reported memory usage declined over time (correlation −0.24), and identical Copilot solutions ranked 77 then 54, showing time-dependent measurement changes.The rank uses previously accepted solutions, including the study’s submissions, so testing can change later ranks; evolving metrics complicate comparisons.Source: §5.1.2 · pp. 6–7, Figure 5 · p. 7
- Leetcode evaluation is constrained by potential training-data contamination and by the need to use problems published after a model’s training data was formed, which complicates reproducibility.The authors also question the reliability of Leetcode-provided runtime, memory, and rank metrics compared with traditional measurement methods.Source: §5.1.2 · pp. 6–7
5.2 RQ2: Are there notable differences in performances between LLMs?
- Mean Cohen’s d was 0.024, indicating negligible practical differences in mean code speed across LLMs despite some statistically significant pairwise differences.StarCoder and Python CodeLlama-13B often performed slightly better; CodeLlama-13-instruct was significantly faster than CodeGen25-7B-mono on 3 of 8 problems, with a 0.02-standard-deviation mean difference.Source: §5.2 · p. 7, Figure 6 · p. 7
5.3 RQ3: Is there an effect of the functional validity of the LLM and its temperature on the generated code’s performance?
- Temperature had no correlation with solution speed (0.05), but was moderately correlated with greater performance variation (0.41); success rate correlated only −0.08 with runtime.Success rate also had close to no correlation (−0.11) with variation in generated-code performance.Source: §5.3 · pp. 7–8, Figure 7 · p. 8
5.4 RQ4: How fast are LLMs compared to humans ?
- LLM solutions had a mean Leetcode rank of 73%, with some faster than 95% of prior submissions, assuming most prior submissions were human-authored.The human comparison used Leetcode ranking because the authors assumed most earlier submissions came from humans.Source: §5.4 · p. 8
6 DISCUSSION
The discussion finds little practical difference in generated-code speed across LLMs, while identifying contamination and measurement constraints in Leetcode-based evaluation. It also limits generalization to algorithmic tasks and treats comparisons against human rankings cautiously.
6.1 Discussion of the results
- Leetcode contamination threatens accurate evaluation and reproducibility as newer problems may enter future training datasets; alternative evaluation data may be needed for replication.The authors suggest pre-filtered datasets such as HumanEval as a possible basis for performance evaluation.Source: §6.1 · p. 8
- Leetcode problems appear harder than HumanEval’s: StarCoder’s pass@1 is 0.09 versus 0.408, and solution length correlates −0.30 with success rate.Average solution length is 425 characters in this dataset versus 180 in HumanEval; the authors also point to greater problem-solving demands.Source: §6.1 · p. 8
- Varying valid-solution rates make pairwise comparisons necessary, while only 24 of 204 problems met the study’s criteria for broad model comparisons.The authors propose comparing generated-solution speed with an optimal solution as a possible path toward a single performance score.Source: §6.1 · p. 9
- Different LLMs show no significant variance in generated-code performance, so developers seeking fast code need not consistently choose the largest model.The authors say a smaller model suffices for this aim.Source: §6.1 · p. 9
6.2 Limits and Threats to Validity
- Prompt or model-configuration changes could affect correctness; Copilot may change through undisclosed retraining, and Leetcode limits submissions to 1,000 per account daily.The authors say all models were configured similarly and suggest CodeNet as a less restricted alternative to Leetcode.Source: §6.2 · p. 9
- Benchmarking relied on Leetcode tests, including three fetched cases; random input generation was unsatisfactory, and memory usage was not measured.The authors note that extracted large test cases could favor some implementations, while random generation of inputs with specific structures or properties proved difficult.Source: §6.2 · p. 9
- Human comparisons use evolving Leetcode rankings with an unknown comparison population, so the resulting rankings warrant caution.The authors describe the comparison as a useful indication of an LLM’s performance position, but not a definitive measure.Source: §6.2 · p. 9
- Because the evaluation uses algorithmic problems, the findings are difficult to generalize across programming fields.The paper notes constraints from limited context and one-shot, self-contained code generation; SQL generation would not fit this study of generalist programming languages.Source: §6.2 · p. 9
7 RELATED WORKS
Prior work examines code-LLM correctness, security, developer interaction, and benchmarking, while some studies use Leetcode or investigate temperature. The authors position their work as the first, to their knowledge, to compare the performance of LLM-generated code.
7 RELATED WORKS
- Prior code-LLM research covers security, bugs, developer interaction, correctness, and benchmarks including HumanEval, MBPP, APPS, and ReCode.The section also notes comparative evaluations of multiple code LLMs.Source: §7 · p. 9
- Earlier Leetcode studies examined code suggestions, prompt changes, generated-code complexity, and uncertainty cues.The cited studies include work on Copilot, Codex, and code-completion uncertainty.Source: §7 · p. 9
- Prior studies examined temperature’s effects on code generation, and this paper reports that higher temperature increases the chance of slow, inefficient code.Earlier work also considered pass@k, nucleus sampling, and temperature tuning.Source: §7 · pp. 9–10
- The authors present their work as the first investigation, to their knowledge, of performance differences in LLM-generated code, complementing prior code-optimization techniques.They propose their methodology as a way to compare future LLMs adapted with performance-improving techniques.Source: §7 · p. 10
8 CONCLUSION
The study finds broadly similar runtime performance across LLMs, while higher temperature increases variability without improving or worsening average solutions. It also flags reliability and contamination concerns in Leetcode-based evaluation.
8 CONCLUSION
- Runtime performance is largely similar across models, regardless of size or training data, while higher generation temperature increases variance but not average solution quality.The study uses a novel methodology to measure and compare runtime speed on algorithmic problems.Source: §8 · p. 10
- Leetcode problems are suitable for performance evaluation, but platform measures require caution because of stability, reliability, and data-contamination concerns.The authors say newer problems are essential to avoid contamination and ensure evaluation validity.Source: §8 · p. 10
- Future research avenues include performance-oriented training datasets and fine-tuning LLMs to improve performance.Source: §8 · p. 10