Source-linked AI summary
Evaluating Language Models for Efficient Code Generation
Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, Lingming Zhang
TL;DR
Existing coding benchmarks can provide unreliable evidence about code efficiency because they use light computation and lack an effective compound metric. DPE creates performance-exercising tasks and scores correct solutions against efficiency-clustered references; its EvalPerf evaluation reports insights across models and consistent, reliable benchmarking across platforms.
Problem
Existing coding benchmarks can give unreliable code-efficiency insights because their tasks often use simplistic inputs and runtime speedups are difficult to aggregate meaningfully across tasks.
Source: §1 · pp. 1–2Method
DPE creates performance-exercising tasks from coding benchmarks and evaluates correct new solutions by profiling them against representative reference solutions.
Source: §2 · p. 3Results
EvalPerf evaluation reveals findings about model sizes, instruction tuning, and prompting, while DPE evaluation indicates reliable and consistent benchmarking across platforms.
Source: §6 · p. 9Takeaways & Limitations
EvalPerf provides a 121-task benchmark for assessing code efficiency, and the paper reports that it can support consistent evaluation across platforms.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We introduce Differential Performance Evaluation (DPE), a framework designed to reliably evaluate Large Language Models (LLMs) for efficient code generation. Traditional coding benchmarks often fail to provide reliable insights into code efficiency, due to their reliance on simplistic test inputs and the absence of effective compound metrics. DPE addresses these issues by focusing on efficiency-demanding programming tasks and establishing an insightful compound metric for performance evaluation. DPE operates in two phases: To curate efficiency datasets, it selects efficiency-demanding tasks from existing coding benchmarks and generates computationally expensive inputs to stress the efficiency of LLM solutions. To assess the code efficiency, DPE profiles the new solution and compares it globally against a set of reference solutions that exhibit distinct efficiency levels, where the matched level defines its efficiency score. As a proof of concept, we use DPE to create EvalPerf, a benchmark with 121 performance-challenging coding tasks. Our comprehensive evaluation draws interesting findings on the efficiency impact of model sizes, instruction tuning, and prompting. For example, while the scaling law fails to account for code efficiency, general instruction tuning benefits both code correctness and efficiency. We also evaluate the evaluation by examining the effectiveness of DPE, showing that EvalPerf is reliable and convenient to use even across platforms.
1 Introduction
Code-efficiency evaluation matters for software quality and code generation, but existing benchmarks can provide unreliable efficiency insights because tasks are too light and aggregate speedup is hard to interpret. DPE addresses these gaps with performance-exercising tasks and a reference-based efficiency score; EvalPerf demonstrates the framework in a 121-task benchmark.
Motivation
- Code efficiency matters alongside correctness because code execution can bottleneck Program-aided Language Models and affect user experience.Efficient code can also support low-latency, scalable, and cost-effective software.Source: §1 · p. 2
Why efficiency evaluation is challenging
- Existing benchmark tasks often use small inputs and simple control flow, making efficiency comparisons noisy and less meaningful at small scales.The paper argues that code efficiency is more meaningfully studied over large-scale data.Source: §1 · p. 2
- Averaging speedups across tasks can misrepresent overall efficiency when speedups vary greatly in scale.The example contrasts a 2× slowdown on 99 tasks with a 100× speedup on one task, yielding an average speedup of 1.495× for the slower model.Source: §1 · p. 2
Contributions
- DPE curates performance-demanding tasks and scores new solutions against efficiency-clustered references; EvalPerf contains 121 tasks.The paper also reports that DPE creates inputs 4.8× more performance-exercising than prior art and that EvalPerf yields consistent evaluations across platforms.Source: §1 · p. 2
2 Differential Performance Evaluation
DPE builds performance-exercising coding tasks from existing benchmarks, then evaluates correct new solutions against representative references. Its pipeline synthesizes costly inputs, filters tasks for reliable and diverse performance, clusters reference solutions, and scores a solution by its matched cluster.
2 Differential Performance Evaluation
- DPE turns existing programming tasks into performance-exercising tasks with challenging test inputs, then compares correct new solutions against reference solutions.The matched reference cluster determines the new solution’s efficiency score.Source: §2 · p. 3
2.1 Valid Solution Curation
- DPE samples plausible solutions from diverse code LLMs and validates them with rigorous correctness tests before profiling.The paper identifies correctness as a prerequisite for performance evaluation.Source: §2.1 · p. 3
2.2 Synthesizing a Synthesizer: Performance-Exercising Input Generation
- SAS synthesizes scale-controllable test generators; exponential sampling increases input scale until a validated solution reaches a time or memory limit.The generator must produce increasingly challenging inputs at larger scale factors, while task-specific structure can make direct large-input generation difficult.Source: §2.2 · pp. 3–4
2.3 Performance-Exercising Task Selection
- Task selection requires sufficiently long executions, low timing variation, and more than one performance cluster.These criteria filter out tasks that do not provide reliable performance distinctions.Source: §2.3 · p. 4
2.4 Adaptive Performance Clustering
- Adaptive clustering sorts mean execution times from slow to fast and segments them according to relative scale.The resulting clusters provide performance references for differentiating new solutions.Source: §2.4 · pp. 4–5
2.5 Efficiency Scoring by References
- DPE retains the slowest solution in each cluster as a reference and scores a validated solution by the cumulative ratio of its matched reference.Dataset-wide scores average over passed tasks; collecting references across diverse efficiency levels is required.Source: §2.5 · p. 5
3 EVALPERF: A Benchmark for Code Efficiency Evaluation
EVALPERF applies DPE to create a benchmark of 121 performance-exercising coding tasks, using computationally challenging inputs and performance-reference solutions. Its construction profiles solutions with hardware instruction counters and filters tasks for sufficient computation and performance diversity.
3 EVALPERF: A Benchmark for Code Efficiency Evaluation
- 121 performance-exercising coding tasks make up EVALPERF, equipped with computationally challenging inputs and performance-reference solutions.The benchmark is built using Differential Performance Evaluation.Source: §3 · p. 5
- The construction starts with HumanEval+ and MBPP+ tasks and samples correct solutions from 21 open LLMs before generating and testing input samplers.The input-generation process increases scale until reaching a 20-second or 16GB limit, then filters broken generators.Source: §3 · p. 6
- Task selection profiles executed assembly instructions with hardware performance counters and filters tasks below 10k instructions or with fewer than four performance clusters.The implementation sets a 20% base clustering threshold and a 10k-instruction weight; the variation criterion is omitted because hardware counters measure cost.Source: §3 · p. 6
- The authors identify extending EVALPERF with more coding tasks as future work.Source: §3 · p. 6
4 Evaluation
The evaluation studies code efficiency across model variants and tests DPE’s cross-platform consistency and usability. It reports benefits from instruction tuning, inconsistent gains from performance-oriented prompting and model scaling, and low variation across test beds.
4.1 Evaluating Code Efficiency
- The study compares open model families and GPT-4 Turbo using base, instruction-tuned, performance-prompted, and chain-of-thought variants.It evaluates correctness on 164 HumanEval+ and 399 MBPP+ tasks and averages efficiency scores over correct solutions, comparing models on common passing solutions.Source: §4.1 · pp. 6–7
- Instruction-tuned models generally produce more efficient correct code than base models, with DeepSeekCoder-6.7B improving DPS by 19%, except StarCoder2-15B.The authors conclude general instruction tuning can benefit code quality beyond correctness, despite not being designed to optimize efficiency.Source: §4.1 · p. 7, Figure 4 · p. 7
- Performance-oriented prompts neither consistently nor noticeably improve code efficiency over basic prompting, and commonly degrade correctness.Source: §4.1 · p. 7
- Seven of 12 model-size pairs favor the larger model on efficiency, but four show more than 1% degradation, including a 6% loss from StarCoder2 base 3B to 7B.Other reported gains include 14% for the instructed 1.3B version and 8.7% from StarCoder2 7B to 15B; the authors say scaling is not explicit for code efficiency.Source: §4.1 · p. 7, Figure 5 · p. 7
4.2 Evaluating the Evaluation
- At the clustering stage, SAS retains 121 tasks while EvalPlus retains 25, a 4.8× relative improvement.This filter requires at least four clusters with different performance characteristics.Source: Table 1 · p. 8
- At most 0.4% coefficient of variation across test beds supports consistent EVALPERF conclusions, with evaluation taking no more than about 15 minutes for most models.The authors attribute low variation to differential comparison against reference solutions and hardware performance counters.Source: §4.2 · p. 8, Table 2 · p. 8
5 Related Work
Related work spans correctness benchmarks and specialized code-efficiency evaluation. The paper positions DPE as a complementary meta-evaluation framework that targets challenging inputs and a stable compound efficiency metric.
5 Related Work
- Code-correctness benchmarks cover diverse task types, including Python problems, multilingual tasks, data science, repository-level generation, software development, security, and code understanding.HumanEval+ and MBPP+ extend test coverage through automated test generation.Source: §5 · p. 9
- Existing code-efficiency benchmarks include PIE, which evaluates LLM optimization of base C++ programs and uses CPU simulators for reproducible profiling.EVALPERF instead targets direct code generation, where a base reference program may not be available, and measures cost with hardware performance counters.Source: §5 · p. 9
- Newer sibling benchmarks add source tasks and efficiency dimensions but retain limitations from insufficient test computation and variation-sensitive average-speedup metrics.DPE complements them with automatically generated performance-exercising inputs and a stable compound metric.Source: §5 · p. 9
6 Conclusion
DPE assesses LLM code efficiency by profiling solutions against representative references and creates EvalPerf, a benchmark of 121 performance-challenging tasks. Evaluation with EvalPerf yields findings on model size, instruction tuning, and prompting, while DPE itself is reported effective, reliable, and convenient across platforms.
6 Conclusion
- DPE profiles new solutions against references with representative performance characteristics to assess code efficiency.Its two phases create performance-exercising tasks and evaluate the global performance of new solutions.Source: §6 · p. 9
- EvalPerf, built using DPE, contains 121 performance-challenging coding tasks.Source: §6 · p. 9
- Evaluation with EvalPerf reveals findings on model sizes, instruction tuning, and prompting, while DPE is reported effective, reliable, and convenient across platforms.Source: §6 · p. 9
A.1 Runtime Variation at Different Runtime Scales
Runtime variation is examined across profiled solutions at different mean runtime scales. Smaller executions show greater sensitivity to system noise, motivating large computations for robust and meaningful performance benchmarking.
A.1 Runtime Variation at Different Runtime Scales
- Smaller executions are more sensitive to system noise, motivating large computations for robust and meaningful performance benchmarking.Solutions were profiled five times on an i7-10700K desktop without other mandatory processes running.Source: §A.1 · p. 14
- Figure 6 plots mean runtime on the X-axis against coefficient of variation on the Y-axis, with blue dots for solutions and purple bars for mean variation by 10× runtime scale.Source: §A.1 · p. 14, Figure 6 · p. 14
- The runtime-scale comparison shows that smaller executions have greater variation despite a clean test bed.Source: §A.1 · p. 14
A.2 Discussion of Performance Measurement and Metrics
The discussion contrasts physical runtime, architecture simulators, and hardware performance counters, noting trade-offs in variation, reproducibility, overhead, and platform fidelity. It also motivates DPS as a global efficiency score with a consistent range across tasks.
A.2 Discussion of Performance Measurement and Metrics
- Hardware performance counters provide low-overhead, highly reproducible instruction counts, while counts can vary across platforms and do not replace simulators for mimicking CPUs unavailable physically.Physical runtime can vary in noisy environments, whereas architecture simulators are reproducible but slower than native execution and may not reflect physical performance.Source: §A.2 · pp. 14–15
- Averaging relative speedups across tasks can obscure holistic efficiency gains because task speedups vary substantially and highly improvable tasks can skew the average.Source: §A.2 · p. 15
- DPS assigns each task a consistent 0–100% score for a solution’s empirical global performance position relative to benchmark solutions.The score is intended to convey solution effectiveness and distance to optimal performance.Source: §A.2 · p. 15
A.3 Extended Results of Code Correctness and Efficiency
The extended results report correctness and DPS/DPSnorm across EvalPerf, with efficiency scores aggregated over each model’s passing tasks. DPSnorm favors a different model than DPS, and EvalPerf correctness results show a reversed scaling trend within the CodeLlama family.
A.3 Extended Results of Code Correctness and Efficiency
- Efficiency scores in this extended analysis are computed over each model’s passing tasks, so correctness affects the task set used for global comparison.Table 3 reports correctness across three task sets and DPS/DPSnorm on EvalPerf, aggregating sample-wise scores from the initial 10 samples by task.Source: §A.3 · p. 15, Table 3 · p. 16
- DeepSeekCoder-6.7B-instruct leads on DPSnorm, whereas GPT-4 Turbo leads on DPS, reflecting different performance patterns across efficiency levels.DPSnorm tends to be lower on EvalPerf tasks, indicating that correct LLM samples are skewed toward slower implementations.Source: §A.3 · p. 15, Figure 8 · p. 17, Figure 7 · p. 17
- On the 121 EvalPerf tasks, smaller CodeLlama models achieve better pass rates, a reversed scaling trend; other findings remain overall consistent with Figures 7 and 8.Source: §A.3 · p. 15, Figure 9 · p. 17, Figure 10 · p. 18