Source-linked AI summary
Mercury: A Code Efficiency Benchmark for Code Large Language Models
Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, See-Kiong Ng
TL;DR
Existing code-generation benchmarks largely assess functional correctness, leaving code efficiency undermeasured. Mercury combines Python tasks, test generators, and historical solutions with a runtime-percentile metric; its results show room to improve efficiency, with DPO outperforming SFT as an efficiency-enhancement baseline in the reported comparison.
Problem
Existing code-generation benchmarks mainly evaluate functional correctness, with limited attention to code efficiency and test-case coverage.
Source: §1 · pp. 1–2Method
Results
Code LLMs show considerable potential to improve efficiency, and DPO yields stable code-efficiency enhancement for models exceeding 6.7B parameters, unlike SFT’s reduction in most Beyond scores.
Source: §1 · p. 3, §4.4 · p. 7Takeaways & Limitations
Mercury provides a benchmark for evaluating code efficiency alongside functional correctness, while the reported baseline comparison favors DPO over SFT for efficiency enhancement.
Source: §1 · p. 3, §4.4 · p. 7Mercury assumes uniformly distributed runtimes; real-world runtime distributions may be more intricate and require more solution samples for precise modeling.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Amidst the recent strides in evaluating Large Language Models for Code (Code LLMs), existing benchmarks have mainly focused on the functional correctness of generated code, neglecting the importance of their computational efficiency. To fill the gap, we present Mercury, the first code efficiency benchmark for Code LLMs. It comprises 1,889 Python tasks, each accompanied by adequate solutions that serve as real-world efficiency baselines, enabling a comprehensive analysis of the runtime distribution. Based on the distribution, we introduce a new metric Beyond, which computes a runtime-percentile-weighted Pass score to reflect functional correctness and code efficiency simultaneously. On Mercury, leading Code LLMs can achieve 65% on Pass, while less than 50% on Beyond. Given that an ideal Beyond score would be aligned with the Pass score, it indicates that while Code LLMs exhibit impressive capabilities in generating functionally correct code, there remains a notable gap in their efficiency. Finally, our empirical experiments reveal that Direct Preference Optimization (DPO) serves as a robust baseline for enhancing code efficiency compared with Supervised Fine Tuning (SFT), which paves a promising avenue for future exploration of efficient code generation. Our code and data are available on GitHub: https://github.com/Elfsong/Mercury.
1 Introduction
Existing code-generation benchmarks emphasize functional correctness while overlooking efficiency, test coverage, and task diversity. Mercury pairs Python tasks with test generators and historical solutions, and introduces Beyond to evaluate code efficiency using runtime percentiles.
1 Introduction
- Existing code-generation benchmarks emphasize functional correctness while overlooking code efficiency, and their small test sets and limited task diversity can miss edge cases and challenge models weakly.The introduction identifies efficiency evaluation, test-case coverage, and task diversity as three shortcomings.Source: §1 · pp. 1–2
- The paper focuses on execution time as its measure of code efficiency, rather than the broader time-and-space performance concept.The authors describe execution time as the performance bottleneck of most code.Source: §1 · p. 2
- Mercury’s 1,889 Python tasks use test case generators and historical solutions to evaluate code efficiency against task-specific runtime distributions.Beyond scores generated code by its runtime percentile among the corresponding solutions.Source: §1 · p. 2, Figure 2 · p. 3
- Mercury introduces an efficiency-focused benchmark and Beyond, alongside baselines for improving efficiency while maintaining functional correctness.The authors report that Code LLMs show considerable potential to improve efficiency.Source: §1 · p. 3
- Mercury distinguishes itself from existing NL2Code benchmarks with distilled solutions and a dedicated test case generator for each task.The Table 1 caption identifies these features but does not supply comparative cell values.Source: Table 1 · p. 3
2 Mercury Datasets
Mercury is built from public LeetCode problems, with evaluation and training splits, task-specific test generators, and historical solutions. Its selection rules and difficulty stratification shape the tasks and runtime distributions used for evaluation.
2 Mercury Datasets
- Mercury contains 1,889 distilled LeetCode tasks, split into a 256-task evaluation benchmark and a training dataset.Mercury-eval averages 18.4 solutions per problem; “Mercury” denotes Mercury-eval unless otherwise specified.Source: §2 · p. 3
- Each task schema includes a plain-text description, test case generator, sampled historical solutions, and prompts with execution entry points.The solutions are tested, and Locality-Sensitive Hashing is used to prevent identical solutions from being included.Source: §2 · p. 4
- Tasks with fewer than two solutions are excluded; selected tasks average 18.4 unique solutions and use only supported data structures and outputs verified as unique.The described supported custom structures are binary trees and linked lists; tasks potentially yielding non-unique answers are excluded.Source: §2 · p. 4
- Mercury’s task filters select problems with enough solutions to form runtime distributions and exclude tasks with unsupported structures or potentially non-unique answers.The selection also limits the dataset to tasks with at least two associated solutions.Source: §2 · p. 4
- Mercury stratifies tasks as Easy, Medium, and Hard, inheriting LeetCode’s difficulty categories to distinguish model capabilities.The authors say this stratification aims to probe the upper bounds of Code LLM capabilities.Source: §2 · p. 4
- For each evaluation task, a generator is drafted with GPT-4 and manually revised until its 24 generated test cases pass LeetCode Online Judge validation.The generators are intended to produce test cases for assessing functional correctness and code efficiency.Source: §2 · p. 4
3 Code Efficiency Metric
Beyond normalizes runtime by comparing each generated solution with a task-specific runtime distribution, combining efficiency with functional correctness. The evaluation also checks whether the metric remains consistent across hardware configurations.
3 Code Efficiency Metric
- Beyond uses task-specific runtime distributions to normalize measurements across tasks with different runtime ranges.Absolute runtime can vary with hardware, operating systems, and interpreters, making it an inconsistent efficiency measure.Source: §3 · p. 4
- For each of N tasks and K LLM solutions, runtime is measured across all test cases and clipped to that task’s historical runtime range.Runtime spans solution instantiation through successful termination across the test cases.Source: §3 · p. 5
- A solution that fails any test receives runtime 0, so Beyond reflects functional correctness as well as speed.Source: §3 · p. 5
- Generated code runs in an isolated sandbox to mitigate the risk of executing malicious code.Source: §3 · p. 5
- Beyond remains consistent across hardware configurations from 0.25 to 1 CPU core for the two evaluated DeepSeek-Coder models.Source: §3 · p. 5
4 Experiments
The experiments compare original models, SFT, and DPO on Mercury, using ten open-source LLMs ranging from 1.3B to 34B parameters. DPO generally improves code efficiency for larger models, while its effects on the Beyond–Pass gap vary with model size.
4.1 Baselines
- SFT trains on fastest-solution completions, whereas DPO uses preference pairs and avoids RLHF’s additional reward-model training and complexity.The paper notes SFT’s susceptibility to catastrophic forgetting.Source: §4.1 · p. 5
4.2 Functional Correctness Benchmarks
- Pass is the fraction of tasks whose generated solutions pass all test cases; HumanEval and MBPP supplement Mercury’s functional-correctness evaluation.Source: §4.2 · p. 5, §4.4 · p. 6
4.3 Experimental Setups
- SFT uses fastest-solution labels, while DPO uses the five most runtime-disparate solution pairs; evaluation uses LoRA and K = 5.Source: §4.3 · p. 6
4.4 Empirical Results
- The experiments report Pass, Beyond, and the Beyond–Pass Gap, without extensive parameter optimization or prompt engineering.Source: §4.4 · p. 6
4.5 Failure Analysis
- Both methods generally reduce generation errors, but SFT aggravates execution errors on most models while DPO mitigates them; test-case errors are most prevalent.Source: §4.5 · p. 7
4.4 Empirical Results
- DPO improves Beyond stably for models above 6.7B and narrows the Beyond–Pass Gap above 15B, while SFT lowers Beyond and smaller models’ Gap widens.On Pass, DPO improves most models except the smallest, while SFT diminishes scores for the two largest models.Source: §4.4 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 8
5 Related Work
Related work traces NL2Code from early rule-based and statistical approaches to Transformer-based LLMs. Prior evaluation has emphasized functional correctness, while newer benchmarks address additional evaluation dimensions.
5 Related Work
- NL2Code research progressed from grammars and domain-specific languages through statistical models to Transformer-based LLMs such as Codex, AlphaCode, CodeGen, PaLM-Coder, and StarCoder.Source: §5 · p. 8
- Code-generation evaluation shifted from code similarity metrics toward test-case fuzzing, with benchmarks also covering multilingual code, data analysis, multi-turn generation, and security.Source: §5 · p. 8
- Table 4 distributes failure cases across code-generation, code-execution, and test-case errors, with results grouped by easy, medium, and hard difficulty.It samples 1,280 solutions per model.Source: Table 4 · p. 9
6 Limitations
Mercury assumes code runtimes are uniformly distributed, although real-world runtime distributions are more intricate.
6 Limitations
- The uniform-runtime assumption simplifies efficiency evaluation with limited solution samples, but more samples may be needed for more precise real-world modeling.Training-data contamination can also compromise benchmark precision for tainted models; the authors plan to add new tasks dynamically through the open-source collection framework.Source: §6 · p. 9
7 Conclusion
Mercury introduces a code-efficiency benchmark using dedicated test generators and ground-truth solutions sampled across difficulty levels. Evaluations find that leading Code LLMs remain less efficient despite strong functional correctness.
7 Conclusion
- Mercury evaluates Code LLMs with dedicated test generators and ground-truth solutions sampled across all difficulty levels from Leetcode.The authors describe it as the first code-efficiency benchmark for NL2Code evaluation.Source: §7 · p. 9
- The evaluation finds considerable room to improve code efficiency even though models are proficient at generating functionally correct code.The conclusion links code efficiency to factual productivity as Code LLM use widens.Source: §7 · p. 9
- Mercury’s open-source dataset collection framework is intended to support future research and innovation.Source: §7 · p. 9
Checklist
The checklist records that the paper describes its scope and limitations and addresses ethics, assets, and experimental reproducibility items.
Checklist
- The checklist says the abstract and introduction describe the paper’s main claims and research scope.Source: Checklist · p. 13
- The authors report that they discuss their work’s limitations in Section 6.Source: Checklist · p. 13
- The checklist marks theoretical results and proofs as not applicable because the paper includes no theoretical results.Source: Checklist · p. 13
- The checklist reports that asset creators are cited and asset licenses, consent, and personal-information or offensive-content issues are addressed in Appendix A.13.Source: Checklist · p. 13
- The paper marks human-subject participant instructions, risks, and compensation items as not applicable.Source: Checklist · p. 13
A.1 Dataset Nutrition Labels
Table 5 is captioned as defining fields in the Mercury dataset.
A.1 Dataset Nutrition Labels
- Table 5 defines the fields within the Mercury dataset.Source: Table 5 · p. 14
A.2 Mercury Data Distribution and Customized Data Structures
These appendix sections describe Mercury’s data structures, sandbox, DPO setup, prompting, and evaluation details. They also document dataset splits, model scope, and data release conditions.
A.2 Mercury Data Distribution and Customized Data Structures
- Mercury supplements built-in Python structures with TreeNode and ListNode to increase data-structure diversity and complexity.Source: §A.2 · p. 14, Figure 4 · p. 14
A.3 Sandbox Details
- Mercury-eval contains 256 balanced tasks, while Mercury-train contains the remaining 1,633 tasks.The sandbox reinitializes for each task and records solution-instantiation and test-case-evaluation time; executions have a 30-second timeout and 2048 MB memory limit.Source: Table 6 · p. 14, §A.3 · p. 14, Figure 5 · p. 15
A.4 DPO Experiment Details
- For DPO, the experiment constructs preferences from faster and slower solutions, initializes a reference model on faster completions, and optimizes the target against that reference.The described gradient increases the likelihood of preferred completions and decreases that of dis-preferred ones, weighted by implicit reward-ordering error and scaled by β.Source: §A.4 · pp. 15–16
A.5 External Libraries Utilized in Mercury
- Mercury imports shared libraries during sandbox initialization to reduce solution failures caused by missing imports.The libraries are imported into a temporary namespace controlled by the sandbox.Source: §A.5 · p. 16, Figure 6 · p. 16
A.6 Model Details
- The evaluated models range from 1.3B to 34B, and the dataset is described as a non-commercial research release under Creative Commons Attribution Non Commercial 4.0.The dataset is hosted on Hugging Face; collection was limited to free-domain LeetCode tasks, excluding paid content.Source: Table 7 · p. 16, §A.13 · p. 19, §A.12 · p. 19
A.7 A Mercury Example
- The appendix’s Mercury-eval example shows functionally correct solutions with substantially different runtimes, while its HumanEval example illustrates how passing limited tests can leave larger inputs vulnerable to timeout or stack overflow.Source: Figure 7 · p. 17, Figure 8 · p. 17
A.9 Prompts for Code Generation
- A unified one-shot prompt template uses placeholders for task content, the code starter, and code completion.The prompt figure identifies lines 1–40 as the one-shot example.Source: §A.9 · p. 18, Figure 9 · p. 18
A.10 Hardware-agnostic Evaluation
- Beyond scores for two DeepSeek-Coder models remain consistent across the tested Intel Skylake CPU configurations; bootstrapped Beyond scores were evaluated over 50 runs per Starcoder2 model.The bootstrapped distribution covers 3B, 7B, and 15B models, with histogram frequency on the y-axis.Source: Figure 10 · p. 19, Figure 11 · p. 19