COFFE: A Code Efficiency Benchmark for Code Generation

1 Introduction

• Code generation converts natural-language instructions into executable code, and LLMs can generate code, reviews, or comments from instructions.The introduction situates this capability within software engineering tasks and potential development-efficiency gains.Source: §1 · p. 1
• Existing correctness benchmarks use small inputs that may not distinguish time complexities, while execution-time measurements vary across repeated runs and machines.The paper also notes that separate correctness and efficiency metrics do not jointly characterize code solutions on both dimensions.Source: §1 · p. 2
• 27.26% and 17.60% RSD on stressful tests exceed the 19.05% and 15.73% RSD on original tests for function-level and file-level code, respectively.CPU instruction count RSD is 0.003%–0.005%, compared with 2.37%–5.65% for execution time.Source: §1 · p. 3
• Current LLMs generate correct but not necessarily time-efficient code, with lower efficiency on file-level tasks and little efficiency advantage from larger models.Larger models generally perform better in correctness, but do not significantly outperform smaller models in efficient code generation.Source: §1 · p. 3
• COFFE evaluates function- and file-level code generation, while STGen uses contracts and expression or generator test cases to produce stressful tests.The proposed efficient@k metric combines correctness and time efficiency using CPU instruction counts.Source: §1 · p. 3

Function-level Code Generation

• Function-level code generation produces one function from a natural-language description, with inputs supplied through its parameters.HumanEval [8] and MBPP [5] are function-level benchmarks.Source: §2 · p. 4
• Function-level test cases can combine inputs for each parameter, while file-level tasks pose challenges because input locations and counts may vary.The function example requires test inputs for the single parameter lst.Source: §2 · p. 4

File-level Code Generation

• File-level code generation produces a complete program file whose inputs are managed through standard-input APIs such as input().Code Contests and APPS [28] are file-level benchmarks.Source: §2 · p. 4
• Function inputs are determined by the parameters, whereas a file-level program’s input count can depend on the inputs themselves.The example file-level program uses an initial input to control how many times a later input is read.Source: §2 · p. 4

3. Methodology

• Table 1 reports original, validated, and selected problem counts, alongside statistics for the finally selected problems.The caption defines “Ori.”, “Val.”, and “Sel.” as original, validated, and finally selected problems.Source: Table 1 · p. 5

3.1 Data Preparation

• COFFE draws on HumanEval [8] and MBPP [5] for function-level tasks, and APPS [28] and CodeContests for file-level tasks.For MBPP, the authors use the common subset of the sanitized version and EvalPlus-verified MBPP+.Source: §3.1 · p. 5
• The authors validate problems by checking conflicts between provided test cases and ground-truth solutions, then select problems that most LLMs can solve.This selection is intended to ensure problems contribute to time-efficiency evaluation.Source: §3.1 · p. 5

3.1.1 Problem Validation.

• Validation removes unsafe file-operation solutions and problems lacking valid ground-truth solutions or test cases.The authors also remove APPS problems whose ground-truth outputs do not exactly match the provided expected outputs.Source: §3.1.1 · p. 5

3.1.2 Problem Selection.

• For CodeContests and APPS, the authors sample temperature-0 solutions from 14 LLMs and remove problems that all models fail.They additionally select 300 APPS problems that more LLMs can solve to balance the function-level and file-level splits.Source: §3.1.2 · p. 5, Figure 2 · p. 6

3.2.1 Overview.

• STGen decomposes stressful test generation into contract generation, test generation, and test–contract pair checking, with incorrect items sent for regeneration.The LLM judge examines conflicts after contract-related assertion errors exceed a threshold.Source: §3.2.1 · pp. 6–7

3.2.2 Phase I: Contract Generation.

• Contracts are assertions recording input types, scale, and internal constraints, helping guide test generation and identify invalid inputs early.STGen generates assertions iteratively from the target program, existing correctness tests, and previously generated assertions.Source: §3.2.2 · p. 7
• STGen inserts contracts at function parameters for function-level programs and around identified input locations for file-level programs.For inputs assigned inside loops, assertions may be inserted after the loop so assigned values can be checked.Source: §3.2.2 · p. 7

3.2.3 Phase II: Stressful Test Case Generation.

• Expression test cases compactly generate structured inputs, while generator test cases handle intricate patterns or an undetermined number of inputs.These code-based formats avoid requiring LLMs to produce overlong raw inputs; expressions suit a determined number of inputs.Source: §3.2.3 · p. 8

3.2.4 Phase III: Test Case-Contract Pair Check.

• When repeated assertion errors exceed a threshold, an LLM judge checks whether contracts or tests caused the conflict and returns incorrect items for regeneration.A contract judged valid is not checked again, and subsequent tests that violate it are rejected.Source: §3.2.4 · p. 8

3.3.1 CPU Instruction Count.

• CPU instruction count is the program-related factor in CPU time and is more stable than execution time when external conditions slow or stall execution.CPU time is expressed as instruction count multiplied by clocks per instruction and clock-cycle time; the latter two depend on the physical machine.Source: §3.3.1 · p. 9

3.3.2 Efficient@k.

• Efficient@k uses CPU instruction counts to address the limits of instruction count alone and is inspired by pass@k.Pass@k estimates the probability that at least one of k sampled solutions passes all test cases; efficient@k ranges from 0 to pass@k.Source: §3.3.2 · pp. 9–10
• Efficient@k combines correctness and time efficiency by estimating whether sampled solutions are correct and faster than the best ground-truth solution.It substitutes the count of correct-and-faster solutions for pass@k’s count of correct solutions, comparing CPU instruction counts rather than absolute values.Source: §3.3.2 · pp. 9–10

• The benchmark includes the five stressful test cases with the highest average CPU instruction counts, selected from 20 generated tests per problem.The authors measure each test 12 times, discard the highest and lowest measurements, then rank by the average.Source: §3.4 · p. 10
• Each CPU-instruction measurement is capped at five seconds, and the repeated measurements are designed to stabilize the reported averages.With 12 measurements per test, the measurement time for one test case will not exceed one minute.Source: §3.4 · p. 10
• COFFE retains existing correctness tests because stressful tests are costly to run, and the correctness tests validate generated solutions.Stressful tests generally take much longer to execute than correctness tests.Source: §3.4 · p. 10

4.1 Research Questions

• The setup asks how instruction counts compare with execution time, how effective STGen is, and how efficient current LLM-generated code is.Source: §4.1 · p. 10

4.2 Metrics

• The evaluation uses RSD for metric stability or test distinguishability, Pearson correlation for linear association, and accuracy and line coverage for test quality.For code efficiency, it uses pass@k, speedup, and efficient@k.Source: §4.2 · pp. 10–11

4.3 Baselines

• Stressful-test baselines adapt instruction prompting, few-shot prompting, and generator-based prompting; the first two produce raw tests and the third produces generator tests.Source: §4.3 · p. 11

4.4 Models

• The study evaluates 14 popular LLMs, including ten open-source and four closed-source models.The model listing distinguishes closed-source models with gray highlighting.Source: §4.4 · p. 11, Table 3 · p. 12

4.5 Implementation Details

• Experiments run on Ubuntu 20.04.4 LTS with an Intel Xeon Platinum 8358P CPU and use Coverage.py for line coverage and Cirron for instruction counts.Source: §4.5 · p. 11

5.1 RQ1: CPU Instruction Count vs. Execution Time

• 0.003%∼0.005% RSD for CPU instruction count is more than 1,000× smaller than execution-time RSD, while Pearson correlations are 0.96∼1.0.The paper concludes CPU instruction count is more suitable for evaluating time efficiency.Source: §5.1 · p. 12, Table 4 · p. 13

5.2 RQ2: Effectiveness of STGen and Distinguishability of Stressful Test Cases

• 98.64% and 98.91% accuracy, alongside 96.01% and 95.17% line coverage, show STGen generates correct tests that cover much of the target code.The paired values are for function-level and file-level splits, respectively.Source: Table 5 · p. 13, §5.2 · pp. 13–14
• 43.10% and 32.08% function-level RSD gains, plus 11.89% and 13.86% file-level gains, show STGen tests better distinguish solutions than original tests.The paired gains are for Llama3.1 and GPT-4o, respectively; expression tests also outperform generator-based prompting on function-level RSD.Source: Table 5 · p. 13, §5.2 · p. 14

5.3 RQ3: Time Efficiency of Code Generated by LLMs

• 46.97% function-level and 46.51% file-level efficient@1 are below the best Pass@1 values of 79.90% and 90.78%, respectively.The comparison indicates a substantial gap between correct and efficient code generation.Source: Table 6 · p. 15, §5.3 · pp. 15–16
• Only three LLMs exceed 1.0 speedup at file level, versus all at function level; efficient@1 also drops more sharply for file-level tasks.The reported pass@1-to-efficient@1 drop is 30%∼45% for function-level generation and 40%∼100% for file-level generation.Source: §5.3 · p. 16
• Larger models generally improve Pass@1 but do not consistently improve efficient@1, with much larger models sometimes achieving similar efficiency to smaller ones.For example, CodeLlama-34b achieves 40.37% efficient@1, close to 39.58% for Llama3.1-405b.Source: §5.3 · p. 16

LLM Researchers

• Researchers should consider code structure and semantics when improving efficiency, and focus more on file-level generation, where current LLMs perform worse than at the function level.Source: §6 · pp. 16–17

Software Practitioners

• Practitioners may choose middle-sized LLMs such as Llama3-70b or CodeLlama-34b for similar efficient-code performance with much lower computational costs.Source: §6 · p. 17

7.1 Threats to Internal Validity

• CPU instruction counts can still contain errors from code-optimization techniques, despite Docker isolation and averaging 10 measurements after removing the highest and lowest.The authors run each measurement 12 times and remove the highest and lowest before averaging.Source: §7.1 · p. 17
• Because no LLM-based stressful-test-generation methods were available for comparison, the authors modified correctness-test generators, which may change their performance.They ran these baselines on GPT-4o and selected the best five of 20 generated tests for most evaluations.Source: §7.1 · p. 17

7.2 Threats to External Validity

• The experiments focus on Python, so results may differ for languages such as C++ and Java due to syntax and coding-style differences.The authors state STGen is language-agnostic and could be extended to other languages.Source: §7.2 · p. 17

8.1 LLMs for Code Generation

• Decoder-only models and general LLMs have demonstrated strong code-generation performance, alongside earlier encoder-decoder models trained on large code corpora.The passage names examples including Codex, CodeGen, Llama3.1, and GPT-4.Source: §8.1 · pp. 17–18

8.2 Code Generation Benchmarks

• Correctness benchmarks cover function- and file-level tasks, while Effibench targets time and memory efficiency but uses difficult problems and execution-time measurements.The paper describes execution time as unreliable for distinguishing code-solution efficiency.Source: §8.2 · p. 18

8.3 LLM-based Test Case Generation

• Prior work generated tests for correctness, but the authors report no prior approach targeting large inputs for code-efficiency evaluation and propose STGen for Python projects.STGen is described as generating stressful test cases with high accuracy and coverage.Source: §8.3 · pp. 18–19

9 Conclusion

• COFFE evaluates LLM-generated code efficiency using STGen stressful tests and efficient@k based on CPU instruction count; the authors evaluate 14 LLMs.The paper says efficient@k stably evaluates both code correctness and time efficiency.Source: §9 · p. 19