The Larger the Better? Improved LLM Code-Generation via Budget Reallocation

1 Introduction

• The comparison generates as many outputs as possible from each model under fixed FLOPs or wall-time budgets, using unit tests to select successful code.The evaluation adapts pass@k to compare models of different sizes under a fixed compute budget.Source: §1 · p. 1
• The proposed alternative to scaling model size is to generate multiple outputs from a smaller model and select the best one.Figure 1 contrasts this strategy with increasing model size while generating a single output.Source: Figure 1 · p. 2
• Generating multiple outputs from 13B models can yield gains of up to 15% over a single generation from a 70B model across five tasks.On the challenging APPS competition split, the 13B model performs best across almost all budgets, with a consistent 5% margin.Source: §1 · p. 2
• Without unit tests, ranking-based selection improves with compute and ranking-model size but still falls short of running a larger model independently.The paper tests ranking 7B outputs using each of the models.Source: §1 · p. 2
• Smaller models are also more hardware-friendly; a 13B model can fit on one A100 GPU, unlike a 70B model.The paper also releases more than 1M Code Llama 7B outputs for HumanEval and MBPP to support research on ranking LLM outputs.Source: §1 · p. 2

2.1 Standard Code Generation Evaluation

• For standard code-generation evaluation, a question counts as correct if at least one of k generated outputs passes all unit tests.Performance for k outputs is estimated by generating n outputs, where n > k; the resulting metric is an unbiased estimator.Source: §2.1 · p. 3

2.2 Comparing LLMs of Different Sizes under a Fixed Budget

• Under a fixed budget, smaller models can generate more outputs so that different model sizes use roughly the same compute.The paper considers FLOPs and wall-time limits, stopping generation when the specified resource limit is reached.Source: §2.2 · p. 3
• FLOPs limits assume perfect hardware utilization, while wall-time limits are more realistic but hardware-specific and not directly comparable across machines.The budgets are expressed through the FLOPs and time used to generate a given number of outputs.Source: §2.2 · p. 3

3.1 Benchmarks

• The evaluation uses HumanEval, MBPP, and the test subset of APPS, with APPS divided into introductory, interview, and competition difficulties.HumanEval has 164 function-completion problems, MBPP has 500 instruction-based examples, and APPS has 5,000 programming problems.Source: §3.1 · p. 3

3.2 Metrics

• FLOPs use is estimated with calflops at input sequence length 128, while wall-time is measured on eight A100 GPUs under optimal throughput utilization.Batch size is optimized per model, and generation time is measured on a subset of approximately 1,000 examples; Table 1 reports model resource use and values normalized to 7B.Source: §3.2 · pp. 3–4, Table 1 · p. 4

3.3 Experiments

• Experiments use Code Llama models from 7B to 70B, plus Llama-3 8B and 70B on HumanEval and MBPP.The setup follows zero-shot prompting for HumanEval, 3-shot for MBPP, and 2-shot for APPS, with specified generation limits and sampling settings.Source: §3.3 · p. 4

4 Small Models Outperform Large Ones under a Fixed Compute Budget

• 60%: In HumanEval, Code Llama 7B and 13B reach this score using one quarter of the time larger models need to reach it.Source: §4 · p. 4
• 5—15%: On HumanEval and MBPP, 7B, 8B, and 13B models outperform 34B and 70B models across fixed-FLOPs and fixed-wall-time budgets, with the largest gaps at small budgets.The small-budget regime is up to 32 normalized FLOPs units and 64 wall-time units.Source: §4 · p. 4, Figure 2 · p. 5, Figure 3 · p. 5
• Repeated smaller-model decoding is equivalent or preferable to larger-model greedy decoding; on APPS, 13B matches or exceeds greedy decoding and is 2–4 times more efficient than 70B at fixed performance.Source: §4 · pp. 5–6
• ≈5%: On APPS competition, 13B outperforms all larger models across budgets, with this margin over 70B under fixed wall-time; on the introductory and interview splits, it beats 34B and is comparable to 70B.Source: §4 · p. 5, Figure 4 · p. 5, Figure 6 · p. 6
• The 7B model is competitive on APPS only at small budgets—up to 8 normalized FLOPs units and 16 wall-time units—and slightly underperforms larger models at higher budgets.The authors attribute this to 7B’s inability to generate enough correct answers and suggest a minimum size may be required for some task difficulty levels.Source: §4 · p. 6
• These comparisons assume unit tests are available to select correct outputs from generated candidates.Source: §4 · p. 6

5.1 Evaluating Rankers

• The estimator counts groups of k generations whose highest-ranked candidate is correct, using pass scores ordered by the ranking policy.The paper also defines rank-scoreflops@f and rank-scoretime@t by choosing the largest k within the corresponding compute or time budget.Source: §5.1 · p. 7

5.2 Large Language Models as Rankers

• Candidates are ranked by their average negative log likelihood, excluding the prompt; this score can be extracted in parallel rather than token by token.The evaluation uses 2,000 Code Llama 7B generations on HumanEval and MBPP, with all four Code Llama sizes as rankers.Source: §5.2 · p. 7
• Larger rankers improve selection, but under fixed compute budgets greedy decoding consistently outperforms ranking multiple generations from a smaller model.The comparison covers HumanEval and MBPP; Figure 8 plots rank-scoretime@t against wall-time, with different rankers and sampling approaches.Source: §5.2 · p. 8, Figure 8 · p. 8
• The LEVER verifier does not improve ranking results in the test-less setting, where its execution-based pruning and execution results are unavailable.Figure 9 reports MBPP results using LEVER with different NLL rankers.Source: §5.2 · p. 8, Figure 9 · p. 8
• The authors release 2,000 Code Llama 7B generations per example for HumanEval and MBPP, totaling 1,328,000 generations.Source: §5.2 · p. 8

6.1 Model Scaling

• Unlike scaling-law studies, this work compares smaller and larger models at fixed budgets rather than proposing new scaling laws.Prior work examined model and dataset scaling, mixture-of-experts scaling, and downstream-task scaling analyses.Source: §6.1 · p. 9
• Fixed-budget advantages for smaller models have also been reported in vision, including multiple image scales and smaller diffusion models.The authors relate generating multiple text outputs from a small model to these findings.Source: §6.1 · p. 9

6.2 Verifiers and Rankers

• Related selection work includes external verifiers for math solutions, LLM text rankers, code-generation selection, and execution-feedback filtering.The paper also distinguishes learned verifier approaches from non-learned filtering and execution-feedback methods.Source: §6.2 · p. 9

7 Discussion & Limitations

• At equal compute, using smaller models can improve LLM code-generation performance.The authors note that HumanEval and MBPP results seem to slightly support eventual saturation at similar performance, but do not fully resolve whether larger models will overtake smaller ones with enough compute.Source: §7 · p. 9
• Compute constraints restricted the study to a limited number of generations per model; the authors state that their conclusions apply to most practical use cases given these costs.Source: §7 · p. 10

8 Conclusion

• Multiple outputs from a 13B model may gain up to 15% over one output from a 70B model across five tasks.Source: §8 · p. 10
• The comparison uses fixed FLOPs and wall-time budgets on execution-based code-generation tasks with unit tests.Source: §8 · p. 10
• Without unit tests or other verifiers, the explored ranking approach falls short of a single output from a larger model.Source: §8 · p. 10
• The authors release over 1M Code Llama 7B samples spanning HumanEval and MBPP to support further research on output ranking.Source: §8 · p. 10

• Figures 10 and 11 present Llama-3 results for HumanEval and MBPP, respectively.Source: A Llama-3 Results · p. 15

• Tables 2 to 4 report precise models’ pass@k results for HumanEval, MBPP, and APPS.Tables 2, 3, and 4 cover HumanEval, MBPP, and APPS, respectively.Source: B Detailed pass@k Results · p. 15, Table 2 · p. 15, Table 3 · p. 16
• The tables show selected k values because reporting results for every k is infeasible.Source: B Detailed pass@k Results · p. 15
• All relevant k values were calculated and used to compute the figures.Source: B Detailed pass@k Results · p. 15