The Limits of Inference Scaling Through Resampling

1 INTRODUCTION

• The paper studies whether unit-test resampling lets less capable models match more capable models on coding benchmarks.It reviews inference-scaling approaches and their known limitations, and reports a qualitative analysis of recurring error types behind weaker models’ larger generalization gap.Source: §1 · pp. 1–2
• For realistic cost-benefit ratios, the optimal number of samples is finite and very low, even when computation is free.False positives incur costs, while correct answers provide benefits; tasks requiring more attempts tend to be harder and more susceptible to false positives.Source: §1 · p. 2
• On HumanEval+, weaker models produce false positives at higher rates, and some cannot match GPT-4o’s Pass@1 even with unlimited resampling.MBPP+ results follow a similar pattern.Source: §1 · p. 2
• Even with an infinite inference budget, weaker models often cannot match the single-invocation performance of a sufficiently strong model.The paper identifies imperfect verifiers and false positives as fundamental limits on resampling-based scaling.Source: §1 · pp. 1–2
• False-positive solutions have lower code quality than true positives across models and tested readability metrics.The metrics include naming conventions, line-level comments, and limits on line length and function length; the paper speculates that similar patterns may hold for simplicity and modularity.Source: §1 · p. 2
• The findings underscore the importance of accurate verifiers, especially for training, where models may exploit verifier weaknesses and raise safety concerns.The paper also warns that using imperfect benchmarks as both verifier and evaluation ground truth can hide gaps and mislead model comparisons.Source: §1 · p. 3
• Imperfect verifiers may mislabel examples selected through rejection sampling, limiting the gains of resampling-based data curation for reasoning models.The paper argues that stronger base models or more accurate verifiers are needed to avoid this bottleneck.Source: §1 · p. 3
• The paper does not claim resampling is the predominant scaling technique, but says verifier gaps could constrain inference-scaling strategies more broadly.It invites research into ways to mitigate the identified issues.Source: §1 · p. 3

• Inference scaling methods include multi-candidate selection and single-solution refinement, and the distinction can be unclear when methods are combined.The paper excludes training improvements from its definition of inference scaling.Source: §2 · p. 3
• Verifier-based resampling can generate candidates repeatedly until one passes, motivating hopes for inference scaling across many orders of magnitude.Unlike methods such as majority voting, its control flow allows indefinite regeneration in principle.Source: §2 · p. 3
• Table 1 organizes inference-scaling techniques by category, description, and known limitations.The boundary between rankers and verifiers can be unclear.Source: Table 1 · p. 4
• Resampling’s usefulness depends on verifier capability: oracle verifiers can avoid false positives, while imperfect verifiers can accept incorrect solutions.False negatives are described as less problematic because additional samples can be generated, though it is unclear whether all-correct-solutions-missed cases arise in practice.Source: §2 · p. 4

• Unit tests can miss incorrect code, motivating the study of whether less capable models produce more false positives than stronger models.The study uses extended HumanEval+ and MBPP+ test suites to assess correctness beyond original benchmark tests.Source: §3 · pp. 4–5
• The experiments use HumanEval+ and MBPP+ hidden tests to assess solutions passing the original benchmarks’ tests.Models produced at least 200 samples per task on HumanEval+ (1,000 for Command Light) and 50 per task on MBPP+; solutions passing the full tests are assumed correct.Source: §3 · p. 5
• Weaker models have higher false-positive probabilities than stronger models across model families, and this relationship scales inversely with true capability.The reported linear relationship appears across Cohere Command, GPT-4o, and Llama-3.1 models.Source: §3 · p. 5
• Models below the Figure 3 cutoff cannot match GPT-4o through resampling, even with an infinite compute budget.The figure relates single-sample accuracy to the highest achievable accuracy using limited unit tests as verifiers.Source: §3 · p. 6, Figure 3 · p. 6
• The observed capability relationship is largely driven by tasks with poor unit tests and becomes more pronounced when analysis is limited to those tasks.Source: §3 · p. 6
• The findings rely on human-generated unit tests; whether LLM-generated unit-test verifiers change them remains open.Source: §3 · p. 6

Experimental setup

• The reward model assigns benefit 1 to a true positive, a cost of 0, 1, 2, 4, or 8 to a false positive, and zero reward when no solution passes within K samples.The study uses 200 samples per task and computes mean reward across 1,000 random permutations of the same samples.Source: §4 · p. 6

Findings

• The analysis compares reward across K and cost-benefit ratios for Llama-3.1 and Code Llama, with crosses marking each setting’s optimal sample count.The figure also includes standard inference-scaling curves with no false-positive cost.Source: Figure 4 · p. 7
• Harder tasks are more susceptible to false positives, so repeated sampling quickly reaches diminishing or negative returns as the false-positive rate increases with K.Task difficulty is strongly bimodal: easy tasks are solved within a few attempts, while false positives are more likely on the remaining harder tasks.Source: §4 · p. 7
• At a cost-benefit ratio of 4, the optimal number of samples is K ≤5 for all four models, and a sufficiently high ratio can make the optimum zero.The expected cost of a false positive can outweigh the benefit of a correct solution.Source: §4 · p. 7
• False-positive rates rise much more dramatically with K for Llama-3.1-70B than for Code Llama and Command models, which can have higher optimal K at low cost-benefit ratios.The authors report no intuitive explanation for this difference.Source: §4 · p. 7
• The authors conclude that weaker models cannot sample their way to top-level performance when the verifier cannot reliably filter false positives.They attribute the limit to false-positive risks quickly outweighing benefits.Source: §4 · p. 7

Experimental setup

• The evaluation compares solutions passing standard unit tests with those passing the more comprehensive test suite, using separate prompts for each readability measure.The prompts and implementation follow Zheng et al. (2024).Source: §5 · p. 8

Findings

• False positives tend to have worse code quality across all tested metrics, consistently across models of varying capabilities.The comparison concerns solutions that pass standard tests but fail the extended tests.Source: §5 · p. 8

Qualitative analysis of false positives

• Manual review of 10 false positives identified logical errors, edge-case failures, and inefficient implementations.Examples include converting floats to integers, mishandling empty lists, and a loop that times out on larger inputs.Source: §5 · pp. 8–9

Findings

• Figure 7 illustrates that false positives are lower-quality code, including less frequent adherence to camelCase naming conventions.The caption reports that this pattern holds across models and all four code-quality metrics tested.Source: Figure 7 · p. 9

6 DISCUSSION

• The experiments pair all generators with the same verifier, which has imperfect coverage but no false negatives.The authors note that model-generated tests could introduce verifier disparities and false negatives, potentially widening the generalization gap.Source: §6 · p. 9
• The authors hypothesize that weaker models paired with imperfect verifiers may produce datasets too low-quality to train competitive reasoning models.They identify this as a bottleneck for resampling-based data curation without stronger base models or more accurate verifiers.Source: §6 · p. 9
• The theoretical results are described as domain-agnostic, while the experiments focus on coding tasks.The authors identify reasoning, web agents, and agent-user interaction as possible areas for future investigation.Source: §6 · pp. 9–10
• The paper distinguishes its inference-time resampling ceiling, which arises from false positives, from reward misalignment in proxy-reward optimization.It also says its findings weaken support for claims that resampling effectively increases accuracy by trading off inference-time compute.Source: §6 · p. 10

A.1 EDGE CASES IN OUR GENERATOR-VERIFIER SETTING

• Chain-of-Verification differs from the framework because it verifies intermediate facts to refine one response rather than scoring candidates for acceptance or rejection.The passage says this makes it less suitable for inference scaling through resampling; Tree of Thoughts can fit the framework by producing aggregate scores and a solvability decision.Source: §A.1 · p. 18

B ADDITIONAL DETAILS ON SECTION 3

• The supplemental figures show false-positive rates across attempt-count bins and relate conditional accuracy after standard tests to single-sample accuracy on tasks with verifier precision below 90%.The latter figure reports a more pronounced relationship for both benchmarks and gives the considered task counts as 70/150 and 128/321.Source: Figure 11 · p. 21, Figure 10 · p. 21

Sample Collection

• Samples were generated at temperature 0.8, with at least 50 samples per model and task and 200 samples for most models.For Command Light, the authors collected 1,000 samples per HumanEval+ task to reduce tasks without any solutions passing the HumanEval unit tests.Source: §B.1 · pp. 21–22

B.1 DETAILS ON DATA AND IMPLEMENTATION

• Figure 12 plots single-sample accuracy against achievable accuracy with infinite compute and limited unit tests, showing upper and lower bounds for HumanEval+ and MBPP+.The bounds set the conditional passing rate to 1 or 0, respectively, for tasks without observed passing solutions.Source: Figure 12 · p. 22

Sample Collection

• For most models, the collected samples came from Liu et al. (2023b) and used the same temperature setting of 0.8 as the authors’ generated samples.On MBPP+, the authors collected 50 samples for each model and task.Source: §B.1 · p. 22

B.2 ADDITIONAL DETAILS ON EXCLUDED TASKS FROM MBPP+

• The MBPP+ creators had already removed 21 broken tasks, reducing the benchmark to 378 tasks.The original exclusions were attributed to unreliable oracle implementations.Source: §B.2 · p. 22
• 350 tasks remained in the final MBPP+ evaluation set after 28 additional tasks with oracle issues were excluded.Those issues included extended-test failures across all models and an oracle exception on large numerical inputs.Source: §B.2 · p. 22
• Another 29 tasks were excluded because solutions passed the extended tests but failed the original tests; excluding them did not significantly impact the final results.The authors planned to report these cases to the benchmark creators.Source: §B.2 · pp. 22–23

C.1 MODEL SETUP

• The appendix models a generator and an imperfect verifier, with easy and hard tasks assigned different success probabilities.It formalizes inference-scaling limits and generalizes the empirical findings to other benchmarks.Source: C ADDITIONAL DETAILS ON SECTION 4 · p. 24, §C.1 · p. 25

PROBABILITY OF REJECTION

• The rejection probability β_i is task-type-specific, and the appendix defines it for easy and hard tasks.The model treats β_i as the probability that the verifier rejects a generated sample for task type i.Source: PROBABILITY OF REJECTION · p. 25

BELIEF UPDATES

• After repeated rejections, the model raises its belief that the task is hard, and Figure 13 shows the easy-task belief decreasing as K grows.The setup classifies tasks with Pass@1 ≥ 0.5 as easy and those below 0.5 as hard.Source: BELIEF UPDATES · pp. 25–26, Table 3 · p. 26

PROBABILITY OF CORRECT AND FALSE POSITIVE SOLUTIONS

• The model weights correct-solution and false-positive probabilities by posterior beliefs about task type; false-positive probability increases at higher attempts.Correct and false-positive outcomes are evaluated conditional on previous attempts being rejected.Source: PROBABILITY OF CORRECT AND FALSE… · p. 26
• As attempts increase, the probability of generating a correct solution falls while the probability of a false positive rises.The model’s expected value peaks at very low K, and the optimal K is at most 3 even with zero computational cost.Source: Figure 13 · p. 24, PROBABILITY OF CORRECT AND FALSE… · p. 26

OPTIMAL NUMBER OF ATTEMPTS

• The model chooses Kopt to maximize total expected reward, balancing the benefit of correct solutions against the cost of false positives.Figure 14 shows optimal attempt counts across models and false-positive-cost to true-positive-benefit ratios.Source: Figure 14 · p. 25, EXPECTED VALUE OF GENERATING… · p. 26, OPTIMAL NUMBER OF ATTEMPTS · p. 27

C.2 INFERENCE SCALING CURVE FOR GPT-4O

• GPT-4o’s inference-scaling curves are flat, with minimal search benefits in line with the empirical task-difficulty distribution.GPT-4o had the highest single-sample accuracy on HumanEval+ among the models in the experiments.Source: Figure 15 · p. 27

D ADDITIONAL DETAILS ON SECTION 5

• False positives tend to be lower-quality code than correct solutions across models and four code-quality metrics.The metrics concern naming conventions, line length, and line-level comments.Source: Figure 16 · p. 28

D.1 DETAILS ON DATA AND IMPLEMENTATION

• For each model and code-quality instruction, the authors generated 50 HumanEval+ samples per task at temperature 0.8 using Zheng et al. (2024)’s implementation.Other parameters were left at the implementation’s defaults.Source: §D.1 · p. 28

D.2 PROMPT EXAMPLES FOR READABILITY METRICS

• The appendix provides prompt templates for each code-readability metric, following Zheng et al. (2024).Source: Figure 17 · p. 29

D.3 QUALITATIVE EXAMPLES OF FALSE POSITIVES

• CodeGen-6B’s HumanEval/30 solution passed standard tests but truncated positive floats by converting them to integers, failing extended tests.The requested behavior was to return only positive numbers from the list.Source: Figure 18 · p. 29, §D.3 · p. 29
• Other HumanEval examples fail extended tests through a timeout-prone loop, lost duplicates, or mishandled empty input.The examples include CodeGen, Code Llama 7B, and CodeT5p 16B implementations.Source: Figure 19 · p. 30, Figure 20 · p. 30, Figure 22 · p. 31
• MBPP examples illustrate wrong algorithms and failures on empty strings or repeated elements despite passing standard tests.The examples include incorrect substring counting, alphabetic-only filtering, and a pattern-checking function returning False for repeated elements.Source: Figure 21 · p. 30, Figure 26 · p. 32, Figure 27 · p. 32
• The examples also show false positives with incorrect pair counting or an absolute-difference result, and a GPT-4o variable violating camelCase.Source: Figure 24 · p. 31, Figure 29 · p. 33, Figure 30 · p. 33

E DECLARATION OF LLM USAGE AND COMPUTE RESOURCES

• Experiments used OpenAI endpoints for coding benchmarks and Together.ai endpoints for HotPotQA analysis with Llama models.The work relied primarily on external APIs and used no GPUs for inference or LLM training.Source: E DECLARATION OF LLM USAGE AND COMPUTE… · p. 33

F IMPACT STATEMENT

• The authors report no additional domain-specific ethical or societal concerns unique to this study.They say broader implications mirror those familiar from developing and using large-scale language models.Source: F IMPACT STATEMENT · p. 33
• The findings reinforce the importance of reliable evaluation metrics and thorough verification for safer, more trustworthy compound AI applications.Source: F IMPACT STATEMENT · p. 33