PaperlayerView exact revision on arXiv

Source-linked AI summary

Fewer Bits, More Tries: Recovering Capability Under Memory Constraints

Tomer Galanti, Josh Susskind, Eran Malach

arXiv:2610.15372v1cs.AI

TL;DR

Memory constraints can make locally deployable models less capable than larger models, motivating the question of whether retries can recover some of that gap. The paper generates and verifies parallel candidates from smaller or compressed models. Across coding benchmarks, retry can recover substantial capability, with a 7B model at eight attempts matching or exceeding a 32B model while using less memory, time, and energy.

  • Problem

    Under a hard memory budget, choosing a model that fits can require sacrificing the one-shot capability associated with larger models.

    Source: §1 · p. 1

  • Method

    The paper uses a smaller or compressed model to generate parallel candidates and selects one using deployment-time verification, including public or model-generated tests.

    Source: §4.1 · p. 4, §6 · p. 10

  • Results

    Across four benchmarks, Coder-7B at K = 8 essentially matches Coder-32B at K = 1 on MBPP+ and exceeds it elsewhere, using about 4.3× less model memory.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    A 7B model with eight parallel attempts matches or exceeds a 32B model with one while using less memory, time, and energy.

    Source: §6 · p. 10

    The results are limited to single-turn code generation and depend on good tests, which the model cannot always write for itself.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult. We ask whether part of this capability gap can instead be recovered by running a smaller or more compressed model multiple times and selecting among its outputs using public or model-generated tests. Single-user local inference can leave hardware underutilized, allowing parallel candidate generation to trade additional test-time computation for accuracy at modest latency cost. Across four code-generation benchmarks, verified retry recovers substantial accuracy lost to model size or compression. A 7B Qwen model with eight parallel attempts essentially matches a 32B model on MBPP+ and exceeds it on the other three benchmarks, while using about $4\times$ less model memory, running up to $6\times$ faster, and using less energy, as measured on Nvidia GPU. We observe similar tradeoff across model sizes, compression levels and hardware choices, although exact benefit depends on the hardware profile and problem setting. These results show that model capacity can be traded for verified test-time computation when correct solutions remain reachable within a modest retry budget.

1 INTRODUCTION

The introduction asks whether retrying and verifying a smaller model can recover capability otherwise lost to memory-driven compression. It reports substantial recovery across coding benchmarks, while noting that success depends on producing and recognizing correct candidates.

1 INTRODUCTION

  • Under a hard memory budget, compression makes the smaller model responsible for fitting while preserving the larger model’s accuracy, although performance scales with model size and training compute.Source: §1 · p. 1
  • Single-user local hardware can run parallel attempts with improved utilization, so runtime need not grow linearly with retry count.The approach asks when verified attempts from a smaller model can substitute for one attempt from a larger model.Source: §1 · p. 1
  • Retry trades one-shot reliability for candidate coverage: a smaller or lower-precision model may still produce a correct solution within several draws, and coverage changes less with size than one-shot accuracy.Source: §1 · p. 1
  • When external tests are unavailable, model-generated tests introduce verification failures alongside generation failures, including tests that cannot distinguish correct from incorrect programs or aggregation that selects the wrong candidate.The paper analyzes these failure modes across coding datasets and aggregation methods.Source: §1 · pp. 1–2
  • Only 2–4 attempts recover the bf16 27B model’s one-shot accuracy across all four benchmarks in a controlled one-bit experiment, with more than an order-of-magnitude less model memory.Source: §1 · p. 2
  • With eight attempts, Coder-7B essentially matches Coder-32B on MBPP+ and exceeds it on the other three benchmarks, while using about 4.3× less model memory.On Nvidia GPUs it also runs 1.3–5.9× faster and uses 1.3–6.7× less energy.Source: §1 · p. 2

2 RELATED WORK

Prior work studies test-time scaling, execution-guided code generation, compression, and inference systems. This paper focuses on when verified retry can trade model scale or precision for accuracy under memory constraints, while holding selectors fixed.

2 RELATED WORK

  • Prior work studies repeated sampling, aggregation, verifier-guided selection, candidate coverage, best-of-N selection, and imperfect verification; this paper characterizes when scale or precision can be exchanged for verified retry and separates generation, test-discrimination, and aggregation errors.Source: §2 · p. 2
  • Execution-guided code generation has used repeated sampling, execution feedback, learned verifiers, and model-generated tests; this work holds selectors fixed to study their value under memory constraints.Prior examples include S* and DiffCodeGen when public tests are unavailable (Li et al., 2025; He et al., 2026).Source: §2 · p. 2
  • Compression research reduces persistent weight memory through distillation, pruning, and quantization, whereas this paper asks how much accuracy verified retry recovers after a checkpoint is chosen for a hard memory budget.Distilled and pruned models are practical deployment alternatives, not controlled descendants of Qwen.Source: §2 · p. 2
  • Unlike speculative decoding, retry keeps only the chosen checkpoint resident and spends hardware headroom on additional candidates, so the study measures wall-clock time and energy.Source: §2 · p. 3

3 WHY PARALLEL RETRY CAN BE INEXPENSIVE

The section uses a roofline model to explain why parallel attempts can add little latency while inference remains bandwidth-bound. Retry headroom depends on batch size and hardware, and capability recovery is useful only if it occurs before that headroom is exhausted.

3 WHY PARALLEL RETRY CAN BE INEXPENSIVE

  • For batch size B, the roofline approximation sets latency to the larger of weight-transfer time and compute time, while the model uses P = Ld^2 parameters and approximately 2BP FLOPs.Activation traffic is omitted to isolate weight reuse and is smaller than weight traffic when B ≪ d.Source: §3 · p. 3
  • Retry remains inexpensive while NK ≤ B⋆(H): below the bandwidth–compute crossover, increasing batch size adds relatively little latency, but beyond it computation limits performance and latency grows approximately linearly.For one prompt with K parallel attempts, B ≈ K; with N concurrent prompts, B ≈ NK.Source: §3 · p. 3
  • For bf16 weights, B⋆ depends on the compute-to-bandwidth ratio FH/βH; the stated values are approximately 153 for A100, 295 for H100, and 290 for B200.Scaling compute and bandwidth together changes runtime but leaves B⋆ unchanged; Apple Silicon’s break-even point is measured empirically.Source: §3 · p. 3
  • A smaller model can spend part of its lower inference cost on additional attempts while remaining no slower than a reference, subject to the system’s retry budget Ksys.Lower numerical precision also reduces weight traffic, although low-precision kernels can change effective compute throughput.Source: §3 · p. 3
  • The useful regime is Krec ≤ Ksys: recovery at or below the system retry budget can match the reference while remaining faster and using less memory; otherwise retry may retain the memory advantage but lose the latency advantage.Source: §3 · p. 3
  • The roofline model is approximate, assumes attempts have similar lengths, and does not quantitatively predict measured speedups; length variation can increase latency when a batch waits for its slowest sequence.Adaptive stopping could mitigate that effect but is outside the fixed-retry setting.Source: §3 · pp. 3–4

4 MAIN RESULTS WITH PUBLIC-TEST VERIFICATION

Under public-test verification, the experiments ask whether retry can recover smaller or compressed models’ accuracy before systems costs erase their efficiency gains. Recovery depends on correct solutions remaining reachable, retry depth, and hardware profile.

4.1 EXPERIMENTAL SETTING

  • Deployments generate K candidates in parallel, select with deployment-time information, and evaluate correctness on held-out tests; public-test accuracy is exec@K, with Krec the smallest tested budget matching reference one-shot accuracy.For each candidate, visible tests are run; passing candidates are ranked by mean token log-probability, with an overall-score fallback if none passes.Source: §4.1 · p. 4

4.2 RESULTS

  • Coder-7B at K = 8 matches Coder-32B at K = 1 on MBPP+ and exceeds it on the other three benchmarks, while using 4.3× less model memory, running 1.3–5.9× faster, and using 1.3–6.7× less energy.Table 1 compares native Qwen2.5-Coder models on matched B200 workloads.Source: §4.2 · p. 5, Table 1 · p. 6
  • On LiveCodeBench, Coder-7B solves 5.7 percentage points fewer tasks than Coder-32B within 50 attempts, showing that retry cannot recover solutions removed from candidate coverage.Across the native scale ladder, candidate coverage varies by 13.1 points on LiveCodeBench versus 1.3–2.4 points on the other three benchmarks.Source: §4.2 · p. 7
  • Across direct bf16-to-lowest-memory quantization comparisons, weight memory falls 2.7–3.6× while accuracy after 50 verified attempts changes by at most 3.1 percentage points; one-bit quantization recovers bf16 one-shot accuracy in K = 2–4.The controlled one-bit experiment holds architecture and parameter count fixed, isolating precision.Source: §4.2 · pp. 6–7, Table 2 · p. 7
  • On MBPP+, Coder-7B reaches 83% of its eventual K = 50 accuracy gain by K = 4 and 93% by K = 8, while wall time rises 1.35× and 1.73×, respectively.Beyond moderate retry depth, accuracy gains diminish while wall time continues to rise.Source: §4.2 · p. 7
  • On M3 Max and M5 Pro, Coder-7B recovers Coder-14B one-shot accuracy in two attempts on HumanEval+, MBPP+, and BigCodeBench and four on LiveCodeBench, with all comparisons favorable.The study uses 64-GiB Apple Silicon systems with llama.cpp.Source: §4.2 · p. 7
  • Coder-3B’s deeper recovery can erase efficiency gains: LiveCodeBench needs K = 16 and is unfavorable in wall time and energy on both chips, despite roughly 4.3× memory savings.MBPP+ needs K = 8 and remains unfavorable in energy on both chips; HumanEval+ and BigCodeBench recover at K = 4 and remain favorable.Source: §4.2 · p. 7
  • At K = 4, Bonsai Q1 0 is slower than bf16 on M3 Max at 0.83–0.87× speedup but faster on M5 Pro at 1.58–1.67×, illustrating device-specific break-even.The roofline calculation anticipates this qualitative hardware dependence but does not predict measured ratios quantitatively.Source: §4.2 · pp. 7–8

5 MODEL-GENERATED VERIFICATION

The section tests whether smaller models can generate and select correct candidates without external tests, and finds that model-generated verification recovers substantial capability while adding distinct selection bottlenecks. Recovery depends on both candidate generation and whether generated tests distinguish and select correct solutions.

5.1 EXPERIMENTAL SETTING

  • When external tests are unavailable, the model generates tests, CodeT selects among candidates, and failed test generation falls back to mean token log-probability ranking.Final correctness is evaluated on separate held-out tests.Source: §5.1 · p. 8, §5 · p. 8
  • Selection errors arise from missing correct candidates, indistinguishable test-outcome signatures, or a selector that fails to exploit informative test outcomes.The gaps from CodeT to sig-oracle@K and from sig-oracle@K to oracle@K measure aggregation and test-discrimination failures, respectively.Source: §5.1 · p. 8

5.2 RESULTS

  • At K=50, CodeT reaches 78.6%, 83.5%, 28.0%, and 53.5% on MBPP+, HumanEval+, LiveCodeBench, and BigCodeBench, respectively—within 3.2 percentage points of Coder-32B one-shot on every benchmark.The 7B checkpoint uses about 10× less model memory and 6.23× less measured process memory.Source: §5.2 · pp. 8–9
  • With eight attempts, Coder-7B is within 1.1–3.9 percentage points of the Coder-32B one-shot reference across four benchmarks.Across 58 verified model–benchmark configurations on three benchmarks, CodeT improves one-shot accuracy by a median 7.3 percentage points.Source: §5.2 · p. 9
  • At K=500, the signature oracle remains 5.4–18.6 percentage points below candidate coverage, while CodeT trails it by 4.3–7.7 points across benchmarks.BigCodeBench is especially constrained because correct and incorrect programs often share generated-test outcomes; on LiveCodeBench, 57% of tasks still lack a correct candidate after 500 attempts.Source: §5.2 · p. 9
  • Parallel retry can recover substantial capability without external tests, but model-generated verification adds a bottleneck; when correct candidates remain reachable, generated-test quality can matter as much as more attempts.Source: §5.2 · p. 9

6 DISCUSSION AND CONCLUSION

The discussion argues that single-user local hardware leaves compute available for parallel attempts, which can recover capability lost to smaller models. The authors limit the evidence to single-turn code generation and identify test quality and interaction design as open issues.

6 DISCUSSION AND CONCLUSION

  • A 7B model with eight parallel attempts matches or exceeds a 32B model with one attempt while using less memory, time, and energy.The authors attribute this tradeoff to the smaller model remaining likely to generate a correct solution given enough attempts.Source: §6 · p. 10
  • The paper contrasts underused single-user local compute with shared datacenter GPUs, where additional generation competes with other users’ demand.Source: §6 · pp. 9–10
  • The results cover single-turn code generation and depend on good tests, which the model cannot always generate itself.Proposed next steps include interactive multi-turn local agents, parallel plans or solutions, improved verification, and training for aggregate success across parallel attempts.Source: §6 · p. 10

A ADDITIONAL EXPERIMENTAL RESULTS

The appendix adds experimental tables and describes complementary experiments on extreme one-bit compression and conventional quantization or model-size changes. It also supplements accuracy with system-cost and memory measurements.

A ADDITIONAL EXPERIMENTAL RESULTS

B ADDITIONAL EVIDENCE FOR THE MEMORY–RETRY TRADEOFF

  • The experiments pair one-bit Qwen3.6-27B compression with conventional quantization and model-size studies of Qwen2.5-Coder, adding wall-clock, energy, and process-memory measurements.Model memory includes checkpoint weights plus generation KV cache; peak process-level measurements are reported separately.Source: B ADDITIONAL EVIDENCE FOR THE… · p. 15

B.1 EXTREME ONE-BIT COMPRESSION AND PRESERVED CANDIDATE COVERAGE

The one-bit experiments show that repeated sampling can largely preserve candidate coverage despite lower one-shot reliability, but recovery budgets do not guarantee time or energy savings. Results also depend on test generation, output formatting, and the serving and accounting choices used.

B.1 EXTREME ONE-BIT COMPRESSION AND PRESERVED CANDIDATE COVERAGE

  • The study compares bf16, 4-bit, and one-bit versions of Qwen3.6-27B and evaluates model-generated verification alongside accuracy on the full task populations.For Qwen3.6/Bonsai BigCodeBench, canonical sanitization raises parse rate from 72.9% to 93.1% and pass@1 from 9.18% to 26.70% on the same generations.Source: §B.1 · pp. 15, 17
  • On HumanEval+, Q1 0 loses 3.8 percentage points in pass@1, yet both Q1 0 and bf16 solve 98.2% of tasks within 50 attempts.On MBPP+, the one-shot gap is 8.3 points and the 50-attempt gap is 1.3 points; on BigCodeBench, pass@1 is 19.2% versus 26.7%, while 50-attempt coverage remains within 1.5 points of bf16.Source: §B.1 · pp. 15–16
  • On LiveCodeBench, only 25.0%, 25.8%, and 26.5% of Q1 0, AWQ4, and bf16 generations contain the expected fenced code block, and many tasks have no correctly formatted candidate within 50 attempts.Among correctly formatted generations, held-out pass rates are 58.3%, 60.2%, and 63.2%; the paper evaluates the fixed deployed pipeline rather than conditioning on correct formatting.Source: §B.1 · p. 16
  • On LiveCodeBench, CodeT@8 is similar across Q1 0, AWQ4, and bf16 at 37.20%, 37.12%, and 38.13%, despite different test-generation success rates.The compact line-based format yields usable tests on 97.7%, 51.4%, and 75.4% of tasks, respectively; failed generation falls back to confidence ranking.Source: §B.1 · p. 16
  • Q1 0 reaches bf16 one-shot accuracy at K=2 on HumanEval+, LiveCodeBench, and BigCodeBench, and at K=4 on MBPP+.On BigCodeBench, Q1 0 rises from 19.22% at one attempt to 29.93% at two, exceeding bf16 one-shot accuracy of 26.70%.Source: §B.1 · p. 16
  • At benchmark-specific recovery budgets versus bf16 at K=1, speedup and energy saving are 0.72×/1.30× on HumanEval+, 0.42×/0.73× on MBPP+, 1.04×/1.61× on LiveCodeBench, and 0.71×/0.92× on BigCodeBench.At equal K on HumanEval+, Q1 0 is faster and uses less energy than bf16; the reported speedup rises from 1.32× at K=1 to 1.92× at K=50.Source: §B.1 · pp. 16–17, Table 6 · p. 17

B.2 PHYSICAL-DEVICE MEASUREMENTS ON APPLE SILICON

Physical-device results show that recovery depth and hardware profile jointly determine whether retry preserves accuracy while reducing serving cost. Conventional quantization greatly reduces weight memory, but its accuracy effects are small and not uniformly directional.

B.2 PHYSICAL-DEVICE MEASUREMENTS ON APPLE SILICON

  • The physical-device experiments measure runtime, energy, power, and swap separately from accuracy, and a generation-path check found at most 1.23-point pass@1 and 1.21-point oracle@K differences on HumanEval+.The M5 Pro LiveCodeBench run used a second chassis with matched reported chip, CPU, memory, and working-set specifications.Source: §B.2 · pp. 17–19, Table 7 · p. 18
  • 1.58–1.67× faster on M5 Pro but slower on M3 Max, Q1 0’s serving outcome reverses across chips; its throughput reaches 6.60–6.85× and 3.48–3.57× bf16, respectively.The measurements establish the serving outcome, not a unique hardware cause; bf16 uses swap in every matched Bonsai comparison, while Q1 0 does not.Source: §B.2 · p. 18
  • 2–4 attempts let Coder-7B recover Coder-14B one-shot accuracy across the four benchmarks, with every measured K ≤ 4 comparison favorable in wall time and energy on both chips.At K = 16 on LiveCodeBench, Coder-3B is unfavorable on both chips; the precise margin depends on hardware.Source: §B.2 · p. 18

B.3 CONVENTIONAL 4/8-BIT QUANTIZATION

  • 2.7×–3.6× lower weight memory accompanies relatively small accuracy changes under conventional 4/8-bit quantization, with the largest change after 50 attempts reaching 3.1 percentage points.The compared checkpoint for each model size was selected by weight memory, not benchmark accuracy.Source: Table 8 · p. 19, §B.3 · p. 19
  • No quantized checkpoint is uniformly best across benchmarks, and small accuracy differences do not vary monotonically with precision.The full precision sweep reports both public-test selection and CodeT; LiveCodeBench CodeT uses independently generated candidate and test pools.Source: §B.3 · pp. 19–20, Table 9 · p. 19

B.4 CANDIDATE COVERAGE UNDER MODEL SCALING

Model scaling often improves single-attempt reliability more than candidate coverage, though LiveCodeBench is an important exception. Retry also reduces sensitivity to precision in a sparse mixture-of-experts model, while its serving-cost profile differs from dense models.

B.4 CANDIDATE COVERAGE UNDER MODEL SCALING

  • 16.3, 7.7, and 15.4 percentage-point one-shot gains from 3B to 32B on MBPP+, HumanEval+, and BigCodeBench contrast with only 1.3, 2.4, and 1.5-point oracle@50 ranges.LiveCodeBench differs: one-shot accuracy rises 17.6 points and coverage changes 13.1 points across the size ladder.Source: §B.4 · p. 20
  • On MBPP+, Coder-32B AWQ4 and Coder-7B AWQ4 both reach 79.9% after 50 attempts, despite checkpoint sizes of 19.33 GB and 5.57 GB; larger models retain an advantage on HumanEval+ and LiveCodeBench.For Coder-7B on LiveCodeBench, public-test selection reaches 33.7% at K = 50 versus 37.1% of tasks with a correct candidate, indicating many remaining tasks lack a generated solution.Source: §B.4 · pp. 20–21

B.5 PRECISION REPLICATION IN A SPARSE MIXTURE-OF-EXPERTS MODEL

  • 0.8, 1.2, and 2.1 percentage-point full-precision-range changes after 50 attempts on MBPP+, HumanEval+, and BigCodeBench show relative stability across Qwen3-Coder MoE precisions, unlike LiveCodeBench’s 4.0-point range.The model has 30.5B total parameters and activates 3.3B per token; DeepSeek-Coder-6.7B was excluded from the precision comparison because its MBPP+ generations were malformed.Source: §B.5 · p. 21
  • 19,520 tokens/s and 2.83× less energy per token on LiveCodeBench distinguish Qwen3-Coder-30B-A3B from dense Coder-32B under the matched protocol.The MoE expert weights still occupy memory, although only a small fraction of the model is active per token.Source: §B.5 · p. 21

B.6 BIGCODEBENCH VERIFICATION LIMITS

Retry outcomes vary across compact checkpoints, budgets, verifiers, and hardware. The results distinguish gains from better candidate selection from limits imposed by candidate coverage, verification evidence, and serving cost.

B.6 BIGCODEBENCH VERIFICATION LIMITS

  • 10.9 points at 3B versus 5.8 points at 32B separate public-test selection from the best available candidate on BigCodeBench, while candidate coverage varies by only 1.5 points across model sizes.Model-generated tests leave an even larger gap for many checkpoints because correct and incorrect programs can yield identical test outcomes.Source: §B.6 · pp. 21–22, Table 12 · p. 22

B.7 ALTERNATIVE COMPACT DEPLOYMENT STRATEGIES

  • Coder-7B exceeds the Coder-32B one-shot reference on HumanEval+ and LiveCodeBench, while Coder-7B AWQ4 and Coder-3B GPTQ8 are statistically consistent with it on all three benchmarks.Granite-Code-3B also exceeds its within-family 34B reference on all three benchmarks while using 9.7× fewer weight bytes; results depend on checkpoint and verifier.Source: §B.7 · pp. 22–23, Figure 5 · p. 23

B.8 PRACTICAL RETRY BUDGETS

  • 83% of Coder-7B’s eventual MBPP+ gain arrives by K = 4 at 1.35× one-shot wall time, and 93% arrives by K = 8 at 1.73× wall time.Beyond about eight attempts, accuracy gains are small relative to the increase in serving time.Source: §B.8 · p. 23, Table 14 · p. 24

B.9 PUBLIC-TEST FALLBACK BEHAVIOR

  • 77.8% to 38.9% is the LiveCodeBench fallback-rate drop for Coder-7B from one to 50 attempts; at K = 50, its remaining advantage from larger models mainly reflects candidates the 7B model never generates.The selector falls back to mean token log-probability only when no candidate passes visible tests.Source: §B.9 · pp. 23–24, Table 15 · p. 24

B.10 SERIAL REASONING VERSUS PARALLEL RETRY

  • Neither eight non-reasoning attempts nor one reasoning attempt wins consistently across MBPP+, HumanEval+, and LiveCodeBench.At K = 50, the 72B model’s oracle@50 is below the 14B Coder model’s despite its higher one-shot accuracy.Source: §B.10 · p. 24, Table 16 · p. 25, §B.11 · p. 25

B.12 ENERGY SCALING UNDER PARALLEL RETRY

  • 6.9–18.6× is the measured energy increase from one to 50 parallel attempts, below the 50× independent-attempt cost; wide-batch AWQ4, however, falls to 0.51–0.62× bf16 speedups at K = 50.Energy comparisons are within benchmarks because workloads differ; quantization reduces memory but does not guarantee faster parallel decoding.Source: §B.12 · pp. 25–26, Figure 6 · p. 26, Figure 7 · p. 26

C.1 AGGREGATION WITH MODEL-GENERATED TESTS

C.2 WHERE MODEL-GENERATED VERIFICATION FAILS

Model-generated verification can fail because selection misses information, tests fail to distinguish programs, or no correct candidate is generated. Increasing test-writer scale does little to close some verification gaps, and test usability varies across models and benchmarks.

C.2 WHERE MODEL-GENERATED VERIFICATION FAILS

  • 6.7, 4.3, 7.7, and 7.0 percentage-point gaps at K = 500 separate CodeT from the best possible selection using observed test outcomes on MBPP+, HumanEval+, LiveCodeBench, and BigCodeBench.A further 6.5, 5.4, 8.9, and 18.6-point gap to candidate coverage reflects information absent from the tests; on LiveCodeBench, 57% of tasks still lack a correct candidate after 500 attempts.Source: §C.2 · p. 27, Figure 8 · p. 28

C.3 EFFECT OF TEST-WRITER SCALE

  • Replacing the 7B test writer with a 32B model changes CodeT accuracy by only 0.5 points on MBPP+ and 1.2 on HumanEval+, with similarly small changes to the outcome-based upper bound.LiveCodeBench is omitted because stored K = 500 data lack candidate programs for a paired comparison.Source: §C.3 · p. 28

C.4 USABILITY OF MODEL-GENERATED TESTS

  • 75% and 69% of R1-Distill-1.5B’s MBPP+ and HumanEval+ tasks have usable tests, but none of its LiveCodeBench test generations are usable.Usability means tests can be parsed and executed; these failures precede evaluation of whether tests distinguish correct from incorrect programs.Source: §C.4 · p. 28, Table 20 · p. 29
  • The controlled Qwen3.6/Bonsai LiveCodeBench experiment reports usable tests for 97.7%, 51.4%, and 75.4% of tasks with Q1 0, AWQ4, and bf16, respectively.The study describes matching candidates, tasks, and measurements across separately produced benchmark, test, checkpoint, and systems data.Source: §C.4 · p. 28, D MEASUREMENT CONSTRUCTION AND… · p. 28

D.1 TASK POPULATIONS, CANDIDATE POOLS, AND MEMORY ACCOUNTING

The appendix specifies the task populations and memory accounting behind the experiments, then checks measured process memory, runtime and energy under particular hardware and serving setups. It also bounds the systems conclusions to executable-feedback tasks and cautions that retry break-even depends on deployment conditions.

D.1 TASK POPULATIONS, CANDIDATE POOLS, AND MEMORY ACCOUNTING

  • Accuracy pools use 50 candidates per task, while the deeper verification study uses up to 500; task populations vary by experiment.EvalPlus uses its full configured HumanEval+ and MBPP+ sets, while LiveCodeBench uses 175 problems from the stated release range.Source: §D.1 · p. 29
  • The memory calculation adds checkpoint weights to KV cache, counting a shared prompt once and each candidate completion separately.It excludes activations, driver state, allocator overhead, and serving-engine workspace, so process memory is measured separately.Source: §D.1 · pp. 29–30
  • Coder-7B AWQ4 at K = 8 uses 5.57 GB for weights and 0.13 GB for KV cache, versus 65.53 GB of weights for Coder-32B bf16 at K = 1.The additional memory for several attempts is small relative to the checkpoint-size difference.Source: §D.1 · p. 30, Table 21 · p. 30

D.2 FROM MODEL MEMORY TO MEASURED PROCESS MEMORY

  • 21.58 GB for Coder-7B bf16 at K = 8 versus 73.02 GB for Coder-32B bf16 at K = 1 yields a 3.38× reduction in measured process memory.Coder-7B AWQ4 at K = 50 peaks at 11.73 GB, or 6.23× less than the 32B reference; process measurements validate overall savings, not precise retry-depth effects.Source: §D.2 · pp. 30–31, Table 22 · p. 31

D.3 RUNTIME AND ENERGY MEASUREMENT PROTOCOL

  • B200/vLLM workloads run 40–60 prompts concurrently, with wall time measured through workload completion and repeated runs used for latency ratios.Model comparisons use consistent serving stacks; Apple comparisons use llama.cpp, and runtime is measured for the checkpoint actually served.Source: §D.3 · p. 31

D.4 SCOPE AND LIMITATIONS

  • The tradeoff is established for code tasks with executable tests, not open-ended generation; reported systems costs omit generating and executing model-made tests.Retry break-even also varies with hardware, workload, model, and serving stack; adaptive stopping would be a different serving strategy from fixed parallel budgets.Source: §D.4 · pp. 31–32
  • Apple energy results are reported only when power records meet coverage and consistency checks, with comparisons measured on the same device within each shard.Some device results combine compatible disjoint shards.Source: §D.4 · p. 32

D.5 VALIDATION OF CANDIDATE AND VERIFIER DATA

This section validates that candidate programs, correctness labels, and generated-test outcomes are correctly matched before paired analyses. It reports which configurations enter the pooled result and describes limits on comparisons across independently generated pools and noisy estimates.

D.5 VALIDATION OF CANDIDATE AND VERIFIER DATA

  • Paired CodeT analyses require candidates, correctness labels, mean log-probabilities, and candidate-by-test outcomes from the same generation.Task names or filenames alone are insufficient to establish a valid match.Source: §D.5 · p. 32
  • 58 verified configurations enter the pooled CodeT improvement: 24 on MBPP+, 21 on HumanEval+, and 13 on BigCodeBench; 17 LiveCodeBench configurations are excluded from that pooled statistic.The LiveCodeBench experiments use independently generated candidate pools and remain valid for within-pool comparisons.Source: §D.5 · p. 32
  • Four duplicate generations with materially different correctness labels are excluded from paired analyses; DeepSeek-6.7B and Magicoder also have unusually high malformed-output rates.These exclusions do not affect the controlled Qwen precision comparisons, native accuracy–time experiments, memory results, fallback analysis, or K = 500 Qwen study.Source: §D.5 · p. 32
  • Pools are excluded from paired analyses only after independent checks identify systematic generation, extraction, or execution failures, not simply for low scores.CodeT requires the full candidate-by-test outcome matrix and checks extraction, decoding, parsing, expected outputs, and line-break preservation.Source: §D.5 · p. 32
  • When a task has no usable generated tests, CodeT falls back to the highest-confidence candidate, keeping test-generation failures in reported accuracy.Native LiveCodeBench generated-test experiments use independent pools, so comparisons with separate public-test runs are descriptive rather than paired.Source: §D.5 · p. 32
  • The reported LiveCodeBench CodeT accuracies are 18.3% for the 3B model and 32.0% for the 32B model.These results are from independently generated pools and are not paired candidate-by-candidate with the separate public-test generation.Source: §D.5 · pp. 32–33
  • The controlled Qwen3.6 LiveCodeBench study evaluates generated tests on existing K = 50 pools of 8,750 candidates per checkpoint across 175 tasks, with no identified pool mismatches.Differences in test usability therefore come from test generation and parsing rather than mismatched candidate pools.Source: §D.5 · p. 33
  • Within each K = 500 experiment, CodeT and the test-pattern upper bound use independent Monte Carlo subsets, so their plotted difference includes sampling variation.Independent 7B generations vary by roughly 1.5–2.3 percentage points in one-shot accuracy, a practical scale below which cross-generation differences should not be overinterpreted.Source: §D.5 · p. 33

Related papers

Loading 2610.15372v1…