PaperlayerView exact revision on arXiv

Source-linked AI summary

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

Mobina Kashaniyan, Ali Jannesari

arXiv:2609.19499v2cs.LGcs.DCcs.PF

TL;DR

Candidate count N tells how many responses are generated, but not whether they run in one batched call or several sequential calls. This paper compares fixed-N generation schedules and measures their systems costs. At N = 8, more calls consistently increase systems cost, with the pattern also appearing across independent A100 jobs and in short-output SciQ/V100 experiments.

  • Problem

    Candidate count N does not specify how candidates are executed, although different schedules for the same budget can have different systems costs and are often not reported fully.

    Source: §I · p. 1

  • Method

    The paper compares four schedules at fixed N = 8 and measures latency, throughput, GPU-hours, utilization, and gross GPU-device energy.

    Source: §I · p. 1, C. Effect of Generation Schedule at… · p. 5

  • Results

    At fixed N = 8, splitting candidates across more calls consistently increases latency, GPU-hours, and gross GPU-device energy while reducing throughput; 8×1 uses 4.64× as much energy for Phi-3 and 4.86× for Qwen as 1×8.

    Source: §VIII · p. 7

  • Takeaways & Limitations

    Multi-candidate evaluations should report candidate count, call structure, batching mode, systems metrics, and the energy-measurement boundary.

    Source: §VI · p. 6

    The experiments cover two models, two GPU architectures, and batch-scheduled Hugging Face generation, so measured ratios should not be assumed to hold unchanged for larger models, other GPU generations, continuous batching, or multi-GPU inference.

    Source: §VII · p. 7

Abstract

from arXiv · show

Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. However, N tells us how many candidates are generated, not how they are executed. The same candidate budget can be produced in one batched generation call or split across several sequential calls with smaller batch sizes. We first study the effect of increasing N on reasoning accuracy using Phi-3-mini and Qwen2.5-1.5B on 500 GSM8K prompts. As expected, increasing N from 1 to 8 improves accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B. However, accuracy alone does not show the systems cost of using a larger candidate budget. We therefore fix N = 8 and compare four generation schedules: 1x8, 2x4, 4x2, and 8x1, where axb denotes a generation calls with b candidates per call. We measure latency, throughput, GPU-hours, and gross GPU-device energy while keeping the total candidate count fixed. On A100 GPUs, eight serial calls use 4.64-4.86x as much gross GPU-device energy and have 5.77-6.12x the P95 latency of one batched call with eight candidates. The same pattern appears across three independently scheduled A100 nodes per model and in short-output SciQ/V100 experiments. These results show that candidate count alone is not enough to describe the systems cost of multi-candidate test-time scaling. When candidates are independent and memory allows it, fewer generation calls with larger batch sizes are more efficient. Evaluations should therefore report not only candidate count and accuracy, but also generation schedule and GPU-level systems metrics.

I. INTRODUCTION

Candidate count alone does not describe how multi-candidate inference is executed, although generation schedules can affect systems cost. This paper compares fixed-budget schedules and reports substantially higher energy for serial generation than for one batched call.

I. INTRODUCTION

  • Candidate count N records how many candidates are generated, not whether they run in one batched call or several sequential calls.Those execution choices can produce different latency, throughput, GPU-hours, utilization, and energy, making systems results harder to reproduce and compare when call structure is unreported.Source: §I · p. 1
  • The paper formalizes a generation schedule and notes that representative test-time-scaling studies often omit calls per query and candidates per call.The schedule is represented as S = (b1, . . . , bC).Source: §I · p. 1
  • 4.64–4.86× as much energy is used by serial execution as by a single eight-candidate batched call on A100 GPUs.The fixed-N comparison measures latency, throughput, GPU-hours, utilization, and gross GPU-device energy.Source: §I · p. 1
  • When candidates are independent and memory allows it, fewer generation calls with larger batches are more efficient, a pattern also evaluated on other A100 nodes and SciQ/V100.The paper also recommends reporting generation schedule and GPU-level systems metrics.Source: §I · p. 1

A. Test-Time Scaling

  • Prior test-time-scaling work studies allocating extra inference compute to extend or refine reasoning trajectories or to generate and combine candidate responses.Prior work reports that additional test-time compute can improve reasoning and sometimes help smaller models approach larger-model performance under similar inference budgets [2]–.Source: A. Test-Time Scaling · p. 2

B. Multi-Candidate Sampling

Prior multi-candidate studies mainly examine candidate budgets or selection methods, while inference-systems research examines execution efficiency. The paper connects these topics and audits what representative studies report about execution details and energy.

B. Multi-Candidate Sampling

  • Prior multi-candidate studies mainly examine how many candidates to generate or how to select among them, rather than the systems cost of executing a fixed candidate budget across different calls.Self-consistency selects the most frequent answer, while best-of-N uses a verifier, reward model, or confidence measure.Source: B. Multi-Candidate Sampling · p. 2

C. Inference Scheduling and Energy

  • Inference-efficiency studies cover batching, scheduling, memory management, resource allocation, and energy, and this work measures generation-call structure at fixed N.Prior systems work includes ORCA, vLLM, and Sarathi-Serve –.Source: C. Inference Scheduling and Energy · p. 2

D. Reporting Practices in Prior Work

  • Representative studies commonly report candidate budgets but omit full generation-call structure or measured energy cost.The audit records a field only when explicitly stated in the paper or supplementary material; “NR” means not reported.Source: D. Reporting Practices in Prior Work · p. 2, Table I · p. 2

A. Candidate-Generation Strategies

The paper separates the total candidate budget from its execution schedule, comparing sequential calls and within-call batches while keeping the candidate count fixed. It also tracks logical token volume to check that schedules are comparable.

A. Candidate-Generation Strategies

  • For N = 8, the compared schedules are 1×8, 2×4, 4×2, and 8×1, with calls sequential on one GPU and candidates within each call generated together.All schedules use the same prompts, decoding settings, answer extraction, and voting procedure; the figure depicts the schedules and shared plurality vote.Source: A. Candidate-Generation Strategies · p. 2, Figure 1 · p. 2
  • Accuracy is evaluated for N ∈ {1, 2, 3, 4, 8} using the first N candidates, while systems measurements are collected separately for each schedule.Candidates are sampled independently across schedules in the systems experiments.Source: A. Candidate-Generation Strategies · pp. 2–3

B. Token Accounting

  • At fixed N = 8, candidate count is identical across schedules and logical generated-token volume remains closely matched.This supports comparing schedule costs while holding the overall candidate budget fixed.Source: B. Token Accounting · p. 3

C. Answer Extraction and Voting

Candidates are converted to answers and combined by plurality vote, with explicit rules for extraction failures and ties. The measured energy interval includes generation and voting, but excludes model loading and warm-up.

C. Answer Extraction and Voting

  • Candidates with the same extracted answer form a group, and the largest group determines the prediction without a verifier, reward model, or token-level score.GSM8K answers are extracted and normalized numerically; SciQ answers are extracted as choices A–D.Source: C. Answer Extraction and Voting · p. 3
  • If answers tie, the first occurring answer wins; in the accuracy analysis this can make N = 2 identical to N = 1 when the first two candidates disagree.The paper also evaluates N = 3 and reports a sensitivity analysis using uniform random tie-breaking.Source: C. Answer Extraction and Voting · p. 3

D. Measurement Boundary and Energy

  • Gross GPU-device energy spans prompt processing, prefill, decoding, all generation calls, answer extraction, and voting; model loading, warm-up, token counting, and grading are excluded.Idle energy is not subtracted, and the measure is neither isolated dynamic computation energy nor whole-node energy.Source: D. Measurement Boundary and Energy · p. 3

IV. EXPERIMENTAL DESIGN AND MEASUREMENT

The evaluation examines accuracy as candidate count grows, systems costs across fixed-budget schedules, and whether scheduling effects persist across nodes and a short-output workload. It uses GSM8K and SciQ with Phi-3 and Qwen on A100 and V100 GPUs.

A. Study Overview

  • The study measures accuracy as N grows, systems costs as candidate count and schedule vary, and robustness across GPU nodes and a short-output workload.Table II summarizes the five experiments used for these goals.Source: A. Study Overview · p. 3, Table II · p. 3
  • The accuracy study uses 500 GSM8K prompts; systems experiments use the same 100 prompts for three repetitions and compare four schedules at fixed N=8.The fixed-budget sweep also tests endpoints on two additional A100 nodes per model and repeats the full schedule sweep on 500 SciQ prompts over V100 GPUs.Source: A. Study Overview · p. 3

C. Models, Workloads, and Hardware

  • Phi-3-mini-4k-instruct and Qwen2.5-1.5B-Instruct are evaluated on GSM8K and SciQ; primary schedule tests use A100-SXM4 80 GB GPUs, while SciQ uses V100 PCIe 32 GB GPUs.The batched-scaling experiment pairs Phi-3 with a V100 and Qwen with an A100, so their absolute systems values are interpreted separately.Source: C. Models, Workloads, and Hardware · p. 4

D. Output-Length Validation

On GSM8K, accuracy rises as the candidate budget increases, while the study also checks output lengths, tie and extraction diagnostics, and statistical variability. SciQ responses are substantially shorter than GSM8K responses.

D. Output-Length Validation

  • Mean responses are 251 and 270 tokens for Phi-3 and Qwen on GSM8K, versus 3.46 and 1.71 tokens on SciQ, respectively.Only 2.0% and 5.25% of GSM8K candidates reach the 512-token limit; on SciQ's 64-token limit, 1.93% of Phi-3 and no Qwen candidates reach it.Source: D. Output-Length Validation · p. 4

E. Statistical Analysis

  • Systems results are means±SD across three repetitions; P95 latency is computed per repetition, while confidence intervals use paired hierarchical bootstraps and cross-node results report the observed range across three jobs.Accuracy intervals use 10,000 prompt-level bootstrap resamples, paired for candidate-budget comparisons.Source: E. Statistical Analysis · p. 4

A. Accuracy Gains from Increasing N

B. Systems Cost of Increasing Batched N

At fixed N=8, splitting candidates across more generation calls raises energy and P95 latency; increasing batched N also raises total query costs while lowering energy per token. Under the studied conditions, fewer calls with larger batches are more efficient when memory allows.

B. Systems Cost of Increasing Batched N

C. Effect of Generation Schedule at Fixed N = 8

D. Cross-Node Robustness

The 8×1 versus 1×8 scheduling effect remained stable across three independently scheduled A100 jobs per model, though its exact ratios may not generalize to other systems.

D. Cross-Node Robustness

  • The 1×8 and 8×1 endpoints were repeated in three independently scheduled A100 jobs per model.Table VII reports the observed ratio ranges across those jobs.Source: D. Cross-Node Robustness · p. 5, Table VII · p. 5
  • Serial throughput stayed at about 17–18% of batched throughput in every independently scheduled A100 job.The observed ranges were narrow compared with the scheduling effect.Source: D. Cross-Node Robustness · p. 5
  • The results are consistent across the tested nodes, but the exact ratios are not assumed to hold on other GPU architectures, clusters, or software environments.The table caption identifies the ratios as 8×1/1×8 ranges across the three jobs per model.Source: D. Cross-Node Robustness · p. 5, Table VII · p. 5

E. Length-Stratified GSM8K Analysis

Across GSM8K response-length groups, the scheduling penalty appears beyond the longest outputs; the same fixed-budget pattern also appears in short-output SciQ on V100 GPUs.

E. Length-Stratified GSM8K Analysis

  • For Phi-3 across GSM8K response-length groups, energy ratios were 4.40–4.77 and latency ratios were 5.45–6.16, with neither ratio monotonic in response length.Figure 3 shows the groupwise ratios; each group contains 25 prompts.Source: E. Length-Stratified GSM8K Analysis · p. 5, Figure 3 · p. 6
  • For Qwen across GSM8K response-length groups, energy ratios were 4.51–5.09 and latency ratios were 5.15–5.88, so the penalty appeared across all four groups rather than only the longest outputs.Figure 4 shows the groupwise ratios; this analysis is descriptive because each group has 25 prompts and schedules independently sample candidates.Source: E. Length-Stratified GSM8K Analysis · p. 5, Figure 4 · p. 6

F. Short-Output Validation on SciQ

  • From 1×8 to 8×1 on SciQ/V100, gross energy increased 2.57× for Phi-3 and 3.34× for Qwen, while throughput fell to 36% and 23% of the batched baseline.Mean latency increased 2.88× for Phi-3 and 4.42× for Qwen; Table VIII normalizes values to 1×8 within each model.Source: F. Short-Output Validation on SciQ · p. 6, Table VIII · p. 6
  • The SciQ/V100 result supports the narrower conclusion that the fixed-N scheduling effect appears in both longer-output GSM8K/A100 and short-output SciQ/V100 settings.The ratio-size difference cannot be attributed only to output length because the GPU differs.Source: F. Short-Output Validation on SciQ · p. 6

VI. DISCUSSION

At fixed N, splitting candidates across more calls raises systems costs, while larger batches are more efficient when candidates are independent and memory permits. The discussion also explains possible overheads and limits the interpretation to measured end-to-end effects.

VI. DISCUSSION

  • For independent candidates, fewer generation calls with larger batches are more efficient when memory allows, but 1×N is not always optimal.Memory limits, request dependencies, continuous batching, and serving constraints may require another schedule.Source: §VI · p. 6
  • Under the measured configurations, moving from 1×8 to 8×1 across one million queries adds about 1.37 MWh and 10,401 GPU-hours for Phi-3, or 1.00 MWh and 9,631 GPU-hours for Qwen.These are linear illustrations of measured configurations, not projections to other deployments.Source: §VI · p. 6
  • Multi-candidate evaluations should report generation calls, candidates per call, batching mode, latency, throughput, GPU-hours, and the energy-measurement boundary alongside N.The discussion says this supports reproducibility and comparison when schedules differ despite matching model, dataset, decoding settings, and candidate count.Source: §VI · p. 6

A. Why Generation Schedule Changes Systems Cost

  • More calls can add framework and synchronization overhead, repeat prompt-processing and generation-start work, and expose less parallel work to the GPU.The measurements combine these possible effects rather than isolating their individual contributions.Source: A. Why Generation Schedule Changes… · p. 7
  • In Phi-3/A100 measurements, mean power fell from 177.8 W for 1×8 to 139.8 W for 8×1 while mean latency rose 5.97×, and the longer execution time outweighed lower average power in total energy.Separating the contributions of prompt processing, call overhead, synchronization, utilization, and other low-level effects requires more detailed profiling.Source: A. Why Generation Schedule Changes… · p. 7

B. Practical Schedule Selection

The paper proposes choosing the largest feasible candidate batch to minimize calls at fixed budget, while treating that heuristic as a starting point rather than a universal optimizer. Its conclusion recommends reporting schedule and systems metrics alongside candidate count.

B. Practical Schedule Selection

  • For independent candidates, use the largest feasible batch size to minimize generation calls while keeping the candidate budget fixed.If eight candidates fit in memory, the paper favors 1×8; if only four fit, it favors 2×4 over schedules with smaller calls.Source: B. Practical Schedule Selection · p. 7
  • The heuristic is not a universal optimizer: the best schedule may vary with model size, sequence length, memory, concurrent traffic, and inference engine.A more general scheduler could predict energy, latency, and memory costs and select the lowest-cost configuration satisfying system constraints.Source: B. Practical Schedule Selection · p. 7

VII. LIMITATIONS AND FUTURE WORK

  • The experiments cover two models, two GPU architectures, and batch-scheduled Hugging Face generation; measured ratios should not be assumed unchanged for other systems.The conclusion supports reporting schedule and systems metrics alongside candidate count so costs can be reproduced and compared.Source: §VII · p. 7, §VIII · p. 7
Loading 2609.19499v2…