PaperlayerView exact revision on arXiv

Source-linked AI summary

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

Atoosa Chegini, Soheil Feizi

arXiv:2606.01682v1cs.CLcs.AIcs.LG

TL;DR

Post-hoc selection cannot intervene before flawed small-model reasoning paths propagate, while PRM guided search requires reward-model training. Chunk-Level Guided Generation uses an off-the-shelf large language model to score fixed-length candidate chunks during generation, substantially improving mathematical reasoning and matching or outperforming PRM guided search on most benchmarks without reward-model training.

  • Problem

    Post-hoc selection cannot intervene after small-model samples have committed to flawed intermediate reasoning steps, motivating generation-time guidance without reward-model training.

  • Method

    Chunk-Level Guided Generation samples fixed-length candidate chunks from a small model and uses an off-the-shelf large model as a likelihood scorer to select each chunk before continuing generation.

  • Results

    Across mathematical reasoning benchmarks, chunk-level guidance substantially outperforms post-hoc selection and matches or outperforms PRM guided search on most benchmarks without reward-model training.

  • Takeaways & Limitations

    Fixed-length chunks make candidate likelihoods directly comparable because large-model likelihoods remain biased toward longer variable-length reasoning steps even after normalization.

  • Takeaways & Limitations

    Evaluation covers only mathematical reasoning and within-family model pairs, leaving other domains, cross-family pairs, and better-calibrated variable-length scoring unexplored.

Abstract

from arXiv · show

Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths. PRM guided search avoids this by scoring candidate continuations during generation, but requires a reward model trained with step-level labels. We propose Chunk-Level Guided Generation, a training-free alternative that uses an off-the-shelf large language model as a process scorer. At each step, a small model samples k fixed-length candidate chunks, while the larger model scores the candidates using likelihoods without generating any text. The selected chunk is committed before the next step, steering generation before errors can propagate. We instantiate this framework with two selection rules: Likelihood-Guided Selection (LGS), which selects the chunk with the highest length-normalized large-model log-probability, and Contrastive-Guided Selection (CGS), which subtracts the small model's log-probability to favor chunks where the large model's preference diverges from the small model's. We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias that persists even after length normalization, and that fixed-length chunks avoid this confound. On GSM8K, MATH, Minerva Math, AMC23, and AIME24 with Qwen2.5-1.5B guided by Qwen2.5-32B and Llama-3.2-1B guided by Llama-3.1-70B, CGS outperforms majority voting by up to 28 pp and, under matched guidance budgets, matches or outperforms Qwen2.5-Math-PRM-72B guided search on most benchmarks without reward-model training. With Qwen2.5-7B guided by Qwen2.5-72B, CGS reaches 81.8% on MATH and 63.6% on Minerva Math at k=16, surpassing majority voting by 4--6 pp. Finally, Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.

1 Introduction

Post-hoc selection cannot correct reasoning errors after complete small-model solutions are generated, while PRM guidance intervenes during generation but requires trained step-level reward models. Chunk-Level Guided Generation uses an off-the-shelf large model to score fixed-length candidate chunks during generation, with strong gains over post-hoc methods and competitive performance with PRM search.

  • Motivation: Post-hoc selection chooses among completed small-model trajectories, so it cannot intervene before an early reasoning error propagates.PRM guided search moves selection inside generation but requires a reward model trained for step-level evaluation.
  • Method: Chunk-Level Guided Generation samples k fixed-length chunks from a small model and commits the highest-scoring continuation according to a larger model’s likelihood.The larger model only scores candidate continuations and does not generate the final answer.
  • Method: LGS selects the highest length-normalized large-model log-probability, whereas CGS subtracts the small model’s likelihood to favor divergent large-model preferences.CGS emphasizes continuations where the larger model contributes information beyond the small model’s preferences.
  • Design Insight: 55.8% →52.6% on MATH shows that large-model Best-of-N likelihood scoring can degrade as k grows, despite the same signal helping at chunk level.The passage identifies timing, rather than the scorer itself, as the bottleneck.
  • Design Insight: Fixed-length chunks avoid the systematic bias toward longer reasoning steps that persists in large-model likelihood scores even after length normalization.Equal candidate lengths make scores directly comparable at each decision point.
  • Results: CGS produces substantially shorter reasoning traces than PRM guided search, whose responses contain up to 80% more tokens on GSM8K.The gains therefore do not come from generating longer solutions.
  • Results: CGS substantially outperforms post-hoc selection across five mathematical benchmarks and three model pairs, while matching or outperforming PRM guided search on most benchmarks without reward-model training.With Qwen2.5-7B guided by Qwen2.5-72B, CGS comes within 1.5 pp of PRM-72B guided search on average.

2 Related Work

Prior work uses post-hoc selection, trained step-level scorers, large-model text contributions, or token-level contrastive decoding. Chunk-Level Guided Generation instead keeps the small model as the sole generator while using the large model to score sampled chunks.

  • Sampling and post-hoc selection: Self-consistency, Best-of-N, and self-certainty generate complete solutions before selecting by final answers, verifiers, or confidence-based rankings.These methods represent the post-hoc selection paradigm.
  • Trained scorers and step-level guidance: Process reward models assign scores to individual reasoning steps using human or automated labels, while related methods still require reward-model training.PRM guided search applies these trained scorers during inference.
  • Large-to-small collaborative generation: Recent large-to-small collaborative methods let larger models generate reflective steps, lookahead segments, or reasoning insights during inference.These approaches differ from the paper’s scoring-only use of the large model.
  • Large-to-small collaborative generation: In Chunk-Level Guided Generation, the small model is the sole generator and the large model only scores fixed-length chunks sampled from it.The large model contributes no generated text.
  • Contrastive and speculative decoding: CGS adapts contrastive decoding’s log π_large(x)−log π_small(x) signal from token-level distribution steering to chunk-level selection among k candidates.The adaptation changes the selection unit from individual tokens to sampled chunks.

3 Method

Chunk-Level Guided Generation samples fixed-length continuations from a small model and uses a larger model to score and select one before continuing generation. The selected chunk is appended according to either likelihood-based scoring rule.

  • At each iteration, the small model samples k candidate continuations of length L from the current context.
  • Each candidate receives a length-normalized log-probability under the relevant language model.
  • Likelihood-Guided Selection chooses the candidate with the highest large-model likelihood.
  • Contrastive-Guided Selection scores candidates by subtracting the small model’s likelihood from the large model’s likelihood.
  • The highest-scoring chunk is appended to the context, and generation repeats until the selected chunk contains an end-of-sequence token.

4 Experiments

The experiments evaluate chunk-level guidance across five mathematical reasoning benchmarks, model pairs, baselines, and scoring setups. They compare when selection occurs and which model supplies the score, including post-hoc, Best-of-N, and PRM-guided methods.

  • Benchmarks and models: Experiments cover GSM8K, MATH, Minerva Math, AMC23, and AIME24 using two small-to-large model pairs.
  • Chunk-level guidance: Chunk-level guidance uses k ∈{8, 16, 32} candidate chunks of length L sampled at temperature 0.7.
  • Baselines: Majority@k generates complete small-model responses and selects the most frequent final answer.
  • Baselines: Self-Certainty and Borda count rerank completed responses using scorer-model confidence without influencing generation.
  • Baselines: Best-of-N selects among completed responses using the guided score, whereas PRM guided search scores candidate continuations at reasoning-step boundaries with a trained PRM.Best-of-N uses L=2048 so responses finish before scoring; the experiments use Qwen2.5-Math-PRM-72B for PRM guidance.

4.3 Comparison with PRM Guided Search

Under matched intervention budgets, CGS generally matches or exceeds PRM guided search across the evaluated model pairs and benchmarks. The advantage is strongest for the Llama pair, while the Qwen pair shows comparable performance on GSM8K and MATH and larger gains on several other datasets.

  • L is selected from {10, 20, 50, 100} so chunk interventions most closely match the average PRM scoring-step count.
  • +7.6 pp on Minerva Math, +2.2 pp on AMC23, and +1.1 pp on AIME24 are CGS’s average gains over PRM guided search for the Qwen pair.On GSM8K and MATH, CGS and PRM differ by at most 0.3 pp across k ∈ {8, 16, 32}.
  • CGS can match or outperform PRM guided search at comparable intervention budgets without training or using a reward model.

4.4 Why Fixed-Length Chunks?

The paper uses fixed-length chunks because large-model likelihoods systematically favor longer naturally delimited reasoning steps even after normalization. Accuracy is stable at short chunk lengths but degrades sharply when chunks approach complete responses.

  • Length bias: Longer naturally delimited reasoning steps receive higher large-model scores even after length normalization.The diagnostic scores the first two reasoning steps across GSM8K, MATH, and Minerva Math.
  • Fixed-length chunks: Fixed-length chunks make candidate scores directly comparable because candidates at each decision point share the same length and context.
  • Chunk-length sensitivity: Performance is stable for L ∈{10, 20} across datasets for the Qwen pair, while L=10 tends to perform best on Minerva Math for the Llama pair.
  • Chunk-length sensitivity: MATH remains stable through L=50, but accuracy begins declining at L=50 on GSM8K and Minerva Math and degrades sharply at L=2048.
  • Chunk-length sensitivity: L=20 is used thereafter because it lies in the stable region across all datasets and model pairs without per-dataset tuning.

4.6 Main Results

Chunk-level guidance substantially outperforms post-hoc selection and often matches or exceeds PRM guided search. CGS generally improves over LGS, while fixed-length chunk evaluation enables reliable large-model guidance.

  • Best-of-N falls below Majority@k on MATH for the Qwen pair at every k and for the Llama pair at k=16 and k=32.
  • 24 out of 30 settings: Self-Certainty (small) underperforms Majority@k, while post-hoc methods remain below chunk-level guidance in nearly all settings.
  • 92.5% on GSM8K and 50.8% on AMC23: CGS gains +12.8 and +16.6 pp over Majority@k on the Qwen pair at k=32.
  • +27.9 pp on GSM8K and +14.3 pp on Minerva Math: CGS gains over Majority@k on the Llama pair at k=32.
  • CGS outperforms LGS by 2.8 pp on the Qwen pair and 2.2 pp on the Llama pair across datasets and k values.
  • Averaged across five datasets and three k values, CGS outperforms PRM by +2.8 pp on the Qwen pair and +4.7 pp on the Llama pair.

4.7 Scaling to a Larger Small Model: Qwen2.5-7B →72B

With Qwen2.5-7B guided by Qwen2.5-72B, CGS remains effective and nearly matches the large-model greedy ceiling, while producing shorter traces than PRM guided search.

  • 63.7% average accuracy: CGS outperforms the best post-hoc method by +3.6 pp and nearly matches the 72B greedy ceiling of 64.1%.
  • 81.8% on MATH and 65.8% on AMC23: CGS surpasses the 72B greedy baseline at k=16.
  • CGS trails PRM-72B guided search by 1.5 pp on average in the 7B-to-72B setting.
  • CGS produces shorter reasoning traces than PRM guided search on all five datasets; on GSM8K, PRM uses 311 tokens versus 173 for CGS.

5 Conclusion

Chunk-Level Guided Generation uses an off-the-shelf large model to score fixed-length chunks during small-model generation. It improves mathematical reasoning without reward-model training by intervening before incorrect paths fully develop.

  • Chunk-Level Guided Generation is a training-free method that scores fixed-length candidate chunks with a large language model during generation.
  • Intervening at each chunk step lets the method steer the small model before incorrect reasoning paths fully develop, addressing post-hoc selection’s core limitation.
  • Across five mathematical benchmarks and two model pairs, chunk-level guidance outperforms majority voting, Best-of-N, and self-certainty-based reranking, while matching or outperforming PRM guided search on most benchmarks without reward-model training.

Limitations

The evaluation is limited to mathematical reasoning and within-family Qwen-to-Qwen and Llama-to-Llama model pairs, leaving broader transfer and cross-family effectiveness untested.

  • All experiments use mathematical reasoning benchmarks, so generalization to coding or commonsense reasoning remains an open question.
  • Only within-family Qwen-to-Qwen and Llama-to-Llama pairs are evaluated; cross-family effectiveness is untested.
  • Whether a better-calibrated variable-length scoring scheme could match fixed-length chunks remains unexplored.

A Prompt Template

The shared instruction template prompts step-by-step reasoning and requires a structured JSON response, with predictions extracted from the “answer” field.

  • The template elicits step-by-step reasoning in a structured JSON response and uses the “answer” field for prediction extraction.
Loading 2606.01682v1…