PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning with Sampling: Your Base Model is Smarter Than You Think

Aayush Karan, Yilun Du

arXiv:2510.14901v1cs.LGcs.AIcs.CL

TL;DR

The paper asks whether RL-posttraining’s reasoning gains are already latent in base models and can be elicited without additional training. It introduces likelihood-based MCMC sampling from a sharpened power distribution, achieving RL-comparable or better single-shot results while preserving diversity. The approach is training-, dataset-, and verifier-free, though MCMC may mix exponentially slowly on long sequences.

  • Problem

    The paper asks whether reasoning capabilities appearing after RL-posttraining are genuinely novel or already present in base models.

  • Method

    The method uses an approximate MCMC algorithm to sample a power distribution defined entirely by base-model likelihoods.

  • Results

    Across multiple models and tasks, sampling reaches RL-comparable single-shot reasoning, sometimes exceeds RL out of domain, and avoids RL’s diversity collapse.

  • Takeaways & Limitations

    Inference-time compute and better use of base-model likelihoods can expand reasoning beyond settings with easily verifiable rewards.

  • Takeaways & Limitations

    MCMC sampling can require exponentially many samples before convergence, especially in high-dimensional long-sequence spaces.

Abstract

from arXiv · show

Frontier reasoning models have exhibited incredible capabilities across a wide array of disciplines, driven by posttraining large language models (LLMs) with reinforcement learning (RL). However, despite the widespread success of this paradigm, much of the literature has been devoted to disentangling truly novel behaviors that emerge during RL but are not present in the base models. In our work, we approach this question from a different angle, instead asking whether comparable reasoning capabilites can be elicited from base models at inference time by pure sampling, without any additional training. Inspired by Markov chain Monte Carlo (MCMC) techniques for sampling from sharpened distributions, we propose a simple iterative sampling algorithm leveraging the base models' own likelihoods. Over different base models, we show that our algorithm offers substantial boosts in reasoning that nearly match and even outperform those from RL on a wide variety of single-shot tasks, including MATH500, HumanEval, and GPQA. Moreover, our sampler avoids the collapse in diversity over multiple samples that is characteristic of RL-posttraining. Crucially, our method does not require training, curated datasets, or a verifier, suggesting broad applicability beyond easily verifiable domains.

1 Introduction

The paper asks whether RL-posttraining creates genuinely new reasoning capabilities or sharpens capabilities already present in base models. It shows that inference-time sampling can approach or exceed RL performance while preserving sample diversity and avoiding training, dataset, and verifier requirements.

  • Motivation: The paper investigates whether RL-posttraining produces novel behaviors or a sharper version of capabilities already present in base models.This question is framed as distribution sharpening.
  • Results: The sampling algorithm achieves single-shot reasoning performance on par with RL-posttraining and can outperform it on some out-of-domain tasks.The comparison spans verifiable and general tasks, including MATH500, HumanEval, GPQA, and AlpacaEval 2.0.
  • Contributions: The method is training-free, dataset-free, and verifier-free, unlike RL methods requiring posttraining infrastructure and reward signals.The paper identifies hyperparameter sweeps, curated datasets, and guaranteed verifier access as weaknesses avoided by the sampler.
  • Contributions: The paper introduces a power distribution target and an approximate MCMC sampler that iteratively resamples token subsequences using base-model likelihoods.The contributions include the power distribution, an approximate sampling algorithm, and evaluations across several models and tasks.
  • Results: Over multiple samples, the sampler avoids the generation-diversity collapse associated with RL-posttraining while retaining strong single-to-few-shot reasoning.The authors describe this as combining reasoning capability with sample diversity.
  • Conclusion: The results suggest that current sampling methods underuse base-model single-shot reasoning capabilities.The authors report this conclusion across their experiments.

2 Related Works

Related work covers RL posttraining, autoregressive MCMC sampling, and annealed sampling. The paper differs from reward- or value-guided approaches by specifying its target distribution entirely through the base LLM.

  • Reinforcement learning for LLMs: RL posttraining has improved LLM performance through human-feedback and verifiable-reward signals, especially on mathematics and other difficult reasoning tasks.The cited literature includes RLHF and RLVR approaches.
  • Autoregressive MCMC sampling with LLMs: Prior autoregressive MCMC methods often tilt base-model sampling toward external rewards or expected future rewards using value functions.Sequential Monte Carlo maintains and updates multiple candidate sequences according to expected future reward.
  • Annealed sampling for diffusion: Annealed sampling targets tempered distributions to improve sampling from complex multimodal distributions and mitigate mode collapse.Related diffusion work uses sequences of annealed distributions to transition toward tilted targets.

3 Preliminaries

The preliminaries define an autoregressive language-model distribution over finite token sequences. Sampling the model token by token from its conditional distributions yields samples from the corresponding joint sequence distribution.

  • Token sequences: A finite token vocabulary generates finite sequences x0:T, with each token xi belonging to the vocabulary.The sequence length T is a nonnegative integer.
  • Autoregressive distributions: An LLM defines a joint distribution over token sequences by learning conditional token distributions p(xt|x<t) autoregressively.The joint distribution follows from the autoregressive factorization.
  • Autoregressive sampling: Standard sampling draws tokens sequentially from these conditionals, thereby sampling from the model’s joint sequence distribution.The procedure follows directly from the autoregressive identity.

4 MCMC Sampling for Power Distributions

The paper targets the sharpened power distribution p^α and develops MCMC procedures to sample from it using base-model likelihoods. This approach favors high-likelihood future paths, avoids the mismatch with ordinary low-temperature sampling, and uses autoregressive intermediate distributions to address convergence challenges.

  • 4 MCMC Sampling for Power Distributions: Distribution sharpening reweights high-likelihood regions upward and low-likelihood regions downward, motivating p^α as a target that can emulate RL posttraining.The paper frames this target around the hypothesis that RL-posttrained models may be sharpened versions of base models.
  • 4.1 Reasoning with Power Distributions: Low-temperature sampling does not sample from the power distribution p^α, despite using temperature τ = 1/α.Their next-token conditional distributions differ because the two strategies aggregate future likelihoods differently.
  • 4.1 Reasoning with Power Distributions: p^α favors tokens with fewer but higher-likelihood future paths, whereas low-temperature sampling favors tokens with several lower-likelihood completions.The distinction arises because p^α exponentiates future-path probabilities before aggregating them, unlike low-temperature sampling.
  • 4.1 Reasoning with Power Distributions: The power distribution implicitly biases generation toward planning for future high-likelihood tokens, which the paper connects to critical windows in reasoning.The paper relates high-average-likelihood choices that trap outputs in low-likelihood futures to reasoning failures.
  • 4.2 The Metropolis-Hastings Algorithm: Metropolis-Hastings enables approximate sampling from the unnormalized p^α distribution using relative likelihoods and a tractable proposal sampler.Candidates are proposed, accepted according to relative target weights, or rejected so the chain remains at its current sequence.
  • 4.3 Power Sampling with Autoregressive MCMC: Autoregressive intermediate distributions initialize successive MCMC processes, helping avoid pathological initializations in the high-dimensional token-sequence space.MCMC can otherwise require exponentially many samples to mix, especially for long sequences.
  • 4.3 Power Sampling with Autoregressive MCMC: The algorithm is single-shot inference-time scaling: extra inference calls improve the sampled sequence while acceptance decisions use only base-model likelihoods.The paper distinguishes this from generating multiple final samples.

5 Experiments

The experiments evaluate power sampling across mathematics, coding, STEM, and helpfulness benchmarks, finding strong single-shot performance, preserved diversity, and sensitivity to sampling hyperparameters and compute.

  • Evaluation: Power sampling is evaluated on MATH500, HumanEval, GPQA, and AlpacaEval 2.0 using Qwen2.5-Math-7B, Qwen2.5-7B, and Phi-3.5-mini-instruct.GRPO baselines are produced by posttraining the same models on the MATH training split.
  • Main results: Up to +51.9% on HumanEval and +25.2% on MATH500, power sampling delivers large single-shot gains across models and tasks.The reported gains include Phi-3.5-mini on HumanEval and Qwen2.5-Math on MATH500.
  • Main results: Power sampling matches GRPO on in-domain MATH500 and GPQA, outperforms GRPO on HumanEval by up to +59.8%, and consistently wins on non-verifiable AlpacaEval 2.0.These results extend beyond tasks with automatically verifiable rewards.
  • Analysis: Power sampling targets higher-likelihood base-model responses while retaining more distributional spread than GRPO, whose responses concentrate near the highest-likelihood peak.The corresponding confidence distributions are similarly high for power sampling and GRPO, and higher confidence correlates with correct reasoning.
  • Analysis: On MATH500, average response lengths are 600 tokens for the base model, 671 for GRPO, and 679 for power sampling.The longer power-sampling responses arise without explicitly encouraging longer generations.
  • Analysis: Power sampling has a strictly better MATH500 pass@k curve than GRPO and the base model, while matching the base model's high-k pass rate.This indicates sustained generation diversity across multiple samples.
  • Hyperparameters: An intermediate α = 4.0 performs best, accuracies remain relatively stable for α ≥2.0, and accuracy rises with MCMC steps until NMCMC = 10.The difference between NMCMC = 2 and NMCMC = 10 is no more than 3-4%, while using at least two steps instead of none yields a substantial 3-4% increase.
  • Hyperparameters: With NMCMC = 10, T = 679, and B = 192, power sampling uses 8.84× as many tokens as standard inference.The reported cost is roughly comparable to one epoch of GRPO training before the passage truncates.

6 Conclusion

The paper presents power sampling as a training-free way to sample from a sharpened base-model distribution using autoregressive MCMC. It reports reasoning performance comparable to, and sometimes better than, RL posttraining, while suggesting that base-model capabilities are underused at sampling time.

  • 6 Conclusion: Power sampling uses classic MCMC with autoregressive generation to approximate sampling from an intractable power distribution over base-model sequences.The method requires no additional training or external signal.
  • 6 Conclusion: The algorithm achieves single-shot reasoning performance on par with, and sometimes better than, a state-of-the-art RL-posttraining algorithm.The conclusion frames power sampling as a direct base-model alternative to RL posttraining.
  • 6 Conclusion: The results suggest that base-model capabilities are underutilized at sampling time and that high-likelihood regions are closely related to strong reasoning capabilities.Additional inference-time compute is presented as a direction for extending reasoning beyond verifiable domains.

A.1 Additional Theoretical Discussion

The theoretical discussion studies pivotal tokens that lead to high- or low-likelihood futures. It argues that power sampling can favor a token with better future completions even when its immediate marginal weight is lower, unlike low-temperature sampling.

  • Definitions: Pivotal tokens are intermediate tokens that strongly influence final-generation quality, distinguished by whether they lead to high- or low-likelihood futures.A positive pivotal token concentrates its marginal weight on a high-likelihood completion, while a negative pivotal token leads to dispersed low-likelihood futures.
  • Formalization: The formal analysis represents a positive pivotal token as placing all marginal weight ε on one future completion, whereas a negative pivotal token distributes marginal weight ε′ across support N.The resulting power-distribution comparison depends on the relationship between these marginal weights and future likelihoods.
  • Comparison: Power sampling can favor a positive pivotal token over a negative one even when the negative token has higher marginal weight.Low-temperature sampling always favors the negative pivotal token in this scenario.
  • Comparison: When the positive token's future likelihood exceeds every future likelihood of the negative token, power sampling upweights the positive token while low-temperature sampling upweights the negative one.This comparison isolates the effect of future-completion likelihood from immediate token mass.
  • Formalization: Power sampling favors the positive token when its individual completion likelihood exceeds the negative token's completion likelihood, even if its marginal mass is smaller.This is the stated mechanism by which power sampling can prefer concentrated, high-quality futures.

A.2 Pass@k Accuracies over Multiple Domains

Power sampling is evaluated against GRPO and the base model across MATH500, GPQA, and HumanEval using Pass@k. It generally improves multi-sample performance while preserving response diversity, though HumanEval shows a different pattern at smaller k.

  • Experimental setup: Power sampling, GRPO, and the base model are compared on MATH500, GPQA, and HumanEval using Pass@k.The plots use α = 4.0 for MATH500 and GPQA and α = 1.67 for HumanEval.
  • Results: For k > 1, power sampling has near-universally better Pass@k performance than GRPO and the base model across in-domain and out-of-domain settings.At large k, it matches or exceeds the base model upper bound.
  • Benchmark differences: MATH500 and GPQA show that GRPO has significantly lower Pass@k performance and diversity even for smaller k.These benchmarks are described as having a singular answer.
  • Benchmark differences: On HumanEval, GRPO outperforms the base model in Pass@k until k = 16.The paper speculates that multiple accepted code solutions may delay the emergence of reduced diversity.

A.3 More Qualitative Examples

The paper includes qualitative comparisons on MATH500 and HumanEval that illustrate cases where its sampling method and GRPO produce different outcomes.

  • MATH500: Table 3 presents a MATH500 example where the proposed method and GRPO are both correct.
  • HumanEval: Table 4 provides a HumanEval comparison using Phi-3.5-mini-instruct.
  • MATH500: Table 5 shows a MATH500 case where GRPO is incorrect while the sampling algorithm succeeds using a distinct method.
Loading 2510.14901v1…