PaperlayerView exact revision on arXiv

Source-linked AI summary

A Critical Look at Power Sampling when Test Time Sharpening LLMs

Abiel Malkani Talwar, Christopher Nemeth, Henry Moss

arXiv:2610.13653v1stat.MLcs.LG

TL;DR

The paper asks whether RLVR-like reasoning gains can come from sharpening the base model’s own distribution, and whether faithfully targeting the power distribution accounts for those gains. It analyzes approximate samplers and tests the target directly through fine-tuning, finding that sampler gains can be recreated by selection while faithful targeting yields limited, model-dependent gains.

  • Problem

    Prior work suggests RLVR often concentrates probability on reasoning paths the base model already produces, raising whether such gains can be recovered by sampling those high-probability regions without reward signals.

    Source: §1 · p. 1

  • Method

    The paper analyzes MCMC and SMC power samplers and fine-tunes the base model with POWER-RLOO to target πα while measuring target fidelity directly.

    Source: §1 · p. 1, §4 · p. 7

  • Results

    Faithful approximation of πα yields only modest, model-dependent gains, while Best-of-N matches or exceeds Power-SMC on reported comparisons.

    Source: §5.2 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    The results suggest power sampling’s value lies more in selecting sequences the base model already deems likely than in faithfully targeting πα.

    Source: §6 · p. 9

    Direct simulation from πα is infeasible because its sequence-wide normalizer is analytically intractable and the distribution lacks an autoregressive structure.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution $\pi_\alpha \propto \pi_{\text{base}}^\alpha$ via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to $\pi_\alpha$ under realistic compute budgets, and that their gains are largely recreated via simple Best-of-$N$ with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating $\pi_\alpha$, raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match $\pi_\alpha$, sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models and sampling tasks. Together, our results suggest that the value of power sampling lies not in faithfully targeting $\pi_\alpha$, but in the simpler goal of extracting sequences the base model already deems likely.

1 INTRODUCTION

The paper asks whether reasoning gains attributed to RLVR can instead come from selecting high-probability outputs of the base model. It tests whether faithfully targeting the power distribution explains those gains, finding that sampler gains can arise without convergence and that faithful targeting yields gains only under limited conditions.

1 INTRODUCTION

  • RLVR gains may amplify reasoning paths the base model already produces, motivating tests of whether clever sampling can recover them without verifier or reward signals.Prior work cited in the passage associates RLVR gains with reduced diversity and probability shifts toward sequences already ranked highly by the base model.Source: §1 · p. 1
  • Recent SMC schemes fail to converge to πα at practical budgets and instead behave as likelihood-guided search, largely insensitive to α.Figure 1 illustrates this contrast: Power-SMC collapses into a narrow likelihood band, while POWER-RLOO shifts toward higher likelihood as α grows.Source: §1 · p. 1, Figure 1 · p. 2
  • A simple Best-of-N variant that resamples high-likelihood partial sequences largely recreates SMC gains, suggesting selection rather than approximation of πα explains them.The supplied passage characterizes the variant as periodically resampling the highest-likelihood partial sequences.Source: §1 · p. 2
  • POWER-RLOO directly targets πα through fine-tuning, exploiting its connection to low-temperature sampling so the adapter learns only a non-local lookahead correction.This method enables a direct test of whether faithful approximation drives reasoning gains.Source: §1 · pp. 1–2
  • Faithfully targeting πα yields accuracy gains beyond low-temperature decoding only when the base model is already reasonably well-aligned with the task.The authors report that this holds despite POWER-RLOO approximating the target more effectively than previous work.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The background contrasts local next-token sharpening with power sampling over whole sequences, then introduces fine-tuning as an alternative way to target a distribution. It also situates the approach among work on self-improvement and on the role of reward quality.

2 BACKGROUND AND RELATED WORK

  • An LLM generates a finite token sequence by sampling each next token conditional on the prompt and previously generated tokens, stopping at EOS or a maximum length.The sequence probability follows from the chain rule.Source: §2 · p. 2

2.1 SHARPENING LLMS AT TEST-TIME

  • Low-temperature sampling sharpens each next-token conditional but cannot account for how a token choice affects the likelihood of later tokens.At τ → 0 it recovers greedy decoding; at τ = 1 it recovers the original conditional.Source: §2.1 · p. 3
  • Power sampling instead raises sequence-level probabilities to a power, but its sequence-wide normalizer is intractable and removes the autoregressive structure needed for direct simulation.The cited work proposes approximate MCMC and SMC samplers, which the paper says fail to sample the target accurately under realistic compute budgets.Source: §2.1 · p. 3

2.2 FINE-TUNING LLMS

  • RL fine-tuning shifts policy probability toward high-reward samples, while a KL regularizer controls divergence from the base model.The passage describes verifiable rewards as automatically checked against a ground-truth answer or test suite.Source: §2.2 · p. 3
  • Prior self-improvement work uses intrinsic model signals instead of external rewards, while other cited studies report gains even with weakly correlated rewards or a single training example.The passage also describes internal-feedback methods that use model confidence and match RLVR performance in-domain.Source: §2.2 · pp. 3–4

3 THE PITFALLS OF EXISTING SHARPENING SCHEMES

The inference-time samplers considered here do not reliably realize the power distribution at practical compute budgets; their behavior instead resembles likelihood-guided selection. Their failure modes include MCMC stagnation, rollout estimates dominated by a best completion, and collapsing particle diversity in Power-SMC.

Overall finding

  • At practical budgets, MCMC, rollout estimation, and Power-SMC collapse to keeping high-likelihood candidates, so their gains cannot be attributed to the power distribution.The samplers are largely insensitive to α.Source: §3 · p. 4

MCMC

  • As α grows, MCMC acceptance becomes dominated by likelihood and its chains stagnate in high-density regions rather than exploring.The comparison with a greedy hill climber supports the interpretation that the reasoning benefit comes from mode-seeking, not convergence to πα.Source: §3 · p. 4, Figure 2 · p. 5

Rollout estimation

  • For α > 1, rollout estimation is effectively controlled by the single best rollout, making the selected block approximately the one with the highest-likelihood best rollout.The completion likelihood terms can vary by hundreds of nats, so their average is dominated by its largest term.Source: §3 · pp. 4–5

Power-SMC

  • At α = 4 with N = 64, ESS falls from 64 to ≈8 within 64 tokens, and the final Power-SMC population has only 6 distinct prefixes.Repeated resampling leaves a few near-identical sequences and a narrow, α-insensitive log-likelihood band.Source: §3 · p. 5, Figure 3 · p. 6

Best-of-N with Resampling

  • Best-of-N with Resampling replaces weighted resampling with deterministic selection by cumulative base-model log-likelihood, whereas Power-SMC ranks prefixes by cumulative Rényi entropy.The baseline samples N low-temperature sequences in parallel, periodically retaining and duplicating the top k prefixes.Source: §3 · pp. 5–6

4 POWER-RLOO

Power-RLOO fine-tunes a model directly toward πα using a KL-regularized RL objective with log-likelihood rewards, avoiding SMC approximations. Its token-level estimator and warm start support direct target tracking and analysis of how α affects the learned distribution and sample quality.

Objective

  • Power-RLOO directly fine-tunes an LLM toward πα, which is the optimum of a KL-regularized RL objective with log-likelihood rewards.The method is intended to test whether faithfully approximating πα improves reasoning without relying on SMC approximations.Source: §4 · p. 6
  • Maximizing the objective is equivalent to minimizing reverse KL divergence to πα; neither β nor the intractable Zα enters the optimization.The objective uses a log-likelihood reward, and log Zα(z) does not depend on θ.Source: §4 · p. 6

RLOO Training Objective

  • Factorizing both log-probabilities yields a token-level reward structure, and a REINFORCE reward-to-go estimator incorporates a leave-one-out baseline.The derivation uses variance-reduction techniques.Source: §4 · p. 6

Efficient fine-tuning by warm start

  • Sampling and scoring at temperature 1/α makes the adapter learn the non-local lookahead twist; the warm start begins closer to πα and requires substantially less training.At initialization, the adapter contribution is zero and the model matches the locally tempered term.Source: §4 · pp. 6–7

Benefits

  • Because πθ is explicit, its likelihoods allow direct tracking of divergence from πα and sensitivity analysis of α on the learned distribution and sample quality.The approach also avoids the block proposals, resampling, and cache management required by MCMC- and SMC-based approaches.Source: §4 · p. 7

5 EXPERIMENTS

The experiments compare power-distribution sharpening with simpler selection methods across three models and two benchmarks. POWER-RLOO’s results depend on model and task, while Best-of-N often matches or exceeds Power-SMC; faithful sharpening can also cause length degeneracy.

5.1 SETUP

  • The evaluation covers Qwen2.5-7B Base and Instruct, Phi-3.5-mini-Instruct, MATH500, and GPQA Diamond.MATH500 contains 500 competition-level mathematics problems; GPQA Diamond contains graduate-level biology, physics, and chemistry questions.Source: §5.1 · p. 7
  • Best-of-N uses 16 low-temperature samples and selects by total or mean log-probability; the resampling baseline retains and duplicates the top four after each 64-token block.The low-temperature sampling setting is τ = 0.25.Source: §5.1 · p. 7

5.2 RESULTS AND ANALYSIS

  • POWER-RLOO is strongest on Qwen2.5-7B-Instruct (78.4% MATH500; 41.0% GPQA), while Best-of-N-mean exceeds Power-SMC on GPQA for every model.On MATH500, Best-of-N-mean matches Power-SMC for Qwen2.5-7B-Instruct (77.0% vs. 76.9%).Source: Table 1 · p. 8, §5.2 · p. 8
  • POWER-RLOO trails Power-SMC on MATH500 for Phi-3.5-mini-Instruct (43.6% vs. 46.6%), while its strongest results occur with the better-aligned Qwen2.5-7B-Instruct.The authors report that faithful sharpening benefits a base model only when it is reasonably aligned with the task.Source: §5.2 · p. 8
  • POWER-RLOO and Power-SMC gains over low-temperature sampling vary by task, with a sizeable Power-SMC increase on Qwen-Base MATH500 (63.1% to 70.6%).On GPQA, the gap between low-temperature decoding and the more sophisticated methods closes.Source: §5.2 · p. 8
  • POWER-RLOO collapses to overly short completions on Qwen-Base, whereas SMC approximations and Best-of-N avoid this length-degeneracy pathology.The authors attribute reasonable output length under sharpening to a strong prior over length, such as that instilled by chat-template tuning in instruct models.Source: §5.2 · pp. 8–9
  • On Qwen2.5-7B-Instruct MATH500, mild sharpening at α = 1.5 improves pass@k at large k, while pass@k decreases as α increases.The authors suggest mild sharpening surfaces reasoning paths present in the base model but rarely sampled at α = 1.Source: Figure 4 · p. 9, §5.2 · p. 9

6 CONCLUSION

The paper finds that MCMC and SMC samplers fail to converge to the power distribution and behave as likelihood-guided search. Directly targeting that distribution with POWER-RLOO enables a test of whether faithful approximation itself improves reasoning, but the reported gains are modest and model-dependent.

6 CONCLUSION

  • The paper reports that MCMC and SMC schemes fail to converge to the power distribution and instead behave as likelihood-guided search.POWER-RLOO directly targets the distribution to investigate whether faithful targeting drives reasoning gains.Source: §6 · p. 9
  • Faithfully approximating the power distribution yields only modest, model-dependent gains and can produce length degeneracy; Best-of-N matches or exceeds sharpening without targeting that distribution.The authors conclude that improvements stem from selection rather than approximation quality.Source: §6 · p. 9

A DERIVING THE TRAINING OBJECTIVE

The training loss is constructed to have the power distribution as its unique minimizer. Its gradient estimator uses conditional independence to remove mean-zero terms and a leave-one-out baseline to reduce variance.

B CONNECTION TO R´ENYI ENTROPY

The per-token training cost connects the power objective to Rényi entropy. This makes the training signal depend on the uncertainty of future continuations, not the sampled token itself.

C LOW TEMPERATURE SAMPLING IN POWER-RLOO

The section examines low-temperature sampling during Power-RLOO training. It reports faster, more stable convergence, with an initial KL advantage and a warm start that carries across increases in α.

D IMPLEMENTATION DETAILS

The implementation uses LoRA adapters on three base models, grouped rollouts, and staged optimization. Training dynamics are tracked with KL estimates, which the authors report converge faster and more stably under low-temperature sampling.

  • The experiments use Qwen2.5-7B-Instruct, Qwen2.5-7B base, and Phi-3.5-mini-instruct with LoRA adapters of rank 16 and α = 32.The adapters target attention and MLP projections; the reference policy is the same model with the adapter disabled.Source: D IMPLEMENTATION DETAILS · pp. 14–15
  • Each optimizer step accumulates 320 rollouts across 10 questions, with prompts capped at 768 tokens and completions at 1564 tokens.Source: D IMPLEMENTATION DETAILS · p. 15
  • LoRA parameters use a 1e−4 learning rate with per-stage cosine decay, optimizer resets, and one on-policy iteration without PPO-style replay or clipping.Source: D IMPLEMENTATION DETAILS · p. 15
  • All runs use seed 0 and a single adapter across the α schedule; Figure 5 tracks KL divergence to the target and reports faster, more stable convergence with low-temperature sampling.Source: D IMPLEMENTATION DETAILS · p. 15, Figure 5 · p. 15

Related papers

Loading 2610.13653v1…