Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution

• OPPD replaces that repeated sampling with a trained model that generates one answer and obtains 94% of the sixteen-candidate sampler’s gain on MATH500.Figure 1 contrasts inference-time sampling and one-generation student output.Source: Figure 1 · p. 2, §1 · p. 2
• With sixteen candidates, power sampling raises Qwen2.5-Math-7B’s MATH500 score from 66.8% for one answer to 79.2%.The sampler must run again for every query.Source: §1 · p. 2
• OPPD trains the student on its own sampler-generated answers, weighted by a frozen teacher’s power distribution, using weighted maximum likelihood.The teacher’s probabilities also set the sampler’s importance weights; a token-level anchor keeps the student near the teacher at temperature 1/α.Source: §1 · p. 2
• The paper reports that OPPD scores higher than same-checkpoint, same-budget GRPO on MATH500, GSM8K, and AIME, without using reference answers.The methods are complementary, and applying OPPD after GRPO further improves the model.Source: §1 · p. 2
• The absorbed sharpening exponent decreases as the token-level anchor coefficient λ increases and rises mainly on answers the student writes itself.A quantity measured before training predicts the best λ for each model.Source: §1 · p. 3

2.1 SHARPENING COMPLETE ANSWERS VERSUS SHARPENING TOKENS

• At α = 2, the power distribution assigns 0.213 to answers beginning with the uncertain token, versus 0.308 under temperature 1/2.The former renormalizes over complete answers; temperature scaling renormalizes at each token.Source: §2.1 · p. 3

2.2 DRAWING SAMPLES FROM THE POWER DISTRIBUTION

• Sequential Monte Carlo extends N candidate answers in parallel and uses importance weights to correct for a proposal distribution that depends only on the prompt and prefix.The power distribution’s normalizing constant sums over all possible answers and cannot be computed.Source: §2.2 · pp. 3–4
• When effective sample size falls below a threshold, the sampler discards low-weight candidates, duplicates high-weight ones, and resets weights equally.ESS approximates the number of candidates with non-negligible weight; during sampling it also estimates divergence between the power distribution and proposal.Source: §2.2 · p. 3
• Sampling from the teacher at temperature 1/α is the unique prefix-dependent proposal that minimizes the conditional variance of incremental importance weights.Under this proposal, the incremental weight equals the normalizing constant for that token position.Source: §2.2 · p. 3

3 METHOD

• OPPD minimizes a divergence from the teacher’s power distribution, whose unavailable normalizer prevents exact evaluation and sampling.The method rewrites the gradient and supplies approximate samples, a proposal, and a loss.Source: §3 · p. 4

3.1 WHAT THE GRADIENT REQUIRES

• For any β > 0, the divergence gradient is an expectation of the student’s log-likelihood gradient under a reweighted target distribution.A single estimator serves every β because the normalizer independent of student parameters drops out of the gradient.Source: §3.1 · p. 4

3.2 WHERE THE ANSWERS COME FROM

• Using the student as proposal lets the sampler’s teacher and proposal probabilities, plus the student’s generation probabilities, supply the training weights without extra model evaluations.A teacher forward pass per token block supplies both importance weights for resampling and training weights for the update.Source: §3.2 · p. 4

3.3 CHOOSING THE PROPOSAL TEMPERATURE

• The student-at-temperature-1/α proposal avoids the student-draw-dependent noise term in the importance weight.With the student as proposal, that term can cause the ESS ratio to remain at 1/N from the first block.Source: §3.3 · pp. 4–5

3.4 THE LOSS

• The sequence-level loss targets the power distribution, while a token-level forward-KL anchor trades that objective against protection from concentrating probability on one answer.The anchor moves the attainable optimum toward the teacher’s next-token distribution at temperature 1/α; λ controls the trade-off.Source: §3.4 · p. 5

3.5 IMPLEMENTATION

• The implementation generates 16 candidates in 64-token blocks and renormalizes recorded training weights after each resampling step.It also uses separate accumulators for importance and training weights, and adjusts weights for candidates that finish early.Source: §3.5 · p. 5

• The absorbed exponent αeff is the median context-level slope from a weighted least-squares fit of student log-probabilities on initialization log-probabilities.The fit uses the 200 tokens most likely under the initialized model, weighted by its probabilities.Source: §4 · p. 6
• A slope of 1 indicates no sharpening, while a slope of α corresponds to the initialized model at temperature 1/α.In self-distillation, the teacher control has αeff = 1.000 exactly.Source: §4 · p. 6
• The exponent fit is reported only when initialization entropy is high enough to distinguish models; elsewhere the paper reports a direct entropy measurement.When the initialized model is nearly deterministic, noise dominates the slope estimate.Source: §4 · p. 6

5.1 ACCURACY

• At one generation, OPPD beats published 64-candidate power sampling by 2.4 points on MATH500 and 3.5 on GSM8K.At temperature 1, it gains 23.0 points on MATH500 and 27.3 on GSM8K over the untrained model; it recovers 94% of the untrained model’s 16-candidate MATH500 gain.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 COMPARISON WITH TRAINING ON VERIFIED REWARDS

• OPPD scores 3.75 points higher on MATH500, 4.00 on GSM8K, and 5.42 on AIME than GRPO trained from the same checkpoint and budget.Both runs use 300 optimizer steps and generate 9,600 answers; the authors caution that this small comparison bounds neither method at a larger budget.Source: §5.2 · p. 7
• After GRPO, OPPD adds 9.25 points on MATH500 and 8.38 on GSM8K, while combined AIME pass@1 reaches 12.92% versus 10.83% with either method alone.The starting GRPO checkpoint moved from 50.25 to 64.00 on MATH500 and from 59.25 to 71.00 on GSM8K.Source: §5.2 · p. 7

5.3 EVIDENCE THAT THE STUDENT MOVES TOWARD THE POWER DISTRIBUTION

• The implied divergence falls 21.6% on Qwen2.5-Math-7B and 26.6%, 24.7%, and 12.6% on Qwen3-8B-Base at N=4, 16, and 64.Along answers from the λ=0 model, the untrained model’s token entropy is 0.32 times its entropy along its own answers, versus 0.47 for ordinary on-policy distillation.Source: §5.3 · p. 7
• The absorbed exponent reaches 2.02 with the sequence-level term alone versus 1.14 for ordinary distillation, and OPPD estimates are higher on student-written contexts.For OPPD, the estimates are 1.207 on initialization contexts and 2.019 on the model’s own answers; Figure 2 shows the exponent decreases as λ increases.Source: §5.3 · pp. 7–8, Figure 2 · p. 8
• The untrained model’s MATH500 gain from lowering temperature predicts the best λ: 19.9 points for Qwen2.5-Math-7B, which does best at λ=0, versus 0.6 for Llama-3.1-8B-Instruct, which does best at λ=1.Llama loses 4.75 MATH500 points without the anchor; Qwen3-8B-Base, with a 13.0-point gain, is trained at λ=0.Source: §5.3 · p. 8

5.4 OTHER MODELS AND SCALE

• OPPD improves across model families and sizes, including deepseek-math-7b-rl, where temperature reduction adds 0.00 MATH500 points but OPPD raises the score from 47.12 to 51.50.On Qwen3-14B-Base, OPPD reaches 84.00 on MATH500, 95.88 on GSM8K, and 16.25 on AIME, above the untrained model’s best at either temperature.Source: §5.4 · p. 8

5.5 CODE GENERATION

• Trained only on mathematics questions, OPPD raises HumanEval pass@1 by 3.51 to 5.34 points; 95% intervals exclude zero except for Qwen2.5-Math-7B at λ=0.The evaluation covers 164 problems with four samples each.Source: §5.5 · p. 8, Table 2 · p. 9

6 RELATED WORK

• Karan & Du (2026) use Metropolis–Hastings for power sampling, while Azizi et al. (2026) provide the batch-parallel sampler used by OPPD.Tomihari & Sato (2026) derive power self-distillation on sampler-produced answers, reaching accuracy at or above power sampling with lower inference cost.Source: §6 · p. 9
• Unlike power self-distillation on stored sampler outputs, OPPD uses the student as the sampler’s proposal and optimizes the Rényi family objective.One teacher pass per block serves both sampling and the update.Source: §6 · p. 9
• Verified-reward methods such as GRPO require answer correctness, whereas OPPD needs only questions and can also be applied after such training.Other confidence-based approaches include entropy minimization, self-certainty, negative token entropy, and majority-vote rewards.Source: §6 · p. 9

B.1 RUNNING THE SAMPLER ON THE TRAINED MODEL

• One generation recovered 11.6 of the 12.4 points gained by sixteen candidates (94%); sampling the trained model added only 1.6 points more.The gain from candidates fell from 12.4 to 5.2 to 1.6 points across the untrained model, λ = 1 and λ = 0.Source: §B.1 · p. 12

B.3 DIVERGENCE AND ENTROPY MEASUREMENTS

• Training reduced the measured divergence to the power distribution by 21.6%; along OPPD’s own answers, the untrained model’s token-entropy ratio was 0.32, versus 0.47 for ordinary distillation.The divergence reduction was larger at λ = 0 than at λ = 1.Source: §B.3 · p. 13, Table 4 · p. 13

B.5 REPLICATION ON TWO FURTHER MODELS

• OPPD improved on the untrained model and ordinary on-policy distillation across three models and both benchmarks; on Qwen3-8B-Base, gains were 12.75 and 17.13 points.For Llama-3.1-8B-Instruct, it gained 3.25 and 3.13 points over the untrained model, and 2.75 on MATH500 over ordinary on-policy distillation.Source: Figure 4 · p. 14, §B.5 · p. 14

B.4 THE COEFFICIENT λ CONTROLS THE ABSORBED EXPONENT

• The absorbed exponent rose to 2.02 without the anchor term, versus 1.14 for ordinary on-policy distillation, and increased mainly on the student’s own answers.The untrained model’s temperature response predicts whether removing the anchor helps: models with little gain from lower temperature can lose accuracy and best-of-4.Source: §B.4 · p. 14, §B.8 · p. 16

B.6 SCALING TO A LARGER MODEL

• At temperature 1, OPPD on Qwen3-14B-Base reached 84.00 on MATH500, 95.88 on GSM8K and 16.25 on AIME, above the untrained model’s best temperature on every benchmark.It also exceeded the untrained model’s best-of-4 on MATH500 and AIME.Source: §B.6 · p. 15, Table 5 · p. 15

B.7 A THIRD MODEL FAMILY

• On deepseek-math-7b-rl, lowering temperature gained 0.00 points on MATH500, while OPPD gained 4.38 points on the same benchmark.At temperature 1, the MATH500 increase had a 95% interval of [+0.88, +7.88].Source: Table 6 · p. 15, §B.7 · p. 16

B.9 APPLYING OPPD AFTER TRAINING WITH VERIFIED REWARDS

• After verified-reward training, OPPD added 9.25 points on MATH500 and 8.38 on GSM8K, showing the two procedures can be combined.The verified-reward checkpoint rose from 50.25 to 64.00 on MATH500 and from 59.25 to 71.00 on GSM8K before OPPD.Source: §B.9 · p. 16, Table 7 · p. 16

B.10 EQUAL-BUDGET COMPARISON WITH GRPO

• OPPD scored 3.75 points higher on MATH500, 4.00 on GSM8K and 5.42 on AIME than equal-budget GRPO, though each margin was only one to two standard errors.Figure 6 compares pass@1, best-of-4 and untrained pass@1 for both methods.Source: §B.10 · p. 17, Figure 6 · p. 17

B.11 RESULTS ON AIME

• On Qwen3-8B-Base, OPPD raises single-attempt AIME accuracy from 7.08% to 14.17% and best-of-4 from 16.67 to 26.67.Training prompts are GSM8K questions, while AIME problems are competition problems the model solves only occasionally.Source: §B.11 · pp. 18–19

B.12 TRAINING ON A DIFFERENT PROMPT SOURCE

• Training on 4,000 OpenR1-Math-220k problem statements gives MATH500 51.13 versus 52.50 and GSM8K 85.88 versus 85.38 for GSM8K-prompt training, within one standard error.Evaluation prompts were removed from the training corpus; the method requires questions, not questions matched to evaluation.Source: §B.12 · p. 19

B.13 TRAINING AGAINST THE STUDENT ITSELF OR A LARGER TEACHER

• On Qwen2.5-Math-1.5B, self-distillation improves by 10.13 and 15.37 points on MATH500 and GSM8K, versus 8.50 and 9.75 with the larger teacher.The self-distillation advantage is 5.62 points on GSM8K (1.5 standard errors) and 1.63 on MATH500 (within noise); the comparison used λ = 1.Source: §B.13 · p. 19

B.14 WHERE THE SAMPLER’S GAIN COMES FROM

• Across eleven measurements, normalized importance weight on correct candidates tracked the fraction of correct candidates within 0.5 points, and weighted draw, highest weight, and majority vote gave the same result.The authors conclude that sampler gains come from discarding candidates during generation, not reranking completed answers.Source: §B.14 · p. 20

C PROOFS

• For a fixed prefix, the variance-minimizing proposal is proportional to p(v)^α, making incremental importance weights constant and conditional variance zero.The proof states this minimizer is unique.Source: C PROOFS · p. 20

D TABLES CORRESPONDING TO THE FIGURES

• Figures 3, 2, 4, and 5 present the measurements graphically, with exact values supplied in the corresponding tables.Source: D TABLES CORRESPONDING TO THE FIGURES · p. 20

E NOTATION AND DEFINITIONS

• Pass@1 estimates single-answer accuracy, while best-of-4 counts a problem solved if any of four answers is correct; self-distillation initializes teacher and student from the same model.The notation section also distinguishes token-level temperature sharpening from sequence-level power sharpening.Source: E NOTATION AND DEFINITIONS · pp. 21–22

B.11 RESULTS ON AIME

• OPPD and GRPO both reach 10.83% single-attempt AIME accuracy; combined training reaches 12.92% pass@1 and 31.67% best-of-4.The untrained model scores 4.17% pass@1 and 13.33% best-of-4; OPPD uses questions rather than reference answers.Source: §B.11 · p. 18, Table 12 · p. 22

F EXTENDED BACKGROUND ON THE SAMPLER

• Sequential Monte Carlo extends candidate answers token by token, updates importance weights incrementally, and resamples when effective sample size falls below κN.The paper uses κ = 0.5 and systematic resampling; unbiased resampling affects variance and candidate diversity, not the approximated distribution.Source: F EXTENDED BACKGROUND ON THE SAMPLER · pp. 22–23

G WHY A WEIGHTED TOKEN-LEVEL KL CANNOT REACH THE POWER

• A weighted token-level KL objective cannot reach the sequence-level power distribution: its weights change context contributions, not the matched next-token distribution.The attainable optimum is the teacher at temperature 1/α; the missing continuation-dependent twist cannot be supplied by an outside weight.Source: G WHY A WEIGHTED TOKEN-LEVEL KL CANNOT… · p. 23

H.1 WHAT WE ARE MINIMIZING

• The derivation minimizes Rényi divergence between the power distribution and student, then rewrites its gradient as an expectation under a weighted answer distribution.The appendix says it repeats Section 3’s derivation for line-by-line checking and contains no new results.Source: §H.1 · p. 23, H THE DERIVATION IN FULL · p. 23, §H.2 · pp. 23–24

H.3 WHY THE WEIGHTS CAN BE TREATED AS CONSTANTS

• Weights computed from the student can be held fixed during differentiation because the derived gradient is already complete, including their dependence on the student parameters.The appendix reports numerical confirmation against brute-force enumeration.Source: §H.3 · p. 24

H.4 FROM AN EXPECTATION TO A WEIGHTED AVERAGE

• The intractable expectation is estimated from proposal-sampled answers using self-normalized importance weights; the estimator is consistent with bias of order 1/N.A shared constant cancels when weights are normalized to sum to one.Source: §H.4 · pp. 24–25

H.5 COMPUTING THE WEIGHTS TOKEN BY TOKEN

• The answer-weight ratio factors token by token, and logarithms turn its product into a sum that the sampler and loss compute from the same three token-level quantities.Logarithms avoid floating-point underflow on long answers; the scalar loss reproduces the estimated divergence gradient, up to estimation error.Source: §H.5 · p. 25, §H.6 · p. 25

I THE SIMPLEST CASE OF THE LOSS

• At β = 1 and self-distillation initialization, the loss is maximum likelihood on the student’s own answers weighted by its confidence along each answer.As training proceeds, weights also favor answers the frozen teacher rates more highly than the current student does.Source: I THE SIMPLEST CASE OF THE LOSS · pp. 25–26

J FULL EXPERIMENTAL CONFIGURATION

• Experiments use six model configurations, including Qwen2.5-Math-7B self-distillation and a Qwen2.5-Math-1.5B student for the λ sweep.The replication models include Qwen3-8B-Base, Qwen3-14B-Base, Llama-3.1-8B-Instruct, and deepseek-math-7b-rl.Source: J FULL EXPERIMENTAL CONFIGURATION · p. 26
• The sampler uses α = 4, 16 candidates, 64-token blocks, β = 0.7, and a student proposal at temperature 1/α, with resampling below ESS = N/2.Teacher distributions are truncated to 64 tokens; retained probability exceeds 0.98 except on the first block of unusually long prompts.Source: J FULL EXPERIMENTAL CONFIGURATION · p. 26
• Training runs for 300 steps with two prompts per step and model-specific learning rates; training answers are limited to 1,024 tokens, or 2,048 for deepseek-math-7b-rl.Optimization uses 10 warmup steps, no weight decay, gradient clipping at 1.0, and 8-bit AdamW.Source: J FULL EXPERIMENTAL CONFIGURATION · p. 26
• Evaluation uses four samples on 200 GSM8K and MATH500 problems with the reference power-sampling prompt builder, parser, and grader.Training and evaluation share each model’s prompt format; GSM8K questions are used except for the OpenR1-Math-220k run, which filters evaluation matches.Source: J FULL EXPERIMENTAL CONFIGURATION · p. 26

K VALIDATION OF THE EVALUATION CODE

• Differences are within 3 points and run in both directions, consistent with sampling variation on a 200-problem subset rather than a systematic difference in protocol. All comparisons in the paper use this evaluation.Source: K VALIDATION OF THE EVALUATION CODE · p. 27

L COMPLETE ACCURACY TABLES

• The complete tables report Qwen3-8B-Base at temperature 1 and Llama-3.1-8B-Instruct at its default temperature 0.6 with nucleus sampling p = 0.9.The table set also gives both decoding temperatures.Source: L COMPLETE ACCURACY TABLES · p. 27, Table 14 · p. 27, Table 15 · p. 27
• On Llama-3.1-8B-Instruct, lowering temperature gains only +0.6 and +1.25 points, so improvements at its default decoding configuration are attributed to training rather than decoding.Source: Table 15 · p. 27

M COMPLETE DIVERGENCE AND ABSORBED-EXPONENT MEASUREMENTS

• The reported divergence is comparable only within a row, because increasing candidate count expands distribution coverage and measured weight spread.Source: M COMPLETE DIVERGENCE AND… · p. 27
• The trained model is a more efficient proposal than its initialization, so fewer candidates are wasted when sampling after training.Source: M COMPLETE DIVERGENCE AND… · p. 27
• Base entropies across rows are not directly comparable because each row uses contexts written by a different model; only the closed fraction of the base-to-target entropy gap is comparable.Source: M COMPLETE DIVERGENCE AND… · p. 28
• The exponent estimator recovers known synthetic exponents to four decimal places, returns 1.998 with added noise, and −0.008 for an unrelated model.Source: M COMPLETE DIVERGENCE AND… · p. 28
• For 7B and 8B models, low base entropy leaves αeff estimates between 1.11 and 1.25 unresolved, so the paper reports them only for higher-entropy contexts.The Qwen2.5-Math-1.5B sweep has base entropy of 0.22 to 0.30 nats; larger models use Table 4's direct entropy measurement.Source: M COMPLETE DIVERGENCE AND… · p. 28

N WHY THE STUDENT AT TEMPERATURE 1 FAILS AS A PROPOSAL

• With the student at temperature 1, ESS/N stays at 1/N and the sampler effectively selects one random candidate; at temperature 1/α, training ESS/N is 0.39–0.64.When student and teacher are identical, measured ESS/N is 0.998.Source: N WHY THE STUDENT AT TEMPERATURE 1… · p. 28

O IMPLEMENTATION CORRECTNESS

• Fifty automated checks exercise the same code as training, including theorem and fixed-weight agreement with brute force to 10^-10 and generation-versus-recomputed log-probability agreement to 4.8 × 10^-7.Source: O IMPLEMENTATION CORRECTNESS · p. 28
• On a small model, sampler-to-target total variation decreases with candidate count (0.166, 0.095, 0.065 at N = 4, 16, 64); loss and gradient also match reference values to 10^-6 and 3 × 10^-8.Source: O IMPLEMENTATION CORRECTNESS · p. 29

P CORRECTING THE WEIGHT ADVANTAGE OF STOPPING EARLY

• In the identical student-teacher control, correcting finished-candidate weights shifts weighted-minus-unweighted length from −25.9 tokens to +0.13 and ESS/N from 0.084 to 0.997.The correction charges finished candidates the mean per-token importance-weight increment of unfinished candidates.Source: P CORRECTING THE WEIGHT ADVANTAGE OF… · p. 29

Q RESAMPLING AND THE TRAINING WEIGHTS

• Resampling can amplify stored training weights on earlier blocks at each step, so the method renormalizes those weights across candidates after every resampling.The sampler's importance-weight accumulator controls resampling; the separate loss accumulator supplies training weights and resets at block boundaries.Source: Q RESAMPLING AND THE TRAINING WEIGHTS · p. 29

R SELECTING AMONG FINISHED ANSWERS

• Table 16 compares the percentage of candidates that are correct with their total normalized importance weight; useful ranking would put the latter above the former.Source: Table 16 · p. 30