LightReasoner: Can Small Language Models Teach Large Language Models Reasoning?

1 Introduction

• Rejection-sampled SFT requires generating and filtering candidate solutions against ground-truth answers, then optimizes every token despite differences in learning value.The introduction presents this resource-intensive procedure as the motivation for more targeted reasoning strategies.Source: §1 · pp. 1–2
• The introduction asks whether weaker models can identify high-impact reasoning moments and provide useful training signals for stronger models.The motivation draws on internal feedback without external labels and evidence that learning progress concentrates on a small subset of tokens.Source: §1 · pp. 1–2
• The framework samples expert–amateur divergence to identify critical reasoning moments, constructs contrastive supervision from their next-token distributions, and fine-tunes the expert to reinforce its advantage.The authors report that the most effective contrasts stem from domain-specific expertise differences, and that step selection and contrastive supervision play critical, mutually reinforcing roles.Source: §1 · p. 2
• LightReasoner matches or surpasses SFT across five models and seven benchmarks, while reducing time cost by 90%, sampled problems by 80%, and tuned tokens by 99%.It also eliminates the need for ground-truth labels.Source: Figure 1 · p. 1, §1 · p. 2

2.1 Preliminaries

• Reasoning quality depends on token-level decisions, while existing annotation or verification signals face resource constraints; SFT here means fine-tuning on rejection-sampled model trajectories, or RFT.The method uses paired next-token distributions from expert and amateur models on the same prefix to expose decision differences.Source: §2.1 · pp. 2–3

2.2 Token Informativeness

• 60% of tokens have KL divergence in [0.0, 0.1), while 20% exceed 0.4, and divergence spikes at demanding arithmetic and logical transitions.Even when top-1 tokens agree, 10% of cases have KL divergence above 0.4; when they disagree, average KL is 1.99 versus 0.166 under agreement.Source: §2.2 · p. 3, Figure 2 · p. 3, Figure 3 · p. 3

2.3 LightReasoner Framework

• The framework has three components: divergence-based step selection, contrastive supervision that captures the expert’s advantage, and an objective that distills that signal back into the expert.Figure 4 depicts sampling distributions, selection at DKL(πE ∥ πA) > β, soft labels vC, and expert alignment.Source: §2.3 · p. 3, Figure 4 · p. 4

2.3.1 Informative Step Selection

• Informative step selection retains steps where expert–amateur KL divergence exceeds a threshold, focusing supervision on bottlenecks rather than routine agreement.The selected steps are those where expert knowledge is decisive.Source: §2.3.1 · pp. 3–4

2.3.2 Contrastive Distributional Supervision

• For selected steps, contrastive supervision constructs soft targets encoding the expert’s relative advantage rather than reducing the target to a one-hot token.Low-confidence tokens are removed from the masked vocabulary to prevent noisy tail probabilities from distorting the signal; the target is normalized into a distribution over tokens.Source: §2.3.2 · p. 4

2.3.3 Self-Distillation Training Objective

• The self-distillation objective trains the expert distribution to match the contrastive target, reinforcing probability on tokens where its advantage over the amateur is pronounced.Because the target entropy term is constant with respect to the expert distribution, the objective is equivalent to cross-entropy minimization.Source: §2.3.3 · pp. 4–5
• LightReasoner consistently improves accuracy across seven math benchmarks for baseline models.Figure 5 summarizes accuracy across the benchmarks.Source: Figure 5 · p. 5

3.1.1 Models and Data

• Math-specialist Experts are paired with generalist Qwen2.5-0.5B; GSM8K CoT prompts, LoRA tuning, and short rollouts structure supervision and reduce exposure to later cascading errors.The study uses GSM8K because its step-wise arithmetic is largely within Expert capability while allowing coherent Amateur outputs.Source: §3.1.1 · p. 5, §3.1.2 · p. 5, §3.1.3 · p. 5

3.2 Performance Improvements (RQ1)

• Across 5 models and 7 benchmarks, LightReasoner matches or surpasses SFT; Qwen2.5-Math-1.5B gains +28.1% on GSM8K and +25.1% on MATH.Despite training on GSM8K, it also improves performance on diverse benchmarks including MATH, SVAMP, and ASDiv.Source: Table 1 · p. 6, §3.2 · pp. 6–7

3.1.4 Evaluation

• Baselines, rejection-sampled SFT variants, and LightReasoner are evaluated across math benchmarks with zero-shot pass@14 as the primary metric.The experiments evaluate basic arithmetic through expert-level mathematical reasoning.Source: §3.1.4 · p. 6

3.3 Efficiency Study (RQ2)

• 90% less total time, 80% fewer sampled problems, and 99% fewer tuned tokens than SFT, while matching or surpassing its performance.Savings use 128-token rollouts, selective token tuning, and Amateur contrast instead of ground-truth verification.Source: §3.2 · p. 7, §3.3 · pp. 7–8

3.4 Domain Expertise Drives Contrast (RQ3)

• Qwen2.5-Math-1.5B gains +12.1% on average with a same-size generalist Amateur; benefits diminish as the Expert–Amateur expertise gap narrows.Pairing a math Expert with the stronger Math-1.5B-Instruct Amateur can yield negligible gains or degradation.Source: Figure 6 · p. 7, §3.4 · p. 8

3.5 Ablation Study (RQ4)

• Removing step selection yields –3.0% on GSM8K, while removing contrastive supervision lowers average performance by 9.2%.Human-curated solutions were weak, and rejection-sampled SFT gained but lagged the contrastive-supervision-only ablation.Source: §3.5 · pp. 8–9
• Removing both mechanisms drops average performance by 12.4%, exceeding the sum of their individual ablations (–9.2% and –0.2%).The authors attribute this superadditive decline to the mechanisms’ mutual dependence: selection focuses contrast, while contrast turns selected steps into learning signals.Source: §3.5 · p. 9

4 Conclusion

• Experiments across diverse benchmarks show LightReasoner outperforms traditional SFT under equal settings, reduces resource demands, and eliminates reliance on labeled ground truth.The conclusion describes it as a practical, efficient solution for building more capable reasoning models.Source: §4 · p. 9
• LightReasoner combines targeted step selection with contrastive supervision to amplify the expert’s strengths.The framework exploits behavioral divergence between expert and amateur models.Source: §4 · p. 9
• The framework is grounded in the premise that critical reasoning steps carry disproportionate learning value.Source: §4 · p. 9

Scope of evaluation

• Experiments focus primarily on mathematical reasoning, leaving generality to domains such as code reasoning for future exploration.Source: Limitations · p. 10

Pairing strategy

• The Expert–Amateur contrast relies on pairs with a balanced capability gap, and more adaptive pairing strategies may yield stronger, more stable signals.Source: Limitations · p. 10

Hyperparameter sensitivity

• Step selection and contrastive supervision add hyperparameters, including α-pruning and β-filtering, that may require careful tuning across tasks.Source: Limitations · p. 10

Model scalability

• Experiments use small and mid-scale open-weight models; extending the approach to larger proprietary models would further demonstrate scalability and practical applicability.Source: Limitations · p. 10

A.1 Contrastive Decoding

• Contrastive decoding contrasts expert and amateur log-probabilities at each decoding step to reweight the expert’s predictions.The amateur acts as a proxy for undesirable behaviors such as repetition or incoherence.Source: §A.1 · p. 12
• Reasoning-focused extensions and approaches that derive an amateur from the expert have broadened contrastive decoding beyond open-ended generation.Later work also interprets contrastive scores as approximations to an idealized target distribution.Source: §A.1 · p. 12
• Contrastive decoding requires both models at inference, increasing memory use and latency, and often needs a substantial capability and size gap within one family.Source: §A.1 · p. 12
• LightReasoner shifts expert–amateur contrast to training and distills it into persistent supervision, removing concurrent model execution at inference and relaxing strict size-disparity requirements.Source: §A.1 · p. 12

A.2 Training on Selective Tokens

• Selective-token approaches target informative reasoning tokens, but identifying them in advance remains challenging; LightReasoner instead uses expert–amateur expertise gaps to locate high-divergence steps.Those steps correspond to reasoning patterns exhibited by the expert but absent in the amateur.Source: §A.2 · pp. 12–13

A.3 Self-Distillation

• LightReasoner treats the amateur as a reference for the expert’s relative strengths, not as a teacher, and combines selective training with a contrastive objective.The approach remains largely off-policy; on-policy extensions are left for future work.Source: §A.3 · p. 13

B The Use of Large Language Models

• The authors used ChatGPT only to polish grammar, phrasing, and readability; they state that all research content is their own.Source: B The Use of Large Language Models · p. 13

C From KL Divergence to Contrast Score

• The appendix motivates using a contrast score for supervision: its log-ratio form coincides with the contrast score proposed in Li et al. (2022).The authors choose that score rather than directly maximizing KL divergence.Source: C From KL Divergence to Contrast Score · p. 13, §19 · pp. 14–15

19: // Phase 2: Contrastive Fine-tuning

• The contrast score is the central mechanism for constructing supervision signals that guide the Expert model during contrastive fine-tuning.Source: §19 · p. 15

D Connection between Selection, Contrast, and Training

• The same log-ratio unifies step selection, contrastive supervision, and the training objective into a mutually reinforcing framework.Step selection identifies informative reasoning steps; contrastive supervision captures the Expert’s margin over the Amateur, and training weights updates toward high-contrast tokens.Source: D Connection between Selection,… · p. 15
• Because Expert and Amateur share a model family, their divergence is treated as a noisy estimate of improvement that repeated sampling and filtering can average toward consistent advantageous directions.The authors note that not all differences reflect genuine improvements; filtering and repeated sampling are described as variance reduction.Source: D Connection between Selection,… · p. 15

E Relation to Reinforcement Learning

• The contrastive target plays an advantage-like role in a policy-gradient-shaped update, but LightReasoner minimizes a loss rather than maximizing expected return.The contrast score is a masked, renormalized log-ratio between Expert and Amateur distributions, and the framework’s advantage signal comes from their divergence rather than environment rewards or human feedback.Source: E Relation to Reinforcement Learning · p. 16

F Entropy Dynamics

• Entropy reduction is concentrated on high-probability actions where the Expert strongly outperforms the Amateur, while low-signal regions retain exploration capacity.When the models assign similar probabilities, the log-ratio is near zero and contributes negligibly to entropy change; the authors say this targeted reduction helps avoid premature entropy collapse.Source: F Entropy Dynamics · p. 17

G.1 Effect of Truncation Length

• Qualitative case studies show LightReasoner-trained models produce full, coherent chain-of-thought solutions, addressing concerns about shallow reasoning.Source: §G.1 · p. 17
• Performance does not change monotonically with sampling rollout length, and different tasks favor different prefix lengths.The authors conclude LightReasoner is largely insensitive to the exact rollout length and attribute gains primarily to informative-step contrast rather than length-dependent factors.Source: §G.1 · p. 17

G.2 Generalization Beyond Math

• LightReasoner improves performance across all three commonsense benchmarks despite training only on GSM8K.The authors suggest this indicates transfer of fundamental reasoning capabilities beyond the training domain.Source: §G.2 · p. 17

• The supplementary details cover LightReasoner and the competitive supervised fine-tuning baseline.Source: H Supplementary Details · pp. 17–18
• Table 7 reports performance under different sampling rollout lengths, with no monotonic trend across lengths.Source: Table 7 · p. 18
• Table 8 presents generalization results on non-math reasoning benchmarks and describes transfer beyond GSM8K training.Source: Table 8 · p. 18

H.1 Datasets

• GSM8K supplies contrastive training samples, while evaluation spans seven math benchmarks from arithmetic to advanced STEM and olympiad problems.The evaluation set comprises MATH, SVAMP, ASDiv, Minerva Math, OlympiadBench, and MMLU-STEM, alongside GSM8K.Source: §H.1 · p. 18

H.2 Baseline Models

• The setup pairs varied Qwen2.5-Math Expert models with a fixed Qwen2.5-0.5B Amateur from the same family.The listed Experts include base and instruct models at 1.5B and 7B, plus DeepSeek-R1-Distill-Qwen-1.5B.Source: §H.2 · p. 18

H.3 LightReasoner

• LightReasoner uses Expert–Amateur pairing to generate training signals, with further hyperparameter details reported for the method.Source: §H.2 · p. 18, §H.3 · p. 18

H.3.1 α-masking

• α-masking excludes tokens below a relative Expert-probability threshold; larger α favors quality, while smaller α admits more diversity at lower reliability.The authors set α = 0.2 after manual inspection, aiming to preserve high-quality candidates without near-one-hot collapse.Source: §H.3.1 · pp. 18–19

H.3.2 β-filtering

• β-filtering retains steps whose Expert–Amateur divergence exceeds a threshold; β regulates single-token versus distributed labels, with low-KL single-token labels offering little useful contrast.A sharply peaked Expert distribution yields a top-1 label, whereas a more spread distribution can retain multiple tokens.Source: §H.3.2 · p. 19
• Removing β increases single-token labels by +35.6% for Qwen2.5-Math-1.5B, +33.5% for Qwen2.5-Math-7B, and +27.4% for Qwen2.5-Math-1.5B-Instruct.The authors fix β = 0.4 based on manual inspection and leave potentially better choices for future work.Source: §H.3.2 · p. 19

H.3.3 Prompting

• A shared prompt template keeps Expert–Amateur comparisons and training KL evaluations on matching contexts, avoiding prompt-mismatch confounds.During sampling, the Amateur receives the Expert trajectory’s prefixes to form contrastive targets; fine-tuning uses those targets.Source: §H.3.3 · p. 19

H.3.4 LoRA Configuration

• LightReasoner uses a shared LoRA configuration across five backbones, with 1000 update steps and bfloat16 training on one NVIDIA H200 GPU.The authors report that representative perplexity curves converge within the 1000-step horizon.Source: §H.3.4 · p. 19, Figure 8 · p. 20, Table 9 · p. 20

H.4 Supervised Fine-tuning (SFT)

• The SFT baseline uses rejection sampling: one GSM8K trajectory per problem is retained only when its final answer matches ground truth, yielding 4000–7000 problems.This simplified baseline is described as a competitive comparison for LightReasoner.Source: §H.4.1 · p. 20, §H.4 · p. 20

H.4.2 Model Fine-tuning

• Canonical SFT trains by teacher forcing on complete correct trajectories, using cross-entropy over their gold tokens and the same LoRA configuration as LightReasoner.SFT runs use a batch size of 4 with gradient accumulation of 4; their curve lengths vary with the number of correct demonstrations, but all runs converge.Source: §H.4.2 · p. 20, Figure 9 · p. 20, Table 10 · p. 20

I Case Study

• The case studies show GSM8K problems that Qwen2.5-Math-1.5B fails but its LightReasoner-enhanced counterpart solves under zero-shot pass@1.The base model uses prompt (37), while the enhanced model uses prompt (36), which the authors report yields its best accuracy and more systematic chain-of-thought reasoning.Source: I Case Study · p. 21
• 42.5% versus 38.4% on GSM8K for Qwen2.5-Math-1.5B with prompts (37) and (36), respectively; the Instruct model reaches 84.8% with (36) versus 56.5% with (37).The paper reports each model under its best-fitting prompt.Source: I Case Study · p. 21
• On Henry’s bike-trip problem, the base trace reaches 5 miles after subtracting the first-stop distance twice, while the LightReasoner-tuned trace computes 45 − 20 = 25 miles.The section presents the tuned output as correct and the base output as incorrect.Source: I Case Study · pp. 23–24