PaperlayerView exact revision on arXiv

Source-linked AI summary

RAST: Reasoning Activation in LLMs via Small-model Transfer

Siru Ouyang, Xinyu Zhu, Zilin Xiao, Minhao Jiang, Yu Meng, Jiawei Han

arXiv:2506.15710v1cs.LGcs.AI

TL;DR

RL can improve LLM reasoning but is resource-intensive at scale, and the paper hypothesizes that it mainly shifts probabilities for a small set of reasoning-relevant tokens. RAST transfers logit adjustments from a smaller RL-tuned model to larger base models at decoding time, consistently improving reasoning across benchmarks with lower GPU memory use and sometimes matching or surpassing RL-trained counterparts.

  • Problem

    RL improves LLM reasoning but requires multiple model copies and extensive GPU workloads, motivating a less costly way to transfer its reasoning benefits to larger models.

    Source: §1 · pp. 1–2

  • Method

    RAST transfers logit-level adjustments between a smaller RL-tuned model and its base counterpart into a larger base model during decoding.

    Source: §2 · p. 2

  • Results

    RAST consistently boosts reasoning performance across settings, often approaching or surpassing more expensive RL-trained ceiling models.

    Source: §1 · p. 2, §3.2 · p. 5, §6 · p. 10

  • Takeaways & Limitations

    The findings suggest an efficient inference-time pathway for eliciting complex reasoning without retraining the larger model.

    Source: §1 · p. 2, §6 · p. 10

    When the RAST prompt differs from the small model’s training prompt, the transferred ∆R1.5B yields relatively smaller gains.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Reinforcement learning (RL) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs), as evidenced by recent successes such as OpenAI's o1 and Deepseek-R1. However, applying RL at scale remains intimidatingly resource-intensive, requiring multiple model copies and extensive GPU workloads. On the other hand, while being powerful, recent studies suggest that RL does not fundamentally endow models with new knowledge; rather, it primarily reshapes the model's output distribution to activate reasoning capabilities latent in the base model. Building on this insight, we hypothesize that the changes in output probabilities induced by RL are largely model-size invariant, opening the door to a more efficient paradigm: training a small model with RL and transferring its induced probability shifts to larger base models. To verify our hypothesis, we conduct a token-level analysis of decoding trajectories and find high alignment in RL-induced output distributions across model scales, validating our hypothesis. Motivated by this, we propose RAST, a simple yet effective method that transfers reasoning behaviors by injecting RL-induced probability adjustments from a small RL-trained model into larger models. Experiments across multiple mathematical reasoning benchmarks show that RAST substantially and consistently enhances the reasoning capabilities of base models while requiring significantly lower GPU memory than direct RL training, sometimes even yielding better performance than the RL-trained counterparts. Our findings offer new insights into the nature of RL-driven reasoning and practical strategies for scaling its benefits without incurring its full computational cost. The project page of RAST is available at https://ozyyshr.github.io/RAST/.

1 Introduction

The introduction argues that RL can activate latent reasoning by selectively shifting a small subset of token probabilities, while direct RL at scale is resource-intensive. It proposes transferring those shifts from a small RL-trained model to larger base models, reporting consistent reasoning gains with lower GPU memory requirements.

1 Introduction

  • RL requires multiple copies of same-sized models and extensive GPU workloads, making large-scale training resource-intensive.Traditional RL algorithms such as PPO also typically require multiple interdependent rollout, replay, and optimization stages.Source: §1 · p. 1
  • The paper hypothesizes that RL selectively adjusts probabilities for a small set of reasoning-behavior tokens while leaving most token probabilities largely unchanged.This builds on the view that RL elicits latent reasoning behaviors rather than imparting fundamentally new knowledge.Source: §1 · p. 2
  • In one decoding-path case, around 96.71% of tokens remained unchanged after RL, with differences concentrated on reasoning-critical behaviors.The cited examples include self-verification, branching, and backtracking.Source: §1 · p. 2, Figure 1 · p. 2
  • RAST transfers reasoning patterns from a small RL-trained model to larger base models, improving reasoning consistently with lower GPU memory and sometimes outperforming RL-trained counterparts.The introduction also reports increased search-space diversity, exemplified by superior pass@k performance.Source: §1 · p. 2

2 Methodology

The methodology tests whether large base models already align with RL-trained decoding paths, then introduces RAST to transfer reasoning-oriented logit adjustments from a smaller RL-tuned model at decoding time.

2 Methodology

  • RAST aims to let a large base model emulate a smaller RL-tuned reasoner without expensive RL training on the large model.The paper distinguishes its reasoning-focused use of logit steering from prior methods directed at task-specific control, style, or toxicity.Source: §2 · p. 2, §2.2 · p. 3

2.1 Preliminary Study

  • The preliminary study feeds an RL-trained model’s decoding path token by token to the base model and checks whether its next-token prediction matches the RL path.The comparison uses the RL-generated prefix regardless of tokens previously generated by the base model, enabling measurement of path recovery.Source: §2.1 · p. 3
  • PCR remained above 95% across model scales, indicating that base models usually match RL decoding paths and differ mainly on tokens reflecting reasoning behaviors.The study used 50 randomly sampled MATH500 trajectories and sample-level averages; the setup used greedy decoding.Source: §2.1 · p. 3

2.2 RAST: Reasoning Activation in LLMs via Small-model Transfer

  • RAST adds the smaller model’s RL-minus-base token-logit differences to the large base model’s logits before softmax, with λ controlling adjustment strength.The transferred signal selectively amplifies reasoning-relevant tokens while preserving base predictions elsewhere.Source: §2.2 · p. 3, Figure 2 · p. 4

3 Unlocking Reasoning Activation

RAST improves mathematical reasoning across Qwen-2.5 scales, sometimes matching or exceeding RL-trained counterparts, and also supports diverse sampling and transfer to other models and code tasks. Its evaluations span six mathematical benchmarks and report gains across model families and reasoning domains.

3.1 Experimental Setup

  • The evaluation covers six mathematical benchmarks, Qwen-2.5 models from 1.5B to 32B, and additional Llama-3.1 and code-task settings.Metrics include pass@k and recovery rate; performance runs use up to 32 samples, with base models evaluated using greedy decoding in Table 1.Source: §3.1 · p. 4, Table 1 · p. 5

3.2 Main Results

  • RAST improves pass@1 and recovery rate across six mathematical benchmarks and Qwen-2.5 scales, sometimes approaching or exceeding the corresponding RL-trained model.For the 32B base, ∆R14B achieves approximate or superior results to the 32B RL-trained model on MATH500, Minerva, and GSM8K, without target-model retraining or RL.Source: Table 1 · p. 5, §3.2 · p. 5

3.4 RAST Generalizes Well to Other Models and Tasks

  • On Llama-3.1-70B, ∆R8B improves AIME24 by +2.8, AMC by +4.4, and Olympiad by +2.3 absolute points.Improvements on MATH500 and GSM8K were particularly high; the authors note the small RL model’s training recipe included those datasets.Source: Table 2 · p. 5, §3.4 · p. 7
  • On Qwen-2.5-14B-1M, RAST improves average performance by +4.4 absolute points across three code reasoning benchmarks.The transferred delta comes from the 7B Code-R1-Zero model, and evaluation uses greedy decoding.Source: Table 3 · p. 5, §3.4 · p. 7

3.2 Main Results

  • On MATH500, the 32B base model’s accuracy rises from 73.7 with ∆R1.5B to 80.7 with ∆R14B; the ceiling model reaches 81.3.Larger delta sources also improve results for the 14B base compared with ∆R1.5B across datasets.Source: §3.2 · p. 6
  • A 32B base reaches 95.3 on GSM8K with ∆R14B, bridging nearly the gap between the base at 93.1 and RL expert at 95.7.Higher-capacity bases can be more receptive, but applying ∆R7B to a 14B base yields a higher recovery rate than applying it to a 32B base.Source: §3.2 · p. 6

3.3 RAST Boosts Reasoning Diversity

  • RAST’s pass@k equals or exceeds the RL-trained model across all benchmarks, and sometimes exceeds the base model on AMC, MATH500, and OlympiadBench.Pass@k rises monotonically with larger k across benchmarks.Source: §3.3 · pp. 6–7, Figure 3 · p. 6

4 Understanding Reasoning Activation

The analysis links cross-scale alignment of RL-induced logit shifts to transferability, examines RAST’s token-level behavior, and reports its memory efficiency and decoding-parameter robustness.

4.1 Similarity of ∆R as Signal for Transferability

  • Recovery rate positively correlates with cosine similarity between cross-scale ∆R pairs, indicating better transferability as delta logits align.The analysis samples 50 MATH500 examples, follows SimpleRL-32B decoding trajectories, and computes ∆R token by token.Source: Figure 4 · p. 7, §4.1 · p. 7

4.2 Token-Level Behavior Shift

  • RAST-guided outputs recover the RL decoding path at PRC 96.37% and show candidate testing, explicit verification, and alternative reasoning.The “check” token’s KLD reaches 837.9, while normal tokens usually remain below 1.0.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 Efficiency Analysis

  • RAST retains high recovery rates while reducing estimated GPU memory and hardware requirements by up to 50%, reaching over 84% in the 32B + ∆R14B setting.The estimates include training the small RL model and RAST inference costs.Source: Table 4 · p. 8, §4.3 · p. 8

4.4 Robustness regarding τ and λ

  • Accuracy remains reasonably good for τ ∈[0.5, 1.0] and λ ∈[0.3, 1.5], indicating robustness to moderate decoding-time parameter fluctuations.Figure 6 varies τ with λ = 0.5 and varies λ with τ = 1.0.Source: Figure 6 · p. 9, §4.4 · p. 9

5 Related Work

Prior work uses RL to improve reasoning quality and explores decoding-time logit manipulation. RAST builds on these lines by transferring RL-induced reasoning signals from smaller models to larger base models.

5.1 Reinforcement Learning for LLM Complex Reasoning

  • Reasoning-focused RL optimizes answer quality with outcome or process supervision, while zero-RL applies RL directly to base LLMs without intermediate supervised fine-tuning.The paper builds on open-source Zero-RL models and studies transferring their elicited reasoning behavior to larger base models.Source: §5.1 · p. 9

5.2 Decoding-time Strategy for Reasoning Enhancement

  • Earlier decoding-time methods manipulate logits through sampling strategies or expert–amateur contrast, with applications spanning generation and downstream tasks.Prior work applied contrastive decoding to factuality, diversity, coherence, translation, retrieval-augmented generation, and knowledge-intensive tasks.Source: §5.2 · p. 9
  • Unlike methods targeting instruction-tuned or supervised models, RAST targets RL-induced reasoning signals and tests their transferability across model scales.For mathematical reasoning, prior work explored contrastive decoding and identifying critical reasoning tokens through contrastive estimation.Source: §5.2 · p. 10

6 Conclusion and Discussion

RAST transfers logit-level guidance from smaller RL-tuned models to enhance reasoning in larger LLMs. Experiments report consistent performance gains, robustness across decoding settings, and alignment between delta signals and reasoning activation.

6 Conclusion and Discussion

  • RAST is a decoding-time framework that transfers logit-level guidance from smaller RL-tuned models to enhance reasoning in larger LLMs.The paper describes the approach as a scalable reasoning-enhancement framework.Source: §6 · p. 10
  • RAST consistently boosts performance, often approaching or surpassing much more expensive ceiling models.Source: §6 · p. 10
  • The analysis reports robustness across decoding hyperparameters and alignment between delta signals and reasoning activation.The paper presents this as a practical and efficient pathway for eliciting complex reasoning and studying model behavior under RL.Source: §6 · p. 10

A.1 Datasets

The appendix describes mathematical and code evaluation datasets, inference prompts, and decoding configurations used in the experiments.

A.1 Datasets

  • Math evaluation uses MATH500 (500 sampled problems), GSM8K (1,319), OlympiadBench (675), AIME 2024 (30), AMC 2023 (40), and Minerva (272).The OlympiadBench subset is open-ended, text-only, and competition-level; Minerva includes 191 numeric and 81 symbolic solutions.Source: §A.1 · p. 16
  • Code evaluation includes LiveCodeBench (880 samples), HumanEval+ (164), and MBPP+ (378).LiveCodeBench continuously collects contest problems; the two “+” datasets adapt HumanEval and MBPP, respectively.Source: §A.1 · pp. 16–17

A.2 Prompt Templates for Inference

  • Math inference uses two prompt templates; ∆R1.5B uses Prompt (b) because the smaller model struggles with the complex chat template and “\boxed” instruction.The input placeholder is replaced with the corresponding problem.Source: §A.2 · p. 17, Figure 7 · p. 17

B Decoding Configurations

Majority@k improves as more inference runs are aggregated, with RAST bridging much of the base-to-RL performance gap on several benchmarks but retaining a wider gap on harder tasks.

B Decoding Configurations

  • Table 5 lists experimental decoding hyperparameters, while Figure 8 illustrates majority@k across six datasets for Mbase, RAST, and MRL.The supplied passages do not specify the hyperparameter values.Source: B Decoding Configurations · p. 17, Figure 8 · p. 18

C.1 Performance of Majority@k

  • Majority@k measures accuracy using the most frequent prediction among k inference runs for each problem, reflecting robustness and consistency rather than diversity.The majority function selects the prediction appearing most often among the k runs.Source: §C.1 · p. 18
  • As k increases, majority@k improves monotonically; RAST bridges much of the Mbase–MRL gap, especially on MATH500 and GSM8K.Figure 8 compares Qwen-2.5-32B as Mbase, RAST using ∆R14B, and its RL-trained counterpart across six mathematical reasoning datasets.Source: §C.1 · p. 18, Figure 8 · p. 18
  • On AIME 2024 and AMC 2023, a wider gap remains between RAST and MRL, although RAST substantially improves over Mbase.The authors associate these harder settings with more divergent reasoning paths and difficulty achieving high consensus.Source: §C.1 · p. 18

C.2 Response Length

RAST generally produces longer outputs than base models, while its response length varies with the transferred expert and can accompany both performance and style changes.

C.2 Response Length

  • On AMC and GSM8K, Qwen-2.5-7B with ∆R1.5B produces 1362.8 and 354.9 average tokens, respectively, exceeding its base and RLZero counterparts.RL-trained models generally increase output length, and RAST often inherits this tendency, particularly with deltas from smaller experts.Source: §C.2 · p. 19
  • With 32B base models, outputs using ∆R14B are always shorter than those using ∆R1.5B, and RAST’s best performance usually coincides with its shortest outputs.The authors report a general negative correlation between performance and generated output length.Source: §C.2 · p. 19
  • RAST affects generation style as well as reasoning behavior, suggesting potential control over verbosity and output structure through transfer signals.The authors frame tuning verbosity and structure as a future direction for decoding-time style modulation.Source: §C.2 · p. 19

C.3 Empirical Token Signals of Reasoning Activation

RAST shifts reasoning-token patterns toward RL-trained behavior while reflecting task-specific reasoning emphases; the section also outlines memory-estimation assumptions and possible extensions.

C.3 Empirical Token Signals of Reasoning Activation

  • AIME 2024 and Olympiad Bench show increased branching-out and backtracking tokens, while AMC 2023 and MATH500 emphasize self-verification tokens such as “check” or “confirm”.The authors use manually curated token sets for branching out, backtracking, and self-verification; these patterns suggest reasoning demands vary by benchmark.Source: §C.3 · pp. 19–20
  • Across datasets and reasoning categories, RAST increases reasoning-related token frequencies relative to Mbase and more closely matches MRL, nearly mirroring its self-verification behavior on Minerva, Olympiad Bench, and GSM8K.Figure 10 reports normalized occurrence rates for the token categories across models.Source: §C.3 · pp. 19–20, Figure 10 · p. 20

D Memory Estimation Details

  • GPU memory estimates cover model weights, optimizer states, and activations and buffers; the stated activation assumptions include gradient checkpointing and 12–25 GB per GPU.With CPU offloading, optimizer states are excluded from GPU memory estimates; the passage estimates 5–8 GB per GPU for auxiliary memory.Source: D Memory Estimation Details · p. 21

E Future Directions

  • An initial Qwen2.5 ensemble of ∆R14B, ∆R7B, and ∆R1.5B reached around 75.6 on MATH500, below single-expert ∆R14B, and added inference memory overhead.The authors suspect similar reasoning behavior across same-family deltas and propose exploring experts with more distinct behaviors.Source: E Future Directions · p. 21
  • The authors propose distilling RAST’s output-space corrections into lightweight LoRA modules, potentially enabling a hub for combining reasoning patterns.They describe this as a future direction rather than a demonstrated result.Source: E Future Directions · p. 21
  • The authors suggest exploring RAST-style activation for code generation, instruction following, or factual grounding, and comparing it with preference-based fine-tuning or reward modeling.These are proposed future applications and comparisons.Source: E Future Directions · pp. 21–22

F Limitations

The authors identify limited theoretical understanding, modest gains on difficult benchmarks, and dependence on a capable RL-trained small expert as limitations of RAST.

F Limitations

  • The theoretical basis for why small-model logit adjustments activate effectively across scales remains underexplored, including when the method works and its failure modes.The authors say a deeper account could support more rigorous guarantees and improve method design.Source: F Limitations · p. 22
  • On difficult tasks such as AIME, gains are relatively modest, and smaller experts may not capture abstract reasoning, multi-step derivations, or domain-specific heuristics.The authors say this may limit reasoning-behavior activation and suggest targeted adaptation as a possible complement.Source: F Limitations · p. 22
  • RAST still requires a reasonably strong RL-trained small expert, and a poorly aligned expert may provide little benefit.The method’s quality and generalization ability are tied to the expert’s effectiveness.Source: F Limitations · p. 22
Loading 2506.15710v1…