Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners

1 Introduction

• SFT-distilled reasoning models can improve RLVR results and convergence, but their long rollout responses require large contexts and substantial compute.The introduction notes that multistage RLVR may cause irreversible performance degradation and still demands significant resources.Source: §1 · p. 1
• ThinkingFree discards thinking content by appending </think>, and training on adapted inputs improves slow-thinking performance while lowering token use.The paper presents TFPI as an adaptation connecting long-CoT distillation with standard RLVR.Source: §1 · p. 1
• TFPI-based RL accelerates convergence, raises the performance ceiling, and yields more token-efficient models without specialized rewards or complex training designs.The introduction describes TFPI as a bridge between long-CoT distillation and standard RLVR.Source: §1 · p. 2, Figure 1 · p. 2
• The paper reports behavioral and parameter-level analyses indicating that TFPI preserves slow-thinking patterns and enables faster rollouts in later long-CoT RL stages.Its contributions also describe generalization across domains when training solely on mathematics.Source: §1 · p. 2

2 Preliminary

• A policy πθ assigns likelihood πθ(y | x) to response y for query x; the query set is D, and responses receive a binary rule-based correctness reward.The reward is 1 when the response aligns with the ground truth and 0 otherwise.Source: §2 · p. 2
• PPO constrains policy updates relative to the old policy through clipping, using token-level importance ratios and advantages estimated by a value model.ε denotes the clipping range of importance ratios.Source: §2 · p. 2
• GRPO avoids a value model by comparing responses to the same query, with each response’s tokens sharing its group-relative advantage.The group size is the number of generated responses per query.Source: §2 · pp. 2–3
• Figure 2 reports output-token averages for thinking and ThinkingFree modes on AIME25, and tracks avg@32 and output tokens during 4K-response-length training.The right panel evaluates AIME24 in thinking mode.Source: Figure 2 · p. 3
• The experiments use DAPO, a widely used RLVR variant, while the paper states that any RLVR algorithm can be applied to TFPI.DAPO is selected for fair comparison.Source: §2 · p. 3

3.1 Thinking-Free Mode Enables More Efficient Reasoning

• ThinkingFree transforms a thinking-formatted query x into x′ by explicitly omitting its thinking content, controlling whether reasoning content is present in the generated output.Responses are generated conditioned on the modified query x′.Source: §3.1 · p. 3
• ThinkingFree reduced output tokens by more than 70% for both DS-1.5B and Qwen3-4B on AIME25.The same trend appeared for the SFT-distilled long-CoT model and the fast–slow fusion model.Source: §3.1 · p. 3

3.2 Thinking-Free Training Is Beneficial to Slow-Thinking

• On AIME25, DAPO training with a 4K response length substantially reduced performance, motivating tests of ThinkingFree rollout with short training contexts.The passage notes that training response length should not be too short for an SFT-distilled LRM because this is detrimental to testing performance.Source: §3.2 · p. 3
• On AIME25, ThinkingFree RL with a 4K response length raised thinking-mode accuracy by approximately 2% and reduced output tokens by around 20%.Standard RLVR with a 4K length reduced avg@32 by more than 40%.Source: §3.2 · pp. 3–4

3.3 Thinking Free Policy Initialization

• TFPI is a dedicated initialization stage for SFT-distilled LRMs, using ThinkingFree RL before standard RLVR to target lower rollout costs, faster convergence, and a higher reasoning ceiling.Its modified objective evaluates the RLVR objective on x′ = ThinkingFree(x); the reward remains unchanged because the operation does not alter the ground-truth answer.Source: §3.3 · p. 4

4.1 Experimental Settings

• Training uses matched hyperparameters and rollout settings; TFPI response lengths progress from 2K→4K→8K for DS-1.5B/DS-7B and 4K→8K→16K for Qwen3-4B.Evaluations cover math, multi-task reasoning, code generation, and instruction following; benchmark outputs are reported as pass@1 accuracy.Source: §4.1 · p. 4

4.2 TFPI Enhances the Slow-Thinking of Distilled Reasoning Models

• TFPI Stage 1 raises DS-1.5B overall average accuracy from 22.0% to 26.7% with a 2K response length, and AIME25 accuracy reaches 30.8% by Stage 3.AIME25 accuracy increases from 23.0% initially to 26.9%, 28.4%, and 30.8% across Stages 1–3; Qwen3-4B TFPI reaches 63.8% overall accuracy versus 60.2% for Direct RL at the same training cost.Source: §4.2 · pp. 4–5, Table 1 · p. 5
• Math-only TFPI training improves out-of-domain results on DS-1.5B: GPQA rises from 16.3% to 29.6%, LiveCodeBench from 17.7% to 19.9%, and IFEval from 36.6% to 40.8%.The passage notes that non-mathematical domain results can fluctuate across successive stages.Source: §4.2 · p. 5

4.3 TFPI as a Foundation for RLVR to Achieve Higher Performance

• Qwen3-4B overall accuracy rises from 62.0% with Direct RL to 65.7% with TFPI+RL under matched training compute.Across model scales, TFPI+RL also achieves higher accuracies than Direct RL; on Qwen3-4B, AIME25 accuracy increases from 70.6% to 76.0% after the subsequent RL stage.Source: §4.3 · p. 5, Table 2 · p. 6
• TFPI+RL is compute-efficient: its three TFPI stages use less than 20% of standard 32K-sequence RL compute, and TFPI-only Qwen3-4B-2507 reaches 89% AIME24 accuracy.TFPI+RL requires about 1.5K H800 GPU hours for Polaris-4B-Preview comparisons, versus approximately 8K H800 GPU hours for Polaris-4B-Preview; TFPI-only DS-1.5B also outperforms DeepScaleR.Source: §4.3 · p. 6

4.4 TFPI Improves the Token Efficiency of Distilled Reasoning Models

• DS-1.5B thinking-free AIME24 accuracy rises from 29.6% initially to 37.5% at TFPI Stage 3, while average output length is 5.3K tokens versus 16.7K in thinking mode.Across reasoning baselines, TFPI stages 2 and 3 outperform almost all baselines in overall accuracy with competitive or lower token use, and TFPI lies on the described accuracy–token-use Pareto frontier.Source: §4.4 · pp. 6–7

5.1 Why TFPI Leads to Better Thinking-Mode Inference?

• Verification ratios in thinking-free training and thinking-mode evaluation follow similar trends, which the authors interpret as verification behavior transferring to thinking-mode inference.Both show a rapid Stage 1 drop, steady growth in Stage 2, and a sharp Stage 3 increase; the proposed explanation for superior performance is exploratory behavior in Stages 2 and 3.Source: §5.1 · p. 7
• TFPI checkpoints traverse a broader parameter-space region before approaching Direct RL’s final checkpoint, while their update similarity to Direct RL increases across nearly all layers.The authors suggest broad exploration may help explain TFPI’s improved reasoning models.Source: §5.1 · p. 7, Figure 4 · p. 8

5.2 About Reasoning Pattern and Rollout Speed

• For DS-1.5B trained with TFPI and evaluated in thinking mode, answer length remains stable at 500–580 tokens, while its answer-to-response ratio rises as total response length decreases.The authors say this preserves the core reasoning pattern of slow-thinking rather than drifting toward excessively extended “slow-slow thinking” behavior.Source: §5.2 · pp. 7–8
• After TFPI, long-CoT RL rollout averages start at 6K tokens and stay below 7K, versus over 9K initially and around 7.5K later for Direct RL.The comparison is for DS-1.5B on the training set.Source: §5.2 · p. 8

LRMs & Efficiency RLVR

• Long-context training is a major RLVR cost, while multi-stage RLVR and GRPO modifications are existing efficiency approaches [He et al., 2025; Yu et al., 2025; Liu et al., 2025b; Wang et al., 2025a].Initialization from SFT-distilled long-CoT models typically yields stronger results than applying RLVR directly to base LLMs.Source: §6 · p. 8
• TFPI is presented as orthogonal to existing efficiency strategies: a lightweight pre-RLVR stage intended to improve efficiency and strengthen slow-thinking inference with minimal training cost.The paper describes it as an alternative path for building token-efficient LRMs.Source: §6 · pp. 8–9

Efficient Reasoning

• Efficient-reasoning methods include length-based reward shaping, fast/slow-thinking integration, and thinking-budget control, and typically rely on specialized rewards or training strategies [Team et al., 2025b; Aggarwal & Welleck, 2025; Arora & Zanette, 2025; Liu et al., 2025a; Fang et al., 2025; Zhang et al., 2025b; Lou et al., 2025; Tu et al., 2025; Jiang et al., 2025; Zhang et al., 2025d; Li et al., 2025b; Hammoud et al., 2025; Wen et al., 2025].The paper contrasts TFPI with these approaches as yielding more efficient LRMs without specialized rewards or training designs.Source: §6 · p. 9

7 Conclusion

• TFPI is a cost-efficient stage between long-CoT distillation and standard RL that accelerates convergence, enhances attainable performance, and promotes token-efficient reasoning without complex reward shaping or elaborate pipelines.The conclusion frames it as an effective and efficient alternative to current RL paradigms.Source: §7 · p. 9

A.1 RLVR Algorithms

• TFPI is orthogonal to RLVR algorithms, and the experiments use DAPO for fair comparison; DAPO adds clip-higher, dynamic sampling, and token-level policy gradients to GRPO.The appendix also states a sampling constraint: among G responses to a query, the number equivalent to the query must be greater than zero and less than G.Source: §A.1 · p. 14

A.2 ThinkingFree Operation

• The DeepSeek thinking-free template keeps the question and instructions but inserts an empty <think>...</think> section after the assistant prompt.The thinking-mode template shown for comparison ends at the assistant prompt.Source: §A.2 · p. 14

B.1 Token Consumption of ThinkingFree

• The token-consumption meta-experiment samples 32 generations per AIME 2025 query, reports average output tokens, and fixes the maximum output length at 32K tokens for both modes.The DS-1.5B and Qwen3-4B decoding settings differ by mode; Table 4 is identified as the parameter summary.Source: §B.1 · p. 14, Table 4 · p. 14

B.2 Detailed Setup of ThinkingFree Training

• The ThinkingFree training meta-experiment uses DAPO with a 4K maximum training output length and evaluates in thinking mode with a 48K maximum output length.Its initial avg@32 is reported as 68.2 rather than 65.6 in Yang et al. (2025a), attributed to using Polaris RoPE scaling.Source: §B.2 · p. 14

C.1 Training Details

• The main experiments use the VeRL codebase and DAPO loss, with shared hyperparameters including batch size 256, learning rate 10^-6, and no KL or entropy loss.The clip-higher settings are εlow = 0.2 and εhigh = 0.28.Source: §C.1 · p. 15
• Rollouts use eight responses per problem on DS-1.5B, Qwen3-4B, and DS-7B, with Polaris-53K as training data and no dataset filtering for fairness.The experiments use 32 H20 GPUs; Table 5 summarizes training steps and time.Source: §C.1 · p. 15, Table 5 · p. 15
• TFPI uses a multi-stage training strategy; Direct RLVR maximum output lengths are 16K for DS-1.5B and DS-7B and 32K for Qwen3-4B.Source: §C.1 · p. 15

C.2 Baselines

• The evaluation compares TFPI and Direct RL under approximately matched compute, includes same-size external LRM references, and standardizes baseline testing to a 32K maximum length.Both main comparison tables evaluate models in thinking mode; the efficiency comparison includes six RL-based baselines.Source: §C.2 · p. 15

C.3 Evaluation Details

• Evaluation spans mathematical reasoning, multi-task reasoning, code generation, and instruction-following benchmarks.The suite includes AIME24/25 and BeyondAIME, GPQA-Diamond, LiveCodeBench, and IFEval.Source: §C.3 · pp. 15–16
• Table 6 details efficient-reasoning baselines from their original papers, while Table 3 unifies evaluation settings for comparison.Source: Table 6 · p. 16
• AIME24 and AIME25 use avg@32 pass@1, BeyondAIME avg@8, GPQA-Diamond and LiveCodeBench pass@1 with 8 samples, and IFEval strict prompt accuracy with 4.LiveCodeBench uses the 2024-08–2025-01 subset aligned with DeepSeek-R1.Source: §C.3 · p. 16
• Evaluation scripts are adapted from DeepscaleR, with vLLM used to accelerate inference; IFEval uses the official paper’s code.Source: §C.3 · p. 16
• Decoding settings vary by table and model family, including 32K-token limits for DS models and up to 48K for Qwen3-4B.The specified temperatures, top-p, and top-k settings differ between thinking and thinking-free modes for Table 3.Source: §C.3 · p. 16

C.4 Source of Some Results in Table 2

• Starred Table 2 results are reference-only because LiveCodeBench subsets and IFEval metrics may differ from this study’s experiments.Those results come from the Seed-1.5-Thinking report and the Qwen3-2507 Hugging Face page.Source: §C.4 · p. 16

D Analysis Details

• The experiment examines whether verification, described as an indicator of slow-thinking capability, generalizes after TFPI training in thinking-free mode.Source: D Analysis Details · p. 16
• Reasoning trajectories are segmented into steps using blank-line delimiters.Source: D Analysis Details · p. 16
• A step is classified as verification-related when it contains phrases such as “wait,” “let me verify,” or “double-check.”Source: D Analysis Details · p. 16

• Because of page limits, Sections 4.3 and 4.4 show representative results, with complete results in Tables 7 and 8, respectively.Source: E More Results · p. 16
• Table 7 compares TFPI+RL with Direct RL across benchmarks, with matched total training compute and all models evaluated in thinking mode.Avg@k is average accuracy over k random generations, equivalent to pass@1.Source: Table 7 · p. 17
• Table 8 compares TFPI’s thinking-free inference mode with efficient-reasoning baselines across tasks, reporting accuracy and average output length in thousands of tokens.Avg@k denotes average accuracy over k generations, equivalent to pass@1; Toks denotes average output length in K tokens.Source: Table 8 · p. 17