PaperlayerView exact revision on arXiv

Source-linked AI summary

Phi-4-reasoning Technical Report

Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, Vibhav Vineet, Yue Wu, Safoora Yousefi, Guoqing Zheng

arXiv:2504.21318v1cs.AIcs.CL

TL;DR

The paper addresses how compact language models can acquire strong, generalizable reasoning abilities and how those abilities should be evaluated robustly. It trains Phi-4-reasoning through curated supervised fine-tuning and develops Phi-4-reasoning-plus with additional reinforcement learning. Across reasoning and general-purpose benchmarks, the models improve substantially over Phi-4 and are competitive with much larger models, while remaining limited by context length, training-domain coverage, and output reliability.

  • Problem

    The paper investigates how smaller language models can develop strong reasoning abilities and how reasoning-model performance and robustness should be assessed beyond single benchmark scores.

  • Method

    Phi-4-reasoning is produced by supervised fine-tuning 14B Phi-4 on carefully curated prompts and o3-mini reasoning demonstrations, while Phi-4-reasoning-plus receives further outcome-based reinforcement learning.

  • Results

    Both models substantially improve over Phi-4 across math, science, coding, algorithmic problem solving, and planning, outperform several larger models, and transfer gains to general-purpose benchmarks.

  • Takeaways & Limitations

    Careful data curation and supervised fine-tuning can support compact reasoning models, with reinforcement learning further amplifying gains within the evaluated scope.

  • Takeaways & Limitations

    The models have a 32k-token context limit, training data focused on STEM, code, safety, and math, and may produce biased, inaccurate, outdated, or truncated responses.

Abstract

from arXiv · show

We introduce Phi-4-reasoning, a 14-billion parameter reasoning model that achieves strong performance on complex reasoning tasks. Trained via supervised fine-tuning of Phi-4 on carefully curated set of "teachable" prompts-selected for the right level of complexity and diversity-and reasoning demonstrations generated using o3-mini, Phi-4-reasoning generates detailed reasoning chains that effectively leverage inference-time compute. We further develop Phi-4-reasoning-plus, a variant enhanced through a short phase of outcome-based reinforcement learning that offers higher performance by generating longer reasoning traces. Across a wide range of reasoning tasks, both models outperform significantly larger open-weight models such as DeepSeek-R1-Distill-Llama-70B model and approach the performance levels of full DeepSeek-R1 model. Our comprehensive evaluations span benchmarks in math and scientific reasoning, coding, algorithmic problem solving, planning, and spatial understanding. Interestingly, we observe a non-trivial transfer of improvements to general-purpose benchmarks as well. In this report, we provide insights into our training data, our training methodologies, and our evaluations. We show that the benefit of careful data curation for supervised fine-tuning (SFT) extends to reasoning language models, and can be further amplified by reinforcement learning (RL). Finally, our evaluation points to opportunities for improving how we assess the performance and robustness of reasoning models.

1 Introduction

Phi-4-reasoning and Phi-4-reasoning-plus combine curated supervised fine-tuning with reinforcement learning to deliver strong, transferable reasoning performance in compact 14B models. Evaluations show broad gains over Phi-4 and larger models, while exposing tradeoffs in compute, benchmark variability, and domain coverage.

  • Models and training: Phi-4-reasoning is a 14-billion-parameter model supervised fine-tuned on Phi-4, while Phi-4-reasoning-plus adds reinforcement learning on approximately 6K math problems.The SFT data contains over 1.4M prompts with reasoning traces generated using o3-mini; prompts span STEM, coding, and safety-focused tasks.
  • Models and training: Careful prompt and response curation, supervised fine-tuning, and reinforcement learning are presented as complementary ingredients for developing efficient reasoning models.The report emphasizes filtering, data mixture, training recipe, and combining distillation/SFT with RL.
  • Evaluation results: Over 50 percentage points of accuracy improvement occurs on AIME 2025 and OmniMath, while coding improves by over 25 percentage points on LiveCodeBench.Both reasoning variants improve over the Phi-4 base model across math, science, coding, algorithmic problem solving, and planning.
  • Evaluation results: 30 to 60 percentage points of improvement on TSP, 3SAT, and BA Calendar Planning indicate generalization to algorithmic and planning tasks not directly targeted during training.These tasks are described as nominally out-of-domain for the models.
  • Evaluation results: Phi-4-reasoning-plus is 22 points better than Phi-4 on IFEval, 16 points better on FlenQA, and 10 points better on ArenaHard.The report also observes modestly improved toxic-language detection, with Phi-4-reasoning showing more balanced neutral-versus-toxic accuracy.
  • Evaluation tradeoffs and limitations: Phi-4-reasoning-plus achieves higher math accuracy while using approximately 1.5× more tokens than Phi-4-reasoning on average.The token-length difference is less pronounced in coding, planning, and spatial tasks; evaluations also highlight variability across repeated runs and gaps between typical and optimal generations.

2 Data Methodology

The methodology builds reasoning supervision from diverse web, synthetic, and licensed prompts, then filters for teachable, difficult, multi-step problems and transforms selected seeds into verifiable training data. Carefully chosen subsets and synthetic responses support transferable reasoning across targeted and general-purpose tasks.

  • Seeds combine diverse web questions with synthetic questions grounded in high-quality filtered web content.The resulting database is designed to maximize diversity and coverage before filtering.
  • Phi-4’s strong baseline motivates filtering prompts at the edge of its abilities rather than training indiscriminately.The selection targets complex multi-step reasoning instead of primarily factual recall.
  • When objective solutions are unavailable, plurality responses from a strong reference model proxy ground truth for estimating seed difficulty.Agreement between weaker-model generations and this proxy is used to assess which seeds are suitable for learning.
  • Carefully chosen data subsets produce broad generalization across reasoning-specific and general-purpose tasks.The reported transfer includes tasks beyond the directly targeted training domains.
  • Selected seeds are rewritten into synthetic datasets aligned with target skills, including short-answer math problems that are easier to verify during RL.Coding seeds may be transformed into word problems, while math seeds may be rewritten for downstream verification.
  • SFT uses synthetic reasoning traces and final answers in structured thinking and answer blocks, alongside alignment and safety data generated with safety guidelines.The same teacher and synthesis pipeline supports math, coding, and safety-related training examples.

3 Phi-4-reasoning: Supervised Finetuning of Phi-4

Phi-4-reasoning is created by supervised fine-tuning Phi-4 with long chain-of-thought examples, explicit reasoning tokens, and a 32K-token context. Experiments refine hyperparameters, synthetic data, system messaging, teacher quality, and modular domain mixtures, producing strong reasoning and broader benchmark performance after SFT.

  • Phi-4-reasoning distills structured reasoning capabilities into the 14-billion-parameter Phi-4 model through supervised fine-tuning.Its architecture retains Phi-4 while adding explicit reasoning-token support and a longer context window.
  • The model uses <think> and </think> tokens to delimit reasoning blocks and supports up to 32K tokens after RoPE adjustment.These changes accommodate additional reasoning tokens beyond Phi-4’s original 16K-token limit.
  • SFT already improves calendar planning and most general-purpose benchmarks without catastrophic forgetting relative to Phi-4.The transfer occurs despite training emphasis on math, coding, and safety.
  • Reasoning quality improves throughout SFT beyond early acquisition of the explicit thinking-token format.The reported progression distinguishes learned reasoning ability from merely copying the structured format.
  • Exploration Stage: SFT experiments use fixed AIME 2024 and GPQA diamond indicators across exploration and scaling stages.Short, limited runs guide recipe selection before findings are aggregated in the final setup.
  • Exploration Stage: A learning rate of 1e−5 provides the best reasoning-performance balance, while higher rates lower training loss but can saturate or degrade downstream evaluations.Weight-decay differences at 1e−4 were within benchmark variation.
  • Exploration Stage: Adding targeted synthetic math data to web-based math problems improves AIME 2022–2024 performance by 3–10%.The synthetic responses pair chain-of-thought with a brief summary and concise final answer.
  • Exploration Stage: The learned ability to produce concise, verifiable answers also supports reinforcement learning on verifiable math problems.This links the SFT data design to later outcome-based optimization.

4 Phi-4-reasoning-plus: A bit of RL on top of Phi-4-reasoning

Phi-4-reasoning-plus applies outcome-based GRPO after supervised fine-tuning, using a rule-based, length-aware reward that balances correctness, repetition, formatting, and response length. Brief RL substantially improves AIME performance, while longer reasoning traces correlate with performance but introduce clipping-related limits.

  • RL Setup: GRPO further enhances Phi-4-reasoning using outcome-based reinforcement learning focused exclusively on mathematical reasoning.The seed dataset contains 72,401 mathematical problems, with 64 subsampled per RL iteration.
  • Reward Function: The rule-based reward incentivizes correctness, penalizes repetition and excessive length, and encourages proper response formatting.This design avoids complexities and potential reward hacking associated with neural reward models.
  • Reward Function: The length-aware accuracy reward uses Lmax = 31,744, Lpos_control = 25,600, and Lneg_control = 3,702 to treat correct and incorrect answers differently.Correct answers are protected from penalties up to Lpos_control, while incorrect answers avoid length penalties only above the minimum control threshold Lneg_control.
  • Reward Function: For correct answers, reward scaling favors concise outputs, whereas incorrect answers receive encouragement to think longer before answering.Formatting violations can override the length-aware accuracy reward with a penalty of −1.0 for an invalid or missing thinking block.
  • Training Observations: 90 GRPO steps increased AIME performance by more than 10%, while additional training produced no further gains and may be limited by 31k-token response clipping.The selected checkpoint was trained for 90 steps over approximately 6k examples, with eight response trajectories per example.
  • Training Observations: Response duration strongly correlates with AIME performance, and incorrect responses tend to grow faster in length than correct responses during training.The authors note that generations exceeding 31k tokens are clipped, which can prevent incorrect answers from being corrected before the final answer.

5 Evaluation

The evaluation examines reasoning-specific and general-purpose capabilities across diverse benchmarks, while also studying robustness, efficiency, safety, and evaluation variance. Phi-4-reasoning and Phi-4-reasoning-plus substantially improve over Phi-4 and perform competitively with much larger models, but results expose task-specific weaknesses and reproducibility challenges.

  • Robustness and evaluation: AIME 2025 results are highly variable, making single-score comparisons unreliable and motivating evaluation over 50 independent runs.The benchmark has only 30 problems, and average-of-5 results can differ by up to 5–10 percentage points across independent runs.
  • Reasoning benchmarks: Phi-4-reasoning and Phi-4-reasoning-plus match or exceed several much larger models despite having 14B parameters.They are comparable with or better than DeepSeek-R1, DeepSeek-R1-Distill-Llama-70B, and o1/o3-mini on math reasoning, and outperform Claude 3.7 Sonnet and Gemini 2 Flash Thinking on all tasks except GPQA and Calendar Planning.
  • Reasoning benchmarks: Phi-4-reasoning-plus is 15% more accurate than Phi-4-reasoning on AIME 2025 and 5% more accurate on Omni-Math and TSP, but longer traces do not consistently improve non-math tasks.Its generations are 1.5x longer on average, while gains are less pronounced on algorithmic reasoning, planning, coding, and spatial understanding.
  • Limitations and opportunities: The evaluation identifies weaknesses in scientific topics, discrete mathematics, harder recent AIME problems, and reliable accuracy across repeated samplings.The authors also note a 1.5x token-length increase for Phi-4-reasoning-plus and emphasize unresolved tradeoffs between extracting better trajectories and inference cost.
  • General-purpose benchmarks: Reasoning improvements transfer to general-purpose evaluations, including more than 10% gains on ArenaHard, HumanEvalPlus, and PhiBench 2.21, plus 3–5% on MMLUPro.IFEval performance surpasses GPT-4o, and the authors attribute the broader gains to simpler reasoning components embedded in these tasks.

6 Limitations

Phi-4-reasoning inherits base-model limitations and adds costs and reliability concerns associated with reasoning. Its 32k-token context and domain-limited training data constrain some complex, extended, and less-represented tasks.

  • Base-model limitations: Phi-4-reasoning inherits declining performance outside English and standard American English, alongside risks of bias, inappropriate content, and plausible factual errors.Coding support is mainly centered on Python and common packages, so other languages and packages require manual API verification.
  • Reasoning-model limitations: Reasoning processes require more computation and time, and models may contradict their own reasoning chains, leading to slower responses or inaccuracies.Understanding and monitoring reasoning steps for transparency remains an active research area.
  • Context and training scope: A 32k-token context can limit complex tasks, truncate outputs, weaken multiturn continuity, and affect performance beyond the STEM, code, safety, and math training domains.SFT data covers STEM, code, and safety, while RL data covers math; generalization to other domains has shown signs but remains bounded by this coverage.

A Benchmarking Details

The benchmarking details list the models, generation settings, and reasoning benchmarks used throughout the report. Evaluations generally use 32,768 maximum tokens, with selected Phi-4-reasoning evaluations using 65,536 despite neither model being properly trained at that length.

  • Model settings: Table 3 lists each studied model with its temperature and maximum token limit for experiments.Phi-4-reasoning and Phi-4-reasoning-plus use 65,536 maximum generation tokens on AIME, HMMT, GPQA, and Codeforces, versus 32,768 elsewhere.
  • Benchmark inventory: Table 4 catalogs the reasoning benchmarks used in the report and identifies where each benchmark can be found.

B Additional Results

Additional analyses examine performance across difficulty, topic, domain, run aggregation, and safety categories. They show uneven performance by task characteristics, gains from repeated sampling, and modest safety improvements with remaining trade-offs.

  • Difficulty analysis: Accuracy and token usage are analyzed across difficulty levels for TSP, BA-Calendar, and Omni-MATH.Omni-MATH difficulty tags are noisy, data are imbalanced across levels, and adjacent-level fluctuations can result.
  • Topic analysis: All models perform lower on discrete mathematics and geometry problems than on other Omni-MATH topics.
  • Domain analysis: Improvements on GPQA are lower in biology and chemistry than in physics across models.Token-usage standard deviations are computed across five repeats within each high-level domain.
  • Run aggregation: Best-of-5 aggregation improves performance across models and benchmarks, indicating further room for improvement.The comparison also reports worst-of-5, average pass@1, and majority-vote results over five independent runs.
  • Safety evaluation: Phi-4-reasoning offers a better toxic-versus-neutral detection balance, with modest improvements and narrowed group-based discrepancies.Detecting toxicity without causing erasure remains challenging because reasoning improvements favor only one category at a time.
Loading 2504.21318v1…