Source-linked AI summary
Phi-4 Technical Report
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, Yi Zhang
TL;DR
Phi-4 addresses whether a small language model can improve reasoning through data quality rather than relying primarily on organic data or teacher distillation. It uses synthetic-data generation, curated organic data, curriculum changes, and post-training innovations, achieving strong reasoning performance for its size and surpassing GPT-4o on GPQA and MATH. Its main limitations include factual hallucinations, imperfect detailed instruction following, and unresolved safety and bias issues.
Problem
Phi-4 addresses the challenge of improving small language models when conventional training relies mainly on organic data and scaling model or dataset size.
Method
Phi-4 uses synthetic data throughout training, curated organic data, revised data mixtures and curriculum, and new supervised fine-tuning and DPO techniques.
Results
Phi-4 performs comparably to or better than much larger models on reasoning tasks and significantly exceeds GPT-4o on GPQA and MATH.
Takeaways & Limitations
The results provide evidence that phi-4’s data-generation and post-training techniques go beyond teacher distillation for STEM-focused question answering.
Takeaways & Limitations
Phi-4 remains limited by factual hallucinations, detailed instruction following, and unresolved bias, inappropriate-content, and safety issues.
Abstract
from arXiv · showhide
We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size -- especially on reasoning-focused benchmarks -- due to improved data, training curriculum, and innovations in the post-training scheme.
1 Introduction
Phi-4 is a 14-billion-parameter model whose data-quality-centered recipe combines synthetic data, curated organic data, curriculum changes, and post-training innovations. It performs comparably to or better than much larger models on reasoning tasks, including surpassing GPT-4o on GPQA and MATH.
- Phi-4 is a 14-billion-parameter model built around synthetic data generation, curriculum and data-mixture optimization, and new post-training techniques.
- Synthetic data forms the bulk of phi-4’s training data and supports reasoning-focused pretraining, midtraining, and post-training through methods including rejection sampling and DPO.
- Phi-4’s benchmark evaluation compares it with similarly costly small models and larger foundation models using OpenAI’s simple-evals framework and additional standard benchmarks.
- Phi-4 performs comparably to or better than much larger models on reasoning-related tasks, including exceeding GPT-4o on GPQA and MATH.
- Simple-evals results may be lower than Meta’s reports because its strict formatting requirements can disadvantage Llama models.
- Fresh November 2024 AMC-10 and AMC-12 tests were used to reduce contamination concerns, and phi-4 outperformed similar-size, open-weight, and larger frontier models.
- The authors emphasize contamination-resistant evaluation, including original-prompt benchmarks and improved decontamination procedures.
- Long-chain-of-thought models such as QwQ use more tokens and parameters on AMC tasks, making their inference cost or latency different from phi-4’s.
2 Approach to Data
Phi-4’s data strategy uses synthetic data to teach structured reasoning and align training with inference, while curated organic data supplies high-quality seeds and complementary training material. The pipeline emphasizes diversity, complexity, accuracy, iterative revision, and validation across reasoning, code, and scientific tasks.
- Purpose of Synthetic Data: Synthetic data is used because it presents reasoning in structured, gradual sequences and more closely matches the formats expected during inference.
- Purpose of Synthetic Data: Synthetic examples avoid the nonlinear editing found in some human-written solutions by presenting reasoning steps in a linearly predictable order.
- Principles: The generation principles prioritize diverse coverage, nuanced complexity, advanced examples, and accurate code, proofs, and explanations.
- Synthetic Data Generation: Phi-4’s pipeline created about 400B unweighted tokens across 50 broad synthetic dataset types using different seeds and multi-stage prompting procedures.
- Seed Curation: Synthetic generation begins with curated seeds from web pages, books, code repositories, questions, and other sources selected for reasoning depth, educational value, difficulty, or logical structure.
- Rewrite and Augment: Seeds are rewritten into exercises, discussions, or reasoning tasks, then refined through self-critique and revision guided by reasoning and factual-accuracy rubrics.
- Validation: Instruction reversal converts existing code into instruction-before-code pairs, while execution tests and scientific-data checks validate appropriate synthetic datasets.
- Organic Data: Organic data remains important: high-quality questions, reasoning-dense web material, academic sources, licensed books, and code provide direct training data and seeds for specialized synthetic pipelines.
3 Pretraining details
Phi-4’s pretraining combines a decoder-only 14B architecture with approximately 10T-token training, a 4K-to-16K midtraining extension, and a data mixture emphasizing synthetic sources. Ablations indicate that repeated synthetic data can outperform additional fresh web tokens for reasoning, while non-synthetic data remains useful for knowledge-heavy capabilities.
- Architecture and training schedule: Phi-4 uses a 14B decoder-only transformer with a default 4096-token context later extended to 16K during midtraining.The architecture closely follows phi-3-medium, with tokenizer and attention changes.
- Architecture and training schedule: Approximately 10T tokens are used for pretraining, followed by midtraining that increases context length from 4K to 16K.Pretraining uses linear warm-up and decay schedules, a peak learning rate of 0.0003, weight decay of 0.1, and global batch size 5760.
- Synthetic-data ablations: More synthetic-data epochs outperform supplying more fresh web tokens under the same training-token horizon in phase 2 pretraining.The 12-epoch synthetic runs perform better than 4-epoch runs that see more unique web tokens, without observed overfitting behavior.
- Synthetic-data ablations: Synthetic-only models underperform on knowledge-heavy benchmarks and show increased hallucinations, despite gains on reasoning-heavy benchmarks.This result motivates retaining complementary non-synthetic data in the pretraining mixture.
- Data mixture: Data-mixture ablations find uniform allocation suboptimal because synthetic data is higher quality, while filtered web and targeted sources are retained to balance knowledge capabilities.Synthetic-heavy variants are marginally better than the final mixture before post-training, and their gap largely closes afterward.
- Data mixture: The final pretraining mixture allocates 40% of tokens to synthetic data, 30% to web and web rewrites, 20% to code, and 10% to targeted acquisitions.Web and web rewrites receive equal shares of their combined 30%; filtered web has approximately 1.3T unique tokens, while code and targeted acquisitions have approximately 820B and 580B.
4 Post-Training
Phi-4’s post-training combines SFT with two DPO rounds, including Pivotal Token Search (PTS) pairs and judge-guided preference data. PTS targets individual tokens that strongly affect answer correctness, while additional data addresses hallucination and safety.
- Post-Training Recipe: Post-training aligns phi-4 through one SFT round and two DPO rounds, with safety and hallucination data mixed into both stages.The first DPO round uses PTS data; the second uses full-length preference pairs.
- Direct Preference Optimization: Judge-guided DPO uses approximately 850k preference pairs labeled by GPT-4o according to response accuracy, style, and detail.Responses are generated from GPT-4o, GPT-4t, and phi-4, then combined into preference pairs.
- Pivotal Token Search: Pivotal Token Search identifies tokens whose selection significantly changes the probability that a completion succeeds.It estimates success probabilities by sampling continuations and checking them with an oracle.
- Pivotal Token Search: PTS creates DPO pairs that isolate preference optimization around a single pivotal token rather than comparing entire divergent responses.The method is motivated by reducing gradient noise from irrelevant tokens and focusing the signal near the point where responses diverge.
- Pivotal Token Search: PTS data covers mathematics, question answering, and coding, while filtering target questions to 0.2 ≤ p(success) ≤ 0.8 because pivotal tokens are rare at the extremes.Examples include choices that are valid but lead to more or less robust solution paths.
- Hallucination Mitigation: SFT data and DPO pairs are also generated to reduce hallucinations, favoring refusal when the model does not know an answer.The paper reports that this process greatly decreases hallucinations in SimpleQA.
5 Benchmarking Considerations
The paper highlights contamination, narrow skill coverage, biased generation-based judging, and guessing in multiple-choice benchmarks as evaluation concerns. PhiBench addresses these issues with original, diverse tasks and is used to guide phi-4’s development.
- Benchmark Limitations: Benchmark contamination can leave uncertainty about generalization because deduplication and decontamination do not reliably catch rephrased examples.The paper states that standard n-gram methods are ineffective against all rephrasing scenarios.
- Benchmark Limitations: Many benchmarks assess narrowly defined skills, which can miss broader model capabilities and weaknesses.Examples include specific grade-level mathematics tasks and isolated Python-function implementation.
- Benchmark Limitations: LLM-based judging can favor style, fluency, or surface qualities over reasoning accuracy and validity, introducing scoring bias.The paper notes that detailed judge notes improved consistency and reduced adverse impact.
- Benchmark Limitations: Multiple-choice benchmarks may reward pattern matching and clever guessing rather than effective use of underlying concepts through reasoning.This is identified as a limitation of multiple-choice evaluation tasks.
- PhiBench: PhiBench uses original questions and diverse tasks to evaluate the skills and reasoning abilities considered important for phi-4’s development.Its tasks extend beyond isolated implementations to include activities such as debugging, code extension, and explanation.
- PhiBench: PhiBench guided dataset-mixture and hyperparameter decisions and supported high-signal studies of model weaknesses and incoming data sources.The benchmark played a central role in optimizing phi-4.
6 Performance on Key Benchmarks
phi-4 performs strongly across benchmark comparisons, surpassing Qwen-2.5-14B-Instruct on most evaluated benchmarks and excelling in STEM and coding tasks. Its weakest scores are on SimpleQA, DROP, and IFEval, with instruction following identified as a specific weakness.
- The benchmark suite includes MMLU, GPQA diamond, MATH, HumanEval, MGSM, SimpleQA, MMLU-pro, HumanEval+, ArenaHard, and IFEval.
- phi-4 outperforms Qwen-2.5-14B-Instruct in 9 out of 12 benchmarks.
- phi-4 outscores its teacher GPT-4o on GPQA and MATH, while scoring higher on HumanEval and HumanEval+ than benchmarked open-weight models.
- phi-4’s weakest benchmark scores are on SimpleQA, DROP, and IFEval, with strict instruction following identified as a real weakness.
7 Safety
phi-4’s safety approach combines post-training alignment, red-teaming, and automated evaluation across multiple responsible-AI harm categories. Internal benchmarking and a two-week independent red-team exercise assessed grounding, safety, and risky behavior.
- phi-4’s responsible-AI approach combines safety alignment in post-training, red-teaming, and automated testing across dozens of harm categories.
- Table 10 reports performance comparisons across models, with lower scores generally better except for Grounding.
- The internal RAI benchmark compares phi-4 with Phi-3, Mistral, Gemma, and Llama models using GPT-4o-simulated multi-turn conversations across five categories.
- Grounding is scored from 0 to 5, where 0 means not grounded and 5 means fully grounded relative to the prompt.
- An independent Microsoft AI Red Team conducted a two-week exercise emulating average and adversarial users in single- and multi-turn scenarios.
8 Weaknesses
phi-4 has important limitations despite strong reasoning and language understanding: factual hallucinations, detailed-instruction failures, reasoning mistakes, verbosity, single-turn optimization, and unresolved safety concerns.
- phi-4 remains limited by its size on factual knowledge and can hallucinate biographies for plausible names; search augmentation may help but cannot eliminate hallucinations.
- phi-4 is less proficient at rigorously following detailed instructions, especially exact formatting, partly because training prioritized Q&A and reasoning data.
- Even on reasoning tasks, phi-4 can make mistakes, such as incorrectly judging whether 9.9 or 9.11 is smaller.
- Chain-of-thought-heavy data can produce unnecessarily elaborate answers, while fine-tuning maximized performance on single-turn queries.
- Bias, inappropriate-content, and safety issues were mitigated through curation, post-training, and red-teaming but were not completely resolved.
A Post-Training Dataset Details
The post-training dataset targets hallucination reduction by pairing questions with correct answers or refusals according to the base model’s reliability. Synthetic prompts and judge-guided preference data support supervised fine-tuning and DPO decisions about when to answer or abstain.
- Post-training data was created to mitigate hallucinations by teaching phi-4 to answer known questions and refuse questions it is likely to answer incorrectly.
- The pipeline estimates solution reliability with repeated phi-4 runs and uses GPT-4o to generate correct answers, refusals, bogus questions, and bogus-question refusals.
- SFT uses correct-answer pairs when phi-4 is usually correct, refusal pairs when it is usually wrong, and refusal pairs for bogus questions.
- DPO prefers correct answers over refusals for intermittently solvable questions and refusals over wrong answers when phi-4 sometimes errs.
- SimpleQA is used to evaluate progress on obscure factual questions, but its F1 score can penalize post-training that replaces wrong answers with refusals.
- Synthetic prompts instruct models to create plausible but unanswerable questions and to produce refusals rather than unsupported guesses.
- Judge-guided DPO uses GPT-4o to compare responses and label preference pairs using structured assessments of answer quality and faults.
B Data Processing
The evaluation pipeline decontaminates training data against many benchmarks using hybrid 13-gram and 7-gram matching, while preserving common Wiki 13-grams. It classifies training examples as contaminated, partially contaminated, or clean based on overlap thresholds.
- The decontamination procedure checks training text against ARC-Easy, MBPP, MATH, GSM8k, HumanEval, MMLUPro, GPQA, and other evaluation benchmarks.
- The algorithm extracts 13-gram features first and returns contaminated when a non-allowed training feature matches benchmark features.
- It then extracts 7-grams, computes overlap ratios, and compares the maximum ratio with an information threshold.
- Examples are labeled contaminated when a 13-gram match occurs, partially contaminated for an information-bearing 7-gram match, or clean when no significant overlap is found.
C AMC Evaluation Details
The AMC evaluation uses recently released contest questions, standardized prompting, repeated stochastic generations, and a grading adjustment for otherwise correct answers that fail formatting instructions. The section also illustrates synthetic-data construction through metadata-guided extraction and iterative revision.
- AMC evaluation: The AMC evaluation uses 78 questions released on or after November 6, 2024, making contamination by previously published models and training datasets less likely.
- AMC evaluation: Questions are formatted as multiple-choice problems with five options and a scoring scheme awarding 6 points for correct answers, 1.5 for unanswered problems, and 0 for incorrect answers.
- AMC evaluation: The final model was selected before its AMC score was measured, although the scores of two other candidate models had already been observed.
- AMC evaluation: Each model produces 10 independent generations at temperature 0.5, and otherwise correct solutions receive credit when they express the correct answer without following the boxing instruction.
- Synthetic data generation: Synthetic reasoning data is built from annotated excerpts and iterative self-revision that generates, critiques, and revises exercises, conversations, and reasoning demands.
D.2 Fill-in-the-middle exercises
The fill-in-the-middle pipeline creates reasoning exercises by removing meaningful spans from code or text and using the remaining context to reconstruct the missing answer. An example shows iterative analysis of graph-path constraints and correction of an overly broad proposed condition.
- Fill-in-the-middle generation removes a meaningful portion of text, uses the remainder as context, and treats the removed content as the ground-truth answer.
- The example tracks graph paths, disallows revisiting start and end nodes, and limits revisits of small lowercase-named nodes.
- The reasoning assigns cused to indicate whether one small node has already been revisited and bad to mark a path invalid after another prohibited revisit.
- The proposed condition if c > 1 captures the general logic but is broader than the correct condition requiring elif c == 2 and not cused.
- The exercise receives a partial-correctness grade of 3 because the proposed solution lacks the precise revisit condition.
D.3 Agents
Agent-task data uses detailed trajectories, planning, reflection, and error correction, with phi-4 examples showing explicit state tracking and legality checks in Tower of Hanoi interactions. The examples emphasize revising plans after rule-violating failures.
- Agents: AgentKit trajectories with detailed chain-of-thought are rewritten into self-contained statements or thoughts for training on longer-horizon tasks.
- Agents: The Tower of Hanoi example presents explicit planning and reflection while analyzing the current configuration and past unsuccessful actions.
- Agents: Repeated failed moves resulted from violating the rule that a larger disk cannot be placed on a smaller one.
- Agents: The revised plan evaluates move legality before execution and selects moving disk 1 from rod B to rod C as the next necessary step.
- Agents: The planned sequence continues by moving disk 0 from A to C and disk 2 from C to B to progress toward the goal configuration.