PaperlayerView exact revision on arXiv

Source-linked AI summary

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, Xiren Zhou

arXiv:2404.14219v4cs.CLcs.AI

TL;DR

Scaling laws [KMH+20, HBM+22, MRB+23] assume fixed data sources, an assumption disrupted as frontier LLMs enable novel interactions with data. The paper trains Phi models and reports that Phi-3.5-MoE achieves above 90% of GPT-4o-mini's average performance across language benchmarks.

  • Problem

    Scaling laws [KMH+20, HBM+22, MRB+23] assume a fixed data source, an assumption disrupted as frontier LLMs enable novel interactions with data.

    Source: §1 · p. 1

  • Method

    The paper trains a chat-finetuned Phi-3-mini transformer on 3.3T tokens and extends its context to 128K with LongRope.

    Source: §2 · p. 2

  • Results

    Above 90% of the average performance of GPT-4o-mini across language benchmarks, Phi-3.5-MoE significantly outperforms other open-source models; Phi-3.5-mini and MoE also outperform larger open-source models on RepoQA.

    Source: §4 · p. 7

  • Takeaways & Limitations

    At ≈1.8GB after 4-bit quantization, Phi-3-mini ran fully offline on iPhone 14 at more than 12 tokens per second.

    Source: §2 · p. 3

    Phi-3.5-Vision can still answer harmful or sensitive inquiries, including deciphering some captchas and describing scam images.

    Source: §7.4 · p. 15

Abstract

from arXiv · show

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce three models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision. The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters, achieves superior performance in language reasoning, math, and code tasks compared to other open-source models of similar scale, such as Llama 3.1 and the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini. Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from phi-3.5-mini, excels in reasoning tasks and is adept at handling both single-image and text prompts, as well as multi-image and text prompts.

1 Introduction

The report introduces phi-3-mini, a 3.8B-parameter model trained on 3.3T tokens that can run locally on a modern phone while achieving quality on par with Mixtral 8x7B and GPT-3.5. It builds on a data recipe of filtered public web data and synthetic data, which the authors say helps smaller models reach capabilities previously associated with much larger ones.

1 Introduction

  • Phi-3-mini has 3.8B parameters, was trained on 3.3T tokens, and achieves quality on par with Mixtral 8x7B and GPT-3.5 while fitting on a modern phone.The report says it uses larger and more advanced versions of the datasets used for phi-2.Source: §1 · p. 1
  • Phi-2, a 2.7B-parameter model trained with LLM-filtered public web data and LLM-created synthetic data, matched models 25 times larger trained on regular data.The report presents phi-3-mini as a continuation of this data recipe.Source: §1 · pp. 1–2
  • Scaling laws assume a fixed data source, but frontier LLMs enable new ways to interact with data, motivating the report’s data-centered approach.The introduction describes this approach as combining LLM-based filtering of public web data with LLM-created synthetic data.Source: §1 · p. 1

2 Technical Specifications

The section specifies phi-3 model architectures, context lengths, training and post-training methods, and deployment characteristics. It describes quality-filtered and synthetic training data, architectural choices for larger variants, and on-device use of phi-3-mini.

Technical Specifications

  • phi-3-mini is a 3.8B-parameter decoder with a 4K default context, extended to 128K in its LongRope version, and trained in bfloat16 on 3.3T tokens.It has 32 layers, 32 heads, a 3072 hidden dimension, and a 320641-token vocabulary; its Llama-2-like block structure and tokenizer support adaptation of Llama-2 packages.Source: §2 · p. 2
  • phi-3-small (7B) alternates dense and blocksparse attention layers to reduce KV-cache use while maintaining long-context retrieval performance.It uses a 1003522-token tiktoken vocabulary, 8192 context, GEGLU, muP-tuned hyperparameters, grouped-query attention with four queries per key, and an additional 10% multilingual data; dedicated training and inference kernels implement blocksparse attention.Source: §2 · p. 2, Figure 1 · p. 3
  • phi-3.5-MoE uses top-2 routing to activate two of 16 GLU experts per token, yielding 6.6B active parameters in a 16×3.8B model.It uses SparseMixer to train the sparse router and shares the tokenizer used by phi-3-medium and phi-3-mini, with vocabulary size 32064.Source: §2 · pp. 2–3
  • 4-bit quantized phi-3-mini occupies approximately 1.8GB and runs fully offline on an iPhone 14 at more than 12 tokens per second.The device used an A16 Bionic chip.Source: §2 · p. 3, Figure 2 · p. 4

Training Methodology

  • Training uses heavily filtered public web data and synthetic data in two phases, moving from general knowledge and language understanding toward reasoning and niche skills.The authors aim for a data-optimal regime by calibrating web-data knowledge level and retaining pages that may improve reasoning; “optimal” is aspirational, not a claim of a proven optimal mixture.Source: §2 · p. 3

Technical Specifications

  • phi-3-medium has 14B parameters, 40 heads, 40 layers, and a 5120 embedding dimension, and was trained for 4.8T tokens using phi-3-mini’s tokenizer, architecture, and data.Some benchmarks improved less from 7B to 14B than from 3.8B to 7B, which the authors suggest may indicate that the data mixture needs further work for 14B parameters.Source: §2 · p. 3

Post-training

  • Post-training combines supervised fine-tuning on curated diverse-domain data with DPO on chat, reasoning, and responsible-AI data to steer away from unwanted behavior.The stages target improvements in math, coding, reasoning, robustness, and safety, and prepare the model for assistant interaction.Source: §2 · p. 5

3 Academic benchmarks

The section compares phi-3 with open-source models and GPT-3.5 on standard reasoning benchmarks using a shared evaluation pipeline. It uses few-shot prompts at temperature 0, without optimizing the pipeline for phi-3.

3 Academic benchmarks

  • All reported benchmark numbers use the same pipeline for comparability, with few-shot prompts at temperature 0 and no phi-3-specific pipeline optimization.The prompts and shot counts come from a Microsoft internal evaluation tool; results may differ from other publications because evaluation choices vary.Source: §3 · p. 5
  • A prompt-format change using ## before Question improved phi-3-mini results across many benchmarks, but was not applied in this evaluation.This caveat indicates that the reported prompt setup did not incorporate that potentially beneficial change.Source: §3 · p. 5
  • The reported 2-shot CoT row [RHS+23] lists 32.8, 34.3, and 29.0 among its entries, with other entries shown as dashes.The supplied passage does not identify the corresponding model columns or benchmark name.Source: §3 · p. 6
  • The reported 2 round ave. row [ZCS+23] lists 8.38, 8.70, 8.91, and 8.35, with other entries shown as dashes.The supplied passage does not identify the corresponding model columns or benchmark name.Source: §3 · p. 6

4 Multilingual and Long Context

Phi-3.5-mini and phi-3.5-MoE add multilingual and long-context capabilities through expanded mid-training data and context extension. The models show stronger multilingual results and competitive long-context performance, with a notable RULER drop at 128K.

4 Multilingual and Long Context

  • Long-rope [DZZ+24a] and mixed context windows extend the context limit from 4K to 128K without compromising performance on 4K-context tasks.Phi-3.5-mini and phi-3.5-MoE also use more multilingual and long-text data during mid-training.Source: §4 · p. 6
  • 69.9 average MMLU-multilingual for phi-3.5-MoE exceeds 55.4 for phi-3.5-mini and 47.3 for phi-3-mini; Figure 4 compares the models.Phi-3.5-mini improves over phi-3-mini in Arabic, Chinese, Russian, Ukrainian, and Vietnamese.Source: §4 · pp. 6–7, Figure 4 · p. 7
  • On RepoQA, both Phi-3.5 models outperform larger open-source models; on RULER, they are comparable to Llama-3.1-8B, but performance drops significantly at 128K.The authors suspect the 128K RULER drop reflects a lack of high-quality long-context mid-training data.Source: §4 · p. 7, Table 1 · p. 7, Table 2 · p. 8
  • Phi-3.5-mini is comparable to Mistral-Nemo-12B and Llama-3.1-8B, while phi-3.5-MoE outperforms other open-source models and reaches above 90% of GPT-4o-mini’s average across language benchmarks.Phi-3.5-MoE’s performance is also comparable to Gemini-1.5 Flash.Source: §4 · p. 7

5 Safety

Phi-3-mini’s safety approach combined post-training alignment, red-teaming, and automated evaluation, and significantly decreased harmful-response rates in Microsoft red-team testing. The other Phi-3 models named in the section underwent a related alignment process and were evaluated using in-house RAI benchmarks.

5 Safety

  • Phi-3-mini’s iterative safety alignment with tailored post-training data significantly decreased harmful-response rates in Microsoft red-team testing.The red team used adversarial multi-turn conversations, so the figure’s harmful-response percentages are inflated. Post-training addressed dozens of RAI harm categories using preference datasets [BJN+22, JLD+23], modifications inspired by [BSA+24], and in-house data.Source: Figure 5 · p. 8, §5 · p. 8
  • Phi-3-small, phi-3-medium, and phi-3.5-MoE underwent the same red-teaming process and used identical safety datasets, with slightly more samples.An independent Microsoft red team iteratively examined phi-3-mini, and its feedback informed additional tailored post-training data.Source: §5 · p. 8
  • In-house RAI benchmarks compared Phi-3 models with phi-2, Mistral-7b-v0.1, Gemma 7b, and Llama-3-instruct-8b using GPT-4-simulated multi-turn conversations across five categories.Ungroundedness was scored from 0 (fully grounded) to 4 (not grounded); harmfulness severity ranged from 0 to 7, with DR-x measuring the percentage of samples scoring at least x.Source: §5 · pp. 8–9

6 Weakness

The report identifies capacity-related limits in phi-3-mini’s factual knowledge and language coverage, alongside unresolved safety challenges despite mitigation efforts.

6 Weakness

  • Limited capacity constrains factual knowledge, reflected in low TriviaQA performance; the report suggests search-engine augmentation as a remedy.An example using HuggingFace’s default Chat-UI is introduced, but the supplied passage does not provide its details.Source: §6 · p. 9
  • Language was mostly restricted to English, though adding multilingual data yielded initial promising results on phi-3-small.Source: §6 · p. 10
  • Hallucinations, bias amplification, inappropriate content, and safety issues remain despite curated data, targeted post-training, and red-teaming; downstream use needs context-specific evaluation.The report says these measures significantly mitigate issues across dimensions, but substantial work remains.Source: §6 · p. 10

7 Phi-3.5-Vision

Phi-3.5-Vision is a 4.2B multimodal model for image and text inputs that performs competitively on single-image and multi-image benchmarks. Safety post-training improves its RAI performance, though the model retains reasoning, grounding, and harmful-response limitations.

7.1 Technical Specifications

  • The 4.2B model combines a CLIP ViT-L/14 image encoder with a phi-3.5-mini decoder, interleaving image and text tokens and dynamically cropping images.Pre-training uses 0.5T visual and text tokens, with images capped at 1344×1344; post-training uses SFT and DPO, including about 33B multimodal SFT tokens, and jointly trains multimodal and text-only tasks.Source: §7.1 · pp. 10, 12

7.2.1 Single-image Benchmarks

  • Phi-3.5-Vision achieves “super competitive” results on nine benchmarks and outperforms competitors on most while being smaller.The evaluation used a 0-shot setup without image scaling or preprocessing; reported results can differ from published baseline numbers because evaluation parameters differ.Source: §7.2.1 · pp. 12–13

7.2.2 Multi-image Benchmarks

  • Phi-3.5-Vision performs competitively or outperforms similarly sized baselines on multi-image understanding benchmarks.For VideoMME, all models received the same 16 sampled frames; on BLINK, the evaluated model selected answers directly rather than using ChatGPT for final selection.Source: §7.2.2 · p. 13, Table 6 · p. 15

7.3 Safety

  • Safety post-training improves Phi-3.5-Vision’s RAI performance across all three benchmarks and nearly all evaluated categories.Evaluation covered one internal and two public benchmarks, including RTVLM and VLGuard.Source: §7.3 · p. 14, Table 7 · p. 15

7.4 Weakness

  • The model can struggle with high-level reasoning, produce ungrounded outputs, and occasionally answer harmful or sensitive inquiries.The report links some failures to OCR capabilities learned through ordinary instruction tuning and proposes adding reasoning-focused and hallucination-related DPO data to improve the balance between helpfulness and harmlessness.Source: §7.4 · pp. 14–15

A Example prompt for benchmarks

The section shows benchmark prompts formatted as multiple-choice questions with labeled options and answers, including examples about anatomy and Greek history.

A Example prompt for benchmarks

Related papers

Loading 2404.14219v4…