PaperlayerView exact revision on arXiv

Source-linked AI summary

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, Xiren Zhou

arXiv:2404.14219v4cs.CLcs.AI

TL;DR

The paper asks whether highly capable language models can be made small enough for local deployment without sacrificing broad performance. It develops compact Phi models using curated data, post-training, and architectural extensions, reporting competitive results across language, long-context, and multimodal evaluations. The models remain bounded by factual-knowledge and safety limitations that require attention in downstream use.

  • Problem

    The paper addresses whether language models can retain broad capability while becoming small enough for local deployment and extending to multilingual, long-context, and multimodal tasks.

  • Method

    The paper develops Phi models with compact decoder architectures, heavily filtered data, safety-oriented post-training, long-context methods, and multimodal or mixture-of-experts extensions.

  • Results

    Phi-3.5-MoE significantly outperforms other open-source models, performs comparably to Gemini-1.5 Flash, and exceeds 90% of GPT-4o-mini’s average performance across language benchmarks.

  • Takeaways & Limitations

    Phi-3-mini supports fully offline phone deployment, while the Phi-3.5 series extends the family toward long-context, multilingual, and multimodal use.

  • Takeaways & Limitations

    The models remain vulnerable to factual inaccuracies, biases, inappropriate content, and safety issues, so downstream use requires evaluation for specific use cases and contexts.

Abstract

from arXiv · show

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce three models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision. The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters, achieves superior performance in language reasoning, math, and code tasks compared to other open-source models of similar scale, such as Llama 3.1 and the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini. Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from phi-3.5-mini, excels in reasoning tasks and is adept at handling both single-image and text prompts, as well as multi-image and text prompts.

1 Introduction

Phi-3 investigates whether carefully selected data can challenge the usual assumption that larger language models are necessary for high capability. The introduction highlights phi-3-mini as a compact model that runs locally on phones while approaching much larger models in quality.

  • Scaling laws traditionally predict improvement from increasing model and dataset size, but frontier models disrupt the assumption of a fixed data source.
  • Carefully curating and optimizing training data can reduce model size without compromising performance, according to the phi-3-mini example.

2 Technical Specifications

The phi-3 family combines compact decoder architectures, specialized data and post-training, and efficiency-oriented designs for local, multilingual, long-context, and sparse-model deployment. Phi-3-mini can run offline on an iPhone, while later variants extend the family through larger models, long context, and mixture-of-experts routing.

  • Phi-3-mini: Phi-3-mini is a 3.8B-parameter transformer decoder trained on 3.3T tokens, with a 4K default context and a 128K LongRope variant.
  • Phi-3.5-MoE: Phi-3.5-MoE uses top2 routing among 16 experts, activating 2 experts per token for 6.6B active parameters within a 16×3.8B model.
  • Local deployment: 4-bit quantized phi-3-mini occupies approximately 1.8GB and generates more than 12 tokens per second fully offline on an iPhone 14 with an A16 Bionic chip.
  • Training data: Phi-3 training uses heavily filtered public web data calibrated toward a data-optimal regime for small models, emphasizing knowledge quality and reasoning-oriented pages.
  • Parameter scaling: Phi-3-small and phi-3-medium extend parameter scaling to 7B and 14B models trained on 4.8T tokens, though some benchmark gains are smaller from 7B to 14B than from 3.8B to 7B.
  • Post-training: Post-training combines supervised fine-tuning and direct preference optimization across mathematics, coding, reasoning, conversation, model identity, and safety data.

3 Academic benchmarks

Phi-3 is evaluated against several open and proprietary baselines using a common benchmark pipeline, with results reported for reasoning and chat-quality measures. The supplied results include MT-bench scores and a prompt-format caveat that can affect comparisons.

  • Phi-3 benchmarks compare against phi-2, Mistral-7B, Mixtral-8x7B, Gemma 7B, Llama-3-instruct-8B, and GPT-3.5 using the same evaluation pipeline.
  • Prompt formatting can materially affect phi-3-mini results, as using ## before the question produced noticeable improvements across many benchmarks but was not used here.
  • 32.8 on the supplied 2-shot chain-of-thought result is reported alongside 34.3 for the compared model and 29.0 for another listed entry.

4 Multilingual and Long Context

The phi-3.5 models extend multilingual and long-context capabilities through additional multilingual and long-text data, long-rope, and mixed context windows. They perform strongly on multilingual and long-context evaluations, while 128K RULER performance remains limited.

  • Long Context: From 4K to 128K, phi-3.5-mini and phi-3.5-MoE expand context length using long-rope and a mixed context window approach.The models incorporate more multilingual and long-text data during mid-training.
  • Multilingual: 69.9 average MMLU-multilingual score is achieved by phi-3.5-MoE, exceeding phi-3.5-mini’s 55.4.phi-3.5-mini also improves over phi-3-mini, which scores 47.3 on average.
  • Multilingual: 55.4 and 47.3 are the average MMLU-multilingual scores for phi-3.5-mini and phi-3-mini, respectively.Improvements are reported in Arabic, Chinese, Russian, Ukrainian, and Vietnamese.
  • Long Context: Both phi-3.5-mini and phi-3.5-MoE outperform larger open-source models on RepoQA and are comparable to Llama-3.1-8B on RULER.The comparison includes Llama-3.1-8B, Mixtral-8x7B, and Mixtral-8x22B.
  • Long Context: 128K-context RULER evaluation shows a significant performance drop, suspected to reflect insufficient high-quality long-context data during mid-training.The authors plan to address this issue in a subsequent release.
  • Language Benchmarks: Phi-3.5-mini is comparable to Mistral-Nemo-12B and Llama-3.1-8B, while phi-3.5-MoE is comparable to Gemini-1.5 Flash across language benchmarks.Phi-3.5-MoE achieves above 90% of GPT-4o-mini’s average performance.

5 Safety

Phi-3 safety development combines alignment, red-teaming, automated testing, and responsible-AI evaluations. Safety alignment reduced harmful response rates, while benchmark results assess grounding and harmfulness across models.

  • Safety Alignment: Safety alignment, red-teaming, automated testing, and evaluations across dozens of harm categories formed phi-3-mini’s responsible-AI process.The process used curated helpfulness and harmlessness preference data plus in-house datasets.
  • Safety Alignment: Significant decreases in harmful response rates followed phi-3-mini’s safety alignment.Figure 5 compares harmful response percentages before and after alignment.
  • Safety Evaluation: The red-team harmful-response percentages are inflated because adversarial multi-turn conversations attempted to induce harmful outputs.The figure therefore reflects an adversarial stress test rather than ordinary usage.
  • Safety Evaluation: Phi-3-small, phi-3-medium, and phi-3.5-MoE underwent the same red-teaming process with identical datasets and slightly more samples.Table 4 compares phi-3 models with phi-2, Mistral-7b-v0.1, Gemma 7b, and Llama-3-instruct-8b.

6 Weakness

Phi-3-mini matches much larger models on language understanding and reasoning but remains constrained by its small capacity. The paper also notes unresolved factual, bias, content-generation, safety, and multilingual limitations.

  • Model Capacity: Phi-3-mini’s size limits performance on tasks requiring substantial factual knowledge, including TriviaQA.The authors suggest search-engine augmentation as a possible remedy.
  • Model Capacity: Phi-3-mini achieves language understanding and reasoning comparable to much larger models but remains fundamentally limited by its size for certain tasks.The limitation is attributed to insufficient capacity for storing factual knowledge.
  • Multilingual Scope: Phi-3-mini was mostly restricted to English, making multilingual capability an important next step.The paper reports initial promising results for phi-3-small with more multilingual data.
  • Responsible AI: Hallucinations, bias reproduction or amplification, inappropriate content generation, and safety issues remain challenges despite responsible-AI efforts.The authors say downstream use should be evaluated for specific use cases and safety contexts.

7 Phi-3.5-Vision

Phi-3.5-Vision is a 4.2-billion-parameter multimodal model that processes image or multi-image inputs with text and generates text. It performs competitively across visual reasoning, multi-image understanding, and multimodal safety evaluations, while retaining important reasoning and safety limitations.

  • Technical Specifications: 4.2 billion parameters define Phi-3.5-Vision, which combines a CLIP ViT-L/14 image encoder with the Phi-3.5-mini transformer decoder.Visual tokens are interleaved with text tokens; dynamic cropping supports high-resolution images and varied aspect ratios.
  • Training: 33B multimodal SFT tokens cover image understanding, charts, PowerPoint, multi-image comparison, video summarization, and safety tasks.DPO uses text and smaller-scale multimodal preference data, while multimodal and text-only tasks are jointly trained.
  • Single-image Benchmarks: Phi-3.5-Vision outperforms competing models on most single-image benchmarks while remaining smaller.The evaluation covers nine benchmarks grouped into Science, Charts, and Generic knowledge, using a 0-shot setup without image preprocessing.
  • Multi-image Benchmarks: Phi-3.5-Vision performs competitively or better than similarly sized baselines on multi-image understanding benchmarks.BLINK and VideoMME use model-selected answers and uniformly sampled 16-frame inputs to keep baseline inputs comparable.
  • Safety: Safety post-training improves Phi-3.5-Vision across three multimodal responsible-AI benchmarks and nearly all evaluated harm categories.The safety data includes text-only and in-house multimodal datasets covering public and internal harm categories.
  • Weaknesses: High-level reasoning remains difficult, and the model can generate ungrounded outputs or answer harmful and sensitive inquiries.The authors identify potential unreliability in sensitive areas such as finance and a trade-off between helpfulness and harmlessness.

A Example prompt for benchmarks

The benchmark example presents multiple-choice questions with answer options and recorded model answers. It includes arithmetic, anatomy, and history questions.

  • Example prompts: The example combines an arithmetic equation with anatomy and ancient-Greece history questions.The displayed items include a body-cavity question about the pituitary gland and a question about the Greek mystery-cult site.
  • Answer format: The questions use lettered answer choices and provide a corresponding answer after each item.The visible examples identify answers A and B for the arithmetic and anatomy questions, respectively.
  • Example prompts: The anatomy question asks which body cavity contains the pituitary gland, with Cranial listed as option B.

B Authors (alphabetical)

The author list names the paper’s contributors in alphabetical order. It includes researchers such as Marah Abdin, Xin Jin, and Adil Salim.

  • Authors: The section lists the paper’s authors alphabetically.
  • Authors: The listed contributors include Marah Abdin, Xin Jin, Adil Salim, and Jyoti Aneja.
  • Authors: The author list spans contributors with names including Nikos Karampatziakis, Michael Santacroce, and Hany Awadalla.
Loading 2404.14219v4…