Source-linked AI summary
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang, Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Perez-Becker, Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang, Yelong Shen, Saksham Singhal, Subhojit Som, Xia Song, Tetyana Sych, Praneetha Vaddamanu, Shuohang Wang, Yiming Wang, Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang, Ziyi Yang, Donghan Yu, Ishmam Zabir, Jianwen Zhang, Li Lyna Zhang, Yunan Zhang, Xiren Zhou
TL;DR
The paper addresses how compact language models can deliver strong capabilities while supporting diverse modalities without requiring separate models or sacrificing language performance. It introduces Phi-4-Mini and unified Phi-4-Multimodal using curated data and modality-specific LoRAs, and reports strong language, multimodal, speech/audio, and reasoning results for their size. The reasoning-enhanced Phi-4-Mini is a separate preview model, and the models retain limitations in factual recall, multilingual capability, and harmful-content control.
Problem
Existing multimodal systems may require fine-tuning the language backbone or multiple models, creating performance and resource constraints for diverse inputs.
Method
The paper trains Phi-4-Mini on high-quality reasoning-rich text and code data, then freezes the language model and adds modality-specific LoRA adapters for vision and speech/audio.
Results
Phi-4-Mini and Phi-4-Multimodal achieve strong size-efficient language, coding, multimodal, speech/audio, and reasoning performance, with unified multimodal inference in one checkpoint.
Takeaways & Limitations
Mixture of LoRAs provides an extensible way to combine multiple modalities while preserving a frozen language backbone and limiting modality interference.
Takeaways & Limitations
The models may fail to recall specific facts, have multilingual capability limited by parameter count, and sometimes produce undesirable content requiring application-level mitigation.
Abstract
from arXiv · showhide
We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language model trained on high-quality web and synthetic data, significantly outperforming recent open-source models of similar size and matching the performance of models twice its size on math and coding tasks requiring complex reasoning. This achievement is driven by a carefully curated synthetic data recipe emphasizing high-quality math and coding datasets. Compared to its predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of 200K tokens to better support multilingual applications, as well as group query attention for more efficient long-sequence generation. Phi-4-Multimodal is a multimodal model that integrates text, vision, and speech/audio input modalities into a single model. Its novel modality extension approach leverages LoRA adapters and modality-specific routers to allow multiple inference modes combining various modalities without interference. For example, it now ranks first in the OpenASR leaderboard to date, although the LoRA component of the speech/audio modality has just 460 million parameters. Phi-4-Multimodal supports scenarios involving (vision + language), (vision + speech), and (speech/audio) inputs, outperforming larger vision-language and speech-language models on a wide range of tasks. Additionally, we experiment to further train Phi-4-Mini to enhance its reasoning capabilities. Despite its compact 3.8-billion-parameter size, this experimental version achieves reasoning performance on par with or surpassing significantly larger models, including DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.
1 Introduction
The report presents compact Phi language and multimodal models that combine strong size-efficient performance with unified support for text, vision, and speech/audio. Its approach uses curated data, frozen language backbones, and modality-specific LoRAs to extend capabilities while limiting interference.
- Unified multimodality: Phi-4-Multimodal unifies text, image, and speech/audio inputs across multiple inference modes in one model checkpoint.Supported modes include text-only, text + image, speech/audio, and speech + image.
- Architecture: Mixture of LoRAs adds modality capabilities while keeping the base language model frozen and reducing interference between modalities.The design also supports adding new LoRAs for additional modalities without affecting existing ones.
- Scope: The reasoning-enhanced Phi-4-Mini is a separate preview model that is not released concurrently with the standard checkpoints.This experimental model is distinct from Phi-4-Mini and Phi-4-Multimodal.
- Training and reasoning: Training uses high-quality, reasoning-rich text data, curated code datasets, and subsequent multimodal expansion after freezing the language model.The report also explores a reasoning-enhanced Phi-4-Mini against substantially larger reasoning systems.
- Performance: Phi-4-Mini achieves state-of-the-art language, coding, and multimodal performance for its size category.Reported strengths include natural-language reasoning and mathematics, code synthesis and debugging, and tasks combining images with text or speech.
- Speech and audio: Phi-4-Multimodal delivers strong multilingual speech recognition and translation and is the first open-sourced model with speech summarization capability.The report identifies speech and audio performance as an exceptional capability.
2 Model architecture
Phi-4-Multimodal extends a shared language backbone to vision and speech/audio through modality-specific LoRAs, supporting multiple combined inference modes while preserving language performance. Its architecture and staged training connect modality encoders and projectors to the frozen or selectively adapted decoder.
- 2.1 Language model architecture: Phi-4-Mini and Phi-4-Multimodal share a decoder-only language backbone with a 200,064-token vocabulary and 128K context length.The vocabulary is intended to support multilingual and multimodal input and output efficiently, while LongRoPE enables the context length.
- 2.1 Language model architecture: Group Query Attention uses 24 query heads and 8 key/value heads, reducing KV-cache consumption to one-third during long-context generation.The backbone also uses fractional RoPE so 25% of each attention head remains positionagnostic.
- 2.2.1 Modality Details: Vision processing combines an image encoder, a projector aligning vision and text embeddings, and a vision LoRA adapter.The image encoder and projector introduce 440M parameters, while the vision adapter contributes another 370M.
- 2.2.1 Modality Details: Speech/audio processing uses 80-dim log-Mel features, an audio encoder, a projector, and a decoder LoRA adapter.The audio encoder and projector introduce 460M parameters, and the audio LoRA consumes another 460M; the speech token rate is 80ms.
- 2.2.2 Training Pipeline: Multimodal training proceeds through vision training, speech/audio training, and joint vision-speech training with selected components frozen or fine-tuned at each stage.Speech training aligns audio and language representations, then uses about 100M curated speech and audio SFT samples for post-training.
3.1 Language training data
Phi-4-Mini’s language training emphasizes improved filtering, math and coding data, synthetic data, and a re-tuned mixture with more reasoning data. Its training recipe also expands instruction-following, code-completion, and reasoning data.
- 3.1 Language training data: Four data improvements target Phi-4-Mini’s language quality: filtering, math and coding data, synthetic data, and mixture composition.The resulting corpus contains 5 trillion pre-training data items and is described as larger and higher quality than Phi-3.5-Mini.
- 3.1 Language training data: A stronger quality classifier improves filtering across multiple languages and dimensions including toxicity, obscurity, and scientific content.The classifier is trained on a larger curated dataset with cleaner positive and negative samples.
- 3.1 Language training data: Instruction-based math and coding data augments the original datasets, improving math, coding, and reasoning results.The coding mixture also includes extensive code-completion examples requiring missing code to be generated inside existing snippets.
- 3.1 Language training data: Phi-4 synthetic data is incorporated with the same processing and decontamination, while ablations re-tune the mixture and increase the reasoning-data ratio.The authors report that the revised mixture boosts model quality.
- 3.1 Language training data: Synthetic chain-of-thought data from larger reasoning models covers varied domains and difficulty levels and is filtered through rule-based and model-based rejection.Correct and incorrect sampled answers are labeled to construct preferred and dis-preferred generations for DPO data.
3.2 Vision-language training data
Phi-4-Multimodal pre-training uses diverse image-text, grounding, OCR, and chart-comprehension data. The objective focuses on next-token prediction over text tokens while disregarding image-token loss.
- 3.2 Vision-language training data: The vision-language dataset includes interleaved image-text documents, image-text pairs, image grounding, OCR-derived synthetic data, realistic images, and chart-comprehension data.These sources provide varied supervision for visual and textual understanding during multimodal pre-training.
- 3.2 Vision-language training data: During multimodal pre-training, the model predicts the next token using text-token loss while disregarding loss associated with image tokens.The pre-training process involves a total of 0.5T tokens.
3.3 Vision-speech training data
Vision-speech training uses synthetic speech paired with selected single-frame and multi-frame vision-language examples. Text queries are converted to audio and the resulting speech quality is evaluated by transcription.
- 3.3 Vision-speech training data: Synthetic vision-speech data covers both single-frame and multi-frame scenarios.The data reuses a selected subset of vision-language supervised fine-tuning examples.
- 3.3 Vision-speech training data: An in-house text-to-speech engine converts selected text queries into audio, excluding examples whose queries are unsuitable for spoken delivery.The synthetic speech quality is measured by transcribing the generated audio.
3.4 Speech and Audio Training Data
Speech/audio training combines alignment-focused pre-training with instruction-following post-training across recognition, translation, question answering, summarization, and broader audio understanding tasks. The data includes real, synthetic, public, and anonymized in-house examples spanning eight languages.
- Data organization: Speech/audio data is divided into pre-training ASR transcriptions for speech-text alignment and post-training examples for instruction following.Post-training covers ASR, AST, SQA, SQQA, summarization, and audio understanding.
- Pre-training: Approximately 2M hours of anonymized speech-text pairs provide strong or weak ASR supervision across eight supported languages for adapter pre-training.The adapter is pre-trained to reduce the gap between speech and text latent spaces.
- Post-training datasets: ASR training uses about 40k hours of anonymized in-house and selected public transcribed speech, contributing 28M weighted SFT examples across eight languages.The data includes approximately 20k hours from each source category.
- Post-training datasets: AST training uses about 30K hours with translations between English and seven other languages, contributing 28M weighted examples through direct and ASR-plus-translation formats.The dataset combines supervised and synthetic translations and includes a chain-of-thought format.
- Post-training datasets: SQA and SQQA data use synthetic question-answer pairs, spoken queries, and language-model responses, contributing 26M weighted SFT examples.Spoken queries are produced with an internal zero-shot TTS system, while responses are generated from ASR transcripts.
- Post-training datasets: Audio post-training includes multi-speaker summarization clips up to 30 minutes, around 17M public audio-understanding examples, safety data, and PII filtering.GPT-4 generates queries, summaries, or question-answer pairs from transcripts and metadata; detected PII examples are removed.
4 Evaluation
Phi-4-Multimodal performs strongly across vision-language, vision-speech, speech, audio, and multimodal tasks, while its frozen language backbone preserves text capability. Phi-4-Mini also shows strong language, math, coding, and enhanced reasoning performance relative to similarly sized and larger models.
- 4.1.1 Vision Benchmarks: Phi-4-Multimodal improves over Phi-3.5-Vision and generally outperforms similarly sized baselines on vision-language benchmarks.It surpasses some closed-source models on chart understanding and science reasoning tasks.
- 4.1.1 Vision Benchmarks: More than 10 points higher performance than InternOmni is achieved on both ShareGPT4o AI2D and ShareGPT4o ChartQA.These results concern vision-speech benchmarks and compare against a larger model.
- 4.1.2 Speech and Audio Benchmarks: Phi-4-Multimodal achieves strong ASR and AST performance, surpassing expert ASR and AST models across the reported test sets.The evaluated sets include CommonVoice, FLEURS, OpenASR, and CoVoST2.
- 4.1.2 Speech and Audio Benchmarks: 5.5% relative WER improvement over nvidia/canary-1B places Phi-4-Multimodal No.1 on the Huggingface OpenASR leaderboard.The model also surpasses WhisperV3 and Qwen2-audio on the reported multilingual ASR evaluation.
- 4.1.2 Speech and Audio Benchmarks: Phi-4-Multimodal is the first open-sourced model with speech summarization capability, with quality close to GPT-4o for adherence and low hallucinations.It can encode long-form audio in one shot and performs competitively on the Golden3 and AMI test sets.
- 4.1.2 Speech and Audio Benchmarks: On SQQA, Phi-4-Multimodal outperforms Qwen2-audio but trails Gemini-2.0-Flash and GPT-4o, indicating stronger conversational chat than general knowledge and reasoning chat.The authors suggest this pattern may reflect heavier weighting of conversational SQQA data during speech/audio post-training.
- 4.2 Language Benchmarks: Across language-understanding benchmarks, Phi-4-Mini outperforms similar-size models and is on par with models twice its size.Its math performance sometimes exceeds similar-sized models by more than 20 points and also surpasses scores from twice-larger models.
- 4.2 Language Benchmarks: Across nine coding benchmarks, Phi-4-Mini outperforms all evaluated 3B and 8B models except Qwen2.5 on average score.The report attributes this performance to high-quality organic and synthetic code data.
5 Safety
Phi-4-Mini and Phi-4-Multimodal use safety post-training, red-teaming, automated evaluations, and multilingual safety data. Evaluations report improved robustness and harmful-content performance, while audio safety remains bounded by voice-only data and the absence of audio-specific jailbreak training.
- Safety approach: Safety development combined post-training alignment, red-teaming, automated testing, and evaluations across dozens of harm categories.Safety datasets included helpfulness and harmlessness preferences, in-house data, and machine-translated coverage for Tier 1 languages.
- Text safety results: Phi-4 models were reported as more robust to known jailbreaks than Phi-3.5-mini and similarly sized competitors.The evaluation also found that the models could detect jailbreak presence and were less likely to comply with harmful prompts in those cases.
- Text safety results: XSTest evaluated both refusal of harmful prompts and avoidance of refusals for innocuous prompts using IPRR and VPRR.IPRR rewards refusing inappropriate prompts, whereas VPRR rewards answering valid prompts rather than refusing them.
- Multilingual safety: Harmful-content evaluations across Tier 1 languages showed improvement for Phi-4-Mini and Phi-4-Multimodal compared with Phi-3.5-mini.Defect Rates were averaged across Violence, Sexual Content, Self-Harm, and Hateful Content categories.
- Audio safety: The audio safety setup used voice-only datasets and did not train against audio-specific jailbreaks.The authors identify these as limitations of the audio safety approach.
- Audio safety: Audio safety evaluation found comparable harmful-content defect rates to text inputs, but red-teaming identified greater susceptibility to context manipulation and persuasive attacks.Speech fairness testing found no subgroup with egregiously worse performance than the overall population.
6 Weaknesses and limitations
The paper identifies limitations linked to model size, multilingual data allocation, harmful outputs, and prohibited biometric uses. It recommends application-level safeguards and states a specific boundary for biometric categorization.
- Model size limits Phi-4-Mini’s ability to remember some specific facts, including Olympic game results.
- Emphasizing coding data reduced the multilingual data ratio and worsened performance in languages other than English.
- Both models can sometimes produce undesirable content, so developers should apply measures such as system prompts and content filters.
- Phi-4-Multimodal is not intended for biometric categorization that infers protected or sensitive attributes from biometric data.
A Prompt for GPT-4 as a Judge on speech benchmarks
The speech-benchmark evaluation uses GPT-4 prompts to judge assistant responses against task-specific criteria. The prompts define reference materials, response comparisons, scoring dimensions, and required output formats.
- GPT-4-0613 serves as the judge model for synthetic MT-Bench, AirBench-Chat, and speech summarization tasks.
- MT-Bench: MT-Bench prompts ask judges to compare assistant answers with reference answers and rate correctness and helpfulness on a 1-to-10 scale.
- MT-Bench: The evaluation templates provide explicit reference-answer and assistant-answer delimiters for judging responses.
- AirBench-Chat: AirBench-Chat evaluation presents detailed audio descriptions and asks judges to score helpfulness, relevance, accuracy, and comprehensiveness.
- AirBench-Chat: AirBench-Chat requires a single output line containing two scores for the compared assistants.
- Speech summarization: Speech summarization is scored from 1 to 7 according to instruction adherence, factual coverage, structure, fluency, and coherence.
B Authors (alphabetical)
The paper provides an alphabetical author list comprising the researchers credited for the work.
- The author list begins with Abdelrahman Abouelenin, Yuxuan Hu, Bo Ren, Atabak Ashfaq, and Xin Jin.
- The middle of the list includes Adam Atkinson, Mahmoud Khademi, Sambuddha Roy, Hany Awadalla, and Dongwoo Kim.
- The list concludes with Dongdong Chen, Mengchen Liu, Yiming Wang, Junkun Chen, Yang Liu, Zhenghao Wang, and Weizhu Che.