PaperlayerView exact revision on arXiv

Source-linked AI summary

Apple Intelligence Foundation Language Models

Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap, Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang, Jianyu Wang, Jiarui Lu, John Peebles, Ke Ye, Mark Lee, Nan Du, Qibin Chen, Quentin Keunebroek, Sam Wiseman, Syd Evans, Tao Lei, Vivek Rathod, Xiang Kong, Xianzhi Du, Yanghao Li, Yongqiang Wang, Yuan Gao, Zaid Ahmed, Zhaoyang Xu, Zhiyun Lu, Al Rashid, Albin Madappally Jose, Alec Doane, Alfredo Bencomo, Allison Vanderby, Andrew Hansen, Ankur Jain, Anupama Mann Anupama, Areeba Kamal, Bugu Wu, Carolina Brum, Charlie Maalouf, Chinguun Erdenebileg, Chris Dulhanty, Daniel Parilla, Dominik Moritz, Doug Kang, Eduardo Jimenez, Evan Ladd, Fangping Shi, Felix Bai, Frank Chu, Fred Hohman, Hadas Kotek, Hannah Gillis Coleman, Jane Li, Jeffrey Bigham, Jeffery Cao, Jeff Lai, Jessica Cheung, Jiulong Shan, Joe Zhou, John Li, Jun Qin, Karanjeet Singh, Karla Vega, Kelvin Zou, Laura Heckman, Lauren Gardiner, Margit Bowler, Maria Cordell, Meng Cao, Nicole Hay, Nilesh Shahdadpuri, Otto Godwin, Pranay Dighe, Pushyami Rachapudi, Ramsey Tantawi, Roman Frigg, Sam Davarnia, Sanskruti Shah, Saptarshi Guha, Sasha Sirovica, Shen Ma, Shuang Ma, Simon Wang, Sulgi Kim, Suma Jayaram, Vaishaal Shankar, Varsha Paidi, Vivek Kumar, Xin Wang, Xin Zheng, Walker Cheng, Yael Shrager, Yang Ye, Yasu Tanaka, Yihao Guo, Yunsong Meng, Zhao Tang Luo, Zhi Ouyang, Alp Aygar, Alvin Wan, Andrew Walkingshaw, Andy Narayanan, Antonie Lin, Arsalan Farooq, Brent Ramerth, Colorado Reed, Chris Bartels, Chris Chaney, David Riazati, Eric Liang Yang, Erin Feldman, Gabriel Hochstrasser, Guillaume Seguin, Irina Belousova, Joris Pelemans, Karen Yang, Keivan Alizadeh Vahid, Liangliang Cao, Mahyar Najibi, Marco Zuliani, Max Horton, Minsik Cho, Nikhil Bhendawade, Patrick Dong, Piotr Maj, Pulkit Agrawal, Qi Shan, Qichen Fu, Regan Poston, Sam Xu, Shuangning Liu, Sushma Rao, Tashweena Heeramun, Thomas Merth, Uday Rayala, Victor Cui, Vivek Rangarajan Sridhar, Wencong Zhang, Wenqi Zhang, Wentao Wu, Xingyu Zhou, Xinwen Liu, Yang Zhao, Yin Xia, Zhile Ren, Zhongzheng Ren

arXiv:2407.21075v2cs.AIcs.CLcs.LG

TL;DR

Apple develops foundation language models for Apple Intelligence across on-device and Private Cloud Compute settings, addressing the need for capable, efficient, and responsible models. The report describes their data, training, and adaptation, and reports strong benchmark and human-evaluation performance against open and closed models.

  • Problem

    Apple Intelligence requires language models that provide broad capability while operating efficiently across Apple devices and server infrastructure and supporting specific user tasks.

  • Method

    The report develops foundation models with responsible-AI safeguards, staged pre-training, feature-specific adaptation, and evaluation across language, reasoning, instruction-following, and user-facing tasks.

  • Results

    AFM models show strong language and reasoning capabilities, with AFM-on-device achieving a 47.7% win rate against Phi-3-mini and AFM-server exceeding a 50% win rate against GPT-3.5 with a 27.4% tie rate.

  • Takeaways & Limitations

    The models form the foundation of Apple Intelligence features such as summarization, writing assistance, and tool use across Apple products.

  • Takeaways & Limitations

    Writing evaluations using an LLM grader have limitations and biases, including length bias.

Abstract

from arXiv · show

We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute. These models are designed to perform a wide range of tasks efficiently, accurately, and responsibly. This report describes the model architecture, the data used to train the model, the training process, how the models are optimized for inference, and the evaluation results. We highlight our focus on Responsible AI and how the principles are applied throughout the model development.

1 Introduction

The report introduces Apple’s foundation models for Apple Intelligence, including an on-device model and a larger server model, and describes their capability, efficiency, and responsible development.

  • AFM-on-device is a ~3 billion parameter language model, while AFM-server is a larger server-based language model for specialized tasks.
  • Apple’s Responsible AI principles emphasize user empowerment, representation, careful design, and privacy protection throughout model development.
  • The report covers model development, training, feature-specific fine-tuning, and evaluation for helpfulness and unintended harm.

2 Architecture

The AFM base models use a dense decoder-only Transformer architecture with design choices targeting memory efficiency, training stability, computational efficiency, and long-context support.

  • AFM base models are dense decoder-only Transformers with shared embeddings, normalization, grouped-query attention, SwiGLU, and RoPE.
  • Pre-normalization with RMSNorm and query/key normalization are used to improve training stability.
  • Grouped-query attention uses 8 key-value heads to reduce KV-cache memory, while RoPE uses a base frequency of 500k for long-context support.
  • Table 1 provides dimensions for AFM-on-device.

3 Pre-training

Pre-training combines curated, licensed, and crawled data with staged training and extensive filtering, while using distillation and pruning to build an efficient on-device model.

  • The dataset combines licensed publisher data, curated public datasets, web-crawled information, filtered code, and specialized math data, including 3 billion Math Q&A tokens from 20 domains.
  • Data quality, much more so than quantity, is identified as the key determinant of downstream model performance.
  • The data pipeline applies extraction, safety and profanity filtering, deduplication, quality filtering, PII removal, and decontamination against 811 benchmarks.
  • AFM pre-training has core, continued, and context-lengthening stages that shift data emphasis toward code, math, licensed data, and synthetic long-context data.
  • AFM-server is trained from scratch, whereas AFM-on-device is distilled and pruned from a larger model.
  • Distillation improves MMLU and GSM8K by about 5% and 3%, respectively, while pruning changes final benchmark results by 0–2%.

4 Post-Training

Apple’s post-training pipeline combines curated human and synthetic data with instruction tuning and reinforcement learning to improve general-purpose capabilities and alignment. Its new algorithms use model committees, preference-aware reward modeling, and scaled distillation to improve performance across model sizes.

  • iTeC and MDLOO are new post-training algorithms that improve model quality during rejection-sampling and reinforcement-learning iterations.
  • Human annotations and synthetic data are combined through extensive curation and filtering, with data quality treated as central to model success.
  • Synthetic data: Synthetic mathematics data expands prompt diversity through rephrasing, breadth and depth evolution, deduplication, solvability checks, and difficulty filtering.
  • Reward modeling: Reward modeling adds preference-level soft labels and single-sided grading as regularization to better represent human judgments.
  • Iterative RLHF: The iterative RLHF framework combines rejection sampling, DPO or IPO, and online reinforcement learning while collecting preferences from a committee of current and prior models.
  • Scaling up distillation: Scaling committee distillation produced the final AFM-on-device model from more than 1M high-quality generated responses, extending RLHF improvements to smaller models.

5 Powering Apple Intelligence features

Apple powers many task-specific experiences by specializing foundation models with runtime-swappable adapters and inference optimizations. Quantization and accuracy-recovery adapters reduce deployment costs while preserving model quality, and feature adapters address task-specific behaviors such as summarization and prompt injection.

  • Runtime-swappable adapters specialize one foundation model for dozens of tasks, enabling on-the-fly task adaptation.
  • Inference optimizations target memory, latency, and power reductions while maintaining overall model quality for on-device and Private Cloud Compute deployment.
  • 5.2 Optimizations: Near-lossless quantization averages less than 4 bits per weight and uses accuracy-recovery adapters to preserve capabilities across diverse product domains.
  • 5.2 Optimizations: Accuracy-recovery adapters require approximately 10 billion tokens, about 0.15% of base-model training, and recover performance closer to the unquantized model.
  • 5.2 Optimizations: Mixed-precision quantization compresses AFM-on-device to about 3.5 bits per weight, with 3.7 bits per weight used in production to meet memory requirements.
  • Summarization: For summarization, a LoRA adapter is fine-tuned on the quantized on-device model to enforce specifications and mitigate prompt injection in input content.

6 Evaluation

AFM models are evaluated across pre-training, post-training, human, instruction-following, tool-use, writing, and mathematical reasoning benchmarks. Across these evaluations, AFM models show strong capabilities, often matching or outperforming comparable models, with AFM-server achieving the best overall function-calling accuracy.

  • Human evaluation: AFM-server scores over 50% wins and a 27.4% tie rate against GPT-3.5, while AFM models are often preferred over open-source and commercial competitors by human graders.
  • Instruction following: AFM-on-device and AFM-server achieve superior instruction-level and prompt-level accuracy on IFEval, while remaining highly competitive on AlpacaEval 2.0 LC.
  • Tool use: AFM-server achieves the best overall accuracy on the Berkeley Function Calling Leaderboard, outperforming Gemini-1.5-Pro-Preview-0514 and GPT-4.
  • Writing: On internal writing benchmarks, AFM-on-device matches or exceeds Gemma-7B and Mistral-7B, while AFM-server significantly outperforms DBRX-Instruct and GPT-3.5 and is comparable to GPT-4.
  • Math: AFM-on-device significantly outperforms Mistral-7B and Gemma-7B on mathematical reasoning despite being less than half their size.

7 Responsible AI

Apple’s Responsible AI approach integrates feature-specific risk assessment, safety-oriented training, filtering, guardrails, and evaluation throughout model development. The models are reported as robust to adversarial prompts and preferred by human graders for safe and helpful responses over competitors.

  • 7.1 Responsible AI: Feature-specific policies contextualize the safety taxonomy, while guardrail models evaluate potential harm during inference.The taxonomy contains 12 primary categories and 51 subcategories, and policies account for each feature’s needs and content.
  • 7.2 Pre-Training: Pre-training safety mitigations avoid overly aggressive filtering so shared foundation models retain general capabilities for downstream feature-specific policies.The models are intended to remain aware of content that downstream features may need to handle carefully or directly.
  • 7.3 Post-Training: Safety alignment is treated as a core post-training task alongside helpfulness, using adversarial data in SFT and RLHF and safety benchmarks in evaluation.The approach is intended to balance helpfulness and harmlessness rather than isolate safety as a separate training stage.
  • 7.3 Post-Training: More than 10% of training data was adversarial or related to safety or sensitive topics, including preference ratings, annotations, and annotator rewrites.This data was used directly or as seed data for synthetic generation.
  • 7.6 Evaluation: AFM-on-device and AFM-server achieved violation rates significantly lower than open-source and commercial models on adversarial safety prompts.Lower violation rates are reported for harmful content, sensitive topics, and factuality evaluations.
  • 7.6 Evaluation: AFM models were preferred by human graders over competitor models for safe and helpful responses.

8 Conclusion

The report introduces AFM-on-device and AFM-server as foundation language models for Apple Intelligence. They are designed for efficient operation across Apple devices and Private Cloud Compute, with adapters supporting specialized everyday tasks and Responsible AI principles guiding their development.

  • 8 Conclusion: AFM-on-device and AFM-server are foundation language models designed to power Apple Intelligence features.
  • 8 Conclusion: The models are designed to run efficiently on iPhone, iPad, and Mac, as well as Apple silicon servers through Private Cloud Compute.
  • 8 Conclusion: The models support tasks including language understanding, instruction following, reasoning, writing, and tool use.
  • 8 Conclusion: Feature-specific adapters are fine-tuned for user experiences such as summarizing emails, messages, and notifications.
  • 8 Conclusion: The models are developed to help users with everyday activities while being grounded in Apple’s core values and Responsible AI.

Foundation Models

The report’s credited contributors include a large group spanning model development, inference efficiency, pre-training, and post-training leadership. The supplied passages identify overall, pre-training, post-training, and inference-efficiency leads among the contributors.

  • Foundation Models: Chong Wang is identified as the inference efficiency lead.
  • Foundation Models: The contributor lists include many named researchers across the report’s foundation-model effort.
  • Foundation Models: Ruoming Pang is identified as the overall lead for the reported work.
  • Foundation Models: Tom Gunter is identified as the pre-train lead, while Vivek Rathod is identified as the post-train lead.

A Core pre-training recipe ablation

The ablation compares AFM’s core pre-training recipe with a baseline under otherwise matched training conditions. AFM’s recipe produces slight improvements across most tasks, but the differences are typically very small.

  • A Core pre-training recipe ablation: The comparison uses AFM’s core pre-training settings against a Wortsman et al. [2023] baseline, with both recipes otherwise identical.Training covers 3.1T tokens using the AFM-on-device architecture, with a different data mixture from official AFM runs.
  • A Core pre-training recipe ablation: AFM’s recipe allows slight improvements across the majority of few-shot tasks, although the differences are typically very small.The table reports mostly 0-shot prompts unless otherwise noted.
  • A Core pre-training recipe ablation: AFM’s recipe demonstrates a slight improvement over the baseline in the core pre-training ablation.The authors interpret this as evidence that the baseline already configures the most important settings well for this model size and training budget.

B Ablations on pruning and distillation

Structural pruning and knowledge distillation each outperform scratch training, while combining them yields the strongest ablation results with far less computation than a longer baseline.

  • B Ablations on pruning and distillation: Pruning and distillation are complementary methods whose combination produces the best performance.The ablation explicitly evaluates them separately and together.
  • B Ablations on pruning and distillation: Combining pruning and distillation improves MMLU and GSM8K substantially, matching or exceeding a baseline trained with 5× more computation.The combined model achieves better or on-par results than the compute-intensive baseline.
  • B Ablations on pruning and distillation: 44.9% MMLU from distillation and 42.9% from pruning exceed the 34.6% scratch baseline trained for 50% more steps.Pruning performs better on CoreEn, whereas distillation performs better on MMLU.
  • B Ablations on pruning and distillation: The ablation uses an early pre-training data mixture, so its results do not directly represent the official AFM runs.Table 6 identifies the training data as an early version differing from the official runs.
  • B Ablations on pruning and distillation: Internal pre-training metrics are not suitable for comparison with published results because the harness uses nonstandard formulations and harder splits.The evaluations omit length normalization and use more difficult test splits where available.

D Long-context evaluation

AFM-server remains effective through 32k tokens on straightforward retrieval, but performance on more complex long-context tasks declines with length, limiting practical context to at most 24k.

  • D Long-context evaluation: AFM-server performs perfectly through ≥32k tokens on needle-in-the-haystack retrieval, but RULER performance declines as context length increases.RULER is a more complex benchmark than NIAH and better exposes long-context degradation.
  • D Long-context evaluation: For tasks beyond retrieval, AFM-server’s practical context length is currently at most 24k.This boundary is inferred from gradually worsening performance on RULER with increasing context length.
  • D Long-context evaluation: RULER averages results over 13 synthetic long-context tasks using 500 examples per task.The benchmark scope is defined in the Table 9 caption.

F Accuracy-recovery adapters ablation

Quantization causes large quality losses, while rank-16 accuracy-recovery LoRA adapters restore performance substantially and retain significant model-size reductions.

  • F Accuracy-recovery adapters ablation: 7–18% higher Alpaca win rate and 5–10% higher GSM8K accuracy result from rank-16 accuracy-recovery LoRA adapters.The recovered models perform much closer to the original unquantized model.
  • F Accuracy-recovery adapters ablation: Quantized models show huge quality drops across both pre-training and post-training metrics relative to unquantized models.Table 10 normalizes results to the unquantized version.
  • F Accuracy-recovery adapters ablation: More aggressive quantization from 3.7 to 3.5 bpw leads the adapters to recover more quality.The recovered models also achieve significant reductions in model size.
Loading 2407.21075v2…