Source-linked AI summary
Textbooks Are All You Need II: phi-1.5 technical report
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, Yin Tat Lee
TL;DR
The paper asks whether smaller language models can achieve capabilities usually associated with much larger models. It trains phi-1.5 primarily on synthetic textbook-like data and finds performance similar to larger models, while releasing the base model for further research despite residual toxicity and coding errors.
Problem
The paper addresses whether high-level language-model capabilities require the extreme parameter and data scales of contemporary LLMs.
Method
The authors train a 1.3-billion-parameter phi-1.5 model using phi-1 code data and roughly 20B tokens of synthetic textbook-like data for common sense and general knowledge.
Results
Phi-1.5 performs similarly to models with an order of magnitude more parameters and exceeds them on common-sense or logical reasoning tasks.
Takeaways & Limitations
The findings suggest that data quality may matter more than scale alone for achieving high-level capabilities in smaller language models.
Takeaways & Limitations
Phi-1.5 has lower toxicity propensity than some base models but remains capable of generating toxic content.
Abstract
from arXiv · showhide
We continue the investigation into the power of smaller Transformer-based language models as initiated by \textbf{TinyStories} -- a 10 million parameter model that can produce coherent English -- and the follow-up work on \textbf{phi-1}, a 1.3 billion parameter model with Python coding performance close to the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to generate ``textbook quality" data as a way to enhance the learning process compared to traditional web data. We follow the ``Textbooks Are All You Need" approach, focusing this time on common sense reasoning in natural language, and create a new 1.3 billion parameter model named \textbf{phi-1.5}, with performance on natural language tasks comparable to models 5x larger, and surpassing most non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic coding. More generally, \textbf{phi-1.5} exhibits many of the traits of much larger LLMs, both good -- such as the ability to ``think step by step" or perform some rudimentary in-context learning -- and bad, including hallucinations and the potential for toxic and biased generations -- encouragingly though, we are seeing improvement on that front thanks to the absence of web data. We open-source \textbf{phi-1.5} to promote further research on these urgent topics.
1 Introduction
The paper asks whether high-level language-model capabilities require enormous scale and investigates a smaller model trained with synthetic, textbook-quality data. Phi-1.5 is presented as a 1.3-billion-parameter model whose benchmark performance approaches much larger systems while supporting research on LLM risks.
- The paper investigates whether a language model can achieve particular capabilities at substantially smaller scale.
- Large models approach trillions of parameters and training tokens, creating economic, scientific, environmental, and governance concerns.
- Phi-1.5 is a 1.3 billion parameter model trained on 30 billion tokens that achieves common-sense reasoning results comparable to models ten times larger.
- Its training data is almost exclusively synthetic, textbook-like data, supporting investigation of toxicity and bias control in language-model generations.
- The raw phi-1.5 model is open-sourced to facilitate research on in-context learning, interpretability, hallucinations, toxicity, and biased outputs.
2 Technical specifications
Phi-1.5 uses the phi-1 Transformer architecture and combines earlier code data with newly generated textbook-like synthetic data. Additional web-only and mixed-data variants probe the contribution of filtered web data.
- Phi-1.5 uses a 24-layer Transformer with 32 heads, 64-dimensional heads, rotary embeddings, a 2048-token context, flash-attention, and the CodeGen-Mono tokenizer.
- Training data combines 7B tokens from phi-1 with roughly 20B synthetic textbook-like tokens covering common sense reasoning and general knowledge.
- Only the 6B-token filtered code dataset from phi-1 is non-synthetic in phi-1.5’s training data.
- The study creates phi-1.5-web-only from 95B filtered web tokens and phi-1.5-web from a mixture of web, code, and synthetic data.
- None of the models receives instruction fine-tuning or RLHF, although they can follow instructions imperfectly in question-answering formats.
3 Benchmark results
Phi-1.5 performs comparably to substantially larger models on common-sense benchmarks and surpasses existing models on mathematics and coding evaluations. Language-understanding results are closer and vary by task, while web data particularly improves reasoning performance.
- Common Sense Reasoning: Phi-1.5 achieves results comparable to Llama2-7B, Falcon-7B, and Vicuna-13B on nearly all common-sense benchmarks.
- Common Sense Reasoning: Phi-1.5-web-only outperforms existing models of similar size using only 15% of Falcon’s refined web dataset.
- Language Understanding and Knowledge: Language-understanding differences from other models are smaller and depend on the task across PIQA, Hellaswag, OpenbookQA, SQUAD, and MMLU.
- Multi-Step Reasoning: Phi-1.5 outperforms all existing models, including Llama 65B, on the reported coding tasks.
- Multi-Step Reasoning: Web data improves mathematics and coding performance, with phi-1.5-web outperforming phi-1.5 on those reasoning tasks.
- Multi-Step Reasoning: Phi-1.5 retains coding performance while training on mixed natural-language and coding tasks, unlike the decreased accuracy often observed for small mixed-task models.
4 Addressing Toxicity and Biases
The paper evaluates phi-1.5’s potential for toxic content generation using manually graded prompts and a ToxiGen-based safety benchmark. Synthetic, textbook-like training data appears to reduce toxic generation relative to internet-trained base models, although phi-1.5 remains imperfect.
- phi-1.5’s textbook-like synthetic training appears to attenuate toxic content generation compared with models trained exclusively on internet data.
- On a self-awareness prompt, phi-1.5 generated theory-of-mind material rather than the violent or repetitive science-fiction completions produced by Falcon-7B and Llama2-7B.
- The authors emphasize that phi-1.5 has a lower propensity for toxic content than some base models but is not immune to toxic generation.
- Safety scores use 6,541 ToxiGen sentences across 13 demographics, combining scaled perplexity with sentence toxicity on a 0–1 scale.
5 Usage of our model
phi-1.5 supports several prompting styles despite being a base model without instruction finetuning, including direct completion, question answering, chat, coding, and step-by-step reasoning. Its outputs demonstrate flexible task adaptation, while also showing imperfect instruction following and coding errors.
- Prompting formats: Base-model prompting supports direct completion, question answering, chat, Python coding, and chain-of-thought formats.The report presents these as standard prompting techniques for phi-1.5 and phi-1.5-web.
- Direct completion: The model adapts a completion to an unexpected rainy-July premise instead of ignoring the weather context.The example contrasts this behavior with earlier completions that discussed playing basketball despite the rain.
- Reasoning and limitations: Step-by-step prompting yields a correct 20-apples answer, but the model may not follow instructions perfectly because it is a base model without alignment finetuning.The report separately notes that generated code sometimes contains errors and that instruction following is imperfect.
- Code explanation: The model generates coherent explanations for socket code by identifying socket creation, binding, listening, accepting connections, and address printing.The explanation follows the code’s main operations, although the report notes intricate mistakes in some code interpretations.
- Flexible generation: The model produces varied responses for factual questions, social-media writing, polite and rude reviews, file-processing code, analogies, and plotting tasks.Examples include a gravitational-wave announcement, game reviews, directory filtering, a mind-lighthouse analogy, and histogram plotting.
6 Discussion
phi-1.5 is a 1.3 billion parameter model trained primarily on curated synthetic data, yet performs similarly to models an order of magnitude larger and exceeds them on reasoning tasks. Its open release supports research into traits and failure modes shared with larger language models.
- 1.3 billion parameters: phi-1.5 performs at a level similar to models with an order of magnitude more parameters.The model was trained primarily on a specially curated “textbook-quality” synthetic dataset.
- phi-1.5 exceeds larger models on common sense or logical reasoning tasks.
- The findings suggest data quality may matter more than scale for language-model capabilities.
- Open-sourcing phi-1.5 facilitates research on in-context learning, bias mitigation, and hallucinations.Although its capabilities remain far below the largest LLMs, it exhibits traits previously seen mainly in much larger models.