Source-linked AI summary
Textbooks Are All You Need
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, Yuanzhi Li
TL;DR
The paper asks whether carefully constructed training data can improve code-generation performance despite using much smaller models and datasets. It trains phi-1 on textbook-quality data and coding exercises, achieving strong benchmark performance while exhibiting broader capabilities after finetuning. The authors also identify important scope limitations, including Python specialization and reduced robustness to prompt errors.
Problem
The paper investigates whether high-quality, textbook-like data can improve code-generation performance beyond what ordinary scaling of compute or model size provides.
Method
The authors pretrain a 1.3B-parameter model on filtered web and GPT-3.5-generated textbook data, then finetune it on synthetic Python coding exercises.
Results
50.6% pass@1 on HumanEval and 55.5% pass@1 on MBPP were achieved, while finetuning also improved external-library and unconventional-task performance.
Takeaways & Limitations
Textbook-quality data can support strong code-generation proficiency at substantially smaller model and dataset scales than competing approaches.
Takeaways & Limitations
Phi-1 is specialized in Python, lacks some domain-specific API knowledge, and is less robust to stylistic variation or grammatical errors in prompts.
Abstract
from arXiv · showhide
We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook quality" data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves 45% on HumanEval.
1 Introduction
The paper explores whether improving data quality can yield coding performance beyond ordinary scaling expectations. It trains phi-1 with textbook-quality data and reports strong benchmark results despite its much smaller scale.
- Motivation: High-quality data is studied as an alternative axis for improving language-model performance beyond scaling compute or network size.The paper focuses on code-generation models and argues that standard code corpora may be less instructive than carefully selected data.
- Results: Despite vastly smaller training scale, phi-1 outperforms competing models on HumanEval and MBPP except GPT-4.WizardCoder scores higher on HumanEval but lower on MBPP, according to the table caption.
- Results: 50.6% pass@1 on HumanEval and 55.5% pass@1 on MBPP were achieved by the 1.3B-parameter phi-1 model.Training used roughly 8 passes over 7B tokens, followed by finetuning on less than 200M tokens.
- Approach: The approach combines filtered web data, synthetically generated data, and existing language models used to synthesize training data for later models.The paper situates its data-generation strategy within an emerging trend of recursive training and focuses on a narrow task.
2 Training details and the importance of high-quality data
The paper argues that carefully selected, textbook-quality code data can outperform conventional scaling strategies, using filtered web code, synthetic textbooks, and synthetic exercises. Filtering improves HumanEval performance, while small exercise-focused finetuning produces the largest reported gain and broader coding capabilities.
- Motivation: Conventional code datasets often contain non-self-contained, trivial, poorly documented, or topic-skewed examples.These properties make snippets less instructive for learning coding concepts.
- Motivation: The authors hypothesize that code-training data should be clear, self-contained, instructive, and balanced, like a good textbook.They link these qualities to stronger natural-language-to-code learning signals.
- Datasets: The training pipeline combines about 6B tokens of filtered code-language data with less than 1B synthetic textbook tokens and about 180M synthetic exercise tokens.CodeTextbook is used for pretraining, while CodeExercises is used for finetuning.
- Results: 29% HumanEval accuracy is achieved by phi-1-base using CodeTextbook alone, while CodeExercises finetuning is described as crucial for larger gains and emergent capabilities.The paper reports that the full dataset combination contains less than 7B tokens.
- Filtering: GPT-4 labels about 100k code samples for educational value, and a random forest predicts sample quality from pretrained code-model embeddings.GPT-4 is used for limited annotation rather than synthetic content generation.
- Results: 17.68% HumanEval accuracy follows filtering, versus 12.19% for unfiltered code, despite the filtered model using 36k rather than 96k training steps.Adding synthetic textbooks further raises accuracy to 20.12% for the 350M-parameter model.
3 Spikes of model capability after finetuning on CodeExercises
Finetuning phi-1-base on the small CodeExercises dataset substantially improves instruction following and enables capabilities beyond the exercises’ basic-library distribution, including external-library use and chat.
- Capability gains: Finetuning on less than 200M tokens of CodeExercises substantially improves phi-1’s performance on tasks absent from the finetuning data.The paper highlights intricate algorithmic tasks and external-library use as examples of this broader improvement.
- Algorithmic reasoning: phi-1 interprets a complex game prompt and generates the correct answer, while phi-1-base struggles with its logical relationships.The 350M-parameter phi-1-small shows partial understanding but produces an incorrect solution.
- External libraries: Finetuning unexpectedly improves use of Pygame and Tkinter even though CodeExercises contains neither library.The authors interpret this as evidence that finetuning can make unrelated tasks easier to distill from pretraining.
- External libraries: phi-1 correctly applies PyGame functions to update and draw a bouncing ball, whereas phi-1-base and phi-1-small produce syntactically correct but semantically irrelevant calls.The comparison separates API knowledge from following the task’s intended logic.
- External libraries: phi-1 implements the Tkinter GUI and its functions correctly, while phi-1-base and phi-1-small use incorrect APIs and meaningless function calls.The only noted phi-1 error is failing to copy “pewpewpew?” exactly.
- Chat capability: phi-1 has better chat capability than phi-1-base despite chat data appearing only during pretraining and not finetuning.
4 Evaluation on unconventional problems with LLM grading
The paper evaluates phi-1 on 50 unconventional coding problems designed to reduce contamination and uses GPT-4 grading for a finer-grained assessment. The new evaluation reproduces HumanEval’s model ranking and increases confidence in phi-1’s reported performance.
- Evaluation design: 50 new coding problems were created without access to CodeExercises or the final model to minimize bias and leakage.They used the HumanEval format and were designed to be unlikely in real-world code or coding exercises.
- LLM grading: GPT-4 grades candidate solutions from 0 to 10 after a short verbal evaluation, providing a finer-grained signal without requiring tests.
- Results: The unconventional-problem grades produce the same model ranking as HumanEval, with phi-1 again significantly outperforming StarCoder.
- Results: Because the problems were designed outside the training distribution and could not contaminate the training data, the results increase confidence in phi-1’s performance validity.
5 Data pruning for unbiased performance evaluation
The paper tests whether CodeExercises contamination explains phi-1’s HumanEval performance using standard overlap analysis and aggressive embedding- and syntax-based pruning. Strong performance remains after pruning, supporting the authors’ contamination assessment.
- Standard contamination analysis: Standard 13-gram analysis found four HumanEval overlaps, but all were false positives after investigation.The analysis compared HumanEval docstrings with generated CodeExercises entries.
- Similarity-based pruning: The pruning study removes CodeExercises files similar to HumanEval using embedding and syntax-based distances rather than relying only on n-gram overlap.Embedding distance uses L2 distance between CodeGen-Mono 350M code embeddings.
- Pruned-data results: After heavily pruning CodeExercises, phi-1 still outperforms StarCoder-Prompted by a large margin on HumanEval.The evaluation separately reports similar and non-similar HumanEval subsets defined by close matches in the original dataset.
- Pruned-data results: HumanEval accuracy is lower for every model on the non-similar subset than on the similar subset.
6 Conclusion
The paper attributes phi-1’s coding performance to textbook-quality data, while recognizing limitations in language coverage, domain-specific knowledge, robustness, and dataset scaling.
- Textbook-quality data enabled phi-1 to surpass almost all open-source coding models despite being 10x smaller in model size and 100x smaller in dataset size.The authors hypothesize that clear, self-contained, instructive, and balanced examples improve learning efficiency.
- phi-1 is specialized in Python coding, restricting its versatility compared with multilingual code models.
- phi-1 lacks domain-specific knowledge for programming with specific APIs or less common packages.
- Structured datasets with limited language and stylistic diversity make phi-1 less robust to stylistic variations.
- The scaling needed to overcome these limitations, in both model size and dataset size, remains unclear.
A Additional examples for Section 3
This appendix extends the Section 3 discussion with additional examples illustrating how finetuning improves task execution beyond the finetuning dataset.
- Additional examples and details further illustrate improvements of finetuned models over the base model on tasks absent from the finetuning dataset.
A.1 Finetuning improves the model’s understanding ability
Finetuning improves the models’ ability to parse complex logical relationships in coding prompts and translate them into correct array and index operations.
- Finetuning enables accurate identification of relevant arrays and indices while correctly implementing linked logical operators.The example includes conditions involving unions, exclusions, row indices, and column indices.
- Before finetuning, the model confuses elements with indices and struggles to determine which array to use.
- The prompt asks for indices of matrix D elements in the union of B and C, with row indices excluded by A and column indices constrained by D.
- Candidate completions differ in whether they iterate over D’s length, rows, or columns and whether they return individual indices or index pairs.
A.2 Finetuning improves the model’s ability to use external libraries
Finetuning improves coding performance on tasks requiring specialized external-library APIs, including PyTorch gradient updates and animated Pyplot line plots.
- PyTorch example: Finetuning improves the model’s ability to implement a modified PyTorch gradient update using truncated SVD components.The requested update computes SVDs of gradients, retains the top k singular values, and applies the resulting update to parameters.
- PyTorch example: phi-1-small uses an incompatible NumPy SVD and assigns its result directly to the parameter, while phi-1-base produces meaningless definitions.
- Pyplot example: The Pyplot task requires an animated line plot of y = cos(x) / sqrt(x + 1), changing from red to green when x[i] exceeds 11.
- Pyplot example: phi-1-small fails to implement the line-plot requirement and phi-1-base updates the template incorrectly by omitting the core per-iteration line update.
B Limitation of phi-1
phi-1’s small size and limited training data constrain its generality, prompt robustness, natural-language handling, and performance on counting and spatial reasoning tasks.
- 1.3B parameters and 7B training tokens restrict phi-1’s capacity for complex tasks such as developing intricate Flask applications.These constraints cannot be overcome solely through finetuning.
- Prompt sensitivity: Longer prompts significantly reduce performance, with the model ignoring, forgetting, or misinterpreting prompt parts.The authors hypothesize that predominantly short exercise prompts contribute to this sensitivity.
- Natural-language sensitivity: phi-1 is less robust with ambiguous natural-language inputs than ChatGPT or StarCoder, including prompts containing “unchanged” or numbered lists.The authors attribute this possibly to filtering data to maintain textbook-level quality.
- Counting and spatial reasoning: The model struggles with counting and spatial reasoning, failing to maintain precise information about quantities and element positions.In a layout example, it generates an extra textfield and misplaces a button.
- Counting and spatial reasoning: In a tkinter layout example, the generated code creates three textfields and applies additional nonsensical window configuration.The reference layout requires two textfields and three buttons, while the output includes an extra textfield and misplaced controls.
C Examples for Section 5
The examples compare code-problem similarity using AST match rates and embedding distance, showing how similar reasoning can persist despite different wording or requested outputs.
- AST match examples: AST match rate 1.0 indicates identical reasoning with substantially different prompt wording.A real-world holes-on-a-line prompt implicitly teaches the same closest-pair reasoning as another array problem.
- AST match examples: AST match rate 0.96 captures similar reasoning and coding concepts despite different requested outputs.The paired tasks return either a number pair or their average.
- AST match examples: AST match rates of 0.9 and 0.83 correspond to code pairs that become less similar.The section uses these examples to illustrate decreasing structural similarity.
- Embedding-distance example: Embedding distance 0.16 identifies problems with similar docstrings, function names, and code structure.Similarity is measured using L2 distance between normalized CodeGen-Mono 350M embeddings.