Source-linked AI summary
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
Ronen Eldan, Yuanzhi Li
TL;DR
Small language models often struggle to generate coherent text, leaving unclear whether the barrier is model scale or the breadth of ordinary corpora. The paper introduces TinyStories and GPT-4-based evaluation, finding that very small or shallow models can produce fluent, consistent stories and display reasoning-related capabilities, while acknowledging limits in ruling out complex template matching.
Problem
It is unclear whether small models fail at coherent language because natural language is intrinsically complex or because standard corpora are excessively broad and diverse.
Method
The paper constructs TinyStories, a GPT-3.5- and GPT-4-generated dataset of simple short stories, and evaluates model outputs with GPT-4 across multiple capabilities.
Results
Models trained on TinyStories can produce fluent, consistent, diverse stories and show reasoning, instruction-following, factual knowledge, scaling behavior, and higher interpretability at much smaller scales.
Takeaways & Limitations
TinyStories enables study of coherent generation and other language capabilities in smaller models and datasets, including preliminary analysis of how capabilities emerge.
Takeaways & Limitations
The evaluation cannot completely rule out complex template matching as an explanation for generated-story novelty.
Abstract
from arXiv · showhide
Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. This raises the question of whether the emergence of the ability to produce coherent English text only occurs at larger scales (with hundreds of millions of parameters or more) and complex architectures (with many layers of global attention). In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities. We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher. This new paradigm overcomes the flaws of standard benchmarks which often requires the model's output to be very structures, and moreover provides a multidimensional score for the model, providing scores for different capabilities such as grammar, creativity and consistency. We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.
1 Introduction
The paper asks whether coherent, fluent language requires large models and broad corpora, and introduces TinyStories to isolate the minimal requirements for language capabilities. It reports that very small or shallow models trained on this dataset can generate coherent stories and exhibit knowledge, reasoning, and instruction-following behaviors.
- Motivation: Small language models around 125M parameters often fail to generate consistent text beyond a few words despite extensive training on large corpora.These models commonly produce incoherent, repetitive, or nonsensical sentences and lose topic continuity.
- Motivation: The paper asks whether small-model failures reflect natural-language complexity or the excessive breadth and diversity of standard training corpora.The question motivates a smaller, more refined dataset for isolating language requirements.
- Contribution: TinyStories is a synthetic dataset of short stories using vocabulary and factual content intended for typical 3- to 4-year-old children.Stories contain two to three paragraphs with simple plots and consistent themes, and were generated by GPT-3.5 and GPT-4.
- Contribution: Below 10 million parameters or with one transformer block, models trained on TinyStories still produce diverse, fluent, and consistent stories while showing reasoning, factual knowledge, and instruction-following abilities.The reported models use an embedding dimension of 256, and their stories are comparable or superior to those from larger, more complex models.
- Empirical comparison: A 28M-parameter TinyStories model is compared with GPT-2XL, a 1.5B-parameter model, using essentially the same architecture and training scheme.The comparison provides an initial illustration of generation quality at substantially different scales.
2 Description of the TinyStories dataset
TinyStories is designed as a restricted synthetic corpus that retains grammar, vocabulary, facts, and reasoning while reducing linguistic breadth. Its instruction-following variant combines stories with randomly selected constraints, summaries, and required sentences or features.
- TinyStories dataset: TinyStories uses short English stories restricted to vocabulary that a typical 3-year-old child would understand.GPT-3.5 and GPT-4 generate the stories under simple-word and short-story constraints.
- TinyStories dataset: The dataset uses about 1,500 basic words divided into nouns, verbs, and adjectives, with three randomly selected words incorporated into each story.Random word selection is intended to increase diversity and cover the vocabulary familiar to a 3- to 4-year-old child.
- TinyStories dataset: Prompts can require story features such as dialogue, bad endings, moral values, plot twists, foreshadowing, or conflict.A sample prompt combines the words decorate, thunder, and ancient with dialogue and a bad-ending requirement.
- TinyStories-Instruct: TinyStories-Instruct prefixes each original story with a random subset of instructions covering words, required sentences, features, and a short summary.This construction enables prompts to compose multiple constraints for controlled story generation.
- TinyStories-Instruct: Each TinyStories-Instruct entry places the selected instructions before the corresponding story, allowing models to generate stories conditioned on those instructions.The dataset includes examples combining summaries, features, sentences, and word lists with the story text.
3 GPT-Eval: A framework for evaluating models trained on TinyStories
GPT-Eval treats story generation as a student-completion exercise and uses GPT-4 to assess grammar, creativity, consistency, age level, and related qualities. The resulting scores track evaluation loss while revealing different capability thresholds for grammar, consistency, creativity, and instruction following.
- Evaluation framework: GPT-Eval gives GPT-4 a story beginning and asks it to assess a model completion as if written by a student.The evaluation considers grammar, creativity, consistency with the beginning, plot sense, and estimated student age.
- Instruction evaluation: TinyStories-Instruct evaluation supplies GPT-4 with both instructions and generated stories, adding a Plot score for plot coherence.The instruction-evaluation set contains 50 instruction sets disjoint from training data.
- Observed trends: GPT-4 evaluation scores increase as evaluation losses decrease during training.Figure 3 tracks evaluation loss and GPT-Eval scores for GPT-Neo models with embedding dimension 768 and different layer counts.
- Observed trends: Grammar plateaus earlier than consistency and creativity, while consistency emerges as hidden size increases from 64 to 128.The paper reports that grammar can be mastered by relatively small models, whereas consistency and creativity require larger sizes.
- Observed trends: An approximately 80M-parameter TinyStories model reaches almost perfect grammar and consistency scores but remains substantially below GPT-4 in creativity.The result suggests creativity continues improving more with model and dataset size than grammar and consistency.
- Observed trends: One-layer models struggle with instruction following, whereas two layers appear sufficient for some instruction-following ability.The paper attributes instruction following as likely relying heavily on global attention and distinguishes it from plot coherence.
4 The performance of small models trained on TinyStories
TinyStories enables models with roughly 1M–35M parameters and 1–8 layers to generate increasingly coherent, context-consistent, instruction-following, and diverse stories. Performance improves with model size, embedding dimension, and depth, while diversity tests provide evidence against simple memorization but do not rule out complex template matching.
- 4 The performance of small models trained on TinyStories: Models ranging from roughly 1M to 35M parameters and 1 to 8 layers can be trained on TinyStories, with all models fitting within 30 hours on one V100 GPU.The experiments vary architecture and scale to study their effects on model quality.
- 4 The performance of small models trained on TinyStories: Larger models generate more coherent, grammatical, diverse, and creative continuations, while smaller models often repeat themselves or produce nonsensical text.The reported quality trend is consistent with GPT-4 evaluation scores.
- 4 The performance of small models trained on TinyStories: Models with more layers better preserve story consistency and logic, whereas shallow models struggle with long-term dependencies despite sometimes producing syntactically correct English.This distinguishes depth-related context tracking from surface-level grammatical correctness.
- 4.2 Knowledge, reasoning and context-tracking: Higher embedding dimensions and more layers improve factual, relevant, and natural continuations; a 33M-parameter model answers most tested prompts correctly, while a 1M-parameter model fails factual prompts.GPT2-XL performs worse than some TinyStories models across the three evaluated categories despite being much larger.
- 4.2 Knowledge, reasoning and context-tracking: Fact knowledge appears more dependent on embedding dimension, whereas context tracking appears more dependent on layer count.The one-layer model gets no consistency prompts right but some facts right, while the 64-dimensional model gets no facts right but maintains consistency several times.
- 4.4.1 Quantitative measurement of similarity using Rouge score.: Generated completions are diverse rather than simple copies: they differ from original endings, use uncommon k-grams, and remain distant from the nearest training example.Instruction-following on prompts disjoint from the dataset further supports the novelty claim.
5 Interpretability
The paper presents preliminary evidence that smaller TinyStories-trained models are more interpretable, with attention heads and MLP neurons exhibiting meaningful, human-identifiable roles. One-layer models reveal positional and semantic attention patterns, while neurons respond to linguistic and story-level functions.
- Smaller TinyStories-trained models provide preliminary evidence of greater interpretability by assigning meaningful functions to attention heads and MLP neurons.The section focuses on attention patterns and neuron activations as windows into model mechanisms.
- Attention heads: One-layer models make attention heads more directly responsible for output generation, potentially making their functions easier to interpret than those in deeper models.The analysis uses a shallow model with one transformer block and applies an existing method to classify attention patterns.
- Attention heads: Attention heads separate into distance-based heads attending to prescribed relative positions and semantic heads attending to story entities such as banana, park, Tom, and Lucy.The observed patterns include multiple positional distances and heads focused on recurring entities or semantically related tokens.
- Attention heads: The attention patterns align with linguistic roles: local heads support short-range grammatical tokens, whereas semantic attention captures entities that neighboring tokens alone cannot predict.The paper contrasts tokens such as “the,” “a,” “and,” and punctuation with story entities such as “banana,” “park,” “Lucy,” and “Tom.”
- Neurons in the MLP: MLP neurons activate on tokens sharing sentence or story roles, including subjects, actions, adjectives, and first protagonist mentions.In the 1M-parameter model, different neurons correspond to pronoun subjects, actions, adjectives, and protagonist introductions.
- Neurons in the MLP: Compared with GPT-XL, the TinyStories model shows more apparent neuron roles, although the interpretability evidence remains preliminary and non-exhaustive.The comparison reports no apparent role for two examined GPT-XL neurons, while the paper explicitly calls for more work before stronger conclusions.
6 Exploring architectures and hyperparameters for NLP with TinyStories
TinyStories serves as a compact testbed for studying how model size, training budget, architecture, and attention-head count affect language-model performance. Experiments indicate polynomial scaling with training FLOPs and improved performance from adding heads in the small-head regime.
- Dataset as a testbed: TinyStories enables training and evaluating language models that are orders of magnitude smaller than state-of-the-art models while retaining basic coherent-text generation.The dataset supports experiments with model architectures and hyperparameters at substantially reduced scale.
- Experimental questions: The experiments examine how to balance model size with learning budget for fixed training FLOPs and how to choose attention heads for a given width and depth.These questions frame the section’s architecture and scaling-law investigations.
- Model size versus training FLOPs: A polynomial scaling law appears between model size and training FLOPs, suggesting the relationship may extend to the smaller-model regime studied with TinyStories.The authors note that this result is evidence rather than a definitive conclusion about universality.
- Choosing the number of heads: For fixed model width and depth, increasing the number of attention heads improves performance across all metrics when the initial number of heads is small.The result is reported for the small-head regime rather than as a universal effect of adding heads.
7 Related Works
The related work situates TinyStories among challenges in scaling, compressing, evaluating, and theoretically analyzing generative language models. Existing benchmarks often constrain or inadequately represent open-ended natural-language generation, motivating alternative evaluation approaches.
- Generative language models: Generative language models often require hundreds of millions or billions of parameters, creating substantial training, inference, and deployment costs.GPT-3 is given as an example with 175 billion parameters and hundreds of petaflops of training compute.
- Model compression: Compression methods such as distillation, pruning, and quantization have generally been more effective for BERT-like models than for autoregressive GPT-like models.The distinction follows from the different modeling and task settings described for the two model families.
- Evaluation: Many generative-model benchmarks require single-word or short-phrase answers, limiting their ability to capture the richness and diversity of natural-language generation.Other benchmarks may be too complicated for small language models, while structured outputs are easier to evaluate but less representative.
- Transformer theory: Existing theory work often focuses on one-transformer-block models because they are easier to analyze than models with multiple blocks.Prior studies are cited as examples of analyzing linguistic tasks and topical modeling within transformer blocks.
8 Conclusion
The paper concludes that TinyStories enables small language models to generate fluent, consistent stories while supporting analysis of reasoning, interpretability, evaluation, and capability emergence. It also identifies unresolved questions about creativity and understanding and proposes refined synthetic datasets for future applications.
- Conclusion: TinyStories trains small language models that produce fluent, consistent, diverse, multi-paragraph stories with almost perfect grammar and reasoning capabilities.The dataset contains short stories using words intended to be understood by typical 3- to 4-year-old children.
- Conclusion: TinyStories enables studying coherent generation, reasoning, and instruction following at smaller model and dataset scales than large internet-trained language models.The authors argue that broad internet corpora may be too large and diverse for small models to capture complex language aspects.
- Limitations and future work: The paper leaves the extent of model creativity and story understanding unresolved because plausible continuations may still arise from template matching.The authors report evidence of genuinely new stories but identify creativity and understanding as continuing challenges.
- Conclusion: The GPT-4 grading paradigm evaluates generated stories across multiple capabilities instead of relying only on highly structured benchmark outputs.It is presented as a framework intended to address limitations of standard benchmarks.
- Conclusion: Initial findings suggest width is more important for factual knowledge, while depth is more important for contextual tracking.The authors characterize these findings as preliminary and not the main focus of the work.
- Future directions: TinyStories may support future research in low-resource or specialized domains, including synthetic datasets for practical applications such as customer-service chatbots.The paper presents these uses as possibilities for future work rather than demonstrated applications.