Source-linked AI summary
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample
TL;DR
Serving language models at scale makes inference efficiency important, motivating models trained on more tokens than typically used. LLaMA uses publicly available data to produce models from 7B to 65B parameters, with LLaMA-13B outperforming GPT-3 on most benchmarks and LLaMA-65B competitive with leading models.
Problem
Existing scaling-law objectives emphasize training compute while overlooking inference budgets, motivating models optimized for performance at different inference costs.
Method
LLaMA trains 7B–65B-parameter language models on more tokens than typically used, relying exclusively on publicly available data.
Results
LLaMA-13B outperforms GPT-3 on most benchmarks despite being more than 10× smaller, while LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B.
Takeaways & Limitations
The released models support open research and can make large language models more accessible, with LLaMA-13B runnable on a single GPU.
Takeaways & Limitations
The toxicity and bias evaluations are not sufficient to fully understand the risks associated with these models.
Abstract
from arXiv · showhide
We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.
1 Introduction
LLaMA trains 7B–65B language models on more tokens to optimize performance across inference budgets, achieving competitive results with publicly available data exclusively. LLaMA-13B outperforms GPT-3 on most benchmarks despite being 10× smaller, supporting broader access to LLM research.
- Inference-aware scaling: A 7B model continues improving beyond 1T tokens, challenging the recommendation to train a 10B model on 200B tokens and emphasizing inference cost in scaling decisions.For a target performance level, a smaller model trained longer can ultimately be cheaper to serve, even if a larger model is cheaper to train.
- Contribution: LLaMA-13B outperforms GPT-3 on most benchmarks despite being 10× smaller, while the model family spans 7B to 65B parameters with competitive performance.The models are designed to achieve strong performance at various inference budgets by training on more tokens than typically used.
- Data and openness: LLaMA uses only publicly available data, unlike Chinchilla, PaLM, and GPT-3, making the models compatible with open-sourcing.Existing models often rely on data that is unavailable or undocumented; OPT, GPT-NeoX, BLOOM, and GLM are noted exceptions, but none are described as competitive with PaLM-62B or Chinchilla.
- Paper scope: The paper presents transformer architecture modifications and training methods, evaluates the models against other LLMs on standard benchmarks, and examines their encoded biases and toxicity.The responsible-AI evaluation uses recent benchmarks focused on biases and toxicity.
2 Approach
LLaMA trains transformer models on a diverse, publicly available dataset using Chinchilla-inspired scaling, modern architectural modifications, and efficiency optimizations. The 1.4T-token corpus is processed with deduplication and quality filtering, while the 65B model trains in approximately 21 days on 2048 A100 GPUs.
- Training data: CommonCrawl contributes 67% and C4 contributes 15% of the mixture, with deduplication, language identification, and quality filtering applied during preprocessing.C4 was included because exploratory experiments found that diverse preprocessed CommonCrawl datasets improved performance.
- Training data: The training corpus contains roughly 1.4T tokens, with most tokens seen once and approximately two epochs applied to Wikipedia and Books.The dataset combines publicly available sources spanning web text, code, encyclopedic, book, scientific, and question-answering domains.
- Architecture: The models use transformer architectures with pre-normalization and RMSNorm, SwiGLU activations, and rotary positional embeddings instead of absolute positional embeddings.These changes are associated with training stability or performance improvements and draw on prior work by Vaswani et al. (2017), Zhang and Sennrich (2019), Shazeer (2020), and Su et al. (2021).
- Optimization: Training uses AdamW with β1 = 0.9, β2 = 0.95, cosine decay to 10% of the maximum learning rate, weight decay 0.1, gradient clipping 1.0, and 2,000 warmup steps.Learning rate and batch size vary with model size.
- Efficiency: Efficient causal attention, activation checkpointing, manual transformer backward functions, model and sequence parallelism, and overlapping computation with GPU communication reduce training memory and runtime.The attention implementation avoids storing attention weights and computing causally masked key/query scores.
- Efficiency: 380 tokens/sec/GPU enables training the 65B-parameter model on 2048 A100 GPUs with 80GB RAM over 1.4T tokens in approximately 21 days.All models use a batch size of 4M tokens; the 33B and 65B models train on 1.4T tokens, while smaller models train on 1.0T tokens.
3 Main results
Across 20 zero-shot and few-shot benchmarks, LLaMA models outperform or match substantially larger general models on many tasks, while trailing on MMLU and showing benchmark-specific training dynamics.
- Common-sense reasoning: LLaMA-65B outperforms Chinchilla-70B on all reported common-sense benchmarks except BoolQ and surpasses PaLM-540B everywhere except BoolQ and WinoGrande; LLaMA-13B outperforms GPT-3 on most benchmarks despite being 10× smaller.
- Closed-book question answering: LLaMA-65B achieves state-of-the-art performance on NaturalQuestions and TriviaQA in the closed-book zero-shot setting.Both benchmarks report exact-match performance without access to supporting documents.
- Mathematical reasoning: LLaMA-65B outperforms Minerva-62B on GSM8k despite not being fine-tuned on mathematical data.
- Code generation: For similar parameter counts, LLaMA outperforms general models on code generation: LLaMA-13B and larger surpass LaMDA-137B, and LLaMA-65B surpasses PaLM-62B.These comparisons use pass@1 on HumanEval and MBPP; the models were not fine-tuned specifically for code.
- MMLU and training dynamics: On MMLU, LLaMA-65B trails Chinchilla-70B and PaLM-540B by a few percentage points on average and across most domains.During training, performance generally improves with perplexity, except for high variance on SIQA and weak correlation on WinoGrande.
4 Instruction Finetuning
Brief instruction finetuning rapidly improves LLaMA-65B’s MMLU performance and instruction-following ability. The resulting LLaMA-I (65B) reaches 68.9% on MMLU and outperforms existing instruction-finetuned models of moderate size, despite the approach’s simplicity.
- 4 Instruction Finetuning: A very small amount of instruction finetuning improves LLaMA-65B’s MMLU performance and further improves its ability to follow instructions.The non-finetuned model already follows basic instructions, while brief finetuning produces rapid gains.
- 4 Instruction Finetuning: The study conducts a single instruction-finetuning experiment using the protocol of Chung et al. (2022).Instruction finetuning is not the paper’s primary focus.
- 4 Instruction Finetuning: 68.9% on MMLU: LLaMA-I (65B) outperforms existing instruction-finetuned models of moderate size, including OPT-IML and Flan-PaLM.The comparison uses 5-shot MMLU results reported in the corresponding papers.
5 Bias, Toxicity and Misinformation
LLaMA-65B exhibits toxicity and social biases, with toxicity increasing with model size and gender-related errors indicating reliance on occupational stereotypes. It scores higher than GPT-3 on TruthfulQA, but its low accuracy still suggests frequent misinformation hallucinations, while these benchmarks do not fully capture model risks.
- Evaluation limitations: Standard toxicity and bias evaluations provide useful risk indicators, but they are insufficient to fully characterize the risks associated with these models.The authors also note that they lack control over the third-party PerspectiveAPI pipeline, complicating comparisons with previous models.
- Toxicity: Toxicity increases with model size, especially for respectful prompts, although comparisons are difficult because studies use different sampling strategies, prompt counts, and API times.LLaMA’s RealToxicityPrompts scores are comparable to prior literature, but the authors caution that the relationship may apply only within a model family.
- Bias: LLaMA-65B compares slightly favorably with GPT-3 and OPT-175B on average bias, but its religion bias is 10% higher than OPT-175B and may originate in CommonCrawl.CrowS-Pairs evaluates gender, religion, race/color, sexual orientation, age, nationality, disability, physical appearance, and socioeconomic status.
- Bias: LLaMA-65B performs better on gender-neutral WinoGender pronouns than on gendered pronouns, likely indicating gender bias and reliance on occupations’ majority genders.The model makes more errors on “gotcha” cases where the pronoun conflicts with the occupation’s majority gender, rather than using sentence evidence.
- Misinformation: LLaMA scores higher than GPT-3 on TruthfulQA for both truthfulness and truthful-informative answers, yet its low correct-answer rate indicates likely hallucination of false claims.TruthfulQA covers 38 categories and is designed to assess risks of misinformation generation.
6 Carbon footprint
The authors estimate training energy use and carbon emissions with a common formula and assumptions, reporting that developing their models consumed approximately 2,638 MWh and emitted 1,015 tCO2eq. Carbon emissions vary with data-center electricity grids, so comparisons with OPT and BLOOM use a shared location assumption.
- 6 Carbon footprint: Carbon emissions depend on the data center’s electricity grid, with BLOOM estimated at 27 tCO2eq and OPT at 82 tCO2eq under their respective grid intensities.BLOOM’s grid emits 0.057 kg CO2eq/KWh, whereas OPT’s emits 0.231 kg CO2eq/KWh.
- 6 Carbon footprint: 2,638 MWh of energy and 1,015 tCO2eq of emissions were estimated for developing the models using 2048 A100-80GB GPUs over approximately five months.The estimate assumes a Power Usage Effectiveness (PUE) of 1.1.
- 6 Carbon footprint: Because training is already completed, releasing the models may reduce future emissions, and some models can run on a single GPU.This is presented as a potential mitigation of additional training-related emissions.
7 Related work
Language modeling evolved from n-gram statistics to neural networks and self-attention architectures, with transformers improving long-range dependency modeling. Research also established that scaling model and dataset sizes improves language-model performance, motivating increasingly large systems.
- Architecture: Language models progressed from n-gram counts and smoothing methods to feed-forward, recurrent, and LSTM neural networks (Graves, 2013).
- Architecture: Transformers based on self-attention improved language modeling, particularly for capturing long-range dependencies (Vaswani et al., 2017; Dai et al., 2019).
- Scaling: Scaling has a long history: models and datasets grew from trillion-token n-gram systems to billion-parameter LSTMs and increasingly large transformers (Jozefowicz et al., 2016).
- Scaling: Large transformer models—including BERT, GPT-2, Megatron-LM, T5, GPT-3, Gopher, Chinchilla, PaLM, OPT, and GLM—achieved improvements across many NLP tasks as scaling continued.GPT-3 (Brown et al., 2020) is identified as a 175-billion-parameter model.
- Scaling: Studies reported power-law relationships linking model and dataset sizes to system performance, including relationships derived specifically for transformer language models (Hestness et al., 2017; Rosenfeld et al., 2019; Kaplan et al., 2020).
8 Conclusion
The paper presents openly released LLaMA models that achieve competitive state-of-the-art performance, including strong comparisons against larger models using only publicly available training data.
- LLaMA models are released openly and are competitive with state-of-the-art foundation models.
- LLaMA-13B outperforms GPT-3 while being more than 10× smaller.
- LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B.
- The models achieve state-of-the-art performance using exclusively publicly available data rather than proprietary datasets.
A Question Answering
This section evaluates LLaMA on Natural Questions and TriviaQA using open-domain question-answering settings, greedy decoding, and exact-match scoring after answer normalization.
- A Question Answering: LLaMA is evaluated on Natural Questions and TriviaQA, using 3,610 open-domain Natural Questions test questions and the filtered TriviaQA development set.The TriviaQA setup differs from GPT-3 and PaLM, which used the unavailable test set of the unfiltered dataset [5].
- A Question Answering: Answers are generated with greedy decoding and truncated at the first line break, final dot, or comma before evaluation.Exact match counts a prediction as correct when it matches any normalized reference answer; normalization lowercases text and removes articles, punctuation, and duplicate whitespace.
- A Question Answering: Figure 3 illustrates the formatted one-shot dataset examples used for Natural Questions and TriviaQA.The figure shows one example for each dataset.
B MMLU
This section reports detailed 5-shot MMLU results across individual domains on the test sets.
- B MMLU: The table presents detailed 5-shot MMLU results for each domain on the test sets.
- B MMLU: Results are organized by individual MMLU domain.
- B MMLU: The reported evaluations use MMLU test sets.
C Generations from LLaMA-65B
This section presents examples generated by LLaMA-65B without instruction finetuning. The examples include mathematical explanations, recommendation-letter writing, code, and creative or fictional content.
- Generations from LLaMA-65B: LLaMA-65B generates varied open-ended content without instruction finetuning, spanning explanations, letters, code, and imaginative writing.The examples include a Fibonacci-sequence explanation, a recommendation letter, a quadratic-roots function, a fictional rap review, and character dialogue.
- Generations from LLaMA-65B: The model continues a Fibonacci-sequence prompt with historical context, naming Leonardo of Pisa and explaining the sequence’s appearance and rapid initial growth.The continuation identifies the sequence as Fibonacci’s and references Liber abaci and the idealized rabbit-population problem.
- Generations from LLaMA-65B: It writes a tailored recommendation letter for a fictional dragon-feeder job, connecting knightly dragon-fighting experience to feeding responsibilities.The letter emphasizes remembering and processing information, understanding dragons’ weaknesses, and knowing safe and preferred foods.
- Generations from LLaMA-65B: Other generations produce imaginative formats, including a fabricated Yann LeCun rap review and dialogue between Gauss and Curie.The rap review discusses deep learning and AI’s future, while the dialogue depicts the scientists discussing the meaning of “geek.”
D Generations from LLaMA-I
LLaMA-I consists of LLaMA-65B fine-tuned with the protocol and instruction dataset from Chung et al. (2022), with examples spanning dialogue, coding, factual explanation, creative writing, and responsible-use messaging.
- LLaMA-I is LLaMA-65B fine-tuned with the protocol and instruction dataset from Chung et al. (2022).
- The model also produces imaginative dialogue and stories, including conversations between the Sun and Pluto, a grain of sand’s journey, and a shouting match between Julius Caesar and Napoleon.
- The generations include practical programming assistance, providing JavaScript HTTP-request examples and Python regular expressions for removing HTML tags and extracting function definitions.
- The examples show multi-turn conversational responses that maintain context while discussing chess openings and distinguishing the Italian and Scotch Games.The model identifies their divergence after White’s third move, giving 3. Bc4 for the Italian Game and 3. Qf3 for the Scotch Game.
- One example drafts guidance urging responsible language-model use, including caution about deceptive or malicious applications and spreading models without precautions or warnings.