The Llama 3 Herd of Models

1 Introduction

• 405B parameters and up to 128K tokens of context characterize the largest Llama 3 model; the family natively supports multilinguality, coding, reasoning, and tool usage.The herd includes 8B, 70B, and 405B models, and all reported results concern Llama 3.1. Although pre-trained on multilingual data, the 8B and 70B models were originally intended for English use.Source: §1 · pp. 1–2
• Approximately 15T multilingual pre-training tokens replace Llama 2’s 1.8T, alongside improved data quality relative to earlier Llama models (Touvron et al., 2023a,b).Pre-training improvements emphasize preprocessing and curation, while post-training improvements emphasize more rigorous quality assurance and filtering.Source: §1 · p. 1
• 3.8 × 10^25 FLOPs pre-train the flagship model, almost 50× the compute used for the largest Llama 2 model.The 405B model trains on 15.6T text tokens and is approximately compute-optimal for its budget. Smaller models train beyond compute-optimal durations to improve performance at the same inference budget, then receive further post-training improvements using the flagship.Source: §1 · pp. 1–2
• Llama 3 favors a standard dense Transformer (Vaswani et al., 2017) and relatively simple post-training to prioritize stability and scalability.Post-training combines supervised finetuning, rejection sampling, and direct preference optimization (DPO; Rafailov et al. (2023)).Source: §1 · p. 2
• The public release includes all three language models under an updated Llama 3 Community License, plus an updated Llama Guard for input and output safety.The release includes both pre-trained and post-trained versions of the 405B language model; Llama Guard builds on Inan et al. (2023).Source: §1 · p. 2
• Multimodal extensions add image recognition, video recognition, and speech understanding, but remain under active development and are not ready for release.The paper reports initial experiments with these extensions alongside its language-modeling results.Source: §1 · p. 2
• The flagship Llama 3 performs on par with leading models such as GPT-4 (OpenAI, 2023a) across varied tasks.Smaller models outperform similarly sized alternatives (Bai et al., 2023; Jiang et al., 2023). Llama 3 also improves the balance between helpfulness and harmlessness over its predecessor (Touvron et al., 2023b).Source: §1 · p. 2, Table 2 · p. 3

Language model pre-training

• 405B parameters are pre-trained on 15.6T tokens with an 8K-token context, followed by continued pre-training extending context to 128K tokens.The Transformer learns through next-token prediction on tokenized multilingual text.Source: §2 · p. 3, Figure 1 · p. 4

Language model post-training

• Post-training aligns the model through repeated supervised finetuning and Direct Preference Optimization (DPO; Rafailov et al., 2024), while adding tool use and safety mitigations.The authors also observe strong improvements in coding and reasoning during post-training.Source: §2 · p. 3

Model capabilities

• At least eight languages are supported for question answering, alongside high-quality code generation, complex reasoning, and out-of-the-box or zero-shot tool use.Experimental multimodal models also recognize image and video content and support speech interaction, but remain under development and are not ready for release.Source: §2 · pp. 3–4

Multi-modal encoder pre-training

• Separate image and speech encoders learn from image-text pairs and self-supervised reconstruction of masked speech inputs using discrete-token representations, respectively.These objectives teach relationships between images and language and the structure of speech signals.Source: §2 · pp. 3–4

Vision adapter training

• Vision adapters connect image representations through cross-attention and aggregate video frames, supporting experimental image and video recognition.Image-adapter training uses text-image pairs and updates the encoder but freezes the language model; a video adapter is trained on paired video-text data.Source: §2 · p. 4

Speech adapter training

• A speech adapter converts encoder outputs into language-model token representations, while an integrated text-to-speech system enables experimental speech interaction.Supervised finetuning jointly updates the speech encoder and adapter while keeping the language model unchanged.Source: §2 · p. 4

3.1.1 Web Data Curation

• Web curation combines safety filtering, structure-preserving HTML extraction, multilevel deduplication, and model-based document quality scoring.Filters exclude unsafe or PII-heavy domains; a custom parser preserves mathematics and code while removing markdown, which performed worse than plain text. Deduplication retains newest URL versions, removes near-duplicates using MinHash, and removes lines occurring more than six times per 30M-document bucket despite some high-quality text loss; DistilRoberta (Sanh et al., 2019) scores quality using Llama 2-generated labels.Source: §3.1.1 · pp. 4–5
• Dedicated code, reasoning, and multilingual pipelines supplement general web curation with domain-specific extraction, filtering, and quality ranking.Code and reasoning classifiers use Llama 2-annotated DistilRoberta models targeting mathematical deduction, STEM reasoning, and code interleaved with natural language, with specialized extraction and heuristics. Multilingual processing identifies 176 languages, deduplicates within languages, applies language-specific filters, and ranks documents using a multilingual Llama 2-based classifier; token allocation balances English and multilingual benchmark performance.Source: §3.1.1 · pp. 5–6

3.1.2 Determining the Data Mix

• 50% general knowledge, 25% mathematics and reasoning, 17% code, and 8% multilingual tokens form the final approximate pre-training data mix.Knowledge classification downsamples overrepresented web categories such as arts and entertainment. Repeated small-model scaling experiments predict larger-model performance for candidate mixtures, which are subsequently evaluated using a larger model on key benchmarks.Source: §3.1.2 · p. 6

3.1.3 Annealing Data

• 24.0% GSM8k (Cobbe et al., 2021) and 6.4% MATH (Hendrycks et al., 2021b) validation gains followed benchmark-training-set annealing for 8B; 405B gains were negligible.The standard annealing mixture instead upsamples high-quality code and mathematics without including common benchmark training sets. Data-quality tests, similar to Blakeney et al. (2024), anneal a half-trained 8B model’s learning rate linearly to zero over 40B tokens, mixing 30% candidate data with 70% default data.Source: §3.1.3 · p. 6

3.2 Model Architecture

• 3.94 versus 3.17 characters per token improves sampled-English compression over Llama 2; 28K additional non-English tokens improve multilingual compression and downstream performance.The dense Transformer (Vaswani et al., 2017) uses a 128K-token vocabulary and grouped query attention (GQA; Ainslie et al. (2023)) with eight key-value heads for faster inference and smaller caches. Cross-document attention masking matters for long-sequence continued pre-training, while a RoPE base frequency of 500,000 supports longer contexts.Source: §3.2 · pp. 6–7

3.2.1 Scaling Laws

• Four orders of magnitude of extrapolation yielded an ARC Challenge forecast that only slightly underestimated the flagship Llama 3 model’s final performance.The forecast maps training FLOPs to task negative log-likelihood and then to accuracy using scaling models and Llama 2 models. IsoFLOPs minima identified compute-optimal models; extrapolation to 3.8 × 10^25 FLOPs suggested 402B parameters and 16.55T tokens, with flatter minima supporting the final 405B choice.Source: §3.2.1 · p. 8, Figure 4 · p. 9

3.3.1 Training Infrastructure

• Up to 16K H100 GPUs train Llama 3 405B, supported by high-throughput checkpoint storage and topology-aware networking.Storage provides 240 PB capacity, 2 TB/s sustained throughput, and 7 TB/s peak throughput for bursty checkpointing. The 24K-GPU RoCE cluster uses 400 Gbps interconnects, full-bandwidth 3,072-GPU pods with 1:7 cross-pod oversubscription, topology-aware scheduling, E-ECMP balancing of 16 flows, and deep buffers to operate without traditional congestion control.Source: §3.3.1 · pp. 9–10

3.3.2 Parallelism for Model Scaling

• 38-43% BF16 Model FLOPs Utilization was achieved through tuning, using four-dimensional parallelism combining tensor, pipeline, context, and fully sharded data parallelism.Utilization fell from 43% on 8K GPUs with DP=64 to 41% on 16K GPUs with DP=128 because each data-parallel group received a smaller batch. Context parallelism supports 128K-token sequences using two balanced chunks per rank and all-gathered keys and values; flexible attention masks are supported, while GQA (Ainslie et al., 2023) and O(S^2) attention versus O(S) communication make all-gather overhead negligible.Source: §3.3.2 · pp. 10–12
• Flexible micro-batch scheduling and balanced pipeline stages address batch-size constraints and memory and computation imbalances.Tunable contiguous micro-batch counts permit fewer micro-batches than pipeline stages or more to hide communication, while removing one Transformer layer from each end stage balances embedding and output overhead. Interleaving (Narayanan et al., 2021), asynchronous point-to-point communication, memory-use controls, and proactive tensor deallocation enable 8K-token training without activation checkpointing.Source: §3.3.2 · pp. 10–11

3.3.3 Collective Communication

• NCCLX improves communication on high-latency networks through latency-aware transfer tuning and prioritized control messages.The [TP, CP, PP, DP] ordering places bandwidth- and latency-sensitive dimensions inward, while outer FSDP tolerates latency through asynchronous weight prefetching and gradient reduction. A memory estimator and performance-projection tool guide parallelism configurations to limit communication overhead and avoid GPU memory overflow.Source: §3.3.3 · p. 12, §3.3.2 · p. 12

3.3.4 Reliability and Operational Challenges

• Higher than 90% effective training time was maintained despite daily maintenance interruptions and frequent failures, with automation handling nearly all incidents.A 54-day snapshot contained 466 interruptions: 47 planned and 419 unexpected, with approximately 78% of unexpected interruptions attributed to confirmed or suspected hardware issues. GPU issues accounted for 58.7% of unexpected interruptions, and significant manual intervention was required only three times.Source: §3.3.4 · p. 13, Table 5 · p. 13
• 1-2% diurnal throughput variation arose from temperature-dependent GPU voltage and frequency scaling, alongside diagnostic and power-management challenges at scale.Reduced startup and checkpointing time, communication tracing, NCCLX–PyTorch state sharing, automatic stall timeouts, and prioritization of suspect communications support efficient operation and failure or straggler diagnosis. Synchronized GPU activity can cause instantaneous power fluctuations of tens of megawatts, an ongoing challenge for larger models.Source: §3.3.4 · pp. 13–14

3.4.1 Initial Pre-Training

• Initial pre-training gradually increases batch size and sequence length while adjusting the data mix for multilinguality, mathematics, recency, and quality.AdamW uses a peak learning rate of 8 × 10^-5, 8,000 warm-up steps, and cosine decay to 8 × 10^-7 over 1,200,000 steps. Batch sizes grow from 4M to 8M to 16M tokens, sequence length increases from 4,096 to 8,192, and training exhibits few loss spikes without interventions for divergence.Source: §3.4.1 · p. 14

3.4.2 Long Context Pre-Training

• 128K-token context is reached through six incremental extensions from 8K, using approximately 800B training tokens.Long-sequence training is deferred because self-attention computation grows quadratically with sequence length. Each extension requires complete recovery on short-context evaluations and perfect needle-in-a-haystack performance at the increased length.Source: §3.4.2 · p. 14

3.4.3 Annealing

• 40M final training tokens combine learning-rate annealing to zero, 128K context, high-quality data upsampling, and Polyak (1991) checkpoint averaging.Averaging checkpoints during annealing produces the final pre-trained model.Source: §3.4.3 · p. 15

4 Post-Training

• Each post-training round applies supervised finetuning (SFT) followed by Direct Preference Optimization (DPO; Rafailov et al., 2024).Several rounds align a pretrained checkpoint with human feedback.Source: §4 · p. 15
• Post-training examples come from human annotations or synthetic generation.The approach combines post-training modeling and data strategies.Source: §4 · p. 15
• Custom data curation targets reasoning, coding, factuality, multilingual capabilities, tool use, long context, and precise instruction following.Section 4.3 details these capability-specific curation strategies.Source: §4 · p. 15

4.1.1 Chat Dialog Format

• A multi-message chat protocol supports tool use through header tokens identifying message sources and destinations and termination tokens controlling human–AI turn-taking.A single dialog turn can contain multiple messages directed to different destinations, such as the user or ipython.Source: §4.1.1 · pp. 15–16

4.1.2 Reward Modeling

• Reward modeling removes the loss margin and jointly processes randomly shuffled, ranked responses, improving training efficiency without reducing accuracy.The model uses preference data after filtering similar-response samples; some examples include an edited response, yielding edited > chosen > rejected rankings. Prompts and their responses are concatenated into single training rows.Source: §4.1.2 · p. 16

4.1.3 Supervised Finetuning

• Supervised finetuning trains pretrained checkpoints on reward-selected and synthetic targets before DPO alignment.Training uses cross-entropy loss on target tokens while masking prompt tokens, even though many targets are model-generated. The largest models use a learning rate of 10^-5 for 8.5K–9K steps, settings that work across rounds and data mixes.Source: §4.1 · p. 15, §4.1.3 · p. 16

4.1.4 Direct Preference Optimization

• DPO (Rafailov et al., 2024) requires less compute than PPO (Schulman et al., 2017) and performs better, especially on IFEval (Zhou et al., 2023).Training primarily uses recent preference batches collected with the previous rounds’ best-performing models, better matching the policy being optimized. The learning rate is 10^-5 and β is 0.1.Source: §4.1.4 · p. 16
• Masking formatting tokens and adding chosen-response negative log-likelihood regularization stabilize DPO training.Header and termination tokens are excluded from both responses’ losses to avoid tail repetition and abrupt termination-token generation. The additional NLL term has coefficient 0.2, helping preserve formatting and prevent decreasing chosen-response log probabilities (Pang et al., 2024; Pal et al., 2024).Source: §4.1.4 · p. 16

4.1.5 Model Averaging

• Model averaging combines data and hyperparameter variants at each RM, SFT, and DPO stage (Izmailov et al., 2019).The averaged models come from experiments using different data versions or hyperparameter settings.Source: §4.1.5 · p. 16

4.1.6 Iterative Rounds

• Six rounds repeat the alignment procedure, collecting new preference annotations and SFT data while sampling synthetic data from the latest models.This iterative approach follows Llama 2.Source: §4.1.6 · p. 17

4.2.1 Preference Data

• Reward modeling uses all available preference data, while DPO uses recent batches; both retain only pairs rated significantly better or better.Annotators compare responses from different models using four preference levels and can improve the chosen response, sometimes producing edited > chosen > rejected rankings. Human evaluation informs annotator feedback and increasingly complex prompts targeting model weaknesses.Source: §4.2.1 · p. 17

4.2.2 SFT Data

• SFT combines human-annotation prompts with rejection-sampled responses, capability-targeted synthetic data, and small amounts of human-curated data.Each round adjusts the mix across topic, complexity, and quality to tune benchmark performance. The final mix repeats some high-quality sources across multiple epochs and downsamples others.Source: §4.2.2 · pp. 17–18
• 10 to 30 outputs per prompt are typically sampled, with a reward model selecting the best candidate, following Bai et al. (2022).Sampling uses the latest chat policy, usually the best previous-round or capability-specific checkpoint. Later rounds introduce system prompts to control tone, style, and formatting for different capabilities.Source: §4.2.2 · p. 18
• Over 2× rejection-sampling throughput improvement comes from adapting PagedAttention with memory-aware admission and shared prompt caches.Requests run only when memory can accommodate a predefined maximum output length, avoiding swap overhead. Key-value cache pages for each prompt are shared across its sampled outputs.Source: §4.2.2 · p. 18

4.2.3 Data Processing and Quality Control

• Rule-based removal and editing address undesirable styles, while reward-model and Llama-based filters retain examples judged high quality by either.High quality means a top-quartile reward-model score or maximum Llama rating on a three-point scale for general English or two-point scale for coding. These signals frequently disagree, but combining them yields the best recall on the internal test set.Source: §4.2.3 · pp. 18–19
• Topic classification, difficulty scoring, and semantic deduplication organize data and prioritize high-quality, complex, nonredundant dialogues.A finetuned Llama 3 8B classifies coarse and fine topics; difficulty uses Llama 3 70B intention counts through Instag and three-point Llama ratings (Liu et al., 2024c). Semantic deduplication (Abbas et al., 2023; Liu et al., 2024c) clusters dialogues using RoBERTa (Liu et al., 2019b), sorts by quality score × difficulty score, and greedily retains examples below a within-cluster cosine-similarity threshold.Source: §4.2.3 · pp. 18–19

4.3.1 Code

• Over 2.7M synthetic coding examples support SFT, with execution feedback addressing the failure of training on 405B’s own generated data to improve performance.A code expert continued-pretrained on 1T tokens with >85% code, then extended to 16K context, supplies annotations and rejection samples; 8B and 70B models benefit from larger-model data. Parser, linter, and test feedback drives revisions, only dialogs passing all checks enter SFT, and about 20% of solutions are initially incorrect but self-correct.Source: §4.3.1 · pp. 19–20
• Programming-language translation improves less-common-language performance on MultiPL-E, while backtranslation supplies documentation, explanation, generation, and debugging data.Translations undergo syntax parsing, compilation, and execution checks. Approximately 1.2M backtranslation dialogs are generated, with reconstructed code compared against the original to select examples with the highest self-verification scores.Source: §4.3.1 · pp. 20–21
• Code-specific system prompts improve rejection-sampled code quality; model-as-judge filtering retains correct, well-styled samples while revising difficult examples to avoid benchmark regressions.For responses that need not be executable, earlier Llama 3 versions assign binary correctness and style scores, retaining only samples totaling 2. Filtering initially removed too many challenging prompts and regressed benchmarks, prompting revisions that preserve both quality and difficulty.Source: §4.3.1 · pp. 21–22

4.3.2 Multilinguality

• Translated synthetic quantitative-reasoning data produces strong MGSM (Shi et al., 2022) gains, complementing multilingual expert training and instruction tuning across seven languages.An expert continued-pretrained on 90% multilingual tokens supports annotation; instruction data targets German, French, Italian, Portuguese, Hindi, Spanish, and Thai. Machine translation is generally avoided because of translationese and bias concerns, with quantitative reasoning an exception; final-round sampling uses temperature 0.6 to balance creativity against unwanted code-switching.Source: §4.3.2 · pp. 22–23

4.3.3 Math and Reasoning

• Mathematical training combines targeted prompts, answer-filtered and self-verified reasoning traces, and outcome/stepwise reward models, using Monte Carlo Tree Search for harder prompts.Prompts come from mathematical pre-training material converted into questions and answers, plus human contributions targeting underperforming skills. Reward models remove traces with invalid intermediate steps and guide search toward valid solutions.Source: §4.3.3 · p. 23
• Code-execution feedback and iterative correction of failed reasoning traces improve mathematical reasoning and learning from mistakes.Training solutions interleave textual reasoning with Python, using execution feedback to eliminate invalid reasoning chains. Incorrect generations are also converted into correction exercises to simulate feedback during inference.Source: §4.3.3 · pp. 23–24

4.3.4 Long Context

• 0.1% synthetic long-context data mixed with short-context SFT data optimizes performance across short- and long-context benchmarks after context expansion from 8K to 128K tokens.Short-context-only SFT causes long-context regressions, so synthetic examples cover question answering, document summarization, and code-repository reasoning. Short-context-only DPO preserves long-context performance when applied to a sufficiently capable long-context SFT checkpoint.Source: §4.3.4 · p. 24

4.3.5 Tool Use

• Llama 3 is trained to use Brave Search for questions about post-cutoff events or information requiring specific web retrieval.Source: §4.3.5 · p. 24
• Llama 3 can plan sequential tool calls and reason between them in multi-turn conversations, using Python for computation and uploaded files, and Wolfram Alpha for mathematical queries.Python supports file-based question answering, summarization, analysis, and visualization; Wolfram Alpha also supports science problems and database retrieval.Source: §4.3.5 · pp. 24–25
• Tool training bootstraps capabilities with synthetic data before collecting progressively harder human feedback on individual assistant messages, covering both tool calls and reasoning about outputs.Annotations progress from single-turn use to dialogs, multi-step tasks, data analysis, and unsatisfactory tool responses; annotators cannot edit tool outputs. Rejection sampling showed no benchmark gains, while additional examples teach activation-dependent tool use and tool-free answers to simple queries.Source: §4.3.5 · pp. 25–26
• Synthetic tool-training data covers single-step trajectories, interleaved multi-step reasoning, and zero-shot single, nested, parallel, and multi-turn function calling.Around 30% of single-step data is filtered for execution or formatting issues; functions mined from the Stack (Kocetkov et al., 2022) ground function-calling queries. Role-specialized Llama 3 agents generate multi-turn API dialogs, and zero-shot evaluation uses unseen tools.Source: §4.3.5 · pp. 25–26

4.3.6 Factuality

• Knowledge probing trains Llama 3 to answer within its knowledge and refuse uncertain questions, using correctness and informativeness judgments to identify consistently informative but incorrect responses.Generated training data aligns responses with factual subsets of pre-training data. A limited labeled factuality dataset additionally addresses sensitive topics with prevalent contradictory or incorrect statements.Source: §4.3.6 · p. 27

4.3.7 Steerability

• Steerability training targets system-prompt control over response length, format, tone, and persona, using preferences collected from conversations that test sustained instruction following.The collected preferences feed reward modeling, rejection sampling, SFT, and DPO.Source: §4.3.7 · p. 28

• The evaluations examine the performance of Llama 3’s pre-trained language model.Source: §5 · p. 28
• The evaluations also examine the performance of Llama 3’s post-trained language model.Source: §5 · p. 28
• A separate evaluation subsection addresses Llama 3’s safety characteristics.Source: §5 · p. 28

5.1.1 Standard Benchmarks

• 95% confidence intervals lower-bound actual capability-estimate variation and are omitted for benchmark scores that are not simple averages.Following Madaan et al. (2024b), intervals assume Gaussian scores, with preliminary bootstrap support for discrete metrics. Comparisons select the better of reproduced and reported competitor scores under comparable or more conservative settings; unavailable competitor results prevent 405B category averages.Source: §5.1.1 · p. 29
• Llama 3 8B outperforms competitors in virtually every evaluated category, while Llama 3 70B surpasses Llama 2 70B and Mixtral 8x22B.The 8B advantage holds for category-average performance and per-category win rates. Improvements over Llama 2 70B are large on most benchmarks, except likely saturated commonsense benchmarks.Source: §5.1.1 · p. 29, Figure 12 · p. 30
• Llama 3 405B performs competitively with similarly sized models and substantially outperforms prior open-source models.Source: Table 9 · p. 30, Table 10 · p. 30, §5.1.1 · p. 30

5.1.2 Model Robustness

• Pre-trained models remain robust to MMLU label variants, few-shot label bias, answer order, and prompt format, particularly Llama 3 405B.Source: §5.1.2 · pp. 30, 32, 33, Figure 13 · p. 32, Figure 14 · p. 32

5.1.3 Adversarial Benchmarks

• Adversarial question answering and mathematical reasoning substantially reduce performance, whereas PAWS paraphrase adversariality appears ineffective against both pre-trained and post-trained models.The adversarial-versus-non-adversarial performance pattern is similar before and after post-training.Source: Figure 15 · p. 33, §5.1.3 · p. 33

5.1.4 Contamination Analysis

• Contamination analysis uses 8-gram overlap and dataset-specific thresholds selected to maximize significant estimated performance gains across the three model sizes.Following Singh et al. (2024), the method compares full-dataset performance with the estimated clean subset, but detection can produce false positives and negatives. Overlap scores are too high to reliably estimate gains for MBPP, HumanEval, MMLU, and MMLU-Pro.Source: §5.1.4 · pp. 33–34
• 52% estimated contamination in Natural Questions has virtually no performance effect, whereas PiQA and HellaSwag show high estimated contamination and performance gains.For SQuAD and MATH, low detection thresholds produce high contamination estimates without performance gains, suggesting either unhelpful contamination or a need for longer matching sequences.Source: §5.1.4 · p. 34

5.2.1 General Knowledge and Instruction-Following Benchmarks

• Llama 3 8B and 70B outperform similarly sized models on knowledge tasks; 405B surpasses GPT-4 and Nemotron 4 340B, while Claude 3.5 Sonnet leads.Knowledge evaluation uses 5-shot MMLU (Hendrycks et al., 2021a) without chain-of-thought and 5-shot chain-of-thought MMLU-Pro (Wang et al., 2024b). All Llama 3 variants outperform comparable models on IFEval (Zhou et al., 2023), measured by averaged prompt-level and instruction-level accuracy under strict and loose constraints.Source: §5.2.1 · p. 35

5.2.2 Proficiency Exams

• Llama 3 405B performs similarly to Claude 3.5 Sonnet and GPT-4 4o on proficiency exams; 70B surpasses GPT-3.5 Turbo and often Nemotron 4 340B.Evaluations use official exams with multiple-choice and generation questions, exclude image-containing questions, and use few-shot prompting when multiple exam sets exist. GRE scores are scaled to 130–170, other exams report accuracy, and GRE questions with multiple correct options require selecting every correct option.Source: §5.2.2 · pp. 35–36, Table 17 · p. 36

5.2.3 Coding Benchmarks

• Llama 3 8B and 70B outperform similarly sized models on Python benchmarks; 405B performs similarly to Claude 3.5 Sonnet and GPT-4o, with GPT-4o strongest.Code-generation results report pass@1. Performance drops significantly on non-Python MultiPL-E, which translates HumanEval and MBPP problems into other programming languages.Source: §5.2.3 · pp. 36–37, Table 18 · p. 37, Table 19 · p. 37

5.2.4 Multilingual Benchmarks

• 91.6% average MGSM performance puts Llama 3 405B ahead of most models, while its multilingual MMLU performance trails GPT-4o by 2%.The 8B and 70B variants lead competitors by wide margins on both tasks; evaluation uses 0-shot chain-of-thought MGSM (Shi et al., 2022) and 5-shot MMLU (Hendrycks et al., 2021a) translated into seven languages. Llama 3 supports eight languages and is not optimized or safety-tuned for others.Source: §5.2.4 · pp. 37–38, Table 20 · p. 38

5.2.5 Math and Reasoning Benchmarks

• Llama 3 405B leads its category on GSM8K and ARC-C, ranks second on MATH, and is competitive with GPT-4 4o on GPQA.The 8B model outperforms similarly sized models on GSM8K, MATH, and GPQA, while 70B significantly outperforms its class across all evaluated math and reasoning benchmarks. Claude 3.5 Sonnet leads GPQA by a significant margin.Source: §5.2.5 · p. 38

5.2.7 Tool Use Performance

• Llama 3 variants lead their counterparts on Nexus; 8B and 70B lead API-Bank size classes, while 405B and 70B rank close second on BFCL.On API-Bank, 405B trails Claude 3.5 Sonnet by 0.6%; on BFCL, 8B leads its category. These are zero-shot function-calling evaluations using Nexus, API-Bank (Li et al., 2023b), and BFCL.Source: §5.2.7 · p. 38

5.2.6 Long Context Benchmarks

• 100% needle retrieval across all document depths and context lengths accompanies near-perfect multi-needle retrieval; Llama 3 405B leads InfiniteBench En.QA and En.MC.Multi-needle extends Needle-in-a-Haystack by inserting four needles and requesting two, averaging recall across ten sequence lengths up to 128k. Llama 3 405B and 70B match or surpass other models on ZeroSCROLLS (Shaham et al., 2023) validation tasks, while 405B gains are particularly significant on InfiniteBench (Zhang et al., 2024) En.QA.Source: §5.2.6 · p. 38, Table 21 · p. 39

5.2.7 Tool Use Performance

• Llama 3 405B significantly outperforms GPT-4o on text-only code execution and plot generation, but underperforms on file uploads.Human evaluation uses 2000 prompts covering these three use cases, comparing against GPT-4o through OpenAI’s Assistants API.Source: §5.2.7 · p. 39

Prompt collection

• About 7,000 prompts cover six individual capabilities and three multiturn capabilities, with uniform sampling across subcategories.Individual capabilities are English, reasoning, coding, Hindi, Spanish, and Portuguese; multiturn capabilities are English, reasoning, and coding. Prompts are roughly 10% easy, 30% medium, and 60% hard, undergo quality assurance, and remain inaccessible to modeling teams to prevent contamination or overfitting.Source: §5.3 · pp. 39–40

Results

• Llama 3 405B outperforms GPT-4o on code execution without plotting or file uploads and on plot generation, but trails on file-upload tasks.Figure 16 reports these human-evaluation comparisons.Source: Figure 16 · p. 40

Evaluation process

• 7-point pairwise preference ratings yield capability-specific win rates, counting only “better” or “much better” judgments as wins.Human evaluations assess aspects beyond standard benchmarks, including tone, verbosity, nuance, and cultural context, to better reflect user experience.Source: §5.3 · pp. 39–40

Results

• Llama 3 405B approximately matches GPT-4 (0125 API), with mixed results against GPT-4o and Claude 3.5 Sonnet.Against GPT-4, nearly all capability win rates are within the margin of error; Llama outperforms it on multiturn reasoning and coding but underperforms on Hindi, Spanish, and Portuguese. The authors characterize Llama 3 405B as the best-performing openly available model.Source: §5.3 · p. 40
• Llama matches GPT-4o on English and Claude 3.5 Sonnet on multilingual prompts, while outperforming Sonnet on single- and multiturn English.It trails Claude 3.5 Sonnet on coding and reasoning. Human preferences are heavily influenced by tone, response structure, and verbosity, which the authors target during post-training.Source: §5.3 · p. 40

Limitations

• Human evaluations remain vulnerable to annotators’ personal biases, backgrounds, and preferences despite thorough quality assurance.Difficulty defining objective response-quality criteria can produce inconsistent or unreliable results.Source: §5.3 · p. 40

5.4.1 Benchmark Construction

• Over 4000 prompts per capability or language test safety violations and false refusals, using adversarial and borderline examples across single-turn and multi-turn interactions.Internal benchmarks address insufficient external coverage and draw on the ML Commons hazard taxonomy (Vidgen et al., 2024). Adversarial prompts measure violations, while borderline prompts test whether models remain helpful when a plausible safe response exists.Source: §5.4.1 · p. 41

5.4.2 Safety Pre-training

• 1.13% and 3.91% average verbatim memorization rates were measured for pre-trained Llama 3 405B at n = 50 and n = 1000, respectively.Pre-training includes filters for likely personally identifiable information; memorization testing measures exact ground-truth continuation inclusion across languages, domains, and sequence lengths, finding rates roughly comparable to equivalently sized Llama 2. Limitations include exact-match measurement and prompt-search choices, with alternatives discussed by Ippolito et al. (2023) and Kassem et al. (2024).Source: Table 24 · p. 42, §5.4.2 · p. 42, §5.4.3 · p. 42

5.4.3 Safety Finetuning

• 8B models require proportionally more safety data than 70B models, while high-quality adversarial and borderline examples help balance violation and false refusal rates.Ablations find data quality more important than quantity, motivating AI-assisted annotation and rigorous quality assurance for human-generated examples. SFT combines helpfulness and safety data, with carefully balanced adversarial and borderline examples and greater borderline emphasis in challenging risk areas.Source: Figure 18 · p. 43, §5.4.3 · p. 43
• Safety DPO, synthetic adversarial examples, and refusal-tone refinement complement supervised safety training.DPO uses adversarial and borderline preferences, nearly orthogonal response pairs, and model-size-specific data mixes; synthetic generation includes guided mutation and Rainbow Teaming (Samvelyan et al., 2024), based on MAP-Elites. Refusal guidelines, rewriting, human editing, and a tone classifier improve response wording.Source: §5.4.3 · p. 43

5.4.4 Safety Results

• Llama 405B improves on Comp. 2’s long-context safety results and is safer than Comp. 1 on long-context and search tasks, with higher false refusals.Targeted SFT mitigates many-shot jailbreaking through 256-shot attacks with little to no impact on FRR or most helpfulness metrics. English safety knowledge does not readily transfer across languages, requiring high-quality language-specific data and iterative balancing of adversarial and borderline examples.Source: Figure 20 · p. 44, §5.4.4 · pp. 45–46
• Llama 3 achieves competitive violation rates with low false refusals; its 405B model with Llama Guard matches or exceeds competing systems’ safety across supported languages.Without Llama Guard, Llama 405B has significantly lower violation rates than the competing standalone open-source model, but higher false refusals. These internal benchmarks are not externally reproducible, and competitors are anonymized.Source: Figure 19 · p. 44, §5.4.4 · pp. 44–45, Figure 21 · p. 45

5.4.5 Cybersecurity and Chemical/Biological Weapons Safety

• 10.4% malicious-prompt compliance for Llama 3 405B versus 3.8% for 70B, and 21.7% prompt-injection success against 405B, reveal cybersecurity vulnerabilities.Larger models generate more insecure code despite higher average BLEU scores; vulnerability identification and exploitation do not outperform traditional tools, and autonomous attack tests achieve no successful exploitation or post-exploitation. An LLM judge rates both models moderately persuasive in spearphishing, judging 70B successful in 24% of attempts and 405B in 14%.Source: §5.4.5 · p. 46
• 62 volunteers showed insignificant cyberattack uplift from Llama 3 405B assistance over internet-only access, across novice and expert cohorts.Chemical/biological testing uses six-hour attack-planning scenarios with two-person low- or moderate-skill teams, comparing internet-only access against Llama-assisted access with search, retrieval, and code execution. Domain experts assess the plans across attack stages and consolidate scientific and operational scores; this supplied part reports the design, not outcomes.Source: §5.4.5 · p. 47
• No significant performance uplift over web-only controls was found, supporting a low assessed risk of Llama 3 releases increasing chemical or biological weapons threats.The finding held in aggregate and separately for 70B and 405B models and chemical and biological scenarios; CBRNE experts validated the results.Source: §5.4.5 · pp. 47–48

5.4.6 Red Teaming

• Multi-turn conversations generally produced more harmful outputs, and models struggled to recover once they began generating violating content.Obfuscated or layered prompts and combined disclaimer attacks exposed failures that informed targeted mitigations. Expert child-safety assessments expanded evaluation benchmarks and informed fine-tuning.Source: §5.4.6 · pp. 48–49
• Mixed-language prompts increased violations; low-resource languages revealed safety gaps, while misunderstood cultural references complicated judgments about harmfulness.Low-resource failures reflected insufficient safety fine-tuning data, weak safety generalization, or testing priorities. Poor overall output quality often limited their practical adversarial usefulness.Source: §5.4.6 · pp. 48–49
• Tool attacks included unsafe chaining, forced calls with fragmented or encoded inputs, and parameter manipulation; several failures occurred in early checkpoints.Attackers could also access tool results even when the model would normally refuse the search or assistance with its results.Source: §5.4.6 · p. 49

5.4.7 System Level Safety

• 8B-parameter Llama Guard 3 classifies prompt and response safety across languages and tool use, including code interpreter abuse.Training covers 13 AI Safety taxonomy categories (Vidgen et al., 2024) plus Code Interpreter Abuse; sample cleaning combines human and Llama 3 annotations, with humans slightly better on borderline prompts. Int8 quantization reduces size by more than 40% with negligible performance impact.Source: §5.4.7 · pp. 49–50, Table 27 · p. 51
• 65% fewer violations on average across benchmarks came with increased false refusals when using Llama Guard 3 compared with base Llama 3.Table 25 evaluates filtering on generations from 405B Llama 3. Developers can enable individual harm categories to adjust the violation–false-refusal tradeoff.Source: Table 25 · p. 50, §5.4.7 · p. 50, Table 26 · p. 51
• Released safeguards include Prompt Guard for jailbreaks and indirect prompt injections, and Code Shield for filtering insecure generated code through static analysis.Prompt Guard uses mDeBERTa-v3-base with 86M parameters and generalizes well across evaluated distributions, including multilingual tests. Code Shield uses the Insecure Code Detector across seven programming languages.Source: §5.4.7 · p. 50

5.4.8 Limitations

• Safety testing cannot exhaustively identify risks: Llama 3 may still generate harmful content, especially beyond English or under skilled adversarial prompting.Malicious developers and users may discover new jailbreaks; the authors encourage responsibility throughout development and deployment and contributions to the released safety tools.Source: §5.4.8 · p. 51

6.1 Pipeline Parallelism

• 16 GPUs across two machines support BF16 inference, using tensor parallelism within machines and pipeline parallelism between machines.Llama 3 405B does not fit on one eight-H100 machine in BF16. High NVLink bandwidth supports tensor parallelism (Shoeybi et al., 2019), while lower-bandwidth, higher-latency inter-node connectivity motivates pipeline parallelism.Source: §6.1 · pp. 51–52
• Micro-batching improves pipeline-parallel inference throughput at the same local batch size, yielding a better throughput–latency trade-off despite increased latency.Tests use two micro-batches with 4,096 input tokens and 256 output tokens; concurrent execution improves pre-fill and decoding throughput, while additional synchronization increases latency. Inference avoids backward-pass pipeline flushes, so training-style pipeline bubbles are not a concern.Source: Figure 24 · p. 52, §6.1 · p. 52

6.2 FP8 Quantization

• FP8 inference quantizes most feedforward parameters and activations using H100 native support, while leaving self-attention parameters unquantized.Feedforward layers account for roughly 50% of inference compute time; dynamic scaling improves accuracy, and optimized CUDA kernels reduce scaling overhead. The FP8 quantization implementation is publicly released.Source: §6 · p. 51, §6.2 · p. 52
• 100,000-response reward-score analysis finds that the proposed FP8 quantization has very limited impact on Llama 3 405B responses compared with BF16.Standard benchmarks can conceal quantization problems: without upper-bounded scaling factors, the model occasionally produces corrupted responses despite strong benchmark performance.Source: Figure 26 · p. 53, §6.2 · p. 53
• Row-wise quantization, scaling factors capped at 1200, and unquantized first and last Transformer layers mitigate FP8 quantization errors.High-perplexity tokens can cause large activations, excessive scaling, underflows, and decoding errors; row-wise quantization works better than tensor-wise quantization. Leaving boundary layers unquantized is akin to Zhang et al. (2021).Source: §6.2 · pp. 52–53
• Up to 50% higher pre-fill throughput and a substantially better decoding throughput–latency trade-off are achieved with FP8 versus two-machine BF16 inference.The evaluation uses Llama 3 405B with 4,096 input tokens and 256 output tokens.Source: §6.2 · p. 53, Figure 27 · p. 54

7 Vision Experiments

• The compositional approach adds image cross-attention first, then temporal aggregation and video cross-attention, while preserving text-only performance.It enables parallel vision and language development, avoids joint-pretraining complexities, and reduces inference computation by keeping full-resolution images out of the language backbone. The multimodal models remain under development and are not ready for release.Source: §7 · p. 54

7.1.1 Image Data

• Image-data preparation combines quality filtering, perceptual deduplication, diversity-oriented resampling, OCR augmentation, and safety mitigations.Deduplication retains one pair per connected component of similar image embeddings using SSCD (Pizzi et al., 2022); n-gram resampling improves low-frequency and fine-grained recognition, while caption augmentation with extracted text improves OCR-dependent tasks. Safety processing removes detected CSAM and NSFW material and blurs faces throughout the training set.Source: §7.1.1 · pp. 54–56

7.1.2 Video Data / 7.4 Pre-training

• Video-data curation filters text, alignment, and motion; video pre-training updates only newly added temporal components on top of image-trained weights.Filtering removes non-English text, excessive text overlays, poorly aligned pairs, and static or low-motion videos, without aesthetic or resolution filtering; videos average 21 seconds, with over 99% under one minute. Video pre-training freezes all other parameters and uniformly samples 16 frames, aggregating them into one effective frame.Source: §7.1.2 · p. 56, §7.4 · p. 58

7.2 Model Architecture

• Cross-attention adapters connect a multi-layer image encoder to the language model, while temporal aggregation and additional attention support video recognition.The enhanced image encoder has 850M parameters; image cross-attention appears after every fourth language self-attention layer and adds approximately 100B parameters to Llama 3 405B. Image pre-training updates vision components while freezing language weights, uses approximately 6B image-text pairs, then anneals on approximately 500M higher-resolution images; video processing accepts up to 64 frames and aggregates consecutive frames.Source: §7.2 · p. 57, §7.4 · p. 58

7.3 Model Scaling

• Scaling vision-adapter training requires load-balanced pipeline stages, image sequence parallelism, and FP32 gradient accumulation to address heterogeneous computation and numerical instability.The 8B and 70B adapters use data and tensor parallelism; 405B training additionally uses pipeline parallelism, with four self-attention layers plus one cross-attention layer per stage and replicated image encoders. Sequence parallelism balances image-token workloads, larger micro-batches exploit short captions, and FP32 accumulation replaces unstable bf16 accumulation.Source: §7.3 · pp. 57–58

7.5 Post-Training / 7.5.1 Supervised Finetuning Data

• Post-training combines multimodal conversational finetuning, DPO, rejection sampling, and final quality tuning to improve chat, human evaluation, and reasoning.Image supervised data combines rewritten academic question-answer pairs, diverse human-authored conversations, and synthetic examples generated through textual image representations. Video data combines converted academic annotations with human questions deliberately requiring temporal understanding rather than a single frame.Source: §7.5 · p. 58, §7.5.1 · pp. 58–59

7.5.2 Supervised Finetuning Recipe

• Image supervised finetuning swaps in instruction-tuned language weights, freezes them, and averages strong vision checkpoints; video finetuning updates only video-specific parameters.Averaging top-performing checkpoints consistently outperforms the best individual grid-search model and reduces hyperparameter sensitivity. Video finetuning increases input length to 64 frames, aggregated into two effective frames, and raises resolution to match image settings.Source: §7.5.2 · p. 59

7.5.3 Preference Data / 7.5.4 Reward Modeling

• Preference training uses human comparisons, deliberately corrupted synthetic answers, and rejection-sampling negatives; the vision reward model builds on vision SFT and the language reward model.Human comparisons include seven-scale ratings and optional corrections to chosen responses. Synthetic negatives introduce meaningful errors into supervised answers, while unselected rejection-sampling generations provide additional on-policy preference pairs.Source: §7.5.3 · pp. 59–60, §7.5.4 · p. 60

7.5.4 Reward Modeling

• Freezing the language reward model while training the vision encoder and cross-attention layers generally improves accuracy, especially for judgments requiring knowledge or language-quality assessment.Training combines human rankings of edited, chosen, and rejected responses with image-related negative perturbations that encourage visual grounding. A weighted regularizer on squared reward logits prevents reward-score drift.Source: §7.5.4 · p. 60

7.5.5 Direct Preference Optimization

• Vision Direct Preference Optimization (DPO; Rafailov et al. (2023)) consistently improves human evaluations over its SFT starting point at every finetuning iteration.Training retains recent human-preference batches and drops sufficiently off-policy data to combat distribution shift. Updating the reference model through an exponential moving average every k steps improves human evaluations compared with keeping it frozen.Source: §7.5.5 · p. 60

7.5.6 Rejection Sampling

• Rejection sampling supplies missing reasoning explanations by generating candidate answers, checking correctness against ground truth, and adding accepted answers to finetuning data.Keeping multiple correct answers per question is useful, but questions with low sampled correctness are excluded because correct final answers can conceal incorrect explanations. The reward model selects the top-K answers for language and style quality.Source: §7.5.6 · p. 60

7.5.7 Quality Tuning

• Quality-Tuning significantly improves human evaluations without harming benchmark generalization when its selective dataset covers diverse tasks and training uses appropriate early stopping.DPO models receive additional training on a small SFT dataset whose samples are rewritten and verified by humans or the best models. Checkpoints are selected purely using benchmarks to retain or improve capabilities.Source: §7.5.7 · pp. 60–61

7.6 Image Recognition Results

• Llama 3-V 405B outperforms GPT-4V on all evaluated image benchmarks, trails Gemini 1.5 Pro and Claude 3.5 Sonnet slightly, and is particularly competitive on document understanding.The vision module performs competitively across image-recognition benchmarks at varying model capacities.Source: Table 29 · p. 61, §7.6 · p. 61

7.7 Video Recognition Results

• 8B and 70B Llama 3 models with small post-trained video adapters achieve competitive zero-shot performance, sometimes surpassing Gemini and GPT-4 alternatives.Performance is strongest on PerceptionTest, and ActivityNet-QA results remain strong using at most 64 frames per video. None of these benchmarks enter training or finetuning.Source: Table 30 · p. 62, §7.7 · p. 62

8.1.1 Speech Understanding

• Approximately 15M unlabeled speech hours pretrain the encoder; supervised data include 230K ASR hours across 34 languages and 90K speech translation hours.Translation training covers 33 languages to English and the reverse, including NLLB-generated data (NLLB Team et al., 2022); dialogue data include 60K speech hours with synthetic responses (Fathullah et al., 2024) and 25K Voicebox-generated hours (Le et al., 2024). Translation evaluation measures BLEU on FLEURS and 15 of 21 CoVoST 2 languages, with Whisper and SeamlessM4T as references.Source: §8.1.1 · pp. 63–64, Table 32 · p. 67, §8.4 · p. 67

8.1.2 Speech Generation

• 55K text-normalization samples and a 50K-hour professional-voice TTS dataset train the speech generation components, augmented with contextual Llama 3 embeddings.Normalization samples pair written and spoken forms with handcrafted transformation rules; prosody training uses extracted linguistic and prosodic features. Embeddings come from the 16th decoder layer of Llama 3 8B and are chunk-aligned with normalization tokens or phone-rate features for streaming training.Source: §8.1.2 · p. 64

8.2.1 Speech Understanding

• A speech encoder and adapter inject speech embeddings directly alongside text tokens, rather than through the vision module’s cross-attention layers.A 1B-parameter Conformer (Gulati et al., 2020) encoder feeds an approximately 100M-parameter adapter, producing representations at 80 ms intervals. The interface supports up to 34 languages and interleaved text–speech input; system prompts select recognition or translation, while their absence enables general spoken dialogue.Source: §8 · p. 63, §8.2.1 · p. 64

8.2.2 Speech Generation

• Llama 3 8B embeddings supply context to text normalization and prosody modeling, supporting semantic correctness, naturalness, and expressiveness.A proprietary streaming TTS system generates waveforms during language-model decoding without fine-tuning Llama 3 for speech generation. A streaming LSTM predicts normalization rules using embedding cross-attention (Kang et al., 2024), while a decoder-only Transformer models prosody with reduced lookahead (Radford et al., 2021).Source: §8 · p. 63, §8.2.2 · p. 65

8.3.1 Speech Understanding

• Target-language-only prompts address multilingual confusion without requiring source-language identification, preserving flexibility for unseen translation directions and potentially code-switched speech.BEST-RQ (Chiu et al., 2022) pretrains the encoder; supervised training optimizes encoder and adapter on ASR, translation, and dialogue while freezing Llama 3. Despite single-turn training, dialogue examples demonstrate zero-shot code-switching and coherent multi-turn interaction.Source: §8.3.1 · pp. 65–66, §8.4 · p. 67, Figure 30 · p. 68

8.3.2 Speech Generation

• Causal masking and dynamic alignment combine fixed phone lookahead with variable token lookahead for streaming prosody synthesis.Training and inference share this alignment, while delayed-pattern prediction captures long-range prosodic dependencies (Kharitonov et al., 2021). Llama 3 8B remains frozen during prosody-model training.Source: §8.3.2 · p. 66

8.4 Speech Understanding Results

• Llama 3 outperforms Whisper and SeamlessM4T across evaluated speech recognition benchmarks and performs similarly to Gemini on MLS English.On MuTox, English added toxicity is below 1%, lowest among compared models, and toxicity removal substantially exceeds addition. Gemini’s MLS comparison uses published results, empty Gemini safety responses count as safe, and Whisper’s FLEURS average excludes Malayalam.Source: §8.4 · pp. 66–67, Table 31 · p. 67, §8.5 · p. 67

8.5 Speech Generation Results

• 60.0% and 63.6% preference rates favor Llama 3 8B prosody modeling over streaming and non-streaming phone-only baselines, respectively.With three-token right context, embedding-augmented text normalization also outperforms embedding-free models using either three-token or full bidirectional context. Both components support token-wise streaming, reducing reliance on long lookahead.Source: §8.5 · pp. 68–69, Table 34 · p. 68, Table 35 · p. 69

9.1 Language

• Almost fifty times Llama 2 70B’s pre-training compute budget supports Llama 3’s 405B model, continuing improvements driven by increased compute and better data.Despite having fewer parameters than PALM (Chowdhery et al., 2023), Llama 3 performs better; the passage attributes this efficiency to improved understanding of scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022).Source: §9.1 · p. 69
• Smaller Llama 3 models train far beyond compute-optimal training, trading additional training compute for inference efficiency and simpler deployment.Distilling larger models into smaller ones, as in Phi (Abdin et al., 2024), offers an alternative route.Source: §9.1 · p. 69
• Llama 3 outperforms mixture-of-experts models Mixtral (Jiang et al., 2024) and Arctic, suggesting dense architectures are not the limiting factor.Llama 3 makes minimal architectural changes relative to Llama 2; architectural choices still involve trade-offs in training efficiency, inference efficiency, and stability at scale.Source: §9.1 · p. 69
• Llama 3-405B is competitive with the closed-weight state-of-the-art, reflecting rapid improvement in open-weight foundation models.The section places Llama 3 within a growing ecosystem of openly available model families.Source: §9.1 · p. 69
• Millions of human instructions and preference judgments support Llama 3’s instruction tuning and human-feedback alignment across multiple post-training rounds.Training combines rejection sampling (Bai et al., 2022), supervised finetuning (Sanh et al., 2022), and Direct Preference Optimization (Rafailov et al., 2023). Earlier Llama 3 versions filter, rewrite, or generate prompts and responses for training.Source: §9.1 · pp. 69–70

9.2 Multimodality

• Llama 3’s image approach combines prior ideas, achieving results comparable with Gemini 1.0 Ultra (Google, 2023) and GPT-4 Vision.Image antecedents include contrastive image–text embedding (Radford et al., 2021), while video adapters align video and language representations and achieve results competitive with the state-of-the-art. Building on compositional speech approaches such as Fathullah et al. (2024), Llama 3 achieves strong performance at larger scales without language-model finetuning, avoiding potential interference with non-speech tasks.Source: §9.2 · p. 70

10 Conclusion

• High-quality data, scale, and simplicity consistently yielded the best results during Llama 3 development.Preliminary experiments with more complex architectures and training recipes did not yield benefits sufficient to outweigh their added development complexity. The authors see substantial further improvements ahead.Source: §10 · p. 70
• Organizational safeguards were pivotal to preventing benchmark contamination and maintaining trustworthy human evaluations.A separate team procured and processed pre-training data with strong incentives to prevent benchmark contamination. Only a small group of researchers uninvolved in model development could perform and access human evaluations.Source: §10 · p. 70
• Sharing development details aims to clarify the key factors in foundation-model development and support more informed public debate about their future.Source: §10 · p. 70
• The preliminary multimodal models remain under active development and are not ready for release; sharing their results early aims to accelerate research.Source: §10 · p. 70
• Following positive safety analyses, the authors publicly release Llama 3 language models to enable societal applications, research scrutiny, and improvements in model quality and safety.They view public release as important to responsible foundation-model development and hope it encourages open, responsible development of AGI.Source: §10 · p. 71

Core Contributors

• 2/3rd of the project's runtime is the minimum participation threshold for core contributors.The core-contributor roster begins with Aaron Grattafiori and ends with Zoe Papakipos.Source: Contributors and Acknowledgements · p. 72, Core Contributors · p. 72

Contributors and Acknowledgements

• 1/5th of the project's runtime is the minimum participation threshold for contributors.Source: Contributors and Acknowledgements · p. 72
• All contributors are listed alphabetically by first name.Source: Contributors and Acknowledgements · p. 72