Llama 2: Open Foundation and Fine-Tuned Chat Models

• The release addresses the gap between publicly available pretrained models and closed product LLMs, whose extensive preference fine-tuning improves usability and safety but requires costly, opaque, and difficult-to-reproduce annotation and computation.Public pretrained models can match closed pretrained competitors, but the introduction states that they had not provided suitable substitutes for closed product LLMs.Source: §1 · p. 3
• Llama 2-Chat generally outperforms existing open-source models on helpfulness and safety benchmarks and appears comparable to some closed-source models in the authors’ human evaluations.The safety evaluations covered approximately 2,000 adversarial single- and multi-turn prompts, but the authors note limitations from prompt coverage, review guidelines, rater subjectivity, and standards potentially favoring Llama 2-Chat.Source: §1 · p. 3, Figure 3 · p. 4
• The models incorporate safety-specific annotation and tuning, red-teaming, and iterative evaluations, while developers are advised to conduct application-specific safety testing and tuning before deployment.Testing to date was conducted in English and could not cover all scenarios; the authors provide a responsible use guide and code examples for safer deployment.Source: §1 · pp. 3–4
• Llama 2 is released in 7B, 13B, and 70B variants after expanding pretraining data by 40%, doubling context length, and adopting grouped-query attention; 34B variants are reported but withheld.The 34B model release was delayed because there was insufficient time for adequate red-teaming.Source: §1 · p. 4
• Llama 2-Chat is a fine-tuned version of Llama 2 optimized for dialogue use cases and released in 7B, 13B, and 70B variants.The broader training process combines supervised fine-tuning with iterative RLHF using rejection sampling and PPO.Source: §1 · p. 4, Figure 4 · p. 5

2 Pretraining

• Llama 2 uses an optimized autoregressive Transformer with robust data cleaning, updated data mixes, 40% more tokens, doubled context length, and GQA for larger-model inference scalability.The models retain standard Transformer components including RMSNorm, SwiGLU, and RoPE, while increasing context length and adding GQA relative to Llama 1.Source: §2 · p. 5, §2.2 · p. 5, Table 1 · p. 6

2.1 Pretraining Data

• The training corpus contains 2 trillion publicly sourced tokens, excludes Meta products and services, removes some high-personal-information sources, and up-samples factual data to reduce hallucinations.The authors describe this amount as a performance–cost trade-off and report additional data investigations in Section 4.1.Source: §2.1 · p. 5

2.2 Training Details

• Pretraining used AdamW with cosine learning-rate decay, 2000 warmup steps, 0.1 weight decay, 1.0 gradient clipping, and a 32k-token BPE vocabulary.All models used a 4M-token global batch size; 34B and 70B models used GQA, and training showed no saturation after 2T tokens.Source: §2.2 · pp. 5–6, Table 1 · p. 6, Figure 5 · p. 6

2.2.1 Training Hardware & Carbon Footprint

• Training used 3.3M A100 GPU hours and produced an estimated 539 tCO2eq, all directly offset by Meta’s sustainability program.RoCE scaled almost as well as InfiniBand up to 2000 GPUs, and the emissions estimate excludes interconnect, non-GPU server, cooling, and hardware-production power demands.Source: §2.2.1 · p. 7, Table 2 · p. 7

2.3 Llama 2 Pretrained Model Evaluation

• Llama 2 outperforms Llama 1, with 70B improving MMLU and BBH by ≈5 and ≈8 points over Llama 1 65B, respectively, while outperforming all open-source models.Llama 2 7B and 30B outperform similarly sized MPT models except on code, and 7B and 34B outperform corresponding Falcon models across all benchmark categories except where not specified; 70B is close to GPT-3.5 on MMLU and GSM8K, on par or better than PaLM on almost all benchmarks, but trails GPT-4 and PaLM-2-L.Source: §2.3 · p. 8, Table 3 · p. 8, Table 4 · p. 8

3 Fine-tuning

• The fine-tuning program combines supervised fine-tuning, initial and iterative reward modeling, RLHF, and Ghost Attention, which helps control dialogue flow across multiple turns.The work required substantial computational and annotation resources, and the broader safety evaluation of fine-tuned models is deferred to Section 4.2.Source: §3 · p. 8

3.1 Supervised Fine-Tuning (SFT)

• SFT quality improved when the authors replaced millions of third-party examples with tens of thousands of higher-quality annotations, stopping at 27,540 examples.They found vendor and platform differences could markedly affect downstream performance, and model-generated SFT outputs were often competitive with handwritten annotations.Source: §3.1 · p. 9

3.2.1 Human Preference Data Collection

• Human preference collection uses diverse binary comparisons focused separately on helpfulness and safety, with annotators rating preference strength and labeling safety outcomes.The dataset contains over 1 million binary comparisons, has more conversation turns, and is longer on average than the compared open-source datasets.Source: §3.2.1 · p. 10

3.2.2 Reward Modeling

• Reward modeling combines binary preference ranking with rating-dependent margins, separate helpfulness and safety models, and mixed internal and open-source preference data.The margin is larger for clearly distinct responses and smaller for similar ones, improving Helpfulness reward-model accuracy on more separable samples; larger models and more data continue improving accuracy.Source: §3.2.2 · pp. 10, 11, 13
• The Helpfulness and Safety reward models each perform best on their corresponding internal test sets and outperform all reported baselines, including GPT-4.Separate models ease the tension between maximizing helpfulness and refusing unsafe prompts; accuracy also declines as preference pairs become more similar.Source: §3.2.2 · pp. 12–13

3.2.3 Iterative Fine-Tuning

• RLHF iterates through rejection sampling and PPO: rejection sampling selects the highest-reward output among K candidates, while PPO updates the policy sequentially with a KL penalty.The KL constraint improves training stability and reduces reward hacking; later iterations combine PPO with rejection-sampling checkpoints, and retaining top samples from all prior iterations mitigates capability regression.Source: §3.2.3 · pp. 13–15
• Rejection sampling’s potential gain increases with more sampled outputs, while higher temperature improves exploration and the optimal RLHF sampling temperature is T ∈[1.2, 1.3] for 10–100 outputs.The optimal temperature changes across iterative model updates, so it must be progressively readjusted under finite compute.Source: §3.2.3 · p. 15
• The iterative fine-tuning procedure combines safety and helpfulness rewards, prioritizes safety for tagged prompts, filters unsafe responses at threshold 0.15, and whitens reward logits for stable KL balancing.The threshold yields precision 0.89 and recall 0.55 on the Meta Safety test set.Source: §3.2.3 · p. 15

3.3 System Message for Multi-Turn Consistency

• Ghost Attention (GAtt), inspired by Context Distillation (Bai et al., 2022b), preserves system-message constraints across 20+ dialogue turns and maintains stronger attention to the system message.The method concatenates an instruction to user messages during synthetic-data construction, then masks losses for previous turns during fine-tuning.Source: §3.3 · pp. 16–17, Figure 10 · p. 17

3.4.1 Model-Based Evaluation

• Model-based evaluation was used to select among RLHF iterations because it reduced cost and increased iteration speed, and reward models were well calibrated against human preference annotations.Major model versions were later validated with human evaluations, and no divergence from human preferences had yet been observed.Source: §3.4.1 · pp. 17–18
• Llama 2-Chat outperformed ChatGPT on both safety and helpfulness after RLHF-V3 according to in-house rewards, while GPT-4 judged the latest model above 60% win-rate.The reward-model comparison may favor Llama 2-Chat, so GPT-4 was used as a more neutral judge.Source: §3.4.1 · p. 18, Figure 11 · p. 18

3.4.2 Human Evaluation

• Human evaluations found Llama 2-Chat significantly outperformed open-source models: 7B beat MPT-7B-chat on 60% of prompts, while 34B exceeded 75% wins against Vicuna-33B and Falcon 40B.The evaluation covered approximately 4,000 helpfulness prompts across single- and multi-turn settings, with three raters per prompt.Source: §3.4.2 · p. 18, Figure 12 · p. 19
• Llama 2-Chat 70B achieved a 36% win rate and 31.5% tie rate against ChatGPT and substantially outperformed PaLM-bison chat on the evaluated prompts.These comparisons were part of human evaluations against open- and closed-source models.Source: §3.4.2 · p. 19
• Human-evaluation reliability varied from 0.37 to 0.55 on Gwet’s AC2, while results were limited by subjective ratings, prompt coverage, missing coding and reasoning tasks, and evaluating only final multi-turn responses.Different prompts or instructions could produce different outcomes.Source: §3.4.2 · p. 19

4.1 Safety in Pretraining

• Pretraining data exhibits Western demographic skew, including “American” in 69.4% of references, while English predominance may limit suitability for other languages.The corpus also contains higher He-pronoun representation than She-pronoun representation, and about 0.2% of evaluated documents receive a toxicity likelihood of at least 0.5.Source: §4.1 · pp. 20–21, Table 9 · p. 21, Figure 13 · p. 21
• 21.37% higher truthfulness and informativeness and 7.61% lower toxicity are observed for Llama 2-7B than Llama 1-7B, although larger Llama 2 models show increased toxicity.Llama 2 does not outperform other models on toxicity metrics overall, likely because its pretraining data was not aggressively filtered; base models therefore require additional safety mitigation before deployment.Source: §4.1 · p. 22, Table 11 · p. 23

4.2 Safety Fine-Tuning

• Safety fine-tuning combines supervised safety demonstrations, a safety-specific RLHF reward model, adversarial prompting, rejection-sampling or PPO optimization, and targeted context distillation.The process first teaches safety guidelines through supervised fine-tuning, then uses preference optimization and safer responses generated with safety preprompts.Source: §4.2 · p. 23

4.2.1 Safety Categories and Annotation Guidelines

• Annotation guidelines organize adversarial prompts by risk category and attack vector, covering criminal or illicit activities, harmful or hateful activities, unqualified advice, and manipulative prompt styles.Annotators are instructed to address immediate safety concerns, explain risks, and provide additional helpful information while iteratively revising the guidelines for newly identified risks.Source: §4.2.1 · pp. 23–24

4.2.2 Safety Supervised Fine-Tuning

• Supervised safety fine-tuning uses trained annotators to create potentially unsafe prompts and corresponding safe, helpful demonstrations for model training.The prompts are produced through annotator red teaming, followed by crafting the responses the model should generate.Source: §4.2.2 · p. 24

4.2.3 Safety RLHF

• Safety RLHF shifts safety reward scores upward and preserves helpfulness scores, indicating improved long-tail safety without notable helpfulness degradation.Increasing safety-data proportions further improves risky-prompt handling while mean helpfulness remains stable, though false refusals increase and remain approximately 0.05% on the helpfulness set even with all safety data.Source: §4.2.3 · pp. 24, 26, Figure 14 · p. 25, Figure 15 · p. 26

4.2.4 Context Distillation for Safety

• Context distillation enhances safety by training on responses generated with safety preprompts, while safety-reward filtering limits degraded relevance and false refusals.Risk-category answer templates improve safety more than generic preprompts, so distillation is applied selectively when the distilled response receives a higher safety reward than the original.Source: §4.2.4 · pp. 27–28, Figure 16 · p. 28

4.3 Red Teaming

• Red teaming by more than 350 internal, contracted, and external participants probed diverse risks and attack vectors, with marginal weapon-production findings mitigated.Participants included domain experts and varied demographic groups, and testing covered criminal planning, trafficking, controlled substances, explicit content, unqualified advice, privacy, malformed inputs, and extended dialogues.Source: §4.3 · pp. 28–29
• Red-teamers used risk categories, non-English prompts and contexts, specialized subteams, and 5-point risk annotations to probe English model outputs across attack vectors.Collected dialogues were analyzed by length, risk area, misinformation topic where applicable, and rated risk, then used for fine-tuning, model feedback training, and other safety-model training.Source: §4.3 · p. 29
• Red teaming found that later models resolved unsafe-response disclaimers followed by harmful content, while earlier models were vulnerable to creative-writing distractions and positive-context obfuscation.These findings describe progressively improved handling of problematic requests during model development.Source: §4.3 · p. 29
• On the 7B model, the red-team robustness metric γ decreased from 1.8 to 0.45 across iterations, while new candidate releases mitigated 90% of previously discovered violating prompts on average.Robustness was defined as violating prompts generated per person per hour, and additional red teaming was expected to improve it further.Source: §4.3 · p. 29

4.4 Safety Evaluation of Llama 2-Chat

• Safety evaluation used roughly 2,000 adversarial prompts, majority-vote judgments from three annotators, and violation percentage as the main metric, with ratings of 1 or 2 counted as violations.The prompt set included 1,351 single-turn and 623 multi-turn prompts, and Llama 2-Chat annotations averaged 0.92 inter-rater reliability under Gwet’s AC2.Source: §4.4 · pp. 29–30
• Llama 2-Chat has comparable or lower overall violation percentages across model sizes than the evaluated baselines, while maintaining higher safety and helpfulness ratings overall.The comparison places ChatGPT and Falcon next, followed by MPT and Vicuna; Falcon’s similar violation percentage but lower helpfulness reflects its concise responses.Source: §4.4 · p. 30, Figure 17 · p. 30
• Multi-turn conversations more often induce unsafe responses, but Llama 2-Chat remains strong relative to baselines, whereas Falcon reverses from strong single-turn performance to weaker multi-turn performance.Falcon’s multi-turn weakness may reflect limited multi-turn supervised fine-tuning data.Source: Figure 18 · p. 30, §4.4 · p. 31
• Llama 2-Chat has relatively more violations for unqualified advice, but comparable or lower violation percentages in the other risk categories across model sizes.The unqualified-advice issue is partly attributed to missing disclaimers such as stating that the model is not a professional.Source: §4.4 · p. 31, Figure 19 · p. 31
• Fine-tuning improves 70B truthfulness from 50.18 to 64.14 and reduces toxicity from 24.60 to 0.01, with Llama 2-Chat achieving the lowest toxicity among compared models.Compared with Falcon and MPT, fine-tuned Llama 2-Chat performs best on toxicity and truthfulness; it also generally increases positive sentiment across demographic groups.Source: §4.4 · p. 31, Table 14 · p. 31

5.1 Learnings and Observations

• RLHF effectively leverages human preference comparisons, removing low-quality responses and enabling models to exceed annotators’ writing abilities in some tasks.The authors argue that human–LLM synergy, rather than dense supervised annotation alone, is central to RLHF’s effectiveness.Source: §5.1 · p. 32
• RLHF dynamically adapts response diversity to prompt type, retaining creativity for poems while reducing diversity and stabilizing answers for factual questions.Figure 21 reports this pattern using Self-BLEU across temperatures, with lower Self-BLEU indicating more diversity.Source: §5.1 · p. 32, Figure 21 · p. 33
• With 1,000 time-focused SFT examples, Llama 2-Chat generalized temporal organization across manually tested examples, despite randomly shuffled next-token training data.The examples included query dates and event dates, allowing responses to depend on temporal context.Source: §5.1 · p. 33, Figure 22 · p. 33
• Tool use emerged zero-shot from alignment: without explicit tool-use annotation, Llama 2-Chat understood tool applications and API arguments and produced sequences of tool calls.The authors also evaluated calculator access and identify tool use as an area requiring further safety research and red teaming.Source: §5.1 · p. 34, Figure 23 · p. 34

5.2 Limitations and Ethical Considerations

• Llama 2-Chat retains common LLM limitations, including stale post-pretraining knowledge, unqualified advice, hallucinations, harmful or biased content, misuse risks, and overcautious refusals.Non-English performance is fragile because the initial model used limited non-English pretraining data, and pretrained-model users require additional tuning and deployment precautions.Source: §5.2 · pp. 34–35

5.3 Responsible Release Strategy

• The authors openly release Llama 2 for research and commercial use under licensing and safety constraints, arguing that collaboration can improve safety, transparency, innovation, and access.They provide replication code, a Responsible Use Guide, and commit to continued mitigation of toxic content and problematic associations.Source: §5.3 · p. 35

Large Language Models

• LLM research progressed from large-scale models such as GPT-3, Gopher, Galactica, and Chinchilla to computationally efficient Llama and increasingly competitive open-source releases.Open-source models including BLOOM, OPT, and Falcon challenge closed-source systems such as GPT-3 and Chinchilla, while production-ready models like ChatGPT, Bard, and Claude retain performance and usability advantages through preference-alignment techniques.Source: §6 · pp. 35–36
• Distillation-based systems such as Vicuna and Alpaca use synthetic instructions to narrow the gap with closed-source models but still fall short of their performance bar.These approaches build on synthetic-instruction training methods described by Honovich et al. (2022) and Wang et al. (2022).Source: §6 · p. 36

Instruction Tuning

• Instruction tuning transfers to unseen tasks, with outcomes shaped by task count, model size, prompt settings, prompt source, and iterative refinement.Related approaches include chain-of-thought prompting, which asks models to explain reasoning on complex problems to increase the likelihood of correct final answers.Source: §6 · p. 36

RLHF

• RLHF fine-tunes models from human feedback to align responses with human expectations, while instruction tuning plus RLHF can improve factuality, toxicity, and helpfulness beyond scaling alone.RLAIF partially automates this pipeline by using model-generated critiques, revisions, and rankings instead of human-labeled data and raters.Source: §6 · p. 36

Known LLM Safety Challenges

• Safety research identifies risks including bias, toxicity, private-data leakage, misleading expertise, harmful-content attacks, cyber and biological threats, job displacement, and over-reliance on LLMs.The literature also develops taxonomies, mitigation strategies, and frameworks for assessing impacts within base systems and broader societal contexts.Source: §6 · p. 36

• Llama 2 spans 7 billion to 70 billion parameters and includes pretrained and fine-tuned models.Source: §7 · p. 36
• The models compete with existing open-source chat models and match some proprietary models on examined evaluation sets, but still lag behind GPT-4.Source: §7 · p. 36
• The authors emphasize methods for aligning Llama 2 with helpfulness and safety principles.Source: §7 · p. 36
• Llama 2 and Llama 2-Chat were responsibly opened to support society and accelerate research, with further Llama 2-Chat safety improvements planned.Source: §7 · p. 36

A.1 Contributions

• Authors are listed alphabetically by last name.Source: §A.1 · p. 46
• Science and engineering leadership included Guillem Cucurull, Naman Goyal, Louis Martin, Thomas Scialom, Ruan Silva, Kevin Stone, and Hugo Touvron.Source: §A.1 · p. 46
• Core contributors included Peter Albert, Nikolay Bashlykov, Prajjwal Bhargava, Moya Chen, David Esiobu, Jeremy Fu, Vedanuj Goswami, Anthony Hartshorn, Rui Hou, Marcin Kardas, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Diana Liskovich, Xavier Martinet, Yuning Mao, Igor Molybog, Todor Mihaylov, Andrew Poulton, Jeremy Reizenstein, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Jacob Xu, Yuchen Zhang, and Iliyan Zarov.Source: §A.1 · p. 46
• Additional contributors included Amjad Almahairi, Yasmine Babaei, Soumya Batra, Lukas Blecher, Dan Bikel, Shruti Bhosale, Cristian Canton Ferrer, Jude Fernandes, Wenyin Fu, Brian Fuller, Cynthia Gao, Saghar Hosseini, Hakan Inan, Isabel Kloumann, Madian Khabsa, Artem Korenev, Viktor Kerkez, Jian Xiang Kuan, Yinghai Lu, Jenya Lee, Pushkar Mishra, Yixin Nie, Rashi Rungta, Alan Schelten, Kalyan Saladi, Adina Williams, and Zheng Yan.Source: §A.1 · p. 46
• The GenAI executive team was thanked for leadership and support, specifically Ahmad Al-Dahle and Manohar Paluri.Source: §A.1 · p. 46

A.2.1 Architecture Changes Compared to Llama 1

• 4096-token context pretraining improves SCROLLS performance over 2048 tokens while preserving SQUAD performance under otherwise identical 150B-token training.The longer window supports longer chat histories, summarization tasks, and documents.Source: §A.2.1 · p. 47, Table 16 · p. 47, Table 17 · p. 47
• KV-cache memory grows substantially with context windows or batch sizes, making shared key and value projections attractive for larger models.The alternatives are MQA with one KV projection or GQA with 8 KV projections.Source: §A.2.1 · p. 47
• GQA performs comparably to MHA on most evaluation tasks and better than MQA on average, motivating its use in the 34B and 70B Llama 2 models.The choice also reflects GQA’s easier inference scaling relative to MQA.Source: §A.2.1 · pp. 47–48, Table 18 · p. 48
• For fixed 30B models trained on 150B tokens, FFN dimensions increased by 1.33× for MQA and 1.3× for GQA to offset reduced attention parameters.These adjustments kept overall parameter counts similar across the attention variants.Source: §A.2.1 · p. 47
• With 8 A100s and tensor parallelism, MQA cannot shard KV heads across GPUs because it has fewer heads than GPUs.Duplicating KV values matches GQA’s cache size, while batch-dimension sharding requires sufficiently large batches and adds communication costs.Source: §A.2.1 · pp. 47–48
• MQA and GQA enable higher throughput at larger batch sizes and similar latency at smaller batches, while MHA runs out of memory sooner.MHA fails at batch size 1024 with 256-token context and batch size 128 with 2k context, whereas MQA and GQA succeed.Source: Figure 24 · p. 48

MMLU details

• The evaluation reports five-shot performance for Llama 2 and other open-source models on the MMLU benchmark.MMLU is identified as the Massive Multitask Language Understanding benchmark.Source: §A.2.2 · p. 48, Table 19 · p. 49

Standard Benchmarks

• Results are also presented for Llama 2 and other models on several standard benchmarks.The supplied passage does not specify the individual benchmark outcomes.Source: §A.2.2 · p. 48, Table 20 · p. 49

Code Generation

• Code-generation comparisons evaluate Llama 2 against popular open-source models on Human-Eval and MBPP.Human-Eval uses zero-shot results, while MBPP uses three-shot results; pass@100 and pass@80 use temperature 0.8 with top-p=0.95, and pass@1 uses temperature 0.1 with top-p=0.95.Source: §A.2.2 · p. 48, Table 21 · p. 49

World Knowledge

• World-knowledge evaluation compares Llama 2 with other open-source models on NaturalQuestions and TriviaQA.NaturalQuestions reports exact-match performance, while TriviaQA reports zero-shot and few-shot exact-match performance on the filtered development set’s Wiki validation subset.Source: §A.2.2 · p. 48, Table 22 · p. 50

Reading Comprehension

• Llama 2 performs best on all reported SQUAD and QUAC evaluation settings except QUAC zero-shot, where Llama 1 30B performs slightly better.The comparison includes zero-shot and few-shot SQUAD results and zero-shot and one-shot QUAC results.Source: §A.2.2 · p. 48, Table 23 · p. 50

Exams

• The section reports fine-grained English AGI Eval results, covering standardized exams across different subjects.Llama 2 is compared with other open-source models.Source: §A.2.2 · p. 48, Table 24 · p. 50

Mathematical Reasoning

• Mathematical-reasoning results compare Llama 2 and other open-source models on GSM8k and MATH.The reported metric is maj1@1.Source: §A.2.2 · p. 51, Table 25 · p. 51

A.3.1 Detailed Statistics of Meta Human Preference Data

• Over 1 million binary model-generation comparisons were collected across 14 weekly Meta human-preference batches covering safety and helpfulness.These statistics are reported in Table 26.Source: §A.3.1 · p. 51
• Later batches contained more samples as additional annotators joined and existing annotators became more efficient with the tasks.Source: §A.3.1 · p. 51
• Multi-turn samples were intentionally increased across batches, raising the average number of tokens per sample and increasing RLHF-data complexity.Source: §A.3.1 · p. 51
• Figure 25 shows similar-response ratings increasing while strong preferences declined as improved Llama 2-Chat models produced two increasingly comparable responses.The shift reflects the iterative model-update and preference-annotation process, which made selecting a better response more difficult.Source: §A.3.1 · p. 51

Curriculum Strategy

• Annotators progressively increased prompt complexity and introduced new skills during helpfulness preference-data collection.The curriculum strategy was developed in collaboration with annotation platforms, and its helpfulness-data application is illustrated in Figure 26.Source: §A.3.2 · p. 51

A.3.3 Ablation on Ranking Loss with Preference Rating-based Margin for Reward Modeling

• The study evaluates two preference rating-based margin variants with different magnitudes against ranking loss without a margin term.The variants are listed in Table 27 and evaluated for the helpfulness reward model.Source: §A.3.3 · p. 51, Table 27 · p. 52
• Preference rating-based margins improve helpfulness reward-model accuracy on more separable response pairs, but larger margins regress performance on similar samples.The ablation compares two margin magnitudes against a baseline without the margin term using per-rating and average accuracy on the Meta Helpful test set.Source: §A.3.3 · p. 51, Table 28 · p. 52
• Margins push reward scores toward more extreme values, with larger margins producing a more significant distribution shift and motivating reward calibration for PPO.The reward-score histograms are shown for the test set, and PPO is noted as sensitive to reward-distribution changes.Source: §A.3.3 · pp. 51–52

A.3.4 Ablation on Ranking Loss with Safety Auxiliary Loss for Reward Modeling

• Safety auxiliary loss improves unsafe-response recall at a 0.5 reward threshold and increases accuracy across all three safety categories.The loss trains the reward model to discriminate safe from unsafe generations, producing a better safety reward signal for RLHF.Source: §A.3.4 · p. 52, Table 29 · p. 53

A.3.5 Additional Results for GAtt

• GAtt preserves 100% attribute-reference accuracy through 20 turns, whereas Llama 2-Chat without GAtt declines from 100% at turn t+1 to 10% at turn t+3 and then 0%.The evaluated attributes were public figures and hobbies, and all examples totaled less than 4048 tokens over the turns.Source: §A.3.5 · p. 54, Table 30 · p. 54

A.3.6 How Far Can Model-Based Evaluation Go?

• Reward models are well calibrated against triple-reviewed 7-point human quality ratings, supporting point-wise use despite training with Pairwise Ranking Loss.The test set covers both helpfulness and safety prompts.Source: §A.3.6 · p. 54, Figure 29 · p. 55

A.3.7 Human Evaluation

• Human evaluation compares Llama 2-Chat with open- and closed-source models using over 4000 diverse single- and multi-turn prompts, randomized side-by-side judgments, and three annotators per pair.Open-source comparisons use 1000-token context and generation limits for fairness, while closed-source models use 2000-token limits.Source: §A.3.7 · pp. 56–57

A.4.1 Tension between Safety and Helpfulness in Reward Modeling

• Safety and helpfulness reward models can disagree: safe responses may receive high safety but low helpfulness scores, while unsafe responses may receive low safety but high helpfulness scores.This motivates optimizing separate reward models for helpfulness and safety.Source: §A.4.1 · p. 58, Figure 32 · p. 64

A.4.2 Qualitative Results on Safety Data Scaling

• Using more safety data makes Llama 2-Chat more likely to refuse offensive content but also more conservative toward benign prompts containing offensive or sensitive words.The model begins refusing offensive content with 50% of safety data and may misinterpret “sex in a pan” as sensitive despite contextual evidence that it is food.Source: §A.4.2 · p. 58, Table 36 · p. 61, Table 37 · p. 62

A.4.5 Safety Errors: False Refusals and Vague Responses

• False refusal rates increase with the percentage of safety data, rising from 0.006% to 0.05% on the helpfulness dataset and from 15% to 27% on the borderline dataset.Context distillation can make responses vague or lead to false refusals. Benign prompts may contain sensitive keywords such as “crack” and “bomb”.Source: Table 40 · p. 65, Table 41 · p. 66, Figure 33 · p. 67

A.4.8 Automatic Safety Benchmark Evaluation Results

• Instruction fine-tuning improves truthfulness by about 20% for 7B and 13B, 24% for 30B, and 14% for 70B Llama 2-Chat versus pretrained versions, while fine-tuned models show effectively zero toxic generations.TruthfulQA results show most models exceed 90% informativeness, whereas pretrained Falcon, MPT, and 7B Llama 1 achieve only about 30–40% truthfulness; automatic safety metrics are not fully comprehensive.Source: §A.4.8 · pp. 69–70, Table 44 · p. 70
• Safety benchmarks may miss adversarial or toxic inputs, demographic variation, conversational context, and post-deployment user experience, so additional product-context testing is needed.The authors identify disaggregated metrics and future comprehensive evaluations as important complements to benchmark-only assessment.Source: §A.4.8 · pp. 70, 72

A.5 Data Annotation

• Human annotators supplied single- and multi-turn dialogue data for supervised fine-tuning and human preferences for reward-model training.Annotators were asked to produce informative, truthful, relevant, clear, and harmless responses, prioritizing harmlessness when necessary.Source: §A.5 · p. 72, §A.5.1 · p. 72

A.5.2 Negative User Experience Categories

• SFT annotation instructions prohibited responses that promote crime, dangerous behavior, offensive or abusive conduct, or sexually explicit content.These categories were used to avoid negative user experiences and were shared with annotators as guidance.Source: §A.5.2 · p. 74

A.5.3 Quality Assurance Process

• Quality assurance required manual review against dialogue consistency, prompt instructions, writing quality, and safety categories, with minor edits permitted and major defects rejected.A team of skilled content managers approved only annotations meeting the guidelines.Source: §A.5.3 · p. 74

A.5.4 Annotator Selection

• Annotator selection tested grammar, comprehension, writing, sensitive-topic alignment, answer ranking, quality-criteria alignment, and production readiness across four assessments.Passing thresholds included 90% on the first grammar section, 80% agreement on the second test, agreement on more than 26 of 31 quality questions, and an average above 4 on response assessment.Source: §A.5.4 · pp. 74–75

A.6 Dataset Contamination

• The contamination analysis matches tokenized evaluation inputs using 10+-token n-grams with a four-token skipgram budget and defines clean and dirty subsets by 20% and 80% contamination thresholds.Samples are classified as clean below 20%, not clean at least 20%, not dirty below 80%, or dirty at least 80%.Source: §A.6 · p. 75
• Only HellaSwag and MMLU-Humanities appear boosted by contamination; the 70B model benefits more than 7B, while MMLU-Overall shows only a small -0.9 delta.No other dataset appears to benefit from contamination for any tested minimum match length L.Source: §A.6 · p. 76, Table 51 · p. 76

A.7 Model Card

• Table 52 presents the Llama 2 model card summarizing details of the models.The model card is attributed to Mitchell et al. (2018) and Anil et al. (2023).Source: Table 52 · p. 77, §A.7 · p. 77