PaperlayerView exact revision on arXiv

Source-linked AI summary

Llama 2: Open Foundation and Fine-Tuned Chat Models

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom

arXiv:2307.09288v2cs.CLcs.AI

TL;DR

Publicly released LLMs had limited suitability as substitutes for closed-source chat models. This paper develops and releases Llama 2 and Llama 2-Chat, which generally outperform existing open-source models and appear comparable to some closed-source models on tested evaluations.

  • Problem

    Publicly released pretrained LLMs had limited suitability as substitutes for closed-source models.

  • Method

    The paper develops and releases 7B–70B pretrained and dialogue-optimized fine-tuned models, using safety-specific tuning, red-teaming, and iterative evaluations.

  • Results

    Llama 2-Chat generally outperforms existing open-source models and appears comparable to some closed-source models on tested helpfulness and safety evaluations.

  • Takeaways & Limitations

    The paper provides openly accessible models and detailed fine-tuning and safety methods for community research and responsible development.

  • Takeaways & Limitations

    Llama 2-Chat remains subject to limitations including outdated knowledge, unqualified advice, and hallucinations.

Abstract

from arXiv · show

In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.

1 Introduction

The paper introduces and publicly releases Llama 2 and Llama 2-Chat, pretrained and dialogue-optimized LLM families available up to 70B parameters. Llama 2-Chat generally outperforms existing open-source models on tested helpfulness and safety benchmarks and appears comparable to some closed-source models in human evaluations, while requiring application-specific safety testing before deployment.

  • Contribution: Llama 2-Chat generally outperforms existing open-source models on the tested helpfulness and safety benchmarks and appears on par with some closed-source models in human evaluations.The paper points readers to Figures 1 and 3 for these evaluations.
  • Contribution: The work develops and releases Llama 2 and Llama 2-Chat, a family of pretrained and fine-tuned LLMs spanning up to 70B parameters.Llama 2-Chat is optimized for dialogue use cases.
  • Released models: Released Llama 2 variants use 7B, 13B, and 70B parameters, while the 34B variant is reported but withheld because it was not sufficiently red-teamed.Llama 2 also uses a new publicly available data mix, a 40% larger pretraining corpus, doubled context length, and grouped-query attention.
  • Limitations: Because testing was conducted in English and could not cover all scenarios, developers should perform safety testing and tuning tailored to each Llama 2-Chat application.The paper acknowledges that Llama 2 carries potential risks like all LLMs.

2 Pretraining

Llama 2 pretraining combines a revised data and architecture recipe with 2 trillion tokens, longer context, and grouped-query attention, while its base models outperform prior and comparable open-source models across most academic benchmark categories. Llama 2 70B is close to GPT-3.5 on MMLU and GSM8K and on par with or better than PaLM on almost all benchmarks, but remains behind GPT-4 and PaLM-2-L.

  • Pretraining approach: The pretraining recipe uses 40% more tokens, doubled context length, more robust data cleaning, updated data mixes, and grouped-query attention to improve inference scalability for larger models.The models adopt most of Llama 1’s transformer architecture while increasing context length and adding GQA; 34B and 70B models use GQA for improved inference scalability.
  • Pretraining data: The corpus contains 2 trillion tokens from publicly available sources, excludes Meta products and services, removes data from selected sites with substantial private information, and up-samples factual sources to improve knowledge and dampen hallucinations.The authors describe this token count as a good performance–cost trade-off.
  • Training infrastructure and carbon footprint: Pretraining used 3.3M A100-80GB GPU hours and produced an estimated 539 tCO2eq, all directly offset by Meta’s sustainability program.The open release strategy means other companies do not need to incur these pretraining costs.
  • Academic benchmark results: Llama 2 models outperform Llama 1 models, with Llama 2 70B improving MMLU and BBH by ≈5 and ≈8 points over Llama 1 65B, respectively; Llama 2 70B also outperforms all open-source models.Llama 2 7B and 30B outperform corresponding-size MPT models on all categories besides code, while Llama 2 7B and 34B outperform Falcon 7B and 40B on all categories.
  • Academic benchmark results: Llama 2 70B is close to GPT-3.5 on MMLU and GSM8K, performs on par with or better than PaLM (540B) on almost all benchmarks, but has significant coding gaps versus GPT-3.5 and large gaps versus GPT-4 and PaLM-2-L.The comparisons use closed-source academic benchmark results summarized in Table 4.

3 Fine-tuning

Section 3 reports experiments with supervised fine-tuning, reward modeling, RLHF, and Ghost Attention for maintaining dialogue instructions across turns. Llama 2-Chat improves substantially over open-source chat models, while GAtt enables sustained multi-turn control.

  • Reward modeling: Reward models perform best on their corresponding Helpfulness and Safety test sets and outperform all baselines, including GPT-4, reflecting tension between the two objectives.A larger margin for more distinct response pairs improves Helpfulness reward-model accuracy, while larger models continue improving as annotation data increases.
  • RLHF and Rejection Sampling: Rejection Sampling gains more potential reward as the number of sampled outputs increases, while RLHF changes the optimal sampling temperature to T ∈[1.2, 1.3] for 10–100 outputs.Rejection Sampling explores multiple outputs before fine-tuning, whereas PPO updates the policy during training and samples one output per prompt.
  • Ghost Attention: Ghost Attention (GAtt), which concatenates a persistent instruction to user messages during fine-tuning, preserves dialogue control for 20+ turns until the context limit.GAtt was applied after RLHF V3 and also maintained an instruction introduced only at inference time, such as “Always answer with Haiku.”
  • Ghost Attention: GAtt maintains larger attention activations to the system message over more dialogue turns, addressing the tendency of initial RLHF models to forget persistent instructions.The method is inspired by Context Distillation (Bai et al., 2022b).
  • Model progression and evaluation: After RLHF-V3, Llama 2-Chat surpasses ChatGPT on both in-house Safety and Helpfulness axes, although reward-based evaluation may favor Llama 2-Chat.The authors use reward improvements to select intermediate RLHF versions before validating major versions with human evaluation.
  • Model progression and evaluation: 36% win rate and 31.5% tie rate against ChatGPT, while Llama 2-Chat 7B beats MPT-7B-chat on 60% of prompts and 34B exceeds a 75% win rate against Vicuna-33B and Falcon 40B.Llama 2-Chat 70B also outperforms PaLM-bison chat by a large percentage on the evaluated prompt set.

4 Safety

Llama 2’s safety work combines proactive red teaming, safety fine-tuning, context distillation, and evaluation of truthfulness, toxicity, bias, and adversarial robustness. Fine-tuned Llama 2-Chat substantially improves truthfulness and toxicity over pretrained Llama 2, while remaining vulnerable to long-tail, multi-turn, and context-specific risks.

  • Safety Fine-Tuning: Llama 2-Chat generalized from only a few thousand safe demonstrations to produce detailed safe responses, address sensitivity, and provide helpful additional information.Safe responses were often more detailed than those written by the average annotator.
  • Safety RLHF: Safety RLHF improved long-tail safety robustness, while preserving helpfulness, by testing checkpoints with and without adversarial prompts using safety and helpfulness reward models.The comparison is presented as score-distribution shifts in Figure 14.
  • Adversarial Robustness: Context distillation and safety preprompts efficiently bootstrap safer responses to adversarial prompts, while red teaming exposed creative-writing and positive-context attacks against early models.Later models recognized problematic content more often, and the latest models resolved the earlier tendency to provide it anyway.
  • Red Teaming: Red-team refinement reduced the 7B model’s violation-triggering prompt rate γ from 1.8 to 0.45 across iterations.Multiple red-teaming rounds over several months measured robustness as the average number of violating prompts generated per expert per hour.
  • Safety Evaluation: Multi-turn conversations were more likely than single-turn conversations to induce violations, although Llama 2-Chat compared well with baselines, especially in multi-turn settings.Across categories, Llama 2-Chat’s main weakness was unqualified advice, while its violation rates were comparable or lower for the other categories.
  • Truthfulness, Toxicity, and Bias: Truthfulness improved from 50.18 to 64.14 and toxicity fell from 24.60 to 0.01 for 70B after fine-tuning; Llama 2-Chat achieved the best toxicity and truthfulness performance against Falcon and MPT.Toxic generations fell to effectively 0% across all Llama 2-Chat sizes, the lowest toxicity level among compared models.

5 Discussion

The discussion highlights emergent capabilities and mechanisms associated with RLHF, including context-dependent diversity, temporal organization, and zero-shot tool use. It also identifies limitations involving factuality, language coverage, harmful content, misuse, and overcautious safety behavior.

  • 5.1 RLHF properties: RLHF proved highly effective and cost- and time-efficient, with success attributed to synergy between model capabilities and human preference feedback.Preference comparisons let reward models identify low-quality outputs while allowing models to explore writing trajectories beyond annotators’ own writing abilities.
  • 5.1 RLHF properties: RLHF dynamically rescales temperature by prompt type, reducing diversity for factual responses while retaining diversity for creative generation.Figure 21 evaluates 10 creative and 10 factual instructions across temperatures and samples 25 responses per setting.
  • 5.1 RLHF properties: With only 1,000 date-focused SFT examples, Llama 2-Chat robustly organized knowledge temporally, suggesting language models internalize time despite shuffled next-token training data.The authors manually tested dozens of examples and observed consistent temporal generalization.
  • 5.1 RLHF properties: Tool use emerged zero-shot from alignment: Llama 2-Chat inferred tool applications and API arguments and could execute sequences without explicit tool-use annotation.This contrasts with approaches requiring millions of trajectories and few-shot examples for individual tools.
  • 5.2 Limitations: Llama 2-Chat retains common LLM limitations, including outdated knowledge, non-factual advice, hallucinations, harmful or biased outputs, and possible misuse for misinformation or cybercrime.Fine-tuning reduced some harmful behaviors, but unresolved risks remain, especially where non-English safety data are limited.
  • 5.2 Limitations: Performance outside English remains fragile, while safety tuning can become overly cautious by declining benign requests or providing excessive safety details.The initial model used limited non-English pretraining data, and the authors acknowledge that further safety work and updated releases are needed.

6 Related Work

Related work spans the evolution of large language models, instruction tuning and RLHF, and the safety risks associated with deploying tuned systems. While instruction tuning, preference optimization, and synthetic-feedback methods improve alignment, open-source models have generally remained below production-ready closed-source systems, and LLMs continue to present substantial safety challenges.

  • Large Language Models: Large language models have scaled beyond 100B parameters, while Chinchilla (Hoffmann et al., 2022) redirected scaling-law attention toward token counts rather than model size.Examples include GPT-3 (Brown et al., 2020), Gopher, and the specialized scientific model Galactica (Taylor et al., 2022).
  • Large Language Models: Production-ready systems such as ChatGPT, Bard, and Claude differ markedly in performance and usability because they use intricate human-preference alignment techniques that remain under active development in open source.Open-source alignment methods continue to be explored and refined (Gudibande et al., 2023).
  • Large Language Models: Distillation-based models such as Vicuna and Alpaca use synthetic instructions, but still fall short of closed-source counterparts.Their training approach draws on synthetic instructions (Honovich et al., 2022; Wang et al., 2022).
  • Instruction Tuning: Instruction tuning enables zero-shot performance on unseen tasks, with effectiveness studied across task count, model size, prompt settings, and human- or model-generated prompts.Follow-up instructions can refine initial generations to make them more useful, engaging, and u…
  • Instruction Tuning: RLHF fine-tunes models from human feedback to align responses with user expectations, while instruction tuning plus RLHF can address factuality, toxicity, and helpfulness problems that scaling alone cannot fix.RL from AI Feedback partially automates this process by using model self-critiques, revisions, and output rankings (Bai et al., 2022b).
  • Known LLM Safety Challenges: LLM safety research identifies risks including bias, toxicity, private-data leakage, malicious use, harmful-content attacks, emergent behaviors, cyber threats, and misuse in biological warfare.These risks span base-system properties and societal-context evaluations, with red-teaming studies documenting successful attacks on tuned models.

7 Conclusion

Llama 2 is a family of 7B–70B pretrained and fine-tuned models that competes with existing open-source chat models and matches some proprietary models on examined evaluations, while still trailing models such as GPT-4.

  • 7 Conclusion: Llama 2 spans 7 billion to 70 billion parameters and is competitive with existing open-source chat models, achieving competency equivalent to some proprietary models on the examined evaluation sets while lagging behind GPT-4.The study also details the methods and techniques used to develop the models, with particular emphasis on fine-tuning and safety improvements.

A Appendix · A.1 Contributions

Appendix A.1 records the authors’ contribution structure, distinguishing alphabetical ordering, leadership roles, core contributors, contributors, and executive-team support.

  • A.1 Contributions: All authors are sorted alphabetically by last name.
  • A.1 Contributions: Science and Engineering Leadership includes Guillem Cucurull, Naman Goyal, Louis Martin, Thomas Scialom, Ruan Silva, Kevin Stone, and Hugo Touvron.
  • A.1 Contributions: Technical and Management Leadership includes Sergey Edunov, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, and Robert Stojnic.
  • A.1 Contributions: The Core Contributors group comprises 28 named contributors spanning research, engineering, and related project work.The passage lists Peter Albert, Nikolay Bashlykov, Prajjwal Bhargava, Moya Chen, David Esiobu, Jeremy Fu, Vedanuj Goswami, Anthony Hartshorn, Rui Hou, Marcin Kardas, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Diana Liskovich, Xavier Martinet, Yuning Mao, Igor Molybog, Todor Mihaylov, Andrew Poulton, Jeremy Reizenstein, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Jacob Xu, Yuchen Zhang, and Iliyan Zarov.
  • A.1 Contributions: The Contributors group comprises 27 named contributors.The passage lists Amjad Almahairi, Yasmine Babaei, Soumya Batra, Lukas Blecher, Dan Bikel, Shruti Bhosale, Cristian Canton Ferrer, Jude Fernandes, Wenyin Fu, Brian Fuller, Cynthia Gao, Saghar Hosseini, Hakan Inan, Isabel Kloumann, Madian Khabsa, Artem Korenev, Viktor Kerkez, Jian Xiang Kuan, Yinghai Lu, Jenya Lee, Pushkar Mishra, Yixin Nie, Rashi Rungta, Alan Schelten, Kalyan Saladi, Adina Williams, and Zheng Yan.
  • A.1 Contributions: The GenAI executive team is thanked for leadership and support, specifically Ahmad Al-Dahle and Manohar Paluri.

A.2 Additional Details for Pretraining · A.2.1 Architecture Changes Compared to Llama 1

Llama 2 extends the context window from 2,048 to 4,096 tokens and evaluates attention alternatives to improve long-context capability and inference scalability. Based on ablations and scaling considerations, the 34B and 70B models use GQA instead of MQA.

  • A.2.1 Architecture Changes Compared to Llama 1: Llama 2 expands the context window from 2,048 to 4,096 tokens, supporting longer chat histories, summarization, and document understanding.The study compares 2k and 4k pretraining on long-context and general-task ablations, with both models trained for 150B tokens under matched settings.
  • A.2.1 Architecture Changes Compared to Llama 1: Grouped-query attention shares key and value projections across multiple heads, reducing KV-cache memory pressure as context windows, batch sizes, and model sizes increase.The motivation is to address KV-cache growth that becomes a bottleneck in large multi-head attention models.
  • A.2.1 Architecture Changes Compared to Llama 1: In the attention ablation, all variants use 150B training tokens and a fixed 30B model size, with FFN dimensions increased to offset reduced attention parameters.The FFN dimension increases by 1.33× for MQA and 1.3× for GQA; Table 18 reports task-specific evaluation conventions.
  • A.2.1 Architecture Changes Compared to Llama 1: With eight A100 GPUs and tensor parallelism, MQA cannot shard KV values across heads; duplicating them makes its KV-cache size equal to GQA.Under this implementation, MQA and GQA behave very similarly, with MQA having a slightly larger FFN dimension.
  • A.2.1 Architecture Changes Compared to Llama 1: Multi-query variants provide higher throughput at larger batch sizes while maintaining similar latency at smaller batches.In the reported setup, MHA runs out of memory at batch size 1024 with 256-token context and at batch size 128 with 2k context, whereas MQA and GQA complete those runs.
  • A.2.1 Architecture Changes Compared to Llama 1: For the 34B and 70B models, Llama 2 selects GQA over MQA based on ablation results and easier inference scaling.This is the final architectural choice reported for the larger models.

A.2.2 Additional Details for Pretrained Models Evaluation

This section details Llama 2’s evaluation across MMLU, standard benchmarks, code generation, world knowledge, reading comprehension, exams, and mathematical reasoning. On reading comprehension, Llama 2 performs best in every reported setting except QUAC 0-shot, where Llama 1 30B is slightly better.

  • MMLU: The evaluation reports five-shot performance on the MMLU benchmark for Llama 2 and other open-source models.MMLU evaluation details are provided in Table 19.
  • Standard Benchmarks: Llama 2 is evaluated on several standard benchmarks, with results presented in Table 20.
  • Code Generation: Code-generation comparisons cover Human-Eval and MBPP, reporting 0-shot and 3-shot results respectively.Pass@100 and pass@80 use temperature 0.8 and top-p=0.95, while pass@1 uses temperature 0.1 and top-p=0.95.
  • World Knowledge: World-knowledge evaluation compares Llama 2 with other open-source models on NaturalQuestions and TriviaQA.Table 22 reports exact-match performance, including zero-shot and few-shot TriviaQA results on the filtered development set.
  • Reading Comprehension: Llama 2 performs best on all SQUAD and QUAC evaluation settings except QUAC 0-shot, where Llama 1 30B performs slightly better.Table 23 reports zero-shot and few-shot SQUAD results and zero-shot and one-shot QUAC results.
  • Exams and Mathematical Reasoning: Additional evaluations cover English AGI Eval exams and mathematical reasoning on GSM8k and MATH.Results are presented in Tables 24 and 25, with maj1@1 reported for the mathematical reasoning tasks.

A.3 Additional Details for Fine-tuning · A.3.1 Detailed Statistics of Meta Human Preference Data

Meta collected over 1 million binary comparisons across 14 weekly batches of Safety and Helpfulness preference data. Over time, similar-response ratings increased while stronger preferences declined as improved Llama 2-Chat models made annotation more challenging.

  • A.3.1 Detailed Statistics of Meta Human Preference Data: Over 1 million binary model-generation comparisons were collected across 14 weekly batches of Meta Safety and Helpfulness preference data.Table 26 reports the detailed batch statistics.
  • A.3.1 Detailed Statistics of Meta Human Preference Data: Later batches contained more samples as additional annotators joined and existing annotators became more efficient with the tasks.
  • A.3.1 Detailed Statistics of Meta Human Preference Data: The data collection was conducted weekly across 14 batches covering both Safety and Helpfulness preferences.
  • A.3.1 Detailed Statistics of Meta Human Preference Data: The collection intentionally increased the number of multi-turn samples over time.
  • A.3.1 Detailed Statistics of Meta Human Preference Data: The share of samples with similar responses increased dramatically across batches, while samples with stronger preferences declined.Figure 25 plots preference-rating changes over batches; similar responses include negligibly better or unsure cases, whereas stronger preferences include significantly better cases.
  • A.3.1 Detailed Statistics of Meta Human Preference Data: This shift reflected iterative model updates and annotation: better-performing Llama 2-Chat models were used for response sampling, making preferences harder to distinguish.

A.3.2 Curriculum Strategy for Meta Human Preference Data

The authors used a curriculum annotation strategy for alignment data, beginning with simple prompts and progressively introducing more complex prompts and new skills for Llama 2-Chat.

  • A.3.2 Curriculum Strategy for Meta Human Preference Data: A curriculum annotation strategy began with relatively simple prompts, then progressively introduced more complex prompts and new skills to Llama 2-Chat.The strategy was developed in close collaboration with annotation platforms during fine-tuning; Figure 26 illustrates its application to helpfulness preference data.

A.3.3 Ablation on Ranking Loss with Preference Rating-based Margin for Reward Modeling

The ablation evaluates preference rating-based margin terms in the helpfulness reward model’s ranking loss against a no-margin baseline. The margin improves accuracy for more separable response pairs but shifts reward scores toward more extreme values, motivating reward calibration.

  • Ablation on Ranking Loss with Preference Rating-based Margin for Reward Modeling: The ablation compares two preference rating-based margin variants with different magnitudes against ranking loss without a margin term.The variants are listed in Table 27 and evaluated on the Meta Helpful test set.
  • Ablation on Ranking Loss with Preference Rating-based Margin for Reward Modeling: Preference rating-based margin terms improve helpfulness reward-model accuracy on more separable response pairs compared with the baseline without a margin term.The evaluation reports both per-rating and average accuracy on the Meta Helpful test set; two margin variants are compared with the no-margin baseline.
  • Ablation on Ranking Loss with Preference Rating-based Margin for Reward Modeling: The margin pushes reward scores toward more extreme values, with larger margins producing a more significant distribution shift that may make PPO sensitive to reward changes.The observed binary-split reward pattern suggests reward calibration as future work because reinforcement-learning algorithms such as PPO can be sensitive to reward-distribution changes.

A.3.4 Ablation on Ranking Loss with Safety Auxiliary Loss for Reward Modeling … A.7 Model Card

The appendix reports that safety-aware reward modeling improves unsafe-response recall and category accuracy, while GAtt preserves attribute references through 20 turns. It also details human-evaluation procedures, safety analyses, annotation practices, and limited evidence of dataset-contamination effects.

  • A.3.4 Ablation on Ranking Loss with Safety Auxiliary Loss for Reward Modeling: The safety auxiliary loss improves recall of unsafe responses at reward threshold 0.5 and increases accuracy across all three safety categories.Teaching the reward model to discriminate safe from unsafe generations provides a better safety reward signal for RLHF.
  • A.3.5 Additional Results for GAtt: 100% accuracy: GAtt preserves defined attributes through 20 turns, whereas Llama 2-Chat without GAtt falls to 10% at turn t+3 and 0% thereafter.The evaluation used public figures and hobbies, with examples under 4048 total tokens; GAtt also generalized to unseen constraints and beyond Llama 1’s 2048-token pretraining window.
  • A.3.6 How Far Can Model-Based Evaluation Go?: Reward models are overall well calibrated with human preference ratings on helpfulness and safety, supporting their use as point-wise metrics despite pairwise-ranking training.The test set used triple reviews and a 7-point Likert scale.
  • A.3.7 Human Evaluation: Human evaluation compares Llama 2-Chat with open- and closed-source models using over 4000 diverse prompts, randomized side-by-side judgments, and three annotators per pair.Open-source comparisons use 1000-token context and generation limits, while closed-source comparisons use 2000 tokens.
  • A.4.1 Tension between Safety and Helpfulness in Reward Modeling; A.4.2 Qualitative Results on Safety Data Scaling; A.4.5 Safety Errors: False Refusals and Vague Responses; A.4.6 Examples of Safety Evaluation: Safety and helpfulness reward scores can conflict, with unsafe responses receiving high helpfulness scores and safe responses receiving high safety but low helpfulness scores.Increasing safety data makes Llama 2-Chat safer, including refusal of offensive content after using 50% of safety data.
  • A.4.5 Safety Errors: False Refusals and Vague Responses; A.4.6 Examples of Safety Evaluation: Safety-data scaling increases false refusals: rates range from 0.006% to 0.05% on helpfulness prompts and from 15% to 27% on borderline prompts.Examples document vague responses and false refusals caused by context distillation and sensitive keywords in otherwise benign prompts.
  • A.5 Data Annotation; A.5.1 SFT Annotation Instructions; A.5.2 Negative User Experience Categories; A.5.3 Quality Assurance Process; A.5.4 Annotator Selection; A.6 Dataset Contamination: The appendix describes safety-focused annotation instructions, quality review, and multi-step annotator selection, while contamination analysis finds effects mainly on HellaSwag and MMLU-Humanities.For MMLU-Overall on the 70B model, the reported clean-subset versus sampling-mean difference is small at -0.9.
Loading 2307.09288v2…