PaperlayerView exact revision on arXiv

Source-linked AI summary

Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, Madian Khabsa

arXiv:2312.06674v1cs.CLcs.AI

TL;DR

Existing moderation tools do not fully support adaptable, role-specific safeguards for conversational AI inputs and outputs. The paper introduces Llama Guard, an instruction-tuned Llama2-7b classifier built around a configurable safety taxonomy. It achieves strong results on internal and public evaluations, including better AUPRC than all baselines on ToxicChat and comparable zero-shot AUPRC to OpenAI’s moderation API on the OpenAI moderation dataset.

  • Problem

    Existing moderation tools do not distinguish user and agent risks, adapt policies, or support fine-tuning for specific use cases.

  • Method

    Llama Guard is an instruction-tuned Llama2-7b input-output safeguard using taxonomy-labeled data and configurable instructions for prompt and response classification.

  • Results

    Llama Guard surpasses other moderation tools on its internal evaluation, achieves better AUPRC than all baselines on ToxicChat, and shows comparable zero-shot AUPRC to OpenAI’s moderation API on the OpenAI moderation dataset.

  • Takeaways & Limitations

    The released model weights support researchers and practitioners in adapting Llama Guard to different taxonomies and use cases.

  • Takeaways & Limitations

    Most fine-tuning and pretraining data is in English, and the authors do not claim perfect policy coverage, so performance may be subpar in some cases.

Abstract

from arXiv · show

We introduce Llama Guard, an LLM-based input-output safeguard model geared towards Human-AI conversation use cases. Our model incorporates a safety risk taxonomy, a valuable tool for categorizing a specific set of safety risks found in LLM prompts (i.e., prompt classification). This taxonomy is also instrumental in classifying the responses generated by LLMs to these prompts, a process we refer to as response classification. For the purpose of both prompt and response classification, we have meticulously gathered a dataset of high quality. Llama Guard, a Llama2-7b model that is instruction-tuned on our collected dataset, albeit low in volume, demonstrates strong performance on existing benchmarks such as the OpenAI Moderation Evaluation dataset and ToxicChat, where its performance matches or exceeds that of currently available content moderation tools. Llama Guard functions as a language model, carrying out multi-class classification and generating binary decision scores. Furthermore, the instruction fine-tuning of Llama Guard allows for the customization of tasks and the adaptation of output formats. This feature enhances the model's capabilities, such as enabling the adjustment of taxonomy categories to align with specific use cases, and facilitating zero-shot or few-shot prompting with diverse taxonomies at the input. We are making Llama Guard model weights available and we encourage researchers to further develop and adapt them to meet the evolving needs of the community for AI safety.

1 Introduction

Llama Guard addresses limitations of existing moderation tools by using an LLM-based safeguard that separately classifies prompts and responses under adaptable safety taxonomies. The released model weights support further customization for AI-agent use cases.

  • Existing moderation tools do not distinguish user safety risks from risks in AI-agent responses, limiting their suitability as input-output guardrails.
  • Existing tools enforce fixed policies, lack fine-tuning access, and use smaller transformer backbones than more capable LLMs.
  • Llama Guard publicly provides an LLM-based safeguard for classifying safety risks in conversational-AI prompts and responses.
  • The model introduces a safety-risk taxonomy covering potential legal and policy risks applicable to developer use cases.
  • Llama Guard is fine-tuned on taxonomy-labeled data and accepts the applicable taxonomy as input, enabling zero-shot or few-shot adaptation to other taxonomies.
  • Different instructions classify human prompts versus AI responses with one model, capturing their distinct user and agent roles.
  • The released model weights allow practitioners and researchers to use, experiment with, and fine-tune Llama Guard without depending on paid APIs.

2 Safety Risk Taxonomy

The taxonomy defines the risk classes and guidelines needed for automated input-output safeguards. Its sample categories cover common legal, policy, and harm risks while remaining intended as a broadly applicable, non-comprehensive taxonomy.

  • Automated input-output safeguards require a risk taxonomy and guidelines specifying which outputs are encouraged or discouraged in each category.
  • The authors created a sample taxonomy because no single standard exists for mitigating unintended outputs from large language models.
  • The taxonomy is designed to be broadly applicable while demonstrating that LLMs can be tuned into adaptable, high-performing classifiers.
  • Risk categories: Violence & Hate covers content encouraging violence, discrimination, slurs, or hateful sentiments based on sensitive personal characteristics.
  • Risk categories: Sexual Content covers encouragement of specific sex acts, including involving someone who could be underage, and sexually explicit statements.
  • Risk categories: Guns & Illegal Weapons covers assistance with illegal acquisition, creation, or use of guns, explosives, biological agents, or chemical weapons.
  • Risk categories: Regulated or Controlled Substances covers assistance with illegally producing, transferring, or consuming drugs, tobacco, alcohol, or cannabis.
  • Risk categories: Suicide & Self Harm covers encouragement or enablement of self-harm, including methods, and requires appropriate health resources when intent is expressed or implied.

3 Building Llama Guard

Llama Guard frames safeguarding as instruction-following classification over guidelines, conversation content, task type, and output format. Its design supports taxonomy adaptation through prompting, fine-tuning, and category-focused data augmentation.

  • Llama Guard fine-tunes an LLM with instruction-following tasks that classify content as safe or unsafe for input-output safeguarding.
  • Task components: Each task supplies numbered risk categories and safety guidelines that determine which content the model should assess.
  • Task components: Tasks specify whether to classify user messages as prompts or agent messages as responses.
  • Task components: The same model separates prompt and response moderation through instruction wording, requiring no significant added effort.
  • Task components: Each task contains a single-turn or multi-turn conversation in which users and agents take turns.
  • Task components: The output format produces safe or unsafe and, for unsafe content, lists violated taxonomy categories.
  • Task components: The model’s prompt terminology distinguishes the agent’s prompt from the “Llama Guard prompt” used to instruct the safeguard.
  • Adaptation: Zero-shot and few-shot prompting adapts Llama Guard to new taxonomies and guidelines without necessarily changing its fine-tuning data.

4 Experiments

The experiments address taxonomy mismatch by measuring both in-domain performance and adaptability to foreign taxonomies. They compare several evaluation strategies, including binary, 1-vs-all, and 1-vs-benign classification.

  • Experimental axes: Taxonomy mismatch makes cross-model and cross-dataset comparisons challenging because models and test sets use different category definitions.Category mappings can be unclear or subjective, and some APIs lack corresponding categories.
  • Experimental axes: The evaluation uses two axes: absolute performance on Llama Guard’s own datasets and adaptability to other taxonomies.For foreign taxonomies, Llama Guard uses zero-shot or few-shot prompting and taxonomy-specific fine-tuning.
  • Evaluation methods: Overall binary classification assigns a positive score by taking the maximum predicted score across all positive categories.The method uses each classifier’s category scores without checking whether a predicted category aligns with the target category.
  • Evaluation methods: Per-category 1-vs-all evaluation treats one target category as positive and all other samples as negative, enabling on-the-fly task customization.Llama Guard changes the model input to tailor classification tasks in both on-policy and off-policy settings.
  • Evaluation methods: Per-category 1-vs-benign evaluation drops samples from other positive categories, retaining only benign examples as negatives.This accommodates fixed category-wise output heads when scores cannot be straightforwardly mapped to foreign target categories.
  • Evaluation caveat: Removing hard negatives for a target category can produce optimistic off-policy results, so the approach is applied consistently to the baseline APIs.The authors identify this as a caveat of their off-policy evaluation procedure.
  • Public benchmarks: The public evaluation includes ToxicChat’s 10k real-world user-AI interaction samples and the OpenAI Moderation Evaluation Dataset’s 1,680 prompts.Llama Guard is adapted to each benchmark by supplying its taxonomy and a brief description in the input prompt.

4.3 Baselines & Evaluation Metrics

The study compares Llama Guard with probability-score-based moderation APIs and GPT-4 using AUPRC-based evaluations. Llama Guard performs strongly in-domain and adapts effectively to external benchmarks with zero-shot target-taxonomy prompts.

  • Baselines: The baselines include OpenAI Moderation API, Perspective API, Azure AI Content Safety API, and GPT-4.These systems differ in category coverage and output formats, including probabilities, integer severity scores, or prompted judgments.
  • Baselines: Azure’s integer category scores are binarized using a threshold from 1-6 selected for the highest average precision on each dataset.This modified max-all procedure compensates for Azure not returning probability scores.
  • Evaluation metrics: AUPRC is the evaluation metric because it measures the precision-recall trade-off for the positive unsafe class and supports threshold selection for use cases.Average precision is infeasible for Azure API and GPT-4 under the reported setup.
  • Reported results: Table 2 reports benchmark-level AUPRC comparisons, while Table 3 reports prompt- and response-classification breakdowns by safety category.Higher AUPRC is better, and Table 3 lists prompt results before response results in each cell.
  • Reported results: Llama Guard achieves very high scores on its own test set overall and across individual categories, indicating a high in-policy performance ceiling.This finding concerns both prompt and response safeguarding evaluation on the model’s own test set.
  • Reported results: Llama Guard performs close to OpenAI’s API on the OpenAI Moderation Evaluation Dataset without examples and outperforms every other method on ToxicChat.The ToxicChat comparison uses a dataset against which none of the evaluated models was trained.

4.5 Studying the Adaptability of the Model

Llama Guard adapts to new safety taxonomies through prompting and fine-tuning. Prompt-based adaptation improves alignment with the OpenAI taxonomy, while fine-tuning on another taxonomy accelerates adaptation to ToxicChat.

  • 4.5.1 Adaptability via Prompting: Providing a taxonomy description improves Llama Guard’s alignment with the OpenAI moderation taxonomy.The comparison covers no adaptation, category adaptation, and few-shot learning on the OpenAI-Mod dataset.
  • 4.5.1 Adaptability via Prompting: Category-wise AUPRC is lower than binary-classification performance because 1-vs-all evaluation penalizes predicting the wrong category for an unsafe sample.The policy mismatch makes this category-level setting harder, including because Llama Guard merges categories distinguished by the OpenAI taxonomy.
  • 4.5.2 Adaptability via Fine-tuning: Fine-tuning on ToxicChat improves performance on that task and tests whether prior training on a different taxonomy helps adaptation.The study uses 10%, 20%, 50%, and 100% of ToxicChat training data and compares against Llama2-7b under the same setup.
  • 4.5.2 Adaptability via Fine-tuning: 20% of ToxicChat data lets Llama Guard perform comparably to Llama2-7b trained on 100%, while equal-data training can yield better performance.
  • 4.5.2 Adaptability via Fine-tuning: In zero-shot evaluation, Llama Guard achieves 0.626 AUPRC, whereas Llama2-7b is assigned zero because it produced malformed outputs.The released Llama Guard model is not further fine-tuned on ToxicChat; the authors invite further cross-taxonomy study.

5 Related Work

Related work spans zero-shot and few-shot LLM inference, human-content moderation, and safeguards for LLM-generated content. LLM safeguarding differs from conventional moderation because it must assess broader harms and prompt-response interactions.

  • Zero-shot and few-shot inference using LLMs: Llama Guard uses zero-shot prompting and in-context few-shot learning to adapt to unseen policies and taxonomies.Few-shot inference adds examples without training on them.
  • Moderation of human-generated content: Human-generated content moderation has produced many datasets, largely from online social networks and often focused on harms such as hate speech.
  • Moderation of human-generated content: Guarding LLM-generated content differs because generated text has different style and length, involves broader harms, and requires handling prompt-response pairs.
  • Guarding LLM-generated content: Safe LLM dialogue systems require checking model responses as well as human-generated content, including responses to offensive inputs.Prior work also surveys the safety landscape and proposes a framework for launch decisions.
  • Guarding LLM-generated content: ToxicChat targets violations in LLM-generated content from user prompts and generations, but related datasets classify user prompts rather than generated outputs.

6 Limitations & Broader Impacts

Llama Guard has limitations tied to its training data, language coverage, policy coverage, and use outside its intended classification setting. The authors also caution that prompting it as a chat model or attacking it with prompt injection may alter its intended behavior.

  • Llama Guard may produce wrong safety judgements when knowledge required for assessment extends beyond its training and pretraining data.
  • Because most fine-tuning and pretraining data is in English, the authors do not guarantee adequate performance in other languages.
  • The fine-tuning labels do not provide perfect policy coverage, so some cases may receive subpar performance.
  • Llama Guard was trained for classification with a limited output space, not as a general chat model.
  • When prompted as a chat model, it may generate unethical or unsafe language because it lacks safety fine-tuning for chat use.
  • As an LLM, Llama Guard may be susceptible to prompt injection attacks that alter or bypass its intended use.

7 Conclusion

The paper introduces Llama Guard as an input-output safeguard for human-AI conversations, together with a safety risk taxonomy and policy-guided training data. It reports stronger or comparable performance than available moderation tools across internal and public evaluations, while also showing adaptation to new policies through fine-tuning.

  • Llama Guard is an LLM-based safeguard for classifying safety risks in human prompts and AI responses.
  • The authors introduce a safety risk taxonomy and policy-guided data used to train Llama Guard.
  • Llama Guard surpasses other content moderation tools on the internal evaluation set in aggregate and per-category performance.
  • On ToxicChat, Llama Guard achieves better AUPRC than all baselines.
  • On the OpenAI moderation dataset, Llama Guard has comparable zero-shot AUPRC and higher AUPRC with in-context examples than the OpenAI moderation API.
  • Further fine-tuning adapts Llama Guard to a novel dataset and is more data-efficient and performant than training from scratch for that dataset.

B Further comparisons

The further comparisons report thresholded precision, recall, and F1 evaluations for prompt and response classification across the safety categories in the authors’ dataset. AUPRC could not be computed for baselines without output probabilities, so these metrics provide a probability-independent comparison.

  • Precision, recall, and F1 are computed at a threshold of 0.5 when baselines do not provide output probabilities.
  • Table 5 breaks down prompt-classification performance by Violence and Hate, Sexual Content, Criminal Planning, Guns and Illegal Weapons, Regulated or Controlled Substances, and Self-Harm.
  • Table 6 provides the corresponding response-classification performance breakdown across the same safety categories.
Loading 2312.06674v1…