Granite Guardian

1 Introduction

• The report frames robust prompt- and response-risk detection as necessary for safe LLM deployment, with safeguards also used for moderation, output evaluation, and RAG pipelines.It identifies transparency in development and deployment as important to trust and accountability.Source: §1 · p. 1
• AUC scores of 0.871 on guardrail and 0.854 on RAG-hallucination benchmarks are reported for Granite Guardian.The report also says the models outperform other open- and closed-source models on deployment-focused metrics including AUC, F1, and ROC.Source: §1 · p. 2
• Granite Guardian unifies harmful-content detection with RAG risks including context relevance, groundedness, and answer relevance in a 2B and 8B model family.The models combine human-annotated data from diverse sources with synthetic data, including data for jailbreak and RAG-related risks.Source: §1 · p. 2
• Prior risk-detection approaches are described as falling into harmful-content and RAG-hallucination categories, which Granite Guardian brings together in one model family.The related-work discussion names Llama Guard and ShieldGemma for harmful-content risks and task-specific approaches for RAG hallucination risks.Source: §1 · p. 2

2 Risks in LLMs

• LLM risks can arise from prompts or responses, which differ because prompts are externally provided while responses are model-generated.The report says Granite Guardian is designed to detect risks in both dimensions.Source: §2 · p. 3

2.1 Types of risks addressed

• Granite Guardian covers social, security, and RAG risks, including unethical behavior, social bias, violence, profanity, sexual content, jailbreaks, and custom-defined risks.Its RAG dimensions are context relevance, groundedness, and answer relevance.Source: §2.1 · p. 3

2.1.1 Harmful content risks

• Harmful-content risks include unsafe behavior or misuse, with explicit detection dimensions spanning violence, profanity, sexual content, unethical behavior, social bias, and jailbreaking.The umbrella harm category is intended for broad, out-of-the-box detection and applies to prompts and responses.Source: §2.1 · p. 3, §2.1.1 · p. 3

2.1.2 RAG hallucination risks

• RAG can still hallucinate when retrieved context is irrelevant or conflicting, so the report identifies context relevance, groundedness, and answer relevance as risks to evaluate.Groundedness concerns claims unsupported by or contradicted by context; answer relevance concerns whether the response addresses the input.Source: §2.1.2 · pp. 4–5

3 Datasets

• The supervised-fine-tuning dataset combines open-source and synthetic data with external human annotations that are processed for training.The report describes the training data as requiring high-quality labeled examples.Source: §3 · p. 5

3.1 Human annotations

• Three individuals independently labeled each prompt-response pair for input and output safety and specific risk categories.Categories included bias, jailbreaking, violence, profanity, sexual content, unethical behavior, AI refusal, and other.Source: §3.1 · p. 5
• The first phase produced 7,000 annotated pairs, while a later phase selected 1,000 low-confidence examples from 11,000 candidates.The later selection included 400 unsafe and 600 safe examples to maintain balance.Source: §3.1 · pp. 5–6

3.2 Synthetic data

• Synthetic data targets complex benign-versus-harmful prompts, jailbreak attacks, and RAG-specific risks.The generation process uses LLMs and organized taxonomies to create complex benign prompts, harmful variants, and responses.Source: §3.2 · p. 7, §3.2.1 · p. 8

3.2.1 Complex benign and harmful data

• Benign prompts draw on ten safety categories, while harmful prompts use a four-category, 13-subcategory taxonomy and adversarial transformations.The transformations use 24 revision strategies, applied singly or as three randomly sampled strategies.Source: §3.2.1 · pp. 8–9
• Jailbreak training data combines seed examples, automated red-teaming, and synthetic generation to capture adversarial styles and underlying intent.The work includes payload splitting and other attack techniques; the resulting samples were subsampled and labeled for jailbreak risk.Source: §3.2.1 · pp. 9–11

3.3 RAG hallucination risk data

• RAG risk samples use HotPotQA and SQuADV2 seeds, with MNLI and SNLI also included for groundedness.Generated negatives cover non-relevant questions, non-relevant contextual answers, and incorrect contextual answers.Source: §3.3 · p. 11

4.1 Safety instruction template

• The safety template specifies whether to inspect a prompt or response, supplies tagged content and a risk definition, and requests a Yes/No decision.The template is designed to accommodate new risk definitions during deployment.Source: §4.1 · p. 13

4.2 Supervised fine-tuning

• The 2B and 8B variants are supervised-fine-tuned from Granite 3.0 instruct models using the safety template as user content and verbalized labels as assistant responses.The instruct variants performed better than base variants in the authors’ experiments; checkpoints were selected by validation cross-entropy loss.Source: §4.2 · p. 13

4.3 Computing probability of risk

• Granite Guardian aggregates top-k token log-likelihoods containing “Yes” or “No” to estimate unsafe and safe class scores.Matching uses lowercase, stripped text to include lexical variations of the labels.Source: §4.3 · p. 13
• With k set to 20, the aggregated scores are softmax-normalized into class confidence, while the first generated token supplies the Yes/No label.Deployment thresholds can be applied to the resulting probability of risk.Source: §4.3 · p. 14

5 Evaluation

• Evaluation covers umbrella harm detection and RAG groundedness, using eight harmfulness datasets and nine TRUE datasets for groundedness.Harmful labels follow each dataset’s safety taxonomy; evaluation focuses on harm and groundedness while standardization across other risk dimensions continues to evolve.Source: §5 · p. 14, §5.3 · p. 15

5.1 Metrics

• AUPRC, AUC, F1, recall, and precision are reported, with threshold-based metrics calculated at 0.5.Additional recall and AUC measurements at false positive rates of 0.1, 0.01, and 0.001 assess performance under low-FPr constraints.Source: §5.1 · p. 14

5.2 Baselines

• Llama Guard and ShieldGemma serve as harmful-content baselines, with safety taxonomies and templates supporting comparison to Granite Guardian’s umbrella harm definition.Llama Guard targets Human-AI conversation scenarios; ShieldGemma evaluates prompts and responses against defined safety policies.Source: §5.2 · pp. 14–15
• Groundedness baselines are ANLI-T5-11B, WeCheck-0.4B, and Llama-3.1-Bespoke-MiniCheck-7B.They use, respectively, adversarially collected hypotheses, weakly labeled synthetic data, and response-level atomic-fact grounding scores.Source: §5.2 · p. 15

Prompt harmfulness

• Prompt-harm benchmarks include real-user ToxicChat queries, manually annotated AegisSafetyTest and SimpleSafetyTests samples, and HarmBench prompts designed to probe jailbreak robustness.HarmBench also includes standard and copyright behaviors; contextual and multimodal behaviors are excluded.Source: Prompt harmfulness · pp. 15–16

Response harmfulness

• Response-harm evaluation includes BeaverTails annotations, SafeRLHF safe-versus-unsafe preference comparisons, and XSTEST-RESP refusal and compliance dimensions.XSTEST-RESP separates response harmfulness, refusal rate, and explicit compliance with harmful requests.Source: Response harmfulness · pp. 16–17

RAG datasets

• TRUE groundedness datasets include summarization outputs from FRANK, SummEval, MNBM, and QAGS.These datasets assess factual consistency or hallucinations in generated summaries across CNN/DailyMail and XSum materials.Source: RAG datasets · p. 17
• Other TRUE tasks include paraphrase assessment with PAWS and groundedness in knowledge-grounded dialogue with BEGIN, Q2, and DialFact.Q2 annotates generated dialogue responses for binary factual consistency against the provided knowledge paragraph.Source: RAG datasets · p. 17

6.1 Harm risk benchmarks

• 0.871 AUC and 0.846 AUPRC on aggregated harmful-content benchmarks are achieved by Granite-Guardian-3.0-8B, which also scores 0.758 F1 at threshold 0.5.The reported ROC curve shows a true positive rate of 0.68 at approximately 0.1 false positive rate.Source: Table 6 · p. 18, Figure 8 · p. 18, §6.1 · p. 19
• 0.782 AUC and 0.746 AUPRC on aggregated harmful-content benchmarks are achieved by Granite-Guardian-3.0-2B, with F1 0.674 and recall 0.747.The 2B model is designed for resource-constrained scenarios and is described as viable for efficiency-critical applications.Source: Table 6 · p. 18, §6.1 · p. 19
• Dataset-specific results report best aggregate AUC and F1 across baselines for the 8B model, with F1/AUC comparisons spanning prompt- and response-harm datasets.The cited results include ToxicChat, BeaverTails, and XSTest-RH.Source: §6.1 · p. 19, Table 7 · p. 19

6.2 RAG hallucination risk benchmarks

• 0.854 average AUC across TRUE groundedness datasets places Granite-Guardian-3.0-8B second overall and makes it the best-performing fully open-source community model.It outperforms other models on three datasets and places second on four others, despite being trained for broader risk detection.Source: §6.2 · pp. 19–20, Table 8 · p. 20

7.1 Usage

• Use the prescribed scoring mode; deviations from intended use or adversarial attacks may produce unexpected, potentially unsafe outputs.The models were trained and tested on English data; custom risk definitions require testing.Source: §7.1 · p. 20

7.2 Limitations

• Assessments may lack nuance when context about creation circumstances, intent, or social conditions is unavailable.The stated scope does not fully accommodate context-awareness, so the authors emphasize thorough testing and informed application.Source: §7.2 · p. 20
• Annotation uses multiple annotations per sample and a diverse annotator pool, but scaling quality and diversity is resource-intensive.The authors also note limited incentives for thorough subcategory coverage and subjectivity in labeling nuanced content.Source: §7.2 · p. 20
• Risk detection lacks standardized definitions for some risks and robust evaluation benchmarks, hindering comprehensive assessment.Source: §7.2 · p. 20

• Granite Guardian detects prompt and response risks, including RAG context relevance, groundedness, answer relevance, jailbreaks, and custom risks.Source: §8 · p. 21
• The models integrate with any LLM and are tailored for enterprise use cases.Source: §8 · p. 21
• The report says Granite Guardian outperforms competitors on benchmarks and is released as open-source.Its training is described as transparent and supported by diverse human annotations.Source: §8 · p. 21

• Table 9 is titled “Risk Definitions.”Source: Table 9 · p. 27

• Privacy categories include sensitive organizational and individual information, while misinformation includes false or misleading information and material harm.Source: B Taxonomy { · p. 27
• Harmful-language categories include social stereotypes and discrimination, violence and physical harm, toxic language or hate speech, and sexual content.Source: B Taxonomy { · p. 28
• Malicious-use categories include cyberattacks, fraud and illegal activities, unethical or unsafe actions, and mental health or over-reliance concerns.Source: B Taxonomy { · p. 29

C Template

• Table 10 defines Primary as the tag setting the safety agent’s focus and Secondary as the tag specifying instruction content alongside Primary.Harm++ refers to all harmful content risks.Source: Table 10 · p. 29
• Table 11 reports AUC and TPr at FPr thresholds of 0.1, 0.01, and 0.001, marking column-best and second-best values.Bold indicates the best value within a column; underlining indicates the second-best.Source: Table 11 · p. 30

D Further Results

• AUC captures behavior across changing thresholds, whereas threshold-fixed metrics such as TPr, FPr, and accuracy describe behavior at fixed thresholds.The threshold can be adjusted to reflect the desired TPr/FPr trade-off.Source: D Further Results · p. 30
• The evaluation fixes FPr at 0.1, 0.01, and 0.001 to assess positive detection while limiting service interruptions.The section notes that low FPr is important for real-time applications.Source: D Further Results · p. 30
• Granite-Guardian-3.0-8B consistently achieves higher partial AUC and TPr than Granite-Guardian-3.0-2B across FPr thresholds, especially for AUC@0.1 and AUC@0.01.Its TPr advantage is also marked at stricter FPr levels, including TPr@0.001.Source: D Further Results · p. 30

D.1 Metrics and datasets fine-grained results

• Figure 9 presents macro F1 scores for Granite Guardian models and baselines across datasets.Source: Figure 9 · p. 30, §D.1 · p. 30
• Figure 10 presents F1 scores for Granite Guardian models and baselines across datasets.Source: §D.1 · p. 30, Figure 10 · p. 31
• Figure 11 presents TPr for Granite Guardian models and baselines across datasets.Source: §D.1 · p. 30, Figure 11 · p. 31
• Figure 12 presents FPr for Granite Guardian models and baselines across datasets.Source: §D.1 · p. 30, Figure 12 · p. 31