PaperlayerView exact revision on arXiv

Source-linked AI summary

Guard Models Are Overconfident Where Base Models Are Uncertain

Jonghyun Hong, MinJae Jung, Minwoo Kim

arXiv:2609.36477v1cs.LGcs.AI

TL;DR

Guard confidence may be unreliable under adversarial attack, despite several guards being nearly calibrated on clean inputs. The paper evaluates five guards and compares them with corresponding base LMs, finding confident false negatives alongside greater base-model uncertainty and later-layer representation divergence.

  • Problem

    Guard confidence drives moderation decisions, but its reliability under adversarial attack remains underexplored.

    Source: §1 · p. 1

  • Method

    The paper evaluates five guard models and compares each with its corresponding base LM using a shared binary safe/unsafe interface.

    Source: §1 · p. 2

  • Results

    Across five guards, adversarial attacks produce high-confidence false negatives, while corresponding base LMs are usually more uncertain on those cases and divergence emerges in later layers.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Guard evaluation should consider whether confidence remains trustworthy, since base-model uncertainty often remains informative where guard confidence fails.

    Source: §1 · p. 2, §5 · p. 9

    The representation evidence is correlational, with no intervention establishing that the observed geometry produces misclassifications.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Guard models are used as safety classifiers, with confidence scores driving downstream moderation decisions. We evaluate five guard models for prompt classification and find that although several are nearly calibrated on clean inputs, adversarial attacks degrade their calibration by an order of magnitude, turning false negatives into high-confidence errors indistinguishable from correct detections. Comparing each guard with its corresponding base LM, we find that uncertainty signals often remain available, with the base model typically expressing uncertainty on the same inputs where the guard fails. Layer-wise analyses localize this guard-base divergence to later layers, where guard models exhibit sharper safe/unsafe separation and lower-rank representations, while adversarial harmful inputs lie closer to the clean-safe region. These findings highlight a mismatch between guard confidence and base model uncertainty under attack.

1 Introduction

The introduction frames guard confidence as a deployment concern and reports that adversarial attacks make guard errors highly confident. Comparing guards with their corresponding base LMs reveals that uncertainty often remains available, with divergence emerging in later layers.

1 Introduction

  • Broader LLM deployment amplifies safety risks including misuse, manipulation, and jailbreak attacks that induce harmful generations.The introduction presents safety and reliability of deployed LLM systems as a major challenge.Source: §1 · p. 1
  • Guardrails monitor user inputs and model outputs, while guard models classify prompts or responses and often drive allow, flag, or block decisions through confidence thresholds.Their role is to enforce policy-specific safety constraints around base models at inference time.Source: §1 · p. 1
  • The reliability of confidence scores used in moderation remains underexplored, motivating matched guard–base comparisons of uncertainty and representations.Prior work reports guard overconfidence under jailbreak inputs and partial mitigation with post-hoc calibration methods.Source: §1 · p. 1
  • The study evaluates five guard models across six attack methods spanning gradient-based, template-based, and LLM-based strategies, including a custom attack.It compares each guard with its corresponding base LM.Source: §1 · pp. 1–2
  • Adversarial attacks degrade guard calibration by an order of magnitude, with false negatives reaching confidence levels comparable to correct detections.Even at a 0.99 threshold, substantial fractions of adversarial harmful prompts remain high-confidence false negatives.Source: §1 · p. 2, Figure 1 · p. 2, Figure 2 · p. 3
  • On most guard false negatives, corresponding base LMs assign higher entropy to the same input, indicating an uncertainty gap.The comparison uses a shared binary safe/unsafe interface and a 5-shot classification prompt for base LMs.Source: §1 · p. 2
  • Guard–base divergence emerges in later layers, where guards show lower unsafe probabilities, lower-rank representations, and sharper safe/unsafe separation.Adversarial harmful inputs often lie closer to the clean-safe region.Source: §1 · p. 2

2 Related Work

Related work distinguishes interactive LLM guard models from content-moderation systems and notes that prior evaluations emphasize detection accuracy more than confidence reliability. It also reviews evidence that pre-trained base models retain useful uncertainty, a premise this paper uses diagnostically rather than as a correction method.

2 Related Work

  • LLM-specific guard models classify prompts or responses against safety taxonomies, while system frameworks address deployment routing and efficiency.Prior benchmarks include HarmBench, AdvBench, ToxiGen, and RigorLLM.Source: §2 · p. 3
  • Prior guard research focuses on detecting harmful content, not whether confidence in those decisions is trustworthy.Content moderation systems detect toxic human-generated text but do not address interactive LLM deployment risks such as jailbreak attempts and unsafe generations.Source: §2 · p. 3
  • Pre-trained base LLMs are reported to retain calibrated uncertainty, and BaseCal and Dual-Align use base-model representations or layer-wise alignment for correction.This work shares the base-model reference premise but uses matched comparison diagnostically rather than proposing a correction method.Source: §2 · p. 3

3 Background

The section defines guard models as prompt-level safety classifiers and explains how their binary verdicts and confidence are computed. It also defines calibration and Expected Calibration Error (ECE) for evaluating those confidence scores.

3 Background

  • Guard models classify prompts as safe or unsafe, deciding whether an input should be allowed or blocked before it reaches a downstream model.Deployed guards implement this classification through autoregressive generation of a safety label.Source: §3 · p. 3
  • A restricted softmax over the safe and unsafe label logits yields a binary safety distribution, measuring decision confidence rather than full-vocabulary probability mass.The unsafe probability is normalized using the logits for both allowed labels.Source: §3 · p. 4
  • Confidence is the probability assigned to the predicted label, while accuracy measures whether that label matches the ground truth.Clean benign prompts are labeled safe and clean harmful prompts unsafe; adversarial prompts remain harmful despite jailbreak wrapping.Source: §3 · p. 4
  • Calibration means confidence matches empirical accuracy, and lower ECE indicates better calibration across confidence bins.ECE compares each bin’s empirical accuracy with its mean confidence.Source: §3 · p. 4

4 Guard Confidence Under Adversarial Attack

Across five guards, adversarial attacks sharply worsen confidence calibration, while comparisons with corresponding base LMs reveal retained uncertainty on many guard failures. Layer-wise analyses locate the divergence in later layers, where guards form sharper, lower-rank representations that can place adversarial harmful inputs near the clean-safe region.

4.1 Experimental Setup

  • Guards were compared with corresponding base LMs on clean and adversarial prompts; the evaluation covered five guard models and six jailbreak methods.The clean pool included 713 harmful and 500 benign prompts; the adversarial pool contained 3,965 harmful prompts.Source: §4.1 · p. 4

4.2 Guard Confidence Becomes Overconfident Under Attack

  • Llama-Guard-3 ECE rose from 0.013 to 0.308 and WildGuard ECE from 0.004 to 0.196 under adversarial inputs.The degradation is concentrated in high-confidence bins.Source: §4.2 · p. 5
  • At τ = 0.99, Llama-Guard-3, WildGuard, and ShieldGemma retain 11%, 12%, and 37% of adversarial harmful prompts as high-confidence false negatives.Because false negatives and true positives receive similarly high confidence, abstaining only when uncertain does not recover these misses.Source: §4.2 · p. 5

4.3 Base LMs Retain Uncertainty Under Attack

  • Across four matched pairs, guard verdict shifts exceeded base-model shifts, with ∆AS ranging from +1.71 for Llama-Guard-3 to +4.91 for ShieldGemma.The comparison uses prompts both models confidently classified as unsafe when clean.Source: §4.3 · p. 6, Figure 3 · p. 6
  • On false negatives, ∆H was positive for 82% of Llama-Guard-2, 80% of Llama-Guard-3, and 93% of WildGuard cases; ShieldGemma was 51%.Positive ∆H means the base model is more uncertain than the guard on the same input.Source: Figure 4 · p. 6, §4.3 · p. 7

4.4 Where and How Overconfidence Arises

  • At the final layer, adversarial unsafe-probability gaps reached approximately 0.45 for Llama-Guard-3, 0.7 for WildGuard, and 0.9 for ShieldGemma.The divergence grows through the final third of layers, while clean harmful-input gaps remain near zero or slightly negative.Source: §4.4 · p. 7, Figure 5 · p. 7
  • Guard representations sharpen the late-layer safe/unsafe boundary, but adversarial harmful inputs lie closer to the clean-safe side where guards confidently misclassify them.Base-model representations remain largely intermixed across clean safe, clean harmful, and adversarial harmful inputs.Source: §4.4 · p. 7, Figure 6 · p. 8
  • At the penultimate layer, base-to-guard effective-rank ratios were roughly 1.4× for Llama-Guard-2, 1.7× for Llama-Guard-3, 3.4× for WildGuard, and 4.0× for ShieldGemma.The compressed guard subspace separates clean inputs but can collapse adversarial harmful inputs toward the clean-safe side.Source: §4.4 · p. 8, Figure 7 · p. 8

5 Conclusion

The conclusion finds that adversarial attacks make guard confidence unreliable even when base-model uncertainty remains informative on the same failures. Layer-wise analyses associate this gap with later-layer differences in probabilities and representation geometry.

5 Conclusion

  • Across five guard models, adversarial false negatives receive confidence comparable to correct detections, making them difficult to remove by thresholding.The conclusion recommends evaluating whether confidence remains trustworthy under attack, not only detection accuracy.Source: §5 · p. 9
  • On the same false-negative cases, base models are usually more uncertain than their corresponding guards, so the failure is not inherent to the inputs alone.The paper identifies base-model uncertainty as a signal a calibration-aware guardrail could draw on.Source: §5 · p. 9
  • Layer-wise analyses localize the gap to later layers, where guards show lower unsafe probabilities, lower-rank representations, and sharper safe/unsafe separation.Adversarial harmful inputs lie closer to the clean-safe region.Source: §5 · p. 9
  • The conclusion calls for evaluating guard confidence under attack alongside detection accuracy.It presents base-model uncertainty as a potentially useful signal, not as a completed calibration method.Source: §5 · p. 9

Limitations

The study identifies later-layer guard–base divergence and associated representation geometry, but its evidence is correlational and does not establish the mechanism behind confident failures.

Limitations

  • The observed logit gap, sharper safe/unsafe boundary, and lower-rank subspace accompany confident guard failures, but the study does not establish that they produce the misclassifications.The authors say causal interventions, such as ablating or steering identified directions, would be needed to establish the mechanism.Source: Limitations · p. 9
  • The analysis is diagnostic: it finds base-model uncertainty remains informative where guard confidence fails, but does not turn that signal into a calibration method.The paper also leaves open the conditions under which the gap could be closed.Source: Limitations · p. 9
  • The study does not intervene on the identified quantities, so representation geometry may co-occur with confident misclassifications rather than cause them.Source: Limitations · p. 9

Ethical Considerations

The paper frames its work as an analysis of guard-model vulnerabilities for safer deployment and reports that experiments were conducted in controlled research settings.

Ethical Considerations

  • Experiments were conducted in controlled research settings and are presented with an emphasis on mitigation, robustness, and calibration-aware evaluation.Source: Ethical Considerations · p. 9
  • The attacks use previously published methods or optimization frameworks, and the datasets come from publicly available benchmarks.Source: Ethical Considerations · p. 9
  • The authors acknowledge dual-use risk in exposing calibration failures and state that no real-world harm was caused or intended; the paper includes potentially offensive model-output examples.Source: Ethical Considerations · p. 9

A Base-LM Prompting and Interface Robustness

The appendix describes the 5-shot prompt and verdict-token interface used to evaluate base LMs alongside guards. It also reports validation of the binary-label interface and identifies the scope of an alternative-interface robustness table.

A.1 Base-LM Prompt and Verdict-Token Validation

  • The base-LM classifier uses a 5-shot prompt with three safe and two unsafe demonstration pairs followed by a query placeholder.Source: §A.1 · p. 12

Prompt: Base 5-shot safety classifier

A.2 Robustness to Base-LM Interfaces

The guard–base discrepancy persists across many alternative base-LM interfaces, while attack-wise calibration varies by attack and model but often shows high confidence despite lower accuracy.

A.2 Robustness to Base-LM Interfaces

  • Across 23 alternative interfaces per matched pair, tests varied instructions, labels, demonstration order, shot count, zero-shot prompting, and policy-aligned prompts.Source: §A.2 · p. 12
  • The base LM had higher entropy on guard false negatives in 79/97 valid settings and lower adversarial ECE in 90/97 settings.Valid interfaces met clean balanced accuracy ≥60% and benign FPR ≤40%; adversarial outcomes were not used for filtering.Source: §A.2 · p. 12

B.1 Attack-wise calibration.

  • Table 4 compares attack-wise output-verdict shifts and intermediate-layer guard–base gaps for Llama-Guard-3 and Llama-3.1 8B.Positive values indicate larger shifts for the guard than for the base model.Source: Table 4 · p. 13
  • Attack-wise miscalibration varied, but adversarial confidence often remained high as empirical accuracy fell, rather than the aggregate pattern arising from one attack.AutoDAN-Turbo and the custom attack often produced severe overconfidence, while WildGuard under GCG or AutoDAN was comparatively well calibrated.Source: §B.1 · p. 13

B.2 Attack-wise Verdict Shifts and Layer Gaps

Attack-wise output shifts and layer gaps vary together across the examined attacks, with larger shifts and positive middle-layer gaps for four attacks; the pattern is correlational.

B.2 Attack-wise Verdict Shifts and Layer Gaps

  • TAP, PAIR, AutoDAN-Turbo, and the custom attack show larger output shifts and positive guard–base gaps from the middle layers onward.The comparison is for Llama-Guard-3 and Llama-3.1 8B.Source: §B.2 · p. 13
  • GCG and AutoDAN show only small output shifts and no sustained positive layer-wise gap.Source: §B.2 · p. 13
  • The attack-wise shift and gap pattern is consistent with calibration differences, but the relationship is correlational rather than causal.Source: §B.2 · p. 13

C Model-Scale Analysis

Across model scales, the ShieldGemma guard–base discrepancy persists while Llama-Guard-3-1B reverses the ordering; response moderation also shows higher adversarial than clean ECE, with important scope limits.

C Model-Scale Analysis

  • Model-scale detection thresholds were selected on clean-benign data to target a 5% benign FPR, without using adversarial examples for threshold selection.Source: C Model-Scale Analysis · p. 13
  • Across ShieldGemma models from 2B to 27B parameters, the guard is less calibrated and detects fewer adversarial inputs than its backbone, while Llama-Guard-3-1B reverses that ordering.Positive guard–base gaps also appear before the output layer for ShieldGemma; model size alone does not explain the discrepancy, and backbone competence or model family may contribute.Source: C Model-Scale Analysis · p. 13, Table 5 · p. 13
  • Figure 8’s attack-wise reliability diagrams show high-confidence bins below the perfect-calibration diagonal across many model–attack pairs.The figure covers five guard models across six jailbreak attacks; blue bars encode empirical accuracy by confidence bin and red regions indicate overconfidence.Source: Figure 8 · p. 14

D Response Moderation

  • For all five guards on WildGuardMix responses, adversarial ECE (.346–.788) exceeds clean ECE (.081–.182).For Llama-Guard-2, Llama-Guard-3, and ShieldGemma, adversarial guard ECE also exceeds that of the corresponding base model.Source: Table 6 · p. 13, D Response Moderation · p. 14
  • Response moderation is a harmful-response sanity check, not a full calibration evaluation, because the adversarial subset contains only unsafe responses.Matched conclusions are avoided for Llama-2 and Mistral because their clean accuracy is low and adversarial accuracy near-perfect, indicating strong unsafe-prediction bias.Source: D Response Moderation · p. 14

Related papers

Loading 2609.36477v1…