PaperlayerView exact revision on arXiv

Source-linked AI summary

Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge

Wonjun Lee, Kyungsik Yang, Gaeun Ji, Vaidehi Patil, Haon Park, Bumsub Ham, Mohit Bansal, Suhyun Kim

arXiv:2610.07532v1cs.CRcs.AIcs.CL

TL;DR

Decoding-stage defenses can trade helpfulness on benign queries for safety and often rely on architecture-specific hidden states. LADE uses first-token output probabilities to identify latent safety signals and classify harmful queries. Across diverse models and attacks, it reports robust defense with a competitive safety-utility trade-off.

  • Problem

    Decoding-stage defenses can degrade helpfulness on benign queries as safety increases and may rely on architecture-specific hidden states, limiting transfer across LLMs.

    Source: §1 · p. 2

  • Method

    LADE extracts latent safety signals from first-token dark knowledge, maps tokens across tokenizers, and uses kNN over mapped tokens to classify queries.

    Source: §1 · p. 2

  • Results

    LADE achieves robust defense and a competitive safety-utility trade-off across diverse LLMs, benchmarks, and jailbreak attacks.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    LADE operates on output probabilities and remains applicable across the evaluated architectures without modification.

    Source: §5 · p. 10

    LADE relies on safety alignment in target LLMs; weakly aligned models show lower discrimination accuracy, so it is not a substitute for alignment.

    Source: M Limitations · p. 24

Abstract

from arXiv · show

LLMs have advanced rapidly, raising growing concerns about their safety. Recent work has proposed approaches to detect and defend against attacks including defenses at decoding stage that leverage models' hidden states. However, existing decoding-stage defenses suffer from two limitations. First, they introduce a trade-off between safety and over-refusal, where strengthening safety degrades the model's helpfulness on benign queries. Second, many of these methods rely on internal hidden states and are thus restricted to specific architectures, incurring substantial overhead and limited generalization across models. To address these limitations, we introduce LADE (Latent Safety Signals for Defense), which leverages latent safety signals extracted by contrasting harmful and benign queries from dark knowledge (i.e., information carried by the output probability distribution beyond its argmax) in the first-token output probability distribution. Our key insight is that, beyond surface-level refusal tokens, the dark knowledge in the first-token distribution contains latent safety signals, defined as tokens whose probabilities differ sharply between harmful and benign queries. We show that these signals consistently align across LLMs, forming a model-agnostic direction that emerges from safety alignment. LADE consists of three components: (1) Extracting Latent Safety Signals from Dark Knowledge, which selects top-k safety-discriminative tokens from the first-token probability distribution; (2) Tokenizer Mapping, which maps these tokens across different tokenizers to enable model-agnostic application; and (3) kNN-based Discrimination, which classifies queries via a k-Nearest Neighbors search over the mapped tokens. Across diverse LLMs and benchmarks, LADE is robust against a wide range of jailbreak attacks and lowers attack success rates while maintaining a competitive safety-utility trade-off.

1 Introduction

Existing decoding-stage defenses can trade helpfulness on benign queries for safety and may depend on architecture-specific hidden states. LADE instead uses first-token output probabilities to identify transferable latent safety signals and reports robust defense with a competitive safety-utility trade-off.

1 Introduction

  • Prior decoding-stage defenses leverage internal hidden states, but can degrade helpfulness on benign queries and limit transfer across architectures [44].The cited limitations include architecture-specific auxiliary modules and substantial overhead [21].Source: §1 · pp. 1–2
  • Latent safety signals in first-token dark knowledge provide a safety-discrimination signal beyond surface refusal tokens.Dark knowledge is fine-grained information in the full output distribution beyond its highest-probability token.Source: §1 · p. 2
  • Signals extracted from a reference LLM transfer across target LLMs despite differences in tokenizers, architectures, and surface refusal patterns.Refusal tokens alone capture only a small subset of safety signals and fail to separate harmful from benign queries, while adding latent signals yields clear separation.Source: §1 · p. 2, Figure 1 · p. 2
  • LADE operates on output probabilities through signal extraction, tokenizer mapping, and kNN-based classification of queries before generation.The kNN stage compares each query with harmful reference queries; the framework does not access internal hidden states.Source: §1 · p. 2
  • LADE substantially reduces attack success rates and maintains a competitive safety-utility trade-off across diverse LLMs, benchmarks, and jailbreak attacks.The reported comparison includes existing defense baselines and dedicated guard models.Source: §1 · p. 3

2 Related work

Jailbreak research has moved from manually crafted prompts to automated and semantic-level attacks. Defenses span input, training, and decoding stages, with decoding approaches limited by over-refusal and model-specific representations.

2.1 Jailbreak attacks on LLMs

  • Jailbreak methods evolved from role-playing and in-context demonstrations to gradient or genetic prompt optimization and semantic-level attacks.Examples include cipher prompts, cognitive overload, multi-turn strategies, and AutoDAN-Turbo's autonomous discovery of attack strategies.Source: §2.1 · p. 3

2.2 Defense mechanisms against jailbreak attacks

  • Input-level, training-level, and decoding-level defenses address jailbreaks, but decoding methods share over-refusal trade-offs and limited transfer from model-specific representations.Input filters can fail against semantic rephrasing, while training defenses incur heavy computation and catastrophic forgetting.Source: §2.2 · p. 3

3 Latent Safety Signals for Defense

LADE selects safety-discriminative first-token tokens by contrasting harmful and benign queries, maps them across tokenizers, and classifies queries by distance to harmful references.

3 Latent Safety Signals for Defense

  • LADE’s three stages select tokens from harmful-benign contrasts, map them across tokenizers, and classify queries by distance to harmful references.The overview describes the stages as signal extraction, tokenizer mapping, and kNN-based discrimination.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Extracting latent safety signals from Dark Knowledge

  • Latent safety signals are tokens whose first-token probabilities differ most between harmful and benign queries, beyond information in the argmax.The method retains the top-k tokens by distributional gap after excluding special and punctuation-only tokens, and stores benign means for later ratio calibration.Source: §3.1 · p. 4

3.2 Tokenizer mapping

  • Tokenizer mapping handles splits by selecting the first meaningful subword and handles duplicate mappings by scaling non-anchor contributions relative to the anchor.The anchor is the duplicate-group slot with the largest reference benign-mean probability; ratio weighting prevents duplicated source tokens from being over-weighted.Source: §3.2 · pp. 4–5

3.3 kNN-based discrimination

  • kNN discrimination forms weighted, normalized query features and filters queries whose mean distance to K nearest harmful references is at or below a bootstrapped threshold.LADE makes this filtering decision before generation begins, unlike decoding-stage defenses that adjust probabilities during generation.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate LADE across diverse LLMs, jailbreak attacks, and harmful/benign query benchmarks. Results examine its safety-utility balance, cross-model transfer, tokenizer mapping, deployment efficiency, and the signals’ relationship to safety alignment.

4.2 Main results

  • LADE achieves the lowest or near-lowest average compliance across five jailbreak attacks and six LLMs, including models where existing defenses break down.The evaluation includes AutoDAN, DeepInception, GCG, PAIR, and LIAR.Source: §4.2 · p. 6, §4.1 · p. 6, Table 1 · p. 7
  • LADE achieves the best average on six of eight LLMs and remains competitive on the others, while reducing over-refusal relative to defenses that sharply raise benign MMLU refusals.On Llama-2-7B-Chat, Self-Reminder and SafeDecoding raise MMLU refusals from 1 to 372 and 393, respectively.Source: §4.2 · p. 6, Table 2 · p. 8
  • With both tokenizer-mapping components, transfer accuracy reaches 0.9660, up from 0.8934 without them; Split contributes more than Ratio.Ratio alone raises accuracy to 0.9413, while enabling Split raises it to 0.9649; combining both yields the most uniform performance.Source: Figure 3 · p. 7, §4.2 · p. 8
  • Cross-model transfer accuracy remains high regardless of the reference LLM, and is comparable to using the same model as reference and target.The evaluation tests every reference-target combination in a 6 × 6 setting.Source: §4.2 · pp. 6–7
  • LADE achieves 0.9689 average accuracy against guard models, ahead of WildGuard-7B at 0.9474, and is more robust to jailbreak attacks.The comparison uses Llama-3-8B-Instruct as both LADE’s reference and target model.Source: Table 3 · p. 8, §4.2 · p. 9
  • On an RTX A6000 with Llama-3-8B-Instruct, LADE takes 34.13 ms per input, uses the least memory, and is about 2.5× and 16.2× faster than two guard models.LADE reuses the target model’s first-token forward pass and requires no additional model.Source: Table 4 · p. 9, §4.2 · p. 9

4.3 Analysis

  • Gemma accuracy rises from 0.6836 on Gemma-7B to 0.9396 on Gemma-7B-it; adding latent signals also improves classification over refusal tokens alone across six LLMs.The Gemma models share a pre-training backbone and differ at the alignment stage; refusal-token performance varies across models.Source: Table 5 · p. 9, §4.3 · pp. 9–10, Figure 4 · p. 10
  • Classification accuracy declines across the first three candidate positions—0.9657, 0.9222, and 0.8449—and combining later positions does not improve on the first alone.The results support using the first decoding position as LADE’s sole input.Source: Table 6 · p. 10, §4.3 · p. 10

5 Conclusion

LADE filters harmful queries using latent safety signals from first-token output probabilities, without relying on hidden states. Its offline preparation and online kNN classification support the paper’s conclusion that it offers robust defense across evaluated architectures.

5 Conclusion

  • LADE achieves robust defense with a competitive safety-utility trade-off across the evaluated architectures, without modification.It operates on first-token output probabilities and filters harmful queries before generation.Source: §5 · p. 10

A Algorithm

  • The offline stage prepares safety-signal tokens, tokenizer mapping, and a calibrated threshold; online, kNN distance to harmful references classifies each query.The procedure uses the target model’s first-token probabilities for online classification.Source: A Algorithm · p. 14
  • LADE’s additional cost comes mainly from one forward pass per query for offline signal extraction and evaluation.The experiments used one NVIDIA RTX A6000 GPU with 48GB memory.Source: A Algorithm · p. 14

B Additional ablations

The ablations examine the effects of signal-set size, extraction and reference datasets, threshold calibration, duplicate-token mapping, and refused-query share. Results identify useful default settings while showing performance remains stable across several choices.

Top-k ablation

  • Accuracy peaks at 0.9657 with k = 500, versus 0.4505 using only the top-1 token; k = 1000 slightly reduces accuracy.The authors attribute the decline at k = 1000 to noise from less informative tokens.Source: B Additional ablations · p. 15, Table 7 · p. 15

Contrast datasets

  • Hex-Phi and XSTest yield the highest extraction-set accuracy, 0.9660; the 12 configurations range from 0.9259 to 0.9660.XSTest consistently leads to higher accuracy across harmful sets, and other dataset combinations also perform strongly.Source: B Additional ablations · p. 15, Table 8 · p. 15

Harmful reference set

  • Hex-Phi as the harmful reference set reaches 0.9657 accuracy, compared with 0.9488 for StrongReject and 0.9299 for AdvBench.The authors adopt Hex-Phi as the default reference set.Source: Table 9 · p. 15, B Additional ablations · p. 16

Threshold quantile

  • Threshold accuracy is highest at ρ = 0.90 (0.9657), falling to 0.9330 at ρ = 0.85 and 0.7773 at ρ = 0.99.The authors explain that thresholds that are too strict exclude boundary cases, while overly loose thresholds include outliers.Source: B Additional ablations · p. 16, Table 10 · p. 16

Duplicate-mapping statistics

  • Benign-side calibration achieves the highest average accuracy; duplicate mappings affect 19.5% of source tokens, and ratio estimation raises accuracy from 0.9649 to 0.9660.The median weight of non-anchor slots in duplicate groups is 0.015.Source: B Additional ablations · pp. 16–17, Table 11 · p. 16

Refusal rate of reference LLM

  • Even using only 19 non-refused harmful queries retains 87.8% of the selected tokens and yields nearly identical classification performance.The findings indicate that LADE does not require explicit refusals to extract latent safety signals.Source: B Additional ablations · p. 17, Table 12 · p. 17

C Robustness against prefilling-based suffix attacks

LADE filters queries using first-token probabilities before generation, a design that supports resistance to prefilling-based suffix attacks. Under the evaluated attack, classification accuracy exceeds 0.99 on average across six LLMs.

Prefilling-based suffix attacks

D Robustness against adaptive attacks

The adaptive-attack evaluation finds a substantial accuracy drop, while adding observed evasion examples improves detection but does not guarantee robustness to re-optimized or unseen attacks. On realistic benign queries, misclassification stays below 10% in most evaluated model-dataset pairs.

D Robustness against adaptive attacks

  • Under the adaptive attack, LADE accuracy drops from 1.000 to 0.480, and its four-model average drops to 0.420.LADE classifies 24 of 50 prompts as harmful, versus 11 for WildGuard-7B and at most 1 for other guard models; evaluation covers one controlled white-box attack.Source: D Robustness against adaptive attacks · p. 18, Table 14 · p. 18
  • Adding an evasion reference set raises AutoDAN detection from 42.00% to 93.50%, but benign misclassification rises from 2.22% to 5.42%.Under re-optimization, augmented LADE accuracy drops from 0.960 to 0.480, so the mitigation does not guarantee robustness to unseen adaptive attacks.Source: D Robustness against adaptive attacks · pp. 18–19, Table 15 · p. 18

E Realistic benign queries

  • On realistic benign queries, misclassification stays below 10% in 18 of 24 model-dataset pairs; Qwen3-8B reaches 12.13% on security queries.Higher percentages appear mainly for Mistral-7B-Instruct-v0.3 and Qwen2-7B-Instruct, especially on JailbreakBench and medical queries.Source: E Realistic benign queries · p. 19, Table 16 · p. 19

F Averages over all datasets

Across all datasets, LADE has the lowest Full Avg. on six of eight LLMs, with a stated exception on Llama-2-13B-Chat. It also retains performance across different alignment methods, judges, and restricted access to first-token probabilities.

F Averages over all datasets

  • On six of eight LLMs, LADE achieves the lowest Full Avg.; on Llama-2-13B-Chat, its Full Avg. exceeds No Defense because of 89 XSTest refusals.Full Avg. covers all seven datasets; Held-out Avg. excludes Hex-Phi and XSTest, so the averages should be read together.Source: F Averages over all datasets · p. 20, Table 17 · p. 20

G Additional safety-aligned LLMs

  • Held-out Avg. is 4.60 on Beaver-v3 and 1.80 on ThinkSafe-Qwen3-8B, supporting transfer across two different safety alignment methods.Beaver-v3 uses Safe RLHF, while ThinkSafe-Qwen3-8B uses self-generated safety alignment for reasoning models.Source: G Additional safety-aligned LLMs · p. 21, Table 18 · p. 21

H Reliability of compliance and refusal judgments

  • Additional judges agree with Qwen3Guard-4B on 84.80%–91.22% and Dic-Judge on 94.31%–95.03%; LADE compliance on held-out harmful queries remains 1.96%–2.82%.The largest difference from the Qwen3Guard-4B result is 0.65 percentage points, and replacing either judge does not change the conclusions.Source: H Reliability of compliance and… · p. 21, Table 19 · p. 21

I Restricted access to output probabilities

  • With m = 20, LADE matches full-distribution filtering decisions on 6,998 of 6,999 queries; at m = 5, balanced accuracy drops only 0.25 percentage points.Queries retain an average of 3.18 non-refusal latent signals at m = 5 and 11.14 at m = 20.Source: Table 20 · p. 21, I Restricted access to output… · p. 22

J Refusal tokens for the refusal-token-only ablation

A refusal-token-only configuration can misclassify benign queries, particularly on Mistral-7B-Instruct-v0.3 and Qwen2-7B-Instruct. Adding latent safety signals improves accuracy, while the ablation’s refusal tokens are derived from each model’s tokenizer.

J Refusal tokens for the refusal-token-only ablation

  • For each model, the refusal-token set is formed by tokenizing the predefined refusal strings listed in Table 21 with that model’s own tokenizer.The resulting token IDs are used as the refusal-token set for that model.Source: J Refusal tokens for the… · p. 22, Table 21
  • All three LLMs classify harmful queries at similar rates, but Mistral-7B-Instruct-v0.3 and Qwen2-7B-Instruct misclassify many more benign queries as harmful.Qwen2-7B-Instruct and Qwen3-8B use the same 22 refusal token IDs, so their performance gap is not due to tokenization.Source: J Refusal tokens for the… · p. 22
  • Adding latent safety signals raises accuracy to 0.9150, 0.9745, and 0.9645 on Mistral-7B-Instruct-v0.3, Qwen2-7B-Instruct, and Qwen3-8B, respectively.The refusal-token-only configuration performs poorly on Mistral-7B-Instruct-v0.3 and Qwen2-7B-Instruct.Source: J Refusal tokens for the… · p. 22

K Failure case

LADE’s failure cases include attacks that shift first-token probabilities and benign queries that overlap harmful references, as well as borderline harms with weak safety-discriminative patterns. The section also specifies the refusal strings used for benign-query refusal detection.

K Failure case

  • 41 of 50 AutoDAN prompts still receive compliant responses from Gemma-7B-it under LADE, versus none on Qwen2-7B-Instruct.On Gemma, AutoDAN shifts probability away from the harmful reference set, concentrating it on a target token shared through duplicate mapping.Source: K Failure case · p. 23
  • Removing affected mapped tokens helps only slightly because L1 normalization redistributes the remaining probability; later decoding positions or a larger harmful reference set may be needed.The paper leaves these possible approaches for future work.Source: K Failure case · p. 23
  • LADE filters 82 of 500 benign Alpaca queries on Mistral-7B-Instruct-v0.3; these queries lie close to that model’s harmful reference set.The pattern is not generally ambiguous: only 1 is filtered on Llama-3-8B-Instruct, 1 on Llama-2-7B-Chat, 14 on Qwen2-7B-Instruct, 9 on Qwen3-8B, and none on Gemma-7B-it.Source: K Failure case · p. 23
  • Failure causes differ: AutoDAN shifts Gemma-7B-it’s first-token distribution, while some benign queries overlap harmful queries in Mistral-7B-Instruct-v0.3’s feature space.Borderline failures involve indirect or context-dependent harms rather than overtly toxic or illegal content; the paper says addressing these requires alignment that accounts for indirect harms.Source: K Failure case · p. 23

L Refusal keywords and phrases used for refusal detection

M Limitations

LADE has four stated limitations: it depends on target-model safety alignment, can still over-refuse, needs first-token probabilities, and uses a fixed signal-token set that an informed attacker could evade.

M Limitations

  • LADE relies on safety alignment because latent safety signals emerge from that process; weakly aligned LLMs yield lower discrimination accuracy.The authors state that LADE is not a substitute for safety alignment, but operates alongside it.Source: M Limitations · p. 24
  • LADE is not entirely free from over-refusal: on Mistral-7B-Instruct-v0.3, it induces more Alpaca refusals than the No-Defense baseline.The authors attribute this to partial overlap between the harmful reference manifold and conversational benign queries on some LLMs.Source: M Limitations · p. 24
  • LADE requires access to target-model first-token output probabilities, which may be limited for closed APIs.The paper reports that the 20 most likely first tokens are enough to reproduce nearly all filtering decisions.Source: M Limitations · p. 24
  • The fixed latent safety signal token set could be targeted by an attacker who knows it and optimizes inputs to avoid filtering.The paper evaluates an adaptive attack and a mitigation in Appendix D.Source: M Limitations · p. 24
Loading 2610.07532v1…