PaperlayerView exact revision on arXiv

Source-linked AI summary

Faithful Latent Harm Monitors Work No Better Than the Models Own Behaviour at a Matched False Alarm Rate

Abhishek Mishra, Sagnik Chatterjee, Suyash Maniyar

arXiv:2610.30727v1cs.AI

TL;DR

It is unclear whether internal harm representations remain useful under distribution shift and provide a signal independent of the model’s refusal behaviour. The study tests whether harm reads reflect request judgments or refusal plans, and evaluates detection on reworded prompts against model behavior and simple baselines. Instruction-token reads do not outperform text baselines, while template-token gains over model refusal track coupling to the refusal decision.

  • Problem

    It is unclear whether internal harm representations remain useful under distribution shift and provide a signal independent of the model’s refusal behaviour.

    Source: §1 · pp. 1–2

  • Method

    The study tests whether harm reads reflect request judgments or refusal plans, and evaluates detection on reworded prompts against model behavior and simple baselines.

    Source: §1 · p. 2

  • Results

    Instruction-token reads do not outperform text baselines, while template-token gains over model refusal track coupling to the refusal decision.

    Source: §1 · p. 2, §8 · p. 16

  • Takeaways & Limitations

    For the readers tested, neither position provides an independent monitoring check that beats model behaviour at a matched false-alarm rate.

    Source: §7 · p. 15

    Behaviour labels rely on string matching of greedy replies, which misses refusals phrased without listed phrases.

    Source: §7 · p. 16

Abstract

from arXiv · show

Large language models have been reported to encode the harmfulness of a request separately from their decision to refuse it, and latent safety monitors such as Latent Guard build on this report by classifying prompts as harmful from the internal activations of the model, with the aim of catching harmful intent even when the model complies. Whether these representations give a signal of harm that survives distribution shift and is independent of model behaviour is unclear. We test this on out-of-distribution prompts made by rewriting existing harmful and benign requests with surface rewording, persuasive framing and jailbreak-style wrappers. Across 15 instruction-tuned LLMs and 38 activation-based readers, we compare harm detection with that of TF-IDF and sentence-embedding classifiers, with independently trained safety classifiers, and with the refusal of each model. Activations are read at the last token of the user instruction, where the model has read the whole request, and at the last chat-template token before the reply, where the planned response can be represented. Because models disagree about which prompts to refuse, we can separate a signal of the harmfulness of the prompt from a signal of the refusal decision of the model. At the end of the instruction, the representation is largely independent of the refusal behaviour of each model but degrades under distribution shift, so that a TF-IDF classifier ranks harm better than the original Latent Guard in every model we test, and linear and MLP probes do about as well as sentence-embedding classifiers trained on the same examples. The representation just before generation detects more, catching 44 percent of harmful prompts at a matched false-alarm rate against 32 percent for the refusal of the model, but that gain follows the refusal decision itself, and part of it comes from refusals that string-matching evaluation misses.

1 Introduction

The paper tests whether latent harm reads capture judgments of requests independently of models’ refusal plans, and whether those reads remain useful on reworded prompts. Instruction-token reads perform no better than text classifiers, while template-token gains follow the model’s refusal decision.

1 Introduction

  • The two research questions ask what latent reads track—request judgment or refusal plan—and how faithfully they detect harm on reworded prompts against baselines and model behaviour.The robustness evaluation includes rewording, persuasion, and jailbreak-style prompts.Source: §1 · pp. 1–2
  • Instruction-token reads largely reflect request judgments independent of each model’s refusal, whereas template-token reads follow the model’s own decision.The distinction is reported across models, with refusability held fixed for the instruction-token comparison.Source: §1 · p. 2
  • At the instruction token, Latent Guard falls below TF-IDF, while linear and MLP probes match sentence-embedding classifiers; template-token gains over refusal track coupling to the model’s decision.The template-token read also beats refusal as labelled by string matching, but some of that gain comes from replies string matching misses as refusals.Source: §1 · p. 2
  • The study evaluates 38 readers on 15 models against text classifiers, an independent safety classifier, and each model’s behaviour, and releases reimplementations, rewording data, labels, and audits.Its framework also uses cross-model disagreement to test which signal a read carries.Source: §1 · p. 2

2 Two signals in a latent read

The section treats a latent harm score as a mixture of a request-level judgment and a model-specific refusal plan, then sets out tests for which signal it carries and where it can be read. It also distinguishes faithfulness from detection quality by comparing monitors with simple text classifiers and model refusal at a matched false-alarm rate.

2.1 Judgment and plan

  • A monitor score combines a judgment about request features with the model’s propensity to refuse, but those components cannot be recovered separately from the score.A score dominated by the refusal plan may move with framing or over-refusal, and cannot catch a request the model is fully willing to answer; faithfulness means the score is dominated by the judgment component.Source: §2.1 · pp. 2–3

2.2 Testing which signal a read carries

  • Harm-detection metrics alone cannot identify whether a score tracks harm or refusal, so the section tests refusal association conditional on harm and cross-model consensus.The tests include controlled regression and within-harm-class comparisons at matched consensus, plus a cross-model disagreement statistic that is zero for a model-blind shared score and positive when scores follow each model’s own behavior.Source: §2.2 · p. 3
  • The faithfulness tests are necessary but not sufficient because a plan component shared across models could pass them.Models disagree on 115–317 items for every pair, but the tests alone cannot rule out a shared plan component.Source: §2.2 · p. 3
  • At a matched false-alarm budget, the monitor’s caught harmful-item share and gain over refusal are measured at the refusal’s benign-item false-alarm rate.Matching this rate avoids confounding ranking with calibration; the table contrasts predictions for wording-based judgment (W), beyond-text judgment (H), and plan (D), across decoupling, beyond-text, and detection tests.Source: §2.2 · p. 4, Table 1 · p. 4

2.3 Where a judgment can be read

  • The two read positions are the last instruction token, where the whole request has been read, and the final pre-reply template token, where the first reply token is produced from the last-layer state.The template tokens after the instruction are identical across requests, and the section classifies each read by controlled tests rather than assuming the positions carry different signals.Source: §2.3 · p. 4, Figure 1 · p. 5

3 Related work

Related work reports separate harm and refusal representations, activation-monitor generalisation problems, and text-based guard and benchmark approaches. This paper positions its evaluation against those lines of work.

Related work

  • Prior work reports separate harmfulness and refusal representations, but also cautions that lexical effects and pooling choices can shape harm directions (Li et al., 2025; Llorente-Saguer, 2026).Other cited work studies refusal directions and broader moral subspaces, while jailbreaks have been reported to suppress harmfulness representations (Arditi et al., 2024; Wollschläger et al., 2025; Reblitz-Richardson, 2026).Source: §3 · p. 5
  • Activation probes have been reported to generalise poorly under distribution shift and hard negatives, while TF-IDF or bag-of-words can match some probe results (Wang et al., 2025; Sadhu et al., 2026; Balani and Panda, 2026).The paper distinguishes its evaluation by using refusal at a matched false-alarm rate, text classifiers trained on the same rows, two read positions across 15 models, and a cross-model test; activation-optimised evasion is out of scope (Bailey et al., 2026).Source: §3 · p. 5
  • Guard models classify prompts against hazard taxonomies, while SORRY-Bench, WildJailbreak, XSTest, and OR-Bench provide linguistic, adversarial, and over-refusal evaluations (Inan et al., 2023; Han et al., 2024; Zeng et al., 2024a; Xie et al., 2025; Jiang et al., 2024; Röttger et al., 2024; Cui et al., 2025).The cited work also identifies refusal-cue shortcuts in guards (Feng et al., 2026), and the paper uses a hosted decision model as a reference.Source: §3 · p. 5

4 Experimental setup

The experiments compare activation-based readers at two prompt positions across 15 instruction-tuned models, using matched training rows and held-out tests. The setup includes behaviour-labelled and reworded prompts, with baselines and bootstrap intervals used to assess readers and detection.

4 Experimental setup

  • The study covers 15 open instruction-tuned models from four developers, run with their own chat templates and no system prompt.Qwen3-8B and Qwen3-32B use thinking disabled; three models are mixture-of-experts.Source: §4 · p. 6
  • The 38 readers include Latent Guard variants, logistic-regression and MLP probes, and four readers designed to track refusal directions at both activation positions.The linear probe uses cross-validation to select a layer and regularisation; the MLP has one hidden ReLU layer.Source: §4 · p. 6
  • Prompt-string baselines are TF-IDF and two all-mpnet-base-v2 sentence-embedding classifiers trained on the same rows as the activation readers.Llama Guard 3 8B serves as a separate zero-shot safety-classifier reference.Source: §4 · p. 6
  • The original Latent Guard recipe uses 100 refused harmful and 100 answered safe examples per model; curated sets add balanced, behaviour-independent training data at 100, 500, and 2,000 examples per class.The original enrichment step changes pooled rewording-set AUROC by at most 0.006.Source: §4 · p. 6
  • Tests are separated from training by leak group; Drew contains 2,248 reworded requests whose surface cues point to the other class while labels remain unchanged.Dbeh has 1,048 prompts from sources absent from training, and Did contains 600 held-out items from curated sources.Source: §4 · p. 6, Table 2 · p. 7
  • Unless stated otherwise, results are medians over 15 models with 95% bootstrap intervals; two models are excluded from cross-model-test medians because their refusal labels are mostly wrong on disagreement items.The exclusion leaves 78 of 105 model pairs.Source: §4 · pp. 6–7

5 RQ1: What do latent reads track?

The instruction-token read is largely decoupled from each model’s refusal decision once consensus is controlled, whereas the template-token read tracks that decision. Layerwise results place the detection gain where refusal coupling emerges.

5 RQ1: What do latent reads track?

  • At tinst, fixed-consensus within-class AUROC is 0.59 on harmful and 0.62 on harmless prompts; one score standard deviation changes refusal odds by a factor of 1.15.The symmetric statistic is +0.04, a sixth of its template-token value.Source: §5 · pp. 7–8, Table 4 · p. 8
  • At tpost, within-class AUROC is 0.90 on harmful and 0.89 on harmless prompts; fixed-consensus AUROC is 0.76 and 0.80 in 14 models, with Π = +0.23.The read follows the model’s own decision, and its within-class AUROC is about as high as the model’s first-token refusal probability.Source: §5 · p. 8, Table 4 · p. 8
  • Template-token coupling rises from 40–50% relative depth and approaches first-token refusal coupling in the last layers; detection gain starts where that coupling begins.No layer at either position is both decoupled from the decision and better than the refusal.Source: §5 · p. 8, Figure 3 · p. 9
  • Across 19 instruction-token readers on reworded prompts, Π is 0.05–0.15; template-token refusal directions reach 0.44–0.48, and every reader beating refusal has Π ≥0.25.The instruction-token residue is real but small; at fixed consensus, six models still separate their own over-refusals on harmless prompts.Source: §5 · p. 8

6 RQ2: How robust is the faithful read?

On reworded prompts, instruction-token reads are mostly independent of refusal behaviour but weaken under distribution shift, while template-token gains track the model’s refusal decision. The apparent template-token advantage is partly over string-matching labels rather than over the model’s actual replies.

6.1 The instruction token

  • Latent Guard’s instruction-token AUROC is below TF-IDF’s 0.75 in every model, while linear and MLP probes do no better than same-row sentence-embedding classifiers.With 2,000 examples per class, the linear probe exceeds embedding LR by +0.015 AUROC in 3 models and trails it in 1; the MLP probe ties embedding MLP (−0.001 AUROC, 3↑3↓).Source: §6.1 · p. 10, Table 5 · p. 11
  • With 2,000 examples per class, the instruction-token probe removes Latent Guard’s persuasion-framing drop and exceeds embedding LR by +0.11 AUROC on framed prompts.The interval is above zero in all 15 models, but this advantage requires 2,000 labels per class; the framing drop persists for Latent Guard with more data and z-scoring.Source: §6.1 · p. 10

6.2 The template token

  • 0.439 of harmful prompts are caught by the template-token Latent Guard at the refusal’s false-alarm rate, versus 0.32 by refusal; its gain tracks the refusal decision.The gain is partly from harmful prompts labelled “answered” by string matching, many of which are hedged, sanitised or refusals in other words.Source: §6.2 · pp. 10–11, Figure 4 · p. 12

6.3 Coupling to the decision predicts the gain

  • Readers with Π ≥0.25 beat refusal by +0.04 to +0.10, while those with Π ≤0.18 gain at most +0.03; across 30 non-refusal-direction readers, coupling and gain correlate at 0.83.The split supports the claim, but the authors give no p-value because readers share statistics, training sets and positions.Source: §6.3 · p. 11, Figure 6 · p. 14

6.4 The operating point and its one exception

  • On the alarming benign set, fitted readers beat refusal by +0.47 to +0.57 at 2,000 examples per class, while embedding MLP and Llama Guard 3 do not.The gain is bounded by over-refusal-derived harmless training examples and comes with more false alarms on jailbreak-style benign prompts; readers fall below refusal when thresholds are set on those prompts.Source: §6.4 · p. 12
  • The comparison depends on the harmful-prompt type: refusal catches 0.047 under persuasion framing but 0.648 under surface rewording, and the paper reports pooled-set conclusions.Most instruction-token readers catch more than refusal under framing (up to 0.26) but less under surface rewording (0.30–0.62).Source: §6.4 · p. 12

6.5 Qualitative examples

  • The template-token read flags some fully compliant replies, but sampled additions were mostly hedged, sanitised or refusals phrased without listed refusal terms.In an audit of 12 sampled reply openings, three or four complied outright; the openings do not establish the content of each full reply.Source: §6.5 · p. 13
  • On the same harmless “murder” wording, Latent Guard flags a lexical false alarm at the instruction token for Llama-3-8B, but follows Gemma-3-12B’s over-refusal at the template token.Qwen3-8B likewise has only its template-token read flag “assassinate procrastination” when it declines the harmful-sounding wording before offering a benign plan.Source: §6.5 · p. 14

7 Discussion

The discussion finds that instruction-token reads mostly track the request, while template-token reads track the model’s refusal decision. The stronger template-token detection does not provide an independent check, and the study’s labels and evaluation setting limit the conclusions.

Findings

  • At the instruction token, a read is mostly a judgment of request wording; at the template token, it is the model’s reply plan and follows its refusal decision.The template-token read detects more, but its gain tracks coupling to the decision and partly reflects replies that string matching mislabels as answers.Source: §7 · p. 15
  • The template-token read’s higher detection reflects the model’s decision, including near-refusals that binary string matching misses.Across all 38 readers, gains over refusal track coupling to the decision.Source: §7 · p. 15

Monitoring implications

  • For an independent check, neither position beats model behavior at a matched false-alarm rate on mixed or jailbreak-style benign traffic.An independently trained classifier beats the instruction-token read as published; a hosted decision model exceeds both latent reads and behavior on the same items, at the cost of an API call per prompt.Source: §7 · p. 15

Limitations

  • The study tests open models up to 32B parameters and two reader positions; behavior labels use string matching on greedy replies, which misses refusals without listed phrases.Other limits include source-level harm labels, one split seed, and a matched-rate threshold set by about 24 flagged benign prompts per model.Source: §7 · p. 16

8 Conclusion

Across 15 open models and 38 readers, the conclusion is that decoupled latent reads do not catch more harmful reworded prompts than model refusal at a matched false-alarm rate. Readers that catch more are coupled to the model’s decision, so they cannot independently check it.

Conclusion

  • A latent harm read can mix a judgment of the request with the model’s plan for its reply, which detection scores alone cannot distinguish.A cross-model disagreement test can separate them.Source: §8 · p. 16
  • Across 15 open models and 38 readers, decoupled reads match a same-row sentence-embedding classifier and catch no more harmful reworded prompts than model refusal at a matched false-alarm rate..Source: §8 · p. 16
  • Every reader that catches more is coupled to the model’s decision, making the better-performing latent monitors an early copy of behavior rather than an independent check.The one decoupled-read advantage on alarming but harmless requests comes with false alarms on jailbreak-style requests.Source: §8 · p. 16

A Reproducing Latent Guard

The reproduction varies Latent Guard’s training recipe and audits the behavior labels used in the study. The reported recipe differences are small, while removing the chat template mostly removes the assistant turn rather than eliciting compliance.

Reproducing Latent Guard

  • The reproduction changes the original setup by using each tokenizer’s chat template, a 256-token reply budget, and the 2025-03 SORRY-Bench release.The original statistic, layer set, and threshold are retained; S0 contains 100 harmful prompts the model refused and 100 safe prompts it answered, by refusal-phrase matching on greedy replies.Source: A Reproducing Latent Guard · p. 20
  • At equal training-set size, the original recipe is within 0.02 of S100, and behavior filtering adds a consistent +0.014 for LG-z.The original draw differs from the appended variant by +0.001 (LG) and +0.004 (LG-z) pooled AUROC, within draw spread.Source: A Reproducing Latent Guard · p. 20
  • Table 8 compares four 100-per-class training recipes at tinst on reworded prompts, reporting pooled AUROC, framing-related AUROC drop, and WildJailbreak pass and flag shares..Source: Table 8 · p. 20
  • Removing the post-instruction template mostly removes the assistant turn rather than eliciting compliance; for Llama-3-8B, actual compliance’s share in the “accepted harmful” cell falls from 0.88 to 0.10.The authors therefore use the full template to label behavior.Source: A Reproducing Latent Guard · p. 20

B Data

The data and reader details describe leakage controls, curated sets, and evaluation conditions, then specify how probes and comparison readers are selected and scored. Test refusal rates vary substantially across prompt types, and the reader results include both probe comparisons and the behavior of Llama Guard 3 under framing.

B Data

  • Leak groups are assigned wholly to one split, real-world sets are test-side, and readers on Dbeh are fit only on S0; curated sets are balanced and nested with round-robin sources.The curated sources span harmful and harmless prompt sets.Source: B Data · p. 20
  • Median refusal rates are 0.65 on surface rewording, 0.05 under persuasion framing, and 0.24 on jailbreak harmful prompts.Rates are 0.04 each on jailbreak benign and alarming benign; Llama-3-8B has a 0.155 refusal rate on jailbreak benign.Source: B Data · pp. 20–21
  • The 236 alarming-benign examples are model-rewritten Alpaca requests checked for unchanged, harmless meaning and more alarming wording, with no further manual check beyond reading 10 pairs.Rewrites with refusal, hedging, echoing, or excessive length are dropped.Source: B Data · p. 21

C Reader details

  • Probe layer and regularization are selected by five-fold cross-validated AUROC, with specified tie-breaks; the same selection rule applies to TF-IDF and embedding classifiers.At 100 examples per class, every tested C reaches about 1 cross-validated AUROC, making tie-breaking consequential.Source: C Reader details · p. 21
  • At tinst with 2,000 examples, pooled AUROC rises from LG’s 0.65 to 0.741 for a one-layer centroid read and 0.806 for the learned direction.At tpost, the centroid direction gets worse with more curated data, falling from 0.811 to 0.736.Source: C Reader details · p. 21
  • The opener set containing apology openers plus “I”, “As”, “No”, and “My” has the best AUROC against own refusal in all 15 models.The comparison uses Dbeh before any detection number.Source: C Reader details · p. 21
  • Llama Guard 3’s ranking changes little under rewording, with component AUROC drops of at most 0.03, but its verdict flags only 11% of persuasion-framed requests whose plain versions it flags.Under framing, its weakness is the threshold rather than the ranking.Source: C Reader details · p. 21

D Label audits and per-model results

Label audits show that string matching can misread replies, particularly under persuasion framing, and that some model-specific labeling errors limit cross-model interpretation without explaining positional contrasts.

D Label audits and per-model results

E Further detection results

Supplementary results show that detection depends on the operating threshold and read position, while the all-reader results report decoupling and detection across the full reader set.

E Further detection results

  • At an in-distribution-matched threshold, the 2,000-example probe flagged 0.72 of framed and 0.79 of jailbreak harmful prompts the model answered, but also flagged at least 0.30 of benign rewordings.The benign-rewording flag rates were 0.30 for original-recipe Latent Guard, 0.40 for Latent Guard at 2,000 examples, 0.35 for the probe, and 0.37 for Llama Guard 3, versus 0.045 for refusal.Source: E Further detection results · p. 23
  • At matched rates on Drew, Latent Guard’s caught-harm share was below refusal at tinst in 13 models and above it at tpost in 12.The three template-token models without a clear gain were Llama-3-8B, Qwen2.5-3B, and Qwen2.5-7B.Source: E Further detection results · p. 23, Table 13 · p. 24
  • Figures 7–9 show pooled AUROC by training size, rewording and persuasion drops, answered-versus-refused hit rates, and paired reader differences.Figure 7 also plots Latent Guard errors on matched WildJailbreak sets; every model lies near passed + flagged = 1.Source: Figure 7 · p. 25, Figure 8 · p. 25, Figure 9 · p. 26

F All readers

  • Table 14 reports all-reader decoupling and detection results on Drew, with bootstrap intervals separate from Table 13’s, so model counts can differ by one.Its metrics include the disagreement statistic, controlled refusal coefficient, caught-harm share, and difference from refusal.Source: F All readers · p. 27, Table 14 · p. 28

G A hosted decision model as reference

Jev serves as an upper reference from a separately built decision model, outperforming activation monitors on Drew while providing a stricter control for request-level judgments than wording alone.

G A hosted decision model as reference

  • Jev reached pooled AUROC 0.90 [0.888, 0.913], above activation monitors (at most 0.84) and Llama Guard 3 (0.81), and caught 0.668 of harmful prompts at refusal’s false-alarm rate (+0.30 over refusal).Its hits did not concentrate on prompts the study model already refused (href = 0.92, hans = 0.52).Source: G A hosted decision model as reference · p. 29
  • Jev reads neither study-model activations nor behaviour, and is reported only as an upper reference because it is hosted and each score costs an API call.Its weights, architecture, size, and training data are unpublished.Source: G A hosted decision model as reference · p. 29
  • Figure 10 plots refusal coefficients with controls and within-class refusal AUROCs; controls remove most tinst coefficient but at most about 40% at tpost.With both controls, Latent Guard’s tpost coefficient is 1.90 in 15 of 15 models.Source: G A hosted decision model as reference · p. 29, Table 15 · p. 29, Figure 10 · p. 30

Related papers

Loading 2610.30727v1…