PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild

Elad David, Max Fomin

arXiv:2610.02413v1cs.LG

TL;DR

Activation probes monitor unsafe requests from model hidden states, but it is unclear whether prompted probing helps because it induces classification or because of the criterion named. The paper tests controlled suffixes across models and datasets, finding that classification-format suffixes improve out-of-distribution detection, with the criterion adding precision at strict thresholds.

  • Problem

    It is unclear whether prompted-probing gains come from inducing classification or from the specific criterion named, despite the deployment need to detect unsafe requests from hidden states.

    Source: §1 · p. 1

  • Method

    The authors test a controlled ladder of post-user suffixes under strict LODO evaluation across three open-weight model families and safety benchmarks.

    Source: §1 · p. 2

  • Results

    Classification suffixes improve out-of-distribution detection by up to ∼4 AUC points, with classification format carrying ranking gains and the named criterion adding precision at strict thresholds.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    A content-free classification suffix can match the intent suffix on ranking, while the best-performing suffix depends on the probe readout.

    Source: §1 · p. 2

    The authors do not claim that probes read ground-truth harmfulness, and the classification-specific benefit is not established for Qwen.

    Source: §5 · p. 7

Abstract

from arXiv · show

LLM agents increasingly rely on activation probes as runtime monitors for prompt injection, jailbreaks, and unsafe requests, reading the model's own hidden state to catch a harmful input before the agent acts on it. A cheap, increasingly common move, borrowed from LLM-as-judge prompting, is to append a short classification instruction after the user's turn and read the probe at that point, to sharpen it: the instruction asks the model to represent the incoming request as a class, concentrating the signal the probe must separate, at negligible serving cost. But does the wording of that suffix matter, and does its benefit hold in the wild, on attack types the probe never saw in training, the regime a deployed monitor faces? We test this with a controlled ladder of post-user suffixes under strict leave-one-dataset-out (LODO) evaluation across 13 safety benchmarks (jailbreak, injection, and benign chat) and three open-weight model families (Llama-3.1-8B, Qwen3.5-9B, Gemma-4-12B). On a single-position probe, a classification suffix consistently improves out-of-distribution detection over no suffix (up to ~4 AUC points); yet which suffix matters: prompting the model to classify the input, even into content-free labels, reliably wins; an off-topic or merely-attentive suffix helps little. The gain comes from the classification format, not the named criterion: a content-free suffix matches the real malicious/benign one, with the criterion adding precision only at strict thresholds. This is not an artifact of the single-position read: the benefit carries to the multi-position pooling probes used in production (attention, multi-max, MLP), though the best-performing suffix there is readout-dependent. Served through a KV-cache fork, it is a cheap drop-in for any activation-probe monitor, though not an automatic win: which suffix helps, and by how much, depends on the model and the readout.

1 Introduction

Activation probes monitor unsafe inputs from model hidden states, but the effect of prompted-probing instructions may depend on classification format or named criterion. This paper tests those components across open-weight models and datasets using a controlled suffix ladder.

Motivation

  • The study asks whether prompted probing works through inducing classification or through the specific criterion named in the instruction.Prior work found task-description instructions improved out-of-distribution safety monitoring, but its instructions were semantically meaningful and the mechanism remained open [Tillman and Mossing, 2025].Source: §1 · p. 1

Contributions

  • The classification suffix improves out-of-distribution detection by up to ∼4 AUC points and is positive in 22 of 27 model–dataset cells.The study extends prior prompted probing to three open-weight families under strict leave-one-dataset-out evaluation.Source: §1 · p. 2
  • A content-free A/B classification suffix matches the intent suffix on ranking, while off-topic or attend-only suffixes do no better than no suffix.Naming the real criterion adds precision only at strict thresholds.Source: §1 · p. 2
  • The benefit also holds for multi-position pooling probes, with 92% of readout×suffix cells beating no suffix, but the best suffix depends on readout.The evaluated readouts are attention, multi-max, and MLP.Source: §1 · p. 2

Scope

  • The paper frames its contribution as controlled ablation of suffix ingredients across suffixes, models, and probe architectures.It extends prompted probing beyond one closed model and isolates instruction components with a length-matched suffix ladder.Source: §1 · p. 2

Suffix ladder

  • The suffix ladder holds the stem constant and includes intent, content-free A/B, reflective, and irrelevant suffixes.The examples differ in whether they classify the input, ask for careful attention, or add unrelated text.Source: §1 · pp. 2–3

2 Background and Related Work

This section situates prompted probing alongside advances in probe architecture and cross-dataset evaluation. It distinguishes transferable classification-format effects from claims that probes detect ground-truth harmfulness.

Prompted probing and probe architectures

  • Prompted probing acts on the input, while multi-position pooling probes advance the readout architecture; this work combines them across single- and multi-position readouts.The cited production-scale architectures include attention, multi-max, and MLP probes.Source: §2 · p. 3

Concepts versus surface features

  • The content-free A/B control addresses attack-vocabulary shortcuts, but the suffix remains an instructional pattern, so the paper does not claim probes read ground-truth harmfulness.Prior work found malicious-input probes can learn trigger words and instructional patterns despite high in-distribution performance [Wang et al., 2025].Source: §2 · p. 3

Format over content

  • The ranking gain is largely independent of the named criterion for Llama and Gemma, an activation-level analogue to format-over-content behavior reported in in-context learning.The effect is small on near-ceiling Qwen.Source: §2 · p. 3

Cross-dataset evaluation

  • Prior LODO work reports an 8–16 point pooled-AUC gap versus in-distribution cross-validation, but does not vary prompting; this paper studies how the readout is prompted.The authors aim for better ranking, not elimination of that gap.Source: §2 · p. 3

Deployment context

  • The paper combines prompted probing with the KV-fork suffix vehicle and its low-FPR operating-point lens, asking what a classification instruction adds under LODO.The vehicle and operating-point framing are attributed to David et al. [2026], while the prompted-probing question builds on Tillman and Mossing [2025].Source: §2 · p. 3

3 Setup

The experiments use three open-weight model families, a single-position linear probe by default, and 13 safety benchmarks evaluated with leave-one-dataset-out splits. A post-user suffix is read on a KV-cache fork and reverted before generation continues.

3 Setup

  • The default probe is a frozen single-position linear probe at the last suffix token, with advanced pooling probes and two-layer robustness checks also evaluated.The models are Llama-3.1-8B, Qwen3.5-9B, and Gemma-4-12B; late layers are used by default.Source: §3 · p. 3
  • Suffixes share the CRITICAL REMINDER – stem and are length-controlled, so adjacent comparisons isolate one ingredient; the intent and A/B suffixes are 29 tokens.The suffix ladder is described in Table 1.Source: §3 · p. 4
  • The evaluation covers 13 chat-safety, jailbreak, and injection benchmarks under LODO, scoring each dataset’s malicious examples against a shared balanced benign pool.Each dataset uses 2,000 prompts where available.Source: §3 · p. 4

3.1 The suffix and its deployment vehicle

  • On a KV-cache fork, the probe reads after the appended post-user suffix and the cache is reverted so the served response remains byte-identical to no-suffix serving.The monitor uses the existing forward pass rather than a second one.Source: §3.1 · p. 4, Figure 1 · p. 4
  • The suffix vehicle adds negligible overhead—a 29-token suffix and a few MB of KV cache—and requires one appended suffix in an existing probe pipeline.The authors inherit the vehicle’s perturbation robustness rather than re-establishing it.Source: §3.1 · p. 4
  • The primary metric is shared-benign AUC, with TPR@1% FPR capturing a strict deployment operating point; results are LODO out-of-fold and reported per dataset.The stricter deployment comparison is against a neutral suffix, because any generic suffix relocates the readout.Source: §3.1 · p. 4

4 Results and Analysis

Across single-position and pooling probes, suffixes improve LODO out-of-distribution ranking, with gains shaped by classification format, model headroom, and readout. Naming the criterion adds precision at a strict false-positive rate, while the best pooling suffix varies by readout.

4.1 Single-position probe

  • A classification suffix consistently improves out-of-distribution detection by up to ∼4 shared-benign AUC points over no suffix, with positive gains in 22 of 27 model–dataset cells; classification format carries the cross-dataset ranking gain.Figure 2 reports ∆shared-benign AUC versus no suffix as a mean over 9 LODO datasets (±SEM).Source: §4.1 · p. 4, Figure 2 · p. 5
  • Classification format, not the named criterion, carries ranking gains: content-free A/B matches intent, while naming the real classes adds low-FPR precision.Off-topic and merely attentive suffixes leave the probe unmoved; the reported precision gain is at a single noisy operating point.Source: §4.1 · pp. 4–5
  • Against the stricter neutral-suffix baseline, classification wins 38 of 52 per-dataset comparisons (73%; sign-test p=6×10−4).The Llama median gain is +1.8 points; its larger mean is inflated by mosscap.Source: §4.1 · p. 5
  • Gains concentrate where the probe has headroom, but bounded AUC and Qwen’s weak trend limit that account; aggregate means can hide dataset-level cancellation.Figure 3 reports pooled r=−0.78, while the Qwen trend is r=−0.29; reflect helps deepset and harmbench but drops mosscap from 83 to 63.Source: §4.1 · p. 5, Figure 3 · p. 6
  • On Gemma, the suffix aligns scattered per-source malicious directions into a shared axis, raising leave-one-source-out transfer AUC from 0.74 to 0.93.Llama and Qwen already have stronger malicious axes without a suffix, with transfer AUCs of 0.93 and 0.96, respectively.Source: §4.1 · p. 6

4.2 Multi-position (pooling) probes

  • Suffixes also lift pooling probes: 22 of 24 model–readout configurations and 110 of 120 readout–rung cells beat no suffix.Table 4 averages over the five-rung ladder, so most configurations improve without selecting a rung.Source: §4.2 · p. 6, Table 4 · p. 7
  • Pooling gains are model-modulated, and the best suffix is readout-dependent; content-free A/B stays within 0.3 AUC points of intent on average.Length and reflect top more readout columns than intent, although cross-readout differences among effective suffixes are at most about one point.Source: §4.2 · pp. 6–7

5 Limitations

The classification-specific benefit is limited and varies by model, while measurement and statistical power constrain how strongly the results can be interpreted. The reported gain is stable across readout layers for Gemma and near-zero across layers for Qwen.

5 Limitations

  • Qwen shows only a weak classification-specific effect, and cross-model magnitudes are confounded by different readout regimes.The Llama and Qwen readouts use L31, while Gemma uses L47.Source: §5 · p. 7
  • The reported improvement is a ranking gain in shared-benign AUC, not recalibration; within-dataset AUC covers only four mixed-class benchmarks.Accuracy@0.5 understates the AUC gain, and within-dataset AUC can misleadingly show reflect as +2.4 on Qwen.Source: §5 · p. 7
  • With only nine datasets, per-cell tests have limited power, so the claims rely on dataset-level sign tests, directional TPR@1% tests, and A/B-versus-intent equivalence tests.The minimum two-sided Wilcoxon p-value is 0.004.Source: §5 · p. 7
  • The effect is not layer-sensitive on Gemma (+3.1 at L40 vs. +3.2 at L47), while Qwen remains near-zero at L27 and L31.The passage characterizes Qwen’s weakness as intrinsic to the model rather than the layer.Source: §5 · p. 7

6 Conclusion

A classification suffix improves malicious-input detection, with the classification format carrying the ranking gain and the real criterion adding precision at low false-positive rates. The paper describes a KV-fork deployment path and identifies limits and open questions around tested inputs and why the format improves separability.

6 Conclusion

  • For prompted activation probes, the classification format carries the ranking gain, while naming the real criterion adds precision at strict false-positive rates.A content-free criterion suffices for the ranking improvement.Source: §6 · p. 7
  • A 29-token post-user suffix can be read through a KV-fork at the same cost as a no-suffix probe, without model fine-tuning or a second prefill.The probe is trained on the end-of-suffix activation.Source: §6 · p. 7
  • The evaluation uses chat-style benchmarks and does not vary input packaging or genre, so extension to multi-turn, tool-call, table, or code inputs remains conjectural.The work tests strict LODO, with each benchmark held out from probe training.Source: §6 · p. 8
  • The study leaves open whether classification makes inputs easier to read along any encoded axis or recruits a safety prior, and does not explain why the representation becomes more separable.The proposed next steps include broader transferable suffix optimization, agentic inputs, and a deeper representational account.Source: §6 · p. 8

A Per-dataset results and cross-model ROC

Across models, classification suffixes improve ranking most on non-saturated datasets, while near-ceiling sets move little and Qwen changes minimally. The per-dataset tables and macro-ROC figures summarize these comparisons across the LODO folds.

A Per-dataset results and cross-model ROC

  • The Llama per-dataset table reports shared-benign AUC by suffix, with values calculated out-of-fold under LODO at N=2000 per dataset.Shared-benign AUC compares each dataset’s malicious examples against a shared benign pool.Source: Table 5 · p. 9, A Per-dataset results and cross-model… · p. 9
  • Classification suffixes lift non-saturated datasets most, including Llama deepset 80→90 and harmbench 81→91, while near-ceiling sets remain nearly unchanged and Qwen moves little.The macro-ROC figures show the same ranking gain as area under the curve.Source: A Per-dataset results and cross-model… · p. 9
  • The Qwen and Gemma tables likewise report per-dataset shared-benign AUC by suffix; Qwen’s macro-ROC curves bunch near a 0.94 baseline, while Gemma separates by area.Figure 4 presents the Qwen and Gemma macro-ROC curves across nine LODO folds.Source: Table 6 · p. 9, Figure 4 · p. 10, Table 7 · p. 10

A.1 Precision at a strict operating point (TPR@1%)

At a strict 1%-FPR operating point, naming the real criterion improves recall over content-free and neutral suffixes on Llama and Gemma, while Qwen stays nearly flat. The effect varies by dataset, and the content-free suffix can fall below no suffix on some Llama datasets.

A.1 Precision at a strict operating point (TPR@1%)

  • At TPR@1%, intent exceeds A/B and reflect on Llama by approximately 28 percentage points and exceeds A/B on Gemma by 6.6 points, each across 9/9 datasets; Qwen is nearly flat.The authors treat the high-variance operating-point increment as directional rather than a calibrated size.Source: §A.1 · p. 10
  • The suffix-ladder comparison reports intent ahead of content-free A/B on Llama and Gemma, A/B below no suffix on Llama, and Qwen flat.The Gemma reflect figure averages eight datasets because wildjailbreak has no reflect run.Source: Table 8 · p. 10
  • The intent increment is broad on Llama and Gemma but varies in size by dataset; on several Llama datasets, A/B falls below no suffix where intent instead lifts.Source: §A.1 · p. 10
  • The appendix tables report per-dataset TPR@1% values by suffix for Llama, Qwen, and Gemma.Source: Table 9 · p. 11, Table 10 · p. 11, Table 11 · p. 11

B Benchmarks

The evaluation uses 13 safety benchmarks, with nine malicious-source sets scored against a shared benign pool. The primary panel reports each benchmark’s role and example counts.

B Benchmarks

  • 13 benchmarks cover malicious examples and benign chat, with nine malicious-source sets scored against the shared benign pool.The four benign-source sets contribute to that pool.Source: B Benchmarks · p. 12
  • The primary panel lists each benchmark’s role and numbers of benign and malicious examples.Counts vary slightly by model, with up to N=2000 examples per dataset.Source: Table 12 · p. 12, B Benchmarks · p. 12

C Probe architectures and settings

Probes are trained on frozen, standardized residual-stream activations, using either a single suffix-token position or multi-position pooling. The readout layers vary by model family.

C Probe architectures and settings

C.1 Full architecture × suffix grid

Across multi-position pooling readouts, classification suffixes usually improve LODO AUC where the baseline has headroom, although the strongest suffix depends on the readout. The single-position analyses link the gain to more aligned malicious representations and show that naming the criterion matters most at strict operating points.

C.1 Full architecture × suffix grid

  • Almost every pooling-readout column improves over no suffix where the baseline has headroom, but the best suffix varies by readout and Qwen is near ceiling.The grid spans six suffix conditions, four readouts, three model families, and two layers.Source: §C.1 · p. 12, Table 14 · p. 13

C.2 What the suffix does to the representation (where there is headroom)

  • Gemma’s leave-one-source-out transfer AUC rises from 0.74 with no suffix to about 0.93 with any suffix, while Llama and Qwen begin at 0.93 and 0.96.On Gemma, source-specific malicious directions shift from mean pairwise cosine 0.18 to 0.90 for classification rungs; the t-SNE is illustrative.Source: §C.2 · p. 14, Figure 5 · p. 14

C.3 Format/content dissociation and ROC detail

  • Content-free A/B and intent suffixes share ranking gains, but intent yields higher precision; on Llama, A/B falls below no suffix at 1% FPR.The Llama macro-ROC reports intent 0.936 and A/B 0.934 versus no-suffix 0.898 in area.Source: §C.3 · p. 15, Figure 6 · p. 15, Figure 7 · p. 15
Loading 2610.02413v1…