PaperlayerView exact revision on arXiv

Source-linked AI summary

When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift

Max Fomin

arXiv:2602.14161v2cs.LG

TL;DR

Prompt-attack classifiers are often evaluated in ways that can reward dataset-specific cues rather than generalizable attack detection, risking overestimation of performance. The paper trains activation-based classifiers across diverse attack and benign datasets, introduces LODO and coefficient-retention analysis, and finds persistent CV–LODO gaps with overlapping dataset-identity and safety directions. It also shows that LODO-weighted attributions can filter dataset artifacts, while a method that closes the gap remains unavailable.

  • Problem

    Current prompt-attack evaluations can overestimate generalization because classifiers may exploit dataset-identity signals rather than transferable attack semantics.

  • Method

    The paper evaluates activation-based classifiers across multiple LLMs and datasets using LODO, analyzes coefficient retention for SAE features, and applies retention to per-prompt attributions.

  • Results

    Standard CV overestimates pooled AUC by 8.0–16.5 percentage points across four LLMs, while dataset-identifying and safety-relevant subspaces partially overlap and LODO-weighted attributions filter dataset artifacts.

  • Takeaways & Limitations

    LODO provides a dataset-level diagnostic for detecting inflated prompt-attack classifier performance and identifying explanations that depend on dataset artifacts.

  • Takeaways & Limitations

    The paper does not present a method that closes the gap, and its activation-based approach requires white-box access to model internals, limiting direct deployment with closed-source APIs.

Abstract

from arXiv · show

Detecting prompt injection, jailbreak attacks, and harmful requests is critical for deploying LLM-based agents safely, yet current evaluation practices in this literature overestimate generalization. We train activation-based classifiers (linear probes on LLM hidden states) on a benchmark of 18 datasets (prompt attacks plus benign sources) and propose Leave-One-Dataset-Out (LODO) evaluation, where the held-out dataset is never seen during training. Across four LLMs from three families (Llama-3.1-8B, Gemma-3-27B, Qwen-3.5-2B/4B), standard cross-validation reports a pooled AUC 8.0-16.5 points higher than LODO; per-dataset held-out-test-vs-LODO accuracy gaps span 1-25 points. To understand the gap, we analyze the LODO stability of a linear probe's per-feature classifier coefficients, defining a retention metric for sparse-autoencoder (SAE) features that flags dataset-dependent shortcuts. 28-44% of top SAE features are shortcuts across models, a dataset-identity classifier reaches 96.6%, and the dataset-identifying and safety-relevant subspaces partially overlap. Standard domain-generalization fixes such as adversarial training, subspace projection, sample reweighting, and class balancing do not close the gap. Finally, we show LODO-weighted SAE attributions filter dataset artifacts for more reliable per-prompt explanations. We release our framework at https://github.com/maxf-zn/prompt-mining so future prompt-attack classifiers can be evaluated under LODO alongside CV.

1 INTRODUCTION

Prompt-attack classifiers are evaluated on mixed benchmarks, but standard train-test evaluation can reward dataset-identity cues instead of transferable attack semantics. The paper introduces LODO to measure generalization to entirely unseen dataset distributions and reports a substantial CV overestimate across models.

  • Motivation: LLM agents process external data such as emails, documents, tool outputs, and API responses, creating vulnerabilities to malicious instructions embedded in that data.Prompt injection exploits the difficulty of distinguishing trusted user instructions from untrusted data.
  • Evaluation gap: Standard evaluation combines multiple attack and benign datasets while placing samples from the same sources in training and test folds.This protocol can produce near-perfect performance without testing transfer to a new dataset distribution.
  • Contribution: LODO trains on N−1 datasets and tests on the held-out dataset, exposing an 8.4-percentage-point CV overestimate in pooled AUC (0.996 vs 0.912).The gap is reported across four models spanning 2B–27B parameters.
  • Evaluation gap: Classifiers can exploit dataset-identity signals, such as formatting associated with WildJailbreak or Enron, rather than learning generalizable attack patterns.Single-class datasets make this shortcut especially easy because dataset identity predicts the class label.
  • Contribution: The paper contributes LODO evaluation, a coefficient-retention shortcut metric, overlap analysis between dataset-identity and safety subspaces, and LODO-weighted explanations.These components target evaluation validity, feature diagnosis, and attribution reliability.

2 RELATED WORK

Related work covers prompt-attack detectors, adversarial-robustness methods, SAE-based interpretability, shortcut learning, and evaluation pitfalls. This paper positions LODO as a dataset-level distribution-shift diagnostic complementary to adversarial testing and deployment-oriented defenses.

  • Prompt-attack detection: Prompt-attack detectors include fine-tuned classifiers and activation or attention-based probes, while text-only guardrails lack the tool-schema interface needed for agentic attacks.TaskTracker holds out attack types but evaluates within constituent datasets, leaving dataset-level generalization unexamined.
  • Evaluation perspectives: Adversarial-robustness methods target worst-case attackers, whereas LODO measures static distribution shift across datasets.The paper treats these as complementary predeployment diagnostics.
  • SAE interpretability: SAEs provide sparse, interpretable features for classification and explanation, but prior work and this paper report weaker SAE-probe performance than raw activations in relevant settings.The paper adds LODO-weighted attribution to remove dataset-artifact features.
  • Shortcut learning: Group DRO and Just Train Twice address shortcuts within a single training distribution, while this setting treats the dataset itself as the shortcut.The distinction motivates dataset-level evaluation and analysis.
  • Evaluation pitfalls: Static evaluations can overstate robustness along either adversarial or distribution-shift axes, so both diagnostics are relevant before deployment.The paper focuses on the distribution-shift axis rather than adaptive attacker behavior.

3 METHODS

The paper builds a multi-domain malicious-versus-benign benchmark, extracts raw and SAE activation features, and evaluates linear probes under dataset-level distribution shift. It also measures feature stability and uses retention to reweight explanations.

  • Dataset and threat model: The benchmark contains 18 datasets and 105K samples spanning harmful requests, jailbreaks, prompt injections, extraction attacks, and benign sources.It contains 47% malicious samples, and each dataset is held out in turn under LODO.
  • Dataset and threat model: The defender has white-box access to LLM activations during inference but cannot modify the underlying model.Text-only baselines are evaluated for comparison.
  • Activation-based classification: Activation probes use the residual-stream vector from the last user-message token after applying the model’s chat template.The study evaluates four main LLMs spanning 2B–27B parameters, with a partial 70B replication.
  • Feature representations: Raw activations are compared with sparse-autoencoder features, which decompose activations into sparse, higher-dimensional representations intended to expose interpretable concepts.The SAE representation used for Llama has 131,072 features.
  • LODO evaluation: LODO trains on all datasets except D_i and evaluates on the unseen D_i, measuring out-of-distribution generalization without exposure to the test dataset’s distribution.The paper reports both per-dataset and pooled held-out predictions and compares LODO with leave-one-category-out evaluation.
  • Shortcut analysis: The coefficient-retention metric compares each feature’s full-classifier coefficient with coefficients learned when individual datasets are held out.Low or sign-flipping retention marks dataset shortcuts, including pure dataset shortcuts and context-dependent shortcuts.
  • Explanations: LODO-weighted explanations multiply each feature’s activation and classifier coefficient by its retention score to downweight dataset shortcuts.A separate dataset classifier tests whether activations make dataset provenance predictable.

4 EXPERIMENTS

Experiments test the benchmark composition, model and baseline comparisons, cross-validation versus LODO, and explanation stability. The study is organized around whether standard evaluation reflects transfer to unseen datasets and whether LODO diagnoses the resulting shortcuts.

  • Experimental questions: The experiments ask how much CV overestimates generalization, whether the gap spans model families, how many features are shortcuts, and whether mitigations reduce it.They also compare activation probes with production baselines and test whether LODO-stable features improve explanations.
  • Benchmark: The benchmark comprises 18 datasets with 105K samples and 47% malicious inputs, covering direct jailbreaks, indirect injection, extraction attacks, and benign sources.LODO uses each dataset as the held-out test set rather than a fixed train-test split.
  • Models and baselines: Models include Llama-3.1-8B, Gemma-3-27B, and Qwen-3.5-2B/4B, compared with guardrails, Llama-as-Judge variants, and dedicated prompt-injection detectors.Evaluation includes 5-fold CV, held-out test sets, and LODO.

5 RESULTS

Across models and evaluation settings, standard benchmarks substantially overestimate prompt-attack classifier generalization because dataset-specific signals act as shortcuts. LODO exposes this gap, identifies unstable SAE features, and shows that common interventions do not reliably remove it.

  • Evaluation gap: 0.996 AUC under 5-fold CV and 0.997 AUC on the held-out test set fall to 0.912 under LODO for Llama-3.1-8B raw activations.The standard evaluations use samples from dataset sources also present during training, whereas LODO holds out an entire dataset.
  • Evaluation gap: 1.2% to 25.4% are the per-dataset accuracy gaps between held-out test sets and LODO, including substantial gaps on mixed-class datasets.The weighted CV–LODO gap across five mixed-class datasets is 12.5pp after excluding the near-single-class BIPIA outlier.
  • Dataset shortcuts: 96.6% accuracy is achieved by a dataset-identity classifier on SAE features, indicating that dataset provenance is readily recoverable from the representations.Within-dataset similarity also exceeds cross-dataset similarity in both text-embedding and activation spaces.
  • Dataset shortcuts: 28% of top SAE features on Llama-3.1-8B and 44% on Gemma-3-27B are shortcuts under the reported retention threshold.Context-dependent shortcuts can show strong class separation yet fail when a dataset is held out, so firing-ratio heuristics alone miss some shortcuts.
  • Production comparison: At threshold 0.9, the probe’s 3.7% FPR is comparable to LlamaGuard’s 3.0% while detection leads occur on jailbreak, indirect, agentic, and mixed attacks.LlamaGuard remains stronger on conversational harmful refusals, where it is specifically trained.
  • Interventions and explanations: 98.1% of samples show top-20 attribution changes after LODO weighting, with demoted features having mean retention 0.265 versus 0.990 for promoted features.This indicates systematic filtering of dataset-dependent features, although human evaluation of the rankings remains future work.

6 DISCUSSION

Existing guardrails do not cover all agentic security surfaces, while activation probes provide complementary coverage. The CV–LODO gap reflects benchmark construction and overlapping dataset-identity and safety directions, leaving disentanglement as an open challenge.

  • Agentic security: Text-only guardrails cannot represent tool schemas or non-alternating roles, limiting coverage of agentic attacks.Activation probes complement rather than replace these systems by covering indirect and agentic surfaces.
  • Understanding the CV–LODO gap: The CV–LODO gap arises because classifiers exploit within-dataset regularities that fail to transfer across dataset distributions.This pattern holds across every tested model family.
  • Understanding the CV–LODO gap: Ablating 14 identified SAE shortcuts changes pooled AUC by only −0.1pp, so the authors do not claim shortcuts cause the gap.Other features compensate through redundant decision boundaries.
  • Understanding the CV–LODO gap: Dataset-identifying and safety-relevant directions partially overlap, suggesting that closing the gap requires methods that disentangle them.The authors identify this as an open challenge rather than a demonstrated causal mechanism.

7 CONCLUSION

LODO exposes substantial overestimation in conventional cross-validation for prompt-attack detection, while dataset-identifying and safety-relevant representations partially overlap. Standard domain-generalization fixes do not close this gap.

  • Conclusion: 8–16pp CV overestimate across four LLMs accompanies per-dataset gaps of 1–25% under LODO.The comparison concerns prompt-attack detection.
  • Conclusion: 89%/96.6% dataset-classifier performance and targeted-projection effects are consistent with directional overlap, not proof of causal entanglement.Targeted subspace projection changes LODO performance by −1.75pp versus −0.02pp for equal-rank random projections.
  • Conclusion: Standard domain-generalization fixes do not close the CV–LODO gap.The conclusion reports this across the evaluated interventions.

8 LIMITATIONS

The study’s evaluation and interpretation have practical scope boundaries: activation access is required, LODO depends on dataset choice and can be costly, and no definitive mitigation is provided.

  • Deployment scope: The activation-based approach requires model-internal access, limiting direct deployment with closed-source APIs.Possible alternatives include an open-weight sidecar classifier or later distillation into text-based classifiers.
  • Evaluation scope: LODO measures static distribution generalization, not robustness to adaptive attackers or novel evasion strategies.Adversarial evaluation remains complementary and is not addressed.
  • Dataset scope: LODO’s conclusions depend on dataset selection: shared methodologies can understate deployment gaps, while unrepresentative datasets can bias estimates in either direction.The paper recommends spanning distinct attack surfaces and formatting conventions.
  • Computational cost: LODO requires training K classifiers for K datasets, which may become prohibitive with expensive fine-tuning.With logistic regression, each fold completes in minutes.
  • Interpretability scope: SAE explanations indicate detected classifier patterns rather than causal effects on the underlying model, and human evaluation remains future work.Individual explanations are therefore indicative rather than exhaustive.
  • Mitigation scope: The paper establishes the gap and tests seven interventions but provides no method that closes it.A definitive solution remains open.

9 ETHICS STATEMENT

This work studies defensive detection of malicious content in LLM-based agents using publicly released safety-research datasets. It introduces no attack methods and assumes the defender operates the classifier.

  • Scope: The experiments detect prompt injection, jailbreak attacks, and harmful requests rather than generating them.All datasets used are publicly released for safety research.
  • Threat model: The threat model assumes the defender operates the classifier, not the attacker.Guardrail limitations are disclosed to motivate stronger defenses, not facilitate evasion.

10 POTENTIAL RISKS

The paper identifies risks in interpreting LODO and benchmark results, while documenting dataset construction and model-dependent generalization behavior. It also releases code and benchmark tooling, but limits additional attacker benefit by withholding trained checkpoints and activations.

  • False sense of security from LODO: LODO measures static distribution generalization across a chosen benchmark, so adaptive attackers, novel attack families, and production drift can still produce undetected failures.The paper recommends pairing LODO with adversarial evaluation and ongoing red-teaming.
  • Release and dual-use: The released framework lowers the barrier for defenders and benchmark authors, but trained probe checkpoints and extracted activations are not released.The authors judge that this creates limited additional uplift for attackers beyond already accessible information.
  • Benchmark construction: Dataset construction combines heterogeneous prompt formats, including tool-use conversations, embedded-context attacks, password-extraction tasks, and email injections.Prompts are normalized into PromptSpec records and chat-template text, with tool schemas preserved when applicable.
  • Activation capture: Position choice creates a performance-generalization trade-off: position −1 reaches mean per-dataset AUC 0.904 versus 0.867 at position −5, but shortcut prevalence rises from 30% to 46%.Raw activation results are more stable across positions.
  • Configuration sensitivity: No single layer and position dominate across datasets, with layer-31 and layer-27 at position −5 reaching 81.8–82.3% weighted aggregate accuracy.For example, llmail favors −5 while harmbench favors −1.
  • Model and dataset dependence: Dataset identity reaches 96.6% accuracy on SAE features, while LODO accuracy varies sharply by model, including InjecAgent results from 0% to 100%.The authors also report that qualitative patterns carry over across models, but the exact 70B cross-validation gap was not quantified.

G MITIGATION EXPERIMENTS

The mitigation experiments test adversarial training, subspace removal, reweighting, balancing, shortcut ablation, and broader holdouts. None closes the generalization gap, while interventions can have strongly heterogeneous per-dataset effects.

  • Representation interventions: DANN changes aggregate performance by only +0.4pp, while per-dataset effects range from +13pp on BIPIA to −9pp on mosscap across seeds.This variance indicates that adversarial training does not reliably remove dataset-dependent behavior.
  • Representation interventions: Subspace projection changes aggregate performance by −1.75pp, with effects from +21.4pp on llmail to −34.4pp on InjecAgent.The targeted subspace contains 9.2% of safety-classifier weight norm versus 0.4% expected randomly, indicating partial overlap between dataset identity and safety signals.
  • Reweighting and balancing: Sample reweighting changes aggregate performance by −1.0pp for inverse-frequency weights and −0.5pp for inverse-sqrt weights.Explicit per-dataset balancing changes performance by 0.0pp, while global balancing changes it by −0.1pp.
  • Shortcut ablation: Zeroing 22 identified Gemma-3-27B SAE shortcuts changes aggregate performance by +0.7pp, the only intervention with a consistent positive sign but insufficient to close the gap.The result supports shortcut analysis as diagnostic rather than a complete mechanistic explanation.
  • Holdout granularity: LOCO reduces aggregate accuracy by −2.6pp and pooled AUC by −2.8pp relative to LODO, placing LODO between standard CV and category-level holdout.The harmful category drops −26.5pp under LOCO, while llmail improves +9.9pp.
  • Holdout granularity: Within-category LODO variation reaches 48pp for harmful attacks, 36pp for indirect injection, and 23pp for jailbreaks.These differences support dataset-level evaluation because formatting, style, and structure contribute beyond attack methodology.
  • Baseline comparison: Logistic regression exceeds LPM by 5.8pp in weighted accuracy, reaching 81.8% versus 76.0% under LODO.The largest gaps occur on indirect injection: +31pp for BIPIA and +37pp for llmail.
  • Dataset distinguishability: Dataset identity reaches 96.6% SAE-feature accuracy, and t-SNE shows distinct dataset clusters that make shortcut exploitation easy.The dataset-classification baseline for 18-way classification is 5.6%.

O.3 RETENTION-METRIC SENSITIVITY

The retention analysis is sensitive to feature count and retention threshold, while alternative coefficient-stability metrics largely agree with the proposed shortcut measure. Removing identified shortcuts barely changes aggregate AUC, and calibration remains dataset-dependent.

  • Top-K and retention threshold: At the 50% retention threshold, shortcut prevalence ranges from 25% at K=20 to 43% at K=200.For K=50, prevalence is 14% at 30% retention, 28% at 50%, and 56% at 70%.
  • Firing-ratio threshold: At K=50 and 50% retention, the 1.5× firing-ratio threshold yields Q1=8 and Q2=6, with 43% context-dependent shortcuts.The 1.5× threshold is described as providing balanced identification.
  • Alternative stability metrics: Sign agreement averages 99.4%, Spearman correlation averages 0.89, and coefficient variation has mean 0.16 across LODO folds.Only 21 of 200 features show sign flips, and all have negative retention.
  • Shortcut ablation: Ablating all identified shortcuts changes mean per-dataset AUC by only −0.1pp, from 0.867 to 0.866.Heterogeneous effects remain, including +2.8pp on deepset and −2.4pp on jayavibhav.
  • Threshold calibration: The pooled F1-optimal threshold is t∗=0.01 with F1=0.848, compared with F1=0.793 at t=0.5.Per-dataset optima range from 0.01 for BIPIA and deepset to 0.73 for jayavibhav.
  • Per-dataset performance: Under LODO, mixed-class dataset AUC ranges from 0.655 on BIPIA to 0.994 on safeguard, while AP ranges from 0.719 on qualifire to 0.987 on safeguard.Figure 6 presents ROC and precision-recall curves for six mixed-class datasets.
  • Baseline comparison: The probe retains advantages over production baselines on indirect injection and agentic attacks, but tight matched-FPR operating points substantially reduce all detection rates.ProtectAI has 4.2% FPR but misses harmful attacks, whereas Deepset reaches 82.4% FPR.
  • Attribution example: SAE attribution identifies roleplay, dangerous-request, and safety-guideline features in a jailbreak example, with each contribution computed from coefficient × activation.The example receives score 0.896 at threshold 0.5.
Loading 2602.14161v2…