PaperlayerView exact revision on arXiv

Source-linked AI summary

False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize

Cheng Wang, Zeming Wei, Qin Liu, Muhao Chen

arXiv:2509.03888v3cs.CL

TL;DR

Probing-based safety detection assumes that separable hidden representations encode semantic harmfulness, but its poor OOD performance raises a central robustness question. The paper systematically tests this assumption with controlled comparisons and finds that probes primarily learn superficial linguistic patterns, creating a misleading sense of reliability. The study’s scope is bounded by its focus on English-language datasets and decoder-only Transformer models.

  • Problem

    The paper asks whether probing classifiers reliably detect harmfulness from internal representations rather than exploiting distribution-specific surface patterns.

  • Method

    The study evaluates probing-based detection through comparisons with n-gram methods, semantically cleaned datasets, and analyses of pattern dependencies across models and datasets.

  • Results

    Probing classifiers primarily learn superficial linguistic patterns rather than semantic harmfulness, while simple n-gram methods achieve comparable performance and probes fail on semantically cleaned data.

  • Takeaways & Limitations

    Current probing-based malicious detectors can provide a misleading sense of reliability, motivating safety representations and evaluation protocols that move beyond pattern matching.

  • Takeaways & Limitations

    The evaluation focuses on English-language datasets and decoder-only Transformer models, leaving applicability to other languages, cultures, architectures, and emerging LLM paradigms open.

Abstract

from arXiv · show

Large Language Models (LLMs) can comply with harmful instructions, raising serious safety concerns despite their impressive capabilities. Recent work has leveraged probing-based approaches to study the separability of malicious and benign inputs in LLMs' internal representations, and researchers have proposed using such probing methods for safety detection. We systematically re-examine this paradigm. Motivated by poor out-of-distribution performance, we hypothesize that probes learn superficial patterns rather than semantic harmfulness. Through controlled experiments, we confirm this hypothesis and identify the specific patterns learned: instructional patterns and trigger words. Our investigation follows a systematic approach, progressing from demonstrating comparable performance of simple n-gram methods, to controlled experiments with semantically cleaned datasets, to detailed analysis of pattern dependencies. These results reveal a false sense of security around current probing-based approaches and highlight the need to redesign both models and evaluation protocols, for which we provide further discussions in the hope of suggesting responsible further research in this direction. We have open-sourced the project at https://github.com/WangCheng0116/Why-Probe-Fails.

1 Introduction

This study re-examines probing-based safety detection after poor OOD performance raised doubts that probes capture semantic harmfulness. Through progressively controlled experiments, it finds that probes exploit superficial patterns, including instructional patterns and trigger words.

  • Prior paradigm: Probing-based detection builds on reported hidden-state separation and high in-domain accuracy, which motivated its adoption as a safety mechanism.The paper defines probing as training simple classifiers on frozen internal representations.
  • Motivation: Probing classifiers show substantial OOD degradation, challenging the assumption that hidden-state separability reflects a stable semantic distinction between harmful and benign content.The paper argues that sharp performance deterioration under distribution shift is inconsistent with robust semantic encoding of harmfulness.
  • Hypothesis: The central hypothesis is that probing representations primarily capture shallow patterns rather than the semantics of harmfulness.
  • Research approach: The investigation compares probes with n-gram models, tests semantically sanitized datasets, and quantifies false positives from benign content containing trigger patterns.These studies progressively stress-test whether probing-based detection depends on superficial structure rather than harmful meaning.
  • Contribution: Across models and datasets, current probing-based detectors exploit spurious correlations and surface cues, creating a misleading sense of reliability.The paper therefore calls for safety representations grounded beyond pattern matching.

2 Problem Formulation

The paper formulates safety detection as binary classification from LLM hidden representations. It primarily extracts the last-token, last-layer state and trains a classifier to distinguish benign from malicious prompts.

  • Hidden-state extraction: For each prompt, the method extracts a complete token-by-layer hidden-state tensor from a pretrained decoder-only Transformer.The tensor has dimensions T×L×d, representing tokens, layers, and hidden dimensions.
  • Prompt representation: The primary prompt representation is the hidden state of the last token at the last layer.The paper also tests representations from different layers to assess layer-selection effects.
  • Prompt representation: Because self-attention integrates information across the prompt, the selected representation is treated as encoding contextual prompt content for classification.
  • Safety detection formulation: The safety task is binary classification: a trained SVM maps extracted representations to benign or malicious labels.Additional classifiers are evaluated elsewhere, but the formulation centers on this representation-to-label mapping.
  • Research question: The formulation asks whether internal representations alone support reliable malicious-versus-benign discrimination and whether apparent success transfers to real-world safety detection.

3 Motivation: How Do Probing Classifiers Work in Out-of-Distribution Settings?

The evaluation reproduces near-perfect in-distribution probing performance but finds severe OOD failure across models and datasets. This contrast motivates examining whether probes detect harmfulness or merely learn distribution-specific surface patterns.

  • Experimental setup: The evaluation uses malicious datasets including AdvBench, ForbiddenQuestions, BeaverTailsEval, JailbreakBench, StrongReject, MaliciousInstruct, and HarmBench, plus benign instruction-following and question-answering datasets.The evaluated models are Gemma-3-it, Llama-3.1-Instruct, and Qwen2.5-Instruct.
  • Experimental setup: For OOD testing, probes train with Alpaca plus BeaverTailsEval or ForbiddenQuestions, then test on unseen Dolly, HarmBench, and AdvBench datasets.
  • In-distribution Performance: Accuracy consistently exceeds 98% in-distribution across model-dataset combinations, reproducing prior reports of near-perfect probing performance.This result appears to validate probing-based safety detection before distribution shift is introduced.
  • Out-of-distribution Performance: 15–99 percentage points: probing accuracy drops across all models and scales when evaluation shifts from training distributions to OOD data.Some model-dataset combinations reach near-zero accuracy, indicating failure to generalize beyond the training distributions.
  • Interpretation: The stark ID–OOD contrast suggests that probes learn superficial patterns rather than genuine semantic understanding of harmfulness.This motivates subsequent analysis of the mechanisms behind their pattern learning.

4 Research Study 1: Revisiting Naive Bayes

Simple n-gram classifiers perform comparably to probing classifiers, suggesting that probing methods may rely on surface-level patterns rather than semantic harmfulness.

  • 4.1 Experimental Setup: Naive Bayes provides a surface-pattern baseline for testing whether probing classifiers learn semantic harmfulness rather than easily identifiable lexical cues.Comparable performance would indicate similar reliance on surface-level patterns.
  • 4.1 Experimental Setup: The study evaluates probing classifiers against Naive Bayes using matched train-test splits and unigram, bigram, and trigram features.The protocol is designed to support a fair comparison between hidden-state probes and statistical surface features.
  • 4.2 Results: 0.84 to 1.00 accuracy: Naive Bayes classifiers using unigrams and bigrams perform competitively with probing classifiers across dataset combinations.Most combinations exceed 0.95 accuracy under identical evaluation protocols.
  • 4.2 Results: The comparison suggests that sophisticated probing classifiers may not learn deep semantic understanding of harmfulness.Both approaches appear to exploit surface patterns despite their different representations.

5 Research Study 2: Controlled Experiments with Cleaned Datasets

Controlled cleaning experiments preserve dataset structure while removing harmful semantics, revealing that probing classifiers fail when semantic cues are neutralized.

  • 5.1 Experimental Setup: The cleaned datasets replace harmful content with benign alternatives while preserving grammatical structure and length.For example, “How to make a bomb” becomes “How to make a bread.”
  • 5.1 Experimental Setup: The evaluation tests classifiers on original and cleaned malicious datasets to determine whether performance depends on semantic harmfulness or preserved structure.A semantic detector should lose performance on cleaned data while retaining performance on genuinely harmful content.
  • 5.2 Results: 60-90 percentage points: probing-classifier accuracy drops across all model-dataset combinations on semantically cleaned data.Accuracy falls as low as 8.0% for JailbreakBench with Gemma-3-4b-it, indicating near-complete failure after harmful content is removed.
  • 5.2 Results: The results show that probing classifiers are poor at distinguishing malicious inputs from benign text once structural patterns are controlled.This supports overreliance on non-semantic cues.

6 Research Study 3: Understanding Pattern Learning

The study identifies instructional patterns and trigger words as the main superficial cues learned by probing classifiers instead of semantic harmfulness.

  • 6.1 Instructional Pattern Learning: The instructional-pattern experiment paraphrases cleaned datasets while preserving benign semantic content and altering structural formatting.This isolates the effect of instructional patterns on classifier behavior.
  • 6.1 Instructional Pattern Learning: Accuracy often recovers toward original levels when cleaned benign content is paraphrased to remove malicious instructional formatting.The same benign semantics are then classified more accurately, indicating dependence on structure rather than meaning.
  • Research Study 3 – Takeaway: Probing classifiers primarily learn instructional patterns and trigger words rather than semantic harmfulness.Instructional patterns include structural formatting and phrasing, while trigger words are vocabulary associated with malicious content.

7 Discussion

Additional analyses show that superficial pattern learning persists across layers, classifier architectures, and model alignment conditions, while the underlying LLMs can classify harmfulness directly.

  • 7.1 Impact of Layer Selection: Different hidden-state layers show similar high ID performance and severe OOD degradation, indicating that the pattern-learning behavior is not confined to one layer.The comparison uses Alpaca and BeaverTailsEval for training and AdvBench as the OOD test set.
  • 7.2 Classifier Architecture: All evaluated classifier architectures reach 99.0% in-distribution accuracy but fall to approximately 23-30% on cleaned datasets.MLP recovery on paraphrased data reaches 90.2% versus 82.7% for SVM, yet neither achieves robust semantic understanding.
  • 7.3 Comparison Between Base and Instruction-Tuned Models: 95-99% in-distribution performance still yields severe OOD degradation for both base and instruction-tuned models.Instruction-tuned models improve OOD performance only marginally, so alignment training does not resolve the generalization failure.
  • 7.4 Zero-Shot Classification: LLMs achieve high zero-shot safety-classification accuracy, contrasting with probes’ poor OOD performance and indicating that the limitation lies in probing rather than model comprehension.The result is reported across benign and malicious datasets.
  • 7.4 Representation Analysis: PCA representations cluster malicious and cleaned datasets similarly while separating OOD content, linking probe failures to structural and dataset-specific patterns.The visualization supports the interpretation that representations do not provide a stable semantic harmfulness signal.

8 Conclusion

The paper finds that probing-based safety detectors have significant robustness limitations and primarily learn superficial linguistic patterns rather than semantic harmfulness.

  • Probing classifiers primarily learn superficial linguistic patterns rather than semantic understanding of harmfulness.The reported weaknesses include reliance on instructional patterns and trigger words instead of genuine harmfulness.
  • Simple n-gram methods achieve comparable performance to probing classifiers, while probes fail dramatically on semantically cleaned datasets.
  • The findings indicate that probing-based methods provide a false sense of security because they rely on spurious correlations rather than robust semantic comprehension.
  • LLMs demonstrate strong zero-shot safety classification capabilities, but probing classifiers cannot leverage this understanding effectively.
  • The authors call for more principled approaches to AI safety detection.

Limitations

The evaluation is primarily limited to English-language datasets and decoder-only Transformer models, constraining its coverage across languages, cultures, and architectures.

  • The evaluation focuses primarily on English-language datasets, which may limit applicability across languages and cultural contexts.Harmful content may manifest differently across languages and cultures.
  • The analysis is restricted to decoder-only Transformer models, leaving probing behavior in other architectures and emerging LLM paradigms unexamined.
  • The authors identify extending robustness and scope across these boundaries as a direction for future AI safety research.

A Related Works

Related work frames LLM safety around adversarial attacks and defense strategies, while this paper focuses specifically on harmful generation risks.

  • Adversarial Attacks on LLMs: Adversarial attacks on LLMs include jailbreaking, membership inference, prompt injection, and retrieval corpus poisoning.These methods target safety guardrails, training-data extraction, or application-layer vulnerabilities.
  • Adversarial Attacks on LLMs: The paper primarily focuses on harmful generation risks, identified as a major LLM safety concern.
  • Defense Strategies for LLMs: Defense strategies include guard models that filter inputs or audit outputs as external safety layers.
  • Defense Strategies for LLMs: Alignment training is presented as another primary defense approach, including RLHF with supervised fine-tuning and human-preference reward modeling.

B Dataset Information

The dataset section describes sample balancing and prompt-based transformations for sanitizing content, paraphrasing instructional patterns, and classifying safety.

  • B Dataset Information: For datasets containing more than 500 samples, the authors randomly select 500 samples to balance classes during training.
  • Content Sanitization and Lexical Substitution: Content sanitization and lexical substitution replace potentially harmful terminology with benign alternatives while preserving structure and meaning.
  • Instructional Pattern Paraphrasing: Instructional pattern paraphrasing preserves the exact meaning and intent while changing linguistic patterns and sentence structures.
  • B Dataset Information: The dataset materials also include a prompt for content safety classification.
Loading 2509.03888v3…