PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Safety From Within: Detecting Harmful Content with Internal Representations

Difan Jiao, Yilun Liu, Ye Yuan, Zhenwei Tang, Linfeng Du, Haolun Wu, Ashton Anderson

arXiv:2604.18519v1cs.AI

TL;DR

Existing guard models rely mainly on terminal-layer representations, motivating better use of safety information distributed across LLM internals. SIREN identifies safety neurons with linear probing and adaptively aggregates them without modifying the underlying model, achieving stronger performance, generalization, and efficiency with far fewer trainable parameters.

  • Problem

    Existing guard models rely on terminal-layer representations and leave safety-relevant features distributed across internal layers insufficiently harnessed for harmfulness detection.

  • Method

    SIREN identifies safety neurons through linear probing and combines them across layers with performance-weighted aggregation to train a plug-and-play harmfulness detector.

  • Results

    SIREN outperforms state-of-the-art open-source guard models across multiple benchmarks in performance, generalization, and efficiency while using 250× fewer parameters.

  • Takeaways & Limitations

    LLM internal states provide a foundation for practical harmfulness detection with streaming generalization and single-forward-pass inference.

  • Takeaways & Limitations

    The current work focuses on binary harmfulness classification, and linear probing may require adaptation for substantially non-transformer architectures or non-linearly represented concepts.

Abstract

from arXiv · show

Guard models are widely used to detect harmful content in user prompts and LLM responses. However, state-of-the-art guard models rely solely on terminal-layer representations and overlook the rich safety-relevant features distributed across internal layers. We present SIREN, a lightweight guard model that harnesses these internal features. By identifying safety neurons via linear probing and combining them through an adaptive layer-weighted strategy, SIREN builds a harmfulness detector from LLM internals without modifying the underlying model. Our comprehensive evaluation shows that SIREN substantially outperforms state-of-the-art open-source guard models across multiple benchmarks while using 250 times fewer trainable parameters. Moreover, SIREN exhibits superior generalization to unseen benchmarks, naturally enables real-time streaming detection, and significantly improves inference efficiency compared to generative guard models. Overall, our results highlight LLM internal states as a promising foundation for practical, high-performance harmfulness detection.

1 Introduction

Existing guard models typically detect harmfulness from terminal-layer representations, leaving safety-relevant information across internal layers underused. SIREN addresses this gap with a lightweight, plug-and-play detector that identifies and adaptively combines internal safety features.

  • Guard models commonly perform harmfulness detection as generative classification from terminal-layer representations.
  • SIREN targets the underused safety-relevant features encoded throughout LLM internal layers.
  • SIREN identifies safety neurons with linear probing and combines them across layers using adaptive weighting.
  • 14M trainable parameters are introduced by SIREN on a 4B backbone, compared with fine-tuning the full 4B parameters for an equivalent guard model.
  • SIREN requires no modifications to the underlying LLM and operates as a plug-and-play component.
  • 250× fewer parameters accompany SIREN’s superior performance, generalization, and efficiency across multiple benchmark evaluations.

2 Related work

Content-safety safeguards include discriminative classifiers and generative guard models, but both mainly rely on terminal-layer information. Prior internal-representation work motivates SIREN’s broader evaluation of harmfulness classification across complete prompts and responses.

  • Content-safety detection is broadly organized around discriminative classifiers and generative guard models.
  • Discriminative safeguards adapt encoder-only transformers such as BERT and RoBERTa with specialized classification heads.
  • Generative guards reformulate safety detection as generative classification and classify prompts and responses with instruction-tuned LLMs.
  • Both paradigms primarily use terminal-layer representations, while generative guards additionally incur autoregressive inference costs.
  • Intermediate LLM layers encode richer task-relevant features across tasks including sentiment analysis, factual knowledge retrieval, and question answering.
  • Prior safety studies leverage internal signals for specific scenarios, whereas SIREN evaluates complete prompts and responses across diverse safety categories and standard guard-model benchmarks.

3 Methodology

SIREN selects safety-relevant neurons within individual LLM layers and aggregates them into cross-layer features for harmfulness classification. It uses sparse linear probing for neuron selection and performance-based layer weighting before downstream classification.

  • Safety Neuron Identification: SIREN first identifies safety neurons within each layer using linear probing for content-safety classification.
  • Safety Neuron Identification: The method extracts layer representations, including residual streams and feedforward activations, using mean pooling over token-level representations.
  • Safety Neuron Identification: Layer-wise probes use L1 regularization, with larger weight magnitudes indicating neurons more relevant to harmfulness detection.
  • Safety Neuron Identification: SIREN selects the minimal set of top-ranked normalized neurons whose cumulative weight exceeds threshold η, sparsifying latent representations.
  • Adaptive Neuron Aggregation: Layer weights are based on validation F1 scores, prioritizing high-performing layers and down-weighting layers with lower task relevance.
  • Adaptive Neuron Aggregation: Weighted safety-neuron activations are concatenated across layers and passed to an MLP, without modifying LLM weights or architecture.

4 Experiments

SIREN is evaluated against open-source guard models across efficacy, generalization, streaming detection, and efficiency. Across these settings, it improves detection while requiring substantially fewer trainable parameters and lower inference cost.

  • Efficacy: SIREN outperforms safety-specialized guard models across four matched backbone pairs, achieving 86.7% versus 83.4% and a 15% improvement over LlamaGuard3-1B.The matched backbones range from 0.6B to 8B parameters.
  • Efficacy: SIREN maintains balanced precision and recall across benchmarks, whereas guard models show greater variance in policy consistency.The precision-recall analysis covers both prompt-level and response-level harmfulness detection.
  • Generalizability: SIREN generalizes to the unseen Think benchmark and consistently outperforms safety-specialized guard models across reasoning-model backbones.Think evaluates safety detection on manually annotated reasoning traces generated by multiple reasoning models.
  • Generalizability: SIREN captures more harmful examples than Qwen3Guard-Stream across all evaluated detection positions during token-by-token generation.Streaming evaluation measures timely detection at the unsafe-span endpoint and delayed detection within grace periods up to 256 tokens.
  • Efficiency: 14M trainable parameters make SIREN 250× smaller than Qwen3Guard-4B, while training on that backbone completes in 6 GPU hours.SIREN’s parameter efficiency comes from training lightweight components over internal representations rather than fine-tuning the full guard model.
  • Efficiency: SIREN’s inference cost is approximately 4× lower than safety-specialized guards because it uses one forward pass and avoids autoregressive generation.The guard-model comparison assumes perfect KV-cache utilization and only four generated tokens, making it a conservative lower bound for guard costs.

5 Discussion

The discussion examines SIREN’s design choices and internal safety encoding, finding that sparse neuron selection and adaptive cross-layer aggregation support strong, efficient detection. Cross-model ensembles further improve performance while retaining efficiency advantages over generative guards.

  • Neuron selection: 571 neurons, or 1.75% of 32,706 features, are selected for Llama3.2-1B at η=0.6 while maintaining strong performance.At η=0.9, selection rises to 4,214 neurons, or 12.9%; performance is stable across η ∈ [0.6, 0.9].
  • Aggregation strategy: Adaptive aggregation consistently outperforms uniform aggregation by approximately 1.0–1.3% across both backbones and all benchmarks.The adaptive layer weights are derived from validation performance of already-trained linear probes.
  • Aggregation strategy: The adaptive strategy adds no additional training cost because its layer weights come directly from validation performance of trained probes.This provides a zero-cost improvement over uniform aggregation.
  • Internal safety encoding: Middle-layer probes peak around 79% average F1 and outperform terminal-layer probes, indicating that internal states contain informative safety features.Individual layer probes reach within 4 points of fine-tuned guard models.
  • Internal safety encoding: SIREN’s cross-layer aggregation improves performance by 8 points over layer-wise probes, supporting richer multi-grained harmfulness representations.Variation in layer-wise probe performance also supports weighting stronger layers more heavily.
  • Cross-model ensemble: 87.7% average F1 is achieved by the best three-model ensemble, surpassing the best single SIREN at 86.7% by approximately 1 percentage point.Cross-architecture Qwen3 + Llama3 combinations tend to outperform same-family pairs, while ensemble inference doubles single-model SIREN cost but remains more efficient than a generative guard.

6 Conclusion

SIREN addresses the limitation of terminal-layer guard models by extracting safety-relevant internal neuron representations and aggregating them across layers. Across evaluation, it outperforms open-source guard models while generalizing to unseen settings and using few trainable parameters with improved inference efficiency.

  • Conclusion: Current guard models primarily use terminal-layer representations and overlook safety-relevant features distributed across internal layers.Generative guards also incur autoregressive inference costs.
  • Conclusion: SIREN identifies safety neurons through L1-regularized probing and combines them across layers with performance-weighted aggregation.The resulting framework is lightweight and plug-and-play.
  • Conclusion: SIREN consistently outperforms state-of-the-art open-source guard models while generalizing to unseen reasoning-trace datasets and streaming harmfulness detection.The evaluation also reports minimal trainable parameters and improved inference efficiency.

Limitations

The paper identifies architectural and task-scope boundaries for SIREN’s current safety-neuron selection approach.

  • Limitations: SIREN’s linear-probing-based neuron selection may require adaptation for architectures unlike standard transformers or concepts not linearly separable within individual layers.The current work also focuses on binary harmfulness classification rather than fine-grained safety taxonomies.

Ethics Consideration

The ethics discussion frames SIREN as a content-moderation tool while noting that its evaluation uses harmful-content datasets and that inherited model biases remain relevant.

  • Research intent and societal benefit: SIREN is intended to improve harmfulness detection in prompts and model responses and contribute to safer LLM deployment.The stated goal is responsible AI development and mitigation of risks from harmful AI-generated content.
  • Dataset contents: The study uses established safety benchmarks containing toxic language, hateful speech, and other potentially offensive material for training and evaluation.The datasets are used solely for safety-research purposes.
  • Dataset contents: Researchers working with harmful-content datasets are expected to maintain rigorous ethical standards and transparency.
  • Bias and fairness: SIREN may inherit biases from the LLM internal representations it uses, leaving bias characterization and mitigation as an important open problem.

A.1 Implementation Details

SIREN is deployed as a plug-and-play component over frozen LLM representations, using selected safety neurons and an MLP classifier. Training combines layer-wise probing, adaptive aggregation, and tuned classifier hyperparameters.

  • Representation extraction and deployment: SIREN attaches to a frozen base LLM through forward hooks and produces harmfulness scores from captured per-layer hidden states.Training-time safety-neuron indices and aggregation weights are reused during deployment.
  • Dataset preprocessing: Seven public safety datasets are split 80/20 for training and validation, with model-specific tokenization and no additional preprocessing.The datasets are ToxicChat, OpenAI Moderation, Aegis, Aegis-2.0, WildGuardMix, PKU-SafeRLHF, and BeaverTails.
  • Linear probing: Each layer receives an L1-regularized logistic-regression probe, and safety neurons are selected by absolute probe-weight magnitude.Regularization is selected by validation macro F1, with early stopping used for search and probe training.
  • MLP classifier training: The top MLP classifier is tuned with Optuna and cross-validation over depth, hidden dimensions, dropout, and learning rate.Trials use early stopping, and the final classifier trains to convergence with the selected hyperparameters.
  • Hyperparameters: The neuron-selection threshold η ranges from 0.6 to 0.9, retaining approximately 10–50% of neurons per layer while preserving discriminative capacity.Experiments use random seed 42, and Table 6 reports the principal training configurations.

B.1 Streaming Harmfulness Detection Details

SIREN extends sequence-level harmfulness detection to streaming by reapplying the same feature extractor and classifier to progressively longer prefixes. Its continuous scores support adjustable, position-aware thresholds, while streaming performance varies across backbone sizes.

  • Streaming pipeline: SIREN evaluates progressively longer generation prefixes to produce a harmfulness score at every token position.Prefix representations use the training-time pooling operator, safety-neuron selection, adaptive layer weights, and an unchanged classifier.
  • Evaluation protocol: Streaming evaluation reuses the trained feature extractor without updating the LLM, probes, or classifier, providing a zero-shot transfer test from sequence-level training.The protocol measures recall at the unsafe-region boundary and 32, 64, 128, and 256 tokens afterward.
  • Threshold flexibility: SIREN’s continuous harmfulness scores allow deployment-specific thresholds that can be relaxed during reasoning and tightened for final responses.This position-aware adjustment requires no additional training or architectural changes.
  • Token-level behavior: Token-level streaming outputs color-code each token by harmfulness level in Qwen3GuardTest examples.The visualization includes user input, reasoning, and response tokens for Qwen3-4B.
  • Scaling behavior: Smaller backbones tend to outperform larger ones in streaming detection, although the authors do not claim a definitive explanation.SIREN on Qwen3-0.6B outperforms Qwen3-4B, and a similar ordering is reported for Qwen3Guard-Stream.
  • Token-level transfer: The learned sparse safety-neuron representation transfers from sentence-level classification to token-level attribution without additional training or fine-tuning.Average-pooled per-token safety-neuron activations form the sentence-level representation used during training.

C Plug-and-Play SIREN on Guard Models

SIREN operates on general-purpose and safety-specialized guard models without modifying their underlying parameters. Its internal-representation design supports plug-and-play deployment and lightweight inference relative to autoregressive guard generation.

  • Guard-model transfer: SIREN maintains improvements across all benchmarks when trained on internal representations of guard models.For Qwen3Guard-4B, the reported average score increases from 83.4% to 87.6%.
  • FLOPs comparison: Safety-specialized guard inference computes the cost of generating K tokens with KV caching, including incremental attention and parameter-matrix multiplication terms.The analysis uses K = 4 tokens as a conservative lower bound for typical guard outputs and assumes 128-token inputs.
  • FLOPs notation: The FLOPs analysis defines L as transformer layers, S as input sequence length, D_h as hidden dimension, and N_params as total model parameters.The first term accounts for attention over previously generated tokens, while the second accounts for parameter matrix multiplications.
  • Inference overhead: SIREN attaches to hidden states already computed during base LLM inference and adds only an MLP forward pass plus negligible neuron-indexing and aggregation costs.The reported indexing and aggregation overhead is approximately 20K FLOPs.
Loading 2604.18519v1…