PaperlayerView exact revision on arXiv

Source-linked AI summary

Trust The Typical

Debargha Ganguly, Sreehari Sankar, Biyao Zhang, Vikash Singh, Kanan Gupta, Harshini Kavuru, Alan Luo, Weicong Chen, Warren Morningstar, Raghu Machiraju, Vipin Chaudhary

arXiv:2602.04581v1cs.CLcs.AIcs.DCcs.LG

TL;DR

Current LLM safety methods rely on reactive detection of known harmful patterns, leaving novel attacks difficult to anticipate. T3 instead models the distribution of safe language use and detects deviations, achieving strong benchmark performance while substantially reducing false positives, with effectiveness contingent on appropriate safe training data.

  • Problem

    Current safety mechanisms defend against explicitly known attack patterns, requiring continually expanded harmful-pattern catalogs and unable to anticipate novel attacks.

  • Method

    T3 learns the geometric structure of safe language use and treats potentially harmful prompts as out-of-distribution deviations from a distribution of acceptable usage.

  • Results

    Across 18 benchmarks, T3 achieves state-of-the-art AUROC with a 10-40x reduction in false positive rates versus specialized safety models, including strong detection of HILL jailbreaks at AUROC 0.98 and FPR@95 4.4%.

  • Takeaways & Limitations

    Modeling safe usage supports generalization across domains and languages, reduced overrefusal, and practical real-time deployment through integration with vLLM.

  • Takeaways & Limitations

    On the intentionally difficult Anthropic hh-rlhf benchmark, where in-distribution safe responses already contain profanity and differ subtly from harmful responses, all methods perform near random chance.

Abstract

from arXiv · show

Current approaches to LLM safety fundamentally rely on a brittle cat-and-mouse game of identifying and blocking known threats via guardrails. We argue for a fresh approach: robust safety comes not from enumerating what is harmful, but from deeply understanding what is safe. We introduce Trust The Typical (T3), a framework that operationalizes this principle by treating safety as an out-of-distribution (OOD) detection problem. T3 learns the distribution of acceptable prompts in a semantic space and flags any significant deviation as a potential threat. Unlike prior methods, it requires no training on harmful examples, yet achieves state-of-the-art performance across 18 benchmarks spanning toxicity, hate speech, jailbreaking, multilingual harms, and over-refusal, reducing false positive rates by up to 40x relative to specialized safety models. A single model trained only on safe English text transfers effectively to diverse domains and over 14 languages without retraining. Finally, we demonstrate production readiness by integrating a GPU-optimized version into vLLM, enabling continuous guardrailing during token generation with less than 6% overhead even under dense evaluation intervals on large-scale workloads.

1 INTRODUCTION

T3 reframes LLM safety as proactive detection of deviations from the distribution of safe language use rather than reactive matching of known harmful patterns. Across broad safety evaluations, it reports strong detection, low false-positive rates, transfer across domains and languages, and low-overhead streaming deployment.

  • Current safety classifiers favor attackers because defenders must continually enumerate evolving harmful patterns while attackers can discover novel prompt structures.
  • T3 models the concentrated semantic distribution of legitimate interactions and flags atypical representations as potential threats.
  • T3 avoids exhaustive harmful-example collection by characterizing acceptable usage without assuming a specific adversarial input form.
  • Across 18 benchmarks, T3 achieves state-of-the-art AUROC while reducing false positive rates by 10-40x versus specialized safety models.
  • T3-OCSVM reports FPR@95 of 2.0% on OffensEval and 3.5% on Davidson, versus 75.2% and 61.7% for the best baseline, and reduces OR-Bench overrefusals by 75%.
  • A model trained only on English safe text reaches 99.6% AUROC on code, 99.8% on HR, and less than 2% variance across 14+ languages.
  • Integrated into vLLM, T3 supports continuous monitoring with sub-6% overhead at 20-token evaluation intervals on 5,000-prompt workloads.

2 RELATED WORKS

The related work connects LLM safety with OOD detection and embedding geometry. T3 combines these ideas into safe-distribution modeling while addressing computational, fine-tuning, and generalization trade-offs in prior approaches.

  • OOD detection is important because models can produce confident but incorrect predictions on novel inputs, including adversarial perturbations.
  • LLM embeddings exhibit isotropy and clean domain-level clusters, making simple distance metrics useful for OOD detection while fine-tuning can fragment those structures.
  • Likelihood-based OOD methods are computationally prohibitive, while representation-based methods face limitations associated with fine-tuning and embedding geometry.
  • Adversarial prompts such as jailbreaks, prompt injections, and role-playing exploits are treated as OOD because they deviate from natural-language usage.
  • T3 learns safe usage directly from curated examples, avoiding dual-model likelihood costs and preserving clean geometric structure for safety detection.

3 METHODOLOGY

T3 formulates harmful-prompt detection as distinguishing safe in-distribution samples from unknown harmful out-of-distribution samples. It uses normalized multi-encoder embeddings, geometric PRDC features, and safe-only density models to produce anomaly scores.

  • Problem Formulation: T3 determines whether prompts or outputs belong to the safe distribution or an unknown harmful distribution before further processing.
  • Embedding Representation: Three sentence transformers generate normalized embeddings, enabling cosine-based comparisons while mitigating encoder-specific scaling artifacts.
  • Geometric Features: For each encoder, T3 computes four geometric features describing relationships between test points and reference distributions through k-nearest-neighbor structure.
  • Geometric Features: PRDC metrics have analytically characterized expectations under matching distributions and can distinguish alternatives in several regimes.
  • Theoretical Properties: Under partial support mismatch, harmful prompts outside safe support yield asymptotic expected precision of 1−α.
  • Theoretical Properties: The asymptotic coverage bound is 1 − E_y∼D_harmful[e^−λk r(y)] < 1 − e^−λk, with λ defined as the reference-to-test sample-size ratio.
  • Anomaly Scoring: T3 aggregates PRDC features across encoders and models their safe-data distribution with either a BIC-selected GMM or an RBF-kernel OCSVM.
  • Anomaly Scoring: The resulting anomaly score is the normalized negative log-likelihood under the fitted safe-data model.

4 RESULTS

T3 is evaluated across diverse safety, adversarial, overrefusal, domain, multilingual, and deployment settings, using safe-only in-distribution data and semantic OOD detection. It generally reduces false alarms while maintaining strong detection and transfers across attacks, domains, and languages without retraining.

  • Evaluation setup: T3 is evaluated on 12 OOD benchmarks covering toxicity, hate speech, multilingual harms, and domain-specific policy violations, alongside specialized safety and representation-based OOD baselines.The broader evaluation also includes adversarial, jailbreaking, and overrefusal experiments.
  • Toxicity and hate speech: T3-OCSVM achieves FPR@95 values of 2.0% on OffensEval and 3.5% on Davidson, versus 75.2% and 61.7% for the best baselines.Across six toxicity and hate speech benchmarks, it also reaches AUROC ≥0.96 on five of six benchmarks.
  • Adversarial and jailbreak detection: On six adversarial and jailbreaking benchmarks, T3 reduces AdvBench FPR@95 to 15.8%, a 4.2× improvement over PolyGuard, despite training only on safe data.Traditional OOD methods typically exceed 97% FPR@95, while PolyGuard flags over 64% of safe prompts on every benchmark.
  • Overrefusal and cold start: On OR-Bench, T3-GMM achieves 22.2% FPR@95 and T3-OCSVM achieves AUROC 0.934, while LLM-generated safety augmentation decreases performance.OR-Bench measures overrefusal on safe-but-challenging prompts; the augmentation may shift borderline-safe prompts toward the harmful embedding distribution.
  • Domain and multilingual transfer: A single model trained on general-purpose English safe text achieves AUROC above 99.5% and FPR@95 below 1% on Code and HR policy violations without domain-specific training.Across 14+ languages, T3-OCSVM shows less than 0.6% AUROC variance, contrasting with baseline variance up to 28%.
  • Deployment efficiency: Continuous vLLM monitoring every 20 tokens incurs 1.5% overhead on 500-prompt workloads and 6% on 5,000-prompt workloads.These measurements were obtained on an NVIDIA H200 GPU with batches of 32 requests.

5 DISCUSSION

T3’s effectiveness depends on how well safe and harmful examples are separated in the reference distribution. It detects semantically disguised attacks when safe data are properly curated, but struggles when harmful patterns already occur among safe examples.

  • AUROC≈0.5 on Anthropic hh-rlhf shows that all evaluated methods perform near random chance when safe and harmful responses are semantically difficult to distinguish.Chosen responses contain toxic content, producing cosine similarity above 0.95 with rejected responses.
  • T3 fails when the in-distribution safe set already contains the harmful patterns it is intended to detect.This makes the typical safe region overlap with harmful usage, undermining OOD separation.
  • AUROC 0.98 and FPR@95 4.4% show strong HILL jailbreak detection with properly curated safe data.HILL disguises harmful imperatives as innocuous learning-style questions.
  • OOD-based safety succeeds on separable manifolds such as HILL and domain adherence but fails when safe and harmful content overlap, as in HH-RLHF.The discussion identifies appropriate in-distribution data curation as a central condition for T3’s effectiveness.

6 CONCLUSION

T3 reframes LLM safety as proactive statistical typicality, while its PRDC-based analysis characterizes how reference and test distributions relate. In the in-distribution limit, the estimated safe manifold converges to the true support and contains new samples with probability approaching 1.

  • T3 moves LLM safety from reactive threat-blocking to proactive modeling of statistical typicality.The framework models safe usage rather than enumerating harmful patterns.
  • PRDC metrics are analyzed for their expected behavior when reference and test samples come from the same or different distributions.The theoretical treatment covers general settings and includes the in-distribution case as a special case.
  • The limiting PRDC analysis provides intuition even where finite-sample expectations remain analytically intractable.Some expressions cannot be simplified further without stronger distributional assumptions.
  • Under standard regularity conditions, the estimated manifold S_m(X) converges to the support of F as the reference sample size m grows.The assumptions include compact support, a bounded density bounded away from zero, and a measure-zero support boundary.
  • As m grows, an in-distribution sample Y_j drawn from F eventually falls inside S_m(X) with probability approaching 1.This establishes the asymptotic behavior of the estimated safe manifold for new samples from the same distribution.

A.2 CONSISTENCY

The analysis establishes when per-point PRDC metrics distinguish in-distribution from out-of-distribution samples. Coverage is maximized when F = G and separates under both support mismatch and smooth density shift.

  • A statistical test is consistent when its probability of distinguishing the null from any alternative converges to 1 as sample size increases.
  • Partial support mismatch: Under partial support mismatch, coverage is strictly lower than in the in-distribution case whenever G assigns positive mass outside supp(F).
  • Same support, different densities: When F = G, the density ratio satisfies r ≡ 1, yielding the reference coverage value 1 − e^−λk.
  • Same support, different densities: When F ≠ G, strict convexity reasoning makes coverage smaller than its F = G value, even when the distributions have identical supports.
  • Different densities in a small region: A positive-mass region where F has density at most (1 − η) times G produces a strict, quantifiable coverage decrease.
  • Overall, precision separates under support mismatch, while coverage separates under both partial support mismatch and common-support covariate shift.

A.3 CONNECTION WITH TWO-SAMPLE TESTS

The paper places PRDC metrics and T3 alongside classical k-nearest-neighbor two-sample tests, showing shared distributional information but important differences in construction and scope. Schilling’s statistic is consistent because differing distributions reduce cross-sample mixing.

  • Classical pooled-graph tests determine whether two sample sets share a distribution, whereas T3 evaluates whether each individual test sample matches a reference distribution.
  • Schilling’s Tk,N statistic measures the proportion of pooled k-nearest-neighbor comparisons whose endpoints share the same original sample label.
  • PRDC computes within-sample neighborhoods and is asymmetric in reference and test sets, while Schilling’s test uses neighborhoods in the pooled sample.
  • The reported computational costs are O(m^2 + mdmaxK) for preprocessing, O(n(m + dmaxK)) per inference batch, and O(m · dmaxK) memory.
  • Under different distributions, insufficient mixing increases Tk,N relative to the null, supporting consistency of Schilling’s test.
  • The PRDC metrics and Schilling’s statistic capture some overlapping information, although the methods are not generally equivalent.

B EXPERIMENT TECHNIQUE DETAILS

The experiments adapt representation-based OOD detectors and safety APIs to a common text-embedding evaluation framework. T3 uses multi-view sentence-transformer representations, while baselines receive method-specific adaptations for binary safety scoring.

  • The evaluation setup prioritizes reproducibility by documenting datasets, models, hyperparameters, and experimental settings.
  • Text OOD methods were adapted to 1024-dimensional Qwen3-Embedding-0.6B embeddings, using an auxiliary binary classifier when vision-specific components were unavailable.
  • Baseline detectors were modified with method-specific procedures, including pseudo-gradients, exact cosine k-NN, pseudo-binary statistics, and embedding-space thresholding.
  • Safety APIs and judge models were converted into standardized numerical safety scores for comparison with OOD metrics.
  • T3 uses a multi-view representation from Qwen3-Embedding-0.6B, BAAI/bge-m3, and intfloat/e5-large-v2.
  • Forte supports zero-shot deployment through self-supervised embeddings, local geometric descriptors, and no additional detector-training stage.

C.3.1 DATASETS

The experiments evaluate over-refusal using safe prompts as in-distribution data and toxic prompts as out-of-distribution data. The setup includes separate training and testing splits and a shared T3 representation configuration.

  • The over-refusal evaluation uses bench-llm/or-bench, designed to measure responses to safe-but-challenging prompts.
  • The in-distribution pool contains 5,000 safe prompts from the or-bench-80k and or-bench-hard-1k subsets.
  • The safe-prompt pool was split into 3,500 training prompts and 1,500 testing prompts.
  • The out-of-distribution set contains 600 toxic prompts from or-bench-toxic, intended to be correctly refused by safety models.
  • T3 uses a multi-view representation from Qwen3-Embedding-0.6B, BAAI/bge-m3, and intfloat/e5-large-v2 across these evaluations.
  • The adversarial and jailbreaking experiments use the same safe id_mix corpus for in-distribution data and benchmark prompts for out-of-distribution data.

C.5 T3/FORTE ALGORITHMIC ABLATIONS

The ablations show that combining all four PRDC metrics gives the most robust detection, while covariance-based ellipsoids are impractical in high-dimensional spaces. T3 remains effective across different sentence-transformer embedding models.

  • FPR@95 falls from approximately 20–35% with T3-PD to approximately 0.7–4% with T3-RC and approximately 1–2% with T3-Full.T3-Full combines Precision, Recall, Density, and Coverage; the full set captures distinct geometric failure modes.
  • Ellipsoidal regions became unstable in high-dimensional embeddings and required prohibitively expensive per-point covariance estimation.Their performance approached random chance, making the method impractical for large datasets.
  • T3 performance remained strong across different sentence-transformer embedding models, suggesting generalization across NLP embedding spaces.The observation is heuristic and consistent with findings from the original Forte work in computer vision.

D INTEGRATING T3 WITH VLLM FOR ONLINE GENERATION GUARDRAILING

T3 is embedded directly into vLLM to monitor safety continuously during token generation while preserving streaming and throughput. Its main-process safety computations overlap with worker-process inference, keeping runtime overhead low.

  • Integration strategy: T3 is the first framework described here to integrate guardrails into online LLM generation, with negligible overhead under dense evaluation frequencies.The integration targets low latency, streaming support, and continuous monitoring on an NVIDIA H200 GPU.
  • Why vLLM: The vLLM target provides scalable KV-cache management, continuous batching, modularity, and multiprocess execution suited to high-throughput guardrailing.PagedAttention supports KV-cache scaling, while continuous batching improves utilization across heterogeneous workloads.
  • Architecture: The Main Process orchestrates requests while the Engine Core schedules batches and allocates computational resources.Communication between the application entry point and Engine Core uses ZeroMQ IPC for asynchronous scheduling and fault isolation.
  • Integration strategy: Direct pipeline embedding avoids external-service serialization, IPC, and network costs while preserving token-by-token streaming.The design reduces evaluation latency to the sub-millisecond range and avoids buffering-based responses.
  • Implementation: Patching process_outputs localizes safety logic at one interception point, avoiding source changes and enabling dynamic guardrail control.The patch is intended to contain performance regressions and simplify debugging and iteration.
  • Implementation: The three-phase process synchronizes request state, accumulates generated text, and applies T3 decisions within standard output handling.The supplied implementation excerpts show text metadata tracking, batch-size checks, toxic-request abortion, and final output construction.
  • Scheduling: T3 evaluations are scheduled around word-count thresholds, with expanded candidate selection or deferred and reduced-set fallbacks to balance responsiveness and efficiency.Concurrent Main Process evaluation overlaps with Worker Process inference through shared-accelerator execution strategies.
  • Runtime evaluation: 1.5% overhead occurs on 500 prompts and 6% overhead on 5,000 prompts under dense monitoring, compared with baseline vLLM.The larger workload takes 40.292 seconds with guardrails versus 38.011 seconds for baseline vLLM.

E.2 EMBEDDING MODEL ABLATION

Embedding ablations indicate that T3’s advantage comes from its manifold-based methodology rather than embedding size or a particular backbone. Text-native OOD baselines retain high false-positive rates, while T3 detects semantically disguised HILL attacks effectively.

  • Embedding model ablation: T3 maintains its performance advantage over baseline OOD methods even when those methods use 4B- and 8B-parameter embedding models.The ablations attribute the gap to T3’s methodology rather than the embedding backbone.
  • Embedding model ablation: With 8× more embedding parameters, baseline methods still exceed 90% FPR@95 on most benchmarks, while T3 remains below 45%.
  • Text-native baselines: Text-native Energy, kNN, and Mahalanobis methods are trained on ID-only data and evaluated on the same splits as T3.Energy uses classifier logits, Mahalanobis uses feature-space distances, and kNN uses embedding-space distances.
  • Text-native baselines: T3 achieves FPR@95 of 1–5% on toxicity and jailbreaking benchmarks, versus typically above 80% for text-native baselines and approximately 95–100% on jailbreaking.kNN reaches approximately 0.80–0.84 AUROC on toxicity benchmarks but retains high false-positive rates.
  • HILL jailbreaks: Despite semantic similarity to benign educational queries, T3 detects HILL jailbreaks with AUROC >0.98 and FPR@95 of 4.35%.The evaluation uses 1,500 safe Dolly prompts and 46 HILL jailbreak prompts.
Loading 2602.04581v1…