PaperlayerView exact revision on arXiv

Source-linked AI summary

A Safe Prototype Is Not a Safety Direction: Reference Dependence and Prompt Confounds in Response-Safety Embeddings

Sahil Kadadekar

arXiv:2610.01801v1cs.LGcs.CLcs.CR

TL;DR

Safe-only cosine to a response centroid is an appealing safety score, but positive examples do not identify which direction separates safe from unsafe responses. The paper audits the score under prompt-controlled comparisons and finds that an explicit reference ranks responses better, while prompt composition can affect uncontrolled evaluations.

  • Problem

    The open question is whether safe-only cosine prototypes identify safety direction when prompt composition can also influence aggregate AUC.

    Source: §1 · pp. 1–2

  • Method

    The study compares positive-centroid and referenced readouts across prompt-grouped corpora, four frozen encoders, and validation-calibrated operating points.

    Source: §4.2 · p. 5

  • Results

    Across 12 cells, prototype AUC was 0.457–0.545 on human-labeled corpora and 0.358–0.405 on jury control, versus 0.588–0.738 and 0.754–0.793 for difference-of-means.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Treat a class mean as a location, not necessarily a safety direction; declare a reference, control prompt composition, compare nuisance baselines, and evaluate a held-out operating point.

    Source: §8 · p. 8

    The conclusion concerns raw positive-centroid cosine and does not rule out other one-class methods or safety-specialized guards.

    Source: §7 · p. 8

Abstract

from arXiv · show

Can response safety be scored by cosine similarity to the mean embedding of known-safe responses? A recent sleeper-agent detector proposes exactly this score, yet the raw positive-centroid rule is not identified: positive observations locate the safe class relative to an encoder origin, but do not determine which direction separates safe from unsafe responses. We audit the rule on two prompt-controlled, human-labeled corpora and one auxiliary jury-labeled source control, using four frozen encoders and prompt-grouped splits. On the human-labeled corpora the safe prototype reaches ROC-AUC 0.457-0.545, with two cells significantly below chance and one above, while an explicit safe-minus-unsafe reference reaches 0.588-0.738 on the same embeddings; on the jury control the prototype is inverted (0.358-0.405) and the reference reaches 0.754-0.793. At validation-calibrated 5% false-safe thresholds, the reference accepts more safe responses on PKU-SafeRLHF (0.153-0.263 versus 0.039-0.061 across encoders) and Aegis (0.189-0.291 versus 0.004-0.045), but not reliably on BeaverTails. A fully unlabeled held-out reference recovers part to most of the referenced ranking, much less when only 5% of the pool is unsafe, whereas 80-634 labeled unsafe responses recover most of it. Prompt-only ablations show that prompt-label composition can inflate uncontrolled evaluations. This is a bounded result about a raw positive centroid, not all one-class methods or safety-specialized guards. A class mean is a location, not necessarily a safety direction; a declared reference with enough unsafe mass identifies orientation.

1 Introduction

A safe-response centroid is an appealing low-cost score, but safe examples alone do not identify which direction separates safe from unsafe responses. The paper audits this raw rule under prompt and label-source controls, finding stronger performance from an explicit reference and prompt-sensitive evaluations.

1 Introduction

  • The audit uses two human-labeled corpora, an auxiliary jury-labeled source control, four frozen encoders, and prompt-grouped splits.The primary human-labeled arms use prompt-matched comparisons; the Aegis arm fixes its label source to llm_jury.Source: §1 · p. 2
  • Across 12 corpus–encoder cells, the raw safe prototype reached AUC 0.457–0.545 on human-labeled corpora and 0.358–0.405 on the jury control, while the difference-of-means reference was above chance everywhere.The reference reached 0.588–0.738 on human-labeled corpora and 0.754–0.793 on the jury control.Source: §1 · p. 2
  • At validation-calibrated 5% and 10% false-safe targets, referencing raised safe acceptance for every encoder on PKU-SafeRLHF and Aegis, but gains on BeaverTails were small and uncertain.The Aegis arm is an auxiliary jury-label source control, not a human-labeled replication.Source: §1 · p. 2
  • Prompt-only prediction fell to near chance on matched cohorts, despite being strong on full-corpus populations, showing that prompt-label composition can inflate uncontrolled evaluations.The study distinguishes prompt-disjointness from prompt control.Source: §1 · p. 2
  • A held-out unlabeled reference recovered part to most of the referenced ranking, with little recovery at 5% unsafe prevalence, while a small labeled unsafe set from other prompts recovered most of it.The paper tests whether an unlabeled reference can replace unsafe labels in finite samples.Source: §1 · p. 2
  • The scope is a raw positive centroid scored by cosine, not all one-class estimators, deployable guards, or safety-specialized models.The paper also notes that Aegis jury labels are not a human replication.Source: §1 · pp. 2–3

2 Related work

Related work spans representation-level safety directions, benign-only detectors, one-class and positive–unlabeled learning, and embedding-geometry confounds. This paper distinguishes the unreferenced positive endpoint from methods that model a reference, class support, or both classes.

2 Related work

  • Prior safety-direction work analyzes or steers refusal representations, while RankOne and multi-layer prototype moderation use referenced readouts; this paper studies the positive endpoint alone.The paper distinguishes these readouts from instruction-tuned guards and safety-specialized encoders, which it does not evaluate.Source: §2 · p. 3
  • Benign-only monitors include safe-centroid cosine, batch-mean drift, and Gaussian or Mahalanobis scores, while contrastive scores model both classes.The paper limits its claim to cosine similarity to a single mean, not covariance or support estimation.Source: §2 · p. 3
  • Classical one-class methods estimate support or density, and positive–unlabeled learning uses an explicit unlabeled sample with assumptions linking it to positive and negative populations.The paper frames orientation as requiring a declared reference distribution.Source: §2 · p. 3
  • Prior work describes shared components and anisotropy in transformer representations, but anisotropy alone is not treated as a universal explanation of semantic failure.The paper measures class-mean quantities and separately tests whether prompt composition masquerades as response-level signal.Source: §2 · p. 3

3 Why a positive prototype is not identified

The expected score gap of a positive prototype depends on the unsafe mean as well as the safe mean, so safe observations alone cannot identify its sign. An explicit difference reference identifies mean orientation, and an unlabeled mixture can provide a collinear reference under stated assumptions.

3 Why a positive prototype is not identified

  • The positive-prototype score uses the normalized safe mean, but nothing forces its expected safe-minus-unsafe gap to be positive.The safe mean is defined from the safe embedding distribution.Source: §3 · p. 3
  • The same safe embedding distribution is compatible with unsafe distributions that give the positive prototype’s expected score gap opposite signs.The proof constructs unsafe means on either side of the safe mean along the prototype axis.Source: §3 · pp. 3–4
  • The difference score uses the safe-minus-unsafe mean direction and identifies mean orientation, but does not guarantee a particular AUC.The proof contrasts its expected gap with the positive-only gap.Source: §3 · p. 4
  • Under an in-domain two-class mixture with positive unsafe prevalence, subtracting the mixture mean from the safe mean is collinear with the class-difference direction in expectation.In finite samples, the reference signal scales with unsafe prevalence and competes with differences between the samples estimating the means.Source: §3 · p. 4

4 Audit design

The audit controls prompt dependence and label provenance through deduplication, prompt-grouped splits, and corpus-specific cohorts. It compares frozen-embedding readouts under cluster-based ranking and validation-calibrated operating-point evaluation.

4.1 Data, provenance, and dependence units

  • Prompt identity is canonicalized, exact prompt–response duplicates are collapsed, and contradictory-label duplicates are removed; splits keep each prompt together with zero exact train/test prompt overlap.BeaverTails retains mixed-label prompts, PKU-SafeRLHF retains discordant native two-response records, and Aegis uses the jury label source; Table 1 summarizes cohort counts and test-group units.Source: §4.1 · p. 4, Table 1 · p. 4
  • Released BeaverTails and PKU-SafeRLHF labels come from human-annotation pipelines, while the primary Aegis cohort uses only the released jury label source.The labels are reused rather than independently validated; Aegis records without usable binary labels or nonblank responses are excluded before deduplication.Source: §4.1 · p. 4, Table 1 · p. 4

4.2 Encoders, readouts, and statistics

  • Four frozen encoders comprise two sentence-transformer models and masked-mean-pooled hidden states from Qwen2.5-0.5B-Instruct and Qwen2.5-3B-Instruct.Sentence-transformer embeddings use standard normalized pooling; the Qwen arm uses final hidden states, L2 normalization, and 512-token truncation.Source: §4.2 · pp. 4–5
  • The main readout comparison is the normalized safe prototype f+ versus the difference-of-means reference f∆, alongside alternative embedding and stylistic baselines.Aegis RankOne is not interpreted because only six training prompts provide mixed-label directions.Source: §4.2 · p. 5
  • ROC-AUC is the primary ranking metric, with 95% intervals from 500 bootstrap draws that resample whole canonical-prompt clusters.Intervals condition on the fitted training direction; operating thresholds are calibrated on validation unsafe responses for 5% or 10% false-safe targets and evaluated with independent validation and test cluster resampling.Source: §4.2 · p. 5

5 Results

Across controlled cohorts, explicit safe-minus-unsafe references outperform the raw positive prototype. Operating-point, prompt-control, transfer, and finite-sample analyses qualify that result.

5.1 Controlled ranking

  • Prototype AUC was 0.457–0.545 on human-labeled corpora and 0.358–0.405 on Aegis; difference-of-means reached 0.588–0.622, 0.668–0.738, and 0.754–0.793, respectively.The prototype was weak on the human-labeled corpora and inverted on the auxiliary jury control.Source: §5.1 · p. 5, Figure 2 · p. 6

5.2 Validation-calibrated operating points

  • At 5% false-safe targets, difference-of-means safe acceptance was 0.153–0.263 versus 0.039–0.061 on PKU-SafeRLHF and 0.189–0.291 versus 0.004–0.045 on Aegis, while BeaverTails gains stayed small.BeaverTails safe acceptance was 0.067–0.095 for the reference versus 0.037–0.082 for the prototype.Source: §5.2 · pp. 6–7

5.3 Prompt composition, geometry, and transfer

  • Prompt-only readouts were 0.679–0.731 on full BeaverTails and 0.741–0.831 on full PKU-SafeRLHF populations, but fell near chance on matched cohorts.They reached 0.522–0.548 on matched BeaverTails and exactly 0.500 on matched PKU-SafeRLHF; Aegis was not prompt-matched and retained prompt-only scores of 0.621–0.649.Source: §5.3 · p. 7
  • Across 12 cells, safe and unsafe means aligned at cos(µs, µu)=0.825–0.999; the expected gap tracked prototype AUC (ρ=0.881, r=0.855), with sign agreement in 10 of 12 cells.These statistics are descriptive because cells share corpora and encoders; both mismatches had |∆+|≤0.005, so the identity is not an AUC claim.Source: §5.3 · p. 7
  • After prompt decontamination, safe-prototype transfer was weak and sign-unstable (AUC 0.253–0.544), while difference-of-means stayed above chance in 24 of 24 transfers (0.532–0.698).Difference-of-means transfer was weak when fit on Aegis.Source: §5.3 · p. 7

5.4 Does an unlabeled reference suffice?

  • A full unlabeled validation population reached 0.581–0.609 on BeaverTails, 0.623–0.668 on PKU-SafeRLHF, and 0.563–0.710 on Aegis; 5%-unsafe pools fell near chance.Labeled references with 105, 634, and 80 unsafe responses reached 0.571–0.612, 0.625–0.693, and 0.720–0.791, respectively; low-prevalence unlabeled references orient less reliably.Source: §5.4 · pp. 7–8

6 Implications for language-model security

A safety readout needs a declared reference and must be evaluated at operating points, not judged by AUC alone. The authors also recommend nuisance baselines and reject post-hoc sign flipping.

6 Implications for language-model security

  • A security readout should name the negative or unlabeled distribution that orients its score, with enough unsafe mass in an unlabeled reference to withstand sampling shift.Source: §6 · p. 8
  • At a 5% false-safe target, the referenced direction accepts at most 0.095 of safe BeaverTails responses, making these readouts diagnostics rather than filters.Above-chance ranking alone does not establish acceptable deployment behavior.Source: §6 · p. 8
  • Compare one-class readouts with response length and simple stylistic features; do not repair a corpus-dependent sign by selecting it after viewing test labels.A declared reference, rather than post-hoc sign choice, identifies orientation.Source: §6 · p. 8

7 Limitations and broader impact

The conclusions are bounded to a raw cosine-scored positive centroid and a specific set of datasets, encoders, and evaluation choices. The study also leaves uncertainty about refitting, deployment thresholds, and real-world effects.

7 Limitations and broader impact

  • The conclusions concern a raw positive cosine centroid, not density models, positive–unlabeled methods, safety-specialized encoders, or supervised guards.Evaluation covers English responses, two SBERT-family encoders, and one Qwen family at two scales; category-conditional performance and truncation-by-label effects were not audited.Source: §7 · p. 8
  • Bootstrap intervals hold the train-fitted direction fixed, and corpus-specific validation thresholds are not portable deployment thresholds.The 5% and 10% diagnostic targets remain too permissive for many high-stakes uses; the study measures ranking and acceptance errors, not behavior changes, attack robustness, or real-world harm.Source: §7 · p. 8
  • The referenced readout is recommended only as a diagnostic baseline because readers might mistake it for a sufficient guard.Source: §7 · p. 8

8 Conclusion

Across controlled cohorts and four encoders, the raw safe centroid is weak or inverted, while an explicit reference improves ranking. The conclusion calls for declared references, prompt controls, nuisance baselines, and held-out operating-point evaluation.

8 Conclusion

  • Across four encoders, the raw safe centroid is weak or inverted and below simple stylistic features, while an explicit reference improves ranking.Source: §8 · p. 8
  • On two of three corpora, referencing also improves calibrated safe acceptance, while an unlabeled reference helps only when it carries unsafe mass.Source: §8 · p. 8
  • The conclusion recommends declaring the reference, controlling prompt composition, comparing nuisance baselines, and evaluating a held-out operating point before calling an embedding score a safety direction.Source: §8 · p. 8

A Ethics, licenses, and reproducibility details

The paper avoids redistributing potentially distressing prompts and responses and documents licensing and reproducibility practices. Its artifact includes tested analysis scripts, but exploratory compute was not completely logged.

A Ethics, licenses, and reproducibility details

  • The artifact contains code, aggregate metrics, row-order hashes, and source manifests, but no prompts, responses, or model weights are redistributed.The public datasets may contain distressing harmful content; the study collected no new human-subject data.Source: A Ethics, licenses, and… · p. 11
  • Each schema-v3 result records dataset and model revisions, SHA-256 digests, dependency versions, commands, split salts, and bootstrap seeds.Thirty-three protocol tests cover core analysis steps, and tested scripts generate the analyses and appendix tables from the JSON files.Source: A Ethics, licenses, and… · p. 11
  • Cold embedding time and total exploratory compute were not completely logged, which the authors report as a reproducibility limitation.Source: A Ethics, licenses, and… · p. 11

B Readout and analysis details

Readouts are fit on the primary train cohort and evaluated on the same corpus’s primary test cohort, using unit-normalized embeddings. The section defines reference-based scores and nuisance baselines, then specifies held-out reference pools and validation-calibrated operating points.

B Readout and analysis details

  • All readouts are fit on the primary train cohort and scored on the same corpus’s primary test cohort; unit-normalized embeddings make dot products cosines.The train safe and unsafe means and pooled train mean define the class and mixture-centered scores.Source: B Readout and analysis details · p. 11
  • The mixture-centered safe mean uses the safe mean minus the pooled train mean, while RankOne averages per-prompt safe-minus-unsafe directions and logistic regression supplies a supervised baseline.In sample, the mixture-centered safe mean ranks identically to the difference of means.Source: B Readout and analysis details · p. 11
  • Scalar nuisance baselines orient log response length by its train-set class means and detect refusal templates in the first 120 response characters.The stylistic logistic-regression baseline adds standardized length, word, line, lexical, punctuation, digit, uppercase, list-formatting, question-mark, exclamation, and refusal features.Source: B Readout and analysis details · p. 11
  • Held-out references use prompt-disjoint validation responses, with unlabeled population pools or pools subsampled by unsafe prevalence, alongside a reference using labeled unsafe responses.Thresholds are set from validation unsafe scores so the validation false-safe rate does not exceed the target.Source: B Readout and analysis details · p. 11

C Per-cell results and statistics

The section reports controlled response-only ranking, operating-point, input-mode, transfer, refusal-sensitivity, and held-out-reference analyses. The supplied passages specify table scopes and identify a prompt-only ablation pattern and held-out reference conditions.

C Per-cell results and statistics

  • Table 4 reports primary controlled response-only ROC-AUC for each audited readout; the in-sample mixture-centered mean equals difference of means in every cell.Intervals are prompt-cluster bootstrap intervals conditional on the train fit; Aegis RankOne is not interpreted because it has only six mixed-label fit groups.Source: Table 4 · p. 12

D Operating points for all encoders

  • Table 6 reports validation-calibrated operating points for all four encoders, with intervals recalibrating thresholds across validation/test prompt-cluster draws.Intervals are conditional on a fixed train fit.Source: Table 6 · p. 13

E Input-mode ablation for all encoders

  • Prompt-only scores are at or near chance after matching, despite strong prompt-only signal on full-corpus populations.Table 7 covers input-mode ablations for all encoders; the Aegis controlled cohort is a jury source control, not prompt-matched.Source: Figure 3 · p. 13, Table 7 · p. 14

F Cross-corpus transfer for all encoders

  • Table 8 reports cross-corpus transfer between primary controlled cohorts after exact prompt decontamination for all four encoders.Its intervals are prompt-cluster intervals on the evaluation cohort.Source: Table 8 · p. 14

G Refusal-template sensitivity

  • Table 9 tests refusal-template sensitivity by removing refusal-opening responses from both fit and test before refitting.A refusal is defined by an explicit template match within the first 120 response characters.Source: Table 9 · p. 15

H Held-out unlabeled reference

  • Held-out references compare a prompt-disjoint validation population, a validation primary cohort, prevalence-subsampled pools, and a pool of labeled unsafe responses.The safe-mean shift between the primary train cohort and full validation population reflects sampling and composition differences, independent of the reference’s unsafe share.Source: Table 10 · p. 15, H Held-out unlabeled reference · p. 15

Related papers

Loading 2610.01801v1…