Source-linked AI summary
Cosine Similarity Is Not Evidence: Measuring the Noise Floor of Interpretability Transfer Under Quantization
Pranav Varshney
TL;DR
Similarity statistics used to assess interpretability transfer under quantization lack the quantities needed to interpret their estimator noise floors. The paper measures class separation and tests transfer against quantized-model nulls; it finds a direction deficit at INT4 but no detected movement at INT8. It also shows that scale-invariant statistics cannot distinguish translation from attenuation of a transferred decision variable.
Problem
Compression-transfer studies report similarity statistics without the sample size and geometry needed to interpret their noise floors, leaving claims of preservation unevaluable.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
A scale-invariant statistic cannot distinguish translation from attenuation, and no detected movement is not an equivalence claim.
Source: §5 · p. 4, §6 · p. 5The separation measurement covers one model, hidden size, and dataset; quantization is simulated and weight-only, while constructed perturbations establish detectability, not failure frequency.
Source: §7 · p. 5
Abstract
from arXiv · showhide
A statistic reported without the quantity needed to interpret it is not evidence. We develop that thesis for a concrete practice in AI safety. Interpretability artifacts are calibrated on full-precision weights, deployed on quantized ones, and certified as surviving the change by scale-invariant statistics (cosine similarity, correlation, AUROC) that are reported without their noise floor. For the difference-in-means direction estimator, the split-half floor is governed by one dimensionless number, $\kappa = n\rho^2/d$. The closed form $\mathbb{E}[\cos] \approx (1+4/\kappa)^{-1}$ is classical; the missing input is the class separation $\rho$, which we measure on real activations; no compression-transfer study we know of reports it. On Qwen2.5-1.5B-Instruct, $\rho = 33$--$61$ across depth, so two independent runs of the estimator agree to $0.978$--$0.994$ by sampling alone. A published cosine of $0.996$ between full-precision and quantized refusal directions therefore cannot be read as preservation without the $n$ it was computed at, which is not reported. Where $n$ is known, we judge each low-bit cosine against the split-half null measured within that quantized model, because a full-precision null assumes the low-bit estimator has the same variance. That assumption is exactly what a null exists to test. The result is plain: at INT4 the direction rotated, and the deficit exceeds the estimator's own noise. At INT8 we detect no movement, which is not an equivalence claim. We also show that a scale-invariant statistic cannot distinguish translation from attenuation of a transferred decision variable, although the two call for opposite remedies. We close with reporting recommendations that cost one forward pass. Code, data, and a one-cell reproduction are released at https://github.com/pvarshh/quantinterp
1 Introduction
Similarity between estimated quantities has a sampling-noise floor, so a high cosine alone cannot establish preservation. The paper develops this measurement-validity problem for interpretability under quantization and identifies the missing quantities and comparisons needed to evaluate transfer.
1 Introduction
- Compression-transfer cosines are unevaluable without the sample size and geometry that determine their estimator’s noise floor.The authors do not claim prior conclusions are wrong; they argue the reported numbers cannot be interpreted as evidence of preservation without those quantities.Source: §1 · p. 1
- Quantization matters for safety because read-mode probes monitor deployed models and control-mode steering vectors or feature clamps change their behavior.These artifacts are computed from particular weights, while deployment usually uses different weights; 4-bit serving is standard.Source: §1 · p. 1
- The difference-in-means direction estimator’s split-half floor is governed by κ = nρ^2/d, a vector-form correction-for-attenuation argument.The closed form is validated against Monte Carlo to 0.0015.Source: §1 · p. 2
- The paper measures class separation ρ on real activations in a form that, with n and d, places a published cosine against its own floor.The authors describe this as the first measurement they know of in compression-transfer literature in that form.Source: §1 · p. 2
- For cross-precision cosines, the paper proposes measuring the split-half null within each quantized model to distinguish rotation from a noisier estimator.A full-precision null would assume the quantized estimator has the same variance.Source: §1 · p. 2
- Translation and subspace attenuation have opposite signatures and remedies, but scale-invariant statistics distinguish neither.Source: §1 · p. 2
2 Related work
Prior interpretability-transfer studies measure different aspects of quantization, so their findings do not settle whether control artifacts transfer.
2 Related work
- Probe rankings can remain nearly unchanged under quantization while refusal behavior falls, and probes can be weak detectors of that damage.Duan [7] reports median ∆AUC −0.0021 to −0.0062 across 840 quantization cells; Kadadekar [10] reports refusal falling 12–68 percentage points under INT4 while output quality holds.Source: §2 · p. 2
- At the feature level, INT7 can improve perplexity while degrading 18.7% of active SAE features, showing an aggregate metric can miss artifact damage.Duan [8] reports this result; Gupte et al. [9] test causal SAE transfer under pruning only.Source: §2 · p. 2
- The paper distinguishes its full-dimensional ρ measurement from Braun et al.’s [2] d′-style separation along a fixed steering direction.It claims the full-dimensional measure together with n and d in the compression-transfer setting.Source: §2 · p. 2
- The paper’s cosine critique is distinct from embedding-geometry critiques [11] [16]: even with isotropic Gaussian activations, estimated directions need a noise floor for interpretation.Source: §2 · p. 2
3 The cosine floor
The expected cosine between two independently estimated directions depends on κ, not on n, d, or ρ separately. Simulation checks the analytic floor, and applying it to a published cosine requires assumptions about unreported quantities.
3 The cosine floor
- E[cos] ≈ (1 + 4/κ)^−1, with κ = nρ^2/d, gives the expected agreement of two independent difference-in-means estimates.The result is a function of κ and nothing else.Source: §3 · p. 2
- The estimator compares two disjoint-half estimates of the class-mean difference, whose true norm is δ and whose within-class scale is σ.Under the stated isotropic Gaussian setup, each estimate has noise variance 4σ^2/n.Source: §3 · p. 2
- The closed form is the vector form of classical correction-for-attenuation or split-half reliability; ρ is the missing input for placing a reported cosine against its floor.Source: §3 · p. 2
- Simulation across eleven settings and four hidden sizes gives maximum absolute error 0.0015, and matching κ values yield matching floors across different (n,d,ρ).At the paper’s operating points, the error is at most 0.0001.Source: §3 · p. 2
- Under the carry that holds per-dimension effect constant, the published cosine 0.996 is at or below its floor for n ≥ 512.The table’s two carries are not equivalent assumptions for moving measured ρ from d = 1536 to d = 4096.Source: Table 2 · p. 3
4 Measuring ρ
The paper measures class separation on real activations to quantify the estimator’s floor, then uses that measurement to assess transfer and propose reporting practices.
4 Measuring ρ
- Measured ρ is 33–61 on Qwen2.5-1.5B-Instruct, yielding a split-half floor of 0.978–0.994 at the study’s operating point.The measurements use d = 1536, n = 256 per class, AdvBench harmful prompts [17], Alpaca harmless instructions, and six depths; the estimator is bias-corrected.Source: §4 · p. 3
- The closed form matches the resampled floor within 0.0008 across six depths; at layer 16, κ = 627 and the estimator’s split-half floor is 0.9937.The authors note that this agreement is not a test of isotropy.Source: §4 · p. 3
- The measured ρ is extrapolated to d = 4096 under two different carries, and neither carry can be decided from one hidden size when n is unreported.Under the per-dimension-effect carry, the reported 0.996 is at or below its floor for n ≥ 512.Source: §4 · p. 3
- Report κ = nρ^2/d, or at minimum n and d, beside similarity statistics between estimated quantities.The paper says each recommendation costs at most one forward pass over the calibration set.Source: §4 · p. 4
- Estimate the null inside the compressed model rather than at full precision, since the latter assumes both estimators have the same variance.Source: §4 · p. 4
- Report “no detected movement,” not “preservation,” unless an equivalence test against a preregistered margin is run.For scale-sensitive diagnosis, regress compressed decision values on full-precision values to distinguish translation from attenuation before recalibrating.Source: §4 · p. 4
5 Testing the refusal direction against its own null
Against each bit-width’s own split-half null, INT4 shows a refusal-direction rotation beyond its estimator’s noise, while INT8 shows no detected movement. The comparison uses a conservative rule whose detectable margin is limited.
5 Testing the refusal direction against its own null
- Quantization-induced prompt-to-prompt activation variation adds estimation error, so the true no-rotation null lies between 1 and the split-half floor.The INT8 result cannot locate that null for INT4 because its weight error is 17 times smaller.Source: §5 · p. 4
- Each low-bit cosine is compared with a split-half null measured within that quantized model, absorbing variance added by the perturbation.The FP16 null uses the same partitions, so the difference between nulls reflects activation changes rather than different draws.Source: §5 · p. 4
- The rule cos < own null−2sd is conservative, not calibrated; its margin uses split-to-split spread because the splits share data.A calibrated test would require modeling quantization noise in the paired-cosine null; smaller rotations may remain undetected.Source: §5 · pp. 4–5, Table 3 · p. 5
- At INT4, the cosine 0.9647 falls below its own null minus twice the spread (0.9803 − 2(0.0052)), detecting rotation beyond that model’s estimation noise.At INT8, no movement is detected.Source: §5 · p. 4, Table 3 · p. 5
6 Translation and attenuation are not the same failure
AUROC can miss score translation and cannot identify attenuation, although the two failure modes have different effects on accuracy and different remedies.
6 Translation and attenuation are not the same failure
- Under pure translation, ΔAUROC is 0.0000 while fixed-threshold accuracy falls from 0.735 to 0.550.Translation preserves ranking, so AUROC alone can label a monitor healthy while it operates near chance.Source: §6 · p. 5
- Attenuation lowers effective SNR and can move AUROC by up to 0.13, but AUROC does not identify which failure occurred.The paper constructs translation and attenuation with known ground truth (n = 1200, d = 64).Source: §6 · p. 5
- Refitting only the decision threshold recovers 91% of accuracy lost to large translation but 17–37% of that lost to attenuation.Distinguishing the modes requires regressing quantized decision values on full-precision ones.Source: §6 · p. 5
7 Discussion and limitations
The discussion withholds refusal-operator outcome rates because its substring classifier has unmeasured agreement with human judgments, and it limits the empirical scope of its measurements. The floor-formula validation reports close simulation agreement, while the extrapolation to a published cosine remains conditional on how class separation carries across model widths.
7 Discussion and limitations
- Outcome rates from 45,000 scored completions are withheld because the substring refusal classifier has unmeasured agreement with human judgment.The authors say a few-point effect from an instrument of unknown bias is not a finding.Source: §7 · p. 5
- The measured ρ covers one model, hidden size, and dataset; quantization is simulated and weight-only, while constructed perturbations establish detectability, not failure frequency.The discussion also reports a data defect concealed by an aggregate and a mechanism confirmed using an average over unlike conditions.Source: §7 · p. 5
A Validation of the floor formula
- Across the eleven-setting simulation grid, matching κ values yield matching floors despite different n, d, and ρ; maximum absolute error is 0.0015.The last two table rows are the paper’s operating points.Source: A Validation of the floor formula · p. 6, Table 4 · p. 7
- At d = 4096, carry (B) places the floor above the reported cosine 0.996 in the shaded right region, but the cosine’s n is unreported.The two carries bracket the floor, leaving the interpretation conditional on the unmeasured ρ carry.Source: Figure 1 · p. 7
B The bias-corrected ρ estimator
The section corrects the separation estimate for sampling noise and examines how threshold refitting fares under translation and attenuation. Translation-related accuracy loss is recovered more fully than attenuation-related loss.
B The bias-corrected ρ estimator
- At d=2048 and n=256 per class, correction estimates ρ as 2.99 (se 0.007) at true 3.0 and 0.44 (se 0.022) at true 0.5.The corrected squared estimate is clamped at zero when noise subtraction is negative; this occurs in 31% of replicates at ρ=0.5, biasing estimates low.Source: B The bias-corrected ρ estimator · p. 7
- Translation leaves AUROC unchanged while calibrated-threshold accuracy falls to 0.55; attenuation moves AUROC by up to 0.13.The regression slope of quantized on full-precision decision values detects but understates subspace attenuation; for injected g=0.6, the slope is 0.686.Source: Table 5 · p. 8
C Translation vs. attenuation: full grid
- Threshold-only refitting recovers 91.0% of accuracy lost to large translation, 17.2% and 37.0% lost to attenuation, and 42.9% under combined perturbations.Recovery is reported only where the accuracy gap exceeds 0.05; the denominator is the gap to a probe retrained at that perturbation.Source: C Translation vs. attenuation: full grid · p. 8
D The deferred causal-transfer arm
The causal-transfer arm ran 45,000 scored completions, but withholds rates because the refusal classifier’s agreement with human judgment is unmeasured. It reports two qualitative observations and specifies validation needed before releasing rates.
D The deferred causal-transfer arm
- The 45,000 completions are classified by substring matching, whose agreement with human judgment is unmeasured, making few-point additive effects difficult to interpret.The additive effects are a few points over a low baseline, within the range an unknown classifier bias could manufacture.Source: D The deferred causal-transfer arm · p. 8
- Directional ablation suppresses refusal on harmful prompts by a large fraction of available headroom at every tested bit-width.The direction is fit on prompts disjoint from evaluation, and effects are headroom-normalized because quantization moves the baseline.Source: D The deferred causal-transfer arm · pp. 8–9
- At c=1, add_relative is the strongest tested operator; at c=4, refusal is exactly zero across every bit-width and seed, while add_absolute is monotone.Output quality was not measured, so the zero rate is also consistent with degenerate generation; neither observation licenses a bit-width comparison.Source: D The deferred causal-transfer arm · pp. 8–9
- Releasing rates requires 200 blind hand labels with a weighted agreement coefficient, per-example scores for paired tests, and a preregistered equivalence margin with TOST.The hand labels are the only human-time item; without TOST, this arm cannot support a preservation claim.Source: D The deferred causal-transfer arm · p. 9
E Self-audit: errors made and what caught them
The self-audit identifies malformed harmless prompts and an average across unlike conditions as sources of misleading findings. Both errors altered how the results and their explanations were interpreted.
E Self-audit: errors made and what caught them
- Malformed Alpaca prompts omitted the passage to summarize, yielding a harmless-prompt refusal baseline of 0.375; repairing the loader lowered it to 0.05.The harmful side was clean: all 64 of 64 flagged responses were unambiguous refusals.Source: E Self-audit: errors made and what… · p. 9
- Contamination lowered ρ at three early layers but raised it at three late layers, refuting the prediction that heterogeneous harmless prompts would make measured ρ a lower bound.Early-layer values changed 40.9→34.3, 38.8→33.6, and 36.3→32.9; late-layer values changed 60.1→61.3, 54.9→59.0, and 52.5→56.1.Source: E Self-audit: errors made and what… · p. 9
- Averaging add_relative over c∈{1,2,4} concealed distinct behaviors and produced a negative result that appeared to support an over-injection mechanism.The operator was strongest at c=1 and exactly zero at c=4; the mechanism was plausible, but the averaged evidence did not establish it.Source: E Self-audit: errors made and what… · p. 9
F Ethics and reproducibility
The paper measures transfer of an existing interpretability intervention rather than introducing a capability. It uses public prompts and provides reproducibility materials, while withholding steering generations for a stated methodological reason.
F Ethics and reproducibility
- The contribution is measuring whether an FP16-derived refusal direction still works after quantization, using Arditi et al.’s published directional ablation rather than introducing a capability.Prompts come from public AdvBench and Alpaca datasets; no new harmful prompts or jailbreaks are published.Source: F Ethics and reproducibility · p. 9
- Released result files contain activation summary statistics but no generations; steering-run completions are withheld for the methodological reason given in Appendix D.Source: F Ethics and reproducibility · p. 10
- All results reproduce from one self-contained notebook, with manifests recording the commit, package versions, seed, and resolved dataset source.Tables 4 and 5 regenerate without a GPU or dataset; Tables 1 and 2 use one measure_rho run, and Table 3 adds one cell in the same session.Source: F Ethics and reproducibility · p. 10