PaperlayerView exact revision on arXiv

Source-linked AI summary

What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models

Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth

arXiv:2608.16852v1cs.AI

TL;DR

Compliance detectors may rely on scenario surface features rather than the governing rule, undermining whether their verdicts measure rule compliance. The paper introduces counterfactual audits and a training-free activation readout, finding rule blindness across tested guards and probes while ICS remains a broad risk signal rather than rule-specific adjudication.

  • Problem

    It is unclear whether compliance detectors use the stated rule or merely recognize scenarios that resemble violations, and conventional accuracy cannot distinguish these possibilities.

  • Method

    The paper audits guards and activation probes with rule and scenario counterfactuals, and introduces ICS, a training-free activation readout calibrated from labelled pairs.

  • Results

    Deleting, permuting, or substituting rules leaves every tested guard and probe essentially unchanged, while a crossed-rule benchmark confirms rule blindness.

  • Takeaways & Limitations

    ICS should be interpreted as a broad compliance-risk direction, not a rule-specific adjudication signal, and the released benchmark enables future rule-use testing.

  • Takeaways & Limitations

    ICS requires first-party activation access, transfers weakly across compliance and safety data, is serialization-sensitive, and is defeated by an adaptive white-box attack.

Abstract

from arXiv · show

Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy. Such monitoring is meaningful only if a detector's verdict depends on the stated rule rather than on surface features of the scenario. We show this condition fails across the current class of compliance detectors, a failure we call rule blindness. Deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard and activation probe we test, including a policy-conditioned guard that correctly cites the governing clause yet barely changes its verdict when that clause is swapped for its permissive counterpart. A purpose-built benchmark crossing two rules with two scenarios, so that neither alone predicts the label, confirms the failure under a design no prior benchmark rules out, and shows that step by step reasoning, not any fast detector we test, is what escapes it. Auditing at scale requires a retraining-free detector, so we introduce the Internal Compliance Score (ICS): a training-free activation readout calibrated from ten labelled pairs and scored by a single projection. We hold ICS to the same scrutiny as the guards it audits: a pre-registered criterion for beating trivial baselines is not met, and a bag-of-words model matches its pooled generalisation exactly. It remains useful because it is inexpensive, letting us audit four deployed guard models, an 8B zero-shot judge, and thirteen benchmarks, and it raises the mechanically verified pass rate when used to rank candidate responses, though an adaptive white-box attack removes this gain. We release the counterfactual protocol and crossed-rule benchmark so rule blindness can be tested in future probe and guard claims.

1 Introduction

The paper identifies rule blindness: compliance detectors can leave their verdicts unchanged when the governing rule is deleted, permuted, or substituted, across regulatory domains and detector types. It introduces the training-free Internal Compliance Score (ICS) to enable inexpensive, repeated auditing while explicitly treating its contribution as structural rather than superior accuracy.

  • Scope: The evaluated regulatory domains span data protection, healthcare privacy, financial regulation, and platform content policy, including twenty domains.This breadth indicates the reported failure is not confined to one jurisdiction or industry.
  • Rule blindness: Rule blindness leaves detection accuracy unchanged when the governing rule is deleted, permuted, or substituted across tested guards and an activation probe.The failure affects both fixed-taxonomy classifiers that never receive the rule and other detectors for different reasons.
  • Contribution and limitation: ICS is the only compared method that is training-free, reads the monitored model directly, and recalibrates to a changed model at negligible cost, although it remains rule-blind.Its claim is structural rather than one of superior accuracy, and it is evaluated against a budget-matched selection null and pre-registered floor criterion.
  • Method: ICS is a training-free activation readout that uses class-mean differences from a small labelled calibration set, requires no gradient step, and uses one projection per case.It scores compliance risk directly from the monitored model.

2 Method: The Internal Compliance Score

The Internal Compliance Score (ICS) is a training-free activation projection calibrated from labelled adherent and violating pairs, then thresholded or ranked at low inference cost. The method explicitly controls sign conventions, norm confounding, disjoint calibration/test splits, and outcome-ablated evaluation against trivial baselines.

  • Score construction: One forward pass, one activation extraction, and one dot product score each new case after offline calibration.Calibration costs 2n forward passes, while the decision stage thresholds, ranks, or routes on the resulting score.
  • Score construction: ICS estimates an adherence direction from class-mean differences and scores each text by its activation projection onto that normalized direction.Higher ICS denotes greater adherence because the direction points from the violating prototype to the adherent prototype.
  • Score construction: 0.566 is the length-only score, exposing norm as a live confound and motivating a cosine variant that ties the raw score while norm alone remains near chance.The raw projection combines activation angle and norm; the method uses no optimizer or hyperparameter sweep, but still calibrates a threshold on labelled validation slices.
  • Evaluation and controls: The evaluation tests whether activations carry adherence information beyond lexical content, sentiment, and length using the margin ∆ = AUROCICS − maxb∈B AUROCb on outcome-ablated inputs.∆≤0 is a reportable outcome, and reported margins use paired-bootstrap confidence intervals with Benjamini–Hochberg correction.
  • Evaluation and controls: Calibration and testing are always disjoint, with either same-distribution calibration or calibrate-once testing on an entirely withheld distribution.The withheld-distribution condition models deployment on inputs unseen during calibration.

3 Evaluation Validity: What Compliance Benchmarks Can Measure

The as-published OmniCompliance task cannot support interpretable probe results because its labels are written into the input and trivial baselines saturate. Removing the explicit verdict barely changes detection, showing that redundant surface cues—not simple contamination—make the benchmark degenerate.

  • Benchmark degeneracy: OmniCompliance embeds the regulatory outcome in each input, causing trivial baselines to saturate and making probe results on the as-published task uninterpretable.Each case contains a rule, scenario, and natural-language OUTCOME field stating the regulatory result.
  • Benchmark degeneracy: 20-domain mean detection changes only from 0.968 →0.952 after removing OUTCOME, indicating redundant scenario cues rather than simple verdict contamination.The explicit field is sufficient but not necessary for the label, leaving little conditional information beyond the text surface.

4 Generalisation Beyond the Calibration Distribution

ICS generalises beyond its calibration distribution under leave-one-distribution-out testing and remains above chance on external benchmarks, but transfer is structured by detector family and compliance generalisation is only marginal.

  • Leave-one-distribution-out: ICS pools compliant/violating and safe/unsafe pairs from eleven distributions, calibrates on one held-out distribution, and tests on another.It fits a single difference-of-means direction using ten labelled pairs per source distribution.
  • Leave-one-distribution-out: 0.728 mean AUROC on unseen distributions, with 10/11 above 0.60, clears the budget-matched null (0.557) but trails per-distribution calibration (0.849).Table 2 reports that ICS clears its null (0.728 vs. 0.557) but ties TF-IDF in the mean.
  • Family transfer: The complete calibrate-on-X, test-on-Y grid shows family-specific block structure, with even the best single calibrator collapsing on the opposite family.The result supports calibrating on the family that will be monitored rather than shipping one universal calibrator.
  • External benchmarks: On seven never-pooled external benchmarks, ICS remains above chance and mostly preserves family structure, but shared-signal compliance transfer is only marginal.The external compliance set is read above chance by the shared signal, not by the compliance-specialised direction.

5 Comparison with Deployed Guards

ICS outperforms deployed guards in in-domain ranking and exposes threshold and mode effects that complicate direct comparisons. Guards remain strong on their safety benchmarks but over-refuse benign inputs, while training-free detectors’ gains depend on calibration and reference choice.

  • In-domain comparison: 0.952 AUROC: ICS beats every deployed guard head-to-head on in-domain OmniCompliance, though this comparison calibrates ICS directly on the test distribution.The comparison is not the paper’s calibrate-once generalisation result because guards run zero-shot outside their safety-training domain.
  • Mode choice: 0.785→0.875 AUROC: WildGuard improves from prompt-slot to response mode on OmniCompliance, with response mode stronger for all four guards.Guards can screen cases as prompts or judge them as responses, and the two modes are not interchangeable.
  • Threshold effects: 0.08–0.22 recall: shipped guard thresholds rarely flag regulatory violations, producing native F1 below ICS’s 0.904; recalibration recovers guard F1 to 0.54–0.80.The passage attributes the native-F1 gap mostly to threshold choice rather than capability, while ICS’s advantage is ranking rather than threshold access.
  • Training-free detectors: GradSafe also beats every deployed guard after re-referencing with sixteen compliance pairs but collapses to chance under its native safety reference.This supports the training-free comparison while showing that reference choice affects detector performance.
  • Home-domain behavior: 75%: WildGuard flags adversarially-benign OR-Bench prompts, illustrating strong home-domain detection alongside substantial over-refusal.The guards are evaluated on seven standard safety benchmarks and benign inputs because compliance is their non-native domain.

6 What the Score Measures

ICS and the compared guards largely ignore the governing rule, even under counterfactual and crossed-rule tests designed to isolate rule–scenario reasoning. A rule-attending repair and policy-conditioned guard also fail to show reliable rule sensitivity.

  • Counterfactual rule tests: Deleting, shuffling, or substituting the rule leaves detection unchanged: mean AUROC moves +0.003, with no BH-significant drop or gain across 20 domains.The three conditions are length-identical, no test text nears 512 tokens, and the pipeline reproduces the published gate bit-exactly.
  • Crossed-rule benchmark: On the crossed benchmark, every cheap detector is at or barely above chance, while chain-of-thought is the positive control that moves.Chance is 6.25% quadruple exact match and 0.500 AUROC; chain-of-thought achieves 40/200 quadruples, with a Wilson CI of 58.9–85.4%.
  • Crossed-rule benchmark: Two baseline dynamic guards switch to safe only 64% and 36% of the time when the violated policy is removed, rather than reliably.LPG was designed to condition on the active policy, but this probe was interpreted as evidence that guards condition inadequately on it.
  • Robustness checks: An earlier fixed-scenario counterfactual reproduces the same qualitative rule-blindness pattern, with the crossed design resolving its polarity confound.The earlier design could be passed by reading rule polarity rather than the rule–scenario interaction.
  • Failed repairs: A purpose-built rule-attending direction reaches 0.605, below its null in 19/20 domains, and deleting the rule still costs nothing.The direction was calibrated from badness-matched rule-varying contrasts and orthogonalised against the refusal axis after conditional activation steering.

7 Score-Guided Candidate Selection

ICS-guided candidate selection improves mechanically verified and LLM-judged compliance outcomes for substantively different responses, while a controlled OmniCompliance test recovers 64–79% of oracle best-of-5 headroom across six models. The classification result is verdict-selection over sampled labels rather than content-selection, whereas the strongest practical gains concern generated responses.

  • Substantive response selection: +5.2pp: ICS-guided selection of instruction-following responses beats random against a mechanical verifier across 541 cases.The gain scales with within-prompt detection AUROC.
  • Substantive response selection: +11.5pp: the same frozen direction, without re-fitting, beats random on LLM-judged regulatory advisory responses across six domains.The improvement survives a coherence control ruling out quality-selection.
  • Verdict-selection sanity check: 64–79%: best-of-5 ICS-guided selection captures this fraction of oracle headroom in every one of six OmniCompliance models from 1B to 9B.The result is significant and positive, with zero significant negative domains; absolute gains shrink for stronger generators while recovered fractions remain 64–79%.
  • Verdict-selection sanity check: The OmniCompliance sanity check selects the candidate with the correct ICS-predicted verdict from pure-temperature samples with ablated inputs, testing verdict-selection over menu labels rather than content-selection.The comparison uses blind no selection, random best-of-5, ICS-guided best-of-5, and oracle best-of-5.

8 Scope and Limitations

The headline claims are bounded by access, transfer, calibration, and specificity limitations. ICS requires first-party activation access, transfers across related task families but needs local threshold recalibration, and captures broad compliance risk rather than reliably rule-specific composition.

  • Access and auditability: ICS requires first-party activation access, making it suited to self-monitoring rather than third-party auditing.
  • Transfer and calibration: Its direction transfers across related compliance and safety task families, with reported values of 0.55/0.48, but its threshold still requires local recalibration.
  • Signal specificity: ICS captures a broad compliance-risk signal rather than reliably representing rule-specific composition.
  • Probe comparison: A trained MLP or logistic probe on the same activations is slightly more accurate than ICS.

9 Conclusion

The Internal Compliance Score is a training-free activation readout that ranks compliance risk using a small calibration set and one projection per case. It retains signal on withheld distributions and can be refit after fine-tuning, unlike a separate guard.

  • Internal Compliance Score: ICS reads the monitored model’s activations without gradient updates, using a difference of means from a small calibration set and one projection per case.It is a training-free readout of the model being monitored.
  • Internal Compliance Score: 0.728 leave-one-out AUROC against a 0.557 null shows useful withheld-distribution signal, with 0.549 as the worst case.These figures quantify ICS performance when calibration data are withheld by leave-one-out evaluation.
  • Internal Compliance Score: ICS can be refit after fine-tuning, a capability unavailable to a separate guard.The score therefore remains adaptable to changes in the monitored model.

A Experimental Setup

The experiments use a twelve-model, single-GPU pipeline centered on Llama-3.2-1B-Instruct, with outcome-ablated 20-domain compliance data and external benchmarks. Detection and selection are calibrated and evaluated against pre-registered baselines using paired resampling, multiplicity correction, and implementation checks.

  • Compute budget: All detection, gate, and cross-model arms run on one GPU per arm, while calibration uses 10–200 labelled pairs per domain and one forward pass per case.The full 20-domain, twelve-model detection sweep therefore requires only low-thousands of forward passes rather than training.
  • Models: The roster spans twelve models, four architecture families, and 1B–72B parameters, with bfloat16 extraction except for the 72B int8 arm.Detection and geometry use Llama-3.2-1B-Instruct; other arms replicate detection and selection.
  • Data: The primary corpus contains 20 outcome-ablated domains, with detection trained on 40% slices and evaluated by AUROC on held-out 16–100-pair test slices.Selection uses separate generation sets of 200 cases per domain, and the gauntlet also includes six external compliance benchmarks.
  • Baselines and statistics: The pre-registered floor pool includes TF-IDF, keyword, length, VADER, verdict-logit, SST-2, and refusal directions, with the gate defined against its per-domain maximum.Statistical procedures include paired bootstrap with B=2000, Benjamini–Hochberg correction across 20 domains, Clopper–Pearson bounds, and 10^-9 AUROC wrapper checks.

B The Pre-Registered Gate · B.1 Full-Detail Tables Referenced from the Main Text · B.2 The Fixed-Scenario Counterfactual, in Full

The pre-registered ICS gate reaches 11/20 domains, below its requirement of at least 12/20, after corrected trivial floors and exact-threshold checks. Full-detail tables document the underlying evaluations, while fixed-scenario counterfactuals show deployed ICS directions remain at chance across model families and scales.

  • B The Pre-Registered Gate: 11/20 domains pass the pre-registered gate, below the requirement of at least 12/20 with a BH-corrected paired-bootstrap CI excluding zero.The maximum trivial floor is the per-domain maximum over trivial baselines, with an SST-2 sentiment direction reaching that maximum in 14/20 domains.
  • B The Pre-Registered Gate: The verdict changed from 13/20 because a sentiment floor was silently dropped and one pass depended on a floating-point BH threshold.The exact-arithmetic threshold is 0.03×20/12 = 0.0499 . . . .
  • B The Pre-Registered Gate: All eleven passes are full-size (n=100) mid-difficulty domains, while nine failures include small-n, ceiling-saturated, corrected-floor, and n=100 mid-floor failures.The passage reports three small-n failures, two ceiling-saturated failures, two corrected-sentiment-floor failures, and failures in GDPR and foundational rights.
  • B.1 Full-Detail Tables Referenced from the Main Text: The five full-detail tables make numbers from score measurement, generalisation, and deployed-guard comparisons checkable without being necessary to follow the main argument.This subsection is an archival reference for underlying results rather than a separate evidentiary claim.
  • B.2 The Fixed-Scenario Counterfactual, in Full: The fixed-scenario test constructs 208 byte-identical scenario pairs across 8 regulatory domains, flipping labels through permissive versus prohibitive rules.Condition-based, length-matched construction removes lexical polarity; scenario-only and cheap rule-cue baselines score 0.49 and 0.71.
  • B.2 The Fixed-Scenario Counterfactual, in Full: Llama holds fully, with 0/20 significant drops on both arms, while Qwen shows the same small hedge on both models.The table characterizes this as architecture-consistent rather than a single-checkpoint artifact.
  • B.2 The Fixed-Scenario Counterfactual, in Full: Fit-on-the-flips results are only an upper bound on relational reading because deployed directions predict the labels throughout the fixed-scenario set.The crossed benchmark is needed to resolve this caveat.
  • B.2 The Fixed-Scenario Counterfactual, in Full: 0.504 and 0.489 are the deployed-direction accuracies for Llama-3.1-8B-Instruct and Qwen3-8B, respectively, indistinguishable from the 1B arm’s 0.502.The chance-level readout replicates across model scale and family because the deployed direction never sees the flips.

B.3 The Crossed Rule-Scenario Benchmark, in Full … C Full Per-Domain Gate Table

The crossed-rule benchmark shows that most fast detectors remain near chance, whereas step-by-step reasoning solves the rule–scenario distinction. Additional replication finds limited, family-dependent transfer, while the full per-domain gate exposes heterogeneous failures including FOUNDATION_RIGHTS.

  • B.3 The Crossed Rule-Scenario Benchmark, in Full: 200 templates across eight domains yield 800 crossed-rule/scenario rows, with quadruples as the resampling and reporting unit and Wilson 95% intervals computed directly.Each template varies regulatory axes including deadlines, exemptions, consent, jurisdiction, actor type, purpose limitation, and regime conditions.
  • B.3 The Crossed Rule-Scenario Benchmark, in Full: 0.500–0.535 AUROC characterizes the primary detector matrix, with 0.0% exact match for frozen ICS and fitted probes and 1.0% for LPG-4B.The matrix includes the Llama-3.1-8B forced-choice judge, Llama Guard 3, Qwen3Guard, and policy-conditioned LPG-4B.
  • B.3 The Crossed Rule-Scenario Benchmark, in Full: 0.670 AUROC and 11.0% exact match are the best cheap-detector results after crossed-data refitting, yet remain weak against 6.25% chance.Llama-3.1-8B logistic reaches 0.616 AUROC and 7.0% exact match, while frozen ICS remains near chance at 0.510 and 0.540 AUROC on the larger models.
  • B.3 The Crossed Rule-Scenario Benchmark, in Full: 0.849 AUROC and 74.4% quadruple exact match show that the identical Llama-3.1-8B judge can solve the benchmark when prompted to reason step by step.On the fixed 40-quadruple subsample, it also reaches 79.5% rule-flip accuracy and 78.2% scenario-flip accuracy.
  • B.4 A Worked Example: One independently sampled finance_crypto quadruple concretely illustrates what a chance-level detector misses when rules swap category-specific thresholds.The example uses template t07 and contrasts a rule with its threshold-swapped counterpart across retail and institutional dormant accounts.
  • B.5 LPG Violated-Policy-Removal Replication: Compliance transfer marginally holds on an external benchmark through the shared signal, while the compliance-specialised direction transfers to unseen scenarios but not another compliance genre.The reported ceiling is not repaired by refitting on fine-tuned models.
  • C Full Per-Domain Gate Table: Table 15 reports all 20 pre-registered domains; every passing domain has n=100, while failures are heterogeneous across small-n, ceiling saturation, and corrected sentiment floors.FOUNDATION_RIGHTS uniquely fails at n=100 against a mid-range floor and worsens under the rule-disjoint restriction, from 0.909 to 0.731.

D Lexical Floors Across Benchmarks … H Matched-Threshold Calibration and Cross-Model Adaptation

Across compliance and safety benchmarks, lexical shortcuts often match or exceed activation probes, while guard performance depends strongly on task framing and generalises poorly across families. Leakage audits and matched evaluations show that apparent detector gains can reflect benchmark construction, selection floors, or input-versus-response mismatch rather than rule-sensitive compliance detection.

  • D Lexical Floors Across Benchmarks: 4/7 compliance benchmarks are degenerate, with policy-blind lexical floors of 0.57–0.98, including the human-expert-labelled benchmark.On AIReg-Bench, TF-IDF reaches 0.957 (grouped 5-fold 0.975), while CompliBench reaches 0.879 and its guideline metadata leaks labels at 0.990.
  • D Lexical Floors Across Benchmarks: On IFEval, ICS beats the form-counting floor in 0 of 7 categories; on FlexBench, its strict response-track lead is only 0.832 vs. 0.802.The FlexBench lead inverts at loose severity, while the prompt-track zero-supervision refusal direction matches supervised TF-IDF outright.
  • D.1 Guard Models on the Same External and Degeneracy Benchmarks: The 8B zero-shot judge is competitive on classic prompt-harm sets but performs at or below chance on nearly every regulatory or legal-compliance benchmark.Reported scores include 0.864 on wildjailbreak, 0.785 on jbb, 0.843 on xstest, and 0.179 on aireg.
  • D.1 Guard Models on the Same External and Degeneracy Benchmarks: Every guard sits near chance on DynaBench, SafePyramid, and CompliBench, where ICS’s apparent edge is largest precisely when the honest lexical floor is high.The comparison indicates that deployed guards do not clearly outperform ICS in these degenerate domains.
  • D.1 Guard Models on the Same External and Degeneracy Benchmarks: Budget-matched selection inflates random directions to 0.50–0.65 on unrelated probing settings, versus 0.49–0.51 for no-selection controls.The compliance inflation of 0.71 is the highest measured, so selection floors must be measured for each setting.
  • E Split Leakage Audit: Rule-disjoint testing leaves AUROC at 0.962 vs. 0.956, with the restricted score at least as high in 13/19 domains.Foundational rights is the adverse domain, declining from 0.909 to 0.731; the restriction is computable in 19/20 domains.
  • F Cross-Family Generalisation (Leave-One-Family-Out): Compliance-to-safety transfer yields mean AUROC 0.552, with every target between 0.50 and 0.60.This stricter leave-one-family-out test fits on four compliance distributions and evaluates seven safety benchmarks.
  • G Applying Guards on Input vs. Response: Response-mode evaluation improves generative guards almost everywhere, decisively for WildGuard by +0.32 mean AUROC, while native compliance thresholds produce recall of only 0.08–0.22.Recalibration raises WildGuard from 0.11 to 0.70 and Qwen3Guard from 0.15 to 0.81, still below ICS’s val-calibrated F1 of 0.904.

H.1 Training-free comparators and model-specific recalibration, in full … M Independent Frontier-Model Validation of the LLM-Judge Claims

Across training-free detection, rule ablations, selection-null controls, and response ranking, ICS is inexpensive and adaptable but substantially affected by selection and surface baselines. Its generated-response gains replicate directionally across judges, while their magnitudes vary widely.

  • H.1 Training-free comparators and model-specific recalibration, in full: AUROC 0.908 lets GradSafe beat every deployed guard after sixteen compliance-pair recalibration, while native safety referencing collapses to 0.564; ICS remains cheapest.ICS uses no trained parameters, needs no extra pass for retained activations, and reaches full accuracy from 20 pairs versus 104–105 for a guard.
  • H.1 Training-free comparators and model-specific recalibration, in full: A ten-pair re-threshold lifts guards to F1 0.54–0.80, yet ICS still leads every matched-budget threshold metric and exceeds the strongest guard, LPG-4B at AUROC 0.943, across nine models.Fresh ICS refits use each monitored model’s own activations, including Llama, Gemma, and Qwen variants, whereas the frozen guard classifies the input case.
  • I Rule Ablation, Visualized: ICS is unchanged under rule deletion, shuffling, or cross-domain substitution, with 0/20 significant drops each, and collapses only when the scenario is removed.TF-IDF behaves similarly and improves when the rule is removed, indicating that the scenario carries the label under these conditions.
  • I.1 Benign Inputs (Over-Flagging): ICS flags 0.44 of XSTest benign inputs and 0.18 of Alpaca inputs, but only 0.327 of adversarially-benign OR-Bench inputs, lower than every guard that fires on compliance there.Lower numerical rates from Llama Guard 3, HarmBench, and Llama Guard 3 8B reflect refusal classifiers that essentially never fire on OR-Bench’s style.
  • J Selection Nulls: the Budget-Matched Detection Floor: A random direction reaches 0.7135 under ICS’s 17-layer argmax selection, while the refusal-budget null reaches 0.7747, making 0.71 the relevant floor rather than 0.50.The refusal direction retains a +0.089 margin, but harmfulness accounts for roughly 37% of the signal against budget-matched nulls.
  • K Selection-Null Transfer Beyond Compliance: Compliance’s selection floor is 0.714, higher than unrelated truthfulness and sentiment settings at 0.50–0.65 against no-selection controls of 0.49–0.51.This transfer supports selection leakage as a general probing property rather than a compliance-specific artifact.
  • L Content-Selection: Generated Responses and the Judged Regulatory Task: +5.2pp over random raises mechanically verified IFEval pass rates, while +11.5pp pooled raises judged regulatory compliance; the latter captures 43% of oracle headroom.The regulatory selector uses a frozen ICS direction never fit on judge labels or generated text, and the gain persists after restricting to coherent candidates.
  • M Independent Frontier-Model Validation of the LLM-Judge Claims: Claude Haiku 4.5, GPT-4o, and Gemini 2.5 Flash all preserve guided > random > first, but the content-selection magnitude varies by an order of magnitude across judges.These independent judges replicate the local zero-shot protocol, while rule-counterfactual agreement is weaker.

N Adversarial Robustness: Verdict- vs. Content-Selection … W Extended Related Work

Across adversarial, calibration, metric, transfer, perturbation, scaling, methodological, and sample-efficiency tests, ICS shows portable detection signal but substantial rule, threshold, and adaptive-attack vulnerabilities. The paper therefore positions ICS as an inexpensive auditing readout whose robustness claims are bounded by lexical comparators, deployment conditions, and prior probe limitations.

  • N Adversarial Robustness: Verdict- vs. Content-Selection: 78.8% captured headroom collapses to the random floor under a poison suffix for verdict-selection, while content-selection retains 79% of headroom and changes verified pass rate by −1.1pp.Verdict-selection inherits the monitor’s case verdict; the content-selection attack excludes candidates whose strict verdict the suffix genuinely flips.
  • N Adversarial Robustness: Verdict- vs. Content-Selection: 0.00 ICS-guided verified pass follows an adaptive white-box GCG attack, which moves faithful ICS by +1.66 versus +0.10 for the fixed suffix.The attack uses 12 tokens and 40 steps on 20 stratified attackable IFEval prompts, while the official verifier still fails 52/53 poisoned responses.
  • O.1 Calibration Sensitivity: which ten pairs, and how many?: 0.014 mean within-domain AUROC standard deviation across 50 random 10-pair draws shows calibration-set choice is stable, while paraphrasing preserves 96% of cross-template transfer.ICS remains within 0.947±0.008 AUROC across the tested instruction templates, but its position sensitivity is separately documented.
  • R Perturbation Robustness: 72.4% of correctly flagged violations flip after one appended compliant-register sentence, whereas paraphrase retains 0.994 and an opposite-verdict prefix changes sensitivity by only 0.012.TF-IDF moves 3.4% under the appended sentence and degrades more under paraphrase than ICS in 18/20 domains.
Loading 2608.16852v1…