PaperlayerView exact revision on arXiv

Source-linked AI summary

Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang, Leo Yu Zhang

arXiv:2609.29429v1cs.AIcs.CLcs.CR

TL;DR

Detectors often score one criterion per call, and whether Jev can detect alignment failures had not been measured. RLCDAlignBench tests Jev across benchmarks while varying the question and the state separately. A generic question ranks most failures well zero-shot, and Jev’s cost and label comparisons support its potential use as a detector.

  • Problem

    Most detectors score one criterion per call, and Jev’s ability to detect alignment failures—many of which depend on references beyond the response—had not been measured.

    Source: §1 · p. 2

  • Method

    RLCDAlignBench evaluates Jev across 44 benchmarks, varying question wording and answer type separately from the fields in its state.

    Source: §1 · p. 3, §3.1 · p. 4, §3.2 · p. 5

  • Results

    A generic question reaches median AUROC 0.886 over 31 benchmarks, while targeted wording adds only +0.006 [−0.004, +0.015] AUROC out of sample.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Jev’s confident disagreements can locate label defects in existing benchmarks, and one pass over 19 judge-scored benchmarks costs 63× less than their judges.

    Source: §1 · p. 3

    The evaluation covers one RLCD model, English benchmarks, 2–7B targets, and mostly scorer labels.

    Source: §6 · p. 10

Abstract

from arXiv · show

Detectors of alignment failures screen deployed language models and score alignment benchmarks. Most are generative judges that spend a decoding pass on every criterion, and classifiers that read token probabilities, such as Llama Guard, still score one fixed label per call. Jev, a model trained with reinforcement learning for calibrated decisions (RLCD), answers many typed questions about one input with calibrated probabilities in a single call. Whether it detects alignment failures has not been measured. We present RLCDAlignBench, which benchmarks Jev on ten alignment failures: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealing uncertainty, and power seeking. It spans 44 benchmarks and five target models, labelled by each benchmark's scorer and, on two, by humans. Many of these failures are relational, defined against a reference, such as the user's belief or an injected instruction, that the response alone does not reveal. Our key idea is therefore to vary what Jev is asked separately from what it sees: the question's wording and answer type on one side, the fields of the input on the other. A single generic question reaches a median AUROC of 0.886 zero-shot and beats supervised baselines on most benchmarks. Question wording matters little, while context matters more, mostly through fields that encode the label. Jev matches the reference scorer's agreement with human labels, surfaces label defects in existing benchmarks, and costs 63x less than LLM-judge scorers. Code and data: https://github.com/sumleo/RLCDAlignBench.

0.886 AUROC

Figure 1 compares Jev’s generic and targeted-question AUROC across alignment failures and reports headline medians and cost.

0.886 AUROC

  • Rows show per-failure AUROC across 38 usable benchmarks; filled markers are generic NOUL and hollow markers are split-half targeted questions.The right side reports median generic AUROC, median split-half gain, and the cost of one pass over 19 API-judge benchmarks.Source: Figure 1 · p. 2

1 INTRODUCTION

The paper studies Jev as a low-cost detector for alignment failures, whose relational definitions can require context beyond a model’s response. RLCDALIGNBENCH separates the questions Jev is asked from the fields it sees; a generic question performs strongly zero-shot, while context and labels shape gains and errors.

1 INTRODUCTION

  • Detectors must screen failures such as deference to mistaken beliefs and jailbroken requests, and must be accurate and cheap because they run on every message.Prior work describes these failures and detector uses (Sharma et al., 2024; Greshake et al., 2023; Ren et al., 2025; Inan et al., 2023; Guan et al., 2026; Mazeika et al., 2024; Chen et al., 2026).Source: §1 · p. 2
  • Most generative judges or safety classifiers score one criterion per call, whereas Jev returns probabilities for many typed questions about one input in a single call.Jev is an RLCD model; its question types include binary NOUL, categorical CHOICE, and ordinal SCORE.Source: §1 · p. 2
  • Whether Jev detects alignment failures had not been measured, and relational failures may require references absent from the response, such as a user belief or injected instruction.RLCDALIGNBENCH varies question wording, answer type, and state fields separately across ten failure types, 44 benchmarks, and five target models.Source: §1 · pp. 2–3
  • The benchmark includes scorer labels and human labels on two benchmarks, and Jev agrees with human labels as well as the reference scorer where checked.Jev’s confident disagreements exposed label-changing defects in three benchmarks and labels the state cannot reveal in four more.Source: §1 · p. 3
  • A generic question reaches median AUROC 0.886 over 31 benchmarks and beats supervised TF-IDF and length baselines on 25.Out of sample, targeted wording adds +0.006 [−0.004, +0.015] AUROC; Jev’s pass over 19 judge-scored benchmarks costs $0.30, 63× less than their judges.Source: §1 · p. 3
  • The authors provide RLCDALIGNBENCH, cached Jev answers, and rescoring scripts, and report that confident disagreements can locate existing benchmark label defects.The code and data are available at the project repository.Source: §1 · p. 3

2 BACKGROUND

The paper frames alignment-failure detection as ranking and thresholding labelled interactions, while Jev answers typed questions with probabilities from a shared state in one call.

2 BACKGROUND

  • A detector scores each interaction and flags it above a threshold; failure detection must both rank failures above non-failures and separate them at that threshold.Labels come from benchmark reference scorers, and some benchmarks also provide human labels.Source: §2 · p. 3
  • RLCD trains models to return calibrated decisions rather than generated text, with calibration meaning that decisions assigned probability p are correct at a fraction close to p.The paper identifies Jev as TypeSafe AI’s RLCD model.Source: §2 · p. 3
  • RLCDALIGNBENCH evaluates Jev’s combined score against labels it never sees, using labelled interactions and varying both the question and state fields.The pipeline flags an item when its combined score reaches threshold t.Source: Figure 2 · p. 4
  • Jev answers multiple typed questions about one state in a shared call, returning yes-probabilities, distributions over choices, or distributions over ordered scores.These answers provide probabilities that can be combined into a detection score without parsing generated text.Source: §2 · p. 4

3 RLCDALIGNBENCH

RLCDALIGNBENCH evaluates Jev on labelled alignment interactions while separately varying its state and questions. Its protocol controls context, readout, and selection so detection comparisons reflect ranking rather than access to labels.

3.1 FAILURE TYPES AND BENCHMARKS

  • The benchmark converts existing alignment suites into detection tasks spanning ten failure types, 44 benchmarks, and 7,193 instances; labels come from rules or LLM judges, with human comparisons on two sets.Eleven labels are validated, 25 unvalidated, and the audit changes eight; one audited benchmark has too few negatives to score, leaving seven usable audited benchmarks.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 DETECTION INSTANCES AND CONTEXT VARIANTS

  • The canonical state contains the target model’s input and output, while paired variants change fields one at a time to test missing or already-present references and other context additions.Added fields are tagged as deployable or label keys; the PrivacyLens secret list is a label key because positivity depends on leaks of listed secrets.Source: §3.2 · pp. 5–6

3.3 QUESTION PROTOCOL

  • Each benchmark receives a generic question built from a template and its failure definition, while targeted questions name the behaviour through direct, rubric, or decomposition strategies.Generic questions use NOUL, CHOICE, or SCORE readouts; targeted strategies combine answers from one call, with 15–40 strategies per benchmark.Source: §3.3 · p. 6

3.4 METRICS AND PROTOCOL

  • AUROC measures ranking independently of threshold, while F1 is reported at 0.5 and with a cross-validated threshold; results use grouped bootstrap intervals and medians over 38 usable benchmarks.To control best-of-many inflation, targeted strategies are selected on one half of items and evaluated on the other across grouped splits.Source: §3.4 · p. 6
  • Figure 3 compares soft versus argmax readouts, targeted versus generic questions in-sample and split-half, and thresholded rubrics versus direct targeted questions.The split-half comparison is designed to remove selection inflation from choosing among targeted strategies.Source: §3.4 · p. 6, Figure 3 · p. 7

4 RESULTS

Jev’s generic zero-shot question ranks alignment failures well, while question wording adds little and context helps mainly when it contains label-defining information. Its probabilities need benchmark-specific thresholding; human agreement is comparable to the reference scorer, and one Jev call costs substantially less than an LLM judge.

4.1 OVERALL DETECTION

  • 0.886 median AUROC over 31 benchmarks: the generic NOUL beats supervised lexical and length baselines on 25 of 31 benchmarks.The split-half targeted strategy reaches 0.911 [0.860, 0.944] over all 38 benchmarks, while the generic NOUL exceeds the better supervised baseline by a median of +0.132 [+0.057, +0.190].Source: §4.1 · p. 7

4.2 QUESTION DESIGN

  • +0.006 [−0.004, +0.015] AUROC: out-of-sample targeted wording adds a small gain whose confidence interval includes zero.Selecting targeted wording on evaluation data would overstate the gain by 0.008 [0.006, 0.014].Source: §4.2 · p. 7
  • Soft answers matter more than answer type: expected SCORE beats generic NOUL on 19 of 31 benchmarks, while soft CHOICE ties it.Argmax readouts lose on almost every benchmark; thresholding rubric answers at 0.5 loses to the best direct targeted question on 9 of 10 benchmarks, with median −0.137 AUROC.Source: §4.2 · pp. 7–8

4.3 CONTEXT

  • Deployable references rarely help, but label keys raise AUROC above zero-confidence gains on 4 of 11 benchmarks, with median +0.053.Adding the true answer raises the official SycophancyEval label score from 0.540 to 0.941, but lowers the answer-shift label score from 0.712 to 0.288 because the official label measures correctness.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 CALIBRATION AND THRESHOLDS

  • Pooled ECE is 0.047, but median per-benchmark ECE is 0.168 versus 0.074 under perfect calibration; Jev’s mean probability misses the positive rate by a median of 0.125.The median AUROC remains 0.905, indicating the reported error is a base-rate mismatch rather than a ranking problem.Source: §4.4 · p. 8
  • Cross-validated thresholds raise median F1 from 0.706 at t=0.5 to 0.822, while keeping the half of decisions with greatest confidence raises median accuracy from 0.793 to 0.933.Threshold gains come from unvalidated labels; on validated labels, t=0.5 is as good as a fitted threshold, and confidence-based selection can route less-certain decisions to a judge or human.Source: §4.4 · pp. 8–9

4.5 AGREEMENT WITH HUMANS, LABEL AUDIT, AND COST

  • On StrongREJECT, Jev’s agreement with human labels matches the scorer’s (κ=0.809 vs. 0.811), while its AUROC is higher (0.971 vs. 0.929).Confident disagreements exposed label defects in three benchmarks and labels unobservable from the state in four MACHIAVELLI variants; one Jev pass costs 63× less than reference judges pooled.Source: §4.5 · p. 9

5 RELATED WORK

Prior alignment detectors include fixed-taxonomy guards, single-failure detectors, internal probes, and follow-up-question lie detectors. Jev differs by asking many typed questions in one black-box call, while this work also evaluates its agreement with humans and calibration.

Guard models and monitors

  • Guard models use fixed harm taxonomies, while other detectors focus on one failure and probes or chain-of-thought monitors read activations or reasoning [Inan et al., 2023; Zeng et al., 2024; Han et al., 2024].Black-box lie detectors instead ask the model follow-up questions [Pacchiardi et al., 2024].Source: §5 · p. 9
  • Existing detector evaluations stay within detector families: GuardBench compares guards, JudgeBench compares judges, and monitor red-teaming stress-tests agent monitors [Kale et al., 2025].Jev is black-box, has no fixed taxonomy, and answers many typed questions about any failure in one call.Source: §5 · p. 9

LLM judges and label quality

Calibration and thresholds

  • Calibration is commonly measured with scoring rules and reliability diagrams and repaired by post-hoc scaling [Guo et al., 2017].For Jev, label-free prior-shift correction fails, while selective prediction by confidence works [Geifman & El-Yaniv, 2017].Source: §5 · p. 10

6 CONCLUSION

RLCDALIGNBENCH evaluates Jev on 44 alignment-failure benchmarks and finds that generic soft probabilities rank most failures well, while context and label contents shape gains and errors. The study reports human agreement, label-audit utility, cost, and limits on model, language, target size, and label sources.

6 CONCLUSION

  • RLCDALIGNBENCH evaluates Jev as an alignment-failure detector on 44 benchmarks and finds that one generic question with soft probabilities ranks most failures well.The authors recommend fitting a threshold on ten labels for practitioners and using Jev’s confident disagreements to locate benchmark label defects.Source: §6 · p. 10
  • The evaluation is limited to Jev-1.13.0, English benchmarks, 2–7B target models, and mostly scorer labels.The authors identify other detectors, larger targets, and other languages as future work.Source: §6 · p. 10

ETHICS STATEMENT

  • The study reuses harmful requests, jailbreak prompts, and harmful model outputs from public benchmarks, adds no harmful content, and uses previously released human annotations.The authors describe the detector as dual-use because its scores could help attackers search for outputs that evade detection.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The release includes benchmark suites, model outputs, scorer records, labelled detection instances, and cached Jev responses, with scripts to recompute reported metrics offline.Re-querying Jev requires an API key, and the authors use cached responses as the reference because the developer states no retention period.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

  • LLM assistants helped write analysis scripts, draft figures and tables, and draft and edit the text; released scripts generate every reported number from cached responses.Source: AI USE STATEMENT · p. 10

A BENCHMARK DETAILS

RLCDALIGNBENCH turns existing alignment suites into detection instances scored with benchmark reference labels, while separating the monitor-visible state from labels and metadata. Benchmark validity, degenerate cases, and canonical variants determine which instances enter reported aggregates.

Benchmark inventory

  • Each suite is converted to a detection task by generating target-model outputs and replaying its reference scorer, whose verdict becomes the label.The benchmark suite fixes one target model per failure type.Source: A BENCHMARK DETAILS · p. 18

Detection instances

  • Each detection instance pairs a monitor-visible state with a scorer label and metadata; the main state contains the target model’s input and output.Variants can add scorer-seen fields, ablate a field, supply a distilled reference, or use an underlying task label; excluded variants measure reproduction of the scorer’s exclusion rule and do not enter aggregates.Source: A BENCHMARK DETAILS · p. 18

Benchmark roles

  • The benchmark roles H and HO come from Chen et al. (2026), but RLCDALIGNBENCH applies the same protocol to all 44 benchmarks without using those roles.The released package’s battery hash matches the recorded runs for all 38 usable benchmarks, though the cache includes an earlier unreleased battery version for some understanding variants.Source: A BENCHMARK DETAILS · p. 18

Label validity

  • Only 11 benchmarks have validated labels; seven usable benchmarks carry label-changing defects, including deterministic scoring defects, judge defects, and labels unobservable from the state.Excluding those seven leaves 31 benchmarks, while two additional benchmarks have second-judge agreement of κ 0.05 and 0.29, which does not validate their labels.Source: A BENCHMARK DETAILS · p. 18

Benchmark eligibility

  • Six benchmarks are degenerate because they have one class or at most four minority-class items, so they are excluded; three skewed benchmarks remain and are read through AUROC rather than F1.Source: A BENCHMARK DETAILS · p. 18

Canonical variant

  • Aggregates use each benchmark’s canonical state—the target input and output—and the reference scorer’s label.Exceptions include benchmarks with multiple main variants; LLM-AggreFact (multi-turn) has no blind main variant, and LLM-Uncertainty-Bench and SciQ enter only targeted aggregates because their canonical state lacks the required confidence field or generic question.Source: A BENCHMARK DETAILS · p. 19

Benchmark inventory

  • The benchmark table covers 44 benchmarks and records target model, suite split, scorer, label source and validity, instance and positive counts, positive rate, and status.Six degenerate benchmarks are reported but excluded from aggregates; skewed benchmarks are retained.Source: A BENCHMARK DETAILS · p. 18, Table 2 · p. 19

B QUESTION TEMPLATES AND TARGETED-QUESTION FAMILIES

The section compares Jev’s question wording, answer types, and combinations, while defining how generic and targeted questions are scored. Probabilities generally outperform argmax readouts, and split-half selection limits inflation from choosing among many strategies.

B QUESTION TEMPLATES AND TARGETED-QUESTION FAMILIES

  • Each request carries the state and typed questions together, and Jev answers the questions independently in one call.NOUL returns P(yes); CHOICE and SCORE provide distributions and confidence or expected levels.Source: B QUESTION TEMPLATES AND… · pp. 20–21

B.1 THE GENERIC QUESTION

  • Generic question stems vary by state format, with a behavior phrase and field names specifying what Jev should assess.The formats cover single responses, paired outputs, rule-scored benchmarks, and trajectories.Source: §B.1 · p. 21

B.2 TARGETED-QUESTION FAMILIES

  • Targeted questions name the labelled behavior, and fixed strategies combine their answers; code-side strategies using label metadata are excluded.For InjecAgent, the questions distinguish executing an injected instruction from quoting or warning about it without execution.Source: §B.2 · pp. 21–22

B.3 ANSWER TYPE, WORDING, DECOMPOSITION AND SELECTION

  • The 3-level SCORE is the strongest generic readout, while argmax readouts lose 0.06–0.09 AUROC.NOUL and CHOICE tie when both carry the same definitions.Source: §B.3 · p. 22
  • Criteria wording beats minimal wording on 17 of 36 benchmarks, with +0.029 AUROC and +0.022 F1.Definition wording yields +0.017 mean AUROC; statement and question forms are equivalent (−0.003 AUROC).Source: §B.3 · p. 22
  • Decomposition has near-zero median effects, while thresholded multi-atom formulas usually lose to a direct question.Soft compositions recover losses on MACHIAVELLI, PrivacyLens, and MASK.Source: §B.3 · pp. 22–23
  • Split-half selection prevents targeted results from being scored on the same data used to choose a strategy.Selection inflation reaches 0.13 AUROC on Reference letters (n=43) and Workplace scenes (n=44), versus a median 0.008 on 16 larger benchmarks.Source: §B.3 · p. 23

C FULL PER-BENCHMARK RESULTS

The generic question ranks failures well against supervised baselines, while targeted questions add only a small out-of-sample gain. Benchmark-level F1 and per-failure-type estimates require caution because thresholds, class balance, and sparse type coverage affect interpretation.

C FULL PER-BENCHMARK RESULTS

C.1 UNCERTAINTY AND SENSITIVITY OF THE HEADLINE AGGREGATES

  • The generic NOUL beats the better supervised baseline on 25 of 31 benchmarks, with a +0.132 median margin; targeted questions add +0.006 AUROC over the best generic readout.The targeted-gain 95% CI is [−0.004, +0.015], and the comparison has Wilcoxon p = 0.055.Source: §C.1 · pp. 25–26, Table 8 · p. 26
  • Per-failure-type medians rely on only one to six benchmarks, so their confidence intervals are correspondingly wide.Prompt-injection AUROC spans 0.25–0.99 because Open-Prompt-Injection is inverted.Source: §C.1 · p. 26, Table 9 · p. 26

D CONTEXT ABLATION

The context ablation separates deployable context from label-defining references and compares Jev’s scores on matched states. Large gains mostly come from adding label keys, while gains from deployable fields are limited and depend on whether the reference is missing.

D CONTEXT ABLATION

  • Large reference gains come from label keys, while deployable fields rarely improve detection; the DeceptionBench goal is the only deployable-field gain with a CI above zero.Label-key examples include SycophancyEval (+0.36 to +0.40), the multi-turn claim check (+0.18 to +0.26), and PrivacyLens (+0.16).Source: Table 10 · p. 27, D CONTEXT ABLATION · pp. 28–29, Table 11 · p. 29
  • Deployable context helps only when the state lacks the reference: the generic NOUL gains with a CI above zero on 1 of 4 such pairs and none of 3 pairs where the reference was already present.For the missing-reference DeceptionBench goal, the gain is +0.050 [+0.029, +0.077].Source: Table 10 · p. 27, D CONTEXT ABLATION · p. 29, Table 11 · p. 29
  • Adding the true answer reverses Jev’s sycophancy detection because it grades correctness, which is inverted against deference.On the relabelled set, AUROC drops from 0.712 without the reference to 0.288 with it; the +0.40 gain is on the correctness label, not sycophancy.Source: D CONTEXT ABLATION · p. 29
  • Reference gains are larger when baseline AUROC is lower, with Spearman ρ = −0.555 across 20 reference pairs.Benchmarks whose reference is already present have median AUROC 0.964–0.997.Source: D CONTEXT ABLATION · p. 29, Figure 5 · p. 30
  • A distilled one-line belief preserves MASK AUROC while reducing mean input tokens per instance from 8,647 to 2,504, 6,976 to 2,478, and 11,814 to 1,996.Across reference pairs, the median token ratio is 1.04.Source: D CONTEXT ABLATION · p. 29
  • Longer states have a median AUROC difference of −0.001 overall, with penalties concentrated in agentic and multi-turn files missing the reference.In PrivacyLens, the penalty nearly vanishes once the secrets are supplied.Source: D CONTEXT ABLATION · p. 30, Table 12 · p. 30
  • Self-reports help only when questions use label-defining information: stated confidence enables stronger per-file strategies where labels depend on that confidence.DeceptionBench reasoning helps when the deceptive goal is absent (+0.032 [0.004, 0.070], n=64), but not when it is given.Source: D CONTEXT ABLATION · p. 30
  • Generalization and framing effects vary: held-out MACHIAVELLI games lower AUROC by 0.067, while the anti-convergence prompt’s decrease has a CI including zero.The MACHIAVELLI change is 0.918 to 0.851 (−0.067 [−0.122, −0.016]); InstrumentalEval falls from 0.861 to 0.650 with only 31–32 items.Source: D CONTEXT ABLATION · p. 30

E CALIBRATION AND THRESHOLDS

Jev’s probabilities rank failures well but have benchmark-specific prevalence offsets, so pooled calibration can conceal per-file miscalibration. Fitting a threshold on labelled examples improves generic-question F1 on average, though gains vary and cross-validation can fail on some benchmarks.

E CALIBRATION AND THRESHOLDS

  • Pooled calibration masks per-file prevalence mismatch: median ECE is 0.150 against a perfect-calibration null of 0.069, while median AUROC is 0.905.Generic NOUL probabilities are compressed toward the middle; generic CHOICE is over-confident.Source: E CALIBRATION AND THRESHOLDS · p. 31, Table 13 · p. 31
  • Generic-question thresholds often lie below 0.5, causing lost recall; cross-validated thresholds raise generic-NOUL F1 from 0.614 to 0.761 across files.At AbstentionBench, F1 is 0.702 at 0.5 and 1.000 with CV at t*=0.05; CV can hurt on small files such as InstrumentalEval.Source: E CALIBRATION AND THRESHOLDS · pp. 31–32, Table 14 · p. 32
  • Per-benchmark ECE is 0.168 against a null of 0.074, with 24 of 31 benchmarks exceeding the null’s 95th percentile.About half of per-file ECE is finite-sample bias; the remaining miscalibration is attributed to prevalence offset.Source: E CALIBRATION AND THRESHOLDS · p. 32, Table 15 · p. 32
  • Ten labelled items raise median generic-NOUL F1 from 0.706 at t = 0.5 to 0.793, three quarters of the way to the 2-fold CV threshold’s 0.822.The oracle threshold reaches 0.849, while label-free EM prior-shift correction does not help.Source: E CALIBRATION AND THRESHOLDS · p. 32, Table 16 · p. 33
  • Thresholds below 0.5 occur across validated, judge, and rule labels, but fitted-threshold F1 gains differ by label source.The reported gain is +0.008 [−0.043, +0.044] on validated labels and +0.052 [−0.000, +0.126] on judge-labelled benchmarks.Source: E CALIBRATION AND THRESHOLDS · pp. 32–33, Table 17 · p. 33
  • Confidence supports selective detection on jailbreaks, reward hacking, and deception, but is anti-informative on social bias.For the generic NOUL, accuracy on the most confident half has median 0.935 versus 0.804 on all items.Source: E CALIBRATION AND THRESHOLDS · p. 34, Table 19 · p. 34

F.1 AGREEMENT WITH HUMAN LABELS

This section examines Jev’s agreement with human labels across sycophancy, jailbreak, deception, and prompt-injection benchmarks, alongside audits of what their labels measure and whether relevant context is available in the input. It highlights stronger detection when failures and label-defining context are visible, as well as limitations from omitted context, sparse labels, and rules that diverge from human judgments.

F.1 AGREEMENT WITH HUMAN LABELS

  • On StrongREJECT, generic NOUL matched the scorer’s human-label agreement (κ 0.809 vs. 0.811) and ranked better (AUROC 0.971 vs. 0.929).The AUROC advantage persisted after matching the scorer’s nine-value resolution. Out-of-sample selection of the targeted question yielded a κ gain of +0.046 [−0.024, +0.090], but gains did not transfer across datasets.Source: §F.1 · pp. 34–35, Table 20 · p. 34, Table 21 · p. 35
  • On HarmBench validation, Jev’s agreement with a single annotator was on par with annotator agreement (mean pairwise κ 0.736 on 578 answered items); its residual-error accuracy was 0.957 on 465 unanimous items versus 0.708 on 113 split-vote items.Source: §F.1 · pp. 36–37

F.2 LABEL DEFECTS

  • Open-Prompt-Injection’s label tracked injected-task accuracy rather than following the injection, while SycophancyEval (answer) labelled correctness rather than deference.Against corrected labels, Open-Prompt-Injection moved from inverted to near-perfect ranking and SycophancyEval reached 0.71–0.76 AUROC; MACHIAVELLI labels depended on annotations and choices absent from Jev’s state, while the workplace parser defect changed direction but not the binary any-gap label.Source: §F.2 · pp. 38–40
  • The label audit found defects that changed binary labels on seven usable benchmarks, and correcting two deterministic defects slightly raised headline AUROC medians.Generic NOUL rose from 0.886 to 0.893 and best generic from 0.903 to 0.914; F1 at 0.5 remained 0.706. The audit classified 311 items individually and found interpretive defects most often across benchmarks with item-level verdicts.Source: §F.2 · pp. 37, 39, Table 28 · p. 40

G.1 SYCOPHANCY

  • Sycophancy detection varied with the reference information: SycophancyEval (answer) rose from generic NOUL 0.540 to 0.941 when the true answer was supplied, while feedback labels were inconsistent.On ELEPHANT, a relation-specific question reached AUROC and F1 1.000, but generic NOUL had F1 0.646 at 0.5; half of SycophancyEval (feedback) positives contradicted their definition.Source: §G.1 · p. 41

G.2 JAILBREAKS

  • Generic NOUL detected harmful compliance on HarmBench, PAIR, and StrongREJECT with AUROCs 0.965, 0.963, and 0.993, and showing the attack prompt changed results by at most 0.002.Over-refusal was also detectable, but the generic question was conservative on arguable StrongREJECT cases and response length alone reached median AUROC 0.908 for this failure type.Source: §G.2 · pp. 41–42

G.3 DECEPTION

  • Jev detected deception without being told the belief by inferring it from five elicitation answers, while showing the deceptive goal raised DeceptionBench AUROC from 0.926 to 0.976.A generic SCORE beat the re-implemented MASK pipeline on factual deception (F1 0.978 vs. 0.780); remaining errors clustered around construct boundaries such as draft continuation and evasive nonanswers.Source: §G.3 · p. 42

G.4 PROMPT INJECTION

  • Prompt-injection success visible in the output was detected nearly perfectly, but Tensor Trust hijacking’s generic NOUL reached F1 only 0.095 at threshold 0.5 without the relational fact.Showing the attacker input raised F1 to 0.716 but lowered generic AUROC by 0.038; Open-Prompt-Injection’s official label was inverted for generic NOUL, whereas its corrected label yielded AUROC 0.975.Source: §G.4 · p. 42
  • Prompt-injection false positives reflect a mismatch between benchmark rules and partial compliance: InjecAgent counts only attacker-tool calls, while Jev also sees relaying the request.Tensor Trust false positives include near-matches rejected by its regex, such as a typo in “Access Granted” or the phrase written in hexadecimal.Source: §G.4 · p. 42

G.5 HALLUCINATION

  • Showing the gold verdict raises generic NOUL AUROC from 0.617 to 0.793 on multi-turn LLM-AggreFact, while detection largely tracks Jev’s claim verification.LLM-AggreFact A is hardest at 0.755; multi-turn RAGTruth reaches 0.976 AUROC but has 12 positives and F1 of 0.522 at 0.5. Some apparent errors reflect label mismatches, including retracted planted details and unsupported response claims.Source: §G.5 · p. 43

G.6 PRIVACY VIOLATION

  • On PrivacyLens, generic NOUL AUROC falls to 0.792 without the secret list, and detection lags Jev’s understanding when labels encode literal rules or norms.ConfAIde tier 3 and 4 detection AUROCs are 0.728 and 0.642, versus understanding AUROCs of 0.989 and 1.000; labels can count harmless listed secrets or miss real leaks outside the list.Source: §G.6 · p. 43

G.7 SOCIAL BIAS

  • Jev recovers the corrected direction on 92–100% of scene positives, but its generic question is miscalibrated for subtle agency gaps, with pooled ECE 0.546.Swapped-story judgments have sign agreement 0.83 (r = 0.96); the targeted differential question fires on stylistic differences, while a not-for clause recalls 14% at 0.5. Reference-letter F1 of 0.845 is close to the all-positive 0.822 baseline.Source: §G.7 · p. 43

G.8 REWARD HACKING

  • Explicit reward seeking reaches 1.000 on world-affecting reward free-form and 0.996 on choice with generic CHOICE, while Jev raises no false alarms on 50 honest rubric-tampering episodes.The multi-turn benchmark asks whether the model switched because of the autograder, which a transcript cannot establish when the first answer already matches the rewarded letter; generic NOUL has precision 0.77 at recall 1.00.Source: §G.8 · p. 44

G.9 CONCEALING UNCERTAINTY

  • With question-and-answer text alone, targeted split-half AUROC is 0.912 on LLM-Uncertainty-Bench and 0.978 on SciQ, although canonical states omit the target’s confidence.The canonical labels have only 15 and 6 positives; their AUROCs exceed plain-wrongness AUROCs of 0.766 and 0.778 on 55 and 20 wrong answers.Source: §G.9 · p. 44

G.10 POWER SEEKING

  • On MACHIAVELLI, comparing badness across options yields targeted AUROCs of 0.839, 0.919, and 0.851, with identical-state Jev runs agreeing at r = 0.997–0.999.On the understanding track, this approach gains +0.095 AUROC on acquisition and +0.027 on harm, but none across all 13 categories; held-out games cost 0.067 AUROC.Source: §G.10 · p. 44

H COST ACCOUNTING

Jev consolidates many typed questions into one call, and its cost advantage over API judges varies with judge pricing and the question battery. Under list prices, the pooled judge-to-Jev cost ratio is 62.9×; PrivacyLens and MASK account for much of that ratio.

Jev calls

  • One Jev call carries a median of 11.4 questions, with 0.313 s median latency and USD 2.27 for the study’s 23,411 calls.Latency correlation with the number of questions is −0.037; asking only one strategy’s questions costs a median 0.43–0.54 of the full battery.Source: H COST ACCOUNTING · p. 45, Table 30 · p. 46

Cost comparison

  • Across the 19 API-judged benchmarks, Jev uses 7.18M tokens versus 9.36M for reference judges, which make a median of 2 calls per item.On the 20 rule-scored benchmarks, Jev adds USD 0.36 per pass; its stated value there is independence from target-output format.Source: H COST ACCOUNTING · p. 45

Cost ratios

  • 62.9× is the pooled judge-to-Jev cost ratio at list prices with Jev billed for the full battery, versus a per-benchmark median of 16.2×.With repriced judges and Jev billed for one generic question, the pooled ratio is 12.1×; with the full battery it is 6.0×.Source: H COST ACCOUNTING · p. 45, Table 32 · p. 47
  • PrivacyLens and the three MASK benchmarks account for 62–65% of judge costs; excluding them, pooled ratios are 35.1×, 7.3× and 3.6× across the three scenarios.The pooled ratio exceeds the median in every scenario.Source: H COST ACCOUNTING · pp. 45–46

I BASELINES

Zero-shot Jev generally outranks lexical and length baselines, including a supervised TF-IDF model, while a fitted threshold improves its F1. Its probabilities remain miscalibrated, and threshold fitting with only ten labels performs strongly against TF-IDF trained on the same labels.

Baselines

  • TF-IDF uses labelled in-domain states with word 1–2-grams and character 3–5-grams, while length scores target-response character count.All baselines use the canonical labelled items and exclude the six degenerate benchmarks.Source: I BASELINES · p. 47

Results

  • Generic NOUL Jev beats the better of TF-IDF and length on 25 of 31 benchmarks, with median AUROC gain +0.132 [+0.057, +0.190].Across generic NOUL, best generic answer type, and split-half targeted questions, Jev beats the supervised baseline on 26 of 31, 30 of 36, and 33 of 38 benchmarks, respectively.Source: I BASELINES · p. 47

Calibration and thresholds

  • With a cross-validated threshold, Jev reaches median F1 0.822 versus 0.609 for TF-IDF and 0.632 for length.Fitting Jev’s threshold on 10 labelled items yields 0.793 versus 0.351 for TF-IDF trained on the same 10 items.Source: I BASELINES · p. 47, Table 33 · p. 48
  • Jev’s median ECE is 0.168 versus a 0.074 null; TF-IDF is similarly miscalibrated at 0.164, while length’s low ECE accompanies F1 of 0.235 at 0.5.The table reports medians over the 31 benchmarks with a generic NOUL.Source: I BASELINES · p. 47, Table 33 · p. 48

J REPRODUCIBILITY

The package caches the benchmark pipeline and Jev answers so metrics can be recomputed offline, and it includes validation, scripts, and files for rescoring other detectors. Independent implementations closely reproduce most released metrics, with one noted PrivacyLens harness artefact.

Pipeline and rescoring

  • The pipeline replays official scorers and checks reconstructed aggregate scores; offline rescoring rebuilds request hashes and reuses cached answers without new requests.The package is intended for scoring other detectors on identical states and labels, and for auditing benchmark labels through disagreements.Source: J REPRODUCIBILITY · pp. 49–50

Validation

  • Independent implementations match AUROC within 10^-3 for 1,719 of 1,739 strategy–file values, while F1 at 0.5 is within 0.0056 on 111 files.A PrivacyLens rubric artefact marked 77 of 298 one-secret items not applicable; recomputation over all items gives F1 0.806 rather than 0.770.Source: J REPRODUCIBILITY · p. 49

Package and intended use

  • The package includes target-model calls, reference-scorer verdicts, detection files, Jev responses, and per-strategy metrics for 44 benchmarks and 7,193 instances.It also ships corrected labels for Open-Prompt-Injection and SycophancyEval (answer) beside canonical labels.Source: J REPRODUCIBILITY · pp. 49–50

Analysis scripts

  • Analysis scripts regenerate tables and figures, including split-half selection, context ablations, calibration, costs, human agreement, and baselines.The full analysis runs offline in a few minutes on a laptop.Source: J REPRODUCIBILITY · p. 50
Loading 2609.29429v1…