From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection

1 Introduction

• Prompt injection turns untrusted text into instructions, so deployed systems must allow low-risk inputs, block high-risk ones, or defer uncertain cases to a reviewer.A score such as 0.9 is useful only if it reliably signals attacks as attacks, domains, or benign prompts change.Source: §1 · p. 1
• The study asks whether Jev's native probabilities support better security decisions than specialized classifiers with post-hoc calibration, alongside detection and gate–reviewer policy questions.Prior work covers human-authored attacks [8], indirect injection, benign hard cases [5], obfuscation and shift [7], and Jev's calibrated-decision training.Source: §1 · p. 1
• Prompts, models, calibration parameters, and gate policy are selected on development and calibration data, then frozen before confirmatory evaluation.The fixed policy maps attack probabilities to ALLOW, DEFER, or BLOCK, with only deferred inputs sent to the frozen LLM reviewer.Source: Figure 1 · p. 2
• On 47,611 confirmatory examples, native decision probabilities did not consistently outperform target-calibrated classifier probabilities, and a strict 1% FNR target required very high escalation.The paper jointly evaluates discrimination, calibration, confident false negatives, latency, and security-deferral behavior.Source: §1 · p. 2

Prompt-injection evaluation

• Prompt-injection benchmarks cover human-authored attacks [8], indirect injection, benign prompts with attack-like terms [5], and obfuscation and distribution shift [7].PIDS-Bench evaluates hard-benign inputs as well as domain and structural shifts under a frozen protocol [7].Source: §2 · p. 2

Calibration and selective prediction

• Temperature scaling adjusts logits while preserving score rankings [4], whereas selective prediction trades automatic coverage for risk [3].For security, the standard selective-risk formulation does not account for different false-positive and false-negative costs or an imperfect reviewer.Source: §2 · p. 2

Decision models

• TypeSafe presents Jev as a typed probabilistic decision model trained with RLCD; this study tests its native probabilities on a frozen external workload against classifiers calibrated on target data.The comparison does not isolate the effects of the models' underlying training objectives.Source: §2 · p. 2

3.1 Data and isolation

• The confirmatory cohort contains 41,981 attacks and 5,630 benign examples across four benchmark families, with pooled and dataset-level metrics reported for its heterogeneous, imbalanced composition.Development, gate-calibration, reviewer-calibration, feasibility, and confirmatory data are separated; models, prompts, parameters, reviewer threshold, and deployable intervals are frozen before confirmatory evaluation.Source: §3.1 · p. 3

3.2 Models

• The study evaluates Jev-native probabilities, raw and temperature-scaled Prompt Guard 2 22M and 86M scores, and a GPT-5.5 judge with temperature-scaled verbalized probabilities.The judge's FNR-first operating point is 0.0527; on reviewer calibration it yields FNR 0 and FPR 98.6%.Source: §3.2 · p. 3

3.3 Metrics

• Detection is measured with AUROC, AUPRC, TPR at 1% FPR, and default-boundary FNR/FPR; calibration uses ECE, Brier score, NLL, and false-negative confidence.For deferral, the empirical confirmatory escalation requirement is the smallest escalation rate at which observed system FNR reaches 1% or less; it is descriptive, not a deployable threshold selected on confirmatory labels. Temperature scaling fits a scalar by minimizing unweighted binary NLL.Source: §3.3 · pp. 3–4

4.1 RQ1: Detection versus decision

• The LLM judge achieves the highest pooled AUROC and AUPRC, Jev outperforms both Prompt Guard variants, and none supports autonomous blocking at low false-positive rates.Jev misses 64.9% of attacks at its native 0.5 boundary; both Prompt Guard variants miss more than 90%.Source: §4.1 · p. 4

4.2 RQ2: Trustworthiness of confidence

• Jev’s native probabilities are not consistently better calibrated than post-hoc calibrated Prompt Guard scores, while temperature scaling improves calibration without changing discrimination.Among attacks missed by raw PG2-86M, mean assigned attack probability is 0.0107; temperature scaling makes missed-attack probabilities less extreme but does not recover attacks at the 0.5 boundary.Source: §4.2 · p. 4, Figure 2 · p. 5

4.3 RQ3: ALLOW–BLOCK–DEFER policies

• At FNR ≤1%, every gate requires more than 91.8% escalation; system FPR remains 40.6%–45.8%, and small escalation differences do not establish a better gate.The empirical crossings are descriptive test-set statistics, not deployment thresholds selected using confirmatory labels.Source: Table 4 · p. 5, §4.3 · pp. 5–6, Figure 3 · p. 6

4.4 Latency and deployment trade-offs

• Local Prompt Guard models are roughly two orders of magnitude faster than the remote LLM judge, while Jev is slower than local classifiers but faster than the reviewer.The latency values characterize the evaluated combination of local and API-based inference, including network and provider overhead.Source: §4.4 · p. 6, Table 5 · p. 6

5 Discussion

• Jev’s native probabilities outperform raw Prompt Guard scores on several pooled metrics but not consistently temperature-scaled classifier scores on target calibration data.Calibration should be evaluated on the target workload rather than inferred from the model’s training objective.Source: §5 · p. 6
• Temperature scaling improves PG2’s ECE, Brier score, and NLL, but leaves AUROC, AUPRC, 0.5-boundary decisions, and uncertainty ordering unchanged.Under substantial distribution shift, calibration cannot compensate for information the classifier failed to capture.Source: §5 · p. 7
• As escalation approaches 100%, system behavior converges toward the reviewer, whose FNR-first operating point has very high FPR.The deferral curves characterize the gate–reviewer system as a whole, not the gate alone.Source: §5 · p. 7
• Because the confirmatory cohort is dominated by BIPIA attacks, pooled metrics characterize this frozen benchmark mixture rather than a natural deployment distribution.ECE and Brier score depend on evaluation distribution, while AUPRC is sensitive to class prevalence; dataset-level calibration is reported separately.Source: §5 · p. 7

6 Threats to Validity and Limitations

• Jev was evaluated with one fixed question and API version, while the LLM judge’s verbalized probabilities are not token-level likelihood probabilities.Different wording or later Jev versions may yield different results.Source: §6 · p. 7
• The confirmatory mixture does not reflect natural production prevalence, and its pooled calibration and cost results should not transfer directly to deployment settings.The illustrative cost model should be replaced with application-specific costs and review budgets.Source: §6 · p. 7
• Single-class subsets make some class-conditional metrics undefined, and latency was measured across local and API inference without controlled-hardware normalization.Only metrics that remain meaningful, such as false-positive behavior, are reported for those subsets.Source: §6 · p. 7
• PG2 is temperature-scaled on target calibration data, whereas the primary Jev evaluation uses native probabilities, leaving Jev’s post-hoc calibration untested.A Jev calibration ablation would help distinguish native-calibration transfer from its ability to be calibrated after adaptation.Source: §6 · p. 7
• Point estimates lack confidence intervals, benchmark transformations may induce dependence, and the evaluation measures detection and routing rather than end-to-end compromise or downstream harm.Small numerical differences, particularly sub-percentage-point RQ3 differences, should not be treated as statistically resolved.Source: §6 · p. 7

7 Conclusion

• Every tested gate reached observed FNR ≤1% only after more than 91.8% of inputs were escalated, while still producing substantial false positives.These crossings are descriptive test statistics, not deployment thresholds selected using confirmatory labels.Source: §7 · p. 7
• Jev ranks attacks better than Prompt Guard 2 but is not consistently better calibrated than temperature-scaled PG2-86M; the LLM judge has strongest discrimination and overall probability estimates at higher latency.The evaluation covers 47,611 held-out examples.Source: §7 · p. 7
• Training for calibrated outputs alone is insufficient for reliable deployment decisions; detection, calibration, reviewer behavior, and policy utility need joint evaluation.Source: §7 · p. 7

A Dataset-level Calibration

• Table 6 reports calibration by dataset for Jev and PG2-86M.Source: Table 6 · p. 8
• The table’s scope is dataset-level calibration for the two named models.Source: Table 6 · p. 8
• Single-class datasets retain valid proper-score summaries but not all discrimination metrics.Source: Table 6 · p. 8

• 1.590979 is the scalar temperature for reviewer prompt llm-judge-candidate-effect-v1.Source: B Frozen Policy Artifacts · p. 8
• 0.052738 is the frozen calibrated-probability threshold for the reviewer decision rule.Source: B Frozen Policy Artifacts · p. 8
• Gate ALLOW/DEFER/BLOCK intervals are selected only on the 2,000-example gate calibration split.Source: B Frozen Policy Artifacts · p. 8
• Primary security–deferral curves use the frozen reviewer rule; oracle and fail-closed review outcomes are ablations, not deployed-system performance.Source: B Frozen Policy Artifacts · p. 8

• 47,611 aligned records under exact-ID joins supply reported rows for the listed model predictions.Source: C Reproducibility Artifacts · p. 8
• The release includes paper metrics.json, a compact security deferral.json reference, and CSV and figure outputs.The compact reference contains primary endpoints and operating-point ablations.Source: C Reproducibility Artifacts · p. 8
• Full security–deferral curves are regenerated offline from bundled prediction artifacts rather than stored as a large reference JSON.Source: C Reproducibility Artifacts · p. 8
• Scripts/reproduce all.sh regenerates the CSV and figure outputs.Source: C Reproducibility Artifacts · p. 8