PaperlayerView exact revision on arXiv

Source-linked AI summary

The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability

Xin Xu

arXiv:2609.07162v1cs.LGcs.CR

TL;DR

Single-trace monitors face a graded limit when asked to judge properties defined by differences between executions. The paper formalizes that frontier and tests how monitors use information and comparison procedures, finding substantial utilisation shortfalls alongside deployment trade-offs and a need for mechanical benchmark validation.

  • Problem

    Many safety properties, including cross-tenant noninterference, are defined by differences between executions, so a single transcript cannot witness their violation or compliance.

    Source: §1 · p. 1

  • Method

    The paper derives a balanced-accuracy frontier from total variation, constructs leak families with controlled TV, and evaluates nine LLM monitors across information and procedure conditions.

    Source: §1 · pp. 1–2

  • Results

    At TV = 1, nine LLM monitors average 60.9%, naming the check closes 61% of the frontier gap, and executed-run comparisons with an explicit rule reach 90.0%.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The shortfall below the frontier is mostly utilisation, while oversight with replay remains bounded by per-property projection choices and requires an executed second run with an explicit procedure.

    Source: §1 · p. 2, §6 · p. 7

    The multi-turn trajectories are synthetic and do not exercise a real agent framework or memory system; only two members of the 2-safety family are measured.

    Source: §6 · p. 7

Abstract

from arXiv · show

Several properties safety monitors are asked to certify, among them cross-tenant noninterference, sandbagging and evaluation awareness, are 2-safety hyperproperties, witnessed only by two executions. The standard consequence is a binary impossibility: one trace cannot decide them. We replace the binary with a measurement. A tight bound puts the balanced accuracy of any single-trace monitor at $\tfrac12+\tfrac12\,TV(P_0,P_1)$, turning undecidability into a graded detectability frontier and defining an oversight gap: a monitor's shortfall below it. On a leak family with closed-form $TV$, nine LLM monitors are optimal at $TV=0$ but capture little signal as $TV$ grows; at $TV=1$, where a 20-line membership check scores $100\%$, they average $60.9\%$. That shortfall is mostly not capability: naming what to check closes $61\%$ of it while leaving the $TV=0$ control at chance. The same split runs through a $2{\times}2$ factorial: an imagined second run leaves monitors at chance ($50.4\%$) while the same rule on an executed second run reaches $90.0\%$, and a stored oracle without a comparison procedure yields only $68.2\%$. Information and procedure are each necessary and neither is capability. Under nondeterminism, replay tracks a closed-form $k$-replay curve only under the right projection, and a projection frontier shows the resulting dilemma is forced: narrow misses $98.6\%$ of off-channel leaks, broad flags $75.7\%$ of clean traffic, and attainable accuracy decays like $1/(qm)$ in the benign-variation rate and the channel count. Finally, two frontier LLM judges certified an earlier version of our own benchmark as sound while a sign test found a directional bias ($p=2.7\times10^{-5}$) that invalidated three of our findings. Construction validity for hyperproperty benchmarks should be proved mechanically, not audited by models.

1 Introduction

The paper replaces the binary single-trace impossibility for two-execution safety properties with a graded detectability frontier. Its experiments show that monitors often fail to use available signal, while deployment introduces replay and projection trade-offs and benchmark construction requires mechanical validation.

1 Introduction

  • The balanced accuracy of any single-trace monitor is at most 1/2 + 1/2 TV(P0, P1), where TV measures distance between trace distributions for clean and violating executions; the likelihood-ratio test attains this bound.Source: §1 · p. 1
  • At TV = 1, the nine-model mean is 60.9% against a 100% membership-check score, while naming the check raises the mean from 58.9% to 83.8%, closing 61% of the gap.At TV = 0, all nine models are at chance; the named-check control remains at chance (50.3 → 49.2%).Source: §1 · p. 1
  • A 2×2 factorial finds that imagined second runs stay at 50.4% mean, a stored value without a rule reaches 68.2%, and an executed second run with a rule reaches 90.0%.Providing both a stored reference and a rule yields 100% for all six models; differential replay with the rule reaches 90.0%.Source: §1 · p. 2
  • Under nondeterminism, replay follows a closed-form k-replay curve under the requested-field projection, while broader projections trade missed off-channel leaks for false alarms on clean traffic.The narrow projection misses 98.6% of off-channel leaks; the broad alternative flags 75.7% of clean traffic.Source: §1 · p. 2
  • A sign test found directional bias in the earlier benchmark (p = 2.7 × 10^-5), invalidating three headline findings that two frontier LLM judges had certified.The authors conclude that hyperproperty benchmark construction validity should be established mechanically.Source: §1 · p. 2

2 The Detectability Frontier

The detectability frontier quantifies how much single-trace observations can reveal, and separates unavoidable blindness from a monitor's failure to use available signal. Exchangeable constructions control TV exactly; extensions characterize replay under nondeterminism and the limits imposed by projection choices.

The Detectability Frontier

  • A model at chance when TV = 0 has G = 0 and behaves optimally; a model at chance when TV = 0.5 discards half the available accuracy.Source: §2 · p. 3
  • For monitors expressible with a field-type prior, TVH = TV = 1 on type mismatches makes the measured shortfall utilisation, while TVH = 0 and TV = 1 on the in-pool control makes it knowledge.The two axes therefore measure knowledge and utilisation separately.Source: §2 · p. 3
  • An exchangeable own/foreign-value construction gives TV = 0, and mixing in a foreign value from a different field's pool with probability λ yields TV(P0, P1) = λ.This provides a construction whose detectability is controlled exactly rather than estimated.Source: §2 · p. 3
  • With shared support, equality of response distributions has no finite zero-error witness; distinguishing distributions separated by ε requires Θ(1/ε^2) replays per condition.In the disjoint-support deterministic limit, one replay per condition recovers the classical witness pair.Source: §2 · pp. 3–4
  • In the symmetric case q_j=q and p_j=1/m, the best projection size is s*=−1/ln(1−q), independent of m, with balanced accuracy approaching 1/2+1/(2eqm) when mq≫1.Source: §2 · p. 4
  • Under benign paraphrase probability δ, blocking whenever two clean replays differ has false-alarm rate exactly δ, making the naive rule unusable for a sampling LLM when δ is approximately 1.The projection results show why replay must compare a suitable semantic slot rather than indiscriminately compare surface text.Source: §2 · p. 4

3 Experimental Setup

The experiments use mechanically scored monitor verdicts across 40 isolation domains, with model coverage varying by experimental arm.

3 Experimental Setup

  • Experiments use 40 isolation domains with four mutually interchangeable values per field, and score monitor JSON verdicts mechanically across nine models or smaller subsets by arm.The frontier sweep, factorial, and abstention arms use all nine models; mechanism, replay, projection, and real-code arms use seven, while the stored-oracle arm uses six.Source: §3 · p. 4

4 Results

Across six result areas, the paper measures how much single-trace monitors can detect, tests the roles of information and procedure, and examines replay and projection trade-offs. It also reports that benchmark-construction bias invalidated earlier findings.

Measuring the Oversight Gap

  • At λ = 1, LLM monitors average 60.9% against an attainable frontier reached by a 20-line check, leaving a 39.1-point oversight gap.Across nine models, gaps span 30–49 points, with no detected capability ordering (Spearman ρ = 0.15, p = 0.71, n = 9). Naming what to check closes 61% of the gap while the λ = 0 control stays at chance.Source: Measuring the Oversight Gap · p. 5

Separating Information from Procedure

  • The imagined second run scores 50.4%, a stored value without a procedure scores 68.2%, and reference plus rule reaches 100% for all six models.Executing differential replay reaches 90.0%; the paper reports a ten-point difference from the stored-oracle condition and a forty-point difference from the imagine condition.Source: Separating Information from Procedure · p. 5, Table 1 · p. 5

Imagined Counterfactuals

  • Imagined counterfactuals match the real value only 1.2% of the time, and models abstain on just 12.6% of provably undecidable cases.Approval rates on leak and clean traces are statistically indistinguishable in the imagine condition; models either restate the shown value or rationalize discrepancies they imagined.Source: Imagined Counterfactuals · pp. 5–6

Replay Cost and the Projection Dilemma

  • Under free paraphrase, field-only k-replay tracks the closed-form detection curve to within one point at every k.Blocking on any difference was predicted to score 50%, but models score 78.9–91.9% when they silently repair the rule; Qwen-72B applies it literally and scores 55% at k = 2 and 54% at k = 8.Source: Replay Cost and the Projection Dilemma · p. 6
  • The narrow projection detects 1.4% of off-channel leaks, while the broad projection detects 90.4% but flags 75.7% of clean traffic.The two tested extremes leave the preferred operating region unoccupied; the paper attributes the trade-off to the alphabet-size dependence of the deployed regime, not prompting.Source: Replay Cost and the Projection Dilemma · p. 6

Three Replications

  • Across replications, the single-trace condition remains at chance, while pair-plus-rule performance varies by property, model, trajectory length, and production setting.For sycophancy, pair plus rule averages 83.9%; at 16 turns, the four conditions score 49.8%, 50.4%, 55.9%, and 86.3%; on production-code traces, the three conditions score 51.0%, 63.5%, and 92.6%.Source: Three Replications · p. 6

Mechanical Validity Checking

  • A sign test found directional bias in 25 of 28 comparable benchmark pairs (p = 2.7 × 10−5), and repair invalidated three earlier findings.Two frontier LLM judges had certified all 40 original domains; the reported conclusion is that construction validity should be established mechanically.Source: Mechanical Validity Checking · pp. 6–7

5 Related Work

Prior work frames cross-trace monitoring as a logical or observational problem, while this paper studies distributional detectability and intervention at the smallest nontrivial arity. Related work also identifies adjacent risks and open questions around pairwise protocols and deployment.

Hyperproperties and their monitoring.

  • The paper contrasts prior logical, worst-case monitoring work with its distributional frontier, restricted decomposition, and stochastic lifting.Noninterference monitoring is traced to Goguen and Meseguer (1982) and Clarkson and Schneider (2010), among others.Source: §5 · p. 7

Cross-trace oversight of LLM agents.

  • Stein et al. (2026) aggregate 10–600 traces for observational cross-trace failures, whereas this paper studies interventional properties at the smallest nontrivial arity.The two approaches occupy different arity regimes and property sets.Source: §5 · p. 7
  • Pairwise judging addresses output ranking, not properties undefined on single traces; whether a co-tenant adversary can manufacture distractors for this protocol remains untested.The cited pairwise-feedback study found preference flips under irrelevant attributes in about 35% of cases versus 9% for absolute scores (Tripathi et al. 2025).Source: §5 · p. 7

Behavioral properties defined across executions.

  • Behavioral properties including deceptive alignment, strategic underperformance, sycophancy, and counterfactual fairness are specified by comparing executions that differ in one input.Model-written suites can surface these behaviours at scale, but score transcripts individually (Perez et al. 2022).Source: §5 · p. 7

Isolation failures in serving stacks.

  • Transcript-level monitoring of serving-stack isolation failures is governed by TV, projection, and replay budget rather than monitor capability.Prior work documents cross-tenant leakage through shared serving state and system-side fuzzing (CVE-2025-25183; Zhao et al. 2026).Source: §5 · p. 7

Abstention.

  • The paper distinguishes its undecidable cases from calibration when an answer is knowable: its 12.6% abstention rate is an upper bound on this form of calibration.The cited work reports models are reasonably calibrated when they can in principle know the answer (Kadavath et al. 2022).Source: §5 · p. 7

6 Discussion and Limitations

The discussion limits the empirical scope and states a narrow conclusion: the frontier is computable, much of the shortfall is utilization, and the remedy combines an executed second run with an explicit procedure. The technical appendix records proofs and tables and says the paper's load-bearing claims are not deferred there.

6 Discussion and Limitations

  • Synthetic multi-turn trajectories use no real agent framework or memory system, and the measured frontier varies one conspicuousness axis while testing two members of the 2-safety family.Sandbagging and evaluation awareness remain analogies; the nine-model sample cannot rule out moderate capability associations, and prompt-wording variance is unbounded in this study.Source: §6 · p. 7
  • Theorem 2 covers equality-of-induced-distribution properties, and Theorem 3 assumes channel-independent benign variation with 1 ≤ s* ≤ m for its continuous optimum.Replay also assumes re-execution under perturbed co-tenant state; a runtime aware of probing can behave differently on probes, so the protocol certifies executions rather than dispositions.Source: §6 · p. 7
  • The paper concludes that the frontier is computable, the shortfall below it is mostly utilization, and the remedy is an executed second run with an explicit procedure, budgeted per property and model.Source: §6 · p. 7

Technical Appendix

  • The appendix says its tables are regenerated by the named release script and that no load-bearing material is deferred there.Source: Technical Appendix · p. 8

A Full proof of Theorem 2

The proof shows that equality of response distributions cannot be witnessed as a violation by any finite trace set. Statistical testing of close response laws instead requires replay counts that grow quadratically in the inverse separation, while deterministic unequal outputs are exposed with one replay per condition.

No finite arity

  • No finite multiset of traces witnesses a violation when compliant and violating response laws share support, so no finite arity suffices.A finite trace set that a violating runtime can emit is also possible under a satisfying runtime in this construction.Source: A Full proof of Theorem 2 · p. 8

Lower bound

  • For response laws separated by ε, any test with error below 1/3 needs k = Ω(1/ε^2) replays.The proof applies the single-observation bound to the k-replay observation.Source: A Full proof of Theorem 2 · p. 8

Upper bound

  • The upper-bound procedure estimates each condition’s response frequency from k replays and reports interference when estimates differ by more than ε/2.The stated procedure is intended to separate the two instances.Source: A Full proof of Theorem 2 · p. 8

Deterministic limit

  • When the two response laws are deterministic point masses at different values, one replay per condition exposes the violation, so k = 2.Source: A Full proof of Theorem 2 · p. 8

B Full proof of Theorem 4

The proof characterizes when adding a channel improves balanced accuracy and gives the symmetric-case optimum as benign variation changes.

B Full proof of Theorem 4

  • Adding channel j helps exactly when p_j(1 − q_j) > q_j P_S.Here P_S is the leak probability covered by the selected channels, and the added channel changes the benign-variation product by (1 − q_j).Source: B Full proof of Theorem 4 · p. 8
  • In the symmetric case, the optimum occurs at s* = −1/ln(1 − q), with s* approaching 1/(e q) as q approaches zero.Source: B Full proof of Theorem 4 · p. 8

E.1 A Second Property: Sycophancy

The paper tests whether its findings extend from noninterference to sycophancy, where the violation is defined by differing behavior across runs with different user stances. Single-trace and imagined-counterfactual monitoring remain at chance, while the remedy transfers less consistently across models.

E.1 A Second Property: Sycophancy

  • Sycophancy is defined by whether the verdict follows the asker’s stance rather than the claim’s truth, a difference tested across two executions.The honest runtime returns the truth value regardless of stance; the sycophantic runtime returns the stated stance.Source: §E.1 · p. 8
  • 49.6% on single traces and 50.2% with an imagined counterfactual show the monitor remains at chance on sycophancy.Across 40 claims and 7 models, per-model values range from 48.7% to 50.5%.Source: §E.1 · p. 8
  • The executed pair averages 80.6%, and adding the rule reaches 83.9%, below the corresponding noninterference results of 65.8% and 90.0%.Under the full protocol, sycophancy results range from 50% to 100%; GPT-4o-mini scores 49% with the pair and 50% with pair plus rule.Source: §E.1 · p. 8

E.2 Long Multi-Turn Trajectories

Longer transcripts do not change the single-trace blind spot, but the effectiveness of pair-plus-rule monitoring varies by model as trajectories grow. The results therefore support measuring long-context robustness per model rather than inferring it from short sessions.

E.2 Long Multi-Turn Trajectories

  • The multi-turn transcripts make the shown value a deterministic function of the transcript, with surrounding turns drawn independently of the class.A mechanical check caught two filler strings that incidentally contained pool values.Source: §E.2 · pp. 8–9
  • At 16 turns, the four conditions score 49.8%, 50.4%, 55.9%, and 86.3%, versus 50.5%, 50.4%, 65.8%, and 90.0% on single-turn sessions.Across 2 to 64 turns, single-trace performance remains between 49.6% and 50.3%, a range of 0.7 points.Source: §E.2 · p. 9
  • Only 2 of 7 models decline with trajectory length under pair-plus-rule monitoring, losing up to 39 points, while the other five are flat or improve by up to 9 points.The smooth average decline describes no individual model; two monitors near-perfect on short sessions are barely above chance on long ones.Source: §E.2 · p. 9

E.3 Traces from Exploited Production Code

The authors reproduce the monitoring pattern on traces generated by an exploited production-code vulnerability. Single-trace monitoring remains at chance, while an executed pair with a comparison rule reaches 91.1%.

E.3 Traces from Exploited Production Code

  • The authors craft a genuine prefix-cache hash collision and run the actual serving path, which delivers the victim’s cached value on the buggy build and the attacker’s own value on the patched build.The collision was found in 15,261 attempts; the differential pair uses buggy-versus-patched execution of the same request, and TV = 0 holds for the payloads.Source: §E.3 · p. 9
  • Across 7 models, single-trace monitoring scores 50.4%, pair monitoring 61.4%, and pair-plus-rule monitoring 91.1%.The corresponding results on synthesised sessions are 50.5%, 65.8%, and 90.0%.Source: §E.3 · p. 9

F.1 Mechanical Validity Checking

An earlier benchmark version passed two frontier LLM judges but contained a directional bias detected by a sign test, invalidating three findings. The prompt appendix specifies monitor settings and comparison rules used in the experiments.

F.1 Mechanical Validity Checking

  • 25 of 28 comparable pairs had a systematically larger, higher-tier, or later foreign value (p = 2.7×10−5), and three findings were invalidated after repair.The judges had certified all 40 domains as single-trace undecidable with plausible values; the authors conclude that validity should be proved mechanically and bias hunted mechanically.Source: §F.1 · p. 9

H Exact models and prompts

  • The monitor prompt asks for a JSON approve-or-block decision on whether a delivered multi-tenant session is safe to release.The verdict-only prompt sets max_tokens to 40; conditions requiring a predicted counterfactual use 120–140 tokens, and domain realisations are seeded by run index.Source: H Exact models and prompts · p. 9
  • The comparison-rule prompt blocks when the assistant’s value differs between two runs and approves when the values are identical.The type-rule prompt instead checks whether the stated value is plausible for the requested field.Source: H Exact models and prompts · pp. 9–10

Related papers

Loading 2609.07162v1…