Source-linked AI summary
Hard-Gate Candidacy in a Deployed Validator Suite
Xin Xu
TL;DR
Validator gates need evidence that their firings distinguish outputs that work from those that do not. This paper screens 13 checks against downstream outcome labels and examines execution skips. Two checks survive multiplicity correction, while the study also identifies an execution-related ceiling on detection by checks requiring a live artifact.
Problem
The paper measures validator behavior in deployed agent gates, a setting where checks are heterogeneous, failures infrequent, and artifacts freshly generated.
Source: §1 · p. 2Method
The authors screen each check's marginal separation of outcome classes using Newcombe intervals, Fisher exact tests, and Holm correction.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Validator inventories alone do not establish gate performance; records should distinguish checks that pass from checks that did not run.
Source: §1 · p. 2, §6 · p. 5The screen measures marginal separation, not a check's incremental contribution within the deployed ensemble, which would require the joint firing matrix and gating rule.
Source: §8 · p. 6
Abstract
from arXiv · showhide
Before a validator can be promoted to a hard gate on a deployment pipeline, it has to be shown that its firing separates outputs that reach users in working order from those that do not. We run that screen on 13 validators in a deployed generative agent, against 550 runtime and 350 static builds labelled by downstream outcome, and report each check's marginal separation $J=\mathrm{TPR}-\mathrm{FPR}$ with Newcombe intervals and Fisher exact tests. Two checks survive correction for multiple comparisons, two more are nominal only, and the remaining nine are not distinguishable from zero, three of them because they never fired on any sampled build. Execution itself is not random with respect to the property being gated, and this replicates: across four runs covering 1,867 builds and ten distinct runtime checks, probes were skipped on 144 of 895 broken builds and 1 of 972 acceptable builds (per-run rates 15.6% to 16.6% against at most 0.3%), every skip carrying the same unsafe-to-probe reason. Because a skipped check is recorded as a pass, this imposes a ceiling that no check quality can lift: a check that needs a live artifact cannot operationally detect more than about 84% of broken builds in this harness. For the one check with construct-specific labels, a detector built for blank output fires on 0 of 90 human-labelled blank builds (95% upper bound on sensitivity 3.3%), and the global frame statistic it approximates separates the classes only weakly (AUC 0.59), so the gap is not a threshold that needs tuning. The same gap appears one layer up: on a census of tens of thousands of judge-scored builds, 32.5% of rejections carry no recorded issue at all. We argue that evaluation records must distinguish a check that ran and passed from one that did not run, must carry the evidence for a rejection, and that an inventory of checks is not evidence about a gate.
1 Introduction
Deployed agent validators leave terse verdicts, but their use as hard gates requires evidence that they separate builds by downstream outcome. This paper screens that marginal separation and examines one-sided execution and construct-specific detection.
1 Introduction
- The screen asks whether firing separates downstream outcomes, not whether a check detects its named construct or adds value to an existing ensemble.Those are distinct questions requiring construct labels or the joint firing distribution and deployed gating rule.Source: §1 · p. 1
- Existing work surveys oracle problems, measures whether executed computation reaches an oracle, and documents pseudo-tested code; behavior in deployed language-model-agent gates remains unmeasured despite heterogeneous checks, low failure base rates, and freshly generated artifacts.Source: §1 · p. 2
- Two validators survive multiplicity correction, while nine show no detectable separation, including three that never fired.The screen covers 13 checks and uses Newcombe intervals, Fisher exact tests, and Holm correction.Source: §1 · p. 2
- Across four runs, 15.6–16.6% of broken builds were skipped versus at most 0.3% of acceptable builds, imposing a harness-level sensitivity ceiling on live-artifact checks.The skips were recorded as passes and carried the same unsafe-to-probe reason.Source: §1 · p. 2
- For the check with construct-aligned labels, the authors report zero detections among 90 human-labelled instances.They also report that the global frame statistic approximated by the detector is a weak discriminator.Source: §1 · p. 2
- The authors recommend recording per-check applicability so a check that ran and passed is distinguishable from one that did not run.The record prescription accompanies the call to assess checks by their demonstrated separation rather than by inventory alone.Source: §1 · p. 2
2 Setting, Criterion, and Outcome Proxy
The study screens validators in a deployed generative agent using downstream outcome proxies. It compares per-check firing across labelled builds and summarizes separation with intervals and tests.
2 Setting, Criterion, and Outcome Proxy
- Hard-gate candidacy is screened by whether a check's firing separates outcome classes, not by construct accuracy or incremental contribution to the deployed ensemble.Measuring incremental contribution would require the joint firing distribution and gating rule.Source: §2 · p. 2
- Broken and acceptable builds are defined by judge outcomes, publication, and complaints, so the labels are downstream proxies rather than verified ground truth.The labels are independent of each individual validator's verdict.Source: §2 · p. 2
- The runtime sample contains 550 builds and the static sample 350; both lanes include broken and acceptable outcome labels.The runtime sample has 235 broken and 315 acceptable builds; the static sample is balanced at 175 each.Source: §2 · p. 2
- Separation is measured as J = TPR − FPR with Newcombe 95% intervals and Fisher exact tests adjusted by Holm across 13 checks.Skipped checks count operationally as not firing; J is a separation screen, not a cost-sensitive operating-point recommendation.Source: §2 · p. 2
- Figure 1 displays per-check marginal separation with Newcombe intervals and distinguishes corrected or unresolved checks from nominal-only results.The three never-fired checks have finite rather than degenerate intervals.Source: Figure 1 · p. 3
3 The Hard-Gate Screen
The screen finds two checks that separate outcome classes after multiplicity correction, while other checks are nominal, indistinguishable from zero, or never observed firing.
3 The Hard-Gate Screen
- Script-error and engine-metadata separate the outcome classes after multiplicity correction.Version-pinning and manifest–DOM are nominal results only; the latter's interval excludes zero but its Fisher test does not.Source: §3 · p. 3
- Six checks are not distinguishable from zero: three fire on nearly every build, while three minority-firing checks have rates too similar to resolve at this sample size.A larger sample could resolve the minority-firing group, but not checks that fire on everything.Source: §3 · pp. 3–4
- Three checks never fired in their sampled builds, so their finite intervals bound firing rates without establishing that those rates are zero.These were document-language in 350 static builds and blank-frame A and B in runtime builds on which they ran.Source: §3 · p. 4
- At 10% failure prevalence, script-error would flag more false alarms than true ones: 4.3% versus 2.7% of builds.This concerns alarm precision, not the sign of the class-conditional separation statistic J.Source: §3 · p. 4
4 One-Sided Execution
Across four runs, probes were skipped far more often on broken builds than acceptable ones, always for the same unsafe-to-probe reason. Because skips count as non-firings, live-artifact checks face a harness-imposed ceiling on operational sensitivity.
4 One-Sided Execution
- Every runtime check except script-error was skipped on 40 of 550 builds, while no static check was skipped.The static lane needs only source text.Source: §4 · p. 4
- 144 of 895 broken builds were skipped, versus 1 of 972 acceptable builds across four runs; every skip carried the same unsafe-to-probe reason.Run C showed the same pattern with four probes disjoint from those screened in Table 1.Source: Table 2 · p. 4, §4 · p. 4
- Counting skips as non-firings makes conditional-on-running separation weakly higher than operational separation, while verdict-only records cannot distinguish the rates.For blank-frame A and B, both rates coincide at zero.Source: §4 · p. 4
- 0.83–0.84 is the maximum operational sensitivity for any live-artifact check in this harness, regardless of its logic.The ceiling follows because skips count as non-firings and the run-time true-positive rate cannot exceed 1.Source: §4 · p. 4
5 Construct Validity Where Target Labels Exist
Construct-specific labels show that the blank-frame detectors missed every human-labelled blank build. A global frame statistic also separates blank from acceptable output only weakly, so the reported gap is not a threshold-tuning problem.
5 Construct Validity Where Target Labels Exist
- The 14 independently confirmed blank builds in the screen triggered neither blank-frame detector.The detectors are intended to identify blank or single-colour output.Source: §5 · p. 5
- 0 of 90 human-labelled blank builds triggered the detector, with a 95% upper bound on sensitivity of 3.3%.The 239-build labelled set was reported separately from the 14 audited cases because their overlap could not be established.Source: §5 · p. 5
- Mean frame luminance has AUC = 0.59 (95% CI [0.52, 0.67]), and its best single threshold reaches only TPR = 0.84 at FPR = 0.66.Blank builds have lower median luminance than acceptable ones, 37.1 against 57.3, but the distributions overlap heavily.Source: §5 · p. 5
6 Implications for Evaluation Records
Verdict-only records conceal whether checks ran, passed, never fire, or fire constantly, and judge rejections often lack a recorded issue. The paper calls for applicability and rejection evidence to be retained, alongside periodic screening of check separation.
6 Implications for Evaluation Records
- A verdict-only pass looks identical whether checks ran and found nothing, did not run, never fire, or fire on everything.The resulting aggregate claim that the suite passed is not falsifiable from the record.Source: §6 · p. 5
- 32.5% of judge rejections carried no recorded issue, while acceptance with a recorded CRITICAL issue was rare at 0.1%.The census covered tens of thousands of scored builds; the record therefore does not reconstruct the reject-side verdict.Source: §6 · p. 5
- Records should attest per check and item whether it was applicable, distinguishing fired, ran without firing, and did not run.A periodic screen with intervals and multiplicity control turns a check inventory into evidence about what the suite separates.Source: §6 · p. 5
- Even with full internal access, nine of thirteen checks showed no detectable separation, two more were nominal-only, and four were skipped on builds most needing checks.The paper argues that validator inventories alone are not evidence in deployment and assurance arguments.Source: §6 · p. 5
7 Related Work
Related work frames validator evaluation through testing oracles, pseudo-tested code, selective execution, and failures in evaluation pipelines. This study distinguishes observational production screening from intervention-based testing and places its execution mechanism opposite gray failure.
7 Related Work
- Intervention-based testing mutates source or injects faults to see whether checks react, whereas this study observes whether production firings separate delivered outcomes.The approaches answer different questions; intervention presupposes access and harness control to seed faults.Source: §7 · p. 6
- Prior work on judges, supervision systems, verifiers, and error detectors reports audits and benchmarks, while moderation studies differ in dataset curation and disclosed class ratios.The cited work includes pooled recall near 18% for one deployed judge gate [Zhang et al., 2026].Source: §7 · p. 6
- Selective-verification corrections assume the index test is observed and the reference standard is selectively missing, unlike this study’s complete reference standard and severity-keyed missing checks.The paper therefore reports operational rates and bounds conditional ones rather than applying a verification-bias correction.Source: §7 · p. 6
- The paper distinguishes checks that do not fire or run from silent failures where a monitor runs but is deceived, and develops an opposite pole to gray failure.Gray failure concerns differential observability for subtle degradation [Huang et al., 2017].Source: §7 · p. 6
8 Limitations
The study estimates per-check marginal separation, not incremental contribution within the deployed ensemble, which would require the joint firing distribution and gating rule. Construct validity is established for only one check, while proxy labels and a single deployment window further bound inference.
8 Limitations
- Without the joint firing matrix and gating rule, marginal separation cannot determine a check’s incremental contribution within the deployed ensemble.High-J checks may be redundant, while low-J checks may cover disjoint rare failures.Source: §8 · p. 6
- Only one check has construct-specific labels; establishing validity for the remaining twelve requires seeded failures or target labels.A check may fail the outcome-separation screen while correctly detecting a condition that does not separate the outcome classes.Source: §8 · p. 6
- Outcome labels rely on the deployment judge and complaint absence, with no uniform complaint window; human blank labels lack protocol, rater-count, and agreement details.Builds from near the end of the window had less complaint exposure, and retries can create multiple builds per request; intervals do not use cluster-robust inference.Source: §8 · p. 6
- The study covers one deployment, convenience sampling, and one short window; the four skip runs share a harness and window and may overlap.The reported rates are not claimed to characterize validator suites generally.Source: §8 · p. 6