PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Self-Decidable: LLMs Cannot Draw the Boundary of What an Agent Verifier Can Check

Anthony Rhodes

arXiv:2610.04699v2cs.AIcs.CLcs.CR

TL;DR

Agent verifiers must route each predicate to a fixed check or a judge, yet models cannot reliably identify that boundary on externally imposed regulatory rules. CoVer uses model agreement to nominate predicates and admits them only after their checks pass intervention-based tests; the measurements show high agreement on clear cases but conflicting errors on regulatory text. The authors conclude that a verifier must construct its boundary through execution rather than asking a model to decide it.

  • Problem

    Agent verifiers repeatedly route predicates from externally imposed rules without human review, but whether a model can identify which criteria need a judge remains unmeasured and unreliable.

    Source: §1 · pp. 1–2

  • Method

    CoVer uses agreement across independent models to nominate checkable predicates, then admits them only when synthesized checks pass intervention, field-control, and deterministic-rewording tests.

    Source: §1 · p. 3, §8 · p. 7, §8.3 · p. 9

  • Results

    Models agree at κ 0.92–0.99 on unambiguous cases but fall to κ 0.20–0.50 on regulatory predicates, with errors running in opposing, domain-dependent directions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A verifier must establish its scope by executing checks rather than trusting a model’s routing label or asking another judge to ratify it.

    Source: §1 · p. 3, §9 · p. 9

    The agent corpus covers 73 predicates from one deployed credit agent, and interventional evaluation uses 36 synthetic action records; regulatory ground truth is an LLM-assigned proxy.

    Source: §9 · p. 9

Abstract

from arXiv · show

A verifier for an agent faces rules of two kinds: the ones a fixed check can settle and the ones that require a judge. A team that derives its own checks fixes that split up front. Where the requirements come from outside, as in finance, healthcare and law, the agent enforces rules it did not write, so the split falls to runtime, recurring for every predicate of every rule on every action at a rate no reviewer can audit. Every escalation scheme assumes a model can make that decision itself, that it is self-decidable. Across six corpora, including the EU AI Act, FINRA guidance and a deployed credit agent, we collect roughly 22,000 labels from four models built by three labs. They agree almost perfectly where the answer is obvious and collapse on regulatory text; their errors run in opposite directions, so no model can be trusted as the conservative choice; and on the deployed agent's own rule-set they err together, over-claiming that a fixed check will do, the direction that never gets escalated. We introduce CoVer (corroborate-then-verify), which treats unanimity as a nomination, admitting a predicate only when the check synthesized for it survives intervention, reading fields the agent cannot write and holding under deterministic rewording. That gate rejects most of what corroboration wrongly admits, at a cost in coverage we report rather than tune away. The obvious alternative, agreement with a reference judge, certifies nothing: it climbs from 30% to 77% across calibration bands while the genuinely decidable share does not move, because a judge drawn from the population under indictment ratifies the blind spot it shares. Self-decidability is not a capability to elicit from a model but a boundary the verifier must construct.

1 Introduction

Agent verification must route each predicate to a fixed check or a judge, but models cannot reliably make that routing decision on externally imposed regulatory rules. The paper measures this failure and proposes CoVer, which uses model agreement only to nominate predicates before testing the synthesized checks.

1 Introduction

  • Externally imposed rules make routing an action-level compliance decision; a misrouted judgment predicate can lead an unsupervised system to certify compliance it never assessed.The requirements are not chosen or rephrased by the deploying party, and compliance applies to each action.Source: §1 · p. 2
  • κ 0.92–0.99 on unambiguous cases falls to κ 0.20–0.50 on regulatory predicates, where four models disagree in opposing, domain-dependent directions.The contested region covers 61% of regulatory-text predicates; the model most cautious on regulatory text can be the most reckless on agent rules.Source: §1 · p. 2, Figure 1 · p. 2
  • Because escalation schemes depend on a reliable routing decision, an over-claimed predicate can bypass escalation and silently automate a judgment call.Routing recurs for every predicate of every rule on every action, without a human in the loop.Source: §1 · pp. 1, 3
  • CoVer admits a predicate only after agreement nominates it and its synthesized check survives intervention, reads fields the agent cannot write, and holds under deterministic rewording.Agreement alone is 97–99% precise where model errors differ, but only 83% precise on the agent’s own rules, where errors move together.Source: §1 · p. 3
  • The study labels 832 predicates from six corpora with four models across three labs, using four rubric variants and two perturbation instruments for roughly 22,000 model calls.The measurements locate disagreement in the consequential middle rather than at clear endpoints.Source: §1 · p. 3

2 Related Work

Prior work distinguishes criteria that fixed checks can settle from those requiring judgment, but often assumes the boundary is drawn offline. This paper focuses on runtime routing and tests whether model disagreement is merely broad unreliability, undirected error, or underspecification.

2 Related Work

  • The paper argues its question is prior to judge accuracy: whether a model can identify which criteria need a judge at all, rather than whether its judgments are correct.Existing surveys document noisy, position- and verbosity-biased, and domain-sensitive judges.Source: §2 · p. 3
  • Unlike hand-applied benchmark guidelines, this work studies runtime routing without a designer in the loop, where the agent must draw the boundary and then verify it.The related work shares the decidable-versus-subjective distinction with Zhou and Shbita [31], but differs in who draws the line and when.Source: §2 · p. 3
  • κ 0.92–0.99 at clear endpoints, anti-correlated regulatory bias, and 57.7% versus 88.8% accuracy under a more prescriptive rubric challenge broad degradation, undirected error, and underspecification.The accuracy comparison is for a decision procedure with worked examples versus unprompted labeling.Source: §2 · p. 3

3 The Setting: Runtime Verification of Agent Actions

Agent actions create repeated routing decisions: each applicable rule decomposes into predicates that must go to code or a judge. The two mistakes have different costs, and errors across predicates compound at the rule level.

3 The Setting: Runtime Verification of Agent Actions

  • Each agent action is checked against applicable rules, each rule breaks into predicates, and every predicate must be routed to code or a judge.The passage cites over a million binding restrictions in the US Code of Federal Regulations, and notes that checks cost microseconds while judge calls cost seconds and tokens.Source: §3 · p. 4
  • Over-claiming a material-conflict disclosure as checkable can certify compliance no one assessed, while under-claiming its three-day deadline sends timestamp arithmetic to a judge.The former risks a missed violation repeated across a fleet; the latter spends model calls, with one year’s inference budget potentially used in four months.Source: §3 · p. 4
  • A rule passes only when all its predicates are routed correctly, so the chance of a fully correct routing degrades geometrically as rules lengthen.Choosing a model to reduce missed violations can increase the inference bill, because model biases run in opposite directions across settings.Source: §3 · p. 4

4 The Decidability Axis, and Why Routing Needs Type

Routing requires identifying a criterion’s type—whether a fixed procedure settles it—not deciding the verdict for a particular case. The distinction is familiar across fields, but agent deployment makes drawing it a runtime task.

4 The Decidability Axis, and Why Routing Needs Type

  • Law, psychology, psychiatry, and reinforcement learning describe related boundaries, but their line is drawn offline by people with time to adjudicate; agents face unsorted predicates at runtime.The paper connects these distinctions to rules versus standards, actuarial versus clinical prediction, and well-structured versus ill-structured problems.Source: §4.1 · p. 4
  • A decidable criterion has an exact answer from a fixed, interpretation-free procedure; a judgment criterion turns on interpretation, degree, or value.Decidability concerns the criterion’s type, not how difficult a particular verdict is to obtain.Source: §4.1 · pp. 4–5
  • The paper uses “decidable” by analogy with computability, not to claim that judgment criteria are formally Turing-undecidable.Source: §4.1 · p. 4
  • On agent rules, every model shifts toward over-claiming, the unsafe direction; GPT-4o moves from the most cautious on regulatory text to the most reckless on the operational rule-set.The reported over-claim denominators are 78 true-judgment predicates for regulatory text and 24 for agent rules.Source: Figure 2 · p. 5

5 Experimental Setup

The experiments test whether models can classify predicates as decidable under a fixed rubric, across six corpora and roughly 22,000 labels. The corpus selection spans clear endpoints, regulatory rules, operational agent rules, and a synthetic control.

5 Experimental Setup

  • Each model classified a single predicate as decidable only when explicitly stated facts supported an exact answer without interpretation, degree-weighing, or value judgment.The blind protocol used a fixed rubric, temperature 0, and no hedging.Source: §5 · p. 5
  • The study covers 832 distinct predicates from six corpora, labelled by four models under four rubric variants and two perturbation instruments, for approximately 22,000 calls.Every reported figure and table is computed directly from the label files.Source: §5 · p. 5
  • Four models from three labs labelled every target across endpoint benchmarks, EU AI Act and FINRA predicates, operational agent rules, and a gold-labelled synthetic control.Ground truth on the real corpora is an LLM-assigned proxy; predicates were decomposed from rule text under a fixed schema.Source: §5 · p. 5

6 Results

Unanimity identifies a high-precision decidable set on synthetic and regulatory data, but its precision drops on operational agent rules where all models over-claim decidability. More broadly, model agreement is high at clear endpoints but weak and unstable on regulatory predicates.

6 Results

  • Agreement was κ 0.20–0.50 across 231 regulatory predicates, versus κ 0.92–0.99 on the clear IFEval and MT-Bench endpoints.The regulatory κ range is measured under the fixed rubric; a more prescriptive rubric raises it to 0.55–0.69 without tracking ground truth.Source: §6 · p. 5
  • On regulatory text, Llama and Sonnet lean toward over-claiming while GPT-4o and Opus lean toward under-claiming; every model flips to over-claiming on agent rules.The operational-rule over-claim rates are GPT-4o 71%, Sonnet 62%, Llama 46%, and Opus 29%, with κ 0.38–0.64.Source: §6 · p. 6
  • The contested zone covers 65% of EU AI Act predicates, 52% of FINRA predicates, and 33% of agent rules, versus 23% of synthetic predicates.The disagreement rate rises with panel size, so these shares are lower bounds.Source: §6 · p. 6
  • Under deterministic-template perturbation, Opus flipped 33% of its borderline labels versus 5% on controls, and GPT-4o flipped 44% versus 7%.Every model destabilized under at least one perturbation; blind human sorting of 53 FINRA rules also produced κ 0.34–0.48.Source: §6 · p. 6
  • 99.2% precision on gold-labelled synthetic data and 97.0% on regulatory text fell to 83% on the agentic corpus, where all four models over-claimed decidability.This gap motivates the paper’s verification stage beyond agreement filtering.Source: §6 · p. 6

7 Is the Boundary Just Our Rubric? A Specification Ablation

The rubric ablation finds that procedural rewording leaves disagreement unchanged, while added specification changes agreement without reliably resolving the decidability boundary. Its effects differ across regulatory text and operational agent rules, and the predicates resolved depend on which equally correct examples are chosen.

Specification ablation

  • Procedural rewording leaves the contested rate flat on every corpus, despite models flipping 6.6–16.3% of their own labels without consistent direction.The study’s red-team manipulations hold corpus, models, and decoding fixed; R-proc isolates surface form and R-spec isolates underspecification.Source: §7 · p. 6
  • Specification lowers contested rates by 30.9 pp on the EU AI Act and 19.6 pp on FINRA, but converted consensus is correct only 57.7% of the time.For the same 71 converted predicates, the unprompted panel’s majority is right 49.3% of the time; a constant-DECIDABLE router would score 74.6%.Source: §7 · p. 6
  • The proxy ground truth uses the rule-set’s grading_method and counts 20 “both” rules as judgment, inflating absolute over-claim magnitudes.Reclassifying “both” as decidable leaves four true-judgment predicates; the model ordering and flip survive, while κ is unaffected.Source: §7 · p. 6
  • On operational agent rules, specification leaves the contested rate at 33%, while half the predicates change membership beneath that unchanged total.Specification is a lever on regulatory text but not agent rules; where it reduces disagreement, it suppresses it rather than resolving it.Source: §7 · p. 7
  • Eight different, equally correct worked examples resolve a different third of predicates (Jaccard 63.3%) and reach 75.9% accuracy, versus 57.7% for R-spec.The benefit of specification therefore depends on an authoring choice the rubric does not record.Source: §7 · p. 7

8 Corroboration Nominates, Intervention Tests, Neither Certifies

COVER uses unanimous model agreement only to nominate predicates, then admits them only when synthesized checks pass intervention-based gates. The results show that unanimity and judge agreement have limits, while intervention rejects some false-decidables at a coverage cost and cannot certify faithfulness.

8.1 Corroboration as a model-agnostic safety floor

  • Unanimity pays for precision with coverage: GPT-4o routes three-quarters of genuinely decidable predicates to a judge, and unanimity inherits that cost.Which single model is safe varies by domain; GPT-4o is the heaviest over-claimer on agent rules.Source: §8.1 · p. 7
  • Unanimity bounds the safety-critical false-decidable rate to 1–3%, but on agent rules its precision falls to 83% at 69% coverage when models share bias.Adding a fifth reasoning-era model leaves the unanimous set identical on both regulatory corpora; precision on agent rules is 83% at 69% coverage.Source: §8.1 · p. 7

8.2 The obvious second layer, and why it fails

  • At τ = 1, judge agreement admits 43/173 predicates at 79.1% precision, but its apparent lift comes from constant reference verdicts, not discrimination among varying predicates.The gate scores 100.0% on 16 admitted predicates with constant verdicts and 66.7% on the 27 varying predicates it admits.Source: §8.2 · p. 8
  • Calibration agreement rises from 29.6% to 76.7% across four buckets while true-decidable rates do not move, so judge agreement measures reproducibility rather than checkability.The reference judge shares the estimator population’s blind spots, and no τ threshold fixes agreement with the wrong target.Source: §8.2 · p. 8

8.3 Verification by intervention: COVER’s second stage

  • After nomination, COVER consults no model further; it admits predicates based on synthesized-check behavior rather than another estimator’s self-report.The check reads named fact-slots from the action record and applies a decision rule over them.Source: §8.3 · p. 8
  • The four gates test structure, groundedness, invariance, and sensitivity; checks using agent-writable fields, missing fields, or exceptions are escalated.Only attested fields that infrastructure produces and the agent cannot write are admissible.Source: §8.3 · pp. 8–9
  • 85.0% precision against a 67.1% base rate is reached with full invariance; on the held-out half, the gate admits 10 predicates at 90.0% precision with one false-decidable.The held-out comparison is unanimity’s 19 predicates at 89.5% precision with two false-decidables; the gate removes 27.4% of judge calls versus unanimity’s 56.2%.Source: §8.3 · p. 9
  • Intervention cannot establish faithfulness: one agent obligation had three deontic clauses, while its rule covered only one.Establishing faithfulness would require putting a judge back in.Source: §8.3 · p. 9

9 Limitations and Conclusion

The conclusion argues that a verifier’s scope must be established through execution rather than model self-report, especially under externally written rules. COVER uses unanimous model labels only to nominate predicates, then applies gates to decide whether synthesized checks can replace judges.

9 Limitations and Conclusion

  • The agentic evidence is narrow: 73 predicates from 20 rule families in one deployed credit agent, with over-claim rates based on 24 judgment predicates and intervals roughly 35 percentage points wide.The direction is reported as robust, but not the magnitudes; the interventional evaluation also uses 36 synthetic action records.Source: §9 · p. 9
  • The conclusion says models agree on obvious cases but err together where it matters, so a verifier’s scope must be established by execution rather than another model judgment.Under externally written rules, routing errors can remain silent and evade escalation.Source: §9 · p. 9

A The COVER Procedure

  • COVER attempts unanimous nominations first, but predicates outside that set may also be admitted if their synthesized checks clear the gates.No estimator’s opinion enters the later gates; unattempted predicates are escalated.Source: A The COVER Procedure · p. 12
  • Table 2 contrasts label rows covering 175 labelled EU AI Act predicates with graduation rows for the 173 predicates the gate ran on; label routers cannot report held-case agreement.The caption identifies held agreement as agreement with a reference judge on held-out cases.Source: Table 2 · p. 12

A.1 Protocol for the interventional evaluation

The section evaluates whether checks and model labels provide stable routing signals, using interventional records, meaning-preserving perturbations, and a human-labelled replication. The findings report chance-level separation by self-consistency, borderline instability under some deterministic reframings, and continued cross-model disagreement in human-labelled rules.

A.1 Protocol for the interventional evaluation

  • The interventional protocol used 73 predicates and 36 synthetic action records, including clear, ambiguous, and absent forms of each judgment-bearing phenomenon.A seeded split reserved half the corpus from operating-point selection, and one record served as a schema exemplar withheld from every gate.Source: §A.1 · p. 12
  • The self-consistency instrument independently re-extracted each predicate’s slots five times at temperature 1.0 on a fixed prefix of 12 records.The record set was bounded because a worst-case statistic over an unbounded set would eventually encounter a flaky read.Source: §A.1 · p. 12
  • AUC 0.42–0.48 across ten self-consistency aggregations failed to separate true-decidable from true-judgment predicates among the 20 that reached the gate.Nine of 13 decidable and five of seven judgment predicates scored perfect self-consistency; the authors note the bounded, selected sample and possible near-deterministic extraction.Source: §A.1 · p. 13

B Perturbation Instruments for the Stability Test

  • The stability test generated meaning-preserving paraphrases, checked equivalence independently, and compared each model’s flip rate with unanimous-clear controls.A second instrument used deterministic surface wrappers with no LLM in the loop.Source: B Perturbation Instruments for the… · p. 13
  • 33% vs 5% for Opus and 44% vs 7% for GPT-4o show higher borderline flip rates than clean-case baselines under deterministic templates.The effect varied by model and perturbation; control baselines were 5–10%, and not every model destabilized under every instrument.Source: B Perturbation Instruments for the… · p. 13
  • κ 0.20–0.50 on regulatory text and 0.41–0.71 on the full policy set show cross-model disagreement as the dominant failure, with within-model instability a secondary signal.The contested subset alone has still lower agreement.Source: B Perturbation Instruments for the… · p. 13

C.2 Replication against human labels

  • Human replication sorted 53 FINRA rules into three enforceability tiers, but its scope is limited by one annotator, whole-rule targets, and intervals 15–30 points wide.The authors identify independent multi-annotator labeling at predicate granularity as the strongest available improvement.Source: §C.2 · p. 14
  • κ 0.34–0.48 against human labels matches the regulatory-text disagreement range, while models contest 34% of 53 FINRA rules.GPT-4o under-claimed 85% of decidable rules and over-claimed 3%; unanimous labels were 100% precise over 10% of decidable rules in the A-versus-C contrast.Source: §C.2 · p. 14

C.3 A reasoning-era panel member

The added reasoning-era labeller was evaluated on policy predicates, agent rules, and calibration endpoints. It was highly permissive on regulatory and agent-rule predicates, while adding it left the unanimous gate unchanged; the authors limit the inference to one 14B open-weight distillation.

C.3 A reasoning-era panel member

  • The reasoning-era model called 73.2% of regulatory predicates decidable and over-claimed 92% of judgment predicates on the operational agent rule-set.It called 95.9% of all agent predicates decidable, and pairwise agreement with incumbents ranged from κ 0.13 to 0.59.Source: §C.3 · p. 15
  • On MT-Bench, the reasoning-era model called 30% of items decidable, compared with 0.0% for GPT-4o, 1.2% for Sonnet and Llama-3.1, and 10.0% for Opus.On IFEval, it called all 120 instructions decidable.Source: §C.3 · p. 15
  • Adding the reasoning-era labeller left the unanimous-DECIDABLE sets unchanged: 33 regulatory predicates at 97.0% precision and 20.9% coverage, and 41 agent predicates at 82.9% precision.The agent-rule result retained the same seven false-decidables as the four-model gate.Source: §C.3 · p. 16
  • The scope is one 14B open-weight distillation, so the result bears on whether deliberation repairs self-decidability, not whether capability does.The paper reports that this model became the panel’s least safe member on the agent corpus where labels would not be escalated.Source: §C.3 · p. 16

D Specification Ablation in Full

The specification ablation shows that rewording can change which predicates models label decidable without resolving their disagreement, while adding worked examples can create agreement that is inaccurate and example-dependent. Deterministic rewording also destabilizes some individual models, and the pattern echoes concerns about reliability and validity separating in psychiatric criteria.

D Specification Ablation in Full

  • Under R-proc, contested rates were flat across every corpus (−6, +3, +7, 0, 0 pp) although individual models flipped 6.6–16.3% of labels; under R-spec, rates fell 30.9 pp on the EU AI Act and 19.6 pp on FINRA.Source: D Specification Ablation in Full · p. 16
  • 57.7% accuracy for R-spec-manufactured regulatory consensus versus 88.8% for unprompted consensus; 29 of R-spec’s 30 errors were under-claims.R-spec converted 71 regulatory predicates from contested to unanimous; the reported Wilson 95% intervals were 46.2–68.5 and 80.5–93.8, respectively.Source: D Specification Ablation in Full · p. 16
  • On operational agent rules, R-spec left the contested rate unchanged at 33%, while 12 predicates became contested and 12 became unanimous.The authors characterize the stability as coincidental, with half the membership churning.Source: D Specification Ablation in Full · p. 17
  • With the rubric otherwise character-identical, swapping the eight correct worked examples changed resolved predicates and yielded 58% versus 76% consensus accuracy.The resolved sets had 63.3% Jaccard overlap; the accuracy gap was 18 points, and all 14 R-spec′ errors were under-claims.Source: D Specification Ablation in Full · p. 17

E Deferred Instruments and Diagnostics

  • On regulatory text, contested predicates span half to two-thirds of the set; adding a model raises the contested rate, so measured rates are lower bounds.A method tested only on synthetic data would show an approximately 20%-borderline world and understate the regulatory-text result.Source: E Deferred Instruments and Diagnostics · p. 17

E.1 Why this is a bug, not appropriate humility

  • The “feature” reading treats borderline cases as appropriate model uncertainty; the paper tests its predictions against a bug reading.The section frames four opposing predictions for these readings.Source: §E.1 · p. 17

E.2 The within-model stability test

  • Under deterministic templates, Opus flipped borderline labels 33% versus 5% on controls, and GPT-4o flipped 44% versus 7%; the effect depends on instrument and model.Every model destabilized under at least one instrument, while Llama’s failure surfaced chiefly as cross-model disagreement.Source: §E.2 · p. 18

E.3 The same failure outside machine learning

  • Psychiatry’s DSM-5 field trials reported questionable-band intraclass κ for six diagnoses and below 0.20 for three; reliability and validity came apart, a pattern the paper says its high-specificity rubric reproduces.The paper distinguishes routing noise from judge-accuracy noise: aggregation cancels independent error, not shared bias, and even a debiased judge does not identify which targets needed judging.Source: §E.3 · pp. 18–19

E.4 Further limitations

The preliminary synthetic enforcement run automates some predicate evaluations with a safety property established by construction, but misses the stated fidelity target at its default threshold. The paper also describes the dual cost of routing errors, acknowledges ambiguity and sampling limits, and identifies additional scope boundaries.

E.5 Inherent annotation disagreement

  • Some contested predicates are genuinely grey, with no fact of the matter to recover where reasonable annotators would split.The paper argues that this does not justify trusting a single estimator’s routing call.Source: §E.4 · p. 19, §E.5 · p. 20

E.4 Further limitations

  • Within-model instability is instrument- and model-dependent, so the paper treats it as corroborating evidence rather than a uniform law.The cross-model result carries the argument.Source: §E.4 · p. 19
  • The capstone uses one deterministic-check model and synthetic data; its safety result is by construction, not a broader empirical claim.The 36 action records are combinatorially generated but still author-built; the 20-predicate self-consistency null is not a corpus-random measurement. The paper also leaves adversarial predicate manipulation unaddressed and tests extended deliberation with a 14B distilled model, not frontier reasoning capability.Source: §E.4 · pp. 19–20

E.6 An honest concession

  • Where a predicate is indeterminate, a single model’s confident label is meaningless; where determinate, no single model reliably finds its type.The paper’s actionable conclusion is that the boundary must come from somewhere other than one estimator’s self-report.Source: §E.6 · p. 20

F The Dual-Cost Model

  • A single mistyped predicate can flip a rule verdict or silently drop a control, and the chance of every routing call being correct decays geometrically with predicate count.The dual-cost model contrasts recurring judge-token savings with safety cost from false-decidable auto-enforcement; the capstone and rubric ablation both trade automation for safety through over-flagging or under-claiming.Source: F The Dual-Cost Model · pp. 20–21, G Enforcement Capstone: Preliminary… · p. 21
  • A permissive single-model choice can incur a false-decidable rate up to ≈0.11, while a conservative choice covers only q ≈0.22–0.26 of decidable work.The model’s comparison says corroboration-plus-verification drives f toward 0 while coverage tracks the corpus’s deterministic content.Source: F The Dual-Cost Model · p. 21

G Enforcement Capstone: Preliminary Operating Point

  • ≈33% of predicate-evaluations were automated deterministically with zero gating-attributable missed violations in the synthetic enforcement run.Safety holds by construction: a predicate is auto-enforced only if its synthesized check matches the reference judge on the held-out graduation set.Source: G Enforcement Capstone: Preliminary… · p. 21
  • The default graduation threshold missed the strict fidelity bar of within 2 points of judge-everywhere, with a roughly 5-point shortfall appearing as over-flagging.This is one preliminary synthetic run using the agreement gate, not the adopted interventional gate; the full automation–fidelity tradeoff remains uncharacterized.Source: G Enforcement Capstone: Preliminary… · p. 21

Related papers

Loading 2610.04699v2…