Source-linked AI summary

READY or Not: Reliable Enterprise Agent Deployment

Veronica Chatrath, Bryan Zhu, Jingxuan Fan, George Pu, Soham Dinesh Tiwari, Soham Dan, Ryan Young, Yuan, Li, Yuang Yao, Apaar Shanker, Minglai Yang, Daniel Yue Zhang, Yunzhong He, Ying Liu, Chenguang Wang, Zhijun Yin, Yuan Xue

arXiv:2609.02095v1cs.AI

TL;DR

Existing benchmarks measure autonomous task performance, but enterprise deployment requires reliability with acceptable human oversight and cost. READY qualifies deployment policies against reliability targets, and in a 16-system clinical-audit study exposed large review differences between similarly accurate agents.

  • Problem

    Enterprise deployment requires evidence about reliability, human oversight, and operating cost beyond autonomous workflow performance.

    Source: §8 · p. 34
  • Method

    READY evaluates workflow execution, selects a minimum-cost oversight policy meeting a target reliability, and statistically qualifies it on held-out cases.

    Source: §2.2 · p. 6, §8 · p. 34
  • Results

    39.2% versus 29.6% human review was required by GPT-5.4 and Sonnet 5, respectively, to qualify at 76% reliability despite autonomous accuracies of 72.8% and 72.5%.

    Source: §1 · pp. 2–3
  • Takeaways & Limitations

    READY supports comparing agents by the human oversight and operating cost required for reliable deployment rather than autonomous performance alone.

    Source: §8 · p. 34

    Empirical validation focuses on trajectory-invariant, terminal accept-or-escalate policies; large-scale validation of trajectory-dependent interventions remains future work.

    Source: §7 · p. 33

Abstract

from arXiv · show

An AI agent can perform well on benchmarks and still be unsuitable for deployment. Existing AI-agent benchmarks measure whether an agent can complete realistic professional work, whereas enterprise deployment asks a different question: whether an agent can meet a required reliability level, under acceptable human oversight, and at tolerable cost. We introduce Reliable Enterprise Agent Deployment (READY), a framework for qualifying AI agents for deployment on enterprise workflows. READY preserves each workflow's own definition of successful execution while applying a common qualification procedure. Given an agent, a workflow, and a class of candidate oversight policies, READY measures the reliability and operating cost of the human-AI system, selects the minimum-cost policy that satisfies a specified reliability target, and statistically qualifies it on held-out cases. The resulting deployment profile characterizes the supported operating point: reliability, human-oversight burden, and cost. READY is implemented as an open testbed that decouples workflow specification, execution, evaluation, and qualification, and runs on existing agent-evaluation infrastructure. In an end-to-end clinical-audit case study spanning 16 agent systems and 750 cases, READY reveals differences hidden by autonomous performance: two systems separated by only 0.3 points in autonomous accuracy (72.8% vs. 72.5%) require 39.2% versus 29.6% human review, respectively, to qualify at the same 76% reliability target under the evaluated oversight policy. READY thus shifts enterprise agent evaluation from how well can the agent perform the work? to under what conditions, and at what cost, can it be reliably deployed? By making those conditions explicit and statistically testable, READY provides a basis for comparing agent systems, setting oversight requirements, and making evidence-based deployment decisions.

1 Introduction

READY reframes enterprise agent evaluation from autonomous capability to reliable deployment of a human–AI system under workflow-specific reliability requirements, oversight, and cost. It selects and statistically qualifies the minimum-cost oversight policy, with clinical-audit results showing that similar autonomous accuracy can mask substantially different review burdens.

  • READY evaluates whether a human–AI system can meet a workflow’s reliability requirement, rather than only whether an agent succeeds autonomously.Workflows retain their own success criteria, including correctness, process requirements, evidence grounding, and policy adherence.Source: §1 · p. 2
  • READY searches candidate oversight policies for the minimum-cost policy satisfying a specified reliability target and additional deployment constraints.Agent executions on representative cases estimate each policy’s reliability and operating cost before policy selection.Source: §1 · p. 2
  • The selected policy is frozen and statistically qualified on held-out cases, producing a deployment profile of reliability, oversight burden, cost, and workflow-specific risk.The profile characterizes the supported operating point rather than autonomous capability alone.Source: §1 · pp. 2–3
  • 0.3 percentage points separate GPT-5.4’s 72.8% and Sonnet 5’s 72.5% autonomous accuracy, yet they require 39.2% versus 29.6% human review to qualify at 76% reliability.Across 16 agent systems and 750 clinical-audit cases, deployment profiles reveal differences hidden by autonomous benchmark performance.Source: §1 · pp. 2–3
  • READY complements capability benchmarks [1] [2] [3], consistency measures such as passk in τ-bench, routing and cascades, and selective prediction or learning-to-defer [8] by qualifying complete human–AI operating points.It reuses evidence from existing workflow evaluations while adding explicit reliability requirements, oversight optimization, operating cost, and held-out qualification.Source: §1 · p. 3
  • READY is implemented as an open, extensible testbed that separates workflow-specific success definitions from a common deployment-qualification procedure and supports new workflows, cases, environments, and evaluators.The clinical-audit seed workflow [4] demonstrates end-to-end use of the interface.Source: §1 · p. 3

2 Design Overview

READY qualifies enterprise agents by selecting the lowest-cost oversight policy that meets workflow-specific reliability and risk requirements, then statistically evaluating that frozen policy on held-out cases. Its deployment profiles report the supported reliability–oversight–cost operating point under explicit workflow and deployment assumptions.

2.1 Evaluation Setting and Reliable Deployment Objective

  • READY preserves each workflow’s success definition while measuring execution outcomes and process requirements, then selects the minimum-cost policy meeting reliability and optional risk constraints.Success may require outcome correctness together with evidence use, procedural adherence, grounding, or policy compliance; unsupported operating points are rejected when no candidate policy is feasible.Source: §2.1 · pp. 5–6

2.2 End-to-End READY Flow

  • READY begins from declarative workflow, evaluation, and deployment specifications, evaluates multiple agents under the same specification, optimizes oversight policies, and qualifies the selected policy on held-out cases.Trajectory-invariant policies can be evaluated by replaying saved evidence, whereas trajectory-dependent interventions require runtime execution or simulation.Source: §2.2 · pp. 6–7
  • READY separates workflow evaluation, oversight-policy optimization, and held-out statistical qualification to produce deployment profiles for each evaluated agentic system.Each profile reports qualification status, reliability and confidence, human-oversight burden, operating cost, workflow-specific risk, and scope assumptions.Source: Figure 1 · p. 4, §2 · p. 4, §2.2 · p. 7

2.3 Running Example: Retrospective Clinical Audit

  • The clinical-audit example evaluates verdict correctness, abstention, evidence and policy grounding, and investigation adherence using a terminal confidence-based accept-or-escalate policy.Because routing occurs after audit completion, candidate policies can reuse the same saved trajectories before the selected operating point is statistically qualified.Source: §2.3 · p. 8

3 Oversight Policy Optimization and Deployment Qualification

This section presents READY’s procedure for optimizing oversight policies to achieve workflow-specific reliability at acceptable cost, while distinguishing human–AI system performance from autonomous agent performance. It describes policy evaluation across terminal, risk-sensitive, and trajectory-dependent oversight settings, followed by selection from development evidence and statistical qualification of a frozen policy on held-out cases, including application to CliniCARE-Bench.

3.1 Qualification Quantities

  • READY evaluates deployed human–AI systems by workflow-specific reliability and operating cost, which depend on autonomous success, observable routing, oversight effectiveness, and intervention cost rather than agent performance alone.Reliability is an aggregate probability over workflow instances, while operating cost may include both agent execution and human review or intervention.Source: §3 · p. 9, §3.1 · p. 9

3.2 Terminal Accept-or-Escalate Oversight

  • Terminal accept-or-escalate policies replay saved autonomous executions to route cases using observable signals, enabling many candidate policies to be evaluated without re-executing the agent.A threshold policy accepts cases with s_i ≥ τ and routes the remainder to review, while deployment-time routing cannot use the unknown autonomous-success label.Source: §3.2 · p. 10

3.3 Cost at a Target Reliability

  • Similar autonomous success rates can yield substantially different deployment operating points because oversight policies differ in how effectively they identify cases requiring intervention.At a target reliability, READY compares the operating cost required for each system to satisfy the same workflow-specific requirement; with constant positive costs, this is equivalent to minimizing review burden.Source: §3.2 · p. 11, §3.3 · p. 11

3.4 Quality of the Routing Signal

  • Better routing signals reduce selective risk and can lower human intervention and operating cost at a fixed reliability target; READY diagnoses routing with risk–coverage metrics including AURC and AUROC.Lower AURC indicates a more favorable risk–coverage tradeoff, while AUROC measures ranking discrimination independently of successful-execution prevalence.Source: §3.4 · p. 12

3.5 Statistical Qualification

  • READY separates development-time policy selection from held-out statistical qualification by freezing the selected policy and requiring a confidence lower bound, rather than a point estimate alone, to clear the deployment target.Qualification claims are conditional when human-review effectiveness is assumed, and sampling or resampling must preserve task-instance and higher-level cluster dependence.Source: §3.5 · pp. 12–14

3.6 Risk-Sensitive Qualification

  • READY can add workflow-specific risk constraints when binary success fails to represent failure consequences, including severe-event probability bounds or tail-risk measures such as Conditional Value at Risk (CVaR).The common objective remains reliable deployment at minimum operating cost, with the additional risk constraint introduced only when needed.Source: §3.6 · p. 15

3.7 Trajectory-Dependent Oversight

  • For trajectory-dependent oversight, intervention decisions based on workflow history change the induced execution trajectory, but the qualification objective remains unchanged; only the evidence-generation process differs.Examples of intervention include approval, correction, and takeover during execution.Source: §3.7 · p. 15
  • For trajectory-invariant terminal policies, saved trajectories can be replayed under alternative routing rules, whereas trajectory-dependent policies require execution or appropriate simulation in the evaluation runtime.Changing a trajectory-dependent policy can alter what the agent, oversight actor, or interacting user subsequently observes and does.Source: §3.7 · p. 15
  • Intervention can improve successful completion while also changing execution length, human effort, recovery behavior, and subsequent outcome distributions, all captured by READY’s reliability, cost, and risk quantities.Policy optimization for trajectory-dependent oversight may repeatedly invoke agent execution and workflow evaluation while searching candidate policies.Source: §3.7 · pp. 15–16
  • READY’s shared procedure evaluates candidate policies by resulting system reliability, cost, and risk, selects one using development evidence, and qualifies the frozen policy on held-out evidence.READY does not prescribe an optimization algorithm; policy representations and search methods can be workflow-dependent [11].Source: §3.7 · p. 16
  • Section 5 applies the qualification methodology to CliniCARE-Bench using the full four-way adjudication space and a class-label-agnostic signal-based accept-or-escalate policy.Sensitivity analyses and a cost-of-conflation control appear in Appendix B, including Appendix B.5.Source: §3.7 · p. 16

4 READY Evaluation Runtime and Open Testbed

READY’s open testbed separates workflow specification, execution, evaluation evidence, and deployment qualification while reusing existing agent-evaluation infrastructure. It produces versioned, statistically supported deployment profiles whose reliability, oversight, and cost claims remain conditional on explicit workflows, systems, policies, and deployment assumptions.

4.1 Runtime Layering and Reference Execution Substrate

  • READY uses a three-layer architecture that separates non-executing specifications, runtime execution, and evidence evaluation while standardizing interfaces across reusable infrastructure.Its reference implementation uses Inspect AI for execution, integration, sandboxing, scoring, intervention, and logging, while READY adds workflow semantics and qualification.Source: §4.1 · p. 16, §4 · p. 16

4.2 Specification Interfaces

  • The specification interfaces separately define the workflow and task population, evaluation semantics and success predicate, deployment assumptions, and versioned development and held-out partitions.Deployment assumptions remain separate from measured evidence, allowing the same execution records to be interpreted under different reliability, oversight, cost, and risk requirements.Source: §4.2 · p. 17

4.3 Execution, Actors, and Policy Evaluation

  • The runtime supports human or simulated oversight and user actors, plus trajectory-invariant replay or policy-dependent execution for evaluating candidate oversight policies.Trajectory-invariant policies can reuse one completed execution across candidates, whereas trajectory-dependent policies require executions generated under each policy.Source: §4.3 · pp. 18–19

4.4 Evaluation Evidence and Workflow Evaluation

  • Each execution retains workflow-relevant evidence for evaluator-defined measurements and success events, while preserving diagnostics, interventions, and repeated-run identity for qualification.Inspect logs can retain outputs, interactions, scores, metadata, usage, transcripts, and intervention records; repeated epochs remain linked to the same task instance.Source: §4.4 · p. 19

4.5 Qualification Handoff and Deployment Profile

  • The testbed converts execution evidence into a deployment profile conditional on the workflow, system, oversight policy, and declared assumptions, rather than a context-free model score.READY separates workflow scoring from qualification: the runtime records what happened, while the qualification layer determines which deployment policy the evidence supports.Source: §4.5 · p. 20

4.6 Contributing and Maintaining Evaluations

  • READY makes new workflows portable and auditable through explicit interfaces, independent validation, versioned lineage, and change-specific maintenance or requalification rules.Assumption changes can reuse evidence, evaluator changes may permit rescoring when inputs were retained, and trajectory or population changes require re-execution.Source: §4.6 · pp. 20–21

5 Empirical Evaluation: Deployment Qualification on CliniCARE-Bench

This section uses CliniCARE-Bench to evaluate READY’s deployment-qualification methodology, focusing on selective autonomy, oversight requirements, and deployment value beyond autonomous performance alone.

  • The evaluation compares deployment value among systems with similar autonomous performance rather than reproducing the full CliniCARE benchmark.Source: §5 · p. 21

5.3 Routing Signal Reliability

  • The evaluation measures whether routing signals support selective autonomy and distinguish more reliable from less reliable autonomous executions.This question is addressed in Section 5.3.Source: §5 · p. 21
  • The evaluation quantifies how much human oversight is required to meet a given reliability target.Source: §5 · p. 21

2. How does the required human-review burden grow as the reliability target increases? (Section 5.4)

READY shows that increasing reliability targets require increasing human-review burden, with deployment comparisons made on a held-out, statistically qualified reliability–oversight frontier rather than autonomous accuracy alone. The frontier also exposes how routing quality, signal granularity, review assumptions, and qualification margins shape deployability.

5.1 Experimental Setup

  • The experiment evaluates 16 systems across 750 cases, split into identical 375-case development and qualification cohorts for paired cross-system comparisons.The natural cohort distribution is preserved in both partitions rather than balanced or stratified.Source: §5.1 · p. 22

5.2 Escalation Policy

  • The evaluated policy is a terminal accept-or-escalate rule using only stated confidence, with one threshold applied across all four adjudication outcomes and δ = 0.05 used to absorb selection bias.The policy is class-agnostic, and thresholds are optimized against a margin-adjusted target rather than fitting the target directly on development data.Source: §5.2 · p. 23

5.3 Autonomous Performance and Routing Quality

  • Autonomous accuracy and routing quality measure different properties: across 16 systems, base accuracy and AUROC have ρ = −0.12, whereas AURC tracks accuracy with ρ = −0.80.Thus, an agent can be accurate while ranking its own failures poorly, making routing quality operationally relevant for oversight.Source: §5.3 · p. 23

5.4 Reliability–Oversight Frontier

  • READY selects the highest-coverage threshold meeting each target on development cases, freezes it, and compares systems at the same reliability requirement rather than the same autonomous coverage.The resulting frontier measures how much work must be routed to review as the organization demands more reliable operation.Source: §5.4 · pp. 23–24

5.5 Held-Out Qualification and Deployment Comparison

  • Held-out qualification requires a one-sided 95% lower confidence bound to meet Y; without the margin, only 46% of frontier policies qualify, compared with 95% using it.At Y = 0.76, qualification is conditional on the assumed human-review success ah, and point-estimate optima can fail the held-out deployment test.Source: §5.5 · pp. 24–25

5.4 Reliability–Oversight Frontier

  • At ah = 0.90, every system’s review burden exceeds 90% by Y = 0.85, while at Y = 0.76 Sonnet 5 qualifies with 29.6% review versus GPT-5.4’s 39.2%.Figure 3 converts routing evidence into the minimum review burden required for increasing reliability targets; no policy can exceed reliability 0.90 under the stated review assumption.Source: §5.6 · p. 25, Figure 3 · p. 26

5.6 Similar Autonomous Performance, Different Deployment Value

  • GPT-5.4 and Sonnet 5 differ by only 0.3 autonomous-accuracy points, yet both qualify at Y = 0.76 with review burdens of 39.2% and 29.6%, respectively.Sonnet 5 has stronger routing quality, with AUROC 0.681 versus 0.601 and AURC 0.203 versus 0.222, so more work remains safely autonomous.Source: §5.6 · pp. 25–26, Figure 3 · p. 26

5.7 Scope and Additional Analyses

  • The analysis isolates terminal accept-or-escalate routing; richer actions such as requesting evidence would introduce workflow-state-dependent trajectories, while Appendix B reports review-assumption and process-aware sensitivity analyses.The case study therefore demonstrates a deployment-level relationship among autonomous performance, routing quality, human oversight, and operating reliability within the evaluated policy class.Source: §5.7 · pp. 26–27

6 Related Work

READY positions itself as a deployment-qualification layer over agent benchmarks and related evaluation methods, integrating executable-task evidence, workflow-specific success, human oversight, operating cost, reliability targets, and held-out statistical support. Table 4 identifies READY as the only surveyed approach combining all six deployment dimensions in one qualification claim.

6 Related Work

  • READY integrates executable and interactive task evidence, target reliability, operating cost, human oversight, and held-out statistical qualification into a single deployment claim.Table 4 presents READY as the integration point across the six related-work families.Source: Table 4 · p. 28, §6 · p. 29

6.1 From Knowledge Exams to Professional-Work Agents

  • READY consumes professional-work benchmarks rather than replacing them, adding a reliability–oversight–cost layer that selects minimum-cost policies meeting prespecified reliability targets.Capability benchmarks measure whether agents complete professional work, while READY asks whether a deployment operating point is statistically supported.Source: §6.1 · p. 29

6.2 Outcome, Process, and Evidence-Based Evaluation

  • READY treats outcome, process, and evidence requirements as workflow-specific evaluator inputs, requiring any process criterion used for qualification to rely on observable execution evidence.The success predicate can include grounding, provenance, tool use, abstention, and outcome correctness without imposing a universal truthfulness metric.Source: §6.2 · p. 30

6.3 Selective Prediction, Abstention, and Learning to Defer

  • READY extends selective prediction and learning to defer by evaluating the human-review outcome, selecting minimum-cost policies for reliability targets, and separately qualifying them on held-out data.It distinguishes task-level abstention from deployment-level escalation and treats human review as part of the operating policy.Source: §6.3 · p. 31

6.4 Confidence, Uncertainty, and Calibration as Routing Signals

  • READY does not require calibrated routing signals; it evaluates their induced ordering and incorporates extra inference or repeated-sampling costs into the operating-cost comparison.A signal need only preferentially remove higher-risk executions under selective acceptance.Source: §6.4 · p. 31

6.5 Cost-Aware Routing, Cascades, and Deferral Economics

  • READY differs from cost-aware cascades by escalating to human reviewers, enforcing a statistically qualified reliability floor, and using workflow-specific success predicates rather than average model quality.Existing routing methods can provide the confidence-based signal, while READY supplies the deployment-qualification procedure.Source: §6.5 · p. 32

6.6 Human Oversight, Scalable Oversight, and Deployment Assurance

  • READY measures the review path rather than assuming human oversight helps, qualifying a specific human–AI configuration by its reliability, oversight burden, and operating cost.This addresses evidence that teaming effectiveness depends on reviewer expertise, interaction design, and overreliance risks.Source: §6.6 · p. 32

6.7 General Agent-Evaluation Frameworks

  • READY complements evaluation harnesses such as Inspect AI, Harbor, and AgentBeats by adding policy selection and held-out statistical qualification above their executable-task infrastructure.Its reference implementation runs on Inspect and can consume Harbor-packaged tasks through the inspect-harbor adapter.Source: §6.7 · p. 33

7 Scope, Limitations, and Future Work

READY qualifies agents only under explicit workflow, oversight, reliability, and cost assumptions, rather than universally certifying them. Its evidence is population- and configuration-specific, while future work targets richer oversight policies, direct human-review measurement, and broader validation.

Scope

  • READY is a conditional deployment-qualification framework, not a universal agent certification or a complete account of organizational production considerations.Its conclusions depend on explicit workflow, oversight, reliability, and cost assumptions.Source: §7 · p. 33

Empirical scope

  • The empirical evaluation centers on trajectory-invariant terminal accept-or-escalate routing in CliniCARE-Bench, while trajectory-dependent policy validation remains future work.Such policies require policy-in-the-loop execution or simulation because interventions can change subsequent execution.Source: §7 · p. 33

Human-oversight assumptions

  • Deployment profiles are conditional on assumed human-review success, latency, and cost when the review path is not directly executed.Future evaluations should measure reviewer effectiveness, review time, intervention type, and recovery cost directly.Source: §7 · p. 33

Population and distribution validity

  • Qualification claims apply only to the evaluated workflow population and versioned configuration, not arbitrary distribution shifts, environment changes, or materially different agent behavior.Continuous monitoring is needed to detect degradation and determine when requalification is required, but monitoring design is outside this work.Source: §7 · p. 33

Risk and deployment context

  • READY optimizes expected operating cost subject to workflow-specific reliability, can add tail-risk constraints, and does not replace security, privacy, legal, regulatory, or governance processes.These broader considerations may become workflow constraints or evaluation criteria while remaining outside the core methodology.Source: §7 · p. 34

Future work

  • Future work will evaluate trajectory-dependent and multi-step oversight, measure human-review effectiveness and economics directly, and validate READY across broader workflows and domains.The open interface supports these extensions without changing the underlying qualification objective.Source: §7 · p. 34

8 Conclusion · A Formal definitions

READY reframes enterprise agent evaluation as qualifying a workflow-specific human-AI deployment at a required reliability and cost, rather than scoring autonomous performance alone. Its formal extensions define selective routing and risk-sensitive qualification while preserving held-out statistical validation and explicit operating assumptions.

8 Conclusion

  • Enterprise deployment asks whether an agent reaches required reliability with acceptable human oversight and operating cost, not simply whether its benchmark score improves.READY therefore compares systems by the cost of reliable deployment and measures how much oversight remains necessary as capabilities change.Source: §8 · p. 34
  • READY selects and statistically qualifies the minimum-cost oversight policy that meets a target reliability, producing a deployment profile conditional on workflow, oversight model, and operating assumptions.The profile characterizes how an agent can be deployed rather than assigning an intrinsic agent score.Source: §8 · p. 34
  • READY separates workflow-specific success definitions from a common qualification methodology and open runtime testbed, enabling joint analysis of autonomous performance, routing, oversight, and deployment reliability.Trajectory-invariant policies can replay saved execution evidence, whereas trajectory-dependent policies require policy-in-the-loop execution or simulation.Source: §8 · p. 34

A.1 Threshold Routing and Monotonicity of the Routing Signal

  • A threshold policy accepts cases autonomously when the routing signal s_i ≥ τ and sends lower-scoring cases to human review, using observed autonomous outcomes during policy development and qualification.Monotone routing requires increasingly selective acceptance to improve retained-case success probability; READY does not require pointwise monotonicity or calibration.Source: §A.1 · p. 48

A.2 Tail-Risk Estimation for Risk-Sensitive Qualification

  • Risk-sensitive qualification can replace binary success with a workflow-defined loss whose upper-tail CVaR captures failure severity, recovery cost, or other consequences.This extension is optional when binary deployment success adequately represents workflow consequences.Source: §A.2 · pp. 48–49
  • READY freezes the selected policy and risk specification before held-out evaluation, computes losses by replay or policy-consistent execution, and uses a one-sided confidence bound for statistical risk qualification.The reference procedure obtains the bound by bootstrap resampling the qualification set, with resampling units matched to task-instance or cluster-level generalization.Source: §A.2 · pp. 49–50
  • Tail-risk qualification must pre-specify the loss, tail level, tolerance, and confidence level, propagate uncertainty in unobserved review outcomes, and report insufficient evidence when the tail sample is too small.For α = 0.99, the estimate is informed mainly by roughly the worst one percent of outcomes, so READY reports effective tail observations alongside estimates and bounds.Source: §A.2 · p. 50

B Supplementary Analyses for CliniCARE-Bench · B.1 Reproducing the Held-Out Lower Bound · 5. Break-even. amin

The supplementary analyses reproduce held-out deployment qualification using frozen thresholds, exact lower bounds, and complete policy calculations. They also define how break-even review assumptions are solved and when policies qualify.

B Supplementary Analyses for CliniCARE-Bench

  • The appendix applies global routing and margin-adjusted threshold selection, choosing thresholds on development data and freezing them before held-out qualification.This supplements the four-way deployment-qualification analysis with held-out lower-bound computation, sensitivity checks, and complete per-system results.Source: B Supplementary Analyses for… · p. 50

B.1 Reproducing the Held-Out Lower Bound

  • For each qualification case, routing accepts cases with s_i ≥ b_τ, then computes accepted count, autonomous successes, and reviewed fraction.The reviewed fraction is (N_qual − n_acc)/N_qual, while p̂_acc = k_acc/n_acc.Source: §B.1 · pp. 50–51
  • The point reliability estimate combines the observed accepted rate with the assumed review success through Equation 32.This produces the estimated reliability used before applying the lower-bound qualification test.Source: §B.1 · p. 51
  • The held-out lower bound uses a one-sided 95% exact Clopper–Pearson bound for accepted-case success, attaching no confidence to the assumed review term.The bound is the α = 0.05 quantile of Beta(k_acc, n_acc − k_acc + 1), with p_acc = 0 when k_acc = 0.Source: §B.1 · pp. 50–51
  • A policy qualifies exactly when its 95% lower confidence bound is at least the target reliability Y.The qualification rule is applied to the held-out qualification partition after threshold selection has been frozen.Source: B Supplementary Analyses for… · p. 50, §B.1 · p. 51

5. Break-even. amin

  • The break-even analysis finds the smallest assumed review success a_h ∈ [0, 1] that raises the lower-bound expression to Y.Because the lower confidence bound is nondecreasing in a_h, this requires a one-dimensional solve.Source: §5 · p. 51
  • Policies are marked — when no assumed review success a_h ∈ [0, 1] suffices to reach the target reliability.This identifies policies that cannot qualify under the evaluated break-even assumption range.Source: §5 · p. 51

B.2 Sensitivity to the Human-Review Assumption

The analysis shows that the declared human-review success assumption, rather than agent capability, substantially determines reachable reliability and review burden. Repeating qualification across alternative assumptions also reveals nonmonotonic qualification outcomes and stepwise review changes caused by held-out testing and threshold granularity.

B.2 Sensitivity to the Human-Review Assumption

  • The analysis recomputes the reliability–oversight frontier and qualification outcomes for ah ∈{0.80, 0.85, 0.90, 0.95, 1.00} without changing execution evidence.The dev-select/freeze/qualify procedure repeats over the same 31-target sweep, evaluating 496 policies per assumption.Source: §B.2 · p. 51
  • Weakening ah from 0.90 to 0.80 raises mean review burden by 23 percentage points and caps deploying-policy reliability at 0.775.At ah = 0.80, 311 of 496 policies route every case to review and qualify only by not deploying; values for ah ≥0.90 are lower bounds because the sweep ends at Y = 0.85.Source: §B.2 · p. 51, Table 5 · p. 52
  • Qualification is not monotone in ah: 472 policies qualify at ah = 0.85 versus 469 at ah = 0.90 because held-out lower-confidence-bound testing can reject a less conservative selected policy.GPT-5.6-Luna at Y = 0.76 qualifies at ah = 0.80 and ah = 0.90, but not at ah = 0.85.Source: §B.2 · pp. 51–52
  • Review burden weakly decreases in ah in discrete steps, reflecting each system’s reachable confidence thresholds rather than continuous responsiveness to the assumption.Gemini-3.1-Pro remains at 100% review for every ah ≤0.95 and drops to 33.6% only at ah = 1.00, illustrating the granularity limit.Source: §B.2 · p. 52

B.3 Process-Aware Success Definition

This section tests whether routing-quality and deployment conclusions persist when success additionally requires no disqualifying process defect. Routing-quality conclusions remain stable, but deployment frontiers and qualification burdens shift substantially.

Process-aware success definition

  • Process defects affect 21.9% of 12,000 runs, and requiring their absence lowers autonomous success by 4.5 to 13.1 percentage points across systems.Defect prevalence ranges from 12.7% for GPT-5.5 to 30.0% for Kimi-K2.7-Code.Source: §B.3 · p. 52

Sensitivity of deployment conclusions

  • Mean review burden rises from 53.0% to 69.8%, and at Y = 0.76 both Gemini Flash systems require 100% review, leaving no reachable deployable operating point.The process-aware definition strengthens the contrast in Section 5.6 because both contrasted systems become undeployable once process defects count against success.Source: §B.3 · p. 53, Table 6 · p. 53

Sensitivity of routing quality

  • Routing quality remains stable: AUROC rankings correlate at 0.87 across 16 systems, per-system AUROC changes by at most 0.032, and Qwen-3.7-Plus and Gemini-3.1-Pro remain among the best and worst signals, respectively.These results show that routing quality measures the same underlying property under either success predicate.Source: §B.3 · p. 53, Table 6 · p. 53

Qualification effects

  • The number of qualifying policies increases from 469 to 488 of 496 under the harder predicate, while GLM-5.2 shifts from not qualifying at 10.9% review to qualifying at 44.8% review.More demanding success criteria induce more conservative development-set thresholds that clear the held-out bound more comfortably, so qualified status is not monotone in predicate difficulty.Source: §B.3 · p. 53

B.4 Complete Per-System Results

Complete per-system results extend qualification across multiple reliability targets and analyze routing behavior by predicted verdict class. They show target-dependent review burdens and within-class routing signal that supports class-agnostic escalation policies.

Qualification across targets

  • 0.8% → 22.4% → 37.9% review across three targets for Opus 5, while Gemini-3.1-Pro remains at 100% for the upper two targets.Review burden exhibits step-function behavior, staying flat across adjacent targets when confidence has few distinct values before jumping.Source: §B.4 · p. 54

Routing behaviour by verdict class

  • 0.856 autonomous success for Yes falls to 0.336 for Indeterminate: Medically Ambiguous, despite mean stated confidence declining only from 0.894 to 0.797.Pooled success is 0.657 for No and 0.544 for Indeterminate: Lack of Data, indicating verdict-class-dependent performance with less proportional confidence variation.Source: §B.4 · p. 54
  • 0.639–0.668 within-class AUROC across Yes, No, ILD, and IMA shows the routing signal retains ranking power within every predicted verdict class.Because the signal is not merely a proxy for predicted class, class-agnostic thresholding does not discard information recoverable through class-conditional rules.Source: §B.4 · p. 54

B.5 Cost-of-Conflation Control: Always-Review-Indeterminate

The always-review-indeterminate control is not more expensive here because indeterminate verdicts are substantially less reliable, but its fixed class constraint prevents a tunable reliability–review frontier. READY therefore favors measured routing signals over inferring escalation from outcome classes, while acknowledging that combining both information sources could improve performance.

Empirical comparison

  • 48.9% versus 53.0% of cases were routed to review by always-review-indeterminate and signal-only policies, respectively, across 496 swept policies.The control required strictly more review in only 206 of 496 cells, so conflation did not produce wasted oversight on this benchmark.Source: §B.5 · p. 54, Table 9 · p. 57
  • At fixed reliability, always-review-indeterminate still required more review on 10 of 16 systems at 0.80 and 6 of 15 at 0.85, with mean differences of −2.2 and −5.2 percentage points.The comparison uses each policy’s minimum review burden at the stated reliability, removing threshold-grid overshoot.Source: §B.5 · p. 55, Table 9 · p. 57

Why the control works

  • Indeterminate adjudications had 0.493 pooled autonomous success versus 0.757 for definite adjudications, making predicted class genuinely informative about autonomous failure.Thus, the control is competitive on this cohort rather than demonstrating the expected oversight waste.Source: §B.5 · p. 55, Table 8 · p. 56

Structural limitation

  • Always-review-indeterminate imposes a hard per-system review floor and cannot trace a tunable reliability frontier because every indeterminate case is escalated.Its competitiveness depends on this benchmark’s indeterminate-class success profile, which the policy neither measures nor detects if it changes.Source: §B.5 · pp. 54–55

Contribution

  • The defended distinction is that escalation should use a measured routing signal rather than be inferred from an outcome class.The signal remains informative within each class, and a deployment allowed to combine both sources could outperform either policy alone.Source: §B.5 · pp. 54–56
Loading 2609.02095v1…