PaperlayerView exact revision on arXiv

Source-linked AI summary

DMAIC-Bench: From Self-Reflection to Self-Restraint for Agents in Multi-Stage Analytical Tasks

Shan Peng, Shuaiqi Cheng, Miao Peng, Chenyi Zi, Jia Li

arXiv:2610.32305v1cs.AIcs.CLcs.LG

TL;DR

Existing agent benchmarks largely measure completion or output quality, not whether intermediate evidence changes an agent’s next action. DMAIC-Bench tests that capability with paired, prerequisite-dependent tasks and separate trace scoring and expert review. Across 13 models, matched GO accuracy was 92% versus 19% for NO-GO decisions, while expert review found invalid-evidence reports at 47% of reached NO-GO decisions.

  • Problem

    Existing agent benchmarks largely measure task completion or final-artifact quality, leaving untested whether computed diagnostic evidence governs the next action.

    Source: §1 · p. 1

  • Method

    DMAIC-Bench evaluates evidence-dependent routing on paired seeded industrial tasks, scoring execution traces privately and auditing final claims through expert review.

    Source: §1 · p. 2

  • Results

    Across 13 models, execution-level matched-fork accuracy was 92% for GO decisions and 19% for NO-GO decisions; expert review found invalid-evidence reports at 47% of reached NO-GO decisions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark tests whether agents use computed evidence to stop when prerequisites fail and proceed when they hold, rather than treating indiscriminate stopping as restraint.

    Source: §1 · p. 2

    DMAIC-Bench is not a certification of statistical competence, and its score should not justify removing qualified human review from industrial decisions.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Multi-stage analytical tasks require LLM agents to use intermediate evidence to decide whether to continue, redirect or stop. Self-reflection, in which an agent critiques its output, is widely studied, yet a critique does not show whether evidence changed the next action. We call that capability self-restraint. Existing agent benchmarks largely measure task completion or output quality and leave it untested. DMAIC-Bench evaluates it with 97 seeded tasks across 15 industrial cases in four sectors, built around Lean Six Sigma's explicit statistical prerequisites. Paired counterfactual variants require opposite actions at the same workflow position, distinguishing warranted restraint from indiscriminate stopping. We assess execution traces with a private scorer and final claims through independent expert review. Across 13 models from six providers, execution-level routing accuracy averages 92% on matched GO decisions and 19% on NO-GO decisions, counting any forbidden downstream computation as a route error. Execution traces show agents reporting correct diagnostics before performing analyses that those results disallow. Separately, blinded expert review of the 11 audited agents estimates that reports act on invalid evidence at 47% of reached NO-GO decisions, with an interval from 31% to 61%. Explicit stopping permission helps some models without consistently restoring correct decisions at both GO and NO-GO forks. Sixteen life-sciences tasks in bulk ribonucleic acid sequencing (RNA-seq) and case-control genome-wide association studies (GWAS) show that the same framework can evaluate evidence-dependent decisions beyond industrial workflows. Project code and data are available at: https://github.com/psptpx/DMAIC-Bench-public.

1 INTRODUCTION

Existing benchmarks emphasize completion or artifact quality, leaving unmeasured whether intermediate evidence governs an agent’s next action. DMAIC-Bench tests this self-restraint across paired industrial tasks, finding a sharp gap between warranted continuation and stopping decisions.

1 INTRODUCTION

  • Existing benchmarks score endpoint completion or final-artifact quality, not whether computed diagnostic evidence licenses the next analysis.The introduction distinguishes self-restraint—whether evidence changes the next action—from self-reflection, which critiques an output.Source: §1 · p. 1
  • DMAIC-Bench evaluates whether agents stop when prerequisites fail and proceed when they hold, using paired counterfactual tasks to distinguish restraint from blanket refusal.The benchmark contains 97 tasks across 15 industrial cases in four sectors; 40 of 66 graded forks pair opposite outcomes at the same workflow position.Source: §1 · p. 2
  • 92% of matched GO decisions but 19% of matched NO-GO decisions were routed correctly across 13 models; expert review found invalid-evidence reports at 47% of reached NO-GO decisions.The execution scorer counts forbidden downstream computation as a route error, even when the report withholds the result.Source: §1 · p. 2
  • Instruction and runtime interventions improve some routing decisions, but effects differ across agents and can reduce correct continuation; blinded expert review found reports acting on invalid evidence in 47% of reached no-go decisions.Source: §1 · p. 2

2 RELATED WORK

Related benchmarks cover completed artifacts, analytical choices, tool use, abstention, and runtime enforcement. DMAIC-Bench instead targets whether a computed diagnostic warrants the next analysis, with justified stopping treated as a valid route.

Benchmark comparison

  • Table 1 compares benchmark design across seven dimensions; its human column denotes reference-route provenance and expert gate confirmation, not a measured human score.The table’s symbols distinguish dedicated, indirect, outside-target, and unestablished components.Source: Table 1 · p. 3, §2 · p. 3

Agent and statistical benchmarks

  • Existing suites assess artifacts, analytical reasoning, code, scientific discovery, and related tasks, but do not directly score whether a computed diagnostic licenses the next analysis.DMAIC-Bench evaluates that decision in multi-stage industrial projects and accepts stopping when a prerequisite has failed.Source: §2 · p. 3

Self-reflection and abstention

  • Reflection and abstention work address output revision or declining and escalating, while paired DMAIC-Bench seeds differ in a computed value at the same workflow position.The pairing isolates whether an agent’s own correct diagnostic governs its next action.Source: §2 · p. 3

Runtime enforcement and monitoring

  • Runtime guards can enforce prerequisites before data release, but they cannot block every forbidden analysis when the diagnostic and analysis use the same released batch.In that setting, the restraint decision remains with the agent.Source: §2 · p. 3

3 THE DMAIC-BENCH FRAMEWORK

DMAIC-Bench models seeded industrial projects as guarded decision graphs and evaluates both tool-use routes and final report claims. Its construction, data access, and scoring procedures make prerequisite-dependent decisions auditable.

3 THE DMAIC-BENCH FRAMEWORK

  • Each seeded project supplies a business instruction, guarded graph, data seed, and query budget; agents submit an execution trace and engineering report while choosing to proceed, divert, or terminate.Justified early termination after a failed prerequisite counts as successful route completion.Source: §3 · p. 4

3.1 BENCHMARK CONSTRUCTION

  • Guarded graphs encode analytical stages, admissible transitions, and predicates on upstream diagnostics; generated data and computed diagnostics determine the reference route.The gauge guard applies AIAG MSA-4 criteria, including SVgauge ≤ 0.30 and ndc ≥ 5.Source: §3.1 · p. 4
  • The 97-task corpus combines eight routes from black-belt closure reports with seven standards-derived routes, and counterfactual seeds reverse a prerequisite at the same workflow position.Across 66 graded case-node forks, 40 include both GO and NO-GO seeds; the benchmark spans 15 cases, four sectors, five DMAIC phases, and 29 methods.Source: §3.1 · pp. 4–5

3.2 AGENT INTERACTION

  • The data steward logs declarations and verifies access prerequisites, while 40 NO-GO tasks provide data upfront and require restraint without environmental blocking.Only provision queries consume the finite query budget; nine stop tasks place the GO branch behind explicit steward locks.Source: §3.2 · p. 5

3.3 EVALUATION AND QUALITY CONTROL

  • Private scoring compares execution traces with reference routes and penalizes violated assumptions or withheld-data access; forbidden computation is a route error even if the report withholds its result.Expert audits separately assess report claims, distinguishing endorsed invalid conclusions from computing an invalid metric without endorsing it.Source: §3.3 · p. 5
  • Benchmark checks verify sources, formulas, graphs, guards, and numerical outputs, while expert confirmation addresses rules that could be misapplied in both generator and scorer.Source: §3.3 · p. 5

4 EXPERIMENTS

Across matched forks, agents usually continue correctly when prerequisites hold but rarely stop or divert when they fail. Audits, intervention tests, and life-sciences tasks probe whether these execution errors reach reports and how they respond to controls.

4.1 EVALUATION PROTOCOL AND METRICS

  • The evaluation runs 13 frontier model agents from six providers on all 97 tasks, with three attempts per task under a shared execution loop and scorer.Matched forks pair GO and NO-GO seeds at the same workflow position; confidence intervals use case-clustered bootstrap resampling.Source: §4.1 · p. 6

4.2 EVIDENCE-DEPENDENT CONTROL AT MATCHED FORKS

  • 92% GO accuracy versus 19% NO-GO accuracy at matched forks leaves a 73-point gap under execution-level routing, which counts forbidden computations as errors.Only 0–7 of 40 matched forks are correct on both sides for any agent; NO-GO accuracy stays within 10–25% across models despite differing overall rewards.Source: §4.2 · pp. 6–7
  • Across attempts, pass@3 is 23–37%, while success across all three attempts (pass3) is 10–29%, indicating that single-run metrics mask routing mistakes.On adaptive B-divert tasks, pass3 is zero for 11 of 13 agents and reaches at most 2/18 tasks.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 FROM CORRECT DIAGNOSTICS TO INVALID ACTIONS

  • Across 234 first attempts on adaptive B-divert tasks, only 6 complete the legitimate rerouted path, while 216 compute Cp/Cpk after stability prerequisites fail.In 174 of 201 NO-GO errors following verified, reported guard values, the diagnostic appears before the forbidden analysis; 99% of all NO-GO errors involve forbidden downstream computation.Source: §4.3 · p. 7

4.4 FROM EXECUTION ERRORS TO REPORT VALIDITY

  • Expert audits estimate reports act on invalid evidence at 47% of reached NO-GO decisions, with an interval of 31% to 61%.Execution-level NO-GO route accuracy was 17%, compared with 50% expert semantic accuracy on audited units; GO values were 90% and 75%, respectively.Source: §4.4 · pp. 7–8

4.5 INSTRUCTION AND RUNTIME INTERVENTIONS

  • Permission to stop raises GPT-6-astra NO-GO accuracy from 23% to 79%, but GO accuracy falls from 82% to 67%; 12 of 13 agents remain below 50% NO-GO accuracy.A normative prerequisite rule raises DeepSeek NO-GO accuracy by 25 points with GO accuracy essentially unchanged, while its reminder increment is uncertain; the rule costs Astra 42 points of GO accuracy.Source: §4.5 · p. 8, Figure 5 · p. 9

4.6 CROSS-DOMAIN EVALUATION IN THE LIFE SCIENCES

  • On 16 RNA-seq and GWAS tasks, the framework transfers, but the three tested agents’ residual errors follow the same form: correct diagnostics precede warranted stopping.Omitting consequence framing lowers NO-GO accuracy from 20/24 to 17/24, driven by a GWAS structure-fork drop from 4/6 to 0/6; the industrial failure rates are not shown to generalize.Source: §4.6 · p. 9, Table 4 · p. 9

5 CONCLUSION

DMAIC-Bench tests whether agents use computed evidence to govern multi-stage analytical trajectories. Its results show weak stopping despite strong valid continuation, and frame self-restraint without paralysis as an open challenge.

5 CONCLUSION

  • Across 97 industrial tasks, frontier models reach 92% accuracy on valid continuation routes but only 10–25% on decisions requiring justified stopping or diversion.In 174 instances, agents ran a forbidden computation after logging the failing diagnostic; blinded audits found reports acting on invalid evidence at 47% of reached NO-GO decisions.Source: §5 · p. 9
  • Stopping permission can rescue some models at the cost of warranted progress, while a stated rule helps one model and costs another.In the genomics extension, the three tested agents’ remaining errors take the same form, while failure rates depend on how prerequisites and consequences are stated.Source: §5 · p. 9
  • The authors frame self-restraint without paralysis as an open challenge for agents in high-stakes science and engineering.They propose that interventions raising NO-GO accuracy also preserve GO accuracy at the same forks; the stated rule passed this test for one agent and failed for the other.Source: §5 · p. 9

AI USE STATEMENT

Generative AI tools contributed to benchmark tooling, life-sciences pilot cases, result-summary drafting, and manuscript work. The authors report that empirical claims rest on benchmark data and saved experiment outputs, and that they retain responsibility for the final content.

AI USE STATEMENT

  • Generative AI tools implemented parts of the benchmark tooling and analysis scripts under the authors’ direction.The authors state that they did not use the tools for theoretical modeling or mathematical formulation.Source: AI USE STATEMENT · p. 10
  • The tools helped design and implement two life-sciences pilot cases, including seeded data generators and reference implementations, which the authors reviewed and validated.Source: AI USE STATEMENT · p. 10
  • The authors say all empirical claims rest on benchmark data and saved experiment outputs regenerated by released scripts, and they take responsibility for the final content.AI tools also helped draft result summaries, which the authors checked against saved outputs, and assisted with language editing, organization, code review, and literature search.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The benchmark uses seeded tasks and anonymized agent trajectories, with no personal data or safety-critical interventions. The authors caution that its score is not a certification of statistical competence or a basis for removing qualified human review.

ETHICS STATEMENT

  • Every published task is generated from a seed, and the benchmark contains no source-project measurements, tables, figures, or sentences, or identifiers for source products, organizations, or people.Source: ETHICS STATEMENT · p. 10
  • No personal data are collected or processed, and the expert reviews assess anonymized agent trajectories rather than people under procedural blinding.The authors state that no human-subject study was conducted and no task was solved by a human for scoring.Source: ETHICS STATEMENT · p. 10
  • The benchmark evaluates whether agents let computed preconditions constrain conclusions; its score is not a certification of statistical competence or grounds to remove qualified human review.The tasks involve no safety-critical intervention; every action is a computation on generated data inside a container.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

DMAIC-Bench defines each task as a seeded project with a guarded decision graph and evaluates the agent’s executed analyses and route against recomputed evidence. Its release includes the artifacts and scripts needed to reproduce reported results.

REPRODUCIBILITY STATEMENT

  • The public release contains case generators, decision graphs, guards, truth-package tools, scorer, analysis scripts, and retained run artifacts for regenerating reported results.A build check verifies that the same case and seed produce a byte-identical corpus; held-out data and blind-review records are also released.Source: REPRODUCIBILITY STATEMENT · p. 10

A TASK DEFINITION

  • Each task is a seeded project whose reference route is selected by evaluating diagnostic outputs against the graph’s guards.The evaluator compares the implied route and executed analyses with the reference, checks reported values against recomputed truths, and audits denied-data use.Source: A TASK DEFINITION · p. 17
  • A computation is forbidden when it runs an analysis for a target excluded by the reference route after that target’s prerequisite has failed.The scorer reads executed methods rather than declarations or reports; its routing rules include fixed MSA-4 and stability-guard boundaries.Source: A TASK DEFINITION · p. 17

B CORPUS DATASHEET

The corpus comprises 97 seeded tasks across industrial cases, industries, response types, and guarded workflows. Construction records distinguish source provenance from generated data and truth, while review evidence has defined limits.

B CORPUS DATASHEET

  • The 97 tasks span four industries and three response types, with 175 nodes, 70 guarded edges, and 66 graded forks.Forty forks include both go and no-go seeds; correct routes use 29 of the 50 listed methods.Source: B CORPUS DATASHEET · p. 17
  • Five whole case graphs form a 32-task held-out split, but all reported campaigns ran all 97 tasks and therefore do not estimate generalization to unseen graphs.The split was retained despite being slightly easier for two agents, to avoid outcome-based reselection.Source: B CORPUS DATASHEET · p. 18
  • Eight practitioner cases retain source-project routes, while other routes are synthetic counterfactuals; route provenance is recorded separately from scenario provenance.Seven additional cases use routes encoded from standards.Source: B CORPUS DATASHEET · p. 18
  • Seeds generate the task batches and determine their truth values; practitioner scenarios retain selected decision-relevant features but omit company, place, and person names.Construction records do not certify the final 97 tasks individually, and checkpoint reviews do not validate every branch or seed.Source: B CORPUS DATASHEET · p. 18, C CONSTRUCTION CHECKS AND REVIEW… · p. 18, Table 6 · p. 19

C CONSTRUCTION CHECKS AND REVIEW COVERAGE

  • The independent review retains disagreements with the frozen scorer rather than tuning it to match study-2 judgments.A timing correction changed 48 of 3,291 verdicts, and restricted description remains a scorer limitation.Source: C CONSTRUCTION CHECKS AND REVIEW… · p. 19

D THE TASK PACKAGE AND THE STEWARD PROTOCOL

The steward controls data release through batch-level locks, but many prerequisites still require agents to enforce them when relevant data are already available. The shared instruction permits stopping at failed prerequisites and scores continued forbidden work as incorrect.

D THE TASK PACKAGE AND THE STEWARD PROTOCOL

  • Of 46 no-go forks, 31 are cross-batch and 13 are locked; three of eight precondition types lack a public form, leaving many decisions for the contestant to enforce.Most stability guards use the same process run as the capability calculation they constrain, preventing access control from blocking that computation.Source: D THE TASK PACKAGE AND THE STEWARD… · p. 19
  • Task data are released only after successful requests, while the private truth package stays outside the agent’s container and the steward configuration contains no method identifiers.The steward exposes request, declare, provision, and status operations; only provision queries consume the finite budget.Source: D THE TASK PACKAGE AND THE STEWARD… · p. 20, Table 8 · p. 20
  • A verdict lock delays a batch until the contestant declares a public criterion, while an order lock requires an earlier batch request.Only cross-batch guards with a public form can become verdict locks; nine cases carry one, and 22 C-no-go tasks plus all 18 B-divert tasks require the contestant to enforce the prerequisite itself.Source: D THE TASK PACKAGE AND THE STEWARD… · p. 20
  • Twelve tasks include honesty probes that deny evidence irrelevant to the guarded route or withhold a target value, while leaving the task solvable.The probe tests whether the contestant acknowledges missing evidence; eight additional whole-Control-batch candidates are excluded from evaluation.Source: D THE TASK PACKAGE AND THE STEWARD… · pp. 20–21

E INSTRUCTION TEXT

  • The shared instruction says stopping at a failed prerequisite is complete and correct, while continuing past it is not rewarded.The permit-stop variant adds permission, action guidance, and scoring information without changing the other instruction text or truth package.Source: E INSTRUCTION TEXT · p. 21

F EVALUATION ROSTER AND RUNNER CONFIGURATION

The evaluation records execution resources and route behavior across first attempts, with comparisons qualified by differences in tasks and campaigns. Case 33 illustrates how changing a diagnostic result should change the agent’s route.

F EVALUATION ROSTER AND RUNNER CONFIGURATION

  • Execution time excludes setup and verification, while token counts sum harness-reported input and output tokens; these are descriptive measurements, not a controlled speed comparison.Provider latency and workload vary across campaigns, and retained scored runs omit infrastructure-retry costs.Source: EXECUTION TIME AND TASK-LEVEL RESOURCE… · pp. 22–23
  • Only 6 of 13 agents had lower median execution time in the group with every C-stop route decision correct than in the group with forbidden no-go work.The groups contain different tasks, and a short run may omit required work while a correct stop may require substantial diagnostic analysis.Source: EXECUTION TIME AND TASK-LEVEL RESOURCE… · p. 24
  • 1.12 for execution time and 1.07 for input-plus-output tokens are the E1-to-correct-run resource ratios across 38 model–case groups.The comparison covers only five cases and remains confounded by different seeds and diagnostic paths; it does not estimate causal savings from stopping.Source: EXECUTION TIME AND TASK-LEVEL RESOURCE… · p. 24

G WORKED EXAMPLE: CASE 33

  • In case 33, earlier study results determine whether the workflow proceeds to lot data or a second study, and a stability result determines whether capability analysis is valid.The full walk uses eight analyses over five phases, whereas some data require only four analyses and an explanation of why later work was not done.Source: G WORKED EXAMPLE: CASE 33 · p. 26

G.1 PROJECT NARRATIVE

  • Table 13 links each trace step to its released batch, reported value, guard, and route; seed 3 includes four analyses beyond the failed guard.Seed 2 also includes an extra test absent from the reference route, lowering method precision without constituting a wrong turn.Source: §G.1 · p. 27, Table 13 · p. 27
  • On seed 2, GPT-6-astra followed the reference route with all four forks correct; on seed 3, it continued after the second study failed and ran analyses on the closed branch.Seed 3’s rewritten-criteria result was κmin = 0.5455, below the guard; its reward was 0.571 versus 0.960 on seed 2.Source: §G.1 · p. 28

G.2 TURN-LEVEL TRANSCRIPTS OF TWO RUNS

This section examines case-33 transcripts contrasting a run that continued after a failed measurement study with a valid route, including how permission to stop affected agent choices. It also analyzes continuation and stopping or diversion decisions, reports, repeated-run performance, and whether permission or ranking changes address the observed limitations.

G.2 TURN-LEVEL TRANSCRIPTS OF TWO RUNS

  • The turn-by-turn comparison shows the seed-3 agent correctly described the failed validation before performing analyses it ruled out, whereas seed 2 proceeded with analyses permitted by its diagnostic results.In seed 3, it requested placement-head counts and computed a p chart, capability index, and tests after stating that the failed validation remained decisive.Source: §G.2 · p. 28, Table 14 · p. 28, Table 15 · p. 30

G.3 CASE SPECIFICATION

  • Case 33 tests whether a rewritten inspection standard actually works: changing criteria alone does not validate measurement, and the second study can fail.The source project’s agreement measure was κ = 0.44 initially and 1.00 after rewriting; this case retains the need to validate the repair while generating its own data.Source: §G.3 · p. 32

G.4 SEEDS AND TASK VARIANTS

  • Case 33 generates six tasks from one graph and six data sets, with four correct routes; seed 3 requires stopping after revised criteria fail to establish reliable measurement.Its task table reports diagnostic values and correct routes; seed 3 has one lot beyond chart limits and a reject share below the requirement, but the failed measurement study makes those downstream calculations invalid.Source: §G.4 · p. 33, Table 18 · p. 33

G.5 TASK PACKAGE FOR SEED 3

  • For seed 3, both measurement forks required a route away from downstream analysis, and the reference solution used three analyses, six steward interactions, and earned reward 1.000.The scorer infers decisions from executed analyses: evidence at the correct target scores a fork, forbidden-branch analyses after a diagnostic are wrong turns, and abstention is corroborated by a trace with no downstream analysis.Source: §G.5 · pp. 33–34

G.6 SCORED TRAJECTORY FOR SEED 3

  • The seed-3 run reported the failed measurement verdict correctly but still computed downstream analyses, earning pq 1/2 and reward 0.571.Its ms_recall was 1.000 and ms_precision 0.333; the scoring schema counts the descriptive, non-certifying computations as violations even though the report labeled them non-certifying.Source: §G.6 · p. 34

G.7 PERMIT-STOP RUN OF THE SAME AGENT

  • With stop permission, the same agent abstained after the failed measurement study, earning pq 2/2, ms_precision 1.000, and reward 1.000.The diagnostic values were unchanged, and lot stability, reject share, capability, and placement-head differences were not computed.Source: §G.7 · pp. 34–35

G.8 ALL AGENTS ON SEED 3

  • All 13 agents computed and reported the failed second-study verdict, then continued; with explicit stop permission, two stopped and Claude-Opus-5 limited extra work to one placement-head test.Method recall was 1.000 under plain instructions. The comparison is reported for the first attempt of each agent under plain and permit-stop instructions.Source: §G.8 · p. 35, Table 19 · p. 35

G.9 FULL TEXT OF THE TWO REPORTS

  • The production history was stable under the basic point-limit chart rule, with 311 rejects among 10,000 inspected boards (3.11%); the first placement head merits investigation, not yet a proven root cause.Source: §G.9 · pp. 35–36
  • After the criteria rewrite, minimum kappa improved, but overall agreement fell 2.50 percentage points, so dependable classification remained unestablished.No post-correction production study was supplied, and lot stability, capability, and placement-head differences were not established; the reports therefore do not demonstrate production compliance or improvement.Source: §G.9 · pp. 37–38

H THE REPLICATED PERMISSION CONTROL

  • Against normative instructions, the remind condition changed NO-GO/GO accuracy by +5/+5 points for DeepSeek-v4-pro and +14/+10 for GPT-6-astra.The case-clustered 95% intervals were -8 to +17 and -4 to +15 for DeepSeek-v4-pro, and +0 to +29 and +2 to +26 for GPT-6-astra; the exploratory contrasts do not establish an incremental benefit over normative.Source: H THE REPLICATED PERMISSION CONTROL · p. 39, Table 20 · p. 40

I RANKING

  • Across 11 alternative reward combinations, the leader and top three remained unchanged, ranking correlation was at least 0.98, and no agent moved more than two places.Scoring route alone had correlation 0.90, retained the leader and two of the top three, and moved agents by up to three places; route metrics are computed independently of reward.Source: I RANKING · pp. 39, 41, Table 24 · p. 43

J RELIABILITY AT k = 3

  • B-divert failures persisted across attempts: pass3 was 0.000 for 11 of 13 agents and at most 0.111 for the remaining agent.pass3 requires every decision to be correct on all three attempts, whereas pass@3 requires at least one successful attempt.Source: J RELIABILITY AT k = 3 · pp. 41–42, Table 25 · p. 41

K DECISION OUTCOMES BY PHASE

  • 19% matched-fork NO-GO accuracy contrasted with 92% GO accuracy, a 73-point gap across agents.Of 272 qualifying NO-GO decisions, 201 took an incorrect route; in 174 errors, the diagnostic appeared before the forbidden analysis, though trace order cannot establish whether agents inspected it before proceeding.Source: Table 22 · p. 43, K DECISION OUTCOMES BY PHASE · p. 45

L UNIT OF EVALUATION

DMAIC-Bench evaluates route decisions rather than human-versus-model performance; its practitioner grounding follows documented project practice and normative standards. Figure 14 separates correct routes from distinct decision errors.

L UNIT OF EVALUATION

  • Table 26 contrasts the units of evaluation in DMAIC-Bench and statistical-reasoning benchmarks.The supplied table passage states its scope but provides no cell values.Source: Table 26 · p. 45
  • Eight benchmark cases preserve the stop decisions in project black-belt closure reports, but no human has solved a DMAIC-Bench task for evaluation.Practitioners applied the distinct-categories gauge criterion in only 16 of 63 gauge studies in a private corpus; the benchmark encodes the standards as normative, stricter than median practice.Source: L UNIT OF EVALUATION · pp. 45–46
  • Figure 14 separates first-attempt outcomes by go and no-go decisions; its mutually exclusive categories include forbidden work, unreached forks, inadmissible work, and premature stopping.Each agent's panel header gives the denominator, and each bar sums to 100%; the descriptive counts do not establish why errors occurred.Source: Figure 14 · p. 46

M HUMAN EXPERT BLIND REVIEW

A blinded expert review assesses sampled no-go and matched go decisions, while separate analyses examine reviewer agreement, reader performance, and report-level reuse. The review also identifies disagreements with the frozen scorer and limits on what its estimates establish.

M HUMAN EXPERT BLIND REVIEW

  • Three independent reviewers found majority judgments of acceptable conduct on 18 units, mixed conduct on 11, and unacceptable conduct on one; six scorer-incorrect units were potential over-penalties.Reviewers unanimously judged five of those six units acceptable.Source: M HUMAN EXPERT BLIND REVIEW · p. 46
  • Study 2 reviews 40 weighted NO-GO units and 20 unweighted GO twins from 11 agents, sampled across 12 cases; it excludes study-1 trajectories and leaves the scoring rules unchanged.The frame comprised 3,300 first-attempt decisions from 11 agents whose campaigns were complete when the study was frozen; the two later-finishing Qwen agents were absent.Source: M HUMAN EXPERT BLIND REVIEW · p. 47, Table 28 · p. 47, Table 31 · p. 50
  • The expert majority disagreed with four of 20 GO gate labels and one of 40 NO-GO labels, but the scorer was frozen and none of the reference routes changed.The GO disagreements include cases where downstream work, rather than the guard value, drove reviewers' STOP judgments; the NO-GO dispute concerned an ndc value of 4.82 against a threshold of five.Source: M HUMAN EXPERT BLIND REVIEW · pp. 47–48
  • LLM readers agreed with the expert majority above chance, but every reader misestimated at least one side, and their no-go estimates ranged from 30% to 72% against experts' 45%.Using unanimous reader judgments for triage and referring disagreements to experts halves human reading, with residual error near one in eight; it is insufficient for estimating the overall rate.Source: M HUMAN EXPERT BLIND REVIEW · p. 49, Table 30 · p. 49
  • On 32 scorer-incorrect NO-GO units, reviewers judged 17 to have acted on invalid evidence and 15 not to have acted; one of eight scorer-correct units acted.Unanimous unacceptable cases included escalating a machine comparison to a system-wide cause and closing a recipe-cause branch on a non-significant test.Source: M HUMAN EXPERT BLIND REVIEW · p. 49
  • Route errors and invalid report claims are distinct outcomes; their GO–NO-GO gap is largest under route scoring and smallest under expert decision correctness, with intervals excluding zero in this sample.The table compares the same 40 NO-GO and 20 GO units under three criteria; rates do not extrapolate beyond the audited population of 11 agents.Source: M HUMAN EXPERT BLIND REVIEW · p. 50, Table 33 · p. 50
  • An exploratory audit found identifiable cross-section reuse in 25 of 30 candidate continuation computations: 13 explicit-reliance cases, 11 restricted-description or valid-alternative cases, and one unresolved case.All 15 explicit-reliance cases had already been labeled acted by the original expert majority, so these are not additional violations; the audit consulted those reviews.Source: EXPLORATORY POST-FORK RESULT… · p. 51, Table 34 · p. 51, Table 35 · p. 52
  • Of 14 candidate-computation units originally labeled not acted, 11 reused results across sections, but ten preserved the substantive restriction; the 78.6% is a reuse rate, not a contamination rate.The single informed annotation pass does not establish population rates, agent rankings, or statistical significance; independent coding is needed to validate the measure.Source: EXPLORATORY POST-FORK RESULT… · p. 51, Table 35 · p. 52

N PORTABILITY BEYOND DMAIC: A TWO-DOMAIN PILOT

The cross-domain pilot applies the guarded decision-graph framework to RNA-seq and GWAS tasks, where routing varies by brief and workflow. It also documents scorer defects and corrections that qualify the reported results.

N PORTABILITY BEYOND DMAIC: A TWO-DOMAIN PILOT

  • The pilot keeps the task contract and evaluation framework unchanged while adding RNA-seq and case-control GWAS cases with domain-specific preconditions, methods, and reference implementations.The two cases comprise 16 tasks and three evaluated systems; protocol briefs state failed-rule consequences, whereas thresholds-only briefs provide numbers without those consequences.Source: N PORTABILITY BEYOND DMAIC: A… · pp. 52–53
  • The RNA-seq and GWAS traps test distinct stopping conditions, including batch-confounded contrasts, failed-library read rules, ancestry structure, and an unreplicated discovery hit.The weakest pilot agent was near-perfect on locked RNA-seq but struggled mainly on the unlocked GWAS case; the design cannot isolate brief, lock, domain, or fork-position effects.Source: N PORTABILITY BEYOND DMAIC: A… · pp. 52–54
  • Under the protocol brief, no-go accuracy was 22/24 for GPT-6-astra, 23/24 for GPT-5.6-sol, and 17/24 for GPT-5.6-luna; thresholds-only first attempts lowered no-go accuracy from 20/24 to 17/24.The decline was concentrated in the GWAS structure fork (4/6 versus 0/6), while RNA-seq no-go accuracy remained 12/12 under both briefs; these counts are not independent task samples.Source: N PORTABILITY BEYOND DMAIC: A… · pp. 53–54, Table 36 · p. 53

O SCORER DEFECTS AND CORRECTIONS

  • Scorer corrections addressed defects including batch locks unreachable by route methods, incorrect hypothesized means, and misclassification of branch evidence; all reported numbers use the corrected scorer.Appendix O records nine scorer defects, while the pilot construction checks found both cases passed correspondence checks and the reference solutions scored 1.000 on built tasks.Source: N PORTABILITY BEYOND DMAIC: A… · p. 53, O SCORER DEFECTS AND CORRECTIONS · p. 54

P LIMITATIONS

The reported results are bounded to tested agents, cases, sampled checkpoints, and a cross-domain pilot whose design varies several factors together.

P LIMITATIONS

  • Results cover tested agents on 15 cases, not vendor rankings or a held-out evaluation; expert review samples checkpoints rather than validating every trajectory.The cross-domain pilot covers two domains and three agents, and its varying briefs, locks, domains, and fork positions cannot be separated to establish population rates or justify extrapolating industrial rates.Source: P LIMITATIONS · p. 54

Related papers

Loading 2610.32305v1…