PaperlayerView exact revision on arXiv

Source-linked AI summary

From Evidence to Action: How Tool-Using Agents Fail

Hongzhan Lin, Shidong Cao, Ziyang Luo, Wenhao Chai, Mong-Li Lee, Wynne Hsu

arXiv:2610.07753v1cs.CLcs.AI

TL;DR

Correct outcomes do not establish that consequential actions were supported by evidence available beforehand, and it has been less directly characterized how this support holds across dependent workflows. The paper introduces SAFEACTBENCH and deterministic trajectory evaluation to study that chain. Across model–harness configurations, static assessment can be strong while interactive execution is weaker, with failures often arising before supported execution.

  • Problem

    Whether each consequential action is supported by evidence established before execution is less directly characterized, despite evaluation of interactive task completion and state changes.

    Source: §1 · pp. 1–2

  • Method

    SAFEACTBENCH evaluates observable trajectories across static decisions and interactive workflows using a provenance-bound Evidence Ledger and deterministic evaluator.

    Source: §1 · pp. 2–3

  • Results

    Strong static action assessment coexists with weaker interactive execution; conditional on evidence completion, CAS is 93.2%–100% for nine configurations, with DeepSeek–Inspect at 83.3%.

    Source: §5.3.1 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    Failures arise not only from missing information but also from how agents use established evidence when deciding and executing actions.

    Source: §6 · p. 9

    The analysis captures observable support rather than internal reliance, and multi-action interventions varying evidence at each action checkpoint remain future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Tool-using agents make consequential changes to external state, yet correct outcomes do not guarantee that their actions were supported by evidence established beforehand. We study where this evidence-to-action chain breaks as agents move from deciding whether to act to executing single actions and dependent workflows. Across ten model-harness configurations, strong static action assessment can coexist with much weaker interactive execution. Failures often begin before execution: agents stop with incomplete investigation or act before required evidence is established. Once required evidence is obtained, single-action execution is usually reliable, while multi-action workflows additionally expose unresolved prerequisites and incomplete execution. For this analysis, we introduce SafeActBench, comprising 656 cases across six operational domains and five protocols that progress from static action judgment and investigated non-action to single- and multi-action workflows. A provenance-bound Evidence Ledger and deterministic trajectory evaluator track what information was established, when actions occurred, and whether downstream dependencies were satisfied. These results show that failures arise not only from missing information, but also from how agents use established evidence when deciding and executing actions.

1 INTRODUCTION

Consequential tool use requires evidence to be established before an action, not merely a correct endpoint. The paper introduces a benchmark and trajectory-based evaluation to locate breaks in that chain.

1 INTRODUCTION

  • A permitted action may succeed with correct arguments yet remain unsupported when required evidence was not established beforehand.The introduction’s duplicate-charge example also shows why later actions can inherit missing support.Source: §1 · p. 1, Figure 1 · p. 2
  • Prior interactive evaluation assesses task completion and state changes, but whether pre-action evidence supports the specific entity and action is less directly characterized.The paper frames support as obtaining current evidence, binding it to the exact target and arguments, and propagating actual results to dependent actions.Source: §1 · pp. 1–2
  • Strong static action assessment can coexist with much weaker interactive execution, with failures often beginning in incomplete investigation or premature action.Execution is usually reliable once required evidence is established, but multi-action workflows also expose unresolved prerequisites and incomplete execution.Source: §1 · pp. 2–3
  • SAFEACTBENCH comprises 656 cases across six operational domains and uses a provenance-bound Evidence Ledger and deterministic trajectory evaluation to localize chain failures.Its protocols span static judgment, investigated non-action, single actions, and dependent multi-action workflows.Source: §1 · pp. 2–3

2 RELATED WORK

Related work evaluates interactive task completion, state-changing behavior, and decisions to seek information or abstain. SAFEACTBENCH focuses on whether evidence established before execution supports the specific action and its dependencies.

Interactive evaluation of tool-using agents

Evidence, policy adherence, and state-changing actions

  • Prior work studies risky tool use, policy violations, stale or manipulated observations, and policy checks bypassed despite reaching the final state.SAFEACTBENCH instead checks pre-execution support for the specific entity and state, plus binding chained actions to prior results and dependencies.Source: §2 · p. 3

Information seeking, abstention, and action calibration

  • Research on information seeking, help, stopping, and abstention makes information acquisition and action decisions central, while SAFEACTBENCH also evaluates action binding and cross-action dependencies.It includes investigated non-action and requires evidence to be established for the specific action in the observed trajectory.Source: §2 · p. 3

3 THE EVIDENCE-TO-ACTION CHAIN

The evidence-to-action chain evaluates whether information available before execution supports the particular action, and whether later actions rely on actual predecessor results. It covers both justified action and investigated non-action.

3 THE EVIDENCE-TO-ACTION CHAIN

  • The framework distinguishes information-gathering calls from consequential actions and evaluates action-specific support from information available before execution.SAFEACTBENCH spans a static decision setting and four interactive regimes, using a provenance-bound ledger and deterministic evaluator to check evidence, exact actions, and dependencies.Source: §3 · p. 3, Figure 2 · p. 4

3.1 EVIDENCE BEFORE ACTION

  • For a target action, evidence must establish the relevant entity, state, and values before execution; matching values from another entity do not suffice.The criterion concerns observable interaction, not what the model internally relied on.Source: §3.1 · p. 3

3.2 ACTING AND NOT ACTING

  • When no action should occur, the agent must investigate the task-relevant evidence and then stop; stopping before resolving obtainable evidence is insufficient.The paper calls this investigated non-action; unresolved or violated prerequisites after investigation mean the agent should not act.Source: §3.2 · pp. 3–4

3.3 CHAINED ACTIONS

  • In chained tasks, dependent actions must wait for required predecessor outputs or resulting states, while independent actions may occur in any dependency-consistent order.A later notification requiring a refund identifier must use the identifier returned by the relevant refund; reaching the intended final state alone is insufficient.Source: §3.3 · p. 4

4 SAFEACTBENCH

SAFEACTBENCH specifies required evidence, supported actions, and expected results, then evaluates trajectories against those requirements. Its protocols progress from static decisions to dependency-constrained workflows.

4 SAFEACTBENCH

  • SAFEACTBENCH cases specify evidence required before an action, the action supported by that evidence, and its expected result or state change.Agents may take different interaction paths when evidence and action dependencies are satisfied.Source: §4 · p. 4

4.1 TASK STRUCTURE

  • Each action’s evidence requirements may cover its target entity, current state, amount, authorization, version, or other task-specific conditions.Action specifications also record the operation, target, arguments, and downstream results or state changes.Source: §4.1 · p. 4

4.2 INTERACTION PROTOCOLS AND EVIDENCE PROVENANCE

  • Legacy requires a structured judgment; V0–V3 progressively require investigated non-action, a supported single action, actual-result propagation, and dependency-respecting execution.V3 permits any topological order consistent with a directed acyclic dependency graph.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 DETERMINISTIC VERIFICATION AND SCOPE

  • The deterministic evaluator checks protocol-specific investigation, tools, targets, arguments, produced results or states, dependencies, and terminal conditions without inspecting hidden reasoning or using an LLM judge.SAFEACTBENCH contains 656 cases across six operational domains.Source: §4.3 · pp. 5–6

5 EXPERIMENTS

Across ten model–harness configurations, strong static action assessment coexists with weaker interactive execution. Trajectory analyses locate failures across investigation, action timing, harness choice, responses to evidence, and behavior under missing evidence.

5.1 EXPERIMENTAL SETUP

  • The evaluation pairs each model’s family-associated harness with Inspect and records trajectories for deterministic checks against private case specifications.The main evaluation reports exact case success by protocol and unweighted protocol- and domain-macro averages.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • GLM–ZCode and DeepSeek–DSH exceed 96% on Legacy, yet interactive performance is much weaker; on the same V1 cases, static accuracy is at least 95% while interactive ECS is at most 52%.DeepSeek–DSH remains around 60% on V1–V3, whereas GLM–ZCode falls between 12.1% and 34.1%.Source: §5.2 · p. 7

5.3.1 WHERE DOES THE EVIDENCE-TO-ACTION CHAIN BREAK?

  • BSR spans 21.7–62.9% and PAR 37.0–66.9%, while CAS is 93.2–100% for nine configurations; V2/V3 also expose unresolved prerequisites and incomplete workflows.Gap and Part. can co-occur, so they are overlapping diagnostics rather than an additive decomposition.Source: §5.3.1 · p. 7, Table 3 · p. 7

5.3.2 HOW DOES HARNESS CHOICE CHANGE AGENT BEHAVIOR?

  • Family-associated harness effects diverge: DeepSeek gains 4.4 percentage points over Inspect, while GLM’s ZCode is 6.8 points below Inspect.On Qwen, the two harnesses disagree on 134 of 570 cases (23.5%) under the stated majority-success criterion.Source: §5.3.2 · p. 8, Table 4 · p. 8

5.3.3 HOW SENSITIVE ARE AGENTS TO RETRIEVED EVIDENCE?

  • Withholding reduces action probability by 37.2–45.2 points, yet 46.5–53.5% of completed Withheld episodes still include an action; contradiction reduces action probability by 54.5–70.0 points on its eligible subset.The controlled interventions used 43 V1 cases, with Contradicted variants constructed for 22 cases.Source: §5.3.3 · p. 8

5.3.4 WHAT CHANGES ACTION UNDER MISSING EVIDENCE?

  • Evidence packages lower action counts to 18/43, 14/43, and 13/43, while requester claims lower them further to 6/43, 10/43, and 6/43 across DeepSeek, GLM, and Qwen.Only Qwen’s evidence-package interval excludes zero; all three requester-claim intervals exclude zero.Source: §5.3.4 · pp. 8–9
  • Under withholding, distractors add 1.0–1.2 information calls per episode without a corresponding clear action-probability change; urgency and warnings show no clear effect.The distractor-related action-probability intervals include zero for all three configurations.Source: §5.3.4 · p. 9, Figure 4 · p. 9

6 CONCLUSION AND FUTURE WORK

The paper frames consequential tool use as an observable evidence-to-action chain and reports gaps between static assessment and interactive execution. Its analysis measures observable support rather than internal reliance, with multi-action interventions identified as future work.

6 CONCLUSION AND FUTURE WORK

  • Across ten configurations, strong static action assessment coexists with weaker interactive execution, with failures arising from incomplete investigation, premature action, and multi-action workflow breakdowns.Controlled interventions also find that agents may continue acting when decisive support is withheld and respond more to a requester’s conflicting claim than to absent evidence alone.Source: §6 · pp. 9–10
  • The paper’s benchmark and evaluator connect investigation, execution, and downstream dependencies in observable trajectories.The conclusion presents this as the study’s framing of consequential tool use.Source: §6 · p. 9
  • The analysis evaluates observable evidence support, not agents’ internal reliance on that evidence.Extending interventions to vary evidence at each multi-action checkpoint is identified as a next step.Source: §6 · p. 10

ETHICS STATEMENT

The benchmark uses synthetic settings with fictional entities, records, and policies, and contains no real user data or personally identifiable information. The authors caution that its results do not certify safety in corresponding real-world domains.

ETHICS STATEMENT

  • SAFEACTBENCH uses synthetic environments with fictional entities, records, and policies.The benchmark contains no real user data or personally identifiable information.Source: ETHICS STATEMENT · p. 10
  • Human validation and trajectory audits were performed by research-team members, with no external human subjects involved.Source: ETHICS STATEMENT · p. 10
  • Results from the six controlled operational domains should not be interpreted as safety certifications for corresponding real-world domains.Source: ETHICS STATEMENT · p. 10

A COMPARISON WITH EXISTING AGENT BENCHMARKS

The comparison distinguishes explicit evaluation requirements from behaviors that may occur incidentally during task completion. SAFEACTBENCH covers evidence establishment and binding, investigated non-action, and propagation of results across dependent actions.

A COMPARISON WITH EXISTING AGENT BENCHMARKS

  • SAFEACTBENCH explicitly scores whether evidence is established and bound to the relevant entity and state, supports execution or investigated non-action, and is propagated across dependent actions.The requirements are checked from observable trajectories with deterministic evaluators.Source: A COMPARISON WITH EXISTING AGENT… · p. 13, Table 5 · p. 13

B BENCHMARK DETAILS AND EXPERIMENTAL SETUP

B.1 BENCHMARK COMPOSITION

  • SAFEACTBENCH contains 656 cases across five protocols and six domains; V2 uses linear dependencies, while V3 has non-linear workflows with forks and joins.All six domains contribute cases to every protocol, and V0–V3 comprise 570 evidence-grounded tasks.Source: §B.1 · p. 13, Table 6 · p. 14

B.2 INVESTIGATION AND WORKFLOW COMPLEXITY

  • Correct trajectories require about 5.6–7.1 specified reads on average, with some cases requiring 17; V3 includes 501 action nodes and 439 dependency edges.V3 has 103 cases with forks and 96 with joins; the table distinguishes specified reads from additional discovery queries.Source: §B.2 · p. 14, Table 7 · p. 14

B.3 BENCHMARK VALIDATION AND EVIDENCE AUDIT

The benchmark audit checked evaluator consistency against human judgments and alternative evidence paths, then documented the agent setup, execution scoring, and diagnostic measures. These procedures define what counts as success and how failures are recorded across the protocols.

B.3 BENCHMARK VALIDATION AND EVIDENCE AUDIT

  • Cases were independently reviewed for omitted prerequisites, unnecessary requirements, and overly restrictive source requirements.Authors drew requirements from task instructions, policy clauses, and tool semantics; disagreements were discussed or adjudicated.Source: §B.3 · p. 14
  • 95.83% action-support agreement (κ=0.87), 93.36% requirement-necessity agreement (κ=0.81), and 86.72% requirement-establishment agreement (κ=0.74) preceded adjudication of all disagreements.Two annotators reviewed 120 stratified cases, and a third adjudicated every disagreement.Source: §B.3 · p. 15
  • The trajectory audit found 2.0% false acceptance and 1.3% false rejection among 300 sampled action-support judgments.Reviewers assessed pre-action observations; the sample was stratified into 150 evaluator-positive and 150 evaluator-negative checkpoints.Source: §B.3 · p. 15

B.4 AGENT AND RUNTIME DETAILS

  • Five model families were each evaluated with a family-associated harness and a shared Inspect ReAct harness, enabling within-model harness comparisons.Both harnesses used the same public task and tool interface under matched per-model settings.Source: §B.4 · p. 15
  • Agents received no private evaluation specifications, reference trajectory, preferred tool sequence, or action order; each case had a 600-second budget and three main-evaluation rollouts.Inference settings were held fixed across each model’s two harnesses.Source: §B.4 · pp. 15–16

B.5 EXECUTION AND FAILURE HANDLING

  • The evaluator checked recorded calls, observations, action receipts, terminal outputs, and state transitions against private case specifications, while workflow dependencies remained hidden.Harness planning, tool selection, and execution behavior were preserved.Source: §B.5 · p. 16
  • Agent-attributable errors remained scored as unsuccessful, while unscored infrastructure failures were excluded from scored-only analyses; the main evaluation had 100% coverage.Across additional intervention and probe episodes, 2,062 of 2,073 planned episodes were scored (99.47%).Source: §B.5 · p. 16

B.6 MEASURES AND FAILURE DIAGNOSTICS

  • Exact case success requires every protocol-specific condition; diagnostics count episodes, and stopping, premature action, conditional success, missing support, and partial workflow execution have distinct definitions.Rates are averaged across three repetitions, while ECS remains binary and diagnostics do not form an exhaustive failure partition.Source: §B.6 · p. 17

C.1 DETAILED MAIN-RESULT ANALYSIS

Results vary substantially across configurations, and strong fixed-action assessment does not ensure successful interactive execution. Matched comparisons show a static advantage, while a balanced control finds a smaller, non-universal advantage with important design constraints.

C.1 DETAILED MAIN-RESULT ANALYSIS

  • Overall ECS ranges from 37.7% to 67.2%; GLM–ZCode scores 97.7% on Legacy but 32.1%, 12.1%, and 34.1% on V1–V3.DeepSeek–DSH scores 96.5% on Legacy and 61.1%, 65.2%, and 59.1% on V1–V3; fixed-action assessment and evidence-grounded interactive completion are not interchangeable settings.Source: §C.1 · p. 18
  • Harness effects vary by model: DeepSeek ECS rises from 57.5% with Inspect to 63.0% with DSH, while Inspect slightly improves GLM overall ECS.Claude and GPT perform better with family-associated harnesses, and Qwen performs better with Qwen Code than Inspect.Source: §C.1 · p. 18

C.2.1 BALANCED STATIC–INTERACTIVE CONTROL

  • The balanced control uses 99 scenarios from 33 V1 tasks, pairing ALLOW, BLOCK, and DEFER variants across static and interactive conditions.Both conditions receive the same candidate, but only static assessment receives reference-selected observations.Source: §C.2.1 · pp. 18–19

C.2 STATIC JUDGMENT AND INTERACTIVE EXECUTION ON MATCHED CASES

  • On the same 100 V1 cases, static accuracy is 95.0%–99.0% and interactive ECS is 28.0%–52.0%, with paired gaps of 43.0, 69.0, and 66.0 points.All reported confidence intervals exclude zero; because evidence availability and responsibility for constructing and executing actions both change, the comparison does not isolate either component.Source: §C.2 · p. 18, Table 9 · p. 19

C.2.1 BALANCED STATIC–INTERACTIVE CONTROL

  • Static accuracy exceeds interactive accuracy in five of six fully covered configurations, with reported gaps of 5.1 points for Claude–Claude Code and GPT–Codex, 6.1 for GPT–Inspect, and 13.1 for GLM–ZCode.GLM–Inspect reverses the direction, while DeepSeek’s available paired subsets show positive gaps of 8.0 points under DSH and 11.1 under Inspect.Source: §C.2.1 · p. 19
  • The balanced comparison includes output-contract reliability and does not isolate evidence reasoning; supplied candidates and reference-selected static evidence also limit its measure of autonomous action construction.After excluding one ambiguous task triple, gap directions remain unchanged for all eight non-Qwen configurations, though the DeepSeek–DSH interval touches zero.Source: §C.2.1 · p. 19

C.3 FAILURE-MODE DECOMPOSITION ACROSS HARNESSES

Failure decompositions show that similar aggregate scores can conceal different behavior, while evidence withholding and historical trajectories expose additional patterns in investigation, action, and workflow readiness. The latter comparisons are descriptive and do not establish effects of imposed query budgets.

C.3 FAILURE-MODE DECOMPOSITION ACROSS HARNESSES

  • V0 failures are mostly incomplete investigation, which fails exact case success despite preserving action restraint and is not an unsafe-action rate.V0 requires both completed investigation and non-action.Source: §C.3 · p. 20
  • In V1, GLM–ZCode has a 0.66 early-action rate, while GLM–Inspect lowers it to 0.29 but raises no-action to 0.37, leaving overall failure rates nearly unchanged.Similar aggregate scores can therefore mask different failure distributions; Figure 5 defines the categories used in the decomposition.Source: §C.3 · p. 20, Figure 5 · p. 21

C.4 BEHAVIOR UNDER EVIDENCE WITHHOLDING

  • Under withholding, 19/42 DeepSeek, 17/43 GLM, and 19/43 Qwen matched pairs shift from action to stopping, but reduced action submission need not mean correct refusal.The intervention uses a frozen cohort of 43 V1 cases per configuration, with one decisive record withheld.Source: §C.4 · pp. 20–22
  • Affected-tool queries rise from 1.12 to 2.57 for DeepSeek, 1.14 to 1.74 for GLM, and 1.30 to 2.26 for Qwen between matched Original and Withheld cases.Among Withheld episodes with an action, 65/66 queried the affected tool, but this does not establish that the missing record was queried or its absence recognized.Source: §C.4 · p. 22

C.5 INVESTIGATION EFFORT AND DECISION READINESS

  • Across the same 570 cases, Qwen has the highest displayed complete-investigation plateau in every protocol, while GLM shows more incomplete investigation and support, especially in V2 and V3.These are historical trajectories, separate from the main-results evaluation.Source: §C.5 · p. 23, Figure 7 · p. 23
  • V3 has a longer tail of normalized information-call counts, but more calls alone do not establish that requirements at recorded decisions were satisfied.The curves summarize completed trajectories, not the effect of imposing different query budgets.Source: §C.5 · pp. 22–23

C.6 ACTION TIMING UNDER ADDITIONAL INTERVENTIONS

Across the additional probes, requester claims lower final action rates than evidence packages, while urgency and warnings show no consistent reduction. Distractors increase retrieval effort under withheld evidence without a demonstrated change in action probability; cross-harness and rollout comparisons retain important uncertainty.

C.6 ACTION TIMING UNDER ADDITIONAL INTERVENTIONS

  • 14.0%, 23.3%, and 14.0% are the requester-claim action rates for DeepSeek, GLM, and Qwen, below evidence-package rates in all three configurations.The cumulative curves reflect both action timing and eventual incidence, so lower curves do not necessarily mean delayed action alone; conditional timing among cases that act requires separate analysis.Source: §C.6 · pp. 23–24, Figure 8 · p. 24

C.7 DISTRACTOR EFFECTS ON RETRIEVAL EFFORT

  • 1.02, 1.19, and 1.02 additional information calls under Withheld evidence are estimated for DeepSeek, GLM, and Qwen, while action-probability changes of +4.7, +4.8, and +9.3 points have intervals including zero.Under Original evidence, mean calls rise by 1.65 for DeepSeek and 0.67 for GLM, with intervals excluding zero, and by 0.33 for Qwen, with an interval including zero; action-probability changes are −2.3, −2.4, and 0.0 points. Total information-call counts may include calls after the first action.Source: §C.7 · pp. 24–25, Figure 9 · p. 25

C.8.1 ADDITIONAL PROMPT-LEVEL PROBES

  • All six paired confidence intervals for urgency and explicit warnings include or reach zero, and the comparisons do not establish a consistent reduction in action submission.Continued action remains common under both prompts; equal aggregate counts also do not establish identical case-level decisions.Source: §C.8.1 · pp. 25–26

C.8.2 DOES EVIDENCE SENSITIVITY DEPEND ON THE HARNESS?

  • Requester claims yield lower observed action rates than evidence packages in all four common-case configurations, though GLM’s paired ordering intervals include zero.DeepSeek’s rates are 16.7% versus 38.9% under DSH and 5.6% versus 22.2% under Inspect; its 36-case subset differs from the earlier 43-case analysis.Source: §C.8.2 · p. 26

C.9 VARIABILITY ACROSS THREE ROLLOUTS

  • Across three repetitions, ECS standard deviations range from 0.2 to 0.7 percentage points, and the direction of the harness difference is consistent by model.The family-associated harness has higher ECS for Claude, GPT, DeepSeek, and Qwen, while Inspect has higher ECS for GLM; this does not establish case-level agreement or significance for every model.Source: §C.9 · pp. 26–27

C.8.2 DOES EVIDENCE SENSITIVITY DEPEND ON THE HARNESS?

  • 45.2 and 70.0 percentage-point action reductions under both DSH and Inspect show that withholding and contradiction reduce observed submission for DeepSeek across harnesses.For GLM, reductions are 37.2 versus 30.2 points under withholding and 54.5 versus 45.5 points under contradiction; the intervals do not establish harness equivalence.Source: §C.8.2 · p. 26, Figure 10 · p. 27

D ILLUSTRATIVE AGENT TRAJECTORIES

This section presents six selected trajectories across six domains and V0–V3 to illustrate traceable failure points—not their frequency or cognitive causes—including uncredited required evidence and an action that was not executed. It also describes the supplementary SCGR study, which structures evidence gathering, reuse, and public checks before submission while leaving substantive judgment and planning to the model.

D ILLUSTRATIVE AGENT TRAJECTORIES

  • The selected examples span six domains and four protocols, with five failures and one success across GPT, Claude, DeepSeek, and GLM.Cases were chosen for source traceability and clear contrasts, not to estimate failure frequency.Source: D ILLUSTRATIVE AGENT TRAJECTORIES · p. 27

D.1 UNAVAILABLE EVIDENCE AND THE WRONG STOP LABEL

  • Claude completed the investigation but submitted BLOCK incorrectly; the evaluator scores terminal labels alongside investigation completeness and non-action, and no refund was attempted.Source: EVALUATOR FINDINGS · p. 29

D.2 AN INCORRECT IDENTIFIER DESPITE COMPLETE EVIDENCE

  • GPT had covered all required evidence but submitted cset_filtered_61 instead of the supplied cites_filtered_v3, so the mismatched action argument was identified and the action was not executed.Source: EVALUATOR FINDINGS · p. 29

D.3 SUCCESSFUL PROPAGATION THROUGH A FIVE-ACTION WORKFLOW

  • In the DeepSeek healthcare V2 run, all five actions completed, and each downstream action used the actual result returned to an earlier model response.This was an observed model run, not a reference solution.Source: EVALUATOR FINDINGS · p. 31

D.4 CORRECT RECEIPT, INCORRECT BUSINESS TARGET

  • GPT correctly propagated the configuration revision but used change ID CHG-DIAG-013 instead of admin-api as the reference ID, causing the second action to be rejected.Source: EVALUATOR FINDINGS · p. 31

EVALUATOR FINDINGS

  • Evaluator findings distinguish assessed outcomes from tool responses; for example, the missing legal-rule evidence was an evaluator finding, not another tool observation.Source: EVALUATOR FINDINGS · p. 32

D.5 A FINAL JOIN BLOCKED BY MISSING RULE EVIDENCE

  • In the legal-financial V3 run, both branch results reached the final call, but closure was deferred because the exact required legal-rule source was not credited.Source: EVALUATOR FINDINGS · p. 32, §D.5 · p. 32

D.6 MAINTENANCE CLOSURE WITHOUT THE REQUIRED INTERLOCK-RULE SOURCE

  • The smart-home V3 example is framed as closure without the required interlock-rule source; visible results show sensor disable, maintenance-record update, and health-watch actions applied before the closing call.Source: §D.6 · pp. 32–33
  • The maintenance-closure submission was rejected because the action was not executed.In a separate evaluator observation, GLM supplied both follow-up receipts after the simulator applied the disable and follow-up actions, but the required interlock-rule source was uncredited; this did not establish wrong receipt binding or an actual physical device change.Source: §D.6 · p. 33, EVALUATOR FINDINGS · p. 33

E SUPPLEMENTARY STUDY: STRUCTURING EVIDENCE BEFORE ACTION

  • SCGR-Select and SCGR-EG differ in evidence acquisition, and SCGR-EG episodes are excluded from SCGR-Select comparisons.Both use public task information, tool specifications, and observed responses, without gold evidence requirements, reference actions, or the evaluator dependency graph; Table 14 reports retained case counts explicitly.Source: E SUPPLEMENTARY STUDY: STRUCTURING… · p. 34, Table 14 · p. 36

E.1 QUESTION-GUIDED EVIDENCE ACQUISITION

  • SCGR-Select validates ordered query batches, executes submitted queries sequentially with fixed arguments, and charges every attempt against the information-call budget.Dependent queries must wait for a later batch; SCGR-Select performs no automatic sweep, candidate expansion, or retry, and permits proceeding when existing public information suffices.Source: §E.1 · p. 34

E.2 EVIDENCE REUSE AND DECISION-MAKING

  • The evidence ledger supports local retrieval and model-assessed evidence bases, while the runtime checks structure and references rather than substantive support.The model identifies evidence as supported, missing, or violated, cites information-call identifiers, and explains relevance; an empty reference list is allowed with an explicit reason.Source: §E.2 · pp. 34–35

E.3 PUBLIC CHECKS AND PROTOCOL SUBMISSION

  • Before submission, deterministic checks cover output structure, public schemas and constraints, parameter bindings, and evidence or planned-result references, but do not establish evidence completeness or repair actions.The agent may revise an invalid candidate once; the adapter independently audits the final candidate without changing benchmark scoring.Source: §E.3 · p. 35
  • SCGR preserves deferred submissions: V2/V3 workflows can symbolically reference earlier planned results, which the existing execution layer resolves after submission.The query phase exposes no action receipts and does not interleave investigation with actions already executed.Source: §E.3 · p. 35

E.4 RESULTS WITH SCGR-EG

SCGR-EG raises overall ECS in all ten configurations, while SCGR-Select produces mixed task-success changes across configurations and protocols.

E.4 RESULTS WITH SCGR-EG

  • 0.3–16.2 percentage-point gains in overall ECS: SCGR-EG improves results in all ten configurations.Benefits remain model-, harness-, and protocol-dependent.Source: §E.4 · p. 36
  • 37.7% to 53.9% ECS for GLM–ZCode was the largest gain, primarily through V2 rising from 12.1% to 58.0% and V3 from 34.1% to 68.0%.Source: §E.4 · p. 36
  • DeepSeek–DSH gained 4.9 points on V3 but lost 1.3 points on V1 and 1.4 points on V2, illustrating protocol-level variation.Source: §E.4 · p. 36

E.5 DOES STRUCTURED EVIDENCE SELECTION IMPROVE TASK SUCCESS?

  • 11.6-point ECS gain for Qwen–Qwen Code contrasted with declines of 9.1 points for Claude–Claude Code, 2.8 for GLM–ZCode, and 2.6 for GPT–Codex.These four configurations cover all 570 V0–V3 cases.Source: §E.5 · p. 36
  • 35.3% ECS for DeepSeek–DSH on 434 retained cases was below the 57.9% baseline on all 570, but differing coverage limits direct comparison.The results indicate structured evidence selection alone does not ensure higher task success.Source: §E.5 · p. 36
Loading 2610.07753v1…