PaperlayerView exact revision on arXiv

Source-linked AI summary

How Strongly Should Task State Influence an LLM Agent?

Chenyu Zhang, Wonbin Kweon, Jiawei Han

arXiv:2609.25686v1cs.AIcs.CLcs.LG

TL;DR

Assigned-work agents must track changing task obligations, but existing systems are evaluated as wholes, leaving the effect of showing, directing, or enforcing state unmeasured. This paper varies state coupling under fixed task rules and paired episodes, and finds that enforcement pays when failures are state-decidable and frequent but can hurt when its judgement is wrong.

  • Problem

    Existing agent systems are evaluated as wholes, so the reliability contribution of displaying, directing, or enforcing task state is not measured.

    Source: §1 · p. 2

  • Method

    The study holds task rules, model, and paired episodes fixed while comparing raw transcript, exact checklist, per-turn directives, and receipt-advanced enforcement against exact-match ground truth.

    Source: §1 · p. 2

  • Results

    Enforcement pays when failures are state-decidable and frequent, while displaying the record matches or beats the gates when due-ness depends on free-text judgement.

    Source: §1 · p. 2, §5.5 · p. 9

  • Takeaways & Limitations

    Enforcing owned state pays when failures are state-decidable and frequent, changes nothing when they are rare, and can hurt when the action turns on a judgement the state cannot make.

    Source: §7 · p. 9

    The study covers one generator in two domains, while budgets, expiry, conflicting obligations, and multi-agent or multi-session state are out of scope.

    Source: §6 · p. 9

Abstract

from arXiv · show

Long-horizon assigned work requires an LLM agent to track the state of a task: which steps are done, blocked, cancelled, or open to repetition. Agent systems either keep this state as text in the prompt and rely on the model to read that text, or move the state into a module that enforces it, and each system is evaluated as a whole, so no one knows how much reliability comes from the state being shown, told, or enforced. We fix the task rules, the model, and paired episodes and vary how strongly task state reaches the agent: a raw transcript, an exact checklist, per-turn directives from a state machine compiled from the brief and advanced only by execution receipts, or an enforcement gate on that machine that refuses state-violating actions; every episode is scored by exact payload matching against dynamic ground truth. Across three models, two reasoning regimes, and two domains, four findings hold without per-turn reasoning: displaying accurate state is unreliable, an unverified ledger the agent writes itself beats an accurate checklist it is shown, directives help in proportion to the model's obedience, and enforcement needs no obedience but is bounded by the correctness of its state and by the matcher that maps requests to steps; per-turn reasoning at a 235B agent compresses these separations without repairing the text rungs. The same gate, compiled from $τ^2$-bench's airline policy, raises a 235B agent's pass$^1$ from 0.39 to 0.54 and changes nothing for a 35B agent that rarely violates the policy; on PM-Bench, where acting turns on recognizing a cue rather than on state, showing the record is the best rung--matching or beating both gates and reversing the ledger-over-checklist finding--and enforcing the matcher's judgement drops a 35B agent below its raw transcript. Enforcement pays when failures are state-decidable and frequent, and hurts when the gate's judgement is wrong.

1 INTRODUCTION

Long-horizon assigned work requires agents to track obligations that change through conversation, including dependencies, cancellation, and whether completed steps may be repeated. The paper measures how reliability changes as task state is displayed, directed, or enforced, finding that stronger coupling generally helps while enforcement is bounded by state correctness.

1 INTRODUCTION

  • Assigned work is a changing graph of obligations: agents must execute eligible steps once, refuse blocked or cancelled work, and distinguish re-mentions from explicit redo orders.The agent must do this from a record it writes itself.Source: §1 · p. 1
  • The study holds task rules, model, and paired episodes fixed while varying state coupling, scoring episodes by exact payload match against executable ground truth.The compared arms are raw transcript, exact checklist, per-turn directives, and an enforcement gate.Source: §1 · p. 2
  • 0.38 → 0.55 → 0.84 → 0.98: on Qwen3-235B without thinking at S=10, strict episode success rises from raw transcript through checklist and directives to enforcement.The introduction also reports that with per-turn thinking the two gates no longer differ significantly, and that external benchmarks distinguish when enforcement pays or hurts.Source: §1 · p. 2
  • The contribution is a measurement ladder that locates each rung’s bound in model use of the record, directive obedience, or state and matcher correctness.The paper also reports a real-tool harness and two external benchmarks, with a gain on τ 2-bench and a loss on PM-Bench.Source: §1 · p. 2

2 RELATED WORK

Prior work treats agent memory largely as retrieval of past information, while assigned work requires prospective action on later requests. Existing systems occupy individual points on the state-coupling ladder, but do not jointly compile state from conversation, advance it on execution evidence, and vary its coupling to behavior.

Agent memory

  • Agent memory retrieves the past, whereas assigned-work obligations are prospective and must guide action on later requests that may not ask about them.The paper frames this distinction using prospective memory.Source: §2 · p. 2

Plans, ledgers, and workflow engines

  • Prior systems differ in whether they own state, verify receipts, direct behavior, enforce rules, or compile state from a natural-language brief.Table 1 places each system on its closest ladder arm, using the strongest arm when a system has several.Source: Table 1 · p. 3
  • Workflow and agent systems typically use developer-authored or pre-authored graphs, while this study compiles state from the user’s brief and advances it only on receipts.Examples include plan-and-execute agents, task ledgers, checkpointed state, and workflow engines.Source: §2 · p. 3

Runtime enforcement and lifecycle benchmarks

  • Runtime enforcers and lifecycle benchmarks do not vary how strongly state acts on the agent, leaving the point where instruction-compliance decay stops unmeasured.Prior systems either own task state without compiling it from conversation or enforce hand-written rules without owning task state.Source: §2 · p. 3

3 TASK STATE AND THE STATE-COUPLING LADDER

The paper represents assigned work as a task-state machine whose status governs whether a requested step may be executed. It compares a text-only record with increasingly strong couplings of shared state to agent behavior, from display to direction to enforcement.

3.1 THE TASK-STATE MACHINE

  • A request resolving to step v is correct only if the agent executes it once when TODO or explicitly authorized for redo, and otherwise takes the prescribed non-execution response.The state tracks steps, prerequisite edges, status, and authorized re-execution; a turn resolving to no step executes nothing.Source: §3.1 · p. 4
  • Revisions update the graph in place, and violations are divided into six channels: re-execution, superseded, premature, omission, unrequested, and refused redo.The measured lifecycle is compiled from conversation and revised from a message.Source: §3.1 · p. 4

3.2 THE COMPILED STATE MACHINE

  • The directive and enforcement arms share an externally owned compiled state machine, while the checklist displays exact reference state and the gates use compiled state.The state machine is compiled from the brief and revision message; compile errors pass into its state uncorrected.Source: §3.2 · p. 4
  • A step becomes DONE only after a valid booking yields a receipt; prose claims of completion and repeated receipts do not advance state.In the testbed, a booking is a reply line with the completion code and current work-order number; the matcher also resolves requests and redo intent.Source: §3.2 · pp. 4–5

3.3 THE STATE-COUPLING LADDER

  • The ladder holds the model, brief, and paired episodes fixed while varying state coupling: raw text, an exact checklist, directives, and enforcement.The checklist is accurate but leaves action unconstrained; directives tell the agent what state implies, while enforcement refuses violating bookings.Source: §3.3 · p. 5
  • Enforcement closes re-execution and cancelled-step violations outright and premature executions up to compile accuracy, but cannot force omissions or authorized redos.State-bound enforcement admits some unrequested eligible-step bookings; the paper also includes memory-retrieval baselines and an unverified self-ledger arm.Source: §3.3 · p. 5

4 TESTBED, EVALUATION FRAMEWORK, AND SETUP

The testbed uses generated assigned-project episodes with changing prerequisites and trap turns, then scores execution against dynamic ground truth by exact receipt matching. The setup varies model, graph size, density, and reasoning regime while keeping requests paired across ladder arms.

4.1 ASSIGNED-PROJECT EPISODES

  • Episodes vary project size and prerequisite density, include out-of-order asks and a mid-project revision, and contain both unauthorized redo probes and a legitimate redo.The worked trap turn re-mentions a completed step with a fresh work order but no redo order; raw transcript and checklist book it again, while directive and enforcement gates do not.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 EVALUATION FRAMEWORK

  • Strict episode success requires every eligible requested step to be executed exactly once, with no blocked, cancelled, or unrequested execution and every authorized redo honored.Execution is scored by exact matching of receipt lines against ground truth, without an LLM judge.Source: §4.2 · p. 6

4.3 COMMON SETUP

  • The main grid uses Qwen3-235B and Qwen3.6-35B across graph sizes and densities, with 128 episodes per cell unless noted.Qwen3-235B runs with and without per-turn thinking; the amended brief states the one-shot rule, unlike the original-brief control.Source: §4.3 · p. 7

5 RESULTS

The results show that text alone does not reliably maintain task lifecycle state, while stronger coupling helps subject to limits in obedience, state accuracy, and request matching. The benefit of enforcement depends on whether failures are state-decidable and how often they occur.

5.1 RQ1: DOES TEXT-BORNE STATE IMPLEMENT THE LIFECYCLE?

  • Text-borne state fails to implement the lifecycle for both open agents, and fresh requests for completed steps can trigger re-execution despite displayed state.Stating the one-shot rule reduces this hazard only for a sufficiently capable reasoning agent; the three stateful rungs change by at most 0.06 between briefs.Source: §5.1 · p. 7

5.2 RQ2: HOW STRONGLY MUST OWNED STATE ACT ON BEHAVIOR?

  • Without thinking, each ladder rung improves success across both open models, while reasoning compresses the gaps and the agent-written ledger beats the accurate checklist.Rendering matters: the directive gain comes from the instruction in the user role; the self-ledger still leaks re-executions and remains below the directive gate.Source: §5.2 · pp. 7–8

5.3 RQ3: WHAT BOUNDS EACH RUNG?

  • Matcher errors sharply bound both gates (directive 0.92→0.58; enforcement 1.00→0.78), while enforcement inherits state errors and directives depend on obedience.Of 68 omissions across the Qwen3-235B enforcement cells, 35 arose from over-blocking eligible steps; directive compliance rises with model capability and reasoning.Source: §5.3 · p. 8

5.4 RQ4: DOES THE LADDER TRANSFER TO REAL TOOLS AND EXTERNAL BENCHMARKS?

  • 0.39→0.54 pass1 for Qwen3-235B on τ^2-bench airline, but the gate changes nothing for Qwen3.6-35B; on PM-Bench, showing records wins.The tool harness reproduces the ladder ordering; the 35B agent’s judgement-bound enforcement falls below its raw transcript on PM-Bench.Source: §5.4 · pp. 8–9, §5.5 · p. 9

5.5 DISCUSSION: WHEN DOES ENFORCING PAY, AND WHEN DOES IT HURT?

  • Enforcement benefits depend on state-decidability and gate accuracy: it helps when failures are frequent and state-decidable, but can hurt when free-text judgement is wrong.On PM-Bench, showing the record beats the gates for Qwen3-235B; for Qwen3.6-35B, matcher-bound enforcement scores 0.55 versus 0.63 for the raw transcript.Source: §5.5 · p. 9

6 LIMITATIONS

The study’s scope is limited to a specific task-state model, generated episodes, and a small set of evaluation settings. Several deployment conditions and broader task types remain untested.

Scope

  • The task-state model covers one-shot or explicitly repeated steps with dependencies and cancellation; budgets, expiry, conflicting obligations, and multi-agent or multi-session state are out of scope.Each rung is one implementation of that coupling.Source: §6 · p. 9

Evaluation setting

  • Episodes come from one generator in two domains, with one held-out rewriter, one ambiguity pattern, and scripted single-user schedules.The original and amended briefs differ in probe schedule as well as wording, except in the Qwen3.6-35B comparison.Source: §6 · p. 9

Evaluation boundaries

  • The real-tool harness uses tools that never fail, while τ^2-bench covers airline only and PM-Bench covers one released week.The Qwen3.6-35B thinking row is partial, closed-model cells contain 32 episodes, and the tool protocol does not show a re-execution mode seen in text tests.Source: §6 · p. 9

7 CONCLUSION

Task-state coupling improves reliability only within the limits of what the model follows and what the state machinery judges correctly.

7 CONCLUSION

  • Displayed task state helps only as far as the model heeds it.Source: §7 · p. 9
  • Directives help only as far as the model obeys, while enforcement depends on correct state and matching.Source: §7 · p. 9
  • Enforcement pays when failures are state-decidable and frequent, changes little when they are rare, and can hurt when judgement is wrong.Source: §7 · p. 9

ETHICS STATEMENT

The evaluation uses synthetic assigned-project episodes and a temporary-workspace tool harness, with no human subjects, personal data, or real transactions.

ETHICS STATEMENT

  • The study uses program-generated episodes and a temporary workspace rather than human subjects, personal data, or real transactions.Source: ETHICS STATEMENT · p. 10
  • The enforcement gate refuses actions that violate owned task state, but also refuses correct actions when its state is wrong.The authors say deployment must monitor this failure mode rather than assume it away.Source: ETHICS STATEMENT · p. 10
  • The authors identify no distinct misuse path beyond those already present for LLM agents generally.Open models were served locally and the closed model accessed through its vendor API under applicable terms.Source: ETHICS STATEMENT · p. 10

LLM USAGE STATEMENT

LLM assistants supported testbed implementation, experiment operations, and drafting, but primary outcomes use exact payload matching rather than an LLM judge.

A.1 SCORER AND ORACLE VALIDATION

  • Primary scoring matches verifiable payload strings exactly against generator-computed ground truth; no LLM judge contributes to the metric.Scripted-agent checks also test that the scorer responds to corrupted module state.Source: LLM USAGE STATEMENT · p. 10, §A.1 · p. 16

LLM USAGE STATEMENT

  • LLM assistants helped build the testbed, scorer, arm implementations, real-tool harness, and analysis scripts, and also supported experiment runs and text editing.The authors report verifying primary result numbers against released logs.Source: LLM USAGE STATEMENT · p. 10

A.2 DECLINE-AWARE SCORING

  • Decline-aware scoring treats specified refusal annotations around a current-nonce reference as citations rather than executions, and both scoring rules are reported.The primary scoring counts such a reference as a booking.Source: §A.2 · p. 16

A.3 REDO-PROBE WORDING STUDY

  • Three independent Claude Fable 5 sessions judged 90/94 probe cases as re-mentions by majority and 88/94 unanimously, with no fresh-order majorities.The raters were LLMs, not human annotators.Source: §A.3 · p. 16

A.4 WHY THE TEXT RUNGS FAIL: A TRAP-TURN PROBE

Trap-turn probes show that agents can repeat completed work despite seeing state, with distinct failure patterns across the two tested Qwen models. Held-out phrasing also exposes a matcher vulnerability in the gate arms.

A.4 WHY THE TEXT RUNGS FAIL: A TRAP-TURN PROBE

  • 0.21 of Qwen3-235B text-protocol trap turns re-executed in the raw arm, versus 0.09 with tools; changing the earlier execution record raised text re-execution to 0.79.The checklist reduced the text rate to 0.10, while a step-naming directive reduced it to 0.01; for Qwen3.6-35B, 0.47 re-executed in the text protocol, often after saying the step was done, but tool-harness re-execution was 0.02 across variants.Source: §A.4 · pp. 16–17, Table 3 · p. 17

A.5 WORDING CONTROL AND MATCHER STRESS

  • Held-out paraphrases left the raw transcript and checklist unchanged but lowered both gates; matcher agreement fell from 1.000 to 0.98 as rewrites were routed to neighbouring steps.The gate could then issue an incorrect ALREADY DONE or BLOCKED directive, causing an owed step to be omitted.Source: §A.5 · p. 18, Table 4 · p. 18

B.1 PROMPTS AND PROTOCOL STRINGS

  • The checklist is appended to user messages and updated from booked work orders, while a gate appends a matched-step directive and enforcement can prepend a rejection notice.The amended brief adds a one-shot rule: confirm a finished step without booking a new work order unless a redo is explicitly ordered.Source: §B.1 · p. 19

B.2 MATCHER, MESSAGE ROLES, AND PROMPT LENGTH

  • The matcher resolves a step each turn, with reported agreement of 0.997–1.000, but filler and revision turns bypass it and small talk remains untested.The two gate arms use shorter prompts than the checklist arm, which appends the rendered state each turn.Source: §B.2 · p. 20

B.3 COMPILE ACCURACY

  • The task graph and revision are compiled from the brief and revision message rather than read from generator ground truth, and compile errors flow into gate state uncorrected.Compiled graphs are cached and shared by the directive and enforcement arms within a cell.Source: §B.3 · p. 20, Table 5 · p. 21

B.4 MODELS AND SERVING

  • The study uses Qwen3-235B, Qwen3.6-35B, and DeepSeek V4 Pro, with 44 turns per episode and model-specific thinking regimes and episode counts.The original Qwen grids and Qwen3.6-35B dense cells use 128 episodes; DeepSeek V4 Pro cells use 32.Source: §B.4 · p. 21, Table 6 · p. 21

B.5 OVERHEAD ACCOUNTING

The appendices account for gate overhead, situate task state relative to memory and workflow systems, and report ladder patterns across graph size, models, and error channels.

B.5 OVERHEAD ACCOUNTING

  • At S=10 and S=15 dense, the gates cost at most 1.00× and 0.88× the raw transcript’s total tokens, while the checklist costs 1.34× and 1.58×.Both gate arms were faster per episode than both text arms in every no-think cell of the main and amended grids.Source: §B.5 · pp. 21–22, Table 7 · p. 22

C.1 AGENT MEMORY IS RETROSPECTIVE

  • Retrieval-based memory is retrospective, whereas task state tracks prospective obligations; Mem0 and top-k exchange memory complete only 0–2 of 128 episodes per cell.The passage identifies omission as their heaviest overall error channel.Source: §C.1 · p. 22

C.2 PLANS AS TEXT, ENGINES AS STATE

  • Plan-and-execute transcripts and prose ledgers keep progress as text, while workflow engines enforce owned state; this study compiles its state from the user’s brief and revision.The related-work comparison notes that those engines do not measure what is lost when the same state is carried only as text.Source: §C.2 · p. 23

C.3 RUNTIME ENFORCEMENT AND INSTRUCTION PERSISTENCE

  • The enforcement gate can refuse a wrong booking but cannot force a correct one, so omissions remain a failure mode.The passage characterizes it as a suppression automaton over task state.Source: §C.3 · p. 24

D.1 LADDER INVENTORY AND STATISTICS

  • At fixed density, the raw transcript stays flat as graph size grows, the three pre-enforcement rungs rise with model capability, and enforcement is flat across models.Corrupting the gate’s state monotonically degrades its success.Source: Figure 4 · p. 25, §D.1 · p. 25

D.2 PER-CHANNEL ERROR RATES

  • For Qwen3-235B, re-execution dominates the text-rung error channels and vanishes under enforcement.The figure also plots the corrupted-state replay; the ladder comparison covers the original-brief setting and the same channel decomposition.Source: §D.2 · p. 25, Figure 6 · p. 26, Figure 5 · p. 26

D.3 REMAINING ENFORCEMENT ERRORS AND DIRECTIVE OBEDIENCE

The remaining errors expose limits on both the agent and the state module: agents can override directives, while incorrect compiled state can misdirect enforcement. Corruption replay shows that obeying an erroneous directive is a major failure path.

D.3 REMAINING ENFORCEMENT ERRORS AND DIRECTIVE OBEDIENCE

  • 33 of the directive gate’s 36 Qwen3-235B errors occurred when the agent executed despite a CANCELLED, BLOCKED, or ALREADY DONE directive.Under decline-aware scoring, strict success rises to 0.92 at S=10 and 0.90 at S=15 dense; same-turn rejection notices eliminate visible replies that look like completed work without changing strict success.Source: §D.3 · p. 26, Table 9 · p. 27, Table 11 · p. 29
  • 855→858 of 896 episodes succeed in the validator-gated hybrid, while omissions fall from 39 to 31 by using enforcement only in episodes its compile validator passes.The remaining omissions in flagged episodes are directive-obedience failures, not gate refusals.Source: §D.3 · p. 27, Table 10 · p. 28

D.4 CORRUPTED-STATE REPLAY

  • At ε=10−3 and 3×10−3, obeyed ALREADY DONE directives account for 11 and 30 omissions, respectively; obeyed BLOCKED directives account for 5 and 10.Across corruption-changed decisions, rebuilding completion state from the accepted-booking log restores the uncorrupted verdict at 43/43, 103/105, and 263/269 decisions for ε=10−3, 3×10−3, and 10−2.Source: §D.4 · pp. 29–30, Table 12 · p. 30

E ORIGINAL BRIEF, AMENDED BRIEF, AND THE CONTROLS BETWEEN THEM

The brief amendment and per-turn reasoning change how sharply the state-coupling ladder separates its rungs. Across graph sizes and a second domain, the text arms remain less reliable than gates without thinking, while thinking narrows several gaps.

E ORIGINAL BRIEF, AMENDED BRIEF, AND THE CONTROLS BETWEEN THEM

  • 0.27→0.38 and 0.34→0.50 are the raw-transcript success changes at Qwen3-235B without thinking after the one-shot sentence; the checklist’s re-executions also fall.The revised schedule contributes +0.03 and +0.06, while the sentence adds +0.08 and +0.10 on the same seeds; the three stateful rungs stay within 0.06 of original values.Source: E ORIGINAL BRIEF, AMENDED BRIEF, AND… · p. 30, Table 14 · p. 32
  • With thinking, the raw transcript rises from 0.29 to 0.80 at S=10 and 0.28 to 0.63 at S=15 dense, while checklist scores also rise and gate scores move little.At Qwen3.6-35B without thinking, the raw transcript rises from 0.12 to 0.32, but the checklist does not move; the two upper-rung contrasts remain significant.Source: E ORIGINAL BRIEF, AMENDED BRIEF, AND… · pp. 30–31, Table 14 · p. 32

E.1 GRAPH SIZE UNDER THE AMENDED BRIEF

  • Without thinking, Qwen3-235B’s raw transcript stays between 0.34 and 0.50 across five graph sizes, while enforcement stays between 0.91 and 0.98; thinking compresses the ladder.With thinking, the raw transcript reaches 0.63–0.82 and only three of fifteen adjacent contrasts survive Holm correction; the gates do not separate in any cell.Source: §E.1 · pp. 32–33, Table 15 · p. 33

E.2 CHECKLIST RENDERING AND DIRECTIVE PLACEMENT

  • Replace-mode checklists never help, and system-role directives hurt both gates; at Qwen3.6-35B, enforcement falls to 0.26 while refusing 70 bookings.Adding only a matched-step line is inconsistent: it lowers success at Qwen3-235B and raises it at Qwen3.6-35B.Source: §E.2 · p. 33, Table 16 · p. 33

E.3 A SECOND DOMAIN: RELEASE ENGINEERING

  • 0.48→0.44→0.78→0.93 is the release-engineering ladder at Qwen3-235B without thinking; both gate contrasts replicate, while checklist and transcript do not separate.The second domain keeps the trap budget, schedule, scoring, and dependency-graph shape distribution unchanged from procurement.Source: §E.3 · p. 34, Table 17 · p. 34

F RETRIEVAL-BASED MEMORY BASELINES AND THE SELF-LEDGER

Retrieval used as the task record largely fails to track the lifecycle, while retrieval added to the transcript can help. The agent-written ledger beats an accurate checklist, and real-tool results show that enforcement errors depend on both agent behavior and compiled state.

F RETRIEVAL-BASED MEMORY BASELINES AND THE SELF-LEDGER

  • 0/128 and 2/128 episodes succeed with top-k retrieval as the record, and 0/128 succeed in both Mem0 cells, versus 0.38 and 0.50 for the raw transcript.Retrieval-as-record failures are omission-heavy because recalled exchanges neither surface every current obligation nor reliably mark finished work.Source: F RETRIEVAL-BASED MEMORY BASELINES AND… · p. 34
  • 0.67 and 0.71 are top-k augment success rates, above the checklist’s 0.55 and 0.59 but below the directive gate’s 0.84 and 0.87.Top-k augment cuts raw-transcript re-executions from 70 to 34 and from 64 to 37; retrieved text can still be ignored.Source: F RETRIEVAL-BASED MEMORY BASELINES AND… · pp. 34–35, Table 18 · p. 36
  • 0.70 and 0.79 are self-ledger success rates, above the checklist’s 0.55 and 0.59, although the self-ledger remains below the directive gate in both cells.Re-executions persist even when the agent’s own ledger already marks the step DONE; on PM-Bench, showing the record reverses this ordering.Source: F RETRIEVAL-BASED MEMORY BASELINES AND… · p. 35, Table 18 · p. 36

G REAL-TOOL HARNESS DETAILS

  • At Qwen3.6-35B, enforcement refuses 63 tool calls but still has 15 premature and 21 unrequested errors; at Qwen3-235B, its sole failure is an omission in a miscompiled episode.Request-bound enforcement changes nothing for the obedient 235B agent, which makes no unrequested prerequisite calls.Source: Table 19 · p. 36, G REAL-TOOL HARNESS DETAILS · p. 37

H EXTERNAL-BENCHMARK CASE STUDY: τ 2-BENCH AIRLINE

In the τ^2-bench airline domain, enforcement improved strict success for Qwen3-235B but made no material difference for Qwen3.6-35B. The gate eliminated executed state-rule violations, while its benefit depended on how often agents attempted violations it could judge.

H EXTERNAL-BENCHMARK CASE STUDY: τ 2-BENCH AIRLINE

  • The gate was injected at tool dispatch as 16 compiled deny rules over six WRITE tools, with READ tools and benchmark scoring left unchanged.The paired runs used different agents and user simulators, and their scores are internally paired rather than comparable to published τ^2-bench scores.Source: H EXTERNAL-BENCHMARK CASE STUDY: τ… · pp. 37–38
  • The ledger and advisory arms reached pass1 rates of 0.52 and 0.49, respectively, while executing four and eight state-violating writes versus zero under enforcement.The arms’ pass1 intervals overlapped; advisory notices were sometimes overridden, including under simulated-user pressure.Source: H EXTERNAL-BENCHMARK CASE STUDY: τ… · p. 38
  • 0.39 to 0.54 pass1 for Qwen3-235B with the gate, while Qwen3.6-35B moved from 0.730 to 0.755 with paired intervals spanning no effect.At Qwen3-235B, wrongful-write episodes fell from 54% to 35%; the 35B multi-trial pass4 rate was 0.560 ungated and 0.520 gated.Source: H EXTERNAL-BENCHMARK CASE STUDY: τ… · pp. 38–39, Table 20 · p. 39
  • A gate veto appeared as an ordinary tool result: after a cancellation refusal, the agent explained the policy branches and passed both database and communication checks.The policy required booking within 24 hours, an airline-cancelled segment, business class, or insurance for cancellation.Source: H EXTERNAL-BENCHMARK CASE STUDY: τ… · p. 39
  • Older shipped task data exposed a reference-policy mismatch, and the authors recommend unit tests, gold replay, compile validation, or an advisory rung before deployment without gold trajectories.The older revision overblocked one of 55 gold write actions; the paired-run task set had no gold overblocks under either audited semantics.Source: H EXTERNAL-BENCHMARK CASE STUDY: τ… · pp. 39–40
  • The gate’s benefit tracked catchability: 24% of Qwen3-235B episodes but only 3.5% of Qwen3.6-35B episodes contained a rule-detectable violation.At 35B, most wrongful writes involved incorrect arguments within eligible calls, which a database-state eligibility gate cannot decide.Source: H EXTERNAL-BENCHMARK CASE STUDY: τ… · p. 40

I EXTERNAL-BENCHMARK CASE STUDY: PM-BENCH

On PM-Bench, where the agent must recognize when an intention is due, showing the intention record was the strongest rung, while matcher-based enforcement could reduce performance. A one-node precursor also reduced repeated execution of consumed one-shot commitments in its testbed.

I EXTERNAL-BENCHMARK CASE STUDY: PM-BENCH

  • The benchmark’s 80-step week scores due intentions triggered by vignette cues, hidden channels, or clock time, with misses, false alarms, and update violations as scored failures.The released single-agent runs expose no commission or dependency violations because completed tasks disappear from the menu.Source: I EXTERNAL-BENCHMARK CASE STUDY:… · p. 41
  • The intention record stores each intention’s trigger, day, and pending, done, or cancelled status; updates are compiled from text and completion follows an execution receipt.The record is compiled from the agent-visible text, and compile errors flow into it uncorrected.Source: I EXTERNAL-BENCHMARK CASE STUDY:… · p. 41
  • The checklist, directive, and enforcement rungs differ in whether they display the record, announce matched due handles, or refuse selections unsupported by the matcher’s evidence.The study repeats one released week across four temperature-0.7 runs per cell, so that spread measures sampling variance rather than variation across scenarios.Source: I EXTERNAL-BENCHMARK CASE STUDY:… · p. 41
  • 0.77 Set-F1 for Qwen3-235B with the checklist versus 0.67 raw and 0.71 with the self-written ledger; Qwen3.6-35B reached 0.72 with the checklist versus 0.63 raw.For Qwen3-235B, false alarms fell from 7.8 to 1.0 per run and update violations from 4.0 to 0.2; time- and channel-cued misses remained unchanged because those cues were hidden.Source: Table 21 · p. 42, I EXTERNAL-BENCHMARK CASE STUDY:… · p. 42
  • The directive gate cut Qwen3-235B misses to 25% but ended at 0.71 Set-F1, below the checklist, as 17.5 of 20.8 false selections per run had been announced.The matcher’s precision was 0.72 and recall was 0.89 on event cues, 0.67 on times, and 0.10 on channel cues.Source: I EXTERNAL-BENCHMARK CASE STUDY:… · p. 42
  • Matcher-bound enforcement reduced Qwen3.6-35B Set-F1 to 0.55, below raw, while the state-bound gate reached 0.66, matching the ledger.The matcher-bound gate refused 21.5 selections per run, including 16.8 due selections; the state-bound gate’s remaining due refusals came from handle-to-intention misresolution.Source: I EXTERNAL-BENCHMARK CASE STUDY:… · p. 42

J THE ONE-NODE CASE

  • 0 repeated executions under the precursor’s guarded trigger, versus 0.87 with a prompt manifest, at 0.88 trigger recall across 100 concurrent commitments.Removing transcript sanitization alone brought re-execution back to 0.64.Source: J THE ONE-NODE CASE · p. 43
Loading 2609.25686v1…