Source-linked AI summary
When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs
Jason Liu
TL;DR
Tool-using agents need evidence-grounded stopping decisions, but common terminal signals do not ensure that required claims are in scope or replayable. The paper instantiates ECT, a typed certificate verifier with deterministic closed-transform replay, and evaluates it on controlled synthetic termination faults. ECT eliminated unsafe and premature unsupported completions in the reported studies while retaining supported completion within the prespecified margin, under declared contract and adapter assumptions.
Problem
Tool-using agents must decide when to stop, while common stopping signals do not necessarily bind final claims to valid, scoped, replayable evidence.
Method
ECT authorizes COMPLETE only when a typed certificate passes task and ledger binding, receipt validation, scope checks, and deterministic closed-transform replay.
Results
ECT produced 0/288 unsafe completions versus 252/288 for the critic core and 0/66 premature unsupported terminations versus 40/66 for the faithful controller, while supported completion was noninferior.
Takeaways & Limitations
ECT makes terminal support checkable and enabled successful recovery in 18/66 eligible incomplete trajectories, with 17 subsequently completing with support.
Takeaways & Limitations
ECT’s conclusions are limited to trace support under declared assumptions: contracts can omit requirements, adapters can fabricate consistent receipts, and the synthetic setting does not establish generality, safety, or alignment.
Abstract
from arXiv · showhide
Tool-using agents must decide when to stop. Existing systems already gate terminal success, certify execution traces, or enforce runtime polici es, but do not test this particular receipt-, scope-, and closed-replay design at the COMPLETE boundary across controlled termination faults. W e instantiate and evaluate Evidence-Carrying Termination (ECT): an agent may return COMPLETE only when a typed certificate binds every required answer claim to valid, in-scope trace evidence and a deterministic replay reconstructs the claimed value. A locked static study crosses 48 ful ly synthetic tasks in six tool-use families with clean execution and eight faults. ECT produced 0/288 unsafe completions versus 252/288 for the inspected termination-critic core (difference -87.50 pp, 95% task-cluster interval [-87.50, -87.50] pp). A fresh, prespecified and frozen 576- trajectory study then compares ECT with the critic core, its faithful controller, and a full-trace LLM critic. On 22 primary held-out task clus ters, ECT produced 0/66 premature unsupported terminations versus 40/66 for the controller (difference -60.61 pp, 95% interval [-78.79, -40.91] pp), while supported completion was 97/132 versus 92/132 (difference 3.79 pp, interval [0.00, 9.09] pp), satisfying a -10-point noninferiority margin. ECT executed successful recovery in 18/66 trajectories, of which 17 subsequently completed with support; all three closed-loop gates p assed. ECT certifies support in a recorded trace under declared assumptions, not external truth, safety, or alignment.
Introduction
Tool-using agents must decide both what to do next and whether enough has been done, yet common stopping signals do not bind each final claim to scoped, replayable evidence. ECT separates certificate proposal from deterministic authorization to stop, combining typed evidence checks with closed-transform replay.
- Stopping too early can turn an unsupported intermediate state into a final answer, while continuing indefinitely wastes calls and can compound risk.
- Common DONE signals, checklists, heuristics, and plausibility judgments do not necessarily identify claim-supporting observations, requested scope, or replayable derived values.The gap matters because termination is part of the control system and terminal evidence may support later assurance arguments.
- ECT separates certificate proposal from stop authorization by checking a typed certificate against a trusted task contract and immutable evidence ledger.Task and ledger digests bind the certificate to the trace; receipt checks validate value path, status, and scope.
- ECT combines digest binding, receipt-level validation, and exact closed-transform replay, while failed checks return continue with reason codes.
- The contribution is completion-specific evaluation across a six-family, 48-task benchmark with controlled faults and frozen comparisons against critic-based baselines.
Related Work
Prior work gates completion, certifies execution, enforces runtime policies, evaluates evidence paths, or studies solvability and help seeking. ECT narrows the focus to unsupported authorization of positive COMPLETE at the termination boundary.
- Earlier systems gate DONE on executed checks, audited task state, termination verifiers, or certificate-backed coverage, so ECT does not claim the first successful-completion gate.
- Certificate-oriented systems already address evidence, hashes, scope, replay, claim decomposition, emission gates, and conformance obligations.
- ECT contributes a narrower completion-specific integration of required-slot contracts, receipt ledgers, digest-bound certificates, exact scope and cardinality checks, and closed-transform replay.
- Existing evaluation work scores final answers, provenance, final state, trajectory artifacts, solvability, missing information, or help seeking; ECT studies unsupported authorization of positive COMPLETE.
Evidence-Carrying Termination
ECT authorizes COMPLETE only when required claims are bound to valid, in-scope receipts and deterministic replay reconstructs each claimed value. Its guarantee is relative to the trusted contract and evidence adapter.
- A certificate contains task identity, descriptor and ledger digests, and claims pairing each required slot with a value, evidence IDs, and a transform.
- ECT uses a closed, non-executable transform language covering collection, aggregation, differences, ratios, extrema, sorting, set operations, top-k, abstention, and windowed differences.
- The verifier rejects missing or extra slots, duplicate or conflicting records, invalid receipts, scope mismatches, impermissible transforms, and replay discrepancies.
- Deterministic Verification: COMPLETE is returned if and only if all required claims pass every check; verifier failures return CONTINUE, while STUCK requires a separate recovery-availability rule.
- Deterministic Verification: For fixed requirements, evidence, and certificate, the verifier is deterministic, order-independent in ledger hashing, and locally reproducible with reason codes.
- Relative Soundness: If verification accepts, every required slot is present, cited receipts are valid and in scope, and replay reconstructs each certified value within tolerance.
- Trust Boundary: The guarantee assumes adequate requirements and faithful adapter records; a malicious adapter or incomplete contract can yield internally consistent but incomplete support.
- Trust Boundary: ECT certifies reconstructibility from the bound trace under declared assumptions, not external truth, authenticity, effects, safety, or alignment.
Benchmark and Protocol
The study uses synthetic six-family tasks with clean snapshots and eight controlled termination faults, then compares frozen policies under prespecified endpoints and inference. Its design isolates declared defects but limits generalization beyond them.
- Tasks, Faults, and Split: Eight generated worlds cover lookup, aggregation, top-k, temporal comparison, hierarchy join, and missing-data abstention, with 36 held-out tasks and dual independent oracles.
- Tasks, Faults, and Split: Each base task has one clean snapshot and eight controlled faults, producing 48 × 9 = 432 snapshots, including 288 held-out fault snapshots.
- Tasks, Faults, and Split: Table 1 applies the clean condition and eight faults to every base task, with Gold denoting the permitted terminal decision at each snapshot.
- Tasks, Faults, and Split: Fault construction changes certificates, receipts, or recovery state while retaining task descriptors and gold answers, isolating declared termination defects without assuming only one verifier predicate fires.
- Policies and Frozen Inputs: The comparison includes six baselines and ECT, with fixed sources, prompts, models, decoding, parsers, retries, and failure mappings before scoring.
- Endpoints and Inference: The primary endpoint is unsafe completion rate among 288 held-out fault snapshots, analyzed with paired task indicators and task-cluster bootstrap intervals.
- Endpoints and Inference: Clean false continuation is reported separately, while secondary endpoints include recovery, quality, fault breakdowns, resource use, and reason localization.
- Endpoints and Inference: Single-check ablations are descriptive, and the exact-transform-contract ablation lacks a matched frozen fault, limiting its interpretation.
Static Results and Ablations
ECT eliminated unsafe completions in the held-out static evaluation, while ablations localized matched fault-containment checks without treating residual cases as evidence that any check was unnecessary.
- Static Evaluation: 0/288 unsafe completions occurred with ECT versus 252/288 for the critic core, a difference of −87.50 pp with a 95% task-cluster interval of [−87.50, −87.50] pp.H1 passed; the six-world sensitivity interval was also [−87.50, −87.50] pp.
- Static Evaluation: 0 clean false continuations occurred for both ECT and the comparator across 36 clean snapshots.Clean false continuation was reported separately from unsafe completion.
- Ablations: Single-check ablations localized validation families containing their matched injected faults while retaining unmatched and residually caught cases.These ablations were descriptive and did not establish that any validation check was unnecessary.
Closed-Loop Confirmation
A fresh, prespecified V2 study evaluated four termination-policy pipelines across matched synthetic tasks, checkpoints, faults, and seeds; ECT reduced premature unsupported termination while retaining noninferior supported completion, at measurable control cost.
- Study Design: 576 trajectories were generated from 24 held-out tasks, two checkpoint conditions, four policy pipelines, and three seeds.The study used worlds 10–13, with twelve fault assignments and matched nominal inputs rather than same-proposal interventions.
- Study Integrity: All 1289 persisted attempt records succeeded, with no recorded parse/provider failure or transport retry.The final manifest validated all 576 cells and provider-audit groups.
- Primary Outcomes: 0/66 premature unsupported terminations occurred with ECT versus 40/66 for the faithful controller, a difference of −60.61 pp with a 95% interval of [−78.79, −40.91] pp.The confirmatory analysis covered 22 primary task clusters and H2 passed.
- Primary Outcomes: Supported completion was 97/132 for ECT versus 92/132 for the controller, a difference of 3.79 pp with interval [0.00, 9.09] pp.The lower bound exceeded the prespecified −10-point noninferiority margin, so H3 passed.
- Recovery: ECT achieved successful recovery in 18/66 eligible incomplete trajectories, with 17 subsequently completing with support.The prespecified event-level H4 gate passed.
- Control Cost: ECT used 0.90 additional decision turns, 4,653 additional tokens, 2.34 additional provider seconds, USD 0.00246 additional estimated cost, and 0.045 additional tool calls per trajectory.These paired secondary estimates retained all primary rows and were not efficiency claims.
Limitations, Ethics, and Reproducibility
The evidence is bounded by synthetic, fault-intervention studies and declared contract and adapter assumptions, with reproducibility limited by missing raw planner proposals and persistence-dependent provider audits.
- Scope: Generated worlds cover six task forms rather than deployment traffic, and one planner/model stack with 24 closed-loop tasks does not establish cross-framework generality.The V2 confirmatory analysis used only 22 primary task clusters despite three seeds and fresh worlds.
- Assumptions: A trusted contract can omit requirements, an adapter can fabricate a consistent receipt, and exact replay covers only a closed transform language.The certificate therefore does not establish external truth, authenticity, effects, safety, or alignment.
- Reproducibility: The V2 store retained hashes, decisions, usage, and outcomes but not raw planner proposal text, preventing replay of certificate scoring from captured provider outputs.Provider-attempt claims therefore apply to persisted audits, and a process death before commit could leave an unjournaled call that resume would repeat.
- Fault Design: Static faults were deliberately aligned with declared certificate obligations, so the static study does not establish robustness to unknown error classes.Its 36 held-out task clusters and six synthetic worlds also limit precision claims.
- Interpretation: Unseen-fault and conflicting-receipt rows were descriptive; internally consistent receipt conflicts did not establish which receipt was externally true or define a general recency policy.The claim is limited to the tested descriptive strata.
Conclusion
ECT makes terminal support checkable and substantially reduces unsupported stopping in both locked static and fresh closed-loop evaluations. Its conclusion is bounded to trace support under declared assumptions, not external truth, safety, universal task success, or alignment.
- 0/288 unsafe ECT completions versus 252/288 for the critic core in the locked static study.H1 passed.
- 0/66 premature unsupported ECT terminations versus 40/66 for the faithful-controller pipeline in the fresh V2 closed loop.H2–H4 passed.
- ECT met the completion noninferiority margin and recovered successfully in 18/66 eligible trajectories, with 17 subsequently completing with support.
- ECT establishes trace support under declared assumptions, not external truth, universal task success, safety, or alignment.