Source-linked AI summary
When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation
Avyay M. Casheekar
TL;DR
Agent evaluations often treat a stopped-run snapshot as final without establishing outcome finality or separation between trials. This paper develops a completion argument and tests it through controlled replay and protocol review, finding that delayed operations can change labels and later-run scores while documentation of these issues is inconsistent.
Problem
Evaluation endpoints may leave outcomes unsettled and allow earlier runs to affect later runs, so they do not alone establish final labels or separate trials.
Method
The paper defines outcome finality and cross-unit separation, develops a completion argument, and proposes an open-effects record for unfinished operations and persistent resources.
Results
Controlled replay shows endpoint and terminal labels diverge under delay and shared state can transfer effects between runs, while ten protocols document unfinished operations and trial separation inconsistently.
Takeaways & Limitations
A final label and separate-trial analysis require resolving, bounding, or representing effects that could change outcomes or connect runs.
Takeaways & Limitations
Neither study estimates how often these mechanisms affect published results, and the replay uses fixed traces without a model API while the review covers ten selected protocols.
Abstract
from arXiv · showhide
Current agent evaluations score models on the state visible at the end of a stopped run which they count as one trial. However, interpreting the score as a final result would require two conditions that the endpoint does not itself necessarily establish: outcome finality and cross-unit separation. These conditions are independent, since reconciling a delayed outcome can settle the label while runs still share state and isolating runs can prevent carryover while the scored outcome remains unfinished. We develop a completion argument that specifies the evidence needed for each decision and argue that a final label is justified only when anything that could still change the claimed outcome is resolved, bounded, or retained as uncertainty. First, in a controlled replay to demonstrate the mechanism where an agent's actions were held fixed, we find that the endpoint and terminal labels differ for every delayed operation, while a delayed write changes the next run's score when service state persists between runs but not after isolation or verified reset. Second, in a review of ten public protocols, we find that all protocols identify when a run stops and what is scored, while unfinished operations and the evidence for treating runs as separate trials are documented less consistently. Finally, we propose an open-effects record that lists operations or resources that may remain relevant after the endpoint, their current status, and whether they could change the scored outcome or affect another run.
1 Introduction
An evaluation endpoint does not by itself establish a final outcome or a separate trial: unfinished operations can change the score, while persistent files, credentials, accounts, or service values can affect later runs. The paper distinguishes outcome finality from cross-unit separation and develops a completion argument specifying evidence for both.
- Motivation: The endpoint stops requested actions but may leave tool calls running or state available to later runs, so it can end interaction without settling the result or trial boundary.A delayed operation can change the scored outcome, while persistent resources can alter later runs.
- Related evidence: Reported cyber-testing incidents show persistence and cross-run exposure through reused accounts, leftover artifacts, external services, and activity continuing beyond the original runtime.These reports demonstrate boundary crossing in those settings but do not establish how frequently it occurs.
- Outcome finality: Outcome finality requires evidence that no remaining effect can change the claimed success or failure label.For a write requested before the endpoint but applied later, the immediate score may differ from the score after completion.
- Cross-unit separation: Cross-unit separation requires that no relevant route connects runs, including shared service state or resources changed by an earlier run.Waiting for a delayed write can establish finality while leaving its effects visible to a subsequent run.
- Completion argument: A completion argument must state the scored outcome, judgment period, observation unit, unfinished operations, and shared resources supporting the final-result claim.When calls finish before returning and the outcome is defined at the endpoint, finality follows directly; separation requires verified fresh state and no later cross-run changes.
2 Two conditions for completion
A stopped run’s endpoint does not by itself establish either outcome finality or cross-unit separation, which require different evidence. A final result is justified only when the claimed outcome is fixed and effects cannot carry into another analysis unit.
- Outcome finality: Outcome finality requires evidence that fixes the named outcome over the protocol’s specified period; remaining uncertainty should be reported as unresolved.“Visible at the endpoint,” “committed within 24 hours,” and “eventually committed” describe different outcomes, and a snapshot cannot establish completion.
- Cross-unit separation: Cross-unit separation requires evidence that one run cannot change another run’s relevant starting conditions or outcome.It concerns effects carried between runs, not the disappearance of every correlation; fresh-state claims require inherited state to be addressed.
- Evidence for completion: Following a delayed operation can settle its outcome without preventing carryover, while isolation can protect later runs while the original operation remains unfinished.Restoring shared state protects later runs only after pending effects can no longer reintroduce it, so generic cleanup claims are insufficient.
- Construction of evaluations: Some evaluations satisfy both conditions by construction, whereas asynchronous evaluations may stop while a service is still applying a write.A synchronous simulator can finish every tool call and restore relevant state before the next run begins.
- Interpretation of endpoints: A snapshot can validly measure state at time t without establishing that the task was completed.The evaluation boundary must be judged against the outcome and analysis unit the report claims to measure.
3 Local stopping and global completion
An endpoint does not by itself establish outcome finality or separation between trials: delayed work may still change a result, and persistent effects may connect runs. Final labels and separate-trial analyses therefore require route-specific evidence or preserved uncertainty.
- Local stopping and global completion: Stopping an agent’s interaction loop does not establish completion when messages or operations remain in transit or leave the evaluator’s observation scope.Distributed-systems termination detection accounts for work beyond an idle process; an agent endpoint may not show that the outcome can no longer change.
- Local stopping and global completion: Cross-run interference occurs when credentials, artifacts, messages, or stored values produced by one run can change another run’s conditions or outcome.A new run identifier does not ensure separate trials when run B can use state produced by run A.
- Local stopping and global completion: Final labels require every relevant route to be blocked, followed until its effect is known, or bounded tightly enough to determine the label; otherwise, report unresolved.A bound suffices only when every value within the justified range yields the same label or conclusion.
- Local stopping and global completion: Runs count separately only when every relevant route between them is blocked or shown unable to change either run’s conditions or outcome.Otherwise, represent the connection or treat connected runs as a larger unit.
- Local stopping and global completion: Following a delayed write can settle run A’s result while leaving its state visible to run B, so finality and cross-run separation must remain distinct.Grouping connected runs can account for carryover without treating them as separate repetitions.
4 Boundary policies change the recorded result
In a controlled replay with fixed traces, scoring before delayed writes changed labels, while shared service state allowed one run to change the next. Isolation, verified reset, and terminal reconciliation prevented these effects, showing that endpoints alone cannot establish final labels or analysis units.
- Controlled replay: Fixed tool calls and operation schedules isolated boundary-policy effects on labels and subsequent run state.The requested operation, delay, and execution order remained fixed, so observed differences came from scoring and state-sharing policies.
- Outcome finality: 150 of 200 nonzero-delay snapshot trials disagreed with terminal labels because scoring preceded the write.Snapshot scoring recorded 50 of 200 successes, whereas reconciliation recorded 200 of 200; verified cancellation assigned failure to 150 pending writes.
- Cross-unit separation: 150 of 200 shared-state pairs exposed run B to run A’s delayed write, while namespacing and verified reset produced 0 of 200 exposures.Running B before A also reduced exposure from 150 of 200 pairs to 0 of 200.
- Analysis units: Grouping preserved 150 exposures but changed the analysis from 400 run rows to 200 pair-level units.The grouped analysis left shared state unchanged and counted each connected pair as one unit.
- Interpretation: Because delays and shared state were imposed by design, the counts do not estimate their frequency or effect size in published benchmarks.The replay demonstrates how boundary policies change recorded results rather than measuring benchmark-population behavior.
5 Documentation across ten protocols
The review of ten public protocols finds that run boundaries and scored state are documented consistently, but unfinished operations and evidence that runs are separate are less consistently reported. These findings concern public documentation and do not establish benchmark validity or outcome finality.
- Review scope: The review examines ten protocols and records run definitions, stopping conditions, scorer-visible state, unfinished operations, persistence, reset or retention, and evidence for separate trials.The sample includes WebArena, WorkArena, OSWorld, SWE-bench, tau-bench, ToolSandbox, TheAgentCompany, RE-Bench, Cybench, and AgentCanary.
- State management: 8 protocols explicitly document reset or deliberate retention, while 2 provide only partial documentation.A partial code means the rule is stated but its scope or evidence is incomplete.
- Unfinished operations: 6 protocols expose potentially persistent or long-running resources without stating whether unfinished operations finish, are cancelled, or are checked before scoring.These resources include shells, browsers, virtual machines, services, and long-running computation.
- Cross-unit separation: 3 protocols provide explicit evidence that runs are separate observations, whereas 7 describe reset, teardown, or fresh provisioning without fully specifying scope or restoration verification.Reset supports separate scoring only when its coverage and successful completion are documented.
- Scope and limitations: The review does not code outcome finality and therefore reports documentation of unfinished operations rather than protocols proven to satisfy finality.Only one reviewer coded the materials, and missing periods were not inferred.
6 Making completion reviewable
A completion argument makes final labels and separate-trial claims reviewable by specifying the outcome, stopping rule, lingering effects, and evidence for resolving or bounding them. An open-effects record makes such evidence inspectable but cannot itself establish finality or separation.
- Completion argument: A reviewable completion argument identifies the scored outcome, observation unit, interaction stop rule, scoring basis, and effects that may remain active or reach another run.It explains how the protocol handles anything that can remain active or pass between runs.
- Completion argument: Finality and separation require environment-specific evidence, such as observing terminal settlement, confirming process-tree completion, verifying fresh namespaces, or bounding effects so every possible value yields the same result.Synchronous tools in closed simulators may establish finality on return, while fresh namespaces without shared credentials may establish separation.
- Open-effects record: An open-effects record tracks potentially persistent operations and resources through handles, endpoint status, possible later states, and routes to another run.At the endpoint, the evaluator queries the system and records whether each effect is terminal, retaining cancellation or removal evidence when applicable.
- Open-effects record: The record makes evidence inspectable but does not establish finality or separation; self-reports and wrappers may be incomplete, system checks remain necessary, and an empty record proves nothing.The limitation applies even when no relevant effect is listed.
- Interpretation and analysis: Endpoint scoring can differ from eventual-outcome scoring, while connected runs can inflate separate-attempt counts and undermine interpretations of metrics such as passk.Protocols should isolate and verify runs, model remaining connections, or count connected sets as one unit; endpoint or stream claims may remain narrower.
7 What the evidence establishes
An endpoint marks when an evaluation stops, but a final label requires unresolved effects to be addressed and a separate trial requires carryover to be blocked. The completion argument and open-effects record make these conditions inspectable, while insufficient evidence calls for narrower scoring or larger analysis units.
- Outcome finality and trial separation: A final label requires evidence that later effects cannot change the claimed outcome, whereas a separate trial requires evidence that carryover is blocked.These decisions can diverge under delayed operations.
- Scope and limitations: The replay and review do not estimate how often these mechanisms affect published results, and their findings are limited by a benign local service, fixed traces, ten selected protocols, one reviewer, and unobserved private implementations.An open environment cannot prove every unknown effect is absent.
- Completion versus safety: Completion differs from safety: waiting can reveal a harmful operation’s outcome while permitting harm, whereas cancellation can improve safety while changing the outcome observed.The paper’s rules govern result meaning and do not replace incident response or safety controls.
- Inspectable evidence: The completion argument and open-effects record document what could still happen, what could carry over, and how each case is handled.When evidence is insufficient, scores should be described more narrowly or connected runs analyzed as a larger unit.
A Documentation review matrix
The documentation review codes public protocols for open work, reset or retention, and separate evidence using an auditable source ledger. The matrix shows that reset mechanisms are often documented, while pending work and the scope of separation evidence remain less consistently specified.
- Review method: The review assigns protocol-level codes for open work, reset or retention, and separate evidence, with E, P, NR, and NA indicating explicit, partial, not reported, and not applicable.Each public protocol is one review unit, and one reviewer assigned each code under the Section 5 rubric; the ledger records the source and rationale for each decision.
- Reset and separation: Virtual-machine and container resets are often explicit, but reviewed sources may not document coverage of services or state outside those environments.This limitation appears for virtual-machine restoration in one protocol and container reset in another.
- Open work and separation: ToolSandbox treats open work as not applicable, but its scenario-local starting state does not by itself establish a separate analysis unit.The environment changes an in-memory execution context through synchronous Python functions, while the sources do not justify the unit of analysis.
- Reset and separation: Some protocols document separate trials through reinitialization or isolation, including τ-bench’s database reset and AgentCanary’s temporary per-task containers.τ-bench defines k independent and identically distributed trials, while AgentCanary resets single-instance tasks and retains state only within declared multi-session instances.
- Open work: Open work is only partially documented or not reported in protocols where agents can run extended computations, background processes, or descendant work.The reviewed sources do not state how live or descendant processes are handled before scoring in TheAgent Company, RE-Bench, Cybench, and some execution-harness settings.
B Controlled replay details
The controlled replay fixes the agent’s tool-call trace while varying delayed-operation scoring and whether service state is shared, isolated, reset, or grouped. It uses a constructed schedule with repeated delayed writes, whose reported counts were checked against implementation artifacts rather than a benchmark population.
- Replay design: The replay holds the tool-call trace fixed while varying delayed-operation scoring and service-state treatment across shared, isolated, reset, and grouped conditions.This design changes protocol handling without changing the agent’s actions.
- Replay schedule: The schedule covers identifiers 1–5, four write delays from 0 to 250 ms, and ten repetitions for every identifier-delay combination.The delays are 0, 25, 100, and 250 ms.
- Execution setup: The standalone runner executes fixed calls without a model, while a local HTTP service retains state between runs and applies each write after its scheduled delay.The service binds to 127.0.0.1 on an ephemeral port and creates no external resources.
- Verification: The reported counts were checked against the frozen schedule, calibration output, run metadata, and 12 implementation tests.The repetitions verify the constructed schedule rather than sample a benchmark population.
C Open-effects record specification
The open-effects record tracks operations or resources that may remain active, change later, or become visible elsewhere. It is updated with system evidence and records how any still-open route is treated in the analysis.
- Record lifecycle: An entry is created for each operation or resource that may remain active, change later, or be visible in another run.The record is updated at the endpoint and after each attempt to observe, cancel, remove, or isolate the effect.
- Endpoint assessment: At the endpoint, the evaluator queries each handle, records current system state, and tests whether later state could alter the outcome or reach another run.Pending or unknown effects remain open rather than being treated as closed.
- Analytical treatment: Open effects may be reported as unresolved, bounded, modeled, or grouped across exposed runs without changing their underlying system status.A separate field records how the analysis treats any route that remains open.
- Evidence sources: Function wrappers, process monitors, transaction logs, and environment instrumentation can populate the record, while an empty agent report does not establish that no relevant route exists.Agent reports may identify effects first visible in the trajectory.