AgentProofBench: Benchmarking Reusable Verifiers for Web Agents

• A remembered answer can yield a convincing decision for the wrong requirement when the public instruction stays fixed but task obligations change with initial state.The changing requirement may concern the current owner of a name, collection contents, or a value to preserve.Source: §1 · p. 1
• Matched continuation histories preserve, change, or withhold the same completed work under different stopping policies, making verifier decisions' execution consequences observable.Rejection can trigger further edits, while acceptance can release the resulting state.Source: §1 · pp. 1–2
• The benchmark tests whether rules recognize permitted terminal outcomes, recover changed-state requirements, and support effective continuation and delivery decisions.These aims build on executable task evaluation, automatic assessment, generated verifiers, and reusable execution policies [20] [22] [23] [31] [8] [16] [24] [19].Source: §1 · p. 2
• AgentProofBench fixes the public instruction and evidence permissions while varying initial business data, then connects the verifier to actions taken after its decision.Measurements link versioned reference predicates, program source, API evidence, and reconstructable business state.Source: §1 · p. 2
• In the execution-validity test, programs with error-free deployment-state panels correctly delivered both saved openings in 32 of 33 assignments, versus 1 of 11 assignments with panels containing errors.Source: §1 · p. 2

• Executable task environments [20] [22] [23] [31], generated tests or rewards [3] [9], and verifier benchmarks [2] [30] inform the benchmark's executable evaluation setting.Related work also includes enterprise-service replicas, verifiable worlds [21] [27], callable scientific requirements [12], and the broader test-oracle problem.Source: §2 · p. 2
• AgentProofBench extends verifier-generation evaluation to matched initial-state requirements and the subsequent execution controlled by verifier decisions.VerifierBench evaluates executable verifier generation across agent benchmarks, including legal alternatives, ordinary errors, and attacks [24].Source: §2 · p. 2
• Prior evidence-based judgment work studies environment and trajectory evidence [8] [18] [32], while AgentProofBench compares fresh API judgments with executable programs under shared states and evidence permissions.Related methods include criterion-based computer-use assessment [16], state-and-trace evidence [17], agentic verification [4] [26], and continuous scores and repeated evaluation [7].Source: §2 · p. 2
• Reusable execution policies, workflow-revision studies, frozen self-verification, computation allocation, and matched-checkpoint audits provide neighboring approaches to reuse and verification [29] [10] [11] [28] [6] [25] [13].Figure 2 connects paired verifier tests with execution from shared histories.Source: §2 · p. 2

3.1 An executable acceptance interface

• The verifier uses a task's instruction, API, and before/after states to judge completion; frozen source runs in a read-only sandbox with runtime state-dependent requirements and bounded resources.Its interface returns Pass, Fail, or Unknown; exceptions, malformed output, or execution-limit exhaustion yield Error.Source: §3.1 · p. 2

3.2 Outcome coverage and reference semantics

• Reference predicates cover identities, values, collections, and relationships, with versioned selectors resolving required objects from each before state.A preservation example resolves Plot B3 from the current before state and checks that its Dormant status survives; a separate audit agrees with 191 of 192 resolved labels (99.5%).Source: §3.2 · p. 3

3.3 Matched initial-state reuse

• Across 48 task families, the reuse study changes a required binding, value, collection, or preservation condition while holding instruction and source code fixed.Qualification checks instruction support, state construction, agreement between native and API references, and a source-valid decision that becomes invalid after the change; replay witnesses test whether the verifier resolves the new condition from the current before state.Source: §3.3 · p. 3

3.4 Verification during continued execution

• The execution study branches four policies from each saved opening: actor stop, fixed neutral continuation, program gate, and online gate.Each opening preserves initial state, current business state, and interaction history; gate feedback signals nonacceptance without program checks or entity identifiers, while executed writes persist even if delivery is withheld.Source: §3.4 · p. 4

3.5 Measurements and comparison units

• Balanced accuracy weights valid acceptance and failed-state rejection equally; joint correctness additionally requires a correct witness verdict, and conditional rates identify their selected support.Execution records correct, incorrect, withheld, and unresolved delivery separately, while family-cluster resampling retains observations from the same original family.Source: §3.5 · p. 4

4.1 Joint tests of initial-state reuse

• 82/96 Flash and 79/96 Pro source programs pass joint correctness; reuse-minus-rebuild BA is -1.56 points [-5.47, 1.57] for Flash and -4.95 [-10.16, 0.00] for Pro.The corresponding rebuild joint counts are 82/96 for Flash and 86/96 for Pro.Source: §4.1 · p. 4, Table 1 · p. 5, Figure 3 · p. 5
• Reuse failures include remembered Elena and Active literals, plus a JSON predicate that misses a required record inside results[0].In the JSON case, the computed required set contains eight of nine qualifying items.Source: §4.1 · pp. 4–5

4.2 Panel correctness and execution outcomes

• 32/33 assignments with error-free related panels deliver both openings correctly, versus 1/11 assignments with panel errors.All 17 program-gate withholdings and all three reference-defined regressions occur in the panel-error group; 13 withheld terminal states satisfy the reference.Source: §4.2 · p. 5, Figure 4 · p. 6
• Neutral continuation delivers 20/22 Flash and 22/22 Pro openings correctly, the highest level among the tested generic-feedback policies.Program gating delivers 17/22 Flash and 16/22 Pro openings; online gating delivers 20/22 and 21/22.Source: §4.2 · p. 6

4.3 Acceptance profiles on shared outcomes

• B12 false acceptance is 3.5% versus 31.7% for API12, while valid-anchor acceptance is 77.8% versus 98.6%.The comparison uses identical before/after states and explicit-condition reference labels across 36 tasks and 29 families.Source: §4.3 · p. 6
• On the shared natural panel, B12 BA is 92.1 [87.7, 95.9]% and API12 BA is 99.0 [97.7, 99.9]%; median assessment time is 0.58 s versus 20.69 s.On the anchored hard suite, BA is 87.1 [80.6, 93.6]% for B12 and 96.2 [93.8, 98.1]% for API12.Source: §4.3 · pp. 7–8

4.4 Construction cost and the reuse horizon

• Median API-fee break-even horizons are twelve assessments for Flash and ten for Pro, with ranges of 2–29 and 2–36, respectively.At ten uses, Flash program-gate means are USD 0.0195 for natural endings and USD 0.0189 for checkpoints; corresponding Pro means are USD 0.0509 and USD 0.0478.Source: §4.4 · p. 8

5 USING THE BENCHMARK

• Frozen verifier outputs can be tested against changed initial-state requirements with complete-panel and joint-witness correctness.The recorded actor openings then support comparisons of completed work, persistent mutations, and delivered state.Source: §5 · p. 8
• Stable task, family, program, and state identities link false acceptances, transfer errors, and execution outcomes to their causes and consequences.This lets generator improvements be assessed against the changed requirement they should recover and the policy outcome they should support.Source: §5 · p. 8
• The companion Evidence Index maps the benchmark entry points to canonical artifacts, while verify(task, before, after) evaluates frozen programs without model calls.Canonical snapshots and exact row deltas reconstruct saved states.Source: §5 · p. 8

6 CONCLUSION

• 32/33 assignments with error-free deployment-state panels had both openings delivered correctly, versus 1/11 assignments with panel errors.Neutral continuation anchors measurement of gate value, and the released resource connects changed requirements to observed execution consequences.Source: §6 · p. 8

A REPRODUCTION AND REFERENCES

• Agent-Diff revision 3bb9c407 supplies snapshot clients and authenticated APIs; Python 3.12 sandboxes use read-only filesystems and bounded scratch space.Qualification acts through ordinary APIs before the task actor is restored and the state saved.Source: A REPRODUCTION AND REFERENCES · p. 9
• The companion Evidence Index maps assessment materials to canonical artifacts and provides reproduction commands; original code and annotations use MIT.The linear_52 example traces an instruction to reference requirements, including preservation of B3’s prepared Dormant status.Source: A REPRODUCTION AND REFERENCES · pp. 9–10
• The natural-state flow retains 556 main-panel states from 776 resolved attempts, with 452 successes and 104 failures.The remaining 141 unique states enter an all-attempt sensitivity; the main panel samples at most six states per task by reference class, independently of verifier predictions.Source: A REPRODUCTION AND REFERENCES · p. 9
• The paired collection comparison shares 99 failed states and 36 successful anchors across 36 tasks and 29 families, with two observations per state.The paired collection contrasts are descriptive family-bootstrap intervals; prespecified hypothesis tests concern reuse–rebuild BA.Source: A REPRODUCTION AND REFERENCES · p. 9
• The semantic audit agrees on 191/192 resolved cases (99.5%, Cohen’s 𝜅= 0.989); three cases are indeterminate.The released audit preserves first-pass labels and reconciliations, while benchmark scores retain frozen reference labels.Source: A REPRODUCTION AND REFERENCES · p. 9
• The formal reuse cohort selects 48 families from 57 retained tasks and 51 families using frozen inventory order and service coverage.Duplicate legal outcomes are omitted across three initial states, while six single-positive tasks retain one valid and two failed roles in every condition.Source: A REPRODUCTION AND REFERENCES · p. 10

B STATISTICAL PROTOCOL

• Reuse-minus-rebuild BA contrasts use centered-bootstrap tests with Holm adjustment; adjusted p-values are 0.4350 for Flash and 0.1254 for Pro.The family bootstrap uses 10,000 resamples, averages replicas within task, and gives Unknown and Error zero correctness credit in complete common panels.Source: B STATISTICAL PROTOCOL · p. 10
• The exploratory panel–delivery association resamples 22 task families while retaining both actor assignments and both openings.Its statistic is the difference in assignment fractions whose two openings are both delivered correctly; 9,999 of 10,000 draws contain both panel groups.Source: B STATISTICAL PROTOCOL · p. 10
• 83/96 Flash and 80/96 Pro programs were correct on both source and related panels; six Flash and three Pro programs were correct on source only.One Pro program was correct on related only, and seven Flash and twelve Pro programs had errors on both; background changes preserve all source-qualified programs.Source: B STATISTICAL PROTOCOL · p. 10
• Source-witness conditional acceptance has 92 Flash and 89 Pro qualifying programs, while related rebuilds accept 5/96 Flash and 2/96 Pro failed replay witnesses.Figure 8 measures forward and reverse transfer; source-witness conditional acceptance and source-panel support use different prerequisites.Source: B STATISTICAL PROTOCOL · p. 10

Development gate

• Neutral feedback qualifies as the development choice, with changes on seven families and repairs on thirteen trajectories.The gate requires integrity on at least six families and feedback-arm changes on at least six families with reference transitions on five distinct trajectories; all 44 trajectories supply an available checkpoint.Source: C POLICY OUTCOMES AND COSTS · p. 10

Costs

• One Linear29 connection timeout leaves the Flash natural online fee unknown and withholds a successful outcome.The cost views use cold, warm, and ten-use fees with the shared actor prefix and complete-usage records.Source: C POLICY OUTCOMES AND COSTS · pp. 10–11

Construction and assessment

• Figure 8 summarizes 6,696 program predictions across 18 construction/assessment cells, with two replicas and 48 families per value.Circle color encodes task-averaged BA on a common 80–100% scale; outlined cells assess the condition used for construction.Source: Figure 8 · p. 11

Policy outcomes

• Among source-qualified assignments (17 Flash, 19 Pro), program gating and neutral continuation each correctly delivered 16/17 Flash assignments; for Pro, corresponding delivery counts were 16/19 and 19/19.Source: C POLICY OUTCOMES AND COSTS · p. 11
• At checkpoints, fixed continuation succeeds on 45/57 Flash/Pro requests, program gating on 48/65, and online gating on 45/62.The three Pro natural regressions occur in Linear41, Linear44, and Linear45, with four, three, and two successful mutations.Source: C POLICY OUTCOMES AND COSTS · p. 11

Cross diagnostic

• Assigned-source versus fresh-Pro agreement is 81.82%/75.00% at natural endings and 85.23%/76.14% at checkpoints for Flash/Pro actors.The online-gate comparison uses 124 matched observations on 121 decoded state pairs from 87 anchors.Source: C POLICY OUTCOMES AND COSTS · p. 11

Construction

• InitialRead minus SelfReview improves natural BA by 12.0 [4.2, 20.4] points for Flash and 12.3 [4.2, 20.8] for Pro.SelfReview uses the instruction and documentation; InitialRead adds initial-state API reads, and Bounded adds execution feedback.Source: D CONSTRUCTION AND ARCHIVES · p. 11

Source diagnosis

• 149 inspected false rejections across 22 programs include 109 representation/identifier cases, eight omitted alternatives, seven exact-match constraints, and nine unrequested preservation conditions.Rule-specific edits changed all 149 inspected decisions to acceptance; eighteen available negative controls returned Fail afterward, versus seventeen beforehand.Source: D CONSTRUCTION AND ARCHIVES · p. 11

Archives

• The archived pilot contains twelve tasks from eleven families and 96 policy outcomes, while its 34 continuations make zero mutating requests.The formal experiment instead uses its own family cohort, matched three-condition panels, and saved checkpoint branches.Source: D CONSTRUCTION AND ARCHIVES · p. 11

Execution outcomes

• Figure 9 links every assigned program to execution outcomes, pairing openings of the same assignment and marking source-panel passes and reference-defined regressions.Horizontal position encodes related-panel errors; marker shape distinguishes natural endings, checkpoints, and regressions, while color denotes correct, incorrect, or withheld delivery.Source: Figure 9 · p. 12