PaperlayerView exact revision on arXiv

Source-linked AI summary

FinalityBench: An Effect-Level Benchmark for Agent Decisions Under Delayed and Conflicting Financial Finality

Abhishek Sharma

arXiv:2609.04706v1cs.AI

TL;DR

FinalityBench asks how agents should act when payment systems disagree and an operation’s outcome may not yet exist. It constructs faulted views from a hidden event log, pairs snapshot-indistinguishable tasks with opposite correct actions, and scores executed monetary effects. Across the benchmark, effect-level loss changes policy rankings, while authoritative finality gating performs best; the model evaluation is narrow and does not establish broad model capability.

  • Problem

    Payment agents must choose among shipping, recapture, refund, or waiting while system views conflict and finality may be unresolved.

  • Method

    FinalityBench derives four operational views from independently faulted deliveries over a hidden event log, uses indistinguishable twin tasks, and scores terminal economic effects against a privileged reference.

  • Results

    Effect-level scoring reorders policy rankings in 7 places; an authoritative finality gate is the most accurate and only policy whose accuracy survives repetition, while unavoidable residual loss prices delayed finality.

  • Takeaways & Limitations

    The benchmark shows that choosing an authoritative finality channel, rather than repeatedly reading replicas, is central to avoiding preventable monetary effects.

  • Takeaways & Limitations

    The model arm uses one vendor, one prompt, one temperature setting, two seeds, and an 86-episode subset, with two attempted models unevaluable due to rate limits.

Abstract

from arXiv · show

A merchant's payment processor, ledger, ERP and bank feed are updated by messages that get delayed, duplicated, dropped and reordered, so for minutes at a time the four hold contradictory beliefs about the same order. An agent resolving the exception must decide whether to ship goods, re-submit a capture, refund or wait, knowing some of those cannot be undone. We present FinalityBench, an executable benchmark for that decision. It keeps a hidden canonical event log and derives each system's view from a separately faulted delivery stream, so disagreement follows from specified fault semantics rather than being authored. Grading is on executed monetary effects: an episode is scored by the merchant's terminal economic position, relative to a privileged reference told when the pending capture resolves. The corpus of 321 tasks includes 45 twin pairs (90 tasks): tasks whose four system views are identical at the decision instant, whose authoritative probes both return unknown, and whose eventual correct dispositions differ. That snapshot indistinguishability is checked under every evaluation seed rather than assumed; equivalence over all interaction traces is not claimed. Over 14,445 graded episodes from nine programmatic policies, ranking by single-task accuracy and by paired loss disagree in 7 places: a ship-on-first-sign policy is second-best by accuracy at 65.7% and worst in the suite by paired loss, because it cannot tell the two members apart. A runtime gating irreversible actions on an authoritative finality probe reaches 85.4% and, unlike every polling policy, loses nothing to pass^5; its residual loss is almost entirely one archetype, which prices finality information directly. Language models reach the same exact rate as the hand-written gate on a stratified subset, lose about twice as much money, and discover the finality-gating strategy without being told it.

I. INTRODUCTION

FinalityBench treats temporary disagreement among payment systems as a decision problem where delayed outcomes, irreversible actions, and asymmetric monetary costs matter. It constructs faulted operational views, indistinguishable counterfactual tasks, and effect-level monetary evaluation to measure agent decisions.

  • Payment exceptions are difficult because systems can temporarily disagree about whether an operation is settled, while shipping cannot be revisited.
  • Waiting may be the only action that resolves uncertainty, while premature shipping or recapture can impose asymmetric monetary losses.
  • Existing benchmarks do not jointly treat finality as an independent variable and grade decisions by dollars of irreversible effect.
  • The benchmark projects a hidden canonical event log into four operational systems through independently faulted delivery streams with system-specific observable signatures.
  • Twin tasks share indistinguishable tool views at the decision instant but require different correct actions, making premature commitment measurable.
  • An effect-level monetary oracle scores terminal economic position against a reference informed when the pending capture resolves.
  • The reference evaluation covers nine programmatic policies over 14,445 graded episodes, while the model arm is limited to a stratified subset.

III. THE ENVIRONMENT

The environment maintains one hidden event history but exposes four separately delivered operational beliefs. Their storage semantics determine which delivery faults are visible, harmless, or economically dangerous.

  • Each task has a hidden append-only event log containing order placement, authorization, capture submission, terminal outcome, and possibly a later chargeback.
  • Each system receives a separately derived delivery stream, with fixed latency and bank batching even when no faults are enabled.
  • Duplicate delivery is invisible to the processor because it stores status by operation.
  • The append-only ledger double-posts duplicates and can expose imbalance after a partial commit.
  • The ERP is keyed by order and therefore idempotent.
  • The bank shows duplicated settlements as separate statement lines, so diagnosing duplicates or partial commits requires cross-system comparison.

B. Fault families

The environment separates profile-controlled delivery faults from task-defined late reversals and gives policies tools whose costs and reversibility shape what information can be obtained before deadlines.

  • Fault families: Delay, duplication, loss, reordering, partial commit, and stale reads are controlled fault families sharing intensity 0.30.
  • Fault families: Reordering is restricted to nearby deliveries and cannot move a delivery before the event it describes.
  • Fault families: Unbounded reordering was removed because it violated modeled causality and could reveal terminal outcomes early.
  • Fault families: Late reversal is task-defined rather than profile-controlled because a late chargeback is a new world event, not a mis-delivered old one.
  • Actions and finality: The authoritative probe bypasses delivery faults and returns unknown until the operation’s terminal event exists.
  • Actions and finality: Reusing a capture’s original idempotency key is a no-op after settlement, whereas a fresh key creates a genuinely new capture.

D. A worked case

A worked lost-settlement case shows why replica readings can encourage duplicate capture while authoritative finality remains unresolved. The benchmark evaluates the resulting economic position against a privileged but non-clairvoyant reference.

  • In the worked case, the capture settles at t = 300, but messages to the processor and ERP are dropped while the agent acts at t = 120.
  • At t = 121, replica views suggest that payment has not arrived, so a fresh-key recapture would later charge the customer twice.
  • The authoritative probe returns unknown at t = 121 and settled from t = 300, making waiting and probing sufficient to ship before the deadline.
  • The environment fixes a one-tick read cost, five-tick authoritative probe cost, a t = 600 ship-by deadline, and an order-value range of $24.99 to $3,499.00.
  • Terminal position accounts for settled cash, goods cost, liability, and dispute fees, with disjoint deductions summing to total loss.
  • Excess loss is the privileged reference position minus the achieved position, functioning as an economic-position gap rather than a proven distance below an optimum.
  • The reference knows when and how the pending capture resolves but not future chargebacks, so it is privileged without being clairvoyant.

IV. PAIRED COUNTERFACTUALS

Paired counterfactuals test whether agents commit without evidence of finality by presenting indistinguishable snapshots whose eventual dispositions differ. The benchmark verifies snapshot equality across seeds, but does not establish equivalence across all interaction traces.

  • Paired counterfactual construction: Twin pairs share identical four-system snapshots and unknown authoritative probes, although one capture settles and the other fails unrecoverably.The best first action can still be wait for both members.
  • Verification boundary: Snapshot equality is verified at the decision instant across evaluation seeds, not equivalence over every possible interaction trace.Waiting, reordered reads, or keyed retries might reveal a difference before either capture resolves.
  • Paired evaluation: A policy committing at the shared snapshot cannot distinguish the pair, so its pair mean prices premature commitment as a gamble.Aggregate accuracy can reward always shipping because settling outcomes are more common than failing ones.
  • Corpus construction: The corpus contains 321 tasks across nine archetypes, including paired cases constructed under regenerated seeds.The generator and seed design prevent tuning against memorized task identities from transferring directly.

B. Policies

The benchmark compares nine programmatic policies under identical execution and grading, with repeated seeded episodes and task-level bootstrap uncertainty. The finality gate leads the reported results, while its remaining loss concentrates in unresolvable tasks where finality arrives after the shipping deadline.

  • Policy suite: Nine policies share one interface and are graded by identical code across the benchmark.The suite includes progressively more sophisticated policies, from random and optimistic shipping through polling and reasoning loops.
  • Reproducibility: The benchmark records canonical logs, schedules, grades, seeds, fingerprints, and code revisions for reproducible experiment generation.Tables and figures are generated from experiment files rather than manually entered numbers.
  • Headline comparison: 85.4% exactness and $31.37 mean excess loss are reported for the finality gate.It is exact on six archetypes but loses $245.16 per unresolvable case.
  • Evaluation design: 14,445 graded episodes reuse the same 321 tasks across seeds, so uncertainty is bootstrapped over 276 task units with twin pairs held together.The procedure uses 2,000 replicates rather than treating episodes as independent observations.
  • Uncertainty and comparisons: The finality gate’s exact rate is 85.4 [79.9, 90.2]%, while its mean gap is $31.37 [16.28, 48.27].On shared task resamples, its mean gap is $-96.81 [-118.46, -77.91] below the ReAct loop’s, with an interval excluding zero.

C. Accuracy is not reliability

Accuracy can reward policies that gamble on favorable outcomes, while paired loss and pass^5 expose whether decisions remain reliable across indistinguishable tasks and seeds. The finality gate stays reliable across seeds, whereas optimistic and polling policies do not.

  • 85.4% to 85.4%: the finality gate’s exact accuracy is unchanged under pass^5, unlike polling policies that collapse across seeds.ReAct falls from 54.0% to 17.4%, and the rule-based procedure from 34.0% to 6.9%.
  • The finality gate’s seed stability is structural: authoritative finality either becomes established before the deadline or it does not, regardless of dropped deliveries.Replica-based policies depend on the fault draw, so their correctness varies across seeds.
  • 65.7% accuracy makes the optimistic policy second-best, but its paired loss of $253.53 makes it worst in the suite.The gate has $103.91 paired loss, and rankings by accuracy versus paired loss disagree in 7 places.
  • The optimistic policy takes irreversible action before the outcome existed on 66.0% of tasks, so aggregate accuracy rewards a favorable gamble.Settling outcomes outnumber failing ones, causing aggregate accuracy to conceal the policy’s procedural failure.
  • Language models match the gate’s exact rate on the evaluated subset but lose roughly twice as much money when wrong.gemini-3.6-flash reaches 93.0% exact against the gate’s 93.0%, with mean gaps of $13.99 and $6.30 respectively.
  • Models discover finality gating without instruction, using probe_processor in 99.6% of episodes and waiting in nearly all of them.The model arm uses a stratified subset rather than the full corpus, and every archetype is represented.

F. Which faults cost what

Message loss is the dominant fault cost, while reordering is harmless and duplicate delivery mainly harms ledger-and-bank-feed majority voting. The finality gate is invariant to delivery faults, but the benchmark shows that replica-based policies lose differently across fault families.

  • Message loss fires 2.26 times per episode and accounts for most of the damage on its own.
  • Reordering fires 0.68 times per episode and changes no evaluated policy’s decision.
  • Duplicate delivery fires 1.49 times per episode and changes decisions only for majority vote, which incurs $21.41 from double-posting.
  • Loss and delay interact negatively: the rule-based procedure loses $89.81 jointly, $13.18 less than their separate costs summed.
  • The finality gate’s mean position gap is identical in every fault arm because it reads a channel unaffected by delivery faults.
  • The benchmark currently demonstrates the value of authoritative finality signals, not robust reasoning over conflicting evidence, because the winning policy avoids those conflicts.

G. Does tuning survive a regenerated split?

Tuning transfers directionally to a regenerated split, but absolute performance changes substantially, especially for the fitted policy. The headline ordering is robust to economic constants, while middle rankings are not.

  • Tuning improved the public split and remained better than default on the regenerated split.
  • The tuned policy degrades by $37.48 across splits, compared with $3.85 for the unfitted control.
  • The tuned knobs encode a generator property—polling more often—rather than an identity memorized from the public draw.
  • Across 45 cost configurations, the finality gate is ahead of every ungated policy in 45 of 45.
  • The full nine-policy ordering is identical in 20 of 45 configurations, while the middle ranking is not stable.

I. Is the reference actually optimal?

The reference was exhaustively checked rather than assumed optimal, and defects found in its first implementation were repaired. Its remaining gap reflects a documented late-chargeback case, while operation identity and unavoidable late finality shape residual loss.

  • Exhaustive search covered 1,215 wait-and-action plans across 40 sampled tasks to test whether any plan beat the reference.
  • The first search found the reference beaten on eight tasks, including cases where it refunded instead of shipping or delayed a known replacement capture.
  • After repair, only 2 of 40 tasks beat the reference, both late_chargeback cases where a correct decision was later punished.
  • The reference takes an irreversible action before the outcome is determined on 86.0% of tasks because it receives privileged resolution timing.
  • The comparison between authoritative and replica-based policies concerns the observed channel more than general reasoning ability.
  • Reusing the original retry reference prevents duplicate effects in these runs, whereas minting a fresh key produces duplicate charges when the original later settles.
  • The finality gate’s residual $245.16 per unresolvable case is deadline cost shared by every unprivileged policy.

IX. LIMITATIONS

FinalityBench’s claims are bounded by a narrow model evaluation, simulated delivery environment, stipulated economics, and a single-order corpus. Its conclusion identifies authoritative finality as valuable while leaving broader conflict reasoning and more complex workflows unsettled.

  • Model evaluation: The model arm uses one vendor, one prompt, one temperature, two seeds, and a stratified 86-episode subset; two attempted models failed under provider rate limits.
  • Model evaluation: The earlier version evaluated no model, so its reported results were properties of the benchmark and hand-written policies only.
  • Model evaluation: The hand-written ReAct-shaped procedure bounds loop-structure performance rather than standing in for an agent that reasons.
  • Simulation: The environment is simulated, with plausible rather than production-measured delivery semantics, latencies, and batch cadence.
  • Economic assumptions: The economics use defensible round numbers rather than estimates from a merchant’s books, and middle policy rankings change when constants move.
  • Corpus scope: Every task is a single order with one pending capture, excluding multi-item, partial-capture, split-shipment, and concurrent-customer exceptions.
  • Scope of conclusion: The benchmark demonstrates the value of authoritative finality signals but does not yet demonstrate reasoning across conflicting evidence because the winning policy avoids it.
Loading 2609.04706v1…