ABF-Bench: A Verifiable Benchmark for Agentic Blockchain Forensics

1 Introduction

• Investigators must adapt as launderers split flows, route funds across assets and ledgers, and change tactics, while keeping conclusions checkable by another investigator.Public ledger records make claimed movements verifiable, though locating and interpreting those records requires investigation.Source: §1 · p. 2
• Real adversaries supply difficult tracing instances, and public ledgers keep claims checkable, making fund tracing a potential verifiable-reward domain for agent evaluation and training.The paper situates this contribution alongside benchmarks that use mechanical checks and can double as training environments.Source: §1 · p. 2
• ABF-Bench turns real laundering incidents into tasks with auditable reference edges, combining multi-hop transfers, DeFi exchanges, cross-chain round trips, custody and mixer deposits, and address poisoning.Conservative rules determine each reference graph, and each reference edge has a verified witness retrievable within the query budget.Source: §1 · p. 3
• The benchmark evaluates evidence-backed tracing through a shared, metered interface where solvers submit operations with supporting records for verification against a hidden reference graph.An operation supported by its records remains valid even if the reference graph omits it.Source: §1 · p. 3
• The harness outperforms both general-purpose agents in final recall, including Codex on the same model, but agents stop early and breadth-first tracing achieves the highest final recall.The harness also acquires evidence faster than fixed-order tracers early in the budget and recognizes undisclosed custody they miss, at comparable validity.Source: §1 · p. 3

Tracing and anti-money laundering

• Bitcoin clustering and taint analysis infer relationships or allocate commingled value [1,27,28], while TRacer, Connector, ABCTracer, and AMLGuard address account, cross-chain, or incident-level tracing [5] [6] [7] [15].ABF-Bench scores recovered movements and acquisition of supporting records from seeds under metered ledger access.Source: §2 · p. 3

Datasets and agent evaluation

• Elliptic and Elliptic2 provide labeled transactions [3] [4], while LO-CARD and XSplicer evaluate protocol-level links [10] [16]; Table 1 compares settings by scored object, record access, and evidence checking.CryptoBench and AI-ABC cover broader cryptocurrency-agent tasks [13].Source: §2 · pp. 3–4, Table 1 · p. 4

Agentic blockchain forensics and verifiable agent benchmarks

• Agentic blockchain forensics studies LLM-led investigations [10] [12], while verifiable benchmarks score runs against fixed checks; ABF-Bench uses public-ledger evidence to evaluate fund-flow graphs.Other cited agent work includes transaction understanding and attack analysis [11] [9]; verifiable benchmarks include test suites, environment state, captured flags, and checkable answers [32] [33] [34] [35].Source: §2 · p. 4

3 Fund Tracing as a Verifiable Task

• The formalization defines tracing tasks, solver reports, and verifier verdicts, while excluding real-world attribution and speculative links between deposits and withdrawals.These boundaries distinguish recorded, evidence-backed movements from conclusions not determined by the task’s records and rules.Source: §3 · p. 4

3.1 Tasks and Reports

• Tasks specify ledger windows, tracing scope, commingling conventions, and admitted operations, whose evidence rules can require receipts and execution traces for exchanges.The current release scores ledger records; published incident addresses enter only as assumptions on seeds and membership.Source: §3.1 · pp. 4–5
• Fund states locate holdings by ledger, account, asset, and record position; claims assert amounts moving between states and cite records within a report grouped by operation.A tracing task combines admitted records, a finite seed set, a declaration, and admitted operation types.Source: §3.1 · p. 5

3.2 Verification

• The verifier judges cited records independently of the reference graph: supported operations are valid, contradicted claims invalid, and unsupported but uncontradicted claims indeterminate.The operation-type label is advisory; verdicts depend on claims and cited records.Source: §3.2 · pp. 5–6

3.3 Why Tracing Is Hard

• Tracing is difficult because investigators must search among branching leads, recognize protocol-specific operations and custodians, assemble evidence, and choose what to report.Fixed procedures settle these stages in advance, whereas investigators adapt to the case at hand.Source: §3.3 · p. 6

4 ABF-Bench

• The benchmark pairs tracing tasks with a shared evidence environment, verifier, and reference retrieval solver; its reference graph and witnesses remain hidden.The solver’s retrieval plan calibrates budgets rather than exposing the answer.Source: §4 · p. 6

4.1 Design Goals

• Tasks are well-posed when instructions determine the reference answer, every edge has a retrievable witness, and unresolved continuations trigger conservative halts.The verifier accepts any evidence satisfying the operation rules, including valid alternatives to the reference witness; records and scoring are deterministic across solvers.Source: §4.1 · pp. 6–7

4.2 Interaction Under a Budget

• Solvers choose ledger queries or submit a report under a shared query budget, with internal computation and working memory treated as part of the solver policy.The public input includes seeds, block windows, hop cap, followed bridges, disclosed stops, available actions, submission rules, and budget; query costs and budgets are known in advance.Source: §4.2 · p. 7

4.3 The Reference Graph

• Tracing halts at custody or unsupported bridge continuations, while purity excludes commingled holdings once outside value arrives.A mixer deposit establishes entry but leaves the exit unresolved; on Bitcoin, purity is assessed over non-dust inputs.Source: §4.3 · p. 8

4.4 Scoring

• Edge recall counts recovered reference edges only when the report is valid and every cited record was retrieved; fidelity separately measures operations accepted by the verifier.Macro-averaged recall is primary, failed runs remain in the average, and valid downstream edges count even when upstream operations are rejected; precision also penalizes submitted out-of-scope edges.Source: §4.4 · pp. 8–9
• The query budget is calibrated from the reference solver’s retrieval actions plus one activity listing per distinct source holding.Budgets remain fixed after graph amendments, and the reference plan fits within the budget on every retained task; query counts measure evidence access, while token and monetary costs are reported separately.Source: §4.4 · p. 9

5 From Incident to Benchmark

• Within each ledger, Ethereum alternates frontier expansion with record replay until no new holdings or unresolved membership decisions remain; Bitcoin uses transaction replay under input-membership rules.Replay can reveal holdings outside discovered paths or extend tracing windows, feeding expansions back into the next round.Source: §5 · p. 9
• Across-ledger linking supplies destination-ledger entry records, while return deliveries can change earlier membership and trigger a fresh build; repeated pairs signal convergence and older pairs signal oscillation.Human reviewers audit graphs and evidence and record seed, boundary, and rule revisions between builds.Source: §5 · p. 10

5.1 The Released Task Suite

• The Bybit-theft suite follows activity for 90 days across Ethereum and Bitcoin, including transfers, DeFi exchanges, cross-chain round trips, custodial and mixer endpoints, and poisoning activity.Seeds derive from the FBI notice and Bybit’s published incident addresses.Source: §5.1 · pp. 10–11
• 33 of 58 authored tasks pass the release filter, which excludes unsettled interpretations and requires verifier-accepted witnesses for retained reference edges.Tasks are selected for operation diversity, crossings, boundaries, and branching, with pairwise overlap capped at 25% of the smaller graph’s edges.Source: §5 · p. 10, §5.1 · p. 11
• Released tasks exercise all four tracing stages; one holding fans out to 28 reference edges, and 113 edges stop into undisclosed custody versus nearly three times fewer into disclosed custody.Tasks disclose 20 custodians and leave the rest for solvers to recognize.Source: §5.1 · p. 11

• Working memory records confirmed operations as findings, unresolved questions as leads with dispositions, and observations as notes for later use.The agent maintains this representation of investigation progress.Source: §6 · p. 11
• Harness tools operate on working memory, preview submissions for checks, and compact context through summarization.Source: §6 · p. 11
• Five prompt heuristics guide control-of-funds tracing, look-alike address checks, record-grounded citations, continued investigation, and submission of every valid edge found.The harness is designed for blockchain forensics across cases, protocols, and services, and is inspired by LOCARD [10].Source: §6 · pp. 11–12

7.1 Hypotheses

• The hypotheses test whether agents acquire evidence efficiently, recognize operations and undisclosed custody, and assemble and report evidence reliably against fixed-order tracers.The authors characterize evaluated solvers on this suite and leave whether the hypotheses hold for agents generally open.Source: §7 · p. 12, §7.1 · p. 12, §7.5 · p. 14

7.2 Solvers; 7.3 Setup

• Fixed-order tracers expand activity under breadth-first, depth-first, or largest-incoming-amount order; agents use the shared evidence interface, with the harness and Codex holding model and reasoning effort fixed.The comparison uses 31 of 33 released tasks; headline results use the first round.Source: §7.2 · pp. 12–13, §7.3 · p. 13

7.4 Main Results

• 0.75 versus 0.47 edge recall separates the same-model harness and Codex, while breadth-first tracing ranks highest and shows at least 0.88 is attainable.The harness beats breadth-first tracing on only 7 of 31 tasks.Source: Table 3 · p. 13, §7.4 · pp. 13–14
• Harness recall falls from 0.82 on the smallest task third to 0.69 on the largest, and its recall correlates more with fan-out than edge count (ρ = −0.66 vs. −0.41).Across repeated runs, within-task recall standard deviation is 0.18 for the harness and 0.17 for Codex, while mean recall agrees within 0.01.Source: §7.4 · p. 14

7.5 Where Agents Gain and Lose

• At quarter budget, harness evidence recall is 39% versus 35% for the best-in-hindsight tracer; that tracer overtakes after 54% of budget.At its own stopping point, the harness leads the best-in-hindsight tracer on 26 of 31 tasks.Source: §7.5 · pp. 14–15, Figure 4 · p. 15
• The tracer overtakes after 54% of budget; the harness leaves 53% unused, with unexamined branches holding a quarter of its and over half of Codex’s reference edges.What continued querying would recover remains unmeasured.Source: §7.5 · p. 15
• The harness claims 83% of retrieved operations versus bfs’s 85%; agents stop at three quarters or more of eligible undisclosed-custody points, unlike tracers, which stop at none.Bfs reaches more reference edges into those points than the harness, 87% against 63%.Source: §7.5 · p. 15, Figure 5 · p. 16
• Harness and Codex reach bfs fidelity, and agents omit at most 2.5% of retrieved transfers; Claude Code cited unretrieved records in 29 of 31 runs.Under the retrieval condition, Claude Code’s fidelity falls to 0.23, while the harness and Codex remain at 0.98.Source: §7.5 · p. 16

• The suite covers one incident, so how far the findings carry to other cases remains to be tested.Source: §8 · p. 17
• The incident is public and predates the evaluated models, which may have encountered it during pre- or post-training.Source: §8 · p. 17
• Tracing rules halt when records leave a continuation unresolved, so edge recall covers only a conservative subset of incident flows.Source: §8 · p. 17
• The comparisons vary harness, model, and query policy together, while recovery with agents’ unused budget remains unmeasured.Source: §8 · p. 17

• ABF-Bench turns a real laundering incident into well-posed cross-chain tasks, and the ABF harness outperforms general-purpose agents, including Codex with the same model.Source: §9 · p. 17
• Agents acquire evidence faster than fixed-order tracers and recognize custody they miss, yet stop early and finish below breadth-first tracing.Source: §9 · p. 17
• Ledger-checkable operation verdicts and real-adversary difficulty make ABF-Bench a candidate environment for training agents with verifiable rewards.Source: §9 · pp. 17–18