PaperlayerView exact revision on arXiv

Source-linked AI summary

Crypto Accounting Bench: Evaluating Frontier and Open-Weight Models on Crypto-Asset Accounting Tasks

Kareem Khattab, Omar Khattab, Mohamed Ibrahem

arXiv:2609.14811v1cs.AIcs.CL

TL;DR

Crypto-asset accounting requires models to combine transaction evidence with organization-specific context to reproduce the entry actually posted. CAB evaluates this task using pseudonymized cases and withheld posted-entry targets. Results show stronger numeric agreement than account-selection accuracy, with account selection and complete-entry composition remaining the main challenges.

  • Problem

    Existing finance benchmarks cover knowledge, auditing, synthetic workflows, or broad deliverables, while reconstructing an organization’s complete posted entry from transaction evidence remains a distinct checkable task.

    Source: §1 · p. 1, §3.1 · p. 2

  • Method

    CAB pairs source evidence with validated, pseudonymized tasks and evaluates model reconstructions against the organization’s recorded, structured journal entry.

    Source: §3.1 · p. 2, §4.1 · p. 2

  • Results

    Across 4,248 trajectories, the best Mean Score was 77.43% and the best Pass@3 was 56.78%; errors mainly involved account selection and complete-entry composition.

    Source: §10 · p. 5

  • Takeaways & Limitations

    Across evaluated model families, progress will require better evidence synthesis, organization-specific account selection, tax-lot reasoning, and verification that every required line is present.

    Source: §10 · p. 5

    The posted-entry target is not an independent audit, and automated grading remains a separate source of uncertainty.

    Source: §9 · p. 5

Abstract

from arXiv · show

We introduce Crypto Accounting Bench (CAB), a benchmark for assessing whether frontier and open-weight language models can reconstruct the complete journal entry that an organization actually posted for a crypto-asset transaction. CAB contains 118 evaluation tasks drawn from 7 pseudonymized organizations. Each task combines transaction mechanics, asset quantities and base-currency values, wallet and legal-entity context, counterparty evidence, related transaction legs, recurrence, tax-lot evidence, and the organization's complete chart of accounts. The target is a balanced structured entry with every required account, side, amount, currency, and full-precision asset quantity. We evaluate 12 models spanning proprietary frontier systems and open-weight releases over 3 independent attempts per task, producing 4,248 trajectories. We report 3 metrics: Mean Score, Best@3, and Pass@3. Pass@3 is the fraction of tasks with at least 1 of 3 attempts that satisfies every rubric criterion and required gate. The leading model reaches 77.43% Mean Score, while the best Pass@3 is 56.78%. Deterministic diagnostics, read from each task's best of 3 attempts and macro-averaged across the 12 models, show higher base-amount agreement (97.8%) than deciding-account accuracy (56.3%). Together with the failure analysis, these results identify account selection and complete-entry composition as the main remaining challenges on CAB.

1 Introduction

CAB asks whether models can reconstruct the complete, balanced journal entry an organization actually posted from transaction evidence. Its results point to account selection and complete-entry composition as the main challenges.

1 Introduction

  • CAB evaluates full posted-entry reconstruction from transaction evidence, where the recorded treatment varies with organization-specific context and accounting conventions.The entry may depend on wallet structures, legal entities, counterparties, account charts, related transaction legs, recurrence, and tax-lot evidence.Source: §1 · p. 1

Contributions

  • The benchmark contributes real crypto-asset accounting evidence and a withheld posted-entry answer key paired with pseudonymized tasks.Its evaluation combines frozen task-local rubrics, LLM-assisted grading, and independent deterministic diagnostics.Source: Contributions · p. 1
  • 77.43% Mean Score for Gemini 3.7 Flash contrasts with the paper’s conclusion that account selection and complete-entry composition remain the main challenges.The paper frames the evaluation question as reproducing an organization’s posted entry, not producing a narrative recommendation.Source: Contributions · p. 1, §1 · p. 1

Evaluation question

  • The evaluation asks whether frontier models can turn heterogeneous transaction evidence into the exact balanced entry used by the organization without seeing the answer key.Source: Evaluation question · p. 1

2 Related Work

CAB is positioned among benchmarks that assess financial knowledge, workflows, and accounting tasks. It differs by testing a single crypto-asset transaction against the complete entry recorded by a real organization.

2 Related Work

  • GDPval [2] grades professional outputs across occupations and sectors; CAB instead checks a structured accounting artifact against a real-ledger entry.Source: §2 · p. 1
  • FinMaster [3], AuditBench [4], and Finch [5] assess broad finance or accounting workflows, while CAB measures one transaction-level accounting decision.Source: §2 · p. 1
  • Closest accounting benchmarks use generated scenarios or synthetic company worlds, whereas CAB targets entries posted by 7 real organizations for individual crypto-asset transactions.The model selects from the organization’s chart of accounts using wallet, legal-entity, counterparty, chain, related-leg, recurrence, and tax-lot evidence.Source: §2 · p. 1

Positioning

  • CAB’s target is descriptive and organization-specific: it tests combining supplied evidence, choosing the organization’s accounts, preserving numeric precision, and meeting double-entry constraints.Source: Positioning · p. 1

3 Benchmark Design

CAB turns recorded entries and their source evidence into model-visible tasks through a one-way pipeline. Each task is validated and provides the transaction and accounting context needed to infer the recorded entry.

3 Benchmark Design

  • CAB is built as a one-way pipeline: recorded entries and source evidence go in, and pseudonymized model-visible tasks come out.Source: §3 · p. 2, Figure 2 · p. 2

3.1 Task construction

  • Each task joins a posted journal entry to its source transaction and accounting context, then projects factual evidence into a model-visible schema.Construction requirements include a balanced non-degenerate entry, complete chart of accounts, reproducible monetary amounts, reachable account choices, and visible evidence for every non-zero reference line.Source: §3.1 · p. 2

3.2 Information available to the model

  • Models receive transaction mechanics and values alongside wallet, legal-entity, counterparty, contract, related-leg, recurrence, tax-lot, and complete chart-of-accounts evidence.The reference entry, correct-account labels, private source identifiers, and authored solution notes are excluded from the inputs.Source: §3.2 · p. 2

4 Dataset

CAB contains 118 tasks from 7 pseudonymized organizations, with structured targets based on the entries those organizations recorded. The dataset spans varied accounting situations, while the large account search space and context-dependent judgments make reconstruction challenging.

4 Dataset

  • The evaluation set contains 118 tasks from 7 pseudonymized organizations, spanning 21 assets, 10 chains plus an exchange-only context, 74 deciding accounts, 2 base currencies, and 8 source-system labels.Source: §4 · p. 2

4.1 Ground truth

  • Each target contains all posted lines, account names, Debit/Credit sides, base-currency amounts, currency, and full-precision asset quantity.The set has 246 non-zero reference lines; all reference entries balance at accounting precision, and graded amounts have visible evidence or follow from double-entry closure.Source: §4.1 · p. 2

4.2 Accounting situations covered

  • The recorded entries cover transfers, income and expense, intercompany movements, swaps, fees, and tax-lot disposals with realized gains or losses.Source: §4.2 · p. 2
  • Exchange-sourced tasks include trading gains and losses, funding income, fees, and a loan drawdown; account choice depends on exchange, strategy, and flow direction.The evidence covers cash and token movements rather than underlying positions, so CAB does not test derivative position accounting.Source: §4.2 · p. 2

Why reconstruction is hard

  • A task supplies a median of 569 chart accounts, while the 118 entries use only 141 distinct accounts.Reconstruction can require wallet role, legal entities, counterparty relationships, exchange strategy, token metadata, and prior postings; 22 of 64 tasks with other-side entity evidence are intercompany tasks.Source: Why reconstruction is hard · p. 2

4.3 Task anatomy

  • The model must infer one complete entry from combined evidence and return valid JSON without inventing accounts.The illustrated entry is explicitly simplified, with illustrative names and values.Source: §4.3 · p. 2

4.4 Privacy

  • The public Hugging Face release is a transformed derivative, not the exact input used in evaluation; source-ledger provenance, real identities, and transformation mappings are not released.The transformations replace identifying names, account codes, blockchain identifiers, and asset and chain labels; they also rescale values and quantities and shift dates.Source: §4.4 · p. 2

5 Experimental Setup

CAB evaluates 12 models on repeated, isolated task attempts and scores their entries through frozen rubrics, criterion grading, and independent structural checks. Its metrics distinguish average partial credit, best-attempt performance, and complete rubric-and-gate passes.

5.1 Models and harness

  • 12 models each attempted every task three times, producing 4,248 trajectories under shared output and tool constraints.Attempts ran in fresh temporary working directories; each pipeline used its highest exposed reasoning setting.Source: §5.1 · pp. 2–3

5.2 LLM-assisted rubric evaluation

  • Four scoring layers combine frozen task-specific criteria, LLM-judge verdicts, local weighted-score summation, and independent deterministic diagnostics.Table 1 gives weights for transfer, fee, and income/expense rubrics covering 76 tasks; other families use separate frozen six-criterion rubrics.Source: §5.2 · p. 3, Table 1 · p. 3

5.3 Metrics

  • Mean Score averages all attempts, Best@3 averages each task’s strongest rubric score, and Pass@3 counts tasks with a perfect rubric score and all required gates on one attempt.Pass@3 is not an entry-equality metric; normalized complete-entry equality is evaluated separately.Source: §5.3 · p. 3

Critical accounting gates

  • Seven required gates add structural and material-correctness requirements beyond the rubric’s accounting criteria.An otherwise correct entry can fail for combining recorded postings or using an account name absent from the reference.Source: Critical accounting gates · p. 3

5.4 Run identity

  • Stored artifact identities and hashes let each scored attempt and aggregate be matched to its exact evaluation inputs and outputs.The shared grading contract covers all 4,248 trajectories.Source: §5.4 · p. 3

6 Main Results

The leaderboard separates average rubric performance from best-attempt and full-pass performance. Across models, partial credit is much more common than a fully passing attempt, and cost shows little association with Mean Score in this evaluation.

6 Main Results

  • Gemini 3.7 Flash leads Mean Score at 77.43%, while Kimi K3 leads Best@3 at 82.37% and Pass@3 at 56.78%.Table 2 ranks models by Mean Score across 118 tasks and three attempts per model.Source: Table 2 · p. 3, §6 · p. 3

Interpretation

  • The top nine models span just 2.10 percentage points in Mean Score, but their full-rubric success rates diverge.The paper reports no confidence intervals and cautions that ordering within this Mean Score band is not settled.Source: Interpretation · p. 3

6.1 Full-rubric success across models

6.2 Rubric quality against completion

  • Across 12 models, Pass@3 correlates with Mean Score at r = 0.74 and Best@3 at r = 0.82; only six reach the above-median quadrant.The best Pass@3 among those six models is 56.78%.Source: §6.2 · p. 4, Figure 4 · p. 4

6.3 Score against inference cost

  • Inference cost has little association with Mean Score (Pearson r = 0.26, n = 12), across $0.019–$0.454 per attempt and 5.51 Mean Score points.Gemini 3.7 Flash scores 77.43% at 52% of Fable 5’s $0.454 per-attempt cost.Source: §6.3 · p. 4, Figure 5 · p. 4

7 Capability Analysis

Best-of-three diagnostics show that models generally recover transaction quantities, direction, and base amounts more reliably than the deciding account. Scores also vary by rubric family, with fee and swap tasks highest and income/expense and intercompany tasks lowest.

7.1 Where the errors concentrate

  • Asset quantity accuracy reaches 100.0%, accounting direction 98.1%, and base-amount agreement 97.8%, versus 56.3% deciding-account accuracy.Diagnostics use each model’s rubric-score-selected attempt per task, then average across tasks and equally across 12 models; the gap is 41.5 points.Source: §7.1 · p. 4, Figure 6 · p. 4

7.2 Rubric families

  • Fee and swap tasks have the highest macro Mean Scores at 87.5% and 84.5%, while income/expense and intercompany tasks score 67.5% and 68.1%.The six rubric families partition all 118 tasks; the realized gain/loss family scores 76.5% on five tasks, so its mean warrants caution.Source: §7.2 · p. 4, Figure 7 · p. 4

8 Failure Analysis

Failures concentrate in account choice and complete-entry composition rather than basic format compliance or numerical accuracy. The most common signals often reflect a substituted account that leaves a reference line unmatched.

8 Failure Analysis

  • Incomplete entries and misposted accounts each appear 2,328 times, while incorrect amounts occur only 198 times across 4,248 trajectories.Debit/Credit failures appear 2,243 times, unsupported postings 2,214 times, and wrong deciding accounts 1,997 times.Source: §8 · p. 4, Figure 8 · p. 5

8.1 Unmatched postings

  • On 2,141 trajectories with the same number of lines as the reference, a wrong account substitutes for the expected account and leaves its reference line unmatched.This explains why misposted-account and incomplete-entry signals have equal totals.Source: §8.1 · p. 5

8.2 Correct family, wrong account

  • Wrong deciding accounts occur 1,997 times, versus 1,073 incorrect treatments; generic accounts can earn treatment-level partial credit while missing the organization-specific account.The relevant evidence can include wallet names, entity relations, contract metadata, and historical recurrence.Source: §8.2 · p. 5

8.3 Intercompany direction

  • Intercompany account choice requires comparing legal entities before using transaction direction to distinguish a payable from a receivable.Movement direction alone does not establish the intercompany relationship.Source: §8.3 · p. 5

8.4 Format compliance

  • Invalid responses, quantity failures, and currency failures are rare; dominant errors arise within otherwise valid, generally balanced candidates.In this evaluation set, models are separated by accounting judgment rather than format compliance.Source: §8.4 · p. 5

Key finding

  • Models usually move the correct value in the correct direction but do not reliably identify every organization-specific account or compose the complete journal entry.Source: Key finding · p. 5

9 Limitations and Responsible Use

CAB measures agreement with organizations’ posted crypto-asset entries, not whether those entries are normatively correct. Its scope is limited to one operation and the described evaluation population, and grading retains judgment uncertainty.

9 Limitations and Responsible Use

  • CAB evaluates only reconstruction of a posted crypto-asset journal entry, not month-end close, tax preparation, audit, consolidation, or external reporting.Results characterize the CAB evaluation population and do not establish that the recorded entry is the only acceptable accounting policy.Source: §9 · p. 5
  • LLM-assisted criterion grading retains judgment uncertainty; deterministic checks and fixed weights do not independently validate the judge’s accounting decisions.The rubric criteria are authored from accounting practice, while the reference entry remains the organization’s posted entry rather than an independently audited one.Source: §9 · p. 5

Responsible deployment

  • A benchmark pass does not authorize autonomous production posting; deployment should retain human review, access controls, provenance, reconciliation, and rollback.Source: Responsible deployment · p. 5

10 Conclusion

CAB evaluates whether models can reconstruct complete, organization-specific entries from crypto-asset evidence. Results show strong amount agreement but persistent account-selection and entry-composition errors, alongside a public educational dataset and evaluation toolkit.

10 Conclusion

  • CAB’s conclusion is a benchmark of complete, organization-specific journal-entry reconstruction from real-world crypto-asset transaction evidence.Source: §10 · p. 5
  • 77.43% is the best Mean Score and 56.78% the best Pass@3 across 4,248 trajectories; quantity accuracy is 100.0% and base-amount agreement 97.8%.Remaining errors concern accounting judgment, especially choosing the deciding account and composing every required line.Source: §10 · p. 5
  • Every evaluated model family has Pass@3 between 45.76% and 56.78%, a pattern consistent with a shared limitation rather than one pipeline’s weakness.The paper identifies evidence synthesis, organization-specific account selection, tax-lot reasoning, and checking that every required line is present as areas for progress.Source: §10 · p. 5

Data and evaluation artifacts

  • The public educational dataset contains transformed counterparts of the same 118 cases used in the evaluation, while reference answers are excluded from model inputs.The open-source toolkit includes the runner, model pipeline templates, deterministic scorer, validation suite, report generator, and leaderboard generator; private source-ledger provenance and organization identities are not released.Source: Data and evaluation artifacts · p. 5

Appendix A Evaluation Configuration

The appendix specifies shared evaluation settings, gate behavior and calibration, entry normalization, task composition, and the semantics of overlapping failure signals.

Appendix A Evaluation Configuration

  • All 12 pipelines shared a 32K-token output maximum, a repeatedly callable bash runner, identical task prompts, and three independent attempts per task.The workspace was isolated but was not an operating-system security sandbox; the same GPT-5.5 high-reasoning judge configuration served all reported evaluations.Source: Appendix A · pp. 5–6, Failure-label semantics · p. 6

Gate definitions and calibration

  • Six of seven required gates are deterministic; setting every amount to 999.99 preserves balance but triggers the material gate, while gates fired on no correct reference entry.Of 1,917 perfect-rubric attempts, 92 still failed a required gate; the material gate is derived from frozen criterion verdicts.Source: Gate definitions and calibration · p. 6

Entry normalization

  • Complete-entry equality requires every reference line to pair with one predicted line, with no missing or extra lines; it is an internal diagnostic, not a reported metric.Comparison uses account identity, Debit/Credit side, base-currency amount, and currency after zero-amount lines are dropped.Source: Entry normalization · p. 6

Appendix B Additional Benchmark Statistics

  • The 118 tasks include 55 inflows and 63 outflows, with 54 external, 22 intercompany, and 42 same-entity derived relationships.Tax-lot evidence appears in 38 tasks, related same-hash records in 60, and prior-receipt history for recurring patterns across all 118.Source: Appendix B · p. 6

Failure-label semantics

  • Failure signals are non-exclusive: every wrong deciding account is also a misposted account, and both signals share the same 2,328 trajectories.Counts indicate diagnostic incidence, not a mutually exclusive causal taxonomy.Source: Failure-label semantics · p. 6
Loading 2609.14811v1…