PaperlayerView exact revision on arXiv

Source-linked AI summary

TreasuryQA Bench: Can AI Agents Work an Archive of Scanned Financial Records?

Ritiz Tambi, Sahar Shayegan, Joshua Meyer, Sun Kim, Ali Khan, Victor Shih, Mehdi Jamei, Andi Partovi

arXiv:2610.38924v1cs.AI

TL;DR

Existing benchmarks do not combine finding documents in an archive with reading scanned pages, leaving this production-like workflow underrepresented. TreasuryQA Bench evaluates agents on scanned Treasury records reached through Dropbox MCP and read with a hosted OCR tool. Across ten agents, results remain far from saturated, with substantial variation in solve rates.

  • Problem

    Existing benchmarks simplify the workflow by supplying documents or pre-extracted text rather than requiring agents to find and read scanned records.

    Source: §1 · p. 1

  • Method

    TreasuryQA Bench tests 70 numeric questions over 1,435 scanned Treasury PDFs, using Dropbox MCP for retrieval and a hosted reader for pages, with programmatic answer keys and range-based grading.

    Source: §1 · p. 1, §3.3 · p. 4, §3.5 · p. 5

  • Results

    Across ten agents, performance remains far from saturated: Claude Opus 5.5 in Claude Code solves 64.8% of attempts, while GPT-5.6 Terra in Codex solves 16.7%.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Most failures are wrong numbers, often from a digit misread, and every agent solves fewer tasks on the blurriest scans.

    Source: §1 · p. 2, §8 · p. 9

    Scores reflect a model in one harness using Reducto, whose version is unpinned; another harness or reader could change scores and rankings.

    Source: §7 · p. 9

Abstract

from arXiv · show

Organisations keep decades of financial records as scanned PDFs on shared drives, and AI agents can now reach those drives through Model Context Protocol (MCP) servers. Existing benchmarks do not mirror this production setting because they simplify the task. In some, the relevant documents are handed to the agent, so it never has to search. In others, the agent searches through documents with pre-extracted text, so it never has to read scanned images. We introduce TreasuryQA Bench, which requires both: 70 questions over 1,435 scanned U.S. Treasury publications, held in Veris's high-fidelity Dropbox twin. In each task the agent searches for the relevant file through Dropbox's MCP server, reads the pages it needs with Reducto's hosted OCR server or its own tools, and replies with one exact figure. A Python grader checks each answer against an accepted range around the correct value. In this paper we present results from ten agents. The best, Claude Opus 5.5 in Claude Code, solves 64.8% of attempts and the worst, GPT-5.6 Terra in Codex, solves 16.7%, so the benchmark is far from saturated. We also test how scan quality affects performance: every agent solves fewer tasks on the blurriest scans. Most failed attempts reply with a wrong number rather than no number, and the most common cause is a digit misread from the scan. Three worked examples are included with their answer keys.

1 Introduction

Existing benchmarks omit parts of the scanned-records workflow, while TreasuryQA Bench combines archive search and scan reading through tools. Across ten agents, performance varies substantially and remains far from saturated.

1 Introduction

  • The benchmark tests the analyst-like sequence of finding a file, locating a table entry, and checking the figure against table totals.Agents choose how to search, which pages to read, and whether to trust returned text.Source: §1 · p. 1

2 Related Work

  • Prior benchmarks typically hand over documents, use pre-extracted text, or grade service state; OfficeQA Pro instead gives agents a local corpus copy.The comparison distinguishes scans, retrieval, and live tools as benchmark dimensions, but the supplied table caption does not provide cell values.Source: §1 · p. 1, Table 1 · p. 2

1 Introduction

  • TreasuryQA Bench asks agents to find files in 1,435 scanned Treasury PDFs and answer 70 numeric questions using Dropbox MCP and a hosted reader.The archive covers U.S. Treasury publications from 1793 to 2024; a Python grader checks answers against accepted ranges.Source: §1 · pp. 1–2
  • 64.8% of attempts are solved by Claude Opus 5.5 in Claude Code, versus 16.7% by GPT-5.6 Terra in Codex; every agent solves fewer tasks on the blurriest scans.The leading Opus result is not significantly higher than Gemini 4 Argon’s 60.0%.Source: §1 · p. 2

2 Related Work

  • Document QA often supplies page images, while collection QA and finance benchmarks commonly use machine-readable text; TreasuryQA requires both archive retrieval and scan reading.A 1% tolerance would accept every wrong number in the example task, including faint 8s read as 6s.Source: §2 · p. 2

3 TreasuryQA Bench

TreasuryQA Bench pairs a scanned archive and tool-mediated search with numeric tasks whose answer keys and grading are programmatically checked. Tasks are designed around figures that can be verified against the tables’ own totals.

3.1 Design and Task Formulation

  • Agents receive one question and must find and read the relevant scanned document through drive and hosted-reader tools, then return one numeric answer.A Python grader, not a model, checks whether the answer falls within the accepted range.Source: §3.1 · p. 3

3.2 Environment

  • The archive contains 1,435 image-only PDFs and 118,524 pages; scans are reached through a Dropbox twin, with selected pages read by Reducto.Some scans are as coarse as 60 ppi; the environment gives each attempt a fresh copy and no archive files or OCR engine locally.Source: §3.2 · p. 4

3.3 Task Construction

  • Of 70 tasks, 55 are new and 15 adapt OfficeQA Pro v2 questions; programmatic oracles compute keys and check them against table totals.Tasks specify period, units, and precision without naming the document or computation, and plausible wrong answers are kept outside the grading range.Source: §3.3 · p. 4

3.4 Quality Control

  • All 70 keys held when checked against scans; in seven tasks, the key follows table totals even though the printed figure disagrees.Authors also revised ambiguous questions and replaced three answer pages that were unreadable at 60 ppi in available copies.Source: §3.4 · p. 5

3.5 Grading and Metrics

  • The grader parses the final ANSWER line and accepts only values in a narrow task-specific range; an LLM judge passed 32 wrong answers.The paper reports pass@1, pass@3, and pass3, with task-level bootstrap intervals and paired comparisons using a Holm correction.Source: §3.5 · p. 5

4 Experimental Setup

The experiment evaluates ten model–harness pairs across repeated attempts on the same tasks.

4 Experimental Setup

  • Ten agents each attempt every task three times under fixed task text, instructions, and settings, for 2,100 attempts total.Each agent is a model in a pinned command-line harness; reported model spend excludes Reducto charges.Source: §4 · p. 5

5 Results

Opus and Argon top the overall ranking, but their difference is not significant; scan clarity, repeatability, and token costs distinguish performance across agents.

5 Results

  • 64.8% of Opus attempts and 60.0% of Argon attempts are solved, a difference that is not significant (Table 2).After correcting for multiple comparisons, no agent is significantly ahead of the one ranked just below it.Source: §5 · pp. 5–6, Table 2 · p. 6
  • Argon is right on 83% of answered attempts, but more than a quarter of its attempts time out; half of Opus’s solves take 5 minutes or less.Most failed attempts return a wrong number, most often because a digit is misread from the scan.Source: §5 · p. 6
  • On the 20 very blurry tasks, Opus solves 60%, Argon 50%, and every other agent 37% or less; Sol falls to 25% (Figure 3).Sol and Argon succeed on different tasks and together solve 62 tasks at least once, versus 57 for Opus alone.Source: §5 · p. 6, Figure 3 · p. 6
  • Agent–task results are usually consistent across three attempts, yet 72 pairs repeat the same wrong number every time.Fewer than half as many pairs have mixed results as chance would predict.Source: §5 · p. 6
  • GLM is cheapest per attempt, while GLM, DeepSeek, Sol, and Opus offer the most accuracy for their cost; Grok costs 14 times DeepSeek and solves fewer than half as many attempts.Cost tracks uncached input and its price: DeepSeek reads almost everything from a cheap cache, while Argon, Gemini 3.8 Flash, and Grok send many uncached tokens (Figure 4).Source: §5 · pp. 6–7, Figure 4 · p. 7

6 Analysis

Agents’ failures span tool-use differences, scan-reading errors, and choices about which figures to trust. Misread digits dominate failures, while agents often repeat one another’s wrong answers.

6.1 How Agents Work

  • Opus is usually right among its fast answers: 40 of 56 attempts; Sonnet, Terra, and Grok are right in at most 40% of theirs, while Argon never answers that quickly.Source: §6.1 · p. 7
  • Models differ widely within the same harness—by 23 points in Claude Code and 35 in Codex—and also choose different reading strategies.Opus reuses uploads in 90% of reads, compared with Sonnet’s 38%; neither Codex agent views page images.Source: §6.1 · p. 7
  • Attempts viewing page images are solved more often (42% versus 29%), but the gap is no longer significant among attempts lasting at least five minutes.Gemini 3.8 Flash still misreads more than half of its answers on very blurry tasks despite viewing page images in nearly all such attempts.Source: §6.1 · p. 7
  • On seven tasks with an incorrect printed figure, agents solve 24% of attempts versus 41% elsewhere, and most failures report the printed figure despite contradictory totals.In 30 failed replies, the agent’s own check pointed to the correct answer.Source: §6.1 · p. 7
  • Agents typically find volumes by filename within a minute, but their Reducto handoff strategies differ; GLM has the most errors and fewest successful first reads.Some agents upload and reuse volumes, while others often pass download links and encounter errors.Source: §6.1 · p. 7, Table 3 · p. 8

6.2 Failed Attempts

  • 36% of failures stem from misread digits, the leading cause for seven of ten agents; the rate rises from 7% on clear scans to 56% on very blurry ones.An independent re-check agreed with the assigned failure cause in 93% of a sample.Source: Figure 5 · p. 8, §6.2 · p. 8
  • Most wrong answers are shared: 62% are also given by another agent, and majority voting would solve at most 50 tasks versus 68 reached by at least one attempt.Of 38 wrong answers given by four or more agents, 29 were not among the known wrong answers.Source: §6.2 · p. 8

6.3 Integrity

  • Thirteen passing attempts used figures from outside sources, which the authors count as failures; restricting reachable sites would prevent them.Four Grok solves were counted although their figures appeared in no tool output, because grading checks only the number.Source: §6.3 · p. 9

7 Limitations

The benchmark’s tasks, grading, and exposure create boundaries on how its scores should be interpreted.

Tasks and keys

  • Tasks are not a random sample: each uses a table cell confirmed by totals, all come from one English-language archive, and each answer is a single number.Keys were checked mostly by model agents, including the top-scoring evaluated model, rather than domain experts.Source: §7 · p. 9

Grading and setup

  • The grader checks only the number: an answer within $1.00 can pass despite a misread cents digit, and abstention scores the same as a wrong figure.Scores use one model, harness, and hosted reader; another harness or reader could change scores and their order.Source: §7 · p. 9

Size and exposure

  • With 70 tasks, pass@1 intervals are wide at 14 to 21 points; the authors ran no human baseline.The PDFs and OfficeQA Pro v2 questions are available on request, and frontier models may have memorized historical financial statistics.Source: §7 · p. 9

8 Conclusion

TreasuryQA Bench tests agents finding, reading, and reporting figures from scanned Treasury records through drive and OCR tools. The conclusion reports substantial differences in agent performance and recurring scan-reading errors.

8 Conclusion

  • The two leading agents, Claude Opus 5.5 and Gemini 4 Argon, solve 64.8% and 60.0% of attempts, respectively; most failures are wrong numbers, often one digit off.Argon often runs out of time, while Opus answers quickly and misreads more; agents tend to report printed figures even when their own checks contradict them.Source: §8 · p. 9

Disclosure

  • The authors work at Veris AI, which builds the Dropbox twin used in the benchmark; TreasuryQA Bench is released with Reducto, whose reader all agents used.Source: Disclosure · p. 9

Reproducibility Statement

  • Each attempt used a fixed agent configuration on a fresh twin copy, but Reducto’s hosted service reports no release version.The paper says grading all stored replies again reproduces all 2,100 verdicts and provides the instructions and three worked examples with keys.Source: Reproducibility Statement · pp. 9–10

LLM Usage Statement

  • LLM agents drafted tasks and answer-key programs, checked keys on scans, and labelled failure causes; the authors reviewed every task, finding, and edit.Scripts computed every reported number from evaluation records.Source: LLM Usage Statement · p. 10

A Instructions

The appendix gives the shared agent instructions and two public worked examples that show how questions map to source pages and answer keys.

A Instructions

  • The shared instructions are verbatim for all ten agents, and their first two sentences in the second paragraph come from OfficeQA Pro.The instructions comprise 1,402 characters.Source: A Instructions · p. 11
  • Agents are told to locate pages in scanned PDFs—some volumes reach 1,000 pages—and read small page ranges with Reducto.The instructions give filename examples and recommend using contents or index pages to find the relevant pages.Source: A Instructions · pp. 11–12
  • When multiple documents report a metric, agents should use the newest revision unless the question specifies a date, calculate at maximum precision, and end with an ANSWER line.Source: A Instructions · p. 12

B Worked Examples

  • Besides the Figure 2 task, two additional tasks are public worked examples.Source: B Worked Examples · p. 12
  • The 1923–1924 Foreign Intercourse example asks for the change in fiscal-year appropriations, excluding no-year, prior-year, and carried-forward amounts.It identifies the combined statements and printed page ranges used.Source: B Worked Examples · p. 12
  • The appropriations example’s key is −$26,840.92 ± $1.00; 15 of 30 attempts solved it, while seven gave 63,159.08 after a $90,000-short 1923 sum.Source: B Worked Examples · p. 12
  • The Census example asks for the percentage change in net outlays from fiscal 1979 to 1980, using the account’s net-outlays subtotal.The stated source is the combined statements for 1979 and 1980.Source: B Worked Examples · p. 12
  • The Census example’s key is 9.11%; 17 of 30 attempts solved it, while eight answered 9.13 using the printed 1979 subtotal.The solution instead uses the subtotal supported by the account rows and appropriation less withdrawals.Source: B Worked Examples · p. 12

C Construction and Answer Keys

The benchmark uses audited oracle answer keys and groups tasks by the operation each question requires. Pass@1 varies across those task buckets.

C Construction and Answer Keys

  • An audit of seven OfficeQA Pro questions found one key unreachable in parsed text, prompting oracle-first construction of 70 keys with 11,339 lines and 1,078 assertions.After the run, eleven dollar ranges were narrowed without changing any verdict.Source: C Construction and Answer Keys · p. 12
  • 25.6% is the lowest bucket Pass@1 for differences between two items in one year, while 63.3% is the highest for percentage change or excess.Pass@1 is measured over all attempts by ten agents; Table 4 also reports each bucket’s task shares and common wrong-answer traps.Source: C Construction and Answer Keys · p. 12, Table 4 · p. 13
  • Each oracle computes dollar sums in exact cents at full precision, rounds once to the requested precision, and recomputes lookups from table totals.When a series is reprinted, every printing among the sources is checked; a figure inconsistent with its table’s totals is excluded.Source: C Construction and Answer Keys · p. 13
  • A second accepted range is used only for a second reasonable reading of the question.The key procedure does not accept figures that fail their own table’s printed totals.Source: C Construction and Answer Keys · p. 13

D Environment

The environment combines a large scanned Treasury archive with candidate-based task construction and specified agent tooling. A grader extracts one answer from each reply using explicit parsing rules.

D Environment

  • Table 5 identifies each agent by model, provider, pinned command-line harness version, and reasoning effort; Codex runs with its sandbox off inside the container.Source: Table 5 · p. 13
  • The archive contains 1,435 files and 118,524 pages, including 1,321 Combined Statements from 1872–2024 and 114 earlier reports from 1793–1893.Scans range from 60–80 to 150 ppi; 582 files exceed the 5 MB get_file_content limit, and download links last 1–15 minutes and work once.Source: D Environment · p. 13
  • Each attempt runs in an unprivileged Debian 12 container with 2 CPUs and 4 GiB, while the harness stops the CLI one minute before the 30-minute limit.Reducto uses its hosted r-1 model; seven tools were available, and two tool descriptions led to refused calls because of server constraints.Source: D Environment · pp. 13–14
  • Candidate search scanned 206,820 parsed tables, confirmed 201 cells against table totals, and tied 36 of 90 OfficeQA Pro v2 questions to a cell.Authoring agents received 110 candidates and dropped 40; tasks draw on 55 volumes with a median of 594 pages.Source: D Environment · p. 13
  • Bootstrap intervals use 4,000 task-level percentile resamples that keep each task’s three attempts together; Holm correction covers nine neighboring-rank comparisons.Source: D Environment · p. 14

E Grader

  • The grader takes the first eligible number on the last ANSWER or Final answer line, or the last amount if no such line exists.It strips markdown and LaTeX wrappers, interprets specified negative-number formats, does not rescale unit words, and recognizes abstention wording before a number.Source: E Grader · p. 14

F Failure Causes

The paper classifies failed attempts by their primary cause, reports agent time and cost accounting, and assesses benchmark validity against a checklist. The checklist records both supporting evidence and limitations.

F Failure Causes

  • Failures are assigned one primary cause: digit misread, kept figure, wrong source, wrong row, wrong method, ungrounded answer, gave up, time limit, or earlier ending.The categories distinguish errors in reading, sourcing, locating, computing, grounding, and committing an answer.Source: F Failure Causes · p. 14

G Cost and Effort

  • Table 6 reports each agent’s total time, tokens per attempt, and mean cost per attempt, with cost calculated at agent-side list prices.Regional uplifts and long-context surcharges are excluded; one Kimi attempt with no recorded tokens or cost counts as zero in totals and means.Source: G Cost and Effort · p. 14, Table 6 · p. 15

H Agentic Benchmark Checklist

  • The checklist marks setup frozen over time as No because Reducto was live and its tool list changed during the run.Tool versions and API-tool availability are Partly; verified oracle keys and solvability are Yes, with 68 tasks solved.Source: H Agentic Benchmark Checklist · p. 15
  • The checklist reports no human baseline, while a trivial agent that gives no number or a known wrong answer fails.The checklist also marks open data and harness as Partly, citing instructions and three worked examples in the paper.Source: H Agentic Benchmark Checklist · p. 15
Loading 2610.38924v1…