PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem

Divya Godara, Sachin Gupta

arXiv:2609.39013v1cs.CLcs.IR

TL;DR

DOCSEM requires systems to return both a numeric answer and the printed identifier of its supporting passage, so a correct calculation from the wrong block fails the joint requirement. EVICALC reads PDF text into blocks, selects a passage, has a language model produce an arithmetic expression, and evaluates that expression in local code. EVICALC obtained 8.61% official test joint accuracy; descriptive analyses document cases but do not establish a single cause of the test result.

  • Problem

    DOCSEM requires systems to return both a numeric answer and the printed identifier of its supporting passage, so a correct calculation from the wrong block fails the joint requirement.

    Source: §1 · p. 1

  • Method

    EVICALC reads PDF text into blocks, selects a passage, has a language model produce an arithmetic expression, and evaluates that expression in local code.

    Source: §3.1 · p. 2

  • Results

    EVICALC obtained 8.61% official test joint accuracy; descriptive analyses document cases but do not establish a single cause of the test result.

    Source: §8 · p. 4

  • Takeaways & Limitations

    The descriptive findings motivate testing extraction and selection changes, but do not demonstrate that proposed changes improve performance.

    Source: §7 · p. 4, §8 · p. 4

    This report concerns one system and task, and differing models, prompts, readers, and metrics prevent attributing the score difference solely to document layout.

    Source: Limitations · p. 4

Abstract

from arXiv · show

EVICALC, our system for the DocSem shared task, achieved 8.61% joint accuracy on 1,730 tasks in the official final test evaluation. It reads a PDF, selects a passage, asks a language model to write an arithmetic expression, and evaluates that expression in local code. Saved intermediate results support inspection of failures. A separate public-validation run achieved 92.17% answer accuracy and 1.00 evidence F1. The configurations and metrics differ, so these scores are not a controlled comparison. Our manual, post-hoc analysis is descriptive: in one inspected case, optical character recognition (OCR) and block grouping merged the relevant passage into another block, and the system answered from unrelated text. An exploratory study of reading page images on 100 documents returned evidence identifiers for only 22 documents. These descriptive findings motivate further evaluation; they do not establish the causes of the overall score.

1 Introduction

DOCSEM requires a numeric answer and the identifier of its supporting passage, so correct arithmetic from the wrong block fails. This paper reports how EVICALC behaved on that task.

  • Each DOCSEM task provides an image-only PDF and paraphrased query, and requires both a numeric answer and the printed identifier of the supporting passage.Source: §1 · p. 1
  • 8.61% joint accuracy was achieved in the official test evaluation; saved passages, calculations, and answers support case-level inspection, not estimates of each error type’s contribution.Source: §1 · p. 1
  • DocVQA introduced document-image question answering (Mathew et al., 2021), while program-aided models delegate calculation to interpreters (Gao et al., 2023); EVICALC uses a restricted arithmetic expression.Source: §1 · p. 1

2 Task and evaluation setting

The task uses PDF-based queries and requires answer-and-evidence outputs; the evaluation reports different measures for validation and test.

  • The participant release has 908 labeled training tasks and 217 validation tasks, while the test release has 1,730 tasks; systems must answer from the supplied PDF and return visible evidence identifiers.Source: §2 · p. 1
  • The test service reports joint accuracy, counting missing predictions as wrong, whereas public validation reports answer accuracy and evidence scores.Source: §2 · p. 1
  • Development used public training labels and validation feedback, but test predictions did not use organizer-held answers, source solutions, or task-indexed answer tables.Source: §2 · p. 1

3 System design

EVICALC reads PDF pages, selects evidence, generates an arithmetic expression, and evaluates it locally; its components and settings varied across experiments.

3 System design

  • The submitted pipeline has replaceable reader, selector, model-adapter, and calculator components; its answer-revision passes leave evidence IDs unchanged, while dashed links indicate saved records or optional checks.Source: §3 · pp. 1–2, Figure 1 · p. 2

3.1 Document reading and evidence selection

  • Tesseract OCR extracts text and positions for local block grouping; the submitted test selector processes all pages, combines query and document-topic words, and can reread a selected block’s printed identifier from a crop.Source: §3.1 · p. 2

3.2 Interpretation and arithmetic

  • The model lists requested quantities, quoted facts, units, assumptions, and an expression; local Python checks and evaluates supported expressions with Decimal at 40-digit precision, but the model may choose wrong quantities or operations.Source: §3.2 · p. 2

3.3 Recorded configuration and reproducibility

  • Submitted v6 used gpt-4o-mini for uncertain selection and calculation, claude-sonnet-5 for identifier crops, and Tesseract on all pages; the structured fact list was not required.Source: §3.3 · p. 2
  • Two answer-only passes changed 162 of 242 and 96 of 139 answers; failed retries kept prior answers, and neither pass reconsidered passage selection.Source: §3.3 · pp. 2–3
  • Initial records include passages, rankings, model settings, replies, calculations, and answers; revision records reconstruct predictions but omit some crop transcriptions and are not fresh model runs.Source: §3.3 · p. 3

4 Official test result

EVICALC achieved 8.61% joint accuracy on the official test. Public-validation observations and post-hoc rule analyses were development findings, not controlled comparisons or new PDF-based evaluations.

  • 8.61% joint accuracy was reported for adaptive v6 on 1,730 official test tasks; its two revision passes changed 207 answers without changing evidence fields, and no separate pass score establishes improvement.Source: §4 · p. 3
  • The 92.17% validation run used claude-opus-5 with quoted facts, explicit assumptions, and a calculation; Table 1 combines PDF-based runs with a saved-output recomputation, under changing settings and corrected labels.Source: §4 · p. 3, Table 1 · p. 3
  • A rule for wording about values applying to each remaining item reached 98.62% when recomputing saved outputs; no new PDF-based run established that the revised rule would reproduce the score.Source: §4 · p. 3

5 Descriptive failure analysis

The descriptive analysis documents observed test-document difficulties and one case where OCR grouping and selection led reasoning to use unrelated evidence. Its case-level observations and exploratory study do not estimate how much each failure contributed to the official score.

5 Descriptive failure analysis

  • The manual, post-hoc analysis describes cases rather than estimating each failure mode’s causal contribution to the official score.Pages, OCR traces, and saved selections were compared after failures had been observed.Source: §5 · p. 3

5.1 Observed document difficulties

  • Inspection found broad query cues matching several passages, administrative notes echoing queries without requested calculations, and damaged OCR text alongside readable unrelated notes.The frequency of these patterns was not counted across the full test split.Source: §5.1 · p. 3
  • In task_010565, OCR merged the virus story into a “Purpose and reading note” block, leaving its identifier absent and outside the selector’s shortlist.Reasoning then solved an unrelated pottedflower passage, which the calculation stage could not correct because it never received the relevant evidence.Source: §5.1 · p. 3

5.2 Identifier reading and passage selection

  • The 20-crop pilot assessed identifier transcription, not whether the selected paragraph answered the query.Transcribing the wrong paragraph’s identifier still gives incorrect evidence.Source: §5.2 · p. 3
  • Table 2 covers an exploratory page-image study on 100 documents, with exact output counts and approximate local correctness judgments rather than organizer labels.The caption identifies the table’s scope but does not provide its cell values.Source: Table 2 · p. 4

6 Exploratory page-image study

The exploratory page-image study built a list of possible problems and matched query words and OCR topic cues to catalog entries. It returned evidence identifiers for 22 of 100 sampled documents, but did not measure answers or run across the full test split.

6 Exploratory page-image study

  • The page-image model listed possible word problems and their page numbers, opening words, topics, and identifiers; local code matched query words and OCR-derived topic cues to the list.Claude Haiku produced catalog entries, while Claude Sonnet reread cropped identifiers; selections still required matching catalog entries to OCR block positions.Source: §6 · p. 4
  • The 100-document sample excluded 181 documents used during design or earlier inspection, and judgments were local rather than organizer labels.The notes describe grading by people and assistants reading rendered pages and crops, with selection hidden during initial passage reading, but report no independent-grader agreement.Source: §6 · p. 4
  • Only 22 of 100 sampled documents returned an evidence identifier, despite local estimates exceeding thresholds; the study did not measure numerical answers or cover the full test split.There was no minimum output count in the thresholds; 45 documents had no selected passage, 18 had no usable crop location, and 15 had conflicting identifier readings.Source: §6 · p. 4
  • More permissive matching increased selections from 55 to 56 but admitted administrative notes in inspected cases.This reused saved catalogs without new model calls and does not establish a limit for other selectors.Source: §6 · p. 4

7 Lessons Learned

The paper recommends checking passage selection before changing calculation instructions and testing changes through controlled comparisons. It presents these as recommendations motivated by observations, not demonstrated remedies.

7 Lessons Learned

  • Check passage selection first, then inspect selected evidence before changing calculation instructions.The virus case motivates inspecting the evidence the system selected.Source: §7 · p. 4
  • Compare readers or selectors on the same documents, changing one component at a time with the solver fixed, to test whether a change improves performance.The observations motivate this evaluation approach but do not demonstrate remedies.Source: §7 · p. 4
  • Report selection, identifier, and answer accuracy separately, alongside coverage; a correctly read identifier can still belong to the wrong passage.The page-image study returned evidence for 22 of 100 documents, and correctness among outputs alone missed this limitation.Source: §7 · p. 4

8 Conclusion

EVICALC achieved 8.61% official test joint accuracy. Descriptive analyses document a missing-evidence case and low evidence-return coverage in a page-image prototype, but establish neither the cause of the score nor improved performance from proposed changes.

8 Conclusion

  • 8.61% official test joint accuracy was EVICALC’s reported result.The conclusion does not attribute the score to a single cause.Source: §8 · p. 4
  • The descriptive analyses include one case of missing relevant evidence and a page-image prototype that returned evidence for only 22 of 100 documents.These findings motivate testing extraction and selection changes.Source: §8 · p. 4
  • The analyses do not establish a single cause of the test result or demonstrate that proposed changes improve performance.The paper frames its findings as motivation for further testing.Source: §8 · p. 4

Limitations

The report covers one system and task, with differing validation and test setups that prevent attributing their score difference solely to document layout. Its analyses are limited by convenience sampling, approximate judgments, post-hoc validation analysis, and variation in hosted model calls.

Limitations

  • Different models, prompts, readers, and metrics across validation and test mean their score difference cannot be attributed solely to document layout.The report concerns one system and task.Source: Limitations · p. 4
  • The first 120 test documents were a convenience sample, and correctness judgments in the 100-document study are approximate.Raw annotation records would be needed to resolve remaining uncertain cases.Source: Limitations · p. 4
  • The validation-output rule was developed after error inspection, so its score is not a prospective test of a new PDF-based system.Source: Limitations · p. 4
  • Hosted model calls can vary between runs.Source: Limitations · p. 4

Ethical considerations

The system uses released PDFs without private answers or source-solution lookup, but a plausible calculation from unrelated text remains a practical risk. Saved intermediate outputs enable inspection but do not guarantee correctness.

Ethical considerations

Related papers

Loading 2609.39013v1…