PaperlayerView exact revision on arXiv

Source-linked AI summary

When Harness Beats Scale, and When Reading Beats Both

Ivan Bondarenko, Nikolay O. Nikitin

arXiv:2609.34366v1cs.CLcs.AIcs.IR

TL;DR

DocSem requires systems to compute answers from a specific quantitative passage and identify the exact layout blocks supporting them. The paper combines retrieval with executable Program-of-Thoughts and evaluates why the system performed differently on labeled data and raster test PDFs.

  • Problem

    DocSem requires a numeric answer grounded in one quantitative passage and the exact layout block identifiers that justify it.

    Source: §1 · p. 1

  • Method

    The system combines hybrid block retrieval with executable Program-of-Thoughts, structured output, self-consistency sampling, and graph-derived entity context.

    Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 3

  • Results

    PoT raised 7B joint accuracy by 0.282 but moved 72B by at most 0.005; test performance was 13.58% joint, rank 149 of 163.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Audit evaluation-input format before tuning architecture: gains on born-digital validation did not transfer to damaged raster test reading.

    Source: §1 · p. 2, §8 · p. 7

    The reading-failure analysis relies on portal scores and submission diagnostics rather than human re-annotation, and stronger document-parsing VLMs were not evaluated.

    Source: Limitations · p. 7

Abstract

from arXiv · show

We describe our system for DocSem, the document-grounded quantitative reasoning shared task at DocInsights 2026, and analyze why it succeeded on labeled data and failed on the test set. The pipeline pairs hybrid block retrieval with Program-of-Thoughts (PoT) generation executed in a sandboxed interpreter, self-consistency sampling, and entity enrichment from chunk-level knowledge graphs. On our held-out split, application architecture moved the metrics far more than model scale did: PoT added 0.282 joint accuracy to a compact 7B model but at most 0.005 to a 72B model, and a 27B model with the full harness matched the 72B (0.884 vs.\ 0.873) at roughly 2.7$\times$ fewer parameters and a quarter of the CO$_2$. We read this through a distinction between world knowledge, which scales steeply with parameters, and language knowledge, which scales gently, and show that structured-output training makes a compact model harness-ready rather than merely small. On the raster, watermarked test PDFs the same system collapsed to 13.58\% joint (rank 149 of 163); a controlled re-rendering of the validation set reproduces the OCR half of the collapse while bounding what the simulation misses. Auditing the physical nature of evaluation inputs precedes architecture, and the leaderboard's bimodality is consistent with reading quality, not reasoning, having separated the field.

1 Introduction

DocSem requires a system to retrieve the single quantitative passage supporting a paraphrased query, compute its answer, and identify the exact evidence blocks. The paper reports that architecture mattered more than scale on labeled data, but raster, watermarked test PDFs exposed a reading failure that reasoning improvements could not overcome.

1 Introduction

  • A 27B model with PoT matched the 72B direct-answering model at 0.884 vs. 0.873 joint, while PoT added 0.282 joint accuracy to 7B and at most 0.005 to 72B.The system entered the test phase at 0.901 joint accuracy on its held-out split and 0.853 on the validation portal, but scored 13.58% on test, rank 149 of 163.Source: §1 · p. 1
  • The authors distinguish steeply scaling world knowledge from more gently scaling language knowledge, which a harness can leave to a compact model.The 27B harness reached 0.64 joint on one GPU at 0.14 kgCO2e, versus eight GPUs and 0.53 kgCO2e for the 72B line.Source: §1 · pp. 1–2
  • On raster test PDFs with watermarks, the system scored 13.58% joint, and answers changed on 89.6% of tasks when OCR was replaced by page-level vision transcription.The reasoning stack was unchanged between the two transcription attempts; the leaderboard also showed a 22-team spike at 67.46% and a long reading-failure tail.Source: §1 · pp. 1–2
  • Structured-output training gave the compact model discipline in rigid formats, while PoT's effect depended on model lineage and taxed the base model's citation discipline.The comparison spans JSON generation and extraction as well as instruction following and arithmetic.Source: §1 · p. 2
  • Chunk-level graph context helped only when distilled to query-nearest entities, while fuller graph views hurt; the paper also reports a controlled degradation study of test-like inputs.The study reproduces the OCR half of the test collapse on re-rendered validation inputs and bounds what that simulation misses.Source: §1 · p. 2

2 Task and the Data Shift We Missed

DocSem pairs a PDF with a paraphrased quantitative query and requires both a numeric answer and exact identifiers for its supporting block. The paper's data audit found a physical-format shift in test PDFs: clean born-digital pages gave way to low-resolution raster scans with watermarks and running headers.

2 Task and the Data Shift We Missed

  • Each DocSem task requires a numeric answer and the exact identifier set for the block or blocks that justify it.All 908 training labels use a single gold evidence block and numeric answers, motivating one identifier by default and canonical decimal strings.Source: §2 · p. 2
  • Test PDFs were low-resolution raster scans with watermarks and running headers, unlike the clean born-digital training and validation pages.Tesseract's median service-word share was 0.091 on test pages versus 0.224 on validation; the authors say checking the pixels would have revealed the shift.Source: §2 · p. 2

3 System

The system transcribes raster test pages before block segmentation, retrieves relevant blocks with a hybrid search pipeline, and generates answers through executed programs. Chunk-level graph entities enrich the prompt, with graph context helping only in top-3 distilled form.

3.1 Document ingestion

  • Raster test pages are transcribed page-by-page by Qwen2.5-VL-7B at 2× zoom before block segmentation, while training and validation PDFs are parsed directly.A targeted Qwen3.8-27B repair pass re-read 38 damaged pages across 26 documents.Source: §3.1 · pp. 2–3

3.2 Retrieval

  • The gte dense-and-sparse retrieval pair uses reciprocal rank fusion, then reranks the fused top-8 to the 3 blocks shown to the generator.On the held-out split, gte reaches recall@1 of 0.901 and 1.000 after reranking, while bge-m3 reaches 0.348 and 0.807, respectively.Source: §3.2 · p. 3

3.3 Program-of-Thoughts generation

  • The generator writes Python using only displayed blocks; a sandboxed interpreter executes it, and self-consistency votes across five sampled programs’ answers.Structured output is enforced server-side, and identifiers snap to valid block tokens when transcription noise corrupts them.Source: §3.3 · p. 3

3.4 Graph context and final assembly

  • Entities from retrieved-passage mini-graphs enrich the generator prompt, and the final test submission merged votes from base PoT, key-entity, and multimodal-escalation runs.The multimodal escalation showed page images of non-unanimous tasks to Qwen3.8-27B.Source: §3.4 · pp. 3–4, Figure 2 · p. 4
  • Graph context helps only in top-3 distilled form, according to the main-line check-split comparison.The table caption contrasts this with fuller graph views.Source: Table 1 · p. 4

4 Results on Labeled Data

On labeled data, the 27B model with the full harness matched or passed the 72B rows, while Program-of-Thoughts (PoT) had sharply different effects on the two 7B models. Cross-task results associate structured-output training with the format discipline that PoT consumes, while showing trade-offs in other abilities.

4 Results on Labeled Data

  • 0.282 joint accuracy was added by PoT for Meno-Lite-0.1, while PoT lowered base Qwen2.5-7B-Instruct by 0.194.Meno-Lite rose from 0.359 to 0.641; the base model fell from 0.691 to 0.497 as its evidence exact match dropped to 0.646.Source: §4 · p. 4
  • Synthetic DocSem facts reduce the value of memorized world knowledge; the design routes arithmetic, passage search, and fact structuring to external tools.The model is left with linguistic demands including query comprehension, fact selection, and faithful program writing.Source: §4 · p. 4
  • On portal-scored validation, the 27B PoT configuration passed the 72B direct configuration, and graph enrichment added 0.005 at portal precision.Single-row differences on 181 tasks have roughly ±4.5-point 95% confidence intervals, so the authors emphasize ordering across model lines and samplers.Source: Table 2 · p. 5, §4 · p. 5
  • 0.884 joint for the 27B full-harness model matched the 72B model’s 0.873, at roughly 2.7× fewer parameters.The 27B model reached 0.901 with key entities; the 72B model scored 0.873 with direct answering and PoT at identical sampling.Source: §4 · pp. 4–5

4.1 Structured-Output Training as a Harness Prerequisite

  • Across IFEval, GSM8K, MMLU, JSONSchemaBench, and extraction tests, the base and adapted 7B models diverged most on rigid output formats.The base model’s JSON validity was 0.302 against 0.981 for the adapted model, while free-form instruction and English world knowledge favored the base.Source: §4.1 · p. 5, Table 3 · p. 5
  • The base model retained small advantages in English world knowledge and free-form instruction, while the adapted model led in Russian narrative knowledge.Russian MMLU was within noise, and extraction had no clean winner: the base led entity recognition while the adapted model led relation extraction.Source: §4.1 · p. 5
  • Structured-output training brought a portable competence in rigid formats, and JSON validity aligned with whether PoT helped or hurt the 7B model.The authors report JSONSchemaBench validity of 0.981 vs. 0.302 and PoT changes of +0.282 for Meno-Lite and −0.194 for base Qwen2.5-7B.Source: §4 · p. 4, §4.1 · pp. 5–6

5 What Happened on Test

The test post-mortem points to reading and semantics as the main failure points: changing the reader changed many answers, while later prediction turnover brought no reliable gain. The leaderboard pattern is consistent with reading quality separating teams, though the authors could not verify other teams’ pipelines.

5 What Happened on Test

  • Changing from Tesseract to page-wise vision transcription changed answers on 89.6% of tasks; the third attempt’s 14.3% row turnover produced no gain beyond sampling noise.The third attempt scored 0.17 percentage points lower on 1,730 tasks; its deadline repair changed nine rows, all within 26 re-read documents.Source: §5 · p. 6
  • Invalid evidence codes fell from 8.7% to 1.7% of tasks, but format sanitation did not move metrics, locating remaining failure in wrong passages, wrong numbers, or both.Gold evidence was always one block, while a third of tasks received three or more blocks.Source: §5 · p. 6
  • The leaderboard’s bimodality is consistent with reading quality, not reasoning, separating teams, although other teams’ pipelines were unverified.A 22-team spike reached 67.46% joint accuracy, and the top-ranked team reached 85% with the same inputs.Source: §5 · p. 6

6 A Controlled Degradation Study

A controlled re-rendering of validation PDFs reproduced the OCR failure mode but did not reproduce the full damage seen on the real test scans. The authors therefore treat the simulation as bounded evidence, not a complete account of vision-reader failure.

6 A Controlled Degradation Study

  • The re-render used 72-dpi grayscale JPEGs with a diagonal watermark and running header, and its median stopword share fell between born-digital and test scans.The pipeline, 72B generator, and sampling matched the validation row used for comparison.Source: §6 · p. 6
  • On synthetic scans, Tesseract answers fell from 0.825 to 0.244 and evidence fell to zero, while the vision-language reader reached 0.756 joint against 0.825 on born-digital input.The genuine test yielded 17.6% answers for the same vision-language reader; the residual gap reflects damage the clean re-render did not capture.Source: §6 · p. 6, Table 6 · p. 7
  • The simulation reproduced the OCR collapse but underestimated vision-transcription damage from dirtier scans, eight-page documents with four times the blocks, and unrecoverable identifier tokens.The authors say rendering alone can zero out the primary metric through the identifier channel, while the simulation’s other half must be checked against real inputs.Source: §6 · p. 7

7 Related Work

The paper situates DocSem among retrieval, program-of-thought, graph-retrieval, ensemble, and tabular-grounding work. It identifies single-passage arithmetic and exact-match block identifiers as boundary conditions shaping which techniques help.

7 Related Work

  • The reasoning stack combines Program-of-Thoughts and self-consistency with chain-of-thought prompting and program-aided methods.The paper cites Chen et al. (2023), Wang et al. (2023), Wei et al. (2022), and Gao et al. (2023).Source: §7 · p. 7
  • For single-passage arithmetic over short documents, only the distilled top-k of graph retrieval earns its tokens.The passage situates this boundary against GraphRAG and lightweight variants, with multi-hop evaluations cited as related work.Source: §7 · p. 7
  • Majority voting over context variants did not beat the best single vote, while exact-match block identifiers make segmentation drift consequential.The paper connects tabular and textual grounding to HybridQA and TaPas, noting DocSem’s exact-match identifier requirement.Source: §7 · p. 7

8 Conclusion: Lessons with Numbers

The conclusion prioritizes auditing evaluation inputs before tuning architecture, then draws lessons about model scale, model-dependent harness effects, and deadline repairs.

8 Conclusion: Lessons with Numbers

  • A 27B model with the full harness matches a 72B model without it at roughly 2.7× fewer parameters and lower reported GPU and CO2e use.The comparison is one GPU and 0.14 kgCO2e versus eight GPUs and 0.53 kgCO2e.Source: §8 · p. 7
  • Auditing the physical inputs first matters: skipping a pixel-level inspection missed watermarked raster scans, accompanying a drop from 0.853 to 0.136 joint accuracy.The 38-page re-read changed nine submission rows and slightly lowered the score, so the visible artifacts were only a small sample of systemic transcription damage.Source: §8 · p. 7
  • PoT's effect is model-dependent: it pays where format discipline exists and taxes where it does not, making a compact model harness-ready rather than merely small.The conclusion ties this difference to structured-output training.Source: §8 · p. 7
  • The validation lesson does not transfer to damaged test reading: the authors say deadline repairs treat symptoms rather than systemic transcription damage.The re-read covered 38 damaged pages and changed nine submission rows.Source: §8 · p. 7

Limitations

The study's limitations constrain the scope and interpretation of its reading-failure analysis, model comparison, and validation estimates.

Limitations

  • The reading-failure analysis uses portal scores and submission diagnostics, not human re-annotation of test documents.The study covers one shared task.Source: Limitations · p. 7
  • The raster-input ceiling remains unknown because only a 7B page-by-page vision reader was evaluated, not stronger document-parsing VLMs.Source: Limitations · p. 7
  • The 7B lineage comparison uses an English benchmark with a Russian-primary model, so it bounds the harness rather than the model.Source: Limitations · p. 7
  • Validation-portal estimates use 217 tasks, with 95% confidence intervals of four to five points; differences below one point should be read accordingly.Source: Limitations · p. 7

Ethics Statement

The statement reports the system's data scope and serving context, then records run-registry results on demonstrations and verification.

Ethics Statement

A Run Registry

  • PoT demonstrations lifted evidence exact match to 1.000 but cost 0.02–0.03 answer accuracy, a trade the authors declined for the final system.The registry contains all 181-task check rows referenced in Table 1; Table 2 validation rows are portal-scored.Source: A Run Registry · p. 9
  • A verification pass that re-derives constants from cited blocks subtracted 0.013 joint by rejecting honest fractions.Source: A Run Registry · p. 9

B Retrieval Benchmark

The retrieval benchmark favors dense vectors with reranking on paraphrased queries, while the accompanying passages describe serving costs, prompt constraints, ontology types, and raster-input parsing choices.

B Retrieval Benchmark

  • The gte pair reached 0.901/0.995/1.000 evidence recall at top-1/3/8 and 1.000 after reranking to top-3; Track C gained 0.022 joint at k=3 on the compact line.BM42 alone had recall@3 of 0.066; Track C was neutral at k=5, and graph-native Track B lost 0.11 joint to vector retrieval.Source: B Retrieval Benchmark · p. 9

C Cost Accounting

  • Mini-graph extraction ran at about four tasks per minute on a ten-instance 7B fleet, while generation at k=5 took thirteen minutes per four-document shard on one 27B server.The full-document baseline skips retrieval but uses prompts that grow with document length.Source: C Cost Accounting · p. 9
  • The compact line reaches 0.64 joint on one GPU, in about twice the wall time the 72B line spends on eight GPUs.The cited serving costs cover check runs measured on the authors' A100 80GB infrastructure, with energy estimated using stated TDP, PUE, and grid-intensity assumptions.Source: C Cost Accounting · p. 9, Table 7 · p. 10

D Prompts and Ontology

  • The extractor uses 18 entity types and 13 relation types, with MEASURED_IN reserved for unit relations.Generator instructions constrain reasoning to displayed blocks, prohibit mixing numbers across passages, and require copied evidence identifiers.Source: D Prompts and Ontology · p. 10

E Degradation Protocol

  • Validation pages were rendered as stamped, low-resolution grayscale JPEGs in image-only PDFs, and generic identifier parsing was used because OCR corrupted known block prefixes.Strict known-mode parsing then returned zero blocks; unchanged paths also invalidate the path-to-hash index unless cleared.Source: E Degradation Protocol · p. 10
Loading 2609.34366v1…