Source-linked AI summary
PageRecall: Measuring Page Selection in Literature-Grounded Question Answering
Aaditya Chauhan
TL;DR
LitTraceQA asks systems to retrieve relevant papers, locate page-level evidence, and answer in a requested format, while separating evidence grounding from answer accuracy. The paper measures page selection and uses whole-paper input when documents fit, with parsing-based anchors for position-named targets. The selector retrieved the gold page only about half the time, whereas the reader usually cited it correctly when given that page.
Problem
LitTraceQA requires systems to retrieve relevant papers, identify page and table or figure evidence, and answer in a requested format, while distinguishing reader errors from page-selection errors.
Source: §1 · p. 1Method
Results
52.6% gold-page recall contrasted with 45 of 48 correct-page citations (94%) when the reader received the gold page.
Source: §1 · p. 1Takeaways & Limitations
The paper recommends measuring selector recall before investing in the reader and avoiding page selection when documents fit in context.
Source: §12 · p. 8Position-dependent degradation in long contexts is the main threat to showing whole papers, and the authors have not measured it directly.
Source: §5 · p. 4
Abstract
from arXiv · showhide
We describe our system for LitTraceQA (GroundLM @ EMNLP 2026): given a research question, retrieve the relevant papers from a pool of 27,487, cite the page and the table or figure where the answer lives, and answer in a requested format. Our main finding is that evidence grounding is limited by retrieval, not by reading. The page selector put the annotator's page, which we call the gold page, in front of the model that locates evidence only about half the time (52.6% gold-page recall), while that model, given the page, cited the right one in 45 of the 48 locators it emitted (94%). When the page was missing it rarely said so: of 45 such cases it returned nothing 14 times, a wrong page 24 times, and a correct page 7 times, so the pipeline failed quietly almost twice as often as it failed visibly. Since the failure was that the right page was never shown, the fix is to stop choosing: each retrieved paper fits in the model's context, so we show it whole. Page ranking survives only as a fallback inside papers too long to fit, which no test-split paper was, and gold-page recall reaches 100% on the papers we can parse. Separately, questions that identify their target by position rather than content, such as "the first author of the 24th reference", are served by parsing rather than retrieval: we resolve the bibliography into an addressable list, which also supplies identifiers the evidence metric scores. The final system scores 0.762 paper $F_1$, 0.441 evidence $F_1$ and 0.920 multiple-choice accuracy on the held-out test split. Because the pipeline depends on a closed model without seed control, we release a harness that verifies the paper's central claims against committed artifacts.
1 Introduction
LitTraceQA separates paper retrieval, page-level evidence selection, and answer accuracy to identify whether grounding errors arise from reading or from the pages shown to the reader. The paper’s measurements point to page selection as the bottleneck and add analyses of tuning, multiple-choice bias, and answer grounding.
1 Introduction
- LitTraceQA scores paper retrieval, coarse evidence locations, and requested-format answers separately, enabling analysis of reader errors versus page-selection errors.Evidence locations include a page and table or figure number.Source: §1 · p. 1
- 52.6% gold-page recall contrasted with 45 of 48 correct-page citations (94%) when given the gold page, pointing to selection as the bottleneck.Without the gold page, the reader emitted a locator in 31 of 45 units, but only 7 were correct.Source: §1 · p. 1
- The development-set choice of how many papers to predict transferred badly to the test split.The system fixed this count from the development distribution.Source: §1 · p. 1
- The paper examines positional bias in multiple-choice labels and a permutation-voting correction, and audits answer components against their cited source text.It reports that groundedness does not predict correctness on this data.Source: §1 · p. 1
2 Related Work
- Prior work evaluates evidence selection at paragraph level or figure/table level, while this paper measures page-level selection and uses a caption index.It also describes deterministic structural anchors and notes that whole-document input is viable only if long-context models use context uniformly.Source: §2 · p. 2
- Page-image retrieval methods sidestep text extraction, and the paper identifies them as stronger selector baselines that remain untested here.LongDocURL and MMLongBench-Doc are cited as possible external validation for evidence locating.Source: §2 · p. 2
3 Task and Data
LitTraceQA uses a pool of 27,487 papers and scores evidence by paper, source type, page, and normalized object identifier. Its system combines metadata retrieval, document parsing, and evidence extraction, with page-level evidence aligned to physical PDF pages.
3 Task and Data
- The pool contains 27,487 papers from 2024–2025 across nine venues, with metadata and PDF URLs but no full text.Development has 55 labelled questions and test has 71; neither split contains freeform questions.Source: §3 · p. 2
- Inference-time inputs omit task family, so the system infers it from the question before choosing a retrieval policy.Evidence scoring uses a coarse key of paper, source type, page, and normalized object id; a correct object on the wrong page scores nothing.Source: §3 · p. 2
4 System
- The system uses a pinned Claude model without seed or temperature control, so the response cache—not the source alone—supports reproducibility.The same model transcribes tables and figures from rendered page images; nothing is fine-tuned.Source: §4 · p. 2
- Paper retrieval indexes title and abstract with acronym-preserving BM25 and dense embeddings, then routes questions through family-specific selection procedures.Multi-paper questions use tiered name matching and an LLM mapping step; corpus-scan questions use venue/year filtering, abstract screening, and full-text checks.Source: §4 · p. 3
- PDF acquisition prioritizes publisher and official proceedings copies before arXiv because pagination is part of the scored evidence key.Every predicted test paper was acquired, and all 42 OpenReview-blocked papers resolved to official copies.Source: §4 · p. 3
- Parsing preserves page-level text and indexes table and figure captions, while the evidence stage extracts locators and facts and re-reads cited visuals.Numeric tokens are checked against page characters, and invalid object ids are dropped; the physical PDF page matches the evidence metric’s granularity.Source: §4 · p. 3
5 Page selection is the bottleneck
The selector exposed the reader to a gold page only about half the time, while the reader usually cited correctly when given that page. Showing whole papers addresses this selection bottleneck when they fit in context; page ranking remains a fallback for papers above the cap.
5 Page selection is the bottleneck
- 52.6% gold-page recall contrasted with 94% correct citations among locators emitted when the reader received the gold page.The selector result is averaged over 97 units; the reader cited the right page in 45 of 48 emitted locators given the gold page.Source: §5 · p. 4
- Without the gold page, the reader emitted a locator in 31 of 45 cases, but only 7 of those locators were correct.All seven correct locators identified a table or figure through the caption index; the model returned nothing in the other 14 cases.Source: §5 · p. 4
- BM25 is a poor within-paper ranker because page-level term statistics are degenerate and pages share topical vocabulary.The passage characterizes within-document page ranking as close to a worst case for BM25.Source: §5 · p. 4
- The dense-selector comparison is a lower bound, not a like-for-like comparison, because dense retrieval used only the first 2,000 page characters.Its score was compared against BM25 over full pages.Source: §5 · p. 4
- Showing the whole paper puts every page before the reader when it fits; BM25 plus anchors remains a fallback above the page cap.The median paper was about 19k tokens, and the fallback did not trigger on the test split.Source: §5 · p. 4
- Showing every page has not been tested against position-dependent degradation, though the evaluated questions were mostly single-page lookups at about 19k tokens.The authors identify this as the main threat to the whole-paper approach and say they have not measured it directly.Source: §5 · p. 4
6 What structural parsing still buys
Parsing handles questions that specify targets by position or literal name, which ordinary page retrieval cannot reliably locate. Whole-paper input made anchors largely redundant for page selection, but parsed bibliography entries and identifiers still support answering and evidence scoring.
6 What structural parsing still buys
- The system adds four deterministic anchors that resolve named tables to caption pages, equation numbers to printed-marker pages, ordinal references to parsed bibliography entries, and distinctive caption terms to object pages.Source: §6 · p. 5
- 95 of 97 measurement units fit under the page cap and saw the whole paper; the move from 52.6% to 100% gold-page recall was not attributable to anchors.For the two units above the cap, BM25 and anchors both caught the gold page; the test split had no paper above the cap.Source: §6 · p. 5
- Parsing supplies the resolved bibliography entry or list and extracts citation_id and equation_id, which the evidence key requires.Omitting those identifiers mismatched gold by construction and cost 0.091 evidence F1 on development before the fix.Source: §6 · p. 5
7 A tuning choice that did not transfer
Padding multi-paper predictions to match development-set paper counts transferred poorly to test, where it over-predicted. Removing padding improved test paper F1 and the composite score; the development sweep also showed the heuristic was mildly harmful on its tuning split.
7 A tuning choice that did not transfer
- 0.840 development precision fell to 0.453 on test as padding invented roughly two papers per question, yielding test paper F1 of 0.517.The development configuration padded mapped clusters to four papers based on the development gold-set distribution.Source: §7 · p. 5
- 0.762 test paper F1 and 0.568 composite followed removal of padding, the largest single improvement reported.The shipped system predicted a mean of 1.6 papers per test question.Source: §7 · p. 5
- Development F1 was 0.840, 0.831, 0.814, and 0.825 for padding targets of 1, 2, 3, and 4, while precision fell from 0.935 to 0.840.The heuristic was mildly harmful on the split it was fitted to, and aggregate evaluation of bundled changes concealed the component's cost.Source: §7 · p. 5
- The authors recommend ablating choices individually and warn that development-set gold-set cardinality may not transfer to test.Predicting set size from the development distribution can over-predict when split cardinalities differ.Source: §7 · p. 6
8 Positional bias in multiple-choice answering
The section examines whether multiple-choice answers reflect option content or label/position preferences, then tests permutation voting as a correction. On test, the method increased accuracy from 0.840 to 0.920, though the sample was small.
8 Positional bias in multiple-choice answering
- Accuracy alone cannot distinguish ordinary errors from label or position preference, whereas the predicted-label distribution can expose such bias.Options are presented as a labelled list, so preferences may attach to a label or to an option's position.Source: §8 · p. 6
- The system chose D 22 times among 50 test questions, about twice as often as chance, despite accuracy of 0.840.The label distribution did not change after upstream paper predictions were corrected, placing the cause in answer synthesis rather than retrieval.Source: §8 · p. 6
- The correction re-asks questions with option texts moved among fixed label slots and takes a majority vote across identity, reversal, and rotation.A three-way split retains the original answer, preventing one dissenting view from overwriting it.Source: §8 · p. 6
- 0.840 to 0.920 test multiple-choice accuracy followed permutation voting; four changed answers were all decided 2–1 by permuted views.The method changed the predicted-label distribution to A 8, B 9, C 13, D 20; the authors note the sample is small.Source: §8 · p. 6
9 Auditing grounding
The paper adds a pre-emit audit to test whether answers are supported by their own cited page text, distinguishing grounding from correctness. The audit found substantial unsupported output, but its verdict was close to independent of correctness on this data.
9 Auditing grounding
- The pre-emit gate audits each answer component against only the raw parsed text of the pages it cites, excluding pipeline-derived facts and transcripts.The same model receives only the question, answer, and cited page text, and treats true-but-unstated claims as unsupported.Source: §9 · p. 6
- 39 of 67 development answer components were supported by their own cited source text, including 25/41 multiple-choice and 14/26 freeform answers.A substantial minority asserted something their cited evidence did not carry.Source: §9 · p. 6
- Unsupported multiple-choice answers were correct 15/16 times versus 22/25 for supported answers; the authors report grounding verdicts as close to independent of correctness on this data.Freeform results showed a weak effect in the expected direction, based on 26 observations.Source: §9 · p. 6
- The evaluation cannot reward abstention: enforcing the gate would reduce multiple-choice accuracy from 0.902 to 0.537 because blank and wrong answers score alike.The authors therefore run the gate in reporting mode and frame this as a task-design observation.Source: §9 · p. 6
10 Results
The page-selection change improved development evidence F1, but evidence remained the system's weakest absolute score. Full page recall alone does not guarantee evidence credit under the evaluation key.
10 Results
- Table 1 reports the final configuration on development and test; its test scores come from the organisers' online evaluator against hidden labels.The ranking metric averages paper F1, evidence F1, and the mean of available answer-type metrics using the macro cell figure.Source: §10 · p. 7
- 0.384→0.456 evidence F1 on development followed page-selection change, with precision rising 0.367→0.475.The additional pages replaced wrong locators with right ones rather than adding noise.Source: §10 · p. 7
- Full gold-page recall is necessary but insufficient for evidence credit, which also requires the right source type and object id and matching PDF page numbering.Evidence remained the system's weakest score in absolute terms.Source: §10 · p. 7
11 Error analysis
The error analysis separates retrieval, evidence-location, and answer failures, showing that remaining retrieval misses are largely recall-related while evidence and table answering have distinct constraints.
11 Error analysis
- Retrieval errors are now mostly recall rather than precision, with test values 0.830 against 0.759; development predictions miss 38 gold papers and add 9 non-gold papers.Missed papers generally reflect failure to map a named method to any pool paper; residual cases lack a distinctive acronym and a strong lexical signal.Source: §11 · p. 7
- Evidence errors stem from PDF-version mismatches, source-type disagreement, and equation bodies absent from extracted text.Explicit source-type routing corrected one equation locator, but questions about equation renderings remain unanswerable from text when the equation body is absent.Source: §11 · p. 7
- On test, table cell accuracy is 0.242 macro against row F1 0.421, with numeric misreads and multi-level headers driving inspected development errors.Multiple-choice errors fell to 4 of 50 after the option-order correction; the test residue cannot be characterized because gold answers are unavailable.Source: §11 · p. 7
12 Conclusion
Stage-specific measurements exposed losses hidden by end-to-end scores and support measuring selector recall before improving the reader. The conclusion recommends showing whole documents when they fit, while noting limits in sample size, acquisition, extraction, and reproducibility.
12 Conclusion
- Stage-specific checks exposed three hidden losses: the selector found the right page only half the time, padding cost 0.245 paper F1, and multiple-choice answers over-favored the final option.The checks were selector recall conditional on correct retrieval, a single-parameter ablation, and the answer-option distribution.Source: §12 · p. 7
- When documents fit in context, the practical recommendation is to measure selector recall before investing in the reader and to skip page selection.The paper reports that the reader looked adequate while the selector found the right page only about half the time.Source: §12 · pp. 7–8
Limitations
- Evaluation is small: 55 development questions and 71 test questions, while the multiple-choice debias rests on six corrected answers.All six corrected answers ran in the predicted direction, but the authors call six a weak basis for a mechanism claim.Source: Limitations · p. 8
- PDF-version mismatch accounted for roughly 32 of 108 development misses before the page-selection change, but its final-configuration share and recoverability are unknown.Gold locators sometimes refer to document revisions the system cannot fetch.Source: Limitations · p. 8
- Absent equation text leaves some rendering questions unanswered without a vision pass, and exact match penalizes terse answers against full-sentence freeform gold.The system does not route to a vision pass for this equation content.Source: Limitations · p. 8
- The closed model exposes no seed control, so exact reproduction depends on the released response cache; the optional test_extra split was not attempted.That split would require on the order of 104 PDFs.Source: Limitations · p. 8
13 Reproducibility
Because a closed model without seed control cannot be reproduced by rerunning the source alone, the paper releases a harness that checks claims against committed artifacts. The reproducibility practices bind calls, runs, and measurements to recorded artifacts and state.
13 Reproducibility
- The released reproduce.py harness checks central claims against available artifacts and reports skips rather than silent success when required material is missing.With the repository and released dataset, it verifies the gold file by hash, recomputes development metrics from committed predictions using the organizers’ evaluator, and validates the submission.Source: §13 · p. 8
- Given parsed PDFs, the harness recomputes the page-selection measurement; given the response cache, it replays the pipeline and requires zero cache misses and byte-identical output.All checks passed at submission.Source: §13 · p. 8
- Reproducibility practices include content-addressed calls, immutable provenance-stamped run artifacts, and pinning measurements to the upstream artifact used.A stage-1 change silently shifted the experiment from 97 units and 0.526 to 90 units and 0.533, motivating artifact-pinned measurements.Source: §13 · p. 8