Lit3R: Retrieve-Relate-Read for Evidence-Grounded Question Answering over Scientific Literature

1 Introduction

• Lit3R adds paper-to-paper expansion to retriever–reader architecture, targeting relevant papers that question-anchored retrieval may miss.This is intended for questions requiring evidence across papers whose lexical or semantic similarity to the question may be weak.Source: §1 · p. 1
• Retrieve iteratively decomposes questions into subqueries, combines BM25 and dense retrieval, reranks candidates, and uses LLM verification to decide whether another round is needed.Relate then fuses the question-anchored ranking with a ranking expanded from retrieved papers.Source: §1 · p. 1
• Read first identifies supporting evidence within individual papers, then synthesizes evidence across papers into an answer with paper identifiers and evidence locators.Source: §1 · p. 1
• Lit3R ranked 4th on the official test-set leaderboard, and experiments report further candidate-paper recall improvements from Relate.Source: §1 · p. 1

2.1 Retriever

• The retriever fuses question-anchored Ranking A with paper-anchored Ranking B to reach related papers that may sit far from the question in lexical and dense space.Source: §2.1 · p. 2

2.1.1 Corpus preparation

• The corpus comprises 27,487 papers and 2,564,545 indexed chunks, extracted from PDFs and split across five content types.The collection covers recent machine learning, computer vision, and natural language processing venues from 2024 and 2025.Source: §2.1.1 · p. 2

2.1.2 Question-anchored retrieval

• Question-anchored retrieval iterates through LLM-generated subqueries, hybrid BM25 and dense search, pseudo-relevance feedback, and verification that directs further queries toward missing evidence.Retrieval stops when evidence is sufficient, no new subquery is generated, or three rounds have run.Source: §2.1.2 · p. 3

2.1.3 Paper-to-paper expansion

• Paper-to-paper expansion retrieves neighbors through embedding similarity, shared references, and BM25 over seed-paper text, then fuses the lists into Ranking B.The system round-robins neighbor lists across seeds before equal-weight fusion.Source: §2.1.3 · pp. 3–4

2.2 Reader

• The reader receives the top 50 candidate papers and processes them in two stages: paper-local reading followed by cross-paper synthesis.Source: §2.2 · p. 4

2.2.1 Local reading

• Paper-local reading assesses relevance and evidence sufficiency for each query–paper pair, passing only papers with valid supporting evidence to synthesis.Source: §2.2.1 · p. 4

2.2.2 Cross-paper synthesis

• Cross-paper synthesis extracts facts with values, units, sources, and supporting chunks, then integrates evidence and performs needed calculations or comparisons.It produces task-formatted answers, including selected options or constructed table rows and columns when required.Source: §2.2.2 · p. 4

3 Experiments

• The 55-question validation set—26 hidden_source_single_paper and 29 multi_paper questions—was used to configure retriever and reader hyperparameters; 71 test questions were used for final evaluation.The task defines hidden_source_single_paper questions as having one gold evidence paper and multi_paper questions as having multiple; multi_paper questions have three to nine gold evidence papers, with a median of four.Source: §3 · p. 4

3.1 Retriever analysis

• The final retriever achieved validation recall@50 of 0.968, while multi-paper recall reached 0.940 at k = 50.At k = 1, recall was 0.846 for single-paper questions and 0.222 for multi-paper questions.Source: §3.1 · p. 5
• Multi-paper recall@50 rose from 0.647 to 0.940 when paper-to-paper expansion augmented Ranking A, while overall recall@50 increased from 0.814 to 0.968.The authors attribute the gains to semantic, bibliographic, and lexical paper-to-paper relations that help recover papers not readily connected to the question.Source: §3.1 · p. 5

3.2 Official test-set results

• Macro paper-retrieval F1 was 0.992, macro evidence F1 was 0.737, and multiple-choice accuracy was perfect; table answering remained more challenging.Source: §3.2 · p. 5, Table 3 · p. 5

• 0.647 to 0.940 multi-paper recall@50 after adding paper-to-paper expansion in a controlled comparison.Source: §4 · p. 5
• The retriever fuses question-anchored and paper-to-paper rankings to create a 50-paper candidate set.Source: §4 · p. 5
• The reader selects evidence papers, identifies grounded evidence, and constructs answers under deterministic validation.Source: §4 · p. 5

• Evaluation covers only LitTraceQA, with the same 55 validation questions used for configuration selection and retriever analysis.Source: Limitations · p. 5
• Generalization to other domains and question types remains untested.Source: Limitations · p. 5
• Experiments do not isolate individual expansion signals or reader-stage contributions to end-to-end performance, or identify causes of remaining evidence-grounding and table-answering errors.Source: Limitations · p. 5

• Table 4 reports the models and inference settings used in the system.Source: Table 4 · p. 6, A Models and inference settings · p. 6

• PDFs are converted with MinerU 3.4.3 using its pipeline backend.Source: B Document conversion models · p. 6
• The backend composes task-specific models rather than a single end-to-end model.Source: B Document conversion models · p. 6
• The backend uses PDF-Extract-Kit-1.0 weights at snapshot ed6b654, and Table 5 lists the models used at each stage.Source: B Document conversion models · p. 6, Table 5 · p. 6

C Prompts

• Each retriever call pairs the Figure 2 system prompt with one user message; Figures 3–5 cover query decomposition, later-round queries, and evidence verification.Calls request JSON output and carry no conversation history.Source: C Prompts · p. 6, Figure 2 · p. 6
• The first retrieval round generates four subqueries; later rounds generate up to four using prior queries and verifier-identified missing information, with a venue-year instruction added when applicable.Retrieval stops if no subqueries are generated; the venue-year line is placed differently in first and later rounds.Source: C Prompts · p. 6, Figure 4 · p. 7, Figure 6 · p. 7
• The reader’s Stage 1 judges each candidate paper and returns relevant papers with supporting chunks; Stage 2 builds the answer from accepted papers and independently grounds requested components.Prompts request JSON and use synthetic worked examples rather than labeled task data.Source: C Prompts · pp. 7–8
• Decision A marks a candidate relevant when it owns a requested item or directly supplies an answer source, operand, or required eligibility fact—not merely a related topic or similar value.A compound method name is treated as one identity, so a paper introducing only a component is not automatically the requested method’s source.Source: C Prompts · p. 8
• Decision B requires supplied context that directly provides usable answer content, such as a value, operand, requested citation, visible figure, or eligibility fact, under the query’s hard constraints.Paper identity or topic alone, an absent value, an incomplete range, or a missing required image is insufficient.Source: C Prompts · pp. 8–9
• A candidate may provide only one operand for a multi-paper answer; table cells are checked against their full headers and setting, while an attached image of the requested figure can itself count as evidence.Coordinated clauses are parsed separately so a constraint stated in one clause does not automatically govern another.Source: C Prompts · pp. 8–9