PaperlayerView exact revision on arXiv

Source-linked AI summary

SLIDERS: Systematic Reviews via Automated Evidence Synthesis and Reconciliation

Harshit Joshi, Priyank Shethia, Jadelynn Dao, Monica S. Lam

arXiv:2604.22294v2cs.CLcs.AI

TL;DR

Systematic reviews need comprehensive, provenance-preserving synthesis across large document collections, but existing retrieval-centered workflows can miss or overwhelm distributed evidence. SLIDERS builds question-specific evidence tables through extraction and reconciliation, and outperforms evaluated baselines across large-corpus and follow-up benchmarks. Its limitations include overhead on shorter contexts, difficulty with exact numerical questions, and risks from extraction, reconciliation, or synthesis errors.

  • Problem

    Systematic-review workflows require comprehensive evidence tracking and synthesis, while retrieval-centered systems may miss distributed evidence or retrieve unmanageable volumes.

  • Method

    SLIDERS generates question-specific evidence variables, extracts structured values and source-grounded excerpts, reconciles related records, and supports answers from the resulting evidence table.

  • Results

    SLIDERS outperforms the best-performing baseline across evaluated systematic-review tasks, including 77.9% and 58.3% on two follow-up benchmarks.

  • Takeaways & Limitations

    Organizing extracted evidence into a coherent, queryable table supports both initial research questions and follow-up analysis across large document collections.

  • Takeaways & Limitations

    SLIDERS incurs overhead on shorter contexts and struggles with exact counting and numerical questions requiring borderline classification, deduplication, normalization, and aggregation.

Abstract

from arXiv · show

Systematic reviews -- which requires comprehensive evidence collection and synthesis from large document corpora in response to targeted research questions -- are foundational in finance, social sciences, and other technical fields. Manual construction of evidence tables is labor-intensive, and recent LLM-based assistants relying on embedding or keyword based search often fail to meet the coverage standards of systematic reviews. We introduce SLIDERS, a novel LLM-based methodology for systematic reviews, by automatically assembling evidence tables tailored to research questions. In addition to extracting structured data from documents, SLIDERS can extract full-text excerpts that serve as direct evidence or as provenance for structured data. Core to SLIDERS is an automated evidence reconciliation agent that writes code to analyze and reconcile extracted evidence, bringing together information fragmented across documents, resolving inconsistencies across excerpts, and synthesizing overlapping findings into a coherent evidence table. In addition, SLIDERS allows users to ask follow-up questions in natural language to further explore the assembled evidence. We evaluate SLIDERS on three systematic-review-style tasks over large document collections. SLIDERS outperforms the best-performing baseline across benchmarks, remains near 90% accuracy across 6M-11M-token corpora. On two new follow-up analysis benchmarks SLIDERS can answer 77.9% and 58.3% followup questions accurately

1 Introduction

SLIDERS targets labor-intensive systematic-review workflows by combining broad evidence collection with structured, provenance-preserving reconciliation and synthesis. It evaluates this approach across large-corpus and follow-up question-answering tasks.

  • Systematic reviews require collecting, reconciling, and synthesizing evidence from eligible documents to answer pre-specified questions.
  • 67.3 weeks is the estimated mean time to complete and publish a review in one PROSPERO-based analysis.
  • Broad evidence coverage and accurate synthesis are difficult when evidence is distributed, redundant, incomplete, or conflicting across large corpora.
  • SLIDERS discovers evidence variables, extracts documents chunk by chunk, reconciles related records, and synthesizes answers from the resulting evidence table.
  • On Browsecomp+, SLIDERS outperforms the best-performing baseline by 10 points; on Wikipedia and SEC follow-up benchmarks, it achieves 77.9% and 58.3%.On context-window-fitting benchmarks, it outperforms all evaluated baselines by up to 16.58 points.

2 Related Works

Related approaches scale document processing through retrieval, chunk-wise aggregation, expanded context, or agentic decomposition. SLIDERS instead maintains an explicit, question-specific evidence representation for systematic-review-style synthesis.

  • Retrieval-augmented systems ground synthesis in retrieved passages but may lack an explicit, question-specific evidence representation.
  • SLIDERS is complementary to these efforts while emphasizing organized evidence-table construction rather than only retrieval or direct aggregation.
  • Chunk-then-aggregate pipelines process document segments locally and combine outputs, often reducing chunk results directly into a final answer or summary.
  • Long-context and agentic systems expand model capacity or divide work across roles and recursive subtasks.

3 The SLIDERS Method

SLIDERS constructs a reusable evidence table by generating question-specific variables, extracting grounded records, and reconciling related evidence into answer-ready groups. Its design preserves provenance while making distributed evidence comparable and manageable.

  • Task formulation: SLIDERS constructs a reusable evidence table with question-relevant variables as columns and document-grounded records linked to source spans and provenance.
  • Generating evidence variables: Evidence variables may include typed values and free-text entries, and are generated from the research question and sampled document metadata.
  • Generating evidence variables: Strong typing and normalization make independently extracted values suitable for selection, aggregation, filtering, and comparison while preserving source text and rationale.
  • Extraction into the evidence table: The extraction task produces normalized values, supporting spans, and rationales, while a relevance gate suppresses extraction from chunks lacking relevant evidence.
  • Evidence reconciliation agent: The reconciled table preserves original provenance spans, excerpts, and extraction rationales while organizing evidence into an answer-ready representation.
  • Evidence reconciliation agent: Primary-key grouping avoids infeasible all-pairs comparison by reconciling records only within groups sharing the same grouping key.The direct procedure would require O(n^2) comparisons over n records.
  • Evidence reconciliation agent: Reconciliation groups records using question-specific primary-key variables and canonicalized values before merging duplicates, combining complementary records, and marking unresolved conflicts.

4 Evaluation on Benchmarks Beyond LLM Context Limits

SLIDERS is evaluated on systematic-review-style benchmarks spanning deep research, reusable follow-up analysis, and long-context question answering. It outperforms competing baselines while retaining strong performance as corpora and evidence-bearing document sets grow.

  • Evaluation setting: SLIDERS evaluates evidence synthesis across BrowseComp+, WikiCeleb100, FinQ100, FinanceBench, Loong, and Oolong.The benchmarks test distributed multi-document research, reusable follow-up analysis, and within-window long-context QA.
  • Benchmark results: 90.00 accuracy on BrowseComp+ exceeds the strongest non-SLIDERS baseline, RLM, by 10 points.Retrieval baselines retrieved only 10.4% and 12.5% of relevant documents on average, and returned zero evidence documents for 57% and 49% of questions.
  • Scalability: As token length rises from roughly 6M to 11M, SLIDERS remains near 90% accuracy while RLM varies more and ends lower.With evidence from 10 or more documents, SLIDERS reaches 100% accuracy while RLM drops to around 70%.
  • Follow-up analysis: On WikiCeleb100, SLIDERS reaches 78.91 versus 59.80 for RLM, while FinQ100 reaches 60.18 versus 28.87 for the best completing baseline.Both benchmarks reuse evidence collected for an initial question to answer related follow-up questions.
  • Model portability: Using Qwen3.5-122B-A10B for all components, SLIDERS outperforms GPT-4.1-based baselines on WikiCeleb100 and FinQ100.On FinQ100, it also improves over the GPT-4.1-based SLIDERS variant.

SLIDERS RLM GPT BM25

Ablation results show that reconciliation is especially important when evidence tables become too large to pass directly to the answer generator. Its benefit is largest on the large follow-up-analysis benchmarks.

  • Ablation results: On BrowseComp+, Direct table reaches 87.33, only 2.67 points below the full system when the evidence table fits in context.Evidence-table construction alone is strong when the complete table can be consumed directly.
  • Ablation results: Removing reconciliation drops WikiCeleb100 performance from 78.91 to 60.50.The result shows reconciliation becomes critical when the evidence table is too large for direct consumption.

5 Evaluation on Benchmarks Within LLM Context Limits

SLIDERS is evaluated on three long-context QA benchmarks that fit within frontier-model context windows. It achieves the strongest average performance, remains stable as inputs grow, and benefits from extracting and reconciling evidence before synthesis.

  • The evaluation compares SLIDERS with base language models, sequential chunk summarization, and recursive language modeling on within-window QA tasks.
  • SLIDERS achieves the best average performance on FinanceBench, Loong, and Oolong, outperforming GPT-4.1 by 9.54 points on average.
  • 64.67 to 40.00: removing chunk-level extraction sharply reduces Oolong performance, where relevant information is distributed across long inputs.
  • Removing reconciliation lowers performance across all three benchmarks, indicating that extracted evidence must be organized and resolved before answer generation.
  • SLIDERS remains above 70% on Loong at the longest inputs, while RLM falls below 50% at the largest token length.

6 Analysis and Discussions

The analysis examines reconciliation behavior, failure modes, and how SLIDERS fits into systematic-review workflows. Reconciliation compresses and organizes evidence, while limitations arise for short contexts, exact numerical reasoning, and some answer-time retrieval or reasoning steps.

  • Reconciliation analysis: 1.98× and 4.51×: reconciliation produces the largest evidence-table compression on WikiCeleb100 and FinQ100, respectively.
  • Reconciliation analysis: Deduplication dominates Loong Papers, consolidation is more common for WikiCeleb100, and conflict resolution dominates financial and legal corpora.
  • Failure analysis: SLIDERS can add unnecessary overhead on short Chinese classification documents under 16K tokens, where distributed-evidence synthesis is limited.
  • Failure analysis: Exact counting and numerical questions remain challenging because they require borderline-case classification, deduplication, value normalization, and aggregation.
  • Workflow integration: SLIDERS complements conventional search and screening by constructing a question-specific evidence table for a smaller corpus, then supporting follow-up questions without reprocessing it.

7 Conclusion

SLIDERS treats systematic-review question answering as construction and reconciliation of a reusable evidence table rather than passage retrieval or direct long-context synthesis. Its results support initial and follow-up analysis, while future work should examine reconciliation correctness and human auditing.

  • SLIDERS extracts structured variables and source-grounded excerpts, then reconciles overlapping evidence, resolves inconsistencies, and preserves provenance.
  • The resulting evidence table supports both original research questions and follow-up analysis across large document collections.
  • Future work should evaluate the semantic correctness of reconciliation decisions and human-in-the-loop workflows for auditing and refining evidence tables.

8 Broad Societal Impact

SLIDERS could reduce the labor and time required for systematic-review-style analyses, but its outputs require expert review because errors and biases may be amplified in high-stakes domains. The paper also describes question decomposition and schema guidance for different document and question types.

  • Potential positive impacts: Automating evidence discovery, extraction, reconciliation, and synthesis could reduce the cost and time required for systematic-review-style analyses.
  • Potential negative impacts: Errors in extraction, reconciliation, or answer synthesis could become apparently authoritative conclusions, especially in medicine, law, finance, and policy.
  • Pipeline design: SLIDERS decomposes the original question into extraction and reconciliation queries so chunk-level processing does not prematurely apply global constraints.
  • Pipeline design: Chunk metadata and document-specific schemas preserve context and align extracted records for question answering or aggregation.

A.4 SLIDERS Implementation Details

SLIDERS uses document-structure metadata and model configurations that combine specialized models for extraction, schema generation, reconciliation, and synthesis.

  • DocLing converts PDFs to markdown so SLIDERS can identify headers, tables, and figures as structural metadata.
  • When source PDFs are unavailable, SLIDERS preprocesses raw text with LLMs to identify tables and section headers.
  • SLIDERS uses GPT-4.1-mini for evidence-variable-aware extraction and GPT-4.1 for variable generation, reconciliation, and final synthesis.A fully open-source variant uses Qwen3.5-122B-A10B for all components.

A.5 Sample Configuration

The sample configuration specifies benchmark inputs, model assignments, extraction behavior, reconciliation settings, and evaluation outputs for SLIDERS.

  • The Loong Finance configuration points to benchmark data and document directories while recording reconciliation statistics in reconciliation_stats.json.
  • The configuration enables schema generation, chunk relevance checks, quote extraction, table merging, and the recon_v2 merge strategy.
  • Reconciliation uses canonicalization, primary-key selection, inspection limits, retry limits, and up to 20 context-generator rows.
  • The benchmark suite spans single-document extraction, mid-scale retrieval and aggregation, and large-corpus multi-hop reasoning.

B.3 Benchmark Construction

WikiCeleb100 and FinQ100 are constructed as follow-up analysis benchmarks that test reuse of evidence tables for related questions over verified document collections.

  • Follow-up questions are derived by reusing and recombining extracted information into temporal cohorts and aggregate financial properties.
  • Gold answers begin with structured extraction but are determined through manual verification against original Wikipedia articles and financial filings.
  • WikiCeleb100 and FinQ100 test whether an evidence table supports downstream questions without reprocessing the full corpus.
  • Removing reconciliation substantially degrades performance, indicating that extraction alone does not solve grouping, deduplication, conflict handling, and table use.
  • SLIDERS currently assumes follow-up questions are answerable from variables induced for the original question.Follow-ups requiring missing variables may fail; the proposed extension is incremental schema expansion and extraction.

B.4 Evaluation Reliability

Evaluation reliability is assessed through repeated LLM judging and human validation, while the included examples illustrate benchmark answers and agent-generated SQL complexity.

  • LLM-as-a-judge evaluation shows standard deviations of 0.47, 0.31, 1.02, 0.21, and 0.38 across five benchmarks.The reported low variance indicates stable evaluation.
  • Human validation across 50 questions per condition yields combined Cohen’s κ = 0.758, indicating substantial agreement.
  • FinanceBench errors for SLIDERS are exclusively false negatives, with 4 FN and 0 FP, making reported scores conservative estimates of true accuracy.
  • The supplied benchmark examples include follow-up answers about artist trends, industry transitions, borrowing concentration, and company debt.
  • The question-answering agent writes non-trivial SQL programs, including joins and calculated depreciation-to-revenue margins.

C.1 Baseline Implementations

The evaluation compares retrieval, full-corpus, extraction, summarization, and recursive-agent baselines, including RLM under matched GPT-4.1 settings. SLIDERS with GPT-4.1 outperforms RLM with GPT-5 across all benchmarks.

  • Baseline families: The baselines comprise retrieval-augmented methods, base language models operating on full corpora, and agentic scaffolding methods.All baselines use the same LLM-as-a-judge evaluation protocol with partial scoring.
  • Retrieval-augmented baselines: Standard RAG retrieves top-k fixed-size 4k-token chunks using Qwen3-4B embeddings before GPT-4.1 generates an answer.LongRAG uses 16k-token chunks to reduce retrievals while retaining more local context per retrieved unit.
  • Retrieval-augmented baselines: GraphRAG extracts entity relations, builds a knowledge graph, summarizes communities bottom-up, and retrieves relevant community summaries at query time.The graph-structured pipeline uses Qwen3-4B as embedder and GPT-4.1 as reader.
  • Full-corpus baseline: Base LMs receive the full corpus in one prompt without retrieval, summarization, or chunking, using GPT-4.1 and Qwen3.5-122B-A10B.Both tested models have 1M-token contexts.
  • Agentic and extraction baselines: DocETL maps question-specific schemas over 16k-token chunks and reduces the extracted fields, while CoA sequentially updates a running summary chunk by chunk.DocETL uses GPT-4.1; CoA uses GPT-4.1-mini workers and a GPT-4.1 manager.
  • Model comparison: SLIDERS with GPT-4.1 still outperforms RLM with GPT-5 across all benchmarks, while RLM performs worse when its main and sub-models are changed to GPT-4.1 and GPT-4.1-mini.RLM recursively chooses between direct sub-document reading and delegation to smaller recursive calls, with a maximum of 30 iterations.

D.2 Cost Analysis

The cost analysis examines per-question expense, latency, and evidence-table reuse across benchmarks. SLIDERS averages $0.79 per question, and reusing its evidence table lowers marginal cost and latency for follow-up questions.

  • Cost drivers: OoLong has the highest per-question cost because of longer input contexts, while Loong Papers has the lowest cost with shorter papers and fewer reconciliation steps.The analysis attributes cost variation to document length, complexity, and the number of reconciliation operations.
  • Cost: SLIDERS averages $0.79 per question across benchmarks and achieves equal or lower cost than RLM.Most cost comes from canonicalizing evidence variables for evidence-table reconciliation.
  • Follow-up efficiency: Reusing the evidence table substantially lowers marginal query cost and latency on FinQ100 follow-up questions.The answer generator takes approximately 25 seconds per question on FinQ100.
  • Error analysis: The relevance-gate estimate rejected 282 of 516 chunks and produced one false negative, for a 0.4% false-negative rate in the sampled evaluation.The authors caution that this estimate comes from a limited sample of pipeline-level failures and may not represent overall gate accuracy.
  • Generality beyond QA: SLIDERS extracted 117 records from three ML papers and aggregated them into coherent per-paper summaries using a five-table schema.The schema included paper summaries, contributions, methods, findings, and conclusions linked by paper title.

E Limitations

SLIDERS has limitations in evidence recoverability, extraction completeness, reconciliation accuracy, latency, evaluation reliability, provenance, and human oversight.

  • Questions whose evidence does not fit typed fields depend on free-text columns, constraining what SLIDERS can recover at answer time.
  • Rare or implicit evidence that the relevance gate underweights could be silently dropped despite its precision benefit.
  • Aggressive canonicalization can collapse genuinely distinct entities, especially with high-cardinality keys or near-duplicate names.
  • 2-3 min end-to-end latency exceeds single-call baselines, limiting suitability for real-time applications.
  • Evaluation retains residual noise from LLM-as-a-judge, and SLIDERS achieves 55% on FinQ100, insufficient for full automation in high-stakes financial analysis.
  • Provenance errors can cause faulty reconciliation decisions, while the current system lacks an interface for editing variables or reconciliation decisions.

G.1 Reconciliation Loop

The reconciliation loop inspects duplicated records, identifies whether conflicts reflect distinct entities or extraction differences, and selects an authoritative row for the target schema. It then verifies that the resulting table preserves the selected row and columns.

  • G.1 Reconciliation Loop: The controller inspects all rows sharing a primary key before routing the case to conflict resolution.The inspection includes metric values, quotes, rationales, and extraction context.
  • G.1 Reconciliation Loop: Conflicting values can represent consolidated company metrics and business-segment breakdowns rather than extraction errors.For 3M FY2022, the eight rows include company-level and segment-level metrics.
  • G.1 Reconciliation Loop: Row_id=1 is selected as the authoritative FY2022 company-level row because it provides explicit quoted metrics from the Total Company table.Its operating_income is 6539, net_sales is 34229, and operating_margin is 19.1%.
  • G.1 Reconciliation Loop: The merge objective is to deduplicate and resolve conflicts for the company-level row, not preserve segment rows as separate events.Rows are retained or excluded according to the schema’s intended entity level.
  • G.1 Reconciliation Loop: Verification confirms that the final table contains exactly row_id=1 with all columns preserved and matching expected values.The verification phase approves the result after one inspection.
  • G.2 Primary Key Selection: Primary-key selection uses voting rounds to identify [’entity_name’, ’fiscal_year_end’, ’item_name’] for cash-flow statement items.The selected key received 3/3 votes and defines unique observations by entity, fiscal year, and line item.
  • G.3 Field Canonicalization: Canonicalization checks related evidence fields to distinguish alternate forms from identifiers or free text before merging.The artist_name_quote field has two distinct values matching the artist name.
Loading 2604.22294v2…