Source-linked AI summary
Local-to-Global Sentence-Level Graph Reranking for Scientific Synthesis
Zheng Dou, Zhao Zhang, Hao Geng, Ningjing Wang, Deqing Wang
TL;DR
Scientific synthesis needs reranking that exposes fine-grained evidence and captures relations among candidate passages, because passage-level independent scoring can limit grounding and coverage. LoG-Reranker combines role-aware local sentence scoring with global sentence-graph refinement to construct structured generator contexts. It consistently improves reranking and synthesis quality across the reported benchmarks, with gains in grounding, coverage, organization, and answer quality.
Problem
Passage-level rerankers bury key scientific details, while independent candidate scoring overlooks complementary and contextual relations needed for grounded, comprehensive synthesis.
Method
LoG-Reranker uses role-aware local sentence scoring and a global sentence graph to refine rankings and organize structured context for generation.
Results
LoG-Reranker consistently outperforms competitive rerankers, achieving the best result in 14 of 15 reranking task-metric comparisons and improving all seven synthesis benchmarks.
Takeaways & Limitations
Sentence-level relevance combined with relational structure provides more informative context for grounded and comprehensive scientific synthesis.
Takeaways & Limitations
Training the graph encoder and global scorer relies on predicted roles and local scores rather than role labels in supervision data, requiring train-inference consistency.
Abstract
from arXiv · showhide
Retrieval-augmented scientific synthesis aims to answer complex research questions by integrating information from multiple papers into comprehensive and well-grounded responses. Since the generator can only synthesize the information selected and organized by the reranker, the quality of the generated synthesis depends critically on the reranked results. However, most rerankers operate at the passage level, which leaves key methodological, empirical, and comparative information buried in long and flat contexts, weakening the grounding of generated claims. Moreover, existing rerankers mainly rely on independent query-candidate scoring which overlooks complementary, contextual, and contrasting relations across scientific candidates, limiting information coverage and the comprehensiveness of the resulting synthesis. To address these limitations, we propose LoG-Reranker, a local-to-global sentence-level graph reranking framework for scientific synthesis. LoG-Reranker performs role-aware local scoring to identify fine-grained, query-relevant sentences and then models their relations on a sentence graph across the candidate set to globally refine sentence rankings. Top-ranked sentences and their connected neighbors are organized into a structured input context for generator to produce more grounded and comprehensive synthesis.Extensive experiments on scientific synthesis and reranking benchmarks show that LoG-Reranker consistently outperforms competitive rerankers, yielding more reliable rankings and improving the quality of generated synthesis.
1 Introduction
Scientific synthesis requires reranking that surfaces sentence-level evidence and models relations among candidates, rather than independently selecting coarse passages. LoG-Reranker addresses these needs through role-aware local scoring and global sentence-graph refinement to provide structured context for generation.
- Motivation: Scientific synthesis integrates multiple papers into long-form, comprehensive, well-grounded answers to complex research questions.RAG grounds responses in external papers and passages, while synthesis emphasizes integration across sources.
- Limitations of Existing Rerankers: Passage-level reranking can bury methodological, empirical, and comparative details in long, flat contexts, weakening claim grounding.The limitation motivates selecting finer-grained information units, such as sentences, before generation.
- Limitations of Existing Rerankers: Independent query-candidate scoring overlooks complementary, contextual, and contrasting relations, limiting coherence and coverage across relevant research aspects.Existing graph approaches still miss fine-grained relations among scientific statements, motivating relational modeling at the sentence level.
- LoG-Reranker: LoG-Reranker combines role-aware local sentence scoring with a sentence graph that globally refines rankings using intra- and cross-passage relations.The framework replaces coarse, independently ranked passages with fine-grained selection and relational modeling across the candidate set.
- LoG-Reranker: LoG-Reranker supplies structured sentence-level context to the generator and improves reranking, citation grounding, information coverage, and answer quality.The proposed context refinement is designed to address missing details and limited coverage in scientific synthesis.
2 Related Work
Related work progresses from query-candidate relevance scoring toward finer-grained information refinement and relational reranking. Existing approaches use compression, sentence selection, contextual dependencies, evidential support, corpus connectivity, semantic proximity, and knowledge graphs.
- Neural and LLM-Based Reranking: Neural rerankers evolved from pretrained cross-encoders to generative, ranking-specific, and listwise methods incorporating inter-candidate comparisons.Listwise approaches use multi-candidate fusion or LLM-based permutation generation.
- Fine-Grained Reranking and Information Refinement: Fine-grained refinement methods compress retrieved information or select and reconstruct relevant sentences instead of treating passages as indivisible units.Later methods preserve contextual dependencies and prioritize evidential support during refinement.
- Graph-Based Reranking: Graph-based rerankers model candidate relations through corpus connectivity, semantic proximity, score propagation, and explicit entity-relation knowledge.These approaches support adaptive candidate expansion and relational enrichment beyond isolated query-candidate scoring.
3 Method
LoG-Reranker reranks retrieved scientific content from sentences to a globally connected graph, combining query-relevant local scoring with cross-candidate relational refinement. It then expands top-ranked sentences with connected context for generation.
- Problem Formulation: LoG-Reranker decomposes retrieved passages into sentence candidates and learns scores over the unified sentence set for downstream ranking.The formulation assigns each sentence a relevance score conditioned on the query and the candidate sentence set.
- Local Sentence Scoring: Local scoring models contextual query–sentence semantics together with compatibility between query intent and sentence roles.The query and passage sentences are jointly encoded, while soft intent and role distributions support role-aware relevance estimation.
- Global Graph Reranking: The global reranker constructs a typed sentence graph with intra-passage adjacency and constrained cross-passage similarity edges.Cross-passage links require compatible predicted roles, semantic similarity, and local relevance, while adjacency edges preserve sentence order and role transitions.
- Global Graph Reranking: A relation-aware graph attention encoder propagates information across typed edges, and global scores combine local relevance with candidate-set relations.The final ranking sorts valid sentence nodes by their global scores after graph encoding.
- Structured Context Construction: Top-ranked sentences are expanded with in-passage and cross-passage neighbors into ordered sentence units for the generation context.This provides the generator with fine-grained source information together with local context and cross-passage connections.
- Training Objectives: Training proceeds in two stages: local sentence ranking and role supervision are optimized first, then the frozen local component initializes global graph reranking.The local ranking objective uses supervision ranks, while an auxiliary objective supervises sentence role distributions.
4 Experiments
Experiments evaluate LoG-Reranker on scientific reranking and synthesis benchmarks, showing consistent gains from sentence-level relevance, relational modeling, and structured context construction.
- Setup: ScholarQABench covers seven synthesis tasks with task-specific generation metrics, while reranking is evaluated using Recall, MAP, and nDCG over top-10 units.The reranking benchmark adds gold or curated pseudo-gold evidence for five labeled tasks; Bio and Neuro are generation-only.
- Reranking Results: LoG-Reranker achieves the best result in 14 of 15 reranking task-metric comparisons, including one tie.It improves MAP over the strongest baseline by 1.36 points on SciFact and 1.31 points on Scholar-Multi.
- Generation Results: LoG-Reranker obtains the best task-specific generation score and citation F1 on all seven benchmarks.All methods use the same OS-8B generator and top-10 input setting; on QASA, ROUGE-L improves by 0.90 points and citation F1 by 1.52 points over corresponding strongest baselines.
- Quality Analysis: LoG-Reranker achieves the highest score in all 12 task-dimension comparisons across coverage, relevance, and organization.Relevance improves on every task, while the largest margins appear in coverage on Scholar-CS and Scholar-Bio and organization on Scholar-Bio and Scholar-Neuro.
- Ablation Study: Removing either local or global modeling reduces all three rubric scores across every task, with removing local modeling decreasing average coverage by 0.360 points.Removing global modeling decreases average organization by 0.344 points, compared with 0.230 for relevance.
- Ablation Study: Removing structured context lowers average organization and coverage by 0.203 and 0.184 points, respectively, while relevance decreases by 0.105.The complete model expands top-ranked sentences with adjacency and cross-passage neighbors, unlike the flat-list ablation.
5 Conclusion
LoG-Reranker integrates fine-grained semantic relevance with sentence-level relational structure for scientific synthesis. Experiments and ablations show consistent improvements and complementary component contributions.
- Conclusion: LoG-Reranker jointly models local query-sentence interactions and global sentence-level graph relations to provide structured generator context.The framework integrates fine-grained semantic relevance with relational structure.
- Conclusion: Experiments show consistent improvements over competitive reranking baselines, while ablations validate the complementary contributions of the proposed components.
A Benchmark and Baseline Details
ScholarQABench spans single-paper and multi-paper scientific QA and synthesis tasks, while ScholarQABench-Rerank adapts five tasks for reranking evaluation. The benchmark compares passage and graph-based baselines using ranking, answer, rubric, and citation metrics.
- Baselines: Compared baselines include cross-encoder, listwise LLM, feature-based, compression, dependency-preserving, sentence-graph, and entity-graph rerankers.Examples include BGE-Reranker, GTE-Reranker, OS-Reranker, RankZephyr, CoRank, RECOMP, EXIT, ChainRAG, and SentGraph.
- Benchmark Composition: ScholarQABench combines three single-paper tasks and four expert-curated multi-paper tasks across classification, short-form QA, and open-ended synthesis.The tasks cover four scientific disciplines and include SciFact, PubMedQA, QASA, ScholarQA-CS, ScholarQA-Bio, ScholarQA-Neuro, and ScholarQA-Multi.
- Evaluation Metrics: ScholarQABench evaluates task-specific answer quality and citation quality using accuracy, ROUGE-L, rubric-based scores, citation precision, and citation recall.Accuracy applies to SciFact and PubMedQA; ROUGE-L applies to QASA; rubric and Prometheus scores support synthesis evaluation.
- Reranking Benchmark: ScholarQABench-Rerank retrieves 50 passages per query and supplements the pool with missing gold or curated pseudo-gold evidence for five labeled tasks.Bio and Neuro are retained for generation evaluation but excluded from reranking because they lack gold evidence or reference contexts.
- Reranking Metrics: Reranking quality is measured with Recall@k, MAP@k, and nDCG@k, with metrics macro-averaged across queries and evaluated at k = 10.Recall measures recovered positives, MAP averages average precision, and nDCG discounts ranking positions logarithmically.
B Training Data Construction
The training-data pipeline parses scientific QA instances, obtains passage- and sentence-level ranking supervision, enriches sentences with semantic roles, and validates the resulting annotations. Manual checks found faithful segmentation and generally reliable ranking judgments.
- Data Extraction: The pipeline extracts 25,915 QA instances from Open Scholar Training Data, including questions, answers, candidate passages, and reference metadata.The retained data include single_paper_qa and multipaper_qa examples.
- Relevance Annotation: Qwen3-Max provides passage-level rankings and sentence-level rankings after decomposing each candidate passage into sentence-like units.Passage rankings reflect contribution to answering the question and supporting the complete answer; sentence rankings provide finer-grained supervision.
- Ranking Consolidation: The pipeline normalizes annotated passage and sentence ranks into strict consecutive orderings and removes fragments or units without meaningful content.Tie-breaking uses original position after annotated rank and relevance information.
- Role Enrichment: UniFAR assigns each retained sentence a soft distribution over background, method, and result roles, producing role-enriched training records.Long passages are windowed before annotation so sentences receive role distributions without truncation.
- Pipeline Statistics: The construction yields 257,546 question-passage pairs and 2,331,189 sentence units before filtering, while automatic validation checks complete ranks and valid role distributions.The pipeline collects 25,915 QA instances, including 14,098 single-paper and 11,817 multi-paper examples.
- Quality Examination: Manual examination accepted all 2,488 inspected sentence units, 29 of 30 sentence-ranking judgments, and 14 of 15 multi-paper passage-ranking judgments.The lower passage-ranking acceptance rate reflects the greater difficulty of prioritizing information across multiple papers.
C Implementation and Hyperparameters
LoG-Reranker’s configuration is relatively stable across tested hyperparameters and random seeds. A moderate role-supervision weight balances role learning with ranking, while results show low variance across reranking and generation tasks.
- Hyperparameter Sensitivity: Hyperparameter sensitivity remains relatively stable around the default configuration on ScholarQA-CS and ScholarQA-Multi.The defaults are λrole = 0.5, τsim = 0.75, δ = 0.25, and k = 8.
- Hyperparameter Sensitivity: A moderate λrole provides the best balance between sentence-role supervision and the primary ranking objective, whereas excessively weak or strong supervision reduces ranking quality.Graph-construction thresholds also affect performance when moved away from suitable settings.
- Random-Seed Stability: Across seeds 13, 42, and 2027, reranking standard deviations range from 0.09 to 0.47, while generation standard deviations are at most 0.23 for task-specific metrics and 0.32 for citation F1.The three multi-paper Rub-Scores vary by no more than 0.011.
- Random-Seed Stability: The average results remain close to seed-13 results across datasets, indicating robustness to random initialization and training-run choice.This stability applies to both reranking and generation evaluations.
D Additional Experiments
Additional experiments examine efficiency, graph structure, input length, and component ablations. LoG-Reranker combines lower input length and efficient local inference with richer graph connectivity, while ablations identify contextual, relational, and adjacency signals as important.
- Efficiency: On ScholarQA-CS with 50 candidates per query, LoG-Reranker’s 2.426-second latency is approximately 2.6× faster than ChainRAG and 40.2× faster than SentGraph.Its pipeline uses 152.52M locally deployed parameters and avoids remote API latency.
- Graph Complexity: LoG-Reranker retains 530.6 sentence nodes on average, an 8.1% reduction versus ChainRAG and SentGraph, while forming 3387.3 relation pairs, 62.4% more than ChainRAG.The graph is more compact in nodes but richer in connections.
- Input Length: LoG-Reranker uses fewer input tokens than every passage-level baseline while achieving the best generation and citation results on ScholarQA-CS and ScholarQA-Multi.On ScholarQA-CS, it obtains Rub-Acc 62.44 and citation score 39.42 with 3173 input tokens.
- Input Length: Aggressive sentence compression reduces input length but loses supporting evidence, whereas LoG enriches selected sentences with targeted local and cross-passage evidence.LoG produces longer inputs than aggressive compressors but remains more compact than passage-level reranking.
- Local Sentence Modeling: Removing contextual encoding reduces Rub-Acc by 3.07 points on ScholarQA-CS and Rub-Score by 0.153 on ScholarQA-Multi.Removing role or intent heads also reduces ranking and generation quality, with the larger decline from removing the role head.
- Graph Structure and Relations: Removing cross-passage edges decreases nDCG@10 by 2.81 and 2.64 points and produces the lowest ScholarQA-Multi Rub-Score of 3.711.In-passage edges, edge constraints, and relation types also contribute to graph quality.
- Structured Context Expansion: Removing adjacency or similarity expansion leaves sentence rankings unchanged but lowers generation scores, with adjacency expansion providing the larger benefit.Citation F1 changes only marginally, suggesting the expansion mainly affects synthesis completeness and organization.