PaperlayerView exact revision on arXiv

Source-linked AI summary

SciLENS: RL-Driven Autonomous Agents for Scientific Localized Evidence Navigation and Synthesis

Leqi Zheng, Jinbo Su, Yuying Li, Chaokun Wang, Weiping Wang, Haitao Li, Jiajun Zhang, Shannan Yan, Zhaolu Kang, Rong Fu, Jie Wu, Fang Niu, Hang Zhang

arXiv:2609.03338v1cs.IR

TL;DR

Scientific literature synthesis agents face limits from proprietary online services and linear text-only processing, which complicates fully local, large-scale synthesis. SciLENS combines a local dual-tier scholarly infrastructure with visualization-augmented reasoning and automated rubric-based alignment. Across six scientific benchmarks, it outperforms open-source baselines and approaches frontier proprietary-model performance.

  • Problem

    Scientific literature synthesis is limited by proprietary online dependencies and text-centric representations that can cause context overflow and factual hallucinations during macro-level synthesis.

  • Method

    SciLENS uses a dual-tier local infrastructure, structural visualization within the reasoning loop, citation-subgraph data synthesis, and reverse-decomposition rubric-based reinforcement learning.

  • Results

    Across six scientific benchmarks, SciLENS outperforms open-source baselines and approaches frontier proprietary-model performance, including GPT-5.2-level performance on scientific QA and synthesis tasks.

  • Takeaways & Limitations

    SciLENS provides a fully localized framework for scientific reasoning and structural synthesis with integrated text-and-chart outputs and evidence-grounded planning.

  • Takeaways & Limitations

    SciLENS’s primary limitation is the transition from a localized experimental sandbox to live industrial deployment.

Abstract

from arXiv · show

Scientific literature synthesis agents increasingly rely on proprietary online services, limiting reproducibility, privacy, and offline deployment. To address this challenge, we introduce SciLENS Scientific Localized Evidence Navigation and Synthesis), a fully local autonomous agent framework operating on a dual-tier infrastructure indexing approximately 12 million academic records. SciLENS pioneers the integration of structural visualization as an actionable tool within the reasoning loop, enabling the agent to compress complex citation topologies into validated data-driven charts and thereby mitigate context exhaustion during macro-level synthesis. To train the agent without human annotation, we develop an automated data synthesis pipeline that extracts multi-hop subgraphs from a citation knowledge graph, verified by cross-model consensus among 20 frontier models. The agent is subsequently aligned through a reverse-decomposition rubric strategy that provides fine-grained process rewards for early planning and strict evidence grounding. Evaluations across six scientific benchmarks encompassing standard QA, citation accuracy, factual reasoning, and structural synthesis demonstrate that SciLENS significantly outperforms open-source baselines and achieves performance comparable to GPT-5.2 and Gemini-3.0-pro. Our source code and data are released at https://github.com/LQgdwind/SciLENS.

1 Introduction

SciLENS addresses limitations in scientific literature synthesis by combining structural visualization, automated training-data generation, fine-grained rubric alignment, and fully local infrastructure. It evaluates this framework across six scientific benchmarks, outperforming open-source baselines and approaching frontier proprietary models.

  • Motivation: Macro-level synthesis remains vulnerable to context overflow and factual hallucinations because existing paradigms rely on linear textual outputs.These paradigms also lack autonomous compression of quantitative citation-topology data into structured visual representations.
  • Contributions: A dual-tier local infrastructure indexes approximately 12 million academic records and supports semantic retrieval, citation-graph traversal, and structural visualization offline.The infrastructure combines MongoDB metadata management with distributed FAISS dense retrieval for sub-second access.
  • Contributions: SciLENS integrates structural visualization into the reasoning loop to produce validated, data-driven charts that summarize complex citation topologies and reduce context burden.The framework treats visualization as an actionable reasoning tool rather than only a final presentation format.
  • Contributions: Automated citation-subgraph synthesis generates multi-hop reasoning trajectories without human annotation, while reverse-decomposition rubrics reward early planning and strict evidence grounding.The generated trajectories are verified through cross-model consensus before rubric-based alignment.
  • Evaluation: Across six benchmarks covering reading comprehension, citation accuracy, factual reasoning, and structural synthesis, SciLENS outperforms open-source baselines and approaches frontier proprietary models.The reported evaluation supports the framework’s performance across multiple scientific reasoning dimensions rather than a single task.

2 Agent Architecture and Visualization-Augmented Reasoning

SciLENS combines deterministic local scholarly access with an offline research toolbox and a continuous reasoning loop. Structural visualization converts complex citation topology into validated chart schemas, complementing retrieval and graph traversal for multimodal synthesis.

  • Localized Infrastructure: SciLENS uses a dual-tier local backend to provide deterministic access to approximately 12 million academic records without relying on online web searches.MongoDB supports exact-match, BM25, metadata, and citation queries, while distributed FAISS supports dense retrieval across GPU shards.
  • Visualization-Augmented Reasoning: Autonomous visualization compresses high-dimensional citation topology into chart schemas, transforming the agent into a multimodal synthesizer of integrated text-and-chart reports.The design targets context exhaustion and hallucinations associated with linear text generation during macro-level synthesis.
  • Research Toolbox: The offline research toolbox combines semantic retrieval, topological traversal, structural visualization, and summarization within a continuous thought-action-observation loop.Its tools support dense and keyword search, metadata extraction, multi-hop citation exploration, shortest-path tracing, and summarization.
  • Visualization-Augmented Reasoning: Visualization tools emit validated line, bar, and scatter chart schemas while enforcing dimensional and numeric consistency before rendering.This validation makes structural outputs data-driven rather than unconstrained textual descriptions.

3 Citation-Graph Data Synthesis and Rubric-Based Alignment

SciLENS constructs citation-grounded training data from topological subgraphs and aligns the agent with staged supervision for multi-tool scientific reasoning. The process combines automated verification, distilled tool-use trajectories, and rubric rewards for planning, evidence grounding, and format compliance.

  • 3.1 Topological Subgraph Sampling: The pipeline extracts cohesive citation subgraphs to create locally coherent contexts for complex scientific question-answer synthesis.It uses random walks over the Open Academic Graph and generates 30,000 connected subgraphs.
  • 3.1 Topological Subgraph Sampling: Random walks begin from sampled seed publications, run for thirty steps, and are repeated ten times per subgraph to capture citation topology.
  • 3.2 Logic-Driven QA Synthesis and Verification: Multi-hop question-answer pairs compound factoid questions with relational links, while self-consistency requires composite questions to exceed atomic-fact resolution.
  • 3.2 Logic-Driven QA Synthesis and Verification: Each candidate is verified by four randomly selected models from a pool of twenty frontier language models for factual correctness and answer uniqueness.A candidate is retained only when all sampled verifiers reach absolute consensus.
  • 3.3 Multi-Step Multi-Tool Agent Learning: SciLENS uses two-stage training: teacher-model distillation first develops tool-use behavior, followed by reinforcement learning for logical rigor.DeepSeek-V3.2 generates approximately twenty thousand complete interaction trajectories for distillation.
  • 3.3 Multi-Step Multi-Tool Agent Learning: Reinforcement learning combines verifiable final-answer rewards with semantic rewards for planning, reverse decomposition, and evidence grounding.The reward framework evaluates agentic ReAct trajectories with multidimensional criteria.
  • 3.3 Multi-Step Multi-Tool Agent Learning: The composite reward combines a binary format score with a rubric-based semantic answer score.The format component enforces required schemas, while the semantic component evaluates weighted instance-specific criteria.
  • 3.3 Multi-Step Multi-Tool Agent Learning: Fine-grained alignment prioritizes structured early planning and meticulous evidence grounding over superficial text generation.

4 Benchmarks and Multidimensional Evaluation

SciLENS is evaluated on six scientific benchmarks covering standard question answering, citation accuracy, factual reasoning, and structural synthesis. The evaluation uses locally deployed judging and enforces train-test separation across multiple data levels.

  • Six benchmarks span reading comprehension, citation accuracy, factual reasoning, and structural synthesis.The evaluation includes established datasets and automatically constructed multi-hop instances.
  • Automatically constructed benchmarks contain 500 verified instances each with strict train-test disjointness at subgraph, seed-paper, and instance levels.
  • All evaluations use a locally deployed Qwen3-30B-A3B judge with zero temperature.Full protocols, metric definitions, and instance-level rubrics are provided in Appendix H.

5 Experiment

Under a unified offline toolbox protocol, SciLENS is compared with proprietary and open-source systems across benchmarks and ablations. The results support gains from alignment, retrieval, graph traversal, and visualization tools.

  • 5.1 Experimental Setup: SciLENS is compared with state-of-the-art proprietary models and leading open-source frameworks across diverse architectures and agent types.
  • 5.1 Experimental Setup: All evaluated models use the identical offline RESEARCHTOOLBOX and 12-million-paper database without web search.This controls for infrastructure and tool-access differences; the zero-shot Qwen3-30B-A3B Base + Tools row isolates training effects.
  • 5.2 Main Results: SciLENS-RL achieves the highest correctness on QASA (47.62), SciFact (88.94), and PubMedQA (77.53), with strongest citation accuracy across datasets.These results indicate robust factual grounding on external evaluations.
  • 5.2 Main Results: SciLENS-RL outperforms all open-source baselines and achieves performance comparable to GPT-5.2 and Gemini-3.0-pro.
  • 5.3 Ablation Studies: Removing retrieval and graph tools causes functional collapse across benchmarks, showing that the tasks require multi-hop evidence gathering.
  • 5.3 Ablation Studies: Removing planning rubrics reduces SSB from 0.7607 to 0.6855, indicating a contribution from process-level supervision.
  • 5.3 Ablation Studies: Removing visualization tools reduces SSB from 0.7607 to 0.5215 while standard QA remains comparatively stable.The ablation directly tests visualization for structural synthesis.

6 Related Work

Related systems span biomedical and multidisciplinary language-model applications, rubric-based alignment, and localized retrieval. SciLENS combines these directions with citation-network traversal and reverse-decomposition process supervision.

  • Language models have been applied across biomedical, medical, geoscience, astronomy, and multidisciplinary scientific domains.
  • SciLENS extends rubric-based reinforcement learning with reverse-decomposition rubrics that supervise early planning and evidence grounding.This differs from approaches that score only final outputs.
  • Localized retrieval frameworks reduce online instability but rely on single-turn retrieval that does not exploit citation networks for multi-hop reasoning.

7 Conclusion

SciLENS is a fully localized agent trained with rubric-based reinforcement learning for scientific synthesis. It combines validated structural visualization, reverse-decomposition rewards, and a dual-tier index of 12M academic records while approaching frontier-model performance.

  • SciLENS is a fully localized agent trained with rubric-based reinforcement learning for scientific synthesis.
  • Structural visualization compresses complex citation topology into validated chart schemas within the reasoning process.
  • Reverse-decomposition rubrics provide fine-grained process rewards for planning and evidence grounding.
  • 12M academic records support the dual-tier infrastructure, while SciLENS approaches GPT-5.2-level performance on scientific QA and synthesis tasks.

Limitations

SciLENS demonstrates preliminary capabilities in complex academic reasoning and structural synthesis, but its main limitation is deployment beyond the localized experimental sandbox.

  • SciLENS demonstrates preliminary capabilities in complex academic reasoning and structural synthesis.
  • The primary limitation is transitioning SciLENS from a localized experimental sandbox to live industrial deployment.

Ethical Considerations

SciLENS uses a fully local academic-research infrastructure and toolbox rather than external web interfaces. Its implementation combines structured storage, distributed retrieval, graph traversal, visualization, summarization, and constrained agent execution.

  • Ethical scope: SciLENS operates on the openly licensed Open Academic Graph without human subjects, private data, or dual-use applications.
  • Research toolbox: RESEARCHTOOLBOX groups semantic retrieval, topological traversal, structural visualization, and text compression into a unified tool interface.
  • Graph traversal: Citation traversal supports multi-hop expansion and bounded shortest-path tracing over local citation networks.
  • Structural visualization: Visualization tools validate chart schemas and enforce dimensional and numeric consistency before rendering.
  • Agent execution: A ReAct-style loop alternates reasoning, tool calls, and tool responses under system-prompt constraints designed for grounding and context control.

E Train–Test Data Disjointness Guarantee

SciLENS constructs train, validation, and test data from citation-graph subgraphs while enforcing separation before question synthesis. Multiple checks establish disjointness at subgraph, seed-paper, paper, edge, question, and rubric levels.

  • Subgraph-level partition: 30,000 topological subgraphs are partitioned before question synthesis into 28,000 training, 1,000 validation, and 1,000 test instances.
  • Seed-paper exclusivity: Test subgraph seed papers are entirely disjoint from training seeds, with the constraint verified programmatically.
  • Paper and edge overlap: 0% paper-level overlap and 0% edge-level overlap are confirmed across training and test splits.
  • Question deduplication: Exact-match and high-similarity deduplication remove residual near-duplicate questions across train and test pools.
  • Rubric independence: Training and evaluation rubrics use different generators and independently authored prompts to reduce template-based overfitting.
  • Training objective: IcePop stabilizes reinforcement learning through selective token masking, gradient regularization, and KL-divergence constraints.
  • Process supervision: The reward framework evaluates final correctness, reverse-decomposition planning, and strict evidence grounding rather than relying on one scalar reward.

H.2 Newly Constructed Benchmark Metrics

SciLENS evaluates structural synthesis and scientific reasoning with locally hosted, multidimensional metrics designed to measure correctness, query alignment, and evidence grounding. The evaluation uses deterministic judging and standardized tool interfaces across commercial and open-source baselines.

  • Benchmark evaluation: Qwen3-30B-A3B serves as the exclusive zero-temperature judge for the Structural Synthesis and Scientific Fact and Reasoning benchmarks.This setup is intended to improve reproducibility and reduce evaluator variance.
  • Structural Synthesis Benchmark: Structural Synthesis evaluates chart_type and chart_data using Answer Consistency, Query Alignment, and Factual Grounding dimensions.Factual Grounding penalizes numerical patterns that contradict evidence or introduce unsupported variables.
  • Scientific Fact and Reasoning Benchmark: Scientific Fact and Reasoning decomposes performance into Answer Semantic Fidelity, Query Intent Satisfaction, and Evidence Grounding Quality.These metrics assess semantic agreement, scope adherence, citation use, and claims exceeding the retrieval evidence.
  • Scoring design: Instance-specific metric weights sum to 100% across each synthesis benchmark’s three dimensions.The weighting scheme reflects different cognitive priorities across queries.
  • Comparison setup: Baselines span GPT-5 variants, Claude and Gemini models, Qwen3, academic research systems, and exploratory web agents.All systems interact with the research toolbox through standardized JSON tool schemas adapted to their interfaces.
  • Statistical methodology: Main-table scores are means and standard deviations over five independent runs, with paired Wilcoxon tests and Holm-Bonferroni correction for significance.Randomness comes from agent generation or provider-side sampling, while the judge remains deterministic.

J.1 Performance Analysis Across Benchmarks

SciLENS performs strongly across structural synthesis, factual reasoning, citation accuracy, and long-horizon tool-use evaluations. Ablations and behavioral analyses attribute these results to tool orchestration, context pruning, and task-adaptive retrieval, graph, and visualization use.

  • Benchmark performance: 0.7607 on Structural Synthesis and 0.7594 on Scientific Fact and Reasoning surpass the reported GPT-5.2 comparisons.The Scientific Fact and Reasoning comparison is 0.7594 versus GPT-5.2 at 0.6578.
  • Benchmark performance: Citation F1 reaches 83.72 on SciFact and 76.54 on PubMedQA.The localized infrastructure supports deep topological searches during literature retrieval.
  • Ablation analysis: Disabling retrieval and graph tools causes functional collapse, while the full aligned policy remains comparable to the base model with tools.The ablation indicates that alignment teaches procedural tool orchestration rather than simply adding parametric scientific knowledge.
  • Context management: Keeping the five most recent tool observations consistently outperforms keeping twenty, while unmanaged context performs worst.The reported explanation is reduced semantic noise alongside preservation of the unpruned reasoning trace.
  • Context management: The discard-all strategy has the highest raw metrics but violates strict continuous pass@1 evaluation by restarting trajectories and resampling.SciLENS therefore adopts Keep Recent 5 as its operational configuration.
  • Tool invocation dynamics: Retrieval accounts for 86% of SciFR operations, whereas SSB shifts toward 19% graph traversal and 10% visualization.SSB retrieval falls to 61%, and GetKhop alone contributes 9% of actions.
  • Training dynamics: Across 1.2K training steps, the composite reward rises steeply before converging near 80.The trajectory is presented as evidence of stable policy optimization.
  • Qualitative analysis: Case studies show SciLENS converting proportional, comparative, and continuous-variable scientific reasoning into pie and line charts.Examples include educational sociology, geochemistry, and biochemical glucose-response analysis.

K.3 Evaluation of Complex Synthesis Scenarios

Complex synthesis evaluations test whether SciLENS can filter distractors, extract structured quantities, and produce validated visual summaries. The cases also expose failure boundaries involving attention hijacking, incomplete retrieval, and parametric fallback.

  • Complex synthesis scenarios: SciLENS filters irrelevant heavy-metal risk assessments while synthesizing groundwater distribution information in a distractor-rich scenario.The scenario is designed to test informational-noise filtering.
  • Complex synthesis scenarios: The agent extracts organizational counts from mixed bibliometric descriptions and renders forty percent and twenty percent allocations accurately.Two companies belong to one group, while three other groups contain one company each.
  • Complex synthesis scenarios: The reported scenarios indicate stable and precise reasoning on dense queries containing distractors and layered descriptions.The evaluation includes proportional analysis, performance correlation, and cross-entity comparison cases.
  • Error case analysis: An error case begins with attention hijacking toward Turkish environmental-policy distractors before the agent redirects to a carbon disclosure framework.The agent then fails to retrieve exact evidence and hallucinates an incorrect stage label from parametric memory.
  • Successful trajectory: The localized judge evaluates the final trajectory with multidimensional rubric scores, including Query Intent Satisfaction and Evidence Grounding Quality.The reward culminates in a composite score after granular pass or fail judgments.
  • Successful trajectory: Trajectory examples show semantic retrieval locating a 2007 immunogenetic study before graph traversal checks its statistic against later literature.The agent then synthesizes a validated answer after exhausting the citation graph.
  • Error case analysis: The judge assigns zero Answer Semantic Fidelity to the hallucinated answer, demonstrating the reward system’s grounding penalty.This case links failed evidence retrieval and parametric fallback to an explicitly penalized outcome.
Loading 2609.03338v1…