PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Turn Agentic Scientific Literature Search via Workflow Induction

Jisen Li, Bingxuan Li, Nanyi Jiang, Xuying Ning, Xiyao Wang, Yifan Shen, Heng Wang, Yuqing Jian, Xiaoxia Wu, Ben Athiwaratkun, Pan Lu, Jiaxuan You, Bingxin Zhao

arXiv:2607.00597v2cs.CLcs.IR

TL;DR

Scientific literature search must accommodate evolving user intent and fine-grained relevance preferences beyond static query matching. PaperPilot induces editable search workflows and improves multi-turn retrieval, raising Hit@5 from 58.0 to 77.0 over a base toolset agent.

  • Problem

    Scientific literature search needs multi-turn interaction because relevance depends on evolving intent and fine-grained preferences such as citation direction, recency, and methodology.

  • Method

    PaperPilot constructs executable DAG workflows from typed search operators and iteratively edits them using user feedback, trained with workflow imitation and preference optimization.

  • Results

    Hit@5 increased from 58.0 to 77.0 over the base Qwen3.5-9B toolset agent, while workflow execution errors fell from 9.5% to 0%.

  • Takeaways & Limitations

    Explicit, editable search workflows provide an effective interface for aligning literature search agents with complex scientific intent.

  • Takeaways & Limitations

    The benchmark focuses primarily on computer science literature and uses simulated user feedback that cannot fully replace human interaction.

Abstract

from arXiv · show

Scientific literature search often requires more than retrieving papers from a single query: users' intents are underspecified, preference-dependent, and evolve through interaction. Existing search agents typically rely on fixed pipelines or implicit language-only reasoning, making their search strategies difficult to control, inspect, and refine. We introduce PaperPilot, a multi-turn literature search agent that frames scientific search as workflow induction. Given an anchor paper and a user query, PaperPilot constructs an executable DAG of paper-search operators, including keyword search, citation expansion, filtering, scoring, reranking, and evidence extraction. User feedback is then used to refine both the query and the workflow itself. We train PaperPilot with supervised workflow imitation and preference optimization over controlled workflow corruptions. Experiments show that PaperPilot-9B improves over the base Qwen3.5-9B toolset agent under multi-turn interaction, increasing Hit@5 from 58.0 to 77.0, MRR from 47.5 to 59.4, and nDCG@10 from 26.8 to 32.5, while reducing workflow execution errors from 9.5% to 0%. These results show that explicit, editable search workflows provide an effective and controllable interface for aligning literature search agents with complex scientific intent.

1 Introduction

Scientific literature search requires adaptive retrieval and interaction because user intent is often underspecified and depends on fine-grained preferences. PaperPilot addresses this by inducing editable, executable search workflows that user feedback can refine, improving multi-turn retrieval quality and workflow stability.

  • Introduction: Literature search requires identifying relevant work, exploring citation and semantic neighborhoods, comparing candidates, filtering distractors, and explaining relevance to research intent.As scientific output grows, search increasingly depends on adaptive retrieval and interaction rather than static query matching.
  • Introduction: Multi-turn interaction clarifies underspecified intent and refines retrieval strategies around citation direction, methodological similarity, recency, benchmark usage, and application domain.Different interpretations of requests such as finding follow-up work can require direct citations, recent extensions, domain-specific papers, or methodological follow-ups.
  • Introduction: PaperPilot constructs an executable DAG of typed paper-search operators, converting feedback into workflow edits instead of treating it only as additional query text.Its operators include keyword search, citation expansion, filtering, and scoring, with workflows adapted to requests such as emerging follow-up work or strong baselines.
  • Introduction: PAPERPILOT-9B is trained with workflow imitation and preference optimization, using teacher trajectories and corrupted successful workflows across five search directions.The directions are predecessor, successor, sibling, benchmark, and survey; the reported benchmark results show improved retrieval quality, workflow stability, and cost-efficiency.
  • Introduction: Hit@5 increased from 58.0 to 77.0, MRR from 47.5 to 59.4, and nDCG@10 from 26.8 to 32.5 over the base Qwen3.5-9B toolset agent under multi-turn interaction.Workflow execution errors decreased from 9.5% to 0%, while evaluation used a reproducible hold-out benchmark with fixed anchors, queries, directions, gold-paper sets, and interaction protocols.

2 PAPERPILOT

PaperPilot formulates multi-turn scientific literature search as adaptive workflow induction, constructing and refining executable DAGs of paper-search operators to match evolving user intent. It is trained through workflow imitation and preference optimization over corrupted workflows.

  • Formulation: PaperPilot treats literature search as an interactive workflow-construction problem conditioned on an anchor paper, user query, candidate papers, and interaction history.The agent may ask clarification questions or apply operators to expand, filter, score, rerank, and explain candidates.
  • Workflow induction: The agent induces executable DAGs by selecting and parameterizing typed operators whose intermediate outputs flow between nodes.The toolset covers sourcing, set operations, filtering, scoring, reranking, keyword generation, evidence extraction, and workflow construction.
  • Workflow induction: Workflow adaptation lets PaperPilot select operators and parameters according to search intent, such as emphasizing citation expansion for strong baselines or recent keyword search for follow-up work.Workflows must remain type-consistent, and their final outputs can be ranked papers, evidenced papers, or structured graphs.
  • Interaction-driven refinement: User feedback drives iterative workflow editing, allowing PaperPilot to update structure, parameters, and candidate sets across turns.Intermediate outputs are cached for reuse in later turns, and the agent can produce ranked lists, comparisons, or clarification responses.
  • Training: Training combines supervised workflow imitation with preference optimization over corrupted workflows to promote valid DAGs and intent-aligned search strategies.The data include 2,723 anchor-query cases, 5,540 workflow supervision examples, and 1,733 hard chosen–rejected workflow pairs.

3 Experiment and Results

PaperPilot is evaluated on a hold-out benchmark spanning five scientific search directions under single-turn and multi-turn protocols. Results show that adaptive workflow induction, concrete workflow edits, and workflow-induction training improve retrieval quality, stability, and cost efficiency.

  • Evaluation setup: The hold-out benchmark covers predecessor, successor, sibling, benchmark, and survey search directions under single-turn and multi-turn protocols.Single-turn retrieval uses the original query; multi-turn retrieval incorporates clarification questions and simulated user feedback.
  • Interaction behavior: Early turns gather missing intent and constraints, whereas later turns emphasize workflow refinement and finalization.The observed behavior separates clarification, workflow editing, and retrieval finalization into distinct interaction stages.
  • Retrieval results: 84.0 Hit@5, 87.0 Hit@10, 89.5 Hit@15, 56.8 Recall@50, 71.8 MRR, and 41.6 nDCG@10 are achieved by GPT-5.4 with Web Search and PAPERPILOT-TOOLSET in multi-turn retrieval.This is the best overall result reported, while GPT-5.4 with Web Search improves from 72.5 to 79.0 Hit@5 and from 60.2 to 65.3 MRR under multi-turn interaction.
  • Workflow stability: 2.0% to 9.5% workflow execution errors occur for Qwen3.5-9B with PAPERPILOT-TOOLSET under multi-turn interaction.The passage identifies workflow editing as a challenge for smaller untrained models.
  • Trained model: 77.0 Hit@5, 83.5 Hit@10, 89.5 Hit@15, 40.0 Recall@50, 59.4 MRR, and 32.5 nDCG@10 are achieved by PAPERPILOT-9B in the multi-turn setting.Compared with the base Qwen3.5-9B PAPERPILOT-TOOLSET agent, PAPERPILOT-9B improves Hit@5 from 58.0 to 77.0, Recall@50 from 34.8 to 40.0, MRR from 47.5 to 59.4, and nDCG@10 from 26.8 to 32.5, while reducing workflow execution errors from 9.5% to 0%.
  • Cost efficiency: 0.1508 dollars per case is achieved by GPT-5.4 with Web Search and PAPERPILOT-TOOLSET, versus 6.0903 dollars per case for OpenAI DeepResearch.OpenAI DeepResearch achieves 72.0 Hit@5 and 53.0 MRR, while PAPERPILOT-9B remains two orders of magnitude cheaper than DeepResearch.

4 Discussion

PaperPilot-9B improves workflow generation and refinement while offering stronger cost-effectiveness than GPT-5.4. Retrieval quality generally peaks with smaller candidate pools, and the human study reports substantially higher satisfaction than baselines.

  • Workflow-level evaluation: PaperPilot-9B achieves stronger cost-effectiveness than GPT-5.4 for workflow generation, reaching relative efficiency 3.26 at $0.00108 per case.GPT-5.4 attains the highest absolute similarity, with 0.7010 whole-workflow cosine similarity and 0.4960 signature similarity, but at substantially higher cost.
  • Workflow-level evaluation: 0.3149 and 0.1957 are PaperPilot-9B’s reported improvements over Qwen3.5-9B in whole-workflow and signature similarity, respectively.The base-model comparisons are reported as increases from 0.0470 to 0.3619 and from 0.0281 to 0.2238.
  • Workflow-level evaluation: 0.9469 final workflow cosine similarity and 0.8777 query-level cosine similarity are PaperPilot-9B’s strongest workflow-editing results.PaperPilot-9B also improves over Qwen3.5-9B across add-node, modify-node, and remove-node refinements, with the largest gain under remove-node editing.
  • Candidate-pool sensitivity: Recall@50 drops from 0.411 at K = 8 to 0.374 at K = 20, while Hit@5 drops from 0.705 at K = 10 to 0.660 at K = 20.Most metrics peak or remain near their peak at K = 8–10; larger pools generally introduce distractors that weaken downstream filtering and reranking.
  • Human study: 74.7% success rate, 2.4 mean top-1 distance, and 4.2 question satisfaction score are reported for PaperPilot in the human study.GPT-5.4 records 32.0% success rate and 2.4 satisfaction score, while OpenAI-DeepResearch records 8.0% success rate; lower Top-1 Distance and NTS are preferred.

5 Conclusion

PaperPilot frames multi-turn scientific literature search as workflow induction, constructing executable, editable workflows that can be refined through user feedback. Trained with workflow imitation and preference optimization, PaperPilot-9B learns to generate and edit valid workflows for evolving user intent.

  • PaperPilot formulates scientific literature search as workflow induction rather than relying on fixed pipelines or implicit language-only reasoning.
  • PaperPilot constructs executable DAG-structured workflows over typed paper-search operators and refines them through user feedback.
  • Supervised workflow imitation and preference optimization over corrupted workflows train PaperPilot-9B to generate and edit valid search workflows.
  • Explicit, editable workflows provide a practical interface for aligning scientific literature search agents with evolving user intent.

Limitations

PaperPilot is limited by its predefined operator library, teacher-generated supervision, and evaluation centered on computer science literature with controlled hidden-gold testing.

  • PaperPilot’s predefined symbolic operator library may omit domain-specific search behaviors and specialized research workflows.
  • Teacher-model trajectories used for workflow supervision may inherit biases or blind spots from the teacher system.
  • The benchmark focuses primarily on computer science literature and uses controlled hidden-gold evaluation for reproducible comparison.

Use of LLMs · A Appendix · A.1 Toolset

The paper uses LLMs for research support rather than substantive content, while PaperPilot’s operator library supports multiple stages of paper search. GPT-5 assists only with language refinement, and the library includes sourcing, filtering, scoring, reranking, keyword generation, and graph construction.

  • Use of LLMs: LLMs are used strictly for research support rather than as sources of substantive content.
  • Use of LLMs: The LLM use cases are testing and training the model and assisting with language refinement during paper writing.
  • Use of LLMs: GPT-5 was used solely to polish text by improving coherence and grammar.
  • Use of LLMs: The authors originated all ideas, logic, results, and technical contributions.
  • A.1 Toolset: PAPERPILOT uses an operator library covering sourcing, filtering, scoring, reranking, keyword generation, and graph construction.
  • A Appendix: The library includes LLM-based keyword generation operators.

A.2 Full Task Formulation

Multi-turn agentic paper search is formulated as an interactive decision-making problem initialized by an anchor paper and user query. The agent iteratively operates on candidate papers using feedback to produce a ranked list aligned with the user’s information need.

  • Task setup: The task starts from an anchor paper p0 and query q, then iteratively expands, filters, and ranks candidates to satisfy the user’s search intent.The query may request follow-up work, strong baselines, method comparisons, or adjacent research directions.
  • Interactive state: At each turn, the agent observes a state containing the query, anchor paper, current candidate set, and interaction history before selecting an action.The policy selects actions according to π(· | st).
  • Search operators: The action space provides operators for retrieval, citation expansion, candidate-set combination, constraint filtering, scoring, reranking, and user clarification.Feedback includes relevance judgments, preference updates, clarification responses, and refinement instructions.
  • Objective: The agent outputs a ranked paper list after a finite horizon, optimizing expected cumulative utility for relevance, feedback alignment, intent coverage, and interaction efficiency.The objective also minimizes unnecessary search operations and user interactions.

B Related Work … C.3 Inference Procedure

The paper positions PaperPilot against literature-search and workflow-induction agents, emphasizing editable paper-search workflows for multi-turn intent alignment. Its evaluation uses structured search cases, simulated clarification feedback with leakage controls, and single- and multi-turn inference procedures.

  • B Related Work: Existing literature-search agents support retrieval, synthesis, structured extraction, idea generation, and scientific discovery, but typically use fixed pipelines or free-form reasoning that users cannot inspect or steer at individual operations.Representative systems include LitLLM, PaperQA2, AutoSurvey, STORM, ResearchAgent, the AI Scientist, ChatCite, PaSa, SPAR, and Elicit.
  • B Related Work: LOT differs from prior workflow-induction systems by editing workflows conditioned on evolving dialogue and using paper-search primitives such as citation expansion, evidence extraction, and NLI filtering.The comparison distinguishes paper-search operators from general code or text-transformation modules.
  • B Related Work: PAPERPILOT targets interactive retrieval by translating user clarifications into controllable search procedures rather than treating retrieval as a black box.The related-work comparison covers conversational clarification, general-purpose deep research, and five capabilities: symbolic workflow induction, workflow refinement, multi-turn dialogue, citation-graph expansion, and evidence grounding.
  • B Related Work: Under strict capability definitions, PAPERPILOT is the only compared system supporting all five listed capabilities.The table uses △ for partial or indirect support and − for capabilities outside a system’s design scope.
  • C.1 Dataset Details: Each benchmark case specifies an anchor paper, query, search direction, gold papers, and n_gold across predecessor, successor, sibling, benchmark, and survey tasks.Cases contain 6–15 gold papers, whose labels combine citation-graph information, human filtering, LLM-assisted synthesis, and intra-related-work cohorts.
  • C.2 User Simulation and Leakage Control: The multi-turn evaluator uses a fixed Qwen3.5-397B-A17B user simulator conditioned on anchor metadata, query, direction, hidden gold metadata, history, and clarification options.The simulator selects an option or produces a short free-form response, while the retrieval agent does not receive the hidden gold papers.
  • C.2 User Simulation and Leakage Control: Leakage control combines prompt restrictions, deterministic title-and-author substring matching, and Qwen3.5-9B judgment of indirect disclosure.The supplied description states that the simulator is instructed not to mention identifying gold-paper information and that responses pass through this pipeline.
  • C.3 Inference Procedure: In single-turn inference, the agent retrieves anchor metadata, runs one retrieval pass, returns the top-50 papers, and is evaluated against the hidden target list; multi-turn inference adds clarification question-answer pairs before retrieval.Fixed-workflow systems use deterministic sourcing, scoring, and reranking, whereas toolset systems generate and execute a symbolic workflow DAG; clarification responses undergo leakage control before entering interaction history.

C.4 Baseline Details · C.5 Evaluation Metrics · C.6 Training Details

The evaluation compares general-purpose, commercial, fixed-workflow, toolset, and trained workflow-induction systems, using retrieval, workflow-generation, and cost-effectiveness metrics. Training uses supervised fine-tuning followed by IPO-style preference optimization with specified LoRA and optimization settings.

  • C.4 Baseline Details: The baselines span web-search agents, commercial deep-research systems, fixed workflows, symbolic toolsets, and trained workflow induction.The comparison covers eight major configurations across single-turn and multi-turn settings.
  • C.4 Baseline Details: DeepResearch is evaluated one-shot because its API supports neither multi-turn requests nor controllable workflow refinement.It receives the same anchor-paper metadata and user query and returns a ranked paper list.
  • C.4 Baseline Details: The fixed PAPERPILOT-WORKFLOW baseline uses deterministic sourcing, scoring, and reranking, isolating tool-use benefits from adaptive workflow induction.PAPERPILOT-TOOLSET instead lets Qwen3.5-9B, Qwen3.5-397B, and GPT-5.4 generate, configure, and execute DAG workflows without workflow-induction training.
  • C.4 Baseline Details: PAPERPILOT-9B is trained to generate and refine DAG workflows using the anchor paper, query, interaction history, and simulated feedback.This tests specialized workflow-induction training beyond prompting a large model with tools.
  • C.5 Evaluation Metrics: Evaluation averages metrics over hold-out cases, analyzes predecessor, successor, sibling, benchmark, and survey directions, and assigns failed or errored cases zero under strict first-pass scoring.Reported retrieval metrics include Hit@5, Hit@10, Hit@15, Recall@50, MRR, nDCG@10, and nDCG@15.
  • C.5 Evaluation Metrics: Workflow-generation evaluation measures TF-IDF cosine similarity for workflow components and cosine, Jaccard, and final similarity changes under add-node, modify-node, and remove-node edits.These metrics assess executable DAG generation and local refinement rather than only final paper retrieval.
  • C.5 Evaluation Metrics: Cost-effectiveness counts successful retrieval cases as those with Hit@5 = 1 and divides successful cases by total inference cost.Cost uses agent-side token usage and public model and embedding list prices, excluding simulator and leak-checker tokens.
  • C.6 Training Details: Supervised fine-tuning runs for 3 epochs at learning rate 2 × 10^-4 with sequence length 14,336 using LoRA over attention and MLP projections; preference optimization uses β = 0.2, 3 epochs, learning rate 3 × 10^-5, sequence length 16,384, and gradient clipping norm 1.0.The preference stage continues from the SFT checkpoint with an IPO-style DPO objective.

C.7 Workflow-Level Evaluation Metrics · C.8 Human Study Details

The paper evaluates generated workflows against reference DAGs using graph-, text-, and refinement-level metrics, and assesses interactive search through a multi-turn human study with six participants across five sessions. The workflow metrics capture structural, parameter-level, and edit alignment, while the human study collects outcome-based and user-reported measures.

  • C.7 Workflow-Level Evaluation Metrics: Workflow generation is evaluated by comparing each generated DAG with a reference workflow using graph-level and text-level metrics.These metrics capture structural correctness and parameter-level alignment.
  • C.7 Workflow-Level Evaluation Metrics: Graph-level evaluation uses node-identifier and operator-type Jaccard similarities to measure structural and operational overlap with the reference.It also reports added, removed, and modified nodes needed to transform the generated workflow into the reference.
  • C.7 Workflow-Level Evaluation Metrics: Text-level evaluation computes TF-IDF cosine similarity for the whole workflow, query-related fields, filter predicates, and function signatures.These measures assess overall, search-formulation, constraint-level, and signature alignment.
  • C.7 Workflow-Level Evaluation Metrics: Refinement evaluation tests whether each edit moves the current workflow closer to the reference next-state workflow.It reports similarity changes after editing and final full-workflow similarity.
  • C.7 Workflow-Level Evaluation Metrics: Positive similarity changes indicate improved alignment with the reference workflow, alongside final full-workflow cosine similarity after refinement.The final similarity is measured after all refinement steps.
  • C.8 Human Study Details: The human study evaluates how well each system supports interactive paper search in realistic multi-turn settings.Participants provide both outcome-based and user-reported metrics after each session.
  • C.8 Human Study Details: Six volunteers from Biology, Social Science, and Computer Science, spanning undergraduate, master’s, and PhD levels, participate in the study.The participants represent different majors and academic levels.
  • C.8 Human Study Details: Participants interact with each system over five sessions, with evaluation covering Success Rate and Top-1 Distance among four reported dimensions.Success Rate is the percentage of retrieved papers users consider satisfactory, while lower Top-1 Distance indicates better ranking alignment.

D Sensitivity Analysis Results

The sensitivity analysis shows that enlarging the first-stage candidate pool does not consistently improve final retrieval quality. Larger pools can add distractors and reduce recall-oriented metrics.

  • Search scale sensitivity: Increasing the first-stage candidate pool does not monotonically improve final retrieval quality.The sensitivity analysis evaluates how search scale affects retrieval outcomes.
  • Search scale sensitivity: Larger first-stage candidate pools often introduce more distractors.The added distractors can make the expanded candidate set less useful for final retrieval.
  • Search scale sensitivity: Larger candidate pools can reduce recall-oriented metrics.Thus, increasing search scale may hurt recall rather than improve it.
Loading 2607.00597v2…